mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-09-19 16:20:50 +00:00
126 KiB
126 KiB
ClickHouse release v21.11.1.8636-prestable FIXME as compared to v21.10.2.15-stable
Backward Incompatible Change
- Remove setting odbc_max_field_size because it is not used anymore. Closes #30413. #30778 (Kseniia Sumarokova).
- Change order of json_path and json in sql json function (to be correct). Closes #30449. #30474 (Kseniia Sumarokova).
- Remove MergeTree table setting
write_final_mark
. It will be alwaystrue
. #30455 (Kseniia Sumarokova). - Function
bayesAB
is removed. This closes #26233. #29934 (alexey-milovidov). - Now ClickHouse Keeper snapshots compressed with
ZSTD
codec by default instead of custom ClickHouse LZ4 block compression. This behavior can be turned off withcompress_snapshots_with_zstd_format
coordination setting (must be equal on all quorum replicas). Backward incompatibility is quite rare and may happen only when new node will send snapshot (happens in case of recovery) to the old node which is unable to read snapshots in ZSTD format. #29417 (alesapin).
New Feature
- Support
ALTER TABLE
for tables inMemory
databases. #30866 (tavplubix). - New function
mapExtractKeyLike
to get the map only kept elements matched specified pattern. #30793 (凌涛). - CompiledExpressionCache limit elements size using
compiled_expression_cache_elements_size
setting. #30667 (Maksim Kita). - Add ability to change nodes configuration (in
.xml
file) for ClickHouse Keeper. #30372 (alesapin). - Added
HashedArray
dictionary type. Closes #30236. #30242 (Maksim Kita). - Added new JSONExtractKeys function ... #30056 (Vitaly).
- Added function
tokens
. That allow to split string into tokens using non-alpha numeric ASCII characters as separators. #29981 (Maksim Kita). - Add support for FreeBSD aarch64. #29952 (MikaelUrankar).
- User can now create dictionaries with comments:
CREATE DICTIONARY ... COMMENT 'vaue'
... #29899 (Vasily Nemkov). - Added columns
data_compressed_bytes
,data_uncompressed_bytes
,marks_bytes
intosystem.data_skipping_indices
. Added columnssecondary_indices_compressed_bytes
,secondary_indices_uncompressed_bytes
,secondary_indices_marks_bytes
intosystem.parts
. Closes #29697. #29896 (Maksim Kita). - Add
table
alias to system.tables anddatabase
alias to system.databases #29677. #29882 (kevin wan). - Predefined configuration for table function remote. Closes #29756. #29774 (Kseniia Sumarokova).
- Returns String with OS Kernel version. #29755 (Memo).
- Added function
ngram
. Closes #29699. #29738 (Maksim Kita). - Support EXISTS(subquery). Closes #6852. #29731 (Kseniia Sumarokova).
-
- Added MD4 and SHA384 functions. #29602 (Nikita Tikhomirov).
- Clickhouse HTTP Server can enable HSTS by set
hsts_max_age
in config.xml with a positive number. #29516 (凌涛). - Huawei OBS Storage support. Closes #24294. #29511 (kevin wan).
- New function
mapContainsKeyLike
to get the map that key matches a simple regular expression. #29471 (凌涛). - Users now can set comments to database in
CREATE DATABASE
statement ... #29429 (Vasily Nemkov). - clickhouse-format support option
--query
. #29325 (凌涛). - Add CapnProto output format, refactor CapnProto input format. #29291 (Kruglov Pavel).
- Implemented
ALTER TABLE x MODIFY COMMENT
. #29264 (Vasily Nemkov). - Adds H3 inspection functions that are missing from ClickHouse but are available via the H3 api: https://h3geo.org/docs/api/inspection. #29209 (Bharat Nallan).
- Allow non-replicated ALTER TABLE FETCH and ATTACH in Replicated databases. #29202 (Kevin Michel).
- Added a setting
output_format_csv_null_representation
: This is the same asoutput_format_tsv_null_representation
but is for CSV output. #29123 (PHO). - Added function
zookeeperSessionUptime()
which returns uptime of current ZooKeeper session in seconds. #28983 (tavplubix). - Implements the
h3ToGeoBoundary
function. #28952 (Ivan Veselov). - Add aggregate function
exponentialMovingAverage
that can be used as window function. This closes #27511. #28914 (alexey-milovidov). - Allow to include subcolumns of table columns into
DESCRIBE
query result (can be enabled by settingdescribe_include_subcolumns
). #28905 (Anton Popov). Executable
,ExecutablePool
added optionsend_chunk_header
. If this option is true then chunk rows_count with line break will be sent to client before chunk. #28833 (Maksim Kita).- Added executable user defined functions. #28803 (Maksim Kita).
- Added
INFORMATION_SCHEMA
database withSCHEMATA
,TABLES
,VIEWS
andCOLUMNS
views to the corresponding tables insystem
database. Closes #9770. #28691 (tavplubix). - Add normalizeUTF8NFC, normalizeUTF8NFD, normalizeUTF8NFKC, normalizeUTF8NFKD functions. #28633 (darkkeks).
- Allow predefined connections to external data sources. Closes #28367. #28577 (Kseniia Sumarokova).
tokenbf_v1
andngram
support Map with key of String of FixedSring type. It enhance data skipping in query with map key filter.sql CREATE TABLE map_tokenbf ( row_id UInt32, map Map(String, String), INDEX map_tokenbf map TYPE ngrambf_v1(4,256,2,0) GRANULARITY 1 ) Engine=MergeTree() Order by id
With table above, the queryselect * from map_tokebf where map['K']='V'
will skip the granule that doesn't contain keyA
. Of course, how many rows will skipped is depended on thegranularity
andindex_granularity
you set. #28511 (凌涛).- Send profile events from server to client. New packet type
ProfileEvents
was introduced. Closes #26177. #28364 (Dmitry Novik). - This closes #27763. #28325 (小路).
- Support adding / deleting tables to replication from PostgreSQL dynamically in database engine MaterializedPostgreSQL. Support alter for database settings. Closes #27573. #28301 (Kseniia Sumarokova).
- This fully closes #4509 and even more. #27933 (Alexey Boykov).
- Add
sparkbar
aggregate function. This closes #26175. #27481 (小路). - Add interactive mode for clickhouse-local. Also merge code of clickhouse-client and clickhouse-local together. Closes #7203. Closes #25516. Closes #22401. #26231 (Kseniia Sumarokova).
- Implementation of a table engine to consume application log files in ClickHouse. closes #6953. #25969 (flynn).
- Added an ability to suspend and resume a process of a merge. This is needed for a better scheduling and controlling of merges execution. #22381. #25165 (Nikita Mikhaylov).
- Added function accurateCastOrDefault(x, T). Closes #21330. Authors @taiyang-li. #23028 (Maksim Kita).
- Logging all successful and failed login, and logout events to a new
system.session_log
table. ... #22415 (Vasily Nemkov). -
- Add Function
toUUIDOrDefault
,toUInt8/16/32/64/256OrDefault
,toInt8/16/32/64/128/256OrDefault
, which enables user defining default value(not null) when string parsing is failed. #21330 (taiyang-li).
- Add Function
- Queries with JOIN ON support OR. #21320 (Ilya Golshtein).
- New asynchronous insert mode allows to accumulate inserted data and store it in a single batch in background. On server-side it controlled by settings
async_insert_threads
,async_insert_max_data_size
andasync_insert_busy_timeout_ms
. For client it can be enabled by settingasync_insert
forINSERT
queries with data inlined in query or in separate buffer (e.g. forINSERT
queries via HTTP protocol). Ifwait_for_async_insert
is true (by default) the client will wait until data will be flushed to table. Implements #18282. #20557 (Ivan).
Performance Improvement
- Queries with
INTO OUTFILE
inclickhouse-client
will use multiple threads. Fix the issue with flickering progress-bar when usingINTO OUTFILE
. This closes #30873. This closes #30872. #30886 (alexey-milovidov). - Dictionaries support read from multiple threads. #30500 (Maksim Kita).
- ColumnDecimal improve performance for filter operation. #30431 (Jun Jin).
- Improve performance of aggregation in order of primary key (with enabled setting
optimize_aggregation_in_order
). #30266 (Anton Popov). - Improve the performance of SUM of nullable floating point numbers. #28906 (Raúl Marín).
- Attempt to improve performance of string comparison. #28767 (alexey-milovidov).
- Reduce default settings for S3 multipart upload part size. #28679 (ianton-ru).
- Speed up bitmapAnd function. #28332 (dddounaiking).
- Removed sub-optimal mutation notifications in
StorageMergeTree
when merges are still going. #27552 (Vladimir Chebotarev). - If query has multiple quantile aggregate functions with the same arguments but different level parameter:. #26657 (hexiaoting).
Improvement
- Rename the columns of SessionLog:
session_id
->auth_id
,session_name
->session_id
,changed_settings
->settings
. #30882 (Vitaly Baranov). - Allow to parse values of
Date
data type in text formats asYYYYMMDD
in addition toYYYY-MM-DD
. This closes #30870. #30871 (alexey-milovidov). - Arrays of all serializable types are now supported by arrayStringConcat. #30840 (Nickita Taranov).
- SQLUserDefinedFunctions support ON CLUSTER. Example
CREATE FUNCTION test_function ON CLUSTER 'cluster' AS x -> x + 1;
. Closes #30666. #30734 (Maksim Kita). - Get memory amount with Docker/cgroups limitations. See #25662. #30574 (Pavel Medvedev).
- Make query, which fetched table structure for PostgreSQL database because, more reliable. #30477 (Kseniia Sumarokova).
- SQLUserDefinedFunctions support
CREATE OR REPLACE
,CREATE IF NOT EXISTS
syntaxes. #30454 (Maksim Kita). - SQLUserDefinedFunctions added DROP IF EXISTS support. Example
DROP FUNCTION IF EXISTS test_function
. #30437 (Maksim Kita). - SQLUserDefinedFunctions support lambdas. Example
CREATE FUNCTION lambda_function AS x -> arrayMap(element -> element * 2, x);
. #30435 (Maksim Kita). - More full support of positional arguments. #30433 (Kseniia Sumarokova).
- Allow extract non-string element as string using JSONExtractString. This is for https://github.com/ClickHouse/ClickHouse/pull/25452#issuecomment-927123287. #30426 (Amos Bird).
- Added an ability to use FINAL clause in SELECT queries from GraphiteMergeTree. #30360 (Nikita Mikhaylov).
- Minor improvements in replica cloning and enqueuing fetch for broken parts, that should avoid extremely rare hanging of
GET_PART
entries in replication queue. #30346 (tavplubix). - Allow symlinks to files in user_files directory for file table function. #30309 (Kseniia Sumarokova).
- Fixed comparison of Date32 with Date, DateTime, DateTime64, String. #30219 (liang.huang).
- Allow to remove
SAMPLE BY
expression fromMergeTree
tables (ALTER TABLE <table> REMOVE SAMPLE BY
). #30180 (Anton Popov). - Support SQL user defined functions for clickhouse-local. #30179 (Maksim Kita).
- Now
Keeper
(as part ofclickhouse-server
) will start asynchronously if it can connect to some other node. #30170 (alesapin). - Now clickhouse-client supports native multi-line editing. #30143 (Amos Bird).
- PolygonDictionary added support for read method if setting
store_polygon_key_column
= true. Closes #30090. #30142 (Maksim Kita). - Reduce amount of redundant compressed data read from disk for some types
SELECT
queries (only for MergeTree engines family). #30111 (alesapin). - Improve solution https://github.com/ClickHouse/ClickHouse/pull/28853 See also https://github.com/ClickHouse/ClickHouse/pull/29928. #30084 (Vitaly Baranov).
- Add ability to print raw profile events to
clickhouse-client
(This can be useful for debugging and for testing). #30064 (Azat Khuzhin). - ProfileEvents::Counters snapshot doesn't store data in std::atomic anymore. #30000 (Dmitry Novik).
- Now clickhouse use DNS cache while communicating with external S3. #29999 (alesapin).
- Fix data-race between flush() and startup() in StorageBuffer. #29930 (Azat Khuzhin).
- Fix
lock-order-inversion
betweenDROP TABLE
forDatabaseMemory
andLiveView
. #29929 (Azat Khuzhin). - Fix lock-order-inversion between periodic dictionary reload and config reload. #29928 (Azat Khuzhin).
- Better exception message while reading column from Arrow-supported formats like
Arrow
,ArrowStream
,Parquet
andORC
. This closes #29926. #29927 (alexey-milovidov). - Update zoneinfo files to 2021c. #29925 (alexey-milovidov).
- Add ability to configure retries and delays between them for
clickhouse-copier
. #29921 (Azat Khuzhin). - Now min-max aggregation over the first expression of primary key is optimized by projection. This is for https://github.com/ClickHouse/ClickHouse/issues/329. #29918 (Amos Bird).
- Add
shutdown_wait_unfinished_queries
server setting to allowing waiting for running queries up toshutdown_wait_unfinished
time. This is for #24451 . #29914 (Amos Bird). - Remove unused variable in s3cluster. #29912 (taiyang-li).
- Add ability to trace peak memory usage (with new trace_type - MemoryPeak). #29858 (Azat Khuzhin).
- Added partitioned table prefix 'p' for the query for fetching replica identity index. #29828 (Shoh Jahon).
- Web UI: render bars in table cells. #29792 (alexey-milovidov).
- Remove some redundant
seek
calls while reading compressed blocks in MergeTree table engines family. #29766 (alesapin). - Apply
max_untracked_memory
/memory_profiler_step
/memory_profiler_sample_probability
during mutate/merge. #29681 (Azat Khuzhin). - Add ClickHouse logo to Play UI. #29674 (alexey-milovidov).
- Make
url
table function to process multiple URLs in parallel. This closes #29670 and closes #29671. #29673 (alexey-milovidov). - Query obfuscator
clickhouse-format --obfuscate
now works with more types of queries. #29672 (alexey-milovidov). - Fixed the issue:
clickhouse-format --obfuscate
cannot process queries with embedded dictionaries (functionsregionTo...
). #29667 (alexey-milovidov). - Fix incorrect nullable processing of JSONFunctions. This fixes #29615 . Mark as improvement because https://github.com/ClickHouse/ClickHouse/pull/28012 is not released. #29659 (Amos Bird).
- Increase
listen_backlog
by default (to match default in newer linux kernel). #29643 (Azat Khuzhin). - Allow user to change log levels without restart. #29586 (Nikolay Degterinsky).
- Reload dictionaries, models, user defined executable functions if servers config
dictionaries_config
,models_config
,user_defined_executable_functions_config
changes. Closes #28142. #29529 (Maksim Kita). - Get rid of pointless restriction on projection name. Now projection name can start with
tmp_
. #29520 (Amos Bird). - Fixed
There is no query or query context has expired
error in mutations with nested subqueries. Do not allow subqueries in mutation if table is replicated andallow_nondeterministic_mutations
setting is disabled. #29495 (tavplubix). - Add support for
IS NULL
/IS NOT NULL
for external dbs (i.e. MySQL). #29463 (Azat Khuzhin). - Enable per-query memory profiler (set to memory_profiler_step=4MiB) globally. #29455 (Azat Khuzhin).
- Transform
isNull
/isNotNull
toIS NULL
/IS NOT NULL
(for external dbs, i.e. MySQL). #29446 (Azat Khuzhin). - Apply config changes to
max_concurrent_queries
during runtime (no need to restart). #29414 (Raúl Marín). - Added setting
use_skip_indexes
. #29405 (Maksim Kita). - fix lost in memory part when freeze. #29376 (Mo Xuan).
- Pass through initial query_id for clickhouse-benchmark (previously if you run remote query via
clickhouse-benchmark
, queries on shards will not be linked to the initial query viainitial_query_id
). #29364 (Azat Khuzhin). - Allow to write number in query as binary literal. Example
SELECT 0b001;
. #29304 (Maksim Kita). - Primary key index and partition filter can work in tuple. #29281 (凌涛).
- Skip indexes
tokenbf_v1
andngrambf_v1
added support for Array data type with key of String of FixedString type. #29280 (Maksim Kita). - Function
has
added support for map data type. #29267 (Maksim Kita). - Add
compress_logs
settings for clickhouse-keeper which allow to compress clickhouse-keeper logs inZSTD
. Implements: #26977. #29223 (alesapin). - Skip indexes
tokenbf_v1
andngrambf_v1
added support for Map data type with key of String of FixedString type. Author @lingtaolf. #29220 (Maksim Kita). - Add ability to disable converting expressions to local filter for external queries (
external_table_strict_query=true
). #29206 (Azat Khuzhin). - Allow to asynchronous reads for remote fs disks. #29205 (Kseniia Sumarokova).
- Add CORS support. Add headers for preflight options request. Closes #18693. #29155 (Filatenkov Artur).
- Added an ability to execute more merges and mutations than the number of threads in background pool. Merges and mutations will be executed step by step according to their sizes (lower is more prioritized). The ratio of the number of tasks to threads to execute is controlled by a setting
background_merges_mutations_concurrency_ratio
, 2 by default. #29140 (Nikita Mikhaylov). - Disable projections when
ARRAY JOIN
is used. Currently projection analysis will break array join aliases. #29139 (Amos Bird). - Fix possible LogicalError, support more types in MsgPack input/output format. #29077 (Kruglov Pavel).
- Allow to input and output LowCardinality columns in ORC input/output format. #29062 (Kruglov Pavel).
- Select from
system.distributed_ddl_queue
might show incorrect values, it's fixed. #29061 (tavplubix). - Correct behaviour with unknown methods for http connection. Solves #29050. #29057 (Filatenkov Artur).
- Fix bug in
clickhouse-keeper-converter
which can lead to some data loss while restoring from ZooKeeper logs (not snapshot). #29030 (小路). - Apply settings from CREATE AS SELECT queries (fixes: #28810). #28962 (Azat Khuzhin).
- add default database setting for alter table ... on cluster ... replace/move partition from/to ... #28955 (anneji-dev).
- gRPC: Allow change server-side compression from client. #28953 (Vitaly Baranov).
- Add time dependency for clickhouse-server unit (systemd and sysvinit init). #28891 (Azat Khuzhin).
- Skip "no data" exception when reading thermal sensors for asynchronous metrics. This closes #28852. #28882 (alexey-milovidov).
- Fixed race condition that might cause
Dictionary not found
error for existing dictionary in rare cases. #28853 (tavplubix). - Relax nested function for If-combinator check (but forbid nested identical combinators). #28828 (Azat Khuzhin).
- Fix uncaught exception during server termination. #28761 (Azat Khuzhin).
- Forbid cleaning of tmp directories that can be used by an active mutation/merge. #28760 (Azat Khuzhin).
- Allow optimization :
optimize_arithmetic_operations_in_aggregate_functions = 1
when alias is used. I cannot figure out why it was disabled. We set aliases in all other TreeOptimizers, except this one. Maybe I'm missing something important) @4ertus2 could you give a bad example?. #28746 (Amos Bird). - Implement
detach_not_byte_identical_parts
setting forReplicatedMergeTree
, that will detach instead of remove not byte-identical parts (after mege/mutate). #28708 (Azat Khuzhin). - Implement
max_suspicious_broken_parts_bytes
setting forMergeTree
(to limit total size of all broken parts, default is1GiB
). #28707 (Azat Khuzhin). - This PR enables expanding macros in RabbitMQ settings, for example:. #28683 (Vitaly Baranov).
- Resolve table dependencies on metadata loading. Closes #8004, closes #15170. #28373 (tavplubix).
- Speed up part loading process with multiple disks are in use. The idea is similar to https://github.com/ClickHouse/ClickHouse/pull/16423 . Prod env shows improvement: 24 min -> 16 min . #28363 (Amos Bird).
- Avoid error "Division by zero" when denominator is Nullable in functions
divide
,intDiv
andmodulo
. Closes #22621. #28352 (Kruglov Pavel). - Reload stacktrace cache when symbol is reloaded. #28137 (Amos Bird).
- This PR allows reading data of a table using the
Log
engine in multiple threads. #28125 (Vitaly Baranov). - Fix misbehavior of NULL column handling in json functions. This fixes #27930. #28012 (Amos Bird).
- Add dedicated Mark/Uncompressed cache for skip indices. Not sure how to test it properly. #27961 (Amos Bird).
- Correct
Codec(AES_128_GCM_SIV)
. #27476 (Filatenkov Artur). - Add compression for
INTO OUTPUT
andFROM INFILE
with additional optional parameter. #27135 (Filatenkov Artur).
Bug Fix
- Fix reading from empty file on encrypted disk. #30494 (Vitaly Baranov).
- Fix releasing query ID and session ID at the end of query processing while handing gRPC call. This PR fixes flaky test test_grpc_protocol/test.py::test_session. #29954 (Vitaly Baranov).
- Fix shutdown of
AccessControlManager
. Now there can't be reloading of the configuration after AccessControlManager has been destroyed. This PR fixes the flaky test test_user_directories/test.py::test_relative_path. #29951 (Vitaly Baranov). - Fix bug in
clickhouse-keeper-converter
which can lead to incorrect ZooKeeper log deserialization. #29071 (小路). - Fix transformation of disjunctions chain to
IN
(controlled by settingsoptimize_min_equality_disjunction_chain_length
) in distributed queries with settingslegacy_column_name_of_tuple_literal = 0
. #28658 (Anton Popov). - Allow using a materialized column as the sharding key in a distributed table even if
insert_allow_materialized_columns=0
:. #28637 (Vitaly Baranov).
Build/Testing/Packaging Improvement
- Recursive submodules are no longer needed for ClickHouse. #30315 (alexey-milovidov).
- ClickHouse can be statically built with Musl. This is added as experiment, it does not support building
odbc-bridge
,library-bridge
, integration with CatBoost and some libraries. #30248 (alexey-milovidov). - Add tests for encrypted disk & encryption codec and ReplicatedMergeTree. #30172 (Vitaly Baranov).
- Enable Protobuf, Arrow, ORC, Parquet for AArch64 and Darwin builds. This closes #29248. This closes #28018. #30015 (alexey-milovidov).
- Add cross-build for PowerPC (powerpc64le). This closes #9589. Enable support for interaction with MySQL for AArch64 and PowerPC. This closes #26301. #30010 (alexey-milovidov).
- Leave only required files in cross-compile toolchains. Include them as submodules (earlier they were downloaded as tarballs). #29974 (alexey-milovidov).
- Fix an build error because of Rename "common" to "base". #29688 (Sergei Semin).
- Turning on experimental constexpr expressions evaluator for clang to speed up template code compilation. #29668 (myrrc).
- Add ability to compile using newer version fo glibc w/o using new symbols. #29594 (Azat Khuzhin).
- N/A Detailed description / Documentation draft:. #29367 (kevin wan).
- Add a test for https://github.com/ClickHouse/ClickHouse/pull/28683. #28934 (Vitaly Baranov).
- Convert
skip_list .json
into first line comments. #28909 (Vitaly Baranov). - Reduce Debug build binary size by clang optimization. #28736 (flynn).
- Now all images for CI will be placed in the separate dockerhub repo. #28656 (alesapin).
- Improve support for build with clang-13. #28046 (Sergei Semin).
- Switch clang builds to 12 version. #20495 (Ilya Yatsishin).
Bug Fix (user-visible misbehaviour in official stable or prestable release
- Fix
ORDER BY ... WITH FILL
with setTO
andFROM
and no rows in result set. #30888 (Anton Popov).
Bug Fix (user-visible misbehaviour in official stable or prestable release)
- Fix set index not used in AND/OR expressions when there are more than two operands. This fixes #30416 . #30887 (Amos Bird).
- Fix crash when projection with hashing function is materialized. This fixes #30861 . The issue is similar to https://github.com/ClickHouse/ClickHouse/pull/28560 which is a lack of proper understanding of the invariant of header's emptyness. #30877 (Amos Bird).
- Fixed ambiguity when extracting auxiliary ZooKeeper name from ZooKeeper path in
ReplicatedMergeTree
. Previously server might fail to start withUnknown auxiliary ZooKeeper name
if ZooKeeper path contains a colon. Fixes #29052. Also it was allowed to specify ZooKeeper path that does not start with slash, but now it's deprecated and creation of new tables with such path is not allowed. Slashes and colons in auxiliary ZooKeeper names are not allowed too. #30822 (tavplubix). - Clean temporary directory when localBackup failed by some reason. #30797 (ianton-ru).
- Fixed a race condition between
REPLACE/MOVE PARTITION
and background merge in non-replicatedMergeTree
that might cause a part of moved/replaced data to remain in partition. Fixes #29327. #30717 (tavplubix). - Fix PREWHERE with WHERE in case of always true PREWHERE. #30668 (Azat Khuzhin).
- Functions for case-insensitive search in UTF8 strings like
positionCaseInsensitiveUTF8
andcountSubstringsCaseInsensitiveUTF8
might find substrings that actually does not match, it's fixed. #30663 (tavplubix). - Update aws-sdk submodule for throttling in Yandex.S3. #30646 (ianton-ru).
- Limit push down optimization could cause a error
Cannot find column
. Fixes #30438. #30562 (Nikolai Kochetov). - Add missing parenthesis for
isNotNull
/isNull
rewrites toIS [NOT] NULL
(fixes queries that has something likeisNotNull(1)+isNotNull(2)
). #30520 (Azat Khuzhin). -
- Fix deadlock on ALTER with scalar subquery to the same table, close #30461. #30492 (Vladimir C).
- Fixed segfault which might happen if session expired during execution of REPLACE PARTITION. #30432 (tavplubix).
- Makes #30162 less possible ... #30370 (Vasily Nemkov).
- Queries with condition like
IN (subquery)
could return incorrect result in case if aggregate projection applied. Fixed creation of sets for projections. #30310 (Amos Bird). - Fix column alias resolution of JOIN queries when projection is enabled. This fixes #30146. #30293 (Amos Bird).
- fix replaceRegexpAll bug. #30292 (Memo).
- Fix ComplexKeyHashedDictionary, ComplexKeySparseHashedDictionary parsing
preallocate
option from layout config. #30246 (Maksim Kita). - Fix
[I]LIKE
function. Closes #28661. #30244 (Nikolay Degterinsky). - Fix crash with shortcircuit and lowcardinality in multiIf. #30243 (Raúl Marín).
- FlatDictionary, HashedDictionary fix bytes_allocated calculation for nullable attributes. #30238 (Maksim Kita).
-
- Allow identifiers staring with numbers in multiple joins. #30230 (Vladimir C).
- Fix reading from
MergeTree
withmax_read_buffer_size=0
(can lead toCan't adjust last granule
LOGICAL_ERROR
, or even data loss). #30192 (Azat Khuzhin). - Fix
pread_fake_async
/pread_threadpool
withmin_bytes_to_use_direct_io
. #30191 (Azat Khuzhin). - Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column. #30189 (Azat Khuzhin).
- Support nullable arguments in function
initializeAggregation
. #30177 (Anton Popov). - Fix error
Port is already connected
for queries withGLOBAL IN
andWITH TOTALS
. Only for 21.9 and 21.10. #30086 (Nikolai Kochetov). - Fix race between MOVE PARTITION and merges/mutations for MergeTree. #30074 (Azat Khuzhin).
- Dropped
Memory
database might reappear after server restart, it's fixed (#29795). Also addedforce_remove_data_recursively_on_drop
setting as a workaround forDirectory not empty
error when droppingOrdinary
database (because it's not possible to remove data leftovers manually in cloud environment). #30054 (tavplubix). - Fix crash of sample by
tuple()
, closes #30004. #30016 (flynn). - try to close issue: #29965. #29976 (hexiaoting).
- Fix possible data-race between
FileChecker
andStorageLog
/StorageStripeLog
. #29959 (Azat Khuzhin). - Fix data-race between
LogSink::writeMarks()
andLogSource
inStorageLog
. #29946 (Azat Khuzhin). - Fix potential resource leak of the concurrent query limit of merge tree tables introduced in https://github.com/ClickHouse/ClickHouse/pull/19544 . #29879 (Amos Bird).
- Fix system tables recreation check (fails to detect changes in enum values). #29857 (Azat Khuzhin).
- MaterializedMySQL: Fix an issue where if the connection to MySQL was lost, only parts of a transaction could be processed. #29837 (Håvard Kvålen).
- Avoid
Timeout exceeded: elapsed 18446744073.709553 seconds
error that might happen in extremely rare cases, presumably due to some bug in kernel. Fixes #29154. #29811 (tavplubix). - Fix bad cast in
ATTACH TABLE ... FROM 'path'
query when non-string literal is used instead of path. It may lead to reading of uninitialized memory. #29790 (alexey-milovidov). - Fix concurrent access to
LowCardinality
duringGROUP BY
(leads to SIGSEGV). #29782 (Azat Khuzhin). - Fix incorrect
GROUP BY
(multiple rows with the same keys in result) in case of distributed query when shards had mixed versions<= 21.3
and>= 21.4
,GROUP BY
key had several columns all with fixed size, and two-level aggregation was activated (seegroup_by_two_level_threshold
andgroup_by_two_level_threshold_bytes
). Fixes #29580. #29735 (Nikolai Kochetov). - Fixed incorrect behaviour of setting
materialized_postgresql_tables_list
at server restart. Found in #28529. #29686 (Kseniia Sumarokova). - Condition in filter predicate could be lost after push-down optimisation. #29625 (Nikolai Kochetov).
- Fix JIT expression compilation with aliases and short-circuit expression evaluation. Closes #29403. #29574 (Maksim Kita).
- Fix rare segfault in
ALTER MODIFY
query when using incorrect table identifier inDEFAULT
expression likex.y.z...
Fixes #29184. #29573 (alesapin). - Fix null deference for
GROUP BY WITH TOTALS HAVING
(when the column fromHAVING
wasn't selected). #29553 (Azat Khuzhin). - Avoid deadlocks when reading and writting on JOIN Engine tables at the same time. #29544 (Raúl Marín).
- Fix bug in check
pathStartsWith
becuase there was bug with the usage ofstd::mismatch
:The behavior is undefined if the second range is shorter than the first range.
. #29531 (Kseniia Sumarokova). - In ODBC bridge add retries for error Invalid cursor state. It is a retriable error. Closes #29473. #29518 (Kseniia Sumarokova).
- Fixed incorrect table name parsing on loading of
Lazy
database. Fixes #29456. #29476 (tavplubix). - Fix possible
Block structure mismatch
for subqueries with pushed-downHAVING
predicate. Fixes #29010. #29475 (Nikolai Kochetov). - Fix Logical error
Cannot capture columns
in functions greatest/least. Closes #29334. #29454 (Kruglov Pavel). - rocksdb: fix race condition during multiple DB opening (and get back some tests that triggers the problem on CI). #29393 (Azat Khuzhin).
- Fix replicated access storage not shutting down cleanly when misconfigured. #29388 (Kevin Michel).
- Remove window function
nth_value
as it is not memory-safe. This closes #29347. #29348 (alexey-milovidov). - Fix vertical merges of projection parts. This fixes #29253 . This PR also fixes several projection merge/mutation issues introduced in https://github.com/ClickHouse/ClickHouse/pull/25165. #29337 (Amos Bird).
- Fix hanging DDL queries on Replicated database while adding a new replica. #29328 (Kevin Michel).
- Fix connection timeouts (
send_timeout
/receive_timeout
). #29282 (Azat Khuzhin). - Fix failed assertion in ReadBufferFromHDFS. Update libhdfs3 library to be able to run in tests in debug. Closes #29251. Closes #27814. #29276 (Kseniia Sumarokova).
- Fix possible
Table columns structure in ZooKeeper is different from local table structure
exception while recreating or creating new replicas ofReplicatedMergeTree
, when one of table columns have default expressions with case-insensitive functions. #29266 (Anton Popov). - Send normal
Database doesn't exist error
(UNKNOWN_DATABASE
) to the client (via TCP) instead ofAttempt to read after eof
(ATTEMPT_TO_READ_AFTER_EOF
). #29229 (Azat Khuzhin). - Fix segfault while inserting into column with type LowCardinality(Nullable) in Avro input format. #29132 (Kruglov Pavel).
- Do not allow to reuse previous credentials in case of inter-server secret (Before INSERT via Buffer/Kafka to Distributed table with interserver secret configured for that cluster, may re-use previously set user for that connection). #29060 (Azat Khuzhin).
- Handle
any_join_distinct_right_table_keys
when join with dictionary, close #29007. #29014 (Vladimir C). - Fix "Not found column ... in block" error, when join on alias column, close #26980. #29008 (Vladimir C).
- Fix the number of threads used in
GLOBAL IN
subquery (it was executed in single threads since #19414 bugfix). #28997 (Nikolai Kochetov). - Fix bad optimizations of ORDER BY if it contains WITH FILL. This closes #28908. This closes #26049. #28910 (alexey-milovidov).
- Fix higher-order array functions (
SIGSEGV
forarrayCompact
/ILLEGAL_COLUMN
forarrayDifference
/arrayCumSumNonNegative
) with consts. #28904 (Azat Khuzhin). - Fix waiting for mutation with
mutations_sync=2
. #28889 (Azat Khuzhin). - Fix queries to external databases (i.e. MySQL) with multiple columns in IN ( i.e.
(k,v) IN ((1, 2))
) (but note that this has some backward incompatibility for theclickhouse-copier
since it uses alias for tuple element). #28888 (Azat Khuzhin). - Fix bug with LowCardinality in short-curcuit function evaluation. Closes #28884. #28887 (Kruglov Pavel).
- Fix reading of subcolumns from compact parts. #28873 (Anton Popov).
- Fixed a race condition between
DROP PART
andREPLACE/MOVE PARTITION
that might cause replicas to diverge in rare cases. #28864 (tavplubix). - Fix expressions compilation with short circuit evaluation. #28821 (Azat Khuzhin).
- Fix extremely rare case when ReplicatedMergeTree replicas can diverge after hard reboot of all replicas. The error looks like
Part ... intersects (previous|next) part ...
. #28817 (alesapin). - Better check for connection usability and also catch any exception in RabbitMQ shutdown just in case. #28797 (Kseniia Sumarokova).
- Fix benign race condition in ReplicatedMergeTreeQueue. Shouldn't be visible for user, but can lead to subtle bugs. #28734 (alesapin).
- Fix possible crash for
SELECT
with partially created aggregate projection in case of exception. #28700 (Amos Bird). - Fix the coredump in the creation of distributed tables, when the parameters passed in are wrong. #28686 (Zhiyong Wang).
- Add Settings.Names, Settings.Values aliases for system.processes table. #28685 (Vitaly).
-
- Fix the number of arguments required by s2RectAdd and s2RectContains functions. #28663 (Bharat Nallan).
- Fix invalid constant type conversion when nullable or lowcardinality primary key is used. #28636 (Amos Bird).
- Fix "Column is not under aggregate function and not in GROUP BY" with PREWHERE (Fixes: #28461). #28502 (Azat Khuzhin).
NO CL ENTRY
- NO CL ENTRY: 'remove some unneeded header files'. #30722 (flynn).
- NO CL ENTRY: 'Revert "Revert "Improve usability of
remote_url_allow_hosts
""'. #30708 (Nikolai Kochetov). - NO CL ENTRY: 'Revert "Improve usability of
remote_url_allow_hosts
"'. #30707 (Nikolai Kochetov). - NO CL ENTRY: 'Revert "Fix style regressions on benchmark page"'. #30652 (alesapin).
- NO CL ENTRY: 'add support of window function in antlr grammar'. #30181 (PHaroZ).
- NO CL ENTRY: 'Link FAQ from Debian installation'. #29836 (Federico Ceratto).
- NO CL ENTRY: 'Revert "Add coroutines example."'. #29829 (Nikolai Kochetov).
- NO CL ENTRY: 'Generate test_results.tsv for fasttest'. #29319 (Dmitry Novik).
- NO CL ENTRY: 'Update rounding-functions.md'. #29249 (quoctan132).
Testing Improvement
- Implemented structure-aware fuzzing approach in ClickHouse for select statement parser. #30012 (Paul).
Backward Incompatible Change
- Remove setting odbc_max_field_size because it is not used anymore. Closes #30413. #30778 (Kseniia Sumarokova).
- Change order of json_path and json in sql json function (to be correct). Closes #30449. #30474 (Kseniia Sumarokova).
- Remove MergeTree table setting
write_final_mark
. It will be alwaystrue
. #30455 (Kseniia Sumarokova). - Function
bayesAB
is removed. This closes #26233. #29934 (alexey-milovidov). - Now ClickHouse Keeper snapshots compressed with
ZSTD
codec by default instead of custom ClickHouse LZ4 block compression. This behavior can be turned off withcompress_snapshots_with_zstd_format
coordination setting (must be equal on all quorum replicas). Backward incompatibility is quite rare and may happen only when new node will send snapshot (happens in case of recovery) to the old node which is unable to read snapshots in ZSTD format. #29417 (alesapin).
New Feature
- Support
ALTER TABLE
for tables inMemory
databases. #30866 (tavplubix). - New function
mapExtractKeyLike
to get the map only kept elements matched specified pattern. #30793 (凌涛). - CompiledExpressionCache limit elements size using
compiled_expression_cache_elements_size
setting. #30667 (Maksim Kita). - Add ability to change nodes configuration (in
.xml
file) for ClickHouse Keeper. #30372 (alesapin). - Added
HashedArray
dictionary type. Closes #30236. #30242 (Maksim Kita). - Added new JSONExtractKeys function ... #30056 (Vitaly).
- Added function
tokens
. That allow to split string into tokens using non-alpha numeric ASCII characters as separators. #29981 (Maksim Kita). - Add support for FreeBSD aarch64. #29952 (MikaelUrankar).
- User can now create dictionaries with comments:
CREATE DICTIONARY ... COMMENT 'vaue'
... #29899 (Vasily Nemkov). - Added columns
data_compressed_bytes
,data_uncompressed_bytes
,marks_bytes
intosystem.data_skipping_indices
. Added columnssecondary_indices_compressed_bytes
,secondary_indices_uncompressed_bytes
,secondary_indices_marks_bytes
intosystem.parts
. Closes #29697. #29896 (Maksim Kita). - Add
table
alias to system.tables anddatabase
alias to system.databases #29677. #29882 (kevin wan). - Predefined configuration for table function remote. Closes #29756. #29774 (Kseniia Sumarokova).
- Returns String with OS Kernel version. #29755 (Memo).
- Added function
ngram
. Closes #29699. #29738 (Maksim Kita). - Support EXISTS(subquery). Closes #6852. #29731 (Kseniia Sumarokova).
-
- Added MD4 and SHA384 functions. #29602 (Nikita Tikhomirov).
- Clickhouse HTTP Server can enable HSTS by set
hsts_max_age
in config.xml with a positive number. #29516 (凌涛). - Huawei OBS Storage support. Closes #24294. #29511 (kevin wan).
- New function
mapContainsKeyLike
to get the map that key matches a simple regular expression. #29471 (凌涛). - Users now can set comments to database in
CREATE DATABASE
statement ... #29429 (Vasily Nemkov). - clickhouse-format support option
--query
. #29325 (凌涛). - Add CapnProto output format, refactor CapnProto input format. #29291 (Kruglov Pavel).
- Implemented
ALTER TABLE x MODIFY COMMENT
. #29264 (Vasily Nemkov). - Adds H3 inspection functions that are missing from ClickHouse but are available via the H3 api: https://h3geo.org/docs/api/inspection. #29209 (Bharat Nallan).
- Allow non-replicated ALTER TABLE FETCH and ATTACH in Replicated databases. #29202 (Kevin Michel).
- Added a setting
output_format_csv_null_representation
: This is the same asoutput_format_tsv_null_representation
but is for CSV output. #29123 (PHO). - Added function
zookeeperSessionUptime()
which returns uptime of current ZooKeeper session in seconds. #28983 (tavplubix). - Implements the
h3ToGeoBoundary
function. #28952 (Ivan Veselov). - Add aggregate function
exponentialMovingAverage
that can be used as window function. This closes #27511. #28914 (alexey-milovidov). - Allow to include subcolumns of table columns into
DESCRIBE
query result (can be enabled by settingdescribe_include_subcolumns
). #28905 (Anton Popov). Executable
,ExecutablePool
added optionsend_chunk_header
. If this option is true then chunk rows_count with line break will be sent to client before chunk. #28833 (Maksim Kita).- Added executable user defined functions. #28803 (Maksim Kita).
- Added
INFORMATION_SCHEMA
database withSCHEMATA
,TABLES
,VIEWS
andCOLUMNS
views to the corresponding tables insystem
database. Closes #9770. #28691 (tavplubix). - Add normalizeUTF8NFC, normalizeUTF8NFD, normalizeUTF8NFKC, normalizeUTF8NFKD functions. #28633 (darkkeks).
- Allow predefined connections to external data sources. Closes #28367. #28577 (Kseniia Sumarokova).
tokenbf_v1
andngram
support Map with key of String of FixedSring type. It enhance data skipping in query with map key filter.sql CREATE TABLE map_tokenbf ( row_id UInt32, map Map(String, String), INDEX map_tokenbf map TYPE ngrambf_v1(4,256,2,0) GRANULARITY 1 ) Engine=MergeTree() Order by id
With table above, the queryselect * from map_tokebf where map['K']='V'
will skip the granule that doesn't contain keyA
. Of course, how many rows will skipped is depended on thegranularity
andindex_granularity
you set. #28511 (凌涛).- Send profile events from server to client. New packet type
ProfileEvents
was introduced. Closes #26177. #28364 (Dmitry Novik). - This closes #27763. #28325 (小路).
- Support adding / deleting tables to replication from PostgreSQL dynamically in database engine MaterializedPostgreSQL. Support alter for database settings. Closes #27573. #28301 (Kseniia Sumarokova).
- This fully closes #4509 and even more. #27933 (Alexey Boykov).
- Add
sparkbar
aggregate function. This closes #26175. #27481 (小路). - Add interactive mode for clickhouse-local. Also merge code of clickhouse-client and clickhouse-local together. Closes #7203. Closes #25516. Closes #22401. #26231 (Kseniia Sumarokova).
- Implementation of a table engine to consume application log files in ClickHouse. closes #6953. #25969 (flynn).
- Added an ability to suspend and resume a process of a merge. This is needed for a better scheduling and controlling of merges execution. #22381. #25165 (Nikita Mikhaylov).
- Added function accurateCastOrDefault(x, T). Closes #21330. Authors @taiyang-li. #23028 (Maksim Kita).
- Logging all successful and failed login, and logout events to a new
system.session_log
table. ... #22415 (Vasily Nemkov). -
- Add Function
toUUIDOrDefault
,toUInt8/16/32/64/256OrDefault
,toInt8/16/32/64/128/256OrDefault
, which enables user defining default value(not null) when string parsing is failed. #21330 (taiyang-li).
- Add Function
- Queries with JOIN ON support OR. #21320 (Ilya Golshtein).
- New asynchronous insert mode allows to accumulate inserted data and store it in a single batch in background. On server-side it controlled by settings
async_insert_threads
,async_insert_max_data_size
andasync_insert_busy_timeout_ms
. For client it can be enabled by settingasync_insert
forINSERT
queries with data inlined in query or in separate buffer (e.g. forINSERT
queries via HTTP protocol). Ifwait_for_async_insert
is true (by default) the client will wait until data will be flushed to table. Implements #18282. #20557 (Ivan).
Performance Improvement
- Queries with
INTO OUTFILE
inclickhouse-client
will use multiple threads. Fix the issue with flickering progress-bar when usingINTO OUTFILE
. This closes #30873. This closes #30872. #30886 (alexey-milovidov). - Dictionaries support read from multiple threads. #30500 (Maksim Kita).
- ColumnDecimal improve performance for filter operation. #30431 (Jun Jin).
- Improve performance of aggregation in order of primary key (with enabled setting
optimize_aggregation_in_order
). #30266 (Anton Popov). - Improve the performance of SUM of nullable floating point numbers. #28906 (Raúl Marín).
- Attempt to improve performance of string comparison. #28767 (alexey-milovidov).
- Reduce default settings for S3 multipart upload part size. #28679 (ianton-ru).
- Speed up bitmapAnd function. #28332 (dddounaiking).
- Removed sub-optimal mutation notifications in
StorageMergeTree
when merges are still going. #27552 (Vladimir Chebotarev). - If query has multiple quantile aggregate functions with the same arguments but different level parameter:. #26657 (hexiaoting).
Improvement
- Rename the columns of SessionLog:
session_id
->auth_id
,session_name
->session_id
,changed_settings
->settings
. #30882 (Vitaly Baranov). - Allow to parse values of
Date
data type in text formats asYYYYMMDD
in addition toYYYY-MM-DD
. This closes #30870. #30871 (alexey-milovidov). - Arrays of all serializable types are now supported by arrayStringConcat. #30840 (Nickita Taranov).
- SQLUserDefinedFunctions support ON CLUSTER. Example
CREATE FUNCTION test_function ON CLUSTER 'cluster' AS x -> x + 1;
. Closes #30666. #30734 (Maksim Kita). - Get memory amount with Docker/cgroups limitations. See #25662. #30574 (Pavel Medvedev).
- Make query, which fetched table structure for PostgreSQL database because, more reliable. #30477 (Kseniia Sumarokova).
- SQLUserDefinedFunctions support
CREATE OR REPLACE
,CREATE IF NOT EXISTS
syntaxes. #30454 (Maksim Kita). - SQLUserDefinedFunctions added DROP IF EXISTS support. Example
DROP FUNCTION IF EXISTS test_function
. #30437 (Maksim Kita). - SQLUserDefinedFunctions support lambdas. Example
CREATE FUNCTION lambda_function AS x -> arrayMap(element -> element * 2, x);
. #30435 (Maksim Kita). - More full support of positional arguments. #30433 (Kseniia Sumarokova).
- Allow extract non-string element as string using JSONExtractString. This is for https://github.com/ClickHouse/ClickHouse/pull/25452#issuecomment-927123287. #30426 (Amos Bird).
- Added an ability to use FINAL clause in SELECT queries from GraphiteMergeTree. #30360 (Nikita Mikhaylov).
- Minor improvements in replica cloning and enqueuing fetch for broken parts, that should avoid extremely rare hanging of
GET_PART
entries in replication queue. #30346 (tavplubix). - Allow symlinks to files in user_files directory for file table function. #30309 (Kseniia Sumarokova).
- Fixed comparison of Date32 with Date, DateTime, DateTime64, String. #30219 (liang.huang).
- Allow to remove
SAMPLE BY
expression fromMergeTree
tables (ALTER TABLE <table> REMOVE SAMPLE BY
). #30180 (Anton Popov). - Support SQL user defined functions for clickhouse-local. #30179 (Maksim Kita).
- Now
Keeper
(as part ofclickhouse-server
) will start asynchronously if it can connect to some other node. #30170 (alesapin). - Now clickhouse-client supports native multi-line editing. #30143 (Amos Bird).
- PolygonDictionary added support for read method if setting
store_polygon_key_column
= true. Closes #30090. #30142 (Maksim Kita). - Reduce amount of redundant compressed data read from disk for some types
SELECT
queries (only for MergeTree engines family). #30111 (alesapin). - Improve solution https://github.com/ClickHouse/ClickHouse/pull/28853 See also https://github.com/ClickHouse/ClickHouse/pull/29928. #30084 (Vitaly Baranov).
- Add ability to print raw profile events to
clickhouse-client
(This can be useful for debugging and for testing). #30064 (Azat Khuzhin). - ProfileEvents::Counters snapshot doesn't store data in std::atomic anymore. #30000 (Dmitry Novik).
- Now clickhouse use DNS cache while communicating with external S3. #29999 (alesapin).
- Fix data-race between flush() and startup() in StorageBuffer. #29930 (Azat Khuzhin).
- Fix
lock-order-inversion
betweenDROP TABLE
forDatabaseMemory
andLiveView
. #29929 (Azat Khuzhin). - Fix lock-order-inversion between periodic dictionary reload and config reload. #29928 (Azat Khuzhin).
- Better exception message while reading column from Arrow-supported formats like
Arrow
,ArrowStream
,Parquet
andORC
. This closes #29926. #29927 (alexey-milovidov). - Update zoneinfo files to 2021c. #29925 (alexey-milovidov).
- Add ability to configure retries and delays between them for
clickhouse-copier
. #29921 (Azat Khuzhin). - Now min-max aggregation over the first expression of primary key is optimized by projection. This is for https://github.com/ClickHouse/ClickHouse/issues/329. #29918 (Amos Bird).
- Add
shutdown_wait_unfinished_queries
server setting to allowing waiting for running queries up toshutdown_wait_unfinished
time. This is for #24451 . #29914 (Amos Bird). - Remove unused variable in s3cluster. #29912 (taiyang-li).
- Add ability to trace peak memory usage (with new trace_type - MemoryPeak). #29858 (Azat Khuzhin).
- Added partitioned table prefix 'p' for the query for fetching replica identity index. #29828 (Shoh Jahon).
- Web UI: render bars in table cells. #29792 (alexey-milovidov).
- Remove some redundant
seek
calls while reading compressed blocks in MergeTree table engines family. #29766 (alesapin). - Apply
max_untracked_memory
/memory_profiler_step
/memory_profiler_sample_probability
during mutate/merge. #29681 (Azat Khuzhin). - Add ClickHouse logo to Play UI. #29674 (alexey-milovidov).
- Make
url
table function to process multiple URLs in parallel. This closes #29670 and closes #29671. #29673 (alexey-milovidov). - Query obfuscator
clickhouse-format --obfuscate
now works with more types of queries. #29672 (alexey-milovidov). - Fixed the issue:
clickhouse-format --obfuscate
cannot process queries with embedded dictionaries (functionsregionTo...
). #29667 (alexey-milovidov). - Fix incorrect nullable processing of JSONFunctions. This fixes #29615 . Mark as improvement because https://github.com/ClickHouse/ClickHouse/pull/28012 is not released. #29659 (Amos Bird).
- Increase
listen_backlog
by default (to match default in newer linux kernel). #29643 (Azat Khuzhin). - Allow user to change log levels without restart. #29586 (Nikolay Degterinsky).
- Reload dictionaries, models, user defined executable functions if servers config
dictionaries_config
,models_config
,user_defined_executable_functions_config
changes. Closes #28142. #29529 (Maksim Kita). - Get rid of pointless restriction on projection name. Now projection name can start with
tmp_
. #29520 (Amos Bird). - Fixed
There is no query or query context has expired
error in mutations with nested subqueries. Do not allow subqueries in mutation if table is replicated andallow_nondeterministic_mutations
setting is disabled. #29495 (tavplubix). - Add support for
IS NULL
/IS NOT NULL
for external dbs (i.e. MySQL). #29463 (Azat Khuzhin). - Enable per-query memory profiler (set to memory_profiler_step=4MiB) globally. #29455 (Azat Khuzhin).
- Transform
isNull
/isNotNull
toIS NULL
/IS NOT NULL
(for external dbs, i.e. MySQL). #29446 (Azat Khuzhin). - Apply config changes to
max_concurrent_queries
during runtime (no need to restart). #29414 (Raúl Marín). - Added setting
use_skip_indexes
. #29405 (Maksim Kita). - fix lost in memory part when freeze. #29376 (Mo Xuan).
- Pass through initial query_id for clickhouse-benchmark (previously if you run remote query via
clickhouse-benchmark
, queries on shards will not be linked to the initial query viainitial_query_id
). #29364 (Azat Khuzhin). - Allow to write number in query as binary literal. Example
SELECT 0b001;
. #29304 (Maksim Kita). - Primary key index and partition filter can work in tuple. #29281 (凌涛).
- Skip indexes
tokenbf_v1
andngrambf_v1
added support for Array data type with key of String of FixedString type. #29280 (Maksim Kita). - Function
has
added support for map data type. #29267 (Maksim Kita). - Add
compress_logs
settings for clickhouse-keeper which allow to compress clickhouse-keeper logs inZSTD
. Implements: #26977. #29223 (alesapin). - Skip indexes
tokenbf_v1
andngrambf_v1
added support for Map data type with key of String of FixedString type. Author @lingtaolf. #29220 (Maksim Kita). - Add ability to disable converting expressions to local filter for external queries (
external_table_strict_query=true
). #29206 (Azat Khuzhin). - Allow to asynchronous reads for remote fs disks. #29205 (Kseniia Sumarokova).
- Add CORS support. Add headers for preflight options request. Closes #18693. #29155 (Filatenkov Artur).
- Added an ability to execute more merges and mutations than the number of threads in background pool. Merges and mutations will be executed step by step according to their sizes (lower is more prioritized). The ratio of the number of tasks to threads to execute is controlled by a setting
background_merges_mutations_concurrency_ratio
, 2 by default. #29140 (Nikita Mikhaylov). - Disable projections when
ARRAY JOIN
is used. Currently projection analysis will break array join aliases. #29139 (Amos Bird). - Fix possible LogicalError, support more types in MsgPack input/output format. #29077 (Kruglov Pavel).
- Allow to input and output LowCardinality columns in ORC input/output format. #29062 (Kruglov Pavel).
- Select from
system.distributed_ddl_queue
might show incorrect values, it's fixed. #29061 (tavplubix). - Correct behaviour with unknown methods for http connection. Solves #29050. #29057 (Filatenkov Artur).
- Fix bug in
clickhouse-keeper-converter
which can lead to some data loss while restoring from ZooKeeper logs (not snapshot). #29030 (小路). - Apply settings from CREATE AS SELECT queries (fixes: #28810). #28962 (Azat Khuzhin).
- add default database setting for alter table ... on cluster ... replace/move partition from/to ... #28955 (anneji-dev).
- gRPC: Allow change server-side compression from client. #28953 (Vitaly Baranov).
- Add time dependency for clickhouse-server unit (systemd and sysvinit init). #28891 (Azat Khuzhin).
- Skip "no data" exception when reading thermal sensors for asynchronous metrics. This closes #28852. #28882 (alexey-milovidov).
- Fixed race condition that might cause
Dictionary not found
error for existing dictionary in rare cases. #28853 (tavplubix). - Relax nested function for If-combinator check (but forbid nested identical combinators). #28828 (Azat Khuzhin).
- Fix uncaught exception during server termination. #28761 (Azat Khuzhin).
- Forbid cleaning of tmp directories that can be used by an active mutation/merge. #28760 (Azat Khuzhin).
- Allow optimization :
optimize_arithmetic_operations_in_aggregate_functions = 1
when alias is used. I cannot figure out why it was disabled. We set aliases in all other TreeOptimizers, except this one. Maybe I'm missing something important) @4ertus2 could you give a bad example?. #28746 (Amos Bird). - Implement
detach_not_byte_identical_parts
setting forReplicatedMergeTree
, that will detach instead of remove not byte-identical parts (after mege/mutate). #28708 (Azat Khuzhin). - Implement
max_suspicious_broken_parts_bytes
setting forMergeTree
(to limit total size of all broken parts, default is1GiB
). #28707 (Azat Khuzhin). - This PR enables expanding macros in RabbitMQ settings, for example:. #28683 (Vitaly Baranov).
- Resolve table dependencies on metadata loading. Closes #8004, closes #15170. #28373 (tavplubix).
- Speed up part loading process with multiple disks are in use. The idea is similar to https://github.com/ClickHouse/ClickHouse/pull/16423 . Prod env shows improvement: 24 min -> 16 min . #28363 (Amos Bird).
- Avoid error "Division by zero" when denominator is Nullable in functions
divide
,intDiv
andmodulo
. Closes #22621. #28352 (Kruglov Pavel). - Reload stacktrace cache when symbol is reloaded. #28137 (Amos Bird).
- This PR allows reading data of a table using the
Log
engine in multiple threads. #28125 (Vitaly Baranov). - Fix misbehavior of NULL column handling in json functions. This fixes #27930. #28012 (Amos Bird).
- Add dedicated Mark/Uncompressed cache for skip indices. Not sure how to test it properly. #27961 (Amos Bird).
- Correct
Codec(AES_128_GCM_SIV)
. #27476 (Filatenkov Artur). - Add compression for
INTO OUTPUT
andFROM INFILE
with additional optional parameter. #27135 (Filatenkov Artur).
Bug Fix
- Fix reading from empty file on encrypted disk. #30494 (Vitaly Baranov).
- Fix releasing query ID and session ID at the end of query processing while handing gRPC call. This PR fixes flaky test test_grpc_protocol/test.py::test_session. #29954 (Vitaly Baranov).
- Fix shutdown of
AccessControlManager
. Now there can't be reloading of the configuration after AccessControlManager has been destroyed. This PR fixes the flaky test test_user_directories/test.py::test_relative_path. #29951 (Vitaly Baranov). - Fix bug in
clickhouse-keeper-converter
which can lead to incorrect ZooKeeper log deserialization. #29071 (小路). - Fix transformation of disjunctions chain to
IN
(controlled by settingsoptimize_min_equality_disjunction_chain_length
) in distributed queries with settingslegacy_column_name_of_tuple_literal = 0
. #28658 (Anton Popov). - Allow using a materialized column as the sharding key in a distributed table even if
insert_allow_materialized_columns=0
:. #28637 (Vitaly Baranov).
Build/Testing/Packaging Improvement
- Recursive submodules are no longer needed for ClickHouse. #30315 (alexey-milovidov).
- ClickHouse can be statically built with Musl. This is added as experiment, it does not support building
odbc-bridge
,library-bridge
, integration with CatBoost and some libraries. #30248 (alexey-milovidov). - Add tests for encrypted disk & encryption codec and ReplicatedMergeTree. #30172 (Vitaly Baranov).
- Enable Protobuf, Arrow, ORC, Parquet for AArch64 and Darwin builds. This closes #29248. This closes #28018. #30015 (alexey-milovidov).
- Add cross-build for PowerPC (powerpc64le). This closes #9589. Enable support for interaction with MySQL for AArch64 and PowerPC. This closes #26301. #30010 (alexey-milovidov).
- Leave only required files in cross-compile toolchains. Include them as submodules (earlier they were downloaded as tarballs). #29974 (alexey-milovidov).
- Fix an build error because of Rename "common" to "base". #29688 (Sergei Semin).
- Turning on experimental constexpr expressions evaluator for clang to speed up template code compilation. #29668 (myrrc).
- Add ability to compile using newer version fo glibc w/o using new symbols. #29594 (Azat Khuzhin).
- N/A Detailed description / Documentation draft:. #29367 (kevin wan).
- Add a test for https://github.com/ClickHouse/ClickHouse/pull/28683. #28934 (Vitaly Baranov).
- Convert
skip_list .json
into first line comments. #28909 (Vitaly Baranov). - Reduce Debug build binary size by clang optimization. #28736 (flynn).
- Now all images for CI will be placed in the separate dockerhub repo. #28656 (alesapin).
- Improve support for build with clang-13. #28046 (Sergei Semin).
- Switch clang builds to 12 version. #20495 (Ilya Yatsishin).
Bug Fix (user-visible misbehaviour in official stable or prestable release
- Fix
ORDER BY ... WITH FILL
with setTO
andFROM
and no rows in result set. #30888 (Anton Popov).
Bug Fix (user-visible misbehaviour in official stable or prestable release)
- Fix set index not used in AND/OR expressions when there are more than two operands. This fixes #30416 . #30887 (Amos Bird).
- Fix crash when projection with hashing function is materialized. This fixes #30861 . The issue is similar to https://github.com/ClickHouse/ClickHouse/pull/28560 which is a lack of proper understanding of the invariant of header's emptyness. #30877 (Amos Bird).
- Fixed ambiguity when extracting auxiliary ZooKeeper name from ZooKeeper path in
ReplicatedMergeTree
. Previously server might fail to start withUnknown auxiliary ZooKeeper name
if ZooKeeper path contains a colon. Fixes #29052. Also it was allowed to specify ZooKeeper path that does not start with slash, but now it's deprecated and creation of new tables with such path is not allowed. Slashes and colons in auxiliary ZooKeeper names are not allowed too. #30822 (tavplubix). - Clean temporary directory when localBackup failed by some reason. #30797 (ianton-ru).
- Fixed a race condition between
REPLACE/MOVE PARTITION
and background merge in non-replicatedMergeTree
that might cause a part of moved/replaced data to remain in partition. Fixes #29327. #30717 (tavplubix). - Fix PREWHERE with WHERE in case of always true PREWHERE. #30668 (Azat Khuzhin).
- Functions for case-insensitive search in UTF8 strings like
positionCaseInsensitiveUTF8
andcountSubstringsCaseInsensitiveUTF8
might find substrings that actually does not match, it's fixed. #30663 (tavplubix). - Update aws-sdk submodule for throttling in Yandex.S3. #30646 (ianton-ru).
- Limit push down optimization could cause a error
Cannot find column
. Fixes #30438. #30562 (Nikolai Kochetov). - Add missing parenthesis for
isNotNull
/isNull
rewrites toIS [NOT] NULL
(fixes queries that has something likeisNotNull(1)+isNotNull(2)
). #30520 (Azat Khuzhin). -
- Fix deadlock on ALTER with scalar subquery to the same table, close #30461. #30492 (Vladimir C).
- Fixed segfault which might happen if session expired during execution of REPLACE PARTITION. #30432 (tavplubix).
- Makes #30162 less possible ... #30370 (Vasily Nemkov).
- Queries with condition like
IN (subquery)
could return incorrect result in case if aggregate projection applied. Fixed creation of sets for projections. #30310 (Amos Bird). - Fix column alias resolution of JOIN queries when projection is enabled. This fixes #30146. #30293 (Amos Bird).
- fix replaceRegexpAll bug. #30292 (Memo).
- Fix ComplexKeyHashedDictionary, ComplexKeySparseHashedDictionary parsing
preallocate
option from layout config. #30246 (Maksim Kita). - Fix
[I]LIKE
function. Closes #28661. #30244 (Nikolay Degterinsky). - Fix crash with shortcircuit and lowcardinality in multiIf. #30243 (Raúl Marín).
- FlatDictionary, HashedDictionary fix bytes_allocated calculation for nullable attributes. #30238 (Maksim Kita).
-
- Allow identifiers staring with numbers in multiple joins. #30230 (Vladimir C).
- Fix reading from
MergeTree
withmax_read_buffer_size=0
(can lead toCan't adjust last granule
LOGICAL_ERROR
, or even data loss). #30192 (Azat Khuzhin). - Fix
pread_fake_async
/pread_threadpool
withmin_bytes_to_use_direct_io
. #30191 (Azat Khuzhin). - Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column. #30189 (Azat Khuzhin).
- Support nullable arguments in function
initializeAggregation
. #30177 (Anton Popov). - Fix error
Port is already connected
for queries withGLOBAL IN
andWITH TOTALS
. Only for 21.9 and 21.10. #30086 (Nikolai Kochetov). - Fix race between MOVE PARTITION and merges/mutations for MergeTree. #30074 (Azat Khuzhin).
- Dropped
Memory
database might reappear after server restart, it's fixed (#29795). Also addedforce_remove_data_recursively_on_drop
setting as a workaround forDirectory not empty
error when droppingOrdinary
database (because it's not possible to remove data leftovers manually in cloud environment). #30054 (tavplubix). - Fix crash of sample by
tuple()
, closes #30004. #30016 (flynn). - try to close issue: #29965. #29976 (hexiaoting).
- Fix possible data-race between
FileChecker
andStorageLog
/StorageStripeLog
. #29959 (Azat Khuzhin). - Fix data-race between
LogSink::writeMarks()
andLogSource
inStorageLog
. #29946 (Azat Khuzhin). - Fix potential resource leak of the concurrent query limit of merge tree tables introduced in https://github.com/ClickHouse/ClickHouse/pull/19544 . #29879 (Amos Bird).
- Fix system tables recreation check (fails to detect changes in enum values). #29857 (Azat Khuzhin).
- MaterializedMySQL: Fix an issue where if the connection to MySQL was lost, only parts of a transaction could be processed. #29837 (Håvard Kvålen).
- Avoid
Timeout exceeded: elapsed 18446744073.709553 seconds
error that might happen in extremely rare cases, presumably due to some bug in kernel. Fixes #29154. #29811 (tavplubix). - Fix bad cast in
ATTACH TABLE ... FROM 'path'
query when non-string literal is used instead of path. It may lead to reading of uninitialized memory. #29790 (alexey-milovidov). - Fix concurrent access to
LowCardinality
duringGROUP BY
(leads to SIGSEGV). #29782 (Azat Khuzhin). - Fix incorrect
GROUP BY
(multiple rows with the same keys in result) in case of distributed query when shards had mixed versions<= 21.3
and>= 21.4
,GROUP BY
key had several columns all with fixed size, and two-level aggregation was activated (seegroup_by_two_level_threshold
andgroup_by_two_level_threshold_bytes
). Fixes #29580. #29735 (Nikolai Kochetov). - Fixed incorrect behaviour of setting
materialized_postgresql_tables_list
at server restart. Found in #28529. #29686 (Kseniia Sumarokova). - Condition in filter predicate could be lost after push-down optimisation. #29625 (Nikolai Kochetov).
- Fix JIT expression compilation with aliases and short-circuit expression evaluation. Closes #29403. #29574 (Maksim Kita).
- Fix rare segfault in
ALTER MODIFY
query when using incorrect table identifier inDEFAULT
expression likex.y.z...
Fixes #29184. #29573 (alesapin). - Fix null deference for
GROUP BY WITH TOTALS HAVING
(when the column fromHAVING
wasn't selected). #29553 (Azat Khuzhin). - Avoid deadlocks when reading and writting on JOIN Engine tables at the same time. #29544 (Raúl Marín).
- Fix bug in check
pathStartsWith
becuase there was bug with the usage ofstd::mismatch
:The behavior is undefined if the second range is shorter than the first range.
. #29531 (Kseniia Sumarokova). - In ODBC bridge add retries for error Invalid cursor state. It is a retriable error. Closes #29473. #29518 (Kseniia Sumarokova).
- Fixed incorrect table name parsing on loading of
Lazy
database. Fixes #29456. #29476 (tavplubix). - Fix possible
Block structure mismatch
for subqueries with pushed-downHAVING
predicate. Fixes #29010. #29475 (Nikolai Kochetov). - Fix Logical error
Cannot capture columns
in functions greatest/least. Closes #29334. #29454 (Kruglov Pavel). - rocksdb: fix race condition during multiple DB opening (and get back some tests that triggers the problem on CI). #29393 (Azat Khuzhin).
- Fix replicated access storage not shutting down cleanly when misconfigured. #29388 (Kevin Michel).
- Remove window function
nth_value
as it is not memory-safe. This closes #29347. #29348 (alexey-milovidov). - Fix vertical merges of projection parts. This fixes #29253 . This PR also fixes several projection merge/mutation issues introduced in https://github.com/ClickHouse/ClickHouse/pull/25165. #29337 (Amos Bird).
- Fix hanging DDL queries on Replicated database while adding a new replica. #29328 (Kevin Michel).
- Fix connection timeouts (
send_timeout
/receive_timeout
). #29282 (Azat Khuzhin). - Fix failed assertion in ReadBufferFromHDFS. Update libhdfs3 library to be able to run in tests in debug. Closes #29251. Closes #27814. #29276 (Kseniia Sumarokova).
- Fix possible
Table columns structure in ZooKeeper is different from local table structure
exception while recreating or creating new replicas ofReplicatedMergeTree
, when one of table columns have default expressions with case-insensitive functions. #29266 (Anton Popov). - Send normal
Database doesn't exist error
(UNKNOWN_DATABASE
) to the client (via TCP) instead ofAttempt to read after eof
(ATTEMPT_TO_READ_AFTER_EOF
). #29229 (Azat Khuzhin). - Fix segfault while inserting into column with type LowCardinality(Nullable) in Avro input format. #29132 (Kruglov Pavel).
- Do not allow to reuse previous credentials in case of inter-server secret (Before INSERT via Buffer/Kafka to Distributed table with interserver secret configured for that cluster, may re-use previously set user for that connection). #29060 (Azat Khuzhin).
- Handle
any_join_distinct_right_table_keys
when join with dictionary, close #29007. #29014 (Vladimir C). - Fix "Not found column ... in block" error, when join on alias column, close #26980. #29008 (Vladimir C).
- Fix the number of threads used in
GLOBAL IN
subquery (it was executed in single threads since #19414 bugfix). #28997 (Nikolai Kochetov). - Fix bad optimizations of ORDER BY if it contains WITH FILL. This closes #28908. This closes #26049. #28910 (alexey-milovidov).
- Fix higher-order array functions (
SIGSEGV
forarrayCompact
/ILLEGAL_COLUMN
forarrayDifference
/arrayCumSumNonNegative
) with consts. #28904 (Azat Khuzhin). - Fix waiting for mutation with
mutations_sync=2
. #28889 (Azat Khuzhin). - Fix queries to external databases (i.e. MySQL) with multiple columns in IN ( i.e.
(k,v) IN ((1, 2))
) (but note that this has some backward incompatibility for theclickhouse-copier
since it uses alias for tuple element). #28888 (Azat Khuzhin). - Fix bug with LowCardinality in short-curcuit function evaluation. Closes #28884. #28887 (Kruglov Pavel).
- Fix reading of subcolumns from compact parts. #28873 (Anton Popov).
- Fixed a race condition between
DROP PART
andREPLACE/MOVE PARTITION
that might cause replicas to diverge in rare cases. #28864 (tavplubix). - Fix expressions compilation with short circuit evaluation. #28821 (Azat Khuzhin).
- Fix extremely rare case when ReplicatedMergeTree replicas can diverge after hard reboot of all replicas. The error looks like
Part ... intersects (previous|next) part ...
. #28817 (alesapin). - Better check for connection usability and also catch any exception in RabbitMQ shutdown just in case. #28797 (Kseniia Sumarokova).
- Fix benign race condition in ReplicatedMergeTreeQueue. Shouldn't be visible for user, but can lead to subtle bugs. #28734 (alesapin).
- Fix possible crash for
SELECT
with partially created aggregate projection in case of exception. #28700 (Amos Bird). - Fix the coredump in the creation of distributed tables, when the parameters passed in are wrong. #28686 (Zhiyong Wang).
- Add Settings.Names, Settings.Values aliases for system.processes table. #28685 (Vitaly).
-
- Fix the number of arguments required by s2RectAdd and s2RectContains functions. #28663 (Bharat Nallan).
- Fix invalid constant type conversion when nullable or lowcardinality primary key is used. #28636 (Amos Bird).
- Fix "Column is not under aggregate function and not in GROUP BY" with PREWHERE (Fixes: #28461). #28502 (Azat Khuzhin).
NO CL ENTRY
- NO CL ENTRY: 'remove some unneeded header files'. #30722 (flynn).
- NO CL ENTRY: 'Revert "Revert "Improve usability of
remote_url_allow_hosts
""'. #30708 (Nikolai Kochetov). - NO CL ENTRY: 'Revert "Improve usability of
remote_url_allow_hosts
"'. #30707 (Nikolai Kochetov). - NO CL ENTRY: 'Revert "Fix style regressions on benchmark page"'. #30652 (alesapin).
- NO CL ENTRY: 'add support of window function in antlr grammar'. #30181 (PHaroZ).
- NO CL ENTRY: 'Link FAQ from Debian installation'. #29836 (Federico Ceratto).
- NO CL ENTRY: 'Revert "Add coroutines example."'. #29829 (Nikolai Kochetov).
- NO CL ENTRY: 'Generate test_results.tsv for fasttest'. #29319 (Dmitry Novik).
- NO CL ENTRY: 'Update rounding-functions.md'. #29249 (quoctan132).