mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-05 22:12:21 +00:00
34 KiB
34 KiB
ClickHouse release v20.11.2.1-stable FIXME as compared to v20.10.1.4881-prestable
Backward Incompatible Change
- Make rankCorr function return nan on insufficient data #16124. #16135 (hexiaoting).
- Aggregate functions
boundingRatio
,rankCorr
,retention
,timeSeriesGroupSum
,timeSeriesGroupRateSum
,windowFunnel
were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. #16407 (Alexey Milovidov). - Remove
ANALYZE
andAST
queries, and make the settingenable_debug_queries
obsolete since now it is the part of full featuredEXPLAIN
query. #16536 (Ivan). - Restrict to use of non-comparable data types (like
AggregateFunction
) in keys (Sorting key, Primary key, Partition key, and so on). #16601 (alesapin). - If some
profile
was specified indistributed_ddl
config section, then this profile could overwrite settings ofdefault
profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. #16635 (Alexander Tokmakov).
New Feature
- #WelchTTest aggregate function implementation. #10351 (antikvist).
- New functions
encrypt
,aes_encrypt_mysql
,decrypt
,aes_decrypt_mysql
. These functions are working slowly (below ClickHouse standards), so we consider it as an experimental feature. #11844 (Vasily Nemkov). -
- Added support of LDAP as a user directory for locally non-existent users. #12736 (Denis Glazachev).
- Added
disable_merges
option for volumes in multi-disk configuration. #13956 (Vladimir Chebotarev). - Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the
system.opentelemetry_span_log
table. #14195 (Alexander Kuzmenkov). - Allows to read/write Single protobuf message at once (w/o length-delimiters). #15199 (filimonov).
- Add function
formatReadableTimeDelta
that format time delta to human readable string ... #15497 (Filipe Caixeta). - Add
tid
andlogTrace
function. This closes #9434. #15803 (flynn). - Add a new option
print_query_id
to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. #15809 (Amos Bird). - Allow specify primary key in column list of CREATE TABLE query. #15823 (Maksim Kita).
- Added setting date_time_output_format. #15845 (Maksim Kita).
- Implement
OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES}
in select Query with order by. related issue:#15367. #15855 (hexiaoting). - Added an aggregate function, which calculates the p-value used for Welch's t-test. #15874 (Nikita Mikhaylov).
- Add max_concurrent_queries_for_all_users setting, see #6636 for use cases. #16154 (nvartolomei).
- Added minimal web UI to ClickHouse. #16158 (Alexey Milovidov).
- Added function
untuple
which is a special function which can introduce new columns to the SELECT list by flattening a named tuple. #16242 (Nikolai Kochetov). - Added toUUIDOrNull, toUUIDOrZero cast functions. #16337 (Maksim Kita).
- Add
system.replicated_fetches
table which shows currently running background fetches. #16428 (alesapin). -
errorCodeToName()
function - return variable name of the error (useful for analyzing query_log and similar) -system.errors
table - shows how many times errors has been happened (respectssystem_events_show_zero_values
). #16438 (Azat Khuzhin).
- Ability to create a docker image on the top of alpine. Uses precompiled binary and glibc components from ubuntu 20.04. #16479 (filimonov).
- Add
log_queries_min_query_duration_ms
, only queries slower then the value of this setting will go toquery_log
/query_thread_log
(i.e. something likeslow_query_log
in mysql). #16529 (Azat Khuzhin). -
Add farmFingerprint64 function. #16570 (Jacob Hayes).
- Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. #3815. #16594 (Amos Bird).
- Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. #16642 (Maksim Kita).
Performance Improvement
- Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. #15454 (Azat Khuzhin).
- Do not merge parts across partitions in SELECT FINAL. #15938 (Kruglov Pavel).
- Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. #16191 (alesapin).
- Improve performance of logical functions a little. #16347 (Alexey Milovidov).
- Improve performance of
quantileMerge
. In previous versions it was obnoxiously slow. This closes #1463. #16643 (Alexey Milovidov). - Improve performance of
-OrNull
and-OrDefault
aggregate functions. #16661 (Alexey Milovidov).
Improvement
- Allow explicitly specify columns list in
CREATE TABLE table AS table_function(...)
query. Fixes #9249 Fixes #14214. #14295 (Alexander Tokmakov). - Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes #11092 which returns wrong count when
max_parallel_replicas > 1
. #15074 (Amos Bird). - Enable parsing enum values by their ids for CSV, TSV and JSON input formats. #15685 (vivarum).
- Add reconnects to
zookeeper-dump-tree
tool. #15711 (Alexey Milovidov). - Remove
MemoryTrackingInBackground*
metrics to avoid potentially misleading results. This fixes #15684. #15813 (Alexey Milovidov). - Change level of some log messages from information to debug, so information messages will not appear for every query. This closes #5293. #15816 (Alexey Milovidov).
- Fix query hang (endless loop) in case of misconfiguration (
connections_with_failover_max_tries
set to 0). #15876 (Azat Khuzhin). - Added boost::program_options to
db_generator
in order to increase its usability. This closes #15940. #15973 (Nikita Mikhaylov). - Treat
INTERVAL '1 hour'
as equivalent toINTERVAL 1 HOUR
, to be compatible with Postgres. This fixes #15637. #15978 (flynn). - Add support of cache layout for Redis dictionaries with complex key. #15985 (Anton Popov).
- Fix rare issue when clickhouse-client may abort on exit due to loading of suggestions. This fixes #16035. #16047 (Alexey Milovidov).
- Now it's allowed to execute
ALTER ... ON CLUSTER
queries regardless of the<internal_replication>
setting in cluster config. #16075 (alesapin). -
- Fix memory_profiler_step/max_untracked_memory for queries via HTTP (test included), and adjusting this value globally in xml config will not help either, since those settings are not applied anyway, only default (4MB) value is used. - Fix query_id for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). #16101 (Azat Khuzhin).
- Add allow_nondeterministic_optimize_skip_unused_shards (to allow non deterministic like rand() or dictGet() in sharding key). #16105 (Azat Khuzhin).
- database_atomic_wait_for_drop_and_detach_synchronously/NO DELAY/SYNC for DROP DATABASE. #16127 (Azat Khuzhin).
- Add support for nested data types (like named tuple) as sub-types. Fixes #15587. #16262 (Ivan).
- If there are no tmp folder in the system (chroot, misconfigutation etc) clickhouse-local will create temporary subfolder in the current directory. #16280 (filimonov).
- Now it's possible to specify
PRIMARY KEY
withoutORDER BY
for MergeTree table engines family. Closes #15591. #16284 (alesapin). - try use cmake version for croaring instead of amalgamation.sh. #16285 (sundyli).
- Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. #16306 (Azat Khuzhin).
- Better diagnostics when client has dropped connection. In previous versions,
Attempt to read after EOF
andBroken pipe
exceptions were logged in server. In new version, it's information messageClient has dropped the connection, cancel the query.
. #16329 (Alexey Milovidov). - Add
TablesToDropQueueSize
metric. It's equal to number of dropped tables, that are waiting for background data removal. #16364 (Alexander Tokmakov). - Fix debug assertion in quantileDeterministic function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes #15683. #16410 (Alexey Milovidov).
- Better read task scheduling for JBOD architecture and
MergeTree
storage. New settingread_backoff_min_concurrency
which serves as the lower limit to the number of reading threads. #16423 (Amos Bird). - Fixed bug for #16263. Also minimized event loop lifetime. Added more efficient queues setup. #16426 (Kseniia Sumarokova).
- Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. #16538 (Amos Bird).
- Add current_database into query_thread_log. #16558 (Azat Khuzhin).
- Subqueries in WITH section (CTE) can reference previous subqueries in WITH section by their name. #16575 (Amos Bird).
-
- Improve scheduling of background task which removes data of dropped tables in
Atomic
databases. -Atomic
databases do not create broken symlink to table data directory if table actually has no data directory. #16584 (Alexander Tokmakov).
- Improve scheduling of background task which removes data of dropped tables in
- Now paratmeterized functions can be used in APPLY column transformer. #16589 (Amos Bird).
- Now
event_time_microseconds
field stores in Decimal64, not UInt64. Removed an incorrect check from Field::get(). #16617 (Nikita Mikhaylov). - Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes #3178. #16619 (Alexey Milovidov).
- Better update of ZooKeeper configuration in runtime. #16630 (sundyli).
- Make the behaviour of
minMap
andmaxMap
more desireable. It will not skip zero values in the result. Fixes #16087. #16631 (Ildus Kurbangaliev). - Better diagnostics on parse errors in input data. Provide row number on
Cannot read all data
errors. #16644 (Alexey Milovidov). - Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like http://domain.com?delete, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like http://domain.com/?delete. #16814 (ianton-ru).
Bug Fix
- Update jemalloc to fix percpu_arena with affinity mask. #15035 (Azat Khuzhin).
- Decrement the
ReadonlyReplica
metric when detaching read-only tables. This fixes #15598. #15592 (sundyli). - Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
- Fix error
Cannot add simple transform to empty Pipe
which happened while reading fromBuffer
table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov). - Possibility to move part to another disk/volume if the first attempt was failed. #15723 (Pavel Kovalenko).
- Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). #15743 (Azat Khuzhin).
- Fix
select count()
inaccuracy for MaterializeMySQL. #15767 (Alexander Tokmakov). - Fix exception
Block structure mismatch
inSELECT ... ORDER BY DESC
queries which were executed afterALTER MODIFY COLUMN
query. Fixes #15800. #15852 (alesapin). - Now exception will be thrown when
ALTER MODIFY COLUMN ... DEFAULT ...
has incompatible default with column type. Fixes #15854. #15858 (alesapin). - Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
- fixes #12513 fix difference expressions with same alias when analyze queries again. #15886 (Winter Zhang).
- Fix incorrect empty result for query from
Distributed
table if query hasWHERE
,PREWHERE
andGLOBAL IN
. Fixes #15792. #15933 (Nikolai Kochetov). - Fixed
DROP TABLE IF EXISTS
failure withTable ... doesn't exist
error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (likeDROP DATABASE
andRENAME TABLE
) FixedDROP/DETACH DATABASE
failure withTable ... doesn't exist
when concurrently executingDROP/DETACH TABLE
. #15934 (Alexander Tokmakov). - Fix a crash when database creation fails. #15954 (Winter Zhang).
- Fix ambiguity in parsing of settings profiles:
CREATE USER ... SETTINGS profile readonly
is now considered as using a profile namedreadonly
, not a setting namedprofile
with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov). - Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (Alexander Tokmakov).
- Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
- Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
- Fix collate name & charset name parser and support
length = 0
for string type. #16008 (Winter Zhang). - Fix
ALTER MODIFY ... ORDER BY
query hang forReplicatedVersionedCollapsingMergeTree
. This fixes #15980. #16011 (alesapin). - Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query
SELECT ... FROM system.parts
should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel). - Fixes #15780 regression, e.g. indexOf([1, 2, 3], toLowCardinality(1)) now is prohibited but it should not be. #16038 (Mike Kot).
- Fix segfault in some cases of wrong aggregation in lambdas. #16082 (Anton Popov).
- Fix the
clickhouse-local
crash when trying to doOPTIMIZE
command. Fixes #16076. #16192 (filimonov). - Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). #16205 (Azat Khuzhin).
- Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (Alexey Milovidov).
- Fix a possible memory leak during
GROUP BY
with string keys, caused by an error inTwoLevelStringHashTable
implementation. #16264 (Amos Bird). - Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
- Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (Alexey Milovidov).
- Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
- Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
- Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
- Fix double free in case of exception in function
dictGet
. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov). - Specifically crafter argument of
round
function withDecimal
was leading to integer division by zero. This fixes #13338. #16451 (Alexey Milovidov). - Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and
slave_parallel_worker
enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason). - Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike Kot).
- Fix query_thread_log.query_duration_ms unit. #16563 (Azat Khuzhin).
- Calculation of
DEFAULT
expressions was involving possible name collisions (that was very unlikely to encounter). This fixes #9359. #16612 (Alexey Milovidov). - Backported in #16750: Fixed #16081. #16613 (Nikita Mikhaylov).
- Backported in #16758: This will fix optimize_read_in_order/optimize_aggregation_in_order with max_threads>0 and expression in ORDER BY. #16637 (Azat Khuzhin).
- The setting
max_parallel_replicas
worked incorrectly if the queried table has no sampling. This fixes #5733. #16675 (Alexey Milovidov). - Backported in #16739: Fix
IN
operator over several columns and tuples with enabledtransform_null_in
setting. Fixes #15310. #16722 (Anton Popov). - Backported in #16784: Mask password in data_path in the system.distribution_queue. #16727 (Azat Khuzhin).
- Backported in #16785: Not for changelog. #16757 (Alexander Tokmakov).
Build/Testing/Packaging Improvement
- Simplify Sys/V init script. #14135 (Alexey Milovidov).
- Fix illegal code style
&vector[idx]
in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . #15815 (Amos Bird). - Check for
#pragma once
in headers. #15818 (Alexey Milovidov). - Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. #15843 (Alexey Milovidov).
- Refuse to build with AppleClang because it's difficult to find out version correspondence. This closes #16072. #16074 (Alexey Milovidov).
- Add scipy to fasttest docker. #16093 (Nikita Mikhaylov).
- Add MySQL BinLog file check tool for MaterializeMySQL database engine. #16223 (Winter Zhang).
- Add flaky check for stateless tests. #16238 (alesapin).
- Converting test tests/queries/0_stateless/01446_json_strings_each_row to a shell script. #16247 (vzakaznikov).
-
- None. #16249 (Denis Glazachev).
- Fixing fails in LDAP external user directory tests. #16363 (vzakaznikov).
- During config removal wrong exit code was expected. #16365 (vzakaznikov).
- Fix LDAP tests by grabbing log size after container is stopped. #16440 (vzakaznikov).
- Improve generation of build files for
ya.make
build system (Arcadia). #16700 (Alexey Milovidov).
Other
- Use only |name_parts| as primary name source and auto-generate full name. #16149 (Ivan).
- Rename struct NullSink from ReadHelpers to NullOutput, because class NullSink exists in Processors/NullSink.h. It's needed to prevent redefinition of 'NullSink' error. #16520 (Kruglov Pavel).
NO CL CATEGORY
- Try to make MergeTreeWriteAheadLog forward compatible. #16094 (nvartolomei).
NO CL ENTRY
- NO CL ENTRY: 'Revert "Write structure of table functions to metadata"'. #15961 (Alexander Tokmakov).
- NO CL ENTRY: 'Revert "scipy"'. #16156 (Nikita Mikhaylov).
- NO CL ENTRY: 'Bump markdown from 3.2.1 to 3.3.2 in /docs/tools'. #16180 (dependabot-preview[bot]).
- NO CL ENTRY: 'Revert "Optionally upload clickhouse binary in fast test"'. #16333 (Alexey Milovidov).
- NO CL ENTRY: 'minor fix.'. #16335 (Xianda Ke).
- NO CL ENTRY: 'Bump tornado from 5.1.1 to 6.1 in /docs/tools'. #16590 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.17 to 0.4.20 in /docs/tools'. #16692 (dependabot-preview[bot]).