mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-13 01:41:59 +00:00
27 KiB
27 KiB
ClickHouse release v20.8.1.4513-prestable FIXME as compared to v20.7.1.4310-prestable
Backward Incompatible Change
- Remove support for the
ODBCDriver
input/output format. This was a deprecated format once used for communication with the ClickHouse ODBC driver, now long superseded by theODBCDriver2
format. Resolves #13629. #13847 (hexiaoting). - Extend
parallel_distributed_insert_select
setting, adding an option to runINSERT
into local table. The setting changes type fromBool
toUInt64
, so the valuesfalse
andtrue
are no longer supported. If you have these values in server configuration, the server will not start. Please replace them with0
and1
, respectively. #14060 (Azat Khuzhin). - Now
OPTIMIZE FINAL
query doesn't recalculate TTL for parts that were added before TTL was created. UseALTER TABLE ... MATERIALIZE TTL
once to calculate them, after thatOPTIMIZE FINAL
will evaluate TTL's properly. This behavior never worked for replicated tables. #14220 (alesapin).
New Feature
- Support
MaterializeMySQL
database engine. Implements #4006. #10851 (Winter Zhang). - Support Kerberos authentication in Kafka, using
krb5
andcyrus-sasl
libraries. #12771 (Ilya Golshtein). - Add types
Int128
,Int256
,UInt256
and related functions for them. Extend Decimals with Decimal256 (precision up to 76 digits). New types are under the settingallow_experimental_bigint_types
. #13097 (Artem Zuikov). - Function
position
now supports optionalstart_pos
argument. #13237 (Vladimir C). - Add
ALTER SAMPLE BY
statement that allows to change table sample clause. #13280 (Amos Bird). - Add new optional section <user_directories> to the main config. #13425 (Vitaly Baranov).
- Add function
normalizeQuery
that replaces literals, sequences of literals and complex aliases with placeholders. Add functionnormalizedQueryHash
that returns identical 64bit hash values for similar queries. It helps to analyze query log. This closes #11271. #13816 (Alexey Milovidov). - Add
quantileExactLow
andquantileExactHigh
implementations with respective aliases formedianExactLow
andmedianExactHigh
. #13818 (Bharat Nallan). - Add function
defaultValueOfTypeName
that returns the default value for a given type. #13877 (hcz). - Added
date_trunc
function that truncates a date/time value to a specified date/time part. #13888 (vladimir-golovchenko). - Add the ability to specify
Default
compression codec for columns that correspond to settings specified inconfig.xml
. Implements: #9074. #14049 (alesapin). - Add setting
min_index_granularity_bytes
that protects against accidentally creating a table with very lowindex_granularity_bytes
setting. #14139 (Bharat Nallan). - Add
countDigits(x)
function that count number of decimal digits in integer or decimal column. AddisDecimalOverflow(d, [p])
function that checks if the value in Decimal column is out of its (or specified) precision. #14151 (Artem Zuikov).
Performance Improvement
- When performing trivial
INSERT SELECT
queries, automatically setmax_threads
to 1 ormax_insert_threads
, and setmax_block_size
tomin_insert_block_size_rows
. Related to #5907. #12195 (flynn). - Optimize
has()
,indexOf()
andcountEqual()
functions forArray(LowCardinality(T))
and constant right arguments. #12550 (Mike Kot). - Slightly improve performance of aggregation by UInt8/UInt16 keys. #13099 (Alexey Milovidov).
- Fail fast if
max_rows_to_read
limit is exceeded on parts scan. The motivation behind this change is to skip ranges scan for all selected parts if it is clear thatmax_rows_to_read
is already exceeded. The change is quite noticeable for queries over big number of parts. #13677 (Roman Khavronenko). - Enable parallel INSERTs for table engines
Null
,Memory
,Distributed
andBuffer
. #14120 (Alexey Milovidov). - Slightly optimize very short queries with LowCardinality. #14129 (Anton Popov).
Improvement
- Added cache layer for DiskS3 (cache to local disk mark and index files). #13076 (Pavel Kovalenko).
- Fix assert when decimal has too large negative exponent. Fixes #13188. #13228 (Alexey Milovidov).
- Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds to system.events. #13336 (ianton-ru).
- Proper remote host checking in S3 redirects (security-related thing). #13404 (Vladimir Chebotarev).
- Enable mixed granularity parts by default. #13449 (alesapin).
- Move parts from DIskLocal to DiskS3 in parallel. #13459 (Pavel Kovalenko).
- Support compound identifiers for custom settings. #13496 (Vitaly Baranov).
- Provide monotonicity for
toDate/toDateTime
functions in more cases. Now the input arguments are saturated more naturally and provides better monotonicity. #13497 (Amos Bird). - In previous versions
lcm
function may produce assertion violation in debug build if called with specifically crafted arguments. This fixes #13368. #13510 (Alexey Milovidov). - Add MergeTree Write-Ahead-Log(WAL) dump tool. #13640 (BohuTANG).
- Avoid too slow queries when arrays are manipulated as fields. Throw exception instead. #13753 (Alexey Milovidov).
-
- Add GTID-Based Replication, it works even when replication topology changes, and supported/prefered in MySQL 5.6/5.7/8.0 2. Add BIT/SET filed type supports 3. Fix up varchar type meta length bug. #13820 (BohuTANG).
- Fix data race in
lgamma
function. This race was caught only intsan
, no side effects a really happened. #13842 (Nikolai Kochetov). - Function
arrayCompact
will compare NaNs bitwise if the type of array elements is Float32/Float64. In previous versions NaNs were always not equal if the type of array elements is Float32/Float64 and were always equal if the type is more complex, like Nullable(Float64). This closes #13857. #13868 (Alexey Milovidov). - Better error message for null value of TabSeparatedRow format. #13906 (jiang tao).
- Fix wrong error for long queries. It was possible to get syntax error other than
Max query size exceeded
for correct query. #13928 (Nikolai Kochetov). - Corrected an error in AvroConfluent format that caused the Kafka table engine to stop processing messages when an abnormally small, malformed, message was received. #13941 (Gervasio Varela).
- Increase limit in -Resample combinator to 1M. #13947 (Mikhail f. Shiryaev).
- Conditional aggregate functions (for example:
avgIf
,sumIf
,maxIf
) should returnNULL
when miss rows and use nullable arguments. #13964 (Winter Zhang). - Slightly better performance of Memory table if it was constructed from a huge number of very small blocks (that's unlikely). Author of the idea: Mark Papadakis. Closes #14043. #14056 (Alexey Milovidov).
- Fix
toDeicmal256()
and attempt to fix potential perf regression in Decimal to Float conversion. #14110 (Artem Zuikov). - Now it's possible to
ALTER TABLE table_name FETCH PARTITION partition_expr FROM 'zk://<host>:<port>/path-in-zookeeper'
. It's useful for shipping data to new clusters. #14155 (Amos Bird).
Bug Fix
- subquery hash values are not enough to distinguish. #8333. #8367 (Amos Bird).
- Removed wrong auth access check when using ClickHouseDictionarySource to query remote tables. #12756 (sundyli).
- Fix access to redis dictionary after connection was dropped once. It may happen with
cache
anddirect
dictionary layouts. #13082 (Anton Popov). - Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes #5779, #12527. #13199 (Vitaly Baranov).
- Fix missing or excessive headers in
TSV/CSVWithNames
formats. This fixes #12504. #13343 (Azat Khuzhin). - Fix possible race in
StorageMemory
. https://clickhouse-test-reports.s3.yandex.net/0/9cac8a7244063d2092ad25d45502611e18d3749c/stress_test_(thread)/stderr.log Have no idea how to write a test. #13416 (Nikolai Kochetov). - Fix wrong code in function
netloc
. This fixes #13335. #13446 (Alexey Milovidov). - Fix premature
ON CLUSTER
timeouts for queries that must be executed on a single replica. Fixes #6704, Fixes #7228, Fixes #13361, Fixes #11884. #13450 (alesapin). - Fix secondary indices corruption in compact parts. #13538 (Anton Popov).
- Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 (Nikita Mikhaylov).
- Concurrent
ALTER ... REPLACE/MOVE PARTITION ...
queries might cause deadlock. It's fixed. #13626 (Alexander Tokmakov). - Fix typo in error message about
The value of 'number_of_free_entries_in_pool_to_lower_max_size_of_merge' setting
. #13678 (Alexey Milovidov). - Fix crash in JOIN with StorageMerge and
set enable_optimize_predicate_expression=1
. #13679 (Artem Zuikov). - Do not optimize any(arrayJoin()) -> arrayJoin() under optimize_move_functions_out_of_any. #13681 (Azat Khuzhin).
- Fix visible data clobbering by progress bar in client in interactive mode. This fixes #12562 and #13369 and #13584 and fixes #12964. #13691 (Alexey Milovidov).
- Fix incorrect message in
clickhouse-server.init
while checking user and group. #13711 (ylchou). - Fixes /replicas_status endpoint response status code when verbose=1. #13722 (javi santana).
- Fix logging Settings.Names/Values when log_queries_min_type > QUERY_START. #13737 (Azat Khuzhin).
- Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of #8602 that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. #13746 (Alexey Milovidov).
- Add range check for h3KRing function. This fixes #13633. #13752 (Alexey Milovidov).
- Fixed
Directory not empty
error when concurrently executingDROP DATABASE
andCREATE TABLE
. #13756 (Alexey Milovidov). - Fix step overflow in range(). #13790 (Azat Khuzhin).
- Fix reading from MergeTree table with INDEX of type SET fails when comparing against NULL. This fixes #13686. #13793 (Amos Bird).
- Fix topK/topKWeighted merge (with non-default parameters). #13817 (Azat Khuzhin).
- Fix incorrect sorting for
FixedString
columns. Fixes #13182. #13887 (Nikolai Kochetov). - Fixed potential deadlock when renaming
Distributed
table. #13922 (Alexander Tokmakov). - Fix wrong results in select queries with
DISTINCT
keyword in caseoptimize_duplicate_order_by_and_distinct
setting is enabled. #13925 (Artem Zuikov). - Fix parser to reject create table as table function with engine. #13940 (hcz).
- Fixed wrong mount point in extra info for
Poco::Exception: no space left on device
. #14050 (Alexander Tokmakov). - Fix pointInPolygon with const 2d array as polygon. #14079 (Alexey Ilyukhov).
- Fix DistributedFilesToInsert metric (zeroed when it should not). #14095 (Azat Khuzhin).
- When waiting for a dictionary update to complete, use the timeout specified by
query_wait_timeout_milliseconds
setting instead of a hard-coded value. #14105 (Nikita Mikhaylov). - Fix formatting of minimal negative decimal numbers. This fixes #14111. #14119 (Alexander Kuzmenkov).
- Fix creation of tables with named tuples. This fixes #13027. #14143 (Alexey Milovidov).
- Fix crash when INERT INTO Kafka engine table from an empty result set with a subquery. ... #14203 (Dongdong Yang).
- Fixed incorrect sorting order if LowCardinality column. This fixes #13958. #14223 (Nikita Mikhaylov).
- Fix crash in mark inclusion search introduced in https://github.com/ClickHouse/ClickHouse/pull/12277 . #14225 (Amos Bird).
- fixes #14231 fix wrong lexer in MaterializeMySQL database engine dump stage. #14232 (Winter Zhang).
- Fix handling of empty transactions in
MaterializeMySQL
database engine. This fixes #14235. #14253 (BohuTANG). - Fix segfault in
clickhouse-odbc-bridge
during schema fetch from some external sources. This PR fixes #13861. #14267 (Vitaly Baranov).
Build/Testing/Packaging Improvement
- Move Dockerfiles from integration tests to
docker/test
directory. docker_compose files are available inrunner
docker container. Docker images are built in CI and not in integration tests. #13448 (Ilya Yatsishin). - Skip PR's from robot-clickhouse. #13489 (Nikita Mikhaylov).
- Fix typos in code with codespell. #13511 (Alexey Milovidov).
- Enable check for typos in code with
codespell
. #13513 (Alexey Milovidov). - Ensure that there is no copy-pasted GPL code. #13514 (Alexey Milovidov).
- Allow to run
clickhouse
binary without configuration. #13515 (Alexey Milovidov). - Added
clickhouse install
script, that is useful if you only have a single binary. #13528 (Alexey Milovidov). - Fix the remaining shellcheck notices. A preparation to enable Shellcheck. #13529 (Alexey Milovidov).
- Enable Shellcheck in CI as a linter of .sh tests. This closes #13168. #13530 (Alexey Milovidov).
- Make sure #10977 is invalid. #13539 (Amos Bird).
- Increasing health-check timeouts for ClickHouse nodes and adding support to dump docker-compose logs if unhealthy containers found. #13612 (vzakaznikov).
- Build ClickHouse with the most fresh tzdata from package repository. #13623 (Alexey Milovidov).
- Removed
-DENABLE_CURL_CLIENT
forcontrib/aws
. #13628 (Vladimir Chebotarev). - Expose version of embedded tzdata via TZDATA_VERSION in system.build_options. #13648 (filimonov).
- Updating LDAP user authentication suite to check that it works with RBAC. #13656 (vzakaznikov).
- Add a CMake option to fail configuration instead of auto-reconfiguration, enabled by default. #13687 (Konstantin).
- Fix link error in shared build. #13700 (Amos Bird).
- FIx cassandra build on Mac OS. #13708 (Ilya Yatsishin).
- Added docker image for style check. Added style check that all docker and docker compose files are located in docker directory. #13724 (Ilya Yatsishin).
- ZooKeeper cannot work reliably in unit tests in CI infrastructure. Using unit tests for ZooKeeper interaction with real ZooKeeper is bad idea from the start (unit tests are not supposed to verify complex distributed systems). We already using integration tests for this purpose and they are better suited. #13745 (Alexey Milovidov).
- Testflows LDAP module: adding missing certificates and dhparam.pem for openldap4. #13780 (vzakaznikov).
- Enabled text-log in stress test to find more bugs. #13855 (Nikita Mikhaylov).
-
- Adding retry logic when bringing up docker-compose cluster * Increasing COMPOSE_HTTP_TIMEOUT. #14112 (vzakaznikov).
- Add the ability to write js-style comments in skip_list.json. #14159 (alesapin).
- Switch tests docker images to use test-base parent. #14167 (Ilya Yatsishin).
- Actually there are no symlinks there, so
-type f
is enough~/workspace/ClickHouse/contrib/cctz/testdata/zoneinfo$ find . -type l -ls | wc -l 0
Closes #14209. #14215 (filimonov).
Other
- Fix readline so it dumps history to file now. #13600 (Amos Bird).
- Create
system
database withAtomic
engine by default. #13680 (Alexander Tokmakov).
NO CL ENTRY
- NO CL ENTRY: 'Bump pymdown-extensions from 7.1 to 8.0 in /docs/tools'. #13645 (dependabot-preview[bot]).
- NO CL ENTRY: 'Bump protobuf from 3.12.4 to 3.13.0 in /docs/tools'. #13824 (dependabot-preview[bot]).