ClickHouse/docs/changelogs/v22.3.16.1190-lts.md

sidebar_position	sidebar_label
1	2023

2023 Changelog

ClickHouse release v22.3.16.1190-lts (bb4e0934e5) FIXME as compared to v22.10.1.1877-stable (98ab5a3c18)

Backward Incompatible Change

• JSONExtract family of functions will now attempt to coerce to the request type. #41502 (Márcio Martins).
• Backported in #43484: Fixed backward incompatibility in (de)serialization of states of min, max, any*, argMin, argMax aggregate functions with String argument. The incompatibility was introduced in https://github.com/ClickHouse/ClickHouse/pull/41431 and affects 22.9, 22.10 and 22.11 branches (fixed since 22.9.6, 22.10.4 and 22.11.2 correspondingly). Some minor releases of 22.3, 22.7 and 22.8 branches are also affected: 22.3.13...22.3.14 (fixed since 22.3.15), 22.8.6...22.8.9 (fixed since 22.8.10), 22.7.6 and newer (will not be fixed in 22.7, we recommend to upgrade from 22.7.* to 22.8.10 or newer). This release note does not concern users that have never used affected versions. Incompatible versions append extra '\0' to strings when reading states of the aggregate functions mentioned above. For example, if an older version saved state of anyState('foobar') to state_column then incompatible version will print 'foobar\0' on anyMerge(state_column). Also incompatible versions write states of the aggregate functions without trailing '\0'. Newer versions (that have the fix) can correctly read data written by all versions including incompatible versions, except one corner case. If an incompatible version saved a state with a string that actually ends with null character, then newer version will trim trailing '\0' when reading state of affected aggregate function. For example, if an incompatible version saved state of anyState('abrac\0dabra\0') to state_column then newer versions will print 'abrac\0dabra' on anyMerge(state_column). The issue also affects distributed queries when an incompatible version works in a cluster together with older or newer versions. #43038 (Raúl Marín).

New Feature

• • Add function displayName, closes #36770. #37681 (hongbin).
• Add Hudi and DeltaLake table engines, read-only, only for tables on S3. #41054 (Daniil Rubin).
• Add 4LW command csnp for manually creating snapshots. Additionally, lgif was added to get Raft information for a specific node (e.g. index of last created snapshot, last committed log index). #41766 (JackyWoo).
• Add function ascii like in spark: https://spark.apache.org/docs/latest/api/sql/#ascii. #42670 (李扬).
• Published function formatReadableDecimalSize. #42774 (Alejandro).

Performance Improvement

• Currently, the only saturable operators are And and Or, and their code paths are affected by this change. #42214 (Zhiguo Zhou).
• match function can use the index if it's a condition on string prefix. This closes #37333. #42458 (clarkcaoliu).
• Support parallel parsing for LineAsString input format. This improves performance just slightly. This closes #42502. #42780 (Kruglov Pavel).
• Keeper performance improvement: improve commit performance for cases when many different nodes have uncommitted states. This should help with cases when a follower node can't sync fast enough. #42926 (Antonio Andelic).

Improvement

• Support type Object inside other types, e.g. Array(JSON). #36969 (Anton Popov).
• Backported in #42527: Fix issue with passing MySQL timeouts for MySQL database engine and MySQL table function. Closes #34168?notification_referrer_id=NT_kwDOAzsV57MzMDMxNjAzNTY5OjU0MjAzODc5. #40751 (Kseniia Sumarokova).
• ClickHouse Client and ClickHouse Local will show progress by default even in non-interactive mode. If /dev/tty is available, the progress will be rendered directly to the terminal, without writing to stderr. It allows to get progress even if stderr is redirected to a file, and the file will not be polluted by terminal escape sequences. The progress can be disabled by --progress false. This closes #32238. #42003 (Alexey Milovidov).
• 1. Add, subtract and negate operations are now available on Intervals. In case when the types of Intervals are different they will be transformed into the Tuple of those types. 2. A tuple of intervals can be added to or subtracted from a Date/DateTime field. 3. Added parsing of Intervals with different types, for example: INTERVAL '1 HOUR 1 MINUTE 1 SECOND'. #42195 (Nikolay Degterinsky).
• • Add notLike to key condition atom map, so condition like NOT LIKE 'prefix%' can use primary index. #42209 (Duc Canh Le).
• Add support for FixedString input to base64 coding functions. #42285 (ltrk2).
• Add columns bytes_on_disk and path to system.detached_parts. Closes #42264. #42303 (chen).
• Added ** glob support for recursive directory traversal to filesystem and S3. resolves #36316. #42376 (SmitaRKulkarni).
• Mask passwords and secret keys both in system.query_log and /var/log/clickhouse-server/*.log and also in error messages. #42484 (Vitaly Baranov).
• Add a new variable call limit in query_info, indicating whether this query is a limit-trivial query. If so, we will adjust the approximate total rows for later estimation. Closes #7071. #42580 (Han Fei).
• Implement ATTACH of MergeTree table for s3_plain disk (plus some fixes for s3_plain). #42628 (Azat Khuzhin).
• Fix no progress indication on INSERT FROM INFILE. Closes #42548. #42634 (chen).
• Add min_age_to_force_merge_on_partition_only setting to optimize old parts for the entire partition only. #42659 (Antonio Andelic).
• Throttling algorithm changed to token bucket. #42665 (Sergei Trifonov).
• Added new field allow_readonly in system.table_functions to allow using table functions in readonly mode resolves #42414 Implementation: * Added a new field allow_readonly to table system.table_functions. * Updated to use new field allow_readonly to allow using table functions in readonly mode. Testing: * Added a test for filesystem tests/queries/0_stateless/02473_functions_in_readonly_mode.sh Documentation: * Updated the english documentation for Table Functions. #42708 (SmitaRKulkarni).
• Allow to use Date32 arguments for formatDateTime and FROM_UNIXTIME functions. #42737 (Roman Vasin).
• Backported in #42839: Update tzdata to 2022f. Mexico will no longer observe DST except near the US border: https://www.timeanddate.com/news/time/mexico-abolishes-dst-2022.html. Chihuahua moves to year-round UTC-6 on 2022-10-30. Fiji no longer observes DST. See https://github.com/google/cctz/pull/235 and https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/1995209. #42796 (Alexey Milovidov).
• Update tzdata to 2022f. Mexico will no longer observe DST except near the US border: https://www.timeanddate.com/news/time/mexico-abolishes-dst-2022.html. Chihuahua moves to year-round UTC-6 on 2022-10-30. Fiji no longer observes DST. See https://github.com/google/cctz/pull/235 and https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/1995209. #42796 (Alexey Milovidov).
• Add FailedAsyncInsertQuery event metric for async inserts. #42814 (Krzysztof Góralski).
• Increase the size of upload part exponentially for backup to S3. #42833 (Vitaly Baranov).

Bug Fix

• Backported in #43829: Updated normaliser to clone the alias ast. resolves #42452 Implementation: * Updated QueryNormalizer to clone alias ast, when its replaced. Previously just assigning the same leads to exception in LogicalExpressinsOptimizer as it would be the same parent being inserted again. * This bug is not seen with new analyser (allow_experimental_analyzer), so no changes for it. I added a test for the same. #42827 (SmitaRKulkarni).

Build/Testing/Packaging Improvement

• Run SQLancer for each pull request and commit to master. SQLancer is an OpenSource fuzzer that focuses on automatic detection of logical bugs. #42397 (Ilya Yatsishin).
• Update to latest zlib-ng. #42463 (Boris Kuschel).
• use llvm l64.lld in macOS suppress ld warnings, close #42282. #42470 (Lloyd-Pottiger).
• • Improve bugfix validation check: fix bug with skipping the check, port separate status in CI, run after check labels and style check. Close #40349. #42702 (Vladimir C).
• Backported in #43050: Wait for all files are in sync before archiving them in integration tests. #42891 (Mikhail f. Shiryaev).
• Wait for all files are in sync before archiving them in integration tests. #42891 (Mikhail f. Shiryaev).
• Use https://github.com/matus-chochlik/ctcache for clang-tidy results caching. #42913 (Mikhail f. Shiryaev).
• Backported in #42963: Before the fix, the user-defined config was preserved by RPM in $file.rpmsave. The PR fixes it and won't replace the user's files from packages. #42936 (Mikhail f. Shiryaev).
• Before the fix, the user-defined config was preserved by RPM in $file.rpmsave. The PR fixes it and won't replace the user's files from packages. #42936 (Mikhail f. Shiryaev).
• Backported in #43039: Add a CI step to mark commits as ready for release; soft-forbid launching a release script from branches but master. #43017 (Mikhail f. Shiryaev).
• Add a CI step to mark commits as ready for release; soft-forbid launching a release script from branches but master. #43017 (Mikhail f. Shiryaev).
• Backported in #44109: Bring sha512 sums back to the building step. #44017 (Mikhail f. Shiryaev).
• Backported in #44431: Kill stress tests after 2.5h in case of hanging process. #44214 (Mikhail f. Shiryaev).
• Backported in #44557: Retry the integration tests on compressing errors. #44529 (Mikhail f. Shiryaev).

Bug Fix (user-visible misbehavior in official stable or prestable release)

• Fix schema inference in s3Cluster and improve in hdfsCluster. #41979 (Kruglov Pavel).
• Fix retries while reading from http table engines / table function. (retrtiable errors could be retries more times than needed, non-retrialble errors resulted in failed assertion in code). #42224 (Kseniia Sumarokova).
• A segmentation fault related to DNS & c-ares has been reported. The below error ocurred in multiple threads: 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008088 [ 356 ] {} <Fatal> BaseDaemon: ######################################## 2022-09-28 15:41:19.008,"2022.09.28 15:41:19.008147 [ 356 ] {} <Fatal> BaseDaemon: (version 22.8.5.29 (official build), build id: 92504ACA0B8E2267) (from thread 353) (no query) Received signal Segmentation fault (11)" 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008196 [ 356 ] {} <Fatal> BaseDaemon: Address: 0xf Access: write. Address not mapped to object. 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008216 [ 356 ] {} <Fatal> BaseDaemon: Stack trace: 0x188f8212 0x1626851b 0x1626a69e 0x16269b3f 0x16267eab 0x13cf8284 0x13d24afc 0x13c5217e 0x14ec2495 0x15ba440f 0x15b9d13b 0x15bb2699 0x1891ccb3 0x1891e00d 0x18ae0769 0x18ade022 0x7f76aa985609 0x7f76aa8aa133 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008274 [ 356 ] {} <Fatal> BaseDaemon: 2. Poco::Net::IPAddress::family() const @ 0x188f8212 in /usr/bin/clickhouse 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008297 [ 356 ] {} <Fatal> BaseDaemon: 3. ? @ 0x1626851b in /usr/bin/clickhouse 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008309 [ 356 ] {} <Fatal> BaseDaemon: 4. ? @ 0x1626a69e in /usr/bin/clickhouse. #42234 (Arthur Passos).
• Fix LOGICAL_ERROR Arguments of 'plus' have incorrect data types which may happen in PK analysis (monotonicity check). Fix invalid PK analysis for monotonic binary functions with first constant argument. #42410 (Nikolai Kochetov).
• Fix incorrect key analysis when key types cannot be inside Nullable. This fixes #42456. #42469 (Amos Bird).
• Fix typo in setting name that led to bad usage of schema inference cache while using setting input_format_csv_use_best_effort_in_schema_inference. Closes #41735. #42536 (Kruglov Pavel).
• Fix create Set with wrong header when data type is LowCardinality. Closes #42460. #42579 (flynn).
• (U)Int128 and (U)Int256 values are correctly checked in PREWHERE. #42605 (Antonio Andelic).
• Fix a bug in ParserFunction that could have led to a segmentation fault. #42724 (Nikolay Degterinsky).
• Fix truncate table does not hold lock correctly. #42728 (flynn).
• Fix possible SIGSEGV for web disks when file does not exists (or OPTIMIZE TABLE FINAL, that also can got the same error eventually). #42767 (Azat Khuzhin).
• Fix auth_type mapping in system.session_log, by including SSL_CERTIFICATE for the enum values. #42782 (Miel Donkers).
• Fix stack-use-after-return under ASAN build in ParserCreateUserQuery. #42804 (Nikolay Degterinsky).
• Fix lowerUTF8()/upperUTF8() in case of symbol was in between 16-byte boundary (very frequent case of you have strings > 16 bytes long). #42812 (Azat Khuzhin).
• Additional bound check was added to lz4 decompression routine to fix misbehaviour in case of malformed input. #42868 (Nikita Taranov).
• Fix rare possible hung on query cancellation. #42874 (Azat Khuzhin).
• • Fix incorrect saved_block_sample with multiple disjuncts in hash join, close #42832. #42876 (Vladimir C).
• A null pointer will be generated when select if as from ‘three table join’ , For example, the SQL:. #42883 (zzsmdfj).
• Fix memory sanitizer report in ClusterDiscovery, close #42763. #42905 (Vladimir C).
• Fix rare NOT_FOUND_COLUMN_IN_BLOCK error when projection is possible to use but there is no projection available. This fixes #42771 . The bug was introduced in https://github.com/ClickHouse/ClickHouse/pull/25563. #42938 (Amos Bird).
• Fix ATTACH TABLE in PostgreSQL database engine if the table contains DATETIME data type. Closes #42817. #42960 (Kseniia Sumarokova).
• Fix lambda parsing. Closes #41848. #42979 (Nikolay Degterinsky).
• Backported in #43512: - Fix several buffer over-reads. #43159 (Raúl Marín).
• Backported in #43750: An issue with the following exception has been reported while trying to read a Parquet file from S3 into ClickHouse:. #43297 (Arthur Passos).
• Backported in #43427: Fixed queries with SAMPLE BY with prewhere optimization on tables using Merge engine. #43315 (Antonio Andelic).
• Backported in #43616: Fix sumMap() for Nullable(Decimal()). #43414 (Azat Khuzhin).
• Backported in #43720: Fixed primary key analysis with conditions involving toString(enum). #43596 (Nikita Taranov).
• Backported in #43885: Fixed ALTER ... RESET SETTING with ON CLUSTER. It could be applied to one replica only. Fixes #43843. #43848 (Elena Torró).
• Backported in #44179: Fix undefined behavior in the quantiles function, which might lead to uninitialized memory. Found by fuzzer. This closes #44066. #44067 (Alexey Milovidov).
• Backported in #44283: Prevent ReadonlyReplica metric from having negative values. #44220 (Antonio Andelic).

Build Improvement

• ... Add support for format ipv6 on s390x. #42412 (Suzy Wang).

NO CL ENTRY

• NO CL ENTRY: 'Revert "Sonar Cloud Workflow"'. #42725 (Alexey Milovidov).
• NO CL ENTRY: 'Fix multipart upload for large S3 object, backport to 22.3'. #44217 (ianton-ru).

NOT FOR CHANGELOG / INSIGNIFICANT

• Build with libcxx(abi) 15 #42513 (Robert Schulze).
• Sonar Cloud Workflow #42534 (Julio Jimenez).
• Invalid type in where for Merge table (logical error) #42576 (Nikolai Kochetov).
• Fix frequent memory drift message and clarify things in comments #42582 (Azat Khuzhin).
• Try to save IDataPartStorage interface #42618 (Anton Popov).
• Analyzer change setting into allow_experimental_analyzer #42649 (Maksim Kita).
• Analyzer IQueryTreeNode remove getName method #42651 (Maksim Kita).
• Minor fix iotest_nonblock build #42658 (Jordi Villar).
• Add tests and doc for some url-related functions #42664 (Vladimir C).
• Update version_date.tsv and changelogs after v22.10.1.1875-stable #42676 (robot-clickhouse).
• Fix error handling in clickhouse_helper.py #42678 (Ilya Yatsishin).
• Fix execution of version_helper.py to use git tweaks #42679 (Mikhail f. Shiryaev).
• MergeTree indexes use RPNBuilderTree #42681 (Maksim Kita).
• Always run BuilderReport and BuilderSpecialReport in all CI types #42684 (Mikhail f. Shiryaev).
• Always run BuilderReport and BuilderSpecialReport in all CI types #42684 (Mikhail f. Shiryaev).
• Update version after release #42699 (Mikhail f. Shiryaev).
• Update version_date.tsv and changelogs after v22.10.1.1877-stable #42700 (robot-clickhouse).
• OrderByLimitByDuplicateEliminationPass improve performance #42704 (Maksim Kita).
• Analyzer improve subqueries representation #42705 (Maksim Kita).
• Update version_date.tsv and changelogs after v22.9.4.32-stable #42712 (robot-clickhouse).
• Update version_date.tsv and changelogs after v22.8.7.34-lts #42713 (robot-clickhouse).
• Update version_date.tsv and changelogs after v22.7.7.24-stable #42714 (robot-clickhouse).
• Move SonarCloud Job to nightly #42718 (Julio Jimenez).
• Update version_date.tsv and changelogs after v22.8.8.3-lts #42738 (robot-clickhouse).
• Minor fix implicit cast CaresPTRResolver #42747 (Jordi Villar).
• Fix build on master #42752 (Igor Nikonov).
• Update version_date.tsv and changelogs after v22.3.14.18-lts #42759 (robot-clickhouse).
• Fix anchor links #42760 (Sergei Trifonov).
• Update version_date.tsv and changelogs after v22.3.14.23-lts #42764 (robot-clickhouse).
• Update README.md #42783 (Yuko Takagi).
• Slightly better code with projections #42794 (Anton Popov).
• Fix some races in MergeTree #42805 (Alexander Tokmakov).
• Fix typo in comments #42809 (Gabriel).
• Fix compilation of LLVM with cmake cache #42816 (Azat Khuzhin).
• Fix link in docs #42821 (Sergei Trifonov).
• Link to proper place in docs #42822 (Sergei Trifonov).
• Fix argument type check in AggregateFunctionAnalysisOfVariance #42823 (Vladimir C).
• Tests/lambda analyzer #42824 (Denny Crane).
• Fix Missing Quotes - Sonar Nightly #42831 (Julio Jimenez).
• Add exclusions from the Snyk scan #42834 (Alexey Milovidov).
• Fix Missing Env Vars - Sonar Nightly #42843 (Julio Jimenez).
• Fix typo #42855 (GoGoWen).
• Add timezone to 02458_datediff_date32 #42857 (Vladimir C).
• Adjust cancel and rerun workflow names to the actual #42862 (Mikhail f. Shiryaev).
• Analyzer subquery in JOIN TREE with aggregation #42865 (Maksim Kita).
• Fix getauxval for sanitizer builds #42866 (Amos Bird).
• Update version_date.tsv and changelogs after v22.10.2.11-stable #42871 (robot-clickhouse).
• Validate Query Tree in debug #42879 (Dmitry Novik).
• changed type name for s3 plain storage #42890 (Aleksandr).
• Cleanup implementation of regexpReplace(All|One) #42907 (Robert Schulze).
• Do not show status for Bugfix validate check in non bugfix PRs #42932 (Vladimir C).
• fix(typo): Passible -> Possible #42933 (Yakko Majuri).
• Pin the cryptography version to not break lambdas #42934 (Mikhail f. Shiryaev).
• Fix: bad cast from type DB::ColumnLowCardinality to DB::ColumnString #42937 (Igor Nikonov).
• Attach thread pool for loading parts to the query #42947 (Azat Khuzhin).
• Fix macOS M1 builds due to sprintf deprecation #42962 (Jordi Villar).
• Less use of CH-specific bit_cast() #42968 (Robert Schulze).
• Remove some utils #42972 (Alexey Milovidov).
• Fix a bug in CAST function parser #42980 (Nikolay Degterinsky).
• Fix old bug to remove refs/head from ref name #42981 (Mikhail f. Shiryaev).
• Add debug information to nightly builds #42997 (Mikhail f. Shiryaev).
• Add on: workflow_call to debug CI #43000 (Mikhail f. Shiryaev).
• Simple fixes for restart replica description #43004 (Igor Nikonov).
• Cleanup match code #43006 (Robert Schulze).
• Fix TSan errors (correctly ignore _exit interception) #43009 (Azat Khuzhin).
• fix bandwidth throttlers initialization order #43015 (Sergei Trifonov).
• Add test for issue #42520 #43027 (Robert Schulze).
• Fix msan warning #43065 (Raúl Marín).
• Update SECURITY.md on new stable tags #43365 (Mikhail f. Shiryaev).
• Use all parameters with prefixes from ssm #43467 (Mikhail f. Shiryaev).
• Temporarily disable test_hive_query #43542 (Alexander Tokmakov).
• Do not checkout submodules recursively #43637 (Mikhail f. Shiryaev).
• Use docker images cache from merged PRs in master and release branches #43664 (Mikhail f. Shiryaev).
• Yet another fix for AggregateFunctionMinMaxAny #43778 (Alexander Tokmakov).
• Fix tags workflow #43942 (Mikhail f. Shiryaev).
• Assign release PRs #44055 (Mikhail f. Shiryaev).
• Fix another bug in AggregateFunctionMinMaxAny #44091 (Alexander Tokmakov).
• Bump libdivide (to gain some new optimizations) #44132 (Azat Khuzhin).
• Add check for submodules sanity #44386 (Mikhail f. Shiryaev).
• Implement a custom central checkout action #44399 (Mikhail f. Shiryaev).