ClickHouse/docs/changelogs/v21.11.1.8636-prestable.md

ClickHouse release v21.11.1.8636-prestable FIXME as compared to v21.10.1.8013-prestable

Backward Incompatible Change

• Now ClickHouse Keeper snapshots compressed with ZSTD codec by default instead of custom ClickHouse LZ4 block compression. This behavior can be turned off with compress_snapshots_with_zstd_format coordination setting (must be equal on all quorum replicas). Backward incompatibility is quite rare and may happen only when new node will send snapshot (happens in case of recovery) to the old node which is unable to read snapshots in ZSTD format. #29417 (alesapin).
• Function bayesAB is removed. This closes #26233. #29934 (Alexey Milovidov).
• Remove MergeTree table setting write_final_mark. It will be always true. #30455 (Kseniia Sumarokova).
• Change order of json_path and json in sql json function (to be correct). Closes #30449. #30474 (Kseniia Sumarokova).
• Remove setting odbc_max_field_size because it is not used anymore. Closes #30413. #30778 (Kseniia Sumarokova).

New Feature

• Add CapnProto output format, refactor CapnProto input format. #29291 (Kruglov Pavel).
• clickhouse-format support option --query. #29325 (凌涛).
• Users now can set comments to database in CREATE DATABASE statement ... #29429 (Vasily Nemkov).
• New function mapContainsKeyLike to get the map that key matches a simple regular expression. #29471 (凌涛).
• Huawei OBS Storage support. Closes #24294. #29511 (kevin wan).
• ClickHouse HTTP Server can enable HSTS by set hsts_max_age in config.xml with a positive number. #29516 (凌涛).
• • Added MD4 and SHA384 functions. #29602 (Nikita Tikhomirov).
• Support EXISTS(subquery). Closes #6852. #29731 (Kseniia Sumarokova).
• Added function ngram. Closes #29699. #29738 (Maksim Kita).
• Returns String with OS Kernel version. #29755 (Memo).
• Predefined configuration for table function remote. Closes #29756. #29774 (Kseniia Sumarokova).
• Add table alias to system.tables and database alias to system.databases #29677. #29882 (kevin wan).
• Added columns data_compressed_bytes, data_uncompressed_bytes, marks_bytes into system.data_skipping_indices. Added columns secondary_indices_compressed_bytes, secondary_indices_uncompressed_bytes, secondary_indices_marks_bytes into system.parts. Closes #29697. #29896 (Maksim Kita).
• User can now create dictionaries with comments: CREATE DICTIONARY ... COMMENT 'vaue' ... #29899 (Vasily Nemkov).
• Add support for FreeBSD aarch64. #29952 (MikaelUrankar).
• Added function tokens. That allow to split string into tokens using non-alpha numeric ASCII characters as separators. #29981 (Maksim Kita).
• Added new JSONExtractKeys function ... #30056 (Vitaly Orlov).
• Added HashedArray dictionary type. Closes #30236. #30242 (Maksim Kita).
• Add ability to change nodes configuration (in .xml file) for ClickHouse Keeper. #30372 (alesapin).
• CompiledExpressionCache limit elements size using compiled_expression_cache_elements_size setting. #30667 (Maksim Kita).
• New function mapExtractKeyLike to get the map only kept elements matched specified pattern. #30793 (凌涛).
• Support ALTER TABLE for tables in Memory databases. #30866 (Alexander Tokmakov).

Performance Improvement

• • Remove branchy code in filter operation with a better implementation with popcnt/ctz which have better performance. #29881 (Jun Jin).
• • To take advantage of X86_64 feature, Use AVX2/AVX512 instructions to accelerate filter operation. #30014 (jasperzhu).
• Improve performance of aggregation in order of primary key (with enabled setting optimize_aggregation_in_order). #30266 (Anton Popov).
• ColumnDecimal improve performance for filter operation. #30431 (Jun Jin).
• Dictionaries support read from multiple threads. #30500 (Maksim Kita).
• • Improve filter bitmask generator function all in one with sse/avx2/avx512 instructions. #30670 (jasperzhu).
• Queries with INTO OUTFILE in clickhouse-client will use multiple threads. Fix the issue with flickering progress-bar when using INTO OUTFILE. This closes #30873. This closes #30872. #30886 (Alexey Milovidov).

Improvement

• Allow to write number in query as binary literal. Example SELECT 0b001;. #29304 (Maksim Kita).
• Pass through initial query_id for clickhouse-benchmark (previously if you run remote query via clickhouse-benchmark, queries on shards will not be linked to the initial query via initial_query_id). #29364 (Azat Khuzhin).
• fix lost in memory part when freeze. #29376 (Mo Xuan).
• Added setting use_skip_indexes. #29405 (Maksim Kita).
• Apply config changes to max_concurrent_queries during runtime (no need to restart). #29414 (Raúl Marín).
• Transform isNull/isNotNull to IS NULL/IS NOT NULL (for external dbs, i.e. MySQL). #29446 (Azat Khuzhin).
• Enable per-query memory profiler (set to memory_profiler_step=4MiB) globally. #29455 (Azat Khuzhin).
• Add support for IS NULL/IS NOT NULL for external dbs (i.e. MySQL). #29463 (Azat Khuzhin).
• Fixed There is no query or query context has expired error in mutations with nested subqueries. Do not allow subqueries in mutation if table is replicated and allow_nondeterministic_mutations setting is disabled. #29495 (Alexander Tokmakov).
• Get rid of pointless restriction on projection name. Now projection name can start with tmp_. #29520 (Amos Bird).
• Reload dictionaries, models, user defined executable functions if servers config dictionaries_config, models_config, user_defined_executable_functions_config changes. Closes #28142. #29529 (Maksim Kita).
• Allow user to change log levels without restart. #29586 (Nikolay Degterinsky).
• Increase listen_backlog by default (to match default in newer linux kernel). #29643 (Azat Khuzhin).
• Fix incorrect nullable processing of JSONFunctions. This fixes #29615 . Mark as improvement because https://github.com/ClickHouse/ClickHouse/pull/28012 is not released. #29659 (Amos Bird).
• Fixed the issue: clickhouse-format --obfuscate cannot process queries with embedded dictionaries (functions regionTo...). #29667 (Alexey Milovidov).
• Query obfuscator clickhouse-format --obfuscate now works with more types of queries. #29672 (Alexey Milovidov).
• Make url table function to process multiple URLs in parallel. This closes #29670 and closes #29671. #29673 (Alexey Milovidov).
• Add ClickHouse logo to Play UI. #29674 (Alexey Milovidov).
• Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge. #29681 (Azat Khuzhin).
• Remove some redundant seek calls while reading compressed blocks in MergeTree table engines family. #29766 (alesapin).
• Web UI: render bars in table cells. #29792 (Alexey Milovidov).
• Added partitioned table prefix 'p' for the query for fetching replica identity index. #29828 (Shoh Jahon).
• Add ability to trace peak memory usage (with new trace_type - MemoryPeak). #29858 (Azat Khuzhin).
• Remove unused variable in s3cluster. #29912 (李扬).
• Add shutdown_wait_unfinished_queries server setting to allowing waiting for running queries up to shutdown_wait_unfinished time. This is for #24451 . #29914 (Amos Bird).
• Now min-max aggregation over the first expression of primary key is optimized by projection. This is for https://github.com/ClickHouse/ClickHouse/issues/329. #29918 (Amos Bird).
• Add ability to configure retries and delays between them for clickhouse-copier. #29921 (Azat Khuzhin).
• Update zoneinfo files to 2021c. #29925 (Alexey Milovidov).
• Better exception message while reading column from Arrow-supported formats like Arrow, ArrowStream, Parquet and ORC. This closes #29926. #29927 (Alexey Milovidov).
• Fix lock-order-inversion between periodic dictionary reload and config reload. #29928 (Azat Khuzhin).
• Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView. #29929 (Azat Khuzhin).
• Fix data-race between flush() and startup() in StorageBuffer. #29930 (Azat Khuzhin).
• Now clickhouse use DNS cache while communicating with external S3. #29999 (alesapin).
• ProfileEvents::Counters snapshot doesn't store data in std::atomic anymore. #30000 (Dmitry Novik).
• Add ability to print raw profile events to clickhouse-client (This can be useful for debugging and for testing). #30064 (Azat Khuzhin).
• Improve solution https://github.com/ClickHouse/ClickHouse/pull/28853 See also https://github.com/ClickHouse/ClickHouse/pull/29928. #30084 (Vitaly Baranov).
• Reduce amount of redundant compressed data read from disk for some types SELECT queries (only for MergeTree engines family). #30111 (alesapin).
• PolygonDictionary added support for read method if setting store_polygon_key_column = true. Closes #30090. #30142 (Maksim Kita).
• Now clickhouse-client supports native multi-line editing. #30143 (Amos Bird).
• Now Keeper (as part of clickhouse-server) will start asynchronously if it can connect to some other node. #30170 (alesapin).
• Support SQL user defined functions for clickhouse-local. #30179 (Maksim Kita).
• Allow to remove SAMPLE BY expression from MergeTree tables (ALTER TABLE <table> REMOVE SAMPLE BY). #30180 (Anton Popov).
• Fixed comparison of Date32 with Date, DateTime, DateTime64, String. #30219 (liang.huang).
• Allow symlinks to files in user_files directory for file table function. #30309 (Kseniia Sumarokova).
• Minor improvements in replica cloning and enqueuing fetch for broken parts, that should avoid extremely rare hanging of GET_PART entries in replication queue. #30346 (Alexander Tokmakov).
• Added an ability to use FINAL clause in SELECT queries from GraphiteMergeTree. #30360 (Nikita Mikhaylov).
• Allow extract non-string element as string using JSONExtractString. This is for https://github.com/ClickHouse/ClickHouse/pull/25452#issuecomment-927123287. #30426 (Amos Bird).
• More full support of positional arguments. #30433 (Kseniia Sumarokova).
• SQLUserDefinedFunctions support lambdas. Example CREATE FUNCTION lambda_function AS x -> arrayMap(element -> element * 2, x);. #30435 (Maksim Kita).
• SQLUserDefinedFunctions added DROP IF EXISTS support. Example DROP FUNCTION IF EXISTS test_function. #30437 (Maksim Kita).
• SQLUserDefinedFunctions support CREATE OR REPLACE, CREATE IF NOT EXISTS syntaxes. #30454 (Maksim Kita).
• Make query, which fetched table structure for PostgreSQL database because, more reliable. #30477 (Kseniia Sumarokova).
• Get memory amount with Docker/cgroups limitations. See #25662. #30574 (Pavel Medvedev).
• SQLUserDefinedFunctions support ON CLUSTER. Example CREATE FUNCTION test_function ON CLUSTER 'cluster' AS x -> x + 1;. Closes #30666. #30734 (Maksim Kita).
• Arrays of all serializable types are now supported by arrayStringConcat. #30840 (Nikita Taranov).
• Allow to parse values of Date data type in text formats as YYYYMMDD in addition to YYYY-MM-DD. This closes #30870. #30871 (Alexey Milovidov).
• Rename the columns of SessionLog: session_id -> auth_id, session_name -> session_id, changed_settings -> settings. #30882 (Vitaly Baranov).

Bug Fix

• Fix shutdown of AccessControlManager. Now there can't be reloading of the configuration after AccessControlManager has been destroyed. This PR fixes the flaky test test_user_directories/test.py::test_relative_path. #29951 (Vitaly Baranov).
• Fix releasing query ID and session ID at the end of query processing while handing gRPC call. This PR fixes flaky test test_grpc_protocol/test.py::test_session. #29954 (Vitaly Baranov).
• Fix reading from empty file on encrypted disk. #30494 (Vitaly Baranov).

Build/Testing/Packaging Improvement

• N/A Detailed description / Documentation draft:. #29367 (kevin wan).
• Add ability to compile using newer version fo glibc w/o using new symbols. #29594 (Azat Khuzhin).
• Turning on experimental constexpr expressions evaluator for clang to speed up template code compilation. #29668 (Mike Kot).
• Fix an build error because of Rename "common" to "base". #29688 (Sergei Semin).
• Leave only required files in cross-compile toolchains. Include them as submodules (earlier they were downloaded as tarballs). #29974 (Alexey Milovidov).
• Add cross-build for PowerPC (powerpc64le). This closes #9589. Enable support for interaction with MySQL for AArch64 and PowerPC. This closes #26301. #30010 (Alexey Milovidov).
• Enable Protobuf, Arrow, ORC, Parquet for AArch64 and Darwin builds. This closes #29248. This closes #28018. #30015 (Alexey Milovidov).
• Add tests for encrypted disk & encryption codec and ReplicatedMergeTree. #30172 (Vitaly Baranov).
• ClickHouse can be statically built with Musl. This is added as experiment, it does not support building odbc-bridge, library-bridge, integration with CatBoost and some libraries. #30248 (Alexey Milovidov).
• Recursive submodules are no longer needed for ClickHouse. #30315 (Alexey Milovidov).
• Added docker image to build docs. #30499 (Ilya Yatsishin).

Bug Fix (user-visible misbehaviour in official stable or prestable release

• Fix ORDER BY ... WITH FILL with set TO and FROM and no rows in result set. #30888 (Anton Popov).

Bug Fix (user-visible misbehaviour in official stable or prestable release)

• Fix hanging DDL queries on Replicated database while adding a new replica. #29328 (Kevin Michel).
• Fix vertical merges of projection parts. This fixes #29253 . This PR also fixes several projection merge/mutation issues introduced in https://github.com/ClickHouse/ClickHouse/pull/25165. #29337 (Amos Bird).
• Remove window function nth_value as it is not memory-safe. This closes #29347. #29348 (Alexey Milovidov).
• Fix replicated access storage not shutting down cleanly when misconfigured. #29388 (Kevin Michel).
• rocksdb: fix race condition during multiple DB opening (and get back some tests that triggers the problem on CI). #29393 (Azat Khuzhin).
• Fix Logical error Cannot capture columns in functions greatest/least. Closes #29334. #29454 (Kruglov Pavel).
• Fix possible Block structure mismatch for subqueries with pushed-down HAVING predicate. Fixes #29010. #29475 (Nikolai Kochetov).
• Fixed incorrect table name parsing on loading of Lazy database. Fixes #29456. #29476 (Alexander Tokmakov).
• In ODBC bridge add retries for error Invalid cursor state. It is a retriable error. Closes #29473. #29518 (Kseniia Sumarokova).
• Fix bug in check pathStartsWith becuase there was bug with the usage of std::mismatch: The behavior is undefined if the second range is shorter than the first range.. #29531 (Kseniia Sumarokova).
• Avoid deadlocks when reading and writting on JOIN Engine tables at the same time. #29544 (Raúl Marín).
• Fix null deference for GROUP BY WITH TOTALS HAVING (when the column from HAVING wasn't selected). #29553 (Azat Khuzhin).
• Fix rare segfault in ALTER MODIFY query when using incorrect table identifier in DEFAULT expression like x.y.z... Fixes #29184. #29573 (alesapin).
• Fix JIT expression compilation with aliases and short-circuit expression evaluation. Closes #29403. #29574 (Maksim Kita).
• Condition in filter predicate could be lost after push-down optimisation. #29625 (Nikolai Kochetov).
• Fixed incorrect behaviour of setting materialized_postgresql_tables_list at server restart. Found in #28529. #29686 (Kseniia Sumarokova).
• Fix incorrect GROUP BY (multiple rows with the same keys in result) in case of distributed query when shards had mixed versions <= 21.3 and >= 21.4, GROUP BY key had several columns all with fixed size, and two-level aggregation was activated (see group_by_two_level_threshold and group_by_two_level_threshold_bytes). Fixes #29580. #29735 (Nikolai Kochetov).
• Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV). #29782 (Azat Khuzhin).
• Fix bad cast in ATTACH TABLE ... FROM 'path' query when non-string literal is used instead of path. It may lead to reading of uninitialized memory. #29790 (Alexey Milovidov).
• Avoid Timeout exceeded: elapsed 18446744073.709553 seconds error that might happen in extremely rare cases, presumably due to some bug in kernel. Fixes #29154. #29811 (Alexander Tokmakov).
• MaterializedMySQL: Fix an issue where if the connection to MySQL was lost, only parts of a transaction could be processed. #29837 (Håvard Kvålen).
• Fix system tables recreation check (fails to detect changes in enum values). #29857 (Azat Khuzhin).
• Fix potential resource leak of the concurrent query limit of merge tree tables introduced in https://github.com/ClickHouse/ClickHouse/pull/19544 . #29879 (Amos Bird).
• Fix data-race between LogSink::writeMarks() and LogSource in StorageLog. #29946 (Azat Khuzhin).
• Fix possible data-race between FileChecker and StorageLog/StorageStripeLog. #29959 (Azat Khuzhin).
• try to close issue: #29965. #29976 (hexiaoting).
• Fix crash of sample by tuple(), closes #30004. #30016 (flynn).
• Dropped Memory database might reappear after server restart, it's fixed (#29795). Also added force_remove_data_recursively_on_drop setting as a workaround for Directory not empty error when dropping Ordinary database (because it's not possible to remove data leftovers manually in cloud environment). #30054 (Alexander Tokmakov).
• Fix race between MOVE PARTITION and merges/mutations for MergeTree. #30074 (Azat Khuzhin).
• Fix error Port is already connected for queries with GLOBAL IN and WITH TOTALS. Only for 21.9 and 21.10. #30086 (Nikolai Kochetov).
• Support nullable arguments in function initializeAggregation. #30177 (Anton Popov).
• Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column. #30189 (Azat Khuzhin).
• Fix pread_fake_async/pread_threadpool with min_bytes_to_use_direct_io. #30191 (Azat Khuzhin).
• Fix reading from MergeTree with max_read_buffer_size=0 (can lead to Can't adjust last granule LOGICAL_ERROR, or even data loss). #30192 (Azat Khuzhin).
• • Allow identifiers staring with numbers in multiple joins. #30230 (Vladimir C).
• FlatDictionary, HashedDictionary fix bytes_allocated calculation for nullable attributes. #30238 (Maksim Kita).
• Fix crash with shortcircuit and lowcardinality in multiIf. #30243 (Raúl Marín).
• Fix [I]LIKE function. Closes #28661. #30244 (Nikolay Degterinsky).
• Fix ComplexKeyHashedDictionary, ComplexKeySparseHashedDictionary parsing preallocate option from layout config. #30246 (Maksim Kita).
• fix replaceRegexpAll bug. #30292 (Memo).
• Fix column alias resolution of JOIN queries when projection is enabled. This fixes #30146. #30293 (Amos Bird).
• Queries with condition like IN (subquery) could return incorrect result in case if aggregate projection applied. Fixed creation of sets for projections. #30310 (Amos Bird).
• Makes #30162 less possible ... #30370 (Vasily Nemkov).
• Fixed segfault which might happen if session expired during execution of REPLACE PARTITION. #30432 (Alexander Tokmakov).
• • Fix deadlock on ALTER with scalar subquery to the same table, close #30461. #30492 (Vladimir C).
• Add missing parenthesis for isNotNull/isNull rewrites to IS [NOT] NULL (fixes queries that has something like isNotNull(1)+isNotNull(2)). #30520 (Azat Khuzhin).
• Limit push down optimization could cause a error Cannot find column. Fixes #30438. #30562 (Nikolai Kochetov).
• Update aws-sdk submodule for throttling in Yandex.S3. #30646 (ianton-ru).
• Functions for case-insensitive search in UTF8 strings like positionCaseInsensitiveUTF8 and countSubstringsCaseInsensitiveUTF8 might find substrings that actually does not match, it's fixed. #30663 (Alexander Tokmakov).
• Fix PREWHERE with WHERE in case of always true PREWHERE. #30668 (Azat Khuzhin).
• Fixed a race condition between REPLACE/MOVE PARTITION and background merge in non-replicated MergeTree that might cause a part of moved/replaced data to remain in partition. Fixes #29327. #30717 (Alexander Tokmakov).
• Clean temporary directory when localBackup failed by some reason. #30797 (ianton-ru).
• Fixed ambiguity when extracting auxiliary ZooKeeper name from ZooKeeper path in ReplicatedMergeTree. Previously server might fail to start with Unknown auxiliary ZooKeeper name if ZooKeeper path contains a colon. Fixes #29052. Also it was allowed to specify ZooKeeper path that does not start with slash, but now it's deprecated and creation of new tables with such path is not allowed. Slashes and colons in auxiliary ZooKeeper names are not allowed too. #30822 (Alexander Tokmakov).
• Fix crash when projection with hashing function is materialized. This fixes #30861 . The issue is similar to https://github.com/ClickHouse/ClickHouse/pull/28560 which is a lack of proper understanding of the invariant of header's emptyness. #30877 (Amos Bird).
• Fix set index not used in AND/OR expressions when there are more than two operands. This fixes #30416 . #30887 (Amos Bird).

NO CL ENTRY

• NO CL ENTRY: 'Generate test_results.tsv for fasttest'. #29319 (Dmitry Novik).
• NO CL ENTRY: 'Revert "Add coroutines example."'. #29829 (Nikolai Kochetov).
• NO CL ENTRY: 'Link FAQ from Debian installation'. #29836 (Federico Ceratto).
• NO CL ENTRY: 'add support of window function in antlr grammar'. #30181 (PHaroZ).
• NO CL ENTRY: 'Revert "Fix style regressions on benchmark page"'. #30652 (alesapin).
• NO CL ENTRY: 'Revert "Improve usability of remote_url_allow_hosts"'. #30707 (Nikolai Kochetov).
• NO CL ENTRY: 'Revert "Revert "Improve usability of remote_url_allow_hosts""'. #30708 (Nikolai Kochetov).
• NO CL ENTRY: 'remove some unneeded header files'. #30722 (flynn).

Testing Improvement

• Implemented structure-aware fuzzing approach in ClickHouse for select statement parser. #30012 (Paul).