ClickHouse/docs/changelogs/v23.10.1.1976-stable.md

sidebar_position	sidebar_label
1	2023

2023 Changelog

ClickHouse release v23.10.1.1976-stable (13adae0e42) FIXME as compared to v23.9.1.1854-stable (8f9a227de1)

Backward Incompatible Change

• Rewrited storage S3Queue completely: changed the way we keep information in zookeeper which allows to make less zookeeper requests, added caching of zookeeper state in cases when we know the state will not change, improved the polling from s3 process to make it less aggressive, changed the way ttl and max set for trached files is maintained, now it is a background process. Added system.s3queue and system.s3queue_log tables. Closes #54998. #54422 (Kseniia Sumarokova).
• There is no longer an option to automatically remove broken data parts. This closes #55174. #55184 (Alexey Milovidov).
• The obsolete in-memory data parts can no longer be read from the write-ahead log. If you have configured in-memory parts before, they have to be removed before the upgrade. #55186 (Alexey Milovidov).
• Remove the integration with Meilisearch. Reason: it was compatible only with the old version 0.18. The recent version of Meilisearch changed the protocol and does not work anymore. Note: we would appreciate it if you help to return it back. #55189 (Alexey Milovidov).
• Rename directory monitor concept into background INSERT. All settings *directory_monitor* had been renamed to distributed_background_insert*. Backward compatibility should be preserved (since old settings had been added as an alias). #55978 (Azat Khuzhin).
• Do not mix-up send_timeout and receive_timeout. #56035 (Azat Khuzhin).
• Comparison of time intervals with different units will throw an exception. This closes #55942. You might have occasionally rely on the previous behavior when the underlying numeric values were compared regardless of the units. #56090 (Alexey Milovidov).

New Feature

• Add function "arrayFold(x1, ..., xn, accum -> expression, array1, ..., arrayn, init_accum)" which applies a lambda function to multiple arrays of the same cardinality and collects the result in an accumulator. #49794 (Lirikl).
• Added aggregation function lttb which uses the Largest-Triangle-Three-Buckets algorithm for downsampling data for visualization. #53145 (Sinan).
• QueryCHECK TABLE has better performance and usability (sends progress updates, cancellable). Support checking particular part with CHECK TABLE ... PART 'part_name'. #53404 (vdimir).
• Added function jsonMergePatch. When working with JSON data as strings, it provides a way to merge these strings (of JSON objects) together to form a single string containing a single JSON object. #54364 (Memo).
• Added a new SQL function, "arrayRandomSample(arr, k)" which returns a sample of k elements from the input array. Similar functionality could previously be achieved only with less convenient syntax, e.g. "SELECT arrayReduce('groupArraySample(3)', range(10))". #54391 (itayisraelov).
• Added new function getHttpHeader to get HTTP request header value used for a request to ClickHouse server. Return empty string if the request is not done over HTTP protocol or there is no such header. #54813 (凌涛).
• Introduce -ArgMin/-ArgMax aggregate combinators which allow to aggregate by min/max values only. One use case can be found in #54818. This PR also reorganize combinators into dedicated folder. #54947 (Amos Bird).
• Allow to drop cache for Protobuf format with SYSTEM DROP SCHEMA FORMAT CACHE [FOR Protobuf]. #55064 (Aleksandr Musorin).
• Add external HTTP Basic authenticator. #55199 (Aleksei Filatov).
• Added function byteSwap which reverses the bytes of unsigned integers. This is particularly useful for reversing values of types which are represented as unsigned integers internally such as IPv4. #55211 (Priyansh Agrawal).
• Added function formatQuery() which returns a formatted version (possibly spanning multiple lines) of a SQL query string. Also added function formatQuerySingleLine() which does the same but the returned string will not contain linebreaks. #55239 (Salvatore Mesoraca).
• Added DWARF input format that reads debug symbols from an ELF executable/library/object file. #55450 (Michael Kolupaev).
• Allow to save unparsed records and errors in RabbitMQ, NATS and FileLog engines. Add virtual columns _error and _raw_message(for NATS and RabbitMQ), _raw_record (for FileLog) that are filled when ClickHouse fails to parse new record. The behaviour is controlled under storage settings nats_handle_error_mode for NATS, rabbitmq_handle_error_mode for RabbitMQ, handle_error_mode for FileLog similar to kafka_handle_error_mode. If it's set to default, en exception will be thrown when ClickHouse fails to parse a record, if it's set to stream, erorr and raw record will be saved into virtual columns. Closes #36035. #55477 (Kruglov Pavel).
• Keeper client improvement: add get_all_children_number command that returns number of all children nodes under a specific path. #55485 (guoxiaolong).
• If a table has a space-filling curve in its key, e.g., ORDER BY mortonEncode(x, y), the conditions on its arguments, e.g., x >= 10 AND x <= 20 AND y >= 20 AND y <= 30 can be used for indexing. A setting analyze_index_with_space_filling_curves is added to enable or disable this analysis. This closes #41195. Continuation of #4538. Continuation of #6286. Continuation of #28130. Continuation of #41753. #55642 (Alexey Milovidov).
• Add setting optimize_trivial_approximate_count_query to use count() approximation for storage EmbeddedRocksDB. Enable trivial count for StorageJoin. #55806 (Duc Canh Le).
• Keeper client improvement: add get_direct_children_number command that returns number of direct children nodes under a path. #55898 (xuzifu666).
• Add statement SHOW SETTING setting_name which is a simpler version of existing statement SHOW SETTINGS. #55979 (Maksim Kita).
• This pr gives possibility to pass data in Npy format to Clickhouse. ``` SELECT * FROM file('example_array.npy', Npy). #55982 (Yarik Briukhovetskyi).
• This PR impleents a new setting called force_optimize_projection_name, it takes a name of projection as an argument. If it's value set to a non-empty string, ClickHouse checks that this projection is used in the query at least once. Closes #55331. #56134 (Yarik Briukhovetskyi).

Performance Improvement

• Add option query_plan_preserve_num_streams_after_window_functions to preserve the number of streams after evaluating window functions to allow parallel stream processing. #50771 (frinkr).
• Release more num_streams if data is small. #53867 (Jiebin Sun).
• RoaringBitmaps being optimized before serialization. #55044 (UnamedRus).
• Posting lists in inverted indexes are now optimized to use the smallest possible representation for internal bitmaps. Depending on the repetitiveness of the data, this may significantly reduce the space consumption of inverted indexes. #55069 (Harry Lee).
• Fix contention on Context lock, this significantly improves performance for a lot of short-running concurrent queries. #55121 (Maksim Kita).
• Improved the performance of inverted index creation by 30%. This was achieved by replacing std::unordered_map with absl::flat_hash_map. #55210 (Harry Lee).
• Support orc filter push down (rowgroup level). #55330 (李扬).
• Improve performance of external aggregation with a lot of temporary files. #55489 (Maksim Kita).
• Set a reasonable size for the marks cache for secondary indices by default to avoid loading the marks over and over again. #55654 (Alexey Milovidov).
• Avoid unnecessary reconstruction of index granules when reading skip indexes. This addresses #55653#issuecomment-1763766009 . #55683 (Amos Bird).
• Cache cast function in set during execution to improve the performance of function IN when set element type doesn't exactly match column type. #55712 (Duc Canh Le).
• Performance improvement for ColumnVector::insertMany and ColumnVector::insertManyFrom. #55714 (frinkr).
• ... Getting values from a map is widely used. In practice, the key structrues are usally the same in the same map column, we could try to predict the next row's key position and reduce the comparisons. #55929 (lgbo).
• ... Fix an issue that struct field prune doesn't work in some cases. For example ```sql INSERT INTO FUNCTION file('test_parquet_struct', Parquet, 'x Tuple(a UInt32, b UInt32, c String)') SELECT tuple(number, rand(), concat('testxxxxxxx' toString(number))) FROM numbers(10);. #56117 (lgbo).

Improvement

• This is the second part of Kusto Query Language dialect support. Phase 1 implementation has been merged. #42510 (larryluogit).
• Op processors IDs are raw ptrs casted to UInt64. Print it in a prettier manner:. #48852 (Vlad Seliverstov).
• Creating a direct dictionary with a lifetime field set will be rejected at create time. Fixes: #27861. #49043 (Rory Crispin).
• Allow parameters in queries with partitions like ALTER TABLE t DROP PARTITION. Closes #49449. #49516 (Nikolay Degterinsky).
• 1.Refactor the code about zookeeper_connection 2.Add a new column xid for zookeeper_connection. #50702 (helifu).
• Add the ability to tune the number of parallel replicas used in a query execution based on the estimation of rows to read. #51692 (Raúl Marín).
• Distributed queries executed in async_socket_for_remote mode (default) now respect max_threads limit. Previously, some queries could create excessive threads (up to max_distributed_connections), causing server performance issues. #53504 (filimonov).
• Display the correct server settings after reload. #53774 (helifu).
• Add support for mathematical minus − character in queries, similar to -. #54100 (Alexey Milovidov).
• Add replica groups to the Replicated database engine. Closes #53620. #54421 (Nikolay Degterinsky).
• This PR will fix UBsan test error here. #54568 (JackyWoo).
• Support asynchronous inserts with external data via native protocol. Previously it worked only if data is inlined into query. #54730 (Anton Popov).
• It is better to retry retriable s3 errors than totally fail the query. Set bigger value to the s3_retry_attempts by default. #54770 (Sema Checherinda).
• Optimised external aggregation memory consumption in case many temporary files were generated. #54798 (Nikita Taranov).
• Add load balancing test_hostname_levenshtein_distance. #54826 (JackyWoo).
• Caching skip-able entries while executing DDL from Zookeeper distributed DDL queue. #54828 (Duc Canh Le).
• Improve hiding secrets in logs. #55089 (Vitaly Baranov).
• Added fields substreams and filenames to the system.parts_columns table. #55108 (Anton Popov).
• For now the projection analysis will be performed only on top of query plan. The setting query_plan_optimize_projection became obsolete (it was enabled by default long time ago). #55112 (Nikita Mikhaylov).
• When function "untuple()" is now called on a tuple with named elements and itself has an alias (e.g. "select untuple(tuple(1)::Tuple(element_alias Int)) AS untuple_alias"), then the result column name is now generated from the untuple alias and the tuple element alias (in the example: "untuple_alias.element_alias"). #55123 (garcher22).
• Added setting describe_include_virtual_columns, which allows to include virtual columns of table into result of DESCRIBE query. Added setting describe_compact_output. If it is set to true, DESCRIBE query returns only names and types of columns without extra information. #55129 (Anton Popov).
• Sometimes OPTIMIZE with optimize_throw_if_noop=1 may fail with an error unknown reason while the real cause of it - different projections in different parts. This behavior is fixed. #55130 (Nikita Mikhaylov).
• Allow to have several MaterializedPostgreSQL tables following the same Postgres table. By default this behaviour is not enabled (for compatibility, because it is backward-incompatible change), but can be turned on with setting materialized_postgresql_use_unique_replication_consumer_identifier. Closes #54918. #55145 (Kseniia Sumarokova).
• Allow to parse negative DateTime64 and DateTime with fractional part from short strings. #55146 (Andrey Zvonov).
• To improve compatibility with MySQL, 1. "information_schema.tables" now includes the new field "table_rows", and 2. "information_schema.columns" now includes the new field "extra". #55215 (Robert Schulze).
• Clickhouse-client won't show "0 rows in set" if it is zero and if exception was thrown. #55240 (Salvatore Mesoraca).
• Support rename table without keyword TABLE like RENAME db.t1 to db.t2. #55373 (凌涛).
• Add internal_replication to system.clusters. #55377 (Konstantin Morozov).
• Select remote proxy resolver based on request protocol, add proxy feature docs and remove DB::ProxyConfiguration::Protocol::ANY. #55430 (Arthur Passos).
• Avoid retrying keeper operations on INSERT after table shutdown. #55519 (Azat Khuzhin).
• Improved overall resilience for ClickHouse in case of many parts within partition (more than 1000). It might reduce the number of TOO_MANY_PARTS errors. #55526 (Nikita Mikhaylov).
• Follow up https://github.com/ClickHouse/ClickHouse/pull/55184 to not to fall into UNKNOWN_SETTING error when the user uses the deleted MergeTree settings. #55557 (Jihyuk Bok).
• Updated dashboard.html() added a if to check if there is one chart, then "maximize" and "drag" buttons are not shown. #55581 (bhavuk2002).
• Functions toDayOfWeek() (MySQL alias: DAYOFWEEK()), toYearWeek() (YEARWEEK()) and toWeek() (WEEK()) now supports String arguments. This makes its behavior consistent with MySQL's behavior. #55589 (Robert Schulze).
• Implement query parameters support for ALTER TABLE ... ACTION PARTITION [ID] {parameter_name:ParameterType}. Merges #49516. Closes #49449. #55604 (alesapin).
• Inverted indexes do not store tokens with too many matches (i.e. row ids in the posting list). This saves space and avoids ineffective index lookups when sequential scans would be equally fast or faster. The previous heuristics (density parameter passed to the index definition) that controlled when tokens would not be stored was too confusing for users. A much simpler heuristics based on parameter max_rows_per_postings_list (default: 64k) is introduced which directly controls the maximum allowed number of row ids in a postings list. #55616 (Harry Lee).
• SHOW COLUMNS now correctly reports type FixedString as BLOB if setting use_mysql_types_in_show_columns is on. Also added two new settings, mysql_map_string_to_text_in_show_columns and mysql_map_fixed_string_to_text_in_show_columns to switch the output for types String and FixedString as TEXT or BLOB. #55617 (Serge Klochkov).
• During ReplicatedMergeTree tables startup clickhouse server checks set of parts for unexpected parts (exists locally, but not in zookeeper). All unexpected parts move to detached directory and instead of them server tries to restore some ancestor (covered) parts. Now server tries to restore closest ancestors instead of random covered parts. #55645 (alesapin).
• The advanced dashboard now supports draggable charts on touch devices. This closes #54206. #55649 (Alexey Milovidov).
• Introduced setting date_time_overflow_behavior with possible values ignore, throw, saturate that controls the overflow behavior when converting from Date, Date32, DateTime64, Integer or Float to Date, Date32, DateTime or DateTime64. #55696 (Andrey Zvonov).
• Improve write performance to rocksdb. #55732 (Duc Canh Le).
• Use the default query format if declared when outputting exception with http_write_exception_in_output_format. #55739 (Raúl Marín).
• Use upstream repo for apache datasketches. #55787 (Nikita Taranov).
• Add support for SHOW MERGES query. #55815 (megao).
• Provide a better message for common MV pitfalls. #55826 (Raúl Marín).
• Reduced memory consumption during loading of hierarchical dictionaries. #55838 (Nikita Taranov).
• All dictionaries support setting dictionary_use_async_executor. #55839 (vdimir).
• If you dropped the current database, you will still be able to run some queries in clickhouse-local and switch to another database. This makes the behavior consistent with clickhouse-client. This closes #55834. #55853 (Alexey Milovidov).
• Functions (add|subtract)(Year|Quarter|Month|Week|Day|Hour|Minute|Second|Millisecond|Microsecond|Nanosecond) now support string-encoded date arguments, e.g. SELECT addDays('2023-10-22', 1). This increases compatibility with MySQL and is needed by Tableau Online. #55869 (Robert Schulze).
• Introduce setting create_table_empty_primary_key_by_default for default ORDER BY (). #55899 (Srikanth Chekuri).
• Prevent excesive memory usage when deserializing AggregateFunctionTopKGenericData. #55947 (Raúl Marín).
• The setting apply_deleted_mask when disabled allows to read rows that where marked as deleted by lightweight DELETE queries. This is useful for debugging. #55952 (Alexander Gololobov).
• Allow skip null values when serailize tuple to json objects, which makes it possible to keep compatiability with spark to_json function, which is also useful for gluten. #55956 (李扬).
• Functions (add|sub)Date() now support string-encoded date arguments, e.g. SELECT addDate('2023-10-22 11:12:13', INTERVAL 5 MINUTE). The same support for string-encoded date arguments is added to the plus and minus operators, e.g. SELECT '2023-10-23' + INTERVAL 1 DAY. This increases compatibility with MySQL and is needed by Tableau Online. #55960 (Robert Schulze).
• Allow unquoted strings with CR in CSV format. Closes #39930. #56046 (Kruglov Pavel).
• On a Keeper with lots of watches AsyncMetrics threads can consume 100% of CPU for noticable time in DB::KeeperStorage::getSessionsWithWatchesCount(). The fix is to avoid traversing heavy watches and list_watches sets. #56054 (Alexander Gololobov).
• Allow to run clickhouse-keeper using embedded config. #56086 (Maksim Kita).
• Set limit of the maximum configuration value for queued.min.messages to avoid problem with start fetching data with Kafka. #56121 (Stas Morozov).
• Fixed typo in SQL function minSampleSizeContinous (renamed minSampleSizeContinuous). Old name is preserved for backward compatibility. This closes: #56139. #56143 (Dorota Szeremeta).
• Print corrupted part path on disk before shutdown server. Before this change if a part is corrupted on disk and server cannot start, it was almost impossible to understand which part is broken. This is fixed. #56181 (Duc Canh Le).

Build/Testing/Packaging Improvement

• If the database is already initialized, it doesn't need to be initialized again upon subsequent launches. This can potentially fix the issue of infinite container restarts when the database fails to load within 1000 attempts (relevant for very large databases and multi-node setups). #50724 (Alexander Nikolaev).
• Resource with source code including submodules is built in Darwin special build task. It may be used to build ClickHouse without checkouting submodules. #51435 (Ilya Yatsishin).
• An error will occur when building ClickHouse with the avx series of instructions enabled. CMake command： shell cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_AVX=ON -DENABLE_AVX2=ON -DENABLE_AVX2_FOR_SPEC_OP=ON Failed message： [1558/11630] Building CXX object contrib/snappy-cmake/CMakeFiles/_snappy.dir/__/snappy/snappy.cc.o FAILED: contrib/snappy-cmake/CMakeFiles/_snappy.dir/__/snappy/snappy.cc.o /usr/bin/ccache /usr/bin/clang++-17 --target=x86_64-linux-gnu --sysroot=/opt/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc -DHAVE_CONFIG_H -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -I/opt/ClickHouse/base/glibc-compatibility/memcpy -isystem /opt/ClickHouse/contrib/snappy -isystem /opt/ClickHouse/build/contrib/snappy-cmake -isystem /opt/ClickHouse/contrib/llvm-project/libcxx/include -isystem /opt/ClickHouse/contrib/llvm-project/libcxxabi/include -isystem /opt/ClickHouse/contrib/libunwind/include --gcc-toolchain=/opt/ClickHouse/cmake/linux/../../contrib/sysroot/linux-x86_64 -fdiagnostics-color=always -Wno-enum-constexpr-conversion -fsized-deallocation -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -mavx -mavx2 -fasynchronous-unwind-tables -ffile-prefix-map=/opt/ClickHouse=. -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -w -O3 -DNDEBUG -D OS_LINUX -Werror -nostdinc++ -std=c++2b -MD -MT contrib/snappy-cmake/CMakeFiles/_snappy.dir/__/snappy/snappy.cc.o -MF contrib/snappy-cmake/CMakeFiles/_snappy.dir/__/snappy/snappy.cc.o.d -o contrib/snappy-cmake/CMakeFiles/_snappy.dir/__/snappy/snappy.cc.o -c /opt/ClickHouse/contrib/snappy/snappy.cc /opt/ClickHouse/contrib/snappy/snappy.cc:1061:3: error: unknown type name '__m256i' 1061 | __m256i data = _mm256_lddqu_si256(static_cast<const __m256i *>(src)); | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1061:55: error: unknown type name '__m256i' 1061 | __m256i data = _mm256_lddqu_si256(static_cast<const __m256i *>(src)); | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1061:18: error: use of undeclared identifier '_mm256_lddqu_si256'; did you mean '_mm_lddqu_si128'? 1061 | __m256i data = _mm256_lddqu_si256(static_cast<const __m256i *>(src)); | ^~~~~~~~~~~~~~~~~~ | _mm_lddqu_si128 /usr/lib/llvm-17/lib/clang/17/include/pmmintrin.h:38:1: note: '_mm_lddqu_si128' declared here 38 | _mm_lddqu_si128(__m128i_u const *__p) | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1061:66: error: cannot initialize a parameter of type 'const __m128i_u *' with an lvalue of type 'const void *' 1061 | __m256i data = _mm256_lddqu_si256(static_cast<const __m256i *>(src)); | ^~~ /usr/lib/llvm-17/lib/clang/17/include/pmmintrin.h:38:34: note: passing argument to parameter '__p' here 38 | _mm_lddqu_si128(__m128i_u const *__p) | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1062:40: error: unknown type name '__m256i' 1062 | _mm256_storeu_si256(reinterpret_cast<__m256i *>(dst), data); | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1065:49: error: unknown type name '__m256i' 1065 | data = _mm256_lddqu_si256(static_cast<const __m256i *>(src) + 1); | ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1065:65: error: arithmetic on a pointer to void 1065 | data = _mm256_lddqu_si256(static_cast<const __m256i *>(src) + 1); | ~~~ ^ /opt/ClickHouse/contrib/snappy/snappy.cc:1066:42: error: unknown type name '__m256i' 1066 | _mm256_storeu_si256(reinterpret_cast<__m256i *>(dst) + 1, data); | ^ 8 errors generated. [1567/11630] Building CXX object contrib/rocksdb-cma...rocksdb.dir/__/rocksdb/db/arena_wrapped_db_iter.cc.o ninja: build stopped: subcommand failed. The reason is that snappy does not enable SNAPPY_HAVE_X86_CRC32. #55049 (monchickey).
• Add instance_env_variables option to integration tests. #55208 (Arthur Passos).
• Solve issue with launching standalone clickhouse-keeper from clickhouse-server package. #55226 (Mikhail f. Shiryaev).
• In tests RabbitMQ version is updated to 3.12.6. Improved logs collection for RabbitMQ tests. #55424 (Ilya Yatsishin).
• Fix integration check python script to use gh api url - Add Readme for CI tests. #55476 (Max K.).
• Fix integration check python script to use gh api url - Add Readme for CI tests. #55716 (Max K.).
• Check sha512 for tgz; use a proper repository for keeper; write only filenames to TGZ.sha512 files for tarball packages. Prerequisite for #31473. #55717 (Mikhail f. Shiryaev).
• Updated to get free port for azurite. #55796 (SmitaRKulkarni).
• Reduce the ouput info. #55938 (helifu).
• Modified the error message difference between openssl and boringssl to fix the functional test. #55975 (MeenaRenganathan22).
• Changes to support the HDFS for s390x. #56128 (MeenaRenganathan22).
• Fix flaky test of jbod balancer by relaxing the Gini coefficient and introducing more determinism in insertions. #56175 (Amos Bird).

Bug Fix (user-visible misbehavior in an official stable release)

• skip hardlinking inverted index files in mutation #47663 (cangyin).
• Fix 'Cannot find column' in read-in-order optimization with ARRAY JOIN #51746 (Nikolai Kochetov).
• Support missed Object(Nullable(json)) subcolumns in query. #54052 (zps).
• Re-add fix for accurateCastOrNull() #54629 (Salvatore Mesoraca).
• Fix detecting DEFAULT for columns of a Distributed table created without AS #55060 (Vitaly Baranov).
• Proper cleanup in case of exception in ctor of ShellCommandSource #55103 (Alexander Gololobov).
• Fix deadlock in LDAP assigned role update #55119 (Julian Maicher).
• Suppress error statistics update for internal exceptions #55128 (Robert Schulze).
• Fix deadlock in backups #55132 (alesapin).
• Fix storage Iceberg files retrieval #55144 (Kseniia Sumarokova).
• Fix partition pruning of extra columns in set. #55172 (Amos Bird).
• Fix recalculation of skip indexes in ALTER UPDATE queries when table has adaptive granularity #55202 (Duc Canh Le).
• Fix for background download in fs cache #55252 (Kseniia Sumarokova).
• Avoid possible memory leaks in compressors in case of missing buffer finalization #55262 (Azat Khuzhin).
• Fix functions execution over sparse columns #55275 (Azat Khuzhin).
• Fix incorrect merging of Nested for SELECT FINAL FROM SummingMergeTree #55276 (Azat Khuzhin).
• Fix bug with inability to drop detached partition in replicated merge tree on top of S3 without zero copy #55309 (alesapin).
• Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge()) #55335 (Azat Khuzhin).
• Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception #55338 (Azat Khuzhin).
• Fix trash optimization (up to a certain extent) #55353 (Alexey Milovidov).
• Fix leak in StorageHDFS #55370 (Azat Khuzhin).
• Fix parsing of arrays in cast operator #55417 (Anton Popov).
• Fix filtering by virtual columns with OR filter in query #55418 (Azat Khuzhin).
• Fix MongoDB connection issues #55419 (Nikolay Degterinsky).
• Fix MySQL interface boolean representation #55427 (Serge Klochkov).
• Fix MySQL text protocol DateTime formatting and LowCardinality(Nullable(T)) types reporting #55479 (Serge Klochkov).
• Make use_mysql_types_in_show_columns affect only SHOW COLUMNS #55481 (Robert Schulze).
• Fix stack symbolizer parsing DW_FORM_ref_addr incorrectly and sometimes crashing #55483 (Michael Kolupaev).
• Destroy fiber in case of exception in cancelBefore in AsyncTaskExecutor #55516 (Kruglov Pavel).
• Fix Query Parameters not working with custom HTTP handlers #55521 (Konstantin Bogdanov).
• Fix checking of non handled data for Values format #55527 (Azat Khuzhin).
• Fix 'Invalid cursor state' in odbc interacting with MS SQL Server #55558 (vdimir).
• Fix max execution time and 'break' overflow mode #55577 (Alexander Gololobov).
• Fix crash in QueryNormalizer with cyclic aliases #55602 (vdimir).
• Disable wrong optimization and add a test #55609 (Alexey Milovidov).
• Merging #52352 #55621 (Alexey Milovidov).
• Add a test to avoid incorrect decimal sorting #55662 (Amos Bird).
• Fix progress bar for s3 and azure Cluster functions with url without globs #55666 (Kruglov Pavel).
• Fix filtering by virtual columns with OR filter in query (resubmit) #55678 (Azat Khuzhin).
• Fixes and improvements for Iceberg storage #55695 (Kruglov Pavel).
• Fix data race in CreatingSetsTransform (v2) #55786 (Azat Khuzhin).
• Throw exception when parsing illegal string as float if precise_float_parsing is true #55861 (李扬).
• Disable predicate pushdown if the CTE contains stateful functions #55871 (Raúl Marín).
• Fix normalize ASTSelectWithUnionQuery strip FORMAT of the query #55887 (flynn).
• Try to fix possible segfault in Native ORC input format #55891 (Kruglov Pavel).
• Fix window functions in case of sparse columns. #55895 (János Benjamin Antal).
• fix: StorageNull supports subcolumns #55912 (FFish).
• Do not write retriable errors for Replicated mutate/merge into error log #55944 (Azat Khuzhin).
• Fix SHOW DATABASES LIMIT <N> #55962 (Raúl Marín).
• Fix autogenerated Protobuf schema with fields with underscore #55974 (Kruglov Pavel).
• Fix dateTime64ToSnowflake64() with non-default scale #55983 (Robert Schulze).
• Fix output/input of Arrow dictionary column #55989 (Kruglov Pavel).
• Fix fetching schema from schema registry in AvroConfluent #55991 (Kruglov Pavel).
• Fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table #55995 (Michael Kolupaev).
• Fix incorrect free space accounting for least_used JBOD policy #56030 (Azat Khuzhin).
• Fix missing scalar issue when evaluating subqueries inside table functions #56057 (Amos Bird).
• Fix wrong query result when http_write_exception_in_output_format=1 #56135 (Kruglov Pavel).
• Fix schema cache for fallback JSON->JSONEachRow with changed settings #56172 (Kruglov Pavel).
• Add error handler to odbc-bridge #56185 (Yakov Olkhovskiy).

NO CL ENTRY

• NO CL ENTRY: 'Revert "Fix libssh+openssl3 & s390x (part 2)"'. #55188 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Support SAMPLE BY for VIEW"'. #55357 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Revert "refine error code of duplicated index in create query""'. #55467 (Han Fei).
• NO CL ENTRY: 'Update mysql.md - Remove the Private Preview Note'. #55486 (Ryadh DAHIMENE).
• NO CL ENTRY: 'Revert "Removed "maximize" and "drag" buttons from dashboard in case of single chart"'. #55623 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Fix filtering by virtual columns with OR filter in query"'. #55657 (Antonio Andelic).
• NO CL ENTRY: 'Revert "Improve ColumnDecimal, ColumnVector getPermutation performance using pdqsort with RadixSort"'. #55682 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Integration check script fix ups"'. #55694 (alesapin).
• NO CL ENTRY: 'Revert "Fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table"'. #56103 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Add function getHttpHeader"'. #56109 (Alexey Milovidov).
• NO CL ENTRY: 'Revert "Fix output/input of Arrow dictionary column"'. #56150 (Alexey Milovidov).

NOT FOR CHANGELOG / INSIGNIFICANT

• Set defaults_for_omitted_fields to true for hive text format #49486 (李扬).
• Fixing join tests with analyzer #49555 (vdimir).
• Make exception about ALTER TABLE ... DROP COLUMN|INDEX|PROJECTION more clear #50181 (Alexander Gololobov).
• ANTI JOIN: Invalid number of rows in Chunk #50944 (Nikolai Kochetov).
• Analyzer: fix row policy #53170 (Dmitry Novik).
• Add a test with Block structure mismatch in grace hash join. #53278 (Nikolai Kochetov).
• Support skip_unused_shards in Analyzer #53282 (Nikolai Kochetov).
• Revert "Revert "Planner prepare filters for analysis"" #53792 (Nikolai Kochetov).
• Add setting allow_experimental_partial_result #54514 (vdimir).
• Fix CI skip build and skip tests checks #54532 (SmitaRKulkarni).
• MergeTreeData::forcefullyMovePartToDetachedAndRemoveFromMemory does not respect mutations #54653 (Nikolai Kochetov).
• Fix azure test by using unique names #54738 (SmitaRKulkarni).
• Use --filter to reduce checkout time #54857 (Mikhail f. Shiryaev).
• Remove tests #54873 (János Benjamin Antal).
• Remove useless path from lockSharedData in StorageReplicatedMergeTree #54989 (Mike Kot).
• Fix broken test #55002 (Kseniia Sumarokova).
• Update version_date.tsv and changelogs after v23.8.3.48-lts #55063 (robot-clickhouse).
• Use source instead of bash for pre-build script #55071 (Nikita Mikhaylov).
• Update s3queue.md to add experimental flag #55093 (Peignon Melvyn).
• Clean data dir and always start an old server version in aggregate functions compatibility test. #55105 (Nikolai Kochetov).
• Update version_date.tsv and changelogs after v23.9.1.1854-stable #55118 (robot-clickhouse).
• Add links to check reports in status comment #55122 (vdimir).
• Bump croaring to 2.0.2 #55127 (Robert Schulze).
• check if block is empty after async insert retries #55143 (Han Fei).
• Improve linker detection on macOS #55147 (Robert Schulze).
• Fix libssh+openssl3 & s390x #55154 (Boris Kuschel).
• Fix file cache temporary file segment range in FileSegment::reserve #55164 (vdimir).
• Fix libssh+openssl3 & s390x (part 2) #55187 (Boris Kuschel).
• Fix wrong test name #55190 (Alexey Milovidov).
• Fix libssh+openssl3 & s390x (Part 2.1) #55192 (Boris Kuschel).
• Reset signals caught by clickhouse-client if a pager is in use #55193 (Azat Khuzhin).
• Update README.md #55209 (Tyler Hannan).
• remove the blocker to grow the metadata file version #55218 (Sema Checherinda).
• Fix syntax highlight in client for spaceship operator #55224 (Azat Khuzhin).
• Fix mypy errors #55228 (Mikhail f. Shiryaev).
• Upgrade MinIO to support accepting non signed requests #55245 (Azat Khuzhin).
• tests: switch test_throttling to S3 over https to make it more production like #55247 (Azat Khuzhin).
• Evaluate defaults during async insert safer #55253 (Vitaly Baranov).
• Fix data race in context #55260 (Kseniia Sumarokova).
• Do not allow tests with state ERROR be overwritten by PASSED #55261 (Azat Khuzhin).
• Fix query formatting for SYSTEM queries #55277 (Azat Khuzhin).
• Context added TSA #55278 (Maksim Kita).
• Improve logging in query cache #55296 (Robert Schulze).
• MaterializedPostgreSQL: remove back check #55297 (Kseniia Sumarokova).
• Make use_mysql_types_in_show_columns independent from the connection #55298 (Robert Schulze).
• Make HandlingRuleHTTPHandlerFactory more stupid, but less error prone #55307 (alesapin).
• Fix tsan issue in croaring #55311 (Robert Schulze).
• Refactorings and better documentation for toStartOfInterval() #55327 (Robert Schulze).
• Review #51946 and partially revert it #55336 (Alexey Milovidov).
• Update README.md #55339 (Tyler Hannan).
• Context locks small fixes #55352 (Maksim Kita).
• Fix bad test 01605_dictinct_two_level #55354 (Alexey Milovidov).
• Fix test_max_rows_to_read_leaf_with_view flakiness (due to prefer_localhost_replica) #55355 (Azat Khuzhin).
• Better exception messages #55356 (Alexey Milovidov).
• Update CHANGELOG.md #55359 (Alexey Milovidov).
• Better recursion depth check #55361 (Alexey Milovidov).
• Merging #43085 #55362 (Alexey Milovidov).
• Fix data-race in web disk #55372 (Azat Khuzhin).
• Disable skim under TSan (Rust does not supports ThreadSanitizer) #55378 (Azat Khuzhin).
• Fix missing thread accounting for insert_distributed_sync=1 #55392 (Azat Khuzhin).
• Improve tests for untuple() #55425 (Robert Schulze).
• Fix test that never worked test_rabbitmq_random_detach #55453 (Ilya Yatsishin).
• Fix out of bound error in system.remote_data_paths + disk web #55468 (alesapin).
• Remove existing moving/ dir if allow_remove_stale_moving_parts is off #55480 (Mike Kot).
• Bump curl to 8.4 #55492 (Robert Schulze).
• Minor fixes for 02882_replicated_fetch_checksums_doesnt_match.sql #55493 (vdimir).
• AggregatingTransform initGenerate race condition fix #55495 (Maksim Kita).
• HashTable resize exception handle fix #55497 (Maksim Kita).
• fix lots of 'Structure does not match' warnings in ci #55503 (Han Fei).
• Cleanup: parallel replica coordinator usage #55515 (Igor Nikonov).
• add k-morozov to trusted contributors #55523 (Mike Kot).
• Forbid create inverted index if setting not enabled #55529 (flynn).
• Better exception messages but without SEGFAULT #55541 (Antonio Andelic).
• Avoid setting same promise twice #55553 (Antonio Andelic).
• Better error message in case when merge selecting task failed. #55554 (Nikita Mikhaylov).
• Add a test #55564 (Kseniia Sumarokova).
• Added healthcheck for LDAP #55571 (Nikita Mikhaylov).
• Fix global context for tests with --gtest_filter #55583 (Azat Khuzhin).
• Fix replica groups for Replicated database engine #55587 (Azat Khuzhin).
• Remove unused protobuf includes #55590 (Raúl Marín).
• Apply Context changes to standalone Keeper #55591 (Antonio Andelic).
• Do not fail if label-to-remove does not exists in PR #55592 (Mikhail f. Shiryaev).
• CI: cast extra column expression pull_request_number to Int32 #55599 (Han Fei).
• Add back a test that was removed by mistake #55605 (Alexander Tokmakov).
• Bump croaring to v2.0.4 #55606 (Robert Schulze).
• byteswap: Add 16/32-byte integer support #55607 (Robert Schulze).
• Revert #54421 #55613 (Alexander Tokmakov).
• Fix: race condition in kusto implementation #55615 (Yakov Olkhovskiy).
• Remove passed tests from analyzer_tech_debt.txt #55618 (Alexey Milovidov).
• Enable 02161_addressToLineWithInlines #55622 (Michael Kolupaev).
• KeyCondition: preparation #55625 (Alexey Milovidov).
• Fix flakiness of test_system_merges (by increasing sleep interval properly) #55627 (Azat Khuzhin).
• fix structure does not match logs again #55628 (Han Fei).
• KeyCondition: small changes #55640 (Alexey Milovidov).
• Resubmit #54421 #55641 (Nikolay Degterinsky).
• Fix some typos #55646 (alesapin).
• Show move/maximize only if there is more than a single chart #55648 (Alexey Milovidov).
• Enable test_query_is_lock_free[detach table] for the analyzer #55668 (Raúl Marín).
• Allow FINAL with parallel replicas with custom key #55679 (Antonio Andelic).
• Fix StorageMaterializedView::isRemote #55681 (vdimir).
• Bump gRPC to 1.34.1 #55693 (Robert Schulze).
• Randomize block_number column setting in ci #55713 (SmitaRKulkarni).
• Use pool for proxied S3 disk http sessions #55718 (Aleksei Filatov).
• Analyzer: fix block stucture mismatch in matview with engine distributed #55741 (vdimir).
• Use diff object again, since JSON API limits the files #55750 (Mikhail f. Shiryaev).
• Big endian platform max intersection fix #55756 (Suzy Wang).
• Remove temporary debug logging in MultiplexedConnections #55764 (Michael Kolupaev).
• Check if partition ID is nullptr #55765 (Antonio Andelic).
• Control Keeper feature flag randomization with env #55766 (Antonio Andelic).
• Added test to check CapnProto cache #55769 (Aleksandr Musorin).
• Query Cache: Only cache initial query #55771 (zhongyuankai).
• Temporarily disable flaky test #55772 (Kseniia Sumarokova).
• Fix test test_postgresql_replica_database_engine_2/test.py::test_replica_consumer #55774 (Kseniia Sumarokova).
• Improve ColumnDecimal, ColumnVector getPermutation performance using pdqsort with RadixSort fix #55775 (Maksim Kita).
• Fix black check #55779 (Kseniia Sumarokova).
• Correctly grep fuzzer.log #55780 (Antonio Andelic).
• Parallel replicas: cleanup, less copying during announcement #55781 (Igor Nikonov).
• Enable test_mutation_simple with the analyzer #55791 (Raúl Marín).
• Improve enrich image #55793 (Mikhail f. Shiryaev).
• Bump gRPC to v1.36.4 #55811 (Robert Schulze).
• Attemp to fix test_dictionaries_redis flakiness #55813 (Raúl Marín).
• Add diagnostic checks for issue #55041 #55835 (Robert Schulze).
• Correct aggregate functions ser/deserialization to be endianness-independent. #55837 (Austin Kothig).
• Bump gRPC to v1.37.1 #55840 (Robert Schulze).
• Fix 00002_log_and_exception_messages_formatting #55844 (Alexander Tokmakov).
• Fix caching objects in pygithub, and changelogs #55845 (Mikhail f. Shiryaev).
• Update version_date.tsv and changelogs after v23.3.14.78-lts #55847 (robot-clickhouse).
• Update version_date.tsv and changelogs after v23.9.2.56-stable #55848 (robot-clickhouse).
• Update version_date.tsv and changelogs after v23.8.4.69-lts #55849 (robot-clickhouse).
• fix node setting in the test #55850 (Sema Checherinda).
• Add load_metadata_threads to describe filesystem cache #55863 (Jordi Villar).
• One final leftover in diff_urls of PRInfo #55874 (Mikhail f. Shiryaev).
• Fix digest check in replicated ddl worker #55877 (Sergei Trifonov).
• Test parallel replicas with rollup #55886 (Raúl Marín).
• Fix some tests with Replicated database #55889 (Alexander Tokmakov).
• Update stress.py #55890 (Alexander Tokmakov).
• Revert "Revert "Revert "Add settings for real-time updates during query execution""" #55893 (Alexey Milovidov).
• Make test system_zookeeper_connection better #55900 (Alexey Milovidov).
• A test 01019_alter_materialized_view_consistent is unstable with Analyzer #55901 (Alexey Milovidov).
• Remove C++ templates, because they are stupid #55910 (Alexey Milovidov).
• Bump gRPC to v1.39.1 #55914 (Robert Schulze).
• Add sanity check to RPNBuilderFunctionTreeNode #55915 (Robert Schulze).
• Bump gRPC to v1.42.0 #55916 (Robert Schulze).
• Set storage.has_lightweight_delete_parts flag when a part has been loaded #55935 (Alexander Gololobov).
• Include information about supported versions in bug report issue template #55937 (Nikita Taranov).
• arrayFold: Switch accumulator and array arguments #55948 (Robert Schulze).
• Fix overrides via connections_credentials in case of root directives exists #55949 (Azat Khuzhin).
• Test for Bug 43644 #55955 (Robert Schulze).
• Bump protobuf to v3.19.6 #55963 (Robert Schulze).
• Fix possible performance test error #55964 (Azat Khuzhin).
• Avoid counting lost parts twice #55987 (Alexander Tokmakov).
• Convert unnecessary std::scoped_lock usage to std::lock_guard #56006 (Robert Schulze).
• Stress tests: Try to wait until server is responsive after gdb detach #56009 (Raúl Marín).
• test_storage_s3_queue - add debug info #56011 (Kseniia Sumarokova).
• Bump protobuf to v21.9 #56014 (Robert Schulze).
• Correct the implementation of function jsonMergePatch #56020 (Anton Popov).
• Fix 02438_sync_replica_lightweight #56023 (Alexander Tokmakov).
• Fix some bad code by making it worse #56026 (Alexander Tokmakov).
• Fix bash completion for mawk (and update format list and add one more delimiter) #56050 (Azat Khuzhin).
• Analyzer: Fix crash on window resolve #56055 (Dmitry Novik).
• Fix function_json_value_return_type_allow_nullable setting name in doc #56056 (vdimir).
• Force shutdown in upgrade test #56074 (Raúl Marín).
• Try enable 01154_move_partition_long with s3 #56080 (Alexander Tokmakov).
• Fix race condition between DROP_RANGE and committing existing block #56083 (Alexander Tokmakov).
• Fix flakiness of 02263_lazy_mark_load #56087 (Michael Kolupaev).
• Make the code less bloated #56091 (Alexey Milovidov).
• Maybe smaller binary #56112 (Alexey Milovidov).
• Remove old trash from unit tests #56113 (Alexey Milovidov).
• Remove some bloat #56114 (Alexey Milovidov).
• Fix test_format_schema_on_server flakiness #56116 (Azat Khuzhin).
• Fix: schedule delayed part checks correctly #56123 (Igor Nikonov).
• Beautify show merges #56124 (Denny Crane).
• Better options for disabling frame pointer omitting #56130 (Nikita Mikhaylov).
• Fix: incorrect brace style in clang-format #56133 (Igor Nikonov).
• Do not try to activate covered parts when handilng unexpected parts #56137 (Alexander Tokmakov).
• Re-fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table #56140 (Michael Kolupaev).
• Update version_date.tsv and changelogs after v23.3.15.29-lts #56145 (robot-clickhouse).
• Update version_date.tsv and changelogs after v23.8.5.16-lts #56146 (robot-clickhouse).
• Update version_date.tsv and changelogs after v23.9.3.12-stable #56147 (robot-clickhouse).
• Update version_date.tsv and changelogs after v23.7.6.111-stable #56148 (robot-clickhouse).
• Fasttest timeout setting #56160 (Max K.).
• Use monotonic clock for part check scheduling #56162 (Igor Nikonov).
• More metrics for fs cache #56165 (Kseniia Sumarokova).
• FileCache minor changes #56168 (Kseniia Sumarokova).
• Update 01414_mutations_and_errors_zookeeper.sh #56176 (Alexander Tokmakov).
• Shard fs cache keys #56194 (Kseniia Sumarokova).
• Do less work when there lots of read requests and watches for same paths #56197 (Alexander Gololobov).
• Remove skip_unused_shards tests from analyzer skiplist #56200 (Nikolai Kochetov).
• Easy tests fix for analyzer #56211 (Nikolai Kochetov).
• Add a warning if delayed accounting is not enabled (breaks OSIOWaitMicroseconds) #56227 (Azat Khuzhin).
• Do not remove part if Too many open files is thrown #56238 (Nikolay Degterinsky).
• Fix ORC commit #56261 (Raúl Marín).
• Fix typo in largestTriangleThreeBuckets.md #56263 (Nikita Taranov).