Merge remote-tracking branch 'upstream/master' into zk_exist_fix2

CHANGELOG.md

@@ -1,3 +1,77 @@
+## ClickHouse release 20.9
+
+### ClickHouse release v20.9.2.20-stable, 2020-09-22
+
+#### New Feature
+
+* Added column transformers `EXCEPT`, `REPLACE`, `APPLY`, which can be applied to the list of selected columns (after `*` or `COLUMNS(...)`). For example, you can write `SELECT * EXCEPT(URL) REPLACE(number + 1 AS number)`. Another example: `select * apply(length) apply(max) from wide_string_table` to find out the maxium length of all string columns. [#14233](https://github.com/ClickHouse/ClickHouse/pull/14233) ([Amos Bird](https://github.com/amosbird)).
+* Added an aggregate function `rankCorr` which computes a rank correlation coefficient. [#11769](https://github.com/ClickHouse/ClickHouse/pull/11769) ([antikvist](https://github.com/antikvist)) [#14411](https://github.com/ClickHouse/ClickHouse/pull/14411) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Added table function `view` which turns a subquery into a table object. This helps passing queries around. For instance, it can be used in remote/cluster table functions. [#12567](https://github.com/ClickHouse/ClickHouse/pull/12567) ([Amos Bird](https://github.com/amosbird)).
+
+#### Bug Fix
+
+* Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)).
+* Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)).
+* Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
+* Fixed inconsistent comparison with primary key of type `FixedString` on index analysis if they're compered with a string of less size. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)).
+* Fix bug which leads to wrong merges assignment if table has partitions with a single part. [#14444](https://github.com/ClickHouse/ClickHouse/pull/14444) ([alesapin](https://github.com/alesapin)).
+* If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)).
+* Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)).
+* Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)).
+* Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes [#14531](https://github.com/ClickHouse/ClickHouse/issues/14531). [#14845](https://github.com/ClickHouse/ClickHouse/pull/14845) ([alesapin](https://github.com/alesapin)).
+* Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)).
+* Fixed missed default database name in metadata of materialized view when executing `ALTER ... MODIFY QUERY`. [#14664](https://github.com/ClickHouse/ClickHouse/pull/14664) ([tavplubix](https://github.com/tavplubix)).
+* Fix possibly incorrect result of function `has` when LowCardinality and Nullable types are involved. [#14591](https://github.com/ClickHouse/ClickHouse/pull/14591) ([Mike](https://github.com/myrrc)).
+* Cleanup data directory after Zookeeper exceptions during CREATE query for tables with ReplicatedMergeTree Engine. [#14563](https://github.com/ClickHouse/ClickHouse/pull/14563) ([Bharat Nallan](https://github.com/bharatnc)).
+* Fix rare segfaults in functions with combinator `-Resample`, which could appear in result of overflow with very large parameters. [#14562](https://github.com/ClickHouse/ClickHouse/pull/14562) ([Anton Popov](https://github.com/CurtizJ)).
+* Check for array size overflow in `topK` aggregate function. Without this check the user may send a query with carefully crafted parameters that will lead to server crash. This closes [#14452](https://github.com/ClickHouse/ClickHouse/issues/14452). [#14467](https://github.com/ClickHouse/ClickHouse/pull/14467) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Proxy restart/start/stop/reload of SysVinit to systemd (if it is used). [#14460](https://github.com/ClickHouse/ClickHouse/pull/14460) ([Azat Khuzhin](https://github.com/azat)).
+* Stop query execution if exception happened in `PipelineExecutor` itself. This could prevent rare possible query hung. [#14334](https://github.com/ClickHouse/ClickHouse/pull/14334) [#14402](https://github.com/ClickHouse/ClickHouse/pull/14402) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Fix crash during `ALTER` query for table which was created `AS table_function`. Fixes [#14212](https://github.com/ClickHouse/ClickHouse/issues/14212). [#14326](https://github.com/ClickHouse/ClickHouse/pull/14326) ([alesapin](https://github.com/alesapin)).
+* Fix exception during ALTER LIVE VIEW query with REFRESH command. LIVE VIEW is an experimental feature. [#14320](https://github.com/ClickHouse/ClickHouse/pull/14320) ([Bharat Nallan](https://github.com/bharatnc)).
+* Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. [#14315](https://github.com/ClickHouse/ClickHouse/pull/14315) ([Azat Khuzhin](https://github.com/azat)).
+* Better check for tuple size in SSD cache complex key external dictionaries. This fixes [#13981](https://github.com/ClickHouse/ClickHouse/issues/13981). [#14313](https://github.com/ClickHouse/ClickHouse/pull/14313) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Disallows `CODEC` on `ALIAS` column type. Fixes [#13911](https://github.com/ClickHouse/ClickHouse/issues/13911). [#14263](https://github.com/ClickHouse/ClickHouse/pull/14263) ([Bharat Nallan](https://github.com/bharatnc)).
+* Fix GRANT ALL statement when executed on a non-global level. [#13987](https://github.com/ClickHouse/ClickHouse/pull/13987) ([Vitaly Baranov](https://github.com/vitlibar)).
+* Fix arrayJoin() capturing in lambda (exception with logical error message was thrown). [#13792](https://github.com/ClickHouse/ClickHouse/pull/13792) ([Azat Khuzhin](https://github.com/azat)).
+
+#### Experimental Feature
+
+* Added `db-generator` tool for random database generation by given SELECT queries. It may faciliate reproducing issues when there is only incomplete bug report from the user. [#14442](https://github.com/ClickHouse/ClickHouse/pull/14442) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) [#10973](https://github.com/ClickHouse/ClickHouse/issues/10973) ([ZeDRoman](https://github.com/ZeDRoman)).
+
+#### Improvement
+
+* Allow using multi-volume storage configuration in storage Distributed. [#14839](https://github.com/ClickHouse/ClickHouse/pull/14839) ([Pavel Kovalenko](https://github.com/Jokser)).
+* Disallow empty time_zone argument in `toStartOf*` type of functions. [#14509](https://github.com/ClickHouse/ClickHouse/pull/14509) ([Bharat Nallan](https://github.com/bharatnc)).
+* MySQL handler returns `OK` for queries like `SET @@var = value`. Such statement is ignored. It is needed because some MySQL drivers send `SET @@` query for setup after handshake https://github.com/ClickHouse/ClickHouse/issues/9336#issuecomment-686222422 . [#14469](https://github.com/ClickHouse/ClickHouse/pull/14469) ([BohuTANG](https://github.com/BohuTANG)).
+* Now TTLs will be applied during merge if they were not previously materialized. [#14438](https://github.com/ClickHouse/ClickHouse/pull/14438) ([alesapin](https://github.com/alesapin)).
+* Now `clickhouse-obfuscator` supports UUID type as proposed in [#13163](https://github.com/ClickHouse/ClickHouse/issues/13163). [#14409](https://github.com/ClickHouse/ClickHouse/pull/14409) ([dimarub2000](https://github.com/dimarub2000)).
+* Added new setting `system_events_show_zero_values` as proposed in [#11384](https://github.com/ClickHouse/ClickHouse/issues/11384). [#14404](https://github.com/ClickHouse/ClickHouse/pull/14404) ([dimarub2000](https://github.com/dimarub2000)).
+* Implicitly convert primary key to not null in `MaterializeMySQL` (Same as `MySQL`). Fixes [#14114](https://github.com/ClickHouse/ClickHouse/issues/14114). [#14397](https://github.com/ClickHouse/ClickHouse/pull/14397) ([Winter Zhang](https://github.com/zhang2014)).
+* Replace wide integers (256 bit) from boost multiprecision with implementation from https://github.com/cerevra/int. 256bit integers are experimental. [#14229](https://github.com/ClickHouse/ClickHouse/pull/14229) ([Artem Zuikov](https://github.com/4ertus2)).
+* Add default compression codec for parts in `system.part_log` with the name `default_compression_codec`. [#14116](https://github.com/ClickHouse/ClickHouse/pull/14116) ([alesapin](https://github.com/alesapin)).
+* Add precision argument for `DateTime` type. It allows to use `DateTime` name instead of `DateTime64`. [#13761](https://github.com/ClickHouse/ClickHouse/pull/13761) ([Winter Zhang](https://github.com/zhang2014)).
+* Added requirepass authorization for `Redis` external dictionary. [#13688](https://github.com/ClickHouse/ClickHouse/pull/13688) ([Ivan Torgashov](https://github.com/it1804)).
+* Improvements in `RabbitMQ` engine: added connection and channels failure handling, proper commits, insert failures handling, better exchanges, queue durability and queue resume opportunity, new queue settings. Fixed tests. [#12761](https://github.com/ClickHouse/ClickHouse/pull/12761) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Support custom codecs in compact parts. [#12183](https://github.com/ClickHouse/ClickHouse/pull/12183) ([Anton Popov](https://github.com/CurtizJ)).
+
+#### Performance Improvement
+
+* Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under optimize_skip_unused_shards and optimize_distributed_group_by_sharding_key). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)).
+* Creating sets for multiple `JOIN` and `IN` in parallel. It may slightly improve performance for queries with several different `IN subquery` expressions. [#14412](https://github.com/ClickHouse/ClickHouse/pull/14412) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
+* Improve Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). [#13939](https://github.com/ClickHouse/ClickHouse/pull/13939) ([fastio](https://github.com/fastio)).
+
+#### Build/Testing/Packaging Improvement
+
+* Lower binary size in debug build by removing debug info from `Functions`. This is needed only for one internal project in Yandex who is using very old linker. [#14549](https://github.com/ClickHouse/ClickHouse/pull/14549) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Prepare for build with clang 11. [#14455](https://github.com/ClickHouse/ClickHouse/pull/14455) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Fix the logic in backport script. In previous versions it was triggered for any labels of 100% red color. It was strange. [#14433](https://github.com/ClickHouse/ClickHouse/pull/14433) ([alexey-milovidov](https://github.com/alexey-milovidov)).
+* Integration tests use default base config. All config changes are explicit with main_configs, user_configs and dictionaries parameters for instance. [#13647](https://github.com/ClickHouse/ClickHouse/pull/13647) ([Ilya Yatsishin](https://github.com/qoega)).
+
+
+
 ## ClickHouse release 20.8
 
 ### ClickHouse release v20.8.2.3-stable, 2020-09-08

CMakeLists.txt

@@ -28,10 +28,11 @@ endforeach()
 
 project(ClickHouse)
 
+# If turned off: e.g. when ENABLE_FOO is ON, but FOO tool was not found, the CMake will continue.
 option(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION
-       "Stop/Fail CMake configuration if some ENABLE_XXX option is defined (either ON or OFF) but is not possible to satisfy"
+   "Stop/Fail CMake configuration if some ENABLE_XXX option is defined (either ON or OFF)
-       ON
+   but is not possible to satisfy" ON)
-)
+
 if(FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION)
     set(RECONFIGURE_MESSAGE_LEVEL FATAL_ERROR)
 else()
@@ -58,7 +59,11 @@ set(CMAKE_DEBUG_POSTFIX "d" CACHE STRING "Generate debug library name with a pos
 # For more info see https://cmake.org/cmake/help/latest/prop_gbl/USE_FOLDERS.html
 set_property(GLOBAL PROPERTY USE_FOLDERS ON)
 
-option(ENABLE_IPO "Enable full link time optimization (it's usually impractical; see also ENABLE_THINLTO)" OFF) # need cmake 3.9+
+# cmake 3.9+ needed.
+# Usually impractical.
+# See also ${ENABLE_THINLTO}
+option(ENABLE_IPO "Full link time optimization")
+
 if(ENABLE_IPO)
     cmake_policy(SET CMP0069 NEW)
     include(CheckIPOSupported)
@@ -93,11 +98,16 @@ message (STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
 
 string (TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
 
-option (USE_STATIC_LIBRARIES "Set to FALSE to use shared libraries" ON)
+option(USE_STATIC_LIBRARIES "Disable to use shared libraries" ON)
-option (MAKE_STATIC_LIBRARIES "Set to FALSE to make shared libraries" ${USE_STATIC_LIBRARIES})
+option(MAKE_STATIC_LIBRARIES "Disable to make shared libraries" ${USE_STATIC_LIBRARIES})
+
 if (NOT MAKE_STATIC_LIBRARIES)
-    option (SPLIT_SHARED_LIBRARIES "DEV ONLY. Keep all internal libs as separate .so for faster linking" OFF)
+    # DEVELOPER ONLY.
-    option (CLICKHOUSE_SPLIT_BINARY "Make several binaries instead one bundled (clickhouse-server, clickhouse-client, ... )" OFF)
+    # Faster linking if turned on.
+    option(SPLIT_SHARED_LIBRARIES "Keep all internal libraries as separate .so files")
+
+    option(CLICKHOUSE_SPLIT_BINARY
+        "Make several binaries (clickhouse-server, clickhouse-client etc.) instead of one bundled")
 endif ()
 
 if (MAKE_STATIC_LIBRARIES AND SPLIT_SHARED_LIBRARIES)
@@ -112,7 +122,8 @@ if (USE_STATIC_LIBRARIES)
     list(REVERSE CMAKE_FIND_LIBRARY_SUFFIXES)
 endif ()
 
-option (ENABLE_FUZZING "Enables fuzzing instrumentation" OFF)
+# Implies ${WITH_COVERAGE}
+option (ENABLE_FUZZING "Fuzzy testing using libfuzzer" OFF)
 
 if (ENABLE_FUZZING)
     message (STATUS "Fuzzing instrumentation enabled")
@@ -144,10 +155,13 @@ if (COMPILER_CLANG)
     endif ()
 endif ()
 
-option (ENABLE_TESTS "Enables tests" ON)
+# If turned `ON`, assumes the user has either the system GTest library or the bundled one.
+option(ENABLE_TESTS "Provide unit_test_dbms target with Google.Test unit tests" ON)
 
 if (OS_LINUX AND NOT UNBUNDLED AND MAKE_STATIC_LIBRARIES AND NOT SPLIT_SHARED_LIBRARIES AND CMAKE_VERSION VERSION_GREATER "3.9.0")
-    option (GLIBC_COMPATIBILITY "Set to TRUE to enable compatibility with older glibc libraries. Only for x86_64, Linux. Implies ENABLE_FASTMEMCPY." ON)
+    # Only for Linux, x86_64.
+    # Implies ${ENABLE_FASTMEMCPY}
+    option(GLIBC_COMPATIBILITY "Enable compatibility with older glibc libraries." ON)
 elseif(GLIBC_COMPATIBILITY)
     message (${RECONFIGURE_MESSAGE_LEVEL} "Glibc compatibility cannot be enabled in current configuration")
 endif ()
@@ -180,7 +194,9 @@ else ()
     set(NO_WHOLE_ARCHIVE --no-whole-archive)
 endif ()
 
-option (ADD_GDB_INDEX_FOR_GOLD "Set to add .gdb-index to resulting binaries for gold linker. NOOP if lld is used." 0)
+# Ignored if `lld` is used
+option(ADD_GDB_INDEX_FOR_GOLD "Add .gdb-index to resulting binaries for gold linker.")
+
 if (NOT CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE")
     if (LINKER_NAME STREQUAL "lld")
         set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--gdb-index")
@@ -201,9 +217,13 @@ if (NOT CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE")
 endif()
 
 cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY) # Not available under freebsd
+
+
 if(NOT AVAILABLE_PHYSICAL_MEMORY OR AVAILABLE_PHYSICAL_MEMORY GREATER 8000)
-    option(COMPILER_PIPE "-pipe compiler option [less /tmp usage, more ram usage]" ON)
+    # Less `/tmp` usage, more RAM usage.
+    option(COMPILER_PIPE "-pipe compiler option" ON)
 endif()
+
 if(COMPILER_PIPE)
     set(COMPILER_FLAGS "${COMPILER_FLAGS} -pipe")
 else()
@@ -214,7 +234,8 @@ if(NOT DISABLE_CPU_OPTIMIZE)
     include(cmake/cpu_features.cmake)
 endif()
 
-option(ARCH_NATIVE "Enable -march=native compiler flag" 0)
+option(ARCH_NATIVE "Add -march=native compiler flag")
+
 if (ARCH_NATIVE)
     set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")
 endif ()
@@ -225,6 +246,7 @@ if (UNBUNDLED AND (COMPILER_GCC OR COMPILER_CLANG))
 else()
     set (_CXX_STANDARD "-std=c++2a")
 endif()
+
 # cmake < 3.12 doesn't support 20. We'll set CMAKE_CXX_FLAGS for now
 # set (CMAKE_CXX_STANDARD 20)
 set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${_CXX_STANDARD}")
@@ -237,7 +259,8 @@ if (COMPILER_GCC OR COMPILER_CLANG)
     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsized-deallocation")
 endif ()
 
-option(WITH_COVERAGE "Build with coverage." 0)
+# Compiler-specific coverage flags e.g. -fcoverage-mapping for gcc
+option(WITH_COVERAGE "Profile the resulting binary/binaries" OFF)
 
 if (WITH_COVERAGE AND COMPILER_CLANG)
     set(COMPILER_FLAGS "${COMPILER_FLAGS} -fprofile-instr-generate -fcoverage-mapping")
@@ -271,10 +294,13 @@ if (COMPILER_CLANG)
     set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fdiagnostics-absolute-paths")
 
     if (NOT ENABLE_TESTS AND NOT SANITIZE)
-        option(ENABLE_THINLTO "Enable Thin LTO. Only applicable for clang. It's also suppressed when building with tests or sanitizers." ON)
+        # https://clang.llvm.org/docs/ThinLTO.html
+        # Applies to clang only.
+        # Disabled when building with tests or sanitizers.
+        option(ENABLE_THINLTO "Clang-specific link time optimization" ON)
     endif()
 
-    # We cannot afford to use LTO when compiling unitests, and it's not enough
+    # We cannot afford to use LTO when compiling unit tests, and it's not enough
     # to only supply -fno-lto at the final linking stage. So we disable it
     # completely.
     if (ENABLE_THINLTO AND NOT ENABLE_TESTS AND NOT SANITIZE)
@@ -287,8 +313,8 @@ if (COMPILER_CLANG)
     endif ()
 
     # Always prefer llvm tools when using clang. For instance, we cannot use GNU ar when llvm LTO is enabled
-
     find_program (LLVM_AR_PATH NAMES "llvm-ar" "llvm-ar-10" "llvm-ar-9" "llvm-ar-8")
+
     if (LLVM_AR_PATH)
         message(STATUS "Using llvm-ar: ${LLVM_AR_PATH}.")
         set (CMAKE_AR ${LLVM_AR_PATH})
@@ -297,30 +323,38 @@ if (COMPILER_CLANG)
     endif ()
 
     find_program (LLVM_RANLIB_PATH NAMES "llvm-ranlib" "llvm-ranlib-10" "llvm-ranlib-9" "llvm-ranlib-8")
+
     if (LLVM_RANLIB_PATH)
         message(STATUS "Using llvm-ranlib: ${LLVM_RANLIB_PATH}.")
         set (CMAKE_RANLIB ${LLVM_RANLIB_PATH})
     else ()
         message(WARNING "Cannot find llvm-ranlib. System ranlib will be used instead. It does not work with ThinLTO.")
     endif ()
+
 elseif (ENABLE_THINLTO)
     message (${RECONFIGURE_MESSAGE_LEVEL} "ThinLTO is only available with CLang")
 endif ()
 
-option (ENABLE_LIBRARIES "Enable all libraries (Global default switch)" ON)
+# Turns on all external libs like s3, kafka, ODBC, ...
+option(ENABLE_LIBRARIES "Enable all external libraries by default" ON)
+
+# We recommend avoiding this mode for production builds because we can't guarantee all needed libraries exist in your
+# system.
+# This mode exists for enthusiastic developers who are searching for trouble.
+# Useful for maintainers of OS packages.
+option (UNBUNDLED "Use system libraries instead of ones in contrib/" OFF)
 
-option (UNBUNDLED "Try find all libraries in system. We recommend to avoid this mode for production builds, because we cannot guarantee exact versions and variants of libraries your system has installed. This mode exists for enthusiastic developers who search for trouble. Also it is useful for maintainers of OS packages." OFF)
 if (UNBUNDLED)
-    set(NOT_UNBUNDLED 0)
+    set(NOT_UNBUNDLED OFF)
 else ()
-    set(NOT_UNBUNDLED 1)
+    set(NOT_UNBUNDLED ON)
 endif ()
 
 if (UNBUNDLED OR NOT (OS_LINUX OR OS_DARWIN))
     # Using system libs can cause a lot of warnings in includes (on macro expansion).
-    option (WERROR "Enable -Werror compiler option" OFF)
+    option(WERROR "Enable -Werror compiler option" OFF)
 else ()
-    option (WERROR "Enable -Werror compiler option" ON)
+    option(WERROR "Enable -Werror compiler option" ON)
 endif ()
 
 if (WERROR)
@@ -362,8 +396,9 @@ else ()
     set (CMAKE_POSITION_INDEPENDENT_CODE ON)
 endif ()
 
-# Using "include-what-you-use" tool.
+# https://github.com/include-what-you-use/include-what-you-use
-option (USE_INCLUDE_WHAT_YOU_USE "Use 'include-what-you-use' tool" OFF)
+option (USE_INCLUDE_WHAT_YOU_USE "Automatically reduce unneeded includes in source code (external tool)" OFF)
+
 if (USE_INCLUDE_WHAT_YOU_USE)
     find_program(IWYU_PATH NAMES include-what-you-use iwyu)
     if (NOT IWYU_PATH)
@@ -375,8 +410,11 @@ if (USE_INCLUDE_WHAT_YOU_USE)
 endif ()
 
 if (ENABLE_TESTS)
-    message (STATUS "Tests are enabled")
+    message (STATUS "Unit tests are enabled")
+else()
+    message(STATUS "Unit tests are disabled")
 endif ()
+
 enable_testing() # Enable for tests without binary
 
 # when installing to /usr - place configs to /etc but for /usr/local place to /usr/local/etc
@@ -386,7 +424,13 @@ else ()
     set (CLICKHOUSE_ETC_DIR "${CMAKE_INSTALL_PREFIX}/etc")
 endif ()
 
-message (STATUS "Building for: ${CMAKE_SYSTEM} ${CMAKE_SYSTEM_PROCESSOR} ${CMAKE_LIBRARY_ARCHITECTURE} ; USE_STATIC_LIBRARIES=${USE_STATIC_LIBRARIES} MAKE_STATIC_LIBRARIES=${MAKE_STATIC_LIBRARIES} SPLIT_SHARED=${SPLIT_SHARED_LIBRARIES} UNBUNDLED=${UNBUNDLED} CCACHE=${CCACHE_FOUND} ${CCACHE_VERSION}")
+message (STATUS
+    "Building for: ${CMAKE_SYSTEM} ${CMAKE_SYSTEM_PROCESSOR} ${CMAKE_LIBRARY_ARCHITECTURE} ;
+    USE_STATIC_LIBRARIES=${USE_STATIC_LIBRARIES}
+    MAKE_STATIC_LIBRARIES=${MAKE_STATIC_LIBRARIES}
+    SPLIT_SHARED=${SPLIT_SHARED_LIBRARIES}
+    UNBUNDLED=${UNBUNDLED}
+    CCACHE=${CCACHE_FOUND} ${CCACHE_VERSION}")
 
 include (GNUInstallDirs)
 include (cmake/contrib_finder.cmake)

README.md

@@ -17,5 +17,5 @@ ClickHouse is an open-source column-oriented database management system that all
 
 ## Upcoming Events
 
-* [eBay migrating from Druid](https://us02web.zoom.us/webinar/register/tZMkfu6rpjItHtaQ1DXcgPWcSOnmM73HLGKL) on September 23, 2020.
 * [ClickHouse for Edge Analytics](https://ones2020.sched.com/event/bWPs) on September 29, 2020.
+* [ClickHouse online meetup (in Russian)](https://clck.ru/R2zB9) on October 1, 2020.

base/common/arithmeticOverflow.h

@@ -31,8 +31,8 @@ namespace common
     template <>
     inline bool addOverflow(__int128 x, __int128 y, __int128 & res)
     {
-        static constexpr __int128 min_int128 = __int128(0x8000000000000000ll) << 64;
+        static constexpr __int128 min_int128 = minInt128();
-        static constexpr __int128 max_int128 = (__int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
+        static constexpr __int128 max_int128 = maxInt128();
         res = x + y;
         return (y > 0 && x > max_int128 - y) || (y < 0 && x < min_int128 - y);
     }
@@ -79,8 +79,8 @@ namespace common
     template <>
     inline bool subOverflow(__int128 x, __int128 y, __int128 & res)
     {
-        static constexpr __int128 min_int128 = __int128(0x8000000000000000ll) << 64;
+        static constexpr __int128 min_int128 = minInt128();
-        static constexpr __int128 max_int128 = (__int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
+        static constexpr __int128 max_int128 = maxInt128();
         res = x - y;
         return (y < 0 && x > max_int128 + y) || (y > 0 && x < min_int128 + y);
     }

base/common/extended_types.h

@@ -13,6 +13,9 @@ using wUInt256 = wide::integer<256, unsigned>;
 static_assert(sizeof(wInt256) == 32);
 static_assert(sizeof(wUInt256) == 32);
 
+static constexpr __int128 minInt128() { return static_cast<unsigned __int128>(1) << 127; }
+static constexpr __int128 maxInt128() { return (static_cast<unsigned __int128>(1) << 127) - 1; }
+
 /// The standard library type traits, such as std::is_arithmetic, with one exception
 /// (std::common_type), are "set in stone". Attempting to specialize them causes undefined behavior.
 /// So instead of using the std type_traits, we use our own version which allows extension.

base/common/itoa.h

@@ -372,7 +372,7 @@ static inline char * writeLeadingMinus(char * pos)
 
 static inline char * writeSIntText(int128_t x, char * pos)
 {
-    static const int128_t min_int128 = int128_t(0x8000000000000000ll) << 64;
+    static constexpr int128_t min_int128 = uint128_t(1) << 127;
 
     if (unlikely(x == min_int128))
     {

base/daemon/BaseDaemon.cpp

@@ -781,7 +781,7 @@ void BaseDaemon::initializeTerminationAndSignalProcessing()
 void BaseDaemon::logRevision() const
 {
     Poco::Logger::root().information("Starting " + std::string{VERSION_FULL}
-        + " with revision " + std::to_string(ClickHouseRevision::get())
+        + " with revision " + std::to_string(ClickHouseRevision::getVersionRevision())
         + ", " + build_id_info
         + ", PID " + std::to_string(getpid()));
 }

benchmark/hardware.sh

@@ -0,0 +1,127 @@
+#!/bin/bash -e
+
+if [[ -n $1 ]]; then
+    SCALE=$1
+else
+    SCALE=100
+fi
+
+TABLE="hits_${SCALE}m_obfuscated"
+DATASET="${TABLE}_v1.tar.xz"
+QUERIES_FILE="queries.sql"
+TRIES=3
+
+AMD64_BIN_URL="https://clickhouse-builds.s3.yandex.net/0/e29c4c3cc47ab2a6c4516486c1b77d57e7d42643/clickhouse_build_check/gcc-10_relwithdebuginfo_none_bundled_unsplitted_disable_False_binary/clickhouse"
+AARCH64_BIN_URL="https://clickhouse-builds.s3.yandex.net/0/e29c4c3cc47ab2a6c4516486c1b77d57e7d42643/clickhouse_special_build_check/clang-10-aarch64_relwithdebuginfo_none_bundled_unsplitted_disable_False_binary/clickhouse"
+
+# Note: on older Ubuntu versions, 'axel' does not support IPv6. If you are using IPv6-only servers on very old Ubuntu, just don't install 'axel'.
+
+FASTER_DOWNLOAD=wget
+if command -v axel >/dev/null; then
+    FASTER_DOWNLOAD=axel
+else
+    echo "It's recommended to install 'axel' for faster downloads."
+fi
+
+if command -v pixz >/dev/null; then
+    TAR_PARAMS='-Ipixz'
+else
+    echo "It's recommended to install 'pixz' for faster decompression of the dataset."
+fi
+
+mkdir -p clickhouse-benchmark-$SCALE
+pushd clickhouse-benchmark-$SCALE
+
+if [[ ! -f clickhouse ]]; then
+    CPU=$(uname -m)
+    if [[ ($CPU == x86_64) || ($CPU == amd64) ]]; then
+        $FASTER_DOWNLOAD "$AMD64_BIN_URL"
+    elif [[ $CPU == aarch64 ]]; then
+        $FASTER_DOWNLOAD "$AARCH64_BIN_URL"
+    else
+        echo "Unsupported CPU type: $CPU"
+        exit 1
+    fi
+fi
+
+chmod a+x clickhouse
+
+if [[ ! -f $QUERIES_FILE ]]; then
+    wget "https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/benchmark/clickhouse/$QUERIES_FILE"
+fi
+
+if [[ ! -d data ]]; then
+    if [[ ! -f $DATASET ]]; then
+        $FASTER_DOWNLOAD "https://clickhouse-datasets.s3.yandex.net/hits/partitions/$DATASET"
+    fi
+
+    tar $TAR_PARAMS --strip-components=1 --directory=. -x -v -f $DATASET
+fi
+
+uptime
+
+echo "Starting clickhouse-server"
+
+./clickhouse server > server.log 2>&1 &
+PID=$!
+
+function finish {
+    kill $PID
+    wait
+}
+trap finish EXIT
+
+echo "Waiting for clickhouse-server to start"
+
+for i in {1..30}; do
+    sleep 1
+    ./clickhouse client --query "SELECT 'The dataset size is: ', count() FROM $TABLE" 2>/dev/null && break || echo '.'
+    if [[ $i == 30 ]]; then exit 1; fi
+done
+
+echo
+echo "Will perform benchmark. Results:"
+echo
+
+cat "$QUERIES_FILE" | sed "s/{table}/${TABLE}/g" | while read query; do
+    sync
+    echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
+
+    echo -n "["
+    for i in $(seq 1 $TRIES); do
+        RES=$(./clickhouse client --max_memory_usage 100000000000 --time --format=Null --query="$query" 2>&1 ||:)
+        [[ "$?" == "0" ]] && echo -n "${RES}" || echo -n "null"
+        [[ "$i" != $TRIES ]] && echo -n ", "
+    done
+    echo "],"
+done
+
+
+echo
+echo "Benchmark complete. System info:"
+echo
+
+echo '----Version, build id-----------'
+./clickhouse local --query "SELECT format('Version: {}, build id: {}', version(), buildId())"
+./clickhouse local --query "SELECT format('The number of threads is: {}', value) FROM system.settings WHERE name = 'max_threads'" --output-format TSVRaw
+./clickhouse local --query "SELECT format('Current time: {}', toString(now(), 'UTC'))"
+echo '----CPU-------------------------'
+cat /proc/cpuinfo | grep -i -F 'model name' | uniq
+lscpu
+echo '----Block Devices---------------'
+lsblk
+echo '----Disk Free and Total--------'
+df -h .
+echo '----Memory Free and Total-------'
+free -h
+echo '----Physical Memory Amount------'
+cat /proc/meminfo | grep MemTotal
+echo '----RAID Info-------------------'
+cat /proc/mdstat
+#echo '----PCI-------------------------'
+#lspci
+#echo '----All Hardware Info-----------'
+#lshw
+echo '--------------------------------'
+
+echo

cmake/analysis.cmake

@@ -1,20 +1,28 @@
-# This file configures static analysis tools that can be integrated to the build process
+# https://clang.llvm.org/extra/clang-tidy/
+option (ENABLE_CLANG_TIDY "Use clang-tidy static analyzer" OFF)
 
-option (ENABLE_CLANG_TIDY "Use 'clang-tidy' static analyzer if present" OFF)
 if (ENABLE_CLANG_TIDY)
     if (${CMAKE_VERSION} VERSION_LESS "3.6.0")
         message(FATAL_ERROR "clang-tidy requires CMake version at least 3.6.")
     endif()
 
     find_program (CLANG_TIDY_PATH NAMES "clang-tidy" "clang-tidy-10" "clang-tidy-9" "clang-tidy-8")
+
     if (CLANG_TIDY_PATH)
-        message(STATUS "Using clang-tidy: ${CLANG_TIDY_PATH}. The checks will be run during build process. See the .clang-tidy file at the root directory to configure the checks.")
+        message(STATUS
-        set (USE_CLANG_TIDY 1)
+            "Using clang-tidy: ${CLANG_TIDY_PATH}.
+            The checks will be run during build process.
+            See the .clang-tidy file at the root directory to configure the checks.")
+
+        set (USE_CLANG_TIDY ON)
+
         # The variable CMAKE_CXX_CLANG_TIDY will be set inside src and base directories with non third-party code.
         # set (CMAKE_CXX_CLANG_TIDY "${CLANG_TIDY_PATH}")
     elseif (FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION)
         message(FATAL_ERROR "clang-tidy is not found")
     else ()
-        message(STATUS "clang-tidy is not found. This is normal - the tool is only used for static code analysis and isn't essential for the build.")
+        message(STATUS
+            "clang-tidy is not found.
+            This is normal - the tool is only used for code static analysis and isn't essential for the build.")
     endif ()
 endif ()

cmake/find/ccache.cmake

@@ -18,7 +18,8 @@ if (NOT CCACHE_FOUND AND NOT DEFINED ENABLE_CCACHE AND NOT COMPILER_MATCHES_CCAC
             "Setting it up will significantly reduce compilation time for 2nd and consequent builds")
 endif()
 
-option(ENABLE_CCACHE "Speedup re-compilations using ccache" ${ENABLE_CCACHE_BY_DEFAULT})
+# https://ccache.dev/
+option(ENABLE_CCACHE "Speedup re-compilations using ccache (external tool)" ${ENABLE_CCACHE_BY_DEFAULT})
 
 if (NOT ENABLE_CCACHE)
     return()

cmake/find/cxx.cmake

@@ -4,13 +4,16 @@ if (NOT USE_LIBCXX)
     if (USE_INTERNAL_LIBCXX_LIBRARY)
         message (${RECONFIGURE_MESSAGE_LEVEL} "Cannot use internal libcxx with USE_LIBCXX=OFF")
     endif()
+
     target_link_libraries(global-libs INTERFACE -l:libstdc++.a -l:libstdc++fs.a) # Always link these libraries as static
     target_link_libraries(global-libs INTERFACE ${EXCEPTION_HANDLING_LIBRARY})
     return()
 endif()
 
 set(USE_INTERNAL_LIBCXX_LIBRARY_DEFAULT ${NOT_UNBUNDLED})
-option (USE_INTERNAL_LIBCXX_LIBRARY "Set to FALSE to use system libcxx and libcxxabi libraries instead of bundled" ${USE_INTERNAL_LIBCXX_LIBRARY_DEFAULT})
+
+option (USE_INTERNAL_LIBCXX_LIBRARY "Disable to use system libcxx and libcxxabi libraries instead of bundled"
+    ${USE_INTERNAL_LIBCXX_LIBRARY_DEFAULT})
 
 if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcxx/CMakeLists.txt")
     if (USE_INTERNAL_LIBCXX_LIBRARY)

cmake/find/gtest.cmake

@@ -1,11 +1,4 @@
-option (ENABLE_GTEST_LIBRARY "Enable gtest library" ${ENABLE_LIBRARIES})
+# included only if ENABLE_TESTS=1
-
-if (NOT ENABLE_GTEST_LIBRARY)
-    if(USE_INTERNAL_GTEST_LIBRARY)
-        message (${RECONFIGURE_MESSAGE_LEVEL} "Cannot use internal Google Test when ENABLE_GTEST_LIBRARY=OFF")
-    endif()
-    return()
-endif()
 
 option (USE_INTERNAL_GTEST_LIBRARY "Set to FALSE to use system Google Test instead of bundled" ${NOT_UNBUNDLED})
 
@@ -15,6 +8,7 @@ if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/googletest/googletest/CMakeList
        message (${RECONFIGURE_MESSAGE_LEVEL} "Can't find internal gtest")
        set (USE_INTERNAL_GTEST_LIBRARY 0)
    endif ()
+
    set (MISSING_INTERNAL_GTEST_LIBRARY 1)
 endif ()
 

cmake/find/sentry.cmake

@@ -1,4 +1,5 @@
 set (SENTRY_LIBRARY "sentry")
+
 set (SENTRY_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/sentry-native/include")
 if (NOT EXISTS "${SENTRY_INCLUDE_DIR}/sentry.h")
     message (WARNING "submodule contrib/sentry-native is missing. to fix try run: \n git submodule update --init --recursive")

cmake/find/snappy.cmake

@@ -1,4 +1,4 @@
-option(USE_SNAPPY "Enable support of snappy library" ${ENABLE_LIBRARIES})
+option(USE_SNAPPY "Enable snappy library" ${ENABLE_LIBRARIES})
 
 if(NOT USE_SNAPPY)
     if (USE_INTERNAL_SNAPPY_LIBRARY)

cmake/fuzzer.cmake

@@ -1,11 +1,12 @@
-option (FUZZER "Enable fuzzer: libfuzzer")
+# see ./CMakeLists.txt for variable declaration
-
 if (FUZZER)
     if (FUZZER STREQUAL "libfuzzer")
         # NOTE: Eldar Zaitov decided to name it "libfuzzer" instead of "fuzzer" to keep in mind another possible fuzzer backends.
-        # NOTE: no-link means that all the targets are built with instrumentation for fuzzer, but only some of them (tests) have entry point for fuzzer and it's not checked.
+        # NOTE: no-link means that all the targets are built with instrumentation for fuzzer, but only some of them
+        # (tests) have entry point for fuzzer and it's not checked.
         set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SAN_FLAGS} -fsanitize=fuzzer-no-link")
         set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SAN_FLAGS} -fsanitize=fuzzer-no-link")
+
         if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
             set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=fuzzer-no-link")
         endif()
@@ -14,7 +15,6 @@ if (FUZZER)
         if (NOT LIB_FUZZING_ENGINE)
             set (LIB_FUZZING_ENGINE "-fsanitize=fuzzer")
         endif ()
-
     else ()
         message (FATAL_ERROR "Unknown fuzzer type: ${FUZZER}")
     endif ()

cmake/limit_jobs.cmake

@@ -6,26 +6,35 @@
 cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY) # Not available under freebsd
 cmake_host_system_information(RESULT NUMBER_OF_LOGICAL_CORES QUERY NUMBER_OF_LOGICAL_CORES)
 
-option(PARALLEL_COMPILE_JOBS "Define the maximum number of concurrent compilation jobs" "")
+# 1 if not set
+option(PARALLEL_COMPILE_JOBS "Maximum number of concurrent compilation jobs" "")
+
+# 1 if not set
+option(PARALLEL_LINK_JOBS "Maximum number of concurrent link jobs" "")
+
 if (NOT PARALLEL_COMPILE_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_COMPILER_MEMORY)
     math(EXPR PARALLEL_COMPILE_JOBS ${AVAILABLE_PHYSICAL_MEMORY}/${MAX_COMPILER_MEMORY})
+
     if (NOT PARALLEL_COMPILE_JOBS)
         set (PARALLEL_COMPILE_JOBS 1)
     endif ()
 endif ()
+
 if (PARALLEL_COMPILE_JOBS AND (NOT NUMBER_OF_LOGICAL_CORES OR PARALLEL_COMPILE_JOBS LESS NUMBER_OF_LOGICAL_CORES))
     set(CMAKE_JOB_POOL_COMPILE compile_job_pool${CMAKE_CURRENT_SOURCE_DIR})
     string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_COMPILE ${CMAKE_JOB_POOL_COMPILE})
     set_property(GLOBAL APPEND PROPERTY JOB_POOLS ${CMAKE_JOB_POOL_COMPILE}=${PARALLEL_COMPILE_JOBS})
 endif ()
 
-option(PARALLEL_LINK_JOBS "Define the maximum number of concurrent link jobs" "")
+
 if (NOT PARALLEL_LINK_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_LINKER_MEMORY)
     math(EXPR PARALLEL_LINK_JOBS ${AVAILABLE_PHYSICAL_MEMORY}/${MAX_LINKER_MEMORY})
+
     if (NOT PARALLEL_LINK_JOBS)
         set (PARALLEL_LINK_JOBS 1)
     endif ()
 endif ()
+
 if (PARALLEL_LINK_JOBS AND (NOT NUMBER_OF_LOGICAL_CORES OR PARALLEL_COMPILE_JOBS LESS NUMBER_OF_LOGICAL_CORES))
     set(CMAKE_JOB_POOL_LINK link_job_pool${CMAKE_CURRENT_SOURCE_DIR})
     string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_LINK ${CMAKE_JOB_POOL_LINK})
@@ -33,5 +42,7 @@ if (PARALLEL_LINK_JOBS AND (NOT NUMBER_OF_LOGICAL_CORES OR PARALLEL_COMPILE_JOBS
 endif ()
 
 if (PARALLEL_COMPILE_JOBS OR PARALLEL_LINK_JOBS)
-    message(STATUS "${CMAKE_CURRENT_SOURCE_DIR}: Have ${AVAILABLE_PHYSICAL_MEMORY} megabytes of memory. Limiting concurrent linkers jobs to ${PARALLEL_LINK_JOBS} and compiler jobs to ${PARALLEL_COMPILE_JOBS}")
+    message(STATUS
+        "${CMAKE_CURRENT_SOURCE_DIR}: Have ${AVAILABLE_PHYSICAL_MEMORY} megabytes of memory.
+        Limiting concurrent linkers jobs to ${PARALLEL_LINK_JOBS} and compiler jobs to ${PARALLEL_COMPILE_JOBS}")
 endif ()

cmake/sanitize.cmake

@@ -1,4 +1,5 @@
-option (SANITIZE "Enable sanitizer: address, memory, thread, undefined" "")
+# Possible values: `address` (ASan), `memory` (MSan), `thread` (TSan), `undefined` (UBSan), and "" (no sanitizing)
+option (SANITIZE "Enable one of the code sanitizers" "")
 
 set (SAN_FLAGS "${SAN_FLAGS} -g -fno-omit-frame-pointer -DSANITIZER")
 

cmake/tools.cmake

@@ -40,7 +40,9 @@ endif ()
 STRING(REGEX MATCHALL "[0-9]+" COMPILER_VERSION_LIST ${CMAKE_CXX_COMPILER_VERSION})
 LIST(GET COMPILER_VERSION_LIST 0 COMPILER_VERSION_MAJOR)
 
+# Example values: `lld-10`, `gold`.
 option (LINKER_NAME "Linker name or full path")
+
 if (COMPILER_GCC AND NOT LINKER_NAME)
     find_program (LLD_PATH NAMES "ld.lld")
     find_program (GOLD_PATH NAMES "ld.gold")

cmake/warnings.cmake

@@ -17,8 +17,9 @@ if (USE_DEBUG_HELPERS)
 endif ()
 
 # Add some warnings that are not available even with -Wall -Wextra -Wpedantic.
-
+# Intended for exploration of new compiler warnings that may be found useful.
-option (WEVERYTHING "Enables -Weverything option with some exceptions. This is intended for exploration of new compiler warnings that may be found to be useful. Only makes sense for clang." ON)
+# Applies to clang only
+option (WEVERYTHING "Enable -Weverything option with some exceptions." ON)
 
 # Control maximum size of stack frames. It can be important if the code is run in fibers with small stack size.
 # Only in release build because debug has too large stack frames.

contrib/protobuf

@@ -1 +1 @@
-Subproject commit d6a10dd3db55d8f7f9e464db9151874cde1f79ec
+Subproject commit 445d1ae73a450b1e94622e7040989aa2048402e3

contrib/protobuf-cmake/CMakeLists.txt

@@ -11,3 +11,7 @@ else ()
 endif ()
 
 add_subdirectory("${protobuf_SOURCE_DIR}/cmake" "${protobuf_BINARY_DIR}")
+
+# We don't want to stop compilation on warnings in protobuf's headers.
+# The following line overrides the value assigned by the command target_include_directories() in libprotobuf.cmake 
+set_property(TARGET libprotobuf PROPERTY INTERFACE_SYSTEM_INCLUDE_DIRECTORIES ${protobuf_SOURCE_DIR}/src)

debian/rules

@@ -36,8 +36,8 @@ endif
 
 CMAKE_FLAGS += -DENABLE_UTILS=0
 
-DEB_CC ?= $(shell which gcc-9 gcc-8 gcc | head -n1)
+DEB_CC ?= $(shell which gcc-10 gcc-9 gcc | head -n1)
-DEB_CXX ?= $(shell which g++-9 g++-8 g++ | head -n1)
+DEB_CXX ?= $(shell which g++-10 g++-9 g++ | head -n1)
 
 ifdef DEB_CXX
     DEB_BUILD_GNU_TYPE := $(shell dpkg-architecture -qDEB_BUILD_GNU_TYPE)

docker/server/entrypoint.sh

@@ -89,7 +89,8 @@ EOT
 fi
 
 if [ -n "$(ls /docker-entrypoint-initdb.d/)" ] || [ -n "$CLICKHOUSE_DB" ]; then
-    $gosu /usr/bin/clickhouse-server --config-file=$CLICKHOUSE_CONFIG &
+    # Listen only on localhost until the initialization is done
+    $gosu /usr/bin/clickhouse-server --config-file=$CLICKHOUSE_CONFIG -- --listen_host=127.0.0.1 &
     pid="$!"
 
     # check if clickhouse is ready to accept connections

docker/test/fasttest/run.sh

@@ -83,7 +83,7 @@ SUBMODULES_TO_UPDATE=(contrib/boost contrib/zlib-ng contrib/libxml2 contrib/poco
 
 git submodule update --init --recursive "${SUBMODULES_TO_UPDATE[@]}" | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/submodule_log.txt
 
-export CMAKE_LIBS_CONFIG="-DENABLE_LIBRARIES=0 -DENABLE_TESTS=0 -DENABLE_UTILS=0 -DENABLE_EMBEDDED_COMPILER=0 -DENABLE_THINLTO=0 -DUSE_UNWIND=1"
+CMAKE_LIBS_CONFIG=(-DENABLE_LIBRARIES=0 -DENABLE_TESTS=0 -DENABLE_UTILS=0 -DENABLE_EMBEDDED_COMPILER=0 -DENABLE_THINLTO=0 -DUSE_UNWIND=1)
 
 export CCACHE_DIR=/ccache
 export CCACHE_BASEDIR=/ClickHouse
@@ -96,8 +96,8 @@ ccache --zero-stats ||:
 
 mkdir build
 cd build
-cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_CXX_COMPILER=clang++-10 -DCMAKE_C_COMPILER=clang-10 "$CMAKE_LIBS_CONFIG" "${FASTTEST_CMAKE_FLAGS[@]}" | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/cmake_log.txt
+cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_CXX_COMPILER=clang++-10 -DCMAKE_C_COMPILER=clang-10 "${CMAKE_LIBS_CONFIG[@]}" "${FASTTEST_CMAKE_FLAGS[@]}" | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/cmake_log.txt
-ninja clickhouse-bundle | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/build_log.txt
+time ninja clickhouse-bundle | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/build_log.txt
 ninja install | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/install_log.txt
 
 
@@ -111,35 +111,10 @@ ln -s /test_output /var/log/clickhouse-server
 cp "$CLICKHOUSE_DIR/programs/server/config.xml" /etc/clickhouse-server/
 cp "$CLICKHOUSE_DIR/programs/server/users.xml" /etc/clickhouse-server/
 
-mkdir -p /etc/clickhouse-server/dict_examples
+# install tests config
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/
+$CLICKHOUSE_DIR/tests/config/install.sh
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/
+# doesn't support SSL
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/
+rm -f /etc/clickhouse-server/config.d/secure_ports.xml
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/custom_settings_prefixes.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/access_management.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/executable_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/disks.xml /etc/clickhouse-server/config.d/
-#ln -s /usr/share/clickhouse-test/config/secure_ports.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/clusters.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/graphite.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/server.key /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/server.crt /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/dhparam.pem /etc/clickhouse-server/
-ln -sf /usr/share/clickhouse-test/config/client_config.xml /etc/clickhouse-client/config.xml
-
-# Keep original query_masking_rules.xml
-ln -s --backup=simple --suffix=_original.xml /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/
 
 # Kill the server in case we are running locally and not in docker
 kill_clickhouse
@@ -216,7 +191,7 @@ TESTS_TO_SKIP=(
     01460_DistributedFilesToInsert
 )
 
-clickhouse-test -j 4 --no-long --testname --shard --zookeeper --skip "${TESTS_TO_SKIP[@]}" 2>&1 | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/test_log.txt
+time clickhouse-test -j 8 --no-long --testname --shard --zookeeper --skip "${TESTS_TO_SKIP[@]}" 2>&1 | ts '%Y-%m-%d %H:%M:%S' | tee /test_output/test_log.txt
 
 
 # substr is to remove semicolon after test name
@@ -234,7 +209,7 @@ then
     kill_clickhouse
 
     # Clean the data so that there is no interference from the previous test run.
-    rm -rvf /var/lib/clickhouse ||:
+    rm -rf /var/lib/clickhouse ||:
     mkdir /var/lib/clickhouse
 
     clickhouse-server --config /etc/clickhouse-server/config.xml --daemon

docker/test/fuzzer/run-fuzzer.sh

@@ -48,7 +48,7 @@ function configure
     cp -av "$repo_dir"/programs/server/config* db
     cp -av "$repo_dir"/programs/server/user* db
     # TODO figure out which ones are needed
-    cp -av "$repo_dir"/tests/config/listen.xml db/config.d
+    cp -av "$repo_dir"/tests/config/config.d/listen.xml db/config.d
     cp -av "$script_dir"/query-fuzzer-tweaks-users.xml db/users.d
 }
 

docker/test/performance-comparison/compare.sh

@@ -114,8 +114,6 @@ function run_tests
     # Just check that the script runs at all
     "$script_dir/perf.py" --help > /dev/null
 
-    changed_test_files=""
-
     # Find the directory with test files.
     if [ -v CHPC_TEST_PATH ]
     then
@@ -130,14 +128,6 @@ function run_tests
     else
         # For PRs, use newer test files so we can test these changes.
         test_prefix=right/performance
-
-        # If only the perf tests were changed in the PR, we will run only these
-        # tests. The list of changed tests in changed-test.txt is prepared in
-        # entrypoint.sh from git diffs, because it has the cloned repo.  Used
-        # to use rsync for that but it was really ugly and not always correct
-        # (e.g. when the reference SHA is really old and has some other
-        # differences to the tested SHA, besides the one introduced by the PR).
-        changed_test_files=$(sed "s/tests\/performance/${test_prefix//\//\\/}/" changed-tests.txt)
     fi
 
     # Determine which tests to run.
@@ -146,25 +136,32 @@ function run_tests
         # Run only explicitly specified tests, if any.
         # shellcheck disable=SC2010
         test_files=$(ls "$test_prefix" | grep "$CHPC_TEST_GREP" | xargs -I{} -n1 readlink -f "$test_prefix/{}")
-    elif [ "$changed_test_files" != "" ]
+    elif [ "$PR_TO_TEST" -ne 0 ] \
+        && [ "$(wc -l < changed-test-definitions.txt)" -gt 0 ] \
+        && [ "$(wc -l < changed-test-scripts.txt)" -eq 0 ] \
+        && [ "$(wc -l < other-changed-files.txt)" -eq 0 ]
     then
-        # Use test files that changed in the PR.
+        # If only the perf tests were changed in the PR, we will run only these
-        test_files="$changed_test_files"
+        # tests. The lists of changed files are prepared in entrypoint.sh because
+        # it has the repository.
+        test_files=$(sed "s/tests\/performance/${test_prefix//\//\\/}/" changed-test-definitions.txt)
     else
         # The default -- run all tests found in the test dir.
         test_files=$(ls "$test_prefix"/*.xml)
     fi
 
-    # For PRs, test only a subset of queries, and run them less times.
+    # For PRs w/o changes in test definitons and scripts, test only a subset of
-    # If the corresponding environment variables are already set, keep
+    # queries, and run them less times. If the corresponding environment variables
-    # those values.
+    # are already set, keep those values.
-    if [ "$PR_TO_TEST" == "0" ]
+    if [ "$PR_TO_TEST" -ne 0 ] \
+        && [ "$(wc -l < changed-test-definitions.txt)" -eq 0 ] \
+        && [ "$(wc -l < changed-test-scripts.txt)" -eq 0 ]
     then
-        CHPC_RUNS=${CHPC_RUNS:-13}
-        CHPC_MAX_QUERIES=${CHPC_MAX_QUERIES:-0}
-    else
         CHPC_RUNS=${CHPC_RUNS:-7}
         CHPC_MAX_QUERIES=${CHPC_MAX_QUERIES:-20}
+    else
+        CHPC_RUNS=${CHPC_RUNS:-13}
+        CHPC_MAX_QUERIES=${CHPC_MAX_QUERIES:-0}
     fi
     export CHPC_RUNS
     export CHPC_MAX_QUERIES
@@ -629,17 +626,53 @@ create table test_time engine Memory as
     from total_client_time_per_query full join queries using (test, query_index)
     group by test;
 
+create view query_runs as select * from file('analyze/query-runs.tsv', TSV,
+    'test text, query_index int, query_id text, version UInt8, time float');
+
+--
+-- Guess the number of query runs used for this test. The number is required to
+-- calculate and check the average query run time in the report.
+-- We have to be careful, because we will encounter:
+--  1) partial queries which run only on one server
+--  2) short queries which run for a much higher number of times
+--  3) some errors that make query run for a different number of times on a
+--     particular server.
+--
+create view test_runs as
+    select test,
+        -- Default to 7 runs if there are only 'short' queries in the test, and
+        -- we can't determine the number of runs.
+        if((ceil(medianOrDefaultIf(t.runs, not short), 0) as r) != 0, r, 7) runs
+    from (
+        select
+            -- The query id is the same for both servers, so no need to divide here.
+            uniqExact(query_id) runs,
+            (test, query_index) in
+                (select * from file('analyze/marked-short-queries.tsv', TSV,
+                    'test text, query_index int'))
+            as short,
+            test, query_index
+        from query_runs
+        group by test, query_index
+        ) t
+    group by test
+    ;
+
 create table test_times_report engine File(TSV, 'report/test-times.tsv') as
     select wall_clock_time_per_test.test, real,
         toDecimal64(total_client_time, 3),
         queries,
         toDecimal64(query_max, 3),
         toDecimal64(real / queries, 3) avg_real_per_query,
-        toDecimal64(query_min, 3)
+        toDecimal64(query_min, 3),
+        runs
     from test_time
-    -- wall clock times are also measured for skipped tests, so don't
+        -- wall clock times are also measured for skipped tests, so don't
-    -- do full join
+        -- do full join
-    left join wall_clock_time_per_test using test
+        left join wall_clock_time_per_test
+            on wall_clock_time_per_test.test = test_time.test
+        full join test_runs
+            on test_runs.test = test_time.test
     order by avg_real_per_query desc;
 
 -- report for all queries page, only main metric
@@ -693,8 +726,8 @@ create view shortness
 create table inconsistent_short_marking_report
     engine File(TSV, 'report/unexpected-query-duration.tsv')
     as select
-        multiIf(marked_short and time > 0.1, '"short" queries must run faster than 0.02 s',
+        multiIf(marked_short and time > 0.1, '\"short\" queries must run faster than 0.02 s',
-                not marked_short and time < 0.02, '"normal" queries must run longer than 0.1 s',
+                not marked_short and time < 0.02, '\"normal\" queries must run longer than 0.1 s',
                 '') problem,
         marked_short, time,
         test, query_index, query_display_name
@@ -1032,7 +1065,7 @@ case "$stage" in
     # to collect the logs. Prefer not to restart, because addresses might change
     # and we won't be able to process trace_log data. Start in a subshell, so that
     # it doesn't interfere with the watchdog through `wait`.
-    ( get_profiles || restart && get_profiles ) ||:
+    ( get_profiles || { restart && get_profiles ; } ) ||:
 
     # Kill the whole process group, because somehow when the subshell is killed,
     # the sleep inside remains alive and orphaned.

docker/test/performance-comparison/entrypoint.sh

@@ -97,13 +97,10 @@ then
     # tests for use by compare.sh. Compare to merge base, because master might be
     # far in the future and have unrelated test changes.
     base=$(git -C right/ch merge-base pr origin/master)
-    git -C right/ch diff --name-only "$base" pr | tee changed-tests.txt
+    git -C right/ch diff --name-only "$base" pr -- . | tee all-changed-files.txt
-    if grep -vq '^tests/performance' changed-tests.txt
+    git -C right/ch diff --name-only "$base" pr -- tests/performance | tee changed-test-definitions.txt
-    then
+    git -C right/ch diff --name-only "$base" pr -- docker/test/performance-comparison | tee changed-test-scripts.txt
-        # Have some other changes besides the tests, so truncate the test list,
+    git -C right/ch diff --name-only "$base" pr -- :!tests/performance :!docker/test/performance-comparison | tee other-changed-files.txt
-        # meaning, run all tests.
-        : > changed-tests.txt
-    fi
 fi
 
 # Set python output encoding so that we can print queries with Russian letters.

docker/test/performance-comparison/perf.py

@@ -15,6 +15,7 @@ import sys
 import time
 import traceback
 import xml.etree.ElementTree as et
+from threading import Thread
 from scipy import stats
 
 def tsv_escape(s):
@@ -23,10 +24,11 @@ def tsv_escape(s):
 parser = argparse.ArgumentParser(description='Run performance test.')
 # Explicitly decode files as UTF-8 because sometimes we have Russian characters in queries, and LANG=C is set.
 parser.add_argument('file', metavar='FILE', type=argparse.FileType('r', encoding='utf-8'), nargs=1, help='test description file')
-parser.add_argument('--host', nargs='*', default=['localhost'], help="Server hostname(s). Corresponds to '--port' options.")
+parser.add_argument('--host', nargs='*', default=['localhost'], help="Space-separated list of server hostname(s). Corresponds to '--port' options.")
-parser.add_argument('--port', nargs='*', default=[9000], help="Server port(s). Corresponds to '--host' options.")
+parser.add_argument('--port', nargs='*', default=[9000], help="Space-separated list of server port(s). Corresponds to '--host' options.")
 parser.add_argument('--runs', type=int, default=1, help='Number of query runs per server.')
 parser.add_argument('--max-queries', type=int, default=None, help='Test no more than this number of queries, chosen at random.')
+parser.add_argument('--queries-to-run', nargs='*', type=int, default=None, help='Space-separated list of indexes of queries to test.')
 parser.add_argument('--long', action='store_true', help='Do not skip the tests tagged as long.')
 parser.add_argument('--print-queries', action='store_true', help='Print test queries and exit.')
 parser.add_argument('--print-settings', action='store_true', help='Print test settings and exit.')
@@ -157,8 +159,11 @@ for t in tables:
             print(f'skipped\t{tsv_escape(skipped_message)}')
             sys.exit(0)
 
-# Run create queries
+# Run create and fill queries. We will run them simultaneously for both servers,
-create_query_templates = [q.text for q in root.findall('create_query')]
+# to save time.
+# The weird search is to keep the relative order of elements, which matters, and
+# etree doesn't support the appropriate xpath query.
+create_query_templates = [q.text for q in root.findall('./*') if q.tag in ('create_query', 'fill_query')]
 create_queries = substitute_parameters(create_query_templates)
 
 # Disallow temporary tables, because the clickhouse_driver reconnects on errors,
@@ -170,23 +175,34 @@ for q in create_queries:
             file = sys.stderr)
         sys.exit(1)
 
-for conn_index, c in enumerate(all_connections):
+def do_create(connection, index, queries):
-    for q in create_queries:
+    for q in queries:
-        c.execute(q)
+        connection.execute(q)
-        print(f'create\t{conn_index}\t{c.last_query.elapsed}\t{tsv_escape(q)}')
+        print(f'create\t{index}\t{connection.last_query.elapsed}\t{tsv_escape(q)}')
 
-# Run fill queries
+threads = [Thread(target = do_create, args = (connection, index, create_queries))
-fill_query_templates = [q.text for q in root.findall('fill_query')]
+                for index, connection in enumerate(all_connections)]
-fill_queries = substitute_parameters(fill_query_templates)
-for conn_index, c in enumerate(all_connections):
-    for q in fill_queries:
-        c.execute(q)
-        print(f'fill\t{conn_index}\t{c.last_query.elapsed}\t{tsv_escape(q)}')
 
-# Run the queries in randomized order, but preserve their indexes as specified
+for t in threads:
-# in the test XML. To avoid using too much time, limit the number of queries
+    t.start()
-# we run per test.
+
-queries_to_run = random.sample(range(0, len(test_queries)), min(len(test_queries), args.max_queries or len(test_queries)))
+for t in threads:
+    t.join()
+
+queries_to_run = range(0, len(test_queries))
+
+if args.max_queries:
+    # If specified, test a limited number of queries chosen at random.
+    queries_to_run = random.sample(range(0, len(test_queries)), min(len(test_queries), args.max_queries))
+
+if args.queries_to_run:
+    # Run the specified queries, with some sanity check.
+    for i in args.queries_to_run:
+        if i < 0 or i >= len(test_queries):
+            print(f'There is no query no. "{i}" in this test, only [{0}-{len(test_queries) - 1}] are present')
+            exit(1)
+
+    queries_to_run = args.queries_to_run
 
 # Run test queries.
 for query_index in queries_to_run:

docker/test/performance-comparison/report.py

@@ -187,8 +187,10 @@ def td(value, cell_attributes = ''):
         cell_attributes = cell_attributes,
         value = value)
 
-def th(x):
+def th(value, cell_attributes = ''):
-    return '<th>' + str(x) + '</th>'
+    return '<th {cell_attributes}>{value}</th>'.format(
+        cell_attributes = cell_attributes,
+        value = value)
 
 def tableRow(cell_values, cell_attributes = [], anchor=None):
     return tr(
@@ -199,8 +201,13 @@ def tableRow(cell_values, cell_attributes = [], anchor=None):
             if a is not None and v is not None]),
         anchor)
 
-def tableHeader(r):
+def tableHeader(cell_values, cell_attributes = []):
-    return tr(''.join([th(f) for f in r]))
+    return tr(
+        ''.join([th(v, a)
+            for v, a in itertools.zip_longest(
+                cell_values, cell_attributes,
+                fillvalue = '')
+            if a is not None and v is not None]))
 
 def tableStart(title):
     cls = '-'.join(title.lower().split(' ')[:3]);
@@ -377,16 +384,16 @@ if args.report == 'main':
             'Ratio of speedup&nbsp;(-) or slowdown&nbsp;(+)',                 # 2
             'Relative difference (new&nbsp;&minus;&nbsp;old) / old',   # 3
             'p&nbsp;<&nbsp;0.01 threshold',                   # 4
-            # Failed                                           # 5
+            '', # Failed                                           # 5
             'Test',                                            # 6
             '#',                                               # 7
             'Query',                                           # 8
             ]
-
-        text += tableHeader(columns)
-
         attrs = ['' for c in columns]
         attrs[5] = None
+
+        text += tableHeader(columns, attrs)
+
         for row in rows:
             anchor = f'{currentTableAnchor()}.{row[6]}.{row[7]}'
             if int(row[5]):
@@ -421,17 +428,17 @@ if args.report == 'main':
             'New,&nbsp;s', #1
             'Relative difference (new&nbsp;-&nbsp;old)/old', #2
             'p&nbsp;&lt;&nbsp;0.01 threshold', #3
-            # Failed #4
+            '', # Failed #4
             'Test', #5
             '#',    #6
             'Query' #7
         ]
-
-        text = tableStart('Unstable Queries')
-        text += tableHeader(columns)
-
         attrs = ['' for c in columns]
         attrs[4] = None
+
+        text = tableStart('Unstable Queries')
+        text += tableHeader(columns, attrs)
+
         for r in unstable_rows:
             anchor = f'{currentTableAnchor()}.{r[5]}.{r[6]}'
             if int(r[4]):
@@ -461,24 +468,25 @@ if args.report == 'main':
             return
 
         columns = [
-            'Test',                                          #0
+            'Test',                                               #0
             'Wall clock time,&nbsp;s',                            #1
             'Total client time,&nbsp;s',                          #2
-            'Total queries',                                 #3
+            'Total queries',                                      #3
             'Longest query<br>(sum for all runs),&nbsp;s',        #4
             'Avg wall clock time<br>(sum for all runs),&nbsp;s',  #5
             'Shortest query<br>(sum for all runs),&nbsp;s',       #6
+            '', # Runs                                            #7
             ]
+        attrs = ['' for c in columns]
+        attrs[7] = None
 
         text = tableStart('Test Times')
-        text += tableHeader(columns)
+        text += tableHeader(columns, attrs)
 
-        nominal_runs = 7  # FIXME pass this as an argument
+        allowed_average_run_time = 1.6 # 30 seconds per test at 7 runs
-        total_runs = (nominal_runs + 1) * 2  # one prewarm run, two servers
-        allowed_average_run_time = allowed_single_run_time + 60 / total_runs; # some allowance for fill/create queries
-        attrs = ['' for c in columns]
         for r in rows:
             anchor = f'{currentTableAnchor()}.{r[0]}'
+            total_runs = (int(r[7]) + 1) * 2  # one prewarm run, two servers
             if float(r[5]) > allowed_average_run_time * total_runs:
                 # FIXME should be 15s max -- investigate parallel_insert
                 slow_average_tests += 1
@@ -580,8 +588,8 @@ elif args.report == 'all-queries':
             return
 
         columns = [
-            # Changed #0
+            '', # Changed #0
-            # Unstable #1
+            '', # Unstable #1
             'Old,&nbsp;s', #2
             'New,&nbsp;s', #3
             'Ratio of speedup&nbsp;(-) or slowdown&nbsp;(+)',                 #4
@@ -591,13 +599,13 @@ elif args.report == 'all-queries':
             '#',                                      #8
             'Query',                                  #9
             ]
-
-        text = tableStart('All Query Times')
-        text += tableHeader(columns)
-
         attrs = ['' for c in columns]
         attrs[0] = None
         attrs[1] = None
+
+        text = tableStart('All Query Times')
+        text += tableHeader(columns, attrs)
+
         for r in rows:
             anchor = f'{currentTableAnchor()}.{r[7]}.{r[8]}'
             if int(r[1]):

docker/test/stateful/run.sh

@@ -8,26 +8,8 @@ dpkg -i package_folder/clickhouse-server_*.deb
 dpkg -i package_folder/clickhouse-client_*.deb
 dpkg -i package_folder/clickhouse-test_*.deb
 
-mkdir -p /etc/clickhouse-server/dict_examples
+# install test configs
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/
+/usr/share/clickhouse-test/config/install.sh
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-
-if [[ -n "$USE_DATABASE_ATOMIC" ]] && [[ "$USE_DATABASE_ATOMIC" -eq 1 ]]; then
-    ln -s /usr/share/clickhouse-test/config/database_atomic_configd.xml /etc/clickhouse-server/config.d/
-    ln -s /usr/share/clickhouse-test/config/database_atomic_usersd.xml /etc/clickhouse-server/users.d/
-fi
 
 function start()
 {

docker/test/stateful_with_coverage/run.sh

@@ -48,28 +48,8 @@ mkdir -p /var/lib/clickhouse
 mkdir -p /var/log/clickhouse-server
 chmod 777 -R /var/log/clickhouse-server/
 
-# Temorary way to keep CI green while moving dictionaries to separate directory
+# install test configs
-mkdir -p /etc/clickhouse-server/dict_examples
+/usr/share/clickhouse-test/config/install.sh
-chmod 777 -R /etc/clickhouse-server/dict_examples
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/; \
-    ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/; \
-    ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/;
-
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-
-# Retain any pre-existing config and allow ClickHouse to load those if required
-ln -s --backup=simple --suffix=_original.xml \
-    /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/
 
 function start()
 {

docker/test/stateless/Dockerfile

@@ -21,9 +21,7 @@ RUN apt-get update -y \
             telnet \
             tree \
             unixodbc \
-            wget \
+            wget
-            zookeeper \
-            zookeeperd
 
 RUN mkdir -p /tmp/clickhouse-odbc-tmp \
    && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \

docker/test/stateless/run.sh

@@ -8,48 +8,9 @@ dpkg -i package_folder/clickhouse-server_*.deb
 dpkg -i package_folder/clickhouse-client_*.deb
 dpkg -i package_folder/clickhouse-test_*.deb
 
-mkdir -p /etc/clickhouse-server/dict_examples
+# install test configs
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/
+/usr/share/clickhouse-test/config/install.sh
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/custom_settings_prefixes.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/access_management.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/executable_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/disks.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/secure_ports.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/clusters.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/graphite.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/server.key /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/server.crt /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/dhparam.pem /etc/clickhouse-server/
 
-# Retain any pre-existing config and allow ClickHouse to load it if required
-ln -s --backup=simple --suffix=_original.xml \
-    /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/
-
-if [[ -n "$USE_POLYMORPHIC_PARTS" ]] && [[ "$USE_POLYMORPHIC_PARTS" -eq 1 ]]; then
-    ln -s /usr/share/clickhouse-test/config/polymorphic_parts.xml /etc/clickhouse-server/config.d/
-fi
-if [[ -n "$USE_DATABASE_ATOMIC" ]] && [[ "$USE_DATABASE_ATOMIC" -eq 1 ]]; then
-    ln -s /usr/share/clickhouse-test/config/database_atomic_configd.xml /etc/clickhouse-server/config.d/
-    ln -s /usr/share/clickhouse-test/config/database_atomic_usersd.xml /etc/clickhouse-server/users.d/
-fi
-
-ln -sf /usr/share/clickhouse-test/config/client_config.xml /etc/clickhouse-client/config.xml
-
-service zookeeper start
-sleep 5
 service clickhouse-server start && sleep 5
 
 if cat /usr/bin/clickhouse-test | grep -q -- "--use-skip-list"; then

docker/test/stateless_unbundled/Dockerfile

@@ -66,9 +66,7 @@ RUN apt-get --allow-unauthenticated update -y \
             unixodbc \
             unixodbc-dev \
             wget \
-            zlib1g-dev \
+            zlib1g-dev
-            zookeeper \
-            zookeeperd
 
 RUN mkdir -p /tmp/clickhouse-odbc-tmp \
    && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \

docker/test/stateless_unbundled/run.sh

@@ -8,48 +8,9 @@ dpkg -i package_folder/clickhouse-server_*.deb
 dpkg -i package_folder/clickhouse-client_*.deb
 dpkg -i package_folder/clickhouse-test_*.deb
 
-mkdir -p /etc/clickhouse-server/dict_examples
+# install test configs
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/
+/usr/share/clickhouse-test/config/install.sh
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/custom_settings_prefixes.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/access_management.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/executable_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/disks.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/secure_ports.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/clusters.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/graphite.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/server.key /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/server.crt /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/dhparam.pem /etc/clickhouse-server/
 
-# Retain any pre-existing config and allow ClickHouse to load it if required
-ln -s --backup=simple --suffix=_original.xml \
-    /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/
-
-if [[ -n "$USE_POLYMORPHIC_PARTS" ]] && [[ "$USE_POLYMORPHIC_PARTS" -eq 1 ]]; then
-    ln -s /usr/share/clickhouse-test/config/polymorphic_parts.xml /etc/clickhouse-server/config.d/
-fi
-if [[ -n "$USE_DATABASE_ATOMIC" ]] && [[ "$USE_DATABASE_ATOMIC" -eq 1 ]]; then
-    ln -s /usr/share/clickhouse-test/config/database_atomic_configd.xml /etc/clickhouse-server/config.d/
-    ln -s /usr/share/clickhouse-test/config/database_atomic_usersd.xml /etc/clickhouse-server/users.d/
-fi
-
-ln -sf /usr/share/clickhouse-test/config/client_config.xml /etc/clickhouse-client/config.xml
-
-service zookeeper start
-sleep 5
 service clickhouse-server start && sleep 5
 
 if cat /usr/bin/clickhouse-test | grep -q -- "--use-skip-list"; then

docker/test/stateless_with_coverage/Dockerfile

@@ -11,8 +11,6 @@ RUN apt-get update -y \
             tzdata \
             fakeroot \
             debhelper \
-            zookeeper \
-            zookeeperd \
             expect \
             python \
             python-lxml \

docker/test/stateless_with_coverage/run.sh

@@ -39,41 +39,8 @@ mkdir -p /var/log/clickhouse-server
 chmod 777 -R /var/lib/clickhouse
 chmod 777 -R /var/log/clickhouse-server/
 
-# Temorary way to keep CI green while moving dictionaries to separate directory
+# install test configs
-mkdir -p /etc/clickhouse-server/dict_examples
+/usr/share/clickhouse-test/config/install.sh
-chmod 777 -R /etc/clickhouse-server/dict_examples
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/dict_examples/; \
-    ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/dict_examples/; \
-    ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/dict_examples/;
-
-ln -s /usr/share/clickhouse-test/config/zookeeper.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/listen.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/metric_log.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/readonly.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/access_management.xml /etc/clickhouse-server/users.d/
-ln -s /usr/share/clickhouse-test/config/ints_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/strings_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/decimals_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/executable_dictionary.xml /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/disks.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/secure_ports.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/clusters.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/graphite.xml /etc/clickhouse-server/config.d/
-ln -s /usr/share/clickhouse-test/config/server.key /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/server.crt /etc/clickhouse-server/
-ln -s /usr/share/clickhouse-test/config/dhparam.pem /etc/clickhouse-server/
-ln -sf /usr/share/clickhouse-test/config/client_config.xml /etc/clickhouse-client/config.xml
-
-# Retain any pre-existing config and allow ClickHouse to load it if required
-ln -s --backup=simple --suffix=_original.xml \
-    /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/
-
-service zookeeper start
-sleep 5
 
 start_clickhouse
 

docker/test/stress/run.sh

@@ -39,9 +39,8 @@ function start()
     done
 }
 
-ln -s /usr/share/clickhouse-test/config/log_queries.xml /etc/clickhouse-server/users.d/
+# install test configs
-ln -s /usr/share/clickhouse-test/config/part_log.xml /etc/clickhouse-server/config.d/
+/usr/share/clickhouse-test/config/install.sh
-ln -s /usr/share/clickhouse-test/config/text_log.xml /etc/clickhouse-server/config.d/
 
 echo "ASAN_OPTIONS='malloc_context_size=10 verbosity=1 allocator_release_to_os_interval_ms=10000'" >> /etc/environment
 

docker/test/stress/stress

@@ -29,7 +29,7 @@ def get_options(i):
     if 0 < i:
         options += " --order=random"
     if i % 2 == 1:
-        options += " --atomic-db-engine"
+        options += " --db-engine=Ordinary"
     return options
 
 

docker/test/testflows/runner/Dockerfile

@@ -35,7 +35,7 @@ RUN apt-get update \
 ENV TZ=Europe/Moscow
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
-RUN pip3 install urllib3 testflows==1.6.42 docker-compose docker dicttoxml kazoo tzlocal
+RUN pip3 install urllib3 testflows==1.6.48 docker-compose docker dicttoxml kazoo tzlocal
 
 ENV DOCKER_CHANNEL stable
 ENV DOCKER_VERSION 17.09.1-ce

docs/_includes/cmake_in_clickhouse_footer.md

@@ -0,0 +1,121 @@
+
+## Developer's guide for adding new CMake options
+
+### Don't be obvious. Be informative.
+
+Bad:
+```cmake
+option (ENABLE_TESTS "Enables testing" OFF)
+```
+
+This description is quite useless as is neither gives the viewer any additional information nor explains the option purpose.
+
+Better:
+
+```cmake
+option(ENABLE_TESTS "Provide unit_test_dbms target with Google.test unit tests" OFF)
+```
+
+If the option's purpose can't be guessed by its name, or the purpose guess may be misleading, or option has some 
+pre-conditions, leave a comment above the `option()` line and explain what it does. 
+The best way would be linking the docs page (if it exists).
+The comment is parsed into a separate column (see below).
+
+Even better:
+
+```cmake
+# implies ${TESTS_ARE_ENABLED}
+# see tests/CMakeLists.txt for implementation detail.
+option(ENABLE_TESTS "Provide unit_test_dbms target with Google.test unit tests" OFF)
+```
+
+### If the option's state could produce unwanted (or unusual) result, explicitly warn the user.
+
+Suppose you have an option that may strip debug symbols from the ClickHouse's part.
+This can speed up the linking process, but produces a binary that cannot be debugged.
+In that case, prefer explicitly raising a warning telling the developer that he may be doing something wrong. 
+Also, such options should be disabled if applies.
+
+Bad:
+```cmake
+option(STRIP_DEBUG_SYMBOLS_FUNCTIONS
+    "Do not generate debugger info for ClickHouse functions.
+    ${STRIP_DSF_DEFAULT})
+
+if (STRIP_DEBUG_SYMBOLS_FUNCTIONS)
+    target_compile_options(clickhouse_functions PRIVATE "-g0")
+endif()
+
+```
+Better:
+
+```cmake
+# Provides faster linking and lower binary size.
+# Tradeoff is the inability to debug some source files with e.g. gdb
+# (empty stack frames and no local variables)."
+option(STRIP_DEBUG_SYMBOLS_FUNCTIONS
+    "Do not generate debugger info for ClickHouse functions."
+    ${STRIP_DSF_DEFAULT})
+
+if (STRIP_DEBUG_SYMBOLS_FUNCTIONS)
+    message(WARNING "Not generating debugger info for ClickHouse functions")
+    target_compile_options(clickhouse_functions PRIVATE "-g0")
+endif()
+```
+
+### In the option's description, explain WHAT the option does rather than WHY it does something.
+
+The WHY explanation should be placed in the comment.
+You may find that the option's name is self-descriptive.
+
+Bad:
+
+```cmake
+option(ENABLE_THINLTO "Enable Thin LTO. Only applicable for clang. It's also suppressed when building with tests or sanitizers." ON)
+```
+
+Better:
+
+```cmake
+# Only applicable for clang.
+# Turned off when building with tests or sanitizers.
+option(ENABLE_THINLTO "Clang-specific link time optimisation" ON).
+```
+
+### Don't assume other developers know as much as you do.
+
+In ClickHouse, there are many tools used that an ordinary developer may not know. If you are in doubt, give a link to
+the tool's docs. It won't take much of your time.
+
+Bad:
+
+```cmake
+option(ENABLE_THINLTO "Enable Thin LTO. Only applicable for clang. It's also suppressed when building with tests or sanitizers." ON)
+```
+
+Better (combined with the above hint):
+
+```cmake
+# https://clang.llvm.org/docs/ThinLTO.html
+# Only applicable for clang.
+# Turned off when building with tests or sanitizers.
+option(ENABLE_THINLTO "Clang-specific link time optimisation" ON).
+```
+
+Other example, bad:
+
+```cmake
+option (USE_INCLUDE_WHAT_YOU_USE "Use 'include-what-you-use' tool" OFF)
+```
+
+Better:
+
+```cmake
+# https://github.com/include-what-you-use/include-what-you-use
+option (USE_INCLUDE_WHAT_YOU_USE "Reduce unneeded #include s (external tool)" OFF)
+```
+
+### Prefer consistent default values.
+
+CMake allows you to pass a plethora of values representing boolean `true/false`, e.g. `1, ON, YES, ...`.
+Prefer the `ON/OFF` values, if possible.

docs/_includes/cmake_in_clickhouse_header.md

@@ -0,0 +1,34 @@
+# CMake in ClickHouse
+
+## TL; DR How to make ClickHouse compile and link faster?
+
+Developer only! This command will likely fulfill most of your needs. Run before calling `ninja`.
+
+```cmake
+cmake .. \
+    -DCMAKE_C_COMPILER=/bin/clang-10 \
+    -DCMAKE_CXX_COMPILER=/bin/clang++-10 \
+    -DCMAKE_BUILD_TYPE=Debug \
+    -DENABLE_CLICKHOUSE_ALL=OFF \
+    -DENABLE_CLICKHOUSE_SERVER=ON \
+    -DENABLE_CLICKHOUSE_CLIENT=ON \
+    -DUSE_STATIC_LIBRARIES=OFF \
+    -DCLICKHOUSE_SPLIT_BINARY=ON \
+    -DSPLIT_SHARED_LIBRARIES=ON \
+    -DENABLE_LIBRARIES=OFF \
+    -DENABLE_UTILS=OFF \
+    -DENABLE_TESTS=OFF
+```
+
+## CMake files types
+
+1. ClickHouse's source CMake files (located in the root directory and in `/src`).
+2. Arch-dependent CMake files (located in `/cmake/*os_name*`).
+3. Libraries finders (search for contrib libraries, located in `/cmake/find`).
+3. Contrib build CMake files (used instead of libraries' own CMake files, located in `/cmake/modules`)
+
+## List of CMake flags
+
+* This list is auto-generated by [this Python script](https://github.com/clickhouse/clickhouse/blob/master/docs/tools/cmake_in_clickhouse_generator.py).
+* The flag name is a link to its position in the code.
+* If an option's default value is itself an option, it's also a link to its position in this list.

docs/en/introduction/adopters.md

@@ -38,7 +38,7 @@ toc_title: Adopters
 | <a href="https://db.com" class="favicon">Deutsche Bank</a>                                     | Finance                         | BI Analytics          | —                                                          | —                                                                            | [Slides in English, October 2019](https://bigdatadays.ru/wp-content/uploads/2019/10/D2-H3-3_Yakunin-Goihburg.pdf)                                                                                                       |
 | <a href="https://www.diva-e.com" class="favicon">Diva-e</a>                                    | Digital consulting              | Main Product          | —                                                          | —                                                                            | [Slides in English, September 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup29/ClickHouse-MeetUp-Unusual-Applications-sd-2019-09-17.pdf)                                               |
 | <a href="https://www.ecwid.com/" class="favicon">Ecwid</a>                                    | E-commerce SaaS | Metrics, Logging      | —                                                          | —                                                                            | [Slides in Russian, April 2019](https://nastachku.ru/var/files/1/presentation/backend/2_Backend_6.pdf)                                                                                                                                                        |
-| <a href="https://www.ebay.com/" class="favicon">eBay</a>                                    | E-commerce | TBA | —                                                          | —                                                                            | [Webinar, Sep 2020](https://altinity.com/webinarspage/2020/09/08/migrating-from-druid-to-next-gen-olap-on-clickhouse-ebays-experience)                                                                                                                                                        |
+| <a href="https://www.ebay.com/" class="favicon">eBay</a>                                    | E-commerce | Logs, Metrics and Events | —                                                          | —                                                                            | [Official website, Sep 2020](https://tech.ebayinc.com/engineering/ou-online-analytical-processing/)                                                                                                                                                        |
 | <a href="https://www.exness.com" class="favicon">Exness</a>                                    | Trading                         | Metrics, Logging      | —                                                          | —                                                                            | [Talk in Russian, May 2019](https://youtu.be/_rpU-TvSfZ8?t=3215)                                                                                                                                                        |
 | <a href="https://fastnetmon.com/" class="favicon">FastNetMon</a> | DDoS Protection | Main Product |  | —                                                                            | [Official website](https://fastnetmon.com/docs-fnm-advanced/fastnetmon-advanced-traffic-persistency/) |
 | <a href="https://www.flipkart.com/" class="favicon">Flipkart</a>                               | e-Commerce                      | —                     | —                                                          | —                                                                            | [Talk in English, July 2020](https://youtu.be/GMiXCMFDMow?t=239)                                                                                               |

docs/en/introduction/info.md

@@ -0,0 +1,10 @@
+---
+toc_priority: 100
+---
+
+# Information support {#information-support}
+
+- Email address: <a class="feedback-email"></a>
+- Phone: <a href="tel:+74957806510">+7-495-780-6510</a>
+
+[Original article](https://clickhouse.tech/docs/en/introduction/info/) <!--hide-->

docs/en/operations/utilities/clickhouse-obfuscator.md

@@ -0,0 +1,42 @@
+# ClickHouse obfuscator
+
+Simple tool for table data obfuscation.
+
+It reads input table and produces output table, that retain some properties of input, but contains different data.
+It allows to publish almost real production data for usage in benchmarks.
+
+It is designed to retain the following properties of data:
+- cardinalities of values (number of distinct values) for every column and for every tuple of columns;
+- conditional cardinalities: number of distinct values of one column under condition on value of another column;
+- probability distributions of absolute value of integers; sign of signed integers; exponent and sign for floats;
+- probability distributions of length of strings;
+- probability of zero values of numbers; empty strings and arrays, NULLs;
+- data compression ratio when compressed with LZ77 and entropy family of codecs;
+- continuity (magnitude of difference) of time values across table; continuity of floating point values.
+- date component of DateTime values;
+- UTF-8 validity of string values;
+- string values continue to look somewhat natural.
+
+Most of the properties above are viable for performance testing:
+
+reading data, filtering, aggregation and sorting will work at almost the same speed
+as on original data due to saved cardinalities, magnitudes, compression ratios, etc.
+
+It works in deterministic fashion: you define a seed value and transform is totally determined by input data and by seed.
+Some transforms are one to one and could be reversed, so you need to have large enough seed and keep it in secret.
+
+It use some cryptographic primitives to transform data, but from the cryptographic point of view,
+It doesn't do anything properly and you should never consider the result as secure, unless you have other reasons for it.
+
+It may retain some data you don't want to publish.
+
+It always leave numbers 0, 1, -1 as is. Also it leaves dates, lengths of arrays and null flags exactly as in source data.
+For example, you have a column IsMobile in your table with values 0 and 1. In transformed data, it will have the same value.
+So, the user will be able to count exact ratio of mobile traffic.
+
+Another example, suppose you have some private data in your table, like user email and you don't want to publish any single email address.
+If your table is large enough and contain multiple different emails and there is no email that have very high frequency than all others,
+It will perfectly anonymize all data. But if you have small amount of different values in a column, it can possibly reproduce some of them.
+And you should take care and look at exact algorithm, how this tool works, and probably fine tune some of it command line parameters.
+
+This tool works fine only with reasonable amount of data (at least 1000s of rows).

docs/en/sql-reference/functions/tuple-map-functions.md

@@ -9,8 +9,7 @@ toc_title: Working with maps
 
 Collect all the keys and sum corresponding values.
 
-Arguments are tuples of two arrays, where items in the first array represent keys, and the second array
+Arguments are tuples of two arrays, where items in the first array represent keys, and the second array contains values for the each key.
-contains values for the each key.
 All key arrays should have same type, and all value arrays should contain items which are promotable to the one type (Int64, UInt64 or Float64).
 The common promoted type is used as a type for the result array.
 
@@ -30,8 +29,7 @@ SELECT mapAdd(([toUInt8(1), 2], [1, 1]), ([toUInt8(1), 2], [1, 1])) as res, toTy
 
 Collect all the keys and subtract corresponding values.
 
-Arguments are tuples of two arrays, where items in the first array represent keys, and the second array
+Arguments are tuples of two arrays, where items in the first array represent keys, and the second array contains values for the each key.
-contains values for the each key.
 All key arrays should have same type, and all value arrays should contain items which are promotable to the one type (Int64, UInt64 or Float64).
 The common promoted type is used as a type for the result array.
 
@@ -45,25 +43,24 @@ SELECT mapSubtract(([toUInt8(1), 2], [toInt32(1), 1]), ([toUInt8(1), 2], [toInt3
 ┌─res────────────┬─type──────────────────────────────┐
 │ ([1,2],[-1,0]) │ Tuple(Array(UInt8), Array(Int64)) │
 └────────────────┴───────────────────────────────────┘
-````
+```
 
 ## mapPopulateSeries {#function-mappopulateseries}
 
 Syntax: `mapPopulateSeries((keys : Array(<IntegerType>), values : Array(<IntegerType>)[, max : <IntegerType>])`
 
-Generates a map, where keys are a series of numbers, from minimum to maximum keys (or `max` argument if it specified) taken from `keys` array with step size of one,
+Generates a map, where keys are a series of numbers, from minimum to maximum keys (or `max` argument if it specified) taken from `keys` array with step size of one, and corresponding values taken from `values` array. If the value is not specified for the key, then it uses default value in the resulting map.
-and corresponding values taken from `values` array. If the value is not specified for the key, then it uses default value in the resulting map.
 For repeated keys only the first value (in order of appearing) gets associated with the key.
 
 The number of elements in `keys` and `values` must be the same for each row.
 
 Returns a tuple of two arrays: keys in sorted order, and values the corresponding keys.
 
-``` sql
+```sql
 select mapPopulateSeries([1,2,4], [11,22,44], 5) as res, toTypeName(res) as type;
 ```
 
-``` text
+```text
 ┌─res──────────────────────────┬─type──────────────────────────────┐
 │ ([1,2,3,4,5],[11,22,0,44,0]) │ Tuple(Array(UInt8), Array(UInt8)) │
 └──────────────────────────────┴───────────────────────────────────┘

docs/ru/introduction/info.md

@@ -7,6 +7,6 @@ toc_priority: 100
 Информационная поддержка ClickHouse осуществляется на всей территории Российской Федерации без ограничений посредством использования телефонной связи и средств электронной почты на русском языке в круглосуточном режиме:
 
 -   Адрес электронной почты: <a class="feedback-email"></a>
--   Телефон: <a href="tel:88002509639">8-800-250-96-39</a> (звонки бесплатны из всех регионов России)
+-   Телефон: <a href="tel:+74957806510">+7-495-780-6510</a>
 
 [Оригинальная статья](https://clickhouse.tech/docs/ru/introduction/info/) <!--hide-->

docs/ru/whats-new/extended-roadmap.md

@@ -97,7 +97,9 @@ Upd. Есть pull request. Upd. Сделано.
 
 Частный случай такой задачи уже есть в https://clickhouse.tech/docs/ru/operations/table_engines/graphitemergetree/ Но это было сделано для конкретной задачи. А надо обобщить.
 
-### 1.10. Пережатие старых данных в фоне {#perezhatie-starykh-dannykh-v-fone}
+### 1.10. + Пережатие старых данных в фоне {#perezhatie-starykh-dannykh-v-fone}
+
+В master, сделал Александр Сапин, https://github.com/ClickHouse/ClickHouse/pull/14494
 
 Будет делать Кирилл Барухов, ВШЭ, экспериментальная реализация к весне 2020. Нужно для Яндекс.Метрики.
 
@@ -138,27 +140,32 @@ Upd: PR [#10463](https://github.com/ClickHouse/ClickHouse/pull/10463)
 
 ### 1.14. Не писать столбцы, полностью состоящие из нулей {#ne-pisat-stolbtsy-polnostiu-sostoiashchie-iz-nulei}
 
-Антон Попов. Q3.
+Антон Попов. Q4.
 В очереди. Простая задача, является небольшим пререквизитом для потенциальной поддержки полуструктурированных данных.
+Upd. В очереди после чтения срезов столбцов.
 
 ### 1.15. Возможность иметь разный первичный ключ в разных кусках {#vozmozhnost-imet-raznyi-pervichnyi-kliuch-v-raznykh-kuskakh}
 
 Сложная задача, только после 1.3.
 Upd. В обсуждении.
+Upd. Взял в работу Amos Bird. Описана концепция. Совпадает с 1.16.
 
 ### 1.16. Несколько физических представлений для одного куска данных {#neskolko-fizicheskikh-predstavlenii-dlia-odnogo-kuska-dannykh}
 
 Сложная задача, только после 1.3 и 1.6. Позволяет компенсировать 21.20.
 Upd. В обсуждении.
+Upd. Взял в работу Amos Bird. Описана концепция, работа на начальной стадии.
 
 ### 1.17. Несколько сортировок для одной таблицы {#neskolko-sortirovok-dlia-odnoi-tablitsy}
 
 Сложная задача, только после 1.3 и 1.6.
 Upd. В обсуждении.
+Upd. Взял в работу Amos Bird. Описана концепция. Совпадает с 1.16.
 
-### 1.18. Отдельное хранение файлов кусков {#otdelnoe-khranenie-failov-kuskov}
+### 1.18. - Отдельное хранение файлов кусков {#otdelnoe-khranenie-failov-kuskov}
 
 Требует 1.3 и 1.6. Полная замена hard links на sym links, что будет лучше для 1.12.
+Отменено.
 
 
 ## 2. Крупные рефакторинги {#krupnye-refaktoringi}
@@ -194,13 +201,14 @@ Upd. Старый код по большей части удалён.
 
 ### 2.5. Версионирование состояний агрегатных функций {#versionirovanie-sostoianii-agregatnykh-funktsii}
 
-В очереди.
+В очереди. Описана схема реализации. Алексей Миловидов.
 
 ### 2.6. Правая часть IN как тип данных. Выполнение IN в виде скалярного подзапроса {#pravaia-chast-in-kak-tip-dannykh-vypolnenie-in-v-vide-skaliarnogo-podzaprosa}
 
 Требует 2.1.
+Отменено.
 
-### 2.7. Нормализация Context {#normalizatsiia-context}
+### 2.7. + Нормализация Context {#normalizatsiia-context}
 
 В очереди. Нужно для YQL.
 
@@ -209,12 +217,14 @@ Upd. Старый код по большей части удалён.
 Upd. Каталог БД вынесен из Context.
 Upd. SharedContext вынесен из Context.
 Upd. Проблема нейтрализована и перестала быть актуальной.
+Upd. Вообще всё стало Ок.
 
 ### 2.8. Декларативный парсер запросов {#deklarativnyi-parser-zaprosov}
 
 Средний приоритет. Нужно для YQL.
 
 Upd. В очереди. Иван Лежанкин.
+Upd. Задача в финальной стадии. Пока рассматривается только как альтернативный парсер, описание которого подойдёт для сторонних приложений.
 
 ### 2.9. + Логгировние в format-стиле {#loggirovnie-v-format-stile}
 
@@ -225,10 +235,12 @@ Upd. В очереди. Иван Лежанкин.
 ### 2.10. Запрашивать у таблиц не столбцы, а срезы {#zaprashivat-u-tablits-ne-stolbtsy-a-srezy}
 
 В очереди.
+В работе, Антон Попов, Q4.
 
-### 2.11. Разбирательство и нормализация функциональности для bitmap {#razbiratelstvo-i-normalizatsiia-funktsionalnosti-dlia-bitmap}
+### 2.11. - Разбирательство и нормализация функциональности для bitmap {#razbiratelstvo-i-normalizatsiia-funktsionalnosti-dlia-bitmap}
 
 В очереди.
+Не актуально.
 
 ### 2.12. Декларативные сигнатуры функций {#deklarativnye-signatury-funktsii}
 
@@ -265,7 +277,7 @@ Upd. Поползновения наблюдаются.
 
 Требует 3.1.
 
-### + 3.3. Исправить катастрофически отвратительно неприемлемый поиск по документации {#ispravit-katastroficheski-otvratitelno-nepriemlemyi-poisk-po-dokumentatsii}
+### 3.3. + Исправить катастрофически отвратительно неприемлемый поиск по документации {#ispravit-katastroficheski-otvratitelno-nepriemlemyi-poisk-po-dokumentatsii}
 
 [Иван Блинков](https://github.com/blinkov/) - очень хороший человек. Сам сайт документации основан на технологиях, не удовлетворяющих требованиям задачи, и эти технологии трудно исправить. Задачу будет делать первый встретившийся нам frontend разработчик, которого мы сможем заставить это сделать.
 
@@ -311,7 +323,6 @@ Upd. Сейчас обсуждается, как сделать другую з
 ### 4.8. Разделить background pool для fetch и merge {#razdelit-background-pool-dlia-fetch-i-merge}
 
 В очереди. Исправить проблему, что восстанавливающаяся реплика перестаёт мержить. Частично компенсируется 4.3.
-Александр Казаков.
 
 
 ## 5. Операции {#operatsii}
@@ -450,6 +461,7 @@ UBSan включен в функциональных тестах, но не в
 ### 7.12. Показывать тестовое покрытие нового кода в PR {#pokazyvat-testovoe-pokrytie-novogo-koda-v-pr}
 
 Пока есть просто показ тестового покрытия всего кода.
+Отложено.
 
 ### 7.13. + Включение аналога -Weverything в gcc {#vkliuchenie-analoga-weverything-v-gcc}
 
@@ -598,7 +610,7 @@ Upd. Сергей Штыков сделал функцию `randomPrintableASCII
 Upd. Илья Яцишин сделал табличную функцию `generateRandom`.
 Upd. Эльдар Заитов добавляет OSS Fuzz.
 Upd. Сделаны randomString, randomFixedString.
-Upd. Сделаны fuzzBits, fuzzBytes.
+Upd. Сделаны fuzzBits.
 
 ### 7.24. Fuzzing лексера и парсера запросов; кодеков и форматов {#fuzzing-leksera-i-parsera-zaprosov-kodekov-i-formatov}
 
@@ -649,10 +661,11 @@ Upd. В Аркадии частично работает небольшая ча
 
 В очереди. Нужно для Яндекс.Метрики.
 
-### 7.32. Обфускация продакшен запросов {#obfuskatsiia-prodakshen-zaprosov}
+### 7.32. + Обфускация продакшен запросов {#obfuskatsiia-prodakshen-zaprosov}
 
 Роман Ильговский. Нужно для Яндекс.Метрики.
-Есть pull request, почти готово: https://github.com/ClickHouse/ClickHouse/pull/10973
+Есть pull request: https://github.com/ClickHouse/ClickHouse/pull/10973
+Готово.
 
 Имея SQL запрос, требуется вывести структуру таблиц, на которых этот запрос будет выполнен, и заполнить эти таблицы случайными данными, такими, что результат этого запроса зависит от выбора подмножества данных.
 
@@ -660,6 +673,8 @@ Upd. В Аркадии частично работает небольшая ча
 
 Обфускация запросов: имея секретные запросы и структуру таблиц, заменить имена полей и константы, чтобы запросы можно было использовать в качестве публично доступных тестов.
 
+Upd. Последняя часть пока не сделана и будет сделана отдельно.
+
 ### 7.33. Выкладывать патч релизы в репозиторий автоматически {#vykladyvat-patch-relizy-v-repozitorii-avtomaticheski}
 
 В очереди. Иван Лежанкин.
@@ -701,10 +716,11 @@ Upd. Частично решён вопрос с visibility - есть како
 
 Altinity. Никто не делает эту задачу.
 
-### 8.2. Поддержка Mongo Atlas URI {#podderzhka-mongo-atlas-uri}
+### 8.2. - Поддержка Mongo Atlas URI {#podderzhka-mongo-atlas-uri}
 
 [Александр Кузьменков](https://github.com/akuzm).
 Upd. Задача взята в работу.
+Все pull requests успешно закрыты.
 
 ### 8.3. + Доработки globs (правильная поддержка диапазонов, уменьшение числа одновременных stream-ов) {#dorabotki-globs-pravilnaia-podderzhka-diapazonov-umenshenie-chisla-odnovremennykh-stream-ov}
 
@@ -721,6 +737,7 @@ Upd. Задача взята в работу.
 ### 8.6. Kerberos аутентификация для HDFS и Kafka {#kerberos-autentifikatsiia-dlia-hdfs-i-kafka}
 
 Андрей Коняев, ArenaData. Он куда-то пропал.
+Upd. В процессе работа для Kafka.
 
 ### 8.7. + Исправление мелочи HDFS на очень старых ядрах Linux {#ispravlenie-melochi-hdfs-na-ochen-starykh-iadrakh-linux}
 
@@ -759,6 +776,8 @@ Upd. В стадии код-ревью.
 
 ### 8.15. Запись данных в CapNProto {#zapis-dannykh-v-capnproto}
 
+Отложено.
+
 ### 8.16. + Поддержка формата Avro {#podderzhka-formata-avro}
 
 Andrew Onyshchuk. Есть pull request. Q1. Сделано.
@@ -814,12 +833,13 @@ Upd. Готово.
 
 Низкий приоритет. Отменено.
 
-### 8.21. Поддержка произвольного количества языков для имён регионов {#podderzhka-proizvolnogo-kolichestva-iazykov-dlia-imion-regionov}
+### 8.21. - Поддержка произвольного количества языков для имён регионов {#podderzhka-proizvolnogo-kolichestva-iazykov-dlia-imion-regionov}
 
 Нужно для БК. Декабрь 2019.
 В декабре для БК сделан минимальный вариант этой задачи.
 Максимальный вариант, вроде, никому не нужен.
 Upd. Всё ещё кажется, что задача не нужна.
+Отменено.
 
 ### 8.22. + Поддержка синтаксиса для переменных в стиле MySQL {#podderzhka-sintaksisa-dlia-peremennykh-v-stile-mysql}
 
@@ -831,6 +851,7 @@ Upd. Сделано теми людьми, кому не запрещено ра
 ### 8.23. Подписка для импорта обновляемых и ротируемых логов в ФС {#podpiska-dlia-importa-obnovliaemykh-i-rotiruemykh-logov-v-fs}
 
 Желательно 2.15.
+Отложено.
 
 
 ## 9. Безопасность {#bezopasnost}
@@ -870,9 +891,10 @@ Upd. Одну причину устранили, но ещё что-то неи
 Upd. Нас заставляют переписать эту библиотеку с одного API на другое, так как старое внезапно устарело. Кажется, что переписывание случайно исправит все проблемы.
 Upd. Ура, нашли причину и исправили.
 
-### 10.3. Возможность чтения данных из статических таблиц в YT словарях {#vozmozhnost-chteniia-dannykh-iz-staticheskikh-tablits-v-yt-slovariakh}
+### 10.3. - Возможность чтения данных из статических таблиц в YT словарях {#vozmozhnost-chteniia-dannykh-iz-staticheskikh-tablits-v-yt-slovariakh}
 
 Нужно для БК и Метрики.
+Отменено.
 
 ### 10.4. - Словарь из YDB (KikiMR) {#slovar-iz-ydb-kikimr}
 
@@ -884,9 +906,11 @@ Upd. Ура, нашли причину и исправили.
 
 Для MySQL сделал Clément Rodriguez.
 
-### 10.6. Словари из Cassandra и Couchbase {#slovari-iz-cassandra-i-couchbase}
+### 10.6. + Словари из Cassandra и Couchbase {#slovari-iz-cassandra-i-couchbase}
 
 Готова Cassandra.
+Couchbase отменён, так как не было спроса.
+Aerospike под вопросом.
 
 ### 10.7. Поддержка Nullable в словарях {#podderzhka-nullable-v-slovariakh}
 
@@ -929,10 +953,14 @@ Upd. Задача в финальной стадии готовности.
 
 ### 10.17. Локальный дамп состояния словаря для быстрого старта сервера {#lokalnyi-damp-sostoianiia-slovaria-dlia-bystrogo-starta-servera}
 
+Отложено.
+
 ### 10.18. Таблица Join или словарь на удалённом сервере как key-value БД для cache словаря {#tablitsa-join-ili-slovar-na-udalionnom-servere-kak-key-value-bd-dlia-cache-slovaria}
 
 ### 10.19. Возможность зарегистрировать некоторые функции, использующие словари, под пользовательскими именами {#vozmozhnost-zaregistrirovat-nekotorye-funktsii-ispolzuiushchie-slovari-pod-polzovatelskimi-imenami}
 
+Отложено.
+
 
 ## 11. Интерфейсы {#interfeisy}
 
@@ -943,6 +971,7 @@ Upd. Задача в финальной стадии готовности.
 Нужно разобраться, как упаковывать Java в статический бинарник, возможно AppImage. Или предоставить максимально простую инструкцию по установке jdbc-bridge. Может быть будет заинтересован Александр Крашенинников, Badoo, так как он разработал jdbc-bridge.
 
 Upd. Александр Крашенинников перешёл в другую компанию и больше не занимается этим.
+Upd. Задачу взял Zhichun Wu.
 
 ### 11.3. + Интеграционные тесты ODBC драйвера путём подключения ClickHouse к самому себе через ODBC {#integratsionnye-testy-odbc-draivera-putiom-podkliucheniia-clickhouse-k-samomu-sebe-cherez-odbc}
 
@@ -960,6 +989,8 @@ Altinity целиком взяли на себя поддержку clickhouse-c
 
 ### 11.7. Интерактивный режим работы программы clickhouse-local {#interaktivnyi-rezhim-raboty-programmy-clickhouse-local}
 
+Отложено.
+
 ### 11.8. + Поддержка протокола PostgreSQL {#podderzhka-protokola-postgresql}
 
 Элбакян Мовсес Андраникович, ВШЭ.
@@ -998,14 +1029,17 @@ Q1. Сделано управление правами полностью, но
 Аутентификация через LDAP - Денис Глазачев.
 [Виталий Баранов](https://github.com/vitlibar) и Денис Глазачев, Altinity. Требует 12.1.
 Q3.
+Upd. Pull request на финальной стадии.
 
 ### 12.4. Подключение IDM системы Яндекса как справочника пользователей и прав доступа {#podkliuchenie-idm-sistemy-iandeksa-kak-spravochnika-polzovatelei-i-prav-dostupa}
 
 Пока низкий приоритет. Нужно для Метрики. Требует 12.3.
+Отложено.
 
 ### 12.5. Pluggable аутентификация с помощью Kerberos (возможно, подключение GSASL) {#pluggable-autentifikatsiia-s-pomoshchiu-kerberos-vozmozhno-podkliuchenie-gsasl}
 
 [Виталий Баранов](https://github.com/vitlibar) и Денис Глазачев, Altinity. Требует 12.1.
+Upd. Есть pull request.
 
 ### 12.6. + Информация о пользователях и квотах в системной таблице {#informatsiia-o-polzovateliakh-i-kvotakh-v-sistemnoi-tablitse}
 
@@ -1033,6 +1067,7 @@ Q3.
 Upd. Не уследили, и задачу стали обсуждать менеджеры.
 Upd. Задачу смотрит Александр Казаков.
 Upd. Задача взята в работу.
+Upd. Задача как будто взята в работу.
 
 
 ## 14. Диалект SQL {#dialekt-sql}
@@ -1041,7 +1076,9 @@ Upd. Задача взята в работу.
 
 Нужно для DataLens. А также для внедрения в BI инструмент Looker.
 
-### 14.2. Поддержка WITH для подзапросов {#podderzhka-with-dlia-podzaprosov}
+### 14.2. + Поддержка WITH для подзапросов {#podderzhka-with-dlia-podzaprosov}
+
+Сделал Amos Bird.
 
 ### 14.3. Поддержка подстановок для множеств в правой части IN {#podderzhka-podstanovok-dlia-mnozhestv-v-pravoi-chasti-in}
 
@@ -1057,11 +1094,13 @@ zhang2014
 
 ### 14.6. Глобальный scope для WITH {#globalnyi-scope-dlia-with}
 
+В обсуждении. Amos Bird.
+
 ### 14.7. Nullable для WITH ROLLUP, WITH CUBE, WITH TOTALS {#nullable-dlia-with-rollup-with-cube-with-totals}
 
 Простая задача.
 
-### 14.8. Модификаторы DISTINCT, ORDER BY для агрегатных функций {#modifikatory-distinct-order-by-dlia-agregatnykh-funktsii}
+### 14.8. + Модификаторы DISTINCT, ORDER BY для агрегатных функций {#modifikatory-distinct-order-by-dlia-agregatnykh-funktsii}
 
 В ClickHouse поддерживается вычисление COUNT(DISTINCT x). Предлагается добавить возможность использования модификатора DISTINCT для всех агрегатных функций. Например, AVG(DISTINCT x) - вычислить среднее значение для всех различных значений x. Под вопросом вариант, в котором фильтрация уникальных значений выполняется по одному выражению, а агрегация по другому.
 
@@ -1069,6 +1108,7 @@ zhang2014
 
 Upd. Есть pull request-ы.
 Upd. DISTINCT готов.
+Upd. ORDER BY отменён и будет заново сделан уже с LIMIT.
 
 ### 14.9. + Поддержка запроса EXPLAIN {#podderzhka-zaprosa-explain}
 
@@ -1079,8 +1119,12 @@ Upd. Есть pull request. Готово.
 
 ### 14.11. Функции для grouping sets {#funktsii-dlia-grouping-sets}
 
+Отложено.
+
 ### 14.12. Функции обработки временных рядов {#funktsii-obrabotki-vremennykh-riadov}
 
+Отложено.
+
 Сложная задача, так как вводит новый класс функций и требует его обработку в оптимизаторе запросов.
 
 В time-series СУБД нужны функции, которые зависят от последовательности значений. Или даже от последовательности значений и их меток времени. Примеры: moving average, exponential smoothing, derivative, Holt-Winters forecast. Вычисление таких функций поддерживается в ClickHouse лишь частично. Так, ClickHouse поддерживает тип данных «массив» и позволяет реализовать эти функции как функции, принимающие массивы. Но гораздо удобнее для пользователя было бы иметь возможность применить такие функции к таблице (промежуточному результату запроса после сортировки).
@@ -1089,6 +1133,8 @@ Upd. Есть pull request. Готово.
 
 ### 14.13. Применимость функций высшего порядка для кортежей и Nested {#primenimost-funktsii-vysshego-poriadka-dlia-kortezhei-i-nested}
 
+После задачи "чтение срезов столбцов".
+
 ### 14.14. Неявные преобразования типов констант {#neiavnye-preobrazovaniia-tipov-konstant}
 
 Сделано для операторов сравнения с константами (подавляющее большинство use cases).
@@ -1180,12 +1226,14 @@ Upd. Секретного изменения в работе не будет, з
 
 ### 16.5. Функции для XML и HTML escape {#funktsii-dlia-xml-i-html-escape}
 
-### 16.6. Функции нормализации и хэширования SQL запросов {#funktsii-normalizatsii-i-kheshirovaniia-sql-zaprosov}
+### 16.6. + Функции нормализации и хэширования SQL запросов {#funktsii-normalizatsii-i-kheshirovaniia-sql-zaprosov}
+
+Алексей Миловидов. Сделано.
 
 
 ## 17. Работа с географическими данными {#rabota-s-geograficheskimi-dannymi}
 
-### 17.1. Гео-словари для определения региона по координатам {#geo-slovari-dlia-opredeleniia-regiona-po-koordinatam}
+### 17.1. + Гео-словари для определения региона по координатам {#geo-slovari-dlia-opredeleniia-regiona-po-koordinatam}
 
 [Андрей Чулков](https://github.com/achulkov2), Антон Кваша, Артур Петуховский, ВШЭ.
 Будет основано на коде от Арслана Урташева.
@@ -1198,6 +1246,7 @@ Upd. Андрей сделал прототип интерфейса и реал
 Upd. Андрей сделал прототип более оптимальной структуры данных.
 Upd. Есть обнадёживающие результаты.
 Upd. В ревью.
+Upd. В релизе.
 
 ### 17.2. GIS типы данных и операции {#gis-tipy-dannykh-i-operatsii}
 
@@ -1227,6 +1276,7 @@ Upd. Есть pull request.
 Александр Кожихов, Максим Кузнецов. Обнаружена фундаментальная проблема в реализации, доделывает предположительно [Николай Кочетов](https://github.com/KochetovNicolai). Он может делегировать задачу кому угодно.
 
 Исправление фундаментальной проблемы - есть PR.
+Фундаментальная проблема решена.
 
 ### 18.2. Агрегатные функции для статистических тестов {#agregatnye-funktsii-dlia-statisticheskikh-testov}
 
@@ -1235,16 +1285,20 @@ Upd. Есть pull request.
 Предлагается реализовать в ClickHouse статистические тесты (Analysis of Variance, тесты нормальности распределения и т. п.) в виде агрегатных функций. Пример: `welchTTest(value, sample_idx)`.
 
 Сделали прототип двух тестов, есть pull request. Также есть pull request для корелляции рангов.
+Upd. Помержили корелляцию рангов, но ещё не помержили сравнение t-test, u-test.
 
 ### 18.3. Инфраструктура для тренировки моделей в ClickHouse {#infrastruktura-dlia-trenirovki-modelei-v-clickhouse}
 
 В очереди.
+Отложено.
 
 
 ## 19. Улучшение работы кластера {#uluchshenie-raboty-klastera}
 
 ### 19.1. Параллельные кворумные вставки без линеаризуемости {#parallelnye-kvorumnye-vstavki-bez-linearizuemosti}
 
+Upd. В работе, ожидается в начале октября.
+
 Репликация данных в ClickHouse по-умолчанию является асинхронной без выделенного мастера. Это значит, что клиент, осуществляющий вставку данных, получает успешный ответ после того, как данные попали на один сервер; репликация данных по остальным серверам осуществляется в другой момент времени. Это ненадёжно, потому что допускает потерю только что вставленных данных при потере лишь одного сервера.
 
 Для решения этой проблемы, в ClickHouse есть возможность включить «кворумную» вставку. Это значит, что клиент, осуществляющий вставку данных, получает успешный ответ после того, как данные попали на несколько (кворум) серверов. Обеспечивается линеаризуемость: клиент, получает успешный ответ после того, как данные попали на несколько реплик, *которые содержат все предыдущие данные, вставленные с кворумом* (такие реплики можно называть «синхронными»), и при запросе SELECT можно выставить настройку, разрешающую только чтение с синхронных реплик.
@@ -1265,6 +1319,7 @@ Upd. Есть pull request.
 
 Upd. Алексей сделал какой-то вариант, но борется с тем, что ничего не работает.
 Upd. Есть pull request на начальной стадии.
+Upd. Взято в работу, но непонятна перспектива, так как не ясно, подлежат ли исправлению некоторые нюансы.
 
 ### 19.3. - Подключение YT Cypress или YDB как альтернативы ZooKeeper {#podkliuchenie-yt-cypress-ili-ydb-kak-alternativy-zookeeper}
 
@@ -1349,9 +1404,9 @@ Upd. Для DISTINCT есть pull request.
 [Vxider](https://github.com/Vxider), ICT
 Есть pull request.
 
-### 21.6. Уменьшение числа потоков для SELECT в случае тривиального INSERT SELECT {#umenshenie-chisla-potokov-dlia-select-v-sluchae-trivialnogo-insert-select}
+### 21.6. + Уменьшение числа потоков для SELECT в случае тривиального INSERT SELECT {#umenshenie-chisla-potokov-dlia-select-v-sluchae-trivialnogo-insert-select}
 
-ucasFL, в разработке.
+ucasFL, в разработке. Готово.
 
 ### 21.7. Кэш результатов запросов {#kesh-rezultatov-zaprosov}
 
@@ -1371,11 +1426,14 @@ Upd. В обсуждении.
 
 Upd. Есть нерабочий прототип, скорее всего будет отложено.
 Upd. Отложено до осени.
+Upd. Отложено до.
 
 ### 21.8.1. Отдельный аллокатор для кэшей с ASLR {#otdelnyi-allokator-dlia-keshei-s-aslr}
 
 В прошлом году задачу пытался сделать Данила Кутенин с помощью lfalloc из Аркадии и mimalloc из Microsoft, но оба решения не были квалифицированы для использования в продакшене. Успешная реализация задачи 21.8 отменит необходимость в этой задаче, поэтому холд.
 
+Upd. Ещё попробовали новый tcmalloc, результаты неудовлетворительные. Пока отменено.
+
 ### 21.9. Исправить push-down выражений с помощью Processors {#ispravit-push-down-vyrazhenii-s-pomoshchiu-processors}
 
 [Николай Кочетов](https://github.com/KochetovNicolai). Требует 2.1.
@@ -1384,7 +1442,7 @@ Upd. Отложено до осени.
 
 Amos Bird.
 
-### 21.11. Peephole оптимизации запросов {#peephole-optimizatsii-zaprosov}
+### 21.11. + Peephole оптимизации запросов {#peephole-optimizatsii-zaprosov}
 
 Руслан Камалов, Михаил Малафеев, Виктор Гришанин, ВШЭ
 
@@ -1399,8 +1457,9 @@ Amos Bird.
 Сделано ещё несколько оптимизаций.
 Upd. Все вышеперечисленные оптимизации доступны в pull requests.
 Upd. Из них почти все помержены, осталась одна.
+Upd. Помержили всё.
 
-### 21.12. Алгебраические оптимизации запросов {#algebraicheskie-optimizatsii-zaprosov}
+### 21.12. + Алгебраические оптимизации запросов {#algebraicheskie-optimizatsii-zaprosov}
 
 Руслан Камалов, Михаил Малафеев, Виктор Гришанин, ВШЭ
 
@@ -1415,6 +1474,7 @@ Upd. Из них почти все помержены, осталась одна
 Несколько оптимизаций есть в PR.
 Upd. Все оптимизации кроме "Обращение инъективных функций в сравнениях на равенство" есть в PR.
 Upd. Из них больше половины помержены, осталось ещё две.
+Upd. Помержили всё.
 
 ### 21.13. Fusion агрегатных функций {#fusion-agregatnykh-funktsii}
 
@@ -1427,6 +1487,7 @@ Constraints позволяют задать выражение, истиннос
 Если выражение содержит равенство, то встретив в запросе одну из частей равенства, её можно заменить на другую часть равенства, если это сделает проще чтение данных или вычисление выражения. Например, задан constraint: `URLDomain = domain(URL)`. Значит, выражение `domain(URL)` можно заменить на `URLDomain`.
 
 Upd. Возможно будет отложено на следующий год.
+Отложено на следующий год.
 
 ### 21.15. Многоступенчатое чтение данных вместо PREWHERE {#mnogostupenchatoe-chtenie-dannykh-vmesto-prewhere}
 
@@ -1442,10 +1503,11 @@ Upd. Возможно будет отложено на следующий год
 
 ### 21.18. Внутренняя параллелизация мержа больших состояний агрегатных функций {#vnutrenniaia-parallelizatsiia-merzha-bolshikh-sostoianii-agregatnykh-funktsii}
 
-### 21.19. Оптимизация сортировки {#optimizatsiia-sortirovki}
+### 21.19. + Оптимизация сортировки {#optimizatsiia-sortirovki}
 
 Василий Морозов, Арслан Гумеров, Альберт Кидрачев, ВШЭ.
 В прошлом году задачу начинал делать другой человек, но не добился достаточного прогресса.
+Upd. Сделаны самые существенные из предложенных вариантов.
 
 \+ 1.  Оптимизация top sort.
 
@@ -1481,11 +1543,13 @@ Upd. Вместо этого будем делать задачу 1.16.
 ### 21.22. Userspace page cache {#userspace-page-cache}
 
 Требует 21.8.
+Отложено.
 
-### 21.23. Ускорение работы с вторичными индексами {#uskorenie-raboty-s-vtorichnymi-indeksami}
+### 21.23. + Ускорение работы с вторичными индексами {#uskorenie-raboty-s-vtorichnymi-indeksami}
 
 zhang2014.
 Есть pull request.
+Готово.
 
 
 ## 22. Долги и недоделанные возможности {#dolgi-i-nedodelannye-vozmozhnosti}
@@ -1679,15 +1743,18 @@ Q1. [Николай Кочетов](https://github.com/KochetovNicolai).
 
 ### 24.2. Экспериментальные алгоритмы сжатия {#eksperimentalnye-algoritmy-szhatiia}
 
+Отложено.
+
 ClickHouse поддерживает LZ4 и ZSTD для сжатия данных. Эти алгоритмы являются парето-оптимальными по соотношению скорости и коэффициентам сжатия среди достаточно известных. Тем не менее, существуют менее известные алгоритмы сжатия, которые могут превзойти их по какому-либо критерию. Из потенциально более быстрых по сравнимом коэффициенте сжатия: Lizard, LZSSE, density. Из более сильных: bsc и csc. Необходимо изучить эти алгоритмы, добавить их поддержку в ClickHouse и исследовать их работу на тестовых датасетах.
 
-### 24.3. Экспериментальные кодеки {#eksperimentalnye-kodeki}
+### 24.3. - Экспериментальные кодеки {#eksperimentalnye-kodeki}
 
 Существуют специализированные алгоритмы кодирования числовых последовательностей: Group VarInt, MaskedVByte, PFOR. Необходимо изучить наиболее эффективные реализации этих алгоритмов. Примеры вы сможете найти на https://github.com/lemire и https://github.com/powturbo/ а также https://github.com/schizofreny/middle-out
 
 Внедрить их в ClickHouse в виде кодеков и изучить их работу на тестовых датасетах.
 
 Upd. Есть два pull requests в начальной стадии, отложено.
+Upd. Отменено.
 
 ### 24.4. Шифрование в ClickHouse на уровне VFS {#shifrovanie-v-clickhouse-na-urovne-vfs}
 
@@ -1697,6 +1764,7 @@ Upd. Есть два pull requests в начальной стадии, отло
 
 Обсуждаются детали реализации. Q3/Q4.
 Виталий Баранов.
+Отложено, после бэкапов.
 
 ### 24.5. Поддержка функций шифрования для отдельных значений {#podderzhka-funktsii-shifrovaniia-dlia-otdelnykh-znachenii}
 
@@ -1706,6 +1774,7 @@ Upd. Есть два pull requests в начальной стадии, отло
     Для этого требуется реализовать функции шифрования и расшифрования, доступные из SQL. Для шифрования реализовать возможность добавления нужного количества случайных бит для исключения одинаковых зашифрованных значений на одинаковых данных. Это позволит реализовать возможность «забывания» данных без удаления строк таблицы: можно шифровать данные разных клиентов разными ключами, и для того, чтобы забыть данные одного клиента, потребуется всего лишь удалить ключ.
     
 Делает Василий Немков, Altinity
+Есть pull request в процессе ревью, исправляем проблемы производительности.
 
 ### 24.6. Userspace RAID {#userspace-raid}
 
@@ -1722,6 +1791,7 @@ RAID позволяет одновременно увеличить надёжн
 Для преодоления этих ограничений, предлагается реализовать в ClickHouse встроенный алгоритм расположения данных на дисках.
 
 Есть pull request на начальной стадии.
+Отложено.
 
 ### 24.7. Вероятностные структуры данных для фильтрации по подзапросам {#veroiatnostnye-struktury-dannykh-dlia-filtratsii-po-podzaprosam}
 
@@ -1762,6 +1832,7 @@ Upd. Есть pull request. В стадии ревью. Готово.
 Рустам Гусейн-заде, ВШЭ.
 
 Есть pull request на промежуточной стадии.
+Отложено.
 
 ### 24.11. User Defined Functions {#user-defined-functions}
 
@@ -1785,7 +1856,7 @@ ClickHouse предоставляет достаточно богатый наб
 
 Upd. В работе два варианта реализации UDF.
 
-### 24.12. GPU offloading {#gpu-offloading}
+### 24.12. - GPU offloading {#gpu-offloading}
 
 Риск состоит в том, что даже известные GPU базы, такие как OmniSci, работают медленнее, чем ClickHouse.
 Преимущество возможно только на полной сортировке и JOIN.
@@ -1794,10 +1865,11 @@ Upd. В работе два варианта реализации UDF.
 В компании nVidia сделали прототип offloading вычисления GROUP BY с некоторыми из агрегатных функций в ClickHouse и обещат предоставить исходники в публичный доступ для дальнейшего развития. Предлагается изучить этот прототип и расширить его применимость для более широкого сценария использования. В качестве альтернативы, предлагается изучить исходные коды системы `OmniSci` или `Alenka` или библиотеку `CUB` https://nvlabs.github.io/cub/ и применить некоторые из алгоритмов в ClickHouse.
 
 Upd. В компании nVidia выложили прототип, теперь нужна интеграция в систему сборки.
-Upd. Интеграция в систему сборки - Иван Лежанкин.
+Upd. Интеграция в систему сборки - Иван Лежанкин (не сделано).
 Upd. Есть прототип bitonic sort.
 Upd. Прототип bitonic sort помержен, но целесообразность под вопросом (он работает медленнее).
 Наверное надо будет подержать и удалить.
+Удалили.
 
 ### 24.13. Stream запросы {#stream-zaprosy}
 
@@ -1819,6 +1891,8 @@ Upd. Есть два прототипа от внешних контрибьют
 
 В прошлом году исследование по этой задаче сделал Егор Соловьёв, ВШЭ и Яндекс.Такси. Его исследование показало, что алгоритм нельзя существенно улучшить путём изменения параметров. Но исследование лажовое, так как рассмотрен только уже использующийся алгоритм. То есть, задача остаётся открытой.
 
+Отложено.
+
 ### 24.17. Экспериментальные способы ускорения параллельного GROUP BY {#eksperimentalnye-sposoby-uskoreniia-parallelnogo-group-by}
 
 Максим Серебряков
@@ -1831,9 +1905,12 @@ Upd. Есть pull request - в большинстве случаев однов
 
 ### 24.19. Промежуточное состояние GROUP BY как структура данных для key-value доступа {#promezhutochnoe-sostoianie-group-by-kak-struktura-dannykh-dlia-key-value-dostupa}
 
+Отложено.
+
 ### 24.20. Short-circuit вычисления некоторых выражений {#short-circuit-vychisleniia-nekotorykh-vyrazhenii}
 
 Два года назад задачу попробовала сделать Анастасия Царькова, ВШЭ и Яндекс, но реализация получилась слишком неудобной и её удалили.
+В обсуждении.
 
 ### 24.21. Реализация в ClickHouse протокола распределённого консенсуса {#realizatsiia-v-clickhouse-protokola-raspredelionnogo-konsensusa}
 
@@ -1851,9 +1928,10 @@ ClickHouse также может использоваться для быстр
 
 Другая экспериментальная задача - реализация эвристик для обработки данных в неизвестном построчном текстовом формате. Детектирование CSV, TSV, JSON, детектирование разделителей и форматов значений.
 
-### 24.23. Минимальная поддержка транзакций для множества вставок/чтений {#minimalnaia-podderzhka-tranzaktsii-dlia-mnozhestva-vstavokchtenii}
+### 24.23. - Минимальная поддержка транзакций для множества вставок/чтений {#minimalnaia-podderzhka-tranzaktsii-dlia-mnozhestva-vstavokchtenii}
 
 Максим Кузнецов, ВШЭ.
+Отменено.
 
 Таблицы типа MergeTree состоят из набора независимых неизменяемых «кусков» данных. При вставках данных (INSERT), формируются новые куски. При модификациях данных (слияние кусков), формируются новые куски, а старые - становятся неактивными и перестают использоваться следующими запросами. Чтение данных (SELECT) производится из снэпшота множества кусков на некоторый момент времени. Таким образом, чтения и вставки не блокируют друг друга.
 
@@ -1863,11 +1941,12 @@ ClickHouse также может использоваться для быстр
 
 Для решения этих проблем, предлагается ввести глобальные метки времени для кусков данных (сейчас уже есть инкрементальные номера кусков, но они выделяются в рамках одной таблицы). Первым шагом сделаем эти метки времени в рамках сервера. Вторым шагом сделаем метки времени в рамках всех серверов, но неточные на основе локальных часов. Третьим шагом сделаем метки времени, выдаваемые сервисом координации.
 
-### 24.24. Реализация алгоритмов differential privacy {#realizatsiia-algoritmov-differential-privacy}
+### 24.24. - Реализация алгоритмов differential privacy {#realizatsiia-algoritmov-differential-privacy}
 
 [\#6874](https://github.com/ClickHouse/ClickHouse/issues/6874)
 
 Артём Вишняков, ВШЭ. Есть pull request.
+Отменено, так как решение имеет низкую практичность.
 
 ### 24.25. Интеграция в ClickHouse функциональности обработки HTTP User Agent {#integratsiia-v-clickhouse-funktsionalnosti-obrabotki-http-user-agent}
 
@@ -1882,6 +1961,7 @@ Upd. Есть pull request. Нужно ещё чистить код библио
 
 Александр Кожихов, ВШЭ и Яндекс.YT.
 Upd. Есть pull request с прототипом.
+Upd. Александ Кузьменков взял задачу в работу.
 
 ### 24.27. Реализация алгоритмов min-hash, sim-hash для нечёткого поиска полудубликатов {#realizatsiia-algoritmov-min-hash-sim-hash-dlia-nechiotkogo-poiska-poludublikatov}
 
@@ -1892,10 +1972,12 @@ ucasFL, ICT.
 Алгоритмы min-hash и sim-hash позволяют вычислить для текста несколько хэш-значений таких, что при небольшом изменении текста, по крайней мере один из хэшей не меняется. Вычисления можно реализовать на n-грамах и словарных шинглах. Предлагается добавить поддержку этих алгоритмов в виде функций в ClickHouse и изучить их применимость для задачи нечёткого поиска полудубликатов.
 
 Есть pull request, есть что доделывать.
+Upd. Николай Кочетов взял задачу в работу.
 
 ### 24.28. Другой sketch для квантилей {#drugoi-sketch-dlia-kvantilei}
 
 Похоже на quantileTiming, но с логарифмическими корзинами. См. DDSketch.
+Отложено.
 
 ### 24.29. Поддержка Arrow Flight {#podderzhka-arrow-flight}
 
@@ -1911,6 +1993,7 @@ Amos Bird, но его решение слишком громоздкое и п
 ### 24.31. Кореллированные подзапросы {#korellirovannye-podzaprosy}
 
 Перепиcывание в JOIN. Не раньше 21.11, 21.12, 21.9. Низкий приоритет.
+Отложено.
 
 ### 24.32. Поддержка GRPC {#podderzhka-grpc}
 
@@ -1925,6 +2008,7 @@ Amos Bird, но его решение слишком громоздкое и п
 Рассматривается вариант - поддержка GRPC в ClickHouse. Здесь есть неочевидные моменты, такие как - эффективная передача массивов данных в column-oriented формате - насколько удобно будет обернуть это в GRPC.
 
 Задача в работе, есть pull request. [#10136](https://github.com/ClickHouse/ClickHouse/pull/10136)
+Upd. Задачу взял в работу Виталий Баранов.
 
 
 ## 25. DevRel {#devrel}
@@ -1970,17 +2054,18 @@ Amos Bird, но его решение слишком громоздкое и п
 
 Екатерина - организация. Upd. Проведено два онлайн митапа на русском и два на английском.
 
-### 25.11. Митапы зарубежные: восток США (Нью Йорк, возможно Raleigh), возможно северо-запад (Сиэтл), Китай (Пекин снова, возможно митап для разработчиков или хакатон), Лондон {#mitapy-zarubezhnye-vostok-ssha-niu-iork-vozmozhno-raleigh-vozmozhno-severo-zapad-sietl-kitai-pekin-snova-vozmozhno-mitap-dlia-razrabotchikov-ili-khakaton-london}
+### 25.11. + Митапы зарубежные: восток США (Нью Йорк, возможно Raleigh), возможно северо-запад (Сиэтл), Китай (Пекин снова, возможно митап для разработчиков или хакатон), Лондон {#mitapy-zarubezhnye-vostok-ssha-niu-iork-vozmozhno-raleigh-vozmozhno-severo-zapad-sietl-kitai-pekin-snova-vozmozhno-mitap-dlia-razrabotchikov-ili-khakaton-london}
 
-[Иван Блинков](https://github.com/blinkov/) - организация. Две штуки в США запланированы. Upd. Два митапа в США и один в Европе проведены.
+[Иван Блинков](https://github.com/blinkov/) - организация. Две штуки в США запланированы. Upd. Два митапа в США и один в Европе проведены. Upd. Все остальные перенесены в онлайн.
 
 ### 25.12. Статья «научная» - про устройство хранения данных и индексов или whitepaper по архитектуре. Есть вариант подать на VLDB {#statia-nauchnaia-pro-ustroistvo-khraneniia-dannykh-i-indeksov-ili-whitepaper-po-arkhitekture-est-variant-podat-na-vldb}
 
 Низкий приоритет. Алексей Миловидов.
 
-### 25.13. Участие во всех мероприятиях Яндекса, которые связаны с разработкой бэкенда, C++ разработкой или с базами данных, возможно участие в DevRel мероприятиях {#uchastie-vo-vsekh-meropriiatiiakh-iandeksa-kotorye-sviazany-s-razrabotkoi-bekenda-c-razrabotkoi-ili-s-bazami-dannykh-vozmozhno-uchastie-v-devrel-meropriiatiiakh}
+### 25.13. + Участие во всех мероприятиях Яндекса, которые связаны с разработкой бэкенда, C++ разработкой или с базами данных, возможно участие в DevRel мероприятиях {#uchastie-vo-vsekh-meropriiatiiakh-iandeksa-kotorye-sviazany-s-razrabotkoi-bekenda-c-razrabotkoi-ili-s-bazami-dannykh-vozmozhno-uchastie-v-devrel-meropriiatiiakh}
 
-Алексей Миловидов и все подготовленные докладчики
+Алексей Миловидов и все подготовленные докладчики.
+Upd. Участвуем.
 
 ### 25.14. Конференции в России: все HighLoad, возможно CodeFest, DUMP или UWDC, возможно C++ Russia {#konferentsii-v-rossii-vse-highload-vozmozhno-codefest-dump-ili-uwdc-vozmozhno-c-russia}
 
@@ -1988,6 +2073,7 @@ Amos Bird, но его решение слишком громоздкое и п
 Upd. Есть Saint HighLoad online.
 Upd. Есть C++ Russia.
 CodeFest, DUMP, UWDC отменились.
+Upd. Добавились Highload Fwdays, Матемаркетинг.
 
 ### 25.15. Конференции зарубежные: Percona, DataOps, попытка попасть на более крупные {#konferentsii-zarubezhnye-percona-dataops-popytka-popast-na-bolee-krupnye}
 
@@ -2009,16 +2095,18 @@ DataOps отменилась.
 Требуется проработать вопрос безопасности и изоляции инстансов (поднятие в контейнерах с ограничениями по сети), подключение тестовых датасетов с помощью copy-on-write файловой системы; органичения ресурсов.
 
 Есть минимальный прототип. Сделал Илья Яцишин. Этот прототип не позволяет делиться ссылками на результаты запросов.
+Upd. На финальной стадии инструмент для экспериментирования с разными версиями ClickHouse.
 
 ### 25.17. Взаимодействие с ВУЗами: ВШЭ, УрФУ, ICT Beijing {#vzaimodeistvie-s-vuzami-vshe-urfu-ict-beijing}
 
 Алексей Миловидов и вся группа разработки.
 Благодаря Robert Hodges добавлен CMU.
 Upd. Взаимодействие с ВШЭ 2019/2020 успешно выполнено.
+Upd. Идёт подготовка к 2020/2021.
 
 ### 25.18. - Лекция в ШАД {#lektsiia-v-shad}
 
-Алексей Миловидов
+Алексей Миловидов.
 
 ### 25.19. - Участие в курсе разработки на C++ в ШАД {#uchastie-v-kurse-razrabotki-na-c-v-shad}
 
@@ -2029,6 +2117,8 @@ Upd. Взаимодействие с ВШЭ 2019/2020 успешно выпол
 Существуют мало известные специализированные СУБД, способные конкурировать с ClickHouse по скорости обработки некоторых классов запросов. Пример: `TDEngine` и `DolphinDB`, `VictoriaMetrics`, а также `Apache Doris` и `LocustDB`. Предлагается изучить и классифицировать архитектурные особенности этих систем - их особенности и преимущества. Установить эти системы, загрузить тестовые данные, изучить производительность. Проанализировать, за счёт чего достигаются преимущества.
 
 Upd. Есть поползновения с TDEngine.
+Upd. Добавили OmniSci, обновили MonetDB.
+Также посмотрели QuestDB и VectorSQL (они не работают).
 
 ### 25.21. Повторное награждение контрибьюторов в Китае {#povtornoe-nagrazhdenie-kontribiutorov-v-kitae}
 
@@ -2038,6 +2128,7 @@ Upd. Ждём снятия ограничений и восстановлени
 
 [Иван Блинков](https://github.com/blinkov/) - организация. Провёл мероприятие для турецкой компании.
 Upd. On-site заменяется на Online.
+Upd. Проведены консультации для нескольких секретных компаний.
 
 ### 25.23. Новый мерч для ClickHouse {#novyi-merch-dlia-clickhouse}
 

docs/tools/build.py

@@ -28,6 +28,7 @@ import test
 import util
 import website
 
+from cmake_in_clickhouse_generator import generate_cmake_flags_files
 
 class ClickHouseMarkdown(markdown.extensions.Extension):
     class ClickHousePreprocessor(markdown.util.Processor):
@@ -184,6 +185,8 @@ def build(args):
         test.test_templates(args.website_dir)
 
     if not args.skip_docs:
+        generate_cmake_flags_files(os.path.join(os.path.dirname(__file__), '..', '..'))
+
         build_docs(args)
         from github import build_releases
         build_releases(args, build_docs)
@@ -200,6 +203,7 @@ def build(args):
 if __name__ == '__main__':
     os.chdir(os.path.join(os.path.dirname(__file__), '..'))
     website_dir = os.path.join('..', 'website')
+
     arg_parser = argparse.ArgumentParser()
     arg_parser.add_argument('--lang', default='en,es,fr,ru,zh,ja,tr,fa')
     arg_parser.add_argument('--blog-lang', default='en,ru')

docs/tools/cmake_in_clickhouse_generator.py

@@ -0,0 +1,152 @@
+import re
+import os
+from typing import TextIO, List, Tuple, Optional, Dict
+
+# name, default value, description
+Entity = Tuple[str, str, str]
+
+# https://regex101.com/r/R6iogw/12
+cmake_option_regex: str = r"^\s*option\s*\(([A-Z_0-9${}]+)\s*(?:\"((?:.|\n)*?)\")?\s*(.*)?\).*$"
+
+ch_master_url: str = "https://github.com/clickhouse/clickhouse/blob/master/"
+
+name_str: str = "<a name=\"{anchor}\"></a>[`{name}`](" + ch_master_url + "{path}#L{line})"
+default_anchor_str: str = "[`{name}`](#{anchor})"
+
+comment_var_regex: str = r"\${(.+)}"
+comment_var_replace: str = "`\\1`"
+
+table_header: str = """
+| Name | Default value | Description | Comment |
+|------|---------------|-------------|---------|
+"""
+
+# Needed to detect conditional variables (those which are defined twice)
+# name -> (path, values)
+entities: Dict[str, Tuple[str, str]] = {}
+
+
+def make_anchor(t: str) -> str:
+    return "".join(["-" if i == "_" else i.lower() for i in t if i.isalpha() or i == "_"])
+
+def process_comment(comment: str) -> str:
+    return re.sub(comment_var_regex, comment_var_replace, comment, flags=re.MULTILINE)
+
+def build_entity(path: str, entity: Entity, line_comment: Tuple[int, str]) -> None:
+    (line, comment) = line_comment
+    (name, description, default) = entity
+
+    if name in entities:
+        return
+
+    # cannot escape the { in macro option description -> invalid AMP html
+    # Skipping "USE_INTERNAL_${LIB_NAME_UC}_LIBRARY"
+    if "LIB_NAME_UC" in name:
+        return
+
+    if len(default) == 0:
+        formatted_default: str = "`OFF`"
+    elif default[0] == "$":
+        formatted_default: str = "`{}`".format(default[2:-1])
+    else:
+        formatted_default: str = "`" + default + "`"
+
+    formatted_name: str = name_str.format(
+        anchor=make_anchor(name),
+        name=name,
+        path=path,
+        line=line if line > 0 else 1)
+
+    formatted_description: str = "".join(description.split("\n"))
+
+    formatted_comment: str = process_comment(comment)
+
+    formatted_entity: str = "| {} | {} | {} | {} |".format(
+        formatted_name, formatted_default, formatted_description, formatted_comment)
+
+    entities[name] = path, formatted_entity
+
+def process_file(root_path: str, input_name: str) -> None:
+    with open(os.path.join(root_path, input_name), 'r') as cmake_file:
+        contents: str = cmake_file.read()
+
+        def get_line_and_comment(target: str) -> Tuple[int, str]:
+            contents_list: List[str] = contents.split("\n")
+            comment: str = ""
+
+            for n, line in enumerate(contents_list):
+                if line.find(target) == -1:
+                    continue
+
+                for maybe_comment_line in contents_list[n - 1::-1]:
+                    if not re.match("\s*#\s*", maybe_comment_line):
+                        break
+
+                    comment = re.sub("\s*#\s*", "", maybe_comment_line) + " " + comment
+
+                return n, comment
+
+        matches: Optional[List[Entity]] = re.findall(cmake_option_regex, contents, re.MULTILINE)
+
+        if matches:
+            for entity in matches:
+                build_entity(os.path.join(root_path[6:], input_name), entity, get_line_and_comment(entity[0]))
+
+def process_folder(root_path:str, name: str) -> None:
+    for root, _, files in os.walk(os.path.join(root_path, name)):
+        for f in files:
+            if f == "CMakeLists.txt" or ".cmake" in f:
+                process_file(root, f)
+
+def generate_cmake_flags_files(root_path: str) -> None:
+    output_file_name: str = os.path.join(root_path, "docs/en/development/cmake-in-clickhouse.md")
+    header_file_name: str = os.path.join(root_path, "docs/_includes/cmake_in_clickhouse_header.md")
+    footer_file_name: str = os.path.join(root_path, "docs/_includes/cmake_in_clickhouse_footer.md")
+
+    process_file(root_path, "CMakeLists.txt")
+    process_file(root_path, "programs/CMakeLists.txt")
+
+    process_folder(root_path, "base")
+    process_folder(root_path, "cmake")
+    process_folder(root_path, "src")
+
+    with open(output_file_name, "w") as f:
+        with open(header_file_name, "r") as header:
+            f.write(header.read())
+
+        sorted_keys: List[str] = sorted(entities.keys())
+        ignored_keys: List[str] = []
+
+        f.write("### ClickHouse modes\n" + table_header)
+
+        for k in sorted_keys:
+            if k.startswith("ENABLE_CLICKHOUSE_"):
+                f.write(entities[k][1] + "\n")
+                ignored_keys.append(k)
+
+        f.write("\n### External libraries\nNote that ClickHouse uses forks of these libraries, see https://github.com/ClickHouse-Extras.\n" +
+            table_header)
+
+        for k in sorted_keys:
+            if k.startswith("ENABLE_") and ".cmake" in entities[k][0]:
+                f.write(entities[k][1] + "\n")
+                ignored_keys.append(k)
+
+        f.write("\n### External libraries system/bundled mode\n" + table_header)
+
+        for k in sorted_keys:
+            if k.startswith("USE_INTERNAL_"):
+                f.write(entities[k][1] + "\n")
+                ignored_keys.append(k)
+
+        f.write("\n### Other flags\n" + table_header)
+
+        for k in sorted(set(sorted_keys).difference(set(ignored_keys))):
+            f.write(entities[k][1] + "\n")
+
+        with open(footer_file_name, "r") as footer:
+            f.write(footer.read())
+
+
+if __name__ == '__main__':
+    generate_cmake_flags_files("../../")

docs/tools/translate/translate.py

@@ -49,13 +49,14 @@ def translate_impl(text, target_language=None):
 
 
 def translate(text, target_language=None):
-    result = []
+    return "".join(
-    for part in re.split(curly_braces_re, text):
+        [
-        if part.startswith('{') and part.endswith('}'):
+            part
-            result.append(part)
+            if part.startswith("{") and part.endswith("}")
-        else:
+            else translate_impl(part, target_language=target_language)
-            result.append(translate_impl(part, target_language=target_language))
+            for part in re.split(curly_braces_re, text)
-    return ''.join(result)
+        ]
+    )
 
 
 def translate_toc(root, lang):

docs/zh/sql-reference/operators/in.md

@@ -3,7 +3,7 @@ machine_translated: true
 machine_translated_rev: 5decc73b5dc60054f19087d3690c4eb99446a6c3
 ---
 
-# 在运营商 {#select-in-operators}
+# IN 操作符 {#select-in-operators}
 
 该 `IN`, `NOT IN`, `GLOBAL IN`，和 `GLOBAL NOT IN` 运算符是单独复盖的，因为它们的功能相当丰富。
 

programs/CMakeLists.txt

@@ -2,31 +2,49 @@ if (USE_CLANG_TIDY)
     set (CMAKE_CXX_CLANG_TIDY "${CLANG_TIDY_PATH}")
 endif ()
 
-# 'clickhouse' binary is a multi purpose tool,
+# The `clickhouse` binary is a multi purpose tool that contains multiple execution modes (client, server, etc.),
-# that contain multiple execution modes (client, server, etc.)
+# each of them may be built and linked as a separate library.
-# each of them is built and linked as a separate library, defined below.
+# If you do not know what modes you need, turn this option OFF and enable SERVER and CLIENT only.
+option (ENABLE_CLICKHOUSE_ALL "Enable all ClickHouse modes by default" ON)
 
-option (ENABLE_CLICKHOUSE_ALL "Enable all tools" ON)
+option (ENABLE_CLICKHOUSE_SERVER "Server mode (main mode)" ${ENABLE_CLICKHOUSE_ALL})
-option (ENABLE_CLICKHOUSE_SERVER "Enable clickhouse-server" ${ENABLE_CLICKHOUSE_ALL})
+option (ENABLE_CLICKHOUSE_CLIENT "Client mode (interactive tui/shell that connects to the server)"
-option (ENABLE_CLICKHOUSE_CLIENT "Enable clickhouse-client" ${ENABLE_CLICKHOUSE_ALL})
+    ${ENABLE_CLICKHOUSE_ALL})
-option (ENABLE_CLICKHOUSE_LOCAL "Enable clickhouse-local" ${ENABLE_CLICKHOUSE_ALL})
+
-option (ENABLE_CLICKHOUSE_BENCHMARK "Enable clickhouse-benchmark" ${ENABLE_CLICKHOUSE_ALL})
+# https://clickhouse.tech/docs/en/operations/utilities/clickhouse-local/
-option (ENABLE_CLICKHOUSE_EXTRACT_FROM_CONFIG "Enable clickhouse-extract-from-config" ${ENABLE_CLICKHOUSE_ALL})
+option (ENABLE_CLICKHOUSE_LOCAL "Local files fast processing mode" ${ENABLE_CLICKHOUSE_ALL})
-option (ENABLE_CLICKHOUSE_COMPRESSOR "Enable clickhouse-compressor" ${ENABLE_CLICKHOUSE_ALL})
+
-option (ENABLE_CLICKHOUSE_COPIER "Enable clickhouse-copier" ${ENABLE_CLICKHOUSE_ALL})
+# https://clickhouse.tech/docs/en/operations/utilities/clickhouse-benchmark/
-option (ENABLE_CLICKHOUSE_FORMAT "Enable clickhouse-format" ${ENABLE_CLICKHOUSE_ALL})
+option (ENABLE_CLICKHOUSE_BENCHMARK "Queries benchmarking mode" ${ENABLE_CLICKHOUSE_ALL})
-option (ENABLE_CLICKHOUSE_OBFUSCATOR "Enable clickhouse-obfuscator" ${ENABLE_CLICKHOUSE_ALL})
+
-option (ENABLE_CLICKHOUSE_GIT_IMPORT "Enable clickhouse-git-import" ${ENABLE_CLICKHOUSE_ALL})
+option (ENABLE_CLICKHOUSE_EXTRACT_FROM_CONFIG "Configs processor (extract values etc.)" ${ENABLE_CLICKHOUSE_ALL})
-option (ENABLE_CLICKHOUSE_ODBC_BRIDGE "Enable clickhouse-odbc-bridge" ${ENABLE_CLICKHOUSE_ALL})
+
+# https://clickhouse.tech/docs/en/operations/utilities/clickhouse-compressor/
+option (ENABLE_CLICKHOUSE_COMPRESSOR "Data compressor and decompressor" ${ENABLE_CLICKHOUSE_ALL})
+
+# https://clickhouse.tech/docs/en/operations/utilities/clickhouse-copier/
+option (ENABLE_CLICKHOUSE_COPIER "Inter-cluster data copying mode" ${ENABLE_CLICKHOUSE_ALL})
+
+option (ENABLE_CLICKHOUSE_FORMAT "Queries pretty-printer and formatter with syntax highlighting"
+    ${ENABLE_CLICKHOUSE_ALL})
+
+# https://clickhouse.tech/docs/en/operations/utilities/clickhouse-obfuscator/
+option (ENABLE_CLICKHOUSE_OBFUSCATOR "Table data obfuscator (convert real data to benchmark-ready one)"
+    ${ENABLE_CLICKHOUSE_ALL})
+
+# https://clickhouse.tech/docs/en/operations/utilities/odbc-bridge/
+option (ENABLE_CLICKHOUSE_ODBC_BRIDGE "HTTP-server working like a proxy to ODBC driver"
+    ${ENABLE_CLICKHOUSE_ALL})
 
 if (CLICKHOUSE_SPLIT_BINARY)
-    option (ENABLE_CLICKHOUSE_INSTALL "Enable clickhouse-install" OFF)
+    option(ENABLE_CLICKHOUSE_INSTALL "Install ClickHouse without .deb/.rpm/.tgz packages (having the binary only)" OFF)
 else ()
-    option (ENABLE_CLICKHOUSE_INSTALL "Enable clickhouse-install" ${ENABLE_CLICKHOUSE_ALL})
+    option(ENABLE_CLICKHOUSE_INSTALL "Install ClickHouse without .deb/.rpm/.tgz packages (having the binary only)"
+        ${ENABLE_CLICKHOUSE_ALL})
 endif ()
 
 if(NOT (MAKE_STATIC_LIBRARIES OR SPLIT_SHARED_LIBRARIES))
-    set(CLICKHOUSE_ONE_SHARED 1)
+    set(CLICKHOUSE_ONE_SHARED ON)
 endif()
 
 configure_file (config_tools.h.in ${ConfigIncludePath}/config_tools.h)

programs/client/Client.cpp

@@ -1167,6 +1167,9 @@ private:
                             dump_of_cloned_ast.str().c_str());
                         fprintf(stderr, "dump after fuzz:\n");
                         fuzz_base->dumpTree(std::cerr);
+
+                        fmt::print(stderr, "IAST::clone() is broken for some AST node. This is a bug. The original AST ('dump before fuzz') and its cloned copy ('dump of cloned AST') refer to the same nodes, which must never happen. This means that their parent node doesn't implement clone() correctly.");
+
                         assert(false);
                     }
 

programs/client/Suggest.cpp

@@ -80,7 +80,7 @@ Suggest::Suggest()
              "WITH",         "TOTALS",   "HAVING", "ORDER",     "COLLATE",  "LIMIT",       "UNION",    "AND",         "OR",      "ASC",
              "IN",           "KILL",     "QUERY",  "SYNC",      "ASYNC",    "TEST",        "BETWEEN",  "TRUNCATE",    "USER",    "ROLE",
              "PROFILE",      "QUOTA",    "POLICY", "ROW",       "GRANT",    "REVOKE",      "OPTION",   "ADMIN",       "EXCEPT",  "REPLACE",
-             "IDENTIFIED",   "HOST",     "NAME",   "READONLY",  "WRITABLE", "PERMISSIVE",  "FOR",      "RESTRICTIVE", "FOR",     "RANDOMIZED",
+             "IDENTIFIED",   "HOST",     "NAME",   "READONLY",  "WRITABLE", "PERMISSIVE",  "FOR",      "RESTRICTIVE", "RANDOMIZED",
              "INTERVAL",     "LIMITS",   "ONLY",   "TRACKING",  "IP",       "REGEXP",      "ILIKE"};
 }
 

programs/copier/ClusterCopier.cpp

@@ -1477,7 +1477,9 @@ TaskStatus ClusterCopier::processPartitionPieceTaskImpl(
     {
         auto create_query_push_ast = rewriteCreateQueryStorage(task_shard.current_pull_table_create_query,
                                                                task_table.table_push, task_table.engine_push_ast);
-        create_query_push_ast->as<ASTCreateQuery &>().if_not_exists = true;
+        auto & create = create_query_push_ast->as<ASTCreateQuery &>();
+        create.if_not_exists = true;
+        InterpreterCreateQuery::prepareOnClusterQuery(create, context, task_table.cluster_push_name);
         String query = queryToString(create_query_push_ast);
 
         LOG_DEBUG(log, "Create destination tables. Query: {}", query);

programs/copier/ClusterCopierApp.cpp

@@ -105,7 +105,7 @@ void ClusterCopierApp::mainImpl()
     ThreadStatus thread_status;
 
     auto * log = &logger();
-    LOG_INFO(log, "Starting clickhouse-copier (id {}, host_id {}, path {}, revision {})", process_id, host_id, process_path, ClickHouseRevision::get());
+    LOG_INFO(log, "Starting clickhouse-copier (id {}, host_id {}, path {}, revision {})", process_id, host_id, process_path, ClickHouseRevision::getVersionRevision());
 
     SharedContextHolder shared_context = Context::createShared();
     auto context = std::make_unique<Context>(Context::createGlobal(shared_context.get()));

programs/copier/Internals.cpp

@@ -215,31 +215,20 @@ Names extractPrimaryKeyColumnNames(const ASTPtr & storage_ast)
     return primary_key_columns;
 }
 
-String extractReplicatedTableZookeeperPath(const ASTPtr & storage_ast)
+bool isReplicatedTableEngine(const ASTPtr & storage_ast)
 {
-    String storage_str = queryToString(storage_ast);
-
     const auto & storage = storage_ast->as<ASTStorage &>();
     const auto & engine = storage.engine->as<ASTFunction &>();
 
     if (!endsWith(engine.name, "MergeTree"))
     {
+        String storage_str = queryToString(storage_ast);
         throw Exception(
                 "Unsupported engine was specified in " + storage_str + ", only *MergeTree engines are supported",
                 ErrorCodes::BAD_ARGUMENTS);
     }
 
-    if (!startsWith(engine.name, "Replicated"))
+    return startsWith(engine.name, "Replicated");
-    {
-        return "";
-    }
-
-    auto replicated_table_arguments = engine.arguments->children;
-
-    auto zk_table_path_ast = replicated_table_arguments[0]->as<ASTLiteral &>();
-    auto zk_table_path_string = zk_table_path_ast.value.safeGet<String>();
-
-    return zk_table_path_string;
 }
 
 ShardPriority getReplicasPriority(const Cluster::Addresses & replicas, const std::string & local_hostname, UInt8 random)

programs/copier/Internals.h

@@ -200,7 +200,7 @@ ASTPtr extractOrderBy(const ASTPtr & storage_ast);
 
 Names extractPrimaryKeyColumnNames(const ASTPtr & storage_ast);
 
-String extractReplicatedTableZookeeperPath(const ASTPtr & storage_ast);
+bool isReplicatedTableEngine(const ASTPtr & storage_ast);
 
 ShardPriority getReplicasPriority(const Cluster::Addresses & replicas, const std::string & local_hostname, UInt8 random);
 

programs/copier/TaskTableAndShard.h

@@ -48,7 +48,7 @@ struct TaskTable
     String getCertainPartitionPieceTaskStatusPath(const String & partition_name, const size_t piece_number) const;
 
 
-    bool isReplicatedTable() const { return engine_push_zk_path != ""; }
+    bool isReplicatedTable() const { return is_replicated_table; }
 
     /// Partitions will be split into number-of-splits pieces.
     /// Each piece will be copied independently. (10 by default)
@@ -78,6 +78,7 @@ struct TaskTable
 
     /// First argument of Replicated...MergeTree()
     String engine_push_zk_path;
+    bool is_replicated_table;
 
     ASTPtr rewriteReplicatedCreateQueryToPlain();
 
@@ -269,7 +270,7 @@ inline TaskTable::TaskTable(TaskCluster & parent, const Poco::Util::AbstractConf
         engine_push_ast = parseQuery(parser_storage, engine_push_str, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH);
         engine_push_partition_key_ast = extractPartitionKey(engine_push_ast);
         primary_key_comma_separated = Nested::createCommaSeparatedStringFrom(extractPrimaryKeyColumnNames(engine_push_ast));
-        engine_push_zk_path = extractReplicatedTableZookeeperPath(engine_push_ast);
+        is_replicated_table = isReplicatedTableEngine(engine_push_ast);
     }
 
     sharding_key_str = config.getString(table_prefix + "sharding_key");
@@ -372,15 +373,18 @@ inline ASTPtr TaskTable::rewriteReplicatedCreateQueryToPlain()
     auto & new_storage_ast = prev_engine_push_ast->as<ASTStorage &>();
     auto & new_engine_ast = new_storage_ast.engine->as<ASTFunction &>();
 
-    auto & replicated_table_arguments = new_engine_ast.arguments->children;
+    /// Remove "Replicated" from name
-
-    /// Delete first two arguments of Replicated...MergeTree() table.
-    replicated_table_arguments.erase(replicated_table_arguments.begin());
-    replicated_table_arguments.erase(replicated_table_arguments.begin());
-
-    /// Remove replicated from name
     new_engine_ast.name = new_engine_ast.name.substr(10);
 
+    if (new_engine_ast.arguments)
+    {
+        auto & replicated_table_arguments = new_engine_ast.arguments->children;
+
+        /// Delete first two arguments of Replicated...MergeTree() table.
+        replicated_table_arguments.erase(replicated_table_arguments.begin());
+        replicated_table_arguments.erase(replicated_table_arguments.begin());
+    }
+
     return new_storage_ast.clone();
 }
 

programs/format/CMakeLists.txt

@@ -5,6 +5,9 @@ set (CLICKHOUSE_FORMAT_LINK
         boost::program_options
         clickhouse_common_io
         clickhouse_parsers
+        clickhouse_functions
+        clickhouse_aggregate_functions
+        clickhouse_table_functions
         dbms
 )
 

programs/format/Format.cpp

@@ -1,13 +1,29 @@
 #include <iostream>
+#include <string_view>
+#include <functional>
 #include <boost/program_options.hpp>
 
 #include <IO/ReadBufferFromFileDescriptor.h>
 #include <IO/ReadHelpers.h>
+#include <IO/WriteBufferFromFileDescriptor.h>
 #include <Parsers/ParserQuery.h>
 #include <Parsers/parseQuery.h>
 #include <Parsers/formatAST.h>
+#include <Parsers/obfuscateQueries.h>
 #include <Common/TerminalSize.h>
 
+#include <Interpreters/Context.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/registerFunctions.h>
+#include <AggregateFunctions/AggregateFunctionFactory.h>
+#include <AggregateFunctions/registerAggregateFunctions.h>
+#include <TableFunctions/TableFunctionFactory.h>
+#include <TableFunctions/registerTableFunctions.h>
+#include <Storages/StorageFactory.h>
+#include <Storages/registerStorages.h>
+#include <DataTypes/DataTypeFactory.h>
+
+
 #pragma GCC diagnostic ignored "-Wunused-function"
 #pragma GCC diagnostic ignored "-Wmissing-declarations"
 
@@ -22,6 +38,8 @@ int mainEntryClickHouseFormat(int argc, char ** argv)
         ("oneline", "format in single line")
         ("quiet,q", "just check syntax, no output on success")
         ("multiquery,n", "allow multiple queries in the same file")
+        ("obfuscate", "obfuscate instead of formatting")
+        ("seed", po::value<std::string>(), "seed (arbitrary string) that determines the result of obfuscation")
     ;
 
     boost::program_options::variables_map options;
@@ -40,10 +58,17 @@ int mainEntryClickHouseFormat(int argc, char ** argv)
         bool oneline = options.count("oneline");
         bool quiet = options.count("quiet");
         bool multiple = options.count("multiquery");
+        bool obfuscate = options.count("obfuscate");
 
-        if (quiet && (hilite || oneline))
+        if (quiet && (hilite || oneline || obfuscate))
         {
-            std::cerr << "Options 'hilite' or 'oneline' have no sense in 'quiet' mode." << std::endl;
+            std::cerr << "Options 'hilite' or 'oneline' or 'obfuscate' have no sense in 'quiet' mode." << std::endl;
+            return 2;
+        }
+
+        if (obfuscate && (hilite || oneline || quiet))
+        {
+            std::cerr << "Options 'hilite' or 'oneline' or 'quiet' have no sense in 'obfuscate' mode." << std::endl;
             return 2;
         }
 
@@ -51,21 +76,66 @@ int mainEntryClickHouseFormat(int argc, char ** argv)
         ReadBufferFromFileDescriptor in(STDIN_FILENO);
         readStringUntilEOF(query, in);
 
-        const char * pos = query.data();
+        if (obfuscate)
-        const char * end = pos + query.size();
-
-        ParserQuery parser(end);
-        do
         {
-            ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH);
+            WordMap obfuscated_words_map;
-            if (!quiet)
+            WordSet used_nouns;
+            SipHash hash_func;
+
+            if (options.count("seed"))
             {
-                formatAST(*res, std::cout, hilite, oneline);
+                std::string seed;
-                if (multiple)
+                hash_func.update(options["seed"].as<std::string>());
-                    std::cout << "\n;\n";
-                std::cout << std::endl;
             }
-        } while (multiple && pos != end);
+
+            SharedContextHolder shared_context = Context::createShared();
+            Context context = Context::createGlobal(shared_context.get());
+            context.makeGlobalContext();
+
+            registerFunctions();
+            registerAggregateFunctions();
+            registerTableFunctions();
+            registerStorages();
+
+            std::unordered_set<std::string> additional_names;
+
+            auto all_known_storage_names = StorageFactory::instance().getAllRegisteredNames();
+            auto all_known_data_type_names = DataTypeFactory::instance().getAllRegisteredNames();
+
+            additional_names.insert(all_known_storage_names.begin(), all_known_storage_names.end());
+            additional_names.insert(all_known_data_type_names.begin(), all_known_data_type_names.end());
+
+            KnownIdentifierFunc is_known_identifier = [&](std::string_view name)
+            {
+                std::string what(name);
+
+                return FunctionFactory::instance().tryGet(what, context) != nullptr
+                    || AggregateFunctionFactory::instance().isAggregateFunctionName(what)
+                    || TableFunctionFactory::instance().isTableFunctionName(what)
+                    || additional_names.count(what);
+            };
+
+            WriteBufferFromFileDescriptor out(STDOUT_FILENO);
+            obfuscateQueries(query, out, obfuscated_words_map, used_nouns, hash_func, is_known_identifier);
+        }
+        else
+        {
+            const char * pos = query.data();
+            const char * end = pos + query.size();
+
+            ParserQuery parser(end);
+            do
+            {
+                ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH);
+                if (!quiet)
+                {
+                    formatAST(*res, std::cout, hilite, oneline);
+                    if (multiple)
+                        std::cout << "\n;\n";
+                    std::cout << std::endl;
+                }
+            } while (multiple && pos != end);
+        }
     }
     catch (...)
     {

programs/server/CMakeLists.txt

@@ -3,7 +3,7 @@ set(CLICKHOUSE_SERVER_SOURCES
     Server.cpp
 )
 
-if (OS_LINUX AND ARCH_AMD64)
+if (OS_LINUX)
     set (LINK_CONFIG_LIB INTERFACE "-Wl,${WHOLE_ARCHIVE} $<TARGET_FILE:clickhouse_server_configs> -Wl,${NO_WHOLE_ARCHIVE}")
 endif ()
 

programs/server/Server.cpp

@@ -273,7 +273,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
 #endif
 #endif
 
-    CurrentMetrics::set(CurrentMetrics::Revision, ClickHouseRevision::get());
+    CurrentMetrics::set(CurrentMetrics::Revision, ClickHouseRevision::getVersionRevision());
     CurrentMetrics::set(CurrentMetrics::VersionInteger, ClickHouseRevision::getVersionInteger());
 
     if (ThreadFuzzer::instance().isEffective())
@@ -297,6 +297,11 @@ int Server::main(const std::vector<std::string> & /*args*/)
     global_context->makeGlobalContext();
     global_context->setApplicationType(Context::ApplicationType::SERVER);
 
+    // Initialize global thread pool. Do it before we fetch configs from zookeeper
+    // nodes (`from_zk`), because ZooKeeper interface uses the pool. We will
+    // ignore `max_thread_pool_size` in configs we fetch from ZK, but oh well.
+    GlobalThreadPool::initialize(config().getUInt("max_thread_pool_size", 10000));
+
     bool has_zookeeper = config().has("zookeeper");
 
     zkutil::ZooKeeperNodeCache main_config_zk_node_cache([&] { return global_context->getZooKeeper(); });
@@ -334,16 +339,23 @@ int Server::main(const std::vector<std::string> & /*args*/)
         {
             if (hasLinuxCapability(CAP_IPC_LOCK))
             {
-                /// Get the memory area with (current) code segment.
+                try
-                /// It's better to lock only the code segment instead of calling "mlockall",
+                {
-                /// because otherwise debug info will be also locked in memory, and it can be huge.
+                    /// Get the memory area with (current) code segment.
-                auto [addr, len] = getMappedArea(reinterpret_cast<void *>(mainEntryClickHouseServer));
+                    /// It's better to lock only the code segment instead of calling "mlockall",
+                    /// because otherwise debug info will be also locked in memory, and it can be huge.
+                    auto [addr, len] = getMappedArea(reinterpret_cast<void *>(mainEntryClickHouseServer));
 
-                LOG_TRACE(log, "Will do mlock to prevent executable memory from being paged out. It may take a few seconds.");
+                    LOG_TRACE(log, "Will do mlock to prevent executable memory from being paged out. It may take a few seconds.");
-                if (0 != mlock(addr, len))
+                    if (0 != mlock(addr, len))
-                    LOG_WARNING(log, "Failed mlock: {}", errnoToString(ErrorCodes::SYSTEM_ERROR));
+                        LOG_WARNING(log, "Failed mlock: {}", errnoToString(ErrorCodes::SYSTEM_ERROR));
-                else
+                    else
-                    LOG_TRACE(log, "The memory map of clickhouse executable has been mlock'ed, total {}", ReadableSize(len));
+                        LOG_TRACE(log, "The memory map of clickhouse executable has been mlock'ed, total {}", ReadableSize(len));
+                }
+                catch (...)
+                {
+                    LOG_WARNING(log, "Cannot mlock: {}", getCurrentExceptionMessage(false));
+                }
             }
             else
             {
@@ -436,9 +448,6 @@ int Server::main(const std::vector<std::string> & /*args*/)
     DateLUT::instance();
     LOG_TRACE(log, "Initialized DateLUT with time zone '{}'.", DateLUT::instance().getTimeZone());
 
-    /// Initialize global thread pool
-    GlobalThreadPool::initialize(config().getUInt("max_thread_pool_size", 10000));
-
     /// Storage with temporary data for processing of heavy queries.
     {
         std::string tmp_path = config().getString("tmp_path", path + "tmp/");
@@ -662,6 +671,10 @@ int Server::main(const std::vector<std::string> & /*args*/)
     total_memory_tracker.setDescription("(total)");
     total_memory_tracker.setMetric(CurrentMetrics::MemoryTracking);
 
+    /// Set current database name before loading tables and databases because
+    /// system logs may copy global context.
+    global_context->setCurrentDatabaseNameInGlobalContext(default_database);
+
     LOG_INFO(log, "Loading metadata from {}", path);
 
     try
@@ -669,11 +682,14 @@ int Server::main(const std::vector<std::string> & /*args*/)
         loadMetadataSystem(*global_context);
         /// After attaching system databases we can initialize system log.
         global_context->initializeSystemLogs();
+        auto & database_catalog = DatabaseCatalog::instance();
         /// After the system database is created, attach virtual system tables (in addition to query_log and part_log)
-        attachSystemTablesServer(*DatabaseCatalog::instance().getSystemDatabase(), has_zookeeper);
+        attachSystemTablesServer(*database_catalog.getSystemDatabase(), has_zookeeper);
         /// Then, load remaining databases
         loadMetadata(*global_context, default_database);
-        DatabaseCatalog::instance().loadDatabases();
+        database_catalog.loadDatabases();
+        /// After loading validate that default database exists
+        database_catalog.assertDatabaseExists(default_database);
     }
     catch (...)
     {
@@ -736,8 +752,6 @@ int Server::main(const std::vector<std::string> & /*args*/)
         LOG_INFO(log, "Query Profiler and TraceCollector are disabled because they require PHDR cache to be created"
             " (otherwise the function 'dl_iterate_phdr' is not lock free and not async-signal safe).");
 
-    global_context->setCurrentDatabase(default_database);
-
     if (has_zookeeper && config().has("distributed_ddl"))
     {
         /// DDL worker should be started after all tables were loaded

release

@@ -95,9 +95,9 @@ then
         exit 3
     fi
 
-    export DEB_CC=${DEB_CC=clang-6.0}
+    export DEB_CC=${DEB_CC=clang-10}
-    export DEB_CXX=${DEB_CXX=clang++-6.0}
+    export DEB_CXX=${DEB_CXX=clang++-10}
-    EXTRAPACKAGES="$EXTRAPACKAGES clang-6.0 lld-6.0"
+    EXTRAPACKAGES="$EXTRAPACKAGES clang-10 lld-10"
 elif [[ $BUILD_TYPE == 'valgrind' ]]; then
     MALLOC_OPTS="-DENABLE_TCMALLOC=0 -DENABLE_JEMALLOC=0"
     VERSION_POSTFIX+="+valgrind"
@@ -118,8 +118,8 @@ echo -e "\nCurrent version is $VERSION_STRING"
 if [ -z "$NO_BUILD" ] ; then
     gen_changelog "$VERSION_STRING" "" "$AUTHOR" ""
     if [ -z "$USE_PBUILDER" ] ; then
-        DEB_CC=${DEB_CC:=`which gcc-9 gcc-8 gcc | head -n1`}
+        DEB_CC=${DEB_CC:=`which gcc-10 gcc-9 gcc | head -n1`}
-        DEB_CXX=${DEB_CXX:=`which g++-9 g++-8 g++ | head -n1`}
+        DEB_CXX=${DEB_CXX:=`which gcc-10 g++-9 g++ | head -n1`}
         # Build (only binary packages).
         debuild --preserve-env -e PATH \
         -e DEB_CC=$DEB_CC -e DEB_CXX=$DEB_CXX -e CMAKE_FLAGS="$CMAKE_FLAGS" \

src/Access/UsersConfigAccessStorage.cpp

@@ -192,7 +192,7 @@ namespace
     }
 
 
-    std::vector<AccessEntityPtr> parseUsers(const Poco::Util::AbstractConfiguration & config, Poco::Logger * log)
+    std::vector<AccessEntityPtr> parseUsers(const Poco::Util::AbstractConfiguration & config)
     {
         Poco::Util::AbstractConfiguration::Keys user_names;
         config.keys("users", user_names);
@@ -200,16 +200,8 @@ namespace
         std::vector<AccessEntityPtr> users;
         users.reserve(user_names.size());
         for (const auto & user_name : user_names)
-        {
+            users.push_back(parseUser(config, user_name));
-            try
+
-            {
-                users.push_back(parseUser(config, user_name));
-            }
-            catch (...)
-            {
-                tryLogCurrentException(log, "Could not parse user " + backQuote(user_name));
-            }
-        }
         return users;
     }
 
@@ -256,12 +248,11 @@ namespace
         }
 
         quota->to_roles.add(user_ids);
-
         return quota;
     }
 
 
-    std::vector<AccessEntityPtr> parseQuotas(const Poco::Util::AbstractConfiguration & config, Poco::Logger * log)
+    std::vector<AccessEntityPtr> parseQuotas(const Poco::Util::AbstractConfiguration & config)
     {
         Poco::Util::AbstractConfiguration::Keys user_names;
         config.keys("users", user_names);
@@ -278,76 +269,63 @@ namespace
         quotas.reserve(quota_names.size());
         for (const auto & quota_name : quota_names)
         {
-            try
+            auto it = quota_to_user_ids.find(quota_name);
-            {
+            const std::vector<UUID> & quota_users = (it != quota_to_user_ids.end()) ? std::move(it->second) : std::vector<UUID>{};
-                auto it = quota_to_user_ids.find(quota_name);
+            quotas.push_back(parseQuota(config, quota_name, quota_users));
-                const std::vector<UUID> & quota_users = (it != quota_to_user_ids.end()) ? std::move(it->second) : std::vector<UUID>{};
-                quotas.push_back(parseQuota(config, quota_name, quota_users));
-            }
-            catch (...)
-            {
-                tryLogCurrentException(log, "Could not parse quota " + backQuote(quota_name));
-            }
         }
         return quotas;
     }
 
 
-    std::vector<AccessEntityPtr> parseRowPolicies(const Poco::Util::AbstractConfiguration & config, Poco::Logger * log)
+    std::vector<AccessEntityPtr> parseRowPolicies(const Poco::Util::AbstractConfiguration & config)
     {
         std::map<std::pair<String /* database */, String /* table */>, std::unordered_map<String /* user */, String /* filter */>> all_filters_map;
+
         Poco::Util::AbstractConfiguration::Keys user_names;
+        config.keys("users", user_names);
 
-        try
+        for (const String & user_name : user_names)
         {
-            config.keys("users", user_names);
+            const String databases_config = "users." + user_name + ".databases";
-            for (const String & user_name : user_names)
+            if (config.has(databases_config))
             {
-                const String databases_config = "users." + user_name + ".databases";
+                Poco::Util::AbstractConfiguration::Keys database_keys;
-                if (config.has(databases_config))
+                config.keys(databases_config, database_keys);
+
+                /// Read tables within databases
+                for (const String & database_key : database_keys)
                 {
-                    Poco::Util::AbstractConfiguration::Keys database_keys;
+                    const String database_config = databases_config + "." + database_key;
-                    config.keys(databases_config, database_keys);
 
-                    /// Read tables within databases
+                    String database_name;
-                    for (const String & database_key : database_keys)
+                    if (((database_key == "database") || (database_key.starts_with("database["))) && config.has(database_config + "[@name]"))
+                        database_name = config.getString(database_config + "[@name]");
+                    else if (size_t bracket_pos = database_key.find('['); bracket_pos != std::string::npos)
+                        database_name = database_key.substr(0, bracket_pos);
+                    else
+                        database_name = database_key;
+
+                    Poco::Util::AbstractConfiguration::Keys table_keys;
+                    config.keys(database_config, table_keys);
+
+                    /// Read table properties
+                    for (const String & table_key : table_keys)
                     {
-                        const String database_config = databases_config + "." + database_key;
+                        String table_config = database_config + "." + table_key;
-
+                        String table_name;
-                        String database_name;
+                        if (((table_key == "table") || (table_key.starts_with("table["))) && config.has(table_config + "[@name]"))
-                        if (((database_key == "database") || (database_key.starts_with("database["))) && config.has(database_config + "[@name]"))
+                            table_name = config.getString(table_config + "[@name]");
-                            database_name = config.getString(database_config + "[@name]");
+                        else if (size_t bracket_pos = table_key.find('['); bracket_pos != std::string::npos)
-                        else if (size_t bracket_pos = database_key.find('['); bracket_pos != std::string::npos)
+                            table_name = table_key.substr(0, bracket_pos);
-                            database_name = database_key.substr(0, bracket_pos);
                         else
-                            database_name = database_key;
+                            table_name = table_key;
 
-                        Poco::Util::AbstractConfiguration::Keys table_keys;
+                        String filter_config = table_config + ".filter";
-                        config.keys(database_config, table_keys);
+                        all_filters_map[{database_name, table_name}][user_name] = config.getString(filter_config);
-
-                        /// Read table properties
-                        for (const String & table_key : table_keys)
-                        {
-                            String table_config = database_config + "." + table_key;
-                            String table_name;
-                            if (((table_key == "table") || (table_key.starts_with("table["))) && config.has(table_config + "[@name]"))
-                                table_name = config.getString(table_config + "[@name]");
-                            else if (size_t bracket_pos = table_key.find('['); bracket_pos != std::string::npos)
-                                table_name = table_key.substr(0, bracket_pos);
-                            else
-                                table_name = table_key;
-
-                            String filter_config = table_config + ".filter";
-                            all_filters_map[{database_name, table_name}][user_name] = config.getString(filter_config);
-                        }
                     }
                 }
             }
         }
-        catch (...)
-        {
-            tryLogCurrentException(log, "Could not parse row policies");
-        }
 
         std::vector<AccessEntityPtr> policies;
         for (auto & [database_and_table_name, user_to_filters] : all_filters_map)
@@ -450,23 +428,14 @@ namespace
 
     std::vector<AccessEntityPtr> parseSettingsProfiles(
         const Poco::Util::AbstractConfiguration & config,
-        const std::function<void(const std::string_view &)> & check_setting_name_function,
+        const std::function<void(const std::string_view &)> & check_setting_name_function)
-        Poco::Logger * log)
     {
         std::vector<AccessEntityPtr> profiles;
         Poco::Util::AbstractConfiguration::Keys profile_names;
         config.keys("profiles", profile_names);
         for (const auto & profile_name : profile_names)
-        {
+            profiles.push_back(parseSettingsProfile(config, profile_name, check_setting_name_function));
-            try
+
-            {
-                profiles.push_back(parseSettingsProfile(config, profile_name, check_setting_name_function));
-            }
-            catch (...)
-            {
-                tryLogCurrentException(log, "Could not parse profile " + backQuote(profile_name));
-            }
-        }
         return profiles;
     }
 }
@@ -520,13 +489,13 @@ void UsersConfigAccessStorage::setConfig(const Poco::Util::AbstractConfiguration
 void UsersConfigAccessStorage::parseFromConfig(const Poco::Util::AbstractConfiguration & config)
 {
     std::vector<std::pair<UUID, AccessEntityPtr>> all_entities;
-    for (const auto & entity : parseUsers(config, getLogger()))
+    for (const auto & entity : parseUsers(config))
         all_entities.emplace_back(generateID(*entity), entity);
-    for (const auto & entity : parseQuotas(config, getLogger()))
+    for (const auto & entity : parseQuotas(config))
         all_entities.emplace_back(generateID(*entity), entity);
-    for (const auto & entity : parseRowPolicies(config, getLogger()))
+    for (const auto & entity : parseRowPolicies(config))
         all_entities.emplace_back(generateID(*entity), entity);
-    for (const auto & entity : parseSettingsProfiles(config, check_setting_name_function, getLogger()))
+    for (const auto & entity : parseSettingsProfiles(config, check_setting_name_function))
         all_entities.emplace_back(generateID(*entity), entity);
     memory_storage.setAll(all_entities);
 }

src/Client/Connection.cpp

@@ -165,14 +165,12 @@ void Connection::sendHello()
         || has_control_character(password))
         throw Exception("Parameters 'default_database', 'user' and 'password' must not contain ASCII control characters", ErrorCodes::BAD_ARGUMENTS);
 
-    auto client_revision = ClickHouseRevision::get();
-
     writeVarUInt(Protocol::Client::Hello, *out);
     writeStringBinary((DBMS_NAME " ") + client_name, *out);
     writeVarUInt(DBMS_VERSION_MAJOR, *out);
     writeVarUInt(DBMS_VERSION_MINOR, *out);
     // NOTE For backward compatibility of the protocol, client cannot send its version_patch.
-    writeVarUInt(client_revision, *out);
+    writeVarUInt(DBMS_TCP_PROTOCOL_VERSION, *out);
     writeStringBinary(default_database, *out);
     /// If interserver-secret is used, one do not need password
     /// (NOTE we do not check for DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET, since we cannot ignore inter-server secret if it was requested)

src/Common/BitHelpers.h

@@ -1,22 +1,12 @@
 #pragma once
 
 #include <cstddef>
+#include <cstdint>
 #include <cassert>
 #include <type_traits>
 #include <common/defines.h>
 
 
-/** Returns log2 of number, rounded down.
-  * Compiles to single 'bsr' instruction on x86.
-  * For zero argument, result is unspecified.
-  */
-inline unsigned int bitScanReverse(unsigned int x)
-{
-    assert(x != 0);
-    return sizeof(unsigned int) * 8 - 1 - __builtin_clz(x);
-}
-
-
 /** For zero argument, result is zero.
   * For arguments with most significand bit set, result is n.
   * For other arguments, returns value, rounded up to power of two.
@@ -41,10 +31,9 @@ inline size_t roundUpToPowerOfTwoOrZero(size_t n)
 
 
 template <typename T>
-inline size_t getLeadingZeroBits(T x)
+inline size_t getLeadingZeroBitsUnsafe(T x)
 {
-    if (!x)
+    assert(x != 0);
-        return sizeof(x) * 8;
 
     if constexpr (sizeof(T) <= sizeof(unsigned int))
     {
@@ -60,10 +49,32 @@ inline size_t getLeadingZeroBits(T x)
     }
 }
 
+
+template <typename T>
+inline size_t getLeadingZeroBits(T x)
+{
+    if (!x)
+        return sizeof(x) * 8;
+
+    return getLeadingZeroBitsUnsafe(x);
+}
+
+/** Returns log2 of number, rounded down.
+  * Compiles to single 'bsr' instruction on x86.
+  * For zero argument, result is unspecified.
+  */
+template <typename T>
+inline uint32_t bitScanReverse(T x)
+{
+    return (std::max<size_t>(sizeof(T), sizeof(unsigned int))) * 8 - 1 - getLeadingZeroBitsUnsafe(x);
+}
+
 // Unsafe since __builtin_ctz()-family explicitly state that result is undefined on x == 0
 template <typename T>
 inline size_t getTrailingZeroBitsUnsafe(T x)
 {
+    assert(x != 0);
+
     if constexpr (sizeof(T) <= sizeof(unsigned int))
     {
         return __builtin_ctz(x);
@@ -88,8 +99,8 @@ inline size_t getTrailingZeroBits(T x)
 }
 
 /** Returns a mask that has '1' for `bits` LSB set:
- * maskLowBits<UInt8>(3) => 00000111
+  * maskLowBits<UInt8>(3) => 00000111
- */
+  */
 template <typename T>
 inline T maskLowBits(unsigned char bits)
 {

src/Common/ClickHouseRevision.cpp

@@ -6,6 +6,6 @@
 
 namespace ClickHouseRevision
 {
-    unsigned get() { return VERSION_REVISION; }
+    unsigned getVersionRevision() { return VERSION_REVISION; }
     unsigned getVersionInteger() { return VERSION_INTEGER; }
 }

src/Common/ClickHouseRevision.h

@@ -2,6 +2,6 @@
 
 namespace ClickHouseRevision
 {
-    unsigned get();
+    unsigned getVersionRevision();
     unsigned getVersionInteger();
 }

src/Common/Config/ConfigReloader.cpp

@@ -138,6 +138,7 @@ void ConfigReloader::reloadIfNewer(bool force, bool throw_on_error, bool fallbac
             if (throw_on_error)
                 throw;
             tryLogCurrentException(log, "Error updating configuration from '" + path + "' config.");
+            return;
         }
 
         LOG_DEBUG(log, "Loaded config '{}', performed update on configuration", path);

src/Common/Macros.cpp

@@ -23,18 +23,15 @@ Macros::Macros(const Poco::Util::AbstractConfiguration & config, const String &
 }
 
 String Macros::expand(const String & s,
-                      size_t level,
+                      MacroExpansionInfo & info) const
-                      const String & database_name,
-                      const String & table_name,
-                      const UUID & uuid) const
 {
     if (s.find('{') == String::npos)
         return s;
 
-    if (level && s.size() > 65536)
+    if (info.level && s.size() > 65536)
         throw Exception("Too long string while expanding macros", ErrorCodes::SYNTAX_ERROR);
 
-    if (level >= 10)
+    if (info.level >= 10)
         throw Exception("Too deep recursion while expanding macros: '" + s + "'", ErrorCodes::SYNTAX_ERROR);
 
     String res;
@@ -64,17 +61,28 @@ String Macros::expand(const String & s,
         /// Prefer explicit macros over implicit.
         if (it != macros.end())
             res += it->second;
-        else if (macro_name == "database" && !database_name.empty())
+        else if (macro_name == "database" && !info.database_name.empty())
-            res += database_name;
+            res += info.database_name;
-        else if (macro_name == "table" && !table_name.empty())
+        else if (macro_name == "table" && !info.table_name.empty())
-            res += table_name;
+            res += info.table_name;
         else if (macro_name == "uuid")
         {
-            if (uuid == UUIDHelpers::Nil)
+            if (info.uuid == UUIDHelpers::Nil)
                 throw Exception("Macro 'uuid' and empty arguments of ReplicatedMergeTree "
                                 "are supported only for ON CLUSTER queries with Atomic database engine",
                                 ErrorCodes::SYNTAX_ERROR);
-            res += toString(uuid);
+            /// For ON CLUSTER queries we don't want to require all macros definitions in initiator's config.
+            /// However, initiator must check that for cross-replication cluster zookeeper_path does not contain {uuid} macro.
+            /// It becomes impossible to check if {uuid} is contained inside some unknown macro.
+            if (info.level)
+                throw Exception("Macro 'uuid' should not be inside another macro", ErrorCodes::SYNTAX_ERROR);
+            res += toString(info.uuid);
+            info.expanded_uuid = true;
+        }
+        else if (info.ignore_unknown)
+        {
+            res += macro_name;
+            info.has_unknown = true;
         }
         else
             throw Exception("No macro '" + macro_name +
@@ -84,7 +92,8 @@ String Macros::expand(const String & s,
         pos = end + 1;
     }
 
-    return expand(res, level + 1, database_name, table_name);
+    ++info.level;
+    return expand(res, info);
 }
 
 String Macros::getValue(const String & key) const
@@ -94,9 +103,20 @@ String Macros::getValue(const String & key) const
     throw Exception("No macro " + key + " in config", ErrorCodes::SYNTAX_ERROR);
 }
 
+
+String Macros::expand(const String & s) const
+{
+    MacroExpansionInfo info;
+    return expand(s, info);
+}
+
 String Macros::expand(const String & s, const StorageID & table_id, bool allow_uuid) const
 {
-    return expand(s, 0, table_id.database_name, table_id.table_name, allow_uuid ? table_id.uuid : UUIDHelpers::Nil);
+    MacroExpansionInfo info;
+    info.database_name = table_id.database_name;
+    info.table_name = table_id.table_name;
+    info.uuid = allow_uuid ? table_id.uuid : UUIDHelpers::Nil;
+    return expand(s, info);
 }
 
 Names Macros::expand(const Names & source_names, size_t level) const
@@ -104,8 +124,12 @@ Names Macros::expand(const Names & source_names, size_t level) const
     Names result_names;
     result_names.reserve(source_names.size());
 
+    MacroExpansionInfo info;
     for (const String & name : source_names)
-        result_names.push_back(expand(name, level));
+    {
+        info.level = level;
+        result_names.push_back(expand(name, info));
+    }
 
     return result_names;
 }

src/Common/Macros.h

@@ -27,15 +27,28 @@ public:
     Macros() = default;
     Macros(const Poco::Util::AbstractConfiguration & config, const String & key);
 
+    struct MacroExpansionInfo
+    {
+        /// Settings
+        String database_name;
+        String table_name;
+        UUID uuid = UUIDHelpers::Nil;
+        bool ignore_unknown = false;
+
+        /// Information about macro expansion
+        size_t level = 0;
+        bool expanded_uuid = false;
+        bool has_unknown = false;
+    };
+
     /** Replace the substring of the form {macro_name} with the value for macro_name, obtained from the config file.
       * If {database} and {table} macros aren`t defined explicitly, expand them as database_name and table_name respectively.
       * level - the level of recursion.
       */
     String expand(const String & s,
-                  size_t level = 0,
+                  MacroExpansionInfo & info) const;
-                  const String & database_name = "",
+
-                  const String & table_name = "",
+    String expand(const String & s) const;
-                  const UUID & uuid = UUIDHelpers::Nil) const;
 
     String expand(const String & s, const StorageID & table_id, bool allow_uuid) const;
 

src/Common/StatusFile.cpp

@@ -37,7 +37,7 @@ StatusFile::FillFunction StatusFile::write_full_info = [](WriteBuffer & out)
 {
     out << "PID: " << getpid() << "\n"
         << "Started at: " << LocalDateTime(time(nullptr)) << "\n"
-        << "Revision: " << ClickHouseRevision::get() << "\n";
+        << "Revision: " << ClickHouseRevision::getVersionRevision() << "\n";
 };
 
 

src/Common/StringUtils/StringUtils.h

@@ -67,10 +67,19 @@ inline bool isASCII(char c)
     return static_cast<unsigned char>(c) < 0x80;
 }
 
+inline bool isLowerAlphaASCII(char c)
+{
+    return (c >= 'a' && c <= 'z');
+}
+
+inline bool isUpperAlphaASCII(char c)
+{
+    return (c >= 'A' && c <= 'Z');
+}
+
 inline bool isAlphaASCII(char c)
 {
-    return (c >= 'a' && c <= 'z')
+    return isLowerAlphaASCII(c) || isUpperAlphaASCII(c);
-        || (c >= 'A' && c <= 'Z');
 }
 
 inline bool isNumericASCII(char c)
@@ -122,6 +131,16 @@ inline bool isPrintableASCII(char c)
     return uc >= 32 && uc <= 126;   /// 127 is ASCII DEL.
 }
 
+inline bool isPunctuationASCII(char c)
+{
+    uint8_t uc = c;
+    return (uc >= 33 && uc <= 47)
+        || (uc >= 58 && uc <= 64)
+        || (uc >= 91 && uc <= 96)
+        || (uc >= 123 && uc <= 125);
+}
+
+
 inline bool isValidIdentifier(const std::string_view & str)
 {
     return !str.empty() && isValidIdentifierBegin(str[0]) && std::all_of(str.begin() + 1, str.end(), isWordCharASCII);

src/Common/ThreadPool.cpp

@@ -13,6 +13,7 @@ namespace DB
     namespace ErrorCodes
     {
         extern const int CANNOT_SCHEDULE_TASK;
+        extern const int LOGICAL_ERROR;
     }
 }
 
@@ -277,7 +278,11 @@ std::unique_ptr<GlobalThreadPool> GlobalThreadPool::the_instance;
 
 void GlobalThreadPool::initialize(size_t max_threads)
 {
-    assert(!the_instance);
+    if (the_instance)
+    {
+        throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR,
+            "The global thread pool is initialized twice");
+    }
 
     the_instance.reset(new GlobalThreadPool(max_threads,
         1000 /*max_free_threads*/, 10000 /*max_queue_size*/,

src/Common/config_version.h.in

@@ -2,18 +2,7 @@
 
 // .h autogenerated by cmake!
 
-#cmakedefine01 USE_DBMS_TCP_PROTOCOL_VERSION
+#cmakedefine VERSION_REVISION @VERSION_REVISION@
-
-#if USE_DBMS_TCP_PROTOCOL_VERSION
-    #include "Core/Defines.h"
-    #ifndef VERSION_REVISION
-        #define VERSION_REVISION DBMS_TCP_PROTOCOL_VERSION
-    #endif
-#else
-    #cmakedefine VERSION_REVISION @VERSION_REVISION@
-#endif
-
-
 #cmakedefine VERSION_NAME "@VERSION_NAME@"
 #define DBMS_NAME VERSION_NAME
 #cmakedefine VERSION_MAJOR @VERSION_MAJOR@

src/Common/randomSeed.cpp

@@ -4,6 +4,7 @@
 #include <Common/Exception.h>
 #include <Common/randomSeed.h>
 #include <Common/SipHash.h>
+#include <common/getThreadId.h>
 #include <common/types.h>
 
 
@@ -19,7 +20,7 @@ namespace DB
 DB::UInt64 randomSeed()
 {
     struct timespec times;
-    if (clock_gettime(CLOCK_THREAD_CPUTIME_ID, &times))
+    if (clock_gettime(CLOCK_MONOTONIC, &times))
         DB::throwFromErrno("Cannot clock_gettime.", DB::ErrorCodes::CANNOT_CLOCK_GETTIME);
 
     /// Not cryptographically secure as time, pid and stack address can be predictable.
@@ -27,7 +28,7 @@ DB::UInt64 randomSeed()
     SipHash hash;
     hash.update(times.tv_nsec);
     hash.update(times.tv_sec);
-    hash.update(getpid());
+    hash.update(getThreadId());
     hash.update(&times);
     return hash.get64();
 }

src/Core/AccurateComparison.h

@@ -4,6 +4,7 @@
 #include <limits>
 #include "Defines.h"
 #include "Types.h"
+#include <common/extended_types.h>
 #include <Common/NaNUtils.h>
 #include <Common/UInt128.h>
 
@@ -382,8 +383,8 @@ inline bool equalsOp<DB::Float32, DB::UInt128>(DB::Float32 f, DB::UInt128 u)
 
 inline bool NO_SANITIZE_UNDEFINED greaterOp(DB::Int128 i, DB::Float64 f)
 {
-    static constexpr Int128 min_int128 = Int128(0x8000000000000000ll) << 64;
+    static constexpr Int128 min_int128 = minInt128();
-    static constexpr Int128 max_int128 = (Int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
+    static constexpr Int128 max_int128 = maxInt128();
 
     if (-MAX_INT64_WITH_EXACT_FLOAT64_REPR <= i && i <= MAX_INT64_WITH_EXACT_FLOAT64_REPR)
         return static_cast<DB::Float64>(i) > f;
@@ -394,8 +395,8 @@ inline bool NO_SANITIZE_UNDEFINED greaterOp(DB::Int128 i, DB::Float64 f)
 
 inline bool NO_SANITIZE_UNDEFINED greaterOp(DB::Float64 f, DB::Int128 i)
 {
-    static constexpr Int128 min_int128 = Int128(0x8000000000000000ll) << 64;
+    static constexpr Int128 min_int128 = minInt128();
-    static constexpr Int128 max_int128 = (Int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
+    static constexpr Int128 max_int128 = maxInt128();
 
     if (-MAX_INT64_WITH_EXACT_FLOAT64_REPR <= i && i <= MAX_INT64_WITH_EXACT_FLOAT64_REPR)
         return f > static_cast<DB::Float64>(i);

src/Core/Defines.h

@@ -70,7 +70,7 @@
 /// Mininum revision supporting interserver secret.
 #define DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET 54441
 
-/// Version of ClickHouse TCP protocol. Set to git tag with latest protocol change.
+/// Version of ClickHouse TCP protocol. Increment it manually when you change the protocol.
 #define DBMS_TCP_PROTOCOL_VERSION 54441
 
 /// The boundary on which the blocks for asynchronous file operations should be aligned.

src/Core/MySQL/PacketsProtocolText.cpp

@@ -77,7 +77,7 @@ ColumnDefinition::ColumnDefinition(
 
 size_t ColumnDefinition::getPayloadSize() const
 {
-    return 13 + getLengthEncodedStringSize("def") + getLengthEncodedStringSize(schema) + getLengthEncodedStringSize(table) + getLengthEncodedStringSize(org_table) + \
+    return 12 + getLengthEncodedStringSize("def") + getLengthEncodedStringSize(schema) + getLengthEncodedStringSize(table) + getLengthEncodedStringSize(org_table) + \
             getLengthEncodedStringSize(name) + getLengthEncodedStringSize(org_name) + getLengthEncodedNumberSize(next_length);
 }
 
@@ -96,7 +96,7 @@ void ColumnDefinition::readPayloadImpl(ReadBuffer & payload)
     payload.readStrict(reinterpret_cast<char *>(&column_length), 4);
     payload.readStrict(reinterpret_cast<char *>(&column_type), 1);
     payload.readStrict(reinterpret_cast<char *>(&flags), 2);
-    payload.readStrict(reinterpret_cast<char *>(&decimals), 2);
+    payload.readStrict(reinterpret_cast<char *>(&decimals), 1);
     payload.ignore(2);
 }
 
@@ -113,7 +113,7 @@ void ColumnDefinition::writePayloadImpl(WriteBuffer & buffer) const
     buffer.write(reinterpret_cast<const char *>(&column_length), 4);
     buffer.write(reinterpret_cast<const char *>(&column_type), 1);
     buffer.write(reinterpret_cast<const char *>(&flags), 2);
-    buffer.write(reinterpret_cast<const char *>(&decimals), 2);
+    buffer.write(reinterpret_cast<const char *>(&decimals), 1);
     writeChar(0x0, 2, buffer);
 }
 

src/Core/Settings.h

@@ -350,7 +350,7 @@ class IColumn;
     M(UInt64, max_live_view_insert_blocks_before_refresh, 64, "Limit maximum number of inserted blocks after which mergeable blocks are dropped and query is re-executed.", 0) \
     M(UInt64, min_free_disk_space_for_temporary_data, 0, "The minimum disk space to keep while writing temporary data used in external sorting and aggregation.", 0) \
     \
-    M(DefaultDatabaseEngine, default_database_engine, DefaultDatabaseEngine::Ordinary, "Default database engine.", 0) \
+    M(DefaultDatabaseEngine, default_database_engine, DefaultDatabaseEngine::Atomic, "Default database engine.", 0) \
     M(Bool, show_table_uuid_in_table_create_query_if_not_nil, false, "For tables in databases with Engine=Atomic show UUID of the table in its CREATE query.", 0) \
     M(Bool, enable_scalar_subquery_optimization, true, "If it is set to true, prevent scalar subqueries from (de)serializing large scalar values and possibly avoid running the same subquery more than once.", 0) \
     M(Bool, optimize_trivial_count_query, true, "Process trivial 'SELECT count() FROM table' query from metadata.", 0) \

src/DataStreams/TemporaryFileStream.h

@@ -1,6 +1,5 @@
 #pragma once
 
-#include <Common/ClickHouseRevision.h>
 #include <DataStreams/IBlockInputStream.h>
 #include <DataStreams/NativeBlockInputStream.h>
 #include <DataStreams/NativeBlockOutputStream.h>
@@ -23,7 +22,7 @@ struct TemporaryFileStream
     TemporaryFileStream(const std::string & path)
         : file_in(path)
         , compressed_in(file_in)
-        , block_in(std::make_shared<NativeBlockInputStream>(compressed_in, ClickHouseRevision::get()))
+        , block_in(std::make_shared<NativeBlockInputStream>(compressed_in, DBMS_TCP_PROTOCOL_VERSION))
     {}
 
     TemporaryFileStream(const std::string & path, const Block & header_)

src/DataTypes/DataTypesDecimal.h

@@ -153,8 +153,9 @@ convertToDecimal(const typename FromDataType::FieldType & value, UInt32 scale)
         auto out = value * static_cast<FromFieldType>(DecimalUtils::scaleMultiplier<ToNativeType>(scale));
         if constexpr (std::is_same_v<ToNativeType, Int128>)
         {
-            static constexpr Int128 min_int128 = Int128(0x8000000000000000ll) << 64;
+            static constexpr Int128 min_int128 = minInt128();
-            static constexpr Int128 max_int128 = (Int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
+            static constexpr Int128 max_int128 = maxInt128();
+
             if (out <= static_cast<ToNativeType>(min_int128) || out >= static_cast<ToNativeType>(max_int128))
                 throw Exception(std::string(ToDataType::family_name) + " convert overflow. Float is out of Decimal range",
                                 ErrorCodes::DECIMAL_OVERFLOW);

src/Dictionaries/CacheDictionary.cpp

@@ -835,7 +835,8 @@ void CacheDictionary::waitForCurrentUpdateFinish(UpdateUnitPtr & update_unit_ptr
         catch (...)
         {
             throw DB::Exception(ErrorCodes::CACHE_DICTIONARY_UPDATE_FAIL,
-                "Dictionary update failed: {}",
+                "Update failed for dictionary '{}': {}",
+                getDictionaryID().getNameForLogs(),
                 getCurrentExceptionMessage(true /*with stack trace*/,
                     true /*check embedded stack trace*/));
         }
@@ -925,7 +926,7 @@ void CacheDictionary::update(UpdateUnitPtr & update_unit_ptr) const
                     else
                         cell.setExpiresAt(std::chrono::time_point<std::chrono::system_clock>::max());
 
-                    update_unit_ptr->getPresentIdHandler()(id, cell_idx);
+                    update_unit_ptr->callPresentIdHandler(id, cell_idx);
                     /// mark corresponding id as found
                     remaining_ids[id] = 1;
                 }
@@ -987,9 +988,9 @@ void CacheDictionary::update(UpdateUnitPtr & update_unit_ptr) const
                 if (was_default)
                     cell.setDefault();
                 if (was_default)
-                    update_unit_ptr->getAbsentIdHandler()(id, cell_idx);
+                    update_unit_ptr->callAbsentIdHandler(id, cell_idx);
                 else
-                    update_unit_ptr->getPresentIdHandler()(id, cell_idx);
+                    update_unit_ptr->callPresentIdHandler(id, cell_idx);
                 continue;
             }
             /// We don't have expired data for that `id` so all we can do is to rethrow `last_exception`.
@@ -1021,7 +1022,7 @@ void CacheDictionary::update(UpdateUnitPtr & update_unit_ptr) const
             setDefaultAttributeValue(attribute, cell_idx);
 
         /// inform caller that the cell has not been found
-        update_unit_ptr->getAbsentIdHandler()(id, cell_idx);
+        update_unit_ptr->callAbsentIdHandler(id, cell_idx);
     }
 
     ProfileEvents::increment(ProfileEvents::DictCacheKeysRequestedMiss, not_found_num);

src/Dictionaries/CacheDictionary.h

@@ -399,16 +399,18 @@ private:
                 absent_id_handler([](Key, size_t){}){}
 
 
-        PresentIdHandler getPresentIdHandler()
+        void callPresentIdHandler(Key key, size_t cell_idx)
         {
             std::lock_guard lock(callback_mutex);
-            return can_use_callback ? present_id_handler : PresentIdHandler{};
+            if (can_use_callback)
+                present_id_handler(key, cell_idx);
         }
 
-        AbsentIdHandler getAbsentIdHandler()
+        void callAbsentIdHandler(Key key, size_t cell_idx)
         {
             std::lock_guard lock(callback_mutex);
-            return can_use_callback ? absent_id_handler : AbsentIdHandler{};
+            if (can_use_callback)
+                absent_id_handler(key, cell_idx);
         }
 
         std::vector<Key> requested_ids;

src/Dictionaries/CacheDictionary.inc.h

@@ -148,7 +148,9 @@ void CacheDictionary::getItemsNumberImpl(
             std::begin(cache_expired_ids), std::end(cache_expired_ids),
             std::back_inserter(required_ids), [](auto & pair) { return pair.first; });
 
-    auto on_cell_updated = [&] (const auto id, const auto cell_idx)
+    auto on_cell_updated =
+    [&attribute_array, &cache_not_found_ids, &cache_expired_ids, &out]
+    (const auto id, const auto cell_idx)
     {
         const auto attribute_value = attribute_array[cell_idx];
 

src/Dictionaries/SSDCacheDictionary.cpp

@@ -53,6 +53,7 @@ namespace ErrorCodes
     extern const int AIO_READ_ERROR;
     extern const int AIO_WRITE_ERROR;
     extern const int BAD_ARGUMENTS;
+    extern const int CACHE_DICTIONARY_UPDATE_FAIL;
     extern const int CANNOT_ALLOCATE_MEMORY;
     extern const int CANNOT_CREATE_DIRECTORY;
     extern const int CANNOT_FSYNC;
@@ -1193,8 +1194,23 @@ void SSDCacheStorage::update(DictionarySourcePtr & source_ptr, const std::vector
             {
                 /// TODO: use old values
 
-                /// We don't have expired data for that `id` so all we can do is to rethrow `last_exception`.
+                // We don't have expired data for that `id` so all we can do is
-                std::rethrow_exception(last_update_exception);
+                // to rethrow `last_exception`. We might have to throw the same
+                // exception for different callers of dictGet() in different
+                // threads, which might then modify the exception object, so we
+                // have to throw a copy.
+                try
+                {
+                    std::rethrow_exception(last_update_exception);
+                }
+                catch (...)
+                {
+                    throw DB::Exception(ErrorCodes::CACHE_DICTIONARY_UPDATE_FAIL,
+                        "Update failed for dictionary '{}': {}",
+                        getPath(),
+                        getCurrentExceptionMessage(true /*with stack trace*/,
+                            true /*check embedded stack trace*/));
+                }
             }
 
             /// Set key

src/Dictionaries/SSDComplexKeyCacheDictionary.cpp

@@ -54,6 +54,7 @@ namespace ErrorCodes
     extern const int AIO_READ_ERROR;
     extern const int AIO_WRITE_ERROR;
     extern const int BAD_ARGUMENTS;
+    extern const int CACHE_DICTIONARY_UPDATE_FAIL;
     extern const int CANNOT_ALLOCATE_MEMORY;
     extern const int CANNOT_CREATE_DIRECTORY;
     extern const int CANNOT_FSYNC;
@@ -1266,8 +1267,23 @@ void SSDComplexKeyCacheStorage::update(
             {
                 /// TODO: use old values.
 
-                /// We don't have expired data for that `id` so all we can do is to rethrow `last_exception`.
+                // We don't have expired data for that `id` so all we can do is
-                std::rethrow_exception(last_update_exception);
+                // to rethrow `last_exception`. We might have to throw the same
+                // exception for different callers of dictGet() in different
+                // threads, which might then modify the exception object, so we
+                // have to throw a copy.
+                try
+                {
+                    std::rethrow_exception(last_update_exception);
+                }
+                catch (...)
+                {
+                    throw DB::Exception(ErrorCodes::CACHE_DICTIONARY_UPDATE_FAIL,
+                        "Update failed for dictionary '{}': {}",
+                        getPath(),
+                        getCurrentExceptionMessage(true /*with stack trace*/,
+                            true /*check embedded stack trace*/));
+                }
             }
 
             std::uniform_int_distribution<UInt64> distribution{lifetime.min_sec, lifetime.max_sec};

src/Functions/CMakeLists.txt

@@ -62,12 +62,10 @@ else()
 endif()
 
 
-option(STRIP_DEBUG_SYMBOLS_FUNCTIONS
+#  Provides faster linking and lower binary size.
-    "Do not generate debugger info for ClickHouse functions.
+#  Tradeoff is the inability to debug some source files with e.g. gdb
-    Provides faster linking and lower binary size.
+#  (empty stack frames and no local variables)."
-    Tradeoff is the inability to debug some source files with e.g. gdb
+option(STRIP_DEBUG_SYMBOLS_FUNCTIONS "Do not generate debugger info for ClickHouse functions" ${STRIP_DSF_DEFAULT})
-    (empty stack frames and no local variables)."
-    ${STRIP_DSF_DEFAULT})
 
 if (STRIP_DEBUG_SYMBOLS_FUNCTIONS)
     message(WARNING "Not generating debugger info for ClickHouse functions")
@@ -115,7 +113,11 @@ if(USE_RAPIDJSON)
     target_include_directories(clickhouse_functions SYSTEM PRIVATE ${RAPIDJSON_INCLUDE_DIR})
 endif()
 
-option(ENABLE_MULTITARGET_CODE "" ON)
+# ClickHouse developers may use platform-dependent code under some macro (e.g. `#ifdef ENABLE_MULTITARGET`).
+# If turned ON, this option defines such macro.
+# See `src/Functions/TargetSpecific.h`
+option(ENABLE_MULTITARGET_CODE "Enable platform-dependent code" ON)
+
 if (ENABLE_MULTITARGET_CODE)
     add_definitions(-DENABLE_MULTITARGET_CODE=1)
 else()

src/Functions/FunctionJoinGet.h

@@ -80,7 +80,7 @@ public:
     DataTypePtr getReturnType(const ColumnsWithTypeAndName &) const override { return {}; } // Not used
 
     bool useDefaultImplementationForNulls() const override { return false; }
-    bool useDefaultImplementationForLowCardinalityColumns() const override { return true; }
+    bool useDefaultImplementationForLowCardinalityColumns() const override { return false; }
 
     bool isVariadic() const override { return true; }
     size_t getNumberOfArguments() const override { return 0; }

src/Functions/FunctionsComparison.h

@@ -1213,7 +1213,7 @@ public:
         const bool left_is_string = isStringOrFixedString(which_left);
         const bool right_is_string = isStringOrFixedString(which_right);
 
-        bool date_and_datetime = (left_type != right_type) &&
+        bool date_and_datetime = (which_left.idx != which_right.idx) &&
             which_left.isDateOrDateTime() && which_right.isDateOrDateTime();
 
         if (left_is_num && right_is_num && !date_and_datetime)

src/Functions/TargetSpecific.h

@@ -11,7 +11,7 @@
  *
  * If compiler is not gcc/clang or target isn't x86_64 or ENABLE_MULTITARGET_CODE
  * was set to OFF in cmake, all code inside these macros will be removed and
- * USE_MUTLITARGE_CODE will be set to 0. Use #if USE_MUTLITARGE_CODE whenever you
+ * USE_MULTITARGET_CODE will be set to 0. Use #if USE_MULTITARGET_CODE whenever you
  * use anything from this namespaces.
  *
  * For similarities there is a macros DECLARE_DEFAULT_CODE, which wraps code

src/Interpreters/ActionsVisitor.cpp

@@ -235,11 +235,7 @@ static Block createBlockFromAST(const ASTPtr & node, const DataTypes & types, co
     return header.cloneWithColumns(std::move(columns));
 }
 
-/** Create a block for set from literal.
+Block createBlockForSet(
-  * 'set_element_types' - types of what are on the left hand side of IN.
-  * 'right_arg' - Literal - Tuple or Array.
-  */
-static Block createBlockForSet(
     const DataTypePtr & left_arg_type,
     const ASTPtr & right_arg,
     const DataTypes & set_element_types,
@@ -280,14 +276,7 @@ static Block createBlockForSet(
     return block;
 }
 
-/** Create a block for set from expression.
+Block createBlockForSet(
-  * 'set_element_types' - types of what are on the left hand side of IN.
-  * 'right_arg' - list of values: 1, 2, 3 or list of tuples: (1, 2), (3, 4), (5, 6).
-  *
-  *  We need special implementation for ASTFunction, because in case, when we interpret
-  *  large tuple or array as function, `evaluateConstantExpression` works extremely slow.
-  */
-static Block createBlockForSet(
     const DataTypePtr & left_arg_type,
     const std::shared_ptr<ASTFunction> & right_arg,
     const DataTypes & set_element_types,
@@ -900,10 +889,11 @@ SetPtr ActionsMatcher::makeSet(const ASTFunction & node, Data & data, bool no_su
           *   in the subquery_for_set object, this subquery is set as source and the temporary table _data1 as the table.
           * - this function shows the expression IN_data1.
           */
-        if (subquery_for_set.source.empty() && data.no_storage_or_local)
+        if (!subquery_for_set.source && data.no_storage_or_local)
         {
             auto interpreter = interpretSubquery(right_in_operand, data.context, data.subquery_depth, {});
-            subquery_for_set.source = QueryPipeline::getPipe(interpreter->execute().pipeline);
+            subquery_for_set.source = std::make_unique<QueryPlan>();
+            interpreter->buildQueryPlan(*subquery_for_set.source);
         }
 
         subquery_for_set.set = set;

src/Interpreters/ActionsVisitor.h

@@ -16,11 +16,37 @@ struct ExpressionAction;
 class ExpressionActions;
 using ExpressionActionsPtr = std::shared_ptr<ExpressionActions>;
 
- /// The case of an explicit enumeration of values.
+/// The case of an explicit enumeration of values.
 SetPtr makeExplicitSet(
     const ASTFunction * node, const Block & sample_block, bool create_ordered_set,
     const Context & context, const SizeLimits & limits, PreparedSets & prepared_sets);
 
+/** Create a block for set from expression.
+  * 'set_element_types' - types of what are on the left hand side of IN.
+  * 'right_arg' - list of values: 1, 2, 3 or list of tuples: (1, 2), (3, 4), (5, 6).
+  *
+  *  We need special implementation for ASTFunction, because in case, when we interpret
+  *  large tuple or array as function, `evaluateConstantExpression` works extremely slow.
+  *
+  *  Note: this and following functions are used in third-party applications in Arcadia, so
+  *  they should be declared in header file.
+  *
+  */
+Block createBlockForSet(
+    const DataTypePtr & left_arg_type,
+    const std::shared_ptr<ASTFunction> & right_arg,
+    const DataTypes & set_element_types,
+    const Context & context);
+
+/** Create a block for set from literal.
+  * 'set_element_types' - types of what are on the left hand side of IN.
+  * 'right_arg' - Literal - Tuple or Array.
+  */
+Block createBlockForSet(
+    const DataTypePtr & left_arg_type,
+    const ASTPtr & right_arg,
+    const DataTypes & set_element_types,
+    const Context & context);
 
 /** For ActionsVisitor
   * A stack of ExpressionActions corresponding to nested lambda expressions.

src/Interpreters/Aggregator.cpp

@@ -844,7 +844,7 @@ void Aggregator::writeToTemporaryFile(AggregatedDataVariants & data_variants, co
     const std::string & path = file->path();
     WriteBufferFromFile file_buf(path);
     CompressedWriteBuffer compressed_buf(file_buf);
-    NativeBlockOutputStream block_out(compressed_buf, ClickHouseRevision::get(), getHeader(false));
+    NativeBlockOutputStream block_out(compressed_buf, DBMS_TCP_PROTOCOL_VERSION, getHeader(false));
 
     LOG_DEBUG(log, "Writing part of aggregation data into temporary file {}.", path);
     ProfileEvents::increment(ProfileEvents::ExternalAggregationWritePart);

src/Interpreters/ClientInfo.cpp

@@ -5,7 +5,6 @@
 #include <IO/WriteHelpers.h>
 #include <Core/Defines.h>
 #include <common/getFQDNOrHostName.h>
-#include <Common/ClickHouseRevision.h>
 #include <unistd.h>
 
 #if !defined(ARCADIA_BUILD)
@@ -44,7 +43,7 @@ void ClientInfo::write(WriteBuffer & out, const UInt64 server_protocol_revision)
         writeBinary(client_name, out);
         writeVarUInt(client_version_major, out);
         writeVarUInt(client_version_minor, out);
-        writeVarUInt(client_revision, out);
+        writeVarUInt(client_tcp_protocol_version, out);
     }
     else if (interface == Interface::HTTP)
     {
@@ -92,7 +91,7 @@ void ClientInfo::read(ReadBuffer & in, const UInt64 client_protocol_revision)
         readBinary(client_name, in);
         readVarUInt(client_version_major, in);
         readVarUInt(client_version_minor, in);
-        readVarUInt(client_revision, in);
+        readVarUInt(client_tcp_protocol_version, in);
     }
     else if (interface == Interface::HTTP)
     {
@@ -111,7 +110,7 @@ void ClientInfo::read(ReadBuffer & in, const UInt64 client_protocol_revision)
         if (client_protocol_revision >= DBMS_MIN_REVISION_WITH_VERSION_PATCH)
             readVarUInt(client_version_patch, in);
         else
-            client_version_patch = client_revision;
+            client_version_patch = client_tcp_protocol_version;
     }
 }
 
@@ -137,7 +136,7 @@ void ClientInfo::fillOSUserHostNameAndVersionInfo()
     client_version_major = DBMS_VERSION_MAJOR;
     client_version_minor = DBMS_VERSION_MINOR;
     client_version_patch = DBMS_VERSION_PATCH;
-    client_revision = ClickHouseRevision::get();
+    client_tcp_protocol_version = DBMS_TCP_PROTOCOL_VERSION;
 }
 
 

src/Interpreters/ClientInfo.h

@@ -69,7 +69,7 @@ public:
     UInt64 client_version_major = 0;
     UInt64 client_version_minor = 0;
     UInt64 client_version_patch = 0;
-    unsigned client_revision = 0;
+    unsigned client_tcp_protocol_version = 0;
 
     /// For http
     HTTPMethod http_method = HTTPMethod::UNKNOWN;