Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview

CHANGELOG.md

@@ -1,9 +1,70 @@
+## ClickHouse release 19.7.3.9, 2019-05-30
+
+### New Features
+* Allow to limit the range of a setting that can be specified by user.
+  These constraints can be set up in user settings profile.
+[#4931](https://github.com/yandex/ClickHouse/pull/4931) ([Vitaly
+Baranov](https://github.com/vitlibar))
+* Add a second version of the function `groupUniqArray` with an optional
+  `max_size` parameter that limits the size of the resulting array. This
+behavior is similar to `groupArray(max_size)(x)` function.
+[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
+Tassery](https://github.com/YiuRULE))
+* For TSVWithNames/CSVWithNames input file formats, column order can now be
+  determined from file header. This is controlled by
+`input_format_with_names_use_header` parameter.
+[#5081](https://github.com/yandex/ClickHouse/pull/5081)
+([Alexander](https://github.com/Akazz))
+
+### Bug Fixes
+* Crash with uncompressed_cache + JOIN during merge (#5197)
+[#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
+Kutenin](https://github.com/danlark1))
+* Segmentation fault on a clickhouse-client query to system tables. #5066
+[#5127](https://github.com/yandex/ClickHouse/pull/5127)
+([Ivan](https://github.com/abyss7))
+* Data loss on heavy load via KafkaEngine (#4736)
+[#5080](https://github.com/yandex/ClickHouse/pull/5080)
+([Ivan](https://github.com/abyss7))
+* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Performance Improvements
+* Use radix sort for sorting by single numeric column in `ORDER BY` without
+  `LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
+[#4439](https://github.com/yandex/ClickHouse/pull/4439)
+([Evgenii Pravda](https://github.com/kvinty),
+[alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Documentation
+* Translate documentation for some table engines to Chinese.
+  [#5107](https://github.com/yandex/ClickHouse/pull/5107),
+[#5094](https://github.com/yandex/ClickHouse/pull/5094),
+[#5087](https://github.com/yandex/ClickHouse/pull/5087)
+([张风啸](https://github.com/AlexZFX)),
+[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
+lee](https://github.com/neverlee))
+ 
+### Build/Testing/Packaging Improvements
+* Print UTF-8 characters properly in `clickhouse-test`.
+  [#5084](https://github.com/yandex/ClickHouse/pull/5084)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Add command line parameter for clickhouse-client to always load suggestion
+  data. [#5102](https://github.com/yandex/ClickHouse/pull/5102)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Resolve some of PVS-Studio warnings.
+  [#5082](https://github.com/yandex/ClickHouse/pull/5082)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Update LZ4 [#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
+  Kutenin](https://github.com/danlark1))
+* Add gperf to build requirements for upcoming pull request #5030.
+  [#5110](https://github.com/yandex/ClickHouse/pull/5110)
+([proller](https://github.com/proller))
+
 ## ClickHouse release 19.6.2.11, 2019-05-13
 
 ### New Features
 * TTL expressions for columns and tables. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
 * Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
-* Added a new aggregate function `simpleLinearRegression(x, y)` which performs linear regression on points (x, y) and returns the parameters of the line. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
 * Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
 * Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
 
@@ -30,6 +91,7 @@
 * Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
 * Partially fix distributed_product_mode = local. It's possible to allow columns of local tables in where/having/order by/... via table aliases. Throw exception if table does not have alias. There's not possible to access to the columns without table aliases yet. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
 * Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
+* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
 
 ### Build/Testing/Packaging Improvements
 * Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
@@ -346,7 +408,7 @@
 * Added support of `Nullable` types in `mysql` table function. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
 * Support for arbitrary constant expressions in `LIMIT` clause. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
 * Added `topKWeighted` aggregate function that takes additional argument with (unsigned integer) weight. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
-* `StorageJoin` now supports `join_overwrite` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
+* `StorageJoin` now supports `join_any_take_last_row` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
 * Added function `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
 * Added `RowBinaryWithNamesAndTypes` format. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
 * Added `IPv4` and `IPv6` data types. More effective implementations of `IPv*` functions. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk))

CHANGELOG_RU.md

@@ -1,13 +1,77 @@
+## ClickHouse release 19.7.3.9, 2019-05-30
+
+### Новые возможности
+* Добавлена возможность ограничить значения конфигурационных параметров,
+  которые может задать пользователь. Эти ограничения устанавливаются в профиле
+настроек пользователя. [#4931](https://github.com/yandex/ClickHouse/pull/4931)
+([Vitaly Baranov](https://github.com/vitlibar))
+* Добавлен вариант функции `groupUniqArray` с дополнительным параметром
+  `max_size`, который ограничивает размер результирующего массива, аналогично
+функции `groupArray(max_size)(x)`.
+[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
+Tassery](https://github.com/YiuRULE))
+* Для входных файлов формата TSVWithNames и CSVWithNames появилась возможность
+  определить порядок колонок в файле исходя из его заголовка. Это поведение
+управляется конфигурационным параметром `input_format_with_names_use_header`.
+[#5081](https://github.com/yandex/ClickHouse/pull/5081)
+([Alexander](https://github.com/Akazz))
+
+### Исправления ошибок
+* Падение в процессе слияния при использовании uncompressed_cache и JOIN
+  (#5197). [#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
+Kutenin](https://github.com/danlark1))
+* Segmentation fault на запросе к системным таблицам (#5066).
+  [#5127](https://github.com/yandex/ClickHouse/pull/5127)
+([Ivan](https://github.com/abyss7))
+* Потеря загружаемых данных при больших потоках загрузки через KafkaEngine
+  (#4736). [#5080](https://github.com/yandex/ClickHouse/pull/5080)
+([Ivan](https://github.com/abyss7))
+* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Улучшения производительности
+* Используется поразрядная сортировка числовых колонок для `ORDER BY` без
+  `LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
+[#4439](https://github.com/yandex/ClickHouse/pull/4439) ([Evgenii
+Pravda](https://github.com/kvinty),
+[alexey-milovidov](https://github.com/alexey-milovidov)
+
+### Документация
+* Документация для некоторых табличных движков переведена на китайский.
+  [#5107](https://github.com/yandex/ClickHouse/pull/5107),
+[#5094](https://github.com/yandex/ClickHouse/pull/5094),
+[#5087](https://github.com/yandex/ClickHouse/pull/5087)
+([张风啸](https://github.com/AlexZFX),
+[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
+lee](https://github.com/neverlee))
+
+### Улучшения сборки, тестирования и пакетирования
+* Правильно отображаются символы в кодировке UTF-8 в `clickhouse-test`.
+  [#5084](https://github.com/yandex/ClickHouse/pull/5084)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Добавлен параметр командной строки для `clickhouse-client`, позволяющий
+  всегда загружать данные подсказок.
+[#5102](https://github.com/yandex/ClickHouse/pull/5102)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлены некоторые предупреждения PVS-Studio.
+  [#5082](https://github.com/yandex/ClickHouse/pull/5082)
+([alexey-milovidov](https://github.com/alexey-milovidov))
+* Обновлена библиотека LZ4.
+  [#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
+Kutenin](https://github.com/danlark1))
+* В зависимости сборки добавлен gperf для поддержки готовящегося PR #5030.
+  [#5110](https://github.com/yandex/ClickHouse/pull/5110)
+([proller](https://github.com/proller))
+
+
 ## ClickHouse release 19.6.2.11, 2019-05-13
 
 ### Новые возможности
 * TTL выражения, позволяющие настроить время жизни и автоматическую очистку данных в таблице или в отдельных её столбцах. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
 * Добавлена поддержка алгоритма сжатия `brotli` в HTTP ответах (`Accept-Encoding: br`). Для тела POST запросов, эта возможность уже существовала. [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
-* Добавлена агрегатная функция `simpleLinearRegression(x, y)` которая осуществляет линейную регрессию на точках (x, y) и возвращает параметры линии. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
 * Добавлена функция `isValidUTF8` для проверки, содержит ли строка валидные данные в кодировке UTF-8. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
 * Добавлены новое правило балансировки (`load_balancing`) `first_or_random` по которому запросы посылаются на первый заданый хост и если он недоступен - на случайные хосты шарда. Полезно для топологий с кросс-репликацией. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
 
-### Эксперемннтальные возможности
+### Экспериментальные возможности
 * Добавлена настройка `index_granularity_bytes` (адаптивная гранулярность индекса) для таблиц семейства MergeTree* . [#4826](https://github.com/yandex/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin))
 
 ### Улучшения
@@ -30,6 +94,7 @@
 * Исправлено зависание на старте сервера если внешний словарь зависит от другого словаря через использование таблицы из БД с движком `Dictionary`. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
 * При использовании `distributed_product_mode = 'local'` корректно работает использование столбцов локальных таблиц в where/having/order by/... через табличные алиасы. Выкидывает исключение если таблица не имеет алиас. Доступ к столбцам без алиасов пока не возможен. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
 * Исправлен потенциально некорректный результат для `SELECT DISTINCT` с `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
+* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
 
 ### Улучшения сборки/тестирования/пакетирования
 * Исправлена неработоспособность тестов, если `clickhouse-server` запущен на удалённом хосте [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
@@ -337,7 +402,7 @@
 * Добавлена поддержка `Nullable` типов в табличной функции `mysql`. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
 * Добавлена поддержка произвольных константных выражений в секции `LIMIT`. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
 * Добавлена агрегатная функция `topKWeighted` - вариант `topK`, позволяющий задавать (целый неотрицательный) вес добавляемого значения. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
-* Движок `Join` теперь поддерживает настройку `join_overwrite`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
+* Движок `Join` теперь поддерживает настройку `join_any_take_last_row`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
 * Добавлена функция `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
 * Добавлена функция `toStartOfTenMinutes`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar))
 * Добавлен формат `RowBinaryWithNamesAndTypes`. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))

CMakeLists.txt

@@ -93,6 +93,11 @@ if (COMPILER_GCC OR COMPILER_CLANG)
     set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wnon-virtual-dtor")
 endif ()
 
+if (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER "8.3.0")
+    # Warnings in protobuf generating
+    set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wno-array-bounds")
+endif ()
+
 if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
     # clang: warning: argument unused during compilation: '-stdlib=libc++'
     # clang: warning: argument unused during compilation: '-specs=/usr/share/dpkg/no-pie-compile.specs' [-Wunused-command-line-argument]

README.md

@@ -12,8 +12,7 @@ ClickHouse is an open-source column-oriented database management system that all
 * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
 
 ## Upcoming Events
-* ClickHouse at [Percona Live 2019](https://www.percona.com/live/19/other-open-source-databases-track) in Austin on May 28-30.
-* [ClickHouse Community Meetup in San Francisco](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/261110652/) on June 4.
 * [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8.
+* [ClickHouse on HighLoad++ Siberia](https://www.highload.ru/siberia/2019/abstracts/5348) on June 24-25.
 * [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
 * [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.

contrib/CMakeLists.txt

@@ -1,8 +1,8 @@
 # Third-party libraries may have substandard code.
 
 if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
-    set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch")
+    set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch -Wno-stringop-truncation")
-    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z")
+    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -Wno-array-bounds -Wno-missing-attributes -Wno-stringop-truncation -std=c++1z")
 elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
     set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch -Wno-string-plus-int")
     set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")

contrib/boost

@@ -1 +1 @@
-Subproject commit 79bf85ea99c05ba4fb6959474d4464ab126f8973
+Subproject commit 8abda007bfe52d78a51548d4594879d6d82a22fa

contrib/croaring/CMakeLists.txt

@@ -1,6 +1,6 @@
 add_library(roaring
 	roaring.c
-	roaring.h
+	roaring/roaring.h
-	roaring.hh)
+	roaring/roaring.hh)
 
 target_include_directories (roaring PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

contrib/croaring/roaring.c

@@ -1,5 +1,5 @@
 /* auto-generated on Tue Dec 18 09:42:59 CST 2018. Do not edit! */
-#include "roaring.h"
+#include "roaring/roaring.h"
 
 /* used for http://dmalloc.com/ Dmalloc - Debug Malloc Library */
 #ifdef DMALLOC

contrib/hyperscan

@@ -1 +1 @@
-Subproject commit 05b0f9064cca4bd55548dedb0a32ed9461146c1e
+Subproject commit 01e6b83f9fbdb4020cd68a5287bf3a0471eeb272

contrib/jemalloc-cmake/CMakeLists.txt

@@ -40,6 +40,10 @@ if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
   list(APPEND SRCS ${JEMALLOC_SOURCE_DIR}/src/zone.c)
 endif()
 
+if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
+    set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
+endif ()
+
 add_library(jemalloc STATIC ${SRCS})
 
 target_include_directories(jemalloc PUBLIC

contrib/libpcg-random/include/pcg_extras.hpp

@@ -81,7 +81,7 @@
     #define PCG_128BIT_CONSTANT(high,low) \
             ((pcg128_t(high) << 64) + low)
 #else
-    #include "pcg_uint128.hpp" // Y_IGNORE
+    #include "pcg_uint128.hpp"
     namespace pcg_extras {
         typedef pcg_extras::uint_x4<uint32_t,uint64_t> pcg128_t;
     }

contrib/librdkafka-cmake/include/librdkafka/rdkafka.h

@@ -1,5 +1,5 @@
 #if __has_include(<rdkafka.h>) // maybe bundled
-#    include_next <rdkafka.h> // Y_IGNORE
+#    include_next <rdkafka.h>
 #else // system
 #    include_next <librdkafka/rdkafka.h>
 #endif

dbms/CMakeLists.txt

@@ -49,19 +49,20 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
     endif ()
 
     if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7)
-        set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
-
         if (WEVERYTHING)
             set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-return-std-move-in-c++11")
         endif ()
     endif ()
 
     if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8)
-        set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wshadow-field -Wstring-plus-int -Wempty-init-stmt")
+        set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow-field -Wstring-plus-int")
+        if(NOT APPLE)
+            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wempty-init-stmt")
+        endif()
     endif ()
 
     if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9)
-        if (WEVERYTHING)
+        if (WEVERYTHING AND NOT APPLE)
             set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-ctad-maybe-unsupported")
         endif ()
     endif ()

dbms/cmake/version.cmake

@@ -1,11 +1,11 @@
 # This strings autochanged from release_lib.sh:
-set(VERSION_REVISION 54420)
+set(VERSION_REVISION 54421)
 set(VERSION_MAJOR 19)
-set(VERSION_MINOR 8)
+set(VERSION_MINOR 9)
 set(VERSION_PATCH 1)
-set(VERSION_GITHASH a76e504f45ff4a74e8c492bd269f022352d5f6d9)
+set(VERSION_GITHASH 0c2aa460651a462f14efc7e995840a244531d373)
-set(VERSION_DESCRIBE v19.8.1.1-testing)
+set(VERSION_DESCRIBE v19.9.1.1-testing)
-set(VERSION_STRING 19.8.1.1)
+set(VERSION_STRING 19.9.1.1)
 # end of autochange
 
 set(VERSION_EXTRA "" CACHE STRING "")

dbms/programs/client/CMakeLists.txt

@@ -1,6 +1,19 @@
-set(CLICKHOUSE_CLIENT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp)
+set(CLICKHOUSE_CLIENT_SOURCES
+    ${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/ConnectionParameters.cpp
+)
+
 set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY})
-set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR})
+set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/include)
+
+include(CheckSymbolExists)
+check_symbol_exists(readpassphrase readpassphrase.h HAVE_READPASSPHRASE)
+configure_file(config_client.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/config_client.h)
+
+if(NOT HAVE_READPASSPHRASE)
+    add_subdirectory(readpassphrase)
+    list(APPEND CLICKHOUSE_CLIENT_LINK PRIVATE readpassphrase)
+endif()
 
 clickhouse_program_add(client)
 

dbms/programs/client/Client.cpp

@@ -67,7 +67,7 @@
 #include <Storages/ColumnsDescription.h>
 
 #if USE_READLINE
-#include "Suggest.h" // Y_IGNORE
+#include "Suggest.h"
 #endif
 
 #ifndef __clang__
@@ -435,7 +435,7 @@ private:
 #if USE_READLINE
                     int res = read_history(history_file.c_str());
                     if (res)
-                        throwFromErrno("Cannot read history from file " + history_file, ErrorCodes::CANNOT_READ_HISTORY);
+                        std::cerr << "Cannot read history from file " + history_file + ": "+ errnoToString(ErrorCodes::CANNOT_READ_HISTORY);
 #endif
                 }
                 else    /// Create history file.
@@ -612,7 +612,7 @@ private:
 
 #if USE_READLINE && HAVE_READLINE_HISTORY
                     if (!history_file.empty() && append_history(1, history_file.c_str()))
-                        throwFromErrno("Cannot append history to file " + history_file, ErrorCodes::CANNOT_APPEND_HISTORY);
+                        std::cerr << "Cannot append history to file " + history_file + ": " + errnoToString(ErrorCodes::CANNOT_APPEND_HISTORY);
 #endif
 
                     prev_input = input;

dbms/programs/client/ConnectionParameters.cpp

@@ -0,0 +1,63 @@
+#include "ConnectionParameters.h"
+#include <fstream>
+#include <iostream>
+#include <Core/Defines.h>
+#include <Core/Protocol.h>
+#include <Core/Types.h>
+#include <IO/ConnectionTimeouts.h>
+#include <Poco/Util/AbstractConfiguration.h>
+#include <Common/Exception.h>
+#include <common/setTerminalEcho.h>
+#include <ext/scope_guard.h>
+#include <readpassphrase.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int BAD_ARGUMENTS;
+}
+
+ConnectionParameters::ConnectionParameters(const Poco::Util::AbstractConfiguration & config)
+{
+    bool is_secure = config.getBool("secure", false);
+    security = is_secure ? Protocol::Secure::Enable : Protocol::Secure::Disable;
+
+    host = config.getString("host", "localhost");
+    port = config.getInt(
+        "port", config.getInt(is_secure ? "tcp_port_secure" : "tcp_port", is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
+
+    default_database = config.getString("database", "");
+    /// changed the default value to "default" to fix the issue when the user in the prompt is blank
+    user = config.getString("user", "default");
+    bool password_prompt = false;
+    if (config.getBool("ask-password", false))
+    {
+        if (config.has("password"))
+            throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
+        password_prompt = true;
+    }
+    else
+    {
+        password = config.getString("password", "");
+        /// if the value of --password is omitted, the password will be set implicitly to "\n"
+        if (password == "\n")
+            password_prompt = true;
+    }
+    if (password_prompt)
+    {
+        std::string prompt{"Password for user (" + user + "): "};
+        char buf[1000] = {};
+        if (auto result = readpassphrase(prompt.c_str(), buf, sizeof(buf), 0))
+            password = result;
+    }
+    compression = config.getBool("compression", true) ? Protocol::Compression::Enable : Protocol::Compression::Disable;
+
+    timeouts = ConnectionTimeouts(
+        Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
+        Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
+        Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
+        Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
+}
+}

dbms/programs/client/ConnectionParameters.h

@@ -1,90 +1,30 @@
 #pragma once
 
-#include <iostream>
+#include <string>
-
-#include <Core/Types.h>
 #include <Core/Protocol.h>
-#include <Core/Defines.h>
-#include <Common/Exception.h>
 #include <IO/ConnectionTimeouts.h>
 
-#include <common/setTerminalEcho.h>
+namespace Poco::Util
-#include <ext/scope_guard.h>
+{
-
+class AbstractConfiguration;
-#include <Poco/Util/AbstractConfiguration.h>
+}
-
 
 namespace DB
 {
-
-namespace ErrorCodes
-{
-    extern const int BAD_ARGUMENTS;
-}
-
 struct ConnectionParameters
 {
-    String host;
+    std::string host;
     UInt16 port{};
-    String default_database;
+    std::string default_database;
-    String user;
+    std::string user;
-    String password;
+    std::string password;
     Protocol::Secure security = Protocol::Secure::Disable;
     Protocol::Compression compression = Protocol::Compression::Enable;
     ConnectionTimeouts timeouts;
 
     ConnectionParameters() {}
 
-    ConnectionParameters(const Poco::Util::AbstractConfiguration & config)
+    ConnectionParameters(const Poco::Util::AbstractConfiguration & config);
-    {
-        bool is_secure = config.getBool("secure", false);
-        security = is_secure
-            ? Protocol::Secure::Enable
-            : Protocol::Secure::Disable;
-
-        host = config.getString("host", "localhost");
-        port = config.getInt("port",
-            config.getInt(is_secure ? "tcp_port_secure" : "tcp_port",
-                is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
-
-        default_database = config.getString("database", "");
-        /// changed the default value to "default" to fix the issue when the user in the prompt is blank
-        user = config.getString("user", "default");
-        bool password_prompt = false;
-        if (config.getBool("ask-password", false))
-        {
-            if (config.has("password"))
-                throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
-            password_prompt = true;
-        }
-        else
-        {
-            password = config.getString("password", "");
-            /// if the value of --password is omitted, the password will be set implicitly to "\n"
-            if (password == "\n")
-                password_prompt = true;
-        }
-        if (password_prompt)
-        {
-            std::cout << "Password for user (" << user << "): ";
-            setTerminalEcho(false);
-
-            SCOPE_EXIT({
-                setTerminalEcho(true);
-            });
-            std::getline(std::cin, password);
-            std::cout << std::endl;
-        }
-        compression = config.getBool("compression", true)
-            ? Protocol::Compression::Enable
-            : Protocol::Compression::Disable;
-
-        timeouts = ConnectionTimeouts(
-            Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
-            Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
-            Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
-            Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
-    }
 };
 
 }

dbms/programs/client/config_client.h.in

@@ -0,0 +1,3 @@
+#pragma once
+
+#cmakedefine HAVE_READPASSPHRASE

dbms/programs/client/readpassphrase/CMakeLists.txt

@@ -0,0 +1,10 @@
+
+# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.c
+# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.h
+
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-result -Wno-reserved-id-macro")
+
+configure_file(includes.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/includes.h)
+add_library(readpassphrase ${CMAKE_CURRENT_SOURCE_DIR}/readpassphrase.c)
+# . to allow #include <readpassphrase.h>
+target_include_directories(readpassphrase PUBLIC . ${CMAKE_CURRENT_BINARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR}/../include)

dbms/programs/client/readpassphrase/includes.h.in

@@ -0,0 +1,9 @@
+#pragma once
+
+#cmakedefine HAVE_READPASSPHRASE
+
+#if !defined(HAVE_READPASSPHRASE)
+#    ifndef _PATH_TTY
+#        define _PATH_TTY "/dev/tty"
+#    endif
+#endif

dbms/programs/client/readpassphrase/readpassphrase.c

@@ -0,0 +1,211 @@
+/*	$OpenBSD: readpassphrase.c,v 1.26 2016/10/18 12:47:18 millert Exp $	*/
+
+/*
+ * Copyright (c) 2000-2002, 2007, 2010
+ *	Todd C. Miller <Todd.Miller@courtesan.com>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ *
+ * Sponsored in part by the Defense Advanced Research Projects
+ * Agency (DARPA) and Air Force Research Laboratory, Air Force
+ * Materiel Command, USAF, under agreement number F39502-99-1-0512.
+ */
+
+/* OPENBSD ORIGINAL: lib/libc/gen/readpassphrase.c */
+
+#include "includes.h"
+
+#ifndef HAVE_READPASSPHRASE
+
+#include <termios.h>
+#include <signal.h>
+#include <ctype.h>
+#include <fcntl.h>
+#include <readpassphrase.h>
+#include <errno.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifndef TCSASOFT
+/* If we don't have TCSASOFT define it so that ORing it it below is a no-op. */
+# define TCSASOFT 0
+#endif
+
+/* SunOS 4.x which lacks _POSIX_VDISABLE, but has VDISABLE */
+#if !defined(_POSIX_VDISABLE) && defined(VDISABLE)
+#  define _POSIX_VDISABLE       VDISABLE
+#endif
+
+static volatile sig_atomic_t signo[_NSIG];
+
+static void handler(int);
+
+char *
+readpassphrase(const char *prompt, char *buf, size_t bufsiz, int flags)
+{
+	ssize_t nr;
+	int input, output, save_errno, i, need_restart;
+	char ch, *p, *end;
+	struct termios term, oterm;
+	struct sigaction sa, savealrm, saveint, savehup, savequit, saveterm;
+	struct sigaction savetstp, savettin, savettou, savepipe;
+
+	/* I suppose we could alloc on demand in this case (XXX). */
+	if (bufsiz == 0) {
+		errno = EINVAL;
+		return(NULL);
+	}
+
+restart:
+	for (i = 0; i < _NSIG; i++)
+		signo[i] = 0;
+	nr = -1;
+	save_errno = 0;
+	need_restart = 0;
+	/*
+	 * Read and write to /dev/tty if available.  If not, read from
+	 * stdin and write to stderr unless a tty is required.
+	 */
+	if ((flags & RPP_STDIN) ||
+	    (input = output = open(_PATH_TTY, O_RDWR)) == -1) {
+		if (flags & RPP_REQUIRE_TTY) {
+			errno = ENOTTY;
+			return(NULL);
+		}
+		input = STDIN_FILENO;
+		output = STDERR_FILENO;
+	}
+
+	/*
+	 * Turn off echo if possible.
+	 * If we are using a tty but are not the foreground pgrp this will
+	 * generate SIGTTOU, so do it *before* installing the signal handlers.
+	 */
+	if (input != STDIN_FILENO && tcgetattr(input, &oterm) == 0) {
+		memcpy(&term, &oterm, sizeof(term));
+		if (!(flags & RPP_ECHO_ON))
+			term.c_lflag &= ~(ECHO | ECHONL);
+#ifdef VSTATUS
+		if (term.c_cc[VSTATUS] != _POSIX_VDISABLE)
+			term.c_cc[VSTATUS] = _POSIX_VDISABLE;
+#endif
+		(void)tcsetattr(input, TCSAFLUSH|TCSASOFT, &term);
+	} else {
+		memset(&term, 0, sizeof(term));
+		term.c_lflag |= ECHO;
+		memset(&oterm, 0, sizeof(oterm));
+		oterm.c_lflag |= ECHO;
+	}
+
+	/*
+	 * Catch signals that would otherwise cause the user to end
+	 * up with echo turned off in the shell.  Don't worry about
+	 * things like SIGXCPU and SIGVTALRM for now.
+	 */
+	sigemptyset(&sa.sa_mask);
+	sa.sa_flags = 0;		/* don't restart system calls */
+	sa.sa_handler = handler;
+	(void)sigaction(SIGALRM, &sa, &savealrm);
+	(void)sigaction(SIGHUP, &sa, &savehup);
+	(void)sigaction(SIGINT, &sa, &saveint);
+	(void)sigaction(SIGPIPE, &sa, &savepipe);
+	(void)sigaction(SIGQUIT, &sa, &savequit);
+	(void)sigaction(SIGTERM, &sa, &saveterm);
+	(void)sigaction(SIGTSTP, &sa, &savetstp);
+	(void)sigaction(SIGTTIN, &sa, &savettin);
+	(void)sigaction(SIGTTOU, &sa, &savettou);
+
+	if (!(flags & RPP_STDIN))
+		(void)write(output, prompt, strlen(prompt));
+	end = buf + bufsiz - 1;
+	p = buf;
+	while ((nr = read(input, &ch, 1)) == 1 && ch != '\n' && ch != '\r') {
+		if (p < end) {
+			if ((flags & RPP_SEVENBIT))
+				ch &= 0x7f;
+			if (isalpha((unsigned char)ch)) {
+				if ((flags & RPP_FORCELOWER))
+					ch = (char)tolower((unsigned char)ch);
+				if ((flags & RPP_FORCEUPPER))
+					ch = (char)toupper((unsigned char)ch);
+			}
+			*p++ = ch;
+		}
+	}
+	*p = '\0';
+	save_errno = errno;
+	if (!(term.c_lflag & ECHO))
+		(void)write(output, "\n", 1);
+
+	/* Restore old terminal settings and signals. */
+	if (memcmp(&term, &oterm, sizeof(term)) != 0) {
+		const int sigttou = signo[SIGTTOU];
+
+		/* Ignore SIGTTOU generated when we are not the fg pgrp. */
+		while (tcsetattr(input, TCSAFLUSH|TCSASOFT, &oterm) == -1 &&
+		    errno == EINTR && !signo[SIGTTOU])
+			continue;
+		signo[SIGTTOU] = sigttou;
+	}
+	(void)sigaction(SIGALRM, &savealrm, NULL);
+	(void)sigaction(SIGHUP, &savehup, NULL);
+	(void)sigaction(SIGINT, &saveint, NULL);
+	(void)sigaction(SIGQUIT, &savequit, NULL);
+	(void)sigaction(SIGPIPE, &savepipe, NULL);
+	(void)sigaction(SIGTERM, &saveterm, NULL);
+	(void)sigaction(SIGTSTP, &savetstp, NULL);
+	(void)sigaction(SIGTTIN, &savettin, NULL);
+	(void)sigaction(SIGTTOU, &savettou, NULL);
+	if (input != STDIN_FILENO)
+		(void)close(input);
+
+	/*
+	 * If we were interrupted by a signal, resend it to ourselves
+	 * now that we have restored the signal handlers.
+	 */
+	for (i = 0; i < _NSIG; i++) {
+		if (signo[i]) {
+			kill(getpid(), i);
+			switch (i) {
+			case SIGTSTP:
+			case SIGTTIN:
+			case SIGTTOU:
+				need_restart = 1;
+			}
+		}
+	}
+	if (need_restart)
+		goto restart;
+
+	if (save_errno)
+		errno = save_errno;
+	return(nr == -1 ? NULL : buf);
+}
+//DEF_WEAK(readpassphrase);
+
+#if 0
+char *
+getpass(const char *prompt)
+{
+	static char buf[_PASSWORD_LEN + 1];
+
+	return(readpassphrase(prompt, buf, sizeof(buf), RPP_ECHO_OFF));
+}
+#endif
+
+static void handler(int s)
+{
+
+	signo[s] = 1;
+}
+#endif /* HAVE_READPASSPHRASE */

dbms/programs/client/readpassphrase/readpassphrase.h

@@ -0,0 +1,56 @@
+// /*	$OpenBSD: readpassphrase.h,v 1.5 2003/06/17 21:56:23 millert Exp $	*/
+
+/*
+ * Copyright (c) 2000, 2002 Todd C. Miller <Todd.Miller@courtesan.com>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ *
+ * Sponsored in part by the Defense Advanced Research Projects
+ * Agency (DARPA) and Air Force Research Laboratory, Air Force
+ * Materiel Command, USAF, under agreement number F39502-99-1-0512.
+ */
+
+/* OPENBSD ORIGINAL: include/readpassphrase.h */
+
+#pragma once
+// #ifndef _READPASSPHRASE_H_
+// #define _READPASSPHRASE_H_
+
+//#include "includes.h"
+#include "config_client.h"
+
+#ifndef HAVE_READPASSPHRASE
+
+#    ifdef __cplusplus
+extern "C" {
+#    endif
+
+
+#    define RPP_ECHO_OFF 0x00 /* Turn off echo (default). */
+#    define RPP_ECHO_ON 0x01 /* Leave echo on. */
+#    define RPP_REQUIRE_TTY 0x02 /* Fail if there is no tty. */
+#    define RPP_FORCELOWER 0x04 /* Force input to lower case. */
+#    define RPP_FORCEUPPER 0x08 /* Force input to upper case. */
+#    define RPP_SEVENBIT 0x10 /* Strip the high bit from input. */
+#    define RPP_STDIN 0x20 /* Read from stdin, not /dev/tty */
+
+char * readpassphrase(const char *, char *, size_t, int);
+
+#    ifdef __cplusplus
+}
+#    endif
+
+
+#endif /* HAVE_READPASSPHRASE */
+
+// #endif /* !_READPASSPHRASE_H_ */

dbms/programs/main.cpp

@@ -15,7 +15,7 @@
 #endif
 
 #if USE_TCMALLOC
-#include <gperftools/malloc_extension.h> // Y_IGNORE
+#include <gperftools/malloc_extension.h>
 #endif
 
 #include <Common/StringUtils/StringUtils.h>

dbms/programs/odbc-bridge/ColumnInfoHandler.cpp

@@ -3,9 +3,9 @@
 #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
 
 #if USE_POCO_SQLODBC
-#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/ODBCException.h>
-#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/SessionImpl.h>
-#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/Utility.h>
 #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
 #endif
 #if USE_POCO_DATAODBC

dbms/programs/odbc-bridge/IdentifierQuoteHandler.cpp

@@ -2,9 +2,9 @@
 #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
 
 #if USE_POCO_SQLODBC
-#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/ODBCException.h>
-#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/SessionImpl.h>
-#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/Utility.h>
 #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
 #endif
 #if USE_POCO_DATAODBC

dbms/programs/odbc-bridge/getIdentifierQuote.cpp

@@ -2,9 +2,9 @@
 #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
 
 #if USE_POCO_SQLODBC
-#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/ODBCException.h>
-#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/SessionImpl.h>
-#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/Utility.h>
 #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
 #endif
 #if USE_POCO_DATAODBC

dbms/programs/odbc-bridge/getIdentifierQuote.h

@@ -8,7 +8,7 @@
 #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
 
 #if USE_POCO_SQLODBC
-#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
+#include <Poco/SQL/ODBC/Utility.h>
 #endif
 #if USE_POCO_DATAODBC
 #include <Poco/Data/ODBC/Utility.h>

dbms/programs/performance-test/PerformanceTest.cpp

@@ -1,6 +1,7 @@
 #include "PerformanceTest.h"
 
 #include <Core/Types.h>
+#include <Common/CpuId.h>
 #include <common/getMemoryAmount.h>
 #include <IO/ReadBufferFromFile.h>
 #include <IO/ReadHelpers.h>
@@ -71,6 +72,7 @@ bool PerformanceTest::checkPreconditions() const
     Strings preconditions;
     config->keys("preconditions", preconditions);
     size_t table_precondition_index = 0;
+    size_t cpu_precondition_index = 0;
 
     for (const std::string & precondition : preconditions)
     {
@@ -136,6 +138,30 @@ bool PerformanceTest::checkPreconditions() const
                 return false;
             }
         }
+
+        if (precondition == "cpu")
+        {
+            std::string precondition_key = "preconditions.cpu[" + std::to_string(cpu_precondition_index++) + "]";
+            std::string flag_to_check = config->getString(precondition_key);
+
+            #define CHECK_CPU_PRECONDITION(OP) \
+            if (flag_to_check == #OP) \
+            { \
+                if (!Cpu::CpuFlagsCache::have_##OP) \
+                { \
+                    LOG_WARNING(log, "CPU doesn't support " << #OP); \
+                    return false; \
+                } \
+            } else
+
+            CPU_ID_ENUMERATE(CHECK_CPU_PRECONDITION)
+            {
+                LOG_WARNING(log, "CPU doesn't support " << flag_to_check);
+                return false;
+            }
+
+            #undef CHECK_CPU_PRECONDITION
+        }
     }
 
     return true;

dbms/programs/server/CMakeLists.txt

@@ -9,6 +9,7 @@ set(CLICKHOUSE_SERVER_SOURCES
     ${CMAKE_CURRENT_SOURCE_DIR}/Server.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/TCPHandler.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
    )
 
 set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})

dbms/programs/server/HTTPHandler.cpp

@@ -312,11 +312,13 @@ void HTTPHandler::processQuery(
             client_supports_http_compression = true;
             http_response_compression_method = CompressionMethod::Zlib;
         }
+#if USE_BROTLI
         else if (http_response_compression_methods == "br")
         {
             client_supports_http_compression = true;
             http_response_compression_method = CompressionMethod::Brotli;
         }
+#endif
     }
 
     /// Client can pass a 'compress' flag in the query string. In this case the query result is
@@ -453,30 +455,6 @@ void HTTPHandler::processQuery(
         return false;
     };
 
-    /// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
-    std::string full_query;
-
-    /// Support for "external data for query processing".
-    if (startsWith(request.getContentType().data(), "multipart/form-data"))
-    {
-        ExternalTablesHandler handler(context, params);
-        params.load(request, istr, handler);
-
-        /// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
-        reserved_param_suffixes.emplace_back("_format");
-        reserved_param_suffixes.emplace_back("_types");
-        reserved_param_suffixes.emplace_back("_structure");
-
-        /// Params are of both form params POST and uri (GET params)
-        for (const auto & it : params)
-            if (it.first == "query")
-                full_query += it.second;
-
-        in = std::make_unique<ReadBufferFromString>(full_query);
-    }
-    else
-        in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
-
     /// Settings can be overridden in the query.
     /// Some parameters (database, default_format, everything used in the code above) do not
     /// belong to the Settings class.
@@ -497,30 +475,63 @@ void HTTPHandler::processQuery(
             settings.readonly = 2;
     }
 
+    bool isExternalData = startsWith(request.getContentType().data(), "multipart/form-data");
+
+    if (isExternalData)
+    {
+        /// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
+        reserved_param_suffixes.reserve(3);
+        /// It is a bug and ambiguity with `date_time_input_format` and `low_cardinality_allow_in_native_format` formats/settings.
+        reserved_param_suffixes.emplace_back("_format");
+        reserved_param_suffixes.emplace_back("_types");
+        reserved_param_suffixes.emplace_back("_structure");
+    }
+
     SettingsChanges settings_changes;
-    for (auto it = params.begin(); it != params.end(); ++it)
+    for (const auto & [key, value] : params)
     {
-        if (it->first == "database")
+        if (key == "database")
         {
-            context.setCurrentDatabase(it->second);
+            context.setCurrentDatabase(value);
         }
-        else if (it->first == "default_format")
+        else if (key == "default_format")
         {
-            context.setDefaultFormat(it->second);
+            context.setDefaultFormat(value);
         }
-        else if (param_could_be_skipped(it->first))
+        else if (param_could_be_skipped(key))
         {
         }
         else
         {
             /// All other query parameters are treated as settings.
-            settings_changes.push_back({it->first, it->second});
+            settings_changes.push_back({key, value});
         }
     }
 
+    /// For external data we also want settings
     context.checkSettingsConstraints(settings_changes);
     context.applySettingsChanges(settings_changes);
 
+    /// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
+    std::string full_query;
+
+    /// Support for "external data for query processing".
+    if (isExternalData)
+    {
+        ExternalTablesHandler handler(context, params);
+        params.load(request, istr, handler);
+
+        /// Params are of both form params POST and uri (GET params)
+        for (const auto & it : params)
+            if (it.first == "query")
+                full_query += it.second;
+
+        in = std::make_unique<ReadBufferFromString>(full_query);
+    }
+    else
+        in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
+
+
     /// HTTP response compression is turned on only if the client signalled that they support it
     /// (using Accept-Encoding header) and 'enable_http_compression' setting is turned on.
     used_output.out->setCompression(client_supports_http_compression && settings.enable_http_compression);

dbms/programs/server/MySQLHandler.cpp

@@ -1,5 +1,5 @@
 #include <DataStreams/copyData.h>
-#include <IO/ReadBufferFromMemory.h>
+#include <IO/ReadBufferFromString.h>
 #include <IO/ReadBufferFromPocoSocket.h>
 #include <IO/WriteBufferFromPocoSocket.h>
 #include <Interpreters/executeQuery.h>
@@ -9,12 +9,14 @@
 #include <Columns/ColumnVector.h>
 #include <Common/config_version.h>
 #include <Common/NetException.h>
+#include <Common/OpenSSLHelpers.h>
 #include <Poco/Crypto/RSAKey.h>
 #include <Poco/Crypto/CipherFactory.h>
 #include <Poco/Net/SecureStreamSocket.h>
 #include <Poco/Net/SSLManager.h>
 #include "MySQLHandler.h"
 #include <limits>
+#include <ext/scope_guard.h>
 
 
 namespace DB
@@ -25,21 +27,32 @@ using Poco::Net::SSLManager;
 
 namespace ErrorCodes
 {
-extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
+    extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
-extern const int UNKNOWN_EXCEPTION;
+    extern const int OPENSSL_ERROR;
 }
 
-uint32_t MySQLHandler::last_connection_id = 0;
+MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id)
-
+    : Poco::Net::TCPServerConnection(socket_)
+    , server(server_)
+    , log(&Poco::Logger::get("MySQLHandler"))
+    , connection_context(server.context())
+    , connection_id(connection_id)
+    , public_key(public_key)
+    , private_key(private_key)
+{
+    server_capability_flags = CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA | CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF;
+    if (ssl_enabled)
+        server_capability_flags |= CLIENT_SSL;
+}
 
 void MySQLHandler::run()
 {
     connection_context = server.context();
-    connection_context.setDefaultFormat("MySQL");
+    connection_context.setDefaultFormat("MySQLWire");
 
     in = std::make_shared<ReadBufferFromPocoSocket>(socket());
     out = std::make_shared<WriteBufferFromPocoSocket>(socket());
-    packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler");
+    packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
 
     try
     {
@@ -49,8 +62,7 @@ void MySQLHandler::run()
          *  This plugin must do the same to stay consistent with historical behavior if it is set to operate as a default plugin.
          *  https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L3994
          */
-        Handshake handshake(connection_id, VERSION_STRING, scramble + '\0');
+        Handshake handshake(server_capability_flags, connection_id, VERSION_STRING + String("-") + VERSION_NAME, scramble + '\0');
-
         packet_sender->sendPacket<Handshake>(handshake, true);
 
         LOG_TRACE(log, "Sent handshake");
@@ -78,15 +90,11 @@ void MySQLHandler::run()
                                         << "\nauth_plugin_name: "
                                         << handshake_response.auth_plugin_name);
 
-        capabilities = handshake_response.capability_flags;
+        client_capability_flags = handshake_response.capability_flags;
-        if (!(capabilities & CLIENT_PROTOCOL_41))
+        if (!(client_capability_flags & CLIENT_PROTOCOL_41))
-        {
             throw Exception("Required capability: CLIENT_PROTOCOL_41.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
-        }
+        if (!(client_capability_flags & CLIENT_PLUGIN_AUTH))
-        if (!(capabilities & CLIENT_PLUGIN_AUTH))
-        {
             throw Exception("Required capability: CLIENT_PLUGIN_AUTH.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
-        }
 
         authenticate(handshake_response, scramble);
         OK_Packet ok_packet(0, handshake_response.capability_flags, 0, 0, 0);
@@ -165,7 +173,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
     };
     read_bytes(3); /// We can find out whether it is SSLRequest of HandshakeResponse by first 3 bytes.
 
-    size_t payload_size = *reinterpret_cast<uint32_t *>(buf) & 0xFFFFFFu;
+    size_t payload_size = unalignedLoad<uint32_t>(buf) & 0xFFFFFFu;
     LOG_TRACE(log, "payload size: " << payload_size);
 
     if (payload_size == SSL_REQUEST_PAYLOAD_SIZE)
@@ -180,7 +188,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
         in = std::make_shared<ReadBufferFromPocoSocket>(*ss);
         out = std::make_shared<WriteBufferFromPocoSocket>(*ss);
         connection_context.sequence_id = 2;
-        packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler");
+        packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
         packet_sender->max_packet_size = connection_context.max_packet_size;
         packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket.
     }
@@ -216,6 +224,10 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
     if (handshake_response.auth_plugin_name != Authentication::SHA256)
     {
         packet_sender->sendPacket(AuthSwitchRequest(Authentication::SHA256, scramble + '\0'), true);
+        if (in->eof())
+            throw Exception(
+                "Client doesn't support authentication method " + String(Authentication::SHA256) + " used by ClickHouse",
+                ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
         packet_sender->receivePacket(response);
         auth_response = response.value;
         LOG_TRACE(log, "Authentication method mismatch.");
@@ -226,31 +238,19 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
         LOG_TRACE(log, "Authentication method match.");
     }
 
-    auto getOpenSSLError = []() -> String
-    {
-        BIO * mem = BIO_new(BIO_s_mem());
-        ERR_print_errors(mem);
-        char * buf = nullptr;
-        long size = BIO_get_mem_data(mem, &buf);
-        String errors_str(buf, size);
-        BIO_free(mem);
-        return errors_str;
-    };
-
     if (auth_response == "\1")
     {
         LOG_TRACE(log, "Client requests public key.");
 
         BIO * mem = BIO_new(BIO_s_mem());
-        if (PEM_write_bio_RSA_PUBKEY(mem, public_key) != 1)
+        SCOPE_EXIT(BIO_free(mem));
+        if (PEM_write_bio_RSA_PUBKEY(mem, &public_key) != 1)
         {
-            LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError());
+            throw Exception("Failed to write public key to memory. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
-            throw Exception("Failed to write public key to memory.", ErrorCodes::UNKNOWN_EXCEPTION);
         }
         char * pem_buf = nullptr;
         long pem_size = BIO_get_mem_data(mem, &pem_buf);
         String pem(pem_buf, pem_size);
-        BIO_free(mem);
 
         LOG_TRACE(log, "Key: " << pem);
 
@@ -271,17 +271,16 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
      *  an empty packet is a blank password, thus the check for auth_response.empty() has to be made too.
      *  https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L4017
      */
-    if (!secure_connection && (!auth_response.empty() && auth_response != "\0"))
+    if (!secure_connection && !auth_response.empty() && auth_response != String("\0", 1))
     {
         LOG_TRACE(log, "Received nonempty password");
         auto ciphertext = reinterpret_cast<unsigned char *>(auth_response.data());
 
-        unsigned char plaintext[RSA_size(private_key)];
+        unsigned char plaintext[RSA_size(&private_key)];
-        int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, private_key, RSA_PKCS1_OAEP_PADDING);
+        int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, &private_key, RSA_PKCS1_OAEP_PADDING);
         if (plaintext_size == -1)
         {
-            LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError());
+            throw Exception("Failed to decrypt auth data. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
-            throw Exception("Failed to decrypt.", ErrorCodes::UNKNOWN_EXCEPTION);
         }
 
         password.resize(plaintext_size);
@@ -324,7 +323,7 @@ void MySQLHandler::comInitDB(const String & payload)
     String database = payload.substr(1);
     LOG_DEBUG(log, "Setting current database to " << database);
     connection_context.setCurrentDatabase(database);
-    packet_sender->sendPacket(OK_Packet(0, capabilities, 0, 0, 1), true);
+    packet_sender->sendPacket(OK_Packet(0, client_capability_flags, 0, 0, 1), true);
 }
 
 void MySQLHandler::comFieldList(const String & payload)
@@ -340,12 +339,12 @@ void MySQLHandler::comFieldList(const String & payload)
         );
         packet_sender->sendPacket(column_definition);
     }
-    packet_sender->sendPacket(OK_Packet(0xfe, capabilities, 0, 0, 0), true);
+    packet_sender->sendPacket(OK_Packet(0xfe, client_capability_flags, 0, 0, 0), true);
 }
 
 void MySQLHandler::comPing()
 {
-    packet_sender->sendPacket(OK_Packet(0x0, capabilities, 0, 0, 0), true);
+    packet_sender->sendPacket(OK_Packet(0x0, client_capability_flags, 0, 0, 0), true);
 }
 
 void MySQLHandler::comQuery(const String & payload)
@@ -354,10 +353,18 @@ void MySQLHandler::comQuery(const String & payload)
     std::function<void(const String &)> set_content_type = [&with_output](const String &) -> void {
         with_output = true;
     };
-    ReadBufferFromMemory query(payload.data() + 1, payload.size() - 1);
+
-    executeQuery(query, *out, true, connection_context, set_content_type, nullptr);
+    String query = payload.substr(1);
+
+    // Translate query from MySQL to ClickHouse.
+    // This is a temporary workaround until ClickHouse supports the syntax "@@var_name".
+    if (query == "select @@version_comment limit 1")  // MariaDB client starts session with that query
+        query = "select ''";
+
+    ReadBufferFromString buf(query);
+    executeQuery(buf, *out, true, connection_context, set_content_type, nullptr);
     if (!with_output)
-        packet_sender->sendPacket(OK_Packet(0x00, capabilities, 0, 0, 0), true);
+        packet_sender->sendPacket(OK_Packet(0x00, client_capability_flags, 0, 0, 0), true);
 }
 
 }

dbms/programs/server/MySQLHandler.h

@@ -4,7 +4,7 @@
 #include <Poco/Net/SecureStreamSocket.h>
 #include <Common/getFQDNOrHostName.h>
 #include <Core/MySQLProtocol.h>
-#include <openssl/evp.h>
+#include <openssl/rsa.h>
 #include "IServer.h"
 
 
@@ -15,21 +15,7 @@ namespace DB
 class MySQLHandler : public Poco::Net::TCPServerConnection
 {
 public:
-    MySQLHandler(
+    MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id);
-        IServer & server_,
-        const Poco::Net::StreamSocket & socket_,
-        RSA * public_key,
-        RSA * private_key)
-            : Poco::Net::TCPServerConnection(socket_)
-            , server(server_)
-            , log(&Poco::Logger::get("MySQLHandler"))
-            , connection_context(server.context())
-            , connection_id(last_connection_id++)
-            , public_key(public_key)
-            , private_key(private_key)
-    {
-        log->setLevel("information");
-    }
 
     void run() final;
 
@@ -55,13 +41,13 @@ private:
 
     std::shared_ptr<MySQLProtocol::PacketSender> packet_sender;
 
-    uint32_t connection_id = 0;
+    size_t connection_id = 0;
 
-    uint32_t capabilities;
+    size_t server_capability_flags;
+    size_t client_capability_flags;
 
-    static uint32_t last_connection_id;
+    RSA & public_key;
-
+    RSA & private_key;
-    RSA * public_key, * private_key;
 
     std::shared_ptr<ReadBuffer> in;
     std::shared_ptr<WriteBuffer> out;

dbms/programs/server/MySQLHandlerFactory.cpp

@@ -0,0 +1,124 @@
+#include <Common/OpenSSLHelpers.h>
+#include <Poco/Crypto/X509Certificate.h>
+#include <Poco/Net/SSLManager.h>
+#include <Poco/Net/TCPServerConnectionFactory.h>
+#include <Poco/Util/Application.h>
+#include <common/logger_useful.h>
+#include <ext/scope_guard.h>
+#include "IServer.h"
+#include "MySQLHandler.h"
+#include "MySQLHandlerFactory.h"
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int CANNOT_OPEN_FILE;
+    extern const int NO_ELEMENTS_IN_CONFIG;
+    extern const int OPENSSL_ERROR;
+    extern const int SYSTEM_ERROR;
+}
+
+MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
+    : server(server_)
+    , log(&Logger::get("MySQLHandlerFactory"))
+{
+    try
+    {
+        Poco::Net::SSLManager::instance().defaultServerContext();
+    }
+    catch (...)
+    {
+        LOG_INFO(log, "Failed to create SSL context. SSL will be disabled. Error: " << getCurrentExceptionMessage(false));
+        ssl_enabled = false;
+    }
+
+    /// Reading rsa keys for SHA256 authentication plugin.
+    try
+    {
+        readRSAKeys();
+    }
+    catch (...)
+    {
+        LOG_WARNING(log, "Failed to read RSA keys. Error: " << getCurrentExceptionMessage(false));
+        generateRSAKeys();
+    }
+}
+
+void MySQLHandlerFactory::readRSAKeys()
+{
+    const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
+    String certificateFileProperty = "openSSL.server.certificateFile";
+    String privateKeyFileProperty = "openSSL.server.privateKeyFile";
+
+    if (!config.has(certificateFileProperty))
+        throw Exception("Certificate file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
+
+    if (!config.has(privateKeyFileProperty))
+        throw Exception("Private key file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
+
+    {
+        String certificateFile = config.getString(certificateFileProperty);
+        FILE * fp = fopen(certificateFile.data(), "r");
+        if (fp == nullptr)
+            throw Exception("Cannot open certificate file: " + certificateFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
+        SCOPE_EXIT(fclose(fp));
+
+        X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
+        SCOPE_EXIT(X509_free(x509));
+        if (x509 == nullptr)
+            throw Exception("Failed to read PEM certificate from " + certificateFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+
+        EVP_PKEY * p = X509_get_pubkey(x509);
+        if (p == nullptr)
+            throw Exception("Failed to get RSA key from X509. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+        SCOPE_EXIT(EVP_PKEY_free(p));
+
+        public_key.reset(EVP_PKEY_get1_RSA(p));
+        if (public_key.get() == nullptr)
+            throw Exception("Failed to get RSA key from ENV_PKEY. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+    }
+
+    {
+        String privateKeyFile = config.getString(privateKeyFileProperty);
+
+        FILE * fp = fopen(privateKeyFile.data(), "r");
+        if (fp == nullptr)
+            throw Exception ("Cannot open private key file " + privateKeyFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
+        SCOPE_EXIT(fclose(fp));
+
+        private_key.reset(PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr));
+        if (!private_key)
+            throw Exception("Failed to read RSA private key from " + privateKeyFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+    }
+}
+
+void MySQLHandlerFactory::generateRSAKeys()
+{
+    LOG_INFO(log, "Generating new RSA key.");
+    public_key.reset(RSA_new());
+    if (!public_key)
+        throw Exception("Failed to allocate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+
+    BIGNUM * e = BN_new();
+    if (!e)
+        throw Exception("Failed to allocate BIGNUM. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+    SCOPE_EXIT(BN_free(e));
+
+    if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(public_key.get(), 2048, e, nullptr))
+        throw Exception("Failed to generate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+
+    private_key.reset(RSAPrivateKey_dup(public_key.get()));
+    if (!private_key)
+        throw Exception("Failed to copy RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
+}
+
+Poco::Net::TCPServerConnection * MySQLHandlerFactory::createConnection(const Poco::Net::StreamSocket & socket)
+{
+    size_t connection_id = last_connection_id++;
+    LOG_TRACE(log, "MySQL connection. Id: " << connection_id << ". Address: " << socket.peerAddress().toString());
+    return new MySQLHandler(server, socket, *public_key, *private_key, ssl_enabled, connection_id);
+}
+
+}

dbms/programs/server/MySQLHandlerFactory.h

@@ -1,16 +1,9 @@
 #pragma once
 
 #include <Poco/Net/TCPServerConnectionFactory.h>
-#include <Poco/Net/SSLManager.h>
+#include <atomic>
-#include <Poco/Crypto/X509Certificate.h>
+#include <openssl/rsa.h>
-#include <common/logger_useful.h>
 #include "IServer.h"
-#include "MySQLHandler.h"
-
-namespace Poco
-{
-class Logger;
-}
 
 namespace DB
 {
@@ -20,94 +13,27 @@ class MySQLHandlerFactory : public Poco::Net::TCPServerConnectionFactory
 private:
     IServer & server;
     Poco::Logger * log;
-    RSA * public_key = nullptr, * private_key = nullptr;
 
+    struct RSADeleter
+    {
+        void operator()(RSA * ptr) { RSA_free(ptr); }
+    };
+    using RSAPtr = std::unique_ptr<RSA, RSADeleter>;
+
+    RSAPtr public_key;
+    RSAPtr private_key;
+
+    bool ssl_enabled = true;
+
+    std::atomic<size_t> last_connection_id = 0;
 public:
-    explicit MySQLHandlerFactory(IServer & server_)
+    explicit MySQLHandlerFactory(IServer & server_);
-        : server(server_), log(&Logger::get("MySQLHandlerFactory"))
-    {
-        /// Reading rsa keys for SHA256 authentication plugin.
-        const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
-        String certificateFileProperty = "openSSL.server.certificateFile";
-        String privateKeyFileProperty = "openSSL.server.privateKeyFile";
 
-        if (!config.has(certificateFileProperty))
+    void readRSAKeys();
-        {
-            LOG_INFO(log, "Certificate file is not set.");
-            generateRSAKeys();
-            return;
-        }
-        if (!config.has(privateKeyFileProperty))
-        {
-            LOG_INFO(log, "Private key file is not set.");
-            generateRSAKeys();
-            return;
-        }
 
-        String certificateFile = config.getString(certificateFileProperty);
+    void generateRSAKeys();
-        FILE * fp = fopen(certificateFile.data(), "r");
-        if (fp == nullptr)
-        {
-            LOG_WARNING(log, "Cannot open certificate file: " << certificateFile << ".");
-            generateRSAKeys();
-            return;
-        }
-        X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
-        EVP_PKEY * p = X509_get_pubkey(x509);
-        public_key = EVP_PKEY_get1_RSA(p);
-        X509_free(x509);
-        EVP_PKEY_free(p);
-        fclose(fp);
 
-        String privateKeyFile = config.getString(privateKeyFileProperty);
+    Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override;
-        fp = fopen(privateKeyFile.data(), "r");
-        if (fp == nullptr)
-        {
-            LOG_WARNING(log, "Cannot open private key file " << privateKeyFile << ".");
-            generateRSAKeys();
-            return;
-        }
-        private_key = PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr);
-        fclose(fp);
-    }
-
-    void generateRSAKeys()
-    {
-        LOG_INFO(log, "Generating new RSA key.");
-        RSA * rsa = RSA_new();
-        if (rsa == nullptr)
-        {
-            throw Exception("Failed to allocate RSA key.", 1002);
-        }
-        BIGNUM * e = BN_new();
-        if (!e)
-        {
-            RSA_free(rsa);
-            throw Exception("Failed to allocate BIGNUM.", 1002);
-        }
-        if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(rsa, 2048, e, nullptr))
-        {
-            RSA_free(rsa);
-            BN_free(e);
-            throw Exception("Failed to generate RSA key.", 1002);
-        }
-        BN_free(e);
-
-        public_key = rsa;
-        private_key = RSAPrivateKey_dup(rsa);
-    }
-
-    Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override
-    {
-        LOG_TRACE(log, "MySQL connection. Address: " << socket.peerAddress().toString());
-        return new MySQLHandler(server, socket, public_key, private_key);
-    }
-
-    ~MySQLHandlerFactory() override
-    {
-        RSA_free(public_key);
-        RSA_free(private_key);
-    }
 };
 
 }

dbms/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h

@@ -1,10 +1,10 @@
 #pragma once
 
-#include <roaring.h>
+#include <roaring/roaring.h>
 #include <IO/ReadHelpers.h>
 #include <IO/WriteHelpers.h>
 #include <boost/noncopyable.hpp>
-#include <roaring.hh>
+#include <roaring/roaring.hh>
 #include <Common/HashTable/SmallTable.h>
 #include <Common/PODArray.h>
 

dbms/src/AggregateFunctions/AggregateFunctionMLMethod.cpp

@@ -3,6 +3,8 @@
 #include <IO/ReadHelpers.h>
 #include <IO/WriteHelpers.h>
 #include <Interpreters/castColumn.h>
+#include <Columns/ColumnArray.h>
+#include <Columns/ColumnTuple.h>
 #include <Common/FieldVisitors.h>
 #include <Common/typeid_cast.h>
 #include "AggregateFunctionFactory.h"
@@ -44,10 +46,10 @@ namespace
         /// Such default parameters were picked because they did good on some tests,
         /// though it still requires to fit parameters to achieve better result
         auto learning_rate = Float64(0.01);
-        auto l2_reg_coef = Float64(0.01);
+        auto l2_reg_coef = Float64(0.1);
-        UInt32 batch_size = 1;
+        UInt32 batch_size = 15;
 
-        std::shared_ptr<IWeightsUpdater> weights_updater = std::make_shared<StochasticGradientDescent>();
+        std::string weights_updater_name = "\'SGD\'";
         std::shared_ptr<IGradientComputer> gradient_computer;
 
         if (!parameters.empty())
@@ -64,19 +66,8 @@ namespace
         }
         if (parameters.size() > 3)
         {
-            if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'SGD\'")
+            weights_updater_name = applyVisitor(FieldVisitorToString(), parameters[3]);
-            {
+            if (weights_updater_name != "\'SGD\'" && weights_updater_name != "\'Momentum\'" && weights_updater_name != "\'Nesterov\'")
-                weights_updater = std::make_shared<StochasticGradientDescent>();
-            }
-            else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Momentum\'")
-            {
-                weights_updater = std::make_shared<Momentum>();
-            }
-            else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Nesterov\'")
-            {
-                weights_updater = std::make_shared<Nesterov>();
-            }
-            else
             {
                 throw Exception("Invalid parameter for weights updater", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
             }
@@ -98,20 +89,19 @@ namespace
         return std::make_shared<Method>(
             argument_types.size() - 1,
             gradient_computer,
-            weights_updater,
+            weights_updater_name,
             learning_rate,
             l2_reg_coef,
             batch_size,
             argument_types,
             parameters);
     }
-
 }
 
 void registerAggregateFunctionMLMethod(AggregateFunctionFactory & factory)
 {
-    factory.registerFunction("linearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>);
+    factory.registerFunction("stochasticLinearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>);
-    factory.registerFunction("logisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>);
+    factory.registerFunction("stochasticLogisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>);
 }
 
 LinearModelData::LinearModelData(
@@ -144,9 +134,34 @@ void LinearModelData::update_state()
 }
 
 void LinearModelData::predict(
-    ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const
+    ColumnVector<Float64>::Container & container,
+    Block & block,
+    size_t offset,
+    size_t limit,
+    const ColumnNumbers & arguments,
+    const Context & context) const
 {
-    gradient_computer->predict(container, block, arguments, weights, bias, context);
+    gradient_computer->predict(container, block, offset, limit, arguments, weights, bias, context);
+}
+
+void LinearModelData::returnWeights(IColumn & to) const
+{
+    size_t size = weights.size() + 1;
+
+    ColumnArray & arr_to = static_cast<ColumnArray &>(to);
+    ColumnArray::Offsets & offsets_to = arr_to.getOffsets();
+
+    size_t old_size = offsets_to.back();
+    offsets_to.push_back(old_size + size);
+
+    typename ColumnFloat64::Container & val_to
+            = static_cast<ColumnFloat64 &>(arr_to.getData()).getData();
+
+    val_to.reserve(old_size + size);
+    for (size_t i = 0; i + 1 < size; ++i)
+        val_to.push_back(weights[i]);
+
+    val_to.push_back(bias);
 }
 
 void LinearModelData::read(ReadBuffer & buf)
@@ -192,7 +207,8 @@ void LinearModelData::merge(const DB::LinearModelData & rhs)
 void LinearModelData::add(const IColumn ** columns, size_t row_num)
 {
     /// first column stores target; features start from (columns + 1)
-    const auto target = (*columns[0])[row_num].get<Float64>();
+    Float64 target = (*columns[0]).getFloat64(row_num);
+
     /// Here we have columns + 1 as first column corresponds to target value, and others - to features
     weights_updater->add_to_batch(
         gradient_batch, *gradient_computer, weights, bias, learning_rate, l2_reg_coef, target, columns + 1, row_num);
@@ -334,42 +350,38 @@ void IWeightsUpdater::add_to_batch(
 void LogisticRegression::predict(
     ColumnVector<Float64>::Container & container,
     Block & block,
+    size_t offset,
+    size_t limit,
     const ColumnNumbers & arguments,
     const std::vector<Float64> & weights,
     Float64 bias,
-    const Context & context) const
+    const Context & /*context*/) const
 {
     size_t rows_num = block.rows();
-    std::vector<Float64> results(rows_num, bias);
+
+    if (offset > rows_num || offset + limit > rows_num)
+        throw Exception("Invalid offset and limit for LogisticRegression::predict. "
+                        "Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
+                        " and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
+
+    std::vector<Float64> results(limit, bias);
 
     for (size_t i = 1; i < arguments.size(); ++i)
     {
         const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
+
         if (!isNativeNumber(cur_col.type))
-        {
             throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
+
+        auto & features_column = cur_col.column;
+
+        for (size_t row_num = 0; row_num < limit; ++row_num)
+            results[row_num] += weights[i - 1] * features_column->getFloat64(offset + row_num);
     }
 
-        /// If column type is already Float64 then castColumn simply returns it
+    container.reserve(container.size() + limit);
-        auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
+    for (size_t row_num = 0; row_num < limit; ++row_num)
-        auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
-
-        if (!features_column)
-        {
-            throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
-        }
-
-        for (size_t row_num = 0; row_num != rows_num; ++row_num)
-        {
-            results[row_num] += weights[i - 1] * features_column->getElement(row_num);
-        }
-    }
-
-    container.reserve(rows_num);
-    for (size_t row_num = 0; row_num != rows_num; ++row_num)
-    {
         container.emplace_back(1 / (1 + exp(-results[row_num])));
-    }
 }
 
 void LogisticRegression::compute(
@@ -385,7 +397,7 @@ void LogisticRegression::compute(
     Float64 derivative = bias;
     for (size_t i = 0; i < weights.size(); ++i)
     {
-        auto value = (*columns[i])[row_num].get<Float64>();
+        auto value = (*columns[i]).getFloat64(row_num);
         derivative += weights[i] * value;
     }
     derivative *= target;
@@ -394,18 +406,20 @@ void LogisticRegression::compute(
     batch_gradient[weights.size()] += learning_rate * target / (derivative + 1);
     for (size_t i = 0; i < weights.size(); ++i)
     {
-        auto value = (*columns[i])[row_num].get<Float64>();
+        auto value = (*columns[i]).getFloat64(row_num);
-        batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * l2_reg_coef * weights[i];
+        batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * learning_rate * l2_reg_coef * weights[i];
     }
 }
 
 void LinearRegression::predict(
     ColumnVector<Float64>::Container & container,
     Block & block,
+    size_t offset,
+    size_t limit,
     const ColumnNumbers & arguments,
     const std::vector<Float64> & weights,
     Float64 bias,
-    const Context & context) const
+    const Context & /*context*/) const
 {
     if (weights.size() + 1 != arguments.size())
     {
@@ -413,36 +427,33 @@ void LinearRegression::predict(
     }
 
     size_t rows_num = block.rows();
-    std::vector<Float64> results(rows_num, bias);
+
+    if (offset > rows_num || offset + limit > rows_num)
+        throw Exception("Invalid offset and limit for LogisticRegression::predict. "
+                        "Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
+                        " and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
+
+    std::vector<Float64> results(limit, bias);
 
     for (size_t i = 1; i < arguments.size(); ++i)
     {
         const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
-        if (!isNativeNumber(cur_col.type))
-        {
-            throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
-        }
 
-        /// If column type is already Float64 then castColumn simply returns it
+        if (!isNativeNumber(cur_col.type))
-        auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
+            throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
-        auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
+
+        auto features_column = cur_col.column;
 
         if (!features_column)
-        {
             throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
+
+        for (size_t row_num = 0; row_num < limit; ++row_num)
+            results[row_num] += weights[i - 1] * features_column->getFloat64(row_num + offset);
     }
 
-        for (size_t row_num = 0; row_num != rows_num; ++row_num)
+    container.reserve(container.size() + limit);
-        {
+    for (size_t row_num = 0; row_num < limit; ++row_num)
-            results[row_num] += weights[i - 1] * features_column->getElement(row_num);
-        }
-    }
-
-    container.reserve(rows_num);
-    for (size_t row_num = 0; row_num != rows_num; ++row_num)
-    {
         container.emplace_back(results[row_num]);
-    }
 }
 
 void LinearRegression::compute(
@@ -458,7 +469,7 @@ void LinearRegression::compute(
     Float64 derivative = (target - bias);
     for (size_t i = 0; i < weights.size(); ++i)
     {
-        auto value = (*columns[i])[row_num].get<Float64>();
+        auto value = (*columns[i]).getFloat64(row_num);
         derivative -= weights[i] * value;
     }
     derivative *= (2 * learning_rate);
@@ -466,8 +477,8 @@ void LinearRegression::compute(
     batch_gradient[weights.size()] += derivative;
     for (size_t i = 0; i < weights.size(); ++i)
     {
-        auto value = (*columns[i])[row_num].get<Float64>();
+        auto value = (*columns[i]).getFloat64(row_num);
-        batch_gradient[i] += derivative * value - 2 * l2_reg_coef * weights[i];
+        batch_gradient[i] += derivative * value - 2 * learning_rate * l2_reg_coef * weights[i];
     }
 }
 

dbms/src/AggregateFunctions/AggregateFunctionMLMethod.h

@@ -3,7 +3,10 @@
 #include <Columns/ColumnVector.h>
 #include <Columns/ColumnsCommon.h>
 #include <Columns/ColumnsNumber.h>
+#include <Common/typeid_cast.h>
 #include <DataTypes/DataTypesNumber.h>
+#include <DataTypes/DataTypeTuple.h>
+#include <DataTypes/DataTypeArray.h>
 #include "IAggregateFunction.h"
 
 namespace DB
@@ -12,6 +15,7 @@ namespace ErrorCodes
 {
     extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
     extern const int BAD_ARGUMENTS;
+    extern const int BAD_CAST;
 }
 
 /**
@@ -39,6 +43,8 @@ public:
     virtual void predict(
         ColumnVector<Float64>::Container & container,
         Block & block,
+        size_t offset,
+        size_t limit,
         const ColumnNumbers & arguments,
         const std::vector<Float64> & weights,
         Float64 bias,
@@ -64,6 +70,8 @@ public:
     void predict(
         ColumnVector<Float64>::Container & container,
         Block & block,
+        size_t offset,
+        size_t limit,
         const ColumnNumbers & arguments,
         const std::vector<Float64> & weights,
         Float64 bias,
@@ -89,6 +97,8 @@ public:
     void predict(
         ColumnVector<Float64>::Container & container,
         Block & block,
+        size_t offset,
+        size_t limit,
         const ColumnNumbers & arguments,
         const std::vector<Float64> & weights,
         Float64 bias,
@@ -215,20 +225,26 @@ public:
 
     void read(ReadBuffer & buf);
 
-    void
+    void predict(
-    predict(ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const;
+        ColumnVector<Float64>::Container & container,
+        Block & block,
+        size_t offset,
+        size_t limit,
+        const ColumnNumbers & arguments,
+        const Context & context) const;
 
+    void returnWeights(IColumn & to) const;
 private:
     std::vector<Float64> weights;
     Float64 bias{0.0};
 
     Float64 learning_rate;
     Float64 l2_reg_coef;
-    UInt32 batch_capacity;
+    UInt64 batch_capacity;
 
-    UInt32 iter_num = 0;
+    UInt64 iter_num = 0;
     std::vector<Float64> gradient_batch;
-    UInt32 batch_size;
+    UInt64 batch_size;
 
     std::shared_ptr<IGradientComputer> gradient_computer;
     std::shared_ptr<IWeightsUpdater> weights_updater;
@@ -253,7 +269,7 @@ public:
     explicit AggregateFunctionMLMethod(
         UInt32 param_num,
         std::shared_ptr<IGradientComputer> gradient_computer,
-        std::shared_ptr<IWeightsUpdater> weights_updater,
+        std::string weights_updater_name,
         Float64 learning_rate,
         Float64 l2_reg_coef,
         UInt32 batch_size,
@@ -265,15 +281,39 @@ public:
         , l2_reg_coef(l2_reg_coef)
         , batch_size(batch_size)
         , gradient_computer(std::move(gradient_computer))
-        , weights_updater(std::move(weights_updater))
+        , weights_updater_name(std::move(weights_updater_name))
     {
     }
 
-    DataTypePtr getReturnType() const override { return std::make_shared<DataTypeNumber<Float64>>(); }
+    /// This function is called when SELECT linearRegression(...) is called
+    DataTypePtr getReturnType() const override
+    {
+        return std::make_shared<DataTypeArray>(std::make_shared<DataTypeFloat64>());
+    }
+
+    /// This function is called from evalMLMethod function for correct predictValues call
+    DataTypePtr getReturnTypeToPredict() const override
+    {
+        return std::make_shared<DataTypeNumber<Float64>>();
+    }
 
     void create(AggregateDataPtr place) const override
     {
-        new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, weights_updater);
+        std::shared_ptr<IWeightsUpdater> new_weights_updater;
+        if (weights_updater_name == "\'SGD\'")
+        {
+            new_weights_updater = std::make_shared<StochasticGradientDescent>();
+        } else if (weights_updater_name == "\'Momentum\'")
+        {
+            new_weights_updater = std::make_shared<Momentum>();
+        } else if (weights_updater_name == "\'Nesterov\'")
+        {
+            new_weights_updater = std::make_shared<Nesterov>();
+        } else
+        {
+            throw Exception("Illegal name of weights updater (should have been checked earlier)", ErrorCodes::LOGICAL_ERROR);
+        }
+        new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, new_weights_updater);
     }
 
     void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
@@ -288,7 +328,13 @@ public:
     void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override { this->data(place).read(buf); }
 
     void predictValues(
-        ConstAggregateDataPtr place, IColumn & to, Block & block, const ColumnNumbers & arguments, const Context & context) const override
+        ConstAggregateDataPtr place,
+        IColumn & to,
+        Block & block,
+        size_t offset,
+        size_t limit,
+        const ColumnNumbers & arguments,
+        const Context & context) const override
     {
         if (arguments.size() != param_num + 1)
             throw Exception(
@@ -296,16 +342,21 @@ public:
                     + ". Required: " + std::to_string(param_num + 1),
                 ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
 
-        auto & column = dynamic_cast<ColumnVector<Float64> &>(to);
+        /// This cast might be correct because column type is based on getReturnTypeToPredict.
+        auto * column = typeid_cast<ColumnFloat64 *>(&to);
+        if (!column)
+            throw Exception("Cast of column of predictions is incorrect. getReturnTypeToPredict must return same value as it is casted to",
+                            ErrorCodes::BAD_CAST);
 
-        this->data(place).predict(column.getData(), block, arguments, context);
+        this->data(place).predict(column->getData(), block, offset, limit, arguments, context);
     }
 
+    /** This function is called if aggregate function without State modifier is selected in a query.
+     *  Inserts all weights of the model into the column 'to', so user may use such information if needed
+     */
     void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
     {
-        std::ignore = place;
+        this->data(place).returnWeights(to);
-        std::ignore = to;
-        throw std::runtime_error("not implemented");
     }
 
     const char * getHeaderFilePath() const override { return __FILE__; }
@@ -316,15 +367,15 @@ private:
     Float64 l2_reg_coef;
     UInt32 batch_size;
     std::shared_ptr<IGradientComputer> gradient_computer;
-    std::shared_ptr<IWeightsUpdater> weights_updater;
+    std::string weights_updater_name;
 };
 
 struct NameLinearRegression
 {
-    static constexpr auto name = "linearRegression";
+    static constexpr auto name = "stochasticLinearRegression";
 };
 struct NameLogisticRegression
 {
-    static constexpr auto name = "logisticRegression";
+    static constexpr auto name = "stochasticLogisticRegression";
 };
 }

dbms/src/AggregateFunctions/IAggregateFunction.h

@@ -48,6 +48,12 @@ public:
     /// Get the result type.
     virtual DataTypePtr getReturnType() const = 0;
 
+    /// Get type which will be used for prediction result in case if function is an ML method.
+    virtual DataTypePtr getReturnTypeToPredict() const
+    {
+        throw Exception("Prediction is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
+    }
+
     virtual ~IAggregateFunction() {}
 
     /** Data manipulating functions. */
@@ -94,9 +100,16 @@ public:
     /// Inserts results into a column.
     virtual void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const = 0;
 
-    /// This function is used for machine learning methods
+    /// Used for machine learning methods. Predict result from trained model.
-    virtual void predictValues(ConstAggregateDataPtr /* place */, IColumn & /*to*/,
+    /// Will insert result into `to` column for rows in range [offset, offset + limit).
-                               Block & /*block*/, const ColumnNumbers & /*arguments*/, const Context & /*context*/) const
+    virtual void predictValues(
+        ConstAggregateDataPtr /* place */,
+        IColumn & /*to*/,
+        Block & /*block*/,
+        size_t /*offset*/,
+        size_t /*limit*/,
+        const ColumnNumbers & /*arguments*/,
+        const Context & /*context*/) const
     {
         throw Exception("Method predictValues is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
     }

dbms/src/Columns/ColumnAggregateFunction.cpp

@@ -35,25 +35,6 @@ void ColumnAggregateFunction::addArena(ArenaPtr arena_)
     arenas.push_back(arena_);
 }
 
-/// This function is used in convertToValues() and predictValues()
-/// and is written here to avoid repetitions
-bool ColumnAggregateFunction::tryFinalizeAggregateFunction(MutableColumnPtr *res_) const
-{
-    if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
-    {
-        auto res = createView();
-        res->set(function_state->getNestedFunction());
-        res->data.assign(data.begin(), data.end());
-        *res_ = std::move(res);
-        return true;
-    }
-
-    MutableColumnPtr res = func->getReturnType()->createColumn();
-    res->reserve(data.size());
-    *res_ = std::move(res);
-    return false;
-}
-
 MutableColumnPtr ColumnAggregateFunction::convertToValues() const
 {
     /** If the aggregate function returns an unfinalized/unfinished state,
@@ -86,17 +67,17 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
         *   AggregateFunction(quantileTiming(0.5), UInt64)
         * into UInt16 - already finished result of `quantileTiming`.
         */
-
+    if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
-    /** Convertion function is used in convertToValues and predictValues
-        * in the similar part of both functions
-        */
-
-    MutableColumnPtr res;
-    if (tryFinalizeAggregateFunction(&res))
     {
+        auto res = createView();
+        res->set(function_state->getNestedFunction());
+        res->data.assign(data.begin(), data.end());
         return res;
     }
 
+    MutableColumnPtr res = func->getReturnType()->createColumn();
+    res->reserve(data.size());
+
     for (auto val : data)
         func->insertResultInto(val, *res);
 
@@ -105,19 +86,27 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
 
 MutableColumnPtr ColumnAggregateFunction::predictValues(Block & block, const ColumnNumbers & arguments, const Context & context) const
 {
-    MutableColumnPtr res;
+    MutableColumnPtr res = func->getReturnTypeToPredict()->createColumn();
-    tryFinalizeAggregateFunction(&res);
+    res->reserve(data.size());
 
     auto ML_function = func.get();
     if (ML_function)
     {
+        if (data.size() == 1)
+        {
+            /// Case for const column. Predict using single model.
+            ML_function->predictValues(data[0], *res, block, 0, block.rows(), arguments, context);
+        }
+        else
+        {
+            /// Case for non-constant column. Use different aggregate function for each row.
             size_t row_num = 0;
             for (auto val : data)
             {
-            ML_function->predictValues(val, *res, block, arguments, context);
+                ML_function->predictValues(val, *res, block, row_num, 1, arguments, context);
                 ++row_num;
             }
-
+        }
     }
     else
     {

dbms/src/Columns/ColumnDecimal.cpp

@@ -94,7 +94,7 @@ ColumnPtr ColumnDecimal<T>::permute(const IColumn::Permutation & perm, size_t li
     for (size_t i = 0; i < size; ++i)
         res_data[i] = data[perm[i]];
 
-    return std::move(res);
+    return res;
 }
 
 template <typename T>
@@ -117,7 +117,7 @@ MutableColumnPtr ColumnDecimal<T>::cloneResized(size_t size) const
         }
     }
 
-    return std::move(res);
+    return res;
 }
 
 template <typename T>
@@ -169,7 +169,7 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter & filt, ssize_t result_
         ++data_pos;
     }
 
-    return std::move(res);
+    return res;
 }
 
 template <typename T>
@@ -202,7 +202,7 @@ ColumnPtr ColumnDecimal<T>::replicate(const IColumn::Offsets & offsets) const
             res_data.push_back(data[i]);
     }
 
-    return std::move(res);
+    return res;
 }
 
 template <typename T>

dbms/src/Columns/ColumnDecimal.h

@@ -187,7 +187,7 @@ ColumnPtr ColumnDecimal<T>::indexImpl(const PaddedPODArray<Type> & indexes, size
     for (size_t i = 0; i < limit; ++i)
         res_data[i] = data[indexes[i]];
 
-    return std::move(res);
+    return res;
 }
 
 }

dbms/src/Columns/ColumnLowCardinality.cpp

@@ -349,7 +349,7 @@ ColumnPtr ColumnLowCardinality::countKeys() const
 
     auto counter = ColumnUInt64::create(dict_size, 0);
     idx.countKeys(counter->getData());
-    return std::move(counter);
+    return counter;
 }
 
 

dbms/src/Columns/ColumnVector.cpp

@@ -215,6 +215,12 @@ UInt64 ColumnVector<T>::get64(size_t n) const
     return ext::bit_cast<UInt64>(data[n]);
 }
 
+template <typename T>
+Float64 ColumnVector<T>::getFloat64(size_t n) const
+{
+    return static_cast<Float64>(data[n]);
+}
+
 template <typename T>
 void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length)
 {

dbms/src/Columns/ColumnVector.h

@@ -202,6 +202,8 @@ public:
 
     UInt64 get64(size_t n) const override;
 
+    Float64 getFloat64(size_t n) const override;
+
     UInt64 getUInt(size_t n) const override
     {
         return UInt64(data[n]);

dbms/src/Columns/IColumn.h

@@ -91,6 +91,13 @@ public:
         throw Exception("Method get64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
     }
 
+    /// If column stores native numeric type, it returns n-th element casted to Float64
+    /// Is used in regression methods to cast each features into uniform type
+    virtual Float64 getFloat64(size_t /*n*/) const
+    {
+        throw Exception("Method getFloat64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
+    }
+
     /** If column is numeric, return value of n-th element, casted to UInt64.
       * For NULL values of Nullable column it is allowed to return arbitrary value.
       * Otherwise throw an exception.

dbms/src/Columns/ReverseIndex.h

@@ -375,7 +375,7 @@ ColumnUInt64::MutablePtr ReverseIndex<IndexType, ColumnType>::calcHashes() const
     for (auto row : ext::range(0, size))
         hash->getElement(row) = getHash(column->getDataAt(row));
 
-    return std::move(hash);
+    return hash;
 }
 
 template <typename IndexType, typename ColumnType>

dbms/src/Columns/getLeastSuperColumn.cpp

@@ -0,0 +1,67 @@
+#include <Columns/getLeastSuperColumn.h>
+#include <Columns/IColumn.h>
+#include <Columns/ColumnConst.h>
+#include <DataTypes/getLeastSupertype.h>
+
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int LOGICAL_ERROR;
+}
+
+static bool sameConstants(const IColumn & a, const IColumn & b)
+{
+    return static_cast<const ColumnConst &>(a).getField() == static_cast<const ColumnConst &>(b).getField();
+}
+
+ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns)
+{
+    if (columns.empty())
+        throw Exception("Logical error: no src columns for supercolumn", ErrorCodes::LOGICAL_ERROR);
+
+    ColumnWithTypeAndName result = *columns[0];
+
+    /// Determine common type.
+
+    size_t num_const = 0;
+    DataTypes types(columns.size());
+    for (size_t i = 0; i < columns.size(); ++i)
+    {
+        types[i] = columns[i]->type;
+        if (columns[i]->column->isColumnConst())
+            ++num_const;
+    }
+
+    result.type = getLeastSupertype(types);
+
+    /// Create supertype column saving constness if possible.
+
+    bool save_constness = false;
+    if (columns.size() == num_const)
+    {
+        save_constness = true;
+        for (size_t i = 1; i < columns.size(); ++i)
+        {
+            const ColumnWithTypeAndName & first = *columns[0];
+            const ColumnWithTypeAndName & other = *columns[i];
+
+            if (!sameConstants(*first.column, *other.column))
+            {
+                save_constness = false;
+                break;
+            }
+        }
+    }
+
+    if (save_constness)
+        result.column = result.type->createColumnConst(0, static_cast<const ColumnConst &>(*columns[0]->column).getField());
+    else
+        result.column = result.type->createColumn();
+
+    return result;
+}
+
+}

dbms/src/Columns/getLeastSuperColumn.h

@@ -0,0 +1,12 @@
+#pragma once
+
+#include <Core/ColumnWithTypeAndName.h>
+
+
+namespace DB
+{
+
+/// getLeastSupertype + related column changes
+ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns);
+
+}

dbms/src/Common/ErrorCodes.cpp

@@ -427,8 +427,10 @@ namespace ErrorCodes
     extern const int BAD_TTL_EXPRESSION = 450;
     extern const int BAD_TTL_FILE = 451;
     extern const int SETTING_CONSTRAINT_VIOLATION = 452;
-    extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 453;
+    extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 453;
-
+    extern const int OPENSSL_ERROR = 454;
+    extern const int SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = 455;
+    extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 456;
     extern const int KEEPER_EXCEPTION = 999;
     extern const int POCO_EXCEPTION = 1000;
     extern const int STD_EXCEPTION = 1001;
@@ -437,8 +439,6 @@ namespace ErrorCodes
 
     extern const int CONDITIONAL_TREE_PARENT_NOT_FOUND = 2001;
     extern const int ILLEGAL_PROJECTION_MANIPULATOR = 2002;
-
-    extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 446;
 }
 
 }

dbms/src/Common/OpenSSLHelpers.cpp

@@ -0,0 +1,18 @@
+#include "OpenSSLHelpers.h"
+#include <ext/scope_guard.h>
+#include <openssl/err.h>
+
+namespace DB
+{
+
+String getOpenSSLErrors()
+{
+    BIO * mem = BIO_new(BIO_s_mem());
+    SCOPE_EXIT(BIO_free(mem));
+    ERR_print_errors(mem);
+    char * buf = nullptr;
+    long size = BIO_get_mem_data(mem, &buf);
+    return String(buf, size);
+}
+
+}

dbms/src/Common/OpenSSLHelpers.h

@@ -0,0 +1,12 @@
+#pragma once
+
+#include <Core/Types.h>
+
+
+namespace DB
+{
+
+/// Returns concatenation of error strings for all errors that OpenSSL has recorded, emptying the error queue.
+String getOpenSSLErrors();
+
+}

dbms/src/Common/OptimizedRegularExpression.h

@@ -6,7 +6,7 @@
 #include <Common/config.h>
 #include <re2/re2.h>
 #if USE_RE2_ST
-    #include <re2_st/re2.h> // Y_IGNORE
+    #include <re2_st/re2.h>
 #else
     #define re2_st re2
 #endif

dbms/src/Common/UInt128.h

@@ -71,7 +71,7 @@ template <typename T> bool inline operator>  (T a, const UInt128 b) { return UIn
 template <typename T> bool inline operator<= (T a, const UInt128 b) { return UInt128(a) <= b; }
 template <typename T> bool inline operator<  (T a, const UInt128 b) { return UInt128(a) < b; }
 
-template <> constexpr bool IsNumber<UInt128> = true;
+template <> inline constexpr bool IsNumber<UInt128> = true;
 template <> struct TypeName<UInt128> { static const char * get() { return "UInt128"; } };
 template <> struct TypeId<UInt128> { static constexpr const TypeIndex value = TypeIndex::UInt128; };
 

dbms/src/Common/getNumberOfPhysicalCPUCores.cpp

@@ -7,7 +7,7 @@
 #   include <Common/Exception.h>
     namespace DB { namespace ErrorCodes { extern const int CPUID_ERROR; }}
 #elif USE_CPUINFO
-#   include <cpuinfo.h> // Y_IGNORE
+#   include <cpuinfo.h>
 #endif
 
 

dbms/src/Common/localBackup.cpp

@@ -1,7 +1,9 @@
-#include <Common/localBackup.h>
+#include "localBackup.h"
+
 #include <Common/createHardLink.h>
 #include <Common/Exception.h>
 #include <Poco/DirectoryIterator.h>
+#include <Poco/Path.h>
 #include <Poco/File.h>
 #include <string>
 #include <iostream>

dbms/src/Common/localBackup.h

@@ -1,8 +1,8 @@
 #pragma once
 
-#include <Poco/Path.h>
 #include <optional>
 
+namespace Poco { class Path; }
 
 namespace DB
 {

dbms/src/Core/MySQLProtocol.h

@@ -1,18 +1,16 @@
 #pragma once
 
-#include <IO/ReadBuffer.h>
+#include <Core/Types.h>
-#include <IO/WriteBuffer.h>
 #include <IO/copyData.h>
+#include <IO/ReadBuffer.h>
 #include <IO/ReadBufferFromPocoSocket.h>
+#include <IO/WriteBuffer.h>
 #include <IO/WriteBufferFromPocoSocket.h>
 #include <IO/WriteBufferFromString.h>
-#include <Core/Types.h>
-#include <Poco/RandomStream.h>
 #include <Poco/Net/StreamSocket.h>
+#include <Poco/RandomStream.h>
 #include <random>
 #include <sstream>
-#include <common/logger_useful.h>
-#include <Poco/Logger.h>
 
 /// Implementation of MySQL wire protocol
 
@@ -157,23 +155,19 @@ public:
     size_t max_packet_size = MAX_PACKET_LENGTH;
 
     /// For reading and writing.
-    PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id, const String logger_name)
+    PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id)
         : sequence_id(sequence_id)
         , in(&in)
         , out(&out)
-        , log(&Poco::Logger::get(logger_name))
     {
-        log->setLevel("information");
     }
 
     /// For writing.
-    PacketSender(WriteBuffer & out, size_t & sequence_id, const String logger_name)
+    PacketSender(WriteBuffer & out, size_t & sequence_id)
         : sequence_id(sequence_id)
         , in(nullptr)
         , out(&out)
-        , log(&Poco::Logger::get(logger_name))
     {
-        log->setLevel("information");
     }
 
     String receivePacketPayload()
@@ -186,8 +180,6 @@ public:
         // packets which are larger than or equal to 16MB are splitted
         do
         {
-            LOG_TRACE(log, "Reading from buffer");
-
             in->readStrict(reinterpret_cast<char *>(&payload_length), 3);
 
             if (payload_length > max_packet_size)
@@ -207,8 +199,6 @@ public:
             }
             sequence_id++;
 
-            LOG_TRACE(log, "Received packet. Sequence-id: " << packet_sequence_id << ", payload length: " << payload_length);
-
             copyData(*in, static_cast<WriteBuffer &>(buf), payload_length);
         } while (payload_length == max_packet_size);
 
@@ -230,9 +220,6 @@ public:
         {
             size_t payload_length = std::min(payload.length() - pos, max_packet_size);
 
-            LOG_TRACE(log, "Writing packet of size " << payload_length << " with sequence-id " << static_cast<int>(sequence_id));
-            LOG_TRACE(log, packetToText(payload));
-
             out->write(reinterpret_cast<const char *>(&payload_length), 3);
             out->write(reinterpret_cast<const char *>(&sequence_id), 1);
             out->write(payload.data() + pos, payload_length);
@@ -241,13 +228,9 @@ public:
             sequence_id++;
         } while (pos < payload.length());
 
-        LOG_TRACE(log, "Packet was sent.");
-
         if (flush)
-        {
             out->next();
     }
-    }
 
     /// Sets sequence-id to 0. Must be called before each command phase.
     void resetSequenceId();
@@ -255,8 +238,6 @@ public:
 private:
     /// Converts packet to text. Is used for debug output.
     static String packetToText(String payload);
-
-    Poco::Logger * log;
 };
 
 
@@ -279,12 +260,11 @@ class Handshake : public WritePacket
     uint32_t status_flags;
     String auth_plugin_data;
 public:
-    explicit Handshake(uint32_t connection_id, String server_version, String auth_plugin_data)
+    explicit Handshake(uint32_t capability_flags, uint32_t connection_id, String server_version, String auth_plugin_data)
         : protocol_version(0xa)
         , server_version(std::move(server_version))
         , connection_id(connection_id)
-        , capability_flags(CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA
+        , capability_flags(capability_flags)
-            | CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF | CLIENT_SSL)
         , character_set(CharacterSet::utf8_general_ci)
         , status_flags(0)
         , auth_plugin_data(auth_plugin_data)

dbms/src/Core/Settings.h

@@ -85,6 +85,7 @@ struct Settings : public SettingsCollection<Settings>
     M(SettingFloat, totals_auto_threshold, 0.5, "The threshold for totals_mode = 'auto'.") \
     \
     M(SettingBool, compile, false, "Whether query compilation is enabled.") \
+    M(SettingBool, allow_suspicious_low_cardinality_types, false, "In CREATE TABLE statement allows specifying LowCardinality modifier for types of small fixed size (8 or less). Enabling this may increase merge times and memory consumption.") \
     M(SettingBool, compile_expressions, false, "Compile some scalar functions and operators to native code.") \
     M(SettingUInt64, min_count_to_compile, 3, "The number of structurally identical queries before they are compiled.") \
     M(SettingUInt64, min_count_to_compile_expression, 3, "The number of identical expressions before they are JIT-compiled") \

dbms/src/Core/Types.h

@@ -32,16 +32,16 @@ using String = std::string;
   */
 template <typename T> constexpr bool IsNumber = false;
 
-template <> constexpr bool IsNumber<UInt8> = true;
+template <> inline constexpr bool IsNumber<UInt8> = true;
-template <> constexpr bool IsNumber<UInt16> = true;
+template <> inline constexpr bool IsNumber<UInt16> = true;
-template <> constexpr bool IsNumber<UInt32> = true;
+template <> inline constexpr bool IsNumber<UInt32> = true;
-template <> constexpr bool IsNumber<UInt64> = true;
+template <> inline constexpr bool IsNumber<UInt64> = true;
-template <> constexpr bool IsNumber<Int8> = true;
+template <> inline constexpr bool IsNumber<Int8> = true;
-template <> constexpr bool IsNumber<Int16> = true;
+template <> inline constexpr bool IsNumber<Int16> = true;
-template <> constexpr bool IsNumber<Int32> = true;
+template <> inline constexpr bool IsNumber<Int32> = true;
-template <> constexpr bool IsNumber<Int64> = true;
+template <> inline constexpr bool IsNumber<Int64> = true;
-template <> constexpr bool IsNumber<Float32> = true;
+template <> inline constexpr bool IsNumber<Float32> = true;
-template <> constexpr bool IsNumber<Float64> = true;
+template <> inline constexpr bool IsNumber<Float64> = true;
 
 template <typename T> struct TypeName;
 
@@ -109,7 +109,7 @@ using Strings = std::vector<String>;
 
 
 using Int128 = __int128;
-template <> constexpr bool IsNumber<Int128> = true;
+template <> inline constexpr bool IsNumber<Int128> = true;
 template <> struct TypeName<Int128> { static const char * get() { return "Int128";  } };
 template <> struct TypeId<Int128>   { static constexpr const TypeIndex value = TypeIndex::Int128; };
 
@@ -159,17 +159,56 @@ template <> struct TypeId<Decimal32>    { static constexpr const TypeIndex value
 template <> struct TypeId<Decimal64>    { static constexpr const TypeIndex value = TypeIndex::Decimal64; };
 template <> struct TypeId<Decimal128>   { static constexpr const TypeIndex value = TypeIndex::Decimal128; };
 
-template <typename T>
+template <typename T> constexpr bool IsDecimalNumber = false;
-constexpr bool IsDecimalNumber = false;
+template <> inline constexpr bool IsDecimalNumber<Decimal32> = true;
-template <> constexpr bool IsDecimalNumber<Decimal32> = true;
+template <> inline constexpr bool IsDecimalNumber<Decimal64> = true;
-template <> constexpr bool IsDecimalNumber<Decimal64> = true;
+template <> inline constexpr bool IsDecimalNumber<Decimal128> = true;
-template <> constexpr bool IsDecimalNumber<Decimal128> = true;
 
 template <typename T> struct NativeType { using Type = T; };
 template <> struct NativeType<Decimal32> { using Type = Int32; };
 template <> struct NativeType<Decimal64> { using Type = Int64; };
 template <> struct NativeType<Decimal128> { using Type = Int128; };
 
+inline const char * getTypeName(TypeIndex idx)
+{
+    switch (idx)
+    {
+        case TypeIndex::Nothing:    return "Nothing";
+        case TypeIndex::UInt8:      return TypeName<UInt8>::get();
+        case TypeIndex::UInt16:     return TypeName<UInt16>::get();
+        case TypeIndex::UInt32:     return TypeName<UInt32>::get();
+        case TypeIndex::UInt64:     return TypeName<UInt64>::get();
+        case TypeIndex::UInt128:    return "UInt128";
+        case TypeIndex::Int8:       return TypeName<Int8>::get();
+        case TypeIndex::Int16:      return TypeName<Int16>::get();
+        case TypeIndex::Int32:      return TypeName<Int32>::get();
+        case TypeIndex::Int64:      return TypeName<Int64>::get();
+        case TypeIndex::Int128:     return TypeName<Int128>::get();
+        case TypeIndex::Float32:    return TypeName<Float32>::get();
+        case TypeIndex::Float64:    return TypeName<Float64>::get();
+        case TypeIndex::Date:       return "Date";
+        case TypeIndex::DateTime:   return "DateTime";
+        case TypeIndex::String:     return TypeName<String>::get();
+        case TypeIndex::FixedString: return "FixedString";
+        case TypeIndex::Enum8:      return "Enum8";
+        case TypeIndex::Enum16:     return "Enum16";
+        case TypeIndex::Decimal32:  return TypeName<Decimal32>::get();
+        case TypeIndex::Decimal64:  return TypeName<Decimal64>::get();
+        case TypeIndex::Decimal128: return TypeName<Decimal128>::get();
+        case TypeIndex::UUID:       return "UUID";
+        case TypeIndex::Array:      return "Array";
+        case TypeIndex::Tuple:      return "Tuple";
+        case TypeIndex::Set:        return "Set";
+        case TypeIndex::Interval:   return "Interval";
+        case TypeIndex::Nullable:   return "Nullable";
+        case TypeIndex::Function:   return "Function";
+        case TypeIndex::AggregateFunction: return "AggregateFunction";
+        case TypeIndex::LowCardinality: return "LowCardinality";
+    }
+
+    __builtin_unreachable();
+}
+
 }
 
 /// Specialization of `std::hash` for the Decimal<T> types.

dbms/src/DataTypes/DataTypeAggregateFunction.h

@@ -36,6 +36,7 @@ public:
     bool canBeInsideNullable() const override { return false; }
 
     DataTypePtr getReturnType() const { return function->getReturnType(); }
+    DataTypePtr getReturnTypeToPredict() const { return function->getReturnTypeToPredict(); }
     DataTypes getArgumentsDataTypes() const { return argument_types; }
 
     /// NOTE These two functions for serializing single values are incompatible with the functions below.

dbms/src/DataTypes/DataTypeLowCardinalityHelpers.cpp

@@ -111,7 +111,7 @@ ColumnPtr recursiveLowCardinalityConversion(const ColumnPtr & column, const Data
         {
             auto col = low_cardinality_type->createColumn();
             static_cast<ColumnLowCardinality &>(*col).insertRangeFromFullColumn(*column, 0, column->size());
-            return std::move(col);
+            return col;
         }
     }
 

dbms/src/DataTypes/DataTypesDecimal.h

@@ -246,9 +246,9 @@ inline UInt32 getDecimalScale(const IDataType & data_type, UInt32 default_value
 ///
 
 template <typename DataType> constexpr bool IsDataTypeDecimal = false;
-template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
+template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
-template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
+template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
-template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
+template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
 
 template <typename DataType> constexpr bool IsDataTypeDecimalOrNumber = IsDataTypeDecimal<DataType> || IsDataTypeNumber<DataType>;
 

dbms/src/DataTypes/DataTypesNumber.h

@@ -38,15 +38,15 @@ using DataTypeFloat32 = DataTypeNumber<Float32>;
 using DataTypeFloat64 = DataTypeNumber<Float64>;
 
 template <typename DataType> constexpr bool IsDataTypeNumber = false;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
-template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
+template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
 
 }

dbms/src/DataTypes/Native.h

@@ -16,7 +16,7 @@
 #pragma GCC diagnostic push
 #pragma GCC diagnostic ignored "-Wunused-parameter"
 
-#include <llvm/IR/IRBuilder.h> // Y_IGNORE
+#include <llvm/IR/IRBuilder.h>
 
 #pragma GCC diagnostic pop
 

dbms/src/Dictionaries/CacheDictionary.cpp

@@ -86,7 +86,7 @@ void CacheDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<K
 {
     const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
 
-    getItemsNumber<UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; });
+    getItemsNumberImpl<UInt64, UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; });
 }
 
 
@@ -207,9 +207,7 @@ void CacheDictionary::isInConstantVector(const Key child_id, const PaddedPODArra
 void CacheDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto null_value = StringRef{std::get<String>(attribute.null_values)};
 
@@ -220,9 +218,7 @@ void CacheDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsString(attribute, ids, out, [&](const size_t row) { return def->getDataAt(row); });
 }
@@ -231,9 +227,7 @@ void CacheDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsString(attribute, ids, out, [&](const size_t) { return StringRef{def}; });
 }

dbms/src/Dictionaries/CacheDictionary.h

@@ -221,11 +221,6 @@ private:
 
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
-
-    template <typename OutputType, typename DefaultGetter>
-    void getItemsNumber(
-        Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;
-
     template <typename AttributeType, typename OutputType, typename DefaultGetter>
     void getItemsNumberImpl(
         Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;

dbms/src/Dictionaries/CacheDictionary.inc.h

@@ -34,34 +34,6 @@ namespace ErrorCodes
     extern const int TYPE_MISMATCH;
 }
 
-template <typename OutputType, typename DefaultGetter>
-void CacheDictionary::getItemsNumber(
-    Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const
-{
-    if (false)
-    {
-    }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) \
-        getItemsNumberImpl<TYPE, OutputType>(attribute, ids, out, std::forward<DefaultGetter>(get_default));
-    DISPATCH(UInt8)
-    DISPATCH(UInt16)
-    DISPATCH(UInt32)
-    DISPATCH(UInt64)
-    DISPATCH(UInt128)
-    DISPATCH(Int8)
-    DISPATCH(Int16)
-    DISPATCH(Int32)
-    DISPATCH(Int64)
-    DISPATCH(Float32)
-    DISPATCH(Float64)
-    DISPATCH(Decimal32)
-    DISPATCH(Decimal64)
-    DISPATCH(Decimal128)
-#undef DISPATCH
-    else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-}
-
 template <typename AttributeType, typename OutputType, typename DefaultGetter>
 void CacheDictionary::getItemsNumberImpl(
     Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const

dbms/src/Dictionaries/CacheDictionary_generate1.cpp.in

@@ -12,13 +12,11 @@ using TYPE = @NAME@;
 void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
     const auto null_value = std::get<TYPE>(attribute.null_values);
 
-    getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return null_value; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return null_value; });
 }
 
 }

dbms/src/Dictionaries/CacheDictionary_generate2.cpp.in

@@ -15,11 +15,9 @@ void CacheDictionary::get@NAME@(const std::string & attribute_name,
       ResultArrayType<TYPE> & out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
-    getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; });
 }
 
 }

dbms/src/Dictionaries/CacheDictionary_generate3.cpp.in

@@ -12,11 +12,9 @@ using TYPE = @NAME@;
 void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const
 {
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
-    getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return def; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return def; });
 }
 
 }

dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp

@@ -77,9 +77,7 @@ void ComplexKeyCacheDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto null_value = StringRef{std::get<String>(attribute.null_values)};
 
@@ -96,9 +94,7 @@ void ComplexKeyCacheDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsString(attribute, key_columns, out, [&](const size_t row) { return def->getDataAt(row); });
 }
@@ -113,9 +109,7 @@ void ComplexKeyCacheDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsString(attribute, key_columns, out, [&](const size_t) { return StringRef{def}; });
 }

dbms/src/Dictionaries/ComplexKeyCacheDictionary.h

@@ -256,34 +256,6 @@ private:
 
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
-    template <typename OutputType, typename DefaultGetter>
-    void
-    getItemsNumber(Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const
-    {
-        if (false)
-        {
-        }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) \
-        getItemsNumberImpl<TYPE, OutputType>(attribute, key_columns, out, std::forward<DefaultGetter>(get_default));
-        DISPATCH(UInt8)
-        DISPATCH(UInt16)
-        DISPATCH(UInt32)
-        DISPATCH(UInt64)
-        DISPATCH(UInt128)
-        DISPATCH(Int8)
-        DISPATCH(Int16)
-        DISPATCH(Int32)
-        DISPATCH(Int64)
-        DISPATCH(Float32)
-        DISPATCH(Float64)
-        DISPATCH(Decimal32)
-        DISPATCH(Decimal64)
-        DISPATCH(Decimal128)
-#undef DISPATCH
-        else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-    }
-
     template <typename AttributeType, typename OutputType, typename DefaultGetter>
     void getItemsNumberImpl(
         Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const

dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate1.cpp.in

@@ -13,12 +13,10 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name, co
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
     const auto null_value = std::get<TYPE>(attribute.null_values);
 
-    getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; });
 }
 }

dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate2.cpp.in

@@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
-    getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; });
 }
 }

dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate3.cpp.in

@@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
     dict_struct.validateKeyTypes(key_types);
 
     auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
-        throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                         ErrorCodes::TYPE_MISMATCH};
 
-    getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return def; });
+    getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return def; });
 }
 }

dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp

@@ -50,13 +50,11 @@ ComplexKeyHashedDictionary::ComplexKeyHashedDictionary(
         dict_struct.validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
         const auto null_value = std::get<TYPE>(attribute.null_values); \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, \
             key_columns, \
             [&](const size_t row, const auto value) { out[row] = value; }, \
@@ -84,9 +82,7 @@ void ComplexKeyHashedDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
 
@@ -108,11 +104,9 @@ void ComplexKeyHashedDictionary::getString(
         dict_struct.validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, \
             key_columns, \
             [&](const size_t row, const auto value) { out[row] = value; }, \
@@ -144,9 +138,7 @@ void ComplexKeyHashedDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -166,11 +158,9 @@ void ComplexKeyHashedDictionary::getString(
         dict_struct.validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
     }
 DECLARE(UInt8)
@@ -199,9 +189,7 @@ void ComplexKeyHashedDictionary::getString(
     dict_struct.validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -566,34 +554,6 @@ ComplexKeyHashedDictionary::createAttributeWithType(const AttributeUnderlyingTyp
 }
 
 
-template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-void ComplexKeyHashedDictionary::getItemsNumber(
-    const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
-{
-    if (false)
-    {
-    }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
-        attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
-    DISPATCH(UInt8)
-    DISPATCH(UInt16)
-    DISPATCH(UInt32)
-    DISPATCH(UInt64)
-    DISPATCH(UInt128)
-    DISPATCH(Int8)
-    DISPATCH(Int16)
-    DISPATCH(Int32)
-    DISPATCH(Int64)
-    DISPATCH(Float32)
-    DISPATCH(Float64)
-    DISPATCH(Decimal32)
-    DISPATCH(Decimal64)
-    DISPATCH(Decimal128)
-#undef DISPATCH
-    else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-}
-
 template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
 void ComplexKeyHashedDictionary::getItemsImpl(
     const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const

dbms/src/Dictionaries/ComplexKeyHashedDictionary.h

@@ -218,16 +218,10 @@ private:
 
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
-
-    template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-    void
-    getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
-
     template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
     void
     getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
 
-
     template <typename T>
     bool setAttributeValueImpl(Attribute & attribute, const StringRef key, const T value);
 

dbms/src/Dictionaries/DictionaryStructure.cpp

@@ -40,44 +40,6 @@ namespace
 } // namespace
 
 
-bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to)
-{
-    if (from == to)
-        return true;
-
-    /** This enum can be somewhat incomplete and the meaning may not coincide with NumberTraits.h.
-      * (for example, because integers can not be converted to floats)
-      * This is normal for a limited usage scope.
-      */
-    if ((from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt16)
-        || (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt32)
-        || (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt64)
-        || (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt32)
-        || (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt64)
-        || (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::UInt64)
-        || (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int16)
-        || (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int32)
-        || (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int64)
-        || (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int32)
-        || (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int64)
-        || (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::Int64)
-
-        || (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int16)
-        || (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int32)
-        || (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int64)
-        || (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int32)
-        || (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int64)
-        || (from == AttributeUnderlyingType::Int32 && to == AttributeUnderlyingType::Int64)
-
-        || (from == AttributeUnderlyingType::Float32 && to == AttributeUnderlyingType::Float64))
-    {
-        return true;
-    }
-
-    return false;
-}
-
-
 AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type)
 {
     static const std::unordered_map<std::string, AttributeUnderlyingType> dictionary{

dbms/src/Dictionaries/DictionaryStructure.h

@@ -13,6 +13,12 @@
 
 namespace DB
 {
+
+namespace ErrorCodes
+{
+    extern const int TYPE_MISMATCH;
+}
+
 enum class AttributeUnderlyingType
 {
     UInt8,
@@ -33,14 +39,18 @@ enum class AttributeUnderlyingType
 };
 
 
-/** For implicit conversions in dictGet functions.
-  */
-bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to);
-
 AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type);
 
 std::string toString(const AttributeUnderlyingType type);
 
+/// Implicit conversions in dictGet functions is disabled.
+inline void checkAttributeType(const std::string & dict_name, const std::string & attribute_name,
+                               AttributeUnderlyingType attribute_type, AttributeUnderlyingType to)
+{
+    if (attribute_type != to)
+        throw Exception{dict_name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute_type)
+            + ", expected " + toString(to), ErrorCodes::TYPE_MISMATCH};
+}
 
 /// Min and max lifetimes for a dictionary or it's entry
 using DictionaryLifetime = ExternalLoadableLifetime;

dbms/src/Dictionaries/FlatDictionary.cpp

@@ -55,7 +55,7 @@ void FlatDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<Ke
 {
     const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
 
-    getItemsNumber<UInt64>(
+    getItemsImpl<UInt64, UInt64>(
         *hierarchical_attribute,
         ids,
         [&](const size_t row, const UInt64 value) { out[row] = value; },
@@ -117,13 +117,11 @@ void FlatDictionary::isInConstantVector(const Key child_id, const PaddedPODArray
     void FlatDictionary::get##TYPE(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
         const auto null_value = std::get<TYPE>(attribute.null_values); \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
     }
 DECLARE(UInt8)
@@ -145,9 +143,7 @@ DECLARE(Decimal128)
 void FlatDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto & null_value = std::get<StringRef>(attribute.null_values);
 
@@ -166,11 +162,9 @@ void FlatDictionary::getString(const std::string & attribute_name, const PaddedP
         ResultArrayType<TYPE> & out) const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
     }
 DECLARE(UInt8)
@@ -193,9 +187,7 @@ void FlatDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -209,11 +201,9 @@ void FlatDictionary::getString(
         const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
     }
 DECLARE(UInt8)
@@ -236,9 +226,7 @@ void FlatDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     FlatDictionary::getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -580,35 +568,6 @@ FlatDictionary::Attribute FlatDictionary::createAttributeWithType(const Attribut
 }
 
 
-template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-void FlatDictionary::getItemsNumber(
-    const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
-{
-    if (false)
-    {
-    }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) \
-        getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
-    DISPATCH(UInt8)
-    DISPATCH(UInt16)
-    DISPATCH(UInt32)
-    DISPATCH(UInt64)
-    DISPATCH(UInt128)
-    DISPATCH(Int8)
-    DISPATCH(Int16)
-    DISPATCH(Int32)
-    DISPATCH(Int64)
-    DISPATCH(Float32)
-    DISPATCH(Float64)
-    DISPATCH(Decimal32)
-    DISPATCH(Decimal64)
-    DISPATCH(Decimal128)
-#undef DISPATCH
-    else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-}
-
-
 template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
 void FlatDictionary::getItemsImpl(
     const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
@@ -701,10 +660,10 @@ void FlatDictionary::setAttributeValue(Attribute & attribute, const Key id, cons
             break;
 
         case AttributeUnderlyingType::Decimal32:
-            setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal128>());
+            setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
             break;
         case AttributeUnderlyingType::Decimal64:
-            setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal128>());
+            setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
             break;
         case AttributeUnderlyingType::Decimal128:
             setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());

dbms/src/Dictionaries/FlatDictionary.h

@@ -206,10 +206,6 @@ private:
 
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
-    template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-    void getItemsNumber(
-        const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
-
     template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
     void getItemsImpl(
         const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;

dbms/src/Dictionaries/HashedDictionary.cpp

@@ -49,7 +49,7 @@ void HashedDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<
 {
     const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
 
-    getItemsNumber<UInt64>(
+    getItemsImpl<UInt64, UInt64>(
         *hierarchical_attribute,
         ids,
         [&](const size_t row, const UInt64 value) { out[row] = value; },
@@ -116,13 +116,11 @@ void HashedDictionary::isInConstantVector(const Key child_id, const PaddedPODArr
         const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
         const auto null_value = std::get<TYPE>(attribute.null_values); \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
     }
 DECLARE(UInt8)
@@ -144,9 +142,7 @@ DECLARE(Decimal128)
 void HashedDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
 
@@ -165,11 +161,9 @@ void HashedDictionary::getString(const std::string & attribute_name, const Padde
         ResultArrayType<TYPE> & out) const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
     }
 DECLARE(UInt8)
@@ -192,9 +186,7 @@ void HashedDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -208,11 +200,9 @@ void HashedDictionary::getString(
         const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE & def, ResultArrayType<TYPE> & out) const \
     { \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
     }
 DECLARE(UInt8)
@@ -235,9 +225,7 @@ void HashedDictionary::getString(
     const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
 {
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -324,7 +312,6 @@ void HashedDictionary::createAttributes()
 void HashedDictionary::blockToAttributes(const Block & block)
 {
     const auto & id_column = *block.safeGetByPosition(0).column;
-    element_count += id_column.size();
 
     for (const size_t attribute_idx : ext::range(0, attributes.size()))
     {
@@ -332,7 +319,8 @@ void HashedDictionary::blockToAttributes(const Block & block)
         auto & attribute = attributes[attribute_idx];
 
         for (const auto row_idx : ext::range(0, id_column.size()))
-            setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]);
+            if (setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]))
+                ++element_count;
     }
 }
 
@@ -567,34 +555,6 @@ HashedDictionary::Attribute HashedDictionary::createAttributeWithType(const Attr
 }
 
 
-template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-void HashedDictionary::getItemsNumber(
-    const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
-{
-    if (false)
-    {
-    }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) \
-        getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
-    DISPATCH(UInt8)
-    DISPATCH(UInt16)
-    DISPATCH(UInt32)
-    DISPATCH(UInt64)
-    DISPATCH(UInt128)
-    DISPATCH(Int8)
-    DISPATCH(Int16)
-    DISPATCH(Int32)
-    DISPATCH(Int64)
-    DISPATCH(Float32)
-    DISPATCH(Float64)
-    DISPATCH(Decimal32)
-    DISPATCH(Decimal64)
-    DISPATCH(Decimal128)
-#undef DISPATCH
-    else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-}
-
 template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
 void HashedDictionary::getItemsImpl(
     const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
@@ -613,69 +573,56 @@ void HashedDictionary::getItemsImpl(
 
 
 template <typename T>
-void HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value)
+bool HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value)
 {
     auto & map = *std::get<CollectionPtrType<T>>(attribute.maps);
-    map.insert({id, value});
+    return map.insert({id, value}).second;
 }
 
-void HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value)
+bool HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value)
 {
     switch (attribute.type)
     {
         case AttributeUnderlyingType::UInt8:
-            setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>());
+            return setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>());
-            break;
         case AttributeUnderlyingType::UInt16:
-            setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>());
+            return setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>());
-            break;
         case AttributeUnderlyingType::UInt32:
-            setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>());
+            return setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>());
-            break;
         case AttributeUnderlyingType::UInt64:
-            setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>());
+            return setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>());
-            break;
         case AttributeUnderlyingType::UInt128:
-            setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>());
+            return setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>());
-            break;
         case AttributeUnderlyingType::Int8:
-            setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>());
+            return setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>());
-            break;
         case AttributeUnderlyingType::Int16:
-            setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>());
+            return setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>());
-            break;
         case AttributeUnderlyingType::Int32:
-            setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>());
+            return setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>());
-            break;
         case AttributeUnderlyingType::Int64:
-            setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>());
+            return setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>());
-            break;
         case AttributeUnderlyingType::Float32:
-            setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>());
+            return setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>());
-            break;
         case AttributeUnderlyingType::Float64:
-            setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>());
+            return setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>());
-            break;
 
         case AttributeUnderlyingType::Decimal32:
-            setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
+            return setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
-            break;
         case AttributeUnderlyingType::Decimal64:
-            setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
+            return setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
-            break;
         case AttributeUnderlyingType::Decimal128:
-            setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
+            return setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
-            break;
 
         case AttributeUnderlyingType::String:
         {
             auto & map = *std::get<CollectionPtrType<StringRef>>(attribute.maps);
             const auto & string = value.get<String>();
             const auto string_in_arena = attribute.string_arena->insert(string.data(), string.size());
-            map.insert({id, StringRef{string_in_arena, string.size()}});
+            return map.insert({id, StringRef{string_in_arena, string.size()}}).second;
-            break;
         }
     }
+
+    throw Exception{"Invalid attribute type", ErrorCodes::BAD_ARGUMENTS};
 }
 
 const HashedDictionary::Attribute & HashedDictionary::getAttribute(const std::string & attribute_name) const

dbms/src/Dictionaries/HashedDictionary.h

@@ -211,18 +211,14 @@ private:
 
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
-    template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-    void getItemsNumber(
-        const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
-
     template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
     void getItemsImpl(
         const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
 
     template <typename T>
-    void setAttributeValueImpl(Attribute & attribute, const Key id, const T value);
+    bool setAttributeValueImpl(Attribute & attribute, const Key id, const T value);
 
-    void setAttributeValue(Attribute & attribute, const Key id, const Field & value);
+    bool setAttributeValue(Attribute & attribute, const Key id, const Field & value);
 
     const Attribute & getAttribute(const std::string & attribute_name) const;
 

dbms/src/Dictionaries/TrieDictionary.cpp

@@ -75,13 +75,11 @@ TrieDictionary::~TrieDictionary()
         validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
         const auto null_value = std::get<TYPE>(attribute.null_values); \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, \
             key_columns, \
             [&](const size_t row, const auto value) { out[row] = value; }, \
@@ -109,9 +107,7 @@ void TrieDictionary::getString(
     validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
 
@@ -133,11 +129,9 @@ void TrieDictionary::getString(
         validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, \
             key_columns, \
             [&](const size_t row, const auto value) { out[row] = value; }, \
@@ -169,9 +163,7 @@ void TrieDictionary::getString(
     validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -191,11 +183,9 @@ void TrieDictionary::getString(
         validateKeyTypes(key_types); \
 \
         const auto & attribute = getAttribute(attribute_name); \
-        if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
+        checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
-            throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
-                            ErrorCodes::TYPE_MISMATCH}; \
 \
-        getItemsNumber<TYPE>( \
+        getItemsImpl<TYPE, TYPE>( \
             attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
     }
 DECLARE(UInt8)
@@ -224,9 +214,7 @@ void TrieDictionary::getString(
     validateKeyTypes(key_types);
 
     const auto & attribute = getAttribute(attribute_name);
-    if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
+    checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
-        throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
-                        ErrorCodes::TYPE_MISMATCH};
 
     getItemsImpl<StringRef, StringRef>(
         attribute,
@@ -507,34 +495,6 @@ TrieDictionary::Attribute TrieDictionary::createAttributeWithType(const Attribut
 }
 
 
-template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-void TrieDictionary::getItemsNumber(
-    const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
-{
-    if (false)
-    {
-    }
-#define DISPATCH(TYPE) \
-    else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
-        attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
-    DISPATCH(UInt8)
-    DISPATCH(UInt16)
-    DISPATCH(UInt32)
-    DISPATCH(UInt64)
-    DISPATCH(UInt128)
-    DISPATCH(Int8)
-    DISPATCH(Int16)
-    DISPATCH(Int32)
-    DISPATCH(Int64)
-    DISPATCH(Float32)
-    DISPATCH(Float64)
-    DISPATCH(Decimal32)
-    DISPATCH(Decimal64)
-    DISPATCH(Decimal128)
-#undef DISPATCH
-    else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
-}
-
 template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
 void TrieDictionary::getItemsImpl(
     const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const

dbms/src/Dictionaries/TrieDictionary.h

@@ -218,10 +218,6 @@ private:
     Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
 
 
-    template <typename OutputType, typename ValueSetter, typename DefaultGetter>
-    void
-    getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
-
     template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
     void
     getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;

dbms/src/Formats/CapnProtoRowInputStream.cpp

@@ -3,13 +3,13 @@
 
 #include <IO/ReadBuffer.h>
 #include <Interpreters/Context.h>
-#include <Formats/CapnProtoRowInputStream.h> // Y_IGNORE
+#include <Formats/CapnProtoRowInputStream.h>
 #include <Formats/FormatFactory.h>
 #include <Formats/BlockInputStreamFromRowInputStream.h>
 #include <Formats/FormatSchemaInfo.h>
-#include <capnp/serialize.h> // Y_IGNORE
+#include <capnp/serialize.h>
-#include <capnp/dynamic.h> // Y_IGNORE
+#include <capnp/dynamic.h>
-#include <capnp/common.h> // Y_IGNORE
+#include <capnp/common.h>
 #include <boost/algorithm/string.hpp>
 #include <boost/range/join.hpp>
 #include <common/logger_useful.h>

dbms/src/Formats/FormatFactory.cpp

@@ -130,7 +130,7 @@ void registerOutputFormatXML(FormatFactory & factory);
 void registerOutputFormatODBCDriver(FormatFactory & factory);
 void registerOutputFormatODBCDriver2(FormatFactory & factory);
 void registerOutputFormatNull(FormatFactory & factory);
-void registerOutputFormatMySQL(FormatFactory & factory);
+void registerOutputFormatMySQLWire(FormatFactory & factory);
 
 /// Input only formats.
 
@@ -169,7 +169,7 @@ FormatFactory::FormatFactory()
     registerOutputFormatODBCDriver(*this);
     registerOutputFormatODBCDriver2(*this);
     registerOutputFormatNull(*this);
-    registerOutputFormatMySQL(*this);
+    registerOutputFormatMySQLWire(*this);
 }
 
 }

dbms/src/Formats/MySQLFormat.cpp

@@ -1,19 +0,0 @@
-#include <DataStreams/MySQLBlockOutputStream.h>
-
-
-namespace DB
-{
-
-void registerOutputFormatMySQL(FormatFactory & factory)
-{
-    factory.registerOutputFormat("MySQL", [](
-        WriteBuffer & buf,
-        const Block & sample,
-        const Context & context,
-        const FormatSettings &)
-    {
-        return std::make_shared<MySQLBlockOutputStream>(buf, sample, const_cast<Context &>(context));
-    });
-}
-
-}

dbms/src/Formats/MySQLWireBlockOutputStream.cpp

@@ -1,4 +1,4 @@
-#include "MySQLBlockOutputStream.h"
+#include "MySQLWireBlockOutputStream.h"
 #include <Core/MySQLProtocol.h>
 #include <Interpreters/ProcessList.h>
 #include <iomanip>
@@ -9,15 +9,15 @@ namespace DB
 
 using namespace MySQLProtocol;
 
-MySQLBlockOutputStream::MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context)
+MySQLWireBlockOutputStream::MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context)
     : header(header)
     , context(context)
-    , packet_sender(new PacketSender(buf, context.sequence_id, "MySQLBlockOutputStream"))
+    , packet_sender(std::make_shared<PacketSender>(buf, context.sequence_id))
 {
     packet_sender->max_packet_size = context.max_packet_size;
 }
 
-void MySQLBlockOutputStream::writePrefix()
+void MySQLWireBlockOutputStream::writePrefix()
 {
     if (header.columns() == 0)
         return;
@@ -26,8 +26,7 @@ void MySQLBlockOutputStream::writePrefix()
 
     for (const ColumnWithTypeAndName & column : header.getColumnsWithTypeAndName())
     {
-        ColumnDefinition column_definition(column.name, CharacterSet::binary, std::numeric_limits<uint32_t>::max(),
+        ColumnDefinition column_definition(column.name, CharacterSet::binary, 0, ColumnType::MYSQL_TYPE_STRING, 0, 0);
-                                           ColumnType::MYSQL_TYPE_STRING, 0, 0);
         packet_sender->sendPacket(column_definition);
     }
 
@@ -37,7 +36,7 @@ void MySQLBlockOutputStream::writePrefix()
     }
 }
 
-void MySQLBlockOutputStream::write(const Block & block)
+void MySQLWireBlockOutputStream::write(const Block & block)
 {
     size_t rows = block.rows();
 
@@ -57,7 +56,7 @@ void MySQLBlockOutputStream::write(const Block & block)
     }
 }
 
-void MySQLBlockOutputStream::writeSuffix()
+void MySQLWireBlockOutputStream::writeSuffix()
 {
     QueryStatus * process_list_elem = context.getProcessListElement();
     CurrentThread::finalizePerformanceCounters();
@@ -79,7 +78,7 @@ void MySQLBlockOutputStream::writeSuffix()
             packet_sender->sendPacket(EOF_Packet(0, 0), true);
 }
 
-void MySQLBlockOutputStream::flush()
+void MySQLWireBlockOutputStream::flush()
 {
     packet_sender->out->next();
 }

dbms/src/Formats/MySQLWireBlockOutputStream.h

@@ -1,7 +1,7 @@
 #pragma once
 
-#include "IBlockOutputStream.h"
 #include <Core/MySQLProtocol.h>
+#include <DataStreams/IBlockOutputStream.h>
 #include <Formats/FormatFactory.h>
 #include <Formats/FormatSettings.h>
 #include <Interpreters/Context.h>
@@ -11,10 +11,10 @@ namespace DB
 
 /** Interface for writing rows in MySQL Client/Server Protocol format.
   */
-class MySQLBlockOutputStream : public IBlockOutputStream
+class MySQLWireBlockOutputStream : public IBlockOutputStream
 {
 public:
-    MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context);
+    MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context);
 
     Block getHeader() const { return header; }
 
@@ -31,6 +31,6 @@ private:
     FormatSettings format_settings;
 };
 
-using MySQLBlockOutputStreamPtr = std::shared_ptr<MySQLBlockOutputStream>;
+using MySQLWireBlockOutputStreamPtr = std::shared_ptr<MySQLWireBlockOutputStream>;
 
 }

dbms/src/Formats/MySQLWireFormat.cpp

@@ -0,0 +1,19 @@
+#include <Formats/MySQLWireBlockOutputStream.h>
+
+
+namespace DB
+{
+
+void registerOutputFormatMySQLWire(FormatFactory & factory)
+{
+    factory.registerOutputFormat("MySQLWire", [](
+        WriteBuffer & buf,
+        const Block & sample,
+        const Context & context,
+        const FormatSettings &)
+    {
+        return std::make_shared<MySQLWireBlockOutputStream>(buf, sample, const_cast<Context &>(context));
+    });
+}
+
+}

dbms/src/Formats/ProtobufSchemas.cpp

@@ -2,8 +2,8 @@
 #if USE_PROTOBUF
 
 #include <Formats/FormatSchemaInfo.h>
-#include <Formats/ProtobufSchemas.h> // Y_IGNORE
+#include <Formats/ProtobufSchemas.h>
-#include <google/protobuf/compiler/importer.h> // Y_IGNORE
+#include <google/protobuf/compiler/importer.h>
 #include <Common/Exception.h>
 
 

dbms/src/Formats/ProtobufWriter.cpp

@@ -7,8 +7,8 @@
 #include <AggregateFunctions/IAggregateFunction.h>
 #include <DataTypes/DataTypesDecimal.h>
 #include <boost/numeric/conversion/cast.hpp>
-#include <google/protobuf/descriptor.h> // Y_IGNORE
+#include <google/protobuf/descriptor.h>
-#include <google/protobuf/descriptor.pb.h> // Y_IGNORE
+#include <google/protobuf/descriptor.pb.h>
 #include <IO/ReadHelpers.h>
 #include <IO/WriteHelpers.h>
 #include "ProtobufWriter.h"

dbms/src/Functions/FunctionBase64Conversion.h

@@ -7,7 +7,7 @@
 #include <Functions/FunctionHelpers.h>
 #include <Functions/GatherUtils/Algorithms.h>
 #include <IO/WriteHelpers.h>
-#include <libbase64.h> // Y_IGNORE
+#include <libbase64.h>
 
 
 namespace DB

dbms/src/Functions/FunctionBinaryArithmetic.h

@@ -24,7 +24,7 @@
 #if USE_EMBEDDED_COMPILER
 #pragma GCC diagnostic push
 #pragma GCC diagnostic ignored "-Wunused-parameter"
-#include <llvm/IR/IRBuilder.h> // Y_IGNORE
+#include <llvm/IR/IRBuilder.h>
 #pragma GCC diagnostic pop
 #endif
 
@@ -354,27 +354,27 @@ template <bool V, typename T> struct Case : std::bool_constant<V> { using type =
 template <typename... Ts> using Switch = typename std::disjunction<Ts..., Case<true, InvalidType>>::type;
 
 template <typename DataType> constexpr bool IsIntegral = false;
-template <> constexpr bool IsIntegral<DataTypeUInt8> = true;
+template <> inline constexpr bool IsIntegral<DataTypeUInt8> = true;
-template <> constexpr bool IsIntegral<DataTypeUInt16> = true;
+template <> inline constexpr bool IsIntegral<DataTypeUInt16> = true;
-template <> constexpr bool IsIntegral<DataTypeUInt32> = true;
+template <> inline constexpr bool IsIntegral<DataTypeUInt32> = true;
-template <> constexpr bool IsIntegral<DataTypeUInt64> = true;
+template <> inline constexpr bool IsIntegral<DataTypeUInt64> = true;
-template <> constexpr bool IsIntegral<DataTypeInt8> = true;
+template <> inline constexpr bool IsIntegral<DataTypeInt8> = true;
-template <> constexpr bool IsIntegral<DataTypeInt16> = true;
+template <> inline constexpr bool IsIntegral<DataTypeInt16> = true;
-template <> constexpr bool IsIntegral<DataTypeInt32> = true;
+template <> inline constexpr bool IsIntegral<DataTypeInt32> = true;
-template <> constexpr bool IsIntegral<DataTypeInt64> = true;
+template <> inline constexpr bool IsIntegral<DataTypeInt64> = true;
 
 template <typename DataType> constexpr bool IsFloatingPoint = false;
-template <> constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
+template <> inline constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
-template <> constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
+template <> inline constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
 
 template <typename DataType> constexpr bool IsDateOrDateTime = false;
-template <> constexpr bool IsDateOrDateTime<DataTypeDate> = true;
+template <> inline constexpr bool IsDateOrDateTime<DataTypeDate> = true;
-template <> constexpr bool IsDateOrDateTime<DataTypeDateTime> = true;
+template <> inline constexpr bool IsDateOrDateTime<DataTypeDateTime> = true;
 
 template <typename T0, typename T1> constexpr bool UseLeftDecimal = false;
-template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true;
+template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true;
-template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true;
+template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true;
-template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true;
+template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true;
 
 template <typename T> using DataTypeFromFieldType = std::conditional_t<std::is_same_v<T, NumberTraits::Error>, InvalidType, DataTypeNumber<T>>;
 

dbms/src/Functions/FunctionMathBinaryFloat64.h

@@ -22,8 +22,8 @@
         #pragma clang diagnostic ignored "-Wshift-negative-value"
     #endif
 
-    #include <vectorf128.h> // Y_IGNORE
+    #include <vectorf128.h>
-    #include <vectormath_exp.h> // Y_IGNORE
+    #include <vectormath_exp.h>
 
     #ifdef __clang__
         #pragma clang diagnostic pop

dbms/src/Functions/FunctionMathUnaryFloat64.h

@@ -21,9 +21,9 @@
         #pragma clang diagnostic ignored "-Wshift-negative-value"
     #endif
 
-    #include <vectorf128.h> // Y_IGNORE
+    #include <vectorf128.h>
-    #include <vectormath_exp.h> // Y_IGNORE
+    #include <vectormath_exp.h>
-    #include <vectormath_trig.h> // Y_IGNORE
+    #include <vectormath_trig.h>
 
     #ifdef __clang__
         #pragma clang diagnostic pop