mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-09-22 09:40:49 +00:00
Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview
This commit is contained in:
Vitaliy Zakaznikov
commit
b65ff910b6
66
CHANGELOG.md
66
CHANGELOG.md
@ -1,9 +1,70 @@
|
||||
## ClickHouse release 19.7.3.9, 2019-05-30
|
||||
|
||||
### New Features
|
||||
* Allow to limit the range of a setting that can be specified by user.
|
||||
These constraints can be set up in user settings profile.
|
||||
[#4931](https://github.com/yandex/ClickHouse/pull/4931) ([Vitaly
|
||||
Baranov](https://github.com/vitlibar))
|
||||
* Add a second version of the function `groupUniqArray` with an optional
|
||||
`max_size` parameter that limits the size of the resulting array. This
|
||||
behavior is similar to `groupArray(max_size)(x)` function.
|
||||
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
|
||||
Tassery](https://github.com/YiuRULE))
|
||||
* For TSVWithNames/CSVWithNames input file formats, column order can now be
|
||||
determined from file header. This is controlled by
|
||||
`input_format_with_names_use_header` parameter.
|
||||
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
|
||||
([Alexander](https://github.com/Akazz))
|
||||
|
||||
### Bug Fixes
|
||||
* Crash with uncompressed_cache + JOIN during merge (#5197)
|
||||
[#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Segmentation fault on a clickhouse-client query to system tables. #5066
|
||||
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Data loss on heavy load via KafkaEngine (#4736)
|
||||
[#5080](https://github.com/yandex/ClickHouse/pull/5080)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Performance Improvements
|
||||
* Use radix sort for sorting by single numeric column in `ORDER BY` without
|
||||
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
|
||||
[#4439](https://github.com/yandex/ClickHouse/pull/4439)
|
||||
([Evgenii Pravda](https://github.com/kvinty),
|
||||
[alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Documentation
|
||||
* Translate documentation for some table engines to Chinese.
|
||||
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
|
||||
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
|
||||
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
|
||||
([张风啸](https://github.com/AlexZFX)),
|
||||
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
|
||||
lee](https://github.com/neverlee))
|
||||
|
||||
### Build/Testing/Packaging Improvements
|
||||
* Print UTF-8 characters properly in `clickhouse-test`.
|
||||
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Add command line parameter for clickhouse-client to always load suggestion
|
||||
data. [#5102](https://github.com/yandex/ClickHouse/pull/5102)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Resolve some of PVS-Studio warnings.
|
||||
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Update LZ4 [#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Add gperf to build requirements for upcoming pull request #5030.
|
||||
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
|
||||
([proller](https://github.com/proller))
|
||||
|
||||
## ClickHouse release 19.6.2.11, 2019-05-13
|
||||
|
||||
### New Features
|
||||
* TTL expressions for columns and tables. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
|
||||
* Added a new aggregate function `simpleLinearRegression(x, y)` which performs linear regression on points (x, y) and returns the parameters of the line. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
|
||||
* Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
|
||||
|
||||
@ -30,6 +91,7 @@
|
||||
* Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Partially fix distributed_product_mode = local. It's possible to allow columns of local tables in where/having/order by/... via table aliases. Throw exception if table does not have alias. There's not possible to access to the columns without table aliases yet. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Build/Testing/Packaging Improvements
|
||||
* Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
@ -346,7 +408,7 @@
|
||||
* Added support of `Nullable` types in `mysql` table function. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
|
||||
* Support for arbitrary constant expressions in `LIMIT` clause. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
|
||||
* Added `topKWeighted` aggregate function that takes additional argument with (unsigned integer) weight. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
|
||||
* `StorageJoin` now supports `join_overwrite` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
|
||||
* `StorageJoin` now supports `join_any_take_last_row` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
|
||||
* Added function `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Added `RowBinaryWithNamesAndTypes` format. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
|
||||
* Added `IPv4` and `IPv6` data types. More effective implementations of `IPv*` functions. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
|
||||
@ -1,13 +1,77 @@
|
||||
## ClickHouse release 19.7.3.9, 2019-05-30
|
||||
|
||||
### Новые возможности
|
||||
* Добавлена возможность ограничить значения конфигурационных параметров,
|
||||
которые может задать пользователь. Эти ограничения устанавливаются в профиле
|
||||
настроек пользователя. [#4931](https://github.com/yandex/ClickHouse/pull/4931)
|
||||
([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Добавлен вариант функции `groupUniqArray` с дополнительным параметром
|
||||
`max_size`, который ограничивает размер результирующего массива, аналогично
|
||||
функции `groupArray(max_size)(x)`.
|
||||
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
|
||||
Tassery](https://github.com/YiuRULE))
|
||||
* Для входных файлов формата TSVWithNames и CSVWithNames появилась возможность
|
||||
определить порядок колонок в файле исходя из его заголовка. Это поведение
|
||||
управляется конфигурационным параметром `input_format_with_names_use_header`.
|
||||
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
|
||||
([Alexander](https://github.com/Akazz))
|
||||
|
||||
### Исправления ошибок
|
||||
* Падение в процессе слияния при использовании uncompressed_cache и JOIN
|
||||
(#5197). [#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Segmentation fault на запросе к системным таблицам (#5066).
|
||||
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Потеря загружаемых данных при больших потоках загрузки через KafkaEngine
|
||||
(#4736). [#5080](https://github.com/yandex/ClickHouse/pull/5080)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Улучшения производительности
|
||||
* Используется поразрядная сортировка числовых колонок для `ORDER BY` без
|
||||
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
|
||||
[#4439](https://github.com/yandex/ClickHouse/pull/4439) ([Evgenii
|
||||
Pravda](https://github.com/kvinty),
|
||||
[alexey-milovidov](https://github.com/alexey-milovidov)
|
||||
|
||||
### Документация
|
||||
* Документация для некоторых табличных движков переведена на китайский.
|
||||
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
|
||||
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
|
||||
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
|
||||
([张风啸](https://github.com/AlexZFX),
|
||||
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
|
||||
lee](https://github.com/neverlee))
|
||||
|
||||
### Улучшения сборки, тестирования и пакетирования
|
||||
* Правильно отображаются символы в кодировке UTF-8 в `clickhouse-test`.
|
||||
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Добавлен параметр командной строки для `clickhouse-client`, позволяющий
|
||||
всегда загружать данные подсказок.
|
||||
[#5102](https://github.com/yandex/ClickHouse/pull/5102)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлены некоторые предупреждения PVS-Studio.
|
||||
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Обновлена библиотека LZ4.
|
||||
[#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* В зависимости сборки добавлен gperf для поддержки готовящегося PR #5030.
|
||||
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
|
||||
([proller](https://github.com/proller))
|
||||
|
||||
|
||||
## ClickHouse release 19.6.2.11, 2019-05-13
|
||||
|
||||
### Новые возможности
|
||||
* TTL выражения, позволяющие настроить время жизни и автоматическую очистку данных в таблице или в отдельных её столбцах. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Добавлена поддержка алгоритма сжатия `brotli` в HTTP ответах (`Accept-Encoding: br`). Для тела POST запросов, эта возможность уже существовала. [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
|
||||
* Добавлена агрегатная функция `simpleLinearRegression(x, y)` которая осуществляет линейную регрессию на точках (x, y) и возвращает параметры линии. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
|
||||
* Добавлена функция `isValidUTF8` для проверки, содержит ли строка валидные данные в кодировке UTF-8. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Добавлены новое правило балансировки (`load_balancing`) `first_or_random` по которому запросы посылаются на первый заданый хост и если он недоступен - на случайные хосты шарда. Полезно для топологий с кросс-репликацией. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
|
||||
|
||||
### Эксперемннтальные возможности
|
||||
### Экспериментальные возможности
|
||||
* Добавлена настройка `index_granularity_bytes` (адаптивная гранулярность индекса) для таблиц семейства MergeTree* . [#4826](https://github.com/yandex/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin))
|
||||
|
||||
### Улучшения
|
||||
@ -30,6 +94,7 @@
|
||||
* Исправлено зависание на старте сервера если внешний словарь зависит от другого словаря через использование таблицы из БД с движком `Dictionary`. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* При использовании `distributed_product_mode = 'local'` корректно работает использование столбцов локальных таблиц в where/having/order by/... через табличные алиасы. Выкидывает исключение если таблица не имеет алиас. Доступ к столбцам без алиасов пока не возможен. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлен потенциально некорректный результат для `SELECT DISTINCT` с `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Улучшения сборки/тестирования/пакетирования
|
||||
* Исправлена неработоспособность тестов, если `clickhouse-server` запущен на удалённом хосте [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
@ -337,7 +402,7 @@
|
||||
* Добавлена поддержка `Nullable` типов в табличной функции `mysql`. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
|
||||
* Добавлена поддержка произвольных константных выражений в секции `LIMIT`. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
|
||||
* Добавлена агрегатная функция `topKWeighted` - вариант `topK`, позволяющий задавать (целый неотрицательный) вес добавляемого значения. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
|
||||
* Движок `Join` теперь поддерживает настройку `join_overwrite`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
|
||||
* Движок `Join` теперь поддерживает настройку `join_any_take_last_row`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
|
||||
* Добавлена функция `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Добавлена функция `toStartOfTenMinutes`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Добавлен формат `RowBinaryWithNamesAndTypes`. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
|
||||
|
||||
@ -93,6 +93,11 @@ if (COMPILER_GCC OR COMPILER_CLANG)
|
||||
set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wnon-virtual-dtor")
|
||||
endif ()
|
||||
|
||||
if (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER "8.3.0")
|
||||
# Warnings in protobuf generating
|
||||
set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wno-array-bounds")
|
||||
endif ()
|
||||
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||
# clang: warning: argument unused during compilation: '-stdlib=libc++'
|
||||
# clang: warning: argument unused during compilation: '-specs=/usr/share/dpkg/no-pie-compile.specs' [-Wunused-command-line-argument]
|
||||
|
||||
@ -12,8 +12,7 @@ ClickHouse is an open-source column-oriented database management system that all
|
||||
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
|
||||
|
||||
## Upcoming Events
|
||||
* ClickHouse at [Percona Live 2019](https://www.percona.com/live/19/other-open-source-databases-track) in Austin on May 28-30.
|
||||
* [ClickHouse Community Meetup in San Francisco](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/261110652/) on June 4.
|
||||
* [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8.
|
||||
* [ClickHouse on HighLoad++ Siberia](https://www.highload.ru/siberia/2019/abstracts/5348) on June 24-25.
|
||||
* [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
|
||||
* [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.
|
||||
|
||||
4
contrib/CMakeLists.txt
vendored
4
contrib/CMakeLists.txt
vendored
@ -1,8 +1,8 @@
|
||||
# Third-party libraries may have substandard code.
|
||||
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch -Wno-stringop-truncation")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -Wno-array-bounds -Wno-missing-attributes -Wno-stringop-truncation -std=c++1z")
|
||||
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch -Wno-string-plus-int")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")
|
||||
|
||||
2
contrib/boost
vendored
2
contrib/boost
vendored
@ -1 +1 @@
|
||||
Subproject commit 79bf85ea99c05ba4fb6959474d4464ab126f8973
|
||||
Subproject commit 8abda007bfe52d78a51548d4594879d6d82a22fa
|
||||
@ -1,6 +1,6 @@
|
||||
add_library(roaring
|
||||
roaring.c
|
||||
roaring.h
|
||||
roaring.hh)
|
||||
roaring/roaring.h
|
||||
roaring/roaring.hh)
|
||||
|
||||
target_include_directories (roaring PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/* auto-generated on Tue Dec 18 09:42:59 CST 2018. Do not edit! */
|
||||
#include "roaring.h"
|
||||
#include "roaring/roaring.h"
|
||||
|
||||
/* used for http://dmalloc.com/ Dmalloc - Debug Malloc Library */
|
||||
#ifdef DMALLOC
|
||||
|
||||
2
contrib/hyperscan
vendored
2
contrib/hyperscan
vendored
@ -1 +1 @@
|
||||
Subproject commit 05b0f9064cca4bd55548dedb0a32ed9461146c1e
|
||||
Subproject commit 01e6b83f9fbdb4020cd68a5287bf3a0471eeb272
|
||||
@ -40,6 +40,10 @@ if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
|
||||
list(APPEND SRCS ${JEMALLOC_SOURCE_DIR}/src/zone.c)
|
||||
endif()
|
||||
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
|
||||
endif ()
|
||||
|
||||
add_library(jemalloc STATIC ${SRCS})
|
||||
|
||||
target_include_directories(jemalloc PUBLIC
|
||||
|
||||
@ -81,7 +81,7 @@
|
||||
#define PCG_128BIT_CONSTANT(high,low) \
|
||||
((pcg128_t(high) << 64) + low)
|
||||
#else
|
||||
#include "pcg_uint128.hpp" // Y_IGNORE
|
||||
#include "pcg_uint128.hpp"
|
||||
namespace pcg_extras {
|
||||
typedef pcg_extras::uint_x4<uint32_t,uint64_t> pcg128_t;
|
||||
}
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
#if __has_include(<rdkafka.h>) // maybe bundled
|
||||
# include_next <rdkafka.h> // Y_IGNORE
|
||||
# include_next <rdkafka.h>
|
||||
#else // system
|
||||
# include_next <librdkafka/rdkafka.h>
|
||||
#endif
|
||||
|
||||
@ -49,19 +49,20 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||
endif ()
|
||||
|
||||
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
|
||||
|
||||
if (WEVERYTHING)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-return-std-move-in-c++11")
|
||||
endif ()
|
||||
endif ()
|
||||
|
||||
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wshadow-field -Wstring-plus-int -Wempty-init-stmt")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow-field -Wstring-plus-int")
|
||||
if(NOT APPLE)
|
||||
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wempty-init-stmt")
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9)
|
||||
if (WEVERYTHING)
|
||||
if (WEVERYTHING AND NOT APPLE)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-ctad-maybe-unsupported")
|
||||
endif ()
|
||||
endif ()
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
# This strings autochanged from release_lib.sh:
|
||||
set(VERSION_REVISION 54420)
|
||||
set(VERSION_REVISION 54421)
|
||||
set(VERSION_MAJOR 19)
|
||||
set(VERSION_MINOR 8)
|
||||
set(VERSION_MINOR 9)
|
||||
set(VERSION_PATCH 1)
|
||||
set(VERSION_GITHASH a76e504f45ff4a74e8c492bd269f022352d5f6d9)
|
||||
set(VERSION_DESCRIBE v19.8.1.1-testing)
|
||||
set(VERSION_STRING 19.8.1.1)
|
||||
set(VERSION_GITHASH 0c2aa460651a462f14efc7e995840a244531d373)
|
||||
set(VERSION_DESCRIBE v19.9.1.1-testing)
|
||||
set(VERSION_STRING 19.9.1.1)
|
||||
# end of autochange
|
||||
|
||||
set(VERSION_EXTRA "" CACHE STRING "")
|
||||
|
||||
@ -1,6 +1,19 @@
|
||||
set(CLICKHOUSE_CLIENT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp)
|
||||
set(CLICKHOUSE_CLIENT_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/ConnectionParameters.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR})
|
||||
set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/include)
|
||||
|
||||
include(CheckSymbolExists)
|
||||
check_symbol_exists(readpassphrase readpassphrase.h HAVE_READPASSPHRASE)
|
||||
configure_file(config_client.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/config_client.h)
|
||||
|
||||
if(NOT HAVE_READPASSPHRASE)
|
||||
add_subdirectory(readpassphrase)
|
||||
list(APPEND CLICKHOUSE_CLIENT_LINK PRIVATE readpassphrase)
|
||||
endif()
|
||||
|
||||
clickhouse_program_add(client)
|
||||
|
||||
|
||||
@ -67,7 +67,7 @@
|
||||
#include <Storages/ColumnsDescription.h>
|
||||
|
||||
#if USE_READLINE
|
||||
#include "Suggest.h" // Y_IGNORE
|
||||
#include "Suggest.h"
|
||||
#endif
|
||||
|
||||
#ifndef __clang__
|
||||
@ -435,7 +435,7 @@ private:
|
||||
#if USE_READLINE
|
||||
int res = read_history(history_file.c_str());
|
||||
if (res)
|
||||
throwFromErrno("Cannot read history from file " + history_file, ErrorCodes::CANNOT_READ_HISTORY);
|
||||
std::cerr << "Cannot read history from file " + history_file + ": "+ errnoToString(ErrorCodes::CANNOT_READ_HISTORY);
|
||||
#endif
|
||||
}
|
||||
else /// Create history file.
|
||||
@ -612,7 +612,7 @@ private:
|
||||
|
||||
#if USE_READLINE && HAVE_READLINE_HISTORY
|
||||
if (!history_file.empty() && append_history(1, history_file.c_str()))
|
||||
throwFromErrno("Cannot append history to file " + history_file, ErrorCodes::CANNOT_APPEND_HISTORY);
|
||||
std::cerr << "Cannot append history to file " + history_file + ": " + errnoToString(ErrorCodes::CANNOT_APPEND_HISTORY);
|
||||
#endif
|
||||
|
||||
prev_input = input;
|
||||
|
||||
63
dbms/programs/client/ConnectionParameters.cpp
Normal file
63
dbms/programs/client/ConnectionParameters.cpp
Normal file
@ -0,0 +1,63 @@
|
||||
#include "ConnectionParameters.h"
|
||||
#include <fstream>
|
||||
#include <iostream>
|
||||
#include <Core/Defines.h>
|
||||
#include <Core/Protocol.h>
|
||||
#include <Core/Types.h>
|
||||
#include <IO/ConnectionTimeouts.h>
|
||||
#include <Poco/Util/AbstractConfiguration.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <common/setTerminalEcho.h>
|
||||
#include <ext/scope_guard.h>
|
||||
#include <readpassphrase.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int BAD_ARGUMENTS;
|
||||
}
|
||||
|
||||
ConnectionParameters::ConnectionParameters(const Poco::Util::AbstractConfiguration & config)
|
||||
{
|
||||
bool is_secure = config.getBool("secure", false);
|
||||
security = is_secure ? Protocol::Secure::Enable : Protocol::Secure::Disable;
|
||||
|
||||
host = config.getString("host", "localhost");
|
||||
port = config.getInt(
|
||||
"port", config.getInt(is_secure ? "tcp_port_secure" : "tcp_port", is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
|
||||
|
||||
default_database = config.getString("database", "");
|
||||
/// changed the default value to "default" to fix the issue when the user in the prompt is blank
|
||||
user = config.getString("user", "default");
|
||||
bool password_prompt = false;
|
||||
if (config.getBool("ask-password", false))
|
||||
{
|
||||
if (config.has("password"))
|
||||
throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
|
||||
password_prompt = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
password = config.getString("password", "");
|
||||
/// if the value of --password is omitted, the password will be set implicitly to "\n"
|
||||
if (password == "\n")
|
||||
password_prompt = true;
|
||||
}
|
||||
if (password_prompt)
|
||||
{
|
||||
std::string prompt{"Password for user (" + user + "): "};
|
||||
char buf[1000] = {};
|
||||
if (auto result = readpassphrase(prompt.c_str(), buf, sizeof(buf), 0))
|
||||
password = result;
|
||||
}
|
||||
compression = config.getBool("compression", true) ? Protocol::Compression::Enable : Protocol::Compression::Disable;
|
||||
|
||||
timeouts = ConnectionTimeouts(
|
||||
Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
|
||||
}
|
||||
}
|
||||
@ -1,90 +1,30 @@
|
||||
#pragma once
|
||||
|
||||
#include <iostream>
|
||||
|
||||
#include <Core/Types.h>
|
||||
#include <string>
|
||||
#include <Core/Protocol.h>
|
||||
#include <Core/Defines.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <IO/ConnectionTimeouts.h>
|
||||
|
||||
#include <common/setTerminalEcho.h>
|
||||
#include <ext/scope_guard.h>
|
||||
|
||||
#include <Poco/Util/AbstractConfiguration.h>
|
||||
|
||||
namespace Poco::Util
|
||||
{
|
||||
class AbstractConfiguration;
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int BAD_ARGUMENTS;
|
||||
}
|
||||
|
||||
struct ConnectionParameters
|
||||
{
|
||||
String host;
|
||||
std::string host;
|
||||
UInt16 port{};
|
||||
String default_database;
|
||||
String user;
|
||||
String password;
|
||||
std::string default_database;
|
||||
std::string user;
|
||||
std::string password;
|
||||
Protocol::Secure security = Protocol::Secure::Disable;
|
||||
Protocol::Compression compression = Protocol::Compression::Enable;
|
||||
ConnectionTimeouts timeouts;
|
||||
|
||||
ConnectionParameters() {}
|
||||
|
||||
ConnectionParameters(const Poco::Util::AbstractConfiguration & config)
|
||||
{
|
||||
bool is_secure = config.getBool("secure", false);
|
||||
security = is_secure
|
||||
? Protocol::Secure::Enable
|
||||
: Protocol::Secure::Disable;
|
||||
|
||||
host = config.getString("host", "localhost");
|
||||
port = config.getInt("port",
|
||||
config.getInt(is_secure ? "tcp_port_secure" : "tcp_port",
|
||||
is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
|
||||
|
||||
default_database = config.getString("database", "");
|
||||
/// changed the default value to "default" to fix the issue when the user in the prompt is blank
|
||||
user = config.getString("user", "default");
|
||||
bool password_prompt = false;
|
||||
if (config.getBool("ask-password", false))
|
||||
{
|
||||
if (config.has("password"))
|
||||
throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
|
||||
password_prompt = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
password = config.getString("password", "");
|
||||
/// if the value of --password is omitted, the password will be set implicitly to "\n"
|
||||
if (password == "\n")
|
||||
password_prompt = true;
|
||||
}
|
||||
if (password_prompt)
|
||||
{
|
||||
std::cout << "Password for user (" << user << "): ";
|
||||
setTerminalEcho(false);
|
||||
|
||||
SCOPE_EXIT({
|
||||
setTerminalEcho(true);
|
||||
});
|
||||
std::getline(std::cin, password);
|
||||
std::cout << std::endl;
|
||||
}
|
||||
compression = config.getBool("compression", true)
|
||||
? Protocol::Compression::Enable
|
||||
: Protocol::Compression::Disable;
|
||||
|
||||
timeouts = ConnectionTimeouts(
|
||||
Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
|
||||
Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
|
||||
}
|
||||
ConnectionParameters(const Poco::Util::AbstractConfiguration & config);
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
3
dbms/programs/client/config_client.h.in
Normal file
3
dbms/programs/client/config_client.h.in
Normal file
@ -0,0 +1,3 @@
|
||||
#pragma once
|
||||
|
||||
#cmakedefine HAVE_READPASSPHRASE
|
||||
10
dbms/programs/client/readpassphrase/CMakeLists.txt
Normal file
10
dbms/programs/client/readpassphrase/CMakeLists.txt
Normal file
@ -0,0 +1,10 @@
|
||||
|
||||
# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.c
|
||||
# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.h
|
||||
|
||||
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-result -Wno-reserved-id-macro")
|
||||
|
||||
configure_file(includes.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/includes.h)
|
||||
add_library(readpassphrase ${CMAKE_CURRENT_SOURCE_DIR}/readpassphrase.c)
|
||||
# . to allow #include <readpassphrase.h>
|
||||
target_include_directories(readpassphrase PUBLIC . ${CMAKE_CURRENT_BINARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR}/../include)
|
||||
9
dbms/programs/client/readpassphrase/includes.h.in
Normal file
9
dbms/programs/client/readpassphrase/includes.h.in
Normal file
@ -0,0 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#cmakedefine HAVE_READPASSPHRASE
|
||||
|
||||
#if !defined(HAVE_READPASSPHRASE)
|
||||
# ifndef _PATH_TTY
|
||||
# define _PATH_TTY "/dev/tty"
|
||||
# endif
|
||||
#endif
|
||||
211
dbms/programs/client/readpassphrase/readpassphrase.c
Normal file
211
dbms/programs/client/readpassphrase/readpassphrase.c
Normal file
@ -0,0 +1,211 @@
|
||||
/* $OpenBSD: readpassphrase.c,v 1.26 2016/10/18 12:47:18 millert Exp $ */
|
||||
|
||||
/*
|
||||
* Copyright (c) 2000-2002, 2007, 2010
|
||||
* Todd C. Miller <Todd.Miller@courtesan.com>
|
||||
*
|
||||
* Permission to use, copy, modify, and distribute this software for any
|
||||
* purpose with or without fee is hereby granted, provided that the above
|
||||
* copyright notice and this permission notice appear in all copies.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
||||
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
||||
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
||||
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
||||
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
||||
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
||||
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
||||
*
|
||||
* Sponsored in part by the Defense Advanced Research Projects
|
||||
* Agency (DARPA) and Air Force Research Laboratory, Air Force
|
||||
* Materiel Command, USAF, under agreement number F39502-99-1-0512.
|
||||
*/
|
||||
|
||||
/* OPENBSD ORIGINAL: lib/libc/gen/readpassphrase.c */
|
||||
|
||||
#include "includes.h"
|
||||
|
||||
#ifndef HAVE_READPASSPHRASE
|
||||
|
||||
#include <termios.h>
|
||||
#include <signal.h>
|
||||
#include <ctype.h>
|
||||
#include <fcntl.h>
|
||||
#include <readpassphrase.h>
|
||||
#include <errno.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#ifndef TCSASOFT
|
||||
/* If we don't have TCSASOFT define it so that ORing it it below is a no-op. */
|
||||
# define TCSASOFT 0
|
||||
#endif
|
||||
|
||||
/* SunOS 4.x which lacks _POSIX_VDISABLE, but has VDISABLE */
|
||||
#if !defined(_POSIX_VDISABLE) && defined(VDISABLE)
|
||||
# define _POSIX_VDISABLE VDISABLE
|
||||
#endif
|
||||
|
||||
static volatile sig_atomic_t signo[_NSIG];
|
||||
|
||||
static void handler(int);
|
||||
|
||||
char *
|
||||
readpassphrase(const char *prompt, char *buf, size_t bufsiz, int flags)
|
||||
{
|
||||
ssize_t nr;
|
||||
int input, output, save_errno, i, need_restart;
|
||||
char ch, *p, *end;
|
||||
struct termios term, oterm;
|
||||
struct sigaction sa, savealrm, saveint, savehup, savequit, saveterm;
|
||||
struct sigaction savetstp, savettin, savettou, savepipe;
|
||||
|
||||
/* I suppose we could alloc on demand in this case (XXX). */
|
||||
if (bufsiz == 0) {
|
||||
errno = EINVAL;
|
||||
return(NULL);
|
||||
}
|
||||
|
||||
restart:
|
||||
for (i = 0; i < _NSIG; i++)
|
||||
signo[i] = 0;
|
||||
nr = -1;
|
||||
save_errno = 0;
|
||||
need_restart = 0;
|
||||
/*
|
||||
* Read and write to /dev/tty if available. If not, read from
|
||||
* stdin and write to stderr unless a tty is required.
|
||||
*/
|
||||
if ((flags & RPP_STDIN) ||
|
||||
(input = output = open(_PATH_TTY, O_RDWR)) == -1) {
|
||||
if (flags & RPP_REQUIRE_TTY) {
|
||||
errno = ENOTTY;
|
||||
return(NULL);
|
||||
}
|
||||
input = STDIN_FILENO;
|
||||
output = STDERR_FILENO;
|
||||
}
|
||||
|
||||
/*
|
||||
* Turn off echo if possible.
|
||||
* If we are using a tty but are not the foreground pgrp this will
|
||||
* generate SIGTTOU, so do it *before* installing the signal handlers.
|
||||
*/
|
||||
if (input != STDIN_FILENO && tcgetattr(input, &oterm) == 0) {
|
||||
memcpy(&term, &oterm, sizeof(term));
|
||||
if (!(flags & RPP_ECHO_ON))
|
||||
term.c_lflag &= ~(ECHO | ECHONL);
|
||||
#ifdef VSTATUS
|
||||
if (term.c_cc[VSTATUS] != _POSIX_VDISABLE)
|
||||
term.c_cc[VSTATUS] = _POSIX_VDISABLE;
|
||||
#endif
|
||||
(void)tcsetattr(input, TCSAFLUSH|TCSASOFT, &term);
|
||||
} else {
|
||||
memset(&term, 0, sizeof(term));
|
||||
term.c_lflag |= ECHO;
|
||||
memset(&oterm, 0, sizeof(oterm));
|
||||
oterm.c_lflag |= ECHO;
|
||||
}
|
||||
|
||||
/*
|
||||
* Catch signals that would otherwise cause the user to end
|
||||
* up with echo turned off in the shell. Don't worry about
|
||||
* things like SIGXCPU and SIGVTALRM for now.
|
||||
*/
|
||||
sigemptyset(&sa.sa_mask);
|
||||
sa.sa_flags = 0; /* don't restart system calls */
|
||||
sa.sa_handler = handler;
|
||||
(void)sigaction(SIGALRM, &sa, &savealrm);
|
||||
(void)sigaction(SIGHUP, &sa, &savehup);
|
||||
(void)sigaction(SIGINT, &sa, &saveint);
|
||||
(void)sigaction(SIGPIPE, &sa, &savepipe);
|
||||
(void)sigaction(SIGQUIT, &sa, &savequit);
|
||||
(void)sigaction(SIGTERM, &sa, &saveterm);
|
||||
(void)sigaction(SIGTSTP, &sa, &savetstp);
|
||||
(void)sigaction(SIGTTIN, &sa, &savettin);
|
||||
(void)sigaction(SIGTTOU, &sa, &savettou);
|
||||
|
||||
if (!(flags & RPP_STDIN))
|
||||
(void)write(output, prompt, strlen(prompt));
|
||||
end = buf + bufsiz - 1;
|
||||
p = buf;
|
||||
while ((nr = read(input, &ch, 1)) == 1 && ch != '\n' && ch != '\r') {
|
||||
if (p < end) {
|
||||
if ((flags & RPP_SEVENBIT))
|
||||
ch &= 0x7f;
|
||||
if (isalpha((unsigned char)ch)) {
|
||||
if ((flags & RPP_FORCELOWER))
|
||||
ch = (char)tolower((unsigned char)ch);
|
||||
if ((flags & RPP_FORCEUPPER))
|
||||
ch = (char)toupper((unsigned char)ch);
|
||||
}
|
||||
*p++ = ch;
|
||||
}
|
||||
}
|
||||
*p = '\0';
|
||||
save_errno = errno;
|
||||
if (!(term.c_lflag & ECHO))
|
||||
(void)write(output, "\n", 1);
|
||||
|
||||
/* Restore old terminal settings and signals. */
|
||||
if (memcmp(&term, &oterm, sizeof(term)) != 0) {
|
||||
const int sigttou = signo[SIGTTOU];
|
||||
|
||||
/* Ignore SIGTTOU generated when we are not the fg pgrp. */
|
||||
while (tcsetattr(input, TCSAFLUSH|TCSASOFT, &oterm) == -1 &&
|
||||
errno == EINTR && !signo[SIGTTOU])
|
||||
continue;
|
||||
signo[SIGTTOU] = sigttou;
|
||||
}
|
||||
(void)sigaction(SIGALRM, &savealrm, NULL);
|
||||
(void)sigaction(SIGHUP, &savehup, NULL);
|
||||
(void)sigaction(SIGINT, &saveint, NULL);
|
||||
(void)sigaction(SIGQUIT, &savequit, NULL);
|
||||
(void)sigaction(SIGPIPE, &savepipe, NULL);
|
||||
(void)sigaction(SIGTERM, &saveterm, NULL);
|
||||
(void)sigaction(SIGTSTP, &savetstp, NULL);
|
||||
(void)sigaction(SIGTTIN, &savettin, NULL);
|
||||
(void)sigaction(SIGTTOU, &savettou, NULL);
|
||||
if (input != STDIN_FILENO)
|
||||
(void)close(input);
|
||||
|
||||
/*
|
||||
* If we were interrupted by a signal, resend it to ourselves
|
||||
* now that we have restored the signal handlers.
|
||||
*/
|
||||
for (i = 0; i < _NSIG; i++) {
|
||||
if (signo[i]) {
|
||||
kill(getpid(), i);
|
||||
switch (i) {
|
||||
case SIGTSTP:
|
||||
case SIGTTIN:
|
||||
case SIGTTOU:
|
||||
need_restart = 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (need_restart)
|
||||
goto restart;
|
||||
|
||||
if (save_errno)
|
||||
errno = save_errno;
|
||||
return(nr == -1 ? NULL : buf);
|
||||
}
|
||||
//DEF_WEAK(readpassphrase);
|
||||
|
||||
#if 0
|
||||
char *
|
||||
getpass(const char *prompt)
|
||||
{
|
||||
static char buf[_PASSWORD_LEN + 1];
|
||||
|
||||
return(readpassphrase(prompt, buf, sizeof(buf), RPP_ECHO_OFF));
|
||||
}
|
||||
#endif
|
||||
|
||||
static void handler(int s)
|
||||
{
|
||||
|
||||
signo[s] = 1;
|
||||
}
|
||||
#endif /* HAVE_READPASSPHRASE */
|
||||
56
dbms/programs/client/readpassphrase/readpassphrase.h
Normal file
56
dbms/programs/client/readpassphrase/readpassphrase.h
Normal file
@ -0,0 +1,56 @@
|
||||
// /* $OpenBSD: readpassphrase.h,v 1.5 2003/06/17 21:56:23 millert Exp $ */
|
||||
|
||||
/*
|
||||
* Copyright (c) 2000, 2002 Todd C. Miller <Todd.Miller@courtesan.com>
|
||||
*
|
||||
* Permission to use, copy, modify, and distribute this software for any
|
||||
* purpose with or without fee is hereby granted, provided that the above
|
||||
* copyright notice and this permission notice appear in all copies.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
|
||||
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
|
||||
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
|
||||
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
||||
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
|
||||
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
|
||||
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
||||
*
|
||||
* Sponsored in part by the Defense Advanced Research Projects
|
||||
* Agency (DARPA) and Air Force Research Laboratory, Air Force
|
||||
* Materiel Command, USAF, under agreement number F39502-99-1-0512.
|
||||
*/
|
||||
|
||||
/* OPENBSD ORIGINAL: include/readpassphrase.h */
|
||||
|
||||
#pragma once
|
||||
// #ifndef _READPASSPHRASE_H_
|
||||
// #define _READPASSPHRASE_H_
|
||||
|
||||
//#include "includes.h"
|
||||
#include "config_client.h"
|
||||
|
||||
#ifndef HAVE_READPASSPHRASE
|
||||
|
||||
# ifdef __cplusplus
|
||||
extern "C" {
|
||||
# endif
|
||||
|
||||
|
||||
# define RPP_ECHO_OFF 0x00 /* Turn off echo (default). */
|
||||
# define RPP_ECHO_ON 0x01 /* Leave echo on. */
|
||||
# define RPP_REQUIRE_TTY 0x02 /* Fail if there is no tty. */
|
||||
# define RPP_FORCELOWER 0x04 /* Force input to lower case. */
|
||||
# define RPP_FORCEUPPER 0x08 /* Force input to upper case. */
|
||||
# define RPP_SEVENBIT 0x10 /* Strip the high bit from input. */
|
||||
# define RPP_STDIN 0x20 /* Read from stdin, not /dev/tty */
|
||||
|
||||
char * readpassphrase(const char *, char *, size_t, int);
|
||||
|
||||
# ifdef __cplusplus
|
||||
}
|
||||
# endif
|
||||
|
||||
|
||||
#endif /* HAVE_READPASSPHRASE */
|
||||
|
||||
// #endif /* !_READPASSPHRASE_H_ */
|
||||
@ -15,7 +15,7 @@
|
||||
#endif
|
||||
|
||||
#if USE_TCMALLOC
|
||||
#include <gperftools/malloc_extension.h> // Y_IGNORE
|
||||
#include <gperftools/malloc_extension.h>
|
||||
#endif
|
||||
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
|
||||
@ -3,9 +3,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -2,9 +2,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -2,9 +2,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -8,7 +8,7 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
#include <Poco/Data/ODBC/Utility.h>
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
#include "PerformanceTest.h"
|
||||
|
||||
#include <Core/Types.h>
|
||||
#include <Common/CpuId.h>
|
||||
#include <common/getMemoryAmount.h>
|
||||
#include <IO/ReadBufferFromFile.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
@ -71,6 +72,7 @@ bool PerformanceTest::checkPreconditions() const
|
||||
Strings preconditions;
|
||||
config->keys("preconditions", preconditions);
|
||||
size_t table_precondition_index = 0;
|
||||
size_t cpu_precondition_index = 0;
|
||||
|
||||
for (const std::string & precondition : preconditions)
|
||||
{
|
||||
@ -136,6 +138,30 @@ bool PerformanceTest::checkPreconditions() const
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (precondition == "cpu")
|
||||
{
|
||||
std::string precondition_key = "preconditions.cpu[" + std::to_string(cpu_precondition_index++) + "]";
|
||||
std::string flag_to_check = config->getString(precondition_key);
|
||||
|
||||
#define CHECK_CPU_PRECONDITION(OP) \
|
||||
if (flag_to_check == #OP) \
|
||||
{ \
|
||||
if (!Cpu::CpuFlagsCache::have_##OP) \
|
||||
{ \
|
||||
LOG_WARNING(log, "CPU doesn't support " << #OP); \
|
||||
return false; \
|
||||
} \
|
||||
} else
|
||||
|
||||
CPU_ID_ENUMERATE(CHECK_CPU_PRECONDITION)
|
||||
{
|
||||
LOG_WARNING(log, "CPU doesn't support " << flag_to_check);
|
||||
return false;
|
||||
}
|
||||
|
||||
#undef CHECK_CPU_PRECONDITION
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
|
||||
@ -9,6 +9,7 @@ set(CLICKHOUSE_SERVER_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/Server.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/TCPHandler.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})
|
||||
|
||||
@ -312,11 +312,13 @@ void HTTPHandler::processQuery(
|
||||
client_supports_http_compression = true;
|
||||
http_response_compression_method = CompressionMethod::Zlib;
|
||||
}
|
||||
#if USE_BROTLI
|
||||
else if (http_response_compression_methods == "br")
|
||||
{
|
||||
client_supports_http_compression = true;
|
||||
http_response_compression_method = CompressionMethod::Brotli;
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
/// Client can pass a 'compress' flag in the query string. In this case the query result is
|
||||
@ -453,30 +455,6 @@ void HTTPHandler::processQuery(
|
||||
return false;
|
||||
};
|
||||
|
||||
/// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
|
||||
std::string full_query;
|
||||
|
||||
/// Support for "external data for query processing".
|
||||
if (startsWith(request.getContentType().data(), "multipart/form-data"))
|
||||
{
|
||||
ExternalTablesHandler handler(context, params);
|
||||
params.load(request, istr, handler);
|
||||
|
||||
/// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
|
||||
reserved_param_suffixes.emplace_back("_format");
|
||||
reserved_param_suffixes.emplace_back("_types");
|
||||
reserved_param_suffixes.emplace_back("_structure");
|
||||
|
||||
/// Params are of both form params POST and uri (GET params)
|
||||
for (const auto & it : params)
|
||||
if (it.first == "query")
|
||||
full_query += it.second;
|
||||
|
||||
in = std::make_unique<ReadBufferFromString>(full_query);
|
||||
}
|
||||
else
|
||||
in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
|
||||
|
||||
/// Settings can be overridden in the query.
|
||||
/// Some parameters (database, default_format, everything used in the code above) do not
|
||||
/// belong to the Settings class.
|
||||
@ -497,30 +475,63 @@ void HTTPHandler::processQuery(
|
||||
settings.readonly = 2;
|
||||
}
|
||||
|
||||
bool isExternalData = startsWith(request.getContentType().data(), "multipart/form-data");
|
||||
|
||||
if (isExternalData)
|
||||
{
|
||||
/// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
|
||||
reserved_param_suffixes.reserve(3);
|
||||
/// It is a bug and ambiguity with `date_time_input_format` and `low_cardinality_allow_in_native_format` formats/settings.
|
||||
reserved_param_suffixes.emplace_back("_format");
|
||||
reserved_param_suffixes.emplace_back("_types");
|
||||
reserved_param_suffixes.emplace_back("_structure");
|
||||
}
|
||||
|
||||
SettingsChanges settings_changes;
|
||||
for (auto it = params.begin(); it != params.end(); ++it)
|
||||
for (const auto & [key, value] : params)
|
||||
{
|
||||
if (it->first == "database")
|
||||
if (key == "database")
|
||||
{
|
||||
context.setCurrentDatabase(it->second);
|
||||
context.setCurrentDatabase(value);
|
||||
}
|
||||
else if (it->first == "default_format")
|
||||
else if (key == "default_format")
|
||||
{
|
||||
context.setDefaultFormat(it->second);
|
||||
context.setDefaultFormat(value);
|
||||
}
|
||||
else if (param_could_be_skipped(it->first))
|
||||
else if (param_could_be_skipped(key))
|
||||
{
|
||||
}
|
||||
else
|
||||
{
|
||||
/// All other query parameters are treated as settings.
|
||||
settings_changes.push_back({it->first, it->second});
|
||||
settings_changes.push_back({key, value});
|
||||
}
|
||||
}
|
||||
|
||||
/// For external data we also want settings
|
||||
context.checkSettingsConstraints(settings_changes);
|
||||
context.applySettingsChanges(settings_changes);
|
||||
|
||||
/// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
|
||||
std::string full_query;
|
||||
|
||||
/// Support for "external data for query processing".
|
||||
if (isExternalData)
|
||||
{
|
||||
ExternalTablesHandler handler(context, params);
|
||||
params.load(request, istr, handler);
|
||||
|
||||
/// Params are of both form params POST and uri (GET params)
|
||||
for (const auto & it : params)
|
||||
if (it.first == "query")
|
||||
full_query += it.second;
|
||||
|
||||
in = std::make_unique<ReadBufferFromString>(full_query);
|
||||
}
|
||||
else
|
||||
in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
|
||||
|
||||
|
||||
/// HTTP response compression is turned on only if the client signalled that they support it
|
||||
/// (using Accept-Encoding header) and 'enable_http_compression' setting is turned on.
|
||||
used_output.out->setCompression(client_supports_http_compression && settings.enable_http_compression);
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
#include <DataStreams/copyData.h>
|
||||
#include <IO/ReadBufferFromMemory.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/ReadBufferFromPocoSocket.h>
|
||||
#include <IO/WriteBufferFromPocoSocket.h>
|
||||
#include <Interpreters/executeQuery.h>
|
||||
@ -9,12 +9,14 @@
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Common/config_version.h>
|
||||
#include <Common/NetException.h>
|
||||
#include <Common/OpenSSLHelpers.h>
|
||||
#include <Poco/Crypto/RSAKey.h>
|
||||
#include <Poco/Crypto/CipherFactory.h>
|
||||
#include <Poco/Net/SecureStreamSocket.h>
|
||||
#include <Poco/Net/SSLManager.h>
|
||||
#include "MySQLHandler.h"
|
||||
#include <limits>
|
||||
#include <ext/scope_guard.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -25,21 +27,32 @@ using Poco::Net::SSLManager;
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
|
||||
extern const int UNKNOWN_EXCEPTION;
|
||||
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
|
||||
extern const int OPENSSL_ERROR;
|
||||
}
|
||||
|
||||
uint32_t MySQLHandler::last_connection_id = 0;
|
||||
|
||||
MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id)
|
||||
: Poco::Net::TCPServerConnection(socket_)
|
||||
, server(server_)
|
||||
, log(&Poco::Logger::get("MySQLHandler"))
|
||||
, connection_context(server.context())
|
||||
, connection_id(connection_id)
|
||||
, public_key(public_key)
|
||||
, private_key(private_key)
|
||||
{
|
||||
server_capability_flags = CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA | CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF;
|
||||
if (ssl_enabled)
|
||||
server_capability_flags |= CLIENT_SSL;
|
||||
}
|
||||
|
||||
void MySQLHandler::run()
|
||||
{
|
||||
connection_context = server.context();
|
||||
connection_context.setDefaultFormat("MySQL");
|
||||
connection_context.setDefaultFormat("MySQLWire");
|
||||
|
||||
in = std::make_shared<ReadBufferFromPocoSocket>(socket());
|
||||
out = std::make_shared<WriteBufferFromPocoSocket>(socket());
|
||||
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler");
|
||||
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
|
||||
|
||||
try
|
||||
{
|
||||
@ -49,8 +62,7 @@ void MySQLHandler::run()
|
||||
* This plugin must do the same to stay consistent with historical behavior if it is set to operate as a default plugin.
|
||||
* https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L3994
|
||||
*/
|
||||
Handshake handshake(connection_id, VERSION_STRING, scramble + '\0');
|
||||
|
||||
Handshake handshake(server_capability_flags, connection_id, VERSION_STRING + String("-") + VERSION_NAME, scramble + '\0');
|
||||
packet_sender->sendPacket<Handshake>(handshake, true);
|
||||
|
||||
LOG_TRACE(log, "Sent handshake");
|
||||
@ -78,15 +90,11 @@ void MySQLHandler::run()
|
||||
<< "\nauth_plugin_name: "
|
||||
<< handshake_response.auth_plugin_name);
|
||||
|
||||
capabilities = handshake_response.capability_flags;
|
||||
if (!(capabilities & CLIENT_PROTOCOL_41))
|
||||
{
|
||||
client_capability_flags = handshake_response.capability_flags;
|
||||
if (!(client_capability_flags & CLIENT_PROTOCOL_41))
|
||||
throw Exception("Required capability: CLIENT_PROTOCOL_41.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
|
||||
}
|
||||
if (!(capabilities & CLIENT_PLUGIN_AUTH))
|
||||
{
|
||||
if (!(client_capability_flags & CLIENT_PLUGIN_AUTH))
|
||||
throw Exception("Required capability: CLIENT_PLUGIN_AUTH.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
|
||||
}
|
||||
|
||||
authenticate(handshake_response, scramble);
|
||||
OK_Packet ok_packet(0, handshake_response.capability_flags, 0, 0, 0);
|
||||
@ -165,7 +173,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
|
||||
};
|
||||
read_bytes(3); /// We can find out whether it is SSLRequest of HandshakeResponse by first 3 bytes.
|
||||
|
||||
size_t payload_size = *reinterpret_cast<uint32_t *>(buf) & 0xFFFFFFu;
|
||||
size_t payload_size = unalignedLoad<uint32_t>(buf) & 0xFFFFFFu;
|
||||
LOG_TRACE(log, "payload size: " << payload_size);
|
||||
|
||||
if (payload_size == SSL_REQUEST_PAYLOAD_SIZE)
|
||||
@ -180,7 +188,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
|
||||
in = std::make_shared<ReadBufferFromPocoSocket>(*ss);
|
||||
out = std::make_shared<WriteBufferFromPocoSocket>(*ss);
|
||||
connection_context.sequence_id = 2;
|
||||
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler");
|
||||
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
|
||||
packet_sender->max_packet_size = connection_context.max_packet_size;
|
||||
packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket.
|
||||
}
|
||||
@ -216,6 +224,10 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
|
||||
if (handshake_response.auth_plugin_name != Authentication::SHA256)
|
||||
{
|
||||
packet_sender->sendPacket(AuthSwitchRequest(Authentication::SHA256, scramble + '\0'), true);
|
||||
if (in->eof())
|
||||
throw Exception(
|
||||
"Client doesn't support authentication method " + String(Authentication::SHA256) + " used by ClickHouse",
|
||||
ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
|
||||
packet_sender->receivePacket(response);
|
||||
auth_response = response.value;
|
||||
LOG_TRACE(log, "Authentication method mismatch.");
|
||||
@ -226,31 +238,19 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
|
||||
LOG_TRACE(log, "Authentication method match.");
|
||||
}
|
||||
|
||||
auto getOpenSSLError = []() -> String
|
||||
{
|
||||
BIO * mem = BIO_new(BIO_s_mem());
|
||||
ERR_print_errors(mem);
|
||||
char * buf = nullptr;
|
||||
long size = BIO_get_mem_data(mem, &buf);
|
||||
String errors_str(buf, size);
|
||||
BIO_free(mem);
|
||||
return errors_str;
|
||||
};
|
||||
|
||||
if (auth_response == "\1")
|
||||
{
|
||||
LOG_TRACE(log, "Client requests public key.");
|
||||
|
||||
BIO * mem = BIO_new(BIO_s_mem());
|
||||
if (PEM_write_bio_RSA_PUBKEY(mem, public_key) != 1)
|
||||
SCOPE_EXIT(BIO_free(mem));
|
||||
if (PEM_write_bio_RSA_PUBKEY(mem, &public_key) != 1)
|
||||
{
|
||||
LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError());
|
||||
throw Exception("Failed to write public key to memory.", ErrorCodes::UNKNOWN_EXCEPTION);
|
||||
throw Exception("Failed to write public key to memory. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
}
|
||||
char * pem_buf = nullptr;
|
||||
long pem_size = BIO_get_mem_data(mem, &pem_buf);
|
||||
String pem(pem_buf, pem_size);
|
||||
BIO_free(mem);
|
||||
|
||||
LOG_TRACE(log, "Key: " << pem);
|
||||
|
||||
@ -271,17 +271,16 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
|
||||
* an empty packet is a blank password, thus the check for auth_response.empty() has to be made too.
|
||||
* https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L4017
|
||||
*/
|
||||
if (!secure_connection && (!auth_response.empty() && auth_response != "\0"))
|
||||
if (!secure_connection && !auth_response.empty() && auth_response != String("\0", 1))
|
||||
{
|
||||
LOG_TRACE(log, "Received nonempty password");
|
||||
auto ciphertext = reinterpret_cast<unsigned char *>(auth_response.data());
|
||||
|
||||
unsigned char plaintext[RSA_size(private_key)];
|
||||
int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, private_key, RSA_PKCS1_OAEP_PADDING);
|
||||
unsigned char plaintext[RSA_size(&private_key)];
|
||||
int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, &private_key, RSA_PKCS1_OAEP_PADDING);
|
||||
if (plaintext_size == -1)
|
||||
{
|
||||
LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError());
|
||||
throw Exception("Failed to decrypt.", ErrorCodes::UNKNOWN_EXCEPTION);
|
||||
throw Exception("Failed to decrypt auth data. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
}
|
||||
|
||||
password.resize(plaintext_size);
|
||||
@ -324,7 +323,7 @@ void MySQLHandler::comInitDB(const String & payload)
|
||||
String database = payload.substr(1);
|
||||
LOG_DEBUG(log, "Setting current database to " << database);
|
||||
connection_context.setCurrentDatabase(database);
|
||||
packet_sender->sendPacket(OK_Packet(0, capabilities, 0, 0, 1), true);
|
||||
packet_sender->sendPacket(OK_Packet(0, client_capability_flags, 0, 0, 1), true);
|
||||
}
|
||||
|
||||
void MySQLHandler::comFieldList(const String & payload)
|
||||
@ -340,12 +339,12 @@ void MySQLHandler::comFieldList(const String & payload)
|
||||
);
|
||||
packet_sender->sendPacket(column_definition);
|
||||
}
|
||||
packet_sender->sendPacket(OK_Packet(0xfe, capabilities, 0, 0, 0), true);
|
||||
packet_sender->sendPacket(OK_Packet(0xfe, client_capability_flags, 0, 0, 0), true);
|
||||
}
|
||||
|
||||
void MySQLHandler::comPing()
|
||||
{
|
||||
packet_sender->sendPacket(OK_Packet(0x0, capabilities, 0, 0, 0), true);
|
||||
packet_sender->sendPacket(OK_Packet(0x0, client_capability_flags, 0, 0, 0), true);
|
||||
}
|
||||
|
||||
void MySQLHandler::comQuery(const String & payload)
|
||||
@ -354,10 +353,18 @@ void MySQLHandler::comQuery(const String & payload)
|
||||
std::function<void(const String &)> set_content_type = [&with_output](const String &) -> void {
|
||||
with_output = true;
|
||||
};
|
||||
ReadBufferFromMemory query(payload.data() + 1, payload.size() - 1);
|
||||
executeQuery(query, *out, true, connection_context, set_content_type, nullptr);
|
||||
|
||||
String query = payload.substr(1);
|
||||
|
||||
// Translate query from MySQL to ClickHouse.
|
||||
// This is a temporary workaround until ClickHouse supports the syntax "@@var_name".
|
||||
if (query == "select @@version_comment limit 1") // MariaDB client starts session with that query
|
||||
query = "select ''";
|
||||
|
||||
ReadBufferFromString buf(query);
|
||||
executeQuery(buf, *out, true, connection_context, set_content_type, nullptr);
|
||||
if (!with_output)
|
||||
packet_sender->sendPacket(OK_Packet(0x00, capabilities, 0, 0, 0), true);
|
||||
packet_sender->sendPacket(OK_Packet(0x00, client_capability_flags, 0, 0, 0), true);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
#include <Poco/Net/SecureStreamSocket.h>
|
||||
#include <Common/getFQDNOrHostName.h>
|
||||
#include <Core/MySQLProtocol.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <openssl/rsa.h>
|
||||
#include "IServer.h"
|
||||
|
||||
|
||||
@ -15,21 +15,7 @@ namespace DB
|
||||
class MySQLHandler : public Poco::Net::TCPServerConnection
|
||||
{
|
||||
public:
|
||||
MySQLHandler(
|
||||
IServer & server_,
|
||||
const Poco::Net::StreamSocket & socket_,
|
||||
RSA * public_key,
|
||||
RSA * private_key)
|
||||
: Poco::Net::TCPServerConnection(socket_)
|
||||
, server(server_)
|
||||
, log(&Poco::Logger::get("MySQLHandler"))
|
||||
, connection_context(server.context())
|
||||
, connection_id(last_connection_id++)
|
||||
, public_key(public_key)
|
||||
, private_key(private_key)
|
||||
{
|
||||
log->setLevel("information");
|
||||
}
|
||||
MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id);
|
||||
|
||||
void run() final;
|
||||
|
||||
@ -55,13 +41,13 @@ private:
|
||||
|
||||
std::shared_ptr<MySQLProtocol::PacketSender> packet_sender;
|
||||
|
||||
uint32_t connection_id = 0;
|
||||
size_t connection_id = 0;
|
||||
|
||||
uint32_t capabilities;
|
||||
size_t server_capability_flags;
|
||||
size_t client_capability_flags;
|
||||
|
||||
static uint32_t last_connection_id;
|
||||
|
||||
RSA * public_key, * private_key;
|
||||
RSA & public_key;
|
||||
RSA & private_key;
|
||||
|
||||
std::shared_ptr<ReadBuffer> in;
|
||||
std::shared_ptr<WriteBuffer> out;
|
||||
|
||||
124
dbms/programs/server/MySQLHandlerFactory.cpp
Normal file
124
dbms/programs/server/MySQLHandlerFactory.cpp
Normal file
@ -0,0 +1,124 @@
|
||||
#include <Common/OpenSSLHelpers.h>
|
||||
#include <Poco/Crypto/X509Certificate.h>
|
||||
#include <Poco/Net/SSLManager.h>
|
||||
#include <Poco/Net/TCPServerConnectionFactory.h>
|
||||
#include <Poco/Util/Application.h>
|
||||
#include <common/logger_useful.h>
|
||||
#include <ext/scope_guard.h>
|
||||
#include "IServer.h"
|
||||
#include "MySQLHandler.h"
|
||||
#include "MySQLHandlerFactory.h"
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int CANNOT_OPEN_FILE;
|
||||
extern const int NO_ELEMENTS_IN_CONFIG;
|
||||
extern const int OPENSSL_ERROR;
|
||||
extern const int SYSTEM_ERROR;
|
||||
}
|
||||
|
||||
MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
|
||||
: server(server_)
|
||||
, log(&Logger::get("MySQLHandlerFactory"))
|
||||
{
|
||||
try
|
||||
{
|
||||
Poco::Net::SSLManager::instance().defaultServerContext();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
LOG_INFO(log, "Failed to create SSL context. SSL will be disabled. Error: " << getCurrentExceptionMessage(false));
|
||||
ssl_enabled = false;
|
||||
}
|
||||
|
||||
/// Reading rsa keys for SHA256 authentication plugin.
|
||||
try
|
||||
{
|
||||
readRSAKeys();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
LOG_WARNING(log, "Failed to read RSA keys. Error: " << getCurrentExceptionMessage(false));
|
||||
generateRSAKeys();
|
||||
}
|
||||
}
|
||||
|
||||
void MySQLHandlerFactory::readRSAKeys()
|
||||
{
|
||||
const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
|
||||
String certificateFileProperty = "openSSL.server.certificateFile";
|
||||
String privateKeyFileProperty = "openSSL.server.privateKeyFile";
|
||||
|
||||
if (!config.has(certificateFileProperty))
|
||||
throw Exception("Certificate file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
|
||||
|
||||
if (!config.has(privateKeyFileProperty))
|
||||
throw Exception("Private key file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
|
||||
|
||||
{
|
||||
String certificateFile = config.getString(certificateFileProperty);
|
||||
FILE * fp = fopen(certificateFile.data(), "r");
|
||||
if (fp == nullptr)
|
||||
throw Exception("Cannot open certificate file: " + certificateFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
|
||||
SCOPE_EXIT(fclose(fp));
|
||||
|
||||
X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
|
||||
SCOPE_EXIT(X509_free(x509));
|
||||
if (x509 == nullptr)
|
||||
throw Exception("Failed to read PEM certificate from " + certificateFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
|
||||
EVP_PKEY * p = X509_get_pubkey(x509);
|
||||
if (p == nullptr)
|
||||
throw Exception("Failed to get RSA key from X509. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
SCOPE_EXIT(EVP_PKEY_free(p));
|
||||
|
||||
public_key.reset(EVP_PKEY_get1_RSA(p));
|
||||
if (public_key.get() == nullptr)
|
||||
throw Exception("Failed to get RSA key from ENV_PKEY. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
}
|
||||
|
||||
{
|
||||
String privateKeyFile = config.getString(privateKeyFileProperty);
|
||||
|
||||
FILE * fp = fopen(privateKeyFile.data(), "r");
|
||||
if (fp == nullptr)
|
||||
throw Exception ("Cannot open private key file " + privateKeyFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
|
||||
SCOPE_EXIT(fclose(fp));
|
||||
|
||||
private_key.reset(PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr));
|
||||
if (!private_key)
|
||||
throw Exception("Failed to read RSA private key from " + privateKeyFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
}
|
||||
}
|
||||
|
||||
void MySQLHandlerFactory::generateRSAKeys()
|
||||
{
|
||||
LOG_INFO(log, "Generating new RSA key.");
|
||||
public_key.reset(RSA_new());
|
||||
if (!public_key)
|
||||
throw Exception("Failed to allocate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
|
||||
BIGNUM * e = BN_new();
|
||||
if (!e)
|
||||
throw Exception("Failed to allocate BIGNUM. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
SCOPE_EXIT(BN_free(e));
|
||||
|
||||
if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(public_key.get(), 2048, e, nullptr))
|
||||
throw Exception("Failed to generate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
|
||||
private_key.reset(RSAPrivateKey_dup(public_key.get()));
|
||||
if (!private_key)
|
||||
throw Exception("Failed to copy RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
|
||||
}
|
||||
|
||||
Poco::Net::TCPServerConnection * MySQLHandlerFactory::createConnection(const Poco::Net::StreamSocket & socket)
|
||||
{
|
||||
size_t connection_id = last_connection_id++;
|
||||
LOG_TRACE(log, "MySQL connection. Id: " << connection_id << ". Address: " << socket.peerAddress().toString());
|
||||
return new MySQLHandler(server, socket, *public_key, *private_key, ssl_enabled, connection_id);
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,16 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#include <Poco/Net/TCPServerConnectionFactory.h>
|
||||
#include <Poco/Net/SSLManager.h>
|
||||
#include <Poco/Crypto/X509Certificate.h>
|
||||
#include <common/logger_useful.h>
|
||||
#include <atomic>
|
||||
#include <openssl/rsa.h>
|
||||
#include "IServer.h"
|
||||
#include "MySQLHandler.h"
|
||||
|
||||
namespace Poco
|
||||
{
|
||||
class Logger;
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -20,94 +13,27 @@ class MySQLHandlerFactory : public Poco::Net::TCPServerConnectionFactory
|
||||
private:
|
||||
IServer & server;
|
||||
Poco::Logger * log;
|
||||
RSA * public_key = nullptr, * private_key = nullptr;
|
||||
|
||||
struct RSADeleter
|
||||
{
|
||||
void operator()(RSA * ptr) { RSA_free(ptr); }
|
||||
};
|
||||
using RSAPtr = std::unique_ptr<RSA, RSADeleter>;
|
||||
|
||||
RSAPtr public_key;
|
||||
RSAPtr private_key;
|
||||
|
||||
bool ssl_enabled = true;
|
||||
|
||||
std::atomic<size_t> last_connection_id = 0;
|
||||
public:
|
||||
explicit MySQLHandlerFactory(IServer & server_)
|
||||
: server(server_), log(&Logger::get("MySQLHandlerFactory"))
|
||||
{
|
||||
/// Reading rsa keys for SHA256 authentication plugin.
|
||||
const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
|
||||
String certificateFileProperty = "openSSL.server.certificateFile";
|
||||
String privateKeyFileProperty = "openSSL.server.privateKeyFile";
|
||||
explicit MySQLHandlerFactory(IServer & server_);
|
||||
|
||||
if (!config.has(certificateFileProperty))
|
||||
{
|
||||
LOG_INFO(log, "Certificate file is not set.");
|
||||
generateRSAKeys();
|
||||
return;
|
||||
}
|
||||
if (!config.has(privateKeyFileProperty))
|
||||
{
|
||||
LOG_INFO(log, "Private key file is not set.");
|
||||
generateRSAKeys();
|
||||
return;
|
||||
}
|
||||
void readRSAKeys();
|
||||
|
||||
String certificateFile = config.getString(certificateFileProperty);
|
||||
FILE * fp = fopen(certificateFile.data(), "r");
|
||||
if (fp == nullptr)
|
||||
{
|
||||
LOG_WARNING(log, "Cannot open certificate file: " << certificateFile << ".");
|
||||
generateRSAKeys();
|
||||
return;
|
||||
}
|
||||
X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
|
||||
EVP_PKEY * p = X509_get_pubkey(x509);
|
||||
public_key = EVP_PKEY_get1_RSA(p);
|
||||
X509_free(x509);
|
||||
EVP_PKEY_free(p);
|
||||
fclose(fp);
|
||||
void generateRSAKeys();
|
||||
|
||||
String privateKeyFile = config.getString(privateKeyFileProperty);
|
||||
fp = fopen(privateKeyFile.data(), "r");
|
||||
if (fp == nullptr)
|
||||
{
|
||||
LOG_WARNING(log, "Cannot open private key file " << privateKeyFile << ".");
|
||||
generateRSAKeys();
|
||||
return;
|
||||
}
|
||||
private_key = PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr);
|
||||
fclose(fp);
|
||||
}
|
||||
|
||||
void generateRSAKeys()
|
||||
{
|
||||
LOG_INFO(log, "Generating new RSA key.");
|
||||
RSA * rsa = RSA_new();
|
||||
if (rsa == nullptr)
|
||||
{
|
||||
throw Exception("Failed to allocate RSA key.", 1002);
|
||||
}
|
||||
BIGNUM * e = BN_new();
|
||||
if (!e)
|
||||
{
|
||||
RSA_free(rsa);
|
||||
throw Exception("Failed to allocate BIGNUM.", 1002);
|
||||
}
|
||||
if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(rsa, 2048, e, nullptr))
|
||||
{
|
||||
RSA_free(rsa);
|
||||
BN_free(e);
|
||||
throw Exception("Failed to generate RSA key.", 1002);
|
||||
}
|
||||
BN_free(e);
|
||||
|
||||
public_key = rsa;
|
||||
private_key = RSAPrivateKey_dup(rsa);
|
||||
}
|
||||
|
||||
Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override
|
||||
{
|
||||
LOG_TRACE(log, "MySQL connection. Address: " << socket.peerAddress().toString());
|
||||
return new MySQLHandler(server, socket, public_key, private_key);
|
||||
}
|
||||
|
||||
~MySQLHandlerFactory() override
|
||||
{
|
||||
RSA_free(public_key);
|
||||
RSA_free(private_key);
|
||||
}
|
||||
Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -1,10 +1,10 @@
|
||||
#pragma once
|
||||
|
||||
#include <roaring.h>
|
||||
#include <roaring/roaring.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <boost/noncopyable.hpp>
|
||||
#include <roaring.hh>
|
||||
#include <roaring/roaring.hh>
|
||||
#include <Common/HashTable/SmallTable.h>
|
||||
#include <Common/PODArray.h>
|
||||
|
||||
|
||||
@ -3,6 +3,8 @@
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <Interpreters/castColumn.h>
|
||||
#include <Columns/ColumnArray.h>
|
||||
#include <Columns/ColumnTuple.h>
|
||||
#include <Common/FieldVisitors.h>
|
||||
#include <Common/typeid_cast.h>
|
||||
#include "AggregateFunctionFactory.h"
|
||||
@ -44,10 +46,10 @@ namespace
|
||||
/// Such default parameters were picked because they did good on some tests,
|
||||
/// though it still requires to fit parameters to achieve better result
|
||||
auto learning_rate = Float64(0.01);
|
||||
auto l2_reg_coef = Float64(0.01);
|
||||
UInt32 batch_size = 1;
|
||||
auto l2_reg_coef = Float64(0.1);
|
||||
UInt32 batch_size = 15;
|
||||
|
||||
std::shared_ptr<IWeightsUpdater> weights_updater = std::make_shared<StochasticGradientDescent>();
|
||||
std::string weights_updater_name = "\'SGD\'";
|
||||
std::shared_ptr<IGradientComputer> gradient_computer;
|
||||
|
||||
if (!parameters.empty())
|
||||
@ -64,19 +66,8 @@ namespace
|
||||
}
|
||||
if (parameters.size() > 3)
|
||||
{
|
||||
if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'SGD\'")
|
||||
{
|
||||
weights_updater = std::make_shared<StochasticGradientDescent>();
|
||||
}
|
||||
else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Momentum\'")
|
||||
{
|
||||
weights_updater = std::make_shared<Momentum>();
|
||||
}
|
||||
else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Nesterov\'")
|
||||
{
|
||||
weights_updater = std::make_shared<Nesterov>();
|
||||
}
|
||||
else
|
||||
weights_updater_name = applyVisitor(FieldVisitorToString(), parameters[3]);
|
||||
if (weights_updater_name != "\'SGD\'" && weights_updater_name != "\'Momentum\'" && weights_updater_name != "\'Nesterov\'")
|
||||
{
|
||||
throw Exception("Invalid parameter for weights updater", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
@ -98,20 +89,19 @@ namespace
|
||||
return std::make_shared<Method>(
|
||||
argument_types.size() - 1,
|
||||
gradient_computer,
|
||||
weights_updater,
|
||||
weights_updater_name,
|
||||
learning_rate,
|
||||
l2_reg_coef,
|
||||
batch_size,
|
||||
argument_types,
|
||||
parameters);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
void registerAggregateFunctionMLMethod(AggregateFunctionFactory & factory)
|
||||
{
|
||||
factory.registerFunction("linearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>);
|
||||
factory.registerFunction("logisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>);
|
||||
factory.registerFunction("stochasticLinearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>);
|
||||
factory.registerFunction("stochasticLogisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>);
|
||||
}
|
||||
|
||||
LinearModelData::LinearModelData(
|
||||
@ -144,9 +134,34 @@ void LinearModelData::update_state()
|
||||
}
|
||||
|
||||
void LinearModelData::predict(
|
||||
ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const
|
||||
{
|
||||
gradient_computer->predict(container, block, arguments, weights, bias, context);
|
||||
gradient_computer->predict(container, block, offset, limit, arguments, weights, bias, context);
|
||||
}
|
||||
|
||||
void LinearModelData::returnWeights(IColumn & to) const
|
||||
{
|
||||
size_t size = weights.size() + 1;
|
||||
|
||||
ColumnArray & arr_to = static_cast<ColumnArray &>(to);
|
||||
ColumnArray::Offsets & offsets_to = arr_to.getOffsets();
|
||||
|
||||
size_t old_size = offsets_to.back();
|
||||
offsets_to.push_back(old_size + size);
|
||||
|
||||
typename ColumnFloat64::Container & val_to
|
||||
= static_cast<ColumnFloat64 &>(arr_to.getData()).getData();
|
||||
|
||||
val_to.reserve(old_size + size);
|
||||
for (size_t i = 0; i + 1 < size; ++i)
|
||||
val_to.push_back(weights[i]);
|
||||
|
||||
val_to.push_back(bias);
|
||||
}
|
||||
|
||||
void LinearModelData::read(ReadBuffer & buf)
|
||||
@ -192,7 +207,8 @@ void LinearModelData::merge(const DB::LinearModelData & rhs)
|
||||
void LinearModelData::add(const IColumn ** columns, size_t row_num)
|
||||
{
|
||||
/// first column stores target; features start from (columns + 1)
|
||||
const auto target = (*columns[0])[row_num].get<Float64>();
|
||||
Float64 target = (*columns[0]).getFloat64(row_num);
|
||||
|
||||
/// Here we have columns + 1 as first column corresponds to target value, and others - to features
|
||||
weights_updater->add_to_batch(
|
||||
gradient_batch, *gradient_computer, weights, bias, learning_rate, l2_reg_coef, target, columns + 1, row_num);
|
||||
@ -334,42 +350,38 @@ void IWeightsUpdater::add_to_batch(
|
||||
void LogisticRegression::predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
const Context & context) const
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
size_t rows_num = block.rows();
|
||||
std::vector<Float64> results(rows_num, bias);
|
||||
|
||||
if (offset > rows_num || offset + limit > rows_num)
|
||||
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
|
||||
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
|
||||
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
std::vector<Float64> results(limit, bias);
|
||||
|
||||
for (size_t i = 1; i < arguments.size(); ++i)
|
||||
{
|
||||
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
|
||||
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
{
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
auto & features_column = cur_col.column;
|
||||
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
results[row_num] += weights[i - 1] * features_column->getFloat64(offset + row_num);
|
||||
}
|
||||
|
||||
/// If column type is already Float64 then castColumn simply returns it
|
||||
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
|
||||
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
|
||||
|
||||
if (!features_column)
|
||||
{
|
||||
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
|
||||
}
|
||||
}
|
||||
|
||||
container.reserve(rows_num);
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
container.reserve(container.size() + limit);
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
container.emplace_back(1 / (1 + exp(-results[row_num])));
|
||||
}
|
||||
}
|
||||
|
||||
void LogisticRegression::compute(
|
||||
@ -385,7 +397,7 @@ void LogisticRegression::compute(
|
||||
Float64 derivative = bias;
|
||||
for (size_t i = 0; i < weights.size(); ++i)
|
||||
{
|
||||
auto value = (*columns[i])[row_num].get<Float64>();
|
||||
auto value = (*columns[i]).getFloat64(row_num);
|
||||
derivative += weights[i] * value;
|
||||
}
|
||||
derivative *= target;
|
||||
@ -394,18 +406,20 @@ void LogisticRegression::compute(
|
||||
batch_gradient[weights.size()] += learning_rate * target / (derivative + 1);
|
||||
for (size_t i = 0; i < weights.size(); ++i)
|
||||
{
|
||||
auto value = (*columns[i])[row_num].get<Float64>();
|
||||
batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * l2_reg_coef * weights[i];
|
||||
auto value = (*columns[i]).getFloat64(row_num);
|
||||
batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * learning_rate * l2_reg_coef * weights[i];
|
||||
}
|
||||
}
|
||||
|
||||
void LinearRegression::predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
const Context & context) const
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
if (weights.size() + 1 != arguments.size())
|
||||
{
|
||||
@ -413,36 +427,33 @@ void LinearRegression::predict(
|
||||
}
|
||||
|
||||
size_t rows_num = block.rows();
|
||||
std::vector<Float64> results(rows_num, bias);
|
||||
|
||||
if (offset > rows_num || offset + limit > rows_num)
|
||||
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
|
||||
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
|
||||
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
std::vector<Float64> results(limit, bias);
|
||||
|
||||
for (size_t i = 1; i < arguments.size(); ++i)
|
||||
{
|
||||
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
{
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
}
|
||||
|
||||
/// If column type is already Float64 then castColumn simply returns it
|
||||
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
|
||||
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
auto features_column = cur_col.column;
|
||||
|
||||
if (!features_column)
|
||||
{
|
||||
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
results[row_num] += weights[i - 1] * features_column->getFloat64(row_num + offset);
|
||||
}
|
||||
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
|
||||
}
|
||||
}
|
||||
|
||||
container.reserve(rows_num);
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
container.reserve(container.size() + limit);
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
container.emplace_back(results[row_num]);
|
||||
}
|
||||
}
|
||||
|
||||
void LinearRegression::compute(
|
||||
@ -458,7 +469,7 @@ void LinearRegression::compute(
|
||||
Float64 derivative = (target - bias);
|
||||
for (size_t i = 0; i < weights.size(); ++i)
|
||||
{
|
||||
auto value = (*columns[i])[row_num].get<Float64>();
|
||||
auto value = (*columns[i]).getFloat64(row_num);
|
||||
derivative -= weights[i] * value;
|
||||
}
|
||||
derivative *= (2 * learning_rate);
|
||||
@ -466,8 +477,8 @@ void LinearRegression::compute(
|
||||
batch_gradient[weights.size()] += derivative;
|
||||
for (size_t i = 0; i < weights.size(); ++i)
|
||||
{
|
||||
auto value = (*columns[i])[row_num].get<Float64>();
|
||||
batch_gradient[i] += derivative * value - 2 * l2_reg_coef * weights[i];
|
||||
auto value = (*columns[i]).getFloat64(row_num);
|
||||
batch_gradient[i] += derivative * value - 2 * learning_rate * l2_reg_coef * weights[i];
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -3,7 +3,10 @@
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnsCommon.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <Common/typeid_cast.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeTuple.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include "IAggregateFunction.h"
|
||||
|
||||
namespace DB
|
||||
@ -12,6 +15,7 @@ namespace ErrorCodes
|
||||
{
|
||||
extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
|
||||
extern const int BAD_ARGUMENTS;
|
||||
extern const int BAD_CAST;
|
||||
}
|
||||
|
||||
/**
|
||||
@ -39,6 +43,8 @@ public:
|
||||
virtual void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -64,6 +70,8 @@ public:
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -89,6 +97,8 @@ public:
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -215,20 +225,26 @@ public:
|
||||
|
||||
void read(ReadBuffer & buf);
|
||||
|
||||
void
|
||||
predict(ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const;
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const;
|
||||
|
||||
void returnWeights(IColumn & to) const;
|
||||
private:
|
||||
std::vector<Float64> weights;
|
||||
Float64 bias{0.0};
|
||||
|
||||
Float64 learning_rate;
|
||||
Float64 l2_reg_coef;
|
||||
UInt32 batch_capacity;
|
||||
UInt64 batch_capacity;
|
||||
|
||||
UInt32 iter_num = 0;
|
||||
UInt64 iter_num = 0;
|
||||
std::vector<Float64> gradient_batch;
|
||||
UInt32 batch_size;
|
||||
UInt64 batch_size;
|
||||
|
||||
std::shared_ptr<IGradientComputer> gradient_computer;
|
||||
std::shared_ptr<IWeightsUpdater> weights_updater;
|
||||
@ -253,7 +269,7 @@ public:
|
||||
explicit AggregateFunctionMLMethod(
|
||||
UInt32 param_num,
|
||||
std::shared_ptr<IGradientComputer> gradient_computer,
|
||||
std::shared_ptr<IWeightsUpdater> weights_updater,
|
||||
std::string weights_updater_name,
|
||||
Float64 learning_rate,
|
||||
Float64 l2_reg_coef,
|
||||
UInt32 batch_size,
|
||||
@ -265,15 +281,39 @@ public:
|
||||
, l2_reg_coef(l2_reg_coef)
|
||||
, batch_size(batch_size)
|
||||
, gradient_computer(std::move(gradient_computer))
|
||||
, weights_updater(std::move(weights_updater))
|
||||
, weights_updater_name(std::move(weights_updater_name))
|
||||
{
|
||||
}
|
||||
|
||||
DataTypePtr getReturnType() const override { return std::make_shared<DataTypeNumber<Float64>>(); }
|
||||
/// This function is called when SELECT linearRegression(...) is called
|
||||
DataTypePtr getReturnType() const override
|
||||
{
|
||||
return std::make_shared<DataTypeArray>(std::make_shared<DataTypeFloat64>());
|
||||
}
|
||||
|
||||
/// This function is called from evalMLMethod function for correct predictValues call
|
||||
DataTypePtr getReturnTypeToPredict() const override
|
||||
{
|
||||
return std::make_shared<DataTypeNumber<Float64>>();
|
||||
}
|
||||
|
||||
void create(AggregateDataPtr place) const override
|
||||
{
|
||||
new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, weights_updater);
|
||||
std::shared_ptr<IWeightsUpdater> new_weights_updater;
|
||||
if (weights_updater_name == "\'SGD\'")
|
||||
{
|
||||
new_weights_updater = std::make_shared<StochasticGradientDescent>();
|
||||
} else if (weights_updater_name == "\'Momentum\'")
|
||||
{
|
||||
new_weights_updater = std::make_shared<Momentum>();
|
||||
} else if (weights_updater_name == "\'Nesterov\'")
|
||||
{
|
||||
new_weights_updater = std::make_shared<Nesterov>();
|
||||
} else
|
||||
{
|
||||
throw Exception("Illegal name of weights updater (should have been checked earlier)", ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, new_weights_updater);
|
||||
}
|
||||
|
||||
void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
|
||||
@ -288,7 +328,13 @@ public:
|
||||
void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override { this->data(place).read(buf); }
|
||||
|
||||
void predictValues(
|
||||
ConstAggregateDataPtr place, IColumn & to, Block & block, const ColumnNumbers & arguments, const Context & context) const override
|
||||
ConstAggregateDataPtr place,
|
||||
IColumn & to,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const override
|
||||
{
|
||||
if (arguments.size() != param_num + 1)
|
||||
throw Exception(
|
||||
@ -296,16 +342,21 @@ public:
|
||||
+ ". Required: " + std::to_string(param_num + 1),
|
||||
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
|
||||
auto & column = dynamic_cast<ColumnVector<Float64> &>(to);
|
||||
/// This cast might be correct because column type is based on getReturnTypeToPredict.
|
||||
auto * column = typeid_cast<ColumnFloat64 *>(&to);
|
||||
if (!column)
|
||||
throw Exception("Cast of column of predictions is incorrect. getReturnTypeToPredict must return same value as it is casted to",
|
||||
ErrorCodes::BAD_CAST);
|
||||
|
||||
this->data(place).predict(column.getData(), block, arguments, context);
|
||||
this->data(place).predict(column->getData(), block, offset, limit, arguments, context);
|
||||
}
|
||||
|
||||
/** This function is called if aggregate function without State modifier is selected in a query.
|
||||
* Inserts all weights of the model into the column 'to', so user may use such information if needed
|
||||
*/
|
||||
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
|
||||
{
|
||||
std::ignore = place;
|
||||
std::ignore = to;
|
||||
throw std::runtime_error("not implemented");
|
||||
this->data(place).returnWeights(to);
|
||||
}
|
||||
|
||||
const char * getHeaderFilePath() const override { return __FILE__; }
|
||||
@ -316,15 +367,15 @@ private:
|
||||
Float64 l2_reg_coef;
|
||||
UInt32 batch_size;
|
||||
std::shared_ptr<IGradientComputer> gradient_computer;
|
||||
std::shared_ptr<IWeightsUpdater> weights_updater;
|
||||
std::string weights_updater_name;
|
||||
};
|
||||
|
||||
struct NameLinearRegression
|
||||
{
|
||||
static constexpr auto name = "linearRegression";
|
||||
static constexpr auto name = "stochasticLinearRegression";
|
||||
};
|
||||
struct NameLogisticRegression
|
||||
{
|
||||
static constexpr auto name = "logisticRegression";
|
||||
static constexpr auto name = "stochasticLogisticRegression";
|
||||
};
|
||||
}
|
||||
|
||||
@ -48,6 +48,12 @@ public:
|
||||
/// Get the result type.
|
||||
virtual DataTypePtr getReturnType() const = 0;
|
||||
|
||||
/// Get type which will be used for prediction result in case if function is an ML method.
|
||||
virtual DataTypePtr getReturnTypeToPredict() const
|
||||
{
|
||||
throw Exception("Prediction is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
virtual ~IAggregateFunction() {}
|
||||
|
||||
/** Data manipulating functions. */
|
||||
@ -94,9 +100,16 @@ public:
|
||||
/// Inserts results into a column.
|
||||
virtual void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const = 0;
|
||||
|
||||
/// This function is used for machine learning methods
|
||||
virtual void predictValues(ConstAggregateDataPtr /* place */, IColumn & /*to*/,
|
||||
Block & /*block*/, const ColumnNumbers & /*arguments*/, const Context & /*context*/) const
|
||||
/// Used for machine learning methods. Predict result from trained model.
|
||||
/// Will insert result into `to` column for rows in range [offset, offset + limit).
|
||||
virtual void predictValues(
|
||||
ConstAggregateDataPtr /* place */,
|
||||
IColumn & /*to*/,
|
||||
Block & /*block*/,
|
||||
size_t /*offset*/,
|
||||
size_t /*limit*/,
|
||||
const ColumnNumbers & /*arguments*/,
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
throw Exception("Method predictValues is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
@ -35,25 +35,6 @@ void ColumnAggregateFunction::addArena(ArenaPtr arena_)
|
||||
arenas.push_back(arena_);
|
||||
}
|
||||
|
||||
/// This function is used in convertToValues() and predictValues()
|
||||
/// and is written here to avoid repetitions
|
||||
bool ColumnAggregateFunction::tryFinalizeAggregateFunction(MutableColumnPtr *res_) const
|
||||
{
|
||||
if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
|
||||
{
|
||||
auto res = createView();
|
||||
res->set(function_state->getNestedFunction());
|
||||
res->data.assign(data.begin(), data.end());
|
||||
*res_ = std::move(res);
|
||||
return true;
|
||||
}
|
||||
|
||||
MutableColumnPtr res = func->getReturnType()->createColumn();
|
||||
res->reserve(data.size());
|
||||
*res_ = std::move(res);
|
||||
return false;
|
||||
}
|
||||
|
||||
MutableColumnPtr ColumnAggregateFunction::convertToValues() const
|
||||
{
|
||||
/** If the aggregate function returns an unfinalized/unfinished state,
|
||||
@ -86,17 +67,17 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
|
||||
* AggregateFunction(quantileTiming(0.5), UInt64)
|
||||
* into UInt16 - already finished result of `quantileTiming`.
|
||||
*/
|
||||
|
||||
/** Convertion function is used in convertToValues and predictValues
|
||||
* in the similar part of both functions
|
||||
*/
|
||||
|
||||
MutableColumnPtr res;
|
||||
if (tryFinalizeAggregateFunction(&res))
|
||||
if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
|
||||
{
|
||||
auto res = createView();
|
||||
res->set(function_state->getNestedFunction());
|
||||
res->data.assign(data.begin(), data.end());
|
||||
return res;
|
||||
}
|
||||
|
||||
MutableColumnPtr res = func->getReturnType()->createColumn();
|
||||
res->reserve(data.size());
|
||||
|
||||
for (auto val : data)
|
||||
func->insertResultInto(val, *res);
|
||||
|
||||
@ -105,19 +86,27 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
|
||||
|
||||
MutableColumnPtr ColumnAggregateFunction::predictValues(Block & block, const ColumnNumbers & arguments, const Context & context) const
|
||||
{
|
||||
MutableColumnPtr res;
|
||||
tryFinalizeAggregateFunction(&res);
|
||||
MutableColumnPtr res = func->getReturnTypeToPredict()->createColumn();
|
||||
res->reserve(data.size());
|
||||
|
||||
auto ML_function = func.get();
|
||||
if (ML_function)
|
||||
{
|
||||
if (data.size() == 1)
|
||||
{
|
||||
/// Case for const column. Predict using single model.
|
||||
ML_function->predictValues(data[0], *res, block, 0, block.rows(), arguments, context);
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Case for non-constant column. Use different aggregate function for each row.
|
||||
size_t row_num = 0;
|
||||
for (auto val : data)
|
||||
{
|
||||
ML_function->predictValues(val, *res, block, arguments, context);
|
||||
ML_function->predictValues(val, *res, block, row_num, 1, arguments, context);
|
||||
++row_num;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@ -94,7 +94,7 @@ ColumnPtr ColumnDecimal<T>::permute(const IColumn::Permutation & perm, size_t li
|
||||
for (size_t i = 0; i < size; ++i)
|
||||
res_data[i] = data[perm[i]];
|
||||
|
||||
return std::move(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -117,7 +117,7 @@ MutableColumnPtr ColumnDecimal<T>::cloneResized(size_t size) const
|
||||
}
|
||||
}
|
||||
|
||||
return std::move(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -169,7 +169,7 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter & filt, ssize_t result_
|
||||
++data_pos;
|
||||
}
|
||||
|
||||
return std::move(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -202,7 +202,7 @@ ColumnPtr ColumnDecimal<T>::replicate(const IColumn::Offsets & offsets) const
|
||||
res_data.push_back(data[i]);
|
||||
}
|
||||
|
||||
return std::move(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
|
||||
@ -187,7 +187,7 @@ ColumnPtr ColumnDecimal<T>::indexImpl(const PaddedPODArray<Type> & indexes, size
|
||||
for (size_t i = 0; i < limit; ++i)
|
||||
res_data[i] = data[indexes[i]];
|
||||
|
||||
return std::move(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -349,7 +349,7 @@ ColumnPtr ColumnLowCardinality::countKeys() const
|
||||
|
||||
auto counter = ColumnUInt64::create(dict_size, 0);
|
||||
idx.countKeys(counter->getData());
|
||||
return std::move(counter);
|
||||
return counter;
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -215,6 +215,12 @@ UInt64 ColumnVector<T>::get64(size_t n) const
|
||||
return ext::bit_cast<UInt64>(data[n]);
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
Float64 ColumnVector<T>::getFloat64(size_t n) const
|
||||
{
|
||||
return static_cast<Float64>(data[n]);
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length)
|
||||
{
|
||||
|
||||
@ -202,6 +202,8 @@ public:
|
||||
|
||||
UInt64 get64(size_t n) const override;
|
||||
|
||||
Float64 getFloat64(size_t n) const override;
|
||||
|
||||
UInt64 getUInt(size_t n) const override
|
||||
{
|
||||
return UInt64(data[n]);
|
||||
|
||||
@ -91,6 +91,13 @@ public:
|
||||
throw Exception("Method get64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
/// If column stores native numeric type, it returns n-th element casted to Float64
|
||||
/// Is used in regression methods to cast each features into uniform type
|
||||
virtual Float64 getFloat64(size_t /*n*/) const
|
||||
{
|
||||
throw Exception("Method getFloat64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
/** If column is numeric, return value of n-th element, casted to UInt64.
|
||||
* For NULL values of Nullable column it is allowed to return arbitrary value.
|
||||
* Otherwise throw an exception.
|
||||
|
||||
@ -375,7 +375,7 @@ ColumnUInt64::MutablePtr ReverseIndex<IndexType, ColumnType>::calcHashes() const
|
||||
for (auto row : ext::range(0, size))
|
||||
hash->getElement(row) = getHash(column->getDataAt(row));
|
||||
|
||||
return std::move(hash);
|
||||
return hash;
|
||||
}
|
||||
|
||||
template <typename IndexType, typename ColumnType>
|
||||
|
||||
67
dbms/src/Columns/getLeastSuperColumn.cpp
Normal file
67
dbms/src/Columns/getLeastSuperColumn.cpp
Normal file
@ -0,0 +1,67 @@
|
||||
#include <Columns/getLeastSuperColumn.h>
|
||||
#include <Columns/IColumn.h>
|
||||
#include <Columns/ColumnConst.h>
|
||||
#include <DataTypes/getLeastSupertype.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
static bool sameConstants(const IColumn & a, const IColumn & b)
|
||||
{
|
||||
return static_cast<const ColumnConst &>(a).getField() == static_cast<const ColumnConst &>(b).getField();
|
||||
}
|
||||
|
||||
ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns)
|
||||
{
|
||||
if (columns.empty())
|
||||
throw Exception("Logical error: no src columns for supercolumn", ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
ColumnWithTypeAndName result = *columns[0];
|
||||
|
||||
/// Determine common type.
|
||||
|
||||
size_t num_const = 0;
|
||||
DataTypes types(columns.size());
|
||||
for (size_t i = 0; i < columns.size(); ++i)
|
||||
{
|
||||
types[i] = columns[i]->type;
|
||||
if (columns[i]->column->isColumnConst())
|
||||
++num_const;
|
||||
}
|
||||
|
||||
result.type = getLeastSupertype(types);
|
||||
|
||||
/// Create supertype column saving constness if possible.
|
||||
|
||||
bool save_constness = false;
|
||||
if (columns.size() == num_const)
|
||||
{
|
||||
save_constness = true;
|
||||
for (size_t i = 1; i < columns.size(); ++i)
|
||||
{
|
||||
const ColumnWithTypeAndName & first = *columns[0];
|
||||
const ColumnWithTypeAndName & other = *columns[i];
|
||||
|
||||
if (!sameConstants(*first.column, *other.column))
|
||||
{
|
||||
save_constness = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (save_constness)
|
||||
result.column = result.type->createColumnConst(0, static_cast<const ColumnConst &>(*columns[0]->column).getField());
|
||||
else
|
||||
result.column = result.type->createColumn();
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
}
|
||||
12
dbms/src/Columns/getLeastSuperColumn.h
Normal file
12
dbms/src/Columns/getLeastSuperColumn.h
Normal file
@ -0,0 +1,12 @@
|
||||
#pragma once
|
||||
|
||||
#include <Core/ColumnWithTypeAndName.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// getLeastSupertype + related column changes
|
||||
ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns);
|
||||
|
||||
}
|
||||
@ -427,8 +427,10 @@ namespace ErrorCodes
|
||||
extern const int BAD_TTL_EXPRESSION = 450;
|
||||
extern const int BAD_TTL_FILE = 451;
|
||||
extern const int SETTING_CONSTRAINT_VIOLATION = 452;
|
||||
extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 453;
|
||||
|
||||
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 453;
|
||||
extern const int OPENSSL_ERROR = 454;
|
||||
extern const int SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = 455;
|
||||
extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 456;
|
||||
extern const int KEEPER_EXCEPTION = 999;
|
||||
extern const int POCO_EXCEPTION = 1000;
|
||||
extern const int STD_EXCEPTION = 1001;
|
||||
@ -437,8 +439,6 @@ namespace ErrorCodes
|
||||
|
||||
extern const int CONDITIONAL_TREE_PARENT_NOT_FOUND = 2001;
|
||||
extern const int ILLEGAL_PROJECTION_MANIPULATOR = 2002;
|
||||
|
||||
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 446;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
18
dbms/src/Common/OpenSSLHelpers.cpp
Normal file
18
dbms/src/Common/OpenSSLHelpers.cpp
Normal file
@ -0,0 +1,18 @@
|
||||
#include "OpenSSLHelpers.h"
|
||||
#include <ext/scope_guard.h>
|
||||
#include <openssl/err.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
String getOpenSSLErrors()
|
||||
{
|
||||
BIO * mem = BIO_new(BIO_s_mem());
|
||||
SCOPE_EXIT(BIO_free(mem));
|
||||
ERR_print_errors(mem);
|
||||
char * buf = nullptr;
|
||||
long size = BIO_get_mem_data(mem, &buf);
|
||||
return String(buf, size);
|
||||
}
|
||||
|
||||
}
|
||||
12
dbms/src/Common/OpenSSLHelpers.h
Normal file
12
dbms/src/Common/OpenSSLHelpers.h
Normal file
@ -0,0 +1,12 @@
|
||||
#pragma once
|
||||
|
||||
#include <Core/Types.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// Returns concatenation of error strings for all errors that OpenSSL has recorded, emptying the error queue.
|
||||
String getOpenSSLErrors();
|
||||
|
||||
}
|
||||
@ -6,7 +6,7 @@
|
||||
#include <Common/config.h>
|
||||
#include <re2/re2.h>
|
||||
#if USE_RE2_ST
|
||||
#include <re2_st/re2.h> // Y_IGNORE
|
||||
#include <re2_st/re2.h>
|
||||
#else
|
||||
#define re2_st re2
|
||||
#endif
|
||||
|
||||
@ -71,7 +71,7 @@ template <typename T> bool inline operator> (T a, const UInt128 b) { return UIn
|
||||
template <typename T> bool inline operator<= (T a, const UInt128 b) { return UInt128(a) <= b; }
|
||||
template <typename T> bool inline operator< (T a, const UInt128 b) { return UInt128(a) < b; }
|
||||
|
||||
template <> constexpr bool IsNumber<UInt128> = true;
|
||||
template <> inline constexpr bool IsNumber<UInt128> = true;
|
||||
template <> struct TypeName<UInt128> { static const char * get() { return "UInt128"; } };
|
||||
template <> struct TypeId<UInt128> { static constexpr const TypeIndex value = TypeIndex::UInt128; };
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@
|
||||
# include <Common/Exception.h>
|
||||
namespace DB { namespace ErrorCodes { extern const int CPUID_ERROR; }}
|
||||
#elif USE_CPUINFO
|
||||
# include <cpuinfo.h> // Y_IGNORE
|
||||
# include <cpuinfo.h>
|
||||
#endif
|
||||
|
||||
|
||||
|
||||
@ -1,7 +1,9 @@
|
||||
#include <Common/localBackup.h>
|
||||
#include "localBackup.h"
|
||||
|
||||
#include <Common/createHardLink.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Poco/DirectoryIterator.h>
|
||||
#include <Poco/Path.h>
|
||||
#include <Poco/File.h>
|
||||
#include <string>
|
||||
#include <iostream>
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
#pragma once
|
||||
|
||||
#include <Poco/Path.h>
|
||||
#include <optional>
|
||||
|
||||
namespace Poco { class Path; }
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
@ -1,18 +1,16 @@
|
||||
#pragma once
|
||||
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
#include <Core/Types.h>
|
||||
#include <IO/copyData.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/ReadBufferFromPocoSocket.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
#include <IO/WriteBufferFromPocoSocket.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <Core/Types.h>
|
||||
#include <Poco/RandomStream.h>
|
||||
#include <Poco/Net/StreamSocket.h>
|
||||
#include <Poco/RandomStream.h>
|
||||
#include <random>
|
||||
#include <sstream>
|
||||
#include <common/logger_useful.h>
|
||||
#include <Poco/Logger.h>
|
||||
|
||||
/// Implementation of MySQL wire protocol
|
||||
|
||||
@ -157,23 +155,19 @@ public:
|
||||
size_t max_packet_size = MAX_PACKET_LENGTH;
|
||||
|
||||
/// For reading and writing.
|
||||
PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id, const String logger_name)
|
||||
PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id)
|
||||
: sequence_id(sequence_id)
|
||||
, in(&in)
|
||||
, out(&out)
|
||||
, log(&Poco::Logger::get(logger_name))
|
||||
{
|
||||
log->setLevel("information");
|
||||
}
|
||||
|
||||
/// For writing.
|
||||
PacketSender(WriteBuffer & out, size_t & sequence_id, const String logger_name)
|
||||
PacketSender(WriteBuffer & out, size_t & sequence_id)
|
||||
: sequence_id(sequence_id)
|
||||
, in(nullptr)
|
||||
, out(&out)
|
||||
, log(&Poco::Logger::get(logger_name))
|
||||
{
|
||||
log->setLevel("information");
|
||||
}
|
||||
|
||||
String receivePacketPayload()
|
||||
@ -186,8 +180,6 @@ public:
|
||||
// packets which are larger than or equal to 16MB are splitted
|
||||
do
|
||||
{
|
||||
LOG_TRACE(log, "Reading from buffer");
|
||||
|
||||
in->readStrict(reinterpret_cast<char *>(&payload_length), 3);
|
||||
|
||||
if (payload_length > max_packet_size)
|
||||
@ -207,8 +199,6 @@ public:
|
||||
}
|
||||
sequence_id++;
|
||||
|
||||
LOG_TRACE(log, "Received packet. Sequence-id: " << packet_sequence_id << ", payload length: " << payload_length);
|
||||
|
||||
copyData(*in, static_cast<WriteBuffer &>(buf), payload_length);
|
||||
} while (payload_length == max_packet_size);
|
||||
|
||||
@ -230,9 +220,6 @@ public:
|
||||
{
|
||||
size_t payload_length = std::min(payload.length() - pos, max_packet_size);
|
||||
|
||||
LOG_TRACE(log, "Writing packet of size " << payload_length << " with sequence-id " << static_cast<int>(sequence_id));
|
||||
LOG_TRACE(log, packetToText(payload));
|
||||
|
||||
out->write(reinterpret_cast<const char *>(&payload_length), 3);
|
||||
out->write(reinterpret_cast<const char *>(&sequence_id), 1);
|
||||
out->write(payload.data() + pos, payload_length);
|
||||
@ -241,13 +228,9 @@ public:
|
||||
sequence_id++;
|
||||
} while (pos < payload.length());
|
||||
|
||||
LOG_TRACE(log, "Packet was sent.");
|
||||
|
||||
if (flush)
|
||||
{
|
||||
out->next();
|
||||
}
|
||||
}
|
||||
|
||||
/// Sets sequence-id to 0. Must be called before each command phase.
|
||||
void resetSequenceId();
|
||||
@ -255,8 +238,6 @@ public:
|
||||
private:
|
||||
/// Converts packet to text. Is used for debug output.
|
||||
static String packetToText(String payload);
|
||||
|
||||
Poco::Logger * log;
|
||||
};
|
||||
|
||||
|
||||
@ -279,12 +260,11 @@ class Handshake : public WritePacket
|
||||
uint32_t status_flags;
|
||||
String auth_plugin_data;
|
||||
public:
|
||||
explicit Handshake(uint32_t connection_id, String server_version, String auth_plugin_data)
|
||||
explicit Handshake(uint32_t capability_flags, uint32_t connection_id, String server_version, String auth_plugin_data)
|
||||
: protocol_version(0xa)
|
||||
, server_version(std::move(server_version))
|
||||
, connection_id(connection_id)
|
||||
, capability_flags(CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA
|
||||
| CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF | CLIENT_SSL)
|
||||
, capability_flags(capability_flags)
|
||||
, character_set(CharacterSet::utf8_general_ci)
|
||||
, status_flags(0)
|
||||
, auth_plugin_data(auth_plugin_data)
|
||||
|
||||
@ -85,6 +85,7 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingFloat, totals_auto_threshold, 0.5, "The threshold for totals_mode = 'auto'.") \
|
||||
\
|
||||
M(SettingBool, compile, false, "Whether query compilation is enabled.") \
|
||||
M(SettingBool, allow_suspicious_low_cardinality_types, false, "In CREATE TABLE statement allows specifying LowCardinality modifier for types of small fixed size (8 or less). Enabling this may increase merge times and memory consumption.") \
|
||||
M(SettingBool, compile_expressions, false, "Compile some scalar functions and operators to native code.") \
|
||||
M(SettingUInt64, min_count_to_compile, 3, "The number of structurally identical queries before they are compiled.") \
|
||||
M(SettingUInt64, min_count_to_compile_expression, 3, "The number of identical expressions before they are JIT-compiled") \
|
||||
|
||||
@ -32,16 +32,16 @@ using String = std::string;
|
||||
*/
|
||||
template <typename T> constexpr bool IsNumber = false;
|
||||
|
||||
template <> constexpr bool IsNumber<UInt8> = true;
|
||||
template <> constexpr bool IsNumber<UInt16> = true;
|
||||
template <> constexpr bool IsNumber<UInt32> = true;
|
||||
template <> constexpr bool IsNumber<UInt64> = true;
|
||||
template <> constexpr bool IsNumber<Int8> = true;
|
||||
template <> constexpr bool IsNumber<Int16> = true;
|
||||
template <> constexpr bool IsNumber<Int32> = true;
|
||||
template <> constexpr bool IsNumber<Int64> = true;
|
||||
template <> constexpr bool IsNumber<Float32> = true;
|
||||
template <> constexpr bool IsNumber<Float64> = true;
|
||||
template <> inline constexpr bool IsNumber<UInt8> = true;
|
||||
template <> inline constexpr bool IsNumber<UInt16> = true;
|
||||
template <> inline constexpr bool IsNumber<UInt32> = true;
|
||||
template <> inline constexpr bool IsNumber<UInt64> = true;
|
||||
template <> inline constexpr bool IsNumber<Int8> = true;
|
||||
template <> inline constexpr bool IsNumber<Int16> = true;
|
||||
template <> inline constexpr bool IsNumber<Int32> = true;
|
||||
template <> inline constexpr bool IsNumber<Int64> = true;
|
||||
template <> inline constexpr bool IsNumber<Float32> = true;
|
||||
template <> inline constexpr bool IsNumber<Float64> = true;
|
||||
|
||||
template <typename T> struct TypeName;
|
||||
|
||||
@ -109,7 +109,7 @@ using Strings = std::vector<String>;
|
||||
|
||||
|
||||
using Int128 = __int128;
|
||||
template <> constexpr bool IsNumber<Int128> = true;
|
||||
template <> inline constexpr bool IsNumber<Int128> = true;
|
||||
template <> struct TypeName<Int128> { static const char * get() { return "Int128"; } };
|
||||
template <> struct TypeId<Int128> { static constexpr const TypeIndex value = TypeIndex::Int128; };
|
||||
|
||||
@ -159,17 +159,56 @@ template <> struct TypeId<Decimal32> { static constexpr const TypeIndex value
|
||||
template <> struct TypeId<Decimal64> { static constexpr const TypeIndex value = TypeIndex::Decimal64; };
|
||||
template <> struct TypeId<Decimal128> { static constexpr const TypeIndex value = TypeIndex::Decimal128; };
|
||||
|
||||
template <typename T>
|
||||
constexpr bool IsDecimalNumber = false;
|
||||
template <> constexpr bool IsDecimalNumber<Decimal32> = true;
|
||||
template <> constexpr bool IsDecimalNumber<Decimal64> = true;
|
||||
template <> constexpr bool IsDecimalNumber<Decimal128> = true;
|
||||
template <typename T> constexpr bool IsDecimalNumber = false;
|
||||
template <> inline constexpr bool IsDecimalNumber<Decimal32> = true;
|
||||
template <> inline constexpr bool IsDecimalNumber<Decimal64> = true;
|
||||
template <> inline constexpr bool IsDecimalNumber<Decimal128> = true;
|
||||
|
||||
template <typename T> struct NativeType { using Type = T; };
|
||||
template <> struct NativeType<Decimal32> { using Type = Int32; };
|
||||
template <> struct NativeType<Decimal64> { using Type = Int64; };
|
||||
template <> struct NativeType<Decimal128> { using Type = Int128; };
|
||||
|
||||
inline const char * getTypeName(TypeIndex idx)
|
||||
{
|
||||
switch (idx)
|
||||
{
|
||||
case TypeIndex::Nothing: return "Nothing";
|
||||
case TypeIndex::UInt8: return TypeName<UInt8>::get();
|
||||
case TypeIndex::UInt16: return TypeName<UInt16>::get();
|
||||
case TypeIndex::UInt32: return TypeName<UInt32>::get();
|
||||
case TypeIndex::UInt64: return TypeName<UInt64>::get();
|
||||
case TypeIndex::UInt128: return "UInt128";
|
||||
case TypeIndex::Int8: return TypeName<Int8>::get();
|
||||
case TypeIndex::Int16: return TypeName<Int16>::get();
|
||||
case TypeIndex::Int32: return TypeName<Int32>::get();
|
||||
case TypeIndex::Int64: return TypeName<Int64>::get();
|
||||
case TypeIndex::Int128: return TypeName<Int128>::get();
|
||||
case TypeIndex::Float32: return TypeName<Float32>::get();
|
||||
case TypeIndex::Float64: return TypeName<Float64>::get();
|
||||
case TypeIndex::Date: return "Date";
|
||||
case TypeIndex::DateTime: return "DateTime";
|
||||
case TypeIndex::String: return TypeName<String>::get();
|
||||
case TypeIndex::FixedString: return "FixedString";
|
||||
case TypeIndex::Enum8: return "Enum8";
|
||||
case TypeIndex::Enum16: return "Enum16";
|
||||
case TypeIndex::Decimal32: return TypeName<Decimal32>::get();
|
||||
case TypeIndex::Decimal64: return TypeName<Decimal64>::get();
|
||||
case TypeIndex::Decimal128: return TypeName<Decimal128>::get();
|
||||
case TypeIndex::UUID: return "UUID";
|
||||
case TypeIndex::Array: return "Array";
|
||||
case TypeIndex::Tuple: return "Tuple";
|
||||
case TypeIndex::Set: return "Set";
|
||||
case TypeIndex::Interval: return "Interval";
|
||||
case TypeIndex::Nullable: return "Nullable";
|
||||
case TypeIndex::Function: return "Function";
|
||||
case TypeIndex::AggregateFunction: return "AggregateFunction";
|
||||
case TypeIndex::LowCardinality: return "LowCardinality";
|
||||
}
|
||||
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/// Specialization of `std::hash` for the Decimal<T> types.
|
||||
|
||||
@ -36,6 +36,7 @@ public:
|
||||
bool canBeInsideNullable() const override { return false; }
|
||||
|
||||
DataTypePtr getReturnType() const { return function->getReturnType(); }
|
||||
DataTypePtr getReturnTypeToPredict() const { return function->getReturnTypeToPredict(); }
|
||||
DataTypes getArgumentsDataTypes() const { return argument_types; }
|
||||
|
||||
/// NOTE These two functions for serializing single values are incompatible with the functions below.
|
||||
|
||||
@ -111,7 +111,7 @@ ColumnPtr recursiveLowCardinalityConversion(const ColumnPtr & column, const Data
|
||||
{
|
||||
auto col = low_cardinality_type->createColumn();
|
||||
static_cast<ColumnLowCardinality &>(*col).insertRangeFromFullColumn(*column, 0, column->size());
|
||||
return std::move(col);
|
||||
return col;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -246,9 +246,9 @@ inline UInt32 getDecimalScale(const IDataType & data_type, UInt32 default_value
|
||||
///
|
||||
|
||||
template <typename DataType> constexpr bool IsDataTypeDecimal = false;
|
||||
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
|
||||
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
|
||||
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
|
||||
template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
|
||||
template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
|
||||
template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
|
||||
|
||||
template <typename DataType> constexpr bool IsDataTypeDecimalOrNumber = IsDataTypeDecimal<DataType> || IsDataTypeNumber<DataType>;
|
||||
|
||||
|
||||
@ -38,15 +38,15 @@ using DataTypeFloat32 = DataTypeNumber<Float32>;
|
||||
using DataTypeFloat64 = DataTypeNumber<Float64>;
|
||||
|
||||
template <typename DataType> constexpr bool IsDataTypeNumber = false;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
|
||||
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
|
||||
template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
|
||||
|
||||
}
|
||||
|
||||
@ -16,7 +16,7 @@
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
|
||||
#pragma GCC diagnostic pop
|
||||
|
||||
|
||||
@ -86,7 +86,7 @@ void CacheDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<K
|
||||
{
|
||||
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
|
||||
|
||||
getItemsNumber<UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; });
|
||||
getItemsNumberImpl<UInt64, UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; });
|
||||
}
|
||||
|
||||
|
||||
@ -207,9 +207,7 @@ void CacheDictionary::isInConstantVector(const Key child_id, const PaddedPODArra
|
||||
void CacheDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto null_value = StringRef{std::get<String>(attribute.null_values)};
|
||||
|
||||
@ -220,9 +218,7 @@ void CacheDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsString(attribute, ids, out, [&](const size_t row) { return def->getDataAt(row); });
|
||||
}
|
||||
@ -231,9 +227,7 @@ void CacheDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsString(attribute, ids, out, [&](const size_t) { return StringRef{def}; });
|
||||
}
|
||||
|
||||
@ -221,11 +221,6 @@ private:
|
||||
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
|
||||
template <typename OutputType, typename DefaultGetter>
|
||||
void getItemsNumber(
|
||||
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename DefaultGetter>
|
||||
void getItemsNumberImpl(
|
||||
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;
|
||||
|
||||
@ -34,34 +34,6 @@ namespace ErrorCodes
|
||||
extern const int TYPE_MISMATCH;
|
||||
}
|
||||
|
||||
template <typename OutputType, typename DefaultGetter>
|
||||
void CacheDictionary::getItemsNumber(
|
||||
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) \
|
||||
getItemsNumberImpl<TYPE, OutputType>(attribute, ids, out, std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename DefaultGetter>
|
||||
void CacheDictionary::getItemsNumberImpl(
|
||||
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const
|
||||
|
||||
@ -12,13 +12,11 @@ using TYPE = @NAME@;
|
||||
void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return null_value; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return null_value; });
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -15,11 +15,9 @@ void CacheDictionary::get@NAME@(const std::string & attribute_name,
|
||||
ResultArrayType<TYPE> & out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; });
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -12,11 +12,9 @@ using TYPE = @NAME@;
|
||||
void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const
|
||||
{
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return def; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return def; });
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -77,9 +77,7 @@ void ComplexKeyCacheDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto null_value = StringRef{std::get<String>(attribute.null_values)};
|
||||
|
||||
@ -96,9 +94,7 @@ void ComplexKeyCacheDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsString(attribute, key_columns, out, [&](const size_t row) { return def->getDataAt(row); });
|
||||
}
|
||||
@ -113,9 +109,7 @@ void ComplexKeyCacheDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsString(attribute, key_columns, out, [&](const size_t) { return StringRef{def}; });
|
||||
}
|
||||
|
||||
@ -256,34 +256,6 @@ private:
|
||||
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
template <typename OutputType, typename DefaultGetter>
|
||||
void
|
||||
getItemsNumber(Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) \
|
||||
getItemsNumberImpl<TYPE, OutputType>(attribute, key_columns, out, std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename DefaultGetter>
|
||||
void getItemsNumberImpl(
|
||||
Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const
|
||||
|
||||
@ -13,12 +13,10 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name, co
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; });
|
||||
}
|
||||
}
|
||||
|
||||
@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; });
|
||||
}
|
||||
}
|
||||
|
||||
@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@))
|
||||
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
|
||||
|
||||
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return def; });
|
||||
getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return def; });
|
||||
}
|
||||
}
|
||||
|
||||
@ -50,13 +50,11 @@ ComplexKeyHashedDictionary::ComplexKeyHashedDictionary(
|
||||
dict_struct.validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, \
|
||||
key_columns, \
|
||||
[&](const size_t row, const auto value) { out[row] = value; }, \
|
||||
@ -84,9 +82,7 @@ void ComplexKeyHashedDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
|
||||
|
||||
@ -108,11 +104,9 @@ void ComplexKeyHashedDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, \
|
||||
key_columns, \
|
||||
[&](const size_t row, const auto value) { out[row] = value; }, \
|
||||
@ -144,9 +138,7 @@ void ComplexKeyHashedDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -166,11 +158,9 @@ void ComplexKeyHashedDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -199,9 +189,7 @@ void ComplexKeyHashedDictionary::getString(
|
||||
dict_struct.validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -566,34 +554,6 @@ ComplexKeyHashedDictionary::createAttributeWithType(const AttributeUnderlyingTyp
|
||||
}
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void ComplexKeyHashedDictionary::getItemsNumber(
|
||||
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
|
||||
attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void ComplexKeyHashedDictionary::getItemsImpl(
|
||||
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
|
||||
@ -218,16 +218,10 @@ private:
|
||||
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void
|
||||
getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void
|
||||
getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
|
||||
template <typename T>
|
||||
bool setAttributeValueImpl(Attribute & attribute, const StringRef key, const T value);
|
||||
|
||||
|
||||
@ -40,44 +40,6 @@ namespace
|
||||
} // namespace
|
||||
|
||||
|
||||
bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to)
|
||||
{
|
||||
if (from == to)
|
||||
return true;
|
||||
|
||||
/** This enum can be somewhat incomplete and the meaning may not coincide with NumberTraits.h.
|
||||
* (for example, because integers can not be converted to floats)
|
||||
* This is normal for a limited usage scope.
|
||||
*/
|
||||
if ((from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt16)
|
||||
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt32)
|
||||
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt64)
|
||||
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt32)
|
||||
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt64)
|
||||
|| (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::UInt64)
|
||||
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int16)
|
||||
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int32)
|
||||
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int64)
|
||||
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int32)
|
||||
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int64)
|
||||
|| (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::Int64)
|
||||
|
||||
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int16)
|
||||
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int32)
|
||||
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int64)
|
||||
|| (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int32)
|
||||
|| (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int64)
|
||||
|| (from == AttributeUnderlyingType::Int32 && to == AttributeUnderlyingType::Int64)
|
||||
|
||||
|| (from == AttributeUnderlyingType::Float32 && to == AttributeUnderlyingType::Float64))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type)
|
||||
{
|
||||
static const std::unordered_map<std::string, AttributeUnderlyingType> dictionary{
|
||||
|
||||
@ -13,6 +13,12 @@
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int TYPE_MISMATCH;
|
||||
}
|
||||
|
||||
enum class AttributeUnderlyingType
|
||||
{
|
||||
UInt8,
|
||||
@ -33,14 +39,18 @@ enum class AttributeUnderlyingType
|
||||
};
|
||||
|
||||
|
||||
/** For implicit conversions in dictGet functions.
|
||||
*/
|
||||
bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to);
|
||||
|
||||
AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type);
|
||||
|
||||
std::string toString(const AttributeUnderlyingType type);
|
||||
|
||||
/// Implicit conversions in dictGet functions is disabled.
|
||||
inline void checkAttributeType(const std::string & dict_name, const std::string & attribute_name,
|
||||
AttributeUnderlyingType attribute_type, AttributeUnderlyingType to)
|
||||
{
|
||||
if (attribute_type != to)
|
||||
throw Exception{dict_name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute_type)
|
||||
+ ", expected " + toString(to), ErrorCodes::TYPE_MISMATCH};
|
||||
}
|
||||
|
||||
/// Min and max lifetimes for a dictionary or it's entry
|
||||
using DictionaryLifetime = ExternalLoadableLifetime;
|
||||
|
||||
@ -55,7 +55,7 @@ void FlatDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<Ke
|
||||
{
|
||||
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
|
||||
|
||||
getItemsNumber<UInt64>(
|
||||
getItemsImpl<UInt64, UInt64>(
|
||||
*hierarchical_attribute,
|
||||
ids,
|
||||
[&](const size_t row, const UInt64 value) { out[row] = value; },
|
||||
@ -117,13 +117,11 @@ void FlatDictionary::isInConstantVector(const Key child_id, const PaddedPODArray
|
||||
void FlatDictionary::get##TYPE(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -145,9 +143,7 @@ DECLARE(Decimal128)
|
||||
void FlatDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto & null_value = std::get<StringRef>(attribute.null_values);
|
||||
|
||||
@ -166,11 +162,9 @@ void FlatDictionary::getString(const std::string & attribute_name, const PaddedP
|
||||
ResultArrayType<TYPE> & out) const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -193,9 +187,7 @@ void FlatDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -209,11 +201,9 @@ void FlatDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -236,9 +226,7 @@ void FlatDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
FlatDictionary::getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -580,35 +568,6 @@ FlatDictionary::Attribute FlatDictionary::createAttributeWithType(const Attribut
|
||||
}
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void FlatDictionary::getItemsNumber(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) \
|
||||
getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void FlatDictionary::getItemsImpl(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
@ -701,10 +660,10 @@ void FlatDictionary::setAttributeValue(Attribute & attribute, const Key id, cons
|
||||
break;
|
||||
|
||||
case AttributeUnderlyingType::Decimal32:
|
||||
setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal128>());
|
||||
setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
|
||||
break;
|
||||
case AttributeUnderlyingType::Decimal64:
|
||||
setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal128>());
|
||||
setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
|
||||
break;
|
||||
case AttributeUnderlyingType::Decimal128:
|
||||
setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
|
||||
|
||||
@ -206,10 +206,6 @@ private:
|
||||
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void getItemsNumber(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void getItemsImpl(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
@ -49,7 +49,7 @@ void HashedDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<
|
||||
{
|
||||
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
|
||||
|
||||
getItemsNumber<UInt64>(
|
||||
getItemsImpl<UInt64, UInt64>(
|
||||
*hierarchical_attribute,
|
||||
ids,
|
||||
[&](const size_t row, const UInt64 value) { out[row] = value; },
|
||||
@ -116,13 +116,11 @@ void HashedDictionary::isInConstantVector(const Key child_id, const PaddedPODArr
|
||||
const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -144,9 +142,7 @@ DECLARE(Decimal128)
|
||||
void HashedDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
|
||||
|
||||
@ -165,11 +161,9 @@ void HashedDictionary::getString(const std::string & attribute_name, const Padde
|
||||
ResultArrayType<TYPE> & out) const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -192,9 +186,7 @@ void HashedDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -208,11 +200,9 @@ void HashedDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE & def, ResultArrayType<TYPE> & out) const \
|
||||
{ \
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -235,9 +225,7 @@ void HashedDictionary::getString(
|
||||
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
|
||||
{
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -324,7 +312,6 @@ void HashedDictionary::createAttributes()
|
||||
void HashedDictionary::blockToAttributes(const Block & block)
|
||||
{
|
||||
const auto & id_column = *block.safeGetByPosition(0).column;
|
||||
element_count += id_column.size();
|
||||
|
||||
for (const size_t attribute_idx : ext::range(0, attributes.size()))
|
||||
{
|
||||
@ -332,7 +319,8 @@ void HashedDictionary::blockToAttributes(const Block & block)
|
||||
auto & attribute = attributes[attribute_idx];
|
||||
|
||||
for (const auto row_idx : ext::range(0, id_column.size()))
|
||||
setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]);
|
||||
if (setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]))
|
||||
++element_count;
|
||||
}
|
||||
}
|
||||
|
||||
@ -567,34 +555,6 @@ HashedDictionary::Attribute HashedDictionary::createAttributeWithType(const Attr
|
||||
}
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void HashedDictionary::getItemsNumber(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) \
|
||||
getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void HashedDictionary::getItemsImpl(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
@ -613,69 +573,56 @@ void HashedDictionary::getItemsImpl(
|
||||
|
||||
|
||||
template <typename T>
|
||||
void HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value)
|
||||
bool HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value)
|
||||
{
|
||||
auto & map = *std::get<CollectionPtrType<T>>(attribute.maps);
|
||||
map.insert({id, value});
|
||||
return map.insert({id, value}).second;
|
||||
}
|
||||
|
||||
void HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value)
|
||||
bool HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value)
|
||||
{
|
||||
switch (attribute.type)
|
||||
{
|
||||
case AttributeUnderlyingType::UInt8:
|
||||
setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>());
|
||||
break;
|
||||
return setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>());
|
||||
case AttributeUnderlyingType::UInt16:
|
||||
setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>());
|
||||
break;
|
||||
return setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>());
|
||||
case AttributeUnderlyingType::UInt32:
|
||||
setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>());
|
||||
break;
|
||||
return setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>());
|
||||
case AttributeUnderlyingType::UInt64:
|
||||
setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>());
|
||||
break;
|
||||
return setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>());
|
||||
case AttributeUnderlyingType::UInt128:
|
||||
setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>());
|
||||
break;
|
||||
return setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>());
|
||||
case AttributeUnderlyingType::Int8:
|
||||
setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>());
|
||||
case AttributeUnderlyingType::Int16:
|
||||
setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>());
|
||||
case AttributeUnderlyingType::Int32:
|
||||
setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>());
|
||||
case AttributeUnderlyingType::Int64:
|
||||
setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>());
|
||||
case AttributeUnderlyingType::Float32:
|
||||
setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>());
|
||||
case AttributeUnderlyingType::Float64:
|
||||
setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>());
|
||||
|
||||
case AttributeUnderlyingType::Decimal32:
|
||||
setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
|
||||
break;
|
||||
return setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
|
||||
case AttributeUnderlyingType::Decimal64:
|
||||
setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
|
||||
break;
|
||||
return setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
|
||||
case AttributeUnderlyingType::Decimal128:
|
||||
setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
|
||||
break;
|
||||
return setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
|
||||
|
||||
case AttributeUnderlyingType::String:
|
||||
{
|
||||
auto & map = *std::get<CollectionPtrType<StringRef>>(attribute.maps);
|
||||
const auto & string = value.get<String>();
|
||||
const auto string_in_arena = attribute.string_arena->insert(string.data(), string.size());
|
||||
map.insert({id, StringRef{string_in_arena, string.size()}});
|
||||
break;
|
||||
return map.insert({id, StringRef{string_in_arena, string.size()}}).second;
|
||||
}
|
||||
}
|
||||
|
||||
throw Exception{"Invalid attribute type", ErrorCodes::BAD_ARGUMENTS};
|
||||
}
|
||||
|
||||
const HashedDictionary::Attribute & HashedDictionary::getAttribute(const std::string & attribute_name) const
|
||||
|
||||
@ -211,18 +211,14 @@ private:
|
||||
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void getItemsNumber(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void getItemsImpl(
|
||||
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename T>
|
||||
void setAttributeValueImpl(Attribute & attribute, const Key id, const T value);
|
||||
bool setAttributeValueImpl(Attribute & attribute, const Key id, const T value);
|
||||
|
||||
void setAttributeValue(Attribute & attribute, const Key id, const Field & value);
|
||||
bool setAttributeValue(Attribute & attribute, const Key id, const Field & value);
|
||||
|
||||
const Attribute & getAttribute(const std::string & attribute_name) const;
|
||||
|
||||
|
||||
@ -75,13 +75,11 @@ TrieDictionary::~TrieDictionary()
|
||||
validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
const auto null_value = std::get<TYPE>(attribute.null_values); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, \
|
||||
key_columns, \
|
||||
[&](const size_t row, const auto value) { out[row] = value; }, \
|
||||
@ -109,9 +107,7 @@ void TrieDictionary::getString(
|
||||
validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
|
||||
|
||||
@ -133,11 +129,9 @@ void TrieDictionary::getString(
|
||||
validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, \
|
||||
key_columns, \
|
||||
[&](const size_t row, const auto value) { out[row] = value; }, \
|
||||
@ -169,9 +163,7 @@ void TrieDictionary::getString(
|
||||
validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -191,11 +183,9 @@ void TrieDictionary::getString(
|
||||
validateKeyTypes(key_types); \
|
||||
\
|
||||
const auto & attribute = getAttribute(attribute_name); \
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
|
||||
ErrorCodes::TYPE_MISMATCH}; \
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
|
||||
\
|
||||
getItemsNumber<TYPE>( \
|
||||
getItemsImpl<TYPE, TYPE>( \
|
||||
attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
|
||||
}
|
||||
DECLARE(UInt8)
|
||||
@ -224,9 +214,7 @@ void TrieDictionary::getString(
|
||||
validateKeyTypes(key_types);
|
||||
|
||||
const auto & attribute = getAttribute(attribute_name);
|
||||
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String))
|
||||
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
|
||||
ErrorCodes::TYPE_MISMATCH};
|
||||
checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
|
||||
|
||||
getItemsImpl<StringRef, StringRef>(
|
||||
attribute,
|
||||
@ -507,34 +495,6 @@ TrieDictionary::Attribute TrieDictionary::createAttributeWithType(const Attribut
|
||||
}
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void TrieDictionary::getItemsNumber(
|
||||
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
{
|
||||
if (false)
|
||||
{
|
||||
}
|
||||
#define DISPATCH(TYPE) \
|
||||
else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
|
||||
attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
|
||||
DISPATCH(UInt8)
|
||||
DISPATCH(UInt16)
|
||||
DISPATCH(UInt32)
|
||||
DISPATCH(UInt64)
|
||||
DISPATCH(UInt128)
|
||||
DISPATCH(Int8)
|
||||
DISPATCH(Int16)
|
||||
DISPATCH(Int32)
|
||||
DISPATCH(Int64)
|
||||
DISPATCH(Float32)
|
||||
DISPATCH(Float64)
|
||||
DISPATCH(Decimal32)
|
||||
DISPATCH(Decimal64)
|
||||
DISPATCH(Decimal128)
|
||||
#undef DISPATCH
|
||||
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void TrieDictionary::getItemsImpl(
|
||||
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
|
||||
|
||||
@ -218,10 +218,6 @@ private:
|
||||
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
|
||||
|
||||
|
||||
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void
|
||||
getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
|
||||
void
|
||||
getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
|
||||
|
||||
@ -3,13 +3,13 @@
|
||||
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Formats/CapnProtoRowInputStream.h> // Y_IGNORE
|
||||
#include <Formats/CapnProtoRowInputStream.h>
|
||||
#include <Formats/FormatFactory.h>
|
||||
#include <Formats/BlockInputStreamFromRowInputStream.h>
|
||||
#include <Formats/FormatSchemaInfo.h>
|
||||
#include <capnp/serialize.h> // Y_IGNORE
|
||||
#include <capnp/dynamic.h> // Y_IGNORE
|
||||
#include <capnp/common.h> // Y_IGNORE
|
||||
#include <capnp/serialize.h>
|
||||
#include <capnp/dynamic.h>
|
||||
#include <capnp/common.h>
|
||||
#include <boost/algorithm/string.hpp>
|
||||
#include <boost/range/join.hpp>
|
||||
#include <common/logger_useful.h>
|
||||
|
||||
@ -130,7 +130,7 @@ void registerOutputFormatXML(FormatFactory & factory);
|
||||
void registerOutputFormatODBCDriver(FormatFactory & factory);
|
||||
void registerOutputFormatODBCDriver2(FormatFactory & factory);
|
||||
void registerOutputFormatNull(FormatFactory & factory);
|
||||
void registerOutputFormatMySQL(FormatFactory & factory);
|
||||
void registerOutputFormatMySQLWire(FormatFactory & factory);
|
||||
|
||||
/// Input only formats.
|
||||
|
||||
@ -169,7 +169,7 @@ FormatFactory::FormatFactory()
|
||||
registerOutputFormatODBCDriver(*this);
|
||||
registerOutputFormatODBCDriver2(*this);
|
||||
registerOutputFormatNull(*this);
|
||||
registerOutputFormatMySQL(*this);
|
||||
registerOutputFormatMySQLWire(*this);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -1,19 +0,0 @@
|
||||
#include <DataStreams/MySQLBlockOutputStream.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
void registerOutputFormatMySQL(FormatFactory & factory)
|
||||
{
|
||||
factory.registerOutputFormat("MySQL", [](
|
||||
WriteBuffer & buf,
|
||||
const Block & sample,
|
||||
const Context & context,
|
||||
const FormatSettings &)
|
||||
{
|
||||
return std::make_shared<MySQLBlockOutputStream>(buf, sample, const_cast<Context &>(context));
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,4 +1,4 @@
|
||||
#include "MySQLBlockOutputStream.h"
|
||||
#include "MySQLWireBlockOutputStream.h"
|
||||
#include <Core/MySQLProtocol.h>
|
||||
#include <Interpreters/ProcessList.h>
|
||||
#include <iomanip>
|
||||
@ -9,15 +9,15 @@ namespace DB
|
||||
|
||||
using namespace MySQLProtocol;
|
||||
|
||||
MySQLBlockOutputStream::MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context)
|
||||
MySQLWireBlockOutputStream::MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context)
|
||||
: header(header)
|
||||
, context(context)
|
||||
, packet_sender(new PacketSender(buf, context.sequence_id, "MySQLBlockOutputStream"))
|
||||
, packet_sender(std::make_shared<PacketSender>(buf, context.sequence_id))
|
||||
{
|
||||
packet_sender->max_packet_size = context.max_packet_size;
|
||||
}
|
||||
|
||||
void MySQLBlockOutputStream::writePrefix()
|
||||
void MySQLWireBlockOutputStream::writePrefix()
|
||||
{
|
||||
if (header.columns() == 0)
|
||||
return;
|
||||
@ -26,8 +26,7 @@ void MySQLBlockOutputStream::writePrefix()
|
||||
|
||||
for (const ColumnWithTypeAndName & column : header.getColumnsWithTypeAndName())
|
||||
{
|
||||
ColumnDefinition column_definition(column.name, CharacterSet::binary, std::numeric_limits<uint32_t>::max(),
|
||||
ColumnType::MYSQL_TYPE_STRING, 0, 0);
|
||||
ColumnDefinition column_definition(column.name, CharacterSet::binary, 0, ColumnType::MYSQL_TYPE_STRING, 0, 0);
|
||||
packet_sender->sendPacket(column_definition);
|
||||
}
|
||||
|
||||
@ -37,7 +36,7 @@ void MySQLBlockOutputStream::writePrefix()
|
||||
}
|
||||
}
|
||||
|
||||
void MySQLBlockOutputStream::write(const Block & block)
|
||||
void MySQLWireBlockOutputStream::write(const Block & block)
|
||||
{
|
||||
size_t rows = block.rows();
|
||||
|
||||
@ -57,7 +56,7 @@ void MySQLBlockOutputStream::write(const Block & block)
|
||||
}
|
||||
}
|
||||
|
||||
void MySQLBlockOutputStream::writeSuffix()
|
||||
void MySQLWireBlockOutputStream::writeSuffix()
|
||||
{
|
||||
QueryStatus * process_list_elem = context.getProcessListElement();
|
||||
CurrentThread::finalizePerformanceCounters();
|
||||
@ -79,7 +78,7 @@ void MySQLBlockOutputStream::writeSuffix()
|
||||
packet_sender->sendPacket(EOF_Packet(0, 0), true);
|
||||
}
|
||||
|
||||
void MySQLBlockOutputStream::flush()
|
||||
void MySQLWireBlockOutputStream::flush()
|
||||
{
|
||||
packet_sender->out->next();
|
||||
}
|
||||
@ -1,7 +1,7 @@
|
||||
#pragma once
|
||||
|
||||
#include "IBlockOutputStream.h"
|
||||
#include <Core/MySQLProtocol.h>
|
||||
#include <DataStreams/IBlockOutputStream.h>
|
||||
#include <Formats/FormatFactory.h>
|
||||
#include <Formats/FormatSettings.h>
|
||||
#include <Interpreters/Context.h>
|
||||
@ -11,10 +11,10 @@ namespace DB
|
||||
|
||||
/** Interface for writing rows in MySQL Client/Server Protocol format.
|
||||
*/
|
||||
class MySQLBlockOutputStream : public IBlockOutputStream
|
||||
class MySQLWireBlockOutputStream : public IBlockOutputStream
|
||||
{
|
||||
public:
|
||||
MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context);
|
||||
MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context);
|
||||
|
||||
Block getHeader() const { return header; }
|
||||
|
||||
@ -31,6 +31,6 @@ private:
|
||||
FormatSettings format_settings;
|
||||
};
|
||||
|
||||
using MySQLBlockOutputStreamPtr = std::shared_ptr<MySQLBlockOutputStream>;
|
||||
using MySQLWireBlockOutputStreamPtr = std::shared_ptr<MySQLWireBlockOutputStream>;
|
||||
|
||||
}
|
||||
19
dbms/src/Formats/MySQLWireFormat.cpp
Normal file
19
dbms/src/Formats/MySQLWireFormat.cpp
Normal file
@ -0,0 +1,19 @@
|
||||
#include <Formats/MySQLWireBlockOutputStream.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
void registerOutputFormatMySQLWire(FormatFactory & factory)
|
||||
{
|
||||
factory.registerOutputFormat("MySQLWire", [](
|
||||
WriteBuffer & buf,
|
||||
const Block & sample,
|
||||
const Context & context,
|
||||
const FormatSettings &)
|
||||
{
|
||||
return std::make_shared<MySQLWireBlockOutputStream>(buf, sample, const_cast<Context &>(context));
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
@ -2,8 +2,8 @@
|
||||
#if USE_PROTOBUF
|
||||
|
||||
#include <Formats/FormatSchemaInfo.h>
|
||||
#include <Formats/ProtobufSchemas.h> // Y_IGNORE
|
||||
#include <google/protobuf/compiler/importer.h> // Y_IGNORE
|
||||
#include <Formats/ProtobufSchemas.h>
|
||||
#include <google/protobuf/compiler/importer.h>
|
||||
#include <Common/Exception.h>
|
||||
|
||||
|
||||
|
||||
@ -7,8 +7,8 @@
|
||||
#include <AggregateFunctions/IAggregateFunction.h>
|
||||
#include <DataTypes/DataTypesDecimal.h>
|
||||
#include <boost/numeric/conversion/cast.hpp>
|
||||
#include <google/protobuf/descriptor.h> // Y_IGNORE
|
||||
#include <google/protobuf/descriptor.pb.h> // Y_IGNORE
|
||||
#include <google/protobuf/descriptor.h>
|
||||
#include <google/protobuf/descriptor.pb.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include "ProtobufWriter.h"
|
||||
|
||||
@ -7,7 +7,7 @@
|
||||
#include <Functions/FunctionHelpers.h>
|
||||
#include <Functions/GatherUtils/Algorithms.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <libbase64.h> // Y_IGNORE
|
||||
#include <libbase64.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -24,7 +24,7 @@
|
||||
#if USE_EMBEDDED_COMPILER
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
@ -354,27 +354,27 @@ template <bool V, typename T> struct Case : std::bool_constant<V> { using type =
|
||||
template <typename... Ts> using Switch = typename std::disjunction<Ts..., Case<true, InvalidType>>::type;
|
||||
|
||||
template <typename DataType> constexpr bool IsIntegral = false;
|
||||
template <> constexpr bool IsIntegral<DataTypeUInt8> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeUInt16> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeUInt32> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeUInt64> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeInt8> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeInt16> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeInt32> = true;
|
||||
template <> constexpr bool IsIntegral<DataTypeInt64> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeUInt8> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeUInt16> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeUInt32> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeUInt64> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeInt8> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeInt16> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeInt32> = true;
|
||||
template <> inline constexpr bool IsIntegral<DataTypeInt64> = true;
|
||||
|
||||
template <typename DataType> constexpr bool IsFloatingPoint = false;
|
||||
template <> constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
|
||||
template <> constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
|
||||
template <> inline constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
|
||||
template <> inline constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
|
||||
|
||||
template <typename DataType> constexpr bool IsDateOrDateTime = false;
|
||||
template <> constexpr bool IsDateOrDateTime<DataTypeDate> = true;
|
||||
template <> constexpr bool IsDateOrDateTime<DataTypeDateTime> = true;
|
||||
template <> inline constexpr bool IsDateOrDateTime<DataTypeDate> = true;
|
||||
template <> inline constexpr bool IsDateOrDateTime<DataTypeDateTime> = true;
|
||||
|
||||
template <typename T0, typename T1> constexpr bool UseLeftDecimal = false;
|
||||
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true;
|
||||
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true;
|
||||
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true;
|
||||
template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true;
|
||||
template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true;
|
||||
template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true;
|
||||
|
||||
template <typename T> using DataTypeFromFieldType = std::conditional_t<std::is_same_v<T, NumberTraits::Error>, InvalidType, DataTypeNumber<T>>;
|
||||
|
||||
|
||||
@ -22,8 +22,8 @@
|
||||
#pragma clang diagnostic ignored "-Wshift-negative-value"
|
||||
#endif
|
||||
|
||||
#include <vectorf128.h> // Y_IGNORE
|
||||
#include <vectormath_exp.h> // Y_IGNORE
|
||||
#include <vectorf128.h>
|
||||
#include <vectormath_exp.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic pop
|
||||
|
||||
@ -21,9 +21,9 @@
|
||||
#pragma clang diagnostic ignored "-Wshift-negative-value"
|
||||
#endif
|
||||
|
||||
#include <vectorf128.h> // Y_IGNORE
|
||||
#include <vectormath_exp.h> // Y_IGNORE
|
||||
#include <vectormath_trig.h> // Y_IGNORE
|
||||
#include <vectorf128.h>
|
||||
#include <vectormath_exp.h>
|
||||
#include <vectormath_trig.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic pop
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user