thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-30 11:32:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of https://github.com/vzakaznikov/ClickHouse into liveview

Browse Source
This commit is contained in:
Vitaliy Zakaznikov 2019-06-07 08:02:59 -04:00
commit b65ff910b6
416 changed files with 11323 additions and 3726 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

66
CHANGELOG.md
View File

@ -1,9 +1,70 @@
## ClickHouse release 19.7.3.9, 2019-05-30
### New Features
* Allow to limit the range of a setting that can be specified by user.
These constraints can be set up in user settings profile.
[#4931](https://github.com/yandex/ClickHouse/pull/4931) ([Vitaly
Baranov](https://github.com/vitlibar))
* Add a second version of the function `groupUniqArray` with an optional
`max_size` parameter that limits the size of the resulting array. This
behavior is similar to `groupArray(max_size)(x)` function.
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
Tassery](https://github.com/YiuRULE))
* For TSVWithNames/CSVWithNames input file formats, column order can now be
determined from file header. This is controlled by
`input_format_with_names_use_header` parameter.
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
([Alexander](https://github.com/Akazz))
### Bug Fixes
* Crash with uncompressed_cache + JOIN during merge (#5197)
[#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
Kutenin](https://github.com/danlark1))
* Segmentation fault on a clickhouse-client query to system tables. #5066
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
([Ivan](https://github.com/abyss7))
* Data loss on heavy load via KafkaEngine (#4736)
[#5080](https://github.com/yandex/ClickHouse/pull/5080)
([Ivan](https://github.com/abyss7))
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
### Performance Improvements
* Use radix sort for sorting by single numeric column in `ORDER BY` without
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
[#4439](https://github.com/yandex/ClickHouse/pull/4439)
([Evgenii Pravda](https://github.com/kvinty),
[alexey-milovidov](https://github.com/alexey-milovidov))
### Documentation
* Translate documentation for some table engines to Chinese.
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
([张风啸](https://github.com/AlexZFX)),
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
lee](https://github.com/neverlee))
### Build/Testing/Packaging Improvements
* Print UTF-8 characters properly in `clickhouse-test`.
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Add command line parameter for clickhouse-client to always load suggestion
data. [#5102](https://github.com/yandex/ClickHouse/pull/5102)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Resolve some of PVS-Studio warnings.
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Update LZ4 [#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
Kutenin](https://github.com/danlark1))
* Add gperf to build requirements for upcoming pull request #5030.
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
([proller](https://github.com/proller))
## ClickHouse release 19.6.2.11, 2019-05-13 ## ClickHouse release 19.6.2.11, 2019-05-13
### New Features ### New Features
* TTL expressions for columns and tables. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ)) * TTL expressions for columns and tables. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
* Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin)) * Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
* Added a new aggregate function `simpleLinearRegression(x, y)` which performs linear regression on points (x, y) and returns the parameters of the line. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
* Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1)) * Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
* Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei)) * Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
@ -30,6 +91,7 @@
* Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar)) * Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
* Partially fix distributed_product_mode = local. It's possible to allow columns of local tables in where/having/order by/... via table aliases. Throw exception if table does not have alias. There's not possible to access to the columns without table aliases yet. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2)) * Partially fix distributed_product_mode = local. It's possible to allow columns of local tables in where/having/order by/... via table aliases. Throw exception if table does not have alias. There's not possible to access to the columns without table aliases yet. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
* Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2)) * Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
### Build/Testing/Packaging Improvements ### Build/Testing/Packaging Improvements
* Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk)) * Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
@ -346,7 +408,7 @@
* Added support of `Nullable` types in `mysql` table function. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin)) * Added support of `Nullable` types in `mysql` table function. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
* Support for arbitrary constant expressions in `LIMIT` clause. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box)) * Support for arbitrary constant expressions in `LIMIT` clause. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
* Added `topKWeighted` aggregate function that takes additional argument with (unsigned integer) weight. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman)) * Added `topKWeighted` aggregate function that takes additional argument with (unsigned integer) weight. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
* `StorageJoin` now supports `join_overwrite` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird) * `StorageJoin` now supports `join_any_take_last_row` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
* Added function `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar)) * Added function `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
* Added `RowBinaryWithNamesAndTypes` format. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer)) * Added `RowBinaryWithNamesAndTypes` format. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
* Added `IPv4` and `IPv6` data types. More effective implementations of `IPv*` functions. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk)) * Added `IPv4` and `IPv6` data types. More effective implementations of `IPv*` functions. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk))

71
CHANGELOG_RU.md
View File

@ -1,13 +1,77 @@
## ClickHouse release 19.7.3.9, 2019-05-30
### Новые возможности
* Добавлена возможность ограничить значения конфигурационных параметров,
которые может задать пользователь. Эти ограничения устанавливаются в профиле
настроек пользователя. [#4931](https://github.com/yandex/ClickHouse/pull/4931)
([Vitaly Baranov](https://github.com/vitlibar))
* Добавлен вариант функции `groupUniqArray` с дополнительным параметром
`max_size`, который ограничивает размер результирующего массива, аналогично
функции `groupArray(max_size)(x)`.
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
Tassery](https://github.com/YiuRULE))
* Для входных файлов формата TSVWithNames и CSVWithNames появилась возможность
определить порядок колонок в файле исходя из его заголовка. Это поведение
управляется конфигурационным параметром `input_format_with_names_use_header`.
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
([Alexander](https://github.com/Akazz))
### Исправления ошибок
* Падение в процессе слияния при использовании uncompressed_cache и JOIN
(#5197). [#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
Kutenin](https://github.com/danlark1))
* Segmentation fault на запросе к системным таблицам (#5066).
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
([Ivan](https://github.com/abyss7))
* Потеря загружаемых данных при больших потоках загрузки через KafkaEngine
(#4736). [#5080](https://github.com/yandex/ClickHouse/pull/5080)
([Ivan](https://github.com/abyss7))
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
### Улучшения производительности
* Используется поразрядная сортировка числовых колонок для `ORDER BY` без
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
[#4439](https://github.com/yandex/ClickHouse/pull/4439) ([Evgenii
Pravda](https://github.com/kvinty),
[alexey-milovidov](https://github.com/alexey-milovidov)
### Документация
* Документация для некоторых табличных движков переведена на китайский.
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
([张风啸](https://github.com/AlexZFX),
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
lee](https://github.com/neverlee))
### Улучшения сборки, тестирования и пакетирования
* Правильно отображаются символы в кодировке UTF-8 в `clickhouse-test`.
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Добавлен параметр командной строки для `clickhouse-client`, позволяющий
всегда загружать данные подсказок.
[#5102](https://github.com/yandex/ClickHouse/pull/5102)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлены некоторые предупреждения PVS-Studio.
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Обновлена библиотека LZ4.
[#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
Kutenin](https://github.com/danlark1))
* В зависимости сборки добавлен gperf для поддержки готовящегося PR #5030.
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
([proller](https://github.com/proller))
## ClickHouse release 19.6.2.11, 2019-05-13 ## ClickHouse release 19.6.2.11, 2019-05-13
### Новые возможности ### Новые возможности
* TTL выражения, позволяющие настроить время жизни и автоматическую очистку данных в таблице или в отдельных её столбцах. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ)) * TTL выражения, позволяющие настроить время жизни и автоматическую очистку данных в таблице или в отдельных её столбцах. [#4212](https://github.com/yandex/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ))
* Добавлена поддержка алгоритма сжатия `brotli` в HTTP ответах (`Accept-Encoding: br`). Для тела POST запросов, эта возможность уже существовала. [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin)) * Добавлена поддержка алгоритма сжатия `brotli` в HTTP ответах (`Accept-Encoding: br`). Для тела POST запросов, эта возможность уже существовала. [#4388](https://github.com/yandex/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin))
* Добавлена агрегатная функция `simpleLinearRegression(x, y)` которая осуществляет линейную регрессию на точках (x, y) и возвращает параметры линии. (from #4668) [#4917](https://github.com/yandex/ClickHouse/pull/4917) ([hcz](https://github.com/hczhcz))
* Добавлена функция `isValidUTF8` для проверки, содержит ли строка валидные данные в кодировке UTF-8. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1)) * Добавлена функция `isValidUTF8` для проверки, содержит ли строка валидные данные в кодировке UTF-8. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
* Добавлены новое правило балансировки (`load_balancing`) `first_or_random` по которому запросы посылаются на первый заданый хост и если он недоступен - на случайные хосты шарда. Полезно для топологий с кросс-репликацией. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei)) * Добавлены новое правило балансировки (`load_balancing`) `first_or_random` по которому запросы посылаются на первый заданый хост и если он недоступен - на случайные хосты шарда. Полезно для топологий с кросс-репликацией. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
### Эксперемннтальные возможности ### Экспериментальные возможности
* Добавлена настройка `index_granularity_bytes` (адаптивная гранулярность индекса) для таблиц семейства MergeTree* . [#4826](https://github.com/yandex/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin)) * Добавлена настройка `index_granularity_bytes` (адаптивная гранулярность индекса) для таблиц семейства MergeTree* . [#4826](https://github.com/yandex/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin))
### Улучшения ### Улучшения
@ -30,6 +94,7 @@
* Исправлено зависание на старте сервера если внешний словарь зависит от другого словаря через использование таблицы из БД с движком `Dictionary`. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar)) * Исправлено зависание на старте сервера если внешний словарь зависит от другого словаря через использование таблицы из БД с движком `Dictionary`. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
* При использовании `distributed_product_mode = 'local'` корректно работает использование столбцов локальных таблиц в where/having/order by/... через табличные алиасы. Выкидывает исключение если таблица не имеет алиас. Доступ к столбцам без алиасов пока не возможен. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2)) * При использовании `distributed_product_mode = 'local'` корректно работает использование столбцов локальных таблиц в where/having/order by/... через табличные алиасы. Выкидывает исключение если таблица не имеет алиас. Доступ к столбцам без алиасов пока не возможен. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
* Исправлен потенциально некорректный результат для `SELECT DISTINCT` с `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2)) * Исправлен потенциально некорректный результат для `SELECT DISTINCT` с `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
### Улучшения сборки/тестирования/пакетирования ### Улучшения сборки/тестирования/пакетирования
* Исправлена неработоспособность тестов, если `clickhouse-server` запущен на удалённом хосте [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk)) * Исправлена неработоспособность тестов, если `clickhouse-server` запущен на удалённом хосте [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
@ -337,7 +402,7 @@
* Добавлена поддержка `Nullable` типов в табличной функции `mysql`. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin)) * Добавлена поддержка `Nullable` типов в табличной функции `mysql`. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
* Добавлена поддержка произвольных константных выражений в секции `LIMIT`. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box)) * Добавлена поддержка произвольных константных выражений в секции `LIMIT`. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
* Добавлена агрегатная функция `topKWeighted` - вариант `topK`, позволяющий задавать (целый неотрицательный) вес добавляемого значения. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman)) * Добавлена агрегатная функция `topKWeighted` - вариант `topK`, позволяющий задавать (целый неотрицательный) вес добавляемого значения. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
* Движок `Join` теперь поддерживает настройку `join_overwrite`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)) * Движок `Join` теперь поддерживает настройку `join_any_take_last_row`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
* Добавлена функция `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar)) * Добавлена функция `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
* Добавлена функция `toStartOfTenMinutes`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar)) * Добавлена функция `toStartOfTenMinutes`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar))
* Добавлен формат `RowBinaryWithNamesAndTypes`. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer)) * Добавлен формат `RowBinaryWithNamesAndTypes`. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))

5
CMakeLists.txt
View File

@ -93,6 +93,11 @@ if (COMPILER_GCC OR COMPILER_CLANG)
set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wnon-virtual-dtor") set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wnon-virtual-dtor")
endif () endif ()
if (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER "8.3.0")
# Warnings in protobuf generating
set (CXX_WARNING_FLAGS "${CXX_WARNING_FLAGS} -Wno-array-bounds")
endif ()
if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang") if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
# clang: warning: argument unused during compilation: '-stdlib=libc++' # clang: warning: argument unused during compilation: '-stdlib=libc++'
# clang: warning: argument unused during compilation: '-specs=/usr/share/dpkg/no-pie-compile.specs' [-Wunused-command-line-argument] # clang: warning: argument unused during compilation: '-specs=/usr/share/dpkg/no-pie-compile.specs' [-Wunused-command-line-argument]

3
README.md
View File

@ -12,8 +12,7 @@ ClickHouse is an open-source column-oriented database management system that all
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person. * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
## Upcoming Events ## Upcoming Events
* ClickHouse at [Percona Live 2019](https://www.percona.com/live/19/other-open-source-databases-track) in Austin on May 28-30.
* [ClickHouse Community Meetup in San Francisco](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/261110652/) on June 4.
* [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8. * [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8.
* [ClickHouse on HighLoad++ Siberia](https://www.highload.ru/siberia/2019/abstracts/5348) on June 24-25.
* [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20. * [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
* [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27. * [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.

4
contrib/CMakeLists.txt vendored
View File

@ -1,8 +1,8 @@
# Third-party libraries may have substandard code. # Third-party libraries may have substandard code.
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch") set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch -Wno-stringop-truncation")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z") set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -Wno-array-bounds -Wno-missing-attributes -Wno-stringop-truncation -std=c++1z")
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang") elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch -Wno-string-plus-int") set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch -Wno-string-plus-int")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z") set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")

2
contrib/boost vendored

@ -1 +1 @@
Subproject commit 79bf85ea99c05ba4fb6959474d4464ab126f8973 Subproject commit 8abda007bfe52d78a51548d4594879d6d82a22fa

4
contrib/croaring/CMakeLists.txt
View File

@ -1,6 +1,6 @@
add_library(roaring add_library(roaring
roaring.c roaring.c
roaring.h roaring/roaring.h
roaring.hh) roaring/roaring.hh)
target_include_directories (roaring PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}) target_include_directories (roaring PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

2
contrib/croaring/roaring.c
View File

@ -1,5 +1,5 @@
/* auto-generated on Tue Dec 18 09:42:59 CST 2018. Do not edit! */ /* auto-generated on Tue Dec 18 09:42:59 CST 2018. Do not edit! */
#include "roaring.h" #include "roaring/roaring.h"
/* used for http://dmalloc.com/ Dmalloc - Debug Malloc Library */ /* used for http://dmalloc.com/ Dmalloc - Debug Malloc Library */
#ifdef DMALLOC #ifdef DMALLOC

0
contrib/croaring/roaring.h → contrib/croaring/roaring/roaring.h
View File

0
contrib/croaring/roaring.hh → contrib/croaring/roaring/roaring.hh
View File

2
contrib/hyperscan vendored

@ -1 +1 @@
Subproject commit 05b0f9064cca4bd55548dedb0a32ed9461146c1e Subproject commit 01e6b83f9fbdb4020cd68a5287bf3a0471eeb272

4
contrib/jemalloc-cmake/CMakeLists.txt
View File

@ -40,6 +40,10 @@ if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
list(APPEND SRCS ${JEMALLOC_SOURCE_DIR}/src/zone.c) list(APPEND SRCS ${JEMALLOC_SOURCE_DIR}/src/zone.c)
endif() endif()
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
endif ()
add_library(jemalloc STATIC ${SRCS}) add_library(jemalloc STATIC ${SRCS})
target_include_directories(jemalloc PUBLIC target_include_directories(jemalloc PUBLIC

2
contrib/libpcg-random/include/pcg_extras.hpp
View File

@ -81,7 +81,7 @@
#define PCG_128BIT_CONSTANT(high,low) \ #define PCG_128BIT_CONSTANT(high,low) \
((pcg128_t(high) << 64) + low) ((pcg128_t(high) << 64) + low)
#else #else
#include "pcg_uint128.hpp" // Y_IGNORE #include "pcg_uint128.hpp"
namespace pcg_extras { namespace pcg_extras {
typedef pcg_extras::uint_x4<uint32_t,uint64_t> pcg128_t; typedef pcg_extras::uint_x4<uint32_t,uint64_t> pcg128_t;
} }

2
contrib/librdkafka-cmake/include/librdkafka/rdkafka.h
View File

@ -1,5 +1,5 @@
#if __has_include(<rdkafka.h>) // maybe bundled #if __has_include(<rdkafka.h>) // maybe bundled
# include_next <rdkafka.h> // Y_IGNORE # include_next <rdkafka.h>
#else // system #else // system
# include_next <librdkafka/rdkafka.h> # include_next <librdkafka/rdkafka.h>
#endif #endif

9
dbms/CMakeLists.txt
View File

@ -49,19 +49,20 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
endif () endif ()
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7) if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
if (WEVERYTHING) if (WEVERYTHING)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-return-std-move-in-c++11") set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-return-std-move-in-c++11")
endif () endif ()
endif () endif ()
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8) if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wshadow-field -Wstring-plus-int -Wempty-init-stmt") set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow-field -Wstring-plus-int")
if(NOT APPLE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wempty-init-stmt")
endif()
endif () endif ()
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9) if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9)
if (WEVERYTHING) if (WEVERYTHING AND NOT APPLE)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-ctad-maybe-unsupported") set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-ctad-maybe-unsupported")
endif () endif ()
endif () endif ()

10
dbms/cmake/version.cmake
View File

@ -1,11 +1,11 @@
# This strings autochanged from release_lib.sh: # This strings autochanged from release_lib.sh:
set(VERSION_REVISION 54420) set(VERSION_REVISION 54421)
set(VERSION_MAJOR 19) set(VERSION_MAJOR 19)
set(VERSION_MINOR 8) set(VERSION_MINOR 9)
set(VERSION_PATCH 1) set(VERSION_PATCH 1)
set(VERSION_GITHASH a76e504f45ff4a74e8c492bd269f022352d5f6d9) set(VERSION_GITHASH 0c2aa460651a462f14efc7e995840a244531d373)
set(VERSION_DESCRIBE v19.8.1.1-testing) set(VERSION_DESCRIBE v19.9.1.1-testing)
set(VERSION_STRING 19.8.1.1) set(VERSION_STRING 19.9.1.1)
# end of autochange # end of autochange
set(VERSION_EXTRA "" CACHE STRING "") set(VERSION_EXTRA "" CACHE STRING "")

17
dbms/programs/client/CMakeLists.txt
View File

@ -1,6 +1,19 @@
set(CLICKHOUSE_CLIENT_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp) set(CLICKHOUSE_CLIENT_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/Client.cpp
${CMAKE_CURRENT_SOURCE_DIR}/ConnectionParameters.cpp
)
set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY}) set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY})
set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR}) set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/include)
include(CheckSymbolExists)
check_symbol_exists(readpassphrase readpassphrase.h HAVE_READPASSPHRASE)
configure_file(config_client.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/config_client.h)
if(NOT HAVE_READPASSPHRASE)
add_subdirectory(readpassphrase)
list(APPEND CLICKHOUSE_CLIENT_LINK PRIVATE readpassphrase)
endif()
clickhouse_program_add(client) clickhouse_program_add(client)

6
dbms/programs/client/Client.cpp
View File

@ -67,7 +67,7 @@
#include <Storages/ColumnsDescription.h> #include <Storages/ColumnsDescription.h>
#if USE_READLINE #if USE_READLINE
#include "Suggest.h" // Y_IGNORE #include "Suggest.h"
#endif #endif
#ifndef __clang__ #ifndef __clang__
@ -435,7 +435,7 @@ private:
#if USE_READLINE #if USE_READLINE
int res = read_history(history_file.c_str()); int res = read_history(history_file.c_str());
if (res) if (res)
throwFromErrno("Cannot read history from file " + history_file, ErrorCodes::CANNOT_READ_HISTORY); std::cerr << "Cannot read history from file " + history_file + ": "+ errnoToString(ErrorCodes::CANNOT_READ_HISTORY);
#endif #endif
} }
else /// Create history file. else /// Create history file.
@ -612,7 +612,7 @@ private:
#if USE_READLINE && HAVE_READLINE_HISTORY #if USE_READLINE && HAVE_READLINE_HISTORY
if (!history_file.empty() && append_history(1, history_file.c_str())) if (!history_file.empty() && append_history(1, history_file.c_str()))
throwFromErrno("Cannot append history to file " + history_file, ErrorCodes::CANNOT_APPEND_HISTORY); std::cerr << "Cannot append history to file " + history_file + ": " + errnoToString(ErrorCodes::CANNOT_APPEND_HISTORY);
#endif #endif
prev_input = input; prev_input = input;

63
dbms/programs/client/ConnectionParameters.cpp Normal file
View File

@ -0,0 +1,63 @@
#include "ConnectionParameters.h"
#include <fstream>
#include <iostream>
#include <Core/Defines.h>
#include <Core/Protocol.h>
#include <Core/Types.h>
#include <IO/ConnectionTimeouts.h>
#include <Poco/Util/AbstractConfiguration.h>
#include <Common/Exception.h>
#include <common/setTerminalEcho.h>
#include <ext/scope_guard.h>
#include <readpassphrase.h>
namespace DB
{
namespace ErrorCodes
{
extern const int BAD_ARGUMENTS;
}
ConnectionParameters::ConnectionParameters(const Poco::Util::AbstractConfiguration & config)
{
bool is_secure = config.getBool("secure", false);
security = is_secure ? Protocol::Secure::Enable : Protocol::Secure::Disable;
host = config.getString("host", "localhost");
port = config.getInt(
"port", config.getInt(is_secure ? "tcp_port_secure" : "tcp_port", is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
default_database = config.getString("database", "");
/// changed the default value to "default" to fix the issue when the user in the prompt is blank
user = config.getString("user", "default");
bool password_prompt = false;
if (config.getBool("ask-password", false))
{
if (config.has("password"))
throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
password_prompt = true;
}
else
{
password = config.getString("password", "");
/// if the value of --password is omitted, the password will be set implicitly to "\n"
if (password == "\n")
password_prompt = true;
}
if (password_prompt)
{
std::string prompt{"Password for user (" + user + "): "};
char buf[1000] = {};
if (auto result = readpassphrase(prompt.c_str(), buf, sizeof(buf), 0))
password = result;
}
compression = config.getBool("compression", true) ? Protocol::Compression::Enable : Protocol::Compression::Disable;
timeouts = ConnectionTimeouts(
Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
}
}

80
dbms/programs/client/ConnectionParameters.h
View File

@ -1,90 +1,30 @@
#pragma once #pragma once
#include <iostream> #include <string>
#include <Core/Types.h>
#include <Core/Protocol.h> #include <Core/Protocol.h>
#include <Core/Defines.h>
#include <Common/Exception.h>
#include <IO/ConnectionTimeouts.h> #include <IO/ConnectionTimeouts.h>
#include <common/setTerminalEcho.h> namespace Poco::Util
#include <ext/scope_guard.h> {
class AbstractConfiguration;
#include <Poco/Util/AbstractConfiguration.h> }
namespace DB namespace DB
{ {
namespace ErrorCodes
{
extern const int BAD_ARGUMENTS;
}
struct ConnectionParameters struct ConnectionParameters
{ {
String host; std::string host;
UInt16 port{}; UInt16 port{};
String default_database; std::string default_database;
String user; std::string user;
String password; std::string password;
Protocol::Secure security = Protocol::Secure::Disable; Protocol::Secure security = Protocol::Secure::Disable;
Protocol::Compression compression = Protocol::Compression::Enable; Protocol::Compression compression = Protocol::Compression::Enable;
ConnectionTimeouts timeouts; ConnectionTimeouts timeouts;
ConnectionParameters() {} ConnectionParameters() {}
ConnectionParameters(const Poco::Util::AbstractConfiguration & config) ConnectionParameters(const Poco::Util::AbstractConfiguration & config);
{
bool is_secure = config.getBool("secure", false);
security = is_secure
? Protocol::Secure::Enable
: Protocol::Secure::Disable;
host = config.getString("host", "localhost");
port = config.getInt("port",
config.getInt(is_secure ? "tcp_port_secure" : "tcp_port",
is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
default_database = config.getString("database", "");
/// changed the default value to "default" to fix the issue when the user in the prompt is blank
user = config.getString("user", "default");
bool password_prompt = false;
if (config.getBool("ask-password", false))
{
if (config.has("password"))
throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
password_prompt = true;
}
else
{
password = config.getString("password", "");
/// if the value of --password is omitted, the password will be set implicitly to "\n"
if (password == "\n")
password_prompt = true;
}
if (password_prompt)
{
std::cout << "Password for user (" << user << "): ";
setTerminalEcho(false);
SCOPE_EXIT({
setTerminalEcho(true);
});
std::getline(std::cin, password);
std::cout << std::endl;
}
compression = config.getBool("compression", true)
? Protocol::Compression::Enable
: Protocol::Compression::Disable;
timeouts = ConnectionTimeouts(
Poco::Timespan(config.getInt("connect_timeout", DBMS_DEFAULT_CONNECT_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("send_timeout", DBMS_DEFAULT_SEND_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("receive_timeout", DBMS_DEFAULT_RECEIVE_TIMEOUT_SEC), 0),
Poco::Timespan(config.getInt("tcp_keep_alive_timeout", 0), 0));
}
}; };
} }

3
dbms/programs/client/config_client.h.in Normal file
View File

@ -0,0 +1,3 @@
#pragma once
#cmakedefine HAVE_READPASSPHRASE

10
dbms/programs/client/readpassphrase/CMakeLists.txt Normal file
View File

@ -0,0 +1,10 @@
# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.c
# wget https://raw.githubusercontent.com/openssh/openssh-portable/master/openbsd-compat/readpassphrase.h
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-result -Wno-reserved-id-macro")
configure_file(includes.h.in ${CMAKE_CURRENT_BINARY_DIR}/include/includes.h)
add_library(readpassphrase ${CMAKE_CURRENT_SOURCE_DIR}/readpassphrase.c)
# . to allow #include <readpassphrase.h>
target_include_directories(readpassphrase PUBLIC . ${CMAKE_CURRENT_BINARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR}/../include)

9
dbms/programs/client/readpassphrase/includes.h.in Normal file
View File

@ -0,0 +1,9 @@
#pragma once
#cmakedefine HAVE_READPASSPHRASE
#if !defined(HAVE_READPASSPHRASE)
# ifndef _PATH_TTY
# define _PATH_TTY "/dev/tty"
# endif
#endif

211
dbms/programs/client/readpassphrase/readpassphrase.c Normal file
View File

@ -0,0 +1,211 @@
/* $OpenBSD: readpassphrase.c,v 1.26 2016/10/18 12:47:18 millert Exp $ */
/*
* Copyright (c) 2000-2002, 2007, 2010
* Todd C. Miller <Todd.Miller@courtesan.com>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*
* Sponsored in part by the Defense Advanced Research Projects
* Agency (DARPA) and Air Force Research Laboratory, Air Force
* Materiel Command, USAF, under agreement number F39502-99-1-0512.
*/
/* OPENBSD ORIGINAL: lib/libc/gen/readpassphrase.c */
#include "includes.h"
#ifndef HAVE_READPASSPHRASE
#include <termios.h>
#include <signal.h>
#include <ctype.h>
#include <fcntl.h>
#include <readpassphrase.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#ifndef TCSASOFT
/* If we don't have TCSASOFT define it so that ORing it it below is a no-op. */
# define TCSASOFT 0
#endif
/* SunOS 4.x which lacks _POSIX_VDISABLE, but has VDISABLE */
#if !defined(_POSIX_VDISABLE) && defined(VDISABLE)
# define _POSIX_VDISABLE VDISABLE
#endif
static volatile sig_atomic_t signo[_NSIG];
static void handler(int);
char *
readpassphrase(const char *prompt, char *buf, size_t bufsiz, int flags)
{
ssize_t nr;
int input, output, save_errno, i, need_restart;
char ch, *p, *end;
struct termios term, oterm;
struct sigaction sa, savealrm, saveint, savehup, savequit, saveterm;
struct sigaction savetstp, savettin, savettou, savepipe;
/* I suppose we could alloc on demand in this case (XXX). */
if (bufsiz == 0) {
errno = EINVAL;
return(NULL);
}
restart:
for (i = 0; i < _NSIG; i++)
signo[i] = 0;
nr = -1;
save_errno = 0;
need_restart = 0;
/*
* Read and write to /dev/tty if available. If not, read from
* stdin and write to stderr unless a tty is required.
*/
if ((flags & RPP_STDIN) ||
(input = output = open(_PATH_TTY, O_RDWR)) == -1) {
if (flags & RPP_REQUIRE_TTY) {
errno = ENOTTY;
return(NULL);
}
input = STDIN_FILENO;
output = STDERR_FILENO;
}
/*
* Turn off echo if possible.
* If we are using a tty but are not the foreground pgrp this will
* generate SIGTTOU, so do it *before* installing the signal handlers.
*/
if (input != STDIN_FILENO && tcgetattr(input, &oterm) == 0) {
memcpy(&term, &oterm, sizeof(term));
if (!(flags & RPP_ECHO_ON))
term.c_lflag &= ~(ECHO | ECHONL);
#ifdef VSTATUS
if (term.c_cc[VSTATUS] != _POSIX_VDISABLE)
term.c_cc[VSTATUS] = _POSIX_VDISABLE;
#endif
(void)tcsetattr(input, TCSAFLUSH|TCSASOFT, &term);
} else {
memset(&term, 0, sizeof(term));
term.c_lflag |= ECHO;
memset(&oterm, 0, sizeof(oterm));
oterm.c_lflag |= ECHO;
}
/*
* Catch signals that would otherwise cause the user to end
* up with echo turned off in the shell. Don't worry about
* things like SIGXCPU and SIGVTALRM for now.
*/
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0; /* don't restart system calls */
sa.sa_handler = handler;
(void)sigaction(SIGALRM, &sa, &savealrm);
(void)sigaction(SIGHUP, &sa, &savehup);
(void)sigaction(SIGINT, &sa, &saveint);
(void)sigaction(SIGPIPE, &sa, &savepipe);
(void)sigaction(SIGQUIT, &sa, &savequit);
(void)sigaction(SIGTERM, &sa, &saveterm);
(void)sigaction(SIGTSTP, &sa, &savetstp);
(void)sigaction(SIGTTIN, &sa, &savettin);
(void)sigaction(SIGTTOU, &sa, &savettou);
if (!(flags & RPP_STDIN))
(void)write(output, prompt, strlen(prompt));
end = buf + bufsiz - 1;
p = buf;
while ((nr = read(input, &ch, 1)) == 1 && ch != '\n' && ch != '\r') {
if (p < end) {
if ((flags & RPP_SEVENBIT))
ch &= 0x7f;
if (isalpha((unsigned char)ch)) {
if ((flags & RPP_FORCELOWER))
ch = (char)tolower((unsigned char)ch);
if ((flags & RPP_FORCEUPPER))
ch = (char)toupper((unsigned char)ch);
}
*p++ = ch;
}
}
*p = '\0';
save_errno = errno;
if (!(term.c_lflag & ECHO))
(void)write(output, "\n", 1);
/* Restore old terminal settings and signals. */
if (memcmp(&term, &oterm, sizeof(term)) != 0) {
const int sigttou = signo[SIGTTOU];
/* Ignore SIGTTOU generated when we are not the fg pgrp. */
while (tcsetattr(input, TCSAFLUSH|TCSASOFT, &oterm) == -1 &&
errno == EINTR && !signo[SIGTTOU])
continue;
signo[SIGTTOU] = sigttou;
}
(void)sigaction(SIGALRM, &savealrm, NULL);
(void)sigaction(SIGHUP, &savehup, NULL);
(void)sigaction(SIGINT, &saveint, NULL);
(void)sigaction(SIGQUIT, &savequit, NULL);
(void)sigaction(SIGPIPE, &savepipe, NULL);
(void)sigaction(SIGTERM, &saveterm, NULL);
(void)sigaction(SIGTSTP, &savetstp, NULL);
(void)sigaction(SIGTTIN, &savettin, NULL);
(void)sigaction(SIGTTOU, &savettou, NULL);
if (input != STDIN_FILENO)
(void)close(input);
/*
* If we were interrupted by a signal, resend it to ourselves
* now that we have restored the signal handlers.
*/
for (i = 0; i < _NSIG; i++) {
if (signo[i]) {
kill(getpid(), i);
switch (i) {
case SIGTSTP:
case SIGTTIN:
case SIGTTOU:
need_restart = 1;
}
}
}
if (need_restart)
goto restart;
if (save_errno)
errno = save_errno;
return(nr == -1 ? NULL : buf);
}
//DEF_WEAK(readpassphrase);
#if 0
char *
getpass(const char *prompt)
{
static char buf[_PASSWORD_LEN + 1];
return(readpassphrase(prompt, buf, sizeof(buf), RPP_ECHO_OFF));
}
#endif
static void handler(int s)
{
signo[s] = 1;
}
#endif /* HAVE_READPASSPHRASE */

56
dbms/programs/client/readpassphrase/readpassphrase.h Normal file
View File

@ -0,0 +1,56 @@
// /* $OpenBSD: readpassphrase.h,v 1.5 2003/06/17 21:56:23 millert Exp $ */
/*
* Copyright (c) 2000, 2002 Todd C. Miller <Todd.Miller@courtesan.com>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*
* Sponsored in part by the Defense Advanced Research Projects
* Agency (DARPA) and Air Force Research Laboratory, Air Force
* Materiel Command, USAF, under agreement number F39502-99-1-0512.
*/
/* OPENBSD ORIGINAL: include/readpassphrase.h */
#pragma once
// #ifndef _READPASSPHRASE_H_
// #define _READPASSPHRASE_H_
//#include "includes.h"
#include "config_client.h"
#ifndef HAVE_READPASSPHRASE
# ifdef __cplusplus
extern "C" {
# endif
# define RPP_ECHO_OFF 0x00 /* Turn off echo (default). */
# define RPP_ECHO_ON 0x01 /* Leave echo on. */
# define RPP_REQUIRE_TTY 0x02 /* Fail if there is no tty. */
# define RPP_FORCELOWER 0x04 /* Force input to lower case. */
# define RPP_FORCEUPPER 0x08 /* Force input to upper case. */
# define RPP_SEVENBIT 0x10 /* Strip the high bit from input. */
# define RPP_STDIN 0x20 /* Read from stdin, not /dev/tty */
char * readpassphrase(const char *, char *, size_t, int);
# ifdef __cplusplus
}
# endif
#endif /* HAVE_READPASSPHRASE */
// #endif /* !_READPASSPHRASE_H_ */

2
dbms/programs/main.cpp
View File

@ -15,7 +15,7 @@
#endif #endif
#if USE_TCMALLOC #if USE_TCMALLOC
#include <gperftools/malloc_extension.h> // Y_IGNORE #include <gperftools/malloc_extension.h>
#endif #endif
#include <Common/StringUtils/StringUtils.h> #include <Common/StringUtils/StringUtils.h>

6
dbms/programs/odbc-bridge/ColumnInfoHandler.cpp
View File

@ -3,9 +3,9 @@
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
#if USE_POCO_SQLODBC #if USE_POCO_SQLODBC
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE #include <Poco/SQL/ODBC/ODBCException.h>
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE #include <Poco/SQL/ODBC/SessionImpl.h>
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE #include <Poco/SQL/ODBC/Utility.h>
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
#endif #endif
#if USE_POCO_DATAODBC #if USE_POCO_DATAODBC

6
dbms/programs/odbc-bridge/IdentifierQuoteHandler.cpp
View File

@ -2,9 +2,9 @@
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
#if USE_POCO_SQLODBC #if USE_POCO_SQLODBC
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE #include <Poco/SQL/ODBC/ODBCException.h>
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE #include <Poco/SQL/ODBC/SessionImpl.h>
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE #include <Poco/SQL/ODBC/Utility.h>
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
#endif #endif
#if USE_POCO_DATAODBC #if USE_POCO_DATAODBC

6
dbms/programs/odbc-bridge/getIdentifierQuote.cpp
View File

@ -2,9 +2,9 @@
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
#if USE_POCO_SQLODBC #if USE_POCO_SQLODBC
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE #include <Poco/SQL/ODBC/ODBCException.h>
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE #include <Poco/SQL/ODBC/SessionImpl.h>
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE #include <Poco/SQL/ODBC/Utility.h>
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC #define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
#endif #endif
#if USE_POCO_DATAODBC #if USE_POCO_DATAODBC

2
dbms/programs/odbc-bridge/getIdentifierQuote.h
View File

@ -8,7 +8,7 @@
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC #if USE_POCO_SQLODBC || USE_POCO_DATAODBC
#if USE_POCO_SQLODBC #if USE_POCO_SQLODBC
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE #include <Poco/SQL/ODBC/Utility.h>
#endif #endif
#if USE_POCO_DATAODBC #if USE_POCO_DATAODBC
#include <Poco/Data/ODBC/Utility.h> #include <Poco/Data/ODBC/Utility.h>

26
dbms/programs/performance-test/PerformanceTest.cpp
View File

@ -1,6 +1,7 @@
#include "PerformanceTest.h" #include "PerformanceTest.h"
#include <Core/Types.h> #include <Core/Types.h>
#include <Common/CpuId.h>
#include <common/getMemoryAmount.h> #include <common/getMemoryAmount.h>
#include <IO/ReadBufferFromFile.h> #include <IO/ReadBufferFromFile.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
@ -71,6 +72,7 @@ bool PerformanceTest::checkPreconditions() const
Strings preconditions; Strings preconditions;
config->keys("preconditions", preconditions); config->keys("preconditions", preconditions);
size_t table_precondition_index = 0; size_t table_precondition_index = 0;
size_t cpu_precondition_index = 0;
for (const std::string & precondition : preconditions) for (const std::string & precondition : preconditions)
{ {
@ -136,6 +138,30 @@ bool PerformanceTest::checkPreconditions() const
return false; return false;
} }
} }
if (precondition == "cpu")
{
std::string precondition_key = "preconditions.cpu[" + std::to_string(cpu_precondition_index++) + "]";
std::string flag_to_check = config->getString(precondition_key);
#define CHECK_CPU_PRECONDITION(OP) \
if (flag_to_check == #OP) \
{ \
if (!Cpu::CpuFlagsCache::have_##OP) \
{ \
LOG_WARNING(log, "CPU doesn't support " << #OP); \
return false; \
} \
} else
CPU_ID_ENUMERATE(CHECK_CPU_PRECONDITION)
{
LOG_WARNING(log, "CPU doesn't support " << flag_to_check);
return false;
}
#undef CHECK_CPU_PRECONDITION
}
} }
return true; return true;

1
dbms/programs/server/CMakeLists.txt
View File

@ -9,6 +9,7 @@ set(CLICKHOUSE_SERVER_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/Server.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Server.cpp
${CMAKE_CURRENT_SOURCE_DIR}/TCPHandler.cpp ${CMAKE_CURRENT_SOURCE_DIR}/TCPHandler.cpp
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp ${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
) )
set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY}) set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})

73
dbms/programs/server/HTTPHandler.cpp
View File

@ -312,11 +312,13 @@ void HTTPHandler::processQuery(
client_supports_http_compression = true; client_supports_http_compression = true;
http_response_compression_method = CompressionMethod::Zlib; http_response_compression_method = CompressionMethod::Zlib;
} }
#if USE_BROTLI
else if (http_response_compression_methods == "br") else if (http_response_compression_methods == "br")
{ {
client_supports_http_compression = true; client_supports_http_compression = true;
http_response_compression_method = CompressionMethod::Brotli; http_response_compression_method = CompressionMethod::Brotli;
} }
#endif
} }
/// Client can pass a 'compress' flag in the query string. In this case the query result is /// Client can pass a 'compress' flag in the query string. In this case the query result is
@ -453,30 +455,6 @@ void HTTPHandler::processQuery(
return false; return false;
}; };
/// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
std::string full_query;
/// Support for "external data for query processing".
if (startsWith(request.getContentType().data(), "multipart/form-data"))
{
ExternalTablesHandler handler(context, params);
params.load(request, istr, handler);
/// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
reserved_param_suffixes.emplace_back("_format");
reserved_param_suffixes.emplace_back("_types");
reserved_param_suffixes.emplace_back("_structure");
/// Params are of both form params POST and uri (GET params)
for (const auto & it : params)
if (it.first == "query")
full_query += it.second;
in = std::make_unique<ReadBufferFromString>(full_query);
}
else
in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
/// Settings can be overridden in the query. /// Settings can be overridden in the query.
/// Some parameters (database, default_format, everything used in the code above) do not /// Some parameters (database, default_format, everything used in the code above) do not
/// belong to the Settings class. /// belong to the Settings class.
@ -497,30 +475,63 @@ void HTTPHandler::processQuery(
settings.readonly = 2; settings.readonly = 2;
} }
bool isExternalData = startsWith(request.getContentType().data(), "multipart/form-data");
if (isExternalData)
{
/// Skip unneeded parameters to avoid confusing them later with context settings or query parameters.
reserved_param_suffixes.reserve(3);
/// It is a bug and ambiguity with `date_time_input_format` and `low_cardinality_allow_in_native_format` formats/settings.
reserved_param_suffixes.emplace_back("_format");
reserved_param_suffixes.emplace_back("_types");
reserved_param_suffixes.emplace_back("_structure");
}
SettingsChanges settings_changes; SettingsChanges settings_changes;
for (auto it = params.begin(); it != params.end(); ++it) for (const auto & [key, value] : params)
{ {
if (it->first == "database") if (key == "database")
{ {
context.setCurrentDatabase(it->second); context.setCurrentDatabase(value);
} }
else if (it->first == "default_format") else if (key == "default_format")
{ {
context.setDefaultFormat(it->second); context.setDefaultFormat(value);
} }
else if (param_could_be_skipped(it->first)) else if (param_could_be_skipped(key))
{ {
} }
else else
{ {
/// All other query parameters are treated as settings. /// All other query parameters are treated as settings.
settings_changes.push_back({it->first, it->second}); settings_changes.push_back({key, value});
} }
} }
/// For external data we also want settings
context.checkSettingsConstraints(settings_changes); context.checkSettingsConstraints(settings_changes);
context.applySettingsChanges(settings_changes); context.applySettingsChanges(settings_changes);
/// Used in case of POST request with form-data, but it isn't expected to be deleted after that scope.
std::string full_query;
/// Support for "external data for query processing".
if (isExternalData)
{
ExternalTablesHandler handler(context, params);
params.load(request, istr, handler);
/// Params are of both form params POST and uri (GET params)
for (const auto & it : params)
if (it.first == "query")
full_query += it.second;
in = std::make_unique<ReadBufferFromString>(full_query);
}
else
in = std::make_unique<ConcatReadBuffer>(*in_param, *in_post_maybe_compressed);
/// HTTP response compression is turned on only if the client signalled that they support it /// HTTP response compression is turned on only if the client signalled that they support it
/// (using Accept-Encoding header) and 'enable_http_compression' setting is turned on. /// (using Accept-Encoding header) and 'enable_http_compression' setting is turned on.
used_output.out->setCompression(client_supports_http_compression && settings.enable_http_compression); used_output.out->setCompression(client_supports_http_compression && settings.enable_http_compression);

93
dbms/programs/server/MySQLHandler.cpp
View File

@ -1,5 +1,5 @@
#include <DataStreams/copyData.h> #include <DataStreams/copyData.h>
#include <IO/ReadBufferFromMemory.h> #include <IO/ReadBufferFromString.h>
#include <IO/ReadBufferFromPocoSocket.h> #include <IO/ReadBufferFromPocoSocket.h>
#include <IO/WriteBufferFromPocoSocket.h> #include <IO/WriteBufferFromPocoSocket.h>
#include <Interpreters/executeQuery.h> #include <Interpreters/executeQuery.h>
@ -9,12 +9,14 @@
#include <Columns/ColumnVector.h> #include <Columns/ColumnVector.h>
#include <Common/config_version.h> #include <Common/config_version.h>
#include <Common/NetException.h> #include <Common/NetException.h>
#include <Common/OpenSSLHelpers.h>
#include <Poco/Crypto/RSAKey.h> #include <Poco/Crypto/RSAKey.h>
#include <Poco/Crypto/CipherFactory.h> #include <Poco/Crypto/CipherFactory.h>
#include <Poco/Net/SecureStreamSocket.h> #include <Poco/Net/SecureStreamSocket.h>
#include <Poco/Net/SSLManager.h> #include <Poco/Net/SSLManager.h>
#include "MySQLHandler.h" #include "MySQLHandler.h"
#include <limits> #include <limits>
#include <ext/scope_guard.h>
namespace DB namespace DB
@ -26,20 +28,31 @@ using Poco::Net::SSLManager;
namespace ErrorCodes namespace ErrorCodes
{ {
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES; extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
extern const int UNKNOWN_EXCEPTION; extern const int OPENSSL_ERROR;
} }
uint32_t MySQLHandler::last_connection_id = 0; MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id)
: Poco::Net::TCPServerConnection(socket_)
, server(server_)
, log(&Poco::Logger::get("MySQLHandler"))
, connection_context(server.context())
, connection_id(connection_id)
, public_key(public_key)
, private_key(private_key)
{
server_capability_flags = CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA | CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF;
if (ssl_enabled)
server_capability_flags |= CLIENT_SSL;
}
void MySQLHandler::run() void MySQLHandler::run()
{ {
connection_context = server.context(); connection_context = server.context();
connection_context.setDefaultFormat("MySQL"); connection_context.setDefaultFormat("MySQLWire");
in = std::make_shared<ReadBufferFromPocoSocket>(socket()); in = std::make_shared<ReadBufferFromPocoSocket>(socket());
out = std::make_shared<WriteBufferFromPocoSocket>(socket()); out = std::make_shared<WriteBufferFromPocoSocket>(socket());
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler"); packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
try try
{ {
@ -49,8 +62,7 @@ void MySQLHandler::run()
* This plugin must do the same to stay consistent with historical behavior if it is set to operate as a default plugin. * This plugin must do the same to stay consistent with historical behavior if it is set to operate as a default plugin.
* https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L3994 * https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L3994
*/ */
Handshake handshake(connection_id, VERSION_STRING, scramble + '\0'); Handshake handshake(server_capability_flags, connection_id, VERSION_STRING + String("-") + VERSION_NAME, scramble + '\0');
packet_sender->sendPacket<Handshake>(handshake, true); packet_sender->sendPacket<Handshake>(handshake, true);
LOG_TRACE(log, "Sent handshake"); LOG_TRACE(log, "Sent handshake");
@ -78,15 +90,11 @@ void MySQLHandler::run()
<< "\nauth_plugin_name: " << "\nauth_plugin_name: "
<< handshake_response.auth_plugin_name); << handshake_response.auth_plugin_name);
capabilities = handshake_response.capability_flags; client_capability_flags = handshake_response.capability_flags;
if (!(capabilities & CLIENT_PROTOCOL_41)) if (!(client_capability_flags & CLIENT_PROTOCOL_41))
{
throw Exception("Required capability: CLIENT_PROTOCOL_41.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES); throw Exception("Required capability: CLIENT_PROTOCOL_41.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
} if (!(client_capability_flags & CLIENT_PLUGIN_AUTH))
if (!(capabilities & CLIENT_PLUGIN_AUTH))
{
throw Exception("Required capability: CLIENT_PLUGIN_AUTH.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES); throw Exception("Required capability: CLIENT_PLUGIN_AUTH.", ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
}
authenticate(handshake_response, scramble); authenticate(handshake_response, scramble);
OK_Packet ok_packet(0, handshake_response.capability_flags, 0, 0, 0); OK_Packet ok_packet(0, handshake_response.capability_flags, 0, 0, 0);
@ -165,7 +173,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
}; };
read_bytes(3); /// We can find out whether it is SSLRequest of HandshakeResponse by first 3 bytes. read_bytes(3); /// We can find out whether it is SSLRequest of HandshakeResponse by first 3 bytes.
size_t payload_size = *reinterpret_cast<uint32_t *>(buf) & 0xFFFFFFu; size_t payload_size = unalignedLoad<uint32_t>(buf) & 0xFFFFFFu;
LOG_TRACE(log, "payload size: " << payload_size); LOG_TRACE(log, "payload size: " << payload_size);
if (payload_size == SSL_REQUEST_PAYLOAD_SIZE) if (payload_size == SSL_REQUEST_PAYLOAD_SIZE)
@ -180,7 +188,7 @@ MySQLProtocol::HandshakeResponse MySQLHandler::finishHandshake()
in = std::make_shared<ReadBufferFromPocoSocket>(*ss); in = std::make_shared<ReadBufferFromPocoSocket>(*ss);
out = std::make_shared<WriteBufferFromPocoSocket>(*ss); out = std::make_shared<WriteBufferFromPocoSocket>(*ss);
connection_context.sequence_id = 2; connection_context.sequence_id = 2;
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id, "MySQLHandler"); packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.sequence_id);
packet_sender->max_packet_size = connection_context.max_packet_size; packet_sender->max_packet_size = connection_context.max_packet_size;
packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket. packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket.
} }
@ -216,6 +224,10 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
if (handshake_response.auth_plugin_name != Authentication::SHA256) if (handshake_response.auth_plugin_name != Authentication::SHA256)
{ {
packet_sender->sendPacket(AuthSwitchRequest(Authentication::SHA256, scramble + '\0'), true); packet_sender->sendPacket(AuthSwitchRequest(Authentication::SHA256, scramble + '\0'), true);
if (in->eof())
throw Exception(
"Client doesn't support authentication method " + String(Authentication::SHA256) + " used by ClickHouse",
ErrorCodes::MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES);
packet_sender->receivePacket(response); packet_sender->receivePacket(response);
auth_response = response.value; auth_response = response.value;
LOG_TRACE(log, "Authentication method mismatch."); LOG_TRACE(log, "Authentication method mismatch.");
@ -226,31 +238,19 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
LOG_TRACE(log, "Authentication method match."); LOG_TRACE(log, "Authentication method match.");
} }
auto getOpenSSLError = []() -> String
{
BIO * mem = BIO_new(BIO_s_mem());
ERR_print_errors(mem);
char * buf = nullptr;
long size = BIO_get_mem_data(mem, &buf);
String errors_str(buf, size);
BIO_free(mem);
return errors_str;
};
if (auth_response == "\1") if (auth_response == "\1")
{ {
LOG_TRACE(log, "Client requests public key."); LOG_TRACE(log, "Client requests public key.");
BIO * mem = BIO_new(BIO_s_mem()); BIO * mem = BIO_new(BIO_s_mem());
if (PEM_write_bio_RSA_PUBKEY(mem, public_key) != 1) SCOPE_EXIT(BIO_free(mem));
if (PEM_write_bio_RSA_PUBKEY(mem, &public_key) != 1)
{ {
LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError()); throw Exception("Failed to write public key to memory. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
throw Exception("Failed to write public key to memory.", ErrorCodes::UNKNOWN_EXCEPTION);
} }
char * pem_buf = nullptr; char * pem_buf = nullptr;
long pem_size = BIO_get_mem_data(mem, &pem_buf); long pem_size = BIO_get_mem_data(mem, &pem_buf);
String pem(pem_buf, pem_size); String pem(pem_buf, pem_size);
BIO_free(mem);
LOG_TRACE(log, "Key: " << pem); LOG_TRACE(log, "Key: " << pem);
@ -271,17 +271,16 @@ void MySQLHandler::authenticate(const HandshakeResponse & handshake_response, co
* an empty packet is a blank password, thus the check for auth_response.empty() has to be made too. * an empty packet is a blank password, thus the check for auth_response.empty() has to be made too.
* https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L4017 * https://github.com/mysql/mysql-server/blob/8.0/sql/auth/sql_authentication.cc#L4017
*/ */
if (!secure_connection && (!auth_response.empty() && auth_response != "\0")) if (!secure_connection && !auth_response.empty() && auth_response != String("\0", 1))
{ {
LOG_TRACE(log, "Received nonempty password"); LOG_TRACE(log, "Received nonempty password");
auto ciphertext = reinterpret_cast<unsigned char *>(auth_response.data()); auto ciphertext = reinterpret_cast<unsigned char *>(auth_response.data());
unsigned char plaintext[RSA_size(private_key)]; unsigned char plaintext[RSA_size(&private_key)];
int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, private_key, RSA_PKCS1_OAEP_PADDING); int plaintext_size = RSA_private_decrypt(auth_response.size(), ciphertext, plaintext, &private_key, RSA_PKCS1_OAEP_PADDING);
if (plaintext_size == -1) if (plaintext_size == -1)
{ {
LOG_TRACE(log, "OpenSSL error:\n" << getOpenSSLError()); throw Exception("Failed to decrypt auth data. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
throw Exception("Failed to decrypt.", ErrorCodes::UNKNOWN_EXCEPTION);
} }
password.resize(plaintext_size); password.resize(plaintext_size);
@ -324,7 +323,7 @@ void MySQLHandler::comInitDB(const String & payload)
String database = payload.substr(1); String database = payload.substr(1);
LOG_DEBUG(log, "Setting current database to " << database); LOG_DEBUG(log, "Setting current database to " << database);
connection_context.setCurrentDatabase(database); connection_context.setCurrentDatabase(database);
packet_sender->sendPacket(OK_Packet(0, capabilities, 0, 0, 1), true); packet_sender->sendPacket(OK_Packet(0, client_capability_flags, 0, 0, 1), true);
} }
void MySQLHandler::comFieldList(const String & payload) void MySQLHandler::comFieldList(const String & payload)
@ -340,12 +339,12 @@ void MySQLHandler::comFieldList(const String & payload)
); );
packet_sender->sendPacket(column_definition); packet_sender->sendPacket(column_definition);
} }
packet_sender->sendPacket(OK_Packet(0xfe, capabilities, 0, 0, 0), true); packet_sender->sendPacket(OK_Packet(0xfe, client_capability_flags, 0, 0, 0), true);
} }
void MySQLHandler::comPing() void MySQLHandler::comPing()
{ {
packet_sender->sendPacket(OK_Packet(0x0, capabilities, 0, 0, 0), true); packet_sender->sendPacket(OK_Packet(0x0, client_capability_flags, 0, 0, 0), true);
} }
void MySQLHandler::comQuery(const String & payload) void MySQLHandler::comQuery(const String & payload)
@ -354,10 +353,18 @@ void MySQLHandler::comQuery(const String & payload)
std::function<void(const String &)> set_content_type = [&with_output](const String &) -> void { std::function<void(const String &)> set_content_type = [&with_output](const String &) -> void {
with_output = true; with_output = true;
}; };
ReadBufferFromMemory query(payload.data() + 1, payload.size() - 1);
executeQuery(query, *out, true, connection_context, set_content_type, nullptr); String query = payload.substr(1);
// Translate query from MySQL to ClickHouse.
// This is a temporary workaround until ClickHouse supports the syntax "@@var_name".
if (query == "select @@version_comment limit 1") // MariaDB client starts session with that query
query = "select ''";
ReadBufferFromString buf(query);
executeQuery(buf, *out, true, connection_context, set_content_type, nullptr);
if (!with_output) if (!with_output)
packet_sender->sendPacket(OK_Packet(0x00, capabilities, 0, 0, 0), true); packet_sender->sendPacket(OK_Packet(0x00, client_capability_flags, 0, 0, 0), true);
} }
} }

28
dbms/programs/server/MySQLHandler.h
View File

@ -4,7 +4,7 @@
#include <Poco/Net/SecureStreamSocket.h> #include <Poco/Net/SecureStreamSocket.h>
#include <Common/getFQDNOrHostName.h> #include <Common/getFQDNOrHostName.h>
#include <Core/MySQLProtocol.h> #include <Core/MySQLProtocol.h>
#include <openssl/evp.h> #include <openssl/rsa.h>
#include "IServer.h" #include "IServer.h"
@ -15,21 +15,7 @@ namespace DB
class MySQLHandler : public Poco::Net::TCPServerConnection class MySQLHandler : public Poco::Net::TCPServerConnection
{ {
public: public:
MySQLHandler( MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key, RSA & private_key, bool ssl_enabled, size_t connection_id);
IServer & server_,
const Poco::Net::StreamSocket & socket_,
RSA * public_key,
RSA * private_key)
: Poco::Net::TCPServerConnection(socket_)
, server(server_)
, log(&Poco::Logger::get("MySQLHandler"))
, connection_context(server.context())
, connection_id(last_connection_id++)
, public_key(public_key)
, private_key(private_key)
{
log->setLevel("information");
}
void run() final; void run() final;
@ -55,13 +41,13 @@ private:
std::shared_ptr<MySQLProtocol::PacketSender> packet_sender; std::shared_ptr<MySQLProtocol::PacketSender> packet_sender;
uint32_t connection_id = 0; size_t connection_id = 0;
uint32_t capabilities; size_t server_capability_flags;
size_t client_capability_flags;
static uint32_t last_connection_id; RSA & public_key;
RSA & private_key;
RSA * public_key, * private_key;
std::shared_ptr<ReadBuffer> in; std::shared_ptr<ReadBuffer> in;
std::shared_ptr<WriteBuffer> out; std::shared_ptr<WriteBuffer> out;

124
dbms/programs/server/MySQLHandlerFactory.cpp Normal file
View File

@ -0,0 +1,124 @@
#include <Common/OpenSSLHelpers.h>
#include <Poco/Crypto/X509Certificate.h>
#include <Poco/Net/SSLManager.h>
#include <Poco/Net/TCPServerConnectionFactory.h>
#include <Poco/Util/Application.h>
#include <common/logger_useful.h>
#include <ext/scope_guard.h>
#include "IServer.h"
#include "MySQLHandler.h"
#include "MySQLHandlerFactory.h"
namespace DB
{
namespace ErrorCodes
{
extern const int CANNOT_OPEN_FILE;
extern const int NO_ELEMENTS_IN_CONFIG;
extern const int OPENSSL_ERROR;
extern const int SYSTEM_ERROR;
}
MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
: server(server_)
, log(&Logger::get("MySQLHandlerFactory"))
{
try
{
Poco::Net::SSLManager::instance().defaultServerContext();
}
catch (...)
{
LOG_INFO(log, "Failed to create SSL context. SSL will be disabled. Error: " << getCurrentExceptionMessage(false));
ssl_enabled = false;
}
/// Reading rsa keys for SHA256 authentication plugin.
try
{
readRSAKeys();
}
catch (...)
{
LOG_WARNING(log, "Failed to read RSA keys. Error: " << getCurrentExceptionMessage(false));
generateRSAKeys();
}
}
void MySQLHandlerFactory::readRSAKeys()
{
const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
String certificateFileProperty = "openSSL.server.certificateFile";
String privateKeyFileProperty = "openSSL.server.privateKeyFile";
if (!config.has(certificateFileProperty))
throw Exception("Certificate file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
if (!config.has(privateKeyFileProperty))
throw Exception("Private key file is not set.", ErrorCodes::NO_ELEMENTS_IN_CONFIG);
{
String certificateFile = config.getString(certificateFileProperty);
FILE * fp = fopen(certificateFile.data(), "r");
if (fp == nullptr)
throw Exception("Cannot open certificate file: " + certificateFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
SCOPE_EXIT(fclose(fp));
X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
SCOPE_EXIT(X509_free(x509));
if (x509 == nullptr)
throw Exception("Failed to read PEM certificate from " + certificateFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
EVP_PKEY * p = X509_get_pubkey(x509);
if (p == nullptr)
throw Exception("Failed to get RSA key from X509. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
SCOPE_EXIT(EVP_PKEY_free(p));
public_key.reset(EVP_PKEY_get1_RSA(p));
if (public_key.get() == nullptr)
throw Exception("Failed to get RSA key from ENV_PKEY. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
}
{
String privateKeyFile = config.getString(privateKeyFileProperty);
FILE * fp = fopen(privateKeyFile.data(), "r");
if (fp == nullptr)
throw Exception ("Cannot open private key file " + privateKeyFile + ".", ErrorCodes::CANNOT_OPEN_FILE);
SCOPE_EXIT(fclose(fp));
private_key.reset(PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr));
if (!private_key)
throw Exception("Failed to read RSA private key from " + privateKeyFile + ". Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
}
}
void MySQLHandlerFactory::generateRSAKeys()
{
LOG_INFO(log, "Generating new RSA key.");
public_key.reset(RSA_new());
if (!public_key)
throw Exception("Failed to allocate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
BIGNUM * e = BN_new();
if (!e)
throw Exception("Failed to allocate BIGNUM. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
SCOPE_EXIT(BN_free(e));
if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(public_key.get(), 2048, e, nullptr))
throw Exception("Failed to generate RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
private_key.reset(RSAPrivateKey_dup(public_key.get()));
if (!private_key)
throw Exception("Failed to copy RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
}
Poco::Net::TCPServerConnection * MySQLHandlerFactory::createConnection(const Poco::Net::StreamSocket & socket)
{
size_t connection_id = last_connection_id++;
LOG_TRACE(log, "MySQL connection. Id: " << connection_id << ". Address: " << socket.peerAddress().toString());
return new MySQLHandler(server, socket, *public_key, *private_key, ssl_enabled, connection_id);
}
}

110
dbms/programs/server/MySQLHandlerFactory.h
View File

@ -1,16 +1,9 @@
#pragma once #pragma once
#include <Poco/Net/TCPServerConnectionFactory.h> #include <Poco/Net/TCPServerConnectionFactory.h>
#include <Poco/Net/SSLManager.h> #include <atomic>
#include <Poco/Crypto/X509Certificate.h> #include <openssl/rsa.h>
#include <common/logger_useful.h>
#include "IServer.h" #include "IServer.h"
#include "MySQLHandler.h"
namespace Poco
{
class Logger;
}
namespace DB namespace DB
{ {
@ -20,94 +13,27 @@ class MySQLHandlerFactory : public Poco::Net::TCPServerConnectionFactory
private: private:
IServer & server; IServer & server;
Poco::Logger * log; Poco::Logger * log;
RSA * public_key = nullptr, * private_key = nullptr;
struct RSADeleter
{
void operator()(RSA * ptr) { RSA_free(ptr); }
};
using RSAPtr = std::unique_ptr<RSA, RSADeleter>;
RSAPtr public_key;
RSAPtr private_key;
bool ssl_enabled = true;
std::atomic<size_t> last_connection_id = 0;
public: public:
explicit MySQLHandlerFactory(IServer & server_) explicit MySQLHandlerFactory(IServer & server_);
: server(server_), log(&Logger::get("MySQLHandlerFactory"))
{
/// Reading rsa keys for SHA256 authentication plugin.
const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
String certificateFileProperty = "openSSL.server.certificateFile";
String privateKeyFileProperty = "openSSL.server.privateKeyFile";
if (!config.has(certificateFileProperty)) void readRSAKeys();
{
LOG_INFO(log, "Certificate file is not set.");
generateRSAKeys();
return;
}
if (!config.has(privateKeyFileProperty))
{
LOG_INFO(log, "Private key file is not set.");
generateRSAKeys();
return;
}
String certificateFile = config.getString(certificateFileProperty); void generateRSAKeys();
FILE * fp = fopen(certificateFile.data(), "r");
if (fp == nullptr)
{
LOG_WARNING(log, "Cannot open certificate file: " << certificateFile << ".");
generateRSAKeys();
return;
}
X509 * x509 = PEM_read_X509(fp, nullptr, nullptr, nullptr);
EVP_PKEY * p = X509_get_pubkey(x509);
public_key = EVP_PKEY_get1_RSA(p);
X509_free(x509);
EVP_PKEY_free(p);
fclose(fp);
String privateKeyFile = config.getString(privateKeyFileProperty); Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override;
fp = fopen(privateKeyFile.data(), "r");
if (fp == nullptr)
{
LOG_WARNING(log, "Cannot open private key file " << privateKeyFile << ".");
generateRSAKeys();
return;
}
private_key = PEM_read_RSAPrivateKey(fp, nullptr, nullptr, nullptr);
fclose(fp);
}
void generateRSAKeys()
{
LOG_INFO(log, "Generating new RSA key.");
RSA * rsa = RSA_new();
if (rsa == nullptr)
{
throw Exception("Failed to allocate RSA key.", 1002);
}
BIGNUM * e = BN_new();
if (!e)
{
RSA_free(rsa);
throw Exception("Failed to allocate BIGNUM.", 1002);
}
if (!BN_set_word(e, 65537) || !RSA_generate_key_ex(rsa, 2048, e, nullptr))
{
RSA_free(rsa);
BN_free(e);
throw Exception("Failed to generate RSA key.", 1002);
}
BN_free(e);
public_key = rsa;
private_key = RSAPrivateKey_dup(rsa);
}
Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override
{
LOG_TRACE(log, "MySQL connection. Address: " << socket.peerAddress().toString());
return new MySQLHandler(server, socket, public_key, private_key);
}
~MySQLHandlerFactory() override
{
RSA_free(public_key);
RSA_free(private_key);
}
}; };
} }

4
dbms/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h
View File

@ -1,10 +1,10 @@
#pragma once #pragma once
#include <roaring.h> #include <roaring/roaring.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <boost/noncopyable.hpp> #include <boost/noncopyable.hpp>
#include <roaring.hh> #include <roaring/roaring.hh>
#include <Common/HashTable/SmallTable.h> #include <Common/HashTable/SmallTable.h>
#include <Common/PODArray.h> #include <Common/PODArray.h>

153
dbms/src/AggregateFunctions/AggregateFunctionMLMethod.cpp
View File

@ -3,6 +3,8 @@
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <Interpreters/castColumn.h> #include <Interpreters/castColumn.h>
#include <Columns/ColumnArray.h>
#include <Columns/ColumnTuple.h>
#include <Common/FieldVisitors.h> #include <Common/FieldVisitors.h>
#include <Common/typeid_cast.h> #include <Common/typeid_cast.h>
#include "AggregateFunctionFactory.h" #include "AggregateFunctionFactory.h"
@ -44,10 +46,10 @@ namespace
/// Such default parameters were picked because they did good on some tests, /// Such default parameters were picked because they did good on some tests,
/// though it still requires to fit parameters to achieve better result /// though it still requires to fit parameters to achieve better result
auto learning_rate = Float64(0.01); auto learning_rate = Float64(0.01);
auto l2_reg_coef = Float64(0.01); auto l2_reg_coef = Float64(0.1);
UInt32 batch_size = 1; UInt32 batch_size = 15;
std::shared_ptr<IWeightsUpdater> weights_updater = std::make_shared<StochasticGradientDescent>(); std::string weights_updater_name = "\'SGD\'";
std::shared_ptr<IGradientComputer> gradient_computer; std::shared_ptr<IGradientComputer> gradient_computer;
if (!parameters.empty()) if (!parameters.empty())
@ -64,19 +66,8 @@ namespace
} }
if (parameters.size() > 3) if (parameters.size() > 3)
{ {
if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'SGD\'") weights_updater_name = applyVisitor(FieldVisitorToString(), parameters[3]);
{ if (weights_updater_name != "\'SGD\'" && weights_updater_name != "\'Momentum\'" && weights_updater_name != "\'Nesterov\'")
weights_updater = std::make_shared<StochasticGradientDescent>();
}
else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Momentum\'")
{
weights_updater = std::make_shared<Momentum>();
}
else if (applyVisitor(FieldVisitorToString(), parameters[3]) == "\'Nesterov\'")
{
weights_updater = std::make_shared<Nesterov>();
}
else
{ {
throw Exception("Invalid parameter for weights updater", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); throw Exception("Invalid parameter for weights updater", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
} }
@ -98,20 +89,19 @@ namespace
return std::make_shared<Method>( return std::make_shared<Method>(
argument_types.size() - 1, argument_types.size() - 1,
gradient_computer, gradient_computer,
weights_updater, weights_updater_name,
learning_rate, learning_rate,
l2_reg_coef, l2_reg_coef,
batch_size, batch_size,
argument_types, argument_types,
parameters); parameters);
} }
} }
void registerAggregateFunctionMLMethod(AggregateFunctionFactory & factory) void registerAggregateFunctionMLMethod(AggregateFunctionFactory & factory)
{ {
factory.registerFunction("linearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>); factory.registerFunction("stochasticLinearRegression", createAggregateFunctionMLMethod<FuncLinearRegression>);
factory.registerFunction("logisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>); factory.registerFunction("stochasticLogisticRegression", createAggregateFunctionMLMethod<FuncLogisticRegression>);
} }
LinearModelData::LinearModelData( LinearModelData::LinearModelData(
@ -144,9 +134,34 @@ void LinearModelData::update_state()
} }
void LinearModelData::predict( void LinearModelData::predict(
ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const ColumnVector<Float64>::Container & container,
Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments,
const Context & context) const
{ {
gradient_computer->predict(container, block, arguments, weights, bias, context); gradient_computer->predict(container, block, offset, limit, arguments, weights, bias, context);
}
void LinearModelData::returnWeights(IColumn & to) const
{
size_t size = weights.size() + 1;
ColumnArray & arr_to = static_cast<ColumnArray &>(to);
ColumnArray::Offsets & offsets_to = arr_to.getOffsets();
size_t old_size = offsets_to.back();
offsets_to.push_back(old_size + size);
typename ColumnFloat64::Container & val_to
= static_cast<ColumnFloat64 &>(arr_to.getData()).getData();
val_to.reserve(old_size + size);
for (size_t i = 0; i + 1 < size; ++i)
val_to.push_back(weights[i]);
val_to.push_back(bias);
} }
void LinearModelData::read(ReadBuffer & buf) void LinearModelData::read(ReadBuffer & buf)
@ -192,7 +207,8 @@ void LinearModelData::merge(const DB::LinearModelData & rhs)
void LinearModelData::add(const IColumn ** columns, size_t row_num) void LinearModelData::add(const IColumn ** columns, size_t row_num)
{ {
/// first column stores target; features start from (columns + 1) /// first column stores target; features start from (columns + 1)
const auto target = (*columns[0])[row_num].get<Float64>(); Float64 target = (*columns[0]).getFloat64(row_num);
/// Here we have columns + 1 as first column corresponds to target value, and others - to features /// Here we have columns + 1 as first column corresponds to target value, and others - to features
weights_updater->add_to_batch( weights_updater->add_to_batch(
gradient_batch, *gradient_computer, weights, bias, learning_rate, l2_reg_coef, target, columns + 1, row_num); gradient_batch, *gradient_computer, weights, bias, learning_rate, l2_reg_coef, target, columns + 1, row_num);
@ -334,43 +350,39 @@ void IWeightsUpdater::add_to_batch(
void LogisticRegression::predict( void LogisticRegression::predict(
ColumnVector<Float64>::Container & container, ColumnVector<Float64>::Container & container,
Block & block, Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments, const ColumnNumbers & arguments,
const std::vector<Float64> & weights, const std::vector<Float64> & weights,
Float64 bias, Float64 bias,
const Context & context) const const Context & /*context*/) const
{ {
size_t rows_num = block.rows(); size_t rows_num = block.rows();
std::vector<Float64> results(rows_num, bias);
if (offset > rows_num || offset + limit > rows_num)
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
std::vector<Float64> results(limit, bias);
for (size_t i = 1; i < arguments.size(); ++i) for (size_t i = 1; i < arguments.size(); ++i)
{ {
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]); const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
if (!isNativeNumber(cur_col.type)) if (!isNativeNumber(cur_col.type))
{
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS); throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
auto & features_column = cur_col.column;
for (size_t row_num = 0; row_num < limit; ++row_num)
results[row_num] += weights[i - 1] * features_column->getFloat64(offset + row_num);
} }
/// If column type is already Float64 then castColumn simply returns it container.reserve(container.size() + limit);
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context); for (size_t row_num = 0; row_num < limit; ++row_num)
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
if (!features_column)
{
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
}
for (size_t row_num = 0; row_num != rows_num; ++row_num)
{
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
}
}
container.reserve(rows_num);
for (size_t row_num = 0; row_num != rows_num; ++row_num)
{
container.emplace_back(1 / (1 + exp(-results[row_num]))); container.emplace_back(1 / (1 + exp(-results[row_num])));
} }
}
void LogisticRegression::compute( void LogisticRegression::compute(
std::vector<Float64> & batch_gradient, std::vector<Float64> & batch_gradient,
@ -385,7 +397,7 @@ void LogisticRegression::compute(
Float64 derivative = bias; Float64 derivative = bias;
for (size_t i = 0; i < weights.size(); ++i) for (size_t i = 0; i < weights.size(); ++i)
{ {
auto value = (*columns[i])[row_num].get<Float64>(); auto value = (*columns[i]).getFloat64(row_num);
derivative += weights[i] * value; derivative += weights[i] * value;
} }
derivative *= target; derivative *= target;
@ -394,18 +406,20 @@ void LogisticRegression::compute(
batch_gradient[weights.size()] += learning_rate * target / (derivative + 1); batch_gradient[weights.size()] += learning_rate * target / (derivative + 1);
for (size_t i = 0; i < weights.size(); ++i) for (size_t i = 0; i < weights.size(); ++i)
{ {
auto value = (*columns[i])[row_num].get<Float64>(); auto value = (*columns[i]).getFloat64(row_num);
batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * l2_reg_coef * weights[i]; batch_gradient[i] += learning_rate * target * value / (derivative + 1) - 2 * learning_rate * l2_reg_coef * weights[i];
} }
} }
void LinearRegression::predict( void LinearRegression::predict(
ColumnVector<Float64>::Container & container, ColumnVector<Float64>::Container & container,
Block & block, Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments, const ColumnNumbers & arguments,
const std::vector<Float64> & weights, const std::vector<Float64> & weights,
Float64 bias, Float64 bias,
const Context & context) const const Context & /*context*/) const
{ {
if (weights.size() + 1 != arguments.size()) if (weights.size() + 1 != arguments.size())
{ {
@ -413,37 +427,34 @@ void LinearRegression::predict(
} }
size_t rows_num = block.rows(); size_t rows_num = block.rows();
std::vector<Float64> results(rows_num, bias);
if (offset > rows_num || offset + limit > rows_num)
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
std::vector<Float64> results(limit, bias);
for (size_t i = 1; i < arguments.size(); ++i) for (size_t i = 1; i < arguments.size(); ++i)
{ {
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]); const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
if (!isNativeNumber(cur_col.type))
{
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
}
/// If column type is already Float64 then castColumn simply returns it if (!isNativeNumber(cur_col.type))
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context); throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
auto features_column = cur_col.column;
if (!features_column) if (!features_column)
{
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR); throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
for (size_t row_num = 0; row_num < limit; ++row_num)
results[row_num] += weights[i - 1] * features_column->getFloat64(row_num + offset);
} }
for (size_t row_num = 0; row_num != rows_num; ++row_num) container.reserve(container.size() + limit);
{ for (size_t row_num = 0; row_num < limit; ++row_num)
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
}
}
container.reserve(rows_num);
for (size_t row_num = 0; row_num != rows_num; ++row_num)
{
container.emplace_back(results[row_num]); container.emplace_back(results[row_num]);
} }
}
void LinearRegression::compute( void LinearRegression::compute(
std::vector<Float64> & batch_gradient, std::vector<Float64> & batch_gradient,
@ -458,7 +469,7 @@ void LinearRegression::compute(
Float64 derivative = (target - bias); Float64 derivative = (target - bias);
for (size_t i = 0; i < weights.size(); ++i) for (size_t i = 0; i < weights.size(); ++i)
{ {
auto value = (*columns[i])[row_num].get<Float64>(); auto value = (*columns[i]).getFloat64(row_num);
derivative -= weights[i] * value; derivative -= weights[i] * value;
} }
derivative *= (2 * learning_rate); derivative *= (2 * learning_rate);
@ -466,8 +477,8 @@ void LinearRegression::compute(
batch_gradient[weights.size()] += derivative; batch_gradient[weights.size()] += derivative;
for (size_t i = 0; i < weights.size(); ++i) for (size_t i = 0; i < weights.size(); ++i)
{ {
auto value = (*columns[i])[row_num].get<Float64>(); auto value = (*columns[i]).getFloat64(row_num);
batch_gradient[i] += derivative * value - 2 * l2_reg_coef * weights[i]; batch_gradient[i] += derivative * value - 2 * learning_rate * l2_reg_coef * weights[i];
} }
} }

87
dbms/src/AggregateFunctions/AggregateFunctionMLMethod.h
View File

@ -3,7 +3,10 @@
#include <Columns/ColumnVector.h> #include <Columns/ColumnVector.h>
#include <Columns/ColumnsCommon.h> #include <Columns/ColumnsCommon.h>
#include <Columns/ColumnsNumber.h> #include <Columns/ColumnsNumber.h>
#include <Common/typeid_cast.h>
#include <DataTypes/DataTypesNumber.h> #include <DataTypes/DataTypesNumber.h>
#include <DataTypes/DataTypeTuple.h>
#include <DataTypes/DataTypeArray.h>
#include "IAggregateFunction.h" #include "IAggregateFunction.h"
namespace DB namespace DB
@ -12,6 +15,7 @@ namespace ErrorCodes
{ {
extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
extern const int BAD_ARGUMENTS; extern const int BAD_ARGUMENTS;
extern const int BAD_CAST;
} }
/** /**
@ -39,6 +43,8 @@ public:
virtual void predict( virtual void predict(
ColumnVector<Float64>::Container & container, ColumnVector<Float64>::Container & container,
Block & block, Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments, const ColumnNumbers & arguments,
const std::vector<Float64> & weights, const std::vector<Float64> & weights,
Float64 bias, Float64 bias,
@ -64,6 +70,8 @@ public:
void predict( void predict(
ColumnVector<Float64>::Container & container, ColumnVector<Float64>::Container & container,
Block & block, Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments, const ColumnNumbers & arguments,
const std::vector<Float64> & weights, const std::vector<Float64> & weights,
Float64 bias, Float64 bias,
@ -89,6 +97,8 @@ public:
void predict( void predict(
ColumnVector<Float64>::Container & container, ColumnVector<Float64>::Container & container,
Block & block, Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments, const ColumnNumbers & arguments,
const std::vector<Float64> & weights, const std::vector<Float64> & weights,
Float64 bias, Float64 bias,
@ -215,20 +225,26 @@ public:
void read(ReadBuffer & buf); void read(ReadBuffer & buf);
void void predict(
predict(ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const; ColumnVector<Float64>::Container & container,
Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments,
const Context & context) const;
void returnWeights(IColumn & to) const;
private: private:
std::vector<Float64> weights; std::vector<Float64> weights;
Float64 bias{0.0}; Float64 bias{0.0};
Float64 learning_rate; Float64 learning_rate;
Float64 l2_reg_coef; Float64 l2_reg_coef;
UInt32 batch_capacity; UInt64 batch_capacity;
UInt32 iter_num = 0; UInt64 iter_num = 0;
std::vector<Float64> gradient_batch; std::vector<Float64> gradient_batch;
UInt32 batch_size; UInt64 batch_size;
std::shared_ptr<IGradientComputer> gradient_computer; std::shared_ptr<IGradientComputer> gradient_computer;
std::shared_ptr<IWeightsUpdater> weights_updater; std::shared_ptr<IWeightsUpdater> weights_updater;
@ -253,7 +269,7 @@ public:
explicit AggregateFunctionMLMethod( explicit AggregateFunctionMLMethod(
UInt32 param_num, UInt32 param_num,
std::shared_ptr<IGradientComputer> gradient_computer, std::shared_ptr<IGradientComputer> gradient_computer,
std::shared_ptr<IWeightsUpdater> weights_updater, std::string weights_updater_name,
Float64 learning_rate, Float64 learning_rate,
Float64 l2_reg_coef, Float64 l2_reg_coef,
UInt32 batch_size, UInt32 batch_size,
@ -265,15 +281,39 @@ public:
, l2_reg_coef(l2_reg_coef) , l2_reg_coef(l2_reg_coef)
, batch_size(batch_size) , batch_size(batch_size)
, gradient_computer(std::move(gradient_computer)) , gradient_computer(std::move(gradient_computer))
, weights_updater(std::move(weights_updater)) , weights_updater_name(std::move(weights_updater_name))
{ {
} }
DataTypePtr getReturnType() const override { return std::make_shared<DataTypeNumber<Float64>>(); } /// This function is called when SELECT linearRegression(...) is called
DataTypePtr getReturnType() const override
{
return std::make_shared<DataTypeArray>(std::make_shared<DataTypeFloat64>());
}
/// This function is called from evalMLMethod function for correct predictValues call
DataTypePtr getReturnTypeToPredict() const override
{
return std::make_shared<DataTypeNumber<Float64>>();
}
void create(AggregateDataPtr place) const override void create(AggregateDataPtr place) const override
{ {
new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, weights_updater); std::shared_ptr<IWeightsUpdater> new_weights_updater;
if (weights_updater_name == "\'SGD\'")
{
new_weights_updater = std::make_shared<StochasticGradientDescent>();
} else if (weights_updater_name == "\'Momentum\'")
{
new_weights_updater = std::make_shared<Momentum>();
} else if (weights_updater_name == "\'Nesterov\'")
{
new_weights_updater = std::make_shared<Nesterov>();
} else
{
throw Exception("Illegal name of weights updater (should have been checked earlier)", ErrorCodes::LOGICAL_ERROR);
}
new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, new_weights_updater);
} }
void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
@ -288,7 +328,13 @@ public:
void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override { this->data(place).read(buf); } void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override { this->data(place).read(buf); }
void predictValues( void predictValues(
ConstAggregateDataPtr place, IColumn & to, Block & block, const ColumnNumbers & arguments, const Context & context) const override ConstAggregateDataPtr place,
IColumn & to,
Block & block,
size_t offset,
size_t limit,
const ColumnNumbers & arguments,
const Context & context) const override
{ {
if (arguments.size() != param_num + 1) if (arguments.size() != param_num + 1)
throw Exception( throw Exception(
@ -296,16 +342,21 @@ public:
+ ". Required: " + std::to_string(param_num + 1), + ". Required: " + std::to_string(param_num + 1),
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
auto & column = dynamic_cast<ColumnVector<Float64> &>(to); /// This cast might be correct because column type is based on getReturnTypeToPredict.
auto * column = typeid_cast<ColumnFloat64 *>(&to);
if (!column)
throw Exception("Cast of column of predictions is incorrect. getReturnTypeToPredict must return same value as it is casted to",
ErrorCodes::BAD_CAST);
this->data(place).predict(column.getData(), block, arguments, context); this->data(place).predict(column->getData(), block, offset, limit, arguments, context);
} }
/** This function is called if aggregate function without State modifier is selected in a query.
* Inserts all weights of the model into the column 'to', so user may use such information if needed
*/
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
{ {
std::ignore = place; this->data(place).returnWeights(to);
std::ignore = to;
throw std::runtime_error("not implemented");
} }
const char * getHeaderFilePath() const override { return __FILE__; } const char * getHeaderFilePath() const override { return __FILE__; }
@ -316,15 +367,15 @@ private:
Float64 l2_reg_coef; Float64 l2_reg_coef;
UInt32 batch_size; UInt32 batch_size;
std::shared_ptr<IGradientComputer> gradient_computer; std::shared_ptr<IGradientComputer> gradient_computer;
std::shared_ptr<IWeightsUpdater> weights_updater; std::string weights_updater_name;
}; };
struct NameLinearRegression struct NameLinearRegression
{ {
static constexpr auto name = "linearRegression"; static constexpr auto name = "stochasticLinearRegression";
}; };
struct NameLogisticRegression struct NameLogisticRegression
{ {
static constexpr auto name = "logisticRegression"; static constexpr auto name = "stochasticLogisticRegression";
}; };
} }

19
dbms/src/AggregateFunctions/IAggregateFunction.h
View File

@ -48,6 +48,12 @@ public:
/// Get the result type. /// Get the result type.
virtual DataTypePtr getReturnType() const = 0; virtual DataTypePtr getReturnType() const = 0;
/// Get type which will be used for prediction result in case if function is an ML method.
virtual DataTypePtr getReturnTypeToPredict() const
{
throw Exception("Prediction is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
}
virtual ~IAggregateFunction() {} virtual ~IAggregateFunction() {}
/** Data manipulating functions. */ /** Data manipulating functions. */
@ -94,9 +100,16 @@ public:
/// Inserts results into a column. /// Inserts results into a column.
virtual void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const = 0; virtual void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const = 0;
/// This function is used for machine learning methods /// Used for machine learning methods. Predict result from trained model.
virtual void predictValues(ConstAggregateDataPtr /* place */, IColumn & /*to*/, /// Will insert result into `to` column for rows in range [offset, offset + limit).
Block & /*block*/, const ColumnNumbers & /*arguments*/, const Context & /*context*/) const virtual void predictValues(
ConstAggregateDataPtr /* place */,
IColumn & /*to*/,
Block & /*block*/,
size_t /*offset*/,
size_t /*limit*/,
const ColumnNumbers & /*arguments*/,
const Context & /*context*/) const
{ {
throw Exception("Method predictValues is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED); throw Exception("Method predictValues is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
} }

49
dbms/src/Columns/ColumnAggregateFunction.cpp
View File

@ -35,25 +35,6 @@ void ColumnAggregateFunction::addArena(ArenaPtr arena_)
arenas.push_back(arena_); arenas.push_back(arena_);
} }
/// This function is used in convertToValues() and predictValues()
/// and is written here to avoid repetitions
bool ColumnAggregateFunction::tryFinalizeAggregateFunction(MutableColumnPtr *res_) const
{
if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
{
auto res = createView();
res->set(function_state->getNestedFunction());
res->data.assign(data.begin(), data.end());
*res_ = std::move(res);
return true;
}
MutableColumnPtr res = func->getReturnType()->createColumn();
res->reserve(data.size());
*res_ = std::move(res);
return false;
}
MutableColumnPtr ColumnAggregateFunction::convertToValues() const MutableColumnPtr ColumnAggregateFunction::convertToValues() const
{ {
/** If the aggregate function returns an unfinalized/unfinished state, /** If the aggregate function returns an unfinalized/unfinished state,
@ -86,17 +67,17 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
* AggregateFunction(quantileTiming(0.5), UInt64) * AggregateFunction(quantileTiming(0.5), UInt64)
* into UInt16 - already finished result of `quantileTiming`. * into UInt16 - already finished result of `quantileTiming`.
*/ */
if (const AggregateFunctionState *function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
/** Convertion function is used in convertToValues and predictValues
* in the similar part of both functions
*/
MutableColumnPtr res;
if (tryFinalizeAggregateFunction(&res))
{ {
auto res = createView();
res->set(function_state->getNestedFunction());
res->data.assign(data.begin(), data.end());
return res; return res;
} }
MutableColumnPtr res = func->getReturnType()->createColumn();
res->reserve(data.size());
for (auto val : data) for (auto val : data)
func->insertResultInto(val, *res); func->insertResultInto(val, *res);
@ -105,19 +86,27 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
MutableColumnPtr ColumnAggregateFunction::predictValues(Block & block, const ColumnNumbers & arguments, const Context & context) const MutableColumnPtr ColumnAggregateFunction::predictValues(Block & block, const ColumnNumbers & arguments, const Context & context) const
{ {
MutableColumnPtr res; MutableColumnPtr res = func->getReturnTypeToPredict()->createColumn();
tryFinalizeAggregateFunction(&res); res->reserve(data.size());
auto ML_function = func.get(); auto ML_function = func.get();
if (ML_function) if (ML_function)
{ {
if (data.size() == 1)
{
/// Case for const column. Predict using single model.
ML_function->predictValues(data[0], *res, block, 0, block.rows(), arguments, context);
}
else
{
/// Case for non-constant column. Use different aggregate function for each row.
size_t row_num = 0; size_t row_num = 0;
for (auto val : data) for (auto val : data)
{ {
ML_function->predictValues(val, *res, block, arguments, context); ML_function->predictValues(val, *res, block, row_num, 1, arguments, context);
++row_num; ++row_num;
} }
}
} }
else else
{ {

8
dbms/src/Columns/ColumnDecimal.cpp
View File

@ -94,7 +94,7 @@ ColumnPtr ColumnDecimal<T>::permute(const IColumn::Permutation & perm, size_t li
for (size_t i = 0; i < size; ++i) for (size_t i = 0; i < size; ++i)
res_data[i] = data[perm[i]]; res_data[i] = data[perm[i]];
return std::move(res); return res;
} }
template <typename T> template <typename T>
@ -117,7 +117,7 @@ MutableColumnPtr ColumnDecimal<T>::cloneResized(size_t size) const
} }
} }
return std::move(res); return res;
} }
template <typename T> template <typename T>
@ -169,7 +169,7 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter & filt, ssize_t result_
++data_pos; ++data_pos;
} }
return std::move(res); return res;
} }
template <typename T> template <typename T>
@ -202,7 +202,7 @@ ColumnPtr ColumnDecimal<T>::replicate(const IColumn::Offsets & offsets) const
res_data.push_back(data[i]); res_data.push_back(data[i]);
} }
return std::move(res); return res;
} }
template <typename T> template <typename T>

2
dbms/src/Columns/ColumnDecimal.h
View File

@ -187,7 +187,7 @@ ColumnPtr ColumnDecimal<T>::indexImpl(const PaddedPODArray<Type> & indexes, size
for (size_t i = 0; i < limit; ++i) for (size_t i = 0; i < limit; ++i)
res_data[i] = data[indexes[i]]; res_data[i] = data[indexes[i]];
return std::move(res); return res;
} }
} }

2
dbms/src/Columns/ColumnLowCardinality.cpp
View File

@ -349,7 +349,7 @@ ColumnPtr ColumnLowCardinality::countKeys() const
auto counter = ColumnUInt64::create(dict_size, 0); auto counter = ColumnUInt64::create(dict_size, 0);
idx.countKeys(counter->getData()); idx.countKeys(counter->getData());
return std::move(counter); return counter;
} }

6
dbms/src/Columns/ColumnVector.cpp
View File

@ -215,6 +215,12 @@ UInt64 ColumnVector<T>::get64(size_t n) const
return ext::bit_cast<UInt64>(data[n]); return ext::bit_cast<UInt64>(data[n]);
} }
template <typename T>
Float64 ColumnVector<T>::getFloat64(size_t n) const
{
return static_cast<Float64>(data[n]);
}
template <typename T> template <typename T>
void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length) void ColumnVector<T>::insertRangeFrom(const IColumn & src, size_t start, size_t length)
{ {

2
dbms/src/Columns/ColumnVector.h
View File

@ -202,6 +202,8 @@ public:
UInt64 get64(size_t n) const override; UInt64 get64(size_t n) const override;
Float64 getFloat64(size_t n) const override;
UInt64 getUInt(size_t n) const override UInt64 getUInt(size_t n) const override
{ {
return UInt64(data[n]); return UInt64(data[n]);

7
dbms/src/Columns/IColumn.h
View File

@ -91,6 +91,13 @@ public:
throw Exception("Method get64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED); throw Exception("Method get64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
} }
/// If column stores native numeric type, it returns n-th element casted to Float64
/// Is used in regression methods to cast each features into uniform type
virtual Float64 getFloat64(size_t /*n*/) const
{
throw Exception("Method getFloat64 is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
}
/** If column is numeric, return value of n-th element, casted to UInt64. /** If column is numeric, return value of n-th element, casted to UInt64.
* For NULL values of Nullable column it is allowed to return arbitrary value. * For NULL values of Nullable column it is allowed to return arbitrary value.
* Otherwise throw an exception. * Otherwise throw an exception.

2
dbms/src/Columns/ReverseIndex.h
View File

@ -375,7 +375,7 @@ ColumnUInt64::MutablePtr ReverseIndex<IndexType, ColumnType>::calcHashes() const
for (auto row : ext::range(0, size)) for (auto row : ext::range(0, size))
hash->getElement(row) = getHash(column->getDataAt(row)); hash->getElement(row) = getHash(column->getDataAt(row));
return std::move(hash); return hash;
} }
template <typename IndexType, typename ColumnType> template <typename IndexType, typename ColumnType>

67
dbms/src/Columns/getLeastSuperColumn.cpp Normal file
View File

@ -0,0 +1,67 @@
#include <Columns/getLeastSuperColumn.h>
#include <Columns/IColumn.h>
#include <Columns/ColumnConst.h>
#include <DataTypes/getLeastSupertype.h>
namespace DB
{
namespace ErrorCodes
{
extern const int LOGICAL_ERROR;
}
static bool sameConstants(const IColumn & a, const IColumn & b)
{
return static_cast<const ColumnConst &>(a).getField() == static_cast<const ColumnConst &>(b).getField();
}
ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns)
{
if (columns.empty())
throw Exception("Logical error: no src columns for supercolumn", ErrorCodes::LOGICAL_ERROR);
ColumnWithTypeAndName result = *columns[0];
/// Determine common type.
size_t num_const = 0;
DataTypes types(columns.size());
for (size_t i = 0; i < columns.size(); ++i)
{
types[i] = columns[i]->type;
if (columns[i]->column->isColumnConst())
++num_const;
}
result.type = getLeastSupertype(types);
/// Create supertype column saving constness if possible.
bool save_constness = false;
if (columns.size() == num_const)
{
save_constness = true;
for (size_t i = 1; i < columns.size(); ++i)
{
const ColumnWithTypeAndName & first = *columns[0];
const ColumnWithTypeAndName & other = *columns[i];
if (!sameConstants(*first.column, *other.column))
{
save_constness = false;
break;
}
}
}
if (save_constness)
result.column = result.type->createColumnConst(0, static_cast<const ColumnConst &>(*columns[0]->column).getField());
else
result.column = result.type->createColumn();
return result;
}
}

12
dbms/src/Columns/getLeastSuperColumn.h Normal file
View File

@ -0,0 +1,12 @@
#pragma once
#include <Core/ColumnWithTypeAndName.h>
namespace DB
{
/// getLeastSupertype + related column changes
ColumnWithTypeAndName getLeastSuperColumn(std::vector<const ColumnWithTypeAndName *> columns);
}

8
dbms/src/Common/ErrorCodes.cpp
View File

@ -427,8 +427,10 @@ namespace ErrorCodes
extern const int BAD_TTL_EXPRESSION = 450; extern const int BAD_TTL_EXPRESSION = 450;
extern const int BAD_TTL_FILE = 451; extern const int BAD_TTL_FILE = 451;
extern const int SETTING_CONSTRAINT_VIOLATION = 452; extern const int SETTING_CONSTRAINT_VIOLATION = 452;
extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 453; extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 453;
extern const int OPENSSL_ERROR = 454;
extern const int SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = 455;
extern const int QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = 456;
extern const int KEEPER_EXCEPTION = 999; extern const int KEEPER_EXCEPTION = 999;
extern const int POCO_EXCEPTION = 1000; extern const int POCO_EXCEPTION = 1000;
extern const int STD_EXCEPTION = 1001; extern const int STD_EXCEPTION = 1001;
@ -437,8 +439,6 @@ namespace ErrorCodes
extern const int CONDITIONAL_TREE_PARENT_NOT_FOUND = 2001; extern const int CONDITIONAL_TREE_PARENT_NOT_FOUND = 2001;
extern const int ILLEGAL_PROJECTION_MANIPULATOR = 2002; extern const int ILLEGAL_PROJECTION_MANIPULATOR = 2002;
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = 446;
} }
} }

18
dbms/src/Common/OpenSSLHelpers.cpp Normal file
View File

@ -0,0 +1,18 @@
#include "OpenSSLHelpers.h"
#include <ext/scope_guard.h>
#include <openssl/err.h>
namespace DB
{
String getOpenSSLErrors()
{
BIO * mem = BIO_new(BIO_s_mem());
SCOPE_EXIT(BIO_free(mem));
ERR_print_errors(mem);
char * buf = nullptr;
long size = BIO_get_mem_data(mem, &buf);
return String(buf, size);
}
}

12
dbms/src/Common/OpenSSLHelpers.h Normal file
View File

@ -0,0 +1,12 @@
#pragma once
#include <Core/Types.h>
namespace DB
{
/// Returns concatenation of error strings for all errors that OpenSSL has recorded, emptying the error queue.
String getOpenSSLErrors();
}

2
dbms/src/Common/OptimizedRegularExpression.h
View File

@ -6,7 +6,7 @@
#include <Common/config.h> #include <Common/config.h>
#include <re2/re2.h> #include <re2/re2.h>
#if USE_RE2_ST #if USE_RE2_ST
#include <re2_st/re2.h> // Y_IGNORE #include <re2_st/re2.h>
#else #else
#define re2_st re2 #define re2_st re2
#endif #endif

2
dbms/src/Common/UInt128.h
View File

@ -71,7 +71,7 @@ template <typename T> bool inline operator> (T a, const UInt128 b) { return UIn
template <typename T> bool inline operator<= (T a, const UInt128 b) { return UInt128(a) <= b; } template <typename T> bool inline operator<= (T a, const UInt128 b) { return UInt128(a) <= b; }
template <typename T> bool inline operator< (T a, const UInt128 b) { return UInt128(a) < b; } template <typename T> bool inline operator< (T a, const UInt128 b) { return UInt128(a) < b; }
template <> constexpr bool IsNumber<UInt128> = true; template <> inline constexpr bool IsNumber<UInt128> = true;
template <> struct TypeName<UInt128> { static const char * get() { return "UInt128"; } }; template <> struct TypeName<UInt128> { static const char * get() { return "UInt128"; } };
template <> struct TypeId<UInt128> { static constexpr const TypeIndex value = TypeIndex::UInt128; }; template <> struct TypeId<UInt128> { static constexpr const TypeIndex value = TypeIndex::UInt128; };

2
dbms/src/Common/getNumberOfPhysicalCPUCores.cpp
View File

@ -7,7 +7,7 @@
# include <Common/Exception.h> # include <Common/Exception.h>
namespace DB { namespace ErrorCodes { extern const int CPUID_ERROR; }} namespace DB { namespace ErrorCodes { extern const int CPUID_ERROR; }}
#elif USE_CPUINFO #elif USE_CPUINFO
# include <cpuinfo.h> // Y_IGNORE # include <cpuinfo.h>
#endif #endif

4
dbms/src/Common/localBackup.cpp
View File

@ -1,7 +1,9 @@
#include <Common/localBackup.h> #include "localBackup.h"
#include <Common/createHardLink.h> #include <Common/createHardLink.h>
#include <Common/Exception.h> #include <Common/Exception.h>
#include <Poco/DirectoryIterator.h> #include <Poco/DirectoryIterator.h>
#include <Poco/Path.h>
#include <Poco/File.h> #include <Poco/File.h>
#include <string> #include <string>
#include <iostream> #include <iostream>

2
dbms/src/Common/localBackup.h
View File

@ -1,8 +1,8 @@
#pragma once #pragma once
#include <Poco/Path.h>
#include <optional> #include <optional>
namespace Poco { class Path; }
namespace DB namespace DB
{ {

36
dbms/src/Core/MySQLProtocol.h
View File

@ -1,18 +1,16 @@
#pragma once #pragma once
#include <IO/ReadBuffer.h> #include <Core/Types.h>
#include <IO/WriteBuffer.h>
#include <IO/copyData.h> #include <IO/copyData.h>
#include <IO/ReadBuffer.h>
#include <IO/ReadBufferFromPocoSocket.h> #include <IO/ReadBufferFromPocoSocket.h>
#include <IO/WriteBuffer.h>
#include <IO/WriteBufferFromPocoSocket.h> #include <IO/WriteBufferFromPocoSocket.h>
#include <IO/WriteBufferFromString.h> #include <IO/WriteBufferFromString.h>
#include <Core/Types.h>
#include <Poco/RandomStream.h>
#include <Poco/Net/StreamSocket.h> #include <Poco/Net/StreamSocket.h>
#include <Poco/RandomStream.h>
#include <random> #include <random>
#include <sstream> #include <sstream>
#include <common/logger_useful.h>
#include <Poco/Logger.h>
/// Implementation of MySQL wire protocol /// Implementation of MySQL wire protocol
@ -157,23 +155,19 @@ public:
size_t max_packet_size = MAX_PACKET_LENGTH; size_t max_packet_size = MAX_PACKET_LENGTH;
/// For reading and writing. /// For reading and writing.
PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id, const String logger_name) PacketSender(ReadBuffer & in, WriteBuffer & out, size_t & sequence_id)
: sequence_id(sequence_id) : sequence_id(sequence_id)
, in(&in) , in(&in)
, out(&out) , out(&out)
, log(&Poco::Logger::get(logger_name))
{ {
log->setLevel("information");
} }
/// For writing. /// For writing.
PacketSender(WriteBuffer & out, size_t & sequence_id, const String logger_name) PacketSender(WriteBuffer & out, size_t & sequence_id)
: sequence_id(sequence_id) : sequence_id(sequence_id)
, in(nullptr) , in(nullptr)
, out(&out) , out(&out)
, log(&Poco::Logger::get(logger_name))
{ {
log->setLevel("information");
} }
String receivePacketPayload() String receivePacketPayload()
@ -186,8 +180,6 @@ public:
// packets which are larger than or equal to 16MB are splitted // packets which are larger than or equal to 16MB are splitted
do do
{ {
LOG_TRACE(log, "Reading from buffer");
in->readStrict(reinterpret_cast<char *>(&payload_length), 3); in->readStrict(reinterpret_cast<char *>(&payload_length), 3);
if (payload_length > max_packet_size) if (payload_length > max_packet_size)
@ -207,8 +199,6 @@ public:
} }
sequence_id++; sequence_id++;
LOG_TRACE(log, "Received packet. Sequence-id: " << packet_sequence_id << ", payload length: " << payload_length);
copyData(*in, static_cast<WriteBuffer &>(buf), payload_length); copyData(*in, static_cast<WriteBuffer &>(buf), payload_length);
} while (payload_length == max_packet_size); } while (payload_length == max_packet_size);
@ -230,9 +220,6 @@ public:
{ {
size_t payload_length = std::min(payload.length() - pos, max_packet_size); size_t payload_length = std::min(payload.length() - pos, max_packet_size);
LOG_TRACE(log, "Writing packet of size " << payload_length << " with sequence-id " << static_cast<int>(sequence_id));
LOG_TRACE(log, packetToText(payload));
out->write(reinterpret_cast<const char *>(&payload_length), 3); out->write(reinterpret_cast<const char *>(&payload_length), 3);
out->write(reinterpret_cast<const char *>(&sequence_id), 1); out->write(reinterpret_cast<const char *>(&sequence_id), 1);
out->write(payload.data() + pos, payload_length); out->write(payload.data() + pos, payload_length);
@ -241,13 +228,9 @@ public:
sequence_id++; sequence_id++;
} while (pos < payload.length()); } while (pos < payload.length());
LOG_TRACE(log, "Packet was sent.");
if (flush) if (flush)
{
out->next(); out->next();
} }
}
/// Sets sequence-id to 0. Must be called before each command phase. /// Sets sequence-id to 0. Must be called before each command phase.
void resetSequenceId(); void resetSequenceId();
@ -255,8 +238,6 @@ public:
private: private:
/// Converts packet to text. Is used for debug output. /// Converts packet to text. Is used for debug output.
static String packetToText(String payload); static String packetToText(String payload);
Poco::Logger * log;
}; };
@ -279,12 +260,11 @@ class Handshake : public WritePacket
uint32_t status_flags; uint32_t status_flags;
String auth_plugin_data; String auth_plugin_data;
public: public:
explicit Handshake(uint32_t connection_id, String server_version, String auth_plugin_data) explicit Handshake(uint32_t capability_flags, uint32_t connection_id, String server_version, String auth_plugin_data)
: protocol_version(0xa) : protocol_version(0xa)
, server_version(std::move(server_version)) , server_version(std::move(server_version))
, connection_id(connection_id) , connection_id(connection_id)
, capability_flags(CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA , capability_flags(capability_flags)
| CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF | CLIENT_SSL)
, character_set(CharacterSet::utf8_general_ci) , character_set(CharacterSet::utf8_general_ci)
, status_flags(0) , status_flags(0)
, auth_plugin_data(auth_plugin_data) , auth_plugin_data(auth_plugin_data)

1
dbms/src/Core/Settings.h
View File

@ -85,6 +85,7 @@ struct Settings : public SettingsCollection<Settings>
M(SettingFloat, totals_auto_threshold, 0.5, "The threshold for totals_mode = 'auto'.") \ M(SettingFloat, totals_auto_threshold, 0.5, "The threshold for totals_mode = 'auto'.") \
\ \
M(SettingBool, compile, false, "Whether query compilation is enabled.") \ M(SettingBool, compile, false, "Whether query compilation is enabled.") \
M(SettingBool, allow_suspicious_low_cardinality_types, false, "In CREATE TABLE statement allows specifying LowCardinality modifier for types of small fixed size (8 or less). Enabling this may increase merge times and memory consumption.") \
M(SettingBool, compile_expressions, false, "Compile some scalar functions and operators to native code.") \ M(SettingBool, compile_expressions, false, "Compile some scalar functions and operators to native code.") \
M(SettingUInt64, min_count_to_compile, 3, "The number of structurally identical queries before they are compiled.") \ M(SettingUInt64, min_count_to_compile, 3, "The number of structurally identical queries before they are compiled.") \
M(SettingUInt64, min_count_to_compile_expression, 3, "The number of identical expressions before they are JIT-compiled") \ M(SettingUInt64, min_count_to_compile_expression, 3, "The number of identical expressions before they are JIT-compiled") \

71
dbms/src/Core/Types.h
View File

@ -32,16 +32,16 @@ using String = std::string;
*/ */
template <typename T> constexpr bool IsNumber = false; template <typename T> constexpr bool IsNumber = false;
template <> constexpr bool IsNumber<UInt8> = true; template <> inline constexpr bool IsNumber<UInt8> = true;
template <> constexpr bool IsNumber<UInt16> = true; template <> inline constexpr bool IsNumber<UInt16> = true;
template <> constexpr bool IsNumber<UInt32> = true; template <> inline constexpr bool IsNumber<UInt32> = true;
template <> constexpr bool IsNumber<UInt64> = true; template <> inline constexpr bool IsNumber<UInt64> = true;
template <> constexpr bool IsNumber<Int8> = true; template <> inline constexpr bool IsNumber<Int8> = true;
template <> constexpr bool IsNumber<Int16> = true; template <> inline constexpr bool IsNumber<Int16> = true;
template <> constexpr bool IsNumber<Int32> = true; template <> inline constexpr bool IsNumber<Int32> = true;
template <> constexpr bool IsNumber<Int64> = true; template <> inline constexpr bool IsNumber<Int64> = true;
template <> constexpr bool IsNumber<Float32> = true; template <> inline constexpr bool IsNumber<Float32> = true;
template <> constexpr bool IsNumber<Float64> = true; template <> inline constexpr bool IsNumber<Float64> = true;
template <typename T> struct TypeName; template <typename T> struct TypeName;
@ -109,7 +109,7 @@ using Strings = std::vector<String>;
using Int128 = __int128; using Int128 = __int128;
template <> constexpr bool IsNumber<Int128> = true; template <> inline constexpr bool IsNumber<Int128> = true;
template <> struct TypeName<Int128> { static const char * get() { return "Int128"; } }; template <> struct TypeName<Int128> { static const char * get() { return "Int128"; } };
template <> struct TypeId<Int128> { static constexpr const TypeIndex value = TypeIndex::Int128; }; template <> struct TypeId<Int128> { static constexpr const TypeIndex value = TypeIndex::Int128; };
@ -159,17 +159,56 @@ template <> struct TypeId<Decimal32> { static constexpr const TypeIndex value
template <> struct TypeId<Decimal64> { static constexpr const TypeIndex value = TypeIndex::Decimal64; }; template <> struct TypeId<Decimal64> { static constexpr const TypeIndex value = TypeIndex::Decimal64; };
template <> struct TypeId<Decimal128> { static constexpr const TypeIndex value = TypeIndex::Decimal128; }; template <> struct TypeId<Decimal128> { static constexpr const TypeIndex value = TypeIndex::Decimal128; };
template <typename T> template <typename T> constexpr bool IsDecimalNumber = false;
constexpr bool IsDecimalNumber = false; template <> inline constexpr bool IsDecimalNumber<Decimal32> = true;
template <> constexpr bool IsDecimalNumber<Decimal32> = true; template <> inline constexpr bool IsDecimalNumber<Decimal64> = true;
template <> constexpr bool IsDecimalNumber<Decimal64> = true; template <> inline constexpr bool IsDecimalNumber<Decimal128> = true;
template <> constexpr bool IsDecimalNumber<Decimal128> = true;
template <typename T> struct NativeType { using Type = T; }; template <typename T> struct NativeType { using Type = T; };
template <> struct NativeType<Decimal32> { using Type = Int32; }; template <> struct NativeType<Decimal32> { using Type = Int32; };
template <> struct NativeType<Decimal64> { using Type = Int64; }; template <> struct NativeType<Decimal64> { using Type = Int64; };
template <> struct NativeType<Decimal128> { using Type = Int128; }; template <> struct NativeType<Decimal128> { using Type = Int128; };
inline const char * getTypeName(TypeIndex idx)
{
switch (idx)
{
case TypeIndex::Nothing: return "Nothing";
case TypeIndex::UInt8: return TypeName<UInt8>::get();
case TypeIndex::UInt16: return TypeName<UInt16>::get();
case TypeIndex::UInt32: return TypeName<UInt32>::get();
case TypeIndex::UInt64: return TypeName<UInt64>::get();
case TypeIndex::UInt128: return "UInt128";
case TypeIndex::Int8: return TypeName<Int8>::get();
case TypeIndex::Int16: return TypeName<Int16>::get();
case TypeIndex::Int32: return TypeName<Int32>::get();
case TypeIndex::Int64: return TypeName<Int64>::get();
case TypeIndex::Int128: return TypeName<Int128>::get();
case TypeIndex::Float32: return TypeName<Float32>::get();
case TypeIndex::Float64: return TypeName<Float64>::get();
case TypeIndex::Date: return "Date";
case TypeIndex::DateTime: return "DateTime";
case TypeIndex::String: return TypeName<String>::get();
case TypeIndex::FixedString: return "FixedString";
case TypeIndex::Enum8: return "Enum8";
case TypeIndex::Enum16: return "Enum16";
case TypeIndex::Decimal32: return TypeName<Decimal32>::get();
case TypeIndex::Decimal64: return TypeName<Decimal64>::get();
case TypeIndex::Decimal128: return TypeName<Decimal128>::get();
case TypeIndex::UUID: return "UUID";
case TypeIndex::Array: return "Array";
case TypeIndex::Tuple: return "Tuple";
case TypeIndex::Set: return "Set";
case TypeIndex::Interval: return "Interval";
case TypeIndex::Nullable: return "Nullable";
case TypeIndex::Function: return "Function";
case TypeIndex::AggregateFunction: return "AggregateFunction";
case TypeIndex::LowCardinality: return "LowCardinality";
}
__builtin_unreachable();
}
} }
/// Specialization of `std::hash` for the Decimal<T> types. /// Specialization of `std::hash` for the Decimal<T> types.

1
dbms/src/DataTypes/DataTypeAggregateFunction.h
View File

@ -36,6 +36,7 @@ public:
bool canBeInsideNullable() const override { return false; } bool canBeInsideNullable() const override { return false; }
DataTypePtr getReturnType() const { return function->getReturnType(); } DataTypePtr getReturnType() const { return function->getReturnType(); }
DataTypePtr getReturnTypeToPredict() const { return function->getReturnTypeToPredict(); }
DataTypes getArgumentsDataTypes() const { return argument_types; } DataTypes getArgumentsDataTypes() const { return argument_types; }
/// NOTE These two functions for serializing single values are incompatible with the functions below. /// NOTE These two functions for serializing single values are incompatible with the functions below.

2
dbms/src/DataTypes/DataTypeLowCardinalityHelpers.cpp
View File

@ -111,7 +111,7 @@ ColumnPtr recursiveLowCardinalityConversion(const ColumnPtr & column, const Data
{ {
auto col = low_cardinality_type->createColumn(); auto col = low_cardinality_type->createColumn();
static_cast<ColumnLowCardinality &>(*col).insertRangeFromFullColumn(*column, 0, column->size()); static_cast<ColumnLowCardinality &>(*col).insertRangeFromFullColumn(*column, 0, column->size());
return std::move(col); return col;
} }
} }

6
dbms/src/DataTypes/DataTypesDecimal.h
View File

@ -246,9 +246,9 @@ inline UInt32 getDecimalScale(const IDataType & data_type, UInt32 default_value
/// ///
template <typename DataType> constexpr bool IsDataTypeDecimal = false; template <typename DataType> constexpr bool IsDataTypeDecimal = false;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true; template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true; template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true; template <> inline constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
template <typename DataType> constexpr bool IsDataTypeDecimalOrNumber = IsDataTypeDecimal<DataType> || IsDataTypeNumber<DataType>; template <typename DataType> constexpr bool IsDataTypeDecimalOrNumber = IsDataTypeDecimal<DataType> || IsDataTypeNumber<DataType>;

20
dbms/src/DataTypes/DataTypesNumber.h
View File

@ -38,15 +38,15 @@ using DataTypeFloat32 = DataTypeNumber<Float32>;
using DataTypeFloat64 = DataTypeNumber<Float64>; using DataTypeFloat64 = DataTypeNumber<Float64>;
template <typename DataType> constexpr bool IsDataTypeNumber = false; template <typename DataType> constexpr bool IsDataTypeNumber = false;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true; template <> inline constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
} }

2
dbms/src/DataTypes/Native.h
View File

@ -16,7 +16,7 @@
#pragma GCC diagnostic push #pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-parameter" #pragma GCC diagnostic ignored "-Wunused-parameter"
#include <llvm/IR/IRBuilder.h> // Y_IGNORE #include <llvm/IR/IRBuilder.h>
#pragma GCC diagnostic pop #pragma GCC diagnostic pop

14
dbms/src/Dictionaries/CacheDictionary.cpp
View File

@ -86,7 +86,7 @@ void CacheDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<K
{ {
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values); const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
getItemsNumber<UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; }); getItemsNumberImpl<UInt64, UInt64>(*hierarchical_attribute, ids, out, [&](const size_t) { return null_value; });
} }
@ -207,9 +207,7 @@ void CacheDictionary::isInConstantVector(const Key child_id, const PaddedPODArra
void CacheDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const void CacheDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto null_value = StringRef{std::get<String>(attribute.null_values)}; const auto null_value = StringRef{std::get<String>(attribute.null_values)};
@ -220,9 +218,7 @@ void CacheDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsString(attribute, ids, out, [&](const size_t row) { return def->getDataAt(row); }); getItemsString(attribute, ids, out, [&](const size_t row) { return def->getDataAt(row); });
} }
@ -231,9 +227,7 @@ void CacheDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsString(attribute, ids, out, [&](const size_t) { return StringRef{def}; }); getItemsString(attribute, ids, out, [&](const size_t) { return StringRef{def}; });
} }

5
dbms/src/Dictionaries/CacheDictionary.h
View File

@ -221,11 +221,6 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename DefaultGetter>
void getItemsNumber(
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;
template <typename AttributeType, typename OutputType, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename DefaultGetter>
void getItemsNumberImpl( void getItemsNumberImpl(
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const; Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const;

28
dbms/src/Dictionaries/CacheDictionary.inc.h
View File

@ -34,34 +34,6 @@ namespace ErrorCodes
extern const int TYPE_MISMATCH; extern const int TYPE_MISMATCH;
} }
template <typename OutputType, typename DefaultGetter>
void CacheDictionary::getItemsNumber(
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) \
getItemsNumberImpl<TYPE, OutputType>(attribute, ids, out, std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename DefaultGetter>
void CacheDictionary::getItemsNumberImpl( void CacheDictionary::getItemsNumberImpl(
Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const Attribute & attribute, const PaddedPODArray<Key> & ids, ResultArrayType<OutputType> & out, DefaultGetter && get_default) const

6
dbms/src/Dictionaries/CacheDictionary_generate1.cpp.in
View File

@ -12,13 +12,11 @@ using TYPE = @NAME@;
void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto null_value = std::get<TYPE>(attribute.null_values); const auto null_value = std::get<TYPE>(attribute.null_values);
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return null_value; }); getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return null_value; });
} }
} }

6
dbms/src/Dictionaries/CacheDictionary_generate2.cpp.in
View File

@ -15,11 +15,9 @@ void CacheDictionary::get@NAME@(const std::string & attribute_name,
ResultArrayType<TYPE> & out) const ResultArrayType<TYPE> & out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; }); getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t row) { return def[row]; });
} }
} }

6
dbms/src/Dictionaries/CacheDictionary_generate3.cpp.in
View File

@ -12,11 +12,9 @@ using TYPE = @NAME@;
void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const void CacheDictionary::get@NAME@(const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const
{ {
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsNumber<TYPE>(attribute, ids, out, [&](const size_t) { return def; }); getItemsNumberImpl<TYPE, TYPE>(attribute, ids, out, [&](const size_t) { return def; });
} }
} }

12
dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp
View File

@ -77,9 +77,7 @@ void ComplexKeyCacheDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto null_value = StringRef{std::get<String>(attribute.null_values)}; const auto null_value = StringRef{std::get<String>(attribute.null_values)};
@ -96,9 +94,7 @@ void ComplexKeyCacheDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsString(attribute, key_columns, out, [&](const size_t row) { return def->getDataAt(row); }); getItemsString(attribute, key_columns, out, [&](const size_t row) { return def->getDataAt(row); });
} }
@ -113,9 +109,7 @@ void ComplexKeyCacheDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsString(attribute, key_columns, out, [&](const size_t) { return StringRef{def}; }); getItemsString(attribute, key_columns, out, [&](const size_t) { return StringRef{def}; });
} }

28
dbms/src/Dictionaries/ComplexKeyCacheDictionary.h
View File

@ -256,34 +256,6 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename DefaultGetter>
void
getItemsNumber(Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) \
getItemsNumberImpl<TYPE, OutputType>(attribute, key_columns, out, std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename DefaultGetter>
void getItemsNumberImpl( void getItemsNumberImpl(
Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const Attribute & attribute, const Columns & key_columns, PaddedPODArray<OutputType> & out, DefaultGetter && get_default) const

6
dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate1.cpp.in
View File

@ -13,12 +13,10 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name, co
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto null_value = std::get<TYPE>(attribute.null_values); const auto null_value = std::get<TYPE>(attribute.null_values);
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; }); getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return null_value; });
} }
} }

6
dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate2.cpp.in
View File

@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; }); getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t row) { return def[row]; });
} }
} }

6
dbms/src/Dictionaries/ComplexKeyCacheDictionary_generate3.cpp.in
View File

@ -18,10 +18,8 @@ void ComplexKeyCacheDictionary::get@NAME@(const std::string & attribute_name,
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
auto & attribute = getAttribute(attribute_name); auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::@NAME@)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::@NAME@);
throw Exception {name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsNumber<TYPE>(attribute, key_columns, out, [&](const size_t) { return def; }); getItemsNumberImpl<TYPE, TYPE>(attribute, key_columns, out, [&](const size_t) { return def; });
} }
} }

58
dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp
View File

@ -50,13 +50,11 @@ ComplexKeyHashedDictionary::ComplexKeyHashedDictionary(
dict_struct.validateKeyTypes(key_types); \ dict_struct.validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
const auto null_value = std::get<TYPE>(attribute.null_values); \ const auto null_value = std::get<TYPE>(attribute.null_values); \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, \ attribute, \
key_columns, \ key_columns, \
[&](const size_t row, const auto value) { out[row] = value; }, \ [&](const size_t row, const auto value) { out[row] = value; }, \
@ -84,9 +82,7 @@ void ComplexKeyHashedDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto & null_value = StringRef{std::get<String>(attribute.null_values)}; const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
@ -108,11 +104,9 @@ void ComplexKeyHashedDictionary::getString(
dict_struct.validateKeyTypes(key_types); \ dict_struct.validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, \ attribute, \
key_columns, \ key_columns, \
[&](const size_t row, const auto value) { out[row] = value; }, \ [&](const size_t row, const auto value) { out[row] = value; }, \
@ -144,9 +138,7 @@ void ComplexKeyHashedDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -166,11 +158,9 @@ void ComplexKeyHashedDictionary::getString(
dict_struct.validateKeyTypes(key_types); \ dict_struct.validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -199,9 +189,7 @@ void ComplexKeyHashedDictionary::getString(
dict_struct.validateKeyTypes(key_types); dict_struct.validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -566,34 +554,6 @@ ComplexKeyHashedDictionary::createAttributeWithType(const AttributeUnderlyingTyp
} }
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void ComplexKeyHashedDictionary::getItemsNumber(
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void ComplexKeyHashedDictionary::getItemsImpl( void ComplexKeyHashedDictionary::getItemsImpl(
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const

6
dbms/src/Dictionaries/ComplexKeyHashedDictionary.h
View File

@ -218,16 +218,10 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void
getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void void
getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const; getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename T> template <typename T>
bool setAttributeValueImpl(Attribute & attribute, const StringRef key, const T value); bool setAttributeValueImpl(Attribute & attribute, const StringRef key, const T value);

38
dbms/src/Dictionaries/DictionaryStructure.cpp
View File

@ -40,44 +40,6 @@ namespace
} // namespace } // namespace
bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to)
{
if (from == to)
return true;
/** This enum can be somewhat incomplete and the meaning may not coincide with NumberTraits.h.
* (for example, because integers can not be converted to floats)
* This is normal for a limited usage scope.
*/
if ((from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt16)
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt32)
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::UInt64)
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt32)
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::UInt64)
|| (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::UInt64)
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int16)
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int32)
|| (from == AttributeUnderlyingType::UInt8 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int32)
|| (from == AttributeUnderlyingType::UInt16 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::UInt32 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int16)
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int32)
|| (from == AttributeUnderlyingType::Int8 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int32)
|| (from == AttributeUnderlyingType::Int16 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::Int32 && to == AttributeUnderlyingType::Int64)
|| (from == AttributeUnderlyingType::Float32 && to == AttributeUnderlyingType::Float64))
{
return true;
}
return false;
}
AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type) AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type)
{ {
static const std::unordered_map<std::string, AttributeUnderlyingType> dictionary{ static const std::unordered_map<std::string, AttributeUnderlyingType> dictionary{

18
dbms/src/Dictionaries/DictionaryStructure.h
View File

@ -13,6 +13,12 @@
namespace DB namespace DB
{ {
namespace ErrorCodes
{
extern const int TYPE_MISMATCH;
}
enum class AttributeUnderlyingType enum class AttributeUnderlyingType
{ {
UInt8, UInt8,
@ -33,14 +39,18 @@ enum class AttributeUnderlyingType
}; };
/** For implicit conversions in dictGet functions.
*/
bool isAttributeTypeConvertibleTo(AttributeUnderlyingType from, AttributeUnderlyingType to);
AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type); AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type);
std::string toString(const AttributeUnderlyingType type); std::string toString(const AttributeUnderlyingType type);
/// Implicit conversions in dictGet functions is disabled.
inline void checkAttributeType(const std::string & dict_name, const std::string & attribute_name,
AttributeUnderlyingType attribute_type, AttributeUnderlyingType to)
{
if (attribute_type != to)
throw Exception{dict_name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute_type)
+ ", expected " + toString(to), ErrorCodes::TYPE_MISMATCH};
}
/// Min and max lifetimes for a dictionary or it's entry /// Min and max lifetimes for a dictionary or it's entry
using DictionaryLifetime = ExternalLoadableLifetime; using DictionaryLifetime = ExternalLoadableLifetime;

65
dbms/src/Dictionaries/FlatDictionary.cpp
View File

@ -55,7 +55,7 @@ void FlatDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<Ke
{ {
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values); const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
getItemsNumber<UInt64>( getItemsImpl<UInt64, UInt64>(
*hierarchical_attribute, *hierarchical_attribute,
ids, ids,
[&](const size_t row, const UInt64 value) { out[row] = value; }, [&](const size_t row, const UInt64 value) { out[row] = value; },
@ -117,13 +117,11 @@ void FlatDictionary::isInConstantVector(const Key child_id, const PaddedPODArray
void FlatDictionary::get##TYPE(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const \ void FlatDictionary::get##TYPE(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ResultArrayType<TYPE> & out) const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
const auto null_value = std::get<TYPE>(attribute.null_values); \ const auto null_value = std::get<TYPE>(attribute.null_values); \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -145,9 +143,7 @@ DECLARE(Decimal128)
void FlatDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const void FlatDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto & null_value = std::get<StringRef>(attribute.null_values); const auto & null_value = std::get<StringRef>(attribute.null_values);
@ -166,11 +162,9 @@ void FlatDictionary::getString(const std::string & attribute_name, const PaddedP
ResultArrayType<TYPE> & out) const \ ResultArrayType<TYPE> & out) const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -193,9 +187,7 @@ void FlatDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -209,11 +201,9 @@ void FlatDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const \ const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE def, ResultArrayType<TYPE> & out) const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -236,9 +226,7 @@ void FlatDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
FlatDictionary::getItemsImpl<StringRef, StringRef>( FlatDictionary::getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -580,35 +568,6 @@ FlatDictionary::Attribute FlatDictionary::createAttributeWithType(const Attribut
} }
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void FlatDictionary::getItemsNumber(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) \
getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void FlatDictionary::getItemsImpl( void FlatDictionary::getItemsImpl(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
@ -701,10 +660,10 @@ void FlatDictionary::setAttributeValue(Attribute & attribute, const Key id, cons
break; break;
case AttributeUnderlyingType::Decimal32: case AttributeUnderlyingType::Decimal32:
setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal128>()); setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
break; break;
case AttributeUnderlyingType::Decimal64: case AttributeUnderlyingType::Decimal64:
setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal128>()); setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
break; break;
case AttributeUnderlyingType::Decimal128: case AttributeUnderlyingType::Decimal128:
setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>()); setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());

4
dbms/src/Dictionaries/FlatDictionary.h
View File

@ -206,10 +206,6 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void getItemsNumber(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void getItemsImpl( void getItemsImpl(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const; const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;

117
dbms/src/Dictionaries/HashedDictionary.cpp
View File

@ -49,7 +49,7 @@ void HashedDictionary::toParent(const PaddedPODArray<Key> & ids, PaddedPODArray<
{ {
const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values); const auto null_value = std::get<UInt64>(hierarchical_attribute->null_values);
getItemsNumber<UInt64>( getItemsImpl<UInt64, UInt64>(
*hierarchical_attribute, *hierarchical_attribute,
ids, ids,
[&](const size_t row, const UInt64 value) { out[row] = value; }, [&](const size_t row, const UInt64 value) { out[row] = value; },
@ -116,13 +116,11 @@ void HashedDictionary::isInConstantVector(const Key child_id, const PaddedPODArr
const \ const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
const auto null_value = std::get<TYPE>(attribute.null_values); \ const auto null_value = std::get<TYPE>(attribute.null_values); \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return null_value; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -144,9 +142,7 @@ DECLARE(Decimal128)
void HashedDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const void HashedDictionary::getString(const std::string & attribute_name, const PaddedPODArray<Key> & ids, ColumnString * out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto & null_value = StringRef{std::get<String>(attribute.null_values)}; const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
@ -165,11 +161,9 @@ void HashedDictionary::getString(const std::string & attribute_name, const Padde
ResultArrayType<TYPE> & out) const \ ResultArrayType<TYPE> & out) const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -192,9 +186,7 @@ void HashedDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const ColumnString * const def, ColumnString * const out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -208,11 +200,9 @@ void HashedDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE & def, ResultArrayType<TYPE> & out) const \ const std::string & attribute_name, const PaddedPODArray<Key> & ids, const TYPE & def, ResultArrayType<TYPE> & out) const \
{ \ { \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -235,9 +225,7 @@ void HashedDictionary::getString(
const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const const std::string & attribute_name, const PaddedPODArray<Key> & ids, const String & def, ColumnString * const out) const
{ {
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -324,7 +312,6 @@ void HashedDictionary::createAttributes()
void HashedDictionary::blockToAttributes(const Block & block) void HashedDictionary::blockToAttributes(const Block & block)
{ {
const auto & id_column = *block.safeGetByPosition(0).column; const auto & id_column = *block.safeGetByPosition(0).column;
element_count += id_column.size();
for (const size_t attribute_idx : ext::range(0, attributes.size())) for (const size_t attribute_idx : ext::range(0, attributes.size()))
{ {
@ -332,7 +319,8 @@ void HashedDictionary::blockToAttributes(const Block & block)
auto & attribute = attributes[attribute_idx]; auto & attribute = attributes[attribute_idx];
for (const auto row_idx : ext::range(0, id_column.size())) for (const auto row_idx : ext::range(0, id_column.size()))
setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]); if (setAttributeValue(attribute, id_column[row_idx].get<UInt64>(), attribute_column[row_idx]))
++element_count;
} }
} }
@ -567,34 +555,6 @@ HashedDictionary::Attribute HashedDictionary::createAttributeWithType(const Attr
} }
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void HashedDictionary::getItemsNumber(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) \
getItemsImpl<TYPE, OutputType>(attribute, ids, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void HashedDictionary::getItemsImpl( void HashedDictionary::getItemsImpl(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const
@ -613,69 +573,56 @@ void HashedDictionary::getItemsImpl(
template <typename T> template <typename T>
void HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value) bool HashedDictionary::setAttributeValueImpl(Attribute & attribute, const Key id, const T value)
{ {
auto & map = *std::get<CollectionPtrType<T>>(attribute.maps); auto & map = *std::get<CollectionPtrType<T>>(attribute.maps);
map.insert({id, value}); return map.insert({id, value}).second;
} }
void HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value) bool HashedDictionary::setAttributeValue(Attribute & attribute, const Key id, const Field & value)
{ {
switch (attribute.type) switch (attribute.type)
{ {
case AttributeUnderlyingType::UInt8: case AttributeUnderlyingType::UInt8:
setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>()); return setAttributeValueImpl<UInt8>(attribute, id, value.get<UInt64>());
break;
case AttributeUnderlyingType::UInt16: case AttributeUnderlyingType::UInt16:
setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>()); return setAttributeValueImpl<UInt16>(attribute, id, value.get<UInt64>());
break;
case AttributeUnderlyingType::UInt32: case AttributeUnderlyingType::UInt32:
setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>()); return setAttributeValueImpl<UInt32>(attribute, id, value.get<UInt64>());
break;
case AttributeUnderlyingType::UInt64: case AttributeUnderlyingType::UInt64:
setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>()); return setAttributeValueImpl<UInt64>(attribute, id, value.get<UInt64>());
break;
case AttributeUnderlyingType::UInt128: case AttributeUnderlyingType::UInt128:
setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>()); return setAttributeValueImpl<UInt128>(attribute, id, value.get<UInt128>());
break;
case AttributeUnderlyingType::Int8: case AttributeUnderlyingType::Int8:
setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>()); return setAttributeValueImpl<Int8>(attribute, id, value.get<Int64>());
break;
case AttributeUnderlyingType::Int16: case AttributeUnderlyingType::Int16:
setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>()); return setAttributeValueImpl<Int16>(attribute, id, value.get<Int64>());
break;
case AttributeUnderlyingType::Int32: case AttributeUnderlyingType::Int32:
setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>()); return setAttributeValueImpl<Int32>(attribute, id, value.get<Int64>());
break;
case AttributeUnderlyingType::Int64: case AttributeUnderlyingType::Int64:
setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>()); return setAttributeValueImpl<Int64>(attribute, id, value.get<Int64>());
break;
case AttributeUnderlyingType::Float32: case AttributeUnderlyingType::Float32:
setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>()); return setAttributeValueImpl<Float32>(attribute, id, value.get<Float64>());
break;
case AttributeUnderlyingType::Float64: case AttributeUnderlyingType::Float64:
setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>()); return setAttributeValueImpl<Float64>(attribute, id, value.get<Float64>());
break;
case AttributeUnderlyingType::Decimal32: case AttributeUnderlyingType::Decimal32:
setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>()); return setAttributeValueImpl<Decimal32>(attribute, id, value.get<Decimal32>());
break;
case AttributeUnderlyingType::Decimal64: case AttributeUnderlyingType::Decimal64:
setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>()); return setAttributeValueImpl<Decimal64>(attribute, id, value.get<Decimal64>());
break;
case AttributeUnderlyingType::Decimal128: case AttributeUnderlyingType::Decimal128:
setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>()); return setAttributeValueImpl<Decimal128>(attribute, id, value.get<Decimal128>());
break;
case AttributeUnderlyingType::String: case AttributeUnderlyingType::String:
{ {
auto & map = *std::get<CollectionPtrType<StringRef>>(attribute.maps); auto & map = *std::get<CollectionPtrType<StringRef>>(attribute.maps);
const auto & string = value.get<String>(); const auto & string = value.get<String>();
const auto string_in_arena = attribute.string_arena->insert(string.data(), string.size()); const auto string_in_arena = attribute.string_arena->insert(string.data(), string.size());
map.insert({id, StringRef{string_in_arena, string.size()}}); return map.insert({id, StringRef{string_in_arena, string.size()}}).second;
break;
} }
} }
throw Exception{"Invalid attribute type", ErrorCodes::BAD_ARGUMENTS};
} }
const HashedDictionary::Attribute & HashedDictionary::getAttribute(const std::string & attribute_name) const const HashedDictionary::Attribute & HashedDictionary::getAttribute(const std::string & attribute_name) const

8
dbms/src/Dictionaries/HashedDictionary.h
View File

@ -211,18 +211,14 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void getItemsNumber(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void getItemsImpl( void getItemsImpl(
const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const; const Attribute & attribute, const PaddedPODArray<Key> & ids, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename T> template <typename T>
void setAttributeValueImpl(Attribute & attribute, const Key id, const T value); bool setAttributeValueImpl(Attribute & attribute, const Key id, const T value);
void setAttributeValue(Attribute & attribute, const Key id, const Field & value); bool setAttributeValue(Attribute & attribute, const Key id, const Field & value);
const Attribute & getAttribute(const std::string & attribute_name) const; const Attribute & getAttribute(const std::string & attribute_name) const;

58
dbms/src/Dictionaries/TrieDictionary.cpp
View File

@ -75,13 +75,11 @@ TrieDictionary::~TrieDictionary()
validateKeyTypes(key_types); \ validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
const auto null_value = std::get<TYPE>(attribute.null_values); \ const auto null_value = std::get<TYPE>(attribute.null_values); \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, \ attribute, \
key_columns, \ key_columns, \
[&](const size_t row, const auto value) { out[row] = value; }, \ [&](const size_t row, const auto value) { out[row] = value; }, \
@ -109,9 +107,7 @@ void TrieDictionary::getString(
validateKeyTypes(key_types); validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
const auto & null_value = StringRef{std::get<String>(attribute.null_values)}; const auto & null_value = StringRef{std::get<String>(attribute.null_values)};
@ -133,11 +129,9 @@ void TrieDictionary::getString(
validateKeyTypes(key_types); \ validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, \ attribute, \
key_columns, \ key_columns, \
[&](const size_t row, const auto value) { out[row] = value; }, \ [&](const size_t row, const auto value) { out[row] = value; }, \
@ -169,9 +163,7 @@ void TrieDictionary::getString(
validateKeyTypes(key_types); validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -191,11 +183,9 @@ void TrieDictionary::getString(
validateKeyTypes(key_types); \ validateKeyTypes(key_types); \
\ \
const auto & attribute = getAttribute(attribute_name); \ const auto & attribute = getAttribute(attribute_name); \
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::TYPE)) \ checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::TYPE); \
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type), \
ErrorCodes::TYPE_MISMATCH}; \
\ \
getItemsNumber<TYPE>( \ getItemsImpl<TYPE, TYPE>( \
attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \
} }
DECLARE(UInt8) DECLARE(UInt8)
@ -224,9 +214,7 @@ void TrieDictionary::getString(
validateKeyTypes(key_types); validateKeyTypes(key_types);
const auto & attribute = getAttribute(attribute_name); const auto & attribute = getAttribute(attribute_name);
if (!isAttributeTypeConvertibleTo(attribute.type, AttributeUnderlyingType::String)) checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::String);
throw Exception{name + ": type mismatch: attribute " + attribute_name + " has type " + toString(attribute.type),
ErrorCodes::TYPE_MISMATCH};
getItemsImpl<StringRef, StringRef>( getItemsImpl<StringRef, StringRef>(
attribute, attribute,
@ -507,34 +495,6 @@ TrieDictionary::Attribute TrieDictionary::createAttributeWithType(const Attribut
} }
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void TrieDictionary::getItemsNumber(
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const
{
if (false)
{
}
#define DISPATCH(TYPE) \
else if (attribute.type == AttributeUnderlyingType::TYPE) getItemsImpl<TYPE, OutputType>( \
attribute, key_columns, std::forward<ValueSetter>(set_value), std::forward<DefaultGetter>(get_default));
DISPATCH(UInt8)
DISPATCH(UInt16)
DISPATCH(UInt32)
DISPATCH(UInt64)
DISPATCH(UInt128)
DISPATCH(Int8)
DISPATCH(Int16)
DISPATCH(Int32)
DISPATCH(Int64)
DISPATCH(Float32)
DISPATCH(Float64)
DISPATCH(Decimal32)
DISPATCH(Decimal64)
DISPATCH(Decimal128)
#undef DISPATCH
else throw Exception("Unexpected type of attribute: " + toString(attribute.type), ErrorCodes::LOGICAL_ERROR);
}
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void TrieDictionary::getItemsImpl( void TrieDictionary::getItemsImpl(
const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const

4
dbms/src/Dictionaries/TrieDictionary.h
View File

@ -218,10 +218,6 @@ private:
Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value); Attribute createAttributeWithType(const AttributeUnderlyingType type, const Field & null_value);
template <typename OutputType, typename ValueSetter, typename DefaultGetter>
void
getItemsNumber(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;
template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter> template <typename AttributeType, typename OutputType, typename ValueSetter, typename DefaultGetter>
void void
getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const; getItemsImpl(const Attribute & attribute, const Columns & key_columns, ValueSetter && set_value, DefaultGetter && get_default) const;

8
dbms/src/Formats/CapnProtoRowInputStream.cpp
View File

@ -3,13 +3,13 @@
#include <IO/ReadBuffer.h> #include <IO/ReadBuffer.h>
#include <Interpreters/Context.h> #include <Interpreters/Context.h>
#include <Formats/CapnProtoRowInputStream.h> // Y_IGNORE #include <Formats/CapnProtoRowInputStream.h>
#include <Formats/FormatFactory.h> #include <Formats/FormatFactory.h>
#include <Formats/BlockInputStreamFromRowInputStream.h> #include <Formats/BlockInputStreamFromRowInputStream.h>
#include <Formats/FormatSchemaInfo.h> #include <Formats/FormatSchemaInfo.h>
#include <capnp/serialize.h> // Y_IGNORE #include <capnp/serialize.h>
#include <capnp/dynamic.h> // Y_IGNORE #include <capnp/dynamic.h>
#include <capnp/common.h> // Y_IGNORE #include <capnp/common.h>
#include <boost/algorithm/string.hpp> #include <boost/algorithm/string.hpp>
#include <boost/range/join.hpp> #include <boost/range/join.hpp>
#include <common/logger_useful.h> #include <common/logger_useful.h>

4
dbms/src/Formats/FormatFactory.cpp
View File

@ -130,7 +130,7 @@ void registerOutputFormatXML(FormatFactory & factory);
void registerOutputFormatODBCDriver(FormatFactory & factory); void registerOutputFormatODBCDriver(FormatFactory & factory);
void registerOutputFormatODBCDriver2(FormatFactory & factory); void registerOutputFormatODBCDriver2(FormatFactory & factory);
void registerOutputFormatNull(FormatFactory & factory); void registerOutputFormatNull(FormatFactory & factory);
void registerOutputFormatMySQL(FormatFactory & factory); void registerOutputFormatMySQLWire(FormatFactory & factory);
/// Input only formats. /// Input only formats.
@ -169,7 +169,7 @@ FormatFactory::FormatFactory()
registerOutputFormatODBCDriver(*this); registerOutputFormatODBCDriver(*this);
registerOutputFormatODBCDriver2(*this); registerOutputFormatODBCDriver2(*this);
registerOutputFormatNull(*this); registerOutputFormatNull(*this);
registerOutputFormatMySQL(*this); registerOutputFormatMySQLWire(*this);
} }
} }

19
dbms/src/Formats/MySQLFormat.cpp
View File

@ -1,19 +0,0 @@
#include <DataStreams/MySQLBlockOutputStream.h>
namespace DB
{
void registerOutputFormatMySQL(FormatFactory & factory)
{
factory.registerOutputFormat("MySQL", [](
WriteBuffer & buf,
const Block & sample,
const Context & context,
const FormatSettings &)
{
return std::make_shared<MySQLBlockOutputStream>(buf, sample, const_cast<Context &>(context));
});
}
}

17
dbms/src/DataStreams/MySQLBlockOutputStream.cpp → dbms/src/Formats/MySQLWireBlockOutputStream.cpp
View File

@ -1,4 +1,4 @@
#include "MySQLBlockOutputStream.h" #include "MySQLWireBlockOutputStream.h"
#include <Core/MySQLProtocol.h> #include <Core/MySQLProtocol.h>
#include <Interpreters/ProcessList.h> #include <Interpreters/ProcessList.h>
#include <iomanip> #include <iomanip>
@ -9,15 +9,15 @@ namespace DB
using namespace MySQLProtocol; using namespace MySQLProtocol;
MySQLBlockOutputStream::MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context) MySQLWireBlockOutputStream::MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context)
: header(header) : header(header)
, context(context) , context(context)
, packet_sender(new PacketSender(buf, context.sequence_id, "MySQLBlockOutputStream")) , packet_sender(std::make_shared<PacketSender>(buf, context.sequence_id))
{ {
packet_sender->max_packet_size = context.max_packet_size; packet_sender->max_packet_size = context.max_packet_size;
} }
void MySQLBlockOutputStream::writePrefix() void MySQLWireBlockOutputStream::writePrefix()
{ {
if (header.columns() == 0) if (header.columns() == 0)
return; return;
@ -26,8 +26,7 @@ void MySQLBlockOutputStream::writePrefix()
for (const ColumnWithTypeAndName & column : header.getColumnsWithTypeAndName()) for (const ColumnWithTypeAndName & column : header.getColumnsWithTypeAndName())
{ {
ColumnDefinition column_definition(column.name, CharacterSet::binary, std::numeric_limits<uint32_t>::max(), ColumnDefinition column_definition(column.name, CharacterSet::binary, 0, ColumnType::MYSQL_TYPE_STRING, 0, 0);
ColumnType::MYSQL_TYPE_STRING, 0, 0);
packet_sender->sendPacket(column_definition); packet_sender->sendPacket(column_definition);
} }
@ -37,7 +36,7 @@ void MySQLBlockOutputStream::writePrefix()
} }
} }
void MySQLBlockOutputStream::write(const Block & block) void MySQLWireBlockOutputStream::write(const Block & block)
{ {
size_t rows = block.rows(); size_t rows = block.rows();
@ -57,7 +56,7 @@ void MySQLBlockOutputStream::write(const Block & block)
} }
} }
void MySQLBlockOutputStream::writeSuffix() void MySQLWireBlockOutputStream::writeSuffix()
{ {
QueryStatus * process_list_elem = context.getProcessListElement(); QueryStatus * process_list_elem = context.getProcessListElement();
CurrentThread::finalizePerformanceCounters(); CurrentThread::finalizePerformanceCounters();
@ -79,7 +78,7 @@ void MySQLBlockOutputStream::writeSuffix()
packet_sender->sendPacket(EOF_Packet(0, 0), true); packet_sender->sendPacket(EOF_Packet(0, 0), true);
} }
void MySQLBlockOutputStream::flush() void MySQLWireBlockOutputStream::flush()
{ {
packet_sender->out->next(); packet_sender->out->next();
} }

8
dbms/src/DataStreams/MySQLBlockOutputStream.h → dbms/src/Formats/MySQLWireBlockOutputStream.h
View File

@ -1,7 +1,7 @@
#pragma once #pragma once
#include "IBlockOutputStream.h"
#include <Core/MySQLProtocol.h> #include <Core/MySQLProtocol.h>
#include <DataStreams/IBlockOutputStream.h>
#include <Formats/FormatFactory.h> #include <Formats/FormatFactory.h>
#include <Formats/FormatSettings.h> #include <Formats/FormatSettings.h>
#include <Interpreters/Context.h> #include <Interpreters/Context.h>
@ -11,10 +11,10 @@ namespace DB
/** Interface for writing rows in MySQL Client/Server Protocol format. /** Interface for writing rows in MySQL Client/Server Protocol format.
*/ */
class MySQLBlockOutputStream : public IBlockOutputStream class MySQLWireBlockOutputStream : public IBlockOutputStream
{ {
public: public:
MySQLBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context); MySQLWireBlockOutputStream(WriteBuffer & buf, const Block & header, Context & context);
Block getHeader() const { return header; } Block getHeader() const { return header; }
@ -31,6 +31,6 @@ private:
FormatSettings format_settings; FormatSettings format_settings;
}; };
using MySQLBlockOutputStreamPtr = std::shared_ptr<MySQLBlockOutputStream>; using MySQLWireBlockOutputStreamPtr = std::shared_ptr<MySQLWireBlockOutputStream>;
} }

19
dbms/src/Formats/MySQLWireFormat.cpp Normal file
View File

@ -0,0 +1,19 @@
#include <Formats/MySQLWireBlockOutputStream.h>
namespace DB
{
void registerOutputFormatMySQLWire(FormatFactory & factory)
{
factory.registerOutputFormat("MySQLWire", [](
WriteBuffer & buf,
const Block & sample,
const Context & context,
const FormatSettings &)
{
return std::make_shared<MySQLWireBlockOutputStream>(buf, sample, const_cast<Context &>(context));
});
}
}

4
dbms/src/Formats/ProtobufSchemas.cpp
View File

@ -2,8 +2,8 @@
#if USE_PROTOBUF #if USE_PROTOBUF
#include <Formats/FormatSchemaInfo.h> #include <Formats/FormatSchemaInfo.h>
#include <Formats/ProtobufSchemas.h> // Y_IGNORE #include <Formats/ProtobufSchemas.h>
#include <google/protobuf/compiler/importer.h> // Y_IGNORE #include <google/protobuf/compiler/importer.h>
#include <Common/Exception.h> #include <Common/Exception.h>

4
dbms/src/Formats/ProtobufWriter.cpp
View File

@ -7,8 +7,8 @@
#include <AggregateFunctions/IAggregateFunction.h> #include <AggregateFunctions/IAggregateFunction.h>
#include <DataTypes/DataTypesDecimal.h> #include <DataTypes/DataTypesDecimal.h>
#include <boost/numeric/conversion/cast.hpp> #include <boost/numeric/conversion/cast.hpp>
#include <google/protobuf/descriptor.h> // Y_IGNORE #include <google/protobuf/descriptor.h>
#include <google/protobuf/descriptor.pb.h> // Y_IGNORE #include <google/protobuf/descriptor.pb.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include "ProtobufWriter.h" #include "ProtobufWriter.h"

2
dbms/src/Functions/FunctionBase64Conversion.h
View File

@ -7,7 +7,7 @@
#include <Functions/FunctionHelpers.h> #include <Functions/FunctionHelpers.h>
#include <Functions/GatherUtils/Algorithms.h> #include <Functions/GatherUtils/Algorithms.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <libbase64.h> // Y_IGNORE #include <libbase64.h>
namespace DB namespace DB

32
dbms/src/Functions/FunctionBinaryArithmetic.h
View File

@ -24,7 +24,7 @@
#if USE_EMBEDDED_COMPILER #if USE_EMBEDDED_COMPILER
#pragma GCC diagnostic push #pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-parameter" #pragma GCC diagnostic ignored "-Wunused-parameter"
#include <llvm/IR/IRBuilder.h> // Y_IGNORE #include <llvm/IR/IRBuilder.h>
#pragma GCC diagnostic pop #pragma GCC diagnostic pop
#endif #endif
@ -354,27 +354,27 @@ template <bool V, typename T> struct Case : std::bool_constant<V> { using type =
template <typename... Ts> using Switch = typename std::disjunction<Ts..., Case<true, InvalidType>>::type; template <typename... Ts> using Switch = typename std::disjunction<Ts..., Case<true, InvalidType>>::type;
template <typename DataType> constexpr bool IsIntegral = false; template <typename DataType> constexpr bool IsIntegral = false;
template <> constexpr bool IsIntegral<DataTypeUInt8> = true; template <> inline constexpr bool IsIntegral<DataTypeUInt8> = true;
template <> constexpr bool IsIntegral<DataTypeUInt16> = true; template <> inline constexpr bool IsIntegral<DataTypeUInt16> = true;
template <> constexpr bool IsIntegral<DataTypeUInt32> = true; template <> inline constexpr bool IsIntegral<DataTypeUInt32> = true;
template <> constexpr bool IsIntegral<DataTypeUInt64> = true; template <> inline constexpr bool IsIntegral<DataTypeUInt64> = true;
template <> constexpr bool IsIntegral<DataTypeInt8> = true; template <> inline constexpr bool IsIntegral<DataTypeInt8> = true;
template <> constexpr bool IsIntegral<DataTypeInt16> = true; template <> inline constexpr bool IsIntegral<DataTypeInt16> = true;
template <> constexpr bool IsIntegral<DataTypeInt32> = true; template <> inline constexpr bool IsIntegral<DataTypeInt32> = true;
template <> constexpr bool IsIntegral<DataTypeInt64> = true; template <> inline constexpr bool IsIntegral<DataTypeInt64> = true;
template <typename DataType> constexpr bool IsFloatingPoint = false; template <typename DataType> constexpr bool IsFloatingPoint = false;
template <> constexpr bool IsFloatingPoint<DataTypeFloat32> = true; template <> inline constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
template <> constexpr bool IsFloatingPoint<DataTypeFloat64> = true; template <> inline constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
template <typename DataType> constexpr bool IsDateOrDateTime = false; template <typename DataType> constexpr bool IsDateOrDateTime = false;
template <> constexpr bool IsDateOrDateTime<DataTypeDate> = true; template <> inline constexpr bool IsDateOrDateTime<DataTypeDate> = true;
template <> constexpr bool IsDateOrDateTime<DataTypeDateTime> = true; template <> inline constexpr bool IsDateOrDateTime<DataTypeDateTime> = true;
template <typename T0, typename T1> constexpr bool UseLeftDecimal = false; template <typename T0, typename T1> constexpr bool UseLeftDecimal = false;
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true; template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> = true;
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true; template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal64>> = true;
template <> constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true; template <> inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal64>, DataTypeDecimal<Decimal32>> = true;
template <typename T> using DataTypeFromFieldType = std::conditional_t<std::is_same_v<T, NumberTraits::Error>, InvalidType, DataTypeNumber<T>>; template <typename T> using DataTypeFromFieldType = std::conditional_t<std::is_same_v<T, NumberTraits::Error>, InvalidType, DataTypeNumber<T>>;

4
dbms/src/Functions/FunctionMathBinaryFloat64.h
View File

@ -22,8 +22,8 @@
#pragma clang diagnostic ignored "-Wshift-negative-value" #pragma clang diagnostic ignored "-Wshift-negative-value"
#endif #endif
#include <vectorf128.h> // Y_IGNORE #include <vectorf128.h>
#include <vectormath_exp.h> // Y_IGNORE #include <vectormath_exp.h>
#ifdef __clang__ #ifdef __clang__
#pragma clang diagnostic pop #pragma clang diagnostic pop

6
dbms/src/Functions/FunctionMathUnaryFloat64.h
View File

@ -21,9 +21,9 @@
#pragma clang diagnostic ignored "-Wshift-negative-value" #pragma clang diagnostic ignored "-Wshift-negative-value"
#endif #endif
#include <vectorf128.h> // Y_IGNORE #include <vectorf128.h>
#include <vectormath_exp.h> // Y_IGNORE #include <vectormath_exp.h>
#include <vectormath_trig.h> // Y_IGNORE #include <vectormath_trig.h>
#ifdef __clang__ #ifdef __clang__
#pragma clang diagnostic pop #pragma clang diagnostic pop

Some files were not shown because too many files have changed in this diff Show More