thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-10-08 09:30:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:yandex/ClickHouse

Browse Source
This commit is contained in:
Ivan Blinkov 2018-09-12 15:37:16 +03:00
commit 9d01b5bd52
293 changed files with 12136 additions and 9669 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
CHANGELOG.draft.md
View File

@ -1,31 +0,0 @@
## RU
## ClickHouse release 18.10.3, 2018-08-13
### Новые возможности:
* поддержка межсерверной репликации по HTTPS
* MurmurHash
* ODBCDriver2 с поддержкой NULL-ов
* поддержка UUID в ключевых колонках (экспериментально)
### Улучшения:
* добавлена поддержка SETTINGS для движка Kafka
* поддежка пустых кусков после мержей в движках Summing, Collapsing and VersionedCollapsing
* удаление старых записей о полностью выполнившихся мутациях
* исправлена логика REPLACE PARTITION для движка RplicatedMergeTree
* добавлена системная таблица system.merge_tree_settings
* в системную таблицу system.tables добавлены столбцы зависимостей: dependencies_database и dependencies_table
* заменен аллокатор, теперь используется jemalloc вместо tcmalloc
* улучшена валидация connection string ODBC
* удалена поддержка CHECK TABLE для распределенных таблиц
* добавлены stateful тесты (пока без данных)
* добавлена опция конфига max_partition_size_to_drop
* добавлена настройка output_format_json_escape_slashes
* добавлена настройка max_fetch_partition_retries_count
* добавлена настройка prefer_localhost_replica
* добавлены libressl, unixodbc и mariadb-connector-c как сабмодули
### Исправление ошибок:
* #2786
* #2777
* #2795

156
CHANGELOG_RU.md
View File

@ -1,3 +1,151 @@
## ClickHouse release 18.12.13, 2018-09-10
### Новые возможности:
* Добавлен тип данных `DECIMAL(digits, scale)` (`Decimal32(scale)`, `Decimal64(scale)`, `Decimal128(scale)`). Возможность доступна под настройкой `allow_experimental_decimal_type`. [#2846](https://github.com/yandex/ClickHouse/pull/2846) [#2970](https://github.com/yandex/ClickHouse/pull/2970) [#3008](https://github.com/yandex/ClickHouse/pull/3008) [#3047](https://github.com/yandex/ClickHouse/pull/3047)
* Модификатор `WITH ROLLUP` для `GROUP BY` (также доступен синтаксис: `GROUP BY ROLLUP(...)`). [#2948](https://github.com/yandex/ClickHouse/pull/2948)
* В запросах с JOIN, звёздочка раскрывается в список столбцов всех таблиц, в соответствии со стандартом SQL. Вернуть старое поведение можно, выставив настройку (уровня пользователя) `asterisk_left_columns_only` в значение 1. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2787)
* Добавлена поддержка JOIN с табличной функцией. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2907)
* Автодополнение по нажатию Tab в clickhouse-client. [Sergey Shcherbin](https://github.com/yandex/ClickHouse/pull/2447)
* Нажатие Ctrl+C в clickhouse-client очищает запрос, если он был введён. [#2877](https://github.com/yandex/ClickHouse/pull/2877)
* Добавлена настройка `join_default_strictness` (значения `''`, `'any'`, `'all'`). Её использование позволяет не указывать `ANY` или `ALL` для `JOIN`. [#2982](https://github.com/yandex/ClickHouse/pull/2982)
* В каждой строчке лога сервера, относящейся к обработке запроса, выводится идентификатор запроса. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* Возможность получения логов выполнения запроса в clickhouse-client (настройка `send_logs_level`). При распределённой обработке запроса, логи отправляются каскадно со всех серверов. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* В таблицах `system.query_log` и `system.processes` (`SHOW PROCESSLIST`) появилась информация о всех изменённых настройках при выполнении запроса (вложенная структура данных `Settings`). Добавлена настройка `log_query_settings`. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* В таблицах `system.query_log` и `system.processes` появилась информация о номерах потоков, участвующих в исполнении запроса (столбец `thread_numbers`). [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* Добавлены счётчики `ProfileEvents`, измеряющие время, потраченное на чтение и запись по сети; чтение и запись на диск; количество сетевых ошибок; время потраченное на ожидании при ограничении сетевой полосы. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* Добавлены счётчики `ProfileEvents`, содержащие системные метрики из rusage (позволяющие получить информацию об использовании CPU в userspace и ядре, page faults, context switches) а также метрики taskstats (позволяющие получить информацию о времени ожидания IO, CPU, а также количество прочитанных и записанных данных с учётом и без учёта page cache). [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* Счётчики `ProfileEvents` учитываются не только глобально, но и на каждый запрос, а также на каждый поток выполнения запроса, что позволяет детально профилировать потребление ресурсов отдельными запросами. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* Добавлена таблица `system.query_thread_log`, содержащая информацию о каждом потоке выполнения запроса. Добавлена настройка `log_query_threads`. [#2482](https://github.com/yandex/ClickHouse/pull/2482)
* В таблицах `system.metrics` и `system.events` появилась встроенная документация. [#3016](https://github.com/yandex/ClickHouse/pull/3016)
* Добавлена функция `arrayEnumerateDense`. [Amos Bird](https://github.com/yandex/ClickHouse/pull/2975)
* Добавлены функции `arrayCumSumNonNegative` и `arrayDifference`. [Aleksey Studnev](https://github.com/yandex/ClickHouse/pull/2942)
* Добавлена агрегатная функция `retention`. [Sundy Li](https://github.com/yandex/ClickHouse/pull/2887)
* Возможность сложения (слияния) состояний агрегатных функций с помощью оператора плюс, а также умножения состояний агрегатных функций на целую неотрицательную константу. [#3062](https://github.com/yandex/ClickHouse/pull/3062) [#3034](https://github.com/yandex/ClickHouse/pull/3034)
* В таблицах семейства MergeTree добавлен виртуальный столбец `_partition_id`. [#3089](https://github.com/yandex/ClickHouse/pull/3089)
### Экспериментальные возможности:
* Добавлен тип данных `LowCardinality(T)`. Тип данных автоматически создаёт локальный словарь значений и позволяет обрабатывать данные без распаковки словаря. [#2830](https://github.com/yandex/ClickHouse/pull/2830)
* Добавлен кэш JIT-скомпилированных функций, а также счётчик числа использований перед компиляцией. Возможность JIT-компиляции выражений включается настройкой `compile_expressions`. [#2990](https://github.com/yandex/ClickHouse/pull/2990) [#3077](https://github.com/yandex/ClickHouse/pull/3077)
### Улучшения:
* Исправлена проблема неограниченного накопления лога репликации в случае наличия заброшенных реплик. Добавлен режим эффективного восстановления реплик после длительного отставания.
* Увеличена производительность при выполнении `GROUP BY` в случае, если есть несколько полей агрегации, одно из которых строковое, а другие - фиксированной длины.
* Увеличена производительность при использовании `PREWHERE` и при неявном переносе выражений в `PREWHERE`.
* Увеличена производительность парсинга текстовых форматов (`CSV`, `TSV`). [Amos Bird](https://github.com/yandex/ClickHouse/pull/2977) [#2980](https://github.com/yandex/ClickHouse/pull/2980)
* Увеличена производительность чтения строк и массивов в бинарных форматах. [Amos Bird](https://github.com/yandex/ClickHouse/pull/2955)
* Увеличена производительность и уменьшено потребление памяти в запросах к таблицам `system.tables` и `system.columns` в случае наличия очень большого количества таблиц на одном сервере. [#2953](https://github.com/yandex/ClickHouse/pull/2953)
* Исправлена проблема низкой производительности в случае наличия большого потока запросов, для которых возвращается ошибка (в `perf top` видна функция `_dl_addr`, при этом сервер использует мало CPU). [#2938](https://github.com/yandex/ClickHouse/pull/2938)
* Прокидывание условий внутрь View (при включенной настройке `enable_optimize_predicate_expression`) [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2907)
* Доработки недостающей функциональности для типа данных `UUID`. [#3074](https://github.com/yandex/ClickHouse/pull/3074) [#2985](https://github.com/yandex/ClickHouse/pull/2985)
* Тип данных `UUID` поддержан в словарях The-Alchemist. [#2822](https://github.com/yandex/ClickHouse/pull/2822)
* Функция `visitParamExtractRaw` корректно работает с вложенными структурами. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2974)
* При использовании настройки `input_format_skip_unknown_fields` корректно работает пропуск значений-объектов в формате `JSONEachRow`. [BlahGeek](https://github.com/yandex/ClickHouse/pull/2958)
* Для выражения `CASE` с условиями, появилась возможность не указывать `ELSE`, что эквивалентно `ELSE NULL`. [#2920](https://github.com/yandex/ClickHouse/pull/2920)
* Возможность конфигурирования operation timeout при работе с ZooKeeper. [urykhy](https://github.com/yandex/ClickHouse/pull/2971)
* Возможность указания смещения для `LIMIT n, m` в виде `LIMIT n OFFSET m`. [#2840](https://github.com/yandex/ClickHouse/pull/2840)
* Возможность использования синтаксиса `SELECT TOP n` в качестве альтернативы для `LIMIT`. [#2840](https://github.com/yandex/ClickHouse/pull/2840)
* Увеличен размер очереди записи в системные таблицы, что позволяет уменьшить количество ситуаций `SystemLog queue is full`.
* В агрегатной функции `windowFunnel` добавлена поддержка событий, подходящих под несколько условий. [Amos Bird](https://github.com/yandex/ClickHouse/pull/2801)
* Возможность использования дублирующихся столбцов в секции `USING` для `JOIN`. [#3006](https://github.com/yandex/ClickHouse/pull/3006)
* Для форматов `Pretty` введено ограничение выравнивания столбцов по ширине. Настройка `output_format_pretty_max_column_pad_width`. В случае более широкого значения, оно всё ещё будет выведено целиком, но остальные ячейки таблицы не будут излишне широкими. [#3003](https://github.com/yandex/ClickHouse/pull/3003)
* В табличной функции `odbc` добавлена возможность указания имени базы данных/схемы. [Amos Bird](https://github.com/yandex/ClickHouse/pull/2885)
* Добавлена возможность использования имени пользователя, заданного в конфигурационном файле `clickhouse-client`. [Vladimir Kozbin](https://github.com/yandex/ClickHouse/pull/2909)
* Счётчик `ZooKeeperExceptions` разделён на три счётчика `ZooKeeperUserExceptions`, `ZooKeeperHardwareExceptions`, `ZooKeeperOtherExceptions`.
* Запросы `ALTER DELETE` работают для материализованных представлений.
* Добавлена рандомизация во времени периодического запуска cleanup thread для таблиц типа `ReplicatedMergeTree`, чтобы избежать периодических всплесков нагрузки в случае очень большого количества таблиц типа `ReplicatedMergeTree`.
* Поддержка запроса `ATTACH TABLE ... ON CLUSTER`. [#3025](https://github.com/yandex/ClickHouse/pull/3025)
### Исправление ошибок:
* Исправлена ошибка в работе таблиц типа `Dictionary` (кидается исключение `Size of offsets doesn't match size of column` или `Unknown compression method`). Ошибка появилась в версии 18.10.3. [#2913](https://github.com/yandex/ClickHouse/issues/2913)
* Исправлена ошибка при мерже данных таблиц типа `CollapsingMergeTree`, если один из кусков данных пустой (такие куски, в свою очередь, образуются при слиянии или при `ALTER DELETE` в случае удаления всех данных), и для слияния был выбран алгоритм `vertical`. [#3049](https://github.com/yandex/ClickHouse/pull/3049)
* Исправлен race condition при `DROP` или `TRUNCATE` таблиц типа `Memory` при одновременном `SELECT`, который мог приводить к падениям сервера. Ошибка появилась в версии 1.1.54388. [#3038](https://github.com/yandex/ClickHouse/pull/3038)
* Исправлена возможность потери данных при вставке в `Replicated` таблицы в случае получения ошибки `Session expired` (потеря данных может быть обнаружена по метрике `ReplicatedDataLoss`). Ошибка возникла в версии 1.1.54378. [#2939](https://github.com/yandex/ClickHouse/pull/2939) [#2949](https://github.com/yandex/ClickHouse/pull/2949) [#2964](https://github.com/yandex/ClickHouse/pull/2964)
* Исправлен segfault при `JOIN ... ON`. [#3000](https://github.com/yandex/ClickHouse/pull/3000)
* Исправлена ошибка поиска имён столбцов в случае, если выражение `WHERE` состоит целиком из квалифицированного имени столбца, как например `WHERE table.column`. [#2994](https://github.com/yandex/ClickHouse/pull/2994)
* Исправлена ошибка вида "Not found column" при выполнении распределённых запросов в случае, если с удалённого сервера запрашивается единственный столбец, представляющий собой выражение IN с подзапросом. [#3087](https://github.com/yandex/ClickHouse/pull/3087)
* Исправлена ошибка `Block structure mismatch in UNION stream: different number of columns`, возникающая при распределённых запросах, если один из шардов локальный, а другой - нет, и если при этом срабатывает оптимизация переноса в `PREWHERE`. [#2226](https://github.com/yandex/ClickHouse/pull/2226) [#3037](https://github.com/yandex/ClickHouse/pull/3037) [#3055](https://github.com/yandex/ClickHouse/pull/3055) [#3065](https://github.com/yandex/ClickHouse/pull/3065) [#3073](https://github.com/yandex/ClickHouse/pull/3073) [#3090](https://github.com/yandex/ClickHouse/pull/3090) [#3093](https://github.com/yandex/ClickHouse/pull/3093)
* Исправлена работа функции `pointInPolygon` для некоторого случая невыпуклых полигонов. [#2910](https://github.com/yandex/ClickHouse/pull/2910)
* Исправлен некорректный результат при сравнении `nan` с целыми числами. [#3024](https://github.com/yandex/ClickHouse/pull/3024)
* Исправлена ошибка в библиотеке `zlib-ng`, которая могла приводить к segfault в редких случаях. [#2854](https://github.com/yandex/ClickHouse/pull/2854)
* Исправлена утечка памяти при вставке в таблицу со столбцами типа `AggregateFunction`, если состояние агрегатной функции нетривиальное (выделяет память отдельно), и если в одном запросе на вставку получается несколько маленьких блоков. [#3084](https://github.com/yandex/ClickHouse/pull/3084)
* Исправлен race condition при одновременном создании и удалении одной и той же таблицы типа `Buffer` или `MergeTree`.
* Исправлена возможность segfault при сравнении кортежей из некоторых нетривиальных типов, таких как, например, кортежей. [#2989](https://github.com/yandex/ClickHouse/pull/2989)
* Исправлена возможность segfault при выполнении некоторых запросов `ON CLUSTER`. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2960)
* Исправлена ошибка в функции `arrayDistinct` в случае `Nullable` элементов массивов. [#2845](https://github.com/yandex/ClickHouse/pull/2845) [#2937](https://github.com/yandex/ClickHouse/pull/2937)
* Возможность `enable_optimize_predicate_expression` корректно поддерживает случаи с `SELECT *`. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2929)
* Исправлена возможность segfault при переинициализации сессии с ZooKeeper. [#2917](https://github.com/yandex/ClickHouse/pull/2917)
* Исправлена возможность блокировки при взаимодействии с ZooKeeper.
* Исправлен некорректный код суммирования вложенных структур данных в `SummingMergeTree`.
* При выделении памяти для состояний агрегатных функций, корректно учитывается выравнивание, что позволяет использовать при реализации состояний агрегатных функций операции, для которых выравнивание является необходимым. [chenxing-xc](https://github.com/yandex/ClickHouse/pull/2808)
### Исправления безопасности:
* Безопасная работа с ODBC источниками данных. Взаимодействие с ODBC драйверами выполняется через отдельный процесс `clickhouse-odbc-bridge`. Ошибки в сторонних ODBC драйверах теперь не приводят к проблемам со стабильностью сервера или уязвимостям. [#2828](https://github.com/yandex/ClickHouse/pull/2828) [#2879](https://github.com/yandex/ClickHouse/pull/2879) [#2886](https://github.com/yandex/ClickHouse/pull/2886) [#2893](https://github.com/yandex/ClickHouse/pull/2893) [#2921](https://github.com/yandex/ClickHouse/pull/2921)
* Исправлена некорректная валидация пути к файлу в табличной функции `catBoostPool`. [#2894](https://github.com/yandex/ClickHouse/pull/2894)
* Содержимое системных таблиц (`tables`, `databases`, `parts`, `columns`, `parts_columns`, `merges`, `mutations`, `replicas`, `replication_queue`) фильтруется согласно конфигурации доступа к базам данных для пользователя (`allow_databases`) [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2856)
### Обратно несовместимые изменения:
* В запросах с JOIN, звёздочка раскрывается в список столбцов всех таблиц, в соответствии со стандартом SQL. Вернуть старое поведение можно, выставив настройку (уровня пользователя) `asterisk_left_columns_only` в значение 1.
### Изменения сборки:
* Добавлен покоммитный запуск большинства интеграционных тестов.
* Добавлен покоммитный запуск проверки стиля кода.
* Корректный выбор реализации `memcpy` при сборке на CentOS7 / Fedora. [Etienne Champetier](https://github.com/yandex/ClickHouse/pull/2912)
* При сборке с помощью clang добавлены некоторые warnings из `-Weverything` в дополнение к обычным `-Wall -Wextra -Werror`. [#2957](https://github.com/yandex/ClickHouse/pull/2957)
* При debug сборке используется debug вариант `jemalloc`.
* Абстрагирован интерфейс библиотеки для взаимодействия с ZooKeeper. [#2950](https://github.com/yandex/ClickHouse/pull/2950)
## ClickHouse release 18.10.3, 2018-08-13
### Новые возможности:
* Возможность использования HTTPS для репликации. [#2760](https://github.com/yandex/ClickHouse/pull/2760)
* Добавлены функции `murmurHash2_64`, `murmurHash3_32`, `murmurHash3_64`, `murmurHash3_128` в дополнение к имеющемуся `murmurHash2_32`. [#2791](https://github.com/yandex/ClickHouse/pull/2791)
* Поддержка Nullable типов в ODBC драйвере ClickHouse (формат вывода `ODBCDriver2`) [#2834](https://github.com/yandex/ClickHouse/pull/2834)
* Поддержка `UUID` в ключевых столбцах.
### Улучшения:
* Удаление кластеров без перезагрузки сервера при их удалении из конфигурационных файлов. [#2777](https://github.com/yandex/ClickHouse/pull/2777)
* Удаление внешних словарей без перезагрузки сервера при их удалении из конфигурационных файлов. [#2779](https://github.com/yandex/ClickHouse/pull/2779)
* Добавлена поддержка `SETTINGS` для движка таблиц `Kafka`. [Alexander Marshalov](https://github.com/yandex/ClickHouse/pull/2781)
* Доработки для типа данных `UUID` (не полностью) Šimon Podlipský. [#2618](https://github.com/yandex/ClickHouse/pull/2618)
* Поддежка пустых кусков после мержей в движках `SummingMergeTree`, `CollapsingMergeTree` and `VersionedCollapsingMergeTree`. [#2815](https://github.com/yandex/ClickHouse/pull/2815)
* Удаление старых записей о полностью выполнившихся мутациях (`ALTER DELETE`) [#2784](https://github.com/yandex/ClickHouse/pull/2784)
* Добавлена таблица `system.merge_tree_settings`. [Kirill Shvakov](https://github.com/yandex/ClickHouse/pull/2841)
* В таблицу `system.tables` добавлены столбцы зависимостей: `dependencies_database` и `dependencies_table`. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2851)
* Добавлена опция конфига `max_partition_size_to_drop`. [#2782](https://github.com/yandex/ClickHouse/pull/2782)
* Добавлена настройка `output_format_json_escape_forward_slashes`. [Alexander Bocharov](https://github.com/yandex/ClickHouse/pull/2812)
* Добавлена настройка `max_fetch_partition_retries_count`. [#2831](https://github.com/yandex/ClickHouse/pull/2831)
* Добавлена настройка `prefer_localhost_replica`, позволяющая отключить предпочтение локальной реплики и хождение на локальную реплику без межпроцессного взаимодействия. [#2832](https://github.com/yandex/ClickHouse/pull/2832)
* Агрегатная функция `quantileExact` возвращает `nan` в случае агрегации по пустому множеству `Float32`/`Float64` типов. [Sundy Li](https://github.com/yandex/ClickHouse/pull/2855)
### Исправление ошибок:
* Убрано излишнее экранирование параметров connection string для ODBC, котрое приводило к невозможности соединения. Ошибка возникла в версии 18.6.0.
* Исправлена логика обработки команд на `REPLACE PARTITION` в очереди репликации. Неправильная логика могла приводить к тому, что при наличии двух `REPLACE` одной и той же партиции, один из них оставался в очереди репликации и не мог выполниться. [#2814](https://github.com/yandex/ClickHouse/pull/2814)
* Исправлена ошибка при мерже, если все куски были пустыми (такие куски, в свою очередь, образуются при слиянии или при `ALTER DELETE` в случае удаления всех данных). Ошибка появилась в версии 18.1.0. [#2930](https://github.com/yandex/ClickHouse/pull/2930)
* Исправлена ошибка при параллельной записи в таблицы типа `Set` или `Join`. [Amos Bird](https://github.com/yandex/ClickHouse/pull/2823)
* Исправлена ошибка `Block structure mismatch in UNION stream: different number of columns`, возникающая при запросах с `UNION ALL` внутри подзапроса, в случае, если один из `SELECT` запросов содержит дублирующиеся имена столбцов. [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2094)
* Исправлена утечка памяти в случае исключения при соединении с MySQL сервером.
* Исправлен некорректный код возврата clickhouse-client в случае ошибочного запроса
* Исправлен некорректная работа materialized views, содержащих DISTINCT. [#2795](https://github.com/yandex/ClickHouse/issues/2795)
### Обратно несовместимые изменения
* Убрана поддержка запросов CHECK TABLE для Distributed таблиц.
### Изменения сборки:
* Заменен аллокатор, теперь используется `jemalloc` вместо `tcmalloc`. На некоторых сценариях ускорение достигает 20%. В то же время, существуют запросы, замедлившиеся до 20%. Потребление памяти на некоторых сценариях примерно на 10% меньше и более стабильно. При высококонкурентной нагрузке, потребление CPU в userspace и в system незначительно вырастает. [#2773](https://github.com/yandex/ClickHouse/pull/2773)
* Использование libressl из submodule. [#1983](https://github.com/yandex/ClickHouse/pull/1983) [#2807](https://github.com/yandex/ClickHouse/pull/2807)
* Использование unixodbc из submodule. [#2789](https://github.com/yandex/ClickHouse/pull/2789)
* Использование mariadb-connector-c из submodule. [#2785](https://github.com/yandex/ClickHouse/pull/2785)
* В репозиторий добавлены файлы функциональных тестов, рассчитывающих на наличие тестовых данных (пока без самих тестовых данных).
## ClickHouse release 18.6.0, 2018-08-02
### Новые возможности:
@ -13,14 +161,14 @@
## ClickHouse release 18.5.1, 2018-07-31
### Новые возможности:
* Добавлена функция хеширования `murmurHash2_32` [#2756](https://github.com/yandex/ClickHouse/pull/2756).
* Добавлена функция хеширования `murmurHash2_32`. [#2756](https://github.com/yandex/ClickHouse/pull/2756).
### Улучшения:
* Добавлена возможность указывать значения в конфигурационных файлах из переменных окружения с помощью атрибута `from_env` [#2741](https://github.com/yandex/ClickHouse/pull/2741).
* Добавлены регистронезависимые версии функций `coalesce`, `ifNull`, `nullIf` [#2752](https://github.com/yandex/ClickHouse/pull/2752).
* Добавлена возможность указывать значения в конфигурационных файлах из переменных окружения с помощью атрибута `from_env`. [#2741](https://github.com/yandex/ClickHouse/pull/2741).
* Добавлены регистронезависимые версии функций `coalesce`, `ifNull`, `nullIf`. [#2752](https://github.com/yandex/ClickHouse/pull/2752).
### Исправление ошибок:
* Исправлена возможная ошибка при старте реплики [#2759](https://github.com/yandex/ClickHouse/pull/2759).
* Исправлена возможная ошибка при старте реплики. [#2759](https://github.com/yandex/ClickHouse/pull/2759).
## ClickHouse release 18.4.0, 2018-07-28

7
PreLoad.cmake
View File

@ -8,7 +8,12 @@
# sudo apt-get install ninja-build
# CLion does not support Ninja
if (NOT ${CMAKE_COMMAND} MATCHES "clion")
# You can add your vote on CLion task tracker:
# https://youtrack.jetbrains.com/issue/CPP-2659
# https://youtrack.jetbrains.com/issue/CPP-870
string(TOLOWER "${CMAKE_COMMAND}" CMAKE_COMMAND_LOWER)
if (NOT ${CMAKE_COMMAND_LOWER} MATCHES "clion")
find_program(NINJA_PATH ninja)
if (NINJA_PATH)
set(CMAKE_GENERATOR "Ninja" CACHE INTERNAL "" FORCE)

2
cmake/find_capnp.cmake
View File

@ -29,7 +29,7 @@ if (ENABLE_CAPNP)
find_library (CAPNP capnp PATHS ${CAPNP_PATHS})
find_library (CAPNPC capnpc PATHS ${CAPNP_PATHS})
find_library (KJ kj PATHS ${CAPNP_PATHS})
set (CAPNP_LIBRARY ${CAPNP} ${CAPNPC} ${KJ})
set (CAPNP_LIBRARY ${CAPNPC} ${CAPNP} ${KJ})
find_path (CAPNP_INCLUDE_DIR NAMES capnp/schema-parser.h PATHS ${CAPNP_INCLUDE_PATHS})
endif ()

2
contrib/poco vendored

@ -1 +1 @@
Subproject commit 3df947389e6d9654919002797bdd86ed190b3963
Subproject commit d7a4383c4d85b51938b62ed5812bc0935245edb3

8
dbms/cmake/version.cmake
View File

@ -2,10 +2,10 @@
set(VERSION_REVISION 54407 CACHE STRING "")
set(VERSION_MAJOR 18 CACHE STRING "")
set(VERSION_MINOR 12 CACHE STRING "")
set(VERSION_PATCH 8 CACHE STRING "")
set(VERSION_GITHASH 199d8734f98fa7d04ebf2119431c5f56a7ed4e5a CACHE STRING "")
set(VERSION_DESCRIBE v18.12.8-testing CACHE STRING "")
set(VERSION_STRING 18.12.8 CACHE STRING "")
set(VERSION_PATCH 13 CACHE STRING "")
set(VERSION_GITHASH c6bb8a340a45474f1009af4eb665506e16808672 CACHE STRING "")
set(VERSION_DESCRIBE v18.12.13-testing CACHE STRING "")
set(VERSION_STRING 18.12.13 CACHE STRING "")
# end of autochange
set(VERSION_EXTRA "" CACHE STRING "")

2
dbms/programs/client/Suggest.h
View File

@ -85,6 +85,8 @@ private:
" UNION ALL "
"SELECT name FROM system.data_type_families"
" UNION ALL "
"SELECT name FROM system.settings"
" UNION ALL "
"SELECT concat(func.name, comb.name) FROM system.functions AS func CROSS JOIN system.aggregate_function_combinators AS comb WHERE is_aggregate";
/// The user may disable loading of databases, tables, columns by setting suggestion_limit to zero.

1
dbms/programs/copier/ClusterCopier.cpp
View File

@ -37,7 +37,6 @@
#include <Interpreters/Context.h>
#include <Interpreters/Cluster.h>
#include <Interpreters/InterpreterFactory.h>
#include <Interpreters/InterpreterInsertQuery.h>
#include <Interpreters/InterpreterExistsQuery.h>
#include <Interpreters/InterpreterShowCreateQuery.h>
#include <Interpreters/InterpreterDropQuery.h>

4
dbms/programs/obfuscator/Obfuscator.cpp
View File

@ -862,9 +862,9 @@ class ModelFactory
public:
ModelPtr get(const IDataType & data_type, UInt64 seed, MarkovModelParameters markov_model_params) const
{
if (data_type.isInteger())
if (isInteger(data_type))
{
if (data_type.isUnsignedInteger())
if (isUnsignedInteger(data_type))
return std::make_unique<UnsignedIntegerModel>(seed);
else
return std::make_unique<SignedIntegerModel>(seed);

4
dbms/programs/server/HTTPHandler.cpp
View File

@ -213,9 +213,7 @@ void HTTPHandler::processQuery(
Context context = server.context();
context.setGlobalContext(server.context());
/// It will forcibly detach query even if unexpected error ocurred and detachQuery() was not called
/// Normal detaching is happen in BlockIO callbacks
CurrentThread::QueryScope query_scope_holder(context);
CurrentThread::QueryScope query_scope(context);
LOG_TRACE(log, "Request URI: " << request.getURI());

11
dbms/programs/server/TCPHandler.cpp
View File

@ -130,6 +130,9 @@ void TCPHandler::runImpl()
Stopwatch watch;
state.reset();
/// Initialized later.
std::optional<CurrentThread::QueryScope> query_scope;
/** An exception during the execution of request (it must be sent over the network to the client).
* The client will be able to accept it, if it did not happen while sending another packet and the client has not disconnected yet.
*/
@ -152,7 +155,7 @@ void TCPHandler::runImpl()
if (!receivePacket())
continue;
CurrentThread::initializeQuery();
query_scope.emplace(query_context);
send_exception_with_stack_trace = query_context.getSettingsRef().calculate_text_stack_trace;
@ -197,6 +200,8 @@ void TCPHandler::runImpl()
sendLogs();
sendEndOfStream();
query_scope.reset();
state.reset();
}
catch (const Exception & e)
@ -265,9 +270,7 @@ void TCPHandler::runImpl()
try
{
/// It will forcibly detach query even if unexpected error ocсurred and detachQuery() was not called
CurrentThread::detachQueryIfNotDetached();
query_scope.reset();
state.reset();
}
catch (...)

2
dbms/src/AggregateFunctions/AggregateFunctionArray.cpp
View File

@ -1,6 +1,6 @@
#include <AggregateFunctions/AggregateFunctionArray.h>
#include <AggregateFunctions/AggregateFunctionCombinatorFactory.h>
#include <Common/typeid_cast.h>
namespace DB
{

3
dbms/src/AggregateFunctions/AggregateFunctionArray.h
View File

@ -1,7 +1,6 @@
#pragma once
#include <Columns/ColumnArray.h>
#include <Common/typeid_cast.h>
#include <DataTypes/DataTypeArray.h>
#include <AggregateFunctions/IAggregateFunction.h>
#include <IO/WriteHelpers.h>
@ -32,7 +31,7 @@ public:
: nested_func(nested_), num_arguments(arguments.size())
{
for (const auto & type : arguments)
if (!typeid_cast<const DataTypeArray *>(type.get()))
if (!isArray(type))
throw Exception("All arguments for aggregate function " + getName() + " must be arrays", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}

2
dbms/src/AggregateFunctions/AggregateFunctionFactory.cpp
View File

@ -126,7 +126,7 @@ AggregateFunctionPtr AggregateFunctionFactory::getImpl(
String nested_name = name.substr(0, name.size() - combinator->getName().size());
DataTypes nested_types = combinator->transformArguments(argument_types);
AggregateFunctionPtr nested_function = getImpl(nested_name, nested_types, parameters, recursion_level + 1);
AggregateFunctionPtr nested_function = get(nested_name, nested_types, parameters, recursion_level + 1);
return combinator->transformAggregateFunction(nested_function, argument_types, parameters);
}

1
dbms/src/AggregateFunctions/AggregateFunctionForEach.cpp
View File

@ -1,5 +1,6 @@
#include <AggregateFunctions/AggregateFunctionForEach.h>
#include <AggregateFunctions/AggregateFunctionCombinatorFactory.h>
#include <Common/typeid_cast.h>
namespace DB

3
dbms/src/AggregateFunctions/AggregateFunctionForEach.h
View File

@ -1,7 +1,6 @@
#pragma once
#include <Columns/ColumnArray.h>
#include <Common/typeid_cast.h>
#include <DataTypes/DataTypeArray.h>
#include <AggregateFunctions/IAggregateFunction.h>
@ -106,7 +105,7 @@ public:
throw Exception("Aggregate function " + getName() + " require at least one argument", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
for (const auto & type : arguments)
if (!typeid_cast<const DataTypeArray *>(type.get()))
if (!isArray(type))
throw Exception("All arguments for aggregate function " + getName() + " must be arrays", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}

8
dbms/src/AggregateFunctions/AggregateFunctionGroupArray.cpp
View File

@ -20,8 +20,9 @@ namespace
template <template <typename, typename> class AggregateFunctionTemplate, typename Data, typename ... TArgs>
static IAggregateFunction * createWithNumericOrTimeType(const IDataType & argument_type, TArgs && ... args)
{
if (typeid_cast<const DataTypeDate *>(&argument_type)) return new AggregateFunctionTemplate<UInt16, Data>(std::forward<TArgs>(args)...);
if (typeid_cast<const DataTypeDateTime *>(&argument_type)) return new AggregateFunctionTemplate<UInt32, Data>(std::forward<TArgs>(args)...);
WhichDataType which(argument_type);
if (which.idx == TypeIndex::Date) return new AggregateFunctionTemplate<UInt16, Data>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::DateTime) return new AggregateFunctionTemplate<UInt32, Data>(std::forward<TArgs>(args)...);
return createWithNumericType<AggregateFunctionTemplate, Data, TArgs...>(argument_type, std::forward<TArgs>(args)...);
}
@ -32,7 +33,8 @@ inline AggregateFunctionPtr createAggregateFunctionGroupArrayImpl(const DataType
if (auto res = createWithNumericOrTimeType<GroupArrayNumericImpl, has_limit>(*argument_type, argument_type, std::forward<TArgs>(args)...))
return AggregateFunctionPtr(res);
if (typeid_cast<const DataTypeString *>(argument_type.get()))
WhichDataType which(argument_type);
if (which.idx == TypeIndex::String)
return std::make_shared<GroupArrayGeneralListImpl<GroupArrayListNodeString, has_limit::value>>(argument_type, std::forward<TArgs>(args)...);
return std::make_shared<GroupArrayGeneralListImpl<GroupArrayListNodeGeneral, has_limit::value>>(argument_type, std::forward<TArgs>(args)...);

2
dbms/src/AggregateFunctions/AggregateFunctionGroupArrayInsertAt.h
View File

@ -79,7 +79,7 @@ public:
if (arguments.size() != 2)
throw Exception("Aggregate function " + getName() + " requires two arguments.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!arguments[1]->isUnsignedInteger())
if (!isUnsignedInteger(arguments[1]))
throw Exception("Second argument of aggregate function " + getName() + " must be integer.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
type = arguments.front();

5
dbms/src/AggregateFunctions/AggregateFunctionGroupUniqArray.cpp
View File

@ -26,8 +26,9 @@ class AggregateFunctionGroupUniqArrayDateTime : public AggregateFunctionGroupUni
static IAggregateFunction * createWithExtraTypes(const DataTypePtr & argument_type)
{
if (typeid_cast<const DataTypeDate *>(argument_type.get())) return new AggregateFunctionGroupUniqArrayDate;
else if (typeid_cast<const DataTypeDateTime *>(argument_type.get())) return new AggregateFunctionGroupUniqArrayDateTime;
WhichDataType which(argument_type);
if (which.idx == TypeIndex::Date) return new AggregateFunctionGroupUniqArrayDate;
else if (which.idx == TypeIndex::DateTime) return new AggregateFunctionGroupUniqArrayDateTime;
else
{
/// Check that we can use plain version of AggreagteFunctionGroupUniqArrayGeneric

2
dbms/src/AggregateFunctions/AggregateFunctionIf.cpp
View File

@ -21,7 +21,7 @@ public:
throw Exception("Incorrect number of arguments for aggregate function with " + getName() + " suffix",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!typeid_cast<const DataTypeUInt8 *>(arguments.back().get()))
if (!isUInt8(arguments.back()))
throw Exception("Illegal type " + arguments.back()->getName() + " of last argument for aggregate function with " + getName() + " suffix",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

3
dbms/src/AggregateFunctions/AggregateFunctionIf.h
View File

@ -2,7 +2,6 @@
#include <DataTypes/DataTypesNumber.h>
#include <Columns/ColumnsNumber.h>
#include <Common/typeid_cast.h>
#include <AggregateFunctions/IAggregateFunction.h>
@ -34,7 +33,7 @@ public:
if (num_arguments == 0)
throw Exception("Aggregate function " + getName() + " require at least one argument", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!typeid_cast<const DataTypeUInt8 *>(types.back().get()))
if (!isUInt8(types.back()))
throw Exception("Last argument for aggregate function " + getName() + " must be UInt8", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}

4
dbms/src/AggregateFunctions/AggregateFunctionMaxIntersections.h
View File

@ -61,10 +61,10 @@ public:
AggregateFunctionIntersectionsMax(AggregateFunctionIntersectionsKind kind_, const DataTypes & arguments)
: kind(kind_)
{
if (!arguments[0]->isNumber())
if (!isNumber(arguments[0]))
throw Exception{getName() + ": first argument must be represented by integer", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!arguments[1]->isNumber())
if (!isNumber(arguments[1]))
throw Exception{getName() + ": second argument must be represented by integer", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!arguments[0]->equals(*arguments[1]))

1
dbms/src/AggregateFunctions/AggregateFunctionMinMaxAny.h
View File

@ -6,7 +6,6 @@
#include <Columns/ColumnVector.h>
#include <Columns/ColumnString.h>
#include <DataTypes/IDataType.h>
#include <Common/typeid_cast.h>
#include <common/StringRef.h>
#include <AggregateFunctions/IAggregateFunction.h>

8
dbms/src/AggregateFunctions/AggregateFunctionQuantile.cpp
View File

@ -29,17 +29,17 @@ AggregateFunctionPtr createAggregateFunctionQuantile(const std::string & name, c
const DataTypePtr & argument_type = argument_types[0];
WhichDataType which(argument_type);
#define CREATE(TYPE) \
if (typeid_cast<const DataType ## TYPE *>(argument_type.get())) \
if (which.idx == TypeIndex::TYPE) \
return std::make_shared<AggregateFunctionQuantile<TYPE, Data<TYPE>, Name, have_second_arg, FloatReturnType, returns_many>>(argument_type, params);
FOR_NUMERIC_TYPES(CREATE)
#undef CREATE
if (typeid_cast<const DataTypeDate *>(argument_type.get()))
if (which.idx == TypeIndex::Date)
return std::make_shared<AggregateFunctionQuantile<
DataTypeDate::FieldType, Data<DataTypeDate::FieldType>, Name, have_second_arg, void, returns_many>>(argument_type, params);
if (typeid_cast<const DataTypeDateTime *>(argument_type.get()))
if (which.idx == TypeIndex::DateTime)
return std::make_shared<AggregateFunctionQuantile<
DataTypeDateTime::FieldType, Data<DataTypeDateTime::FieldType>, Name, have_second_arg, void, returns_many>>(argument_type, params);

3
dbms/src/AggregateFunctions/AggregateFunctionRetention.h
View File

@ -10,7 +10,6 @@
#include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h>
#include <Common/ArenaAllocator.h>
#include <Common/typeid_cast.h>
#include <ext/range.h>
#include <bitset>
@ -81,7 +80,7 @@ public:
for (const auto i : ext::range(0, arguments.size()))
{
auto cond_arg = arguments[i].get();
if (!typeid_cast<const DataTypeUInt8 *>(cond_arg))
if (!isUInt8(cond_arg))
throw Exception{"Illegal type " + cond_arg->getName() + " of argument " + toString(i) + " of aggregate function "
+ getName() + ", must be UInt8",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};

5
dbms/src/AggregateFunctions/AggregateFunctionSequenceMatch.h
View File

@ -6,7 +6,6 @@
#include <Columns/ColumnsNumber.h>
#include <ext/range.h>
#include <Common/PODArray.h>
#include <Common/typeid_cast.h>
#include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h>
#include <bitset>
@ -155,7 +154,7 @@ public:
ErrorCodes::TOO_MANY_ARGUMENTS_FOR_FUNCTION};
const auto time_arg = arguments.front().get();
if (!typeid_cast<const DataTypeDateTime *>(time_arg))
if (!WhichDataType(time_arg).isDateTime())
throw Exception{"Illegal type " + time_arg->getName() + " of first argument of aggregate function "
+ derived().getName() + ", must be DateTime",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -163,7 +162,7 @@ public:
for (const auto i : ext::range(1, arg_count))
{
const auto cond_arg = arguments[i].get();
if (!typeid_cast<const DataTypeUInt8 *>(cond_arg))
if (!isUInt8(cond_arg))
throw Exception{"Illegal type " + cond_arg->getName() + " of argument " + toString(i + 1) +
" of aggregate function " + derived().getName() + ", must be UInt8",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};

5
dbms/src/AggregateFunctions/AggregateFunctionTopK.cpp
View File

@ -37,8 +37,9 @@ class AggregateFunctionTopKDateTime : public AggregateFunctionTopK<DataTypeDateT
static IAggregateFunction * createWithExtraTypes(const DataTypePtr & argument_type, UInt64 threshold)
{
if (typeid_cast<const DataTypeDate *>(argument_type.get())) return new AggregateFunctionTopKDate(threshold);
if (typeid_cast<const DataTypeDateTime *>(argument_type.get())) return new AggregateFunctionTopKDateTime(threshold);
WhichDataType which(argument_type);
if (which.idx == TypeIndex::Date) return new AggregateFunctionTopKDate(threshold);
if (which.idx == TypeIndex::DateTime) return new AggregateFunctionTopKDateTime(threshold);
/// Check that we can use plain version of AggregateFunctionTopKGeneric
if (argument_type->isValueUnambiguouslyRepresentedInContiguousMemoryRegion())

22
dbms/src/AggregateFunctions/AggregateFunctionUniq.cpp
View File

@ -45,17 +45,18 @@ AggregateFunctionPtr createAggregateFunctionUniq(const std::string & name, const
AggregateFunctionPtr res(createWithNumericType<AggregateFunctionUniq, Data>(*argument_types[0]));
WhichDataType which(argument_type);
if (res)
return res;
else if (typeid_cast<const DataTypeDate *>(&argument_type))
else if (which.isDate())
return std::make_shared<AggregateFunctionUniq<DataTypeDate::FieldType, Data>>();
else if (typeid_cast<const DataTypeDateTime *>(&argument_type))
else if (which.isDateTime())
return std::make_shared<AggregateFunctionUniq<DataTypeDateTime::FieldType, Data>>();
else if (typeid_cast<const DataTypeString *>(&argument_type) || typeid_cast<const DataTypeFixedString *>(&argument_type))
else if (which.isStringOrFixedString())
return std::make_shared<AggregateFunctionUniq<String, Data>>();
else if (typeid_cast<const DataTypeUUID *>(&argument_type))
else if (which.isUUID())
return std::make_shared<AggregateFunctionUniq<DataTypeUUID::FieldType, Data>>();
else if (typeid_cast<const DataTypeTuple *>(&argument_type))
else if (which.isTuple())
{
if (use_exact_hash_function)
return std::make_shared<AggregateFunctionUniqVariadic<DataForVariadic, true, true>>(argument_types);
@ -90,17 +91,18 @@ AggregateFunctionPtr createAggregateFunctionUniq(const std::string & name, const
AggregateFunctionPtr res(createWithNumericType<AggregateFunctionUniq, Data>(*argument_types[0]));
WhichDataType which(argument_type);
if (res)
return res;
else if (typeid_cast<const DataTypeDate *>(&argument_type))
else if (which.isDate())
return std::make_shared<AggregateFunctionUniq<DataTypeDate::FieldType, Data<DataTypeDate::FieldType>>>();
else if (typeid_cast<const DataTypeDateTime *>(&argument_type))
else if (which.isDateTime())
return std::make_shared<AggregateFunctionUniq<DataTypeDateTime::FieldType, Data<DataTypeDateTime::FieldType>>>();
else if (typeid_cast<const DataTypeString *>(&argument_type) || typeid_cast<const DataTypeFixedString *>(&argument_type))
else if (which.isStringOrFixedString())
return std::make_shared<AggregateFunctionUniq<String, Data<String>>>();
else if (typeid_cast<const DataTypeUUID *>(&argument_type))
else if (which.isUUID())
return std::make_shared<AggregateFunctionUniq<DataTypeUUID::FieldType, Data<DataTypeUUID::FieldType>>>();
else if (typeid_cast<const DataTypeTuple *>(&argument_type))
else if (which.isTuple())
{
if (use_exact_hash_function)
return std::make_shared<AggregateFunctionUniqVariadic<DataForVariadic, true, true>>(argument_types);

11
dbms/src/AggregateFunctions/AggregateFunctionUniqUpTo.cpp
View File

@ -54,17 +54,18 @@ AggregateFunctionPtr createAggregateFunctionUniqUpTo(const std::string & name, c
AggregateFunctionPtr res(createWithNumericType<AggregateFunctionUniqUpTo>(*argument_types[0], threshold));
WhichDataType which(argument_type);
if (res)
return res;
else if (typeid_cast<const DataTypeDate *>(&argument_type))
else if (which.isDate())
return std::make_shared<AggregateFunctionUniqUpTo<DataTypeDate::FieldType>>(threshold);
else if (typeid_cast<const DataTypeDateTime *>(&argument_type))
else if (which.isDateTime())
return std::make_shared<AggregateFunctionUniqUpTo<DataTypeDateTime::FieldType>>(threshold);
else if (typeid_cast<const DataTypeString *>(&argument_type) || typeid_cast<const DataTypeFixedString*>(&argument_type))
else if (which.isStringOrFixedString())
return std::make_shared<AggregateFunctionUniqUpTo<String>>(threshold);
else if (typeid_cast<const DataTypeUUID *>(&argument_type))
else if (which.isUUID())
return std::make_shared<AggregateFunctionUniqUpTo<DataTypeUUID::FieldType>>(threshold);
else if (typeid_cast<const DataTypeTuple *>(&argument_type))
else if (which.isTuple())
{
if (use_exact_hash_function)
return std::make_shared<AggregateFunctionUniqUpToVariadic<true, true>>(argument_types, threshold);

5
dbms/src/AggregateFunctions/AggregateFunctionWindowFunnel.h
View File

@ -9,7 +9,6 @@
#include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h>
#include <Common/ArenaAllocator.h>
#include <Common/typeid_cast.h>
#include <ext/range.h>
#include <AggregateFunctions/IAggregateFunction.h>
@ -190,14 +189,14 @@ public:
AggregateFunctionWindowFunnel(const DataTypes & arguments, const Array & params)
{
const auto time_arg = arguments.front().get();
if (!typeid_cast<const DataTypeDateTime *>(time_arg) && !typeid_cast<const DataTypeUInt32 *>(time_arg))
if (!WhichDataType(time_arg).isDateTime() && !WhichDataType(time_arg).isUInt32())
throw Exception{"Illegal type " + time_arg->getName() + " of first argument of aggregate function " + getName()
+ ", must be DateTime or UInt32"};
for (const auto i : ext::range(1, arguments.size()))
{
auto cond_arg = arguments[i].get();
if (!typeid_cast<const DataTypeUInt8 *>(cond_arg))
if (!isUInt8(cond_arg))
throw Exception{"Illegal type " + cond_arg->getName() + " of argument " + toString(i + 1) + " of aggregate function "
+ getName() + ", must be UInt8",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};

83
dbms/src/AggregateFunctions/Helpers.h
View File

@ -1,17 +1,8 @@
#pragma once
#include <Common/typeid_cast.h>
#include <DataTypes/DataTypesNumber.h>
#include <DataTypes/DataTypeEnum.h>
#include <DataTypes/IDataType.h>
#include <AggregateFunctions/IAggregateFunction.h>
#define FOR_UNSIGNED_INTEGER_TYPES(M) \
M(UInt8) \
M(UInt16) \
M(UInt32) \
M(UInt64)
#define FOR_NUMERIC_TYPES(M) \
M(UInt8) \
M(UInt16) \
@ -24,20 +15,6 @@
M(Float32) \
M(Float64)
#define FOR_NUMERIC_TYPES_AND_ENUMS(M) \
M(UInt8, DataTypeUInt8) \
M(UInt16, DataTypeUInt16) \
M(UInt32, DataTypeUInt32) \
M(UInt64, DataTypeUInt64) \
M(Int8, DataTypeInt8) \
M(Int16, DataTypeInt16) \
M(Int32, DataTypeInt32) \
M(Int64, DataTypeInt64) \
M(Float32, DataTypeFloat32) \
M(Float64, DataTypeFloat64) \
M(UInt8, DataTypeEnum8) \
M(UInt16, DataTypeEnum16)
namespace DB
{
@ -46,30 +23,39 @@ namespace DB
template <template <typename, typename ... TArgs> class AggregateFunctionTemplate, typename ... TArgs>
static IAggregateFunction * createWithNumericType(const IDataType & argument_type, TArgs && ... args)
{
#define DISPATCH(FIELDTYPE, DATATYPE) \
if (typeid_cast<const DATATYPE *>(&argument_type)) return new AggregateFunctionTemplate<FIELDTYPE>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES_AND_ENUMS(DISPATCH)
WhichDataType which(argument_type);
#define DISPATCH(TYPE) \
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<TYPE>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (which.idx == TypeIndex::Enum8) return new AggregateFunctionTemplate<UInt8>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::Enum16) return new AggregateFunctionTemplate<UInt16>(std::forward<TArgs>(args)...);
return nullptr;
}
template <template <typename, typename> class AggregateFunctionTemplate, typename Data, typename ... TArgs>
static IAggregateFunction * createWithNumericType(const IDataType & argument_type, TArgs && ... args)
{
#define DISPATCH(FIELDTYPE, DATATYPE) \
if (typeid_cast<const DATATYPE *>(&argument_type)) return new AggregateFunctionTemplate<FIELDTYPE, Data>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES_AND_ENUMS(DISPATCH)
WhichDataType which(argument_type);
#define DISPATCH(TYPE) \
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<TYPE, Data>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (which.idx == TypeIndex::Enum8) return new AggregateFunctionTemplate<UInt8, Data>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::Enum16) return new AggregateFunctionTemplate<UInt16, Data>(std::forward<TArgs>(args)...);
return nullptr;
}
template <template <typename, typename> class AggregateFunctionTemplate, template <typename> class Data, typename ... TArgs>
static IAggregateFunction * createWithNumericType(const IDataType & argument_type, TArgs && ... args)
{
#define DISPATCH(FIELDTYPE, DATATYPE) \
if (typeid_cast<const DATATYPE *>(&argument_type)) return new AggregateFunctionTemplate<FIELDTYPE, Data<FIELDTYPE>>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES_AND_ENUMS(DISPATCH)
WhichDataType which(argument_type);
#define DISPATCH(TYPE) \
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<TYPE, Data<TYPE>>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (which.idx == TypeIndex::Enum8) return new AggregateFunctionTemplate<UInt8, Data<UInt8>>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::Enum16) return new AggregateFunctionTemplate<UInt16, Data<UInt16>>(std::forward<TArgs>(args)...);
return nullptr;
}
@ -77,10 +63,11 @@ static IAggregateFunction * createWithNumericType(const IDataType & argument_typ
template <template <typename, typename> class AggregateFunctionTemplate, template <typename> class Data, typename ... TArgs>
static IAggregateFunction * createWithUnsignedIntegerType(const IDataType & argument_type, TArgs && ... args)
{
#define DISPATCH(TYPE) \
if (typeid_cast<const DataType ## TYPE *>(&argument_type)) return new AggregateFunctionTemplate<TYPE, Data<TYPE>>(std::forward<TArgs>(args)...);
FOR_UNSIGNED_INTEGER_TYPES(DISPATCH)
#undef DISPATCH
WhichDataType which(argument_type);
if (which.idx == TypeIndex::UInt8) return new AggregateFunctionTemplate<UInt8, Data<UInt8>>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::UInt16) return new AggregateFunctionTemplate<UInt16, Data<UInt16>>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::UInt32) return new AggregateFunctionTemplate<UInt32, Data<UInt32>>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::UInt64) return new AggregateFunctionTemplate<UInt64, Data<UInt64>>(std::forward<TArgs>(args)...);
return nullptr;
}
@ -90,21 +77,29 @@ static IAggregateFunction * createWithUnsignedIntegerType(const IDataType & argu
template <typename FirstType, template <typename, typename> class AggregateFunctionTemplate, typename ... TArgs>
static IAggregateFunction * createWithTwoNumericTypesSecond(const IDataType & second_type, TArgs && ... args)
{
#define DISPATCH(FIELDTYPE, DATATYPE) \
if (typeid_cast<const DATATYPE *>(&second_type)) return new AggregateFunctionTemplate<FirstType, FIELDTYPE>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES_AND_ENUMS(DISPATCH)
WhichDataType which(second_type);
#define DISPATCH(TYPE) \
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<FirstType, TYPE>(std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (which.idx == TypeIndex::Enum8) return new AggregateFunctionTemplate<FirstType, UInt8>(std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::Enum16) return new AggregateFunctionTemplate<FirstType, UInt16>(std::forward<TArgs>(args)...);
return nullptr;
}
template <template <typename, typename> class AggregateFunctionTemplate, typename ... TArgs>
static IAggregateFunction * createWithTwoNumericTypes(const IDataType & first_type, const IDataType & second_type, TArgs && ... args)
{
#define DISPATCH(FIELDTYPE, DATATYPE) \
if (typeid_cast<const DATATYPE *>(&first_type)) \
return createWithTwoNumericTypesSecond<FIELDTYPE, AggregateFunctionTemplate>(second_type, std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES_AND_ENUMS(DISPATCH)
WhichDataType which(first_type);
#define DISPATCH(TYPE) \
if (which.idx == TypeIndex::TYPE) \
return createWithTwoNumericTypesSecond<TYPE, AggregateFunctionTemplate>(second_type, std::forward<TArgs>(args)...);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (which.idx == TypeIndex::Enum8)
return createWithTwoNumericTypesSecond<UInt8, AggregateFunctionTemplate>(second_type, std::forward<TArgs>(args)...);
if (which.idx == TypeIndex::Enum16)
return createWithTwoNumericTypesSecond<UInt16, AggregateFunctionTemplate>(second_type, std::forward<TArgs>(args)...);
return nullptr;
}

30
dbms/src/AggregateFunctions/HelpersMinMaxAny.h
View File

@ -4,11 +4,10 @@
#include <AggregateFunctions/AggregateFunctionArgMinMax.h>
#include <AggregateFunctions/FactoryHelpers.h>
#include <AggregateFunctions/Helpers.h>
#include <Common/typeid_cast.h>
#include <DataTypes/DataTypeDate.h>
#include <DataTypes/DataTypeDateTime.h>
#include <DataTypes/DataTypeString.h>
#include <DataTypes/DataTypesNumber.h>
namespace DB
@ -23,16 +22,17 @@ static IAggregateFunction * createAggregateFunctionSingleValue(const String & na
const DataTypePtr & argument_type = argument_types[0];
WhichDataType which(argument_type);
#define DISPATCH(TYPE) \
if (typeid_cast<const DataType ## TYPE *>(argument_type.get())) return new AggregateFunctionTemplate<Data<SingleValueDataFixed<TYPE>>>(argument_type);
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<Data<SingleValueDataFixed<TYPE>>>(argument_type);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (typeid_cast<const DataTypeDate *>(argument_type.get()))
if (which.idx == TypeIndex::Date)
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDate::FieldType>>>(argument_type);
if (typeid_cast<const DataTypeDateTime *>(argument_type.get()))
if (which.idx == TypeIndex::DateTime)
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDateTime::FieldType>>>(argument_type);
if (typeid_cast<const DataTypeString *>(argument_type.get()))
if (which.idx == TypeIndex::String)
return new AggregateFunctionTemplate<Data<SingleValueDataString>>(argument_type);
return new AggregateFunctionTemplate<Data<SingleValueDataGeneric>>(argument_type);
@ -43,17 +43,18 @@ static IAggregateFunction * createAggregateFunctionSingleValue(const String & na
template <template <typename> class MinMaxData, typename ResData>
static IAggregateFunction * createAggregateFunctionArgMinMaxSecond(const DataTypePtr & res_type, const DataTypePtr & val_type)
{
WhichDataType which(val_type);
#define DISPATCH(TYPE) \
if (typeid_cast<const DataType ## TYPE *>(val_type.get())) \
if (which.idx == TypeIndex::TYPE) \
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<TYPE>>>>(res_type, val_type);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (typeid_cast<const DataTypeDate *>(val_type.get()))
if (which.idx == TypeIndex::Date)
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDate::FieldType>>>>(res_type, val_type);
if (typeid_cast<const DataTypeDateTime*>(val_type.get()))
if (which.idx == TypeIndex::DateTime)
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDateTime::FieldType>>>>(res_type, val_type);
if (typeid_cast<const DataTypeString*>(val_type.get()))
if (which.idx == TypeIndex::String)
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataString>>>(res_type, val_type);
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataGeneric>>>(res_type, val_type);
@ -68,17 +69,18 @@ static IAggregateFunction * createAggregateFunctionArgMinMax(const String & name
const DataTypePtr & res_type = argument_types[0];
const DataTypePtr & val_type = argument_types[1];
WhichDataType which(res_type);
#define DISPATCH(TYPE) \
if (typeid_cast<const DataType ## TYPE *>(res_type.get())) \
if (which.idx == TypeIndex::TYPE) \
return createAggregateFunctionArgMinMaxSecond<MinMaxData, SingleValueDataFixed<TYPE>>(res_type, val_type);
FOR_NUMERIC_TYPES(DISPATCH)
#undef DISPATCH
if (typeid_cast<const DataTypeDate *>(res_type.get()))
if (which.idx == TypeIndex::Date)
return createAggregateFunctionArgMinMaxSecond<MinMaxData, SingleValueDataFixed<DataTypeDate::FieldType>>(res_type, val_type);
if (typeid_cast<const DataTypeDateTime*>(res_type.get()))
if (which.idx == TypeIndex::DateTime)
return createAggregateFunctionArgMinMaxSecond<MinMaxData, SingleValueDataFixed<DataTypeDateTime::FieldType>>(res_type, val_type);
if (typeid_cast<const DataTypeString*>(res_type.get()))
if (which.idx == TypeIndex::String)
return createAggregateFunctionArgMinMaxSecond<MinMaxData, SingleValueDataString>(res_type, val_type);
return createAggregateFunctionArgMinMaxSecond<MinMaxData, SingleValueDataGeneric>(res_type, val_type);

144
dbms/src/Columns/ColumnAggregateFunction.cpp
View File

@ -33,8 +33,6 @@ void ColumnAggregateFunction::addArena(ArenaPtr arena_)
MutableColumnPtr ColumnAggregateFunction::convertToValues() const
{
const IAggregateFunction * function = func.get();
/** If the aggregate function returns an unfinalized/unfinished state,
* then you just need to copy pointers to it and also shared ownership of data.
*
@ -65,33 +63,73 @@ MutableColumnPtr ColumnAggregateFunction::convertToValues() const
* AggregateFunction(quantileTiming(0.5), UInt64)
* into UInt16 - already finished result of `quantileTiming`.
*/
if (const AggregateFunctionState * function_state = typeid_cast<const AggregateFunctionState *>(function))
if (const AggregateFunctionState * function_state = typeid_cast<const AggregateFunctionState *>(func.get()))
{
auto res = createView();
res->set(function_state->getNestedFunction());
res->getData().assign(getData().begin(), getData().end());
res->data.assign(data.begin(), data.end());
return res;
}
MutableColumnPtr res = function->getReturnType()->createColumn();
res->reserve(getData().size());
MutableColumnPtr res = func->getReturnType()->createColumn();
res->reserve(data.size());
for (auto val : getData())
function->insertResultInto(val, *res);
for (auto val : data)
func->insertResultInto(val, *res);
return res;
}
void ColumnAggregateFunction::ensureOwnership()
{
if (src)
{
/// We must copy all data from src and take ownership.
size_t size = data.size();
Arena & arena = createOrGetArena();
size_t size_of_state = func->sizeOfData();
size_t align_of_state = func->alignOfData();
size_t rollback_pos = 0;
try
{
for (size_t i = 0; i < size; ++i)
{
ConstAggregateDataPtr old_place = data[i];
data[i] = arena.alignedAlloc(size_of_state, align_of_state);
func->create(data[i]);
++rollback_pos;
func->merge(data[i], old_place, &arena);
}
}
catch (...)
{
/// If we failed to take ownership, destroy all temporary data.
if (!func->hasTrivialDestructor())
for (size_t i = 0; i < rollback_pos; ++i)
func->destroy(data[i]);
throw;
}
/// Now we own all data.
src.reset();
}
}
void ColumnAggregateFunction::insertRangeFrom(const IColumn & from, size_t start, size_t length)
{
const ColumnAggregateFunction & from_concrete = static_cast<const ColumnAggregateFunction &>(from);
if (start + length > from_concrete.getData().size())
if (start + length > from_concrete.data.size())
throw Exception("Parameters start = " + toString(start) + ", length = " + toString(length)
+ " are out of bound in ColumnAggregateFunction::insertRangeFrom method"
" (data.size() = "
+ toString(from_concrete.getData().size())
+ toString(from_concrete.data.size())
+ ").",
ErrorCodes::PARAMETER_OUT_OF_BOUND);
@ -112,14 +150,14 @@ void ColumnAggregateFunction::insertRangeFrom(const IColumn & from, size_t start
size_t old_size = data.size();
data.resize(old_size + length);
memcpy(&data[old_size], &from_concrete.getData()[start], length * sizeof(data[0]));
memcpy(&data[old_size], &from_concrete.data[start], length * sizeof(data[0]));
}
}
ColumnPtr ColumnAggregateFunction::filter(const Filter & filter, ssize_t result_size_hint) const
{
size_t size = getData().size();
size_t size = data.size();
if (size != filter.size())
throw Exception("Size of filter doesn't match size of column.", ErrorCodes::SIZES_OF_COLUMNS_DOESNT_MATCH);
@ -127,14 +165,14 @@ ColumnPtr ColumnAggregateFunction::filter(const Filter & filter, ssize_t result_
return cloneEmpty();
auto res = createView();
auto & res_data = res->getData();
auto & res_data = res->data;
if (result_size_hint)
res_data.reserve(result_size_hint > 0 ? result_size_hint : size);
for (size_t i = 0; i < size; ++i)
if (filter[i])
res_data.push_back(getData()[i]);
res_data.push_back(data[i]);
/// To save RAM in case of too strong filtering.
if (res_data.size() * 2 < res_data.capacity())
@ -146,7 +184,7 @@ ColumnPtr ColumnAggregateFunction::filter(const Filter & filter, ssize_t result_
ColumnPtr ColumnAggregateFunction::permute(const Permutation & perm, size_t limit) const
{
size_t size = getData().size();
size_t size = data.size();
if (limit == 0)
limit = size;
@ -158,9 +196,9 @@ ColumnPtr ColumnAggregateFunction::permute(const Permutation & perm, size_t limi
auto res = createView();
res->getData().resize(limit);
res->data.resize(limit);
for (size_t i = 0; i < limit; ++i)
res->getData()[i] = getData()[perm[i]];
res->data[i] = data[perm[i]];
return res;
}
@ -175,9 +213,9 @@ ColumnPtr ColumnAggregateFunction::indexImpl(const PaddedPODArray<Type> & indexe
{
auto res = createView();
res->getData().resize(limit);
res->data.resize(limit);
for (size_t i = 0; i < limit; ++i)
res->getData()[i] = getData()[indexes[i]];
res->data[i] = data[indexes[i]];
return res;
}
@ -188,14 +226,14 @@ INSTANTIATE_INDEX_IMPL(ColumnAggregateFunction)
void ColumnAggregateFunction::updateHashWithValue(size_t n, SipHash & hash) const
{
WriteBufferFromOwnString wbuf;
func->serialize(getData()[n], wbuf);
func->serialize(data[n], wbuf);
hash.update(wbuf.str().c_str(), wbuf.str().size());
}
/// NOTE: Highly overestimates size of a column if it was produced in AggregatingBlockInputStream (it contains size of other columns)
size_t ColumnAggregateFunction::byteSize() const
{
size_t res = getData().size() * sizeof(getData()[0]);
size_t res = data.size() * sizeof(data[0]);
for (const auto & arena : arenas)
res += arena->size();
@ -207,7 +245,7 @@ size_t ColumnAggregateFunction::byteSize() const
/// Like byteSize(), highly overestimates size
size_t ColumnAggregateFunction::allocatedBytes() const
{
size_t res = getData().allocated_bytes();
size_t res = data.allocated_bytes();
for (const auto & arena : arenas)
res += arena->size();
@ -225,7 +263,7 @@ Field ColumnAggregateFunction::operator[](size_t n) const
Field field = String();
{
WriteBufferFromString buffer(field.get<String &>());
func->serialize(getData()[n], buffer);
func->serialize(data[n], buffer);
}
return field;
}
@ -235,18 +273,19 @@ void ColumnAggregateFunction::get(size_t n, Field & res) const
res = String();
{
WriteBufferFromString buffer(res.get<String &>());
func->serialize(getData()[n], buffer);
func->serialize(data[n], buffer);
}
}
StringRef ColumnAggregateFunction::getDataAt(size_t n) const
{
return StringRef(reinterpret_cast<const char *>(&getData()[n]), sizeof(getData()[n]));
return StringRef(reinterpret_cast<const char *>(&data[n]), sizeof(data[n]));
}
void ColumnAggregateFunction::insertData(const char * pos, size_t /*length*/)
{
getData().push_back(*reinterpret_cast<const AggregateDataPtr *>(pos));
ensureOwnership();
data.push_back(*reinterpret_cast<const AggregateDataPtr *>(pos));
}
void ColumnAggregateFunction::insertFrom(const IColumn & from, size_t n)
@ -254,24 +293,26 @@ void ColumnAggregateFunction::insertFrom(const IColumn & from, size_t n)
/// Must create new state of aggregate function and take ownership of it,
/// because ownership of states of aggregate function cannot be shared for individual rows,
/// (only as a whole, see comment above).
ensureOwnership();
insertDefault();
insertMergeFrom(from, n);
}
void ColumnAggregateFunction::insertFrom(ConstAggregateDataPtr place)
{
ensureOwnership();
insertDefault();
insertMergeFrom(place);
}
void ColumnAggregateFunction::insertMergeFrom(ConstAggregateDataPtr place)
{
func->merge(getData().back(), place, &createOrGetArena());
func->merge(data.back(), place, &createOrGetArena());
}
void ColumnAggregateFunction::insertMergeFrom(const IColumn & from, size_t n)
{
insertMergeFrom(static_cast<const ColumnAggregateFunction &>(from).getData()[n]);
insertMergeFrom(static_cast<const ColumnAggregateFunction &>(from).data[n]);
}
Arena & ColumnAggregateFunction::createOrGetArena()
@ -281,47 +322,54 @@ Arena & ColumnAggregateFunction::createOrGetArena()
return *arenas.back().get();
}
static void pushBackAndCreateState(ColumnAggregateFunction::Container & data, Arena & arena, IAggregateFunction * func)
{
data.push_back(arena.alignedAlloc(func->sizeOfData(), func->alignOfData()));
try
{
func->create(data.back());
}
catch (...)
{
data.pop_back();
throw;
}
}
void ColumnAggregateFunction::insert(const Field & x)
{
IAggregateFunction * function = func.get();
ensureOwnership();
Arena & arena = createOrGetArena();
getData().push_back(arena.alignedAlloc(function->sizeOfData(), function->alignOfData()));
function->create(getData().back());
pushBackAndCreateState(data, arena, func.get());
ReadBufferFromString read_buffer(x.get<const String &>());
function->deserialize(getData().back(), read_buffer, &arena);
func->deserialize(data.back(), read_buffer, &arena);
}
void ColumnAggregateFunction::insertDefault()
{
IAggregateFunction * function = func.get();
ensureOwnership();
Arena & arena = createOrGetArena();
getData().push_back(arena.alignedAlloc(function->sizeOfData(), function->alignOfData()));
function->create(getData().back());
pushBackAndCreateState(data, arena, func.get());
}
StringRef ColumnAggregateFunction::serializeValueIntoArena(size_t n, Arena & dst, const char *& begin) const
{
IAggregateFunction * function = func.get();
WriteBufferFromArena out(dst, begin);
function->serialize(getData()[n], out);
func->serialize(data[n], out);
return out.finish();
}
const char * ColumnAggregateFunction::deserializeAndInsertFromArena(const char * src_arena)
{
IAggregateFunction * function = func.get();
ensureOwnership();
/** Parameter "src_arena" points to Arena, from which we will deserialize the state.
* And "dst_arena" is another Arena, that aggregate function state will use to store its data.
*/
Arena & dst_arena = createOrGetArena();
getData().push_back(dst_arena.alignedAlloc(function->sizeOfData(), function->alignOfData()));
function->create(getData().back());
pushBackAndCreateState(data, dst_arena, func.get());
/** We will read from src_arena.
* There is no limit for reading - it is assumed, that we can read all that we need after src_arena pointer.
@ -331,7 +379,7 @@ const char * ColumnAggregateFunction::deserializeAndInsertFromArena(const char *
* Probably this will not work under UBSan.
*/
ReadBufferFromMemory read_buffer(src_arena, std::numeric_limits<char *>::max() - src_arena);
function->deserialize(getData().back(), read_buffer, &dst_arena);
func->deserialize(data.back(), read_buffer, &dst_arena);
return read_buffer.position();
}
@ -358,7 +406,7 @@ ColumnPtr ColumnAggregateFunction::replicate(const IColumn::Offsets & offsets) c
return cloneEmpty();
auto res = createView();
auto & res_data = res->getData();
auto & res_data = res->data;
res_data.reserve(offsets.back());
IColumn::Offset prev_offset = 0;
@ -399,7 +447,7 @@ MutableColumns ColumnAggregateFunction::scatter(IColumn::ColumnIndex num_columns
void ColumnAggregateFunction::getPermutation(bool /*reverse*/, size_t /*limit*/, int /*nan_direction_hint*/, IColumn::Permutation & res) const
{
size_t s = getData().size();
size_t s = data.size();
res.resize(s);
for (size_t i = 0; i < s; ++i)
res[i] = i;

5
dbms/src/Columns/ColumnAggregateFunction.h
View File

@ -74,6 +74,11 @@ private:
return res;
}
/// If we have another column as a source (owner of data), copy all data to ourself and reset source.
/// This is needed before inserting new elements, because we must own these elements (to destroy them in destructor),
/// but ownership of different elements cannot be mixed by different columns.
void ensureOwnership();
ColumnAggregateFunction(const AggregateFunctionPtr & func_)
: func(func_)
{

2
dbms/src/Columns/ColumnArray.h
View File

@ -137,7 +137,7 @@ private:
/** Non-constant arrays of constant values are quite rare.
* Most functions can not work with them, and does not create such columns as a result.
* An exception is the function `replicate`(see FunctionsMiscellaneous.h), which has service meaning for the implementation of lambda functions.
* An exception is the function `replicate` (see FunctionsMiscellaneous.h), which has service meaning for the implementation of lambda functions.
* Only for its sake is the implementation of the `replicate` method for ColumnArray(ColumnConst).
*/
ColumnPtr replicateConst(const Offsets & replicate_offsets) const;

1
dbms/src/Common/ErrorCodes.cpp
View File

@ -393,6 +393,7 @@ namespace ErrorCodes
extern const int REPLICA_STATUS_CHANGED = 416;
extern const int EXPECTED_ALL_OR_ANY = 417;
extern const int UNKNOWN_JOIN_STRICTNESS = 418;
extern const int CANNOT_ADD_DIFFERENT_AGGREGATE_STATES = 419;
extern const int KEEPER_EXCEPTION = 999;
extern const int POCO_EXCEPTION = 1000;

2
dbms/src/Core/Block.cpp
View File

@ -317,7 +317,7 @@ Columns Block::getColumns() const
}
MutableColumns Block::mutateColumns() const
MutableColumns Block::mutateColumns()
{
size_t num_columns = data.size();
MutableColumns columns(num_columns);

2
dbms/src/Core/Block.h
View File

@ -115,7 +115,7 @@ public:
MutableColumns cloneEmptyColumns() const;
/** Get columns from block for mutation. Columns in block will be nullptr. */
MutableColumns mutateColumns() const;
MutableColumns mutateColumns();
/** Replace columns in a block */
void setColumns(MutableColumns && columns);

2
dbms/src/Core/DecimalComparison.h
View File

@ -20,7 +20,7 @@ namespace ErrorCodes
}
///
inline bool allowDecimalComparison(const IDataType * left_type, const IDataType * right_type)
inline bool allowDecimalComparison(const DataTypePtr & left_type, const DataTypePtr & right_type)
{
if (isDecimal(left_type))
{

8
dbms/src/Core/Types.h
View File

@ -59,7 +59,7 @@ template <> struct TypeName<String> { static const char * get() { return "Strin
enum class TypeIndex
{
None = 0,
Nothing = 0,
UInt8,
UInt16,
UInt32,
@ -84,6 +84,12 @@ enum class TypeIndex
UUID,
Array,
Tuple,
Set,
Interval,
Nullable,
Function,
AggregateFunction,
LowCardinality,
};
template <typename T> struct TypeId;

17
dbms/src/DataStreams/ColumnGathererStream.cpp
View File

@ -100,15 +100,18 @@ void ColumnGathererStream::readSuffixImpl()
return;
double seconds = profile_info.total_stopwatch.elapsedSeconds();
std::stringstream speed;
if (seconds)
speed << ", " << profile_info.rows / seconds << " rows/sec., "
<< profile_info.bytes / 1048576.0 / seconds << " MiB/sec.";
LOG_TRACE(log, std::fixed << std::setprecision(2)
std::stringstream message;
message << std::fixed << std::setprecision(2)
<< "Gathered column " << column_name
<< " (" << static_cast<double>(profile_info.bytes) / profile_info.rows << " bytes/elem.)"
<< " in " << seconds << " sec."
<< speed.str());
<< " in " << seconds << " sec.";
if (seconds)
message << ", " << profile_info.rows / seconds << " rows/sec., "
<< profile_info.bytes / 1048576.0 / seconds << " MiB/sec.";
LOG_TRACE(log, message.str());
}
}

23
dbms/src/DataStreams/RollupBlockInputStream.cpp
View File

@ -1,4 +1,5 @@
#include <DataStreams/RollupBlockInputStream.h>
#include <DataStreams/finalizeBlock.h>
#include <DataTypes/DataTypeAggregateFunction.h>
#include <Columns/ColumnAggregateFunction.h>
#include <Columns/FilterDescription.h>
@ -7,22 +8,6 @@
namespace DB
{
static void finalize(Block & block)
{
for (size_t i = 0; i < block.columns(); ++i)
{
ColumnWithTypeAndName & current = block.getByPosition(i);
const DataTypeAggregateFunction * unfinalized_type = typeid_cast<const DataTypeAggregateFunction *>(current.type.get());
if (unfinalized_type)
{
current.type = unfinalized_type->getReturnType();
if (current.column)
current.column = typeid_cast<const ColumnAggregateFunction &>(*current.column).convertToValues();
}
}
}
RollupBlockInputStream::RollupBlockInputStream(
const BlockInputStreamPtr & input_, const Aggregator::Params & params_) : aggregator(params_),
keys(params_.keys)
@ -36,7 +21,7 @@ RollupBlockInputStream::RollupBlockInputStream(
Block RollupBlockInputStream::getHeader() const
{
Block res = children.at(0)->getHeader();
finalize(res);
finalizeBlock(res);
return res;
}
@ -58,7 +43,7 @@ Block RollupBlockInputStream::readImpl()
rollup_block = aggregator.mergeBlocks(rollup_blocks, false);
Block finalized = rollup_block;
finalize(finalized);
finalizeBlock(finalized);
return finalized;
}
@ -66,7 +51,7 @@ Block RollupBlockInputStream::readImpl()
current_key = keys.size() - 1;
rollup_block = block;
finalize(block);
finalizeBlock(block);
return block;
}

14
dbms/src/DataStreams/SquashingBlockInputStream.cpp
View File

@ -4,9 +4,9 @@
namespace DB
{
SquashingBlockInputStream::SquashingBlockInputStream(const BlockInputStreamPtr & src,
size_t min_block_size_rows, size_t min_block_size_bytes)
: transform(min_block_size_rows, min_block_size_bytes)
SquashingBlockInputStream::SquashingBlockInputStream(
const BlockInputStreamPtr & src, size_t min_block_size_rows, size_t min_block_size_bytes)
: header(src->getHeader()), transform(min_block_size_rows, min_block_size_bytes)
{
children.emplace_back(src);
}
@ -23,9 +23,13 @@ Block SquashingBlockInputStream::readImpl()
if (!block)
all_read = true;
SquashingTransform::Result result = transform.add(std::move(block));
SquashingTransform::Result result = transform.add(block.mutateColumns());
if (result.ready)
return result.block;
{
if (result.columns.empty())
return {};
return header.cloneWithColumns(std::move(result.columns));
}
}
}

3
dbms/src/DataStreams/SquashingBlockInputStream.h
View File

@ -16,12 +16,13 @@ public:
String getName() const override { return "Squashing"; }
Block getHeader() const override { return children.at(0)->getHeader(); }
Block getHeader() const override { return header; }
protected:
Block readImpl() override;
private:
Block header;
SquashingTransform transform;
bool all_read = false;
};

10
dbms/src/DataStreams/SquashingBlockOutputStream.cpp
View File

@ -5,16 +5,16 @@ namespace DB
{
SquashingBlockOutputStream::SquashingBlockOutputStream(BlockOutputStreamPtr & dst, size_t min_block_size_rows, size_t min_block_size_bytes)
: output(dst), transform(min_block_size_rows, min_block_size_bytes)
: output(dst), header(output->getHeader()), transform(min_block_size_rows, min_block_size_bytes)
{
}
void SquashingBlockOutputStream::write(const Block & block)
{
SquashingTransform::Result result = transform.add(Block(block));
SquashingTransform::Result result = transform.add(Block(block).mutateColumns());
if (result.ready)
output->write(result.block);
output->write(header.cloneWithColumns(std::move(result.columns)));
}
@ -26,8 +26,8 @@ void SquashingBlockOutputStream::finalize()
all_written = true;
SquashingTransform::Result result = transform.add({});
if (result.ready && result.block)
output->write(result.block);
if (result.ready && !result.columns.empty())
output->write(header.cloneWithColumns(std::move(result.columns)));
}

3
dbms/src/DataStreams/SquashingBlockOutputStream.h
View File

@ -14,7 +14,7 @@ class SquashingBlockOutputStream : public IBlockOutputStream
public:
SquashingBlockOutputStream(BlockOutputStreamPtr & dst, size_t min_block_size_rows, size_t min_block_size_bytes);
Block getHeader() const override { return output->getHeader(); }
Block getHeader() const override { return header; }
void write(const Block & block) override;
void flush() override;
@ -26,6 +26,7 @@ public:
private:
BlockOutputStreamPtr output;
Block header;
SquashingTransform transform;
bool all_written = false;

63
dbms/src/DataStreams/SquashingTransform.cpp
View File

@ -10,37 +10,38 @@ SquashingTransform::SquashingTransform(size_t min_block_size_rows, size_t min_bl
}
SquashingTransform::Result SquashingTransform::add(Block && block)
SquashingTransform::Result SquashingTransform::add(MutableColumns && columns)
{
if (!block)
return Result(std::move(accumulated_block));
/// End of input stream.
if (columns.empty())
return Result(std::move(accumulated_columns));
/// Just read block is alredy enough.
if (isEnoughSize(block.rows(), block.bytes()))
if (isEnoughSize(columns))
{
/// If no accumulated data, return just read block.
if (!accumulated_block)
return Result(std::move(block));
if (accumulated_columns.empty())
return Result(std::move(columns));
/// Return accumulated data (may be it has small size) and place new block to accumulated data.
accumulated_block.swap(block);
return Result(std::move(block));
/// Return accumulated data (maybe it has small size) and place new block to accumulated data.
columns.swap(accumulated_columns);
return Result(std::move(columns));
}
/// Accumulated block is already enough.
if (accumulated_block && isEnoughSize(accumulated_block.rows(), accumulated_block.bytes()))
if (!accumulated_columns.empty() && isEnoughSize(accumulated_columns))
{
/// Return accumulated data and place new block to accumulated data.
accumulated_block.swap(block);
return Result(std::move(block));
columns.swap(accumulated_columns);
return Result(std::move(columns));
}
append(std::move(block));
append(std::move(columns));
if (isEnoughSize(accumulated_block.rows(), accumulated_block.bytes()))
if (isEnoughSize(accumulated_columns))
{
Block res;
res.swap(accumulated_block);
MutableColumns res;
res.swap(accumulated_columns);
return Result(std::move(res));
}
@ -49,23 +50,35 @@ SquashingTransform::Result SquashingTransform::add(Block && block)
}
void SquashingTransform::append(Block && block)
void SquashingTransform::append(MutableColumns && columns)
{
if (!accumulated_block)
if (accumulated_columns.empty())
{
accumulated_block = std::move(block);
accumulated_columns = std::move(columns);
return;
}
size_t columns = block.columns();
size_t rows = block.rows();
for (size_t i = 0, size = columns.size(); i < size; ++i)
accumulated_columns[i]->insertRangeFrom(*columns[i], 0, columns[i]->size());
}
for (size_t i = 0; i < columns; ++i)
bool SquashingTransform::isEnoughSize(const MutableColumns & columns)
{
size_t rows = 0;
size_t bytes = 0;
for (const auto & column : columns)
{
MutableColumnPtr mutable_column = (*std::move(accumulated_block.getByPosition(i).column)).mutate();
mutable_column->insertRangeFrom(*block.getByPosition(i).column, 0, rows);
accumulated_block.getByPosition(i).column = std::move(mutable_column);
if (!rows)
rows = column->size();
else if (rows != column->size())
throw Exception("Sizes of columns doesn't match", ErrorCodes::SIZES_OF_COLUMNS_DOESNT_MATCH);
bytes += column->byteSize();
}
return isEnoughSize(rows, bytes);
}

11
dbms/src/DataStreams/SquashingTransform.h
View File

@ -29,25 +29,26 @@ public:
struct Result
{
bool ready = false;
Block block;
MutableColumns columns;
Result(bool ready_) : ready(ready_) {}
Result(Block && block_) : ready(true), block(std::move(block_)) {}
Result(MutableColumns && columns) : ready(true), columns(std::move(columns)) {}
};
/** Add next block and possibly returns squashed block.
* At end, you need to pass empty block. As the result for last (empty) block, you will get last Result with ready = true.
*/
Result add(Block && block);
Result add(MutableColumns && columns);
private:
size_t min_block_size_rows;
size_t min_block_size_bytes;
Block accumulated_block;
MutableColumns accumulated_columns;
void append(Block && block);
void append(MutableColumns && columns);
bool isEnoughSize(const MutableColumns & columns);
bool isEnoughSize(size_t rows, size_t bytes) const;
};

6
dbms/src/DataStreams/SummingSortedBlockInputStream.cpp
View File

@ -76,7 +76,7 @@ SummingSortedBlockInputStream::SummingSortedBlockInputStream(
}
else
{
bool is_agg_func = checkDataType<DataTypeAggregateFunction>(column.type.get());
bool is_agg_func = WhichDataType(column.type).isAggregateFunction();
if (!column.type->isSummable() && !is_agg_func)
{
column_numbers_not_to_aggregate.push_back(i);
@ -273,7 +273,7 @@ Block SummingSortedBlockInputStream::readImpl()
for (auto & desc : columns_to_aggregate)
{
// Wrap aggregated columns in a tuple to match function signature
if (!desc.is_agg_func_type && checkDataType<DataTypeTuple>(desc.function->getReturnType().get()))
if (!desc.is_agg_func_type && isTuple(desc.function->getReturnType()))
{
size_t tuple_size = desc.column_numbers.size();
MutableColumns tuple_columns(tuple_size);
@ -292,7 +292,7 @@ Block SummingSortedBlockInputStream::readImpl()
/// Place aggregation results into block.
for (auto & desc : columns_to_aggregate)
{
if (!desc.is_agg_func_type && checkDataType<DataTypeTuple>(desc.function->getReturnType().get()))
if (!desc.is_agg_func_type && isTuple(desc.function->getReturnType()))
{
/// Unpack tuple into block.
size_t tuple_size = desc.column_numbers.size();

24
dbms/src/DataStreams/TotalsHavingBlockInputStream.cpp
View File

@ -1,4 +1,5 @@
#include <DataStreams/TotalsHavingBlockInputStream.h>
#include <DataStreams/finalizeBlock.h>
#include <Interpreters/ExpressionActions.h>
#include <DataTypes/DataTypeAggregateFunction.h>
#include <Columns/ColumnAggregateFunction.h>
@ -53,23 +54,6 @@ TotalsHavingBlockInputStream::TotalsHavingBlockInputStream(
}
static void finalize(Block & block)
{
for (size_t i = 0; i < block.columns(); ++i)
{
ColumnWithTypeAndName & current = block.getByPosition(i);
const DataTypeAggregateFunction * unfinalized_type = typeid_cast<const DataTypeAggregateFunction *>(current.type.get());
if (unfinalized_type)
{
current.type = unfinalized_type->getReturnType();
if (current.column)
current.column = typeid_cast<const ColumnAggregateFunction &>(*current.column).convertToValues();
}
}
}
Block TotalsHavingBlockInputStream::getTotals()
{
if (!totals)
@ -87,7 +71,7 @@ Block TotalsHavingBlockInputStream::getTotals()
}
totals = children.at(0)->getHeader().cloneWithColumns(std::move(current_totals));
finalize(totals);
finalizeBlock(totals);
}
if (totals && expression)
@ -101,7 +85,7 @@ Block TotalsHavingBlockInputStream::getHeader() const
{
Block res = children.at(0)->getHeader();
if (final)
finalize(res);
finalizeBlock(res);
if (expression)
expression->execute(res);
return res;
@ -129,7 +113,7 @@ Block TotalsHavingBlockInputStream::readImpl()
finalized = block;
if (final)
finalize(finalized);
finalizeBlock(finalized);
total_keys += finalized.rows();

24
dbms/src/DataStreams/finalizeBlock.cpp Normal file
View File

@ -0,0 +1,24 @@
#include <DataStreams/finalizeBlock.h>
#include <DataTypes/DataTypeAggregateFunction.h>
#include <Columns/ColumnAggregateFunction.h>
#include <Common/typeid_cast.h>
namespace DB
{
void finalizeBlock(Block & block)
{
for (size_t i = 0; i < block.columns(); ++i)
{
ColumnWithTypeAndName & current = block.getByPosition(i);
const DataTypeAggregateFunction * unfinalized_type = typeid_cast<const DataTypeAggregateFunction *>(current.type.get());
if (unfinalized_type)
{
current.type = unfinalized_type->getReturnType();
if (current.column)
current.column = typeid_cast<const ColumnAggregateFunction &>(*current.column).convertToValues();
}
}
}
}

9
dbms/src/DataStreams/finalizeBlock.h Normal file
View File

@ -0,0 +1,9 @@
#pragma once
#include <Core/Block.h>
namespace DB
{
/// Converts aggregate function columns with non-finalized states to final values
void finalizeBlock(Block & block);
}

2
dbms/src/DataTypes/DataTypeAggregateFunction.h
View File

@ -30,8 +30,8 @@ public:
AggregateFunctionPtr getFunction() const { return function; }
std::string getName() const override;
const char * getFamilyName() const override { return "AggregateFunction"; }
TypeIndex getTypeId() const override { return TypeIndex::AggregateFunction; }
bool canBeInsideNullable() const override { return false; }

1
dbms/src/DataTypes/DataTypeDate.h
View File

@ -23,7 +23,6 @@ public:
void deserializeTextCSV(IColumn & column, ReadBuffer & istr, const FormatSettings & settings) const override;
bool canBeUsedAsVersion() const override { return true; }
bool isDateOrDateTime() const override { return true; }
bool canBeInsideNullable() const override { return true; }
bool equals(const IDataType & rhs) const override;

1
dbms/src/DataTypes/DataTypeDateTime.h
View File

@ -48,7 +48,6 @@ public:
void deserializeTextCSV(IColumn & column, ReadBuffer & istr, const FormatSettings & settings) const override;
bool canBeUsedAsVersion() const override { return true; }
bool isDateOrDateTime() const override { return true; }
bool canBeInsideNullable() const override { return true; }
bool equals(const IDataType & rhs) const override;

1
dbms/src/DataTypes/DataTypeEnum.h
View File

@ -30,7 +30,6 @@ public:
bool isValueUnambiguouslyRepresentedInContiguousMemoryRegion() const override { return true; }
bool haveMaximumSizeOfValue() const override { return true; }
bool isCategorial() const override { return true; }
bool isEnum() const override { return true; }
bool canBeInsideNullable() const override { return true; }
bool isComparable() const override { return true; }
};

2
dbms/src/DataTypes/DataTypeFixedString.h
View File

@ -77,7 +77,7 @@ public:
bool haveSubtypes() const override { return false; }
bool isComparable() const override { return true; }
bool isValueUnambiguouslyRepresentedInContiguousMemoryRegion() const override { return true; }
bool isFixedString() const override { return true; }
bool isValueUnambiguouslyRepresentedInFixedSizeContiguousMemoryRegion() const override { return true; }
bool haveMaximumSizeOfValue() const override { return true; }
size_t getSizeOfValueInMemory() const override { return n; }
bool isCategorial() const override { return true; }

1
dbms/src/DataTypes/DataTypeFunction.h
View File

@ -24,6 +24,7 @@ public:
std::string getName() const override;
const char * getFamilyName() const override { return "Function"; }
TypeIndex getTypeId() const override { return TypeIndex::Function; }
const DataTypes & getArgumentTypes() const
{

1
dbms/src/DataTypes/DataTypeInterval.h
View File

@ -55,6 +55,7 @@ public:
std::string getName() const override { return std::string("Interval") + kindToString(); }
const char * getFamilyName() const override { return "Interval"; }
TypeIndex getTypeId() const override { return TypeIndex::Interval; }
bool equals(const IDataType & rhs) const override;

1
dbms/src/DataTypes/DataTypeNothing.h
View File

@ -16,6 +16,7 @@ public:
static constexpr bool is_parametric = false;
const char * getFamilyName() const override { return "Nothing"; }
TypeIndex getTypeId() const override { return TypeIndex::Nothing; }
MutableColumnPtr createColumn() const override;

1
dbms/src/DataTypes/DataTypeNullable.h
View File

@ -16,6 +16,7 @@ public:
explicit DataTypeNullable(const DataTypePtr & nested_data_type_);
std::string getName() const override { return "Nullable(" + nested_data_type->getName() + ")"; }
const char * getFamilyName() const override { return "Nullable"; }
TypeIndex getTypeId() const override { return TypeIndex::Nullable; }
void enumerateStreams(const StreamCallback & callback, SubstreamPath & path) const override;

1
dbms/src/DataTypes/DataTypeSet.h
View File

@ -14,6 +14,7 @@ class DataTypeSet final : public IDataTypeDummy
public:
static constexpr bool is_parametric = true;
const char * getFamilyName() const override { return "Set"; }
TypeIndex getTypeId() const override { return TypeIndex::Set; }
bool equals(const IDataType & rhs) const override { return typeid(rhs) == typeid(*this); }
bool isParametric() const override { return true; }
};

1
dbms/src/DataTypes/DataTypeString.h
View File

@ -59,7 +59,6 @@ public:
bool isComparable() const override { return true; }
bool canBeComparedWithCollation() const override { return true; }
bool isValueUnambiguouslyRepresentedInContiguousMemoryRegion() const override { return true; }
bool isString() const override { return true; }
bool isCategorial() const override { return true; }
bool canBeInsideNullable() const override { return true; }
};

12
dbms/src/DataTypes/DataTypeWithDictionary.cpp
View File

@ -42,9 +42,9 @@ DataTypeWithDictionary::DataTypeWithDictionary(DataTypePtr dictionary_type_)
if (dictionary_type->isNullable())
inner_type = static_cast<const DataTypeNullable &>(*dictionary_type).getNestedType();
if (!inner_type->isStringOrFixedString()
&& !inner_type->isDateOrDateTime()
&& !inner_type->isNumber())
if (!isStringOrFixedString(inner_type)
&& !isDateOrDateTime(inner_type)
&& !isNumber(inner_type))
throw Exception("DataTypeWithDictionary is supported only for numbers, strings, Date or DateTime, but got "
+ dictionary_type->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
@ -769,15 +769,15 @@ MutableColumnUniquePtr DataTypeWithDictionary::createColumnUniqueImpl(const IDat
if (auto * nullable_type = typeid_cast<const DataTypeNullable *>(&keys_type))
type = nullable_type->getNestedType().get();
if (type->isString())
if (isString(type))
return creator((ColumnString *)(nullptr));
if (type->isFixedString())
if (isFixedString(type))
return creator((ColumnFixedString *)(nullptr));
if (typeid_cast<const DataTypeDate *>(type))
return creator((ColumnVector<UInt16> *)(nullptr));
if (typeid_cast<const DataTypeDateTime *>(type))
return creator((ColumnVector<UInt32> *)(nullptr));
if (type->isNumber())
if (isNumber(type))
{
MutableColumnUniquePtr column;
TypeListNumbers::forEach(CreateColumnVector(column, *type, creator));

8
dbms/src/DataTypes/DataTypeWithDictionary.h
View File

@ -20,6 +20,7 @@ public:
return "LowCardinality(" + dictionary_type->getName() + ")";
}
const char * getFamilyName() const override { return "LowCardinality"; }
TypeIndex getTypeId() const override { return TypeIndex::LowCardinality; }
void enumerateStreams(const StreamCallback & callback, SubstreamPath & path) const override;
@ -126,20 +127,13 @@ public:
bool isSummable() const override { return dictionary_type->isSummable(); }
bool canBeUsedInBitOperations() const override { return dictionary_type->canBeUsedInBitOperations(); }
bool canBeUsedInBooleanContext() const override { return dictionary_type->canBeUsedInBooleanContext(); }
bool isNumber() const override { return false; }
bool isInteger() const override { return false; }
bool isUnsignedInteger() const override { return false; }
bool isDateOrDateTime() const override { return false; }
bool isValueRepresentedByNumber() const override { return dictionary_type->isValueRepresentedByNumber(); }
bool isValueRepresentedByInteger() const override { return dictionary_type->isValueRepresentedByInteger(); }
bool isValueUnambiguouslyRepresentedInContiguousMemoryRegion() const override { return true; }
bool isString() const override { return false; }
bool isFixedString() const override { return false; }
bool haveMaximumSizeOfValue() const override { return dictionary_type->haveMaximumSizeOfValue(); }
size_t getMaximumSizeOfValueInMemory() const override { return dictionary_type->getMaximumSizeOfValueInMemory(); }
size_t getSizeOfValueInMemory() const override { return dictionary_type->getSizeOfValueInMemory(); }
bool isCategorial() const override { return false; }
bool isEnum() const override { return false; }
bool isNullable() const override { return false; }
bool onlyNull() const override { return false; }
bool withDictionary() const override { return true; }

14
dbms/src/DataTypes/DataTypesDecimal.h
View File

@ -260,13 +260,13 @@ inline UInt32 getDecimalScale(const IDataType & data_type)
///
template <typename DataType> constexpr bool IsDecimal = false;
template <> constexpr bool IsDecimal<DataTypeDecimal<Decimal32>> = true;
template <> constexpr bool IsDecimal<DataTypeDecimal<Decimal64>> = true;
template <> constexpr bool IsDecimal<DataTypeDecimal<Decimal128>> = true;
template <typename DataType> constexpr bool IsDataTypeDecimal = false;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal32>> = true;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal64>> = true;
template <> constexpr bool IsDataTypeDecimal<DataTypeDecimal<Decimal128>> = true;
template <typename FromDataType, typename ToDataType>
inline std::enable_if_t<IsDecimal<FromDataType> && IsDecimal<ToDataType>, typename ToDataType::FieldType>
inline std::enable_if_t<IsDataTypeDecimal<FromDataType> && IsDataTypeDecimal<ToDataType>, typename ToDataType::FieldType>
convertDecimals(const typename FromDataType::FieldType & value, UInt32 scale_from, UInt32 scale_to)
{
ToDataType type_to(ToDataType::maxPrecision(), scale_to);
@ -285,7 +285,7 @@ convertDecimals(const typename FromDataType::FieldType & value, UInt32 scale_fro
}
template <typename FromDataType, typename ToDataType>
inline std::enable_if_t<IsDecimal<FromDataType> && !IsDecimal<ToDataType>, typename ToDataType::FieldType>
inline std::enable_if_t<IsDataTypeDecimal<FromDataType> && !IsDataTypeDecimal<ToDataType>, typename ToDataType::FieldType>
convertFromDecimal(const typename FromDataType::FieldType & value, UInt32 scale [[maybe_unused]])
{
if (scale > FromDataType::maxPrecision())
@ -298,7 +298,7 @@ convertFromDecimal(const typename FromDataType::FieldType & value, UInt32 scale
}
template <typename FromDataType, typename ToDataType>
inline std::enable_if_t<!IsDecimal<FromDataType> && IsDecimal<ToDataType>, typename ToDataType::FieldType>
inline std::enable_if_t<!IsDataTypeDecimal<FromDataType> && IsDataTypeDecimal<ToDataType>, typename ToDataType::FieldType>
convertToDecimal(const typename FromDataType::FieldType & value, UInt32 scale [[maybe_unused]])
{
if (scale > ToDataType::maxPrecision())

15
dbms/src/DataTypes/DataTypesNumber.h
View File

@ -15,10 +15,7 @@ class DataTypeNumber final : public DataTypeNumberBase<T>
bool canBeUsedAsVersion() const override { return true; }
bool isSummable() const override { return true; }
bool canBeUsedInBitOperations() const override { return true; }
bool isUnsignedInteger() const override { return isInteger() && std::is_unsigned_v<T>; }
bool canBeUsedInBooleanContext() const override { return true; }
bool isNumber() const override { return true; }
bool isInteger() const override { return std::is_integral_v<T>; }
bool canBeInsideNullable() const override { return true; }
};
@ -33,4 +30,16 @@ using DataTypeInt64 = DataTypeNumber<Int64>;
using DataTypeFloat32 = DataTypeNumber<Float32>;
using DataTypeFloat64 = DataTypeNumber<Float64>;
template <typename DataType> constexpr bool IsDataTypeNumber = false;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt8>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt16>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<UInt64>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int8>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int16>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Int64>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float32>> = true;
template <> constexpr bool IsDataTypeNumber<DataTypeNumber<Float64>> = true;
}

100
dbms/src/DataTypes/IDataType.h
View File

@ -46,7 +46,7 @@ public:
virtual const char * getFamilyName() const = 0;
/// Unique type number or zero
virtual TypeIndex getTypeId() const { return TypeIndex::None; }
virtual TypeIndex getTypeId() const = 0;
/** Binary serialization for range of values in column - for writing to disk/network, etc.
*
@ -342,17 +342,6 @@ public:
*/
virtual bool canBeUsedInBooleanContext() const { return false; }
/** Integers, floats, not Nullable. Not Enums. Not Date/DateTime.
*/
virtual bool isNumber() const { return false; }
/** Integers. Not Nullable. Not Enums. Not Date/DateTime.
*/
virtual bool isInteger() const { return false; }
virtual bool isUnsignedInteger() const { return false; }
virtual bool isDateOrDateTime() const { return false; }
/** Numbers, Enums, Date, DateTime. Not nullable.
*/
virtual bool isValueRepresentedByNumber() const { return false; }
@ -376,13 +365,9 @@ public:
virtual bool isValueUnambiguouslyRepresentedInFixedSizeContiguousMemoryRegion() const
{
return isValueRepresentedByNumber() || isFixedString();
return isValueRepresentedByNumber();
}
virtual bool isString() const { return false; }
virtual bool isFixedString() const { return false; }
virtual bool isStringOrFixedString() const { return isString() || isFixedString(); }
/** Example: numbers, Date, DateTime, FixedString, Enum... Nullable and Tuple of such types.
* Counterexamples: String, Array.
* It's Ok to return false for AggregateFunction despite the fact that some of them have fixed size state.
@ -401,8 +386,6 @@ public:
*/
virtual bool isCategorial() const { return false; }
virtual bool isEnum() const { return false; }
virtual bool isNullable() const { return false; }
/** Is this type can represent only NULL value? (It also implies isNullable)
@ -423,11 +406,20 @@ public:
};
struct DataTypeExtractor
/// Some sugar to check data type of IDataType
struct WhichDataType
{
TypeIndex idx;
DataTypeExtractor(const IDataType * data_type)
WhichDataType(const IDataType & data_type)
: idx(data_type.getTypeId())
{}
WhichDataType(const IDataType * data_type)
: idx(data_type->getTypeId())
{}
WhichDataType(const DataTypePtr & data_type)
: idx(data_type->getTypeId())
{}
@ -437,6 +429,7 @@ struct DataTypeExtractor
bool isUInt64() const { return idx == TypeIndex::UInt64; }
bool isUInt128() const { return idx == TypeIndex::UInt128; }
bool isUInt() const { return isUInt8() || isUInt16() || isUInt32() || isUInt64() || isUInt128(); }
bool isNativeUInt() const { return isUInt8() || isUInt16() || isUInt32() || isUInt64(); }
bool isInt8() const { return idx == TypeIndex::Int8; }
bool isInt16() const { return idx == TypeIndex::Int16; }
@ -444,6 +437,7 @@ struct DataTypeExtractor
bool isInt64() const { return idx == TypeIndex::Int64; }
bool isInt128() const { return idx == TypeIndex::Int128; }
bool isInt() const { return isInt8() || isInt16() || isInt32() || isInt64() || isInt128(); }
bool isNativeInt() const { return isInt8() || isInt16() || isInt32() || isInt64(); }
bool isDecimal32() const { return idx == TypeIndex::Decimal32; }
bool isDecimal64() const { return idx == TypeIndex::Decimal64; }
@ -469,27 +463,75 @@ struct DataTypeExtractor
bool isUUID() const { return idx == TypeIndex::UUID; }
bool isArray() const { return idx == TypeIndex::Array; }
bool isTuple() const { return idx == TypeIndex::Tuple; }
bool isSet() const { return idx == TypeIndex::Set; }
bool isInterval() const { return idx == TypeIndex::Interval; }
bool isNothing() const { return idx == TypeIndex::Nothing; }
bool isNullable() const { return idx == TypeIndex::Nullable; }
bool isFunction() const { return idx == TypeIndex::Function; }
bool isAggregateFunction() const { return idx == TypeIndex::AggregateFunction; }
};
/// IDataType helpers (alternative for IDataType virtual methods)
/// IDataType helpers (alternative for IDataType virtual methods with single point of truth)
inline bool isEnum(const IDataType * data_type)
inline bool isDateOrDateTime(const DataTypePtr & data_type) { return WhichDataType(data_type).isDateOrDateTime(); }
inline bool isEnum(const DataTypePtr & data_type) { return WhichDataType(data_type).isEnum(); }
inline bool isDecimal(const DataTypePtr & data_type) { return WhichDataType(data_type).isDecimal(); }
inline bool isTuple(const DataTypePtr & data_type) { return WhichDataType(data_type).isTuple(); }
inline bool isArray(const DataTypePtr & data_type) { return WhichDataType(data_type).isArray(); }
template <typename T>
inline bool isUInt8(const T & data_type)
{
return DataTypeExtractor(data_type).isEnum();
return WhichDataType(data_type).isUInt8();
}
inline bool isDecimal(const IDataType * data_type)
template <typename T>
inline bool isUnsignedInteger(const T & data_type)
{
return DataTypeExtractor(data_type).isDecimal();
return WhichDataType(data_type).isUInt();
}
inline bool isNotDecimalButComparableToDecimal(const IDataType * data_type)
template <typename T>
inline bool isInteger(const T & data_type)
{
DataTypeExtractor which(data_type);
WhichDataType which(data_type);
return which.isInt() || which.isUInt();
}
inline bool isCompilableType(const IDataType * data_type)
template <typename T>
inline bool isNumber(const T & data_type)
{
WhichDataType which(data_type);
return which.isInt() || which.isUInt() || which.isFloat();
}
template <typename T>
inline bool isString(const T & data_type)
{
return WhichDataType(data_type).isString();
}
template <typename T>
inline bool isFixedString(const T & data_type)
{
return WhichDataType(data_type).isFixedString();
}
template <typename T>
inline bool isStringOrFixedString(const T & data_type)
{
return WhichDataType(data_type).isStringOrFixedString();
}
inline bool isNotDecimalButComparableToDecimal(const DataTypePtr & data_type)
{
WhichDataType which(data_type);
return which.isInt() || which.isUInt();
}
inline bool isCompilableType(const DataTypePtr & data_type)
{
return data_type->isValueRepresentedByNumber() && !isDecimal(data_type);
}

15
dbms/src/DataTypes/Native.h
View File

@ -66,6 +66,21 @@ static inline llvm::Type * toNativeType(llvm::IRBuilderBase & builder, const IDa
return nullptr;
}
static inline bool canBeNativeType(const IDataType & type)
{
if (auto * nullable = typeid_cast<const DataTypeNullable *>(&type))
return canBeNativeType(*nullable->getNestedType());
return typeIsEither<DataTypeInt8, DataTypeUInt8>(type)
|| typeIsEither<DataTypeInt16, DataTypeUInt16, DataTypeDate>(type)
|| typeIsEither<DataTypeInt32, DataTypeUInt32, DataTypeDateTime>(type)
|| typeIsEither<DataTypeInt64, DataTypeUInt64, DataTypeInterval>(type)
|| typeIsEither<DataTypeUUID>(type)
|| typeIsEither<DataTypeFloat32>(type)
|| typeIsEither<DataTypeFloat64>(type)
|| typeid_cast<const DataTypeFixedString *>(&type);
}
static inline llvm::Type * toNativeType(llvm::IRBuilderBase & builder, const DataTypePtr & type)
{
return toNativeType(builder, *type);

6
dbms/src/DataTypes/getMostSubtype.cpp
View File

@ -213,7 +213,7 @@ DataTypePtr getMostSubtype(const DataTypes & types, bool throw_if_result_is_noth
for (const auto & type : types)
{
if (type->isFixedString())
if (isFixedString(type))
{
have_string = true;
if (!fixed_string_type)
@ -221,7 +221,7 @@ DataTypePtr getMostSubtype(const DataTypes & types, bool throw_if_result_is_noth
else if (!type->equals(*fixed_string_type))
return getNothingOrThrow(" because some of them are FixedStrings with different length");
}
else if (type->isString())
else if (isString(type))
have_string = true;
else
all_strings = false;
@ -243,7 +243,7 @@ DataTypePtr getMostSubtype(const DataTypes & types, bool throw_if_result_is_noth
for (const auto & type : types)
{
if (type->isDateOrDateTime())
if (isDateOrDateTime(type))
have_date_or_datetime = true;
else
all_date_or_datetime = false;

2
dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp
View File

@ -289,7 +289,7 @@ void ComplexKeyHashedDictionary::updateData()
auto stream = source_ptr->loadUpdatedAll();
stream->readPrefix();
while (const auto block = stream->read())
while (Block block = stream->read())
{
const auto saved_key_column_ptrs = ext::map<Columns>(ext::range(0, keys_size), [&](const size_t key_idx)
{

2
dbms/src/Dictionaries/FlatDictionary.cpp
View File

@ -315,7 +315,7 @@ void FlatDictionary::updateData()
auto stream = source_ptr->loadUpdatedAll();
stream->readPrefix();
while (const auto block = stream->read())
while (Block block = stream->read())
{
const auto &saved_id_column = *saved_block->safeGetByPosition(0).column;
const auto &update_id_column = *block.safeGetByPosition(0).column;

2
dbms/src/Dictionaries/HashedDictionary.cpp
View File

@ -307,7 +307,7 @@ void HashedDictionary::updateData()
auto stream = source_ptr->loadUpdatedAll();
stream->readPrefix();
while (const auto block = stream->read())
while (Block block = stream->read())
{
const auto &saved_id_column = *saved_block->safeGetByPosition(0).column;
const auto &update_id_column = *block.safeGetByPosition(0).column;

2
dbms/src/Formats/CSVRowInputStream.cpp
View File

@ -227,7 +227,7 @@ bool CSVRowInputStream::parseRowAndPrintDiagnosticInfo(MutableColumns & columns,
if (curr_position < prev_position)
throw Exception("Logical error: parsing is non-deterministic.", ErrorCodes::LOGICAL_ERROR);
if (data_types[i]->isNumber() || data_types[i]->isDateOrDateTime())
if (isNumber(data_types[i]) || isDateOrDateTime(data_types[i]))
{
/// An empty string instead of a value.
if (curr_position == prev_position)

2
dbms/src/Formats/TabSeparatedRowInputStream.cpp
View File

@ -195,7 +195,7 @@ bool TabSeparatedRowInputStream::parseRowAndPrintDiagnosticInfo(MutableColumns &
if (curr_position < prev_position)
throw Exception("Logical error: parsing is non-deterministic.", ErrorCodes::LOGICAL_ERROR);
if (data_types[i]->isNumber() || data_types[i]->isDateOrDateTime())
if (isNumber(data_types[i]) || isDateOrDateTime(data_types[i]))
{
/// An empty string instead of a value.
if (curr_position == prev_position)

40
dbms/src/Functions/CMakeLists.txt
View File

@ -32,46 +32,6 @@ generate_function_register(Arithmetic
FunctionIntExp10
)
generate_function_register(Array
FunctionArray
FunctionArrayElement
FunctionHas
FunctionIndexOf
FunctionCountEqual
FunctionArrayEnumerate
FunctionArrayEnumerateUniq
FunctionArrayEnumerateDense
FunctionArrayUniq
FunctionArrayDistinct
FunctionEmptyArrayUInt8
FunctionEmptyArrayUInt16
FunctionEmptyArrayUInt32
FunctionEmptyArrayUInt64
FunctionEmptyArrayInt8
FunctionEmptyArrayInt16
FunctionEmptyArrayInt32
FunctionEmptyArrayInt64
FunctionEmptyArrayFloat32
FunctionEmptyArrayFloat64
FunctionEmptyArrayDate
FunctionEmptyArrayDateTime
FunctionEmptyArrayString
FunctionEmptyArrayToSingle
FunctionRange
FunctionArrayReduce
FunctionArrayReverse
FunctionArrayConcat
FunctionArraySlice
FunctionArrayPushBack
FunctionArrayPushFront
FunctionArrayPopBack
FunctionArrayPopFront
FunctionArrayHasAll
FunctionArrayHasAny
FunctionArrayIntersect
FunctionArrayResize
)
generate_function_register(Projection
FunctionOneOrZero
FunctionProject

59
dbms/src/Functions/EmptyImpl.h Normal file
View File

@ -0,0 +1,59 @@
#include <cstring>
#include <Columns/ColumnString.h>
#include <Functions/FunctionFactory.h>
namespace DB
{
namespace ErrorCodes
{
extern const int LOGICAL_ERROR;
}
template <bool negative = false>
struct EmptyImpl
{
/// If the function will return constant value for FixedString data type.
static constexpr auto is_fixed_to_constant = false;
static void vector(const ColumnString::Chars_t & /*data*/, const ColumnString::Offsets & offsets, PaddedPODArray<UInt8> & res)
{
size_t size = offsets.size();
ColumnString::Offset prev_offset = 1;
for (size_t i = 0; i < size; ++i)
{
res[i] = negative ^ (offsets[i] == prev_offset);
prev_offset = offsets[i] + 1;
}
}
/// Only make sense if is_fixed_to_constant.
static void vector_fixed_to_constant(const ColumnString::Chars_t & /*data*/, size_t /*n*/, UInt8 & /*res*/)
{
throw Exception("Logical error: 'vector_fixed_to_constant method' is called", ErrorCodes::LOGICAL_ERROR);
}
static void vector_fixed_to_vector(const ColumnString::Chars_t & data, size_t n, PaddedPODArray<UInt8> & res)
{
std::vector<char> empty_chars(n);
size_t size = data.size() / n;
for (size_t i = 0; i < size; ++i)
res[i] = negative ^ (0 == memcmp(&data[i * size], empty_chars.data(), n));
}
static void array(const ColumnString::Offsets & offsets, PaddedPODArray<UInt8> & res)
{
size_t size = offsets.size();
ColumnString::Offset prev_offset = 0;
for (size_t i = 0; i < size; ++i)
{
res[i] = negative ^ (offsets[i] == prev_offset);
prev_offset = offsets[i];
}
}
};
}

7
dbms/src/Functions/FunctionHelpers.h
View File

@ -20,13 +20,6 @@ const Type * checkAndGetDataType(const IDataType * data_type)
return typeid_cast<const Type *>(data_type);
}
template <typename Type>
bool checkDataType(const IDataType * data_type)
{
return checkAndGetDataType<Type>(data_type);
}
template <typename Type>
const Type * checkAndGetColumn(const IColumn * column)
{

88
dbms/src/Functions/FunctionNumericPredicate.h Normal file
View File

@ -0,0 +1,88 @@
#include <Functions/IFunction.h>
#include <Functions/FunctionHelpers.h>
#include <DataTypes/DataTypesNumber.h>
#include <Columns/ColumnsNumber.h>
#include <ext/range.h>
namespace DB
{
namespace ErrorCodes
{
extern const int ILLEGAL_COLUMN;
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
}
template <typename Impl>
class FunctionNumericPredicate : public IFunction
{
public:
static constexpr auto name = Impl::name;
static FunctionPtr create(const Context &)
{
return std::make_shared<FunctionNumericPredicate>();
}
String getName() const override
{
return name;
}
size_t getNumberOfArguments() const override
{
return 1;
}
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!isNumber(arguments.front()))
throw Exception{"Argument for function " + getName() + " must be number", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
return std::make_shared<DataTypeUInt8>();
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
const auto in = block.getByPosition(arguments.front()).column.get();
if ( !execute<UInt8>(block, in, result)
&& !execute<UInt16>(block, in, result)
&& !execute<UInt32>(block, in, result)
&& !execute<UInt64>(block, in, result)
&& !execute<Int8>(block, in, result)
&& !execute<Int16>(block, in, result)
&& !execute<Int32>(block, in, result)
&& !execute<Int64>(block, in, result)
&& !execute<Float32>(block, in, result)
&& !execute<Float64>(block, in, result))
throw Exception{"Illegal column " + in->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN};
}
template <typename T>
bool execute(Block & block, const IColumn * in_untyped, const size_t result)
{
if (const auto in = checkAndGetColumn<ColumnVector<T>>(in_untyped))
{
const auto size = in->size();
auto out = ColumnUInt8::create(size);
const auto & in_data = in->getData();
auto & out_data = out->getData();
for (const auto i : ext::range(0, size))
out_data[i] = Impl::execute(in_data[i]);
block.getByPosition(result).column = std::move(out);
return true;
}
return false;
}
};
}

139
dbms/src/Functions/FunctionStartsEndsWith.h Normal file
View File

@ -0,0 +1,139 @@
#include <Functions/IFunction.h>
#include <Functions/FunctionHelpers.h>
#include <Functions/GatherUtils/GatherUtils.h>
#include <Functions/GatherUtils/Sources.h>
#include <DataTypes/DataTypeString.h>
#include <DataTypes/DataTypesNumber.h>
#include <Columns/ColumnString.h>
namespace DB
{
using namespace GatherUtils;
namespace ErrorCodes
{
extern const int ILLEGAL_COLUMN;
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
}
struct NameStartsWith
{
static constexpr auto name = "startsWith";
};
struct NameEndsWith
{
static constexpr auto name = "endsWith";
};
template <typename Name>
class FunctionStartsEndsWith : public IFunction
{
public:
static constexpr auto name = Name::name;
static FunctionPtr create(const Context &)
{
return std::make_shared<FunctionStartsEndsWith>();
}
String getName() const override
{
return name;
}
size_t getNumberOfArguments() const override
{
return 2;
}
bool useDefaultImplementationForConstants() const override
{
return true;
}
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!isStringOrFixedString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!isStringOrFixedString(arguments[1]))
throw Exception("Illegal type " + arguments[1]->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return std::make_shared<DataTypeUInt8>();
}
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
{
const IColumn * haystack_column = block.getByPosition(arguments[0]).column.get();
const IColumn * needle_column = block.getByPosition(arguments[1]).column.get();
auto col_res = ColumnVector<UInt8>::create();
typename ColumnVector<UInt8>::Container & vec_res = col_res->getData();
vec_res.resize(input_rows_count);
if (const ColumnString * haystack = checkAndGetColumn<ColumnString>(haystack_column))
dispatch<StringSource>(StringSource(*haystack), needle_column, vec_res);
else if (const ColumnFixedString * haystack = checkAndGetColumn<ColumnFixedString>(haystack_column))
dispatch<FixedStringSource>(FixedStringSource(*haystack), needle_column, vec_res);
else if (const ColumnConst * haystack = checkAndGetColumnConst<ColumnString>(haystack_column))
dispatch<ConstSource<StringSource>>(ConstSource<StringSource>(*haystack), needle_column, vec_res);
else if (const ColumnConst * haystack = checkAndGetColumnConst<ColumnFixedString>(haystack_column))
dispatch<ConstSource<FixedStringSource>>(ConstSource<FixedStringSource>(*haystack), needle_column, vec_res);
else
throw Exception("Illegal combination of columns as arguments of function " + getName(), ErrorCodes::ILLEGAL_COLUMN);
block.getByPosition(result).column = std::move(col_res);
}
private:
template <typename HaystackSource>
void dispatch(HaystackSource haystack_source, const IColumn * needle_column, PaddedPODArray<UInt8> & res_data) const
{
if (const ColumnString * needle = checkAndGetColumn<ColumnString>(needle_column))
execute<HaystackSource, StringSource>(haystack_source, StringSource(*needle), res_data);
else if (const ColumnFixedString * needle = checkAndGetColumn<ColumnFixedString>(needle_column))
execute<HaystackSource, FixedStringSource>(haystack_source, FixedStringSource(*needle), res_data);
else if (const ColumnConst * needle = checkAndGetColumnConst<ColumnString>(needle_column))
execute<HaystackSource, ConstSource<StringSource>>(haystack_source, ConstSource<StringSource>(*needle), res_data);
else if (const ColumnConst * needle = checkAndGetColumnConst<ColumnFixedString>(needle_column))
execute<HaystackSource, ConstSource<FixedStringSource>>(haystack_source, ConstSource<FixedStringSource>(*needle), res_data);
else
throw Exception("Illegal combination of columns as arguments of function " + getName(), ErrorCodes::ILLEGAL_COLUMN);
}
template <typename HaystackSource, typename NeedleSource>
static void execute(HaystackSource haystack_source, NeedleSource needle_source, PaddedPODArray<UInt8> & res_data)
{
size_t row_num = 0;
while (!haystack_source.isEnd())
{
auto haystack = haystack_source.getWhole();
auto needle = needle_source.getWhole();
if (needle.size > haystack.size)
{
res_data[row_num] = false;
}
else
{
if constexpr (std::is_same_v<Name, NameStartsWith>)
{
res_data[row_num] = StringRef(haystack.data, needle.size) == StringRef(needle.data, needle.size);
}
else /// endsWith
{
res_data[row_num] = StringRef(haystack.data + haystack.size - needle.size, needle.size) == StringRef(needle.data, needle.size);
}
}
haystack_source.next();
needle_source.next();
++row_num;
}
}
};
}

101
dbms/src/Functions/FunctionStringOrArrayToT.h Normal file
View File

@ -0,0 +1,101 @@
#include <DataTypes/DataTypeString.h>
#include <DataTypes/DataTypesNumber.h>
#include <Functions/IFunction.h>
#include <Functions/FunctionHelpers.h>
#include <Columns/ColumnVector.h>
#include <Columns/ColumnString.h>
#include <Columns/ColumnFixedString.h>
#include <Columns/ColumnArray.h>
namespace DB
{
namespace ErrorCodes
{
extern const int ILLEGAL_COLUMN;
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
}
template <typename Impl, typename Name, typename ResultType>
class FunctionStringOrArrayToT : public IFunction
{
public:
static constexpr auto name = Name::name;
static FunctionPtr create(const Context &)
{
return std::make_shared<FunctionStringOrArrayToT>();
}
String getName() const override
{
return name;
}
size_t getNumberOfArguments() const override
{
return 1;
}
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!isStringOrFixedString(arguments[0])
&& !isArray(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return std::make_shared<DataTypeNumber<ResultType>>();
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
const ColumnPtr column = block.getByPosition(arguments[0]).column;
if (const ColumnString * col = checkAndGetColumn<ColumnString>(column.get()))
{
auto col_res = ColumnVector<ResultType>::create();
typename ColumnVector<ResultType>::Container & vec_res = col_res->getData();
vec_res.resize(col->size());
Impl::vector(col->getChars(), col->getOffsets(), vec_res);
block.getByPosition(result).column = std::move(col_res);
}
else if (const ColumnFixedString * col = checkAndGetColumn<ColumnFixedString>(column.get()))
{
if (Impl::is_fixed_to_constant)
{
ResultType res = 0;
Impl::vector_fixed_to_constant(col->getChars(), col->getN(), res);
block.getByPosition(result).column = block.getByPosition(result).type->createColumnConst(col->size(), toField(res));
}
else
{
auto col_res = ColumnVector<ResultType>::create();
typename ColumnVector<ResultType>::Container & vec_res = col_res->getData();
vec_res.resize(col->size());
Impl::vector_fixed_to_vector(col->getChars(), col->getN(), vec_res);
block.getByPosition(result).column = std::move(col_res);
}
}
else if (const ColumnArray * col = checkAndGetColumn<ColumnArray>(column.get()))
{
auto col_res = ColumnVector<ResultType>::create();
typename ColumnVector<ResultType>::Container & vec_res = col_res->getData();
vec_res.resize(col->size());
Impl::array(col->getOffsets(), vec_res);
block.getByPosition(result).column = std::move(col_res);
}
else
throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_COLUMN);
}
};
}

76
dbms/src/Functions/FunctionStringToString.h Normal file
View File

@ -0,0 +1,76 @@
#include <DataTypes/DataTypeString.h>
#include <Columns/ColumnString.h>
#include <Columns/ColumnFixedString.h>
#include <Functions/FunctionHelpers.h>
#include <Functions/IFunction.h>
namespace DB
{
namespace ErrorCodes
{
extern const int ILLEGAL_COLUMN;
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
}
template <typename Impl, typename Name, bool is_injective = false>
class FunctionStringToString : public IFunction
{
public:
static constexpr auto name = Name::name;
static FunctionPtr create(const Context &)
{
return std::make_shared<FunctionStringToString>();
}
String getName() const override
{
return name;
}
size_t getNumberOfArguments() const override
{
return 1;
}
bool isInjective(const Block &) override
{
return is_injective;
}
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!isStringOrFixedString(arguments[0]))
throw Exception(
"Illegal type " + arguments[0]->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return arguments[0];
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
const ColumnPtr column = block.getByPosition(arguments[0]).column;
if (const ColumnString * col = checkAndGetColumn<ColumnString>(column.get()))
{
auto col_res = ColumnString::create();
Impl::vector(col->getChars(), col->getOffsets(), col_res->getChars(), col_res->getOffsets());
block.getByPosition(result).column = std::move(col_res);
}
else if (const ColumnFixedString * col = checkAndGetColumn<ColumnFixedString>(column.get()))
{
auto col_res = ColumnFixedString::create(col->getN());
Impl::vector_fixed(col->getChars(), col->getN(), col_res->getChars());
block.getByPosition(result).column = std::move(col_res);
}
else
throw Exception(
"Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_COLUMN);
}
};
}

264
dbms/src/Functions/FunctionsArithmetic.h
View File

@ -44,6 +44,7 @@ namespace ErrorCodes
extern const int LOGICAL_ERROR;
extern const int TOO_LESS_ARGUMENTS_FOR_FUNCTION;
extern const int DECIMAL_OVERFLOW;
extern const int CANNOT_ADD_DIFFERENT_AGGREGATE_STATES;
}
@ -1019,14 +1020,14 @@ public:
/// DateTime, but if both operands are Dates, their type must be the same (e.g. Date - DateTime is invalid).
using ResultDataType = Switch<
/// Decimal cases
Case<!allow_decimal && (IsDecimal<LeftDataType> || IsDecimal<RightDataType>), InvalidType>,
Case<IsDecimal<LeftDataType> && IsDecimal<RightDataType> && UseLeftDecimal<LeftDataType, RightDataType>, LeftDataType>,
Case<IsDecimal<LeftDataType> && IsDecimal<RightDataType>, RightDataType>,
Case<IsDecimal<LeftDataType> && !IsDecimal<RightDataType> && IsIntegral<RightDataType>, LeftDataType>,
Case<!IsDecimal<LeftDataType> && IsDecimal<RightDataType> && IsIntegral<LeftDataType>, RightDataType>,
Case<!allow_decimal && (IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>), InvalidType>,
Case<IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType> && UseLeftDecimal<LeftDataType, RightDataType>, LeftDataType>,
Case<IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType>, RightDataType>,
Case<IsDataTypeDecimal<LeftDataType> && !IsDataTypeDecimal<RightDataType> && IsIntegral<RightDataType>, LeftDataType>,
Case<!IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType> && IsIntegral<LeftDataType>, RightDataType>,
/// Decimal <op> Real is not supported (traditional DBs convert Decimal <op> Real to Real)
Case<IsDecimal<LeftDataType> && !IsDecimal<RightDataType> && !IsIntegral<RightDataType>, InvalidType>,
Case<!IsDecimal<LeftDataType> && IsDecimal<RightDataType> && !IsIntegral<LeftDataType>, InvalidType>,
Case<IsDataTypeDecimal<LeftDataType> && !IsDataTypeDecimal<RightDataType> && !IsIntegral<RightDataType>, InvalidType>,
Case<!IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType> && !IsIntegral<LeftDataType>, InvalidType>,
/// number <op> number -> see corresponding impl
Case<!IsDateOrDateTime<LeftDataType> && !IsDateOrDateTime<RightDataType>,
DataTypeFromFieldType<typename Op::ResultType>>,
@ -1130,6 +1131,123 @@ class FunctionBinaryArithmetic : public IFunction
return FunctionFactory::instance().get(function_name.str(), context);
}
bool isAggregateMultiply(const DataTypePtr & type0, const DataTypePtr & type1) const
{
if constexpr (!std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>)
return false;
WhichDataType which0(type0);
WhichDataType which1(type1);
return (which0.isAggregateFunction() && which1.isNativeUInt())
|| (which0.isNativeUInt() && which1.isAggregateFunction());
}
bool isAggregateAddition(const DataTypePtr & type0, const DataTypePtr & type1) const
{
if constexpr (!std::is_same_v<Op<UInt8, UInt8>, PlusImpl<UInt8, UInt8>>)
return false;
WhichDataType which0(type0);
WhichDataType which1(type1);
return which0.isAggregateFunction() && which1.isAggregateFunction();
}
/// Multiply aggregation state by integer constant: by merging it with itself specified number of times.
void executeAggregateMultiply(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) const
{
ColumnNumbers new_arguments = arguments;
if (WhichDataType(block.getByPosition(new_arguments[1]).type).isAggregateFunction())
std::swap(new_arguments[0], new_arguments[1]);
if (!block.getByPosition(new_arguments[1]).column->isColumnConst())
throw Exception{"Illegal column " + block.getByPosition(new_arguments[1]).column->getName()
+ " of argument of aggregation state multiply. Should be integer constant", ErrorCodes::ILLEGAL_COLUMN};
const ColumnAggregateFunction * column = typeid_cast<const ColumnAggregateFunction *>(block.getByPosition(new_arguments[0]).column.get());
IAggregateFunction * function = column->getAggregateFunction().get();
auto arena = std::make_shared<Arena>();
auto column_to = ColumnAggregateFunction::create(column->getAggregateFunction(), Arenas(1, arena));
column_to->reserve(input_rows_count);
auto column_from = ColumnAggregateFunction::create(column->getAggregateFunction(), Arenas(1, arena));
column_from->reserve(input_rows_count);
for (size_t i = 0; i < input_rows_count; ++i)
{
column_to->insertDefault();
column_from->insertFrom(column->getData()[i]);
}
auto & vec_to = column_to->getData();
auto & vec_from = column_from->getData();
UInt64 m = typeid_cast<const ColumnConst *>(block.getByPosition(new_arguments[1]).column.get())->getValue<UInt64>();
/// We use exponentiation by squaring algorithm to perform multiplying aggregate states by N in O(log(N)) operations
/// https://en.wikipedia.org/wiki/Exponentiation_by_squaring
while (m)
{
if (m % 2)
{
for (size_t i = 0; i < input_rows_count; ++i)
function->merge(vec_to[i], vec_from[i], arena.get());
--m;
}
else
{
for (size_t i = 0; i < input_rows_count; ++i)
function->merge(vec_from[i], vec_from[i], arena.get());
m /= 2;
}
}
block.getByPosition(result).column = std::move(column_to);
}
/// Merge two aggregation states together.
void executeAggregateAddition(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) const
{
const ColumnAggregateFunction * columns[2];
for (size_t i = 0; i < 2; ++i)
columns[i] = typeid_cast<const ColumnAggregateFunction *>(block.getByPosition(arguments[i]).column.get());
auto column_to = ColumnAggregateFunction::create(columns[0]->getAggregateFunction());
column_to->reserve(input_rows_count);
for(size_t i = 0; i < input_rows_count; ++i)
{
column_to->insertFrom(columns[0]->getData()[i]);
column_to->insertMergeFrom(columns[1]->getData()[i]);
}
block.getByPosition(result).column = std::move(column_to);
}
void executeDateTimeIntervalPlusMinus(Block & block, const ColumnNumbers & arguments,
size_t result, size_t input_rows_count, const FunctionBuilderPtr & function_builder) const
{
ColumnNumbers new_arguments = arguments;
/// Interval argument must be second.
if (WhichDataType(block.getByPosition(arguments[0]).type).isInterval())
std::swap(new_arguments[0], new_arguments[1]);
/// Change interval argument type to its representation
Block new_block = block;
new_block.getByPosition(new_arguments[1]).type = std::make_shared<DataTypeNumber<DataTypeInterval::FieldType>>();
ColumnsWithTypeAndName new_arguments_with_type_and_name =
{new_block.getByPosition(new_arguments[0]), new_block.getByPosition(new_arguments[1])};
auto function = function_builder->build(new_arguments_with_type_and_name);
function->execute(new_block, new_arguments, result, input_rows_count);
block.getByPosition(result).column = new_block.getByPosition(result).column;
}
public:
static constexpr auto name = Name::name;
static FunctionPtr create(const Context & context) { return std::make_shared<FunctionBinaryArithmetic>(context); }
@ -1151,11 +1269,21 @@ public:
/// Special case when multiply aggregate function state
if (isAggregateMultiply(arguments[0], arguments[1]))
{
if (checkDataType<DataTypeAggregateFunction>(arguments[0].get()))
if (WhichDataType(arguments[0]).isAggregateFunction())
return arguments[0];
return arguments[1];
}
/// Special case - addition of two aggregate functions states
if (isAggregateAddition(arguments[0], arguments[1]))
{
if (!arguments[0]->equals(*arguments[1]))
throw Exception("Cannot add aggregate states of different functions: "
+ arguments[0]->getName() + " and " + arguments[1]->getName(), ErrorCodes::CANNOT_ADD_DIFFERENT_AGGREGATE_STATES);
return arguments[0];
}
/// Special case when the function is plus or minus, one of arguments is Date/DateTime and another is Interval.
if (auto function_builder = getFunctionForIntervalArithmetic(arguments[0], arguments[1]))
{
@ -1165,7 +1293,7 @@ public:
new_arguments[i].type = arguments[i];
/// Interval argument must be second.
if (checkDataType<DataTypeInterval>(new_arguments[0].type.get()))
if (WhichDataType(new_arguments[0].type).isInterval())
std::swap(new_arguments[0], new_arguments[1]);
/// Change interval argument to its representation
@ -1183,16 +1311,16 @@ public:
using ResultDataType = typename BinaryOperationTraits<Op, LeftDataType, RightDataType>::ResultDataType;
if constexpr (!std::is_same_v<ResultDataType, InvalidType>)
{
if constexpr (IsDecimal<LeftDataType> && IsDecimal<RightDataType>)
if constexpr (IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType>)
{
constexpr bool is_multiply = std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>;
constexpr bool is_division = std::is_same_v<Op<UInt8, UInt8>, DivideFloatingImpl<UInt8, UInt8>>;
ResultDataType result_type = decimalResultType(left, right, is_multiply, is_division);
type_res = std::make_shared<ResultDataType>(result_type.getPrecision(), result_type.getScale());
}
else if constexpr (IsDecimal<LeftDataType>)
else if constexpr (IsDataTypeDecimal<LeftDataType>)
type_res = std::make_shared<LeftDataType>(left.getPrecision(), left.getScale());
else if constexpr (IsDecimal<RightDataType>)
else if constexpr (IsDataTypeDecimal<RightDataType>)
type_res = std::make_shared<RightDataType>(right.getPrecision(), right.getScale());
else
type_res = std::make_shared<ResultDataType>();
@ -1206,92 +1334,26 @@ public:
return type_res;
}
bool isAggregateMultiply(const DataTypePtr & type0, const DataTypePtr & type1) const
{
if constexpr (!std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>)
return false;
auto is_uint_type = [](const DataTypePtr & type)
{
return checkDataType<DataTypeUInt8>(type.get()) || checkDataType<DataTypeUInt16>(type.get())
|| checkDataType<DataTypeUInt32>(type.get()) || checkDataType<DataTypeUInt64>(type.get());
};
return ((checkDataType<DataTypeAggregateFunction>(type0.get()) && is_uint_type(type1))
|| (is_uint_type(type0) && checkDataType<DataTypeAggregateFunction>(type1.get())));
}
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
{
/// Special case when multiply aggregate function state
if (isAggregateMultiply(block.getByPosition(arguments[0]).type, block.getByPosition(arguments[1]).type))
{
ColumnNumbers new_arguments = arguments;
if (checkDataType<DataTypeAggregateFunction>(block.getByPosition(new_arguments[1]).type.get()))
std::swap(new_arguments[0], new_arguments[1]);
executeAggregateMultiply(block, arguments, result, input_rows_count);
return;
}
const ColumnAggregateFunction * column = typeid_cast<const ColumnAggregateFunction *>(block.getByPosition(new_arguments[0]).column.get());
IAggregateFunction * function = column->getAggregateFunction().get();
auto arena = std::make_shared<Arena>();
auto column_to = ColumnAggregateFunction::create(column->getAggregateFunction(), Arenas(1, arena));
column_to->reserve(input_rows_count);
auto column_from = ColumnAggregateFunction::create(column->getAggregateFunction(), Arenas(1, arena));
column_from->reserve(input_rows_count);
for (size_t i = 0; i < input_rows_count; ++i)
{
column_to->insertDefault();
column_from->insertFrom(column->getData()[i]);
}
auto & vec_to = column_to->getData();
auto & vec_from = column_from->getData();
UInt64 m = block.getByPosition(new_arguments[1]).column->getUInt(0);
/// We use exponentiation by squaring algorithm to perform multiplying aggregate states by N in O(log(N)) operations
/// https://en.wikipedia.org/wiki/Exponentiation_by_squaring
while (m)
{
if (m % 2)
{
for (size_t i = 0; i < input_rows_count; ++i)
function->merge(vec_to[i], vec_from[i], arena.get());
--m;
}
else
{
for (size_t i = 0; i < input_rows_count; ++i)
function->merge(vec_from[i], vec_from[i], arena.get());
m /= 2;
}
}
block.getByPosition(result).column = std::move(column_to);
/// Special case - addition of two aggregate functions states
if (isAggregateAddition(block.getByPosition(arguments[0]).type, block.getByPosition(arguments[1]).type))
{
executeAggregateAddition(block, arguments, result, input_rows_count);
return;
}
/// Special case when the function is plus or minus, one of arguments is Date/DateTime and another is Interval.
if (auto function_builder = getFunctionForIntervalArithmetic(block.getByPosition(arguments[0]).type, block.getByPosition(arguments[1]).type))
{
ColumnNumbers new_arguments = arguments;
/// Interval argument must be second.
if (checkDataType<DataTypeInterval>(block.getByPosition(arguments[0]).type.get()))
std::swap(new_arguments[0], new_arguments[1]);
/// Change interval argument type to its representation
Block new_block = block;
new_block.getByPosition(new_arguments[1]).type = std::make_shared<DataTypeNumber<DataTypeInterval::FieldType>>();
ColumnsWithTypeAndName new_arguments_with_type_and_name =
{new_block.getByPosition(new_arguments[0]), new_block.getByPosition(new_arguments[1])};
auto function = function_builder->build(new_arguments_with_type_and_name);
function->execute(new_block, new_arguments, result, input_rows_count);
block.getByPosition(result).column = new_block.getByPosition(result).column;
executeDateTimeIntervalPlusMinus(block, arguments, result, input_rows_count, function_builder);
return;
}
@ -1304,7 +1366,7 @@ public:
using ResultDataType = typename BinaryOperationTraits<Op, LeftDataType, RightDataType>::ResultDataType;
if constexpr (!std::is_same_v<ResultDataType, InvalidType>)
{
constexpr bool result_is_decimal = IsDecimal<LeftDataType> || IsDecimal<RightDataType>;
constexpr bool result_is_decimal = IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>;
constexpr bool is_multiply = std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>;
constexpr bool is_division = std::is_same_v<Op<UInt8, UInt8>, DivideFloatingImpl<UInt8, UInt8>>;
@ -1316,7 +1378,7 @@ public:
using ColVecResult = std::conditional_t<IsDecimalNumber<ResultType>, ColumnDecimal<ResultType>, ColumnVector<ResultType>>;
/// Decimal operations need scale. Operations are on result type.
using OpImpl = std::conditional_t<IsDecimal<ResultDataType>,
using OpImpl = std::conditional_t<IsDataTypeDecimal<ResultDataType>,
DecimalBinaryOperation<T0, T1, Op, ResultType>,
BinaryOperationImpl<T0, T1, Op<T0, T1>, ResultType>>;
@ -1332,7 +1394,7 @@ public:
ResultDataType type = decimalResultType(left, right, is_multiply, is_division);
typename ResultDataType::FieldType scale_a = type.scaleFactorFor(left, is_multiply);
typename ResultDataType::FieldType scale_b = type.scaleFactorFor(right, is_multiply || is_division);
if constexpr (IsDecimal<RightDataType> && is_division)
if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
scale_a = right.getScaleMultiplier();
auto res = OpImpl::constant_constant(col_left->template getValue<T0>(), col_right->template getValue<T1>(),
@ -1373,7 +1435,7 @@ public:
typename ResultDataType::FieldType scale_a = type.scaleFactorFor(left, is_multiply);
typename ResultDataType::FieldType scale_b = type.scaleFactorFor(right, is_multiply || is_division);
if constexpr (IsDecimal<RightDataType> && is_division)
if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
scale_a = right.getScaleMultiplier();
OpImpl::constant_vector(col_left_const->template getValue<T0>(), col_right->getData(), vec_res,
@ -1393,7 +1455,7 @@ public:
typename ResultDataType::FieldType scale_a = type.scaleFactorFor(left, is_multiply);
typename ResultDataType::FieldType scale_b = type.scaleFactorFor(right, is_multiply || is_division);
if constexpr (IsDecimal<RightDataType> && is_division)
if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
scale_a = right.getScaleMultiplier();
if (auto col_right = checkAndGetColumn<ColVecT1>(col_right_raw))
{
@ -1439,7 +1501,7 @@ public:
using RightDataType = std::decay_t<decltype(right)>;
using ResultDataType = typename BinaryOperationTraits<Op, LeftDataType, RightDataType>::ResultDataType;
using OpSpec = Op<typename LeftDataType::FieldType, typename RightDataType::FieldType>;
return !std::is_same_v<ResultDataType, InvalidType> && !IsDecimal<ResultDataType> && OpSpec::compilable;
return !std::is_same_v<ResultDataType, InvalidType> && !IsDataTypeDecimal<ResultDataType> && OpSpec::compilable;
});
}
@ -1452,7 +1514,7 @@ public:
using RightDataType = std::decay_t<decltype(right)>;
using ResultDataType = typename BinaryOperationTraits<Op, LeftDataType, RightDataType>::ResultDataType;
using OpSpec = Op<typename LeftDataType::FieldType, typename RightDataType::FieldType>;
if constexpr (!std::is_same_v<ResultDataType, InvalidType> && !IsDecimal<ResultDataType> && OpSpec::compilable)
if constexpr (!std::is_same_v<ResultDataType, InvalidType> && !IsDataTypeDecimal<ResultDataType> && OpSpec::compilable)
{
auto & b = static_cast<llvm::IRBuilder<> &>(builder);
auto type = std::make_shared<ResultDataType>();
@ -1522,7 +1584,7 @@ public:
using DataType = std::decay_t<decltype(type)>;
using T0 = typename DataType::FieldType;
if constexpr (IsDecimal<DataType>)
if constexpr (IsDataTypeDecimal<DataType>)
{
if constexpr (!allow_decimal)
return false;
@ -1545,7 +1607,7 @@ public:
using DataType = std::decay_t<decltype(type)>;
using T0 = typename DataType::FieldType;
if constexpr (IsDecimal<DataType>)
if constexpr (IsDataTypeDecimal<DataType>)
{
if constexpr (allow_decimal)
{
@ -1585,7 +1647,7 @@ public:
return castType(arguments[0].get(), [&](const auto & type)
{
using DataType = std::decay_t<decltype(type)>;
return !IsDecimal<DataType> && Op<typename DataType::FieldType>::compilable;
return !IsDataTypeDecimal<DataType> && Op<typename DataType::FieldType>::compilable;
});
}
@ -1597,7 +1659,7 @@ public:
using DataType = std::decay_t<decltype(type)>;
using T0 = typename DataType::FieldType;
using T1 = typename Op<T0>::ResultType;
if constexpr (!std::is_same_v<T1, InvalidType> && !IsDecimal<DataType> && Op<T0>::compilable)
if constexpr (!std::is_same_v<T1, InvalidType> && !IsDataTypeDecimal<DataType> && Op<T0>::compilable)
{
auto & b = static_cast<llvm::IRBuilder<> &>(builder);
auto * v = nativeCast(b, types[0], values[0](), std::make_shared<DataTypeNumber<T1>>());
@ -1906,17 +1968,17 @@ public:
throw Exception{"Number of arguments for function " + getName() + " doesn't match: passed "
+ toString(arguments.size()) + ", should be at least 2.", ErrorCodes::TOO_LESS_ARGUMENTS_FOR_FUNCTION};
const auto first_arg = arguments.front().get();
const auto & first_arg = arguments.front();
if (!first_arg->isInteger())
if (!isInteger(first_arg))
throw Exception{"Illegal type " + first_arg->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
for (const auto i : ext::range(1, arguments.size()))
{
const auto pos_arg = arguments[i].get();
const auto & pos_arg = arguments[i];
if (!pos_arg->isUnsignedInteger())
if (!isUnsignedInteger(pos_arg))
throw Exception{"Illegal type " + pos_arg->getName() + " of " + toString(i) + " argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}

3791
dbms/src/Functions/FunctionsArray.cpp
View File

File diff suppressed because it is too large Load Diff

34
dbms/src/Functions/FunctionsCoding.h
View File

@ -150,12 +150,12 @@ public:
", expected FixedString(" + toString(ipv6_bytes_length) + ")",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!checkDataType<DataTypeUInt8>(arguments[1].get()))
if (!WhichDataType(arguments[1]).isUInt8())
throw Exception("Illegal type " + arguments[1]->getName() +
" of argument 2 of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!checkDataType<DataTypeUInt8>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt8())
throw Exception("Illegal type " + arguments[2]->getName() +
" of argument 3 of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -266,7 +266,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -519,7 +519,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!checkDataType<DataTypeUInt32>(arguments[0].get()))
if (!WhichDataType(arguments[0]).isUInt32())
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName() + ", expected UInt32",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -579,7 +579,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -714,7 +714,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!checkDataType<DataTypeUInt64>(arguments[0].get()))
if (!WhichDataType(arguments[0]).isUInt64())
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName() + ", expected UInt64",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -843,7 +843,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1006,7 +1006,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
/// String or FixedString(36)
if (!arguments[0]->isString())
if (!isString(arguments[0]))
{
const auto ptr = checkAndGetDataType<DataTypeFixedString>(arguments[0].get());
if (!ptr || ptr->getN() != uuid_text_length)
@ -1151,13 +1151,11 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString()
&& !arguments[0]->isFixedString()
&& !arguments[0]->isDateOrDateTime()
&& !checkDataType<DataTypeUInt8>(arguments[0].get())
&& !checkDataType<DataTypeUInt16>(arguments[0].get())
&& !checkDataType<DataTypeUInt32>(arguments[0].get())
&& !checkDataType<DataTypeUInt64>(arguments[0].get()))
WhichDataType which(arguments[0]);
if (!which.isStringOrFixedString()
&& !which.isDateOrDateTime()
&& !which.isUInt())
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1370,7 +1368,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1460,7 +1458,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isInteger())
if (!isInteger(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1543,7 +1541,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isStringOrFixedString())
if (!isStringOrFixedString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

12
dbms/src/Functions/FunctionsComparison.h
View File

@ -817,7 +817,7 @@ private:
const IColumn * column_number = left_is_num ? col_left_untyped : col_right_untyped;
const IDataType * number_type = left_is_num ? left_type.get() : right_type.get();
DataTypeExtractor which(number_type);
WhichDataType which(number_type);
const bool legal_types = which.isDateOrDateTime() || which.isEnum() || which.isUUID();
@ -1077,8 +1077,8 @@ public:
/// Get result types by argument types. If the function does not apply to these arguments, throw an exception.
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
DataTypeExtractor left(arguments[0].get());
DataTypeExtractor right(arguments[1].get());
WhichDataType left(arguments[0].get());
WhichDataType right(arguments[1].get());
const DataTypeTuple * left_tuple = checkAndGetDataType<DataTypeTuple>(arguments[0].get());
const DataTypeTuple * right_tuple = checkAndGetDataType<DataTypeTuple>(arguments[1].get());
@ -1159,9 +1159,9 @@ public:
{
executeTuple(block, result, col_with_type_and_name_left, col_with_type_and_name_right, input_rows_count);
}
else if (isDecimal(left_type.get()) || isDecimal(right_type.get()))
else if (isDecimal(left_type) || isDecimal(right_type))
{
if (!allowDecimalComparison(left_type.get(), right_type.get()))
if (!allowDecimalComparison(left_type, right_type))
throw Exception("No operation " + getName() + " between " + left_type->getName() + " and " + right_type->getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1193,7 +1193,7 @@ public:
auto isFloatingPoint = &typeIsEither<DataTypeFloat32, DataTypeFloat64>;
if ((isBigInteger(*types[0]) && isFloatingPoint(*types[1])) || (isBigInteger(*types[1]) && isFloatingPoint(*types[0])))
return false; /// TODO: implement (double, int_N where N > double's mantissa width)
return isCompilableType(types[0].get()) && isCompilableType(types[1].get());
return isCompilableType(types[0]) && isCompilableType(types[1]);
}
llvm::Value * compileImpl(llvm::IRBuilderBase & builder, const DataTypes & types, ValuePlaceholders values) const override

52
dbms/src/Functions/FunctionsConditional.cpp
View File

@ -1,8 +1,9 @@
#include <Functions/FunctionsConditional.h>
#include <Functions/FunctionsArray.h>
#include <Functions/FunctionsTransform.h>
#include <Functions/FunctionFactory.h>
#include <Columns/ColumnNullable.h>
#include <Columns/ColumnConst.h>
#include <DataTypes/getLeastSupertype.h>
#include <Interpreters/castColumn.h>
#include <vector>
@ -205,7 +206,7 @@ DataTypePtr FunctionMultiIf::getReturnTypeImpl(const DataTypes & args) const
nested_type = arg.get();
}
if (!checkDataType<DataTypeUInt8>(nested_type))
if (!WhichDataType(nested_type).isUInt8())
throw Exception{"Illegal type " + arg->getName() + " of argument (condition) "
"of function " + getName() + ". Must be UInt8.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -251,22 +252,15 @@ DataTypePtr FunctionCaseWithExpression::getReturnTypeImpl(const DataTypes & args
/// See the comments in executeImpl() to understand why we actually have to
/// get the return type of a transform function.
/// Get the return types of the arrays that we pass to the transform function.
ColumnsWithTypeAndName src_array_types;
ColumnsWithTypeAndName dst_array_types;
/// Get the types of the arrays that we pass to the transform function.
DataTypes src_array_types;
DataTypes dst_array_types;
for (size_t i = 1; i < (args.size() - 1); ++i)
{
if ((i % 2) != 0)
src_array_types.push_back({nullptr, args[i], {}});
else
dst_array_types.push_back({nullptr, args[i], {}});
}
for (size_t i = 1; i < args.size() - 1; ++i)
((i % 2) ? src_array_types : dst_array_types).push_back(args[i]);
FunctionArray fun_array{context};
DataTypePtr src_array_type = fun_array.getReturnType(src_array_types);
DataTypePtr dst_array_type = fun_array.getReturnType(dst_array_types);
DataTypePtr src_array_type = std::make_shared<DataTypeArray>(getLeastSupertype(src_array_types));
DataTypePtr dst_array_type = std::make_shared<DataTypeArray>(getLeastSupertype(dst_array_types));
/// Finally get the return type of the transform function.
FunctionTransform fun_transform;
@ -291,29 +285,31 @@ void FunctionCaseWithExpression::executeImpl(Block & block, const ColumnNumbers
/// Create the arrays required by the transform function.
ColumnNumbers src_array_args;
ColumnsWithTypeAndName src_array_types;
ColumnsWithTypeAndName src_array_elems;
DataTypes src_array_types;
ColumnNumbers dst_array_args;
ColumnsWithTypeAndName dst_array_types;
ColumnsWithTypeAndName dst_array_elems;
DataTypes dst_array_types;
for (size_t i = 1; i < (args.size() - 1); ++i)
{
if ((i % 2) != 0)
if (i % 2)
{
src_array_args.push_back(args[i]);
src_array_types.push_back(block.getByPosition(args[i]));
src_array_elems.push_back(block.getByPosition(args[i]));
src_array_types.push_back(block.getByPosition(args[i]).type);
}
else
{
dst_array_args.push_back(args[i]);
dst_array_types.push_back(block.getByPosition(args[i]));
dst_array_elems.push_back(block.getByPosition(args[i]));
dst_array_types.push_back(block.getByPosition(args[i]).type);
}
}
FunctionArray fun_array{context};
DataTypePtr src_array_type = fun_array.getReturnType(src_array_types);
DataTypePtr dst_array_type = fun_array.getReturnType(dst_array_types);
DataTypePtr src_array_type = std::make_shared<DataTypeArray>(getLeastSupertype(src_array_types));
DataTypePtr dst_array_type = std::make_shared<DataTypeArray>(getLeastSupertype(dst_array_types));
Block temp_block = block;
@ -323,8 +319,10 @@ void FunctionCaseWithExpression::executeImpl(Block & block, const ColumnNumbers
size_t dst_array_pos = temp_block.columns();
temp_block.insert({nullptr, dst_array_type, ""});
fun_array.execute(temp_block, src_array_args, src_array_pos, input_rows_count);
fun_array.execute(temp_block, dst_array_args, dst_array_pos, input_rows_count);
auto fun_array = FunctionFactory::instance().get("array", context);
fun_array->build(src_array_elems)->execute(temp_block, src_array_args, src_array_pos, input_rows_count);
fun_array->build(dst_array_elems)->execute(temp_block, dst_array_args, dst_array_pos, input_rows_count);
/// Execute transform.
FunctionTransform fun_transform;

4
dbms/src/Functions/FunctionsConditional.h
View File

@ -122,7 +122,7 @@ public:
bool isCompilableImpl(const DataTypes & types) const override
{
for (const auto & type : types)
if (!isCompilableType(removeNullable(type).get()))
if (!isCompilableType(removeNullable(type)))
return false;
return true;
}
@ -895,7 +895,7 @@ public:
return makeNullable(getReturnTypeImpl({
removeNullable(arguments[0]), arguments[1], arguments[2]}));
if (!checkDataType<DataTypeUInt8>(arguments[0].get()))
if (!WhichDataType(arguments[0]).isUInt8())
throw Exception("Illegal type " + arguments[0]->getName() + " of first argument (condition) of function if. Must be UInt8.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

22
dbms/src/Functions/FunctionsConsistentHashing.h
View File

@ -106,7 +106,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isInteger())
if (!isInteger(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of the first argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -115,7 +115,7 @@ public:
+ ", got " + arguments[0]->getName(),
ErrorCodes::BAD_ARGUMENTS);
if (!arguments[1]->isInteger())
if (!isInteger(arguments[1]))
throw Exception("Illegal type " + arguments[1]->getName() + " of the second argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -178,21 +178,23 @@ private:
const IDataType * hash_type = block.getByPosition(arguments[0]).type.get();
auto res_col = ColumnVector<ResultType>::create();
if (checkDataType<DataTypeUInt8>(hash_type))
WhichDataType which(hash_type);
if (which.isUInt8())
executeType<UInt8>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeUInt16>(hash_type))
else if (which.isUInt16())
executeType<UInt16>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeUInt32>(hash_type))
else if (which.isUInt32())
executeType<UInt32>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeUInt64>(hash_type))
else if (which.isUInt64())
executeType<UInt64>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeInt8>(hash_type))
else if (which.isInt8())
executeType<Int8>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeInt16>(hash_type))
else if (which.isInt16())
executeType<Int16>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeInt32>(hash_type))
else if (which.isInt32())
executeType<Int32>(hash_col, num_buckets, res_col.get());
else if (checkDataType<DataTypeInt64>(hash_type))
else if (which.isInt64())
executeType<Int64>(hash_col, num_buckets, res_col.get());
else
throw Exception("Illegal type " + hash_type->getName() + " of the first argument of function " + getName(),

2
dbms/src/Functions/FunctionsConversion.cpp
View File

@ -20,7 +20,7 @@ void throwExceptionForIncompletelyParsedValue(
else
message_buf << " at begin of string";
if (to_type.isNumber())
if (isNumber(to_type))
message_buf << ". Note: there are to" << to_type.getName() << "OrZero and to" << to_type.getName() << "OrNull functions, which returns zero/NULL instead of throwing exception.";
throw Exception(message_buf.str(), ErrorCodes::CANNOT_PARSE_TEXT);

65
dbms/src/Functions/FunctionsConversion.h
View File

@ -110,7 +110,7 @@ struct ConvertImpl
if (const ColVecFrom * col_from = checkAndGetColumn<ColVecFrom>(named_from.column.get()))
{
typename ColVecTo::MutablePtr col_to = nullptr;
if constexpr (IsDecimal<ToDataType>)
if constexpr (IsDataTypeDecimal<ToDataType>)
{
UInt32 scale = additions;
col_to = ColVecTo::create(0, scale);
@ -125,11 +125,11 @@ struct ConvertImpl
for (size_t i = 0; i < size; ++i)
{
if constexpr (IsDecimal<FromDataType> && IsDecimal<ToDataType>)
if constexpr (IsDataTypeDecimal<FromDataType> && IsDataTypeDecimal<ToDataType>)
vec_to[i] = convertDecimals<FromDataType, ToDataType>(vec_from[i], vec_from.getScale(), vec_to.getScale());
else if constexpr (IsDecimal<FromDataType>)
else if constexpr (IsDataTypeDecimal<FromDataType>)
vec_to[i] = convertFromDecimal<FromDataType, ToDataType>(vec_from[i], vec_from.getScale());
else if constexpr (IsDecimal<ToDataType>)
else if constexpr (IsDataTypeDecimal<ToDataType>)
vec_to[i] = convertToDecimal<FromDataType, ToDataType>(vec_from[i], vec_to.getScale());
else
vec_to[i] = static_cast<ToFieldType>(vec_from[i]);
@ -490,7 +490,7 @@ struct ConvertThroughParsing
size_t size = input_rows_count;
typename ColVecTo::MutablePtr col_to = nullptr;
if constexpr (IsDecimal<ToDataType>)
if constexpr (IsDataTypeDecimal<ToDataType>)
{
UInt32 scale = additions;
ToDataType check_bounds_in_ctor(ToDataType::maxPrecision(), scale);
@ -533,7 +533,7 @@ struct ConvertThroughParsing
ReadBufferFromMemory read_buffer(&(*chars)[current_offset], string_size);
if constexpr (IsDecimal<ToDataType>)
if constexpr (IsDataTypeDecimal<ToDataType>)
{
ToDataType::readText(vec_to[i], read_buffer, ToDataType::maxPrecision(), vec_to.getScale());
}
@ -802,7 +802,7 @@ public:
|| std::is_same_v<Name, NameToUnixTimestamp>;
if (!(to_date_or_time
|| (std::is_same_v<Name, NameToString> && checkDataType<DataTypeDateTime>(arguments[0].type.get()))))
|| (std::is_same_v<Name, NameToString> && WhichDataType(arguments[0].type).isDateTime())))
{
throw Exception("Number of arguments for function " + getName() + " doesn't match: passed "
+ toString(arguments.size()) + ", should be 1.",
@ -880,7 +880,7 @@ private:
using LeftDataType = typename Types::LeftType;
using RightDataType = typename Types::RightType;
if constexpr (IsDecimal<RightDataType>)
if constexpr (IsDataTypeDecimal<RightDataType>)
{
if (arguments.size() != 2)
throw Exception{"Function " + getName() + " expects 2 arguments for Decimal.",
@ -950,7 +950,7 @@ public:
+ toString(arguments.size()) + ", should be 1 or 2. Second argument (time zone) is optional only make sense for DateTime.",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!arguments[0].type->isStringOrFixedString())
if (!isStringOrFixedString(arguments[0].type))
throw Exception("Illegal type " + arguments[0].type->getName() + " of first argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -963,7 +963,7 @@ public:
+ toString(arguments.size()) + ", should be 1. Second argument makes sense only when converting to DateTime.",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!arguments[1].type->isString())
if (!isString(arguments[1].type))
throw Exception("Illegal type " + arguments[1].type->getName() + " of 2nd argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
@ -1020,11 +1020,11 @@ public:
DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
{
if (!arguments[1].type->isUnsignedInteger())
if (!isUnsignedInteger(arguments[1].type))
throw Exception("Second argument for function " + getName() + " must be unsigned integer", ErrorCodes::ILLEGAL_COLUMN);
if (!arguments[1].column)
throw Exception("Second argument for function " + getName() + " must be constant", ErrorCodes::ILLEGAL_COLUMN);
if (!arguments[0].type->isStringOrFixedString())
if (!isStringOrFixedString(arguments[0].type))
throw Exception(getName() + " is only implemented for types String and FixedString", ErrorCodes::NOT_IMPLEMENTED);
const size_t n = arguments[1].column->getUInt(0);
@ -1140,8 +1140,8 @@ struct ToIntMonotonicity
}
/// If type is same, too. (Enum has separate case, because it is different data type)
if (checkDataType<DataTypeNumber<T>>(&type) ||
checkDataType<DataTypeEnum<T>>(&type))
if (checkAndGetDataType<DataTypeNumber<T>>(&type) ||
checkAndGetDataType<DataTypeEnum<T>>(&type))
return { true, true, true };
/// In other cases, if range is unbounded, we don't know, whether function is monotonic or not.
@ -1149,8 +1149,7 @@ struct ToIntMonotonicity
return {};
/// If converting from float, for monotonicity, arguments must fit in range of result type.
if (checkDataType<DataTypeFloat32>(&type)
|| checkDataType<DataTypeFloat64>(&type))
if (WhichDataType(type).isFloat())
{
Float64 left_float = left.get<Float64>();
Float64 right_float = right.get<Float64>();
@ -1460,7 +1459,7 @@ private:
static WrapperType createFixedStringWrapper(const DataTypePtr & from_type, const size_t N)
{
if (!from_type->isStringOrFixedString())
if (!isStringOrFixedString(from_type))
throw Exception{"CAST AS FixedString is only implemented for types String and FixedString", ErrorCodes::NOT_IMPLEMENTED};
return [N] (Block & block, const ColumnNumbers & arguments, const size_t result, size_t /*input_rows_count*/)
@ -1469,6 +1468,24 @@ private:
};
}
WrapperType createUUIDWrapper(const DataTypePtr & from_type, const DataTypeUUID * const, bool requested_result_is_nullable) const
{
if (requested_result_is_nullable)
throw Exception{"CAST AS Nullable(UUID) is not implemented", ErrorCodes::NOT_IMPLEMENTED};
FunctionPtr function = FunctionTo<DataTypeUUID>::Type::create(context);
/// Check conversion using underlying function
{
function->getReturnType(ColumnsWithTypeAndName(1, { nullptr, from_type, "" }));
}
return [function] (Block & block, const ColumnNumbers & arguments, const size_t result, size_t input_rows_count)
{
function->execute(block, arguments, result, input_rows_count);
};
}
template <typename FieldType>
WrapperType createDecimalWrapper(const DataTypePtr & from_type, const DataTypeDecimal<FieldType> * to_type) const
{
@ -1628,7 +1645,7 @@ private:
return createStringToEnumWrapper<ColumnString, EnumType>();
else if (checkAndGetDataType<DataTypeFixedString>(from_type.get()))
return createStringToEnumWrapper<ColumnFixedString, EnumType>();
else if (from_type->isNumber() || from_type->isEnum())
else if (isNumber(from_type) || isEnum(from_type))
{
auto function = Function::create(context);
@ -1878,7 +1895,7 @@ private:
{
if (from_type->equals(*to_type))
return createIdentityWrapper(from_type);
else if (checkDataType<DataTypeNothing>(from_type.get()))
else if (WhichDataType(from_type).isNothing())
return createNothingWrapper(to_type.get());
WrapperType ret;
@ -1920,6 +1937,14 @@ private:
ret = createDecimalWrapper(from_type, checkAndGetDataType<ToDataType>(to_type.get()));
return true;
}
if constexpr (std::is_same_v<ToDataType, DataTypeUUID>)
{
if (isStringOrFixedString(from_type))
{
ret = createUUIDWrapper(from_type, checkAndGetDataType<ToDataType>(to_type.get()), requested_result_is_nullable);
return true;
}
}
return false;
};
@ -2027,7 +2052,7 @@ private:
return monotonicityForType(type);
else if (const auto type = checkAndGetDataType<DataTypeString>(to_type))
return monotonicityForType(type);
else if (from_type->isEnum())
else if (isEnum(from_type))
{
if (const auto type = checkAndGetDataType<DataTypeEnum8>(to_type))
return monotonicityForType(type);

42
dbms/src/Functions/FunctionsDateTime.h
View File

@ -637,14 +637,14 @@ public:
{
if (arguments.size() == 1)
{
if (!arguments[0].type->isDateOrDateTime())
if (!isDateOrDateTime(arguments[0].type))
throw Exception("Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() +
". Should be a date or a date with time", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
else if (arguments.size() == 2)
{
if (!checkDataType<DataTypeDateTime>(arguments[0].type.get())
|| !checkDataType<DataTypeString>(arguments[1].type.get()))
if (!WhichDataType(arguments[0].type).isDateTime()
|| !WhichDataType(arguments[1].type).isString())
throw Exception(
"Function " + getName() + " supports 1 or 2 arguments. The 1st argument "
"must be of type Date or DateTime. The 2nd argument (optional) must be "
@ -670,10 +670,11 @@ public:
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
{
const IDataType * from_type = block.getByPosition(arguments[0]).type.get();
WhichDataType which(from_type);
if (checkDataType<DataTypeDate>(from_type))
if (which.isDate())
DateTimeTransformImpl<DataTypeDate::FieldType, typename ToDataType::FieldType, Transform>::execute(block, arguments, result, input_rows_count);
else if (checkDataType<DataTypeDateTime>(from_type))
else if (which.isDateTime())
DateTimeTransformImpl<DataTypeDateTime::FieldType, typename ToDataType::FieldType, Transform>::execute(block, arguments, result, input_rows_count);
else
throw Exception("Illegal type " + block.getByPosition(arguments[0]).type->getName() + " of argument of function " + getName(),
@ -945,20 +946,20 @@ public:
+ toString(arguments.size()) + ", should be 2 or 3",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!arguments[1].type->isNumber())
if (!isNumber(arguments[1].type))
throw Exception("Second argument for function " + getName() + " (delta) must be number",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (arguments.size() == 2)
{
if (!arguments[0].type->isDateOrDateTime())
if (!isDateOrDateTime(arguments[0].type))
throw Exception{"Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() +
". Should be a date or a date with time", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}
else
{
if (!checkDataType<DataTypeDateTime>(arguments[0].type.get())
|| !checkDataType<DataTypeString>(arguments[2].type.get()))
if (!WhichDataType(arguments[0].type).isDateTime()
|| !WhichDataType(arguments[2].type).isString())
throw Exception(
"Function " + getName() + " supports 2 or 3 arguments. The 1st argument "
"must be of type Date or DateTime. The 2nd argument must be number. "
@ -968,7 +969,7 @@ public:
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
if (checkDataType<DataTypeDate>(arguments[0].type.get()))
if (WhichDataType(arguments[0].type).isDate())
{
if (std::is_same_v<decltype(Transform::execute(DataTypeDate::FieldType(), 0, std::declval<DateLUTImpl>())), UInt16>)
return std::make_shared<DataTypeDate>();
@ -990,10 +991,11 @@ public:
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
const IDataType * from_type = block.getByPosition(arguments[0]).type.get();
WhichDataType which(from_type);
if (checkDataType<DataTypeDate>(from_type))
if (which.isDate())
DateTimeAddIntervalImpl<DataTypeDate::FieldType, Transform>::execute(block, arguments, result);
else if (checkDataType<DataTypeDateTime>(from_type))
else if (which.isDateTime())
DateTimeAddIntervalImpl<DataTypeDateTime::FieldType, Transform>::execute(block, arguments, result);
else
throw Exception("Illegal type " + block.getByPosition(arguments[0]).type->getName() + " of argument of function " + getName(),
@ -1032,19 +1034,19 @@ public:
+ toString(arguments.size()) + ", should be 3 or 4",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("First argument for function " + getName() + " (unit) must be String",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!arguments[1]->isDateOrDateTime())
if (!isDateOrDateTime(arguments[1]))
throw Exception("Second argument for function " + getName() + " must be Date or DateTime",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!arguments[2]->isDateOrDateTime())
if (!isDateOrDateTime(arguments[2]))
throw Exception("Third argument for function " + getName() + " must be Date or DateTime",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (arguments.size() == 4 && !arguments[3]->isString())
if (arguments.size() == 4 && !isString(arguments[3]))
throw Exception("Fourth argument for function " + getName() + " (timezone) must be String",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1296,7 +1298,7 @@ public:
+ toString(arguments.size()) + ", should be 2",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!checkDataType<DataTypeDateTime>(arguments[0].type.get()))
if (!WhichDataType(arguments[0].type).isDateTime())
throw Exception{"Illegal type " + arguments[0].type->getName() + " of argument of function " + getName() +
". Should be DateTime", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -1326,7 +1328,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!checkDataType<DataTypeDateTime>(arguments[0].get()))
if (!WhichDataType(arguments[0]).isDateTime())
throw Exception("Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ". Must be DateTime.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -1453,11 +1455,11 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!checkDataType<DataTypeDateTime>(arguments[0].get()))
if (!WhichDataType(arguments[0]).isDateTime())
throw Exception("Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ". Must be DateTime.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (!checkDataType<DataTypeUInt32>(arguments[1].get()))
if (!WhichDataType(arguments[1]).isUInt32())
throw Exception("Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ". Must be UInt32.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

56
dbms/src/Functions/FunctionsExternalDictionaries.h
View File

@ -82,12 +82,12 @@ private:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[1].get()) &&
!checkDataType<DataTypeTuple>(arguments[1].get()))
if (!WhichDataType(arguments[1]).isUInt64() &&
!isTuple(arguments[1]))
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -230,27 +230,27 @@ private:
throw Exception{"Number of arguments for function " + getName() + " doesn't match: passed "
+ toString(arguments.size()) + ", should be 3 or 4.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH};
if (!arguments[0]->isString())
if (!isString(arguments[0]))
{
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}
if (!arguments[1]->isString())
if (!isString(arguments[1]))
{
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}
if (!checkDataType<DataTypeUInt64>(arguments[2].get()) &&
!checkDataType<DataTypeTuple>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt64() &&
!isTuple(arguments[2]))
{
throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName()
+ ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}
/// This is for the case of range dictionaries.
if (arguments.size() == 4 && !checkDataType<DataTypeDate>(arguments[3].get()))
if (arguments.size() == 4 && !WhichDataType(arguments[3]).isDate())
{
throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName()
+ ", must be Date.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -467,22 +467,22 @@ private:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() +
", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!arguments[1]->isString())
if (!isString(arguments[1]))
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() +
", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[2].get()) &&
!checkDataType<DataTypeTuple>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt64() &&
!isTuple(arguments[2]))
{
throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName()
+ ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}
if (!arguments[3]->isString())
if (!isString(arguments[3]))
throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName() +
", must be String.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -735,20 +735,20 @@ private:
if (arguments.size() != 3 && arguments.size() != 4)
throw Exception{"Function " + getName() + " takes 3 or 4 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH};
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!arguments[1]->isString())
if (!isString(arguments[1]))
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[2].get()) &&
!checkDataType<DataTypeTuple>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt64() &&
!isTuple(arguments[2]))
throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName()
+ ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (arguments.size() == 4 && !checkDataType<DataTypeDate>(arguments[3].get()))
if (arguments.size() == 4 && !WhichDataType(arguments[3]).isDate())
throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName()
+ ", must be Date.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -1010,20 +1010,20 @@ private:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!arguments[1]->isString())
if (!isString(arguments[1]))
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[2].get()) &&
!checkDataType<DataTypeTuple>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt64() &&
!isTuple(arguments[2]))
throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName()
+ ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataType>(arguments[3].get()))
if (!checkAndGetDataType<DataType>(arguments[3].get()))
throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName()
+ ", must be " + String(DataType{}.getFamilyName()) + ".", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -1252,11 +1252,11 @@ private:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[1].get()))
if (!WhichDataType(arguments[1]).isUInt64())
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
@ -1408,15 +1408,15 @@ private:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[1].get()))
if (!WhichDataType(arguments[1]).isUInt64())
throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName()
+ ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (!checkDataType<DataTypeUInt64>(arguments[2].get()))
if (!WhichDataType(arguments[2]).isUInt64())
throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName()
+ ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};

2
dbms/src/Functions/FunctionsExternalModels.cpp
View File

@ -32,7 +32,7 @@ DataTypePtr FunctionModelEvaluate::getReturnTypeImpl(const DataTypes & arguments
throw Exception("Function " + getName() + " expects at least 2 arguments",
ErrorCodes::TOO_LESS_ARGUMENTS_FOR_FUNCTION);
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of first argument of function " + getName()
+ ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

2
dbms/src/Functions/FunctionsFindCluster.h
View File

@ -116,7 +116,7 @@ public:
const auto type_x = arguments[0];
if (!type_x->isNumber())
if (!isNumber(type_x))
throw Exception{"Unsupported type " + type_x->getName() + " of first argument of function " + getName() + " must be a numeric type",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};

6
dbms/src/Functions/FunctionsFormatting.h
View File

@ -44,9 +44,9 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
const IDataType * type = arguments[0].get();
const DataTypePtr & type = arguments[0];
if (!type->isInteger())
if (!isInteger(type))
throw Exception("Cannot format " + type->getName() + " as bitmask string", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return std::make_shared<DataTypeString>();
@ -139,7 +139,7 @@ public:
{
const IDataType & type = *arguments[0];
if (!type.isNumber())
if (!isNumber(type))
throw Exception("Cannot format " + type.getName() + " as size in bytes", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return std::make_shared<DataTypeString>();

4
dbms/src/Functions/FunctionsGeo.cpp
View File

@ -133,7 +133,7 @@ public:
for (auto j : ext::range(0, elements.size()))
{
if (!elements[j]->isNumber())
if (!isNumber(elements[j]))
{
throw Exception(getMsgPrefix(i) + " must contains numeric tuple at position " + toString(j + 1),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -162,7 +162,7 @@ public:
const Columns & tuple_columns = tuple_col->getColumns();
const DataTypes & tuple_types = typeid_cast<const DataTypeTuple &>(*block.getByPosition(arguments[0]).type).getElements();
bool use_float64 = checkDataType<DataTypeFloat64>(tuple_types[0].get()) || checkDataType<DataTypeFloat64>(tuple_types[1].get());
bool use_float64 = WhichDataType(tuple_types[0]).isFloat64() || WhichDataType(tuple_types[1]).isFloat64();
auto & result_column = block.safeGetByPosition(result).column;

4
dbms/src/Functions/FunctionsGeo.h
View File

@ -61,7 +61,7 @@ private:
for (const auto arg_idx : ext::range(0, arguments.size()))
{
const auto arg = arguments[arg_idx].get();
if (!checkDataType<DataTypeFloat64>(arg))
if (!WhichDataType(arg).isFloat64())
throw Exception(
"Illegal type " + arg->getName() + " of argument " + std::to_string(arg_idx + 1) + " of function " + getName() + ". Must be Float64",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -213,7 +213,7 @@ private:
for (const auto arg_idx : ext::range(0, arguments.size()))
{
const auto arg = arguments[arg_idx].get();
if (!checkDataType<DataTypeFloat64>(arg))
if (!WhichDataType(arg).isFloat64())
{
throw Exception(
"Illegal type " + arg->getName() + " of argument " + std::to_string(arg_idx + 1) + " of function " + getName() + ". Must be Float64",

97
dbms/src/Functions/FunctionsHashing.h
View File

@ -200,7 +200,7 @@ public:
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (!arguments[0]->isString())
if (!isString(arguments[0]))
throw Exception("Illegal type " + arguments[0]->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -303,17 +303,18 @@ public:
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
const IDataType * from_type = block.getByPosition(arguments[0]).type.get();
WhichDataType which(from_type);
if (checkDataType<DataTypeUInt8>(from_type)) executeType<UInt8>(block, arguments, result);
else if (checkDataType<DataTypeUInt16>(from_type)) executeType<UInt16>(block, arguments, result);
else if (checkDataType<DataTypeUInt32>(from_type)) executeType<UInt32>(block, arguments, result);
else if (checkDataType<DataTypeUInt64>(from_type)) executeType<UInt64>(block, arguments, result);
else if (checkDataType<DataTypeInt8>(from_type)) executeType<Int8>(block, arguments, result);
else if (checkDataType<DataTypeInt16>(from_type)) executeType<Int16>(block, arguments, result);
else if (checkDataType<DataTypeInt32>(from_type)) executeType<Int32>(block, arguments, result);
else if (checkDataType<DataTypeInt64>(from_type)) executeType<Int64>(block, arguments, result);
else if (checkDataType<DataTypeDate>(from_type)) executeType<UInt16>(block, arguments, result);
else if (checkDataType<DataTypeDateTime>(from_type)) executeType<UInt32>(block, arguments, result);
if (which.isUInt8()) executeType<UInt8>(block, arguments, result);
else if (which.isUInt16()) executeType<UInt16>(block, arguments, result);
else if (which.isUInt32()) executeType<UInt32>(block, arguments, result);
else if (which.isUInt64()) executeType<UInt64>(block, arguments, result);
else if (which.isInt8()) executeType<Int8>(block, arguments, result);
else if (which.isInt16()) executeType<Int16>(block, arguments, result);
else if (which.isInt32()) executeType<Int32>(block, arguments, result);
else if (which.isInt64()) executeType<Int64>(block, arguments, result);
else if (which.isDate()) executeType<UInt16>(block, arguments, result);
else if (which.isDateTime()) executeType<UInt32>(block, arguments, result);
else
throw Exception("Illegal type " + block.getByPosition(arguments[0]).type->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -479,23 +480,25 @@ private:
template <bool first>
void executeAny(const IDataType * from_type, const IColumn * icolumn, ColumnUInt64::Container & vec_to)
{
if (checkDataType<DataTypeUInt8>(from_type)) executeIntType<UInt8, first>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt16>(from_type)) executeIntType<UInt16, first>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt32>(from_type)) executeIntType<UInt32, first>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt64>(from_type)) executeIntType<UInt64, first>(icolumn, vec_to);
else if (checkDataType<DataTypeInt8>(from_type)) executeIntType<Int8, first>(icolumn, vec_to);
else if (checkDataType<DataTypeInt16>(from_type)) executeIntType<Int16, first>(icolumn, vec_to);
else if (checkDataType<DataTypeInt32>(from_type)) executeIntType<Int32, first>(icolumn, vec_to);
else if (checkDataType<DataTypeInt64>(from_type)) executeIntType<Int64, first>(icolumn, vec_to);
else if (checkDataType<DataTypeEnum8>(from_type)) executeIntType<Int8, first>(icolumn, vec_to);
else if (checkDataType<DataTypeEnum16>(from_type)) executeIntType<Int16, first>(icolumn, vec_to);
else if (checkDataType<DataTypeDate>(from_type)) executeIntType<UInt16, first>(icolumn, vec_to);
else if (checkDataType<DataTypeDateTime>(from_type)) executeIntType<UInt32, first>(icolumn, vec_to);
else if (checkDataType<DataTypeFloat32>(from_type)) executeIntType<Float32, first>(icolumn, vec_to);
else if (checkDataType<DataTypeFloat64>(from_type)) executeIntType<Float64, first>(icolumn, vec_to);
else if (checkDataType<DataTypeString>(from_type)) executeString<first>(icolumn, vec_to);
else if (checkDataType<DataTypeFixedString>(from_type)) executeString<first>(icolumn, vec_to);
else if (checkDataType<DataTypeArray>(from_type)) executeArray<first>(from_type, icolumn, vec_to);
WhichDataType which(from_type);
if (which.isUInt8()) executeIntType<UInt8, first>(icolumn, vec_to);
else if (which.isUInt16()) executeIntType<UInt16, first>(icolumn, vec_to);
else if (which.isUInt32()) executeIntType<UInt32, first>(icolumn, vec_to);
else if (which.isUInt64()) executeIntType<UInt64, first>(icolumn, vec_to);
else if (which.isInt8()) executeIntType<Int8, first>(icolumn, vec_to);
else if (which.isInt16()) executeIntType<Int16, first>(icolumn, vec_to);
else if (which.isInt32()) executeIntType<Int32, first>(icolumn, vec_to);
else if (which.isInt64()) executeIntType<Int64, first>(icolumn, vec_to);
else if (which.isEnum8()) executeIntType<Int8, first>(icolumn, vec_to);
else if (which.isEnum16()) executeIntType<Int16, first>(icolumn, vec_to);
else if (which.isDate()) executeIntType<UInt16, first>(icolumn, vec_to);
else if (which.isDateTime()) executeIntType<UInt32, first>(icolumn, vec_to);
else if (which.isFloat32()) executeIntType<Float32, first>(icolumn, vec_to);
else if (which.isFloat64()) executeIntType<Float64, first>(icolumn, vec_to);
else if (which.isString()) executeString<first>(icolumn, vec_to);
else if (which.isFixedString()) executeString<first>(icolumn, vec_to);
else if (which.isArray()) executeArray<first>(from_type, icolumn, vec_to);
else
throw Exception("Unexpected type " + from_type->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -602,23 +605,23 @@ public:
const ColumnWithTypeAndName & col = block.getByPosition(arguments[0]);
const IDataType * from_type = col.type.get();
const IColumn * icolumn = col.column.get();
WhichDataType which(from_type);
if (checkDataType<DataTypeUInt8>(from_type)) executeIntType<UInt8>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt16>(from_type)) executeIntType<UInt16>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt32>(from_type)) executeIntType<UInt32>(icolumn, vec_to);
else if (checkDataType<DataTypeUInt64>(from_type)) executeIntType<UInt64>(icolumn, vec_to);
else if (checkDataType<DataTypeInt8>(from_type)) executeIntType<Int8>(icolumn, vec_to);
else if (checkDataType<DataTypeInt16>(from_type)) executeIntType<Int16>(icolumn, vec_to);
else if (checkDataType<DataTypeInt32>(from_type)) executeIntType<Int32>(icolumn, vec_to);
else if (checkDataType<DataTypeInt64>(from_type)) executeIntType<Int64>(icolumn, vec_to);
else if (checkDataType<DataTypeEnum8>(from_type)) executeIntType<Int8>(icolumn, vec_to);
else if (checkDataType<DataTypeEnum16>(from_type)) executeIntType<Int16>(icolumn, vec_to);
else if (checkDataType<DataTypeDate>(from_type)) executeIntType<UInt16>(icolumn, vec_to);
else if (checkDataType<DataTypeDateTime>(from_type)) executeIntType<UInt32>(icolumn, vec_to);
else if (checkDataType<DataTypeFloat32>(from_type)) executeIntType<Float32>(icolumn, vec_to);
else if (checkDataType<DataTypeFloat64>(from_type)) executeIntType<Float64>(icolumn, vec_to);
else if (checkDataType<DataTypeString>(from_type)) executeString(icolumn, vec_to);
else if (checkDataType<DataTypeFixedString>(from_type)) executeString(icolumn, vec_to);
if (which.isUInt8()) executeIntType<UInt8>(icolumn, vec_to);
else if (which.isUInt16()) executeIntType<UInt16>(icolumn, vec_to);
else if (which.isUInt32()) executeIntType<UInt32>(icolumn, vec_to);
else if (which.isUInt64()) executeIntType<UInt64>(icolumn, vec_to);
else if (which.isInt8()) executeIntType<Int8>(icolumn, vec_to);
else if (which.isInt16()) executeIntType<Int16>(icolumn, vec_to);
else if (which.isInt32()) executeIntType<Int32>(icolumn, vec_to);
else if (which.isInt64()) executeIntType<Int64>(icolumn, vec_to);
else if (which.isEnum8()) executeIntType<Int8>(icolumn, vec_to);
else if (which.isEnum16()) executeIntType<Int16>(icolumn, vec_to);
else if (which.isDate()) executeIntType<UInt16>(icolumn, vec_to);
else if (which.isDateTime()) executeIntType<UInt32>(icolumn, vec_to);
else if (which.isFloat32()) executeIntType<Float32>(icolumn, vec_to);
else if (which.isFloat64()) executeIntType<Float64>(icolumn, vec_to);
else if (which.isStringOrFixedString()) executeString(icolumn, vec_to);
else
throw Exception("Unexpected type " + from_type->getName() + " of argument of function " + getName(),
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -843,13 +846,13 @@ public:
toString(arg_count) + ", should be 1 or 2.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH};
const auto first_arg = arguments.front().get();
if (!checkDataType<DataTypeString>(first_arg))
if (!WhichDataType(first_arg).isString())
throw Exception{"Illegal type " + first_arg->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
if (arg_count == 2)
{
const auto second_arg = arguments.back().get();
if (!second_arg->isInteger())
const auto & second_arg = arguments.back();
if (!isInteger(second_arg))
throw Exception{"Illegal type " + second_arg->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT};
}

34
dbms/src/Functions/FunctionsHigherOrder.h
View File

@ -310,20 +310,15 @@ struct ArraySumImpl
static DataTypePtr getReturnType(const DataTypePtr & expression_return, const DataTypePtr & /*array_element*/)
{
if (checkDataType<DataTypeUInt8>(&*expression_return) ||
checkDataType<DataTypeUInt16>(&*expression_return) ||
checkDataType<DataTypeUInt32>(&*expression_return) ||
checkDataType<DataTypeUInt64>(&*expression_return))
WhichDataType which(expression_return);
if (which.isNativeUInt())
return std::make_shared<DataTypeUInt64>();
if (checkDataType<DataTypeInt8>(&*expression_return) ||
checkDataType<DataTypeInt16>(&*expression_return) ||
checkDataType<DataTypeInt32>(&*expression_return) ||
checkDataType<DataTypeInt64>(&*expression_return))
if (which.isNativeInt())
return std::make_shared<DataTypeInt64>();
if (checkDataType<DataTypeFloat32>(&*expression_return) ||
checkDataType<DataTypeFloat64>(&*expression_return))
if (which.isFloat())
return std::make_shared<DataTypeFloat64>();
throw Exception("arraySum cannot add values of type " + expression_return->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -602,20 +597,15 @@ struct ArrayCumSumImpl
static DataTypePtr getReturnType(const DataTypePtr & expression_return, const DataTypePtr & /*array_element*/)
{
if (checkDataType<DataTypeUInt8>(&*expression_return) ||
checkDataType<DataTypeUInt16>(&*expression_return) ||
checkDataType<DataTypeUInt32>(&*expression_return) ||
checkDataType<DataTypeUInt64>(&*expression_return))
WhichDataType which(expression_return);
if (which.isNativeUInt())
return std::make_shared<DataTypeArray>(std::make_shared<DataTypeUInt64>());
if (checkDataType<DataTypeInt8>(&*expression_return) ||
checkDataType<DataTypeInt16>(&*expression_return) ||
checkDataType<DataTypeInt32>(&*expression_return) ||
checkDataType<DataTypeInt64>(&*expression_return))
if (which.isNativeInt())
return std::make_shared<DataTypeArray>(std::make_shared<DataTypeInt64>());
if (checkDataType<DataTypeFloat32>(&*expression_return) ||
checkDataType<DataTypeFloat64>(&*expression_return))
if (which.isFloat())
return std::make_shared<DataTypeArray>(std::make_shared<DataTypeFloat64>());
throw Exception("arrayCumSum cannot add values of type " + expression_return->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -824,7 +814,7 @@ public:
DataTypePtr nested_type = array_type->getNestedType();
if (Impl::needBoolean() && !checkDataType<DataTypeUInt8>(&*nested_type))
if (Impl::needBoolean() && !WhichDataType(nested_type).isUInt8())
throw Exception("The only argument for function " + getName() + " must be array of UInt8. Found "
+ arguments[0].type->getName() + " instead.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
@ -845,7 +835,7 @@ public:
/// The types of the remaining arguments are already checked in getLambdaArgumentTypes.
DataTypePtr return_type = data_type_function->getReturnType();
if (Impl::needBoolean() && !checkDataType<DataTypeUInt8>(&*return_type))
if (Impl::needBoolean() && !WhichDataType(return_type).isUInt8())
throw Exception("Expression for function " + getName() + " must return UInt8, found "
+ return_type->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);

Some files were not shown because too many files have changed in this diff Show More