Merge branch 'master' into mutations-introspection

Conflicts:
	dbms/src/Interpreters/InterpreterKillQueryQuery.cpp

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,30 @@
+---
+name: Bug report
+about: Create a report to help us improve ClickHouse
+title: ''
+labels: bug, issue
+assignees: ''
+
+---
+
+(you don't have to follow this form)
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**How to reproduce**
+* Which ClickHouse server version to use
+* Which interface to use, if matters
+* Non-default settings, if any
+* `CREATE TABLE` statements for all tables involved
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/yandex/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Queries to run that lead to unexpected result
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Error message and/or stacktrace**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,22 @@
+---
+name: Feature request
+about: Suggest an idea for ClickHouse
+title: ''
+labels: feature
+assignees: ''
+
+---
+
+(you don't have to follow this form)
+
+**Use case.**
+A clear and concise description of what is the intended usage scenario is.
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.gitmodules

@@ -64,3 +64,6 @@
 [submodule "contrib/cppkafka"]
 	path = contrib/cppkafka
 	url = https://github.com/mfontanini/cppkafka.git
+[submodule "contrib/pdqsort"]
+	path = contrib/pdqsort
+	url = https://github.com/orlp/pdqsort

CHANGELOG.md

@@ -8,7 +8,7 @@
 * Added functions `left`, `right`, `trim`, `ltrim`, `rtrim`, `timestampadd`, `timestampsub` for SQL standard compatibility. [#3826](https://github.com/yandex/ClickHouse/pull/3826) ([Ivan Blinkov](https://github.com/blinkov))
 * Support for write in `HDFS` tables and `hdfs` table function. [#4084](https://github.com/yandex/ClickHouse/pull/4084) ([alesapin](https://github.com/alesapin))
 * Added functions to search for multiple constant strings from big haystack: `multiPosition`, `multiSearch` ,`firstMatch` also with `-UTF8`, `-CaseInsensitive`, and `-CaseInsensitiveUTF8` variants. [#4053](https://github.com/yandex/ClickHouse/pull/4053) ([Danila Kutenin](https://github.com/danlark1))
-* Pruning of unused shards if `SELECT` query filters by sharding key (setting `distributed_optimize_skip_select_on_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([Ivan](https://github.com/abyss7))
+* Pruning of unused shards if `SELECT` query filters by sharding key (setting `distributed_optimize_skip_select_on_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([Gleb Kanterov](https://github.com/kanterov), [Ivan](https://github.com/abyss7))
 * Allow `Kafka` engine to ignore some number of parsing errors per block. [#4094](https://github.com/yandex/ClickHouse/pull/4094) ([Ivan](https://github.com/abyss7))
 * Added support for `CatBoost` multiclass models evaluation. Function `modelEvaluate` returns tuple with per-class raw predictions for multiclass models. `libcatboostmodel.so` should be built with [#607](https://github.com/catboost/catboost/pull/607). [#3959](https://github.com/yandex/ClickHouse/pull/3959) ([KochetovNicolai](https://github.com/KochetovNicolai))
 * Added functions `filesystemAvailable`, `filesystemFree`, `filesystemCapacity`. [#4097](https://github.com/yandex/ClickHouse/pull/4097) ([Boris Granveaud](https://github.com/bgranvea))

CHANGELOG_RU.md

@@ -1,3 +1,112 @@
+## ClickHouse release 19.1.6, 2019-01-24
+
+### Новые возможности:
+
+* Задание формата сжатия для отдельных столбцов. [#3899](https://github.com/yandex/ClickHouse/pull/3899) [#4111](https://github.com/yandex/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin), [Winter Zhang](https://github.com/zhang2014), [Anatoly](https://github.com/Sindbag))
+* Формат сжатия `Delta`. [#4052](https://github.com/yandex/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin))
+* Изменение формата сжатия запросом `ALTER`. [#4054](https://github.com/yandex/ClickHouse/pull/4054) ([alesapin](https://github.com/alesapin))
+* Добавлены функции `left`, `right`, `trim`, `ltrim`, `rtrim`, `timestampadd`, `timestampsub` для совместимости со стандартом SQL. [#3826](https://github.com/yandex/ClickHouse/pull/3826) ([Ivan Blinkov](https://github.com/blinkov))
+* Поддержка записи в движок `HDFS` и табличную функцию `hdfs`. [#4084](https://github.com/yandex/ClickHouse/pull/4084) ([alesapin](https://github.com/alesapin))
+* Добавлены функции поиска набора константных строк в тексте: `multiPosition`, `multiSearch` ,`firstMatch` также с суффиксами `-UTF8`, `-CaseInsensitive`, и `-CaseInsensitiveUTF8`. [#4053](https://github.com/yandex/ClickHouse/pull/4053) ([Danila Kutenin](https://github.com/danlark1))
+* Пропуск неиспользуемых шардов в случае, если запрос `SELECT` содержит фильтрацию по ключу шардирования (настройка `distributed_optimize_skip_select_on_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([Gleb Kanterov](https://github.com/kanterov), [Ivan](https://github.com/abyss7))
+* Пропуск строк в случае ошибки парсинга для движка `Kafka` (настройка `kafka_skip_broken_messages`). [#4094](https://github.com/yandex/ClickHouse/pull/4094) ([Ivan](https://github.com/abyss7))
+* Поддержка применения мультиклассовых моделей `CatBoost`. Функция `modelEvaluate` возвращает кортеж в случае использования мультиклассовой модели. `libcatboostmodel.so` should be built with [#607](https://github.com/catboost/catboost/pull/607). [#3959](https://github.com/yandex/ClickHouse/pull/3959) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Добавлены функции `filesystemAvailable`, `filesystemFree`, `filesystemCapacity`. [#4097](https://github.com/yandex/ClickHouse/pull/4097) ([Boris Granveaud](https://github.com/bgranvea))
+* Добавлены функции хеширования `xxHash64` и `xxHash32`. [#3905](https://github.com/yandex/ClickHouse/pull/3905) ([filimonov](https://github.com/filimonov))
+* Добавлена функция хеширования `gccMurmurHash` (GCC flavoured Murmur hash), использующая те же hash seed, что и [gcc](https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/libstdc%2B%2B-v3/include/bits/functional_hash.h#L191) [#4000](https://github.com/yandex/ClickHouse/pull/4000) ([sundyli](https://github.com/sundy-li))
+* Добавлены функции хеширования `javaHash`, `hiveHash`. [#3811](https://github.com/yandex/ClickHouse/pull/3811) ([shangshujie365](https://github.com/shangshujie365))
+* Добавлена функция `remoteSecure`. Функция работает аналогично `remote`, но использует безопасное соединение. [#4088](https://github.com/yandex/ClickHouse/pull/4088) ([proller](https://github.com/proller))
+
+
+### Экспериментальные возможности:
+
+* Эмуляция запросов с несколькими секциями `JOIN` (настройка `allow_experimental_multiple_joins_emulation`). [#3946](https://github.com/yandex/ClickHouse/pull/3946) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Исправления ошибок:
+
+* Ограничен размер кеша скомпилированных выражений в случае, если не указана настройка `compiled_expression_cache_size` для экономии потребляемой памяти. [#4041](https://github.com/yandex/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin))
+* Исправлена проблема зависания потоков, выполняющих запрос `ALTER` для таблиц семейства `Replicated`, а также потоков, обновляющих конфигурацию из ZooKeeper. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3891](https://github.com/yandex/ClickHouse/issues/3891) [#3934](https://github.com/yandex/ClickHouse/pull/3934) ([Alex Zatelepin](https://github.com/ztlpn))
+* Исправлен race condition в случае выполнения распределенной задачи запроса `ALTER`. Race condition приводил к состоянию, когда более чем одна реплика пыталась выполнить задачу, в результате чего все такие реплики, кроме одной, падали с ошибкой обращения к ZooKeeper. [#3904](https://github.com/yandex/ClickHouse/pull/3904) ([Alex Zatelepin](https://github.com/ztlpn))
+* Исправлена проблема обновления настройки `from_zk`. Настройка, указанная в файле конфигурации, не обновлялась в случае, если запрос к ZooKeeper падал по timeout. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3947](https://github.com/yandex/ClickHouse/pull/3947) ([Alex Zatelepin](https://github.com/ztlpn))
+* Исправлена ошибка в вычислении сетевого префикса при указании IPv4 маски подсети. [#3945](https://github.com/yandex/ClickHouse/pull/3945) ([alesapin](https://github.com/alesapin))
+* Исправлено падение (`std::terminate`) в редком сценарии, когда новый поток не мог быть создан из-за нехватки ресурсов. [#3956](https://github.com/yandex/ClickHouse/pull/3956) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлено падение табличной функции `remote` в случае, когда не удавалось получить структуру таблицы из-за ограничений пользователя. [#4009](https://github.com/yandex/ClickHouse/pull/4009) ([alesapin](https://github.com/alesapin))
+* Исправлена утечка сетевых сокетов. Сокеты создавались в пуле и никогда не закрывались. При создании потока, создавались новые сокеты в случае, если все доступные использовались. [#4017](https://github.com/yandex/ClickHouse/pull/4017) ([Alex Zatelepin](https://github.com/ztlpn))
+* Исправлена проблема закрывания `/proc/self/fd` раньше, чем все файловые дескрипторы были прочитаны из `/proc` после создания процесса `odbc-bridge`. [#4120](https://github.com/yandex/ClickHouse/pull/4120) ([alesapin](https://github.com/alesapin))
+* Исправлен баг в монотонном преобразовании String в UInt в случае использования String в первичном ключе. [#3870](https://github.com/yandex/ClickHouse/pull/3870) ([Winter Zhang](https://github.com/zhang2014))
+* Исправлен баг в вычислении монотонности функции преобразования типа целых значений. [#3921](https://github.com/yandex/ClickHouse/pull/3921) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлено падение в функциях `arrayEnumerateUniq`, `arrayEnumerateDense` при передаче невалидных аргументов. [#3909](https://github.com/yandex/ClickHouse/pull/3909) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлен undefined behavior в StorageMerge. [#3910](https://github.com/yandex/ClickHouse/pull/3910) ([Amos Bird](https://github.com/amosbird))
+* Исправлено падение в функциях `addDays`, `subtractDays`. [#3913](https://github.com/yandex/ClickHouse/pull/3913) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена проблема, в результате которой функции `round`, `floor`, `trunc`, `ceil` могли возвращать неверный результат для отрицательных целочисленных аргументов с большим значением. [#3914](https://github.com/yandex/ClickHouse/pull/3914) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена проблема, в результате которой 'kill query sync' приводил к падению сервера. [#3916](https://github.com/yandex/ClickHouse/pull/3916) ([muVulDeePecker](https://github.com/fancyqlx))
+* Исправлен баг, приводящий к большой задержке в случае пустой очереди репликации. [#3928](https://github.com/yandex/ClickHouse/pull/3928) [#3932](https://github.com/yandex/ClickHouse/pull/3932) ([alesapin](https://github.com/alesapin))
+* Исправлено избыточное использование памяти в случае вставки в таблицу с `LowCardinality` в первичном ключе. [#3955](https://github.com/yandex/ClickHouse/pull/3955) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Исправлена сериализация пустых массивов типа `LowCardinality` для формата `Native`. [#3907](https://github.com/yandex/ClickHouse/issues/3907) [#4011](https://github.com/yandex/ClickHouse/pull/4011) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Исправлен неверный результат в случае использования distinct для числового столбца `LowCardinality`. [#3895](https://github.com/yandex/ClickHouse/issues/3895) [#4012](https://github.com/yandex/ClickHouse/pull/4012) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Исправлена компиляция вычисления агрегатных функций для ключа `LowCardinality` (для случая, когда включена настройка `compile`). [#3886](https://github.com/yandex/ClickHouse/pull/3886) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Исправлена передача пользователя и пароля для запросов с реплик. [#3957](https://github.com/yandex/ClickHouse/pull/3957) ([alesapin](https://github.com/alesapin)) ([小路](https://github.com/nicelulu))
+* Исправлен очень редкий race condition возникающий при перечислении таблиц из базы данных типа `Dictionary` во время перезагрузки словарей. [#3970](https://github.com/yandex/ClickHouse/pull/3970) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлен неверный результат в случае использования HAVING с ROLLUP или CUBE. [#3756](https://github.com/yandex/ClickHouse/issues/3756) [#3837](https://github.com/yandex/ClickHouse/pull/3837) ([Sam Chou](https://github.com/reflection))
+* Исправлена проблема с алиасами столбцов для запросов с `JOIN ON` над распределенными таблицами. [#3980](https://github.com/yandex/ClickHouse/pull/3980) ([Winter Zhang](https://github.com/zhang2014))
+* Исправлена ошибка в реализации функции `quantileTDigest` (нашел Artem Vakhrushev). Эта ошибка никогда не происходит в ClickHouse и актуальна только для тех, кто использует кодовую базу ClickHouse напрямую в качестве библиотеки. [#3935](https://github.com/yandex/ClickHouse/pull/3935) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Улучшения:
+
+* Добавлена поддержка `IF NOT EXISTS` в выражении `ALTER TABLE ADD COLUMN`, `IF EXISTS` в выражении `DROP/MODIFY/CLEAR/COMMENT COLUMN`. [#3900](https://github.com/yandex/ClickHouse/pull/3900) ([Boris Granveaud](https://github.com/bgranvea))
+* Функция `parseDateTimeBestEffort` теперь поддерживает форматы `DD.MM.YYYY`, `DD.MM.YY`, `DD-MM-YYYY`, `DD-Mon-YYYY`, `DD/Month/YYYY` и аналогичные. [#3922](https://github.com/yandex/ClickHouse/pull/3922) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* `CapnProtoInputStream` теперь поддерживает jagged структуры. [#4063](https://github.com/yandex/ClickHouse/pull/4063) ([Odin Hultgren Van Der Horst](https://github.com/Miniwoffer))
+* Улучшение usability: добавлена проверка, что сервер запущен от пользователя, совпадающего с владельцем директории данных. Запрещен запуск от пользователя root в случае, если root не владеет директорией с данными. [#3785](https://github.com/yandex/ClickHouse/pull/3785) ([sergey-v-galtsev](https://github.com/sergey-v-galtsev))
+* Улучшена логика проверки столбцов, необходимых для JOIN, на стадии анализа запроса. [#3930](https://github.com/yandex/ClickHouse/pull/3930) ([Artem Zuikov](https://github.com/4ertus2))
+* Уменьшено число поддерживаемых соединений в случае большого числа распределенных таблиц. [#3726](https://github.com/yandex/ClickHouse/pull/3726) ([Winter Zhang](https://github.com/zhang2014))
+* Добавлена поддержка строки с totals для запроса с `WITH TOTALS` через ODBC драйвер. [#3836](https://github.com/yandex/ClickHouse/pull/3836) ([Maksim Koritckiy](https://github.com/nightweb))
+* Поддержано использование `Enum` в качестве чисел в функции `if`. [#3875](https://github.com/yandex/ClickHouse/pull/3875) ([Ivan](https://github.com/abyss7))
+* Добавлена настройка `low_cardinality_allow_in_native_format`. Если она выключена, то тип `LowCadrinality` не используется в формате `Native`. [#3879](https://github.com/yandex/ClickHouse/pull/3879) ([KochetovNicolai](https://github.com/KochetovNicolai))
+* Удалены некоторые избыточные объекты из кеша скомпилированных выражений для уменьшения потребления памяти. [#4042](https://github.com/yandex/ClickHouse/pull/4042) ([alesapin](https://github.com/alesapin))
+* Добавлена проверка того, что в запрос `SET send_logs_level = 'value'` передается верное значение. [#3873](https://github.com/yandex/ClickHouse/pull/3873) ([Sabyanin Maxim](https://github.com/s-mx))
+* Добавлена проверка типов для функций преобразования типов. [#3896](https://github.com/yandex/ClickHouse/pull/3896) ([Winter Zhang](https://github.com/zhang2014))
+
+### Улучшения производительности:
+
+* Добавлена настройка `use_minimalistic_part_header_in_zookeeper` для движка MergeTree. Если настройка включена, Replicated таблицы будут хранить метаданные куска в компактном виде (в соответствующем znode для этого куска). Это может значительно уменьшить размер для ZooKeeper snapshot (особенно для таблиц с большим числом столбцов). После включения данной настройки будет невозможно сделать откат к версии, которая эту настройку не поддерживает. [#3960](https://github.com/yandex/ClickHouse/pull/3960) ([Alex Zatelepin](https://github.com/ztlpn))
+* Добавлена реализация функций `sequenceMatch` и `sequenceCount` на основе конечного автомата в случае, если последовательность событий не содержит условия на время. [#4004](https://github.com/yandex/ClickHouse/pull/4004) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
+* Улучшена производительность сериализации целых чисел. [#3968](https://github.com/yandex/ClickHouse/pull/3968) ([Amos Bird](https://github.com/amosbird))
+* Добавлен zero left padding для PODArray. Теперь элемент с индексом -1 является валидным нулевым значением. Эта особенность используется для удаления условного выражения при вычислении оффсетов массивов. [#3920](https://github.com/yandex/ClickHouse/pull/3920) ([Amos Bird](https://github.com/amosbird))
+* Откат версии `jemalloc`, приводящей к деградации производительности. [#4018](https://github.com/yandex/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Обратно несовместимые изменения:
+
+* Удалена недокументированная возможность `ALTER MODIFY PRIMARY KEY`, замененная выражением `ALTER MODIFY ORDER BY`. [#3887](https://github.com/yandex/ClickHouse/pull/3887) ([Alex Zatelepin](https://github.com/ztlpn))
+* Удалена функция `shardByHash`. [#3833](https://github.com/yandex/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Запрещено использование скалярных подзапросов с результатом, имеющим тип `AggregateFunction`. [#3865](https://github.com/yandex/ClickHouse/pull/3865) ([Ivan](https://github.com/abyss7))
+
+### Улучшения сборки/тестирования/пакетирования:
+
+* Добавлена поддержка сборки под PowerPC (`ppc64le`). [#4132](https://github.com/yandex/ClickHouse/pull/4132) ([Danila Kutenin](https://github.com/danlark1))
+* Функциональные stateful тесты запускаются на публично доступных данных. [#3969](https://github.com/yandex/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена ошибка, при которой сервер не мог запуститься с сообщением `bash: /usr/bin/clickhouse-extract-from-config: Operation not permitted` при использовании Docker или systemd-nspawn. [#4136](https://github.com/yandex/ClickHouse/pull/4136) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Обновлена библиотека `rdkafka` до версии v1.0.0-RC5. Использована cppkafka на замену интерфейса языка C. [#4025](https://github.com/yandex/ClickHouse/pull/4025) ([Ivan](https://github.com/abyss7))
+* Обновлена библиотека `mariadb-client`. Исправлена проблема, обнаруженная с использованием UBSan. [#3924](https://github.com/yandex/ClickHouse/pull/3924) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправления для сборок с UBSan. [#3926](https://github.com/yandex/ClickHouse/pull/3926) [#3021](https://github.com/yandex/ClickHouse/pull/3021) [#3948](https://github.com/yandex/ClickHouse/pull/3948) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Добавлены покоммитные запуски тестов с UBSan сборкой.
+* Добавлены покоммитные запуски тестов со статическим анализатором PVS-Studio.
+* Исправлены проблемы, найденные с использованием PVS-Studio. [#4013](https://github.com/yandex/ClickHouse/pull/4013) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлены проблемы совместимости glibc. [#4100](https://github.com/yandex/ClickHouse/pull/4100) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Docker образы перемещены на Ubuntu 18.10, добавлена совместимость с glibc >= 2.28 [#3965](https://github.com/yandex/ClickHouse/pull/3965) ([alesapin](https://github.com/alesapin))
+* Добавлена переменная окружения `CLICKHOUSE_DO_NOT_CHOWN`, позволяющая не делать shown директории для Docker образа сервера. [#3967](https://github.com/yandex/ClickHouse/pull/3967) ([alesapin](https://github.com/alesapin))
+* Включены большинство предупреждений из `-Weverything` для clang. Включено `-Wpedantic`. [#3986](https://github.com/yandex/ClickHouse/pull/3986) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Добавлены некоторые предупреждения, специфичные только для clang 8. [#3993](https://github.com/yandex/ClickHouse/pull/3993) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* При использовании динамической линковки используется `libLLVM` вместо библиотеки `LLVM`. [#3989](https://github.com/yandex/ClickHouse/pull/3989) ([Orivej Desh](https://github.com/orivej))
+* Добавлены переменные окружения для параметров `TSan`, `UBSan`, `ASan` в тестовом Docker образе. [#4072](https://github.com/yandex/ClickHouse/pull/4072) ([alesapin](https://github.com/alesapin))
+* Debian пакет `clickhouse-server` будет рекомендовать пакет `libcap2-bin` для того, чтобы использовать утилиту `setcap` для настроек. Данный пакет опционален. [#4093](https://github.com/yandex/ClickHouse/pull/4093) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Уменьшено время сборки, убраны ненужные включения заголовочных файлов. [#3898](https://github.com/yandex/ClickHouse/pull/3898) ([proller](https://github.com/proller))
+* Добавлены тесты производительности для функций хеширования. [#3918](https://github.com/yandex/ClickHouse/pull/3918) ([filimonov](https://github.com/filimonov))
+* Исправлены циклические зависимости библиотек. [#3958](https://github.com/yandex/ClickHouse/pull/3958) ([proller](https://github.com/proller))
+* Улучшена компиляция при малом объеме памяти. [#4030](https://github.com/yandex/ClickHouse/pull/4030) ([proller](https://github.com/proller))
+* Добавлен тестовый скрипт для воспроизведения деградации производительности в `jemalloc`. [#4036](https://github.com/yandex/ClickHouse/pull/4036) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправления опечаток в комментариях и строковых литералах. [#4122](https://github.com/yandex/ClickHouse/pull/4122) ([maiha](https://github.com/maiha))
+* Исправления опечаток в комментариях. [#4089](https://github.com/yandex/ClickHouse/pull/4089) ([Evgenii Pravda](https://github.com/kvinty))
+
 ## ClickHouse release 18.16.1, 2018-12-21
 
 ### Исправления ошибок:

CMakeLists.txt

@@ -96,7 +96,7 @@ option (ENABLE_TESTS "Enables tests" ON)
 if (CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64")
     option (USE_INTERNAL_MEMCPY "Use internal implementation of 'memcpy' function instead of provided by libc. Only for x86_64." ON)
 
-    if (OS_LINUX AND NOT UNBUNDLED AND MAKE_STATIC_LIBRARIES)
+    if (OS_LINUX AND NOT UNBUNDLED AND MAKE_STATIC_LIBRARIES AND CMAKE_VERSION VERSION_GREATER "3.9.0")
         option (GLIBC_COMPATIBILITY "Set to TRUE to enable compatibility with older glibc libraries. Only for x86_64, Linux. Implies USE_INTERNAL_MEMCPY." ON)
         if (GLIBC_COMPATIBILITY)
             message (STATUS "Some symbols from glibc will be replaced for compatibility")
@@ -253,6 +253,7 @@ endif()
 include (cmake/find_libgsasl.cmake)
 include (cmake/find_libxml2.cmake)
 include (cmake/find_protobuf.cmake)
+include (cmake/find_pdqsort.cmake)
 include (cmake/find_hdfs3.cmake)
 include (cmake/find_consistent-hashing.cmake)
 include (cmake/find_base64.cmake)

README.md

@@ -13,4 +13,5 @@ ClickHouse is an open-source column-oriented database management system that all
 
 ## Upcoming Events
 
-* [C++ ClickHouse and CatBoost Sprints](https://events.yandex.ru/events/ClickHouse/2-feb-2019/) in Moscow on February 2.
+* [ClickHouse Community Meetup](https://www.eventbrite.com/e/meetup-clickhouse-in-the-wild-deployment-success-stories-registration-55305051899) in San Francisco on February 19.
+* [ClickHouse Community Meetup](https://www.eventbrite.com/e/clickhouse-meetup-in-madrid-registration-55376746339) in Madrid on April 2.

cmake/find_pdqsort.cmake

@@ -0,0 +1,2 @@
+set(PDQSORT_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/pdqsort)
+message(STATUS "Using pdqsort: ${PDQSORT_INCLUDE_DIR}")

cmake/find_protobuf.cmake

@@ -1,5 +1,11 @@
 option(USE_INTERNAL_PROTOBUF_LIBRARY "Set to FALSE to use system protobuf instead of bundled" ${NOT_UNBUNDLED})
 
+if(OS_FREEBSD AND SANITIZE STREQUAL "address")
+    # ../contrib/protobuf/src/google/protobuf/arena_impl.h:45:10: fatal error: 'sanitizer/asan_interface.h' file not found
+    set(MISSING_INTERNAL_PROTOBUF_LIBRARY 1)
+    set(USE_INTERNAL_PROTOBUF_LIBRARY 0)
+endif()
+
 if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/protobuf/cmake/CMakeLists.txt")
    if(USE_INTERNAL_PROTOBUF_LIBRARY)
        message(WARNING "submodule contrib/protobuf is missing. to fix try run: \n git submodule update --init --recursive")

cmake/find_re2.cmake

@@ -5,13 +5,24 @@ if (NOT USE_INTERNAL_RE2_LIBRARY)
     find_path (RE2_INCLUDE_DIR NAMES re2/re2.h PATHS ${RE2_INCLUDE_PATHS})
 endif ()
 
+string(FIND ${CMAKE_CURRENT_BINARY_DIR} " " _have_space)
+if(_have_space GREATER 0)
+    message(WARNING "Using spaces in build path [${CMAKE_CURRENT_BINARY_DIR}] highly not recommended. Library re2st will be disabled.")
+    set (MISSING_INTERNAL_RE2_ST_LIBRARY 1)
+endif()
+
 if (RE2_LIBRARY AND RE2_INCLUDE_DIR)
     set (RE2_ST_LIBRARY ${RE2_LIBRARY})
-else ()
+elseif (NOT MISSING_INTERNAL_RE2_LIBRARY)
     set (USE_INTERNAL_RE2_LIBRARY 1)
     set (RE2_LIBRARY re2)
-    set (RE2_ST_LIBRARY re2_st)
-    set (USE_RE2_ST 1)
+    set (RE2_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/re2)
+    if (NOT MISSING_INTERNAL_RE2_ST_LIBRARY)
+        set (RE2_ST_LIBRARY re2_st)
+        set (USE_RE2_ST 1)
+    else ()
+        set (RE2_ST_LIBRARY ${RE2_LIBRARY})
+    endif ()
 endif ()
 
 message (STATUS "Using re2: ${RE2_INCLUDE_DIR} : ${RE2_LIBRARY}; ${RE2_ST_INCLUDE_DIR} : ${RE2_ST_LIBRARY}")

contrib/CMakeLists.txt

@@ -8,6 +8,8 @@ elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
     set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")
 endif ()
 
+set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL 1)
+
 if (USE_INTERNAL_BOOST_LIBRARY)
     add_subdirectory (boost-cmake)
 endif ()

contrib/base64-cmake/CMakeLists.txt

@@ -39,5 +39,10 @@ add_library(base64 ${LINK_MODE}
         ${LIBRARY_DIR}/lib/codecs.h
         ${CMAKE_CURRENT_BINARY_DIR}/config.h)
 
-target_compile_options(base64 PRIVATE ${base64_SSSE3_opt} ${base64_SSE41_opt} ${base64_SSE42_opt} ${base64_AVX_opt} ${base64_AVX2_opt})
+set_source_files_properties(${LIBRARY_DIR}/lib/arch/avx/codec.c PROPERTIES COMPILE_FLAGS -mavx)
+set_source_files_properties(${LIBRARY_DIR}/lib/arch/avx2/codec.c PROPERTIES COMPILE_FLAGS -mavx2)
+set_source_files_properties(${LIBRARY_DIR}/lib/arch/sse41/codec.c PROPERTIES COMPILE_FLAGS -msse4.1)
+set_source_files_properties(${LIBRARY_DIR}/lib/arch/sse42/codec.c PROPERTIES COMPILE_FLAGS -msse4.2)
+set_source_files_properties(${LIBRARY_DIR}/lib/arch/ssse3/codec.c PROPERTIES COMPILE_FLAGS -mssse3)
+
 target_include_directories(base64 PRIVATE ${LIBRARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR})

contrib/pdqsort

@@ -0,0 +1 @@
+Subproject commit 08879029ab8dcb80a70142acb709e3df02de5d37

dbms/CMakeLists.txt

@@ -206,6 +206,10 @@ target_link_libraries (clickhouse_common_io
     ${CMAKE_DL_LIBS}
 )
 
+target_include_directories(clickhouse_common_io SYSTEM BEFORE PUBLIC ${PDQSORT_INCLUDE_DIR})
+
+target_include_directories(clickhouse_common_io SYSTEM BEFORE PUBLIC ${RE2_INCLUDE_DIR})
+
 if(CPUID_LIBRARY)
     target_link_libraries(clickhouse_common_io PRIVATE ${CPUID_LIBRARY})
 endif()
@@ -235,9 +239,6 @@ target_link_libraries (dbms
     Threads::Threads
 )
 
-if (NOT USE_INTERNAL_RE2_LIBRARY)
-    target_include_directories (dbms SYSTEM BEFORE PRIVATE ${RE2_INCLUDE_DIR})
-endif ()
 
 if (NOT USE_INTERNAL_BOOST_LIBRARY)
     target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
@@ -257,7 +258,6 @@ if (USE_POCO_SQLODBC)
     endif()
 endif()
 
-#if (Poco_Data_FOUND AND NOT USE_INTERNAL_POCO_LIBRARY)
 if (Poco_Data_FOUND)
     target_include_directories (clickhouse_common_io SYSTEM PRIVATE ${Poco_Data_INCLUDE_DIR})
     target_include_directories (dbms SYSTEM PRIVATE ${Poco_Data_INCLUDE_DIR})
@@ -284,6 +284,7 @@ target_link_libraries (dbms PRIVATE ${Poco_Foundation_LIBRARY})
 
 if (USE_ICU)
     target_link_libraries (dbms PRIVATE ${ICU_LIBRARIES})
+    target_include_directories (dbms SYSTEM PRIVATE ${ICU_INCLUDE_DIRS})
 endif ()
 
 if (USE_CAPNP)

dbms/programs/CMakeLists.txt

@@ -28,11 +28,18 @@ add_subdirectory (copier)
 add_subdirectory (format)
 add_subdirectory (clang)
 add_subdirectory (obfuscator)
-add_subdirectory (odbc-bridge)
+
+if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
+    add_subdirectory (odbc-bridge)
+endif ()
 
 if (CLICKHOUSE_SPLIT_BINARY)
     set (CLICKHOUSE_ALL_TARGETS clickhouse-server clickhouse-client clickhouse-local clickhouse-benchmark clickhouse-performance-test
-            clickhouse-extract-from-config clickhouse-compressor clickhouse-format clickhouse-copier clickhouse-odbc-bridge)
+            clickhouse-extract-from-config clickhouse-compressor clickhouse-format clickhouse-copier)
+
+    if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
+        list (APPEND CLICKHOUSE_ALL_TARGETS clickhouse-odbc-bridge)
+    endif ()
 
     if (USE_EMBEDDED_COMPILER)
         list (APPEND CLICKHOUSE_ALL_TARGETS clickhouse-clang clickhouse-lld)
@@ -85,9 +92,6 @@ else ()
     if (USE_EMBEDDED_COMPILER)
         target_link_libraries (clickhouse PRIVATE clickhouse-compiler-lib)
     endif ()
-    if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
-        target_link_libraries (clickhouse PRIVATE clickhouse-odbc-bridge-lib)
-    endif()
 
     set (CLICKHOUSE_BUNDLE)
     if (ENABLE_CLICKHOUSE_SERVER)
@@ -135,15 +139,14 @@ else ()
         install (FILES ${CMAKE_CURRENT_BINARY_DIR}/clickhouse-format DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
         list(APPEND CLICKHOUSE_BUNDLE clickhouse-format)
     endif ()
-    if (ENABLE_CLICKHOUSE_COPIER)
+    if (ENABLE_CLICKHOUSE_OBFUSCATOR)
         add_custom_target (clickhouse-obfuscator ALL COMMAND ${CMAKE_COMMAND} -E create_symlink clickhouse clickhouse-obfuscator DEPENDS clickhouse)
         install (FILES ${CMAKE_CURRENT_BINARY_DIR}/clickhouse-obfuscator DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
         list(APPEND CLICKHOUSE_BUNDLE clickhouse-obfuscator)
     endif ()
     if (ENABLE_CLICKHOUSE_ODBC_BRIDGE)
-        add_custom_target (clickhouse-odbc-bridge ALL COMMAND ${CMAKE_COMMAND} -E create_symlink clickhouse clickhouse-odbc-bridge DEPENDS clickhouse)
-        install (FILES ${CMAKE_CURRENT_BINARY_DIR}/clickhouse-odbc-bridge DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
-        list(APPEND CLICKHOUSE_BUNDLE clickhouse-odbc-bridge)
+        # just to be able to run integration tests
+        add_custom_target (clickhouse-odbc-bridge-copy ALL COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_CURRENT_BINARY_DIR}/odbc-bridge/clickhouse-odbc-bridge clickhouse-odbc-bridge DEPENDS clickhouse-odbc-bridge)
     endif ()
 
 

dbms/programs/client/Client.cpp

@@ -1542,12 +1542,19 @@ public:
         po::options_description main_description("Main options", line_length, min_description_length);
         main_description.add_options()
             ("help", "produce help message")
-            ("config-file,c", po::value<std::string>(), "config-file path")
+            ("config-file,C", po::value<std::string>(), "config-file path")
+            ("config,c", po::value<std::string>(), "config-file path (another shorthand)")
             ("host,h", po::value<std::string>()->default_value("localhost"), "server host")
             ("port", po::value<int>()->default_value(9000), "server port")
             ("secure,s", "Use TLS connection")
             ("user,u", po::value<std::string>()->default_value("default"), "user")
-            ("password", po::value<std::string>(), "password")
+            /** If "--password [value]" is used but the value is omitted, the bad argument exception will be thrown.
+              * implicit_value is used to avoid this exception (to allow user to type just "--password")
+              * Since currently boost provides no way to check if a value has been set implicitly for an option,
+              * the "\n" is used to distinguish this case because there is hardly a chance an user would use "\n"
+              * as the password.
+              */
+            ("password", po::value<std::string>()->implicit_value("\n"), "password")
             ("ask-password", "ask-password")
             ("query_id", po::value<std::string>(), "query_id")
             ("query,q", po::value<std::string>(), "query")
@@ -1585,13 +1592,11 @@ public:
             ("structure", po::value<std::string>(), "structure")
             ("types", po::value<std::string>(), "types")
         ;
-
         /// Parse main commandline options.
         po::parsed_options parsed = po::command_line_parser(
             common_arguments.size(), common_arguments.data()).options(main_description).run();
         po::variables_map options;
         po::store(parsed, options);
-
         if (options.count("version") || options.count("V"))
         {
             showClientVersion();
@@ -1649,9 +1654,14 @@ public:
         APPLY_FOR_SETTINGS(EXTRACT_SETTING)
 #undef EXTRACT_SETTING
 
+        if (options.count("config-file") && options.count("config"))
+            throw Exception("Two or more configuration files referenced in arguments", ErrorCodes::BAD_ARGUMENTS);
+
         /// Save received data into the internal config.
         if (options.count("config-file"))
             config().setString("config-file", options["config-file"].as<std::string>());
+        if (options.count("config"))
+            config().setString("config-file", options["config"].as<std::string>());
         if (options.count("host") && !options["host"].defaulted())
             config().setString("host", options["host"].as<std::string>());
         if (options.count("query_id"))
@@ -1710,11 +1720,11 @@ public:
 
 int mainEntryClickHouseClient(int argc, char ** argv)
 {
-    DB::Client client;
-
     try
     {
+        DB::Client client;
         client.init(argc, argv);
+        return client.run();
     }
     catch (const boost::program_options::error & e)
     {
@@ -1726,6 +1736,4 @@ int mainEntryClickHouseClient(int argc, char ** argv)
         std::cerr << DB::getCurrentExceptionMessage(true) << std::endl;
         return 1;
     }
-
-    return client.run();
 }

dbms/programs/client/ConnectionParameters.h

@@ -8,7 +8,7 @@
 #include <Common/Exception.h>
 #include <IO/ConnectionTimeouts.h>
 
-#include <common/SetTerminalEcho.h>
+#include <common/setTerminalEcho.h>
 #include <ext/scope_guard.h>
 
 #include <Poco/Util/AbstractConfiguration.h>
@@ -48,27 +48,33 @@ struct ConnectionParameters
                 is_secure ? DBMS_DEFAULT_SECURE_PORT : DBMS_DEFAULT_PORT));
 
         default_database = config.getString("database", "");
-        user = config.getString("user", "");
-
+        /// changed the default value to "default" to fix the issue when the user in the prompt is blank
+        user = config.getString("user", "default");
+        bool password_prompt = false;
         if (config.getBool("ask-password", false))
         {
             if (config.has("password"))
                 throw Exception("Specified both --password and --ask-password. Remove one of them", ErrorCodes::BAD_ARGUMENTS);
-
-            std::cout << "Password for user " << user << ": ";
-            SetTerminalEcho(false);
-
-            SCOPE_EXIT({
-                SetTerminalEcho(true);
-            });
-            std::getline(std::cin, password);
-            std::cout << std::endl;
+            password_prompt = true;
         }
         else
         {
             password = config.getString("password", "");
+            /// if the value of --password is omitted, the password will be set implicitly to "\n"
+            if (password == "\n")
+                password_prompt = true;
         }
+        if (password_prompt)
+        {
+            std::cout << "Password for user (" << user << "): ";
+            setTerminalEcho(false);
 
+            SCOPE_EXIT({
+                setTerminalEcho(true);
+            });
+            std::getline(std::cin, password);
+            std::cout << std::endl;
+        }
         compression = config.getBool("compression", true)
             ? Protocol::Compression::Enable
             : Protocol::Compression::Disable;

dbms/programs/copier/ClusterCopier.cpp

@@ -1179,7 +1179,7 @@ protected:
     /// Removes MATERIALIZED and ALIAS columns from create table query
     static ASTPtr removeAliasColumnsFromCreateQuery(const ASTPtr & query_ast)
     {
-        const ASTs & column_asts = typeid_cast<ASTCreateQuery &>(*query_ast).columns->children;
+        const ASTs & column_asts = typeid_cast<ASTCreateQuery &>(*query_ast).columns_list->columns->children;
         auto new_columns = std::make_shared<ASTExpressionList>();
 
         for (const ASTPtr & column_ast : column_asts)
@@ -1198,8 +1198,13 @@ protected:
 
         ASTPtr new_query_ast = query_ast->clone();
         ASTCreateQuery & new_query = typeid_cast<ASTCreateQuery &>(*new_query_ast);
-        new_query.columns = new_columns.get();
-        new_query.children.at(0) = std::move(new_columns);
+
+        auto new_columns_list = std::make_shared<ASTColumns>();
+        new_columns_list->set(new_columns_list->columns, new_columns);
+        new_columns_list->set(
+                new_columns_list->indices, typeid_cast<ASTCreateQuery &>(*query_ast).columns_list->indices->clone());
+
+        new_query.replace(new_query.columns_list, new_columns_list);
 
         return new_query_ast;
     }
@@ -1217,7 +1222,7 @@ protected:
         res->table = new_table.second;
 
         res->children.clear();
-        res->set(res->columns, create.columns->clone());
+        res->set(res->columns_list, create.columns_list->clone());
         res->set(res->storage, new_storage_ast->clone());
 
         return res;

dbms/programs/local/LocalServer.cpp

@@ -297,7 +297,7 @@ void LocalServer::processQueries()
 
         try
         {
-            executeQuery(read_buf, write_buf, /* allow_into_outfile = */ true, *context, {});
+            executeQuery(read_buf, write_buf, /* allow_into_outfile = */ true, *context, {}, {});
         }
         catch (...)
         {

dbms/programs/main.cpp

@@ -56,9 +56,6 @@ int mainEntryClickHouseClusterCopier(int argc, char ** argv);
 #if ENABLE_CLICKHOUSE_OBFUSCATOR || !defined(ENABLE_CLICKHOUSE_OBFUSCATOR)
 int mainEntryClickHouseObfuscator(int argc, char ** argv);
 #endif
-#if ENABLE_CLICKHOUSE_ODBC_BRIDGE || !defined(ENABLE_CLICKHOUSE_ODBC_BRIDGE)
-int mainEntryClickHouseODBCBridge(int argc, char ** argv);
-#endif
 
 
 #if USE_EMBEDDED_COMPILER
@@ -105,9 +102,6 @@ std::pair<const char *, MainFunc> clickhouse_applications[] =
 #if ENABLE_CLICKHOUSE_OBFUSCATOR || !defined(ENABLE_CLICKHOUSE_OBFUSCATOR)
     {"obfuscator", mainEntryClickHouseObfuscator},
 #endif
-#if ENABLE_CLICKHOUSE_ODBC_BRIDGE || !defined(ENABLE_CLICKHOUSE_ODBC_BRIDGE)
-    {"odbc-bridge", mainEntryClickHouseODBCBridge},
-#endif
 
 #if USE_EMBEDDED_COMPILER
     {"clang", mainEntryClickHouseClang},

dbms/programs/odbc-bridge/CMakeLists.txt

@@ -9,7 +9,7 @@ add_library (clickhouse-odbc-bridge-lib ${LINK_MODE}
     validateODBCConnectionString.cpp
 )
 
-target_link_libraries (clickhouse-odbc-bridge-lib PRIVATE clickhouse_dictionaries daemon dbms clickhouse_common_io)
+target_link_libraries (clickhouse-odbc-bridge-lib PRIVATE daemon dbms clickhouse_common_io)
 target_include_directories (clickhouse-odbc-bridge-lib PUBLIC ${ClickHouse_SOURCE_DIR}/libs/libdaemon/include)
 
 if (USE_POCO_SQLODBC)
@@ -33,8 +33,11 @@ if (ENABLE_TESTS)
     add_subdirectory (tests)
 endif ()
 
-if (CLICKHOUSE_SPLIT_BINARY)
-    add_executable (clickhouse-odbc-bridge odbc-bridge.cpp)
-    target_link_libraries (clickhouse-odbc-bridge PRIVATE clickhouse-odbc-bridge-lib)
-    install (TARGETS clickhouse-odbc-bridge ${CLICKHOUSE_ALL_TARGETS} RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
-endif ()
+# clickhouse-odbc-bridge is always a separate binary.
+# Reason: it must not export symbols from SSL, mariadb-client, etc. to not break ABI compatibility with ODBC drivers.
+# For this reason, we disabling -rdynamic linker flag. But we do it in strange way:
+SET(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "")
+
+add_executable (clickhouse-odbc-bridge odbc-bridge.cpp)
+target_link_libraries (clickhouse-odbc-bridge PRIVATE clickhouse-odbc-bridge-lib)
+install (TARGETS clickhouse-odbc-bridge RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)

dbms/programs/performance-test/ConfigPreprocessor.cpp

@@ -1,5 +1,6 @@
 #include "ConfigPreprocessor.h"
 #include <Core/Types.h>
+#include <Poco/Path.h>
 #include <regex>
 namespace DB
 {
@@ -14,7 +15,11 @@ std::vector<XMLConfigurationPtr> ConfigPreprocessor::processConfig(
 
     std::vector<XMLConfigurationPtr> result;
     for (const auto & path : paths)
+    {
         result.emplace_back(new XMLConfiguration(path));
+        result.back()->setString("path", Poco::Path(path).absolute().toString());
+    }
+
     /// Leave tests:
     removeConfigurationsIf(result, FilterType::Tag, tests_tags, true);
     removeConfigurationsIf(result, FilterType::Name, tests_names, true);

dbms/programs/performance-test/PerformanceTest.cpp

@@ -25,12 +25,14 @@ PerformanceTest::PerformanceTest(
     Connection & connection_,
     InterruptListener & interrupt_listener_,
     const PerformanceTestInfo & test_info_,
-    Context & context_)
+    Context & context_,
+    const std::vector<size_t> & queries_to_run_)
     : config(config_)
     , connection(connection_)
     , interrupt_listener(interrupt_listener_)
     , test_info(test_info_)
     , context(context_)
+    , queries_to_run(queries_to_run_)
     , log(&Poco::Logger::get("PerformanceTest"))
 {
 }
@@ -128,12 +130,43 @@ UInt64 PerformanceTest::calculateMaxExecTime() const
     return result;
 }
 
+
+void PerformanceTest::prepare() const
+{
+    for (const auto & query : test_info.create_queries)
+    {
+        LOG_INFO(log, "Executing create query '" << query << "'");
+        connection.sendQuery(query);
+    }
+
+    for (const auto & query : test_info.fill_queries)
+    {
+        LOG_INFO(log, "Executing fill query '" << query << "'");
+        connection.sendQuery(query);
+    }
+
+}
+
+void PerformanceTest::finish() const
+{
+    for (const auto & query : test_info.drop_queries)
+    {
+        LOG_INFO(log, "Executing drop query '" << query << "'");
+        connection.sendQuery(query);
+    }
+}
+
 std::vector<TestStats> PerformanceTest::execute()
 {
     std::vector<TestStats> statistics_by_run;
+    size_t query_count;
+    if (queries_to_run.empty())
+        query_count = test_info.queries.size();
+    else
+        query_count = queries_to_run.size();
     size_t total_runs = test_info.times_to_run * test_info.queries.size();
     statistics_by_run.resize(total_runs);
-    LOG_INFO(log, "Totally will run cases " << total_runs << " times");
+    LOG_INFO(log, "Totally will run cases " << test_info.times_to_run * query_count << " times");
     UInt64 max_exec_time = calculateMaxExecTime();
     if (max_exec_time != 0)
         LOG_INFO(log, "Test will be executed for a maximum of " << max_exec_time / 1000. << " seconds");
@@ -146,9 +179,13 @@ std::vector<TestStats> PerformanceTest::execute()
 
         for (size_t query_index = 0; query_index < test_info.queries.size(); ++query_index)
         {
-            size_t statistic_index = number_of_launch * test_info.queries.size() + query_index;
-
-            queries_with_indexes.push_back({test_info.queries[query_index], statistic_index});
+            if (queries_to_run.empty() || std::find(queries_to_run.begin(), queries_to_run.end(), query_index) != queries_to_run.end())
+            {
+                size_t statistic_index = number_of_launch * test_info.queries.size() + query_index;
+                queries_with_indexes.push_back({test_info.queries[query_index], statistic_index});
+            }
+            else
+                LOG_INFO(log, "Will skip query " << test_info.queries[query_index] << " by index");
         }
 
         if (got_SIGINT)

dbms/programs/performance-test/PerformanceTest.h

@@ -22,15 +22,19 @@ public:
         Connection & connection_,
         InterruptListener & interrupt_listener_,
         const PerformanceTestInfo & test_info_,
-        Context & context_);
+        Context & context_,
+        const std::vector<size_t> & queries_to_run_);
 
     bool checkPreconditions() const;
+    void prepare() const;
     std::vector<TestStats> execute();
+    void finish() const;
 
     const PerformanceTestInfo & getTestInfo() const
     {
         return test_info;
     }
+
     bool checkSIGINT() const
     {
         return got_SIGINT;
@@ -51,6 +55,7 @@ private:
     PerformanceTestInfo test_info;
     Context & context;
 
+    std::vector<size_t> queries_to_run;
     Poco::Logger * log;
 
     bool got_SIGINT = false;

dbms/programs/performance-test/PerformanceTestInfo.cpp

@@ -83,12 +83,14 @@ PerformanceTestInfo::PerformanceTestInfo(
     : profiles_file(profiles_file_)
 {
     test_name = config->getString("name");
+    path = config->getString("path");
     applySettings(config);
     extractQueries(config);
     processSubstitutions(config);
     getExecutionType(config);
     getStopConditions(config);
     getMetrics(config);
+    extractAuxiliaryQueries(config);
 }
 
 void PerformanceTestInfo::applySettings(XMLConfigurationPtr config)
@@ -268,4 +270,16 @@ void PerformanceTestInfo::getMetrics(XMLConfigurationPtr config)
         checkMetricsInput(metrics, exec_type);
 }
 
+void PerformanceTestInfo::extractAuxiliaryQueries(XMLConfigurationPtr config)
+{
+    if (config->has("create_query"))
+        create_queries = getMultipleValuesFromConfig(*config, "", "create_query");
+
+    if (config->has("fill_query"))
+        fill_queries = getMultipleValuesFromConfig(*config, "", "fill_query");
+
+    if (config->has("drop_query"))
+        drop_queries = getMultipleValuesFromConfig(*config, "", "drop_query");
+}
+
 }

dbms/programs/performance-test/PerformanceTestInfo.h

@@ -29,6 +29,7 @@ public:
     PerformanceTestInfo(XMLConfigurationPtr config, const std::string & profiles_file_);
 
     std::string test_name;
+    std::string path;
     std::string main_metric;
 
     Strings queries;
@@ -42,6 +43,10 @@ public:
     std::string profiles_file;
     std::vector<TestStopConditions> stop_conditions_by_run;
 
+    Strings create_queries;
+    Strings fill_queries;
+    Strings drop_queries;
+
 private:
     void applySettings(XMLConfigurationPtr config);
     void extractQueries(XMLConfigurationPtr config);
@@ -49,6 +54,7 @@ private:
     void getExecutionType(XMLConfigurationPtr config);
     void getStopConditions(XMLConfigurationPtr config);
     void getMetrics(XMLConfigurationPtr config);
+    void extractAuxiliaryQueries(XMLConfigurationPtr config);
 };
 
 }

dbms/programs/performance-test/PerformanceTestSuite.cpp

@@ -11,12 +11,13 @@
 #include <boost/filesystem.hpp>
 #include <boost/program_options.hpp>
 
-#include <Poco/Util/XMLConfiguration.h>
-#include <Poco/Logger.h>
+#include <Poco/AutoPtr.h>
 #include <Poco/ConsoleChannel.h>
 #include <Poco/FormattingChannel.h>
+#include <Poco/Logger.h>
+#include <Poco/Path.h>
 #include <Poco/PatternFormatter.h>
-
+#include <Poco/Util/XMLConfiguration.h>
 
 #include <common/logger_useful.h>
 #include <Client/Connection.h>
@@ -25,7 +26,6 @@
 #include <IO/ConnectionTimeouts.h>
 #include <IO/UseSSL.h>
 #include <Interpreters/Settings.h>
-#include <Poco/AutoPtr.h>
 #include <Common/Exception.h>
 #include <Common/InterruptListener.h>
 
@@ -70,6 +70,7 @@ public:
         Strings && skip_names_,
         Strings && tests_names_regexp_,
         Strings && skip_names_regexp_,
+        const std::unordered_map<std::string, std::vector<size_t>> query_indexes_,
         const ConnectionTimeouts & timeouts)
         : connection(host_, port_, default_database_, user_,
             password_, timeouts, "performance-test", Protocol::Compression::Enable,
@@ -80,6 +81,7 @@ public:
         , skip_tags(std::move(skip_tags_))
         , skip_names(std::move(skip_names_))
         , skip_names_regexp(std::move(skip_names_regexp_))
+        , query_indexes(query_indexes_)
         , lite_output(lite_output_)
         , profiles_file(profiles_file_)
         , input_files(input_files_)
@@ -128,6 +130,7 @@ private:
     const Strings & skip_tags;
     const Strings & skip_names;
     const Strings & skip_names_regexp;
+    std::unordered_map<std::string, std::vector<size_t>> query_indexes;
 
     Context global_context = Context::createGlobal();
     std::shared_ptr<ReportBuilder> report_builder;
@@ -198,19 +201,26 @@ private:
     {
         PerformanceTestInfo info(test_config, profiles_file);
         LOG_INFO(log, "Config for test '" << info.test_name << "' parsed");
-        PerformanceTest current(test_config, connection, interrupt_listener, info, global_context);
+        PerformanceTest current(test_config, connection, interrupt_listener, info, global_context, query_indexes[info.path]);
 
         current.checkPreconditions();
         LOG_INFO(log, "Preconditions for test '" << info.test_name << "' are fullfilled");
-
+        LOG_INFO(log, "Preparing for run, have " << info.create_queries.size()
+            << " create queries and " << info.fill_queries.size() << " fill queries");
+        current.prepare();
+        LOG_INFO(log, "Prepared");
         LOG_INFO(log, "Running test '" << info.test_name << "'");
         auto result = current.execute();
         LOG_INFO(log, "Test '" << info.test_name << "' finished");
 
+        LOG_INFO(log, "Running post run queries");
+        current.finish();
+        LOG_INFO(log, "Postqueries finished");
+
         if (lite_output)
-            return {report_builder->buildCompactReport(info, result), current.checkSIGINT()};
+            return {report_builder->buildCompactReport(info, result, query_indexes[info.path]), current.checkSIGINT()};
         else
-            return {report_builder->buildFullReport(info, result), current.checkSIGINT()};
+            return {report_builder->buildFullReport(info, result, query_indexes[info.path]), current.checkSIGINT()};
     }
 
 };
@@ -282,6 +292,29 @@ static std::vector<std::string> getInputFiles(const po::variables_map & options,
     return input_files;
 }
 
+std::unordered_map<std::string, std::vector<std::size_t>> getTestQueryIndexes(const po::basic_parsed_options<char> & parsed_opts)
+{
+    std::unordered_map<std::string, std::vector<std::size_t>> result;
+    const auto & options = parsed_opts.options;
+    for (size_t i = 0; i < options.size() - 1; ++i)
+    {
+        const auto & opt = options[i];
+        if (opt.string_key == "input-files")
+        {
+            if (options[i + 1].string_key == "query-indexes")
+            {
+                const std::string & test_path = Poco::Path(opt.value[0]).absolute().toString();
+                for (const auto & query_num_str : options[i + 1].value)
+                {
+                    size_t query_num = std::stoul(query_num_str);
+                    result[test_path].push_back(query_num);
+                }
+            }
+        }
+    }
+    return result;
+}
+
 int mainEntryClickHousePerformanceTest(int argc, char ** argv)
 try
 {
@@ -307,24 +340,18 @@ try
         ("skip-names", value<Strings>()->multitoken(), "Do not run tests with name")
         ("names-regexp", value<Strings>()->multitoken(), "Run tests with names matching regexp")
         ("skip-names-regexp", value<Strings>()->multitoken(), "Do not run tests with names matching regexp")
+        ("input-files", value<Strings>()->multitoken(), "Input .xml files")
+        ("query-indexes", value<std::vector<size_t>>()->multitoken(), "Input query indexes")
         ("recursive,r", "Recurse in directories to find all xml's");
 
-    /// These options will not be displayed in --help
-    po::options_description hidden("Hidden options");
-    hidden.add_options()
-        ("input-files", value<std::vector<std::string>>(), "");
-
-    /// But they will be legit, though. And they must be given without name
-    po::positional_options_description positional;
-    positional.add("input-files", -1);
-
     po::options_description cmdline_options;
-    cmdline_options.add(desc).add(hidden);
+    cmdline_options.add(desc);
 
     po::variables_map options;
-    po::store(
-        po::command_line_parser(argc, argv).
-        options(cmdline_options).positional(positional).run(), options);
+    po::basic_parsed_options<char> parsed = po::command_line_parser(argc, argv).options(cmdline_options).run();
+    auto queries_with_indexes = getTestQueryIndexes(parsed);
+    po::store(parsed, options);
+
     po::notify(options);
 
     Poco::AutoPtr<Poco::PatternFormatter> formatter(new Poco::PatternFormatter("%Y.%m.%d %H:%M:%S.%F <%p> %s: %t"));
@@ -371,6 +398,7 @@ try
         std::move(skip_names),
         std::move(tests_names_regexp),
         std::move(skip_names_regexp),
+        queries_with_indexes,
         timeouts);
     return performance_test_suite.run();
 }

dbms/programs/performance-test/ReportBuilder.cpp

@@ -35,7 +35,8 @@ std::string ReportBuilder::getCurrentTime() const
 
 std::string ReportBuilder::buildFullReport(
     const PerformanceTestInfo & test_info,
-    std::vector<TestStats> & stats) const
+    std::vector<TestStats> & stats,
+    const std::vector<std::size_t> & queries_to_run) const
 {
     JSONString json_output;
 
@@ -46,6 +47,7 @@ std::string ReportBuilder::buildFullReport(
     json_output.set("server_version", server_version);
     json_output.set("time", getCurrentTime());
     json_output.set("test_name", test_info.test_name);
+    json_output.set("path", test_info.path);
     json_output.set("main_metric", test_info.main_metric);
 
     auto has_metric = [&test_info] (const std::string & metric_name)
@@ -84,6 +86,9 @@ std::string ReportBuilder::buildFullReport(
     std::vector<JSONString> run_infos;
     for (size_t query_index = 0; query_index < test_info.queries.size(); ++query_index)
     {
+        if (!queries_to_run.empty() && std::find(queries_to_run.begin(), queries_to_run.end(), query_index) == queries_to_run.end())
+            continue;
+
         for (size_t number_of_launch = 0; number_of_launch < test_info.times_to_run; ++number_of_launch)
         {
             size_t stat_index = number_of_launch * test_info.queries.size() + query_index;
@@ -96,6 +101,7 @@ std::string ReportBuilder::buildFullReport(
 
             auto query = std::regex_replace(test_info.queries[query_index], QUOTE_REGEX, "\\\"");
             runJSON.set("query", query);
+            runJSON.set("query_index", query_index);
             if (!statistics.exception.empty())
                 runJSON.set("exception", statistics.exception);
 
@@ -170,13 +176,17 @@ std::string ReportBuilder::buildFullReport(
 
 std::string ReportBuilder::buildCompactReport(
     const PerformanceTestInfo & test_info,
-    std::vector<TestStats> & stats) const
+    std::vector<TestStats> & stats,
+    const std::vector<std::size_t> & queries_to_run) const
 {
 
     std::ostringstream output;
 
     for (size_t query_index = 0; query_index < test_info.queries.size(); ++query_index)
     {
+        if (!queries_to_run.empty() && std::find(queries_to_run.begin(), queries_to_run.end(), query_index) == queries_to_run.end())
+            continue;
+
         for (size_t number_of_launch = 0; number_of_launch < test_info.times_to_run; ++number_of_launch)
         {
             if (test_info.queries.size() > 1)
@@ -191,5 +201,4 @@ std::string ReportBuilder::buildCompactReport(
     }
     return output.str();
 }
-
 }

dbms/programs/performance-test/ReportBuilder.h

@@ -9,14 +9,18 @@ namespace DB
 class ReportBuilder
 {
 public:
-    explicit ReportBuilder(const std::string & server_version_);
+    ReportBuilder(const std::string & server_version_);
     std::string buildFullReport(
         const PerformanceTestInfo & test_info,
-        std::vector<TestStats> & stats) const;
+        std::vector<TestStats> & stats,
+        const std::vector<std::size_t> & queries_to_run) const;
+
 
     std::string buildCompactReport(
         const PerformanceTestInfo & test_info,
-        std::vector<TestStats> & stats) const;
+        std::vector<TestStats> & stats,
+        const std::vector<std::size_t> & queries_to_run) const;
+
 private:
     std::string server_version;
     std::string hostname;

dbms/programs/server/HTTPHandler.cpp

@@ -4,6 +4,7 @@
 #include <Poco/File.h>
 #include <Poco/Net/HTTPBasicCredentials.h>
 #include <Poco/Net/HTTPServerRequest.h>
+#include <Poco/Net/HTTPServerRequestImpl.h>
 #include <Poco/Net/HTTPServerResponse.h>
 #include <Poco/Net/NetException.h>
 
@@ -558,12 +559,51 @@ void HTTPHandler::processQuery(
     client_info.http_method = http_method;
     client_info.http_user_agent = request.get("User-Agent", "");
 
+    auto appendCallback = [&context] (ProgressCallback callback)
+    {
+        auto prev = context.getProgressCallback();
+
+        context.setProgressCallback([prev, callback] (const Progress & progress)
+        {
+            if (prev)
+                prev(progress);
+
+            callback(progress);
+        });
+    };
+
     /// While still no data has been sent, we will report about query execution progress by sending HTTP headers.
     if (settings.send_progress_in_http_headers)
-        context.setProgressCallback([&used_output] (const Progress & progress) { used_output.out->onProgress(progress); });
+        appendCallback([&used_output] (const Progress & progress) { used_output.out->onProgress(progress); });
+
+    if (settings.readonly > 0 && settings.cancel_http_readonly_queries_on_client_close)
+    {
+        Poco::Net::StreamSocket & socket = dynamic_cast<Poco::Net::HTTPServerRequestImpl &>(request).socket();
+
+        appendCallback([&context, &socket](const Progress &)
+        {
+            /// Assume that at the point this method is called no one is reading data from the socket any more.
+            /// True for read-only queries.
+            try
+            {
+                char b;
+                int status = socket.receiveBytes(&b, 1, MSG_DONTWAIT | MSG_PEEK);
+                if (status == 0)
+                    context.killCurrentQuery();
+            }
+            catch (Poco::TimeoutException &)
+            {
+            }
+            catch (...)
+            {
+                context.killCurrentQuery();
+            }
+        });
+    }
 
     executeQuery(*in, *used_output.out_maybe_delayed_and_compressed, /* allow_into_outfile = */ false, context,
-        [&response] (const String & content_type) { response.setContentType(content_type); });
+        [&response] (const String & content_type) { response.setContentType(content_type); },
+        [&response] (const String & current_query_id) { response.add("Query-Id", current_query_id); });
 
     if (used_output.hasDelayed())
     {

dbms/programs/server/Server.cpp

@@ -11,6 +11,7 @@
 #include <Poco/DirectoryIterator.h>
 #include <Poco/Net/HTTPServer.h>
 #include <Poco/Net/NetException.h>
+#include <Poco/Util/HelpFormatter.h>
 #include <ext/scope_guard.h>
 #include <common/logger_useful.h>
 #include <common/ErrorHandlers.h>
@@ -47,6 +48,7 @@
 #include "MetricsTransmitter.h"
 #include <Common/StatusFile.h>
 #include "TCPHandlerFactory.h"
+#include "Common/config_version.h"
 
 #if defined(__linux__)
 #include <Common/hasLinuxCapability.h>
@@ -116,6 +118,26 @@ void Server::uninitialize()
     BaseDaemon::uninitialize();
 }
 
+int Server::run()
+{
+    if (config().hasOption("help"))
+    {
+        Poco::Util::HelpFormatter helpFormatter(Server::options());
+        std::stringstream header;
+        header << commandName() << " [OPTION] [-- [ARG]...]\n";
+        header << "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010";
+        helpFormatter.setHeader(header.str());
+        helpFormatter.format(std::cout);
+        return 0;
+    }
+    if (config().hasOption("version"))
+    {
+        std::cout << DBMS_NAME << " server version " << VERSION_STRING << "." << std::endl;
+        return 0;
+    }
+    return Application::run();
+}
+
 void Server::initialize(Poco::Util::Application & self)
 {
     BaseDaemon::initialize(self);
@@ -127,6 +149,21 @@ std::string Server::getDefaultCorePath() const
     return getCanonicalPath(config().getString("path", DBMS_DEFAULT_PATH)) + "cores";
 }
 
+void Server::defineOptions(Poco::Util::OptionSet & _options)
+{
+    _options.addOption(
+        Poco::Util::Option("help", "h", "show help and exit")
+            .required(false)
+            .repeatable(false)
+            .binding("help"));
+    _options.addOption(
+        Poco::Util::Option("version", "V", "show version and exit")
+            .required(false)
+            .repeatable(false)
+            .binding("version"));
+    BaseDaemon::defineOptions(_options);
+}
+
 int Server::main(const std::vector<std::string> & /*args*/)
 {
     Logger * log = &logger();

dbms/programs/server/Server.h

@@ -21,6 +21,8 @@ namespace DB
 class Server : public BaseDaemon, public IServer
 {
 public:
+    using ServerApplication::run;
+
     Poco::Util::LayeredConfiguration & config() const override
     {
         return BaseDaemon::config();
@@ -41,7 +43,10 @@ public:
         return BaseDaemon::isCancelled();
     }
 
+    void defineOptions(Poco::Util::OptionSet & _options) override;
 protected:
+    int run() override;
+
     void initialize(Application & self) override;
 
     void uninitialize() override;

dbms/programs/server/config.d/listen.xml

@@ -1 +0,0 @@
-<yandex><listen_host>0.0.0.0</listen_host></yandex>

dbms/programs/server/config.d/zookeeper.xml

@@ -1,16 +1,8 @@
 <yandex>
-    <zookeeper>
+<!--    <zookeeper>
         <node>
             <host>localhost</host>
             <port>2181</port>
         </node>
-        <node>
-            <host>yandex.ru</host>
-            <port>2181</port>
-        </node>
-        <node>
-            <host>111.0.1.2</host>
-            <port>2181</port>
-        </node>
-    </zookeeper>
+    </zookeeper>-->
 </yandex>

dbms/src/AggregateFunctions/AggregateFunctionEntropy.cpp

@@ -0,0 +1,58 @@
+#include <AggregateFunctions/AggregateFunctionFactory.h>
+#include <AggregateFunctions/AggregateFunctionEntropy.h>
+#include <AggregateFunctions/FactoryHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+}
+
+namespace
+{
+
+AggregateFunctionPtr createAggregateFunctionEntropy(const std::string & name, const DataTypes & argument_types, const Array & parameters)
+{
+    assertNoParameters(name, parameters);
+    if (argument_types.empty())
+        throw Exception("Incorrect number of arguments for aggregate function " + name,
+                        ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+
+    WhichDataType which(argument_types[0]);
+    if (isNumber(argument_types[0]))
+    {
+        if (which.isUInt64())
+        {
+            return std::make_shared<AggregateFunctionEntropy<UInt64>>();
+        }
+        else if (which.isInt64())
+        {
+            return std::make_shared<AggregateFunctionEntropy<Int64>>();
+        }
+        else if (which.isInt32())
+        {
+            return std::make_shared<AggregateFunctionEntropy<Int32>>();
+        }
+        else if (which.isUInt32())
+        {
+            return std::make_shared<AggregateFunctionEntropy<UInt32>>();
+        }
+        else if (which.isUInt128())
+        {
+            return std::make_shared<AggregateFunctionEntropy<UInt128, true>>();
+        }
+    }
+
+    return std::make_shared<AggregateFunctionEntropy<UInt128>>();
+}
+
+}
+
+void registerAggregateFunctionEntropy(AggregateFunctionFactory & factory)
+{
+    factory.registerFunction("entropy", createAggregateFunctionEntropy);
+}
+
+}

dbms/src/AggregateFunctions/AggregateFunctionEntropy.h

@@ -0,0 +1,152 @@
+#pragma once
+
+#include <AggregateFunctions/FactoryHelpers.h>
+#include <Common/HashTable/HashMap.h>
+#include <Common/NaNUtils.h>
+
+#include <AggregateFunctions/IAggregateFunction.h>
+#include <AggregateFunctions/UniqVariadicHash.h>
+#include <Columns/ColumnArray.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <IO/ReadHelpers.h>
+#include <IO/WriteHelpers.h>
+
+#include <cmath>
+
+namespace DB
+{
+
+/** Calculates Shannon Entropy, using HashMap and computing empirical distribution function
+  */
+template <typename Value, bool is_hashed>
+struct EntropyData
+{
+    using Weight = UInt64;
+    using HashingMap = HashMap <
+    Value, Weight,
+    HashCRC32<Value>,
+    HashTableGrower<4>,
+    HashTableAllocatorWithStackMemory<sizeof(std::pair<Value, Weight>) * (1 << 3)>
+    >;
+
+    using TrivialMap = HashMap <
+    Value, Weight,
+    UInt128TrivialHash,
+    HashTableGrower<4>,
+    HashTableAllocatorWithStackMemory<sizeof(std::pair<Value, Weight>) * (1 << 3)>
+    >;
+
+    /// If column value is UInt128 then there is no need to hash values
+    using Map = std::conditional_t<is_hashed, TrivialMap, HashingMap>;
+
+    Map map;
+
+    void add(const Value & x)
+    {
+        if (!isNaN(x))
+            ++map[x];
+    }
+
+    void add(const Value & x, const Weight & weight)
+    {
+        if (!isNaN(x))
+            map[x] += weight;
+    }
+
+    void merge(const EntropyData & rhs)
+    {
+        for (const auto & pair : rhs.map)
+            map[pair.first] += pair.second;
+    }
+
+    void serialize(WriteBuffer & buf) const
+    {
+        map.write(buf);
+    }
+
+    void deserialize(ReadBuffer & buf)
+    {
+        typename Map::Reader reader(buf);
+        while (reader.next())
+        {
+            const auto &pair = reader.get();
+            map[pair.first] = pair.second;
+        }
+    }
+
+    Float64 get() const
+    {
+        Float64 shannon_entropy = 0;
+        UInt64 total_value = 0;
+        for (const auto & pair : map)
+        {
+            total_value += pair.second;
+        }
+        Float64 cur_proba;
+        Float64 log2e = 1 / std::log(2);
+        for (const auto & pair : map)
+        {
+            cur_proba = Float64(pair.second) / total_value;
+            shannon_entropy -= cur_proba * std::log(cur_proba) * log2e;
+        }
+
+        return shannon_entropy;
+    }
+};
+
+template <typename Value, bool is_hashed = false>
+class AggregateFunctionEntropy final : public IAggregateFunctionDataHelper<EntropyData<Value, is_hashed>,
+        AggregateFunctionEntropy<Value>>
+{
+public:
+    AggregateFunctionEntropy()
+    {}
+
+    String getName() const override { return "entropy"; }
+
+    DataTypePtr getReturnType() const override
+    {
+        return std::make_shared<DataTypeNumber<Float64>>();
+    }
+
+    void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
+    {
+        if constexpr (!std::is_same_v<UInt128, Value>)
+        {
+            /// Here we manage only with numerical types
+            const auto &column = static_cast<const ColumnVector <Value> &>(*columns[0]);
+            this->data(place).add(column.getData()[row_num]);
+        }
+        else
+        {
+            this->data(place).add(UniqVariadicHash<true, false>::apply(1, columns, row_num));
+
+        }
+    }
+
+    void merge(AggregateDataPtr place, ConstAggregateDataPtr rhs, Arena *) const override
+    {
+        this->data(place).merge(this->data(rhs));
+    }
+
+    void serialize(ConstAggregateDataPtr place, WriteBuffer & buf) const override
+    {
+        this->data(const_cast<AggregateDataPtr>(place)).serialize(buf);
+    }
+
+    void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override
+    {
+        this->data(place).deserialize(buf);
+    }
+
+    void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
+    {
+        auto &column = dynamic_cast<ColumnVector<Float64> &>(to);
+        column.getData().push_back(this->data(place).get());
+    }
+
+    const char * getHeaderFilePath() const override { return __FILE__; }
+
+};
+
+}

dbms/src/AggregateFunctions/AggregateFunctionFactory.cpp

@@ -128,7 +128,11 @@ AggregateFunctionPtr AggregateFunctionFactory::getImpl(
         return combinator->transformAggregateFunction(nested_function, argument_types, parameters);
     }
 
-    throw Exception("Unknown aggregate function " + name, ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION);
+    auto hints = this->getHints(name);
+    if (!hints.empty())
+        throw Exception("Unknown aggregate function " + name + ". Maybe you meant: " + toString(hints), ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION);
+    else
+        throw Exception("Unknown aggregate function " + name, ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION);
 }
 
 

dbms/src/AggregateFunctions/QuantileExact.h

@@ -19,7 +19,7 @@ namespace ErrorCodes
 /** Calculates quantile by collecting all values into array
   *  and applying n-th element (introselect) algorithm for the resulting array.
   *
-  * It use O(N) memory and it is very inefficient in case of high amount of identical values.
+  * It uses O(N) memory and it is very inefficient in case of high amount of identical values.
   * But it is very CPU efficient for not large datasets.
   */
 template <typename Value>

dbms/src/AggregateFunctions/QuantileExactWeighted.h

@@ -14,7 +14,7 @@ namespace ErrorCodes
 
 /** Calculates quantile by counting number of occurrences for each value in a hash map.
   *
-  * It use O(distinct(N)) memory. Can be naturally applied for values with weight.
+  * It uses O(distinct(N)) memory. Can be naturally applied for values with weight.
   * In case of many identical values, it can be more efficient than QuantileExact even when weight is not used.
   */
 template <typename Value>

dbms/src/AggregateFunctions/registerAggregateFunctions.cpp

@@ -27,6 +27,7 @@ void registerAggregateFunctionUniqUpTo(AggregateFunctionFactory &);
 void registerAggregateFunctionTopK(AggregateFunctionFactory &);
 void registerAggregateFunctionsBitwise(AggregateFunctionFactory &);
 void registerAggregateFunctionsMaxIntersections(AggregateFunctionFactory &);
+void registerAggregateFunctionEntropy(AggregateFunctionFactory &);
 
 void registerAggregateFunctionCombinatorIf(AggregateFunctionCombinatorFactory &);
 void registerAggregateFunctionCombinatorArray(AggregateFunctionCombinatorFactory &);
@@ -65,6 +66,7 @@ void registerAggregateFunctions()
         registerAggregateFunctionsMaxIntersections(factory);
         registerAggregateFunctionHistogram(factory);
         registerAggregateFunctionRetention(factory);
+        registerAggregateFunctionEntropy(factory);
     }
 
     {

dbms/src/Columns/ColumnVector.cpp

@@ -12,6 +12,7 @@
 #include <Columns/ColumnsCommon.h>
 #include <DataStreams/ColumnGathererStream.h>
 #include <ext/bit_cast.h>
+#include <pdqsort.h>
 
 #ifdef __SSE2__
     #include <emmintrin.h>
@@ -90,9 +91,9 @@ void ColumnVector<T>::getPermutation(bool reverse, size_t limit, int nan_directi
     else
     {
         if (reverse)
-            std::sort(res.begin(), res.end(), greater(*this, nan_direction_hint));
+            pdqsort(res.begin(), res.end(), greater(*this, nan_direction_hint));
         else
-            std::sort(res.begin(), res.end(), less(*this, nan_direction_hint));
+            pdqsort(res.begin(), res.end(), less(*this, nan_direction_hint));
     }
 }
 

dbms/src/Common/CurrentThread.h

@@ -69,7 +69,7 @@ public:
     static void finalizePerformanceCounters();
 
     /// Returns a non-empty string if the thread is attached to a query
-    static std::string getCurrentQueryID();
+    static const std::string & getQueryId();
 
     /// Non-master threads call this method in destructor automatically
     static void detachQuery();

dbms/src/Common/IFactoryWithAliases.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include <Common/Exception.h>
+#include <Common/NamePrompter.h>
 #include <Core/Types.h>
 #include <Poco/String.h>
 
@@ -105,6 +106,12 @@ public:
         return aliases.count(name) || case_insensitive_aliases.count(name);
     }
 
+    std::vector<String> getHints(const String & name) const
+    {
+        static const auto registered_names = getAllRegisteredNames();
+        return prompter.getHints(name, registered_names);
+    }
+
     virtual ~IFactoryWithAliases() {}
 
 private:
@@ -120,6 +127,12 @@ private:
 
     /// Case insensitive aliases
     AliasMap case_insensitive_aliases;
+
+    /**
+      * prompter for names, if a person makes a typo for some function or type, it
+      * helps to find best possible match (in particular, edit distance is one or two symbols)
+      */
+    NamePrompter</*MistakeFactor=*/2, /*MaxNumHints=*/2> prompter;
 };
 
 }

dbms/src/Common/NamePrompter.h

@@ -0,0 +1,83 @@
+#pragma once
+
+#include <Core/Types.h>
+
+#include <algorithm>
+#include <cctype>
+#include <queue>
+#include <utility>
+
+namespace DB
+{
+template <size_t MistakeFactor, size_t MaxNumHints>
+class NamePrompter
+{
+public:
+    using DistanceIndex = std::pair<size_t, size_t>;
+    using DistanceIndexQueue = std::priority_queue<DistanceIndex>;
+
+    static std::vector<String> getHints(const String & name, const std::vector<String> & prompting_strings)
+    {
+        DistanceIndexQueue queue;
+        for (size_t i = 0; i < prompting_strings.size(); ++i)
+            appendToQueue(i, name, queue, prompting_strings);
+        return release(queue, prompting_strings);
+    }
+
+private:
+    static size_t levenshteinDistance(const String & lhs, const String & rhs)
+    {
+        size_t n = lhs.size();
+        size_t m = rhs.size();
+        std::vector<std::vector<size_t>> dp(n + 1, std::vector<size_t>(m + 1));
+
+        for (size_t i = 1; i <= n; ++i)
+            dp[i][0] = i;
+
+        for (size_t i = 1; i <= m; ++i)
+            dp[0][i] = i;
+
+        for (size_t j = 1; j <= m; ++j)
+        {
+            for (size_t i = 1; i <= n; ++i)
+            {
+                if (std::tolower(lhs[i - 1]) == std::tolower(rhs[j - 1]))
+                    dp[i][j] = dp[i - 1][j - 1];
+                else
+                    dp[i][j] = std::min(dp[i - 1][j] + 1, std::min(dp[i][j - 1] + 1, dp[i - 1][j - 1] + 1));
+            }
+        }
+
+        return dp[n][m];
+    }
+
+    static void appendToQueue(size_t ind, const String & name, DistanceIndexQueue & queue, const std::vector<String> & prompting_strings)
+    {
+        if (prompting_strings[ind].size() <= name.size() + MistakeFactor && prompting_strings[ind].size() + MistakeFactor >= name.size())
+        {
+            size_t distance = levenshteinDistance(prompting_strings[ind], name);
+            if (distance <= MistakeFactor)
+            {
+                queue.emplace(distance, ind);
+                if (queue.size() > MaxNumHints)
+                    queue.pop();
+            }
+        }
+    }
+
+    static std::vector<String> release(DistanceIndexQueue & queue, const std::vector<String> & prompting_strings)
+    {
+        std::vector<String> ans;
+        ans.reserve(queue.size());
+        while (!queue.empty())
+        {
+            auto top = queue.top();
+            queue.pop();
+            ans.push_back(prompting_strings[top.second]);
+        }
+        std::reverse(ans.begin(), ans.end());
+        return ans;
+    }
+};
+
+}

dbms/src/Common/SharedLibrary.cpp

@@ -1,9 +1,9 @@
 #include "SharedLibrary.h"
 #include <string>
-#include <dlfcn.h>
 #include <boost/core/noncopyable.hpp>
 #include "Exception.h"
 
+
 namespace DB
 {
 namespace ErrorCodes
@@ -12,9 +12,9 @@ namespace ErrorCodes
     extern const int CANNOT_DLSYM;
 }
 
-SharedLibrary::SharedLibrary(const std::string & path)
+SharedLibrary::SharedLibrary(const std::string & path, int flags)
 {
-    handle = dlopen(path.c_str(), RTLD_LAZY);
+    handle = dlopen(path.c_str(), flags);
     if (!handle)
         throw Exception(std::string("Cannot dlopen: ") + dlerror(), ErrorCodes::CANNOT_DLOPEN);
 }

dbms/src/Common/SharedLibrary.h

@@ -1,5 +1,6 @@
 #pragma once
 
+#include <dlfcn.h>
 #include <memory>
 #include <string>
 #include <boost/noncopyable.hpp>
@@ -8,12 +9,12 @@
 namespace DB
 {
 
-    /** Allows you to open a dynamic library and get a pointer to a function from it.
+/** Allows you to open a dynamic library and get a pointer to a function from it.
   */
 class SharedLibrary : private boost::noncopyable
 {
 public:
-    explicit SharedLibrary(const std::string & path);
+    explicit SharedLibrary(const std::string & path, int flags = RTLD_LAZY);
 
     ~SharedLibrary();
 

dbms/src/Common/ThreadStatus.h

@@ -116,7 +116,7 @@ public:
         return thread_state.load(std::memory_order_relaxed);
     }
 
-    String getQueryID();
+    const std::string & getQueryId() const;
 
     /// Starts new query and create new thread group for it, current thread becomes master thread of the query
     void initializeQuery();
@@ -160,6 +160,8 @@ protected:
     /// Use it only from current thread
     Context * query_context = nullptr;
 
+    String query_id;
+
     /// A logs queue used by TCPHandler to pass logs to a client
     InternalTextLogsQueueWeakPtr logs_queue_ptr;
 

dbms/src/Common/XDBCBridgeHelper.h

@@ -262,13 +262,7 @@ struct ODBCBridgeMixin
 
 
         std::vector<std::string> cmd_args;
-        path.setFileName(
-#if CLICKHOUSE_SPLIT_BINARY
-            "clickhouse-odbc-bridge"
-#else
-            "clickhouse"
-#endif
-        );
+        path.setFileName("clickhouse-odbc-bridge");
 
         std::stringstream command;
 

dbms/src/Compression/CachedCompressedReadBuffer.cpp

@@ -20,7 +20,7 @@ void CachedCompressedReadBuffer::initInput()
     if (!file_in)
     {
         file_in = createReadBufferFromFileBase(path, estimated_size, aio_threshold, buf_size);
-        compressed_in = &*file_in;
+        compressed_in = file_in.get();
 
         if (profile_callback)
             file_in->setProfileCallback(profile_callback, clock_type);
@@ -30,11 +30,12 @@ void CachedCompressedReadBuffer::initInput()
 
 bool CachedCompressedReadBuffer::nextImpl()
 {
+
     /// Let's check for the presence of a decompressed block in the cache, grab the ownership of this block, if it exists.
     UInt128 key = cache->hash(path, file_pos);
     owned_cell = cache->get(key);
 
-    if (!owned_cell)
+    if (!owned_cell || !codec)
     {
         /// If not, read it from the file.
         initInput();
@@ -42,7 +43,6 @@ bool CachedCompressedReadBuffer::nextImpl()
 
         owned_cell = std::make_shared<UncompressedCacheCell>();
 
-
         size_t size_decompressed;
         size_t size_compressed_without_checksum;
         owned_cell->compressed_size = readCompressedData(size_decompressed, size_compressed_without_checksum);
@@ -50,7 +50,7 @@ bool CachedCompressedReadBuffer::nextImpl()
         if (owned_cell->compressed_size)
         {
             owned_cell->data.resize(size_decompressed + codec->getAdditionalSizeAtTheEndOfBuffer());
-            decompress(owned_cell->data.data(), size_decompressed, owned_cell->compressed_size);
+            decompress(owned_cell->data.data(), size_decompressed, size_compressed_without_checksum);
 
             /// Put data into cache.
             cache->set(key, owned_cell);

dbms/src/Compression/CompressedReadBufferFromFile.cpp

@@ -23,7 +23,7 @@ bool CompressedReadBufferFromFile::nextImpl()
     if (!size_compressed)
         return false;
 
-    memory.resize(size_decompressed + LZ4::ADDITIONAL_BYTES_AT_END_OF_BUFFER);
+    memory.resize(size_decompressed + codec->getAdditionalSizeAtTheEndOfBuffer());
     working_buffer = Buffer(memory.data(), &memory[size_decompressed]);
 
     decompress(working_buffer.begin(), size_decompressed, size_compressed_without_checksum);
@@ -91,7 +91,7 @@ size_t CompressedReadBufferFromFile::readBig(char * to, size_t n)
             return bytes_read;
 
         /// If the decompressed block fits entirely where it needs to be copied.
-        if (size_decompressed + LZ4::ADDITIONAL_BYTES_AT_END_OF_BUFFER <= n - bytes_read)
+        if (size_decompressed + codec->getAdditionalSizeAtTheEndOfBuffer() <= n - bytes_read)
         {
             decompress(to + bytes_read, size_decompressed, size_compressed_without_checksum);
             bytes_read += size_decompressed;
@@ -101,7 +101,7 @@ size_t CompressedReadBufferFromFile::readBig(char * to, size_t n)
         {
             size_compressed = new_size_compressed;
             bytes += offset();
-            memory.resize(size_decompressed + LZ4::ADDITIONAL_BYTES_AT_END_OF_BUFFER);
+            memory.resize(size_decompressed + codec->getAdditionalSizeAtTheEndOfBuffer());
             working_buffer = Buffer(memory.data(), &memory[size_decompressed]);
             pos = working_buffer.begin();
 

dbms/src/Core/BackgroundSchedulePool.h

@@ -153,6 +153,4 @@ private:
     void attachToThreadGroup();
 };
 
-using BackgroundSchedulePoolPtr = std::shared_ptr<BackgroundSchedulePool>;
-
 }

dbms/src/DataStreams/CreatingSetsBlockInputStream.cpp

@@ -120,17 +120,7 @@ void CreatingSetsBlockInputStream::createOne(SubqueryForSet & subquery)
 
         if (!done_with_join)
         {
-            for (const auto & name_with_alias : subquery.joined_block_aliases)
-            {
-                if (block.has(name_with_alias.first))
-                {
-                    auto pos = block.getPositionByName(name_with_alias.first);
-                    auto column = block.getByPosition(pos);
-                    block.erase(pos);
-                    column.name = name_with_alias.second;
-                    block.insert(std::move(column));
-                }
-            }
+            subquery.renameColumns(block);
 
             if (subquery.joined_block_actions)
                 subquery.joined_block_actions->execute(block);

dbms/src/DataStreams/MergingSortedBlockInputStream.h

@@ -157,7 +157,7 @@ protected:
     using QueueWithCollation = std::priority_queue<SortCursorWithCollation>;
     QueueWithCollation queue_with_collation;
 
-    /// Used in Vertical merge algorithm to gather non-PK columns (on next step)
+    /// Used in Vertical merge algorithm to gather non-PK/non-index columns (on next step)
     /// If it is not nullptr then it should be populated during execution
     WriteBuffer * out_row_sources_buf;
 

dbms/src/DataStreams/ParallelInputsProcessor.h

@@ -183,7 +183,8 @@ private:
         try
         {
             setThreadName("ParalInputsProc");
-            CurrentThread::attachTo(thread_group);
+            if (thread_group)
+                CurrentThread::attachTo(thread_group);
 
             while (!finish)
             {

dbms/src/DataTypes/DataTypeFactory.cpp

@@ -7,7 +7,7 @@
 #include <Common/typeid_cast.h>
 #include <Poco/String.h>
 #include <Common/StringUtils/StringUtils.h>
-
+#include <IO/WriteHelpers.h>
 
 namespace DB
 {
@@ -87,7 +87,11 @@ DataTypePtr DataTypeFactory::get(const String & family_name_param, const ASTPtr
             return it->second(parameters);
     }
 
-    throw Exception("Unknown data type family: " + family_name, ErrorCodes::UNKNOWN_TYPE);
+    auto hints = this->getHints(family_name);
+    if (!hints.empty())
+        throw Exception("Unknown data type family: " + family_name + ". Maybe you meant: " + toString(hints), ErrorCodes::UNKNOWN_TYPE);
+    else
+        throw Exception("Unknown data type family: " + family_name, ErrorCodes::UNKNOWN_TYPE);
 }
 
 

dbms/src/Databases/DatabaseDictionary.cpp

@@ -20,9 +20,8 @@ namespace ErrorCodes
     extern const int SYNTAX_ERROR;
 }
 
-DatabaseDictionary::DatabaseDictionary(const String & name_, const Context & context)
+DatabaseDictionary::DatabaseDictionary(const String & name_)
     : name(name_),
-      external_dictionaries(context.getExternalDictionaries()),
       log(&Logger::get("DatabaseDictionary(" + name + ")"))
 {
 }
@@ -31,23 +30,21 @@ void DatabaseDictionary::loadTables(Context &, ThreadPool *, bool)
 {
 }
 
-Tables DatabaseDictionary::loadTables()
+Tables DatabaseDictionary::listTables(const Context & context)
 {
-    auto objects_map = external_dictionaries.getObjectsMap();
+    auto objects_map = context.getExternalDictionaries().getObjectsMap();
     const auto & dictionaries = objects_map.get();
 
     Tables tables;
     for (const auto & pair : dictionaries)
     {
-        const std::string & dict_name = pair.first;
-        if (deleted_tables.count(dict_name))
-            continue;
         auto dict_ptr = std::static_pointer_cast<IDictionaryBase>(pair.second.loadable);
         if (dict_ptr)
         {
             const DictionaryStructure & dictionary_structure = dict_ptr->getStructure();
             auto columns = StorageDictionary::getNamesAndTypes(dictionary_structure);
-            tables[dict_name] = StorageDictionary::create(dict_name, ColumnsDescription{columns}, dictionary_structure, dict_name);
+            const std::string & dict_name = pair.first;
+            tables[dict_name] = StorageDictionary::create(dict_name, ColumnsDescription{columns}, context, true, dict_name);
         }
     }
 
@@ -55,23 +52,21 @@ Tables DatabaseDictionary::loadTables()
 }
 
 bool DatabaseDictionary::isTableExist(
-    const Context & /*context*/,
+    const Context & context,
     const String & table_name) const
 {
-    auto objects_map = external_dictionaries.getObjectsMap();
+    auto objects_map = context.getExternalDictionaries().getObjectsMap();
     const auto & dictionaries = objects_map.get();
-    return dictionaries.count(table_name) && !deleted_tables.count(table_name);
+    return dictionaries.count(table_name);
 }
 
 StoragePtr DatabaseDictionary::tryGetTable(
-    const Context & /*context*/,
+    const Context & context,
     const String & table_name) const
 {
-    auto objects_map = external_dictionaries.getObjectsMap();
+    auto objects_map = context.getExternalDictionaries().getObjectsMap();
     const auto & dictionaries = objects_map.get();
 
-    if (deleted_tables.count(table_name))
-        return {};
     {
         auto it = dictionaries.find(table_name);
         if (it != dictionaries.end())
@@ -81,7 +76,7 @@ StoragePtr DatabaseDictionary::tryGetTable(
             {
                 const DictionaryStructure & dictionary_structure = dict_ptr->getStructure();
                 auto columns = StorageDictionary::getNamesAndTypes(dictionary_structure);
-                return StorageDictionary::create(table_name, ColumnsDescription{columns}, dictionary_structure, table_name);
+                return StorageDictionary::create(table_name, ColumnsDescription{columns}, context, true, table_name);
             }
         }
     }
@@ -89,17 +84,17 @@ StoragePtr DatabaseDictionary::tryGetTable(
     return {};
 }
 
-DatabaseIteratorPtr DatabaseDictionary::getIterator(const Context & /*context*/)
+DatabaseIteratorPtr DatabaseDictionary::getIterator(const Context & context)
 {
-    return std::make_unique<DatabaseSnapshotIterator>(loadTables());
+    return std::make_unique<DatabaseSnapshotIterator>(listTables(context));
 }
 
-bool DatabaseDictionary::empty(const Context & /*context*/) const
+bool DatabaseDictionary::empty(const Context & context) const
 {
-    auto objects_map = external_dictionaries.getObjectsMap();
+    auto objects_map = context.getExternalDictionaries().getObjectsMap();
     const auto & dictionaries = objects_map.get();
     for (const auto & pair : dictionaries)
-        if (pair.second.loadable && !deleted_tables.count(pair.first))
+        if (pair.second.loadable)
             return false;
     return true;
 }
@@ -115,23 +110,19 @@ void DatabaseDictionary::attachTable(const String & /*table_name*/, const Storag
 }
 
 void DatabaseDictionary::createTable(
-    const Context & /*context*/,
-    const String & /*table_name*/,
-    const StoragePtr & /*table*/,
-    const ASTPtr & /*query*/)
+    const Context &,
+    const String &,
+    const StoragePtr &,
+    const ASTPtr &)
 {
     throw Exception("DatabaseDictionary: createTable() is not supported", ErrorCodes::NOT_IMPLEMENTED);
 }
 
 void DatabaseDictionary::removeTable(
-    const Context & context,
-    const String & table_name)
+    const Context &,
+    const String &)
 {
-    if (!isTableExist(context, table_name))
-        throw Exception("Table " + name + "." + table_name + " doesn't exist.", ErrorCodes::UNKNOWN_TABLE);
-
-    auto objects_map = external_dictionaries.getObjectsMap();
-    deleted_tables.insert(table_name);
+    throw Exception("DatabaseDictionary: removeTable() is not supported", ErrorCodes::NOT_IMPLEMENTED);
 }
 
 void DatabaseDictionary::renameTable(
@@ -147,6 +138,7 @@ void DatabaseDictionary::alterTable(
     const Context &,
     const String &,
     const ColumnsDescription &,
+    const IndicesDescription &,
     const ASTModifier &)
 {
     throw Exception("DatabaseDictionary: alterTable() is not supported", ErrorCodes::NOT_IMPLEMENTED);

dbms/src/Databases/DatabaseDictionary.h

@@ -15,7 +15,6 @@ namespace Poco
 
 namespace DB
 {
-class ExternalDictionaries;
 
 /* Database to store StorageDictionary tables
  * automatically creates tables for all dictionaries
@@ -23,7 +22,7 @@ class ExternalDictionaries;
 class DatabaseDictionary : public IDatabase
 {
 public:
-    DatabaseDictionary(const String & name_, const Context & context);
+    DatabaseDictionary(const String & name_);
 
     String getDatabaseName() const override;
 
@@ -72,6 +71,7 @@ public:
         const Context & context,
         const String & name,
         const ColumnsDescription & columns,
+        const IndicesDescription & indices,
         const ASTModifier & engine_modifier) override;
 
     time_t getTableMetadataModificationTime(
@@ -93,13 +93,10 @@ public:
 private:
     const String name;
     mutable std::mutex mutex;
-    const ExternalDictionaries & external_dictionaries;
-    std::unordered_set<String> deleted_tables;
 
     Poco::Logger * log;
 
-    Tables loadTables();
-
+    Tables listTables(const Context & context);
     ASTPtr getCreateTableQueryImpl(const Context & context, const String & table_name, bool throw_on_error) const;
 };
 

dbms/src/Databases/DatabaseFactory.cpp

@@ -23,7 +23,7 @@ DatabasePtr DatabaseFactory::get(
     else if (engine_name == "Memory")
         return std::make_shared<DatabaseMemory>(database_name);
     else if (engine_name == "Dictionary")
-        return std::make_shared<DatabaseDictionary>(database_name, context);
+        return std::make_shared<DatabaseDictionary>(database_name);
 
     throw Exception("Unknown database engine: " + engine_name, ErrorCodes::UNKNOWN_DATABASE_ENGINE);
 }

dbms/src/Databases/DatabaseMemory.cpp

@@ -53,6 +53,7 @@ void DatabaseMemory::alterTable(
     const Context &,
     const String &,
     const ColumnsDescription &,
+    const IndicesDescription &,
     const ASTModifier &)
 {
     throw Exception("DatabaseMemory: alterTable() is not supported", ErrorCodes::NOT_IMPLEMENTED);

dbms/src/Databases/DatabaseMemory.h

@@ -48,6 +48,7 @@ public:
         const Context & context,
         const String & name,
         const ColumnsDescription & columns,
+        const IndicesDescription & indices,
         const ASTModifier & engine_modifier) override;
 
     time_t getTableMetadataModificationTime(

dbms/src/Databases/DatabaseOrdinary.cpp

@@ -510,6 +510,7 @@ void DatabaseOrdinary::alterTable(
     const Context & context,
     const String & table_name,
     const ColumnsDescription & columns,
+    const IndicesDescription & indices,
     const ASTModifier & storage_modifier)
 {
     /// Read the definition of the table and replace the necessary parts with new ones.
@@ -531,7 +532,14 @@ void DatabaseOrdinary::alterTable(
     ASTCreateQuery & ast_create_query = typeid_cast<ASTCreateQuery &>(*ast);
 
     ASTPtr new_columns = InterpreterCreateQuery::formatColumns(columns);
-    ast_create_query.replace(ast_create_query.columns, new_columns);
+    ASTPtr new_indices = InterpreterCreateQuery::formatIndices(indices);
+
+    ast_create_query.columns_list->replace(ast_create_query.columns_list->columns, new_columns);
+
+    if (ast_create_query.columns_list->indices)
+        ast_create_query.columns_list->replace(ast_create_query.columns_list->indices, new_indices);
+    else
+        ast_create_query.columns_list->set(ast_create_query.columns_list->indices, new_indices);
 
     if (storage_modifier)
         storage_modifier(*ast_create_query.storage);

dbms/src/Databases/DatabaseOrdinary.h

@@ -42,6 +42,7 @@ public:
         const Context & context,
         const String & name,
         const ColumnsDescription & columns,
+        const IndicesDescription & indices,
         const ASTModifier & engine_modifier) override;
 
     time_t getTableMetadataModificationTime(

dbms/src/Databases/DatabasesCommon.cpp

@@ -68,10 +68,10 @@ std::pair<String, StoragePtr> createTableFromDefinition(
     /// We do not directly use `InterpreterCreateQuery::execute`, because
     /// - the database has not been created yet;
     /// - the code is simpler, since the query is already brought to a suitable form.
-    if (!ast_create_query.columns)
+    if (!ast_create_query.columns_list || !ast_create_query.columns_list->columns)
         throw Exception("Missing definition of columns.", ErrorCodes::EMPTY_LIST_OF_COLUMNS_PASSED);
 
-    ColumnsDescription columns = InterpreterCreateQuery::getColumnsDescription(*ast_create_query.columns, context);
+    ColumnsDescription columns = InterpreterCreateQuery::getColumnsDescription(*ast_create_query.columns_list->columns, context);
 
     return
     {

dbms/src/Databases/IDatabase.h

@@ -3,6 +3,7 @@
 #include <Core/Types.h>
 #include <Core/NamesAndTypes.h>
 #include <Storages/ColumnsDescription.h>
+#include <Storages/IndicesDescription.h>
 #include <ctime>
 #include <memory>
 #include <functional>
@@ -115,6 +116,7 @@ public:
         const Context & context,
         const String & name,
         const ColumnsDescription & columns,
+        const IndicesDescription & indices,
         const ASTModifier & engine_modifier) = 0;
 
     /// Returns time of table's metadata change, 0 if there is no corresponding metadata file.

dbms/src/Dictionaries/ClickHouseDictionarySource.cpp

@@ -54,7 +54,7 @@ ClickHouseDictionarySource::ClickHouseDictionarySource(
     const Poco::Util::AbstractConfiguration & config,
     const std::string & config_prefix,
     const Block & sample_block,
-    Context & context)
+    Context & context_)
     : update_time{std::chrono::system_clock::from_time_t(0)}
     , dict_struct{dict_struct_}
     , host{config.getString(config_prefix + ".host")}
@@ -69,11 +69,13 @@ ClickHouseDictionarySource::ClickHouseDictionarySource(
     , invalidate_query{config.getString(config_prefix + ".invalidate_query", "")}
     , query_builder{dict_struct, db, table, where, IdentifierQuotingStyle::Backticks}
     , sample_block{sample_block}
-    , context(context)
+    , context(context_)
     , is_local{isLocalAddress({host, port}, context.getTCPPort())}
     , pool{is_local ? nullptr : createPool(host, port, secure, db, user, password, context)}
     , load_all_query{query_builder.composeLoadAllQuery()}
 {
+    /// We should set user info even for the case when the dictionary is loaded in-process (without TCP communication).
+    context.setUser(user, password, Poco::Net::SocketAddress("127.0.0.1", 0), {});
 }
 
 
@@ -182,7 +184,8 @@ std::string ClickHouseDictionarySource::doInvalidateQuery(const std::string & re
 {
     if (is_local)
     {
-        auto input_block = executeQuery(request, context, true).in;
+        Context query_context = context;
+        auto input_block = executeQuery(request, query_context, true).in;
         return readInvalidateQuery(*input_block);
     }
     else
@@ -201,7 +204,8 @@ void registerDictionarySourceClickHouse(DictionarySourceFactory & factory)
                                  const Poco::Util::AbstractConfiguration & config,
                                  const std::string & config_prefix,
                                  Block & sample_block,
-                                 Context & context) -> DictionarySourcePtr {
+                                 Context & context) -> DictionarySourcePtr
+    {
         return std::make_unique<ClickHouseDictionarySource>(dict_struct, config, config_prefix + ".clickhouse", sample_block, context);
     };
     factory.registerSource("clickhouse", createTableSource);

dbms/src/Dictionaries/ClickHouseDictionarySource.h

@@ -2,6 +2,7 @@
 
 #include <memory>
 #include <Client/ConnectionPoolWithFailover.h>
+#include <Interpreters/Context.h>
 #include "DictionaryStructure.h"
 #include "ExternalQueryBuilder.h"
 #include "IDictionarySource.h"
@@ -65,7 +66,7 @@ private:
     mutable std::string invalidate_query_response;
     ExternalQueryBuilder query_builder;
     Block sample_block;
-    Context & context;
+    Context context;
     const bool is_local;
     ConnectionPoolWithFailoverPtr pool;
     const std::string load_all_query;

dbms/src/Dictionaries/DictionaryFactory.cpp

@@ -14,7 +14,6 @@ namespace ErrorCodes
 
 void DictionaryFactory::registerLayout(const std::string & layout_type, Creator create_layout)
 {
-    //LOG_DEBUG(log, "Register dictionary layout type `" + layout_type + "`");
     if (!registered_layouts.emplace(layout_type, std::move(create_layout)).second)
         throw Exception("DictionaryFactory: the layout name '" + layout_type + "' is not unique", ErrorCodes::LOGICAL_ERROR);
 }

dbms/src/Dictionaries/ExecutableDictionarySource.cpp

@@ -234,7 +234,8 @@ void registerDictionarySourceExecutable(DictionarySourceFactory & factory)
                                  const Poco::Util::AbstractConfiguration & config,
                                  const std::string & config_prefix,
                                  Block & sample_block,
-                                 const Context & context) -> DictionarySourcePtr {
+                                 Context & context) -> DictionarySourcePtr
+    {
         if (dict_struct.has_expressions)
             throw Exception{"Dictionary source of type `executable` does not support attribute expressions", ErrorCodes::LOGICAL_ERROR};
 

dbms/src/Dictionaries/FileDictionarySource.cpp

@@ -56,7 +56,8 @@ void registerDictionarySourceFile(DictionarySourceFactory & factory)
                                  const Poco::Util::AbstractConfiguration & config,
                                  const std::string & config_prefix,
                                  Block & sample_block,
-                                 const Context & context) -> DictionarySourcePtr {
+                                 Context & context) -> DictionarySourcePtr
+    {
         if (dict_struct.has_expressions)
             throw Exception{"Dictionary source of type `file` does not support attribute expressions", ErrorCodes::LOGICAL_ERROR};
 

dbms/src/Dictionaries/HTTPDictionarySource.cpp

@@ -157,7 +157,8 @@ void registerDictionarySourceHTTP(DictionarySourceFactory & factory)
                                  const Poco::Util::AbstractConfiguration & config,
                                  const std::string & config_prefix,
                                  Block & sample_block,
-                                 const Context & context) -> DictionarySourcePtr {
+                                 Context & context) -> DictionarySourcePtr
+    {
         if (dict_struct.has_expressions)
             throw Exception{"Dictionary source of type `http` does not support attribute expressions", ErrorCodes::LOGICAL_ERROR};
 

dbms/src/Dictionaries/LibraryDictionarySource.cpp

@@ -121,21 +121,23 @@ LibraryDictionarySource::LibraryDictionarySource(
     const DictionaryStructure & dict_struct_,
     const Poco::Util::AbstractConfiguration & config,
     const std::string & config_prefix,
-    Block & sample_block,
-    const Context & context)
+    Block & sample_block)
     : log(&Logger::get("LibraryDictionarySource"))
     , dict_struct{dict_struct_}
     , config_prefix{config_prefix}
     , path{config.getString(config_prefix + ".path", "")}
     , sample_block{sample_block}
-    , context(context)
 {
     if (!Poco::File(path).exists())
         throw Exception(
             "LibraryDictionarySource: Can't load lib " + toString() + ": " + Poco::File(path).path() + " - File doesn't exist",
             ErrorCodes::FILE_DOESNT_EXIST);
     description.init(sample_block);
-    library = std::make_shared<SharedLibrary>(path);
+    library = std::make_shared<SharedLibrary>(path, RTLD_LAZY
+#if defined(RTLD_DEEPBIND) // Does not exists in freebsd
+        | RTLD_DEEPBIND
+#endif
+    );
     settings = std::make_shared<CStringsHolder>(getLibSettings(config, config_prefix + lib_config_settings));
     if (auto libNew = library->tryGet<decltype(lib_data) (*)(decltype(&settings->strings), decltype(&ClickHouseLibrary::log))>(
             "ClickHouseDictionary_v3_libNew"))
@@ -148,7 +150,6 @@ LibraryDictionarySource::LibraryDictionarySource(const LibraryDictionarySource &
     , config_prefix{other.config_prefix}
     , path{other.path}
     , sample_block{other.sample_block}
-    , context(other.context)
     , library{other.library}
     , description{other.description}
     , settings{other.settings}
@@ -284,8 +285,9 @@ void registerDictionarySourceLibrary(DictionarySourceFactory & factory)
                                  const Poco::Util::AbstractConfiguration & config,
                                  const std::string & config_prefix,
                                  Block & sample_block,
-                                 const Context & context) -> DictionarySourcePtr {
-        return std::make_unique<LibraryDictionarySource>(dict_struct, config, config_prefix + ".library", sample_block, context);
+                                 const Context &) -> DictionarySourcePtr
+    {
+        return std::make_unique<LibraryDictionarySource>(dict_struct, config, config_prefix + ".library", sample_block);
     };
     factory.registerSource("library", createTableSource);
 }

dbms/src/Dictionaries/LibraryDictionarySource.h

@@ -32,8 +32,7 @@ public:
         const DictionaryStructure & dict_struct_,
         const Poco::Util::AbstractConfiguration & config,
         const std::string & config_prefix,
-        Block & sample_block,
-        const Context & context);
+        Block & sample_block);
 
     LibraryDictionarySource(const LibraryDictionarySource & other);
 
@@ -70,7 +69,6 @@ private:
     const std::string config_prefix;
     const std::string path;
     Block sample_block;
-    const Context & context;
     SharedLibraryPtr library;
     ExternalResultDescription description;
     std::shared_ptr<CStringsHolder> settings;

dbms/src/Functions/CMakeLists.txt

@@ -36,6 +36,7 @@ endif ()
 
 if (USE_ICU)
     target_link_libraries (clickhouse_functions PRIVATE ${ICU_LIBRARIES})
+    target_include_directories(clickhouse_functions SYSTEM PRIVATE ${ICU_INCLUDE_DIRS})
 endif ()
 
 if (USE_VECTORCLASS)

dbms/src/Functions/FunctionFactory.cpp

@@ -6,6 +6,8 @@
 
 #include <Poco/String.h>
 
+#include <IO/WriteHelpers.h>
+
 namespace DB
 {
 
@@ -43,7 +45,13 @@ FunctionBuilderPtr FunctionFactory::get(
 {
     auto res = tryGet(name, context);
     if (!res)
-        throw Exception("Unknown function " + name, ErrorCodes::UNKNOWN_FUNCTION);
+    {
+        auto hints = this->getHints(name);
+        if (!hints.empty())
+            throw Exception("Unknown function " + name + ". Maybe you meant: " + toString(hints), ErrorCodes::UNKNOWN_FUNCTION);
+        else
+            throw Exception("Unknown function " + name, ErrorCodes::UNKNOWN_FUNCTION);
+    }
     return res;
 }
 

dbms/src/Functions/FunctionIfBase.h

@@ -15,9 +15,26 @@ class FunctionIfBase : public IFunction
 public:
     bool isCompilableImpl(const DataTypes & types) const override
     {
+        /// It's difficult to compare Date and DateTime - cannot use JIT compilation.
+        bool has_date = false;
+        bool has_datetime = false;
+
         for (const auto & type : types)
-            if (!isCompilableType(removeNullable(type)))
+        {
+            auto type_removed_nullable = removeNullable(type);
+            WhichDataType which(type_removed_nullable);
+
+            if (which.isDate())
+                has_date = true;
+            if (which.isDateTime())
+                has_datetime = true;
+
+            if (has_date && has_datetime)
                 return false;
+
+            if (!isCompilableType(type_removed_nullable))
+                return false;
+        }
         return true;
     }
 

dbms/src/Functions/FunctionsComparison.h

@@ -1146,10 +1146,16 @@ public:
         const DataTypePtr & left_type = col_with_type_and_name_left.type;
         const DataTypePtr & right_type = col_with_type_and_name_right.type;
 
+        WhichDataType which_left{left_type};
+        WhichDataType which_right{right_type};
+
         const bool left_is_num = col_left_untyped->isNumeric();
         const bool right_is_num = col_right_untyped->isNumeric();
 
-        if (left_is_num && right_is_num)
+        bool date_and_datetime = (left_type != right_type) &&
+            which_left.isDateOrDateTime() && which_right.isDateOrDateTime();
+
+        if (left_is_num && right_is_num && !date_and_datetime)
         {
             if (!(executeNumLeftType<UInt8>(block, result, col_left_untyped, col_right_untyped)
                 || executeNumLeftType<UInt16>(block, result, col_left_untyped, col_right_untyped)
@@ -1203,7 +1209,10 @@ public:
     {
         auto isBigInteger = &typeIsEither<DataTypeInt64, DataTypeUInt64, DataTypeUUID>;
         auto isFloatingPoint = &typeIsEither<DataTypeFloat32, DataTypeFloat64>;
-        if ((isBigInteger(*types[0]) && isFloatingPoint(*types[1])) || (isBigInteger(*types[1]) && isFloatingPoint(*types[0])))
+        if ((isBigInteger(*types[0]) && isFloatingPoint(*types[1]))
+            || (isBigInteger(*types[1]) && isFloatingPoint(*types[0]))
+            || (WhichDataType(types[0]).isDate() && WhichDataType(types[1]).isDateTime())
+            || (WhichDataType(types[1]).isDate() && WhichDataType(types[0]).isDateTime()))
             return false; /// TODO: implement (double, int_N where N > double's mantissa width)
         return isCompilableType(types[0]) && isCompilableType(types[1]);
     }

dbms/src/Functions/FunctionsEmbeddedDictionaries.h

@@ -186,7 +186,7 @@ public:
         : owned_dict(owned_dict_)
     {
         if (!owned_dict)
-            throw Exception("Dictionaries was not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
+            throw Exception("Embedded dictionaries were not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
     }
 
     String getName() const override
@@ -280,7 +280,7 @@ public:
         : owned_dict(owned_dict_)
     {
         if (!owned_dict)
-            throw Exception("Dictionaries was not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
+            throw Exception("Embedded dictionaries were not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
     }
 
     String getName() const override
@@ -418,7 +418,7 @@ public:
     : owned_dict(owned_dict_)
     {
         if (!owned_dict)
-            throw Exception("Dictionaries was not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
+            throw Exception("Embedded dictionaries were not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
     }
 
     String getName() const override
@@ -690,7 +690,7 @@ public:
         : owned_dict(owned_dict_)
     {
         if (!owned_dict)
-            throw Exception("Dictionaries was not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
+            throw Exception("Embedded dictionaries were not loaded. You need to check configuration file.", ErrorCodes::DICTIONARIES_WAS_NOT_LOADED);
     }
 
     String getName() const override

dbms/src/Functions/IFunction.h

@@ -151,6 +151,8 @@ public:
 
 #endif
 
+    virtual bool isStateful() const { return false; }
+
     /** Should we evaluate this function while constant folding, if arguments are constants?
       * Usually this is true. Notable counterexample is function 'sleep'.
       * If we will call it during query analysis, we will sleep extra amount of time.
@@ -230,6 +232,9 @@ public:
     /// Get the main function name.
     virtual String getName() const = 0;
 
+    /// Override and return true if function needs to depend on the state of the data.
+    virtual bool isStateful() const { return false; }
+
     /// Override and return true if function could take different number of arguments.
     virtual bool isVariadic() const { return false; }
 
@@ -322,6 +327,9 @@ class IFunction : public std::enable_shared_from_this<IFunction>,
 {
 public:
     String getName() const override = 0;
+
+    bool isStateful() const override { return false; }
+
     /// TODO: make const
     void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override = 0;
 
@@ -478,6 +486,7 @@ public:
     }
 
     String getName() const override { return function->getName(); }
+    bool isStateful() const override { return function->isStateful(); }
     bool isVariadic() const override { return function->isVariadic(); }
     size_t getNumberOfArguments() const override { return function->getNumberOfArguments(); }
 

dbms/src/Functions/bitSwapLastTwo.cpp

@@ -0,0 +1,51 @@
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionUnaryArithmetic.h>
+#include <DataTypes/NumberTraits.h>
+
+namespace DB
+{
+
+template <typename A>
+struct BitSwapLastTwoImpl
+{
+    using ResultType = UInt8;
+
+    static inline ResultType apply(A a)
+    {
+        return static_cast<ResultType>(
+                ((static_cast<ResultType>(a) & 1) << 1) | ((static_cast<ResultType>(a) >> 1) & 1));
+    }
+
+#if USE_EMBEDDED_COMPILER
+    static constexpr bool compilable = true;
+
+    static inline llvm::Value * compile(llvm::IRBuilder<> & b, llvm::Value * arg, bool)
+    {
+        if (!arg->getType()->isIntegerTy())
+            throw Exception("__bitSwapLastTwo expected an integral type", ErrorCodes::LOGICAL_ERROR);
+        return b.CreateOr(
+                b.CreateShl(b.CreateAnd(arg, 1), 1),
+                b.CreateAnd(b.CreateLShr(arg, 1), 1)
+                );
+    }
+#endif
+};
+
+struct NameBitSwapLastTwo { static constexpr auto name = "__bitSwapLastTwo"; };
+using FunctionBitSwapLastTwo = FunctionUnaryArithmetic<BitSwapLastTwoImpl, NameBitSwapLastTwo, true>;
+
+template <> struct FunctionUnaryArithmeticMonotonicity<NameBitSwapLastTwo>
+{
+    static bool has() { return false; }
+    static IFunction::Monotonicity get(const Field &, const Field &)
+    {
+        return {};
+    }
+};
+
+void registerFunctionBitSwapLastTwo(FunctionFactory & factory)
+{
+    factory.registerFunction<FunctionBitSwapLastTwo>();
+}
+
+}

dbms/src/Functions/blockNumber.cpp

@@ -27,6 +27,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 0;

dbms/src/Functions/finalizeAggregation.cpp

@@ -33,6 +33,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 1;

dbms/src/Functions/if.cpp

@@ -22,6 +22,7 @@
 #include <Functions/GatherUtils/Algorithms.h>
 #include <Functions/FunctionIfBase.h>
 #include <Functions/FunctionFactory.h>
+#include <Interpreters/castColumn.h>
 
 
 namespace DB
@@ -168,7 +169,8 @@ class FunctionIf : public FunctionIfBase</*null_is_false=*/false>
 {
 public:
     static constexpr auto name = "if";
-    static FunctionPtr create(const Context &) { return std::make_shared<FunctionIf>(); }
+    static FunctionPtr create(const Context & context) { return std::make_shared<FunctionIf>(context); }
+    FunctionIf(const Context & context) : context(context) {}
 
 private:
     template <typename T0, typename T1>
@@ -588,6 +590,72 @@ private:
         return true;
     }
 
+    void executeGeneric(const ColumnUInt8 * cond_col, Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count)
+    {
+        /// Convert both columns to the common type (if needed).
+
+        const ColumnWithTypeAndName & arg1 = block.getByPosition(arguments[1]);
+        const ColumnWithTypeAndName & arg2 = block.getByPosition(arguments[2]);
+
+        DataTypePtr common_type = getLeastSupertype({arg1.type, arg2.type});
+
+        ColumnPtr col_then = castColumn(arg1, common_type, context);
+        ColumnPtr col_else = castColumn(arg2, common_type, context);
+
+        MutableColumnPtr result_column = common_type->createColumn();
+        result_column->reserve(input_rows_count);
+
+        bool then_is_const = col_then->isColumnConst();
+        bool else_is_const = col_else->isColumnConst();
+
+        const auto & cond_array = cond_col->getData();
+
+        if (then_is_const && else_is_const)
+        {
+            const IColumn & then_nested_column = static_cast<const ColumnConst &>(*col_then).getDataColumn();
+            const IColumn & else_nested_column = static_cast<const ColumnConst &>(*col_else).getDataColumn();
+
+            for (size_t i = 0; i < input_rows_count; ++i)
+            {
+                if (cond_array[i])
+                    result_column->insertFrom(then_nested_column, 0);
+                else
+                    result_column->insertFrom(else_nested_column, 0);
+            }
+        }
+        else if (then_is_const)
+        {
+            const IColumn & then_nested_column = static_cast<const ColumnConst &>(*col_then).getDataColumn();
+
+            for (size_t i = 0; i < input_rows_count; ++i)
+            {
+                if (cond_array[i])
+                    result_column->insertFrom(then_nested_column, 0);
+                else
+                    result_column->insertFrom(*col_else, i);
+            }
+        }
+        else if (else_is_const)
+        {
+            const IColumn & else_nested_column = static_cast<const ColumnConst &>(*col_else).getDataColumn();
+
+            for (size_t i = 0; i < input_rows_count; ++i)
+            {
+                if (cond_array[i])
+                    result_column->insertFrom(*col_then, i);
+                else
+                    result_column->insertFrom(else_nested_column, 0);
+            }
+        }
+        else
+        {
+            for (size_t i = 0; i < input_rows_count; ++i)
+                result_column->insertFrom(cond_array[i] ? *col_then : *col_else, i);
+        }
+
+        block.getByPosition(result).column = std::move(result_column);
+    }
+
     bool executeForNullableCondition(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count)
     {
         const ColumnWithTypeAndName & arg_cond = block.getByPosition(arguments[0]);
@@ -873,6 +941,14 @@ public:
         const ColumnWithTypeAndName & arg_then = block.getByPosition(arguments[1]);
         const ColumnWithTypeAndName & arg_else = block.getByPosition(arguments[2]);
 
+        /// A case for identical then and else (pointers are the same).
+        if (arg_then.column.get() == arg_else.column.get())
+        {
+            /// Just point result to them.
+            block.getByPosition(result).column = arg_then.column;
+            return;
+        }
+
         const ColumnUInt8 * cond_col = typeid_cast<const ColumnUInt8 *>(arg_cond.column.get());
         const ColumnConst * cond_const_col = checkAndGetColumnConst<ColumnVector<UInt8>>(arg_cond.column.get());
         ColumnPtr materialized_cond_col;
@@ -919,17 +995,17 @@ public:
         if (auto rigth_array = checkAndGetDataType<DataTypeArray>(arg_else.type.get()))
             right_id = rigth_array->getNestedType()->getTypeId();
 
-        bool executed_with_nums = callOnBasicTypes<true, true, true, true>(left_id, right_id, call);
-
-        if (!(executed_with_nums
+        if (!(callOnBasicTypes<true, true, true, false>(left_id, right_id, call)
             || executeTyped<UInt128, UInt128>(cond_col, block, arguments, result, input_rows_count)
             || executeString(cond_col, block, arguments, result)
             || executeGenericArray(cond_col, block, arguments, result)
             || executeTuple(block, arguments, result, input_rows_count)))
-            throw Exception("Illegal columns " + arg_then.column->getName() + " and " + arg_else.column->getName()
-                + " of second (then) and third (else) arguments of function " + getName(),
-                ErrorCodes::ILLEGAL_COLUMN);
+        {
+            executeGeneric(cond_col, block, arguments, result, input_rows_count);
+        }
     }
+
+    const Context & context;
 };
 
 void registerFunctionIf(FunctionFactory & factory)

dbms/src/Functions/registerFunctionsArithmetic.cpp

@@ -33,6 +33,8 @@ void registerFunctionRoundToExp2(FunctionFactory & factory);
 void registerFunctionRoundDuration(FunctionFactory & factory);
 void registerFunctionRoundAge(FunctionFactory & factory);
 
+void registerFunctionBitSwapLastTwo(FunctionFactory & factory);
+
 void registerFunctionsArithmetic(FunctionFactory & factory)
 {
     registerFunctionPlus(factory);
@@ -64,6 +66,9 @@ void registerFunctionsArithmetic(FunctionFactory & factory)
     registerFunctionRoundToExp2(factory);
     registerFunctionRoundDuration(factory);
     registerFunctionRoundAge(factory);
+
+    /// Not for external use.
+    registerFunctionBitSwapLastTwo(factory);
 }
 
 }

dbms/src/Functions/rowNumberInAllBlocks.cpp

@@ -27,6 +27,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 0;

dbms/src/Functions/rowNumberInBlock.cpp

@@ -22,6 +22,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 0;

dbms/src/Functions/runningAccumulate.cpp

@@ -41,6 +41,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 1;

dbms/src/Functions/runningDifference.h

@@ -130,6 +130,11 @@ public:
         return name;
     }
 
+    bool isStateful() const override
+    {
+        return true;
+    }
+
     size_t getNumberOfArguments() const override
     {
         return 1;

dbms/src/IO/InterserverWriteBuffer.cpp

@@ -1,111 +0,0 @@
-#include <IO/InterserverWriteBuffer.h>
-#include <IO/WriteBufferFromOStream.h>
-
-#include <Poco/Version.h>
-#include <Poco/URI.h>
-#include <Poco/Net/HTTPRequest.h>
-#include <Poco/Net/HTTPResponse.h>
-
-#include <common/logger_useful.h>
-
-
-namespace DB
-{
-
-namespace ErrorCodes
-{
-    extern const int CANNOT_WRITE_TO_OSTREAM;
-    extern const int RECEIVED_ERROR_FROM_REMOTE_IO_SERVER;
-}
-
-InterserverWriteBuffer::InterserverWriteBuffer(const std::string & host_, int port_,
-    const std::string & endpoint_,
-    const std::string & path_,
-    bool compress_,
-    size_t buffer_size_,
-    const Poco::Timespan & connection_timeout,
-    const Poco::Timespan & send_timeout,
-    const Poco::Timespan & receive_timeout)
-    : WriteBuffer(nullptr, 0), host(host_), port(port_), path(path_)
-{
-    std::string encoded_path;
-    Poco::URI::encode(path, "&#", encoded_path);
-
-    std::string encoded_endpoint;
-    Poco::URI::encode(endpoint_, "&#", encoded_endpoint);
-
-    std::string compress_str = compress_ ? "true" : "false";
-    std::string encoded_compress;
-    Poco::URI::encode(compress_str, "&#", encoded_compress);
-
-    std::stringstream uri;
-    uri << "http://" << host << ":" << port
-        << "/?endpoint=" << encoded_endpoint
-        << "&compress=" << encoded_compress
-        << "&path=" << encoded_path;
-
-    std::string uri_str = Poco::URI(uri.str()).getPathAndQuery();
-
-    session.setHost(host);
-    session.setPort(port);
-    session.setKeepAlive(true);
-
-    /// set the timeout
-#if POCO_CLICKHOUSE_PATCH || POCO_VERSION >= 0x02000000
-    session.setTimeout(connection_timeout, send_timeout, receive_timeout);
-#else
-    session.setTimeout(connection_timeout);
-    static_cast <void> (send_timeout);
-    static_cast <void> (receive_timeout);
-#endif
-
-    Poco::Net::HTTPRequest request(Poco::Net::HTTPRequest::HTTP_POST, uri_str, Poco::Net::HTTPRequest::HTTP_1_1);
-
-    request.setChunkedTransferEncoding(true);
-
-    ostr = &session.sendRequest(request);
-    impl = std::make_unique<WriteBufferFromOStream>(*ostr, buffer_size_);
-    set(impl->buffer().begin(), impl->buffer().size());
-}
-
-InterserverWriteBuffer::~InterserverWriteBuffer()
-{
-    try
-    {
-        finalize();
-    }
-    catch (...)
-    {
-        tryLogCurrentException(__PRETTY_FUNCTION__);
-    }
-}
-
-void InterserverWriteBuffer::nextImpl()
-{
-    if (!offset() || finalized)
-        return;
-
-    /// For correct work with AsynchronousWriteBuffer, which replaces buffers.
-    impl->set(buffer().begin(), buffer().size());
-
-    impl->position() = pos;
-
-    impl->next();
-}
-
-void InterserverWriteBuffer::finalize()
-{
-    if (finalized)
-        return;
-
-    next();
-
-    finalized = true;
-}
-
-void InterserverWriteBuffer::cancel()
-{
-    finalized = true;
-}
-
-}

dbms/src/IO/InterserverWriteBuffer.h

@@ -1,54 +0,0 @@
-#pragma once
-
-#include <IO/WriteBuffer.h>
-#include <IO/HashingWriteBuffer.h>
-
-#include <Poco/Net/HTTPClientSession.h>
-
-namespace DB
-{
-
-namespace
-{
-
-constexpr auto DEFAULT_REMOTE_WRITE_BUFFER_CONNECTION_TIMEOUT = 1;
-constexpr auto DEFAULT_REMOTE_WRITE_BUFFER_RECEIVE_TIMEOUT = 1800;
-constexpr auto DEFAULT_REMOTE_WRITE_BUFFER_SEND_TIMEOUT = 1800;
-
-}
-
-/** Allows you to write a file to a remote server.
-  */
-class InterserverWriteBuffer final : public WriteBuffer
-{
-public:
-    InterserverWriteBuffer(const std::string & host_, int port_,
-        const std::string & endpoint_,
-        const std::string & path_,
-        bool compress_ = false,
-        size_t buffer_size_ = DBMS_DEFAULT_BUFFER_SIZE,
-        const Poco::Timespan & connection_timeout = Poco::Timespan(DEFAULT_REMOTE_WRITE_BUFFER_CONNECTION_TIMEOUT, 0),
-        const Poco::Timespan & send_timeout = Poco::Timespan(DEFAULT_REMOTE_WRITE_BUFFER_SEND_TIMEOUT, 0),
-        const Poco::Timespan & receive_timeout = Poco::Timespan(DEFAULT_REMOTE_WRITE_BUFFER_RECEIVE_TIMEOUT, 0));
-
-    ~InterserverWriteBuffer() override;
-    void finalize();
-    void cancel();
-
-private:
-    void nextImpl() override;
-
-private:
-    std::string host;
-    int port;
-    std::string path;
-
-    Poco::Net::HTTPClientSession session;
-    std::ostream * ostr;    /// this is owned by session
-    std::unique_ptr<WriteBuffer> impl;
-
-    /// Sent all the data and renamed the file
-    bool finalized = false;
-};
-
-}

dbms/src/Interpreters/ActionsVisitor.cpp

@@ -357,7 +357,18 @@ void ActionsVisitor::visit(const ASTPtr & ast)
             ? context.getQueryContext()
             : context;
 
-        const FunctionBuilderPtr & function_builder = FunctionFactory::instance().get(node->name, function_context);
+        FunctionBuilderPtr function_builder;
+        try
+        {
+            function_builder = FunctionFactory::instance().get(node->name, function_context);
+        }
+        catch (DB::Exception & e)
+        {
+            auto hints = AggregateFunctionFactory::instance().getHints(node->name);
+            if (!hints.empty())
+                e.addMessage("Or unknown aggregate function " + node->name + ". Maybe you meant: " + toString(hints));
+            e.rethrow();
+        }
 
         Names argument_names;
         DataTypes argument_types;

dbms/src/Interpreters/ActionsVisitor.h

@@ -3,6 +3,7 @@
 #include <Parsers/IAST.h>
 #include <Interpreters/PreparedSets.h>
 #include <Interpreters/ExpressionActions.h>
+#include <Interpreters/SubqueryForSet.h>
 
 
 namespace DB
@@ -11,32 +12,6 @@ namespace DB
 class Context;
 class ASTFunction;
 
-class Join;
-using JoinPtr = std::shared_ptr<Join>;
-
-/// Information on what to do when executing a subquery in the [GLOBAL] IN/JOIN section.
-struct SubqueryForSet
-{
-    /// The source is obtained using the InterpreterSelectQuery subquery.
-    BlockInputStreamPtr source;
-
-    /// If set, build it from result.
-    SetPtr set;
-    JoinPtr join;
-    /// Apply this actions to joined block.
-    ExpressionActionsPtr joined_block_actions;
-    /// Rename column from joined block from this list.
-    NamesWithAliases joined_block_aliases;
-
-    /// If set, put the result into the table.
-    /// This is a temporary table for transferring to remote servers for distributed query processing.
-    StoragePtr table;
-};
-
-/// ID of subquery -> what to do with it.
-using SubqueriesForSets = std::unordered_map<String, SubqueryForSet>;
-
-
  /// The case of an explicit enumeration of values.
 SetPtr makeExplicitSet(
     const ASTFunction * node, const Block & sample_block, bool create_ordered_set,

dbms/src/Interpreters/AnalyzedJoin.cpp

@@ -16,8 +16,7 @@ namespace DB
 ExpressionActionsPtr AnalyzedJoin::createJoinedBlockActions(
     const JoinedColumnsList & columns_added_by_join,
     const ASTSelectQuery * select_query_with_join,
-    const Context & context,
-    NameSet & required_columns_from_joined_table) const
+    const Context & context) const
 {
     if (!select_query_with_join)
         return nullptr;
@@ -48,8 +47,14 @@ ExpressionActionsPtr AnalyzedJoin::createJoinedBlockActions(
 
     ASTPtr query = expression_list;
     auto syntax_result = SyntaxAnalyzer(context).analyze(query, source_column_names, required_columns);
-    ExpressionAnalyzer analyzer(query, syntax_result, context, {}, required_columns);
-    auto joined_block_actions = analyzer.getActions(false);
+    ExpressionAnalyzer analyzer(query, syntax_result, context, {}, required_columns_set);
+    return analyzer.getActions(false);
+}
+
+NameSet AnalyzedJoin::getRequiredColumnsFromJoinedTable(const JoinedColumnsList & columns_added_by_join,
+                                                        const ExpressionActionsPtr & joined_block_actions) const
+{
+    NameSet required_columns_from_joined_table;
 
     auto required_action_columns = joined_block_actions->getRequiredColumns();
     required_columns_from_joined_table.insert(required_action_columns.begin(), required_action_columns.end());
@@ -63,7 +68,7 @@ ExpressionActionsPtr AnalyzedJoin::createJoinedBlockActions(
         if (!sample.has(column.name_and_type.name))
             required_columns_from_joined_table.insert(column.name_and_type.name);
 
-    return joined_block_actions;
+    return required_columns_from_joined_table;
 }
 
 const JoinedColumnsList & AnalyzedJoin::getColumnsFromJoinedTable(

dbms/src/Interpreters/AnalyzedJoin.h

@@ -64,9 +64,11 @@ struct AnalyzedJoin
     ExpressionActionsPtr createJoinedBlockActions(
         const JoinedColumnsList & columns_added_by_join, /// Subset of available_joined_columns.
         const ASTSelectQuery * select_query_with_join,
-        const Context & context,
-        NameSet & required_columns_from_joined_table /// Columns which will be used in query from joined table.
-    ) const;
+        const Context & context) const;
+
+    /// Columns which will be used in query from joined table.
+    NameSet getRequiredColumnsFromJoinedTable(const JoinedColumnsList & columns_added_by_join,
+                                              const ExpressionActionsPtr & joined_block_actions) const;
 
     const JoinedColumnsList & getColumnsFromJoinedTable(const NameSet & source_columns,
                                                         const Context & context,

dbms/src/Interpreters/Context.cpp

@@ -1,8 +1,8 @@
 #include <map>
 #include <set>
-#include <boost/functional/hash/hash.hpp>
+#include <optional>
+#include <memory>
 #include <Poco/Mutex.h>
-#include <Poco/File.h>
 #include <Poco/UUID.h>
 #include <Poco/Net/IPAddress.h>
 #include <common/logger_useful.h>
@@ -98,7 +98,7 @@ struct ContextShared
 {
     Logger * log = &Logger::get("Context");
 
-    std::shared_ptr<IRuntimeComponentsFactory> runtime_components_factory;
+    std::unique_ptr<IRuntimeComponentsFactory> runtime_components_factory;
 
     /// For access of most of shared objects. Recursive mutex.
     mutable std::recursive_mutex mutex;
@@ -124,12 +124,12 @@ struct ContextShared
     ConfigurationPtr config;                                /// Global configuration settings.
 
     Databases databases;                                    /// List of databases and tables in them.
-    mutable std::shared_ptr<EmbeddedDictionaries> embedded_dictionaries;    /// Metrica's dictionaries. Have lazy initialization.
-    mutable std::shared_ptr<ExternalDictionaries> external_dictionaries;
-    mutable std::shared_ptr<ExternalModels> external_models;
+    mutable std::optional<EmbeddedDictionaries> embedded_dictionaries;    /// Metrica's dictionaries. Have lazy initialization.
+    mutable std::optional<ExternalDictionaries> external_dictionaries;
+    mutable std::optional<ExternalModels> external_models;
     String default_profile_name;                            /// Default profile name used for default values.
     String system_profile_name;                             /// Profile used by system processes
-    std::shared_ptr<ISecurityManager> security_manager;     /// Known users.
+    std::unique_ptr<ISecurityManager> security_manager;     /// Known users.
     Quotas quotas;                                          /// Known quotas for resource use.
     mutable UncompressedCachePtr uncompressed_cache;        /// The cache of decompressed blocks.
     mutable MarkCachePtr mark_cache;                        /// Cache of marks in compressed files.
@@ -138,18 +138,19 @@ struct ContextShared
     ViewDependencies view_dependencies;                     /// Current dependencies
     ConfigurationPtr users_config;                          /// Config with the users, profiles and quotas sections.
     InterserverIOHandler interserver_io_handler;            /// Handler for interserver communication.
-    BackgroundProcessingPoolPtr background_pool;            /// The thread pool for the background work performed by the tables.
-    BackgroundSchedulePoolPtr schedule_pool;                /// A thread pool that can run different jobs in background (used in replicated tables)
+    std::optional<BackgroundProcessingPool> background_pool; /// The thread pool for the background work performed by the tables.
+    std::optional<BackgroundSchedulePool> schedule_pool;    /// A thread pool that can run different jobs in background (used in replicated tables)
     MultiVersion<Macros> macros;                            /// Substitutions extracted from config.
-    std::unique_ptr<Compiler> compiler;                     /// Used for dynamic compilation of queries' parts if it necessary.
+    std::optional<Compiler> compiler;                     /// Used for dynamic compilation of queries' parts if it necessary.
     std::shared_ptr<DDLWorker> ddl_worker;                  /// Process ddl commands from zk.
     /// Rules for selecting the compression settings, depending on the size of the part.
     mutable std::unique_ptr<CompressionCodecSelector> compression_codec_selector;
-    std::unique_ptr<MergeTreeSettings> merge_tree_settings; /// Settings of MergeTree* engines.
+    std::optional<MergeTreeSettings> merge_tree_settings; /// Settings of MergeTree* engines.
     size_t max_table_size_to_drop = 50000000000lu;          /// Protects MergeTree tables from accidental DROP (50GB by default)
     size_t max_partition_size_to_drop = 50000000000lu;      /// Protects MergeTree partitions from accidental DROP (50GB by default)
     String format_schema_path;                              /// Path to a directory that contains schema files used by input formats.
     ActionLocksManagerPtr action_locks_manager;             /// Set of storages' action lockers
+    SystemLogsPtr system_logs;                              /// Used to log queries and operations on parts
 
     /// Named sessions. The user could specify session identifier to reuse settings and temporary tables in subsequent requests.
 
@@ -206,7 +207,7 @@ struct ContextShared
 
     Context::ConfigReloadCallback config_reload_callback;
 
-    ContextShared(std::shared_ptr<IRuntimeComponentsFactory> runtime_components_factory_)
+    ContextShared(std::unique_ptr<IRuntimeComponentsFactory> runtime_components_factory_)
         : runtime_components_factory(std::move(runtime_components_factory_)), macros(std::make_unique<Macros>())
     {
         /// TODO: make it singleton (?)
@@ -243,6 +244,8 @@ struct ContextShared
             return;
         shutdown_called = true;
 
+        system_logs.reset();
+
         /** At this point, some tables may have threads that block our mutex.
           * To complete them correctly, we will copy the current list of tables,
           *  and ask them all to finish their work.
@@ -263,6 +266,15 @@ struct ContextShared
             std::lock_guard lock(mutex);
             databases.clear();
         }
+
+        /// Preemptive destruction is important, because these objects may have a refcount to ContextShared (cyclic reference).
+        /// TODO: Get rid of this.
+
+        embedded_dictionaries.reset();
+        external_dictionaries.reset();
+        external_models.reset();
+        background_pool.reset();
+        schedule_pool.reset();
     }
 
 private:
@@ -276,11 +288,10 @@ private:
 Context::Context() = default;
 
 
-Context Context::createGlobal(std::shared_ptr<IRuntimeComponentsFactory> runtime_components_factory)
+Context Context::createGlobal(std::unique_ptr<IRuntimeComponentsFactory> runtime_components_factory)
 {
     Context res;
-    res.runtime_components_factory = runtime_components_factory;
-    res.shared = std::make_shared<ContextShared>(runtime_components_factory);
+    res.shared = std::make_shared<ContextShared>(std::move(runtime_components_factory));
     res.quota = std::make_shared<QuotaForIntervals>();
     return res;
 }
@@ -290,18 +301,7 @@ Context Context::createGlobal()
     return createGlobal(std::make_unique<RuntimeComponentsFactory>());
 }
 
-Context::~Context()
-{
-    try
-    {
-        /// Destroy system logs while at least one Context is alive
-        system_logs.reset();
-    }
-    catch (...)
-    {
-        tryLogCurrentException(__PRETTY_FUNCTION__);
-    }
-}
+Context::~Context() = default;
 
 
 InterserverIOHandler & Context::getInterserverIOHandler() { return shared->interserver_io_handler; }
@@ -1077,6 +1077,13 @@ void Context::setCurrentQueryId(const String & query_id)
     client_info.current_query_id = query_id_to_set;
 }
 
+void Context::killCurrentQuery()
+{
+    if (process_list_elem)
+    {
+        process_list_elem->cancelQuery(true);
+    }
+};
 
 String Context::getDefaultFormat() const
 {
@@ -1181,9 +1188,9 @@ EmbeddedDictionaries & Context::getEmbeddedDictionariesImpl(const bool throw_on_
 
     if (!shared->embedded_dictionaries)
     {
-        auto geo_dictionaries_loader = runtime_components_factory->createGeoDictionariesLoader();
+        auto geo_dictionaries_loader = shared->runtime_components_factory->createGeoDictionariesLoader();
 
-        shared->embedded_dictionaries = std::make_shared<EmbeddedDictionaries>(
+        shared->embedded_dictionaries.emplace(
             std::move(geo_dictionaries_loader),
             *this->global_context,
             throw_on_error);
@@ -1202,9 +1209,9 @@ ExternalDictionaries & Context::getExternalDictionariesImpl(const bool throw_on_
         if (!this->global_context)
             throw Exception("Logical error: there is no global context", ErrorCodes::LOGICAL_ERROR);
 
-        auto config_repository = runtime_components_factory->createExternalDictionariesConfigRepository();
+        auto config_repository = shared->runtime_components_factory->createExternalDictionariesConfigRepository();
 
-        shared->external_dictionaries = std::make_shared<ExternalDictionaries>(
+        shared->external_dictionaries.emplace(
             std::move(config_repository),
             *this->global_context,
             throw_on_error);
@@ -1222,9 +1229,9 @@ ExternalModels & Context::getExternalModelsImpl(bool throw_on_error) const
         if (!this->global_context)
             throw Exception("Logical error: there is no global context", ErrorCodes::LOGICAL_ERROR);
 
-        auto config_repository = runtime_components_factory->createExternalModelsConfigRepository();
+        auto config_repository = shared->runtime_components_factory->createExternalModelsConfigRepository();
 
-        shared->external_models = std::make_shared<ExternalModels>(
+        shared->external_models.emplace(
             std::move(config_repository),
             *this->global_context,
             throw_on_error);
@@ -1342,7 +1349,7 @@ BackgroundProcessingPool & Context::getBackgroundPool()
 {
     auto lock = getLock();
     if (!shared->background_pool)
-        shared->background_pool = std::make_shared<BackgroundProcessingPool>(settings.background_pool_size);
+        shared->background_pool.emplace(settings.background_pool_size);
     return *shared->background_pool;
 }
 
@@ -1350,7 +1357,7 @@ BackgroundSchedulePool & Context::getSchedulePool()
 {
     auto lock = getLock();
     if (!shared->schedule_pool)
-        shared->schedule_pool = std::make_shared<BackgroundSchedulePool>(settings.background_schedule_pool_size);
+        shared->schedule_pool.emplace(settings.background_schedule_pool_size);
     return *shared->schedule_pool;
 }
 
@@ -1529,7 +1536,7 @@ Compiler & Context::getCompiler()
     auto lock = getLock();
 
     if (!shared->compiler)
-        shared->compiler = std::make_unique<Compiler>(shared->path + "build/", 1);
+        shared->compiler.emplace(shared->path + "build/", 1);
 
     return *shared->compiler;
 }
@@ -1542,7 +1549,7 @@ void Context::initializeSystemLogs()
     if (!global_context)
         throw Exception("Logical error: no global context for system logs", ErrorCodes::LOGICAL_ERROR);
 
-    system_logs = std::make_shared<SystemLogs>(*global_context, getConfigRef());
+    shared->system_logs = std::make_shared<SystemLogs>(*global_context, getConfigRef());
 }
 
 
@@ -1550,10 +1557,10 @@ QueryLog * Context::getQueryLog()
 {
     auto lock = getLock();
 
-    if (!system_logs || !system_logs->query_log)
+    if (!shared->system_logs || !shared->system_logs->query_log)
         return nullptr;
 
-    return system_logs->query_log.get();
+    return shared->system_logs->query_log.get();
 }
 
 
@@ -1561,10 +1568,10 @@ QueryThreadLog * Context::getQueryThreadLog()
 {
     auto lock = getLock();
 
-    if (!system_logs || !system_logs->query_thread_log)
+    if (!shared->system_logs || !shared->system_logs->query_thread_log)
         return nullptr;
 
-    return system_logs->query_thread_log.get();
+    return shared->system_logs->query_thread_log.get();
 }
 
 
@@ -1573,16 +1580,16 @@ PartLog * Context::getPartLog(const String & part_database)
     auto lock = getLock();
 
     /// System logs are shutting down.
-    if (!system_logs || !system_logs->part_log)
+    if (!shared->system_logs || !shared->system_logs->part_log)
         return nullptr;
 
     /// Will not log operations on system tables (including part_log itself).
     /// It doesn't make sense and not allow to destruct PartLog correctly due to infinite logging and flushing,
     /// and also make troubles on startup.
-    if (part_database == system_logs->part_log_database)
+    if (part_database == shared->system_logs->part_log_database)
         return nullptr;
 
-    return system_logs->part_log.get();
+    return shared->system_logs->part_log.get();
 }
 
 
@@ -1612,7 +1619,7 @@ const MergeTreeSettings & Context::getMergeTreeSettings() const
     if (!shared->merge_tree_settings)
     {
         auto & config = getConfigRef();
-        shared->merge_tree_settings = std::make_unique<MergeTreeSettings>();
+        shared->merge_tree_settings.emplace();
         shared->merge_tree_settings->loadFromConfig("merge_tree", config);
     }
 
@@ -1727,7 +1734,6 @@ void Context::reloadConfig() const
 
 void Context::shutdown()
 {
-    system_logs.reset();
     shared->shutdown();
 }
 

dbms/src/Interpreters/Context.h

@@ -113,8 +113,6 @@ private:
     using Shared = std::shared_ptr<ContextShared>;
     Shared shared;
 
-    std::shared_ptr<IRuntimeComponentsFactory> runtime_components_factory;
-
     ClientInfo client_info;
     ExternalTablesInitializer external_tables_initializer_callback;
 
@@ -133,7 +131,6 @@ private:
     Context * query_context = nullptr;
     Context * session_context = nullptr;    /// Session context or nullptr. Could be equal to this.
     Context * global_context = nullptr;     /// Global context or nullptr. Could be equal to this.
-    SystemLogsPtr system_logs;              /// Used to log queries and operations on parts
 
     UInt64 session_close_cycle = 0;
     bool session_is_used = false;
@@ -149,7 +146,7 @@ private:
 
 public:
     /// Create initial Context with ContextShared and etc.
-    static Context createGlobal(std::shared_ptr<IRuntimeComponentsFactory> runtime_components_factory);
+    static Context createGlobal(std::unique_ptr<IRuntimeComponentsFactory> runtime_components_factory);
     static Context createGlobal();
 
     Context(const Context &) = default;
@@ -236,6 +233,8 @@ public:
     void setCurrentDatabase(const String & name);
     void setCurrentQueryId(const String & query_id);
 
+    void killCurrentQuery();
+
     void setInsertionTable(std::pair<String, String> && db_and_table) { insertion_table = db_and_table; }
     const std::pair<String, String> & getInsertionTable() const { return insertion_table; }
 

dbms/src/Interpreters/CrossToInnerJoinVisitor.cpp

@@ -0,0 +1,225 @@
+#include <Common/typeid_cast.h>
+#include <Interpreters/CrossToInnerJoinVisitor.h>
+#include <Interpreters/DatabaseAndTableWithAlias.h>
+#include <Interpreters/IdentifierSemantic.h>
+#include <Parsers/ASTSelectQuery.h>
+#include <Parsers/ASTTablesInSelectQuery.h>
+#include <Parsers/ASTIdentifier.h>
+#include <Parsers/ASTFunction.h>
+#include <Parsers/ASTExpressionList.h>
+#include <Parsers/ParserTablesInSelectQuery.h>
+#include <Parsers/ExpressionListParsers.h>
+#include <Parsers/parseQuery.h>
+#include <IO/WriteHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+    extern const int LOGICAL_ERROR;
+}
+
+/// It checks if where expression could be moved to JOIN ON expression partially or entirely.
+class CheckExpressionVisitorData
+{
+public:
+    using TypeToVisit = const ASTFunction;
+
+    CheckExpressionVisitorData(const std::vector<DatabaseAndTableWithAlias> & tables_)
+        : tables(tables_)
+        , save_where(false)
+        , flat_ands(true)
+    {}
+
+    void visit(const ASTFunction & node, ASTPtr & ast)
+    {
+        if (node.name == "and")
+        {
+            if (!node.arguments || node.arguments->children.empty())
+                throw Exception("Logical error: function requires argiment", ErrorCodes::LOGICAL_ERROR);
+
+            for (auto & child : node.arguments->children)
+            {
+                if (auto func = typeid_cast<const ASTFunction *>(child.get()))
+                {
+                    if (func->name == "and")
+                        flat_ands = false;
+                    visit(*func, child);
+                }
+                else
+                    save_where = true;
+            }
+        }
+        else if (node.name == "equals")
+        {
+            if (checkEquals(node))
+                asts_to_join_on.push_back(ast);
+            else
+                save_where = true;
+        }
+        else
+            save_where = true;
+    }
+
+    bool matchAny() const { return !asts_to_join_on.empty(); }
+    bool matchAll() const { return matchAny() && !save_where; }
+    bool canReuseWhere() const { return matchAll() && flat_ands; }
+
+    ASTPtr makeOnExpression()
+    {
+        if (asts_to_join_on.size() == 1)
+            return asts_to_join_on[0]->clone();
+
+        std::vector<ASTPtr> arguments;
+        arguments.reserve(asts_to_join_on.size());
+        for (auto & ast : asts_to_join_on)
+            arguments.emplace_back(ast->clone());
+
+        return makeASTFunction("and", std::move(arguments));
+    }
+
+private:
+    const std::vector<DatabaseAndTableWithAlias> & tables;
+    std::vector<ASTPtr> asts_to_join_on;
+    bool save_where;
+    bool flat_ands;
+
+    bool checkEquals(const ASTFunction & node)
+    {
+        if (!node.arguments)
+            throw Exception("Logical error: function requires argiment", ErrorCodes::LOGICAL_ERROR);
+        if (node.arguments->children.size() != 2)
+            return false;
+
+        auto left = typeid_cast<const ASTIdentifier *>(node.arguments->children[0].get());
+        auto right = typeid_cast<const ASTIdentifier *>(node.arguments->children[1].get());
+        if (!left || !right)
+            return false;
+
+        return checkIdentifiers(*left, *right);
+    }
+
+    /// Check if the identifiers are from different joined tables. If it's a self joint, tables should have aliases.
+    /// select * from t1 a cross join t2 b where a.x = b.x
+    bool checkIdentifiers(const ASTIdentifier & left, const ASTIdentifier & right)
+    {
+        /// {best_match, berst_table_pos}
+        std::pair<size_t, size_t> left_best{0, 0};
+        std::pair<size_t, size_t> right_best{0, 0};
+
+        for (size_t i = 0; i < tables.size(); ++i)
+        {
+            size_t match = IdentifierSemantic::canReferColumnToTable(left, tables[i]);
+            if (match > left_best.first)
+            {
+                left_best.first = match;
+                left_best.second = i;
+            }
+
+            match = IdentifierSemantic::canReferColumnToTable(right, tables[i]);
+            if (match > right_best.first)
+            {
+                right_best.first = match;
+                right_best.second = i;
+            }
+        }
+
+        return left_best.first && right_best.first && (left_best.second != right_best.second);
+    }
+};
+
+
+static bool extractTableName(const ASTTableExpression & expr, std::vector<DatabaseAndTableWithAlias> & names)
+{
+    /// Subselects are not supported.
+    if (!expr.database_and_table_name)
+        return false;
+
+    names.emplace_back(DatabaseAndTableWithAlias(expr));
+    return true;
+}
+
+
+static ASTPtr getCrossJoin(ASTSelectQuery & select, std::vector<DatabaseAndTableWithAlias> & table_names)
+{
+    if (!select.tables)
+        return {};
+
+    auto tables = typeid_cast<const ASTTablesInSelectQuery *>(select.tables.get());
+    if (!tables)
+        return {};
+
+    size_t num_tables = tables->children.size();
+    if (num_tables != 2)
+        return {};
+
+    auto left = typeid_cast<const ASTTablesInSelectQueryElement *>(tables->children[0].get());
+    auto right = typeid_cast<const ASTTablesInSelectQueryElement *>(tables->children[1].get());
+    if (!left || !right || !right->table_join)
+        return {};
+
+    if (auto join = typeid_cast<const ASTTableJoin *>(right->table_join.get()))
+    {
+        if (join->kind == ASTTableJoin::Kind::Cross ||
+            join->kind == ASTTableJoin::Kind::Comma)
+        {
+            if (!join->children.empty())
+                throw Exception("Logical error: CROSS JOIN has expressions", ErrorCodes::LOGICAL_ERROR);
+
+            auto & left_expr = typeid_cast<const ASTTableExpression &>(*left->table_expression);
+            auto & right_expr = typeid_cast<const ASTTableExpression &>(*right->table_expression);
+
+            table_names.reserve(2);
+            if (extractTableName(left_expr, table_names) &&
+                extractTableName(right_expr, table_names))
+                return right->table_join;
+        }
+    }
+
+    return {};
+}
+
+
+std::vector<ASTPtr *> CrossToInnerJoinMatcher::visit(ASTPtr & ast, Data & data)
+{
+    if (auto * t = typeid_cast<ASTSelectQuery *>(ast.get()))
+        visit(*t, ast, data);
+    return {};
+}
+
+void CrossToInnerJoinMatcher::visit(ASTSelectQuery & select, ASTPtr & ast, Data & data)
+{
+    using CheckExpressionMatcher = OneTypeMatcher<CheckExpressionVisitorData, false>;
+    using CheckExpressionVisitor = InDepthNodeVisitor<CheckExpressionMatcher, true>;
+
+    std::vector<DatabaseAndTableWithAlias> table_names;
+    ASTPtr ast_join = getCrossJoin(select, table_names);
+    if (!ast_join)
+        return;
+
+    CheckExpressionVisitor::Data visitor_data{table_names};
+    CheckExpressionVisitor(visitor_data).visit(select.where_expression);
+
+    if (visitor_data.matchAny())
+    {
+        auto & join = typeid_cast<ASTTableJoin &>(*ast_join);
+        join.kind = ASTTableJoin::Kind::Inner;
+        join.strictness = ASTTableJoin::Strictness::All;
+
+        if (visitor_data.canReuseWhere())
+            join.on_expression.swap(select.where_expression);
+        else
+            join.on_expression = visitor_data.makeOnExpression();
+
+        if (visitor_data.matchAll())
+            select.where_expression.reset();
+
+        join.children.push_back(join.on_expression);
+    }
+
+    ast = ast->clone(); /// rewrite AST in right manner
+    data.done = true;
+}
+
+}

dbms/src/Interpreters/CrossToInnerJoinVisitor.h

@@ -0,0 +1,30 @@
+#pragma once
+
+#include <Interpreters/InDepthNodeVisitor.h>
+
+namespace DB
+{
+
+class ASTSelectQuery;
+
+/// AST transformer. It replaces cross joins with equivalented inner join if possible.
+class CrossToInnerJoinMatcher
+{
+public:
+    struct Data
+    {
+        bool done = false;
+    };
+
+    static constexpr const char * label = "JoinToSubqueryTransform";
+
+    static bool needChildVisit(ASTPtr &, const ASTPtr &) { return true; }
+    static std::vector<ASTPtr *> visit(ASTPtr & ast, Data & data);
+
+private:
+    static void visit(ASTSelectQuery & select, ASTPtr & ast, Data & data);
+};
+
+using CrossToInnerJoinVisitor = InDepthNodeVisitor<CrossToInnerJoinMatcher, true>;
+
+}

dbms/src/Interpreters/DDLWorker.cpp

@@ -528,7 +528,7 @@ bool DDLWorker::tryExecuteQuery(const String & query, const DDLTask & task, Exec
     {
         current_context = std::make_unique<Context>(context);
         current_context->setCurrentQueryId(""); // generate random query_id
-        executeQuery(istr, ostr, false, *current_context, nullptr);
+        executeQuery(istr, ostr, false, *current_context, {}, {});
     }
     catch (...)
     {

dbms/src/Interpreters/DatabaseAndTableWithAlias.h

@@ -27,7 +27,7 @@ struct DatabaseAndTableWithAlias
     DatabaseAndTableWithAlias() = default;
     DatabaseAndTableWithAlias(const ASTPtr & identifier_node, const String & current_database = "");
     DatabaseAndTableWithAlias(const ASTIdentifier & identifier, const String & current_database = "");
-    DatabaseAndTableWithAlias(const ASTTableExpression & table_expression, const String & current_database);
+    DatabaseAndTableWithAlias(const ASTTableExpression & table_expression, const String & current_database = "");
 
     /// "alias." or "table." if alias is empty
     String getQualifiedNamePrefix() const;

dbms/src/Interpreters/ExpressionActions.cpp

@@ -160,15 +160,13 @@ ExpressionAction ExpressionAction::arrayJoin(const NameSet & array_joined_column
 ExpressionAction ExpressionAction::ordinaryJoin(
     std::shared_ptr<const Join> join_,
     const Names & join_key_names_left,
-    const NamesAndTypesList & columns_added_by_join_,
-    const NameSet & columns_added_by_join_from_right_keys_)
+    const NamesAndTypesList & columns_added_by_join_)
 {
     ExpressionAction a;
     a.type = JOIN;
     a.join = std::move(join_);
     a.join_key_names_left = join_key_names_left;
     a.columns_added_by_join = columns_added_by_join_;
-    a.columns_added_by_join_from_right_keys = columns_added_by_join_from_right_keys_;
     return a;
 }
 
@@ -463,7 +461,7 @@ void ExpressionAction::execute(Block & block, bool dry_run) const
 
         case JOIN:
         {
-            join->joinBlock(block, join_key_names_left, columns_added_by_join_from_right_keys);
+            join->joinBlock(block, join_key_names_left, columns_added_by_join);
             break;
         }
 
@@ -1115,7 +1113,8 @@ BlockInputStreamPtr ExpressionActions::createStreamWithNonJoinedDataIfFullOrRigh
 {
     for (const auto & action : actions)
         if (action.join && (action.join->getKind() == ASTTableJoin::Kind::Full || action.join->getKind() == ASTTableJoin::Kind::Right))
-            return action.join->createStreamWithNonJoinedRows(source_header, action.join_key_names_left, max_block_size);
+            return action.join->createStreamWithNonJoinedRows(
+                source_header, action.join_key_names_left, action.columns_added_by_join, max_block_size);
 
     return {};
 }