mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-09-20 08:40:50 +00:00
Merge branch 'master' of github.com:yandex/ClickHouse into DOCAPI-4197-limitations-settings
This commit is contained in:
BayoNet
commit
9230721cdb
63
CHANGELOG.md
63
CHANGELOG.md
@ -1,3 +1,65 @@
|
||||
## ClickHouse release 19.7.3.9, 2019-05-30
|
||||
|
||||
### New Features
|
||||
* Allow to limit the range of a setting that can be specified by user.
|
||||
These constraints can be set up in user settings profile.
|
||||
[#4931](https://github.com/yandex/ClickHouse/pull/4931) ([Vitaly
|
||||
Baranov](https://github.com/vitlibar))
|
||||
* Add a second version of the function `groupUniqArray` with an optional
|
||||
`max_size` parameter that limits the size of the resulting array. This
|
||||
behavior is similar to `groupArray(max_size)(x)` function.
|
||||
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
|
||||
Tassery](https://github.com/YiuRULE))
|
||||
* For TSVWithNames/CSVWithNames input file formats, column order can now be
|
||||
determined from file header. This is controlled by
|
||||
`input_format_with_names_use_header` parameter.
|
||||
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
|
||||
([Alexander](https://github.com/Akazz))
|
||||
|
||||
### Bug Fixes
|
||||
* Crash with uncompressed_cache + JOIN during merge (#5197)
|
||||
[#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Segmentation fault on a clickhouse-client query to system tables. #5066
|
||||
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Data loss on heavy load via KafkaEngine (#4736)
|
||||
[#5080](https://github.com/yandex/ClickHouse/pull/5080)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Performance Improvements
|
||||
* Use radix sort for sorting by single numeric column in `ORDER BY` without
|
||||
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
|
||||
[#4439](https://github.com/yandex/ClickHouse/pull/4439)
|
||||
([Evgenii Pravda](https://github.com/kvinty),
|
||||
[alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Documentation
|
||||
* Translate documentation for some table engines to Chinese.
|
||||
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
|
||||
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
|
||||
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
|
||||
([张风啸](https://github.com/AlexZFX)),
|
||||
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
|
||||
lee](https://github.com/neverlee))
|
||||
|
||||
### Build/Testing/Packaging Improvements
|
||||
* Print UTF-8 characters properly in `clickhouse-test`.
|
||||
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Add command line parameter for clickhouse-client to always load suggestion
|
||||
data. [#5102](https://github.com/yandex/ClickHouse/pull/5102)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Resolve some of PVS-Studio warnings.
|
||||
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Update LZ4 [#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Add gperf to build requirements for upcoming pull request #5030.
|
||||
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
|
||||
([proller](https://github.com/proller))
|
||||
|
||||
## ClickHouse release 19.6.2.11, 2019-05-13
|
||||
|
||||
### New Features
|
||||
@ -29,6 +91,7 @@
|
||||
* Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Partially fix distributed_product_mode = local. It's possible to allow columns of local tables in where/having/order by/... via table aliases. Throw exception if table does not have alias. There's not possible to access to the columns without table aliases yet. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Build/Testing/Packaging Improvements
|
||||
* Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
|
||||
@ -1,3 +1,68 @@
|
||||
## ClickHouse release 19.7.3.9, 2019-05-30
|
||||
|
||||
### Новые возможности
|
||||
* Добавлена возможность ограничить значения конфигурационных параметров,
|
||||
которые может задать пользователь. Эти ограничения устанавливаются в профиле
|
||||
настроек пользователя. [#4931](https://github.com/yandex/ClickHouse/pull/4931)
|
||||
([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Добавлен вариант функции `groupUniqArray` с дополнительным параметром
|
||||
`max_size`, который ограничивает размер результирующего массива, аналогично
|
||||
функции `groupArray(max_size)(x)`.
|
||||
[#5026](https://github.com/yandex/ClickHouse/pull/5026) ([Guillaume
|
||||
Tassery](https://github.com/YiuRULE))
|
||||
* Для входных файлов формата TSVWithNames и CSVWithNames появилась возможность
|
||||
определить порядок колонок в файле исходя из его заголовка. Это поведение
|
||||
управляется конфигурационным параметром `input_format_with_names_use_header`.
|
||||
[#5081](https://github.com/yandex/ClickHouse/pull/5081)
|
||||
([Alexander](https://github.com/Akazz))
|
||||
|
||||
### Исправления ошибок
|
||||
* Падение в процессе слияния при использовании uncompressed_cache и JOIN
|
||||
(#5197). [#5133](https://github.com/yandex/ClickHouse/pull/5133) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* Segmentation fault на запросе к системным таблицам (#5066).
|
||||
[#5127](https://github.com/yandex/ClickHouse/pull/5127)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Потеря загружаемых данных при больших потоках загрузки через KafkaEngine
|
||||
(#4736). [#5080](https://github.com/yandex/ClickHouse/pull/5080)
|
||||
([Ivan](https://github.com/abyss7))
|
||||
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Улучшения производительности
|
||||
* Используется поразрядная сортировка числовых колонок для `ORDER BY` без
|
||||
`LIMIT`. [#5106](https://github.com/yandex/ClickHouse/pull/5106),
|
||||
[#4439](https://github.com/yandex/ClickHouse/pull/4439) ([Evgenii
|
||||
Pravda](https://github.com/kvinty),
|
||||
[alexey-milovidov](https://github.com/alexey-milovidov)
|
||||
|
||||
### Документация
|
||||
* Документация для некоторых табличных движков переведена на китайский.
|
||||
[#5107](https://github.com/yandex/ClickHouse/pull/5107),
|
||||
[#5094](https://github.com/yandex/ClickHouse/pull/5094),
|
||||
[#5087](https://github.com/yandex/ClickHouse/pull/5087)
|
||||
([张风啸](https://github.com/AlexZFX),
|
||||
[#5068](https://github.com/yandex/ClickHouse/pull/5068) ([never
|
||||
lee](https://github.com/neverlee))
|
||||
|
||||
### Улучшения сборки, тестирования и пакетирования
|
||||
* Правильно отображаются символы в кодировке UTF-8 в `clickhouse-test`.
|
||||
[#5084](https://github.com/yandex/ClickHouse/pull/5084)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Добавлен параметр командной строки для `clickhouse-client`, позволяющий
|
||||
всегда загружать данные подсказок.
|
||||
[#5102](https://github.com/yandex/ClickHouse/pull/5102)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлены некоторые предупреждения PVS-Studio.
|
||||
[#5082](https://github.com/yandex/ClickHouse/pull/5082)
|
||||
([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Обновлена библиотека LZ4.
|
||||
[#5040](https://github.com/yandex/ClickHouse/pull/5040) ([Danila
|
||||
Kutenin](https://github.com/danlark1))
|
||||
* В зависимости сборки добавлен gperf для поддержки готовящегося PR #5030.
|
||||
[#5110](https://github.com/yandex/ClickHouse/pull/5110)
|
||||
([proller](https://github.com/proller))
|
||||
|
||||
|
||||
## ClickHouse release 19.6.2.11, 2019-05-13
|
||||
|
||||
### Новые возможности
|
||||
@ -6,7 +71,7 @@
|
||||
* Добавлена функция `isValidUTF8` для проверки, содержит ли строка валидные данные в кодировке UTF-8. [#4934](https://github.com/yandex/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Добавлены новое правило балансировки (`load_balancing`) `first_or_random` по которому запросы посылаются на первый заданый хост и если он недоступен - на случайные хосты шарда. Полезно для топологий с кросс-репликацией. [#5012](https://github.com/yandex/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei))
|
||||
|
||||
### Эксперемннтальные возможности
|
||||
### Экспериментальные возможности
|
||||
* Добавлена настройка `index_granularity_bytes` (адаптивная гранулярность индекса) для таблиц семейства MergeTree* . [#4826](https://github.com/yandex/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin))
|
||||
|
||||
### Улучшения
|
||||
@ -29,6 +94,7 @@
|
||||
* Исправлено зависание на старте сервера если внешний словарь зависит от другого словаря через использование таблицы из БД с движком `Dictionary`. [#4962](https://github.com/yandex/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* При использовании `distributed_product_mode = 'local'` корректно работает использование столбцов локальных таблиц в where/having/order by/... через табличные алиасы. Выкидывает исключение если таблица не имеет алиас. Доступ к столбцам без алиасов пока не возможен. [#4986](https://github.com/yandex/ClickHouse/pull/4986) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлен потенциально некорректный результат для `SELECT DISTINCT` с `JOIN` [#5001](https://github.com/yandex/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлен очень редкий data race condition который мог произойти при выполнении запроса с UNION ALL включающего минимум два SELECT из таблиц system.columns, system.tables, system.parts, system.parts_tables или таблиц семейства Merge и одновременно выполняющихся запросов ALTER столбцов соответствующих таблиц. [#5189](https://github.com/yandex/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
### Улучшения сборки/тестирования/пакетирования
|
||||
* Исправлена неработоспособность тестов, если `clickhouse-server` запущен на удалённом хосте [#4713](https://github.com/yandex/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
|
||||
@ -12,8 +12,6 @@ ClickHouse is an open-source column-oriented database management system that all
|
||||
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
|
||||
|
||||
## Upcoming Events
|
||||
* ClickHouse at [Percona Live 2019](https://www.percona.com/live/19/other-open-source-databases-track) in Austin on May 28-30.
|
||||
* [ClickHouse Community Meetup in San Francisco](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/261110652/) on June 4.
|
||||
* [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8.
|
||||
* [ClickHouse on HighLoad++ Siberia](https://www.highload.ru/siberia/2019/abstracts/5348) on June 24-25.
|
||||
* [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
|
||||
* [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.
|
||||
|
||||
@ -1,9 +1,28 @@
|
||||
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/rapidjson/include/rapidjson/rapidjson.h")
|
||||
message (WARNING "submodule contrib/rapidjson is missing. to fix try run: \n git submodule update --init --recursive")
|
||||
option(ENABLE_RAPIDJSON "Use rapidjson" ON)
|
||||
if(NOT ENABLE_RAPIDJSON)
|
||||
return()
|
||||
endif()
|
||||
|
||||
option (USE_RAPIDJSON "Use rapidjson" ON)
|
||||
option(USE_INTERNAL_RAPIDJSON_LIBRARY "Set to FALSE to use system rapidjson library instead of bundled" ${NOT_UNBUNDLED})
|
||||
|
||||
if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/rapidjson/include/rapidjson/rapidjson.h")
|
||||
if(USE_INTERNAL_RAPIDJSON_LIBRARY)
|
||||
message(WARNING "submodule contrib/rapidjson is missing. to fix try run: \n git submodule update --init --recursive")
|
||||
set(USE_INTERNAL_RAPIDJSON_LIBRARY 0)
|
||||
endif()
|
||||
set(MISSING_INTERNAL_RAPIDJSON_LIBRARY 1)
|
||||
endif()
|
||||
|
||||
if(NOT USE_INTERNAL_RAPIDJSON_LIBRARY)
|
||||
find_path(RAPIDJSON_INCLUDE_DIR NAMES rapidjson/rapidjson.h PATHS ${RAPIDJSON_INCLUDE_PATHS})
|
||||
endif()
|
||||
|
||||
if(RAPIDJSON_INCLUDE_DIR)
|
||||
set(USE_RAPIDJSON 1)
|
||||
elseif(NOT MISSING_INTERNAL_RAPIDJSON_LIBRARY)
|
||||
set(RAPIDJSON_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/rapidjson/include")
|
||||
set(USE_INTERNAL_RAPIDJSON_LIBRARY 1)
|
||||
set(USE_RAPIDJSON 1)
|
||||
endif()
|
||||
|
||||
message(STATUS "Using rapidjson=${USE_RAPIDJSON}: ${RAPIDJSON_INCLUDE_DIR}")
|
||||
|
||||
2
contrib/boost
vendored
2
contrib/boost
vendored
@ -1 +1 @@
|
||||
Subproject commit 8abda007bfe52d78a51548d4594879d6d82a22fa
|
||||
Subproject commit 830e51edb59c4f37a8638138581e1e56c29ac44f
|
||||
@ -1,5 +1,5 @@
|
||||
set(BROTLI_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/brotli/c)
|
||||
set(BROTLI_BINARY_DIR ${CMAKE_BINARY_DIR}/contrib/brotli/c)
|
||||
set(BROTLI_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/brotli/c)
|
||||
set(BROTLI_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/brotli/c)
|
||||
|
||||
set(SRCS
|
||||
${BROTLI_SOURCE_DIR}/dec/bit_reader.c
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
set(CPPKAFKA_DIR ${CMAKE_SOURCE_DIR}/contrib/cppkafka)
|
||||
set(CPPKAFKA_DIR ${ClickHouse_SOURCE_DIR}/contrib/cppkafka)
|
||||
|
||||
set(SRCS
|
||||
${CPPKAFKA_DIR}/src/configuration.cpp
|
||||
|
||||
2
contrib/hyperscan
vendored
2
contrib/hyperscan
vendored
@ -1 +1 @@
|
||||
Subproject commit ed17d34a7c786512471946f9105eaa8d925f34c3
|
||||
Subproject commit 01e6b83f9fbdb4020cd68a5287bf3a0471eeb272
|
||||
@ -1,4 +1,4 @@
|
||||
set(JEMALLOC_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/jemalloc)
|
||||
set(JEMALLOC_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/jemalloc)
|
||||
|
||||
set(SRCS
|
||||
${JEMALLOC_SOURCE_DIR}/src/arena.c
|
||||
|
||||
@ -9,7 +9,7 @@ endif()
|
||||
|
||||
SET(WITH_KERBEROS false)
|
||||
# project and source dir
|
||||
set(HDFS3_ROOT_DIR ${CMAKE_SOURCE_DIR}/contrib/libhdfs3)
|
||||
set(HDFS3_ROOT_DIR ${ClickHouse_SOURCE_DIR}/contrib/libhdfs3)
|
||||
set(HDFS3_SOURCE_DIR ${HDFS3_ROOT_DIR}/src)
|
||||
set(HDFS3_COMMON_DIR ${HDFS3_SOURCE_DIR}/common)
|
||||
|
||||
|
||||
@ -81,7 +81,7 @@
|
||||
#define PCG_128BIT_CONSTANT(high,low) \
|
||||
((pcg128_t(high) << 64) + low)
|
||||
#else
|
||||
#include "pcg_uint128.hpp" // Y_IGNORE
|
||||
#include "pcg_uint128.hpp"
|
||||
namespace pcg_extras {
|
||||
typedef pcg_extras::uint_x4<uint32_t,uint64_t> pcg128_t;
|
||||
}
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
set(RDKAFKA_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/librdkafka/src)
|
||||
set(RDKAFKA_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/librdkafka/src)
|
||||
|
||||
set(SRCS
|
||||
${RDKAFKA_SOURCE_DIR}/crc32c.c
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
#if __has_include(<rdkafka.h>) // maybe bundled
|
||||
# include_next <rdkafka.h> // Y_IGNORE
|
||||
# include_next <rdkafka.h>
|
||||
#else // system
|
||||
# include_next <librdkafka/rdkafka.h>
|
||||
#endif
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(LIBXML2_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/libxml2)
|
||||
set(LIBXML2_BINARY_DIR ${CMAKE_BINARY_DIR}/contrib/libxml2)
|
||||
set(LIBXML2_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libxml2)
|
||||
set(LIBXML2_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/libxml2)
|
||||
|
||||
set(SRCS
|
||||
${LIBXML2_SOURCE_DIR}/SAX.c
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(MARIADB_CLIENT_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/mariadb-connector-c)
|
||||
set(MARIADB_CLIENT_BINARY_DIR ${CMAKE_BINARY_DIR}/contrib/mariadb-connector-c)
|
||||
set(MARIADB_CLIENT_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/mariadb-connector-c)
|
||||
set(MARIADB_CLIENT_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/mariadb-connector-c)
|
||||
|
||||
set(SRCS
|
||||
#${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/bmove_upp.c
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(ODBC_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/unixodbc)
|
||||
set(ODBC_BINARY_DIR ${CMAKE_BINARY_DIR}/contrib/unixodbc)
|
||||
set(ODBC_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/unixodbc)
|
||||
set(ODBC_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/unixodbc)
|
||||
|
||||
|
||||
set(SRCS
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
# This strings autochanged from release_lib.sh:
|
||||
set(VERSION_REVISION 54421)
|
||||
set(VERSION_REVISION 54422)
|
||||
set(VERSION_MAJOR 19)
|
||||
set(VERSION_MINOR 9)
|
||||
set(VERSION_MINOR 10)
|
||||
set(VERSION_PATCH 1)
|
||||
set(VERSION_GITHASH 0c2aa460651a462f14efc7e995840a244531d373)
|
||||
set(VERSION_DESCRIBE v19.9.1.1-testing)
|
||||
set(VERSION_STRING 19.9.1.1)
|
||||
set(VERSION_GITHASH 864a5830f5b5fa67c6b7126e8f2615d03ee890ce)
|
||||
set(VERSION_DESCRIBE v19.10.1.1-prestable)
|
||||
set(VERSION_STRING 19.10.1.1)
|
||||
# end of autochange
|
||||
|
||||
set(VERSION_EXTRA "" CACHE STRING "")
|
||||
|
||||
@ -65,9 +65,10 @@
|
||||
#include <AggregateFunctions/registerAggregateFunctions.h>
|
||||
#include <Common/Config/configReadClient.h>
|
||||
#include <Storages/ColumnsDescription.h>
|
||||
#include <common/argsToConfig.h>
|
||||
|
||||
#if USE_READLINE
|
||||
#include "Suggest.h" // Y_IGNORE
|
||||
#include "Suggest.h"
|
||||
#endif
|
||||
|
||||
#ifndef __clang__
|
||||
@ -1549,7 +1550,7 @@ public:
|
||||
* where possible args are file, name, format, structure, types.
|
||||
* Split these groups before processing.
|
||||
*/
|
||||
using Arguments = std::vector<const char *>;
|
||||
using Arguments = std::vector<std::string>;
|
||||
|
||||
Arguments common_arguments{""}; /// 0th argument is ignored.
|
||||
std::vector<Arguments> external_tables_arguments;
|
||||
@ -1671,8 +1672,7 @@ public:
|
||||
("types", po::value<std::string>(), "types")
|
||||
;
|
||||
/// Parse main commandline options.
|
||||
po::parsed_options parsed = po::command_line_parser(
|
||||
common_arguments.size(), common_arguments.data()).options(main_description).run();
|
||||
po::parsed_options parsed = po::command_line_parser(common_arguments).options(main_description).run();
|
||||
po::variables_map options;
|
||||
po::store(parsed, options);
|
||||
po::notify(options);
|
||||
@ -1705,8 +1705,7 @@ public:
|
||||
for (size_t i = 0; i < external_tables_arguments.size(); ++i)
|
||||
{
|
||||
/// Parse commandline options related to external tables.
|
||||
po::parsed_options parsed_tables = po::command_line_parser(
|
||||
external_tables_arguments[i].size(), external_tables_arguments[i].data()).options(external_description).run();
|
||||
po::parsed_options parsed_tables = po::command_line_parser(external_tables_arguments[i]).options(external_description).run();
|
||||
po::variables_map external_options;
|
||||
po::store(parsed_tables, external_options);
|
||||
|
||||
@ -1802,6 +1801,9 @@ public:
|
||||
}
|
||||
if (options.count("suggestion_limit"))
|
||||
config().setInt("suggestion_limit", options["suggestion_limit"].as<int>());
|
||||
|
||||
argsToConfig(common_arguments, config(), 100);
|
||||
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -16,7 +16,6 @@
|
||||
#include <pcg_random.hpp>
|
||||
#include <common/logger_useful.h>
|
||||
#include <Common/ThreadPool.h>
|
||||
#include <daemon/OwnPatternFormatter.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/ZooKeeper/ZooKeeper.h>
|
||||
#include <Common/ZooKeeper/KeeperException.h>
|
||||
|
||||
@ -34,6 +34,7 @@
|
||||
#include <Dictionaries/registerDictionaries.h>
|
||||
#include <boost/program_options/options_description.hpp>
|
||||
#include <boost/program_options.hpp>
|
||||
#include <common/argsToConfig.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -59,6 +60,23 @@ void LocalServer::initialize(Poco::Util::Application & self)
|
||||
{
|
||||
Poco::Util::Application::initialize(self);
|
||||
|
||||
/// Load config files if exists
|
||||
if (config().has("config-file") || Poco::File("config.xml").exists())
|
||||
{
|
||||
const auto config_path = config().getString("config-file", "config.xml");
|
||||
ConfigProcessor config_processor(config_path, false, true);
|
||||
config_processor.setConfigPath(Poco::Path(config_path).makeParent().toString());
|
||||
auto loaded_config = config_processor.loadConfig();
|
||||
config_processor.savePreprocessedConfig(loaded_config, loaded_config.configuration->getString("path", "."));
|
||||
config().add(loaded_config.configuration.duplicate(), PRIO_DEFAULT, false);
|
||||
}
|
||||
|
||||
if (config().has("logger") || config().has("logger.level") || config().has("logger.log"))
|
||||
{
|
||||
buildLoggers(config(), logger());
|
||||
}
|
||||
else
|
||||
{
|
||||
// Turn off server logging to stderr
|
||||
if (!config().has("verbose"))
|
||||
{
|
||||
@ -66,6 +84,7 @@ void LocalServer::initialize(Poco::Util::Application & self)
|
||||
Poco::Logger::root().setChannel(Poco::AutoPtr<Poco::NullChannel>(new Poco::NullChannel()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void LocalServer::applyCmdSettings()
|
||||
{
|
||||
@ -110,16 +129,6 @@ try
|
||||
return Application::EXIT_OK;
|
||||
}
|
||||
|
||||
/// Load config files if exists
|
||||
if (config().has("config-file") || Poco::File("config.xml").exists())
|
||||
{
|
||||
const auto config_path = config().getString("config-file", "config.xml");
|
||||
ConfigProcessor config_processor(config_path, false, true);
|
||||
config_processor.setConfigPath(Poco::Path(config_path).makeParent().toString());
|
||||
auto loaded_config = config_processor.loadConfig();
|
||||
config_processor.savePreprocessedConfig(loaded_config, loaded_config.configuration->getString("path", DBMS_DEFAULT_PATH));
|
||||
config().add(loaded_config.configuration.duplicate(), PRIO_DEFAULT, false);
|
||||
}
|
||||
|
||||
context = std::make_unique<Context>(Context::createGlobal());
|
||||
context->setGlobalContext(*context);
|
||||
@ -428,6 +437,8 @@ void LocalServer::init(int argc, char ** argv)
|
||||
("stacktrace", "print stack traces of exceptions")
|
||||
("echo", "print query before execution")
|
||||
("verbose", "print query and other debugging info")
|
||||
("logger.log", po::value<std::string>(), "Log file name")
|
||||
("logger.level", po::value<std::string>(), "Log level")
|
||||
("ignore-error", "do not stop processing if a query failed")
|
||||
("version,V", "print version information and exit")
|
||||
;
|
||||
@ -481,8 +492,17 @@ void LocalServer::init(int argc, char ** argv)
|
||||
config().setBool("echo", true);
|
||||
if (options.count("verbose"))
|
||||
config().setBool("verbose", true);
|
||||
if (options.count("logger.log"))
|
||||
config().setString("logger.log", options["logger.log"].as<std::string>());
|
||||
if (options.count("logger.level"))
|
||||
config().setString("logger.level", options["logger.level"].as<std::string>());
|
||||
if (options.count("ignore-error"))
|
||||
config().setBool("ignore-error", true);
|
||||
|
||||
std::vector<std::string> arguments;
|
||||
for (int arg_num = 1; arg_num < argc; ++arg_num)
|
||||
arguments.emplace_back(argv[arg_num]);
|
||||
argsToConfig(arguments, config(), 100);
|
||||
}
|
||||
|
||||
void LocalServer::applyCmdOptions()
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
#include <Core/Settings.h>
|
||||
#include <Poco/Util/Application.h>
|
||||
#include <memory>
|
||||
#include <loggers/Loggers.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -13,7 +14,7 @@ class Context;
|
||||
/// Lightweight Application for clickhouse-local
|
||||
/// No networking, no extra configs and working directories, no pid and status files, no dictionaries, no logging.
|
||||
/// Quiet mode by default
|
||||
class LocalServer : public Poco::Util::Application
|
||||
class LocalServer : public Poco::Util::Application, public Loggers
|
||||
{
|
||||
public:
|
||||
LocalServer();
|
||||
|
||||
@ -15,7 +15,7 @@
|
||||
#endif
|
||||
|
||||
#if USE_TCMALLOC
|
||||
#include <gperftools/malloc_extension.h> // Y_IGNORE
|
||||
#include <gperftools/malloc_extension.h>
|
||||
#endif
|
||||
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
|
||||
@ -3,9 +3,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -2,9 +2,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -123,7 +123,7 @@ void ODBCBridge::initialize(Application & self)
|
||||
|
||||
config().setString("logger", "ODBCBridge");
|
||||
|
||||
buildLoggers(config());
|
||||
buildLoggers(config(), logger());
|
||||
log = &logger();
|
||||
hostname = config().getString("listen-host", "localhost");
|
||||
port = config().getUInt("http-port");
|
||||
|
||||
@ -2,9 +2,9 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/ODBCException.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/ODBCException.h>
|
||||
#include <Poco/SQL/ODBC/SessionImpl.h>
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#define POCO_SQL_ODBC_CLASS Poco::SQL::ODBC
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
|
||||
@ -8,7 +8,7 @@
|
||||
#if USE_POCO_SQLODBC || USE_POCO_DATAODBC
|
||||
|
||||
#if USE_POCO_SQLODBC
|
||||
#include <Poco/SQL/ODBC/Utility.h> // Y_IGNORE
|
||||
#include <Poco/SQL/ODBC/Utility.h>
|
||||
#endif
|
||||
#if USE_POCO_DATAODBC
|
||||
#include <Poco/Data/ODBC/Utility.h>
|
||||
|
||||
@ -310,7 +310,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
/// Initialize DateLUT early, to not interfere with running time of first query.
|
||||
LOG_DEBUG(log, "Initializing DateLUT.");
|
||||
DateLUT::instance();
|
||||
LOG_TRACE(log, "Initialized DateLUT with time zone `" << DateLUT::instance().getTimeZone() << "'.");
|
||||
LOG_TRACE(log, "Initialized DateLUT with time zone '" << DateLUT::instance().getTimeZone() << "'.");
|
||||
|
||||
/// Directory with temporary data for processing of heavy queries.
|
||||
{
|
||||
@ -405,7 +405,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
main_config_zk_changed_event,
|
||||
[&](ConfigurationPtr config)
|
||||
{
|
||||
buildLoggers(*config);
|
||||
buildLoggers(*config, logger());
|
||||
global_context->setClustersConfig(config);
|
||||
global_context->setMacros(std::make_unique<Macros>(*config, "macros"));
|
||||
},
|
||||
|
||||
@ -1,8 +1,6 @@
|
||||
#include <iomanip>
|
||||
#include <ext/scope_guard.h>
|
||||
#include <Poco/Net/NetException.h>
|
||||
#include <daemon/OwnSplitChannel.h>
|
||||
|
||||
#include <Common/ClickHouseRevision.h>
|
||||
#include <Common/CurrentThread.h>
|
||||
#include <Common/Stopwatch.h>
|
||||
|
||||
@ -45,12 +45,12 @@ namespace
|
||||
|
||||
/// Such default parameters were picked because they did good on some tests,
|
||||
/// though it still requires to fit parameters to achieve better result
|
||||
auto learning_rate = Float64(0.00001);
|
||||
auto learning_rate = Float64(0.01);
|
||||
auto l2_reg_coef = Float64(0.1);
|
||||
UInt32 batch_size = 15;
|
||||
|
||||
std::string weights_updater_name = "\'SGD\'";
|
||||
std::shared_ptr<IGradientComputer> gradient_computer;
|
||||
std::string weights_updater_name = "SGD";
|
||||
std::unique_ptr<IGradientComputer> gradient_computer;
|
||||
|
||||
if (!parameters.empty())
|
||||
{
|
||||
@ -66,20 +66,19 @@ namespace
|
||||
}
|
||||
if (parameters.size() > 3)
|
||||
{
|
||||
weights_updater_name = applyVisitor(FieldVisitorToString(), parameters[3]);
|
||||
if (weights_updater_name != "\'SGD\'" && weights_updater_name != "\'Momentum\'" && weights_updater_name != "\'Nesterov\'")
|
||||
{
|
||||
throw Exception("Invalid parameter for weights updater", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
weights_updater_name = parameters[3].safeGet<String>();
|
||||
if (weights_updater_name != "SGD" && weights_updater_name != "Momentum" && weights_updater_name != "Nesterov")
|
||||
throw Exception("Invalid parameter for weights updater. The only supported are 'SGD', 'Momentum' and 'Nesterov'",
|
||||
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
|
||||
if (std::is_same<Method, FuncLinearRegression>::value)
|
||||
{
|
||||
gradient_computer = std::make_shared<LinearRegression>();
|
||||
gradient_computer = std::make_unique<LinearRegression>();
|
||||
}
|
||||
else if (std::is_same<Method, FuncLogisticRegression>::value)
|
||||
{
|
||||
gradient_computer = std::make_shared<LogisticRegression>();
|
||||
gradient_computer = std::make_unique<LogisticRegression>();
|
||||
}
|
||||
else
|
||||
{
|
||||
@ -88,7 +87,7 @@ namespace
|
||||
|
||||
return std::make_shared<Method>(
|
||||
argument_types.size() - 1,
|
||||
gradient_computer,
|
||||
std::move(gradient_computer),
|
||||
weights_updater_name,
|
||||
learning_rate,
|
||||
l2_reg_coef,
|
||||
@ -134,9 +133,14 @@ void LinearModelData::update_state()
|
||||
}
|
||||
|
||||
void LinearModelData::predict(
|
||||
ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const
|
||||
{
|
||||
gradient_computer->predict(container, block, arguments, weights, bias, context);
|
||||
gradient_computer->predict(container, block, offset, limit, arguments, weights, bias, context);
|
||||
}
|
||||
|
||||
void LinearModelData::returnWeights(IColumn & to) const
|
||||
@ -345,43 +349,39 @@ void IWeightsUpdater::add_to_batch(
|
||||
void LogisticRegression::predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
const Context & context) const
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
size_t rows_num = block.rows();
|
||||
std::vector<Float64> results(rows_num, bias);
|
||||
|
||||
if (offset > rows_num || offset + limit > rows_num)
|
||||
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
|
||||
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
|
||||
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
std::vector<Float64> results(limit, bias);
|
||||
|
||||
for (size_t i = 1; i < arguments.size(); ++i)
|
||||
{
|
||||
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
|
||||
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
{
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
auto & features_column = cur_col.column;
|
||||
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
results[row_num] += weights[i - 1] * features_column->getFloat64(offset + row_num);
|
||||
}
|
||||
|
||||
/// If column type is already Float64 then castColumn simply returns it
|
||||
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
|
||||
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
|
||||
|
||||
if (!features_column)
|
||||
{
|
||||
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
|
||||
}
|
||||
}
|
||||
|
||||
container.reserve(rows_num);
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
container.reserve(container.size() + limit);
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
container.emplace_back(1 / (1 + exp(-results[row_num])));
|
||||
}
|
||||
}
|
||||
|
||||
void LogisticRegression::compute(
|
||||
std::vector<Float64> & batch_gradient,
|
||||
@ -413,10 +413,12 @@ void LogisticRegression::compute(
|
||||
void LinearRegression::predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
const Context & context) const
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
if (weights.size() + 1 != arguments.size())
|
||||
{
|
||||
@ -424,37 +426,34 @@ void LinearRegression::predict(
|
||||
}
|
||||
|
||||
size_t rows_num = block.rows();
|
||||
std::vector<Float64> results(rows_num, bias);
|
||||
|
||||
if (offset > rows_num || offset + limit > rows_num)
|
||||
throw Exception("Invalid offset and limit for LogisticRegression::predict. "
|
||||
"Block has " + toString(rows_num) + " rows, but offset is " + toString(offset) +
|
||||
" and limit is " + toString(limit), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
std::vector<Float64> results(limit, bias);
|
||||
|
||||
for (size_t i = 1; i < arguments.size(); ++i)
|
||||
{
|
||||
const ColumnWithTypeAndName & cur_col = block.getByPosition(arguments[i]);
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
{
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
}
|
||||
|
||||
/// If column type is already Float64 then castColumn simply returns it
|
||||
auto features_col_ptr = castColumn(cur_col, std::make_shared<DataTypeFloat64>(), context);
|
||||
auto features_column = typeid_cast<const ColumnFloat64 *>(features_col_ptr.get());
|
||||
if (!isNativeNumber(cur_col.type))
|
||||
throw Exception("Prediction arguments must have numeric type", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
auto features_column = cur_col.column;
|
||||
|
||||
if (!features_column)
|
||||
{
|
||||
throw Exception("Unexpectedly cannot dynamically cast features column " + std::to_string(i), ErrorCodes::LOGICAL_ERROR);
|
||||
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
results[row_num] += weights[i - 1] * features_column->getFloat64(row_num + offset);
|
||||
}
|
||||
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
results[row_num] += weights[i - 1] * features_column->getElement(row_num);
|
||||
}
|
||||
}
|
||||
|
||||
container.reserve(rows_num);
|
||||
for (size_t row_num = 0; row_num != rows_num; ++row_num)
|
||||
{
|
||||
container.reserve(container.size() + limit);
|
||||
for (size_t row_num = 0; row_num < limit; ++row_num)
|
||||
container.emplace_back(results[row_num]);
|
||||
}
|
||||
}
|
||||
|
||||
void LinearRegression::compute(
|
||||
std::vector<Float64> & batch_gradient,
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnsCommon.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <Common/typeid_cast.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeTuple.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
@ -36,12 +37,13 @@ public:
|
||||
Float64 l2_reg_coef,
|
||||
Float64 target,
|
||||
const IColumn ** columns,
|
||||
size_t row_num)
|
||||
= 0;
|
||||
size_t row_num) = 0;
|
||||
|
||||
virtual void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -67,6 +69,8 @@ public:
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -92,6 +96,8 @@ public:
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const std::vector<Float64> & weights,
|
||||
Float64 bias,
|
||||
@ -194,8 +200,7 @@ private:
|
||||
};
|
||||
|
||||
|
||||
/**
|
||||
* LinearModelData is a class which manages current state of learning
|
||||
/** LinearModelData is a class which manages current state of learning
|
||||
*/
|
||||
class LinearModelData
|
||||
{
|
||||
@ -218,8 +223,13 @@ public:
|
||||
|
||||
void read(ReadBuffer & buf);
|
||||
|
||||
void
|
||||
predict(ColumnVector<Float64>::Container & container, Block & block, const ColumnNumbers & arguments, const Context & context) const;
|
||||
void predict(
|
||||
ColumnVector<Float64>::Container & container,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const;
|
||||
|
||||
void returnWeights(IColumn & to) const;
|
||||
private:
|
||||
@ -228,17 +238,16 @@ private:
|
||||
|
||||
Float64 learning_rate;
|
||||
Float64 l2_reg_coef;
|
||||
UInt32 batch_capacity;
|
||||
UInt64 batch_capacity;
|
||||
|
||||
UInt32 iter_num = 0;
|
||||
UInt64 iter_num = 0;
|
||||
std::vector<Float64> gradient_batch;
|
||||
UInt32 batch_size;
|
||||
UInt64 batch_size;
|
||||
|
||||
std::shared_ptr<IGradientComputer> gradient_computer;
|
||||
std::shared_ptr<IWeightsUpdater> weights_updater;
|
||||
|
||||
/**
|
||||
* The function is called when we want to flush current batch and update our weights
|
||||
/** The function is called when we want to flush current batch and update our weights
|
||||
*/
|
||||
void update_state();
|
||||
};
|
||||
@ -256,7 +265,7 @@ public:
|
||||
|
||||
explicit AggregateFunctionMLMethod(
|
||||
UInt32 param_num,
|
||||
std::shared_ptr<IGradientComputer> gradient_computer,
|
||||
std::unique_ptr<IGradientComputer> gradient_computer,
|
||||
std::string weights_updater_name,
|
||||
Float64 learning_rate,
|
||||
Float64 l2_reg_coef,
|
||||
@ -288,19 +297,15 @@ public:
|
||||
void create(AggregateDataPtr place) const override
|
||||
{
|
||||
std::shared_ptr<IWeightsUpdater> new_weights_updater;
|
||||
if (weights_updater_name == "\'SGD\'")
|
||||
{
|
||||
if (weights_updater_name == "SGD")
|
||||
new_weights_updater = std::make_shared<StochasticGradientDescent>();
|
||||
} else if (weights_updater_name == "\'Momentum\'")
|
||||
{
|
||||
else if (weights_updater_name == "Momentum")
|
||||
new_weights_updater = std::make_shared<Momentum>();
|
||||
} else if (weights_updater_name == "\'Nesterov\'")
|
||||
{
|
||||
else if (weights_updater_name == "Nesterov")
|
||||
new_weights_updater = std::make_shared<Nesterov>();
|
||||
} else
|
||||
{
|
||||
else
|
||||
throw Exception("Illegal name of weights updater (should have been checked earlier)", ErrorCodes::LOGICAL_ERROR);
|
||||
}
|
||||
|
||||
new (place) Data(learning_rate, l2_reg_coef, param_num, batch_size, gradient_computer, new_weights_updater);
|
||||
}
|
||||
|
||||
@ -316,7 +321,13 @@ public:
|
||||
void deserialize(AggregateDataPtr place, ReadBuffer & buf, Arena *) const override { this->data(place).read(buf); }
|
||||
|
||||
void predictValues(
|
||||
ConstAggregateDataPtr place, IColumn & to, Block & block, const ColumnNumbers & arguments, const Context & context) const override
|
||||
ConstAggregateDataPtr place,
|
||||
IColumn & to,
|
||||
Block & block,
|
||||
size_t offset,
|
||||
size_t limit,
|
||||
const ColumnNumbers & arguments,
|
||||
const Context & context) const override
|
||||
{
|
||||
if (arguments.size() != param_num + 1)
|
||||
throw Exception(
|
||||
@ -325,17 +336,12 @@ public:
|
||||
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
|
||||
/// This cast might be correct because column type is based on getReturnTypeToPredict.
|
||||
ColumnVector<Float64> * column;
|
||||
try
|
||||
{
|
||||
column = &dynamic_cast<ColumnVector<Float64> &>(to);
|
||||
} catch (const std::bad_cast &)
|
||||
{
|
||||
auto * column = typeid_cast<ColumnFloat64 *>(&to);
|
||||
if (!column)
|
||||
throw Exception("Cast of column of predictions is incorrect. getReturnTypeToPredict must return same value as it is casted to",
|
||||
ErrorCodes::BAD_CAST);
|
||||
}
|
||||
|
||||
this->data(place).predict(column->getData(), block, arguments, context);
|
||||
this->data(place).predict(column->getData(), block, offset, limit, arguments, context);
|
||||
}
|
||||
|
||||
/** This function is called if aggregate function without State modifier is selected in a query.
|
||||
|
||||
@ -117,11 +117,11 @@ namespace
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
void registerAggregateFunctionUniqCombined(AggregateFunctionFactory & factory)
|
||||
{
|
||||
factory.registerFunction("uniqCombined", createAggregateFunctionUniqCombined);
|
||||
}
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -63,7 +63,7 @@ namespace detail
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace detail
|
||||
}
|
||||
|
||||
|
||||
template <typename Key, UInt8 K>
|
||||
@ -231,4 +231,4 @@ public:
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -100,9 +100,16 @@ public:
|
||||
/// Inserts results into a column.
|
||||
virtual void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const = 0;
|
||||
|
||||
/// This function is used for machine learning methods
|
||||
virtual void predictValues(ConstAggregateDataPtr /* place */, IColumn & /*to*/,
|
||||
Block & /*block*/, const ColumnNumbers & /*arguments*/, const Context & /*context*/) const
|
||||
/// Used for machine learning methods. Predict result from trained model.
|
||||
/// Will insert result into `to` column for rows in range [offset, offset + limit).
|
||||
virtual void predictValues(
|
||||
ConstAggregateDataPtr /* place */,
|
||||
IColumn & /*to*/,
|
||||
Block & /*block*/,
|
||||
size_t /*offset*/,
|
||||
size_t /*limit*/,
|
||||
const ColumnNumbers & /*arguments*/,
|
||||
const Context & /*context*/) const
|
||||
{
|
||||
throw Exception("Method predictValues is not supported for " + getName(), ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
@ -92,13 +92,21 @@ MutableColumnPtr ColumnAggregateFunction::predictValues(Block & block, const Col
|
||||
auto ML_function = func.get();
|
||||
if (ML_function)
|
||||
{
|
||||
if (data.size() == 1)
|
||||
{
|
||||
/// Case for const column. Predict using single model.
|
||||
ML_function->predictValues(data[0], *res, block, 0, block.rows(), arguments, context);
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Case for non-constant column. Use different aggregate function for each row.
|
||||
size_t row_num = 0;
|
||||
for (auto val : data)
|
||||
{
|
||||
ML_function->predictValues(val, *res, block, arguments, context);
|
||||
ML_function->predictValues(val, *res, block, row_num, 1, arguments, context);
|
||||
++row_num;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@ -527,7 +527,7 @@ ConfigProcessor::LoadedConfig ConfigProcessor::loadConfig(bool allow_zk_includes
|
||||
XMLDocumentPtr config_xml = processConfig(&has_zk_includes);
|
||||
|
||||
if (has_zk_includes && !allow_zk_includes)
|
||||
throw Poco::Exception("Error while loading config `" + path + "': from_zk includes are not allowed!");
|
||||
throw Poco::Exception("Error while loading config '" + path + "': from_zk includes are not allowed!");
|
||||
|
||||
ConfigurationPtr configuration(new Poco::Util::XMLConfiguration(config_xml));
|
||||
|
||||
|
||||
@ -87,7 +87,7 @@ void ConfigReloader::reloadIfNewer(bool force, bool throw_on_error, bool fallbac
|
||||
ConfigProcessor::LoadedConfig loaded_config;
|
||||
try
|
||||
{
|
||||
LOG_DEBUG(log, "Loading config `" << path << "'");
|
||||
LOG_DEBUG(log, "Loading config '" << path << "'");
|
||||
|
||||
loaded_config = config_processor.loadConfig(/* allow_zk_includes = */ true);
|
||||
if (loaded_config.has_zk_includes)
|
||||
@ -102,7 +102,7 @@ void ConfigReloader::reloadIfNewer(bool force, bool throw_on_error, bool fallbac
|
||||
if (throw_on_error)
|
||||
throw;
|
||||
|
||||
tryLogCurrentException(log, "ZooKeeper error when loading config from `" + path + "'");
|
||||
tryLogCurrentException(log, "ZooKeeper error when loading config from '" + path + "'");
|
||||
return;
|
||||
}
|
||||
catch (...)
|
||||
@ -110,7 +110,7 @@ void ConfigReloader::reloadIfNewer(bool force, bool throw_on_error, bool fallbac
|
||||
if (throw_on_error)
|
||||
throw;
|
||||
|
||||
tryLogCurrentException(log, "Error loading config from `" + path + "'");
|
||||
tryLogCurrentException(log, "Error loading config from '" + path + "'");
|
||||
return;
|
||||
}
|
||||
config_processor.savePreprocessedConfig(loaded_config, preprocessed_dir);
|
||||
@ -134,7 +134,7 @@ void ConfigReloader::reloadIfNewer(bool force, bool throw_on_error, bool fallbac
|
||||
{
|
||||
if (throw_on_error)
|
||||
throw;
|
||||
tryLogCurrentException(log, "Error updating configuration from `" + path + "' config.");
|
||||
tryLogCurrentException(log, "Error updating configuration from '" + path + "' config.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
30
dbms/src/Common/IPv6ToBinary.cpp
Normal file
30
dbms/src/Common/IPv6ToBinary.cpp
Normal file
@ -0,0 +1,30 @@
|
||||
#include "IPv6ToBinary.h"
|
||||
#include <Poco/Net/IPAddress.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
std::array<char, 16> IPv6ToBinary(const Poco::Net::IPAddress & address)
|
||||
{
|
||||
std::array<char, 16> res;
|
||||
|
||||
if (Poco::Net::IPAddress::IPv6 == address.family())
|
||||
{
|
||||
memcpy(res.data(), address.addr(), 16);
|
||||
}
|
||||
else if (Poco::Net::IPAddress::IPv4 == address.family())
|
||||
{
|
||||
/// Convert to IPv6-mapped address.
|
||||
memset(res.data(), 0, 10);
|
||||
res[10] = '\xFF';
|
||||
res[11] = '\xFF';
|
||||
memcpy(&res[12], address.addr(), 4);
|
||||
}
|
||||
else
|
||||
memset(res.data(), 0, 16);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
}
|
||||
11
dbms/src/Common/IPv6ToBinary.h
Normal file
11
dbms/src/Common/IPv6ToBinary.h
Normal file
@ -0,0 +1,11 @@
|
||||
#include <array>
|
||||
|
||||
namespace Poco { namespace Net { class IPAddress; }}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// Convert IP address to 16-byte array with IPv6 data (big endian). If it's an IPv4, map it to IPv6.
|
||||
std::array<char, 16> IPv6ToBinary(const Poco::Net::IPAddress & address);
|
||||
|
||||
}
|
||||
@ -6,7 +6,7 @@
|
||||
#include <Common/config.h>
|
||||
#include <re2/re2.h>
|
||||
#if USE_RE2_ST
|
||||
#include <re2_st/re2.h> // Y_IGNORE
|
||||
#include <re2_st/re2.h>
|
||||
#else
|
||||
#define re2_st re2
|
||||
#endif
|
||||
|
||||
@ -329,8 +329,7 @@ class StringSearcher<false, true> : private StringSearcherBase
|
||||
private:
|
||||
/// string to be searched for
|
||||
const UInt8 * const needle;
|
||||
const size_t needle_size;
|
||||
const UInt8 * const needle_end = needle + needle_size;
|
||||
const UInt8 * const needle_end;
|
||||
/// lower and uppercase variants of the first character in `needle`
|
||||
UInt8 l{};
|
||||
UInt8 u{};
|
||||
@ -345,7 +344,7 @@ private:
|
||||
|
||||
public:
|
||||
StringSearcher(const char * const needle_, const size_t needle_size)
|
||||
: needle{reinterpret_cast<const UInt8 *>(needle_)}, needle_size{needle_size}
|
||||
: needle{reinterpret_cast<const UInt8 *>(needle_)}, needle_end{needle + needle_size}
|
||||
{
|
||||
if (0 == needle_size)
|
||||
return;
|
||||
@ -430,7 +429,7 @@ public:
|
||||
|
||||
const UInt8 * search(const UInt8 * haystack, const UInt8 * const haystack_end) const
|
||||
{
|
||||
if (0 == needle_size)
|
||||
if (needle == needle_end)
|
||||
return haystack;
|
||||
|
||||
while (haystack < haystack_end)
|
||||
@ -528,8 +527,7 @@ class StringSearcher<true, ASCII> : private StringSearcherBase
|
||||
private:
|
||||
/// string to be searched for
|
||||
const UInt8 * const needle;
|
||||
const size_t needle_size;
|
||||
const UInt8 * const needle_end = needle + needle_size;
|
||||
const UInt8 * const needle_end;
|
||||
/// first character in `needle`
|
||||
UInt8 first{};
|
||||
|
||||
@ -543,7 +541,7 @@ private:
|
||||
|
||||
public:
|
||||
StringSearcher(const char * const needle_, const size_t needle_size)
|
||||
: needle{reinterpret_cast<const UInt8 *>(needle_)}, needle_size{needle_size}
|
||||
: needle{reinterpret_cast<const UInt8 *>(needle_)}, needle_end{needle + needle_size}
|
||||
{
|
||||
if (0 == needle_size)
|
||||
return;
|
||||
@ -616,7 +614,7 @@ public:
|
||||
|
||||
const UInt8 * search(const UInt8 * haystack, const UInt8 * const haystack_end) const
|
||||
{
|
||||
if (0 == needle_size)
|
||||
if (needle == needle_end)
|
||||
return haystack;
|
||||
|
||||
while (haystack < haystack_end)
|
||||
@ -715,10 +713,9 @@ using UTF8CaseInsensitiveStringSearcher = StringSearcher<false, false>;
|
||||
struct LibCASCIICaseSensitiveStringSearcher
|
||||
{
|
||||
const char * const needle;
|
||||
const size_t needle_size;
|
||||
|
||||
LibCASCIICaseSensitiveStringSearcher(const char * const needle, const size_t needle_size)
|
||||
: needle(needle), needle_size(needle_size) {}
|
||||
LibCASCIICaseSensitiveStringSearcher(const char * const needle, const size_t /* needle_size */)
|
||||
: needle(needle) {}
|
||||
|
||||
const UInt8 * search(const UInt8 * haystack, const UInt8 * const haystack_end) const
|
||||
{
|
||||
@ -737,10 +734,9 @@ struct LibCASCIICaseSensitiveStringSearcher
|
||||
struct LibCASCIICaseInsensitiveStringSearcher
|
||||
{
|
||||
const char * const needle;
|
||||
const size_t needle_size;
|
||||
|
||||
LibCASCIICaseInsensitiveStringSearcher(const char * const needle, const size_t needle_size)
|
||||
: needle(needle), needle_size(needle_size) {}
|
||||
LibCASCIICaseInsensitiveStringSearcher(const char * const needle, const size_t /* needle_size */)
|
||||
: needle(needle) {}
|
||||
|
||||
const UInt8 * search(const UInt8 * haystack, const UInt8 * const haystack_end) const
|
||||
{
|
||||
|
||||
@ -60,4 +60,4 @@ public:
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -4,7 +4,6 @@
|
||||
#include <vector>
|
||||
#include <stdint.h>
|
||||
#include <string.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Core/Types.h>
|
||||
#include <Poco/UTF8Encoding.h>
|
||||
#include <Poco/Unicode.h>
|
||||
@ -345,6 +344,7 @@ public:
|
||||
|
||||
auto callback = [this](const VolnitskyTraits::Ngram ngram, const int offset) { return this->putNGramBase(ngram, offset); };
|
||||
/// ssize_t is used here because unsigned can't be used with condition like `i >= 0`, unsigned always >= 0
|
||||
/// And also adding from the end guarantees that we will find first occurence because we will lookup bigger offsets first.
|
||||
for (auto i = static_cast<ssize_t>(needle_size - sizeof(VolnitskyTraits::Ngram)); i >= 0; --i)
|
||||
VolnitskyTraits::putNGram<CaseSensitive, ASCII>(this->needle + i, i + 1, this->needle, callback);
|
||||
}
|
||||
@ -436,94 +436,6 @@ public:
|
||||
fallback_searchers.reserve(needles.size());
|
||||
}
|
||||
|
||||
template <typename ResultType, typename AnsCallback>
|
||||
void searchAllPositions(
|
||||
const ColumnString::Chars & haystack_data,
|
||||
const ColumnString::Offsets & haystack_offsets,
|
||||
const AnsCallback & ans_callback,
|
||||
ResultType & ans)
|
||||
{
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
const size_t needles_size = needles.size();
|
||||
|
||||
/// something can be uninitialized after
|
||||
std::fill(ans.begin(), ans.end(), 0);
|
||||
|
||||
while (!reset())
|
||||
{
|
||||
size_t fallback_size = fallback_needles.size();
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0, from = 0; j < haystack_string_size; ++j, from += needles_size)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
for (size_t i = 0; i < fallback_size; ++i)
|
||||
{
|
||||
const UInt8 * ptr = fallback_searchers[fallback_needles[i]].search(haystack, haystack_end);
|
||||
if (ptr != haystack_end)
|
||||
ans[from + fallback_needles[i]] = ans_callback(haystack, ptr);
|
||||
}
|
||||
|
||||
/// check if we have one non empty volnitsky searcher
|
||||
if (step != std::numeric_limits<size_t>::max())
|
||||
{
|
||||
const auto * pos = haystack + step - sizeof(VolnitskyTraits::Ngram);
|
||||
for (; pos <= haystack_end - sizeof(VolnitskyTraits::Ngram); pos += step)
|
||||
{
|
||||
for (size_t cell_num = VolnitskyTraits::toNGram(pos) % VolnitskyTraits::hash_size; hash[cell_num].off;
|
||||
cell_num = (cell_num + 1) % VolnitskyTraits::hash_size)
|
||||
{
|
||||
if (pos >= haystack + hash[cell_num].off - 1)
|
||||
{
|
||||
const auto * res = pos - (hash[cell_num].off - 1);
|
||||
const size_t ind = hash[cell_num].id;
|
||||
if (ans[from + ind] == 0 && res + needles[ind].size <= haystack_end)
|
||||
{
|
||||
if (fallback_searchers[ind].compare(res))
|
||||
{
|
||||
ans[from + ind] = ans_callback(haystack, res);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
template <typename ResultType>
|
||||
void search(const ColumnString::Chars & haystack_data, const ColumnString::Offsets & haystack_offsets, ResultType & ans)
|
||||
{
|
||||
auto callback = [this](const UInt8 * haystack, const UInt8 * haystack_end) -> bool
|
||||
{
|
||||
return this->searchOne(haystack, haystack_end);
|
||||
};
|
||||
searchInternal(haystack_data, haystack_offsets, callback, ans);
|
||||
}
|
||||
|
||||
template <typename ResultType>
|
||||
void searchIndex(const ColumnString::Chars & haystack_data, const ColumnString::Offsets & haystack_offsets, ResultType & ans)
|
||||
{
|
||||
auto callback = [this](const UInt8 * haystack, const UInt8 * haystack_end) -> size_t
|
||||
{
|
||||
return this->searchOneIndex(haystack, haystack_end);
|
||||
};
|
||||
searchInternal(haystack_data, haystack_offsets, callback, ans);
|
||||
}
|
||||
|
||||
template <typename ResultType, typename CountCharsCallback>
|
||||
void searchFirstPosition(const ColumnString::Chars & haystack_data, const ColumnString::Offsets & haystack_offsets, const CountCharsCallback & count_chars_callback, ResultType & ans)
|
||||
{
|
||||
auto callback = [this, &count_chars_callback](const UInt8 * haystack, const UInt8 * haystack_end) -> UInt64
|
||||
{
|
||||
return this->searchOneFirstPosition(haystack, haystack_end, count_chars_callback);
|
||||
};
|
||||
searchInternal(haystack_data, haystack_offsets, callback, ans);
|
||||
}
|
||||
|
||||
private:
|
||||
/**
|
||||
* This function is needed to initialize hash table
|
||||
* Returns `true` if there is nothing to initialize
|
||||
@ -532,15 +444,15 @@ private:
|
||||
* We actually destroy the hash table and initialize it with uninitialized needles
|
||||
* and search through the haystack again.
|
||||
* The actual usage of this function is like this:
|
||||
* while (!reset())
|
||||
* while (hasMoreToSearch())
|
||||
* {
|
||||
* search inside the haystack with the known needles
|
||||
* }
|
||||
*/
|
||||
bool reset()
|
||||
bool hasMoreToSearch()
|
||||
{
|
||||
if (last == needles.size())
|
||||
return true;
|
||||
return false;
|
||||
|
||||
memset(hash, 0, sizeof(hash));
|
||||
fallback_needles.clear();
|
||||
@ -585,28 +497,7 @@ private:
|
||||
}
|
||||
fallback_searchers.emplace_back(cur_needle_data, cur_needle_size);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
template <typename OneSearcher, typename ResultType>
|
||||
inline void searchInternal(
|
||||
const ColumnString::Chars & haystack_data,
|
||||
const ColumnString::Offsets & haystack_offsets,
|
||||
const OneSearcher & search_fallback,
|
||||
ResultType & ans)
|
||||
{
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
while (!reset())
|
||||
{
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0; j < haystack_string_size; ++j)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
ans[j] = search_fallback(haystack, haystack_end);
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
inline bool searchOne(const UInt8 * haystack, const UInt8 * haystack_end) const
|
||||
@ -638,7 +529,7 @@ private:
|
||||
return false;
|
||||
}
|
||||
|
||||
inline size_t searchOneIndex(const UInt8 * haystack, const UInt8 * haystack_end) const
|
||||
inline size_t searchOneFirstIndex(const UInt8 * haystack, const UInt8 * haystack_end) const
|
||||
{
|
||||
const size_t fallback_size = fallback_needles.size();
|
||||
|
||||
@ -676,7 +567,7 @@ private:
|
||||
}
|
||||
|
||||
template <typename CountCharsCallback>
|
||||
inline UInt64 searchOneFirstPosition(const UInt8 * haystack, const UInt8 * haystack_end, const CountCharsCallback & callback) const
|
||||
inline UInt64 searchOneFirstPosition(const UInt8 * haystack, const UInt8 * haystack_end, const CountCharsCallback & count_chars) const
|
||||
{
|
||||
const size_t fallback_size = fallback_needles.size();
|
||||
|
||||
@ -684,7 +575,7 @@ private:
|
||||
|
||||
for (size_t i = 0; i < fallback_size; ++i)
|
||||
if (auto pos = fallback_searchers[fallback_needles[i]].search(haystack, haystack_end); pos != haystack_end)
|
||||
ans = std::min(ans, callback(haystack, pos));
|
||||
ans = std::min<UInt64>(ans, pos - haystack);
|
||||
|
||||
/// check if we have one non empty volnitsky searcher
|
||||
if (step != std::numeric_limits<size_t>::max())
|
||||
@ -700,14 +591,46 @@ private:
|
||||
const auto res = pos - (hash[cell_num].off - 1);
|
||||
const size_t ind = hash[cell_num].id;
|
||||
if (res + needles[ind].size <= haystack_end && fallback_searchers[ind].compare(res))
|
||||
ans = std::min(ans, callback(haystack, res));
|
||||
ans = std::min<UInt64>(ans, res - haystack);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (ans == std::numeric_limits<UInt64>::max())
|
||||
return 0;
|
||||
return ans;
|
||||
return count_chars(haystack, haystack + ans);
|
||||
}
|
||||
|
||||
template <typename CountCharsCallback, typename AnsType>
|
||||
inline void searchOneAll(const UInt8 * haystack, const UInt8 * haystack_end, AnsType * ans, const CountCharsCallback & count_chars) const
|
||||
{
|
||||
const size_t fallback_size = fallback_needles.size();
|
||||
for (size_t i = 0; i < fallback_size; ++i)
|
||||
{
|
||||
const UInt8 * ptr = fallback_searchers[fallback_needles[i]].search(haystack, haystack_end);
|
||||
if (ptr != haystack_end)
|
||||
ans[fallback_needles[i]] = count_chars(haystack, ptr);
|
||||
}
|
||||
|
||||
/// check if we have one non empty volnitsky searcher
|
||||
if (step != std::numeric_limits<size_t>::max())
|
||||
{
|
||||
const auto * pos = haystack + step - sizeof(VolnitskyTraits::Ngram);
|
||||
for (; pos <= haystack_end - sizeof(VolnitskyTraits::Ngram); pos += step)
|
||||
{
|
||||
for (size_t cell_num = VolnitskyTraits::toNGram(pos) % VolnitskyTraits::hash_size; hash[cell_num].off;
|
||||
cell_num = (cell_num + 1) % VolnitskyTraits::hash_size)
|
||||
{
|
||||
if (pos >= haystack + hash[cell_num].off - 1)
|
||||
{
|
||||
const auto * res = pos - (hash[cell_num].off - 1);
|
||||
const size_t ind = hash[cell_num].id;
|
||||
if (ans[ind] == 0 && res + needles[ind].size <= haystack_end && fallback_searchers[ind].compare(res))
|
||||
ans[ind] = count_chars(haystack, res);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void putNGramBase(const VolnitskyTraits::Ngram ngram, const int offset, const size_t num)
|
||||
|
||||
@ -36,7 +36,7 @@ ZooKeeperNodeCache::ZNode ZooKeeperNodeCache::get(const std::string & path, Coor
|
||||
|
||||
zkutil::ZooKeeperPtr zookeeper = get_zookeeper();
|
||||
if (!zookeeper)
|
||||
throw DB::Exception("Could not get znode: `" + path + "'. ZooKeeper not configured.", DB::ErrorCodes::NO_ZOOKEEPER);
|
||||
throw DB::Exception("Could not get znode: '" + path + "'. ZooKeeper not configured.", DB::ErrorCodes::NO_ZOOKEEPER);
|
||||
|
||||
for (const auto & invalidated_path : invalidated_paths)
|
||||
path_to_cached_znode.erase(invalidated_path);
|
||||
|
||||
65
dbms/src/Common/convertMySQLDataType.cpp
Normal file
65
dbms/src/Common/convertMySQLDataType.cpp
Normal file
@ -0,0 +1,65 @@
|
||||
#include <Common/convertMySQLDataType.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
ASTPtr dataTypeConvertToQuery(const DataTypePtr & data_type)
|
||||
{
|
||||
WhichDataType which(data_type);
|
||||
|
||||
if (!which.isNullable())
|
||||
return std::make_shared<ASTIdentifier>(data_type->getName());
|
||||
|
||||
return makeASTFunction("Nullable", dataTypeConvertToQuery(typeid_cast<const DataTypeNullable *>(data_type.get())->getNestedType()));
|
||||
}
|
||||
|
||||
DataTypePtr convertMySQLDataType(const String & mysql_data_type, bool is_nullable, bool is_unsigned, size_t length)
|
||||
{
|
||||
DataTypePtr res;
|
||||
if (mysql_data_type == "tinyint")
|
||||
{
|
||||
if (is_unsigned)
|
||||
res = std::make_shared<DataTypeUInt8>();
|
||||
else
|
||||
res = std::make_shared<DataTypeInt8>();
|
||||
}
|
||||
else if (mysql_data_type == "smallint")
|
||||
{
|
||||
if (is_unsigned)
|
||||
res = std::make_shared<DataTypeUInt16>();
|
||||
else
|
||||
res = std::make_shared<DataTypeInt16>();
|
||||
}
|
||||
else if (mysql_data_type == "int" || mysql_data_type == "mediumint")
|
||||
{
|
||||
if (is_unsigned)
|
||||
res = std::make_shared<DataTypeUInt32>();
|
||||
else
|
||||
res = std::make_shared<DataTypeInt32>();
|
||||
}
|
||||
else if (mysql_data_type == "bigint")
|
||||
{
|
||||
if (is_unsigned)
|
||||
res = std::make_shared<DataTypeUInt64>();
|
||||
else
|
||||
res = std::make_shared<DataTypeInt64>();
|
||||
}
|
||||
else if (mysql_data_type == "float")
|
||||
res = std::make_shared<DataTypeFloat32>();
|
||||
else if (mysql_data_type == "double")
|
||||
res = std::make_shared<DataTypeFloat64>();
|
||||
else if (mysql_data_type == "date")
|
||||
res = std::make_shared<DataTypeDate>();
|
||||
else if (mysql_data_type == "datetime" || mysql_data_type == "timestamp")
|
||||
res = std::make_shared<DataTypeDateTime>();
|
||||
else if (mysql_data_type == "binary")
|
||||
res = std::make_shared<DataTypeFixedString>(length);
|
||||
else
|
||||
/// Also String is fallback for all unknown types.
|
||||
res = std::make_shared<DataTypeString>();
|
||||
if (is_nullable)
|
||||
res = std::make_shared<DataTypeNullable>(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
}
|
||||
26
dbms/src/Common/convertMySQLDataType.h
Normal file
26
dbms/src/Common/convertMySQLDataType.h
Normal file
@ -0,0 +1,26 @@
|
||||
#pragma once
|
||||
|
||||
#include <Core/Types.h>
|
||||
#include <Parsers/IAST.h>
|
||||
#include <Parsers/ASTFunction.h>
|
||||
#include <Parsers/ASTIdentifier.h>
|
||||
#include <DataTypes/IDataType.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeDate.h>
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
#include <DataTypes/DataTypeFixedString.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// Convert data type to query. for example
|
||||
/// DataTypeUInt8 -> ASTIdentifier(UInt8)
|
||||
/// DataTypeNullable(DataTypeUInt8) -> ASTFunction(ASTIdentifier(UInt8))
|
||||
ASTPtr dataTypeConvertToQuery(const DataTypePtr & data_type);
|
||||
|
||||
/// Convert MySQL type to ClickHouse data type.
|
||||
DataTypePtr convertMySQLDataType(const String & mysql_data_type, bool is_nullable, bool is_unsigned, size_t length);
|
||||
|
||||
}
|
||||
@ -7,7 +7,7 @@
|
||||
# include <Common/Exception.h>
|
||||
namespace DB { namespace ErrorCodes { extern const int CPUID_ERROR; }}
|
||||
#elif USE_CPUINFO
|
||||
# include <cpuinfo.h> // Y_IGNORE
|
||||
# include <cpuinfo.h>
|
||||
#endif
|
||||
|
||||
|
||||
|
||||
@ -21,4 +21,4 @@ private:
|
||||
Block readImpl() override { return {}; }
|
||||
};
|
||||
|
||||
} /// namespace DB
|
||||
}
|
||||
|
||||
@ -85,4 +85,4 @@ public:
|
||||
String getName() const override { return name; }
|
||||
};
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -98,7 +98,7 @@ public:
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
void registerDataTypeDomainIPv4AndIPv6(DataTypeFactory & factory)
|
||||
{
|
||||
@ -115,4 +115,4 @@ void registerDataTypeDomainIPv4AndIPv6(DataTypeFactory & factory)
|
||||
});
|
||||
}
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -23,7 +23,7 @@ static void deserializeFromString(const DataTypeCustomSimpleTextSerialization &
|
||||
domain.deserializeText(column, istr, settings);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -85,4 +85,4 @@ void DataTypeCustomSimpleTextSerialization::serializeTextXML(const IColumn & col
|
||||
writeXMLString(serializeToString(*this, column, row_num, settings), ostr);
|
||||
}
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -50,4 +50,4 @@ public:
|
||||
void serializeTextXML(const IColumn & column, size_t row_num, WriteBuffer & ostr, const FormatSettings & settings) const override;
|
||||
};
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -212,7 +212,7 @@ DataTypePtr createDecimal(UInt64 precision_value, UInt64 scale_value)
|
||||
throw Exception("Wrong precision", ErrorCodes::ARGUMENT_OUT_OF_BOUND);
|
||||
|
||||
if (static_cast<UInt64>(scale_value) > precision_value)
|
||||
throw Exception("Negative scales and scales larger than presicion are not supported", ErrorCodes::ARGUMENT_OUT_OF_BOUND);
|
||||
throw Exception("Negative scales and scales larger than precision are not supported", ErrorCodes::ARGUMENT_OUT_OF_BOUND);
|
||||
|
||||
if (precision_value <= maxDecimalPrecision<Decimal32>())
|
||||
return std::make_shared<DataTypeDecimal<Decimal32>>(precision_value, scale_value);
|
||||
|
||||
@ -320,12 +320,27 @@ inline std::enable_if_t<IsDataTypeNumber<FromDataType> && IsDataTypeDecimal<ToDa
|
||||
convertToDecimal(const typename FromDataType::FieldType & value, UInt32 scale)
|
||||
{
|
||||
using FromFieldType = typename FromDataType::FieldType;
|
||||
using ToNativeType = typename ToDataType::FieldType::NativeType;
|
||||
|
||||
if constexpr (std::is_floating_point_v<FromFieldType>)
|
||||
{
|
||||
if (!std::isfinite(value))
|
||||
throw Exception("Decimal convert overflow. Cannot convert infinity or NaN to decimal", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
return value * ToDataType::getScaleMultiplier(scale);
|
||||
|
||||
auto out = value * ToDataType::getScaleMultiplier(scale);
|
||||
if constexpr (std::is_same_v<ToNativeType, Int128>)
|
||||
{
|
||||
static constexpr __int128 min_int128 = __int128(0x8000000000000000ll) << 64;
|
||||
static constexpr __int128 max_int128 = (__int128(0x7fffffffffffffffll) << 64) + 0xffffffffffffffffll;
|
||||
if (out < min_int128 || out > max_int128)
|
||||
throw Exception("Decimal convert overflow. Float is out of Decimal range", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
}
|
||||
else
|
||||
{
|
||||
if (out < std::numeric_limits<ToNativeType>::min() || out > std::numeric_limits<ToNativeType>::max())
|
||||
throw Exception("Decimal convert overflow. Float is out of Decimal range", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@ -16,7 +16,7 @@
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
|
||||
#pragma GCC diagnostic pop
|
||||
|
||||
|
||||
@ -2,22 +2,44 @@
|
||||
#include <Databases/DatabaseOrdinary.h>
|
||||
#include <Databases/DatabaseMemory.h>
|
||||
#include <Databases/DatabaseDictionary.h>
|
||||
#include <Parsers/ASTLiteral.h>
|
||||
#include <Parsers/formatAST.h>
|
||||
#include <Parsers/ASTCreateQuery.h>
|
||||
#include <Common/parseAddress.h>
|
||||
#include <Common/config.h>
|
||||
|
||||
#if USE_MYSQL
|
||||
|
||||
#include <Databases/DatabaseMySQL.h>
|
||||
|
||||
#endif
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int BAD_ARGUMENTS;
|
||||
extern const int UNKNOWN_DATABASE_ENGINE;
|
||||
}
|
||||
|
||||
|
||||
DatabasePtr DatabaseFactory::get(
|
||||
const String & engine_name,
|
||||
const String & database_name,
|
||||
const String & metadata_path,
|
||||
const ASTStorage * engine_define,
|
||||
Context & context)
|
||||
{
|
||||
String engine_name = engine_define->engine->name;
|
||||
|
||||
if (engine_name != "MySQL" && engine_define->engine->arguments)
|
||||
throw Exception("Database engine " + engine_name + " cannot have arguments", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
if (engine_define->engine->parameters || engine_define->partition_by || engine_define->primary_key || engine_define->order_by ||
|
||||
engine_define->sample_by || engine_define->settings)
|
||||
throw Exception("Database engine " + engine_name + " cannot have parameters, primary_key, order_by, sample_by, settings",
|
||||
ErrorCodes::UNKNOWN_ELEMENT_IN_AST);
|
||||
|
||||
if (engine_name == "Ordinary")
|
||||
return std::make_shared<DatabaseOrdinary>(database_name, metadata_path, context);
|
||||
else if (engine_name == "Memory")
|
||||
@ -25,6 +47,29 @@ DatabasePtr DatabaseFactory::get(
|
||||
else if (engine_name == "Dictionary")
|
||||
return std::make_shared<DatabaseDictionary>(database_name);
|
||||
|
||||
#if USE_MYSQL
|
||||
|
||||
else if (engine_name == "MySQL")
|
||||
{
|
||||
const ASTFunction * engine = engine_define->engine;
|
||||
const auto & arguments = engine->arguments->children;
|
||||
|
||||
if (arguments.size() != 4)
|
||||
throw Exception("MySQL Database require mysql_hostname, mysql_database_name, mysql_username, mysql_password arguments.",
|
||||
ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
const auto & mysql_host_name = arguments[0]->as<ASTLiteral>()->value.safeGet<String>();
|
||||
const auto & mysql_database_name = arguments[1]->as<ASTLiteral>()->value.safeGet<String>();
|
||||
const auto & mysql_user_name = arguments[2]->as<ASTLiteral>()->value.safeGet<String>();
|
||||
const auto & mysql_user_password = arguments[3]->as<ASTLiteral>()->value.safeGet<String>();
|
||||
|
||||
auto parsed_host_port = parseAddress(mysql_host_name, 3306);
|
||||
return std::make_shared<DatabaseMySQL>(context, database_name, parsed_host_port.first, parsed_host_port.second, mysql_database_name,
|
||||
mysql_user_name, mysql_user_password);
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
throw Exception("Unknown database engine: " + engine_name, ErrorCodes::UNKNOWN_DATABASE_ENGINE);
|
||||
}
|
||||
|
||||
|
||||
@ -3,17 +3,18 @@
|
||||
#include <Common/ThreadPool.h>
|
||||
#include <Databases/IDatabase.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class ASTStorage;
|
||||
|
||||
class DatabaseFactory
|
||||
{
|
||||
public:
|
||||
static DatabasePtr get(
|
||||
const String & engine_name,
|
||||
const String & database_name,
|
||||
const String & metadata_path,
|
||||
const ASTStorage * engine_define,
|
||||
Context & context);
|
||||
};
|
||||
|
||||
|
||||
397
dbms/src/Databases/DatabaseMySQL.cpp
Normal file
397
dbms/src/Databases/DatabaseMySQL.cpp
Normal file
@ -0,0 +1,397 @@
|
||||
#include <Common/config.h>
|
||||
|
||||
#if USE_MYSQL
|
||||
|
||||
#include <Databases/DatabaseMySQL.h>
|
||||
#include <Common/parseAddress.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Formats/MySQLBlockInputStream.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeDate.h>
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
#include <DataTypes/DataTypeFixedString.h>
|
||||
#include <Storages/StorageMySQL.h>
|
||||
#include <Parsers/ParserCreateQuery.h>
|
||||
#include <Parsers/parseQuery.h>
|
||||
#include <Common/setThreadName.h>
|
||||
#include <Parsers/ASTCreateQuery.h>
|
||||
#include <Common/convertMySQLDataType.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int UNKNOWN_TABLE;
|
||||
}
|
||||
|
||||
static constexpr const std::chrono::seconds cleaner_sleep_time{30};
|
||||
|
||||
String toQueryStringWithQuote(const std::vector<String> & quote_list)
|
||||
{
|
||||
WriteBufferFromOwnString quote_list_query;
|
||||
quote_list_query << "(";
|
||||
|
||||
for (size_t index = 0; index < quote_list.size(); ++index)
|
||||
{
|
||||
if (index)
|
||||
quote_list_query << ",";
|
||||
|
||||
quote_list_query << quote << quote_list[index];
|
||||
}
|
||||
|
||||
quote_list_query << ")";
|
||||
return quote_list_query.str();
|
||||
}
|
||||
|
||||
DatabaseMySQL::DatabaseMySQL(
|
||||
const Context & context_, const String & database_name_, const String & mysql_host_name_, const UInt16 & mysql_port_,
|
||||
const String & mysql_database_name_, const String & mysql_user_name_, const String & mysql_user_password_)
|
||||
: global_context(context_), database_name(database_name_), mysql_host_name(mysql_host_name_), mysql_port(mysql_port_),
|
||||
mysql_database_name(mysql_database_name_), mysql_user_name(mysql_user_name_), mysql_user_password(mysql_user_password_),
|
||||
mysql_pool(mysql_database_name, mysql_host_name, mysql_user_name, mysql_user_password, mysql_port)
|
||||
{
|
||||
}
|
||||
|
||||
bool DatabaseMySQL::empty(const Context &) const
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
|
||||
fetchTablesIntoLocalCache();
|
||||
|
||||
return local_tables_cache.empty();
|
||||
}
|
||||
|
||||
DatabaseIteratorPtr DatabaseMySQL::getIterator(const Context &)
|
||||
{
|
||||
Tables tables;
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
|
||||
fetchTablesIntoLocalCache();
|
||||
|
||||
for (const auto & local_table : local_tables_cache)
|
||||
tables[local_table.first] = local_table.second.storage;
|
||||
|
||||
return std::make_unique<DatabaseSnapshotIterator>(tables);
|
||||
}
|
||||
|
||||
bool DatabaseMySQL::isTableExist(const Context & context, const String & name) const
|
||||
{
|
||||
return bool(tryGetTable(context, name));
|
||||
}
|
||||
|
||||
StoragePtr DatabaseMySQL::tryGetTable(const Context &, const String & mysql_table_name) const
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
|
||||
fetchTablesIntoLocalCache();
|
||||
|
||||
if (local_tables_cache.find(mysql_table_name) != local_tables_cache.end())
|
||||
return local_tables_cache[mysql_table_name].storage;
|
||||
|
||||
return StoragePtr{};
|
||||
}
|
||||
|
||||
ASTPtr DatabaseMySQL::tryGetCreateTableQuery(const Context &, const String & table_name) const
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
|
||||
fetchTablesIntoLocalCache();
|
||||
|
||||
if (local_tables_cache.find(table_name) == local_tables_cache.end())
|
||||
throw Exception("MySQL table " + mysql_database_name + "." + table_name + " doesn't exist..", ErrorCodes::UNKNOWN_TABLE);
|
||||
|
||||
return local_tables_cache[table_name].create_table_query;
|
||||
}
|
||||
|
||||
time_t DatabaseMySQL::getTableMetadataModificationTime(const Context &, const String & table_name)
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
|
||||
fetchTablesIntoLocalCache();
|
||||
|
||||
if (local_tables_cache.find(table_name) == local_tables_cache.end())
|
||||
throw Exception("MySQL table " + mysql_database_name + "." + table_name + " doesn't exist..", ErrorCodes::UNKNOWN_TABLE);
|
||||
|
||||
return time_t(local_tables_cache[table_name].modification_time);
|
||||
}
|
||||
|
||||
ASTPtr DatabaseMySQL::getCreateDatabaseQuery(const Context &) const
|
||||
{
|
||||
const auto & create_query = std::make_shared<ASTCreateQuery>();
|
||||
create_query->database = database_name;
|
||||
|
||||
const auto & storage = std::make_shared<ASTStorage>();
|
||||
storage->set(storage->engine, makeASTFunction("MySQL",
|
||||
std::make_shared<ASTLiteral>(mysql_host_name + ":" + toString(mysql_port)), std::make_shared<ASTLiteral>(mysql_database_name),
|
||||
std::make_shared<ASTLiteral>(mysql_user_name), std::make_shared<ASTLiteral>(mysql_user_password)));
|
||||
|
||||
create_query->set(create_query->storage, storage);
|
||||
return create_query;
|
||||
}
|
||||
|
||||
void DatabaseMySQL::fetchTablesIntoLocalCache() const
|
||||
{
|
||||
const auto & tables_with_modification_time = fetchTablesWithModificationTime();
|
||||
|
||||
destroyLocalCacheExtraTables(tables_with_modification_time);
|
||||
fetchLatestTablesStructureIntoCache(tables_with_modification_time);
|
||||
}
|
||||
|
||||
void DatabaseMySQL::destroyLocalCacheExtraTables(const std::map<String, UInt64> & tables_with_modification_time) const
|
||||
{
|
||||
for (auto iterator = local_tables_cache.begin(); iterator != local_tables_cache.end();)
|
||||
{
|
||||
if (tables_with_modification_time.find(iterator->first) != tables_with_modification_time.end())
|
||||
++iterator;
|
||||
else
|
||||
{
|
||||
outdated_tables.emplace_back(iterator->second.storage);
|
||||
iterator = local_tables_cache.erase(iterator);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void DatabaseMySQL::fetchLatestTablesStructureIntoCache(const std::map<String, UInt64> &tables_modification_time) const
|
||||
{
|
||||
std::vector<String> wait_update_tables_name;
|
||||
for (const auto & table_modification_time : tables_modification_time)
|
||||
{
|
||||
const auto & it = local_tables_cache.find(table_modification_time.first);
|
||||
|
||||
/// Outdated or new table structures
|
||||
if (it == local_tables_cache.end() || table_modification_time.second > it->second.modification_time)
|
||||
wait_update_tables_name.emplace_back(table_modification_time.first);
|
||||
}
|
||||
|
||||
std::map<String, NamesAndTypesList> tables_and_columns = fetchTablesColumnsList(wait_update_tables_name);
|
||||
|
||||
for (const auto & table_and_columns : tables_and_columns)
|
||||
{
|
||||
const auto & table_name = table_and_columns.first;
|
||||
const auto & columns_name_and_type = table_and_columns.second;
|
||||
const auto & table_modification_time = tables_modification_time.at(table_name);
|
||||
|
||||
const auto & iterator = local_tables_cache.find(table_name);
|
||||
if (iterator != local_tables_cache.end())
|
||||
{
|
||||
outdated_tables.emplace_back(iterator->second.storage);
|
||||
local_tables_cache.erase(iterator);
|
||||
}
|
||||
|
||||
local_tables_cache[table_name] = createStorageInfo(table_name, columns_name_and_type, table_modification_time);
|
||||
}
|
||||
}
|
||||
|
||||
static ASTPtr getTableColumnsCreateQuery(const NamesAndTypesList & names_and_types_list)
|
||||
{
|
||||
const auto & table_columns_list_ast = std::make_shared<ASTColumns>();
|
||||
const auto & columns_expression_list = std::make_shared<ASTExpressionList>();
|
||||
|
||||
for (const auto & table_column_name_and_type : names_and_types_list)
|
||||
{
|
||||
const auto & column_declaration = std::make_shared<ASTColumnDeclaration>();
|
||||
column_declaration->name = table_column_name_and_type.name;
|
||||
column_declaration->type = dataTypeConvertToQuery(table_column_name_and_type.type);
|
||||
columns_expression_list->children.emplace_back(column_declaration);
|
||||
}
|
||||
|
||||
table_columns_list_ast->set(table_columns_list_ast->columns, columns_expression_list);
|
||||
return table_columns_list_ast;
|
||||
}
|
||||
|
||||
static ASTPtr getTableStorageCreateQuery(
|
||||
const String & host_name, const UInt16 & port,
|
||||
const String & database_name, const String & table_name,
|
||||
const String & user_name, const String & password)
|
||||
{
|
||||
const auto & table_storage = std::make_shared<ASTStorage>();
|
||||
const auto & storage_engine = std::make_shared<ASTFunction>();
|
||||
|
||||
storage_engine->name = "MySQL";
|
||||
storage_engine->arguments = std::make_shared<ASTExpressionList>();
|
||||
storage_engine->children.push_back(storage_engine->arguments);
|
||||
|
||||
storage_engine->arguments->children = {
|
||||
std::make_shared<ASTLiteral>(host_name + ":" + toString(port)),
|
||||
std::make_shared<ASTLiteral>(database_name), std::make_shared<ASTLiteral>(table_name),
|
||||
std::make_shared<ASTLiteral>(user_name), std::make_shared<ASTLiteral>(password)
|
||||
};
|
||||
|
||||
|
||||
table_storage->set(table_storage->engine, storage_engine);
|
||||
return table_storage;
|
||||
}
|
||||
|
||||
DatabaseMySQL::MySQLStorageInfo DatabaseMySQL::createStorageInfo(
|
||||
const String & table_name, const NamesAndTypesList & columns_name_and_type, const UInt64 & table_modification_time) const
|
||||
{
|
||||
const auto & mysql_table = StorageMySQL::create(
|
||||
table_name, std::move(mysql_pool), mysql_database_name, table_name,
|
||||
false, "", ColumnsDescription{columns_name_and_type}, global_context);
|
||||
|
||||
const auto & create_table_query = std::make_shared<ASTCreateQuery>();
|
||||
|
||||
create_table_query->table = table_name;
|
||||
create_table_query->database = database_name;
|
||||
create_table_query->set(create_table_query->columns_list, getTableColumnsCreateQuery(columns_name_and_type));
|
||||
create_table_query->set(create_table_query->storage, getTableStorageCreateQuery(
|
||||
mysql_host_name, mysql_port, mysql_database_name, table_name, mysql_user_name, mysql_user_password));
|
||||
|
||||
MySQLStorageInfo storage_info;
|
||||
storage_info.storage = mysql_table;
|
||||
storage_info.create_table_query = create_table_query;
|
||||
storage_info.modification_time = table_modification_time;
|
||||
|
||||
return storage_info;
|
||||
}
|
||||
|
||||
std::map<String, UInt64> DatabaseMySQL::fetchTablesWithModificationTime() const
|
||||
{
|
||||
Block tables_status_sample_block
|
||||
{
|
||||
{ std::make_shared<DataTypeString>(), "table_name" },
|
||||
{ std::make_shared<DataTypeDateTime>(), "modification_time" },
|
||||
};
|
||||
|
||||
WriteBufferFromOwnString query;
|
||||
query << "SELECT"
|
||||
" TABLE_NAME AS table_name, "
|
||||
" CREATE_TIME AS modification_time "
|
||||
" FROM INFORMATION_SCHEMA.TABLES "
|
||||
" WHERE TABLE_SCHEMA = " << quote << mysql_database_name;
|
||||
|
||||
std::map<String, UInt64> tables_with_modification_time;
|
||||
MySQLBlockInputStream result(mysql_pool.Get(), query.str(), tables_status_sample_block, DEFAULT_BLOCK_SIZE);
|
||||
|
||||
while (Block block = result.read())
|
||||
{
|
||||
size_t rows = block.rows();
|
||||
for (size_t index = 0; index < rows; ++index)
|
||||
{
|
||||
String table_name = (*block.getByPosition(0).column)[index].safeGet<String>();
|
||||
tables_with_modification_time[table_name] = (*block.getByPosition(1).column)[index].safeGet<UInt64>();
|
||||
}
|
||||
}
|
||||
|
||||
return tables_with_modification_time;
|
||||
}
|
||||
|
||||
std::map<String, NamesAndTypesList> DatabaseMySQL::fetchTablesColumnsList(const std::vector<String> & tables_name) const
|
||||
{
|
||||
std::map<String, NamesAndTypesList> tables_and_columns;
|
||||
|
||||
if (tables_name.empty())
|
||||
return tables_and_columns;
|
||||
|
||||
Block tables_columns_sample_block
|
||||
{
|
||||
{ std::make_shared<DataTypeString>(), "table_name" },
|
||||
{ std::make_shared<DataTypeString>(), "column_name" },
|
||||
{ std::make_shared<DataTypeString>(), "column_type" },
|
||||
{ std::make_shared<DataTypeUInt8>(), "is_nullable" },
|
||||
{ std::make_shared<DataTypeUInt8>(), "is_unsigned" },
|
||||
{ std::make_shared<DataTypeUInt64>(), "length" },
|
||||
};
|
||||
|
||||
WriteBufferFromOwnString query;
|
||||
query << "SELECT "
|
||||
" TABLE_NAME AS table_name,"
|
||||
" COLUMN_NAME AS column_name,"
|
||||
" DATA_TYPE AS column_type,"
|
||||
" IS_NULLABLE = 'YES' AS is_nullable,"
|
||||
" COLUMN_TYPE LIKE '%unsigned' AS is_unsigned,"
|
||||
" CHARACTER_MAXIMUM_LENGTH AS length"
|
||||
" FROM INFORMATION_SCHEMA.COLUMNS"
|
||||
" WHERE TABLE_SCHEMA = " << quote << mysql_database_name
|
||||
<< " AND TABLE_NAME IN " << toQueryStringWithQuote(tables_name) << " ORDER BY ORDINAL_POSITION";
|
||||
|
||||
const auto & external_table_functions_use_nulls = global_context.getSettings().external_table_functions_use_nulls;
|
||||
MySQLBlockInputStream result(mysql_pool.Get(), query.str(), tables_columns_sample_block, DEFAULT_BLOCK_SIZE);
|
||||
while (Block block = result.read())
|
||||
{
|
||||
size_t rows = block.rows();
|
||||
for (size_t i = 0; i < rows; ++i)
|
||||
{
|
||||
String table_name = (*block.getByPosition(0).column)[i].safeGet<String>();
|
||||
tables_and_columns[table_name].emplace_back((*block.getByPosition(1).column)[i].safeGet<String>(),
|
||||
convertMySQLDataType(
|
||||
(*block.getByPosition(2).column)[i].safeGet<String>(),
|
||||
(*block.getByPosition(3).column)[i].safeGet<UInt64>() &&
|
||||
external_table_functions_use_nulls,
|
||||
(*block.getByPosition(4).column)[i].safeGet<UInt64>(),
|
||||
(*block.getByPosition(5).column)[i].safeGet<UInt64>()));
|
||||
}
|
||||
}
|
||||
return tables_and_columns;
|
||||
}
|
||||
|
||||
void DatabaseMySQL::shutdown()
|
||||
{
|
||||
std::map<String, MySQLStorageInfo> tables_snapshot;
|
||||
{
|
||||
std::lock_guard lock(mutex);
|
||||
tables_snapshot = local_tables_cache;
|
||||
}
|
||||
|
||||
for (const auto & table_snapshot : tables_snapshot)
|
||||
table_snapshot.second.storage->shutdown();
|
||||
|
||||
std::lock_guard lock(mutex);
|
||||
local_tables_cache.clear();
|
||||
}
|
||||
|
||||
void DatabaseMySQL::cleanOutdatedTables()
|
||||
{
|
||||
setThreadName("MySQLDBCleaner");
|
||||
|
||||
std::unique_lock lock{mutex};
|
||||
|
||||
while (!quit.load(std::memory_order_relaxed))
|
||||
{
|
||||
for (auto iterator = outdated_tables.begin(); iterator != outdated_tables.end();)
|
||||
{
|
||||
if (!iterator->unique())
|
||||
++iterator;
|
||||
else
|
||||
{
|
||||
const auto table_lock = (*iterator)->lockAlterIntention(RWLockImpl::NO_QUERY);
|
||||
|
||||
(*iterator)->shutdown();
|
||||
(*iterator)->is_dropped = true;
|
||||
iterator = outdated_tables.erase(iterator);
|
||||
}
|
||||
}
|
||||
|
||||
cond.wait_for(lock, cleaner_sleep_time);
|
||||
}
|
||||
}
|
||||
|
||||
DatabaseMySQL::~DatabaseMySQL()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!quit)
|
||||
{
|
||||
{
|
||||
quit = true;
|
||||
std::lock_guard lock{mutex};
|
||||
}
|
||||
cond.notify_one();
|
||||
thread.join();
|
||||
}
|
||||
|
||||
shutdown();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
#endif
|
||||
125
dbms/src/Databases/DatabaseMySQL.h
Normal file
125
dbms/src/Databases/DatabaseMySQL.h
Normal file
@ -0,0 +1,125 @@
|
||||
#pragma once
|
||||
|
||||
#include <Common/config.h>
|
||||
|
||||
#if USE_MYSQL
|
||||
|
||||
#include <mysqlxx/Pool.h>
|
||||
#include <Databases/DatabasesCommon.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/** Real-time access to table list and table structure from remote MySQL
|
||||
* It doesn't make any manipulations with filesystem.
|
||||
* All tables are created by calling code after real-time pull-out structure from remote MySQL
|
||||
*/
|
||||
class DatabaseMySQL : public IDatabase
|
||||
{
|
||||
public:
|
||||
~DatabaseMySQL() override;
|
||||
|
||||
DatabaseMySQL(const Context & context_, const String & database_name_, const String & mysql_host_name_, const UInt16 & mysql_port_,
|
||||
const String & mysql_database_name_, const String & mysql_user_name_, const String & mysql_user_password_);
|
||||
|
||||
String getEngineName() const override { return "MySQL"; }
|
||||
|
||||
String getDatabaseName() const override { return database_name; }
|
||||
|
||||
bool empty(const Context & context) const override;
|
||||
|
||||
DatabaseIteratorPtr getIterator(const Context & context) override;
|
||||
|
||||
ASTPtr getCreateDatabaseQuery(const Context & context) const override;
|
||||
|
||||
bool isTableExist(const Context & context, const String & name) const override;
|
||||
|
||||
StoragePtr tryGetTable(const Context & context, const String & name) const override;
|
||||
|
||||
ASTPtr tryGetCreateTableQuery(const Context & context, const String & name) const override;
|
||||
|
||||
time_t getTableMetadataModificationTime(const Context & context, const String & name) override;
|
||||
|
||||
void shutdown() override;
|
||||
|
||||
StoragePtr detachTable(const String &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support detach table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
void loadTables(Context &, ThreadPool *, bool) override
|
||||
{
|
||||
/// do nothing
|
||||
}
|
||||
|
||||
void removeTable(const Context &, const String &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support remove table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
void attachTable(const String &, const StoragePtr &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support attach table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
void renameTable(const Context &, const String &, IDatabase &, const String &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support rename table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
void createTable(const Context &, const String &, const StoragePtr &, const ASTPtr &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support create table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
void alterTable(const Context &, const String &, const ColumnsDescription &, const IndicesDescription &, const ASTModifier &) override
|
||||
{
|
||||
throw Exception("MySQL database engine does not support alter table.", ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
private:
|
||||
struct MySQLStorageInfo
|
||||
{
|
||||
StoragePtr storage;
|
||||
UInt64 modification_time;
|
||||
ASTPtr create_table_query;
|
||||
};
|
||||
|
||||
const Context global_context;
|
||||
const String database_name;
|
||||
const String mysql_host_name;
|
||||
const UInt16 mysql_port;
|
||||
const String mysql_database_name;
|
||||
const String mysql_user_name;
|
||||
const String mysql_user_password;
|
||||
|
||||
mutable std::mutex mutex;
|
||||
std::atomic<bool> quit{false};
|
||||
std::condition_variable cond;
|
||||
|
||||
mutable mysqlxx::Pool mysql_pool;
|
||||
mutable std::vector<StoragePtr> outdated_tables;
|
||||
mutable std::map<String, MySQLStorageInfo> local_tables_cache;
|
||||
|
||||
|
||||
void cleanOutdatedTables();
|
||||
|
||||
void fetchTablesIntoLocalCache() const;
|
||||
|
||||
std::map<String, UInt64> fetchTablesWithModificationTime() const;
|
||||
|
||||
DatabaseMySQL::MySQLStorageInfo createStorageInfo(
|
||||
const String & table_name, const NamesAndTypesList & columns_name_and_type, const UInt64 & table_modification_time) const;
|
||||
|
||||
std::map<String, NamesAndTypesList> fetchTablesColumnsList(const std::vector<String> & tables_name) const;
|
||||
|
||||
void destroyLocalCacheExtraTables(const std::map<String, UInt64> & tables_with_modification_time) const;
|
||||
|
||||
void fetchLatestTablesStructureIntoCache(const std::map<String, UInt64> & tables_modification_time) const;
|
||||
|
||||
ThreadFromGlobalPool thread{&DatabaseMySQL::cleanOutdatedTables, this};
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
#endif
|
||||
@ -37,7 +37,7 @@ namespace
|
||||
return DictionaryTypedSpecialAttribute{std::move(name), std::move(expression), DataTypeFactory::instance().get(type_name)};
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
|
||||
AttributeUnderlyingType getAttributeUnderlyingType(const std::string & type)
|
||||
|
||||
@ -69,5 +69,5 @@ IRegionsHierarchyDataSourcePtr RegionsHierarchiesDataProvider::getHierarchySourc
|
||||
return std::make_shared<RegionsHierarchyDataSource>(hierarchy_file_path);
|
||||
}
|
||||
|
||||
throw Poco::Exception("Regions hierarchy `" + name + "` not found");
|
||||
throw Poco::Exception("Regions hierarchy '" + name + "' not found");
|
||||
}
|
||||
|
||||
@ -39,7 +39,7 @@ const DB::IColumn & unwrapNullableColumn(const DB::IColumn & column)
|
||||
return column;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
@ -3,13 +3,13 @@
|
||||
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Formats/CapnProtoRowInputStream.h> // Y_IGNORE
|
||||
#include <Formats/CapnProtoRowInputStream.h>
|
||||
#include <Formats/FormatFactory.h>
|
||||
#include <Formats/BlockInputStreamFromRowInputStream.h>
|
||||
#include <Formats/FormatSchemaInfo.h>
|
||||
#include <capnp/serialize.h> // Y_IGNORE
|
||||
#include <capnp/dynamic.h> // Y_IGNORE
|
||||
#include <capnp/common.h> // Y_IGNORE
|
||||
#include <capnp/serialize.h>
|
||||
#include <capnp/dynamic.h>
|
||||
#include <capnp/common.h>
|
||||
#include <boost/algorithm/string.hpp>
|
||||
#include <boost/range/join.hpp>
|
||||
#include <common/logger_useful.h>
|
||||
|
||||
@ -7,7 +7,6 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
|
||||
void PrettySpaceBlockOutputStream::write(const Block & block)
|
||||
{
|
||||
UInt64 max_rows = format_settings.pretty.max_rows;
|
||||
@ -26,12 +25,6 @@ void PrettySpaceBlockOutputStream::write(const Block & block)
|
||||
Widths name_widths;
|
||||
calculateWidths(block, widths, max_widths, name_widths, format_settings);
|
||||
|
||||
/// Do not align on too long values.
|
||||
if (terminal_width > 80)
|
||||
for (size_t i = 0; i < columns; ++i)
|
||||
if (max_widths[i] > terminal_width / 2)
|
||||
max_widths[i] = terminal_width / 2;
|
||||
|
||||
/// Names
|
||||
for (size_t i = 0; i < columns; ++i)
|
||||
{
|
||||
|
||||
@ -2,8 +2,8 @@
|
||||
#if USE_PROTOBUF
|
||||
|
||||
#include <Formats/FormatSchemaInfo.h>
|
||||
#include <Formats/ProtobufSchemas.h> // Y_IGNORE
|
||||
#include <google/protobuf/compiler/importer.h> // Y_IGNORE
|
||||
#include <Formats/ProtobufSchemas.h>
|
||||
#include <google/protobuf/compiler/importer.h>
|
||||
#include <Common/Exception.h>
|
||||
|
||||
|
||||
|
||||
@ -7,8 +7,8 @@
|
||||
#include <AggregateFunctions/IAggregateFunction.h>
|
||||
#include <DataTypes/DataTypesDecimal.h>
|
||||
#include <boost/numeric/conversion/cast.hpp>
|
||||
#include <google/protobuf/descriptor.h> // Y_IGNORE
|
||||
#include <google/protobuf/descriptor.pb.h> // Y_IGNORE
|
||||
#include <google/protobuf/descriptor.h>
|
||||
#include <google/protobuf/descriptor.pb.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include "ProtobufWriter.h"
|
||||
|
||||
@ -7,7 +7,7 @@
|
||||
#include <Functions/FunctionHelpers.h>
|
||||
#include <Functions/GatherUtils/Algorithms.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <libbase64.h> // Y_IGNORE
|
||||
#include <libbase64.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -24,7 +24,7 @@
|
||||
#if USE_EMBEDDED_COMPILER
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
|
||||
@ -22,8 +22,8 @@
|
||||
#pragma clang diagnostic ignored "-Wshift-negative-value"
|
||||
#endif
|
||||
|
||||
#include <vectorf128.h> // Y_IGNORE
|
||||
#include <vectormath_exp.h> // Y_IGNORE
|
||||
#include <vectorf128.h>
|
||||
#include <vectormath_exp.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic pop
|
||||
|
||||
@ -21,9 +21,9 @@
|
||||
#pragma clang diagnostic ignored "-Wshift-negative-value"
|
||||
#endif
|
||||
|
||||
#include <vectorf128.h> // Y_IGNORE
|
||||
#include <vectormath_exp.h> // Y_IGNORE
|
||||
#include <vectormath_trig.h> // Y_IGNORE
|
||||
#include <vectorf128.h>
|
||||
#include <vectormath_exp.h>
|
||||
#include <vectormath_trig.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic pop
|
||||
|
||||
@ -13,7 +13,7 @@
|
||||
#if USE_EMBEDDED_COMPILER
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
|
||||
@ -23,5 +23,6 @@ void registerFunctionsBitmap(FunctionFactory & factory)
|
||||
|
||||
factory.registerFunction<FunctionBitmapHasAll>();
|
||||
factory.registerFunction<FunctionBitmapHasAny>();
|
||||
factory.registerFunction<FunctionBitmapContains>();
|
||||
}
|
||||
}
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
#include <Columns/ColumnArray.h>
|
||||
#include <Columns/ColumnConst.h>
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <DataTypes/DataTypeAggregateFunction.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
@ -55,6 +56,15 @@ namespace ErrorCodes
|
||||
*
|
||||
* Two bitmap andnot calculation, return cardinality:
|
||||
* bitmapAndnotCardinality: bitmap,bitmap -> integer
|
||||
*
|
||||
* Determine if a bitmap contains the given integer:
|
||||
* bitmapContains: bitmap,integer -> bool
|
||||
*
|
||||
* Judge if a bitmap is superset of the another one:
|
||||
* bitmapHasAll: bitmap,bitmap -> bool
|
||||
*
|
||||
* Judge if the intersection of two bitmap is nonempty:
|
||||
* bitmapHasAny: bitmap,bitmap -> bool
|
||||
*/
|
||||
|
||||
template <typename Name>
|
||||
@ -362,6 +372,95 @@ struct BitmapHasAnyImpl
|
||||
}
|
||||
};
|
||||
|
||||
class FunctionBitmapContains : public IFunction
|
||||
{
|
||||
public:
|
||||
static constexpr auto name = "bitmapContains";
|
||||
|
||||
static FunctionPtr create(const Context &) { return std::make_shared<FunctionBitmapContains>(); }
|
||||
|
||||
String getName() const override { return name; }
|
||||
|
||||
bool isVariadic() const override { return false; }
|
||||
|
||||
size_t getNumberOfArguments() const override { return 2; }
|
||||
|
||||
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
|
||||
{
|
||||
auto bitmap_type0 = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
|
||||
if (!(bitmap_type0 && bitmap_type0->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
|
||||
throw Exception(
|
||||
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
|
||||
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
auto arg_type1 = typeid_cast<const DataTypeNumber<UInt32> *>(arguments[1].get());
|
||||
if (!(arg_type1))
|
||||
throw Exception(
|
||||
"Second argument for function " + getName() + " must be UInt32 but it has type " + arguments[1]->getName() + ".",
|
||||
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
|
||||
return std::make_shared<DataTypeNumber<UInt8>>();
|
||||
}
|
||||
|
||||
bool useDefaultImplementationForConstants() const override { return true; }
|
||||
|
||||
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
|
||||
{
|
||||
auto col_to = ColumnVector<UInt8>::create(input_rows_count);
|
||||
typename ColumnVector<UInt8>::Container & vec_to = col_to->getData();
|
||||
const IDataType * from_type = block.getByPosition(arguments[0]).type.get();
|
||||
|
||||
const DataTypeAggregateFunction * aggr_type = typeid_cast<const DataTypeAggregateFunction *>(from_type);
|
||||
WhichDataType which(aggr_type->getArgumentsDataTypes()[0]);
|
||||
if (which.isUInt8())
|
||||
executeIntType<UInt8>(block, arguments, input_rows_count, vec_to);
|
||||
else if (which.isUInt16())
|
||||
executeIntType<UInt16>(block, arguments, input_rows_count, vec_to);
|
||||
else if (which.isUInt32())
|
||||
executeIntType<UInt32>(block, arguments, input_rows_count, vec_to);
|
||||
else if (which.isUInt64())
|
||||
executeIntType<UInt64>(block, arguments, input_rows_count, vec_to);
|
||||
else
|
||||
throw Exception(
|
||||
"Unexpected type " + from_type->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
|
||||
block.getByPosition(result).column = std::move(col_to);
|
||||
}
|
||||
|
||||
private:
|
||||
template <typename T>
|
||||
void executeIntType(
|
||||
Block & block, const ColumnNumbers & arguments, size_t input_rows_count, typename ColumnVector<UInt8>::Container & vec_to)
|
||||
{
|
||||
const IColumn * columns[2];
|
||||
bool isColumnConst[2];
|
||||
const PaddedPODArray<AggregateDataPtr> * container0;
|
||||
const PaddedPODArray<UInt32> * container1;
|
||||
|
||||
for (size_t i = 0; i < 2; ++i)
|
||||
{
|
||||
columns[i] = block.getByPosition(arguments[i]).column.get();
|
||||
isColumnConst[i] = typeid_cast<const ColumnConst*>(columns[i])!=nullptr;
|
||||
}
|
||||
if (isColumnConst[0])
|
||||
container0 = &typeid_cast<const ColumnAggregateFunction*>(typeid_cast<const ColumnConst*>(columns[0])->getDataColumnPtr().get())->getData();
|
||||
else
|
||||
container0 = &typeid_cast<const ColumnAggregateFunction*>(columns[0])->getData();
|
||||
if (isColumnConst[1])
|
||||
container1 = &typeid_cast<const ColumnUInt32*>(typeid_cast<const ColumnConst*>(columns[1])->getDataColumnPtr().get())->getData();
|
||||
else
|
||||
container1 = &typeid_cast<const ColumnUInt32*>(columns[1])->getData();
|
||||
|
||||
for (size_t i = 0; i < input_rows_count; ++i)
|
||||
{
|
||||
const AggregateDataPtr dataPtr0 = isColumnConst[0] ? (*container0)[0] : (*container0)[i];
|
||||
const UInt32 data1 = isColumnConst[1] ? (*container1)[0] : (*container1)[i];
|
||||
const AggregateFunctionGroupBitmapData<T>& bd0
|
||||
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr0);
|
||||
vec_to[i] = bd0.rbs.rb_contains(data1);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
template <template <typename> class Impl, typename Name, typename ToType>
|
||||
class FunctionBitmapCardinality : public IFunction
|
||||
{
|
||||
@ -430,20 +529,32 @@ private:
|
||||
Block & block, const ColumnNumbers & arguments, size_t input_rows_count, typename ColumnVector<ToType>::Container & vec_to)
|
||||
{
|
||||
const ColumnAggregateFunction * columns[2];
|
||||
bool isColumnConst[2];
|
||||
for (size_t i = 0; i < 2; ++i)
|
||||
{
|
||||
if (auto argument_column_const = typeid_cast<const ColumnConst*>(block.getByPosition(arguments[i]).column.get()))
|
||||
{
|
||||
columns[i] = typeid_cast<const ColumnAggregateFunction*>(argument_column_const->getDataColumnPtr().get());
|
||||
else
|
||||
columns[i] = typeid_cast<const ColumnAggregateFunction *>(block.getByPosition(arguments[i]).column.get());
|
||||
isColumnConst[i] = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
columns[i] = typeid_cast<const ColumnAggregateFunction*>(block.getByPosition(arguments[i]).column.get());
|
||||
isColumnConst[i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
const PaddedPODArray<AggregateDataPtr> & container0 = columns[0]->getData();
|
||||
const PaddedPODArray<AggregateDataPtr> & container1 = columns[1]->getData();
|
||||
|
||||
for (size_t i = 0; i < input_rows_count; ++i)
|
||||
{
|
||||
const AggregateDataPtr dataPtr0 = isColumnConst[0] ? container0[0] : container0[i];
|
||||
const AggregateDataPtr dataPtr1 = isColumnConst[1] ? container1[0] : container1[i];
|
||||
const AggregateFunctionGroupBitmapData<T> & bd1
|
||||
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(columns[0]->getData()[i]);
|
||||
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr0);
|
||||
const AggregateFunctionGroupBitmapData<T> & bd2
|
||||
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(columns[1]->getData()[i]);
|
||||
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr1);
|
||||
vec_to[i] = Impl<T>::apply(bd1, bd2);
|
||||
}
|
||||
}
|
||||
|
||||
@ -335,7 +335,7 @@ public:
|
||||
const IColumn * longitude = block.getByPosition(arguments[0]).column.get();
|
||||
const IColumn * latitude = block.getByPosition(arguments[1]).column.get();
|
||||
|
||||
const UInt64 precision_value = std::min(GEOHASH_MAX_TEXT_LENGTH,
|
||||
const UInt64 precision_value = std::min<UInt64>(GEOHASH_MAX_TEXT_LENGTH,
|
||||
arguments.size() == 3 ? block.getByPosition(arguments[2]).column->get64(0) : GEOHASH_MAX_TEXT_LENGTH);
|
||||
|
||||
ColumnPtr & res_column = block.getByPosition(result).column;
|
||||
|
||||
@ -12,7 +12,7 @@
|
||||
|
||||
#include <Common/config.h>
|
||||
#if USE_XXHASH
|
||||
# include <xxhash.h> // Y_IGNORE
|
||||
# include <xxhash.h>
|
||||
#endif
|
||||
|
||||
#if USE_SSL
|
||||
|
||||
@ -18,7 +18,7 @@
|
||||
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@
|
||||
#endif
|
||||
|
||||
#if USE_RE2_ST
|
||||
# include <re2_st/re2.h> // Y_IGNORE
|
||||
# include <re2_st/re2.h>
|
||||
#else
|
||||
# define re2_st re2
|
||||
#endif
|
||||
|
||||
@ -307,7 +307,26 @@ struct MultiSearchAllPositionsImpl
|
||||
{
|
||||
return 1 + Impl::countChars(reinterpret_cast<const char *>(start), reinterpret_cast<const char *>(end));
|
||||
};
|
||||
Impl::createMultiSearcherInBigHaystack(needles).searchAllPositions(haystack_data, haystack_offsets, res_callback, res);
|
||||
|
||||
auto searcher = Impl::createMultiSearcherInBigHaystack(needles);
|
||||
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
const size_t needles_size = needles.size();
|
||||
|
||||
/// Something can be uninitialized after the search itself
|
||||
std::fill(res.begin(), res.end(), 0);
|
||||
|
||||
while (searcher.hasMoreToSearch())
|
||||
{
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0, from = 0; j < haystack_string_size; ++j, from += needles_size)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
searcher.searchOneAll(haystack, haystack_end, res.data() + from, res_callback);
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@ -323,7 +342,22 @@ struct MultiSearchImpl
|
||||
const std::vector<StringRef> & needles,
|
||||
PaddedPODArray<UInt8> & res)
|
||||
{
|
||||
Impl::createMultiSearcherInBigHaystack(needles).search(haystack_data, haystack_offsets, res);
|
||||
auto searcher = Impl::createMultiSearcherInBigHaystack(needles);
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
size_t iteration = 0;
|
||||
while (searcher.hasMoreToSearch())
|
||||
{
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0; j < haystack_string_size; ++j)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
if (iteration == 0 || !res[j])
|
||||
res[j] = searcher.searchOne(haystack, haystack_end);
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
++iteration;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@ -343,7 +377,28 @@ struct MultiSearchFirstPositionImpl
|
||||
{
|
||||
return 1 + Impl::countChars(reinterpret_cast<const char *>(start), reinterpret_cast<const char *>(end));
|
||||
};
|
||||
Impl::createMultiSearcherInBigHaystack(needles).searchFirstPosition(haystack_data, haystack_offsets, res_callback, res);
|
||||
auto searcher = Impl::createMultiSearcherInBigHaystack(needles);
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
size_t iteration = 0;
|
||||
while (searcher.hasMoreToSearch())
|
||||
{
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0; j < haystack_string_size; ++j)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
if (iteration == 0 || res[j] == 0)
|
||||
res[j] = searcher.searchOneFirstPosition(haystack, haystack_end, res_callback);
|
||||
else
|
||||
{
|
||||
UInt64 result = searcher.searchOneFirstPosition(haystack, haystack_end, res_callback);
|
||||
if (result != 0)
|
||||
res[j] = std::min(result, res[j]);
|
||||
}
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
++iteration;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@ -359,7 +414,23 @@ struct MultiSearchFirstIndexImpl
|
||||
const std::vector<StringRef> & needles,
|
||||
PaddedPODArray<UInt64> & res)
|
||||
{
|
||||
Impl::createMultiSearcherInBigHaystack(needles).searchIndex(haystack_data, haystack_offsets, res);
|
||||
auto searcher = Impl::createMultiSearcherInBigHaystack(needles);
|
||||
const size_t haystack_string_size = haystack_offsets.size();
|
||||
size_t iteration = 0;
|
||||
while (searcher.hasMoreToSearch())
|
||||
{
|
||||
size_t prev_offset = 0;
|
||||
for (size_t j = 0; j < haystack_string_size; ++j)
|
||||
{
|
||||
const auto * haystack = &haystack_data[prev_offset];
|
||||
const auto * haystack_end = haystack + haystack_offsets[j] - prev_offset - 1;
|
||||
/// hasMoreToSearch traverse needles in increasing order
|
||||
if (iteration == 0 || res[j] == 0)
|
||||
res[j] = searcher.searchOneFirstIndex(haystack, haystack_end);
|
||||
prev_offset = haystack_offsets[j];
|
||||
}
|
||||
++iteration;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -187,7 +187,7 @@ inline Encoded base32Decode(const char * encoded_string, size_t encoded_length)
|
||||
return result;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -227,6 +227,6 @@ void geohashDecode(const char * encoded_string, size_t encoded_len, Float64 * lo
|
||||
*latitude = decodeCoordinate(lat_encoded, -90, 90, singleCoordBitsPrecision(precision, LATITUDE));
|
||||
}
|
||||
|
||||
} // namespace GeoUtils
|
||||
}
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -25,7 +25,7 @@
|
||||
#if USE_EMBEDDED_COMPILER
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
|
||||
@ -234,8 +234,8 @@ void FunctionArrayEnumerateRankedExtended<Derived>::executeImpl(
|
||||
{
|
||||
throw Exception(
|
||||
getName() + ": Passed array number " + std::to_string(array_num) + " depth ("
|
||||
+ std::to_string(arrays_depths.depths[array_num]) + ") is more than the actual array depth (" + std::to_string(col_depth)
|
||||
+ ").",
|
||||
+ std::to_string(arrays_depths.depths[array_num]) + ") is more than the actual array depth ("
|
||||
+ std::to_string(col_depth) + ").",
|
||||
ErrorCodes::SIZES_OF_ARRAYS_DOESNT_MATCH);
|
||||
}
|
||||
|
||||
@ -255,7 +255,7 @@ void FunctionArrayEnumerateRankedExtended<Derived>::executeImpl(
|
||||
executeMethodImpl(offsets_by_depth, data_columns, arrays_depths, res_values);
|
||||
|
||||
ColumnPtr result_nested_array = std::move(res_nested);
|
||||
for (int depth = arrays_depths.max_array_depth - 1; depth >= 0; --depth)
|
||||
for (ssize_t depth = arrays_depths.max_array_depth - 1; depth >= 0; --depth)
|
||||
result_nested_array = ColumnArray::create(std::move(result_nested_array), offsetsptr_by_depth[depth]);
|
||||
|
||||
block.getByPosition(result).column = result_nested_array;
|
||||
@ -321,6 +321,7 @@ void FunctionArrayEnumerateRankedExtended<Derived>::executeMethodImpl(
|
||||
|
||||
std::vector<size_t> indices_by_depth(arrays_depths.max_array_depth);
|
||||
std::vector<size_t> current_offset_n_by_depth(arrays_depths.max_array_depth);
|
||||
std::vector<size_t> last_offset_by_depth(arrays_depths.max_array_depth, 0); // For skipping empty arrays
|
||||
|
||||
UInt32 rank = 0;
|
||||
|
||||
@ -330,6 +331,24 @@ void FunctionArrayEnumerateRankedExtended<Derived>::executeMethodImpl(
|
||||
{
|
||||
bool want_clear = false;
|
||||
|
||||
/// Skipping offsets if no data in this array
|
||||
if (prev_off == off)
|
||||
{
|
||||
want_clear = true;
|
||||
++indices_by_depth[0];
|
||||
|
||||
for (ssize_t depth = current_offset_depth - 1; depth >= 0; --depth)
|
||||
{
|
||||
const auto offsets_by_depth_size = offsets_by_depth[depth]->size();
|
||||
while (last_offset_by_depth[depth] == (*offsets_by_depth[depth])[current_offset_n_by_depth[depth]])
|
||||
{
|
||||
if (current_offset_n_by_depth[depth] + 1 >= offsets_by_depth_size)
|
||||
break; // only one empty array: SELECT arrayEnumerateUniqRanked([]);
|
||||
++current_offset_n_by_depth[depth];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// For each element at the depth we want to look.
|
||||
for (size_t j = prev_off; j < off; ++j)
|
||||
{
|
||||
@ -356,14 +375,21 @@ void FunctionArrayEnumerateRankedExtended<Derived>::executeMethodImpl(
|
||||
|
||||
// Debug: DUMP(off, prev_off, j, columns_indices, res_values[j], columns);
|
||||
|
||||
for (int depth = current_offset_depth - 1; depth >= 0; --depth)
|
||||
for (ssize_t depth = current_offset_depth - 1; depth >= 0; --depth)
|
||||
{
|
||||
/// Skipping offsets for empty arrays
|
||||
while (last_offset_by_depth[depth] == (*offsets_by_depth[depth])[current_offset_n_by_depth[depth]])
|
||||
{
|
||||
++current_offset_n_by_depth[depth];
|
||||
}
|
||||
|
||||
++indices_by_depth[depth];
|
||||
|
||||
if (indices_by_depth[depth] == (*offsets_by_depth[depth])[current_offset_n_by_depth[depth]])
|
||||
{
|
||||
if (static_cast<int>(arrays_depths.clear_depth) == depth + 1)
|
||||
want_clear = true;
|
||||
last_offset_by_depth[depth] = (*offsets_by_depth[depth])[current_offset_n_by_depth[depth]];
|
||||
++current_offset_n_by_depth[depth];
|
||||
}
|
||||
else
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class FunctionArrayEnumerateUniqRanked : public FunctionArrayEnumerateRankedExtended<FunctionArrayEnumerateUniqRanked>
|
||||
{
|
||||
using Base = FunctionArrayEnumerateRankedExtended<FunctionArrayEnumerateUniqRanked>;
|
||||
|
||||
@ -53,10 +53,10 @@ public:
|
||||
|
||||
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
|
||||
{
|
||||
if (!arguments.size())
|
||||
if (arguments.empty())
|
||||
throw Exception("Function " + getName() + " requires at least one argument", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
const DataTypeAggregateFunction * type = checkAndGetDataType<DataTypeAggregateFunction>(arguments[0].get());
|
||||
const auto * type = checkAndGetDataType<DataTypeAggregateFunction>(arguments[0].get());
|
||||
if (!type)
|
||||
throw Exception("Argument for function " + getName() + " must have type AggregateFunction - state of aggregate function.",
|
||||
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
@ -66,19 +66,21 @@ public:
|
||||
|
||||
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
|
||||
{
|
||||
if (!arguments.size())
|
||||
if (arguments.empty())
|
||||
throw Exception("Function " + getName() + " requires at least one argument", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
const ColumnConst * column_with_states
|
||||
= typeid_cast<const ColumnConst *>(&*block.getByPosition(arguments[0]).column);
|
||||
const auto * model = block.getByPosition(arguments[0]).column.get();
|
||||
|
||||
if (const auto * column_with_states = typeid_cast<const ColumnConst *>(model))
|
||||
model = column_with_states->getDataColumnPtr().get();
|
||||
|
||||
if (!column_with_states)
|
||||
const auto * agg_function = typeid_cast<const ColumnAggregateFunction *>(model);
|
||||
|
||||
if (!agg_function)
|
||||
throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName()
|
||||
+ " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN);
|
||||
|
||||
block.getByPosition(result).column =
|
||||
typeid_cast<const ColumnAggregateFunction *>(&*column_with_states->getDataColumnPtr())->predictValues(block, arguments, context);
|
||||
block.getByPosition(result).column = agg_function->predictValues(block, arguments, context);
|
||||
}
|
||||
|
||||
const Context & context;
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
#if USE_BROTLI
|
||||
|
||||
#include "BrotliReadBuffer.h"
|
||||
#include <brotli/decode.h> // Y_IGNORE
|
||||
#include <brotli/decode.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
@ -49,4 +49,4 @@ private:
|
||||
bool put_delimiter = false;
|
||||
};
|
||||
|
||||
} // namespace DB
|
||||
}
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
#include <Poco/URI.h>
|
||||
|
||||
#if USE_HDFS
|
||||
#include <hdfs/hdfs.h> // Y_IGNORE
|
||||
#include <hdfs/hdfs.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
@ -158,6 +158,26 @@ void setTimeouts(Poco::Net::HTTPClientSession & session, const ConnectionTimeout
|
||||
auto retry_timeout = timeouts.connection_timeout.totalMicroseconds();
|
||||
auto session = pool_ptr->second->get(retry_timeout);
|
||||
|
||||
/// We store exception messages in session data.
|
||||
/// Poco HTTPSession also stores exception, but it can be removed at any time.
|
||||
const auto & sessionData = session->sessionData();
|
||||
if (!sessionData.empty())
|
||||
{
|
||||
auto msg = Poco::AnyCast<std::string>(sessionData);
|
||||
if (!msg.empty())
|
||||
{
|
||||
LOG_TRACE((&Logger::get("HTTPCommon")), "Failed communicating with " << host << " with error '" << msg << "' will try to reconnect session");
|
||||
/// Host can change IP
|
||||
const auto ip = DNSResolver::instance().resolveHost(host).toString();
|
||||
if (ip != session->getHost())
|
||||
{
|
||||
session->reset();
|
||||
session->setHost(ip);
|
||||
session->attachSessionData({});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
setTimeouts(*session, timeouts);
|
||||
|
||||
return session;
|
||||
|
||||
@ -2,10 +2,10 @@
|
||||
|
||||
#if USE_HDFS
|
||||
|
||||
#include <IO/ReadBufferFromHDFS.h> // Y_IGNORE
|
||||
#include <IO/ReadBufferFromHDFS.h>
|
||||
#include <IO/HDFSCommon.h>
|
||||
#include <Poco/URI.h>
|
||||
#include <hdfs/hdfs.h> // Y_IGNORE
|
||||
#include <hdfs/hdfs.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
#include <IO/HTTPCommon.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/ReadBufferFromIStream.h>
|
||||
#include <Poco/Any.h>
|
||||
#include <Poco/Net/HTTPBasicCredentials.h>
|
||||
#include <Poco/Net/HTTPClientSession.h>
|
||||
#include <Poco/Net/HTTPRequest.h>
|
||||
@ -69,6 +70,8 @@ namespace detail
|
||||
|
||||
LOG_TRACE((&Logger::get("ReadWriteBufferFromHTTP")), "Sending request to " << uri.toString());
|
||||
|
||||
try
|
||||
{
|
||||
auto & stream_out = session->sendRequest(request);
|
||||
|
||||
if (out_stream_callback)
|
||||
@ -78,6 +81,14 @@ namespace detail
|
||||
|
||||
impl = std::make_unique<ReadBufferFromIStream>(*istr, buffer_size_);
|
||||
}
|
||||
catch (const Poco::Exception & e)
|
||||
{
|
||||
/// We use session data storage as storage for exception text
|
||||
/// Depend on it we can deduce to reconnect session or reresolve session host
|
||||
session->attachSessionData(e.message());
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
bool nextImpl() override
|
||||
|
||||
@ -3,9 +3,9 @@
|
||||
#if USE_HDFS
|
||||
|
||||
#include <Poco/URI.h>
|
||||
#include <IO/WriteBufferFromHDFS.h> // Y_IGNORE
|
||||
#include <IO/WriteBufferFromHDFS.h>
|
||||
#include <IO/HDFSCommon.h>
|
||||
#include <hdfs/hdfs.h> // Y_IGNORE
|
||||
#include <hdfs/hdfs.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
#endif
|
||||
|
||||
#if USE_TCMALLOC
|
||||
#include <gperftools/malloc_extension.h> // Y_IGNORE
|
||||
#include <gperftools/malloc_extension.h>
|
||||
|
||||
/// Initializing malloc extension in global constructor as required.
|
||||
struct MallocExtensionInitializer
|
||||
|
||||
@ -197,7 +197,7 @@ void Clusters::updateClusters(const Poco::Util::AbstractConfiguration & config,
|
||||
for (const auto & key : config_keys)
|
||||
{
|
||||
if (key.find('.') != String::npos)
|
||||
throw Exception("Cluster names with dots are not supported: `" + key + "`", ErrorCodes::SYNTAX_ERROR);
|
||||
throw Exception("Cluster names with dots are not supported: '" + key + "'", ErrorCodes::SYNTAX_ERROR);
|
||||
|
||||
impl.emplace(key, std::make_shared<Cluster>(config, settings, config_name + "." + key));
|
||||
}
|
||||
|
||||
@ -1524,7 +1524,7 @@ UInt16 Context::getTCPPort() const
|
||||
auto lock = getLock();
|
||||
|
||||
auto & config = getConfigRef();
|
||||
return config.getInt("tcp_port");
|
||||
return config.getInt("tcp_port", DBMS_DEFAULT_PORT);
|
||||
}
|
||||
|
||||
std::optional<UInt16> Context::getTCPPortSecure() const
|
||||
|
||||
@ -200,16 +200,16 @@ static std::unique_ptr<zkutil::Lock> createSimpleZooKeeperLock(
|
||||
|
||||
static bool isSupportedAlterType(int type)
|
||||
{
|
||||
static const std::unordered_set<int> supported_alter_types{
|
||||
ASTAlterCommand::ADD_COLUMN,
|
||||
ASTAlterCommand::DROP_COLUMN,
|
||||
ASTAlterCommand::MODIFY_COLUMN,
|
||||
ASTAlterCommand::DROP_PARTITION,
|
||||
ASTAlterCommand::DELETE,
|
||||
ASTAlterCommand::UPDATE,
|
||||
static const std::unordered_set<int> unsupported_alter_types{
|
||||
ASTAlterCommand::ATTACH_PARTITION,
|
||||
ASTAlterCommand::REPLACE_PARTITION,
|
||||
ASTAlterCommand::FETCH_PARTITION,
|
||||
ASTAlterCommand::FREEZE_PARTITION,
|
||||
ASTAlterCommand::FREEZE_ALL,
|
||||
ASTAlterCommand::NO_TYPE,
|
||||
};
|
||||
|
||||
return supported_alter_types.count(type) != 0;
|
||||
return unsupported_alter_types.count(type) == 0;
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -19,38 +19,30 @@
|
||||
#pragma GCC diagnostic ignored "-Wunused-parameter"
|
||||
#pragma GCC diagnostic ignored "-Wnon-virtual-dtor"
|
||||
|
||||
/** Y_IGNORE marker means that this header is not analyzed by Arcadia build system.
|
||||
* "Arcadia" is the name of internal Yandex source code repository.
|
||||
* ClickHouse have limited support for build in Arcadia
|
||||
* (ClickHouse source code is used in another Yandex products as a library).
|
||||
* Some libraries are not enabled when build inside Arcadia is used,
|
||||
* that what does Y_IGNORE indicate.
|
||||
*/
|
||||
|
||||
#include <llvm/Analysis/TargetTransformInfo.h> // Y_IGNORE
|
||||
#include <llvm/Config/llvm-config.h> // Y_IGNORE
|
||||
#include <llvm/IR/BasicBlock.h> // Y_IGNORE
|
||||
#include <llvm/IR/DataLayout.h> // Y_IGNORE
|
||||
#include <llvm/IR/DerivedTypes.h> // Y_IGNORE
|
||||
#include <llvm/IR/Function.h> // Y_IGNORE
|
||||
#include <llvm/IR/IRBuilder.h> // Y_IGNORE
|
||||
#include <llvm/IR/LLVMContext.h> // Y_IGNORE
|
||||
#include <llvm/IR/Mangler.h> // Y_IGNORE
|
||||
#include <llvm/IR/Module.h> // Y_IGNORE
|
||||
#include <llvm/IR/Type.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/ExecutionEngine.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/JITSymbol.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/SectionMemoryManager.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/Orc/CompileUtils.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/Orc/IRCompileLayer.h> // Y_IGNORE
|
||||
#include <llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h> // Y_IGNORE
|
||||
#include <llvm/Target/TargetMachine.h> // Y_IGNORE
|
||||
#include <llvm/MC/SubtargetFeature.h> // Y_IGNORE
|
||||
#include <llvm/Support/DynamicLibrary.h> // Y_IGNORE
|
||||
#include <llvm/Support/Host.h> // Y_IGNORE
|
||||
#include <llvm/Support/TargetRegistry.h> // Y_IGNORE
|
||||
#include <llvm/Support/TargetSelect.h> // Y_IGNORE
|
||||
#include <llvm/Transforms/IPO/PassManagerBuilder.h> // Y_IGNORE
|
||||
#include <llvm/Analysis/TargetTransformInfo.h>
|
||||
#include <llvm/Config/llvm-config.h>
|
||||
#include <llvm/IR/BasicBlock.h>
|
||||
#include <llvm/IR/DataLayout.h>
|
||||
#include <llvm/IR/DerivedTypes.h>
|
||||
#include <llvm/IR/Function.h>
|
||||
#include <llvm/IR/IRBuilder.h>
|
||||
#include <llvm/IR/LLVMContext.h>
|
||||
#include <llvm/IR/Mangler.h>
|
||||
#include <llvm/IR/Module.h>
|
||||
#include <llvm/IR/Type.h>
|
||||
#include <llvm/ExecutionEngine/ExecutionEngine.h>
|
||||
#include <llvm/ExecutionEngine/JITSymbol.h>
|
||||
#include <llvm/ExecutionEngine/SectionMemoryManager.h>
|
||||
#include <llvm/ExecutionEngine/Orc/CompileUtils.h>
|
||||
#include <llvm/ExecutionEngine/Orc/IRCompileLayer.h>
|
||||
#include <llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h>
|
||||
#include <llvm/Target/TargetMachine.h>
|
||||
#include <llvm/MC/SubtargetFeature.h>
|
||||
#include <llvm/Support/DynamicLibrary.h>
|
||||
#include <llvm/Support/Host.h>
|
||||
#include <llvm/Support/TargetRegistry.h>
|
||||
#include <llvm/Support/TargetSelect.h>
|
||||
#include <llvm/Transforms/IPO/PassManagerBuilder.h>
|
||||
|
||||
#pragma GCC diagnostic pop
|
||||
|
||||
|
||||
@ -110,9 +110,7 @@ BlockIO InterpreterCreateQuery::createDatabase(ASTCreateQuery & create)
|
||||
const ASTStorage & storage = *create.storage;
|
||||
const ASTFunction & engine = *storage.engine;
|
||||
/// Currently, there are no database engines, that support any arguments.
|
||||
if (engine.arguments || engine.parameters || storage.partition_by || storage.primary_key
|
||||
|| storage.order_by || storage.sample_by || storage.settings ||
|
||||
(create.columns_list && create.columns_list->indices && !create.columns_list->indices->children.empty()))
|
||||
if ((create.columns_list && create.columns_list->indices && !create.columns_list->indices->children.empty()))
|
||||
{
|
||||
std::stringstream ostr;
|
||||
formatAST(storage, ostr, false, false);
|
||||
@ -129,7 +127,7 @@ BlockIO InterpreterCreateQuery::createDatabase(ASTCreateQuery & create)
|
||||
String metadata_path = path + "metadata/" + database_name_escaped + "/";
|
||||
Poco::File(metadata_path).createDirectory();
|
||||
|
||||
DatabasePtr database = DatabaseFactory::get(database_engine_name, database_name, metadata_path, context);
|
||||
DatabasePtr database = DatabaseFactory::get(database_name, metadata_path, create.storage, context);
|
||||
|
||||
/// Will write file with database metadata, if needed.
|
||||
String metadata_file_tmp_path = path + "metadata/" + database_name_escaped + ".sql.tmp";
|
||||
|
||||
@ -127,7 +127,7 @@ String generateFilterActions(ExpressionActionsPtr & actions, const StoragePtr &
|
||||
return expr_list->children.at(0)->getColumnName();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
InterpreterSelectQuery::InterpreterSelectQuery(
|
||||
const ASTPtr & query_ptr_,
|
||||
|
||||
@ -133,14 +133,14 @@ static void validateUpdateColumns(
|
||||
for (const auto & col : storage->getColumns().getMaterialized())
|
||||
{
|
||||
if (col.name == column_name)
|
||||
throw Exception("Cannot UPDATE materialized column `" + column_name + "`", ErrorCodes::CANNOT_UPDATE_COLUMN);
|
||||
throw Exception("Cannot UPDATE materialized column " + backQuote(column_name), ErrorCodes::CANNOT_UPDATE_COLUMN);
|
||||
}
|
||||
|
||||
throw Exception("There is no column `" + column_name + "` in table", ErrorCodes::NO_SUCH_COLUMN_IN_TABLE);
|
||||
throw Exception("There is no column " + backQuote(column_name) + " in table", ErrorCodes::NO_SUCH_COLUMN_IN_TABLE);
|
||||
}
|
||||
|
||||
if (key_columns.count(column_name))
|
||||
throw Exception("Cannot UPDATE key column `" + column_name + "`", ErrorCodes::CANNOT_UPDATE_COLUMN);
|
||||
throw Exception("Cannot UPDATE key column " + backQuote(column_name), ErrorCodes::CANNOT_UPDATE_COLUMN);
|
||||
|
||||
auto materialized_it = column_to_affected_materialized.find(column_name);
|
||||
if (materialized_it != column_to_affected_materialized.end())
|
||||
@ -148,8 +148,8 @@ static void validateUpdateColumns(
|
||||
for (const String & materialized : materialized_it->second)
|
||||
{
|
||||
if (key_columns.count(materialized))
|
||||
throw Exception("Updated column `" + column_name + "` affects MATERIALIZED column `"
|
||||
+ materialized + "`, which is a key column. Cannot UPDATE it.",
|
||||
throw Exception("Updated column " + backQuote(column_name) + " affects MATERIALIZED column "
|
||||
+ backQuote(materialized) + ", which is a key column. Cannot UPDATE it.",
|
||||
ErrorCodes::CANNOT_UPDATE_COLUMN);
|
||||
}
|
||||
}
|
||||
|
||||
@ -46,7 +46,7 @@ String qualifiedName(ASTIdentifier * identifier, const String & prefix)
|
||||
return identifier->getAliasOrColumnName();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
}
|
||||
|
||||
PredicateExpressionsOptimizer::PredicateExpressionsOptimizer(
|
||||
ASTSelectQuery * ast_select_, ExtractedSettings && settings_, const Context & context_)
|
||||
|
||||
@ -1,4 +1,6 @@
|
||||
#include <Common/ProfileEvents.h>
|
||||
#include <Common/IPv6ToBinary.h>
|
||||
#include <Common/ClickHouseRevision.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Columns/ColumnFixedString.h>
|
||||
@ -7,11 +9,10 @@
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
#include <DataTypes/DataTypeDate.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypeFixedString.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypeFactory.h>
|
||||
#include <Interpreters/QueryLog.h>
|
||||
#include <Interpreters/ProfileEventsExt.h>
|
||||
#include <Common/ClickHouseRevision.h>
|
||||
#include <Poco/Net/IPAddress.h>
|
||||
#include <array>
|
||||
|
||||
@ -44,11 +45,11 @@ Block QueryLogElement::createBlock()
|
||||
{std::make_shared<DataTypeUInt8>(), "is_initial_query"},
|
||||
{std::make_shared<DataTypeString>(), "user"},
|
||||
{std::make_shared<DataTypeString>(), "query_id"},
|
||||
{std::make_shared<DataTypeFixedString>(16), "address"},
|
||||
{DataTypeFactory::instance().get("IPv6"), "address"},
|
||||
{std::make_shared<DataTypeUInt16>(), "port"},
|
||||
{std::make_shared<DataTypeString>(), "initial_user"},
|
||||
{std::make_shared<DataTypeString>(), "initial_query_id"},
|
||||
{std::make_shared<DataTypeFixedString>(16), "initial_address"},
|
||||
{DataTypeFactory::instance().get("IPv6"), "initial_address"},
|
||||
{std::make_shared<DataTypeUInt16>(), "initial_port"},
|
||||
{std::make_shared<DataTypeUInt8>(), "interface"},
|
||||
{std::make_shared<DataTypeString>(), "os_user"},
|
||||
@ -73,29 +74,6 @@ Block QueryLogElement::createBlock()
|
||||
}
|
||||
|
||||
|
||||
static std::array<char, 16> IPv6ToBinary(const Poco::Net::IPAddress & address)
|
||||
{
|
||||
std::array<char, 16> res;
|
||||
|
||||
if (Poco::Net::IPAddress::IPv6 == address.family())
|
||||
{
|
||||
memcpy(res.data(), address.addr(), 16);
|
||||
}
|
||||
else if (Poco::Net::IPAddress::IPv4 == address.family())
|
||||
{
|
||||
/// Convert to IPv6-mapped address.
|
||||
memset(res.data(), 0, 10);
|
||||
res[10] = '\xFF';
|
||||
res[11] = '\xFF';
|
||||
memcpy(&res[12], address.addr(), 4);
|
||||
}
|
||||
else
|
||||
memset(res.data(), 0, 16);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
|
||||
void QueryLogElement::appendToBlock(Block & block) const
|
||||
{
|
||||
MutableColumns columns = block.mutateColumns();
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user