mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-22 07:31:57 +00:00
Merge branch 'master' of github.com:yandex/ClickHouse into fix_fetch_of_existing_part
This commit is contained in:
alesapin
commit
6a3b543e10
1
.gitignore
vendored
1
.gitignore
vendored
@ -205,6 +205,7 @@ configure-stamp
|
||||
|
||||
*.bin
|
||||
*.mrk
|
||||
*.mrk2
|
||||
|
||||
.dupload.conf
|
||||
|
||||
|
||||
17
CHANGELOG.md
17
CHANGELOG.md
@ -1,3 +1,20 @@
|
||||
## ClickHouse release 19.9.4.1, 2019-07-05
|
||||
|
||||
### Bug Fix
|
||||
* Fix segfault in Delta codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. [#5786](https://github.com/yandex/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin))
|
||||
* Fix rare bug in checking of part with LowCardinality column. [#5832](https://github.com/yandex/ClickHouse/pull/5832) ([alesapin](https://github.com/alesapin))
|
||||
* Fix segfault in TTL merge with non-physical columns in block. [#5819](https://github.com/yandex/ClickHouse/pull/5819) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Fix potential infinite sleeping of low-priority queries. [#5842](https://github.com/yandex/ClickHouse/pull/5842) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Fix how ClickHouse determines default time zone as UCT instead of UTC. [#5828](https://github.com/yandex/ClickHouse/pull/5828) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Fix bug about executing distributed DROP/ALTER/TRUNCATE/OPTIMIZE ON CLUSTER queries on follower replica before leader replica. Now they will be executed directly on leader replica. [#5757](https://github.com/yandex/ClickHouse/pull/5757) ([alesapin](https://github.com/alesapin))
|
||||
* Fix race condition, which cause that some queries may not appear in query_log instantly after SYSTEM FLUSH LOGS query. [#5685](https://github.com/yandex/ClickHouse/pull/5685) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Added missing support for constant arguments to `evalMLModel` function. [#5820](https://github.com/yandex/ClickHouse/pull/5820) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
## ClickHouse release 19.7.6.1, 2019-07-05
|
||||
|
||||
### Bug Fix
|
||||
* Fix performance regression in some queries with JOIN. [#5192](https://github.com/yandex/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014))
|
||||
|
||||
## ClickHouse release 19.9.2.4, 2019-06-24
|
||||
|
||||
### New Feature
|
||||
|
||||
127
CHANGELOG_RU.md
127
CHANGELOG_RU.md
@ -1,3 +1,119 @@
|
||||
## ClickHouse release 19.9.4.1, 2019-07-05
|
||||
|
||||
### Исправления ошибок
|
||||
* Исправлен segmentation fault в кодеке сжатия Delta в колонках с величинами размером меньше 32 бит. Ошибка могла приводить к повреждениям памяти. [#5786](https://github.com/yandex/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin))
|
||||
* Исправлена ошибка в проверке кусков в LowCardinality колонках. [#5832](https://github.com/yandex/ClickHouse/pull/5832) ([alesapin](https://github.com/alesapin))
|
||||
* Исправлен segmentation fault при слиянии кусков с истекшим TTL в случае, когда в блоке присутствуют столбцы, не входящие в структуру таблицы. [#5819](https://github.com/yandex/ClickHouse/pull/5819) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Исправлена существовавшая возможность ухода в бесконечное ожидание на низко-приоритетных запросах. [#5842](https://github.com/yandex/ClickHouse/pull/5842) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлена ошибка определения таймзоны по умолчанию (UCT вместо UTC). [#5828](https://github.com/yandex/ClickHouse/pull/5828) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлена ошибка в распределенных запросах вида DROP/ALTER/TRUNCATE/OPTIMIZE ON CLUSTER. [#5757](https://github.com/yandex/ClickHouse/pull/5757) ([alesapin](https://github.com/alesapin))
|
||||
* Исправлена ошибка, которая при распределенных запросах могла привести к тому, что некоторые запросы не появлялись в query_log сразу после SYSTEM FLUSH LOGS запроса. [#5685](https://github.com/yandex/ClickHouse/pull/5685) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Добавлена отсутствовавшая поддержка константных аргументов для функции `evalMLModel`. [#5820](https://github.com/yandex/ClickHouse/pull/5820) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
|
||||
## ClickHouse release 19.7.6.1, 2019-07-05
|
||||
|
||||
### Исправления ошибок
|
||||
* Исправлена просадка производительности в методе JOIN в некоторых видах запросов. [#5192](https://github.com/yandex/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014))
|
||||
|
||||
## ClickHouse release 19.8.3.8, 2019-06-11
|
||||
|
||||
### Новые возможности
|
||||
* Добавлены функции для работы с JSON. [#4686](https://github.com/yandex/ClickHouse/pull/4686) ([hcz](https://github.com/hczhcz)) [#5124](https://github.com/yandex/ClickHouse/pull/5124). ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Добавлена функция `basename`, аналогичная функции `basename` в различных языках программирования (`os.path.basename` в python, `basename` в PHP, и т.д.). Работает с UNIX и Windows подобными путями. [#5136](https://github.com/yandex/ClickHouse/pull/5136) ([Guillaume Tassery](https://github.com/YiuRULE))
|
||||
* Добавлена возможность указать смещение в секции LIMIT BY с помощью синтаксиса `LIMIT n, m BY` или `LIMIT m OFFSET n BY`. [#5138](https://github.com/yandex/ClickHouse/pull/5138) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Добавлен новый тип данных `SimpleAggregateFunction`, который позволяет создавать столбцы с легковесной агрегацией в AggregatingMergeTree. Он может использоваться только с простыми функциями, такими как `any`,` anyLast`, `sum`,` min`, `max`.
|
||||
[#4629](https://github.com/yandex/ClickHouse/pull/4629) ([Boris Granveaud](https://github.com/bgranvea))
|
||||
* Добавлена поддержка неконстантных аргументов в функции `ngramDistance` [#5198](https://github.com/yandex/ClickHouse/pull/5198) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Добавлены функции `skewPop`, `skewSamp`, `kurtPop` и `kurtSamp`, которые вычисляют для последовательности коэффициент асимметрии, выборочный коэффициент асимметрии, коэффициент эксцесса и выборочный коэффициент эксцесса соответсвенно. [#5200](https://github.com/yandex/ClickHouse/pull/5200) ([hcz](https://github.com/hczhcz))
|
||||
* Добавлена поддержка операции `RENAME` для движка `MaterializedView`. [#5209](https://github.com/yandex/ClickHouse/pull/5209) ([Guillaume Tassery](https://github.com/YiuRULE))
|
||||
* Сервер, позволяющий подключаться к ClickHouse через клиент MySQL. [#4715](https://github.com/yandex/ClickHouse/pull/4715) ([Yuriy Baranov](https://github.com/yurriy))
|
||||
* Добавлены функции `toDecimal*OrZero` и `toDecimal*OrNull`. [#5291](https://github.com/yandex/ClickHouse/pull/5291) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Добавлена поддержка Decimal в функциях: `quantile`, `quantiles`, `median`, `quantileExactWeighted`, `quantilesExactWeighted`, `medianExactWeighted`. [#5304](https://github.com/yandex/ClickHouse/pull/5304) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Добавлена функция `toValidUTF8`, которая заменяет все некорректные символы в кодировке UTF-8 на заменяющий символ <20> (U+FFFD). [#5322](https://github.com/yandex/ClickHouse/pull/5322) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Добавлена функция `format`, которая выполняет подстановку в константный шаблон (упрощенный шаблон форматирования, используемый в Python) строк, переданных в аргументах функции. [#5330](https://github.com/yandex/ClickHouse/pull/5330) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Добавлена системная таблица `system.detached_parts`, содержащая информацию о кусках из директории `detached` для таблиц семейства `MergeTree`. [#5353](https://github.com/yandex/ClickHouse/pull/5353) ([akuzm](https://github.com/akuzm))
|
||||
* Добавлена функция `ngramSearch` для вычисления несимметричной разности между `needle` и `haystack`. [#5418](https://github.com/yandex/ClickHouse/pull/5418)[#5422](https://github.com/yandex/ClickHouse/pull/5422) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Реализованы основные методы машинного обучения (стохастическая линейная регрессия и логистическая регрессия) в виде агрегатных функций. Для вычисления весов могут использоваться различные методы (градиентный спуск, градиентный спуск с накоплением импульса, метод Нестерова). Также поддерживаются mini-batches произвольного размера. [#4943](https://github.com/yandex/ClickHouse/pull/4943) ([Quid37](https://github.com/Quid37))
|
||||
* Добавлены функции `geohashEncode` и `geohashDecode`. [#5003](https://github.com/yandex/ClickHouse/pull/5003) ([Vasily Nemkov](https://github.com/Enmk))
|
||||
* Добавлена агрегатная функция `timeSeriesGroupSum`, которая агрегирует временные ряды в которых не совпадают моменты. Функция использует линейную интерполяцию между двумя значениями времени, а затем суммирует значения для одного и того же момента. Добавлена агрегатная функция `timeSeriesGroupRateSum`, которая вычисляет производные по timestamp для рядов, а затем суммирует полученные производные для всех рядов для одного значения timestamp. [#4542](https://github.com/yandex/ClickHouse/pull/4542) ([Yangkuan Liu](https://github.com/LiuYangkuan))
|
||||
* Добавлены функции `IPv4CIDRtoIPv4Range` and `IPv6CIDRtoIPv6Range`, которые вычисляют диапазон подсети, используя CIDR. [#5095](https://github.com/yandex/ClickHouse/pull/5095) ([Guillaume Tassery](https://github.com/YiuRULE))
|
||||
* Добавлен заголовок X-ClickHouse-Summary при отправке HTTP запроса с включенной настройкой `send_progress_in_http_headers`. Он содержит инфорцию о X-ClickHouse-Progress, а также информацию о том сколько строк и байт было записано в таблицу и подобное. [#5116](https://github.com/yandex/ClickHouse/pull/5116) ([Guillaume Tassery](https://github.com/YiuRULE))
|
||||
|
||||
### Улучшения
|
||||
* Добавлена настройка `max_parts_in_total` (по умолчанию: 100 000) для таблиц семейства MergeTree, предотвращающая неправильное задание ключа партиционирования #5166. [#5171](https://github.com/yandex/ClickHouse/pull/5171) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* `clickhouse-obfuscator`: теперь seed вычисляется для столбцов путем комбинирования начального seed и имени колонки, а не ее позиции. Это необходимо для того, чтобы оставить таблицы пригодными для JOIN-ов при преобразовании датасетов с несколькими таблицами. [#5178](https://github.com/yandex/ClickHouse/pull/5178) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Добавлены функции `JSONExtractRaw`, `JSONExtractKeyAndValues`. Функции вида `jsonExtract<type>` переименованы в `JSONExtract<type>`. Теперь, в случае ошибки эти функции возвращают соответсвующие типу значения, а не `NULL`. Теперь возвращаемый тип передается в функцию `JSONExtract` последним параметром. Если недоступны иструкции AVX2, то JSON функции выполняются с использованием бибилиотеки RapidJSON. Обновлена библиотека Simdjson. [#5235](https://github.com/yandex/ClickHouse/pull/5235) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Теперь функции `if` and `multiIf` не зависят от условных `Nullable` типов. [#5238](https://github.com/yandex/ClickHouse/pull/5238) ([Jian Wu](https://github.com/janplus))
|
||||
* Оператор `IN` теперь возвращает `Null` для значений равных `Null` (аналогично функции `Equal`). [#5152](https://github.com/yandex/ClickHouse/pull/5152) ([Jian Wu](https://github.com/janplus))
|
||||
* Добавлена проверка ограничения по времени выполенения запроса через каждые (flush_interval / poll_timeout) строк при чтении из движка Kafka. Это позволяет чаще прерывать чтение из движка Kafka и чаще проверять ограничение по времени на более высоком уровне конвеера выполнения запроса. [#5249](https://github.com/yandex/ClickHouse/pull/5249) ([Ivan](https://github.com/abyss7))
|
||||
* Теперь библиотека rdkafka линкуется c bundled библиотекой SASL. Это позволяет использовать SASL SCRAM аутентификацию. [#5253](https://github.com/yandex/ClickHouse/pull/5253) ([Ivan](https://github.com/abyss7))
|
||||
* clickhouse-server: более информативные сообщения об ошибках прослушивания. [#5268](https://github.com/yandex/ClickHouse/pull/5268) ([proller](https://github.com/proller))
|
||||
* clickhouse-copier: добавлена поддержка словарей в функциях в секции `<sharding_key>`. [#5270](https://github.com/yandex/ClickHouse/pull/5270) ([proller](https://github.com/proller))
|
||||
* Добавлена настройка `kafka_commit_every_batch` для регулирования чтения данных из движка Kafka. Позволяет установить вид чтения: после каждого полученного пакета сообщений или после того как блок записан в таблицу. [#5308](https://github.com/yandex/ClickHouse/pull/5308) ([Ivan](https://github.com/abyss7))
|
||||
* Добавлена поддержка беззнаковых целых типов отличных от UInt32 в функциях `windowFunnel`, `sequenceMatch`. [#5339](https://github.com/yandex/ClickHouse/pull/5339) [#5320](https://github.com/yandex/ClickHouse/pull/5320) ([sundyli](https://github.com/sundy-li))
|
||||
* В движке Merge виртуальный столбец с названием `_table` (содержащий название таблицы) теперь недоступен, если он присутствует в таблицах-источниках. [#5325](https://github.com/yandex/ClickHouse/pull/5325) ([Ivan](https://github.com/abyss7))
|
||||
* Более понятные сообщения об ошибках, если чексуммы не совпадают вероятнее всего из-за ошибок на аппаратном уровне. [#5355](https://github.com/yandex/ClickHouse/pull/5355) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Для движка `Merge` добавлена проверка на то, что таблицы-источники поддерживают семплирование. [#5366](https://github.com/yandex/ClickHouse/pull/5366) ([Ivan](https://github.com/abyss7))
|
||||
* Теперь после использования MySQL в качестве внешнего словаря закрывается соединение. [#5395](https://github.com/yandex/ClickHouse/pull/5395) ([Clément Rodriguez](https://github.com/clemrodriguez))
|
||||
* Улучшения в протоколе MySQL Wire. Формат переименован из `MySQL` to `MySQLWire`. Теперь SSL недоступно, если не удалось создать SSL контекст. [#5419](https://github.com/yandex/ClickHouse/pull/5419) ([Yuriy Baranov](https://github.com/yurriy))
|
||||
* clickhouse-client: теперь можно запустить клиент, если файл истории недоступен. [#5431](https://github.com/yandex/ClickHouse/pull/5431) ([proller](https://github.com/proller))
|
||||
* Теперь учитываются настройки запроса при асинхронных вставках в таблицу типа Distributed. [#4936](https://github.com/yandex/ClickHouse/pull/4936) ([TCeason](https://github.com/TCeason))
|
||||
* Переименованы функции `leastSqr` в `simpleLinearRegression`, `LinearRegression` в `linearRegression`, `LogisticRegression` в `logisticRegression`. [#5391](https://github.com/yandex/ClickHouse/pull/5391) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||
|
||||
### Улучшения производительности
|
||||
* Паралелльная обработка кусков в запросах `ALTER ... MODIFY`. [#4639](https://github.com/yandex/ClickHouse/pull/4639) ([Ivan Kush](https://github.com/IvanKush))
|
||||
* Оптимизации в обработке регулярных выражений. [#5193](https://github.com/yandex/ClickHouse/pull/5193) [#5191](https://github.com/yandex/ClickHouse/pull/5191) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Теперь столбец правой таблицы, который используется только в секции `JOIN ON` не добавляется в результат. [#5260](https://github.com/yandex/ClickHouse/pull/5260) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Теперь чтение из буфера движка Kafka приостанавливается после первого пустого ответа. Это позволяет не вызывать несколько раз метод `ReadBuffer::next()` для пустого результата. [#5283](https://github.com/yandex/ClickHouse/pull/5283) ([Ivan](https://github.com/abyss7))
|
||||
* Оптимизация функции `concat` для нескольких аргументов. [#5357](https://github.com/yandex/ClickHouse/pull/5357) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Оптимизация запросов. Прокидывание выражения оператора `IN` при конвертации `commа/cross join` в `inner`. [#5396](https://github.com/yandex/ClickHouse/pull/5396) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Улучшение реализации LZ4 для более быстрого разжатия данных. [#5070](https://github.com/yandex/ClickHouse/pull/5070) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Реализована MSD-сортировка, а также на ее основе частичная сортировка. [#5129](https://github.com/yandex/ClickHouse/pull/5129) ([Evgenii Pravda](https://github.com/kvinty))
|
||||
|
||||
### Исправления ошибок
|
||||
* Исправлено прокидывание необходимых для чтения столбцов в join. [#5192](https://github.com/yandex/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014))
|
||||
* Исправлено неправильное поведение команды `sudo service clickhouse-server forcerestart` в случае, если сервер запущен при помощи systemd. [#5204](https://github.com/yandex/ClickHouse/pull/5204) ([proller](https://github.com/proller))
|
||||
* Исправлены коды ошибок при передаче кусков (раньше даже при ошибках возвращался код 200). [#5216](https://github.com/yandex/ClickHouse/pull/5216) ([proller](https://github.com/proller))
|
||||
* Исправлено использование типа SimpleAggregateFunction со строками длиннее чем `MAX_SMALL_STRING_SIZE` [#5311](https://github.com/yandex/ClickHouse/pull/5311) ([Azat Khuzhin](https://github.com/azat))
|
||||
* Исправлена ошибка конвертации типа `Decimal` в `Nullable(Decimal)` в секции IN. Добавлена возможность конвертации между различными типами Decimal (including different scales). [#5350](https://github.com/yandex/ClickHouse/pull/5350) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлено затирание регистров FPU в библиотеке simdjson, которое приводило к неправильным вычислениям в агрегатных функциях `uniqHLL` и `uniqCombined`, а также математических функциях, таких как `log`. [#5354](https://github.com/yandex/ClickHouse/pull/5354) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлены обработка константных и неконстантных аргументов одновременно в JSON функциях. [#5435](https://github.com/yandex/ClickHouse/pull/5435) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* Исправлен возвращаемый тип в функции `quantileExact` с типом `Decimal`. [#5304](https://github.com/yandex/ClickHouse/pull/5304) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
|
||||
### Улучшения сборки, тестирования и пакетирования
|
||||
* Исправлены срабатывания address-санитайзера, показывающие потенциальную ошибку use-after-free.[#5139](https://github.com/yandex/ClickHouse/pull/5139) [#5143](https://github.com/yandex/ClickHouse/pull/5143) [#5393](https://github.com/yandex/ClickHouse/pull/5393) ([Ivan](https://github.com/abyss7))
|
||||
* Тесты производительности перемещены в одну директорию. [#5158](https://github.com/yandex/ClickHouse/pull/5158) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлены некоторые тесты производительности. [#5255](https://github.com/yandex/ClickHouse/pull/5255) ([alesapin](https://github.com/alesapin))
|
||||
* Добавлена утилита, позволяющая вычислять чексуммы, полученные в результате возможных инверсий битов. Она помогает отлаживать аппаратные ошибки. [#5334](https://github.com/yandex/ClickHouse/pull/5334) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Улучшение скрипта для запуска интеграционных тестов. [#5340](https://github.com/yandex/ClickHouse/pull/5340)[#5360](https://github.com/yandex/ClickHouse/pull/5360) ([filimonov](https://github.com/filimonov))
|
||||
* Добавлена инструкция о том, как писать тесты производительности. [#5408](https://github.com/yandex/ClickHouse/pull/5408) ([alesapin](https://github.com/alesapin))
|
||||
* Добавлена возможность делать подстановки в запросах create, fill и drop в тестах производительности. [#5367](https://github.com/yandex/ClickHouse/pull/5367) ([Olga Khvostikova](https://github.com/stavrolia))
|
||||
|
||||
## ClickHouse release 19.7.5.27, 2019-06-09
|
||||
|
||||
### Новые возможности
|
||||
* Добавлены функции для битмапов `bitmapHasAny` и `bitmapHasAll` (аналогично функциям `hasAny` и `hasAll` для массивов). [#5279](https://github.com/yandex/ClickHouse/pull/5279) ([Sergi Vladykin](https://github.com/svladykin))
|
||||
|
||||
### Исправления ошибок
|
||||
* Исправлен segfault при использовании `minmax` индекса со значениями `Null`. [#5246](https://github.com/yandex/ClickHouse/pull/5246) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||
* Исправлена ошибка 'Not found column', возникавшая в некоторых распределенных запросах. [#5407](https://github.com/yandex/ClickHouse/pull/5407) ([Constantin S. Pan](https://github.com/kvap))
|
||||
* Исправлена ошибка "Column '0' already exists" в запросах `SELECT .. PREWHERE` со столбцами с указанным `DEFAULT` [#5397](https://github.com/yandex/ClickHouse/pull/5397) ([proller](https://github.com/proller))
|
||||
* Исправлен запрос `ALTER MODIFY TTL` для таблиц типа `ReplicatedMergeTree`. [#5539](https://github.com/yandex/ClickHouse/pull/5539/commits) ([Anton Popov](https://github.com/CurtizJ))
|
||||
* Теперь сервер не падает, если подписчики Kafka не смогли запуститься. [#5285](https://github.com/yandex/ClickHouse/pull/5285) ([Ivan](https://github.com/abyss7))
|
||||
* Исправлены bitmap функции. [#5359](https://github.com/yandex/ClickHouse/pull/5359) ([Andy Yang](https://github.com/andyyzh))
|
||||
* Исправлено значение `element_count` в таблице `system.dictionaries` для словарей типа hashed (теперь не подсчитываются дубликаты). [#5440](https://github.com/yandex/ClickHouse/pull/5440) ([Azat Khuzhin](https://github.com/azat))
|
||||
* Для определения имени таймзоны теперь используется переменная среды TZ. В некоторых случаях это помогает правильно определять таймзону по умолчанию. [#5443](https://github.com/yandex/ClickHouse/pull/5443) ([Ivan](https://github.com/abyss7))
|
||||
* Убрано неявное преобразование целых типов в функциях `dictGetT`, т.к. оно работало неправильно. Вместо этого теперь бросается исключение. [#5446](https://github.com/yandex/ClickHouse/pull/5446) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлены настройки во внешних данных HTTP запроса. [#5455](https://github.com/yandex/ClickHouse/pull/5455) ([Danila Kutenin](https://github.com/danlark1))
|
||||
* Исправлена ошибка, приводящая к тому, что куски удалялись только из файловой системы, но не из Zookeeper. [#5520](https://github.com/yandex/ClickHouse/pull/5520) ([alesapin](https://github.com/alesapin))
|
||||
* Исправлен segfault в функции `bitmapHasAny`. [#5528](https://github.com/yandex/ClickHouse/pull/5528) ([Zhichang Yu](https://github.com/yuzhichang))
|
||||
* Исправлена ошибка, приводящая к тому, что при репликации не вычислялся заново адрес хоста, даже если DNS кеш был сброшен. [#5534](https://github.com/yandex/ClickHouse/pull/5534) ([alesapin](https://github.com/alesapin))
|
||||
* Исправлен запрос `DROP INDEX IF EXISTS`. Теперь запрос `ALTER TABLE ... DROP INDEX IF EXISTS ...` не выкидывает исключение. [#5524](https://github.com/yandex/ClickHouse/pull/5524) ([Gleb Novikov](https://github.com/NanoBjorn))
|
||||
* Исправлен тип результируещего столбца в запросе `UNION ALL`. Раньше могли быть случаи, когда тип и данные столбца были несогласованы. [#5503](https://github.com/yandex/ClickHouse/pull/5503) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлена ошибка, вследствие которой поток `DDLWorker`-а мог быть остановлен, если одна нода удалила `znode` из очереди задач, а другая уже получила ее, но не успела выполнить. [#5489](https://github.com/yandex/ClickHouse/pull/5489) ([Azat Khuzhin](https://github.com/azat))
|
||||
* Исправлена вставка в Distributed таблицы с материализованныси столбцами. [#5429](https://github.com/yandex/ClickHouse/pull/5429) ([Azat Khuzhin](https://github.com/azat))
|
||||
|
||||
|
||||
## ClickHouse release 19.7.3.9, 2019-05-30
|
||||
|
||||
### Новые возможности
|
||||
@ -63,6 +179,17 @@ Kutenin](https://github.com/danlark1))
|
||||
([proller](https://github.com/proller))
|
||||
|
||||
|
||||
## ClickHouse release 19.6.3.18, 2019-06-13
|
||||
|
||||
### Исправления ошибок
|
||||
* Исправлено прокидывание условий оператора IN для запросов с табличными функциями `mysql` and `odbc` и соответсвующими движками таблиц. [#5313](https://github.com/yandex/ClickHouse/pull/5313) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Исправлен deadlock в Zookeeper. [#5297](https://github.com/yandex/ClickHouse/pull/5297) ([github1youlc](https://github.com/github1youlc))
|
||||
* Для формата CSV добавлена возможность указывать значения типа Decimal в кавычках. [#5284](https://github.com/yandex/ClickHouse/pull/5284) ([Artem Zuikov](https://github.com/4ertus2)
|
||||
* Убрана возможность преобразований из float Inf/NaN в Decimal (теперь бросается исключение). [#5282](https://github.com/yandex/ClickHouse/pull/5282) ([Artem Zuikov](https://github.com/4ertus2))
|
||||
* Исправлен data race в запросе `RENAME`. [#5247](https://github.com/yandex/ClickHouse/pull/5247) ([Winter Zhang](https://github.com/zhang2014))
|
||||
* Отключен LFAlloc. Использование LFAlloc могло приводить к ошибкам MAP_FAILED при использовании UncompressedCache и в результате к падению запросов на высоконагруженных серверах. [cfdba93](https://github.com/yandex/ClickHouse/commit/cfdba938ce22f16efeec504f7f90206a515b1280)([Danila Kutenin](https://github.com/danlark1))
|
||||
|
||||
|
||||
## ClickHouse release 19.6.2.11, 2019-05-13
|
||||
|
||||
### Новые возможности
|
||||
|
||||
@ -202,6 +202,9 @@ set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${COMPILER_FLAGS} -fn
|
||||
set (CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} -O3 ${CMAKE_C_FLAGS_ADD}")
|
||||
set (CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -O0 -g3 -ggdb3 -fno-inline ${CMAKE_C_FLAGS_ADD}")
|
||||
|
||||
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--no-undefined")
|
||||
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--no-undefined")
|
||||
|
||||
include (cmake/use_libcxx.cmake)
|
||||
include (cmake/find_unwind.cmake)
|
||||
|
||||
@ -279,6 +282,7 @@ endif ()
|
||||
if (DEFAULT_LIBS)
|
||||
# Add default libs to all targets as the last dependency.
|
||||
set(CMAKE_CXX_STANDARD_LIBRARIES ${DEFAULT_LIBS})
|
||||
set(CMAKE_C_STANDARD_LIBRARIES ${DEFAULT_LIBS})
|
||||
endif ()
|
||||
|
||||
if (NOT MAKE_STATIC_LIBRARIES)
|
||||
@ -458,4 +462,14 @@ if (GLIBC_COMPATIBILITY OR USE_INTERNAL_UNWIND_LIBRARY_FOR_EXCEPTION_HANDLING)
|
||||
add_default_dependencies(boost_program_options_internal)
|
||||
add_default_dependencies(boost_system_internal)
|
||||
add_default_dependencies(boost_regex_internal)
|
||||
add_default_dependencies(roaring)
|
||||
add_default_dependencies(btrie)
|
||||
add_default_dependencies(cpuid)
|
||||
add_default_dependencies(mysqlclient)
|
||||
add_default_dependencies(zlib)
|
||||
add_default_dependencies(thrift)
|
||||
add_default_dependencies(brotli)
|
||||
add_default_dependencies(libprotobuf)
|
||||
add_default_dependencies(base64)
|
||||
add_default_dependencies(readpassphrase)
|
||||
endif ()
|
||||
|
||||
2
LICENSE
2
LICENSE
@ -1,3 +1,5 @@
|
||||
Copyright 2016-2019 Yandex LLC
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
1
contrib/CMakeLists.txt
vendored
1
contrib/CMakeLists.txt
vendored
@ -251,7 +251,6 @@ if(USE_INTERNAL_GTEST_LIBRARY)
|
||||
add_subdirectory(${ClickHouse_SOURCE_DIR}/contrib/googletest/googletest ${CMAKE_CURRENT_BINARY_DIR}/googletest)
|
||||
# avoid problems with <regexp.h>
|
||||
target_compile_definitions (gtest INTERFACE GTEST_HAS_POSIX_RE=0)
|
||||
target_include_directories (gtest SYSTEM INTERFACE ${ClickHouse_SOURCE_DIR}/contrib/googletest/include)
|
||||
elseif(GTEST_SRC_DIR)
|
||||
add_subdirectory(${GTEST_SRC_DIR}/googletest ${CMAKE_CURRENT_BINARY_DIR}/googletest)
|
||||
target_compile_definitions(gtest INTERFACE GTEST_HAS_POSIX_RE=0)
|
||||
|
||||
2
contrib/librdkafka
vendored
2
contrib/librdkafka
vendored
@ -1 +1 @@
|
||||
Subproject commit 8681f884020e880a4c6cda3cfc672f0669e1f38e
|
||||
Subproject commit 6160ec275a5bb0a4088ede3c5f2afde638bbef65
|
||||
@ -12,6 +12,7 @@ set(SRCS
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_background.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_broker.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_buf.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_cert.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_cgrp.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_conf.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_event.c
|
||||
@ -34,6 +35,7 @@ set(SRCS
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_sasl.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_sasl_plain.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_sasl_scram.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_ssl.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_subscription.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_timer.c
|
||||
${RDKAFKA_SOURCE_DIR}/rdkafka_topic.c
|
||||
|
||||
2
contrib/poco
vendored
2
contrib/poco
vendored
@ -1 +1 @@
|
||||
Subproject commit 29439cf7fa32c1a2d62d925bb6d6a3f14668a4a2
|
||||
Subproject commit ece721f1085e3894cb5286e8560af84cd1445326
|
||||
2
contrib/simdjson
vendored
2
contrib/simdjson
vendored
@ -1 +1 @@
|
||||
Subproject commit 2151ad7f34cf773a23f086e941d661f8a8873144
|
||||
Subproject commit 3bd3116cf8faf6d482dc31423b16533bfa2696f7
|
||||
@ -11,8 +11,9 @@ set(SIMDJSON_SRC
|
||||
${SIMDJSON_SRC_DIR}/stage2_build_tape.cpp
|
||||
${SIMDJSON_SRC_DIR}/parsedjson.cpp
|
||||
${SIMDJSON_SRC_DIR}/parsedjsoniterator.cpp
|
||||
${SIMDJSON_SRC_DIR}/simdjson.cpp
|
||||
)
|
||||
|
||||
add_library(${SIMDJSON_LIBRARY} ${SIMDJSON_SRC})
|
||||
target_include_directories(${SIMDJSON_LIBRARY} PUBLIC "${SIMDJSON_INCLUDE_DIR}")
|
||||
target_include_directories(${SIMDJSON_LIBRARY} SYSTEM PUBLIC "${SIMDJSON_INCLUDE_DIR}")
|
||||
target_compile_options(${SIMDJSON_LIBRARY} PRIVATE -mavx2 -mbmi -mbmi2 -mpclmul)
|
||||
|
||||
@ -48,7 +48,7 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow -Wshadow-uncaptured-local -Wextra-semi -Wcomma -Winconsistent-missing-destructor-override -Wunused-exception-parameter -Wcovered-switch-default -Wold-style-cast -Wrange-loop-analysis -Wunused-member-function -Wunreachable-code -Wunreachable-code-return -Wnewline-eof -Wembedded-directive -Wgnu-case-range -Wunused-macros -Wconditional-uninitialized -Wdeprecated -Wundef -Wreserved-id-macro -Wredundant-parens -Wzero-as-null-pointer-constant")
|
||||
|
||||
if (WEVERYTHING)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-missing-noreturn -Wno-padded -Wno-switch-enum -Wno-shadow-field-in-constructor -Wno-deprecated-dynamic-exception-spec -Wno-float-equal -Wno-weak-vtables -Wno-shift-sign-overflow -Wno-sign-conversion -Wno-conversion -Wno-exit-time-destructors -Wno-undefined-func-template -Wno-documentation-unknown-command -Wno-missing-variable-declarations -Wno-unused-template -Wno-global-constructors -Wno-c99-extensions -Wno-missing-prototypes -Wno-weak-template-vtables -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types -Wno-double-promotion -Wno-disabled-macro-expansion -Wno-used-but-marked-unused -Wno-vla-extension -Wno-vla -Wno-packed")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-padded -Wno-switch-enum -Wno-shadow-field-in-constructor -Wno-deprecated-dynamic-exception-spec -Wno-float-equal -Wno-weak-vtables -Wno-shift-sign-overflow -Wno-sign-conversion -Wno-conversion -Wno-exit-time-destructors -Wno-undefined-func-template -Wno-documentation-unknown-command -Wno-missing-variable-declarations -Wno-unused-template -Wno-global-constructors -Wno-c99-extensions -Wno-missing-prototypes -Wno-weak-template-vtables -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types -Wno-double-promotion -Wno-disabled-macro-expansion -Wno-vla-extension -Wno-vla -Wno-packed")
|
||||
|
||||
# TODO Enable conversion, sign-conversion, double-promotion warnings.
|
||||
endif ()
|
||||
@ -114,6 +114,11 @@ add_headers_and_sources(dbms src/Storages/MergeTree)
|
||||
add_headers_and_sources(dbms src/Client)
|
||||
add_headers_and_sources(dbms src/Formats)
|
||||
|
||||
if(USE_RDKAFKA)
|
||||
add_headers_and_sources(dbms src/Storages/Kafka)
|
||||
endif()
|
||||
|
||||
|
||||
list (APPEND clickhouse_common_io_sources ${CONFIG_BUILD})
|
||||
list (APPEND clickhouse_common_io_headers ${CONFIG_VERSION} ${CONFIG_COMMON})
|
||||
|
||||
@ -222,6 +227,13 @@ target_link_libraries(clickhouse_common_io
|
||||
roaring
|
||||
)
|
||||
|
||||
if (USE_RDKAFKA)
|
||||
target_link_libraries(dbms PRIVATE ${CPPKAFKA_LIBRARY} ${RDKAFKA_LIBRARY})
|
||||
if(NOT USE_INTERNAL_RDKAFKA_LIBRARY)
|
||||
target_include_directories(dbms SYSTEM BEFORE PRIVATE ${RDKAFKA_INCLUDE_DIR})
|
||||
endif()
|
||||
endif()
|
||||
|
||||
|
||||
if(RE2_INCLUDE_DIR)
|
||||
target_include_directories(clickhouse_common_io SYSTEM BEFORE PUBLIC ${RE2_INCLUDE_DIR})
|
||||
@ -246,6 +258,8 @@ target_link_libraries (dbms
|
||||
PRIVATE
|
||||
clickhouse_parsers
|
||||
clickhouse_common_config
|
||||
clickhouse_common_zookeeper
|
||||
string_utils # FIXME: not sure if it's private
|
||||
PUBLIC
|
||||
clickhouse_common_io
|
||||
PRIVATE
|
||||
@ -255,6 +269,7 @@ target_link_libraries (dbms
|
||||
PRIVATE
|
||||
${BTRIE_LIBRARIES}
|
||||
${Boost_PROGRAM_OPTIONS_LIBRARY}
|
||||
${Boost_FILESYSTEM_LIBRARY}
|
||||
PUBLIC
|
||||
${Boost_SYSTEM_LIBRARY}
|
||||
Threads::Threads
|
||||
@ -318,10 +333,6 @@ if (USE_CAPNP)
|
||||
endif ()
|
||||
endif ()
|
||||
|
||||
if (USE_RDKAFKA)
|
||||
target_link_libraries (dbms PRIVATE clickhouse_storage_kafka)
|
||||
endif ()
|
||||
|
||||
if (USE_PARQUET)
|
||||
target_link_libraries(dbms PRIVATE ${PARQUET_LIBRARY})
|
||||
if (NOT USE_INTERNAL_PARQUET_LIBRARY OR USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||
@ -376,6 +387,10 @@ if (ENABLE_TESTS AND USE_GTEST)
|
||||
# attach all dbms gtest sources
|
||||
grep_gtest_sources(${ClickHouse_SOURCE_DIR}/dbms dbms_gtest_sources)
|
||||
add_executable(unit_tests_dbms ${dbms_gtest_sources})
|
||||
|
||||
# gtest framework has substandard code
|
||||
target_compile_options(unit_tests_dbms PRIVATE -Wno-zero-as-null-pointer-constant -Wno-undef -Wno-sign-compare -Wno-used-but-marked-unused -Wno-missing-noreturn)
|
||||
|
||||
target_link_libraries(unit_tests_dbms PRIVATE ${GTEST_BOTH_LIBRARIES} clickhouse_functions clickhouse_parsers dbms clickhouse_common_zookeeper)
|
||||
add_check(unit_tests_dbms)
|
||||
endif ()
|
||||
|
||||
@ -61,6 +61,8 @@ public:
|
||||
randomize(randomize_), max_iterations(max_iterations_), max_time(max_time_),
|
||||
json_path(json_path_), settings(settings_), global_context(Context::createGlobal()), pool(concurrency)
|
||||
{
|
||||
global_context.makeGlobalContext();
|
||||
|
||||
std::cerr << std::fixed << std::setprecision(3);
|
||||
|
||||
/// This is needed to receive blocks with columns of AggregateFunction data type
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(CLICKHOUSE_BENCHMARK_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/Benchmark.cpp)
|
||||
set(CLICKHOUSE_BENCHMARK_LINK PRIVATE clickhouse_aggregate_functions clickhouse_common_config clickhouse_common_io ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_BENCHMARK_LINK PRIVATE dbms clickhouse_aggregate_functions clickhouse_common_config ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_BENCHMARK_INCLUDE SYSTEM PRIVATE ${PCG_RANDOM_INCLUDE_DIR})
|
||||
|
||||
clickhouse_program_add(benchmark)
|
||||
|
||||
@ -3,7 +3,7 @@ set(CLICKHOUSE_CLIENT_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/ConnectionParameters.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_CLIENT_LINK PRIVATE clickhouse_common_config clickhouse_functions clickhouse_aggregate_functions clickhouse_common_io clickhouse_parsers string_utils ${LINE_EDITING_LIBS} ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_CLIENT_INCLUDE SYSTEM PRIVATE ${READLINE_INCLUDE_DIR} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/include)
|
||||
|
||||
include(CheckSymbolExists)
|
||||
|
||||
@ -218,6 +218,7 @@ private:
|
||||
|
||||
configReadClient(config(), home_path);
|
||||
|
||||
context.makeGlobalContext();
|
||||
context.setApplicationType(Context::ApplicationType::CLIENT);
|
||||
|
||||
/// settings and limits could be specified in config file, but passed settings has higher priority
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
# Also in utils
|
||||
|
||||
set(CLICKHOUSE_COMPRESSOR_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/Compressor.cpp)
|
||||
set(CLICKHOUSE_COMPRESSOR_LINK PRIVATE clickhouse_compression clickhouse_common_io ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_COMPRESSOR_LINK PRIVATE clickhouse_compression clickhouse_common_io clickhouse_parsers ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
#set(CLICKHOUSE_COMPRESSOR_INCLUDE SYSTEM PRIVATE ...)
|
||||
|
||||
clickhouse_program_add(compressor)
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(CLICKHOUSE_COPIER_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/ClusterCopier.cpp)
|
||||
set(CLICKHOUSE_COPIER_LINK PRIVATE clickhouse_functions clickhouse_table_functions clickhouse_aggregate_functions clickhouse_dictionaries PUBLIC daemon)
|
||||
set(CLICKHOUSE_COPIER_LINK PRIVATE clickhouse_common_zookeeper clickhouse_parsers clickhouse_functions clickhouse_table_functions clickhouse_aggregate_functions clickhouse_dictionaries string_utils PUBLIC daemon)
|
||||
set(CLICKHOUSE_COPIER_INCLUDE SYSTEM PRIVATE ${PCG_RANDOM_INCLUDE_DIR})
|
||||
|
||||
clickhouse_program_add(copier)
|
||||
|
||||
@ -2171,10 +2171,10 @@ void ClusterCopierApp::mainImpl()
|
||||
<< "revision " << ClickHouseRevision::get() << ")");
|
||||
|
||||
auto context = std::make_unique<Context>(Context::createGlobal());
|
||||
context->makeGlobalContext();
|
||||
SCOPE_EXIT(context->shutdown());
|
||||
|
||||
context->setConfig(loaded_config.configuration);
|
||||
context->setGlobalContext(*context);
|
||||
context->setApplicationType(Context::ApplicationType::LOCAL);
|
||||
context->setPath(process_path);
|
||||
|
||||
@ -2201,6 +2201,10 @@ void ClusterCopierApp::mainImpl()
|
||||
|
||||
copier->init();
|
||||
copier->process(ConnectionTimeouts::getTCPTimeoutsWithoutFailover(context->getSettingsRef()));
|
||||
|
||||
/// Reset ZooKeeper before removing ClusterCopier.
|
||||
/// Otherwise zookeeper watch can call callback which use already removed ClusterCopier object.
|
||||
context->resetZooKeeper();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(CLICKHOUSE_EXTRACT_FROM_CONFIG_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/ExtractFromConfig.cpp)
|
||||
set(CLICKHOUSE_EXTRACT_FROM_CONFIG_LINK PRIVATE clickhouse_common_config clickhouse_common_io ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_EXTRACT_FROM_CONFIG_LINK PRIVATE clickhouse_common_config clickhouse_common_io clickhouse_common_zookeeper ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
#set(CLICKHOUSE_EXTRACT_FROM_CONFIG_INCLUDE SYSTEM PRIVATE ...)
|
||||
|
||||
clickhouse_program_add(extract-from-config)
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
set(CLICKHOUSE_LOCAL_SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/LocalServer.cpp)
|
||||
set(CLICKHOUSE_LOCAL_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_LOCAL_LINK PRIVATE clickhouse_storages_system clickhouse_dictionaries clickhouse_common_config clickhouse_common_io clickhouse_functions clickhouse_aggregate_functions clickhouse_parsers clickhouse_table_functions ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
#set(CLICKHOUSE_LOCAL_INCLUDE SYSTEM PRIVATE ...)
|
||||
|
||||
clickhouse_program_add(local)
|
||||
|
||||
@ -131,7 +131,7 @@ try
|
||||
|
||||
|
||||
context = std::make_unique<Context>(Context::createGlobal());
|
||||
context->setGlobalContext(*context);
|
||||
context->makeGlobalContext();
|
||||
context->setApplicationType(Context::ApplicationType::LOCAL);
|
||||
tryInitPath();
|
||||
|
||||
@ -275,8 +275,8 @@ void LocalServer::processQueries()
|
||||
if (!parse_res.second)
|
||||
throw Exception("Cannot parse and execute the following part of query: " + String(parse_res.first), ErrorCodes::SYNTAX_ERROR);
|
||||
|
||||
context->setSessionContext(*context);
|
||||
context->setQueryContext(*context);
|
||||
context->makeSessionContext();
|
||||
context->makeQueryContext();
|
||||
|
||||
context->setUser("default", "", Poco::Net::SocketAddress{}, "");
|
||||
context->setCurrentQueryId("");
|
||||
|
||||
@ -1024,6 +1024,7 @@ try
|
||||
}
|
||||
|
||||
Context context = Context::createGlobal();
|
||||
context.makeGlobalContext();
|
||||
|
||||
ReadBufferFromFileDescriptor file_in(STDIN_FILENO);
|
||||
WriteBufferFromFileDescriptor file_out(STDOUT_FILENO);
|
||||
|
||||
@ -11,7 +11,7 @@ set(CLICKHOUSE_ODBC_BRIDGE_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/validateODBCConnectionString.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_ODBC_BRIDGE_LINK PRIVATE dbms clickhouse_common_io PUBLIC daemon)
|
||||
set(CLICKHOUSE_ODBC_BRIDGE_LINK PRIVATE dbms clickhouse_parsers PUBLIC daemon)
|
||||
set(CLICKHOUSE_ODBC_BRIDGE_INCLUDE PUBLIC ${ClickHouse_SOURCE_DIR}/libs/libdaemon/include)
|
||||
|
||||
if (USE_POCO_SQLODBC)
|
||||
|
||||
@ -160,7 +160,7 @@ int ODBCBridge::main(const std::vector<std::string> & /*args*/)
|
||||
http_params->setKeepAliveTimeout(keep_alive_timeout);
|
||||
|
||||
context = std::make_shared<Context>(Context::createGlobal());
|
||||
context->setGlobalContext(*context);
|
||||
context->makeGlobalContext();
|
||||
|
||||
auto server = Poco::Net::HTTPServer(
|
||||
new HandlerFactory("ODBCRequestHandlerFactory-factory", keep_alive_timeout, context), server_pool, socket, http_params);
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
set(CLICKHOUSE_PERFORMANCE_TEST_SOURCES
|
||||
set(CLICKHOUSE_PERFORMANCE_TEST_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/JSONString.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/StopConditionsSet.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/TestStopConditions.cpp
|
||||
@ -12,7 +12,7 @@ set(CLICKHOUSE_PERFORMANCE_TEST_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/PerformanceTestSuite.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_PERFORMANCE_TEST_LINK PRIVATE dbms clickhouse_common_io clickhouse_common_config ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_PERFORMANCE_TEST_LINK PRIVATE dbms clickhouse_common_config ${Boost_FILESYSTEM_LIBRARY} ${Boost_PROGRAM_OPTIONS_LIBRARY})
|
||||
set(CLICKHOUSE_PERFORMANCE_TEST_INCLUDE SYSTEM PRIVATE ${PCG_RANDOM_INCLUDE_DIR})
|
||||
|
||||
clickhouse_program_add(performance-test)
|
||||
|
||||
@ -89,6 +89,7 @@ public:
|
||||
, input_files(input_files_)
|
||||
, log(&Poco::Logger::get("PerformanceTestSuite"))
|
||||
{
|
||||
global_context.makeGlobalContext();
|
||||
global_context.getSettingsRef().copyChangesFrom(cmd_settings);
|
||||
if (input_files.size() < 1)
|
||||
throw Exception("No tests were specified", ErrorCodes::BAD_ARGUMENTS);
|
||||
|
||||
@ -12,7 +12,7 @@ set(CLICKHOUSE_SERVER_SOURCES
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
|
||||
)
|
||||
|
||||
set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})
|
||||
set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io clickhouse_common_config clickhouse_common_zookeeper clickhouse_parsers string_utils PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})
|
||||
if (USE_POCO_NETSSL)
|
||||
set(CLICKHOUSE_SERVER_LINK ${CLICKHOUSE_SERVER_LINK} PRIVATE ${Poco_NetSSL_LIBRARY} ${Poco_Crypto_LIBRARY})
|
||||
endif ()
|
||||
|
||||
@ -211,7 +211,6 @@ void HTTPHandler::processQuery(
|
||||
Output & used_output)
|
||||
{
|
||||
Context context = server.context();
|
||||
context.setGlobalContext(server.context());
|
||||
|
||||
CurrentThread::QueryScope query_scope(context);
|
||||
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
#include "MetricsTransmitter.h"
|
||||
|
||||
#include <Interpreters/AsynchronousMetrics.h>
|
||||
#include <Interpreters/Context.h>
|
||||
|
||||
#include <Common/CurrentMetrics.h>
|
||||
#include <Common/Exception.h>
|
||||
@ -16,6 +15,17 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
MetricsTransmitter::MetricsTransmitter(
|
||||
const Poco::Util::AbstractConfiguration & config, const std::string & config_name, const AsynchronousMetrics & async_metrics)
|
||||
: async_metrics(async_metrics), config_name(config_name)
|
||||
{
|
||||
interval_seconds = config.getInt(config_name + ".interval", 60);
|
||||
send_events = config.getBool(config_name + ".events", true);
|
||||
send_metrics = config.getBool(config_name + ".metrics", true);
|
||||
send_asynchronous_metrics = config.getBool(config_name + ".asynchronous_metrics", true);
|
||||
}
|
||||
|
||||
|
||||
MetricsTransmitter::~MetricsTransmitter()
|
||||
{
|
||||
try
|
||||
@ -38,10 +48,7 @@ MetricsTransmitter::~MetricsTransmitter()
|
||||
|
||||
void MetricsTransmitter::run()
|
||||
{
|
||||
const auto & config = context.getConfigRef();
|
||||
auto interval = config.getInt(config_name + ".interval", 60);
|
||||
|
||||
const std::string thread_name = "MetrTx" + std::to_string(interval);
|
||||
const std::string thread_name = "MetrTx" + std::to_string(interval_seconds);
|
||||
setThreadName(thread_name.c_str());
|
||||
|
||||
const auto get_next_time = [](size_t seconds)
|
||||
@ -60,7 +67,7 @@ void MetricsTransmitter::run()
|
||||
|
||||
while (true)
|
||||
{
|
||||
if (cond.wait_until(lock, get_next_time(interval), [this] { return quit; }))
|
||||
if (cond.wait_until(lock, get_next_time(interval_seconds), [this] { return quit; }))
|
||||
break;
|
||||
|
||||
transmit(prev_counters);
|
||||
@ -70,14 +77,12 @@ void MetricsTransmitter::run()
|
||||
|
||||
void MetricsTransmitter::transmit(std::vector<ProfileEvents::Count> & prev_counters)
|
||||
{
|
||||
const auto & config = context.getConfigRef();
|
||||
auto async_metrics_values = async_metrics.getValues();
|
||||
|
||||
GraphiteWriter::KeyValueVector<ssize_t> key_vals{};
|
||||
key_vals.reserve(ProfileEvents::end() + CurrentMetrics::end() + async_metrics_values.size());
|
||||
|
||||
|
||||
if (config.getBool(config_name + ".events", true))
|
||||
if (send_events)
|
||||
{
|
||||
for (size_t i = 0, end = ProfileEvents::end(); i < end; ++i)
|
||||
{
|
||||
@ -90,7 +95,7 @@ void MetricsTransmitter::transmit(std::vector<ProfileEvents::Count> & prev_count
|
||||
}
|
||||
}
|
||||
|
||||
if (config.getBool(config_name + ".metrics", true))
|
||||
if (send_metrics)
|
||||
{
|
||||
for (size_t i = 0, end = CurrentMetrics::end(); i < end; ++i)
|
||||
{
|
||||
@ -101,7 +106,7 @@ void MetricsTransmitter::transmit(std::vector<ProfileEvents::Count> & prev_count
|
||||
}
|
||||
}
|
||||
|
||||
if (config.getBool(config_name + ".asynchronous_metrics", true))
|
||||
if (send_asynchronous_metrics)
|
||||
{
|
||||
for (const auto & name_value : async_metrics_values)
|
||||
{
|
||||
@ -112,4 +117,5 @@ void MetricsTransmitter::transmit(std::vector<ProfileEvents::Count> & prev_count
|
||||
if (key_vals.size())
|
||||
BaseDaemon::instance().writeToGraphite(key_vals, config_name);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -9,14 +9,21 @@
|
||||
#include <Common/ThreadPool.h>
|
||||
|
||||
|
||||
namespace Poco
|
||||
{
|
||||
namespace Util
|
||||
{
|
||||
class AbstractConfiguration;
|
||||
}
|
||||
}
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class AsynchronousMetrics;
|
||||
class Context;
|
||||
|
||||
|
||||
/** Automatically sends
|
||||
/** Automatically sends
|
||||
* - difference of ProfileEvents;
|
||||
* - values of CurrentMetrics;
|
||||
* - values of AsynchronousMetrics;
|
||||
@ -25,33 +32,29 @@ class Context;
|
||||
class MetricsTransmitter
|
||||
{
|
||||
public:
|
||||
MetricsTransmitter(Context & context_,
|
||||
const AsynchronousMetrics & async_metrics_,
|
||||
const std::string & config_name_)
|
||||
: context(context_)
|
||||
, async_metrics(async_metrics_)
|
||||
, config_name(config_name_)
|
||||
{
|
||||
}
|
||||
MetricsTransmitter(const Poco::Util::AbstractConfiguration & config, const std::string & config_name, const AsynchronousMetrics & async_metrics);
|
||||
~MetricsTransmitter();
|
||||
|
||||
private:
|
||||
void run();
|
||||
void transmit(std::vector<ProfileEvents::Count> & prev_counters);
|
||||
|
||||
Context & context;
|
||||
|
||||
const AsynchronousMetrics & async_metrics;
|
||||
const std::string config_name;
|
||||
|
||||
std::string config_name;
|
||||
UInt32 interval_seconds;
|
||||
bool send_events;
|
||||
bool send_metrics;
|
||||
bool send_asynchronous_metrics;
|
||||
|
||||
bool quit = false;
|
||||
std::mutex mutex;
|
||||
std::condition_variable cond;
|
||||
ThreadFromGlobalPool thread{&MetricsTransmitter::run, this};
|
||||
|
||||
static constexpr auto profile_events_path_prefix = "ClickHouse.ProfileEvents.";
|
||||
static constexpr auto current_metrics_path_prefix = "ClickHouse.Metrics.";
|
||||
static constexpr auto asynchronous_metrics_path_prefix = "ClickHouse.AsynchronousMetrics.";
|
||||
static inline constexpr auto profile_events_path_prefix = "ClickHouse.ProfileEvents.";
|
||||
static inline constexpr auto current_metrics_path_prefix = "ClickHouse.Metrics.";
|
||||
static inline constexpr auto asynchronous_metrics_path_prefix = "ClickHouse.AsynchronousMetrics.";
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -48,7 +48,7 @@ MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & so
|
||||
void MySQLHandler::run()
|
||||
{
|
||||
connection_context = server.context();
|
||||
connection_context.setSessionContext(connection_context);
|
||||
connection_context.makeSessionContext();
|
||||
connection_context.setDefaultFormat("MySQLWire");
|
||||
|
||||
in = std::make_shared<ReadBufferFromPocoSocket>(socket());
|
||||
|
||||
@ -187,7 +187,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
* settings, available functions, data types, aggregate functions, databases...
|
||||
*/
|
||||
global_context = std::make_unique<Context>(Context::createGlobal());
|
||||
global_context->setGlobalContext(*global_context);
|
||||
global_context->makeGlobalContext();
|
||||
global_context->setApplicationType(Context::ApplicationType::SERVER);
|
||||
|
||||
bool has_zookeeper = config().has("zookeeper");
|
||||
@ -525,18 +525,26 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Initialize a watcher updating DNS cache in case of network errors
|
||||
dns_cache_updater = std::make_unique<DNSCacheUpdater>(*global_context);
|
||||
/// Initialize a watcher periodically updating DNS cache
|
||||
dns_cache_updater = std::make_unique<DNSCacheUpdater>(*global_context, config().getInt("dns_cache_update_period", 15));
|
||||
}
|
||||
|
||||
#if defined(__linux__)
|
||||
if (!TaskStatsInfoGetter::checkPermissions())
|
||||
{
|
||||
LOG_INFO(log, "It looks like the process has no CAP_NET_ADMIN capability, 'taskstats' performance statistics will be disabled."
|
||||
" It could happen due to incorrect ClickHouse package installation."
|
||||
" You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'."
|
||||
" Note that it will not work on 'nosuid' mounted filesystems."
|
||||
" It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.");
|
||||
" It could happen due to incorrect ClickHouse package installation."
|
||||
" You could resolve the problem manually with 'sudo setcap cap_net_admin=+ep /usr/bin/clickhouse'."
|
||||
" Note that it will not work on 'nosuid' mounted filesystems."
|
||||
" It also doesn't work if you run clickhouse-server inside network namespace as it happens in some containers.");
|
||||
}
|
||||
|
||||
if (!hasLinuxCapability(CAP_SYS_NICE))
|
||||
{
|
||||
LOG_INFO(log, "It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_nice' will have no effect."
|
||||
" It could happen due to incorrect ClickHouse package installation."
|
||||
" You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'."
|
||||
" Note that it will not work on 'nosuid' mounted filesystems.");
|
||||
}
|
||||
#else
|
||||
LOG_INFO(log, "TaskStats is not implemented for this OS. IO accounting will be disabled.");
|
||||
@ -773,6 +781,8 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
|
||||
main_config_reloader->start();
|
||||
users_config_reloader->start();
|
||||
if (dns_cache_updater)
|
||||
dns_cache_updater->start();
|
||||
|
||||
{
|
||||
std::stringstream message;
|
||||
@ -823,6 +833,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
log, "Closed connections." << (current_connections ? " But " + toString(current_connections) + " remains."
|
||||
" Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>" : ""));
|
||||
|
||||
dns_cache_updater.reset();
|
||||
main_config_reloader.reset();
|
||||
users_config_reloader.reset();
|
||||
});
|
||||
@ -850,7 +861,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
for (const auto & graphite_key : DB::getMultipleKeysFromConfig(config(), "", "graphite"))
|
||||
{
|
||||
metrics_transmitters.emplace_back(std::make_unique<MetricsTransmitter>(
|
||||
*global_context, async_metrics, graphite_key));
|
||||
global_context->getConfigRef(), graphite_key, async_metrics));
|
||||
}
|
||||
|
||||
SessionCleaner session_cleaner(*global_context);
|
||||
|
||||
@ -54,7 +54,7 @@ void TCPHandler::runImpl()
|
||||
ThreadStatus thread_status;
|
||||
|
||||
connection_context = server.context();
|
||||
connection_context.setSessionContext(connection_context);
|
||||
connection_context.makeSessionContext();
|
||||
|
||||
Settings global_settings = connection_context.getSettings();
|
||||
|
||||
@ -120,17 +120,26 @@ void TCPHandler::runImpl()
|
||||
|
||||
while (1)
|
||||
{
|
||||
/// Set context of request.
|
||||
query_context = connection_context;
|
||||
|
||||
/// We are waiting for a packet from the client. Thus, every `POLL_INTERVAL` seconds check whether we need to shut down.
|
||||
while (!static_cast<ReadBufferFromPocoSocket &>(*in).poll(global_settings.poll_interval * 1000000) && !server.isCancelled())
|
||||
;
|
||||
/// We are waiting for a packet from the client. Thus, every `poll_interval` seconds check whether we need to shut down.
|
||||
{
|
||||
Stopwatch idle_time;
|
||||
while (!static_cast<ReadBufferFromPocoSocket &>(*in).poll(global_settings.poll_interval * 1000000) && !server.isCancelled())
|
||||
{
|
||||
if (idle_time.elapsedSeconds() > global_settings.idle_connection_timeout)
|
||||
{
|
||||
LOG_TRACE(log, "Closing idle connection");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// If we need to shut down, or client disconnects.
|
||||
if (server.isCancelled() || in->eof())
|
||||
break;
|
||||
|
||||
/// Set context of request.
|
||||
query_context = connection_context;
|
||||
|
||||
Stopwatch watch;
|
||||
state.reset();
|
||||
|
||||
|
||||
@ -36,6 +36,12 @@ struct QueryState
|
||||
QueryProcessingStage::Enum stage = QueryProcessingStage::Complete;
|
||||
Protocol::Compression compression = Protocol::Compression::Disable;
|
||||
|
||||
/// A queue with internal logs that will be passed to client. It must be
|
||||
/// destroyed after input/output blocks, because they may contain other
|
||||
/// threads that use this queue.
|
||||
InternalTextLogsQueuePtr logs_queue;
|
||||
BlockOutputStreamPtr logs_block_out;
|
||||
|
||||
/// From where to read data for INSERT.
|
||||
std::shared_ptr<ReadBuffer> maybe_compressed_in;
|
||||
BlockInputStreamPtr block_in;
|
||||
@ -64,10 +70,6 @@ struct QueryState
|
||||
/// Timeouts setter for current query
|
||||
std::unique_ptr<TimeoutSetter> timeout_setter;
|
||||
|
||||
/// A queue with internal logs that will be passed to client
|
||||
InternalTextLogsQueuePtr logs_queue;
|
||||
BlockOutputStreamPtr logs_block_out;
|
||||
|
||||
void reset()
|
||||
{
|
||||
*this = QueryState();
|
||||
|
||||
@ -82,14 +82,6 @@ public:
|
||||
{
|
||||
if (!returns_many && levels.size() > 1)
|
||||
throw Exception("Aggregate function " + getName() + " require one parameter or less", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
|
||||
if constexpr (std::is_same_v<Data, QuantileTiming<Value>>)
|
||||
{
|
||||
/// QuantileTiming only supports integers (it works only for unsigned integers but signed are also accepted for convenience).
|
||||
if (!isInteger(argument_type))
|
||||
throw Exception("Argument for function " + std::string(Name::name) + " must be integer, but it has type "
|
||||
+ argument_type->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
}
|
||||
|
||||
String getName() const override { return Name::name; }
|
||||
@ -111,16 +103,21 @@ public:
|
||||
|
||||
void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override
|
||||
{
|
||||
/// Out of range conversion may occur. This is Ok.
|
||||
auto value = static_cast<const ColVecType &>(*columns[0]).getData()[row_num];
|
||||
|
||||
const auto & column = static_cast<const ColVecType &>(*columns[0]);
|
||||
if constexpr (std::is_same_v<Data, QuantileTiming<Value>>)
|
||||
{
|
||||
/// QuantileTiming only supports integers.
|
||||
if (isNaN(value) || value > std::numeric_limits<Value>::max() || value < std::numeric_limits<Value>::min())
|
||||
return;
|
||||
}
|
||||
|
||||
if constexpr (has_second_arg)
|
||||
this->data(place).add(
|
||||
column.getData()[row_num],
|
||||
value,
|
||||
columns[1]->getUInt(row_num));
|
||||
else
|
||||
this->data(place).add(column.getData()[row_num]);
|
||||
this->data(place).add(value);
|
||||
}
|
||||
|
||||
void merge(AggregateDataPtr place, ConstAggregateDataPtr rhs, Arena *) const override
|
||||
|
||||
@ -73,7 +73,7 @@ void Connection::connect(const ConnectionTimeouts & timeouts)
|
||||
|
||||
current_resolved_address = DNSResolver::instance().resolveAddress(host, port);
|
||||
|
||||
socket->connect(current_resolved_address, timeouts.connection_timeout);
|
||||
socket->connect(*current_resolved_address, timeouts.connection_timeout);
|
||||
socket->setReceiveTimeout(timeouts.receive_timeout);
|
||||
socket->setSendTimeout(timeouts.send_timeout);
|
||||
socket->setNoDelay(true);
|
||||
@ -533,12 +533,9 @@ void Connection::sendExternalTablesData(ExternalTablesData & data)
|
||||
LOG_DEBUG(log_wrapper.get(), msg.rdbuf());
|
||||
}
|
||||
|
||||
Poco::Net::SocketAddress Connection::getResolvedAddress() const
|
||||
std::optional<Poco::Net::SocketAddress> Connection::getResolvedAddress() const
|
||||
{
|
||||
if (connected)
|
||||
return current_resolved_address;
|
||||
|
||||
return DNSResolver::instance().resolveAddress(host, port);
|
||||
return current_resolved_address;
|
||||
}
|
||||
|
||||
|
||||
@ -595,7 +592,9 @@ Connection::Packet Connection::receivePacket()
|
||||
|
||||
switch (res.type)
|
||||
{
|
||||
case Protocol::Server::Data:
|
||||
case Protocol::Server::Data: [[fallthrough]];
|
||||
case Protocol::Server::Totals: [[fallthrough]];
|
||||
case Protocol::Server::Extremes:
|
||||
res.block = receiveData();
|
||||
return res;
|
||||
|
||||
@ -611,16 +610,6 @@ Connection::Packet Connection::receivePacket()
|
||||
res.profile_info = receiveProfileInfo();
|
||||
return res;
|
||||
|
||||
case Protocol::Server::Totals:
|
||||
/// Block with total values is passed in same form as ordinary block. The only difference is packed id.
|
||||
res.block = receiveData();
|
||||
return res;
|
||||
|
||||
case Protocol::Server::Extremes:
|
||||
/// Same as above.
|
||||
res.block = receiveData();
|
||||
return res;
|
||||
|
||||
case Protocol::Server::Log:
|
||||
res.block = receiveLogData();
|
||||
return res;
|
||||
@ -720,11 +709,14 @@ void Connection::initBlockLogsInput()
|
||||
void Connection::setDescription()
|
||||
{
|
||||
auto resolved_address = getResolvedAddress();
|
||||
description = host + ":" + toString(resolved_address.port());
|
||||
auto ip_address = resolved_address.host().toString();
|
||||
description = host + ":" + toString(port);
|
||||
|
||||
if (host != ip_address)
|
||||
description += ", " + ip_address;
|
||||
if (resolved_address)
|
||||
{
|
||||
auto ip_address = resolved_address->host().toString();
|
||||
if (host != ip_address)
|
||||
description += ", " + ip_address;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -63,7 +63,7 @@ public:
|
||||
Poco::Timespan sync_request_timeout_ = Poco::Timespan(DBMS_DEFAULT_SYNC_REQUEST_TIMEOUT_SEC, 0))
|
||||
:
|
||||
host(host_), port(port_), default_database(default_database_),
|
||||
user(user_), password(password_), current_resolved_address(host, port),
|
||||
user(user_), password(password_),
|
||||
client_name(client_name_),
|
||||
compression(compression_),
|
||||
secure(secure_),
|
||||
@ -168,9 +168,6 @@ public:
|
||||
size_t outBytesCount() const { return out ? out->count() : 0; }
|
||||
size_t inBytesCount() const { return in ? in->count() : 0; }
|
||||
|
||||
/// Returns initially resolved address
|
||||
Poco::Net::SocketAddress getResolvedAddress() const;
|
||||
|
||||
private:
|
||||
String host;
|
||||
UInt16 port;
|
||||
@ -180,12 +177,15 @@ private:
|
||||
|
||||
/// Address is resolved during the first connection (or the following reconnects)
|
||||
/// Use it only for logging purposes
|
||||
Poco::Net::SocketAddress current_resolved_address;
|
||||
std::optional<Poco::Net::SocketAddress> current_resolved_address;
|
||||
|
||||
/// For messages in log and in exceptions.
|
||||
String description;
|
||||
void setDescription();
|
||||
|
||||
/// Returns resolved address if it was resolved.
|
||||
std::optional<Poco::Net::SocketAddress> getResolvedAddress() const;
|
||||
|
||||
String client_name;
|
||||
|
||||
bool connected = false;
|
||||
|
||||
@ -1,4 +0,0 @@
|
||||
if(USE_GTEST)
|
||||
add_executable(column_unique column_unique.cpp)
|
||||
target_link_libraries(column_unique PRIVATE dbms ${GTEST_BOTH_LIBRARIES})
|
||||
endif()
|
||||
@ -7,12 +7,6 @@
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <unordered_map>
|
||||
@ -14,8 +14,9 @@ namespace DB
|
||||
namespace Cpu
|
||||
{
|
||||
|
||||
#if defined(__x86_64__) || defined(__i386__)
|
||||
inline UInt64 _xgetbv(UInt32 xcr) noexcept
|
||||
#if (defined(__x86_64__) || defined(__i386__))
|
||||
/// Our version is independent of -mxsave option, because we do dynamic dispatch.
|
||||
inline UInt64 our_xgetbv(UInt32 xcr) noexcept
|
||||
{
|
||||
UInt32 eax;
|
||||
UInt32 edx;
|
||||
@ -185,7 +186,7 @@ bool haveAVX() noexcept
|
||||
// http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf
|
||||
// https://bugs.chromium.org/p/chromium/issues/detail?id=375968
|
||||
return haveOSXSAVE() // implies haveXSAVE()
|
||||
&& (_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
|
||||
&& (our_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
|
||||
&& ((CpuInfo(0x1).ecx >> 28) & 1u); // AVX bit
|
||||
#else
|
||||
return false;
|
||||
@ -217,8 +218,8 @@ bool haveAVX512F() noexcept
|
||||
#if defined(__x86_64__) || defined(__i386__)
|
||||
// https://software.intel.com/en-us/articles/how-to-detect-knl-instruction-support
|
||||
return haveOSXSAVE() // implies haveXSAVE()
|
||||
&& (_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
|
||||
&& ((_xgetbv(0) >> 5) & 7u) == 7u // ZMM state is enabled by OS
|
||||
&& (our_xgetbv(0) & 6u) == 6u // XMM state and YMM state are enabled by OS
|
||||
&& ((our_xgetbv(0) >> 5) & 7u) == 7u // ZMM state is enabled by OS
|
||||
&& CpuInfo(0x0).eax >= 0x7 // leaf 7 is present
|
||||
&& ((CpuInfo(0x7).ebx >> 16) & 1u); // AVX512F bit
|
||||
#else
|
||||
|
||||
@ -23,7 +23,7 @@ void CurrentThread::updatePerformanceCounters()
|
||||
{
|
||||
if (unlikely(!current_thread))
|
||||
return;
|
||||
get().updatePerformanceCounters();
|
||||
current_thread->updatePerformanceCounters();
|
||||
}
|
||||
|
||||
ThreadStatus & CurrentThread::get()
|
||||
@ -36,35 +36,35 @@ ThreadStatus & CurrentThread::get()
|
||||
|
||||
ProfileEvents::Counters & CurrentThread::getProfileEvents()
|
||||
{
|
||||
return current_thread ? get().performance_counters : ProfileEvents::global_counters;
|
||||
return current_thread ? current_thread->performance_counters : ProfileEvents::global_counters;
|
||||
}
|
||||
|
||||
MemoryTracker * CurrentThread::getMemoryTracker()
|
||||
{
|
||||
if (unlikely(!current_thread))
|
||||
return nullptr;
|
||||
return &get().memory_tracker;
|
||||
return ¤t_thread->memory_tracker;
|
||||
}
|
||||
|
||||
void CurrentThread::updateProgressIn(const Progress & value)
|
||||
{
|
||||
if (unlikely(!current_thread))
|
||||
return;
|
||||
get().progress_in.incrementPiecewiseAtomically(value);
|
||||
current_thread->progress_in.incrementPiecewiseAtomically(value);
|
||||
}
|
||||
|
||||
void CurrentThread::updateProgressOut(const Progress & value)
|
||||
{
|
||||
if (unlikely(!current_thread))
|
||||
return;
|
||||
get().progress_out.incrementPiecewiseAtomically(value);
|
||||
current_thread->progress_out.incrementPiecewiseAtomically(value);
|
||||
}
|
||||
|
||||
void CurrentThread::attachInternalTextLogsQueue(const std::shared_ptr<InternalTextLogsQueue> & logs_queue)
|
||||
{
|
||||
if (unlikely(!current_thread))
|
||||
return;
|
||||
get().attachInternalTextLogsQueue(logs_queue);
|
||||
current_thread->attachInternalTextLogsQueue(logs_queue);
|
||||
}
|
||||
|
||||
std::shared_ptr<InternalTextLogsQueue> CurrentThread::getInternalTextLogsQueue()
|
||||
@ -73,10 +73,10 @@ std::shared_ptr<InternalTextLogsQueue> CurrentThread::getInternalTextLogsQueue()
|
||||
if (unlikely(!current_thread))
|
||||
return nullptr;
|
||||
|
||||
if (get().getCurrentState() == ThreadStatus::ThreadState::Died)
|
||||
if (current_thread->getCurrentState() == ThreadStatus::ThreadState::Died)
|
||||
return nullptr;
|
||||
|
||||
return get().getInternalTextLogsQueue();
|
||||
return current_thread->getInternalTextLogsQueue();
|
||||
}
|
||||
|
||||
ThreadGroupStatusPtr CurrentThread::getGroup()
|
||||
@ -84,7 +84,7 @@ ThreadGroupStatusPtr CurrentThread::getGroup()
|
||||
if (unlikely(!current_thread))
|
||||
return nullptr;
|
||||
|
||||
return get().getThreadGroup();
|
||||
return current_thread->getThreadGroup();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include <common/StringRef.h>
|
||||
#include <Common/ThreadStatus.h>
|
||||
|
||||
|
||||
@ -69,7 +70,7 @@ public:
|
||||
static void finalizePerformanceCounters();
|
||||
|
||||
/// Returns a non-empty string if the thread is attached to a query
|
||||
static const std::string & getQueryId();
|
||||
static StringRef getQueryId();
|
||||
|
||||
/// Non-master threads call this method in destructor automatically
|
||||
static void detachQuery();
|
||||
|
||||
@ -1,14 +1,23 @@
|
||||
#include "DNSResolver.h"
|
||||
#include <common/SimpleCache.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/ProfileEvents.h>
|
||||
#include <Core/Names.h>
|
||||
#include <Core/Types.h>
|
||||
#include <Poco/Net/DNS.h>
|
||||
#include <Poco/Net/NetException.h>
|
||||
#include <Poco/NumberParser.h>
|
||||
#include <Poco/Logger.h>
|
||||
#include <common/logger_useful.h>
|
||||
#include <arpa/inet.h>
|
||||
#include <atomic>
|
||||
#include <optional>
|
||||
|
||||
namespace ProfileEvents
|
||||
{
|
||||
extern Event NetworkErrors;
|
||||
}
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -67,7 +76,6 @@ static void splitHostAndPort(const std::string & host_and_port, std::string & ou
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
static Poco::Net::IPAddress resolveIPAddressImpl(const std::string & host)
|
||||
{
|
||||
/// NOTE: Poco::Net::DNS::resolveOne(host) doesn't work for IP addresses like 127.0.0.2
|
||||
@ -75,15 +83,22 @@ static Poco::Net::IPAddress resolveIPAddressImpl(const std::string & host)
|
||||
return Poco::Net::SocketAddress(host, 0U).host();
|
||||
}
|
||||
|
||||
|
||||
struct DNSResolver::Impl
|
||||
{
|
||||
SimpleCache<decltype(resolveIPAddressImpl), &resolveIPAddressImpl> cache_host;
|
||||
|
||||
std::mutex drop_mutex;
|
||||
std::mutex update_mutex;
|
||||
|
||||
/// Cached server host name
|
||||
std::mutex mutex;
|
||||
std::optional<String> host_name;
|
||||
|
||||
/// Store hosts, which was asked to resolve from last update of DNS cache.
|
||||
NameSet new_hosts;
|
||||
|
||||
/// Store all hosts, which was whenever asked to resolve
|
||||
NameSet known_hosts;
|
||||
|
||||
/// If disabled, will not make cache lookups, will resolve addresses manually on each call
|
||||
std::atomic<bool> disable_cache{false};
|
||||
};
|
||||
@ -93,28 +108,43 @@ DNSResolver::DNSResolver() : impl(std::make_unique<DNSResolver::Impl>()) {}
|
||||
|
||||
Poco::Net::IPAddress DNSResolver::resolveHost(const std::string & host)
|
||||
{
|
||||
return !impl->disable_cache ? impl->cache_host(host) : resolveIPAddressImpl(host);
|
||||
if (impl->disable_cache)
|
||||
return resolveIPAddressImpl(host);
|
||||
|
||||
addToNewHosts(host);
|
||||
return impl->cache_host(host);
|
||||
}
|
||||
|
||||
Poco::Net::SocketAddress DNSResolver::resolveAddress(const std::string & host_and_port)
|
||||
{
|
||||
if (impl->disable_cache)
|
||||
return Poco::Net::SocketAddress(host_and_port);
|
||||
|
||||
String host;
|
||||
UInt16 port;
|
||||
splitHostAndPort(host_and_port, host, port);
|
||||
|
||||
return !impl->disable_cache ? Poco::Net::SocketAddress(impl->cache_host(host), port) : Poco::Net::SocketAddress(host_and_port);
|
||||
addToNewHosts(host);
|
||||
return Poco::Net::SocketAddress(impl->cache_host(host), port);
|
||||
}
|
||||
|
||||
Poco::Net::SocketAddress DNSResolver::resolveAddress(const std::string & host, UInt16 port)
|
||||
{
|
||||
return !impl->disable_cache ? Poco::Net::SocketAddress(impl->cache_host(host), port) : Poco::Net::SocketAddress(host, port);
|
||||
if (impl->disable_cache)
|
||||
return Poco::Net::SocketAddress(host, port);
|
||||
|
||||
addToNewHosts(host);
|
||||
return Poco::Net::SocketAddress(impl->cache_host(host), port);
|
||||
}
|
||||
|
||||
void DNSResolver::dropCache()
|
||||
{
|
||||
impl->cache_host.drop();
|
||||
|
||||
std::unique_lock lock(impl->mutex);
|
||||
std::scoped_lock lock(impl->update_mutex, impl->drop_mutex);
|
||||
|
||||
impl->known_hosts.clear();
|
||||
impl->new_hosts.clear();
|
||||
impl->host_name.reset();
|
||||
}
|
||||
|
||||
@ -128,7 +158,7 @@ String DNSResolver::getHostName()
|
||||
if (impl->disable_cache)
|
||||
return Poco::Net::DNS::hostName();
|
||||
|
||||
std::unique_lock lock(impl->mutex);
|
||||
std::lock_guard lock(impl->drop_mutex);
|
||||
|
||||
if (!impl->host_name.has_value())
|
||||
impl->host_name.emplace(Poco::Net::DNS::hostName());
|
||||
@ -136,6 +166,61 @@ String DNSResolver::getHostName()
|
||||
return *impl->host_name;
|
||||
}
|
||||
|
||||
bool DNSResolver::updateCache()
|
||||
{
|
||||
{
|
||||
std::lock_guard lock(impl->drop_mutex);
|
||||
for (auto & host : impl->new_hosts)
|
||||
impl->known_hosts.insert(std::move(host));
|
||||
impl->new_hosts.clear();
|
||||
|
||||
impl->host_name.emplace(Poco::Net::DNS::hostName());
|
||||
}
|
||||
|
||||
std::lock_guard lock(impl->update_mutex);
|
||||
|
||||
bool updated = false;
|
||||
String lost_hosts;
|
||||
for (const auto & host : impl->known_hosts)
|
||||
{
|
||||
try
|
||||
{
|
||||
updated |= updateHost(host);
|
||||
}
|
||||
catch (const Poco::Net::NetException &)
|
||||
{
|
||||
ProfileEvents::increment(ProfileEvents::NetworkErrors);
|
||||
|
||||
if (!lost_hosts.empty())
|
||||
lost_hosts += ", ";
|
||||
lost_hosts += host;
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
}
|
||||
}
|
||||
|
||||
if (!lost_hosts.empty())
|
||||
LOG_INFO(&Logger::get("DNSResolver"), "Cached hosts not found: " << lost_hosts);
|
||||
|
||||
return updated;
|
||||
}
|
||||
|
||||
bool DNSResolver::updateHost(const String & host)
|
||||
{
|
||||
/// Usage of updateHost implies that host is already in cache and there is no extra computations
|
||||
auto old_value = impl->cache_host(host);
|
||||
impl->cache_host.update(host);
|
||||
return old_value != impl->cache_host(host);
|
||||
}
|
||||
|
||||
void DNSResolver::addToNewHosts(const String & host)
|
||||
{
|
||||
std::lock_guard lock(impl->drop_mutex);
|
||||
impl->new_hosts.insert(host);
|
||||
}
|
||||
|
||||
DNSResolver::~DNSResolver() = default;
|
||||
|
||||
|
||||
|
||||
@ -4,13 +4,15 @@
|
||||
#include <memory>
|
||||
#include <ext/singleton.h>
|
||||
#include <Core/Types.h>
|
||||
#include <Core/Names.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// A singleton implementing DNS names resolving with optional permanent DNS cache
|
||||
/// The cache could be updated only manually via drop() method
|
||||
/// A singleton implementing DNS names resolving with optional DNS cache
|
||||
/// The cache is being updated asynchronous in separate thread (see DNSCacheUpdater)
|
||||
/// or it could be updated manually via drop() method.
|
||||
class DNSResolver : public ext::singleton<DNSResolver>
|
||||
{
|
||||
public:
|
||||
@ -34,9 +36,13 @@ public:
|
||||
/// Drops all caches
|
||||
void dropCache();
|
||||
|
||||
/// Updates all known hosts in cache.
|
||||
/// Returns true if IP of any host has been changed.
|
||||
bool updateCache();
|
||||
|
||||
~DNSResolver();
|
||||
|
||||
protected:
|
||||
private:
|
||||
|
||||
DNSResolver();
|
||||
|
||||
@ -44,6 +50,11 @@ protected:
|
||||
|
||||
struct Impl;
|
||||
std::unique_ptr<Impl> impl;
|
||||
|
||||
/// Returns true if IP of host has been changed.
|
||||
bool updateHost(const String & host);
|
||||
|
||||
void addToNewHosts(const String & host);
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -433,6 +433,7 @@ namespace ErrorCodes
|
||||
extern const int UNKNOWN_QUERY_PARAMETER = 456;
|
||||
extern const int BAD_QUERY_PARAMETER = 457;
|
||||
extern const int CANNOT_UNLINK = 458;
|
||||
extern const int CANNOT_SET_THREAD_PRIORITY = 459;
|
||||
|
||||
extern const int KEEPER_EXCEPTION = 999;
|
||||
extern const int POCO_EXCEPTION = 1000;
|
||||
|
||||
@ -167,6 +167,16 @@ public:
|
||||
}
|
||||
|
||||
auto min = counter_list.back();
|
||||
// The key doesn't exist and cannot fit in the current top K, but
|
||||
// the new key has a bigger weight and is virtually more present
|
||||
// compared to the element who is less present on the set. This part
|
||||
// of the code is useful for the function topKWeighted
|
||||
if (increment > min->count)
|
||||
{
|
||||
destroyLastElement();
|
||||
push(new Counter(arena.emplace(key), increment, error, hash));
|
||||
return;
|
||||
}
|
||||
const size_t alpha_mask = alpha_map.size() - 1;
|
||||
auto & alpha = alpha_map[hash & alpha_mask];
|
||||
if (alpha + increment < min->count)
|
||||
@ -333,6 +343,17 @@ private:
|
||||
alpha_map.clear();
|
||||
}
|
||||
|
||||
void destroyLastElement()
|
||||
{
|
||||
auto last_element = counter_list.back();
|
||||
auto cell = counter_map.find(last_element->key, last_element->hash);
|
||||
cell->setZero();
|
||||
counter_map.reinsert(cell, last_element->hash);
|
||||
counter_list.pop_back();
|
||||
arena.free(last_element->key);
|
||||
delete last_element;
|
||||
}
|
||||
|
||||
HashMap<TKey, Counter *, Hash, Grower, Allocator> counter_map;
|
||||
std::vector<Counter *> counter_list;
|
||||
std::vector<UInt64> alpha_map;
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#pragma once
|
||||
|
||||
#include <common/StringRef.h>
|
||||
#include <Common/ProfileEvents.h>
|
||||
#include <Common/MemoryTracker.h>
|
||||
|
||||
@ -85,6 +86,8 @@ public:
|
||||
UInt32 thread_number = 0;
|
||||
/// Linux's PID (or TGID) (the same id is shown by ps util)
|
||||
Int32 os_thread_id = -1;
|
||||
/// Also called "nice" value. If it was changed to non-zero (when attaching query) - will be reset to zero when query is detached.
|
||||
Int32 os_thread_priority = 0;
|
||||
|
||||
/// TODO: merge them into common entity
|
||||
ProfileEvents::Counters performance_counters{VariableContext::Thread};
|
||||
@ -114,7 +117,7 @@ public:
|
||||
return thread_state.load(std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
const std::string & getQueryId() const;
|
||||
StringRef getQueryId() const;
|
||||
|
||||
/// Starts new query and create new thread group for it, current thread becomes master thread of the query
|
||||
void initializeQuery();
|
||||
|
||||
@ -5,11 +5,6 @@
|
||||
#include <iostream>
|
||||
#include <chrono>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <Common/ShellCommand.h>
|
||||
|
||||
@ -153,7 +153,7 @@ void formatIPv6(const unsigned char * src, char *& dst, UInt8 zeroed_tail_bytes_
|
||||
}
|
||||
|
||||
/// Was it a trailing run of 0x00's?
|
||||
if (best.base != -1 && size_t(best.base + best.len) == words.size())
|
||||
if (best.base != -1 && size_t(best.base) + size_t(best.len) == words.size())
|
||||
*dst++ = ':';
|
||||
|
||||
*dst++ = '\0';
|
||||
|
||||
@ -23,9 +23,7 @@ namespace DB
|
||||
* - the routing rules that affect which network interface we go to the specified address are not checked.
|
||||
*/
|
||||
bool isLocalAddress(const Poco::Net::SocketAddress & address, UInt16 clickhouse_port);
|
||||
|
||||
bool isLocalAddress(const Poco::Net::SocketAddress & address);
|
||||
|
||||
bool isLocalAddress(const Poco::Net::IPAddress & address);
|
||||
|
||||
/// Returns number of different bytes in hostnames, used for load balancing
|
||||
|
||||
@ -1,8 +1,3 @@
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <Common/RWLock.h>
|
||||
|
||||
@ -9,11 +9,6 @@
|
||||
#include <chrono>
|
||||
#include <thread>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
|
||||
|
||||
@ -1,10 +1,5 @@
|
||||
#include <Common/ThreadPool.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
/** Reproduces bug in ThreadPool.
|
||||
|
||||
@ -2,11 +2,6 @@
|
||||
#include <iostream>
|
||||
#include <Common/ThreadPool.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
/// Test for thread self-removal when number of free threads in pool is too large.
|
||||
|
||||
@ -2,11 +2,6 @@
|
||||
#include <iostream>
|
||||
#include <Common/ThreadPool.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
|
||||
|
||||
@ -2,11 +2,6 @@
|
||||
#include <stdexcept>
|
||||
#include <Common/ThreadPool.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
|
||||
|
||||
@ -1,10 +1,5 @@
|
||||
#include <Common/escapeForFileName.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
|
||||
|
||||
@ -34,7 +34,7 @@ struct AggregateIndependent
|
||||
{
|
||||
results.reserve(num_threads);
|
||||
for (size_t i = 0; i < num_threads; ++i)
|
||||
results.emplace_back(new Map);
|
||||
results.emplace_back(std::make_unique<Map>());
|
||||
|
||||
for (size_t i = 0; i < num_threads; ++i)
|
||||
{
|
||||
@ -77,7 +77,7 @@ struct AggregateIndependentWithSequentialKeysOptimization
|
||||
{
|
||||
results.reserve(num_threads);
|
||||
for (size_t i = 0; i < num_threads; ++i)
|
||||
results.emplace_back(new Map);
|
||||
results.emplace_back(std::make_unique<Map>());
|
||||
|
||||
for (size_t i = 0; i < num_threads; ++i)
|
||||
{
|
||||
|
||||
@ -20,11 +20,6 @@
|
||||
#include <bitset>
|
||||
#include <string.h>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
using namespace DB;
|
||||
|
||||
@ -717,8 +717,8 @@ class WriteBuffer;
|
||||
/// It is assumed that all elements of the array have the same type.
|
||||
void readBinary(Array & x, ReadBuffer & buf);
|
||||
|
||||
inline void readText(Array &, ReadBuffer &) { throw Exception("Cannot read Array.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
inline void readQuoted(Array &, ReadBuffer &) { throw Exception("Cannot read Array.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void readText(Array &, ReadBuffer &) { throw Exception("Cannot read Array.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void readQuoted(Array &, ReadBuffer &) { throw Exception("Cannot read Array.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
|
||||
/// It is assumed that all elements of the array have the same type.
|
||||
/// Also write size and type into buf. UInt64 and Int64 is written in variadic size form
|
||||
@ -726,16 +726,16 @@ void writeBinary(const Array & x, WriteBuffer & buf);
|
||||
|
||||
void writeText(const Array & x, WriteBuffer & buf);
|
||||
|
||||
inline void writeQuoted(const Array &, WriteBuffer &) { throw Exception("Cannot write Array quoted.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void writeQuoted(const Array &, WriteBuffer &) { throw Exception("Cannot write Array quoted.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
|
||||
void readBinary(Tuple & x, ReadBuffer & buf);
|
||||
|
||||
inline void readText(Tuple &, ReadBuffer &) { throw Exception("Cannot read Tuple.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
inline void readQuoted(Tuple &, ReadBuffer &) { throw Exception("Cannot read Tuple.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void readText(Tuple &, ReadBuffer &) { throw Exception("Cannot read Tuple.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void readQuoted(Tuple &, ReadBuffer &) { throw Exception("Cannot read Tuple.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
|
||||
void writeBinary(const Tuple & x, WriteBuffer & buf);
|
||||
|
||||
void writeText(const Tuple & x, WriteBuffer & buf);
|
||||
|
||||
inline void writeQuoted(const Tuple &, WriteBuffer &) { throw Exception("Cannot write Tuple quoted.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
[[noreturn]] inline void writeQuoted(const Tuple &, WriteBuffer &) { throw Exception("Cannot write Tuple quoted.", ErrorCodes::NOT_IMPLEMENTED); }
|
||||
}
|
||||
|
||||
@ -62,6 +62,7 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingSeconds, tcp_keep_alive_timeout, 0, "") \
|
||||
M(SettingMilliseconds, queue_max_wait_ms, 5000, "The wait time in the request queue, if the number of concurrent requests exceeds the maximum.") \
|
||||
M(SettingUInt64, poll_interval, DBMS_DEFAULT_POLL_INTERVAL, "Block at the query wait loop on the server for the specified number of seconds.") \
|
||||
M(SettingUInt64, idle_connection_timeout, 3600, "Close idle TCP connections after specified number of seconds.") \
|
||||
M(SettingUInt64, distributed_connections_pool_size, DBMS_DEFAULT_DISTRIBUTED_CONNECTIONS_POOL_SIZE, "Maximum number of connections with one remote server in the pool.") \
|
||||
M(SettingUInt64, connections_with_failover_max_tries, DBMS_CONNECTION_POOL_WITH_FAILOVER_DEFAULT_MAX_TRIES, "The maximum number of attempts to connect to replicas.") \
|
||||
M(SettingBool, extremes, false, "Calculate minimums and maximums of the result columns. They can be output in JSON-formats.") \
|
||||
@ -125,12 +126,14 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingUInt64, mark_cache_min_lifetime, 10000, "If the maximum size of mark_cache is exceeded, delete only records older than mark_cache_min_lifetime seconds.") \
|
||||
\
|
||||
M(SettingFloat, max_streams_to_max_threads_ratio, 1, "Allows you to use more sources than the number of threads - to more evenly distribute work across threads. It is assumed that this is a temporary solution, since it will be possible in the future to make the number of sources equal to the number of threads, but for each source to dynamically select available work for itself.") \
|
||||
M(SettingFloat, max_streams_multiplier_for_merge_tables, 5, "Ask more streams when reading from Merge table. Streams will be spread across tables that Merge table will use. This allows more even distribution of work across threads and especially helpful when merged tables differ in size.") \
|
||||
\
|
||||
M(SettingString, network_compression_method, "LZ4", "Allows you to select the method of data compression when writing.") \
|
||||
\
|
||||
M(SettingInt64, network_zstd_compression_level, 1, "Allows you to select the level of ZSTD compression.") \
|
||||
\
|
||||
M(SettingUInt64, priority, 0, "Priority of the query. 1 - the highest, higher value - lower priority; 0 - do not use priorities.") \
|
||||
M(SettingInt64, os_thread_priority, 0, "If non zero - set corresponding 'nice' value for query processing threads. Can be used to adjust query priority for OS scheduler.") \
|
||||
\
|
||||
M(SettingBool, log_queries, 0, "Log requests and write the log to the system table.") \
|
||||
\
|
||||
|
||||
@ -275,7 +275,7 @@ namespace details
|
||||
{
|
||||
static void serializeName(const StringRef & name, WriteBuffer & buf);
|
||||
static String deserializeName(ReadBuffer & buf);
|
||||
static void throwNameNotFound(const StringRef & name);
|
||||
[[noreturn]] static void throwNameNotFound(const StringRef & name);
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@ -68,6 +68,7 @@ try
|
||||
CollapsingFinalBlockInputStream collapsed(inputs, descr, "Sign");
|
||||
|
||||
Context context = Context::createGlobal();
|
||||
context.makeGlobalContext();
|
||||
WriteBufferFromFileDescriptor out_buf(STDERR_FILENO);
|
||||
BlockOutputStreamPtr output = context.getOutputFormat("TabSeparated", out_buf, block1);
|
||||
|
||||
|
||||
@ -35,6 +35,7 @@ try
|
||||
ASTPtr ast = parseQuery(parser, input.data(), input.data() + input.size(), "", 0);
|
||||
|
||||
Context context = Context::createGlobal();
|
||||
context.makeGlobalContext();
|
||||
|
||||
NamesAndTypesList source_columns = {{"number", std::make_shared<DataTypeUInt64>()}};
|
||||
auto syntax_result = SyntaxAnalyzer(context, {}).analyze(ast, source_columns);
|
||||
|
||||
@ -40,6 +40,7 @@ try
|
||||
std::cerr << std::endl;
|
||||
|
||||
Context context = Context::createGlobal();
|
||||
context.makeGlobalContext();
|
||||
|
||||
NamesAndTypesList source_columns = {{"number", std::make_shared<DataTypeUInt64>()}};
|
||||
auto syntax_result = SyntaxAnalyzer(context, {}).analyze(ast, source_columns);
|
||||
|
||||
@ -1,9 +1,3 @@
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
#include <Core/Block.h>
|
||||
#include <Columns/ColumnVector.h>
|
||||
|
||||
@ -23,6 +23,7 @@ int main(int, char **)
|
||||
try
|
||||
{
|
||||
Context context = Context::createGlobal();
|
||||
context.makeGlobalContext();
|
||||
Settings settings = context.getSettings();
|
||||
|
||||
context.setPath("./");
|
||||
|
||||
@ -350,9 +350,20 @@ Field DataTypeEnum<Type>::castToValue(const Field & value_or_name) const
|
||||
template class DataTypeEnum<Int8>;
|
||||
template class DataTypeEnum<Int16>;
|
||||
|
||||
static void checkASTStructure(const ASTPtr & child)
|
||||
{
|
||||
const auto * func = child->as<ASTFunction>();
|
||||
if (!func
|
||||
|| func->name != "equals"
|
||||
|| func->parameters
|
||||
|| !func->arguments
|
||||
|| func->arguments->children.size() != 2)
|
||||
throw Exception("Elements of Enum data type must be of form: 'name' = number, where name is string literal and number is an integer",
|
||||
ErrorCodes::UNEXPECTED_AST_STRUCTURE);
|
||||
}
|
||||
|
||||
template <typename DataTypeEnum>
|
||||
static DataTypePtr create(const ASTPtr & arguments)
|
||||
static DataTypePtr createExact(const ASTPtr & arguments)
|
||||
{
|
||||
if (!arguments || arguments->children.empty())
|
||||
throw Exception("Enum data type cannot be empty", ErrorCodes::EMPTY_DATA_PASSED);
|
||||
@ -365,15 +376,9 @@ static DataTypePtr create(const ASTPtr & arguments)
|
||||
/// Children must be functions 'equals' with string literal as left argument and numeric literal as right argument.
|
||||
for (const ASTPtr & child : arguments->children)
|
||||
{
|
||||
const auto * func = child->as<ASTFunction>();
|
||||
if (!func
|
||||
|| func->name != "equals"
|
||||
|| func->parameters
|
||||
|| !func->arguments
|
||||
|| func->arguments->children.size() != 2)
|
||||
throw Exception("Elements of Enum data type must be of form: 'name' = number, where name is string literal and number is an integer",
|
||||
ErrorCodes::UNEXPECTED_AST_STRUCTURE);
|
||||
checkASTStructure(child);
|
||||
|
||||
const auto * func = child->as<ASTFunction>();
|
||||
const auto * name_literal = func->arguments->children[0]->as<ASTLiteral>();
|
||||
const auto * value_literal = func->arguments->children[1]->as<ASTLiteral>();
|
||||
|
||||
@ -397,11 +402,38 @@ static DataTypePtr create(const ASTPtr & arguments)
|
||||
return std::make_shared<DataTypeEnum>(values);
|
||||
}
|
||||
|
||||
static DataTypePtr create(const ASTPtr & arguments)
|
||||
{
|
||||
if (!arguments || arguments->children.empty())
|
||||
throw Exception("Enum data type cannot be empty", ErrorCodes::EMPTY_DATA_PASSED);
|
||||
|
||||
/// Children must be functions 'equals' with string literal as left argument and numeric literal as right argument.
|
||||
for (const ASTPtr & child : arguments->children)
|
||||
{
|
||||
checkASTStructure(child);
|
||||
|
||||
const auto * func = child->as<ASTFunction>();
|
||||
const auto * value_literal = func->arguments->children[1]->as<ASTLiteral>();
|
||||
|
||||
if (!value_literal
|
||||
|| (value_literal->value.getType() != Field::Types::UInt64 && value_literal->value.getType() != Field::Types::Int64))
|
||||
throw Exception("Elements of Enum data type must be of form: 'name' = number, where name is string literal and number is an integer",
|
||||
ErrorCodes::UNEXPECTED_AST_STRUCTURE);
|
||||
|
||||
Int64 value = value_literal->value.get<Int64>();
|
||||
|
||||
if (value > std::numeric_limits<Int8>::max() || value < std::numeric_limits<Int8>::min())
|
||||
return createExact<DataTypeEnum16>(arguments);
|
||||
}
|
||||
|
||||
return createExact<DataTypeEnum8>(arguments);
|
||||
}
|
||||
|
||||
void registerDataTypeEnum(DataTypeFactory & factory)
|
||||
{
|
||||
factory.registerDataType("Enum8", create<DataTypeEnum<Int8>>);
|
||||
factory.registerDataType("Enum16", create<DataTypeEnum<Int16>>);
|
||||
factory.registerDataType("Enum8", createExact<DataTypeEnum<Int8>>);
|
||||
factory.registerDataType("Enum16", createExact<DataTypeEnum<Int16>>);
|
||||
factory.registerDataType("Enum", create);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -241,7 +241,7 @@ static DataTypePtr create(const ASTPtr & arguments)
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
static DataTypePtr createExect(const ASTPtr & arguments)
|
||||
static DataTypePtr createExact(const ASTPtr & arguments)
|
||||
{
|
||||
if (!arguments || arguments->children.size() != 1)
|
||||
throw Exception("Decimal data type family must have exactly two arguments: precision and scale",
|
||||
@ -260,9 +260,9 @@ static DataTypePtr createExect(const ASTPtr & arguments)
|
||||
|
||||
void registerDataTypeDecimal(DataTypeFactory & factory)
|
||||
{
|
||||
factory.registerDataType("Decimal32", createExect<Decimal32>, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerDataType("Decimal64", createExect<Decimal64>, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerDataType("Decimal128", createExect<Decimal128>, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerDataType("Decimal32", createExact<Decimal32>, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerDataType("Decimal64", createExact<Decimal64>, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerDataType("Decimal128", createExact<Decimal128>, DataTypeFactory::CaseInsensitive);
|
||||
|
||||
factory.registerDataType("Decimal", create, DataTypeFactory::CaseInsensitive);
|
||||
factory.registerAlias("DEC", "Decimal", DataTypeFactory::CaseInsensitive);
|
||||
|
||||
@ -4,11 +4,6 @@
|
||||
|
||||
#include <sstream>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
|
||||
|
||||
@ -27,6 +27,7 @@ public:
|
||||
|
||||
/// copy-constructor is provided in order to support cloneability
|
||||
ClickHouseDictionarySource(const ClickHouseDictionarySource & other);
|
||||
ClickHouseDictionarySource & operator=(const ClickHouseDictionarySource &) = delete;
|
||||
|
||||
BlockInputStreamPtr loadAll() override;
|
||||
|
||||
|
||||
@ -23,6 +23,7 @@ public:
|
||||
const Context & context);
|
||||
|
||||
ExecutableDictionarySource(const ExecutableDictionarySource & other);
|
||||
ExecutableDictionarySource & operator=(const ExecutableDictionarySource &) = delete;
|
||||
|
||||
BlockInputStreamPtr loadAll() override;
|
||||
|
||||
|
||||
@ -26,6 +26,7 @@ public:
|
||||
const Context & context);
|
||||
|
||||
HTTPDictionarySource(const HTTPDictionarySource & other);
|
||||
HTTPDictionarySource & operator=(const HTTPDictionarySource &) = delete;
|
||||
|
||||
BlockInputStreamPtr loadAll() override;
|
||||
|
||||
|
||||
@ -35,6 +35,7 @@ public:
|
||||
Block & sample_block);
|
||||
|
||||
LibraryDictionarySource(const LibraryDictionarySource & other);
|
||||
LibraryDictionarySource & operator=(const LibraryDictionarySource &) = delete;
|
||||
|
||||
~LibraryDictionarySource() override;
|
||||
|
||||
|
||||
@ -37,6 +37,7 @@ public:
|
||||
|
||||
/// copy-constructor is provided in order to support cloneability
|
||||
MySQLDictionarySource(const MySQLDictionarySource & other);
|
||||
MySQLDictionarySource & operator=(const MySQLDictionarySource &) = delete;
|
||||
|
||||
BlockInputStreamPtr loadAll() override;
|
||||
|
||||
|
||||
@ -36,6 +36,7 @@ public:
|
||||
|
||||
/// copy-constructor is provided in order to support cloneability
|
||||
XDBCDictionarySource(const XDBCDictionarySource & other);
|
||||
XDBCDictionarySource & operator=(const XDBCDictionarySource &) = delete;
|
||||
|
||||
BlockInputStreamPtr loadAll() override;
|
||||
|
||||
|
||||
@ -42,7 +42,7 @@ namespace
|
||||
|
||||
Int64 decodeZigZag(UInt64 n) { return static_cast<Int64>((n >> 1) ^ (~(n & 1) + 1)); }
|
||||
|
||||
void unknownFormat()
|
||||
[[noreturn]] void unknownFormat()
|
||||
{
|
||||
throw Exception("Protobuf messages are corrupted or don't match the provided schema. Please note that Protobuf stream is length-delimited: every message is prefixed by its length in varint.", ErrorCodes::UNKNOWN_PROTOBUF_FORMAT);
|
||||
}
|
||||
|
||||
@ -11,6 +11,7 @@ add_library(clickhouse_functions ${clickhouse_functions_sources})
|
||||
target_link_libraries(clickhouse_functions
|
||||
PUBLIC
|
||||
clickhouse_dictionaries
|
||||
clickhouse_dictionaries_embedded
|
||||
dbms
|
||||
${CONSISTENT_HASHING_LIBRARY}
|
||||
consistent-hashing-sumbur
|
||||
@ -19,6 +20,9 @@ target_link_libraries(clickhouse_functions
|
||||
${METROHASH_LIBRARIES}
|
||||
murmurhash
|
||||
${BASE64_LIBRARY}
|
||||
|
||||
PRIVATE
|
||||
${Boost_FILESYSTEM_LIBRARY}
|
||||
)
|
||||
|
||||
if (OPENSSL_CRYPTO_LIBRARY)
|
||||
|
||||
@ -41,7 +41,7 @@ struct CRC32Impl
|
||||
}
|
||||
}
|
||||
|
||||
static void array(const ColumnString::Offsets & /*offsets*/, PaddedPODArray<UInt32> & /*res*/)
|
||||
[[noreturn]] static void array(const ColumnString::Offsets & /*offsets*/, PaddedPODArray<UInt32> & /*res*/)
|
||||
{
|
||||
throw Exception("Cannot apply function CRC32 to Array argument", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
|
||||
@ -100,7 +100,7 @@ struct ExtractRaw
|
||||
|
||||
for (auto extract_begin = pos; pos != end; ++pos)
|
||||
{
|
||||
if (*pos == current_expect_end)
|
||||
if (current_expect_end && *pos == current_expect_end)
|
||||
{
|
||||
expects_end.pop_back();
|
||||
current_expect_end = expects_end.empty() ? 0 : expects_end.back();
|
||||
@ -192,7 +192,7 @@ struct ExtractParamImpl
|
||||
|
||||
/// We check that the entry does not pass through the boundaries of strings.
|
||||
if (pos + needle.size() < begin + offsets[i])
|
||||
res[i] = ParamExtractor::extract(pos + needle.size(), begin + offsets[i]);
|
||||
res[i] = ParamExtractor::extract(pos + needle.size(), begin + offsets[i] - 1); /// don't include terminating zero
|
||||
else
|
||||
res[i] = 0;
|
||||
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
include(${ClickHouse_SOURCE_DIR}/cmake/dbms_glob_sources.cmake)
|
||||
add_headers_and_sources(clickhouse_functions_gatherutils .)
|
||||
add_library(clickhouse_functions_gatherutils ${clickhouse_functions_gatherutils_sources} ${clickhouse_functions_gatherutils_headers})
|
||||
target_link_libraries(clickhouse_functions_gatherutils PRIVATE clickhouse_common_io)
|
||||
target_link_libraries(clickhouse_functions_gatherutils PRIVATE dbms)
|
||||
|
||||
if (CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE" OR CMAKE_BUILD_TYPE_UC STREQUAL "RELWITHDEBINFO" OR CMAKE_BUILD_TYPE_UC STREQUAL "MINSIZEREL")
|
||||
# Won't generate debug info for files with heavy template instantiation to achieve faster linking and lower size.
|
||||
|
||||
@ -7,18 +7,8 @@
|
||||
#include <Common/Exception.h>
|
||||
#include <Core/Types.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic push
|
||||
#pragma clang diagnostic ignored "-Wold-style-cast"
|
||||
#pragma clang diagnostic ignored "-Wnewline-eof"
|
||||
#endif
|
||||
|
||||
#include <simdjson/jsonparser.h>
|
||||
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic pop
|
||||
#endif
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -42,7 +32,7 @@ struct SimdJSONParser
|
||||
|
||||
bool parse(const StringRef & json) { return !json_parse(json.data, json.size, pj); }
|
||||
|
||||
using Iterator = ParsedJson::iterator;
|
||||
using Iterator = simdjson::ParsedJson::iterator;
|
||||
Iterator getRoot() { return Iterator{pj}; }
|
||||
|
||||
static bool isInt64(const Iterator & it) { return it.is_integer(); }
|
||||
@ -143,7 +133,7 @@ struct SimdJSONParser
|
||||
}
|
||||
|
||||
private:
|
||||
ParsedJson pj;
|
||||
simdjson::ParsedJson pj;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
include(${ClickHouse_SOURCE_DIR}/cmake/dbms_glob_sources.cmake)
|
||||
add_headers_and_sources(clickhouse_functions_url .)
|
||||
add_library(clickhouse_functions_url ${clickhouse_functions_url_sources} ${clickhouse_functions_url_headers})
|
||||
target_link_libraries(clickhouse_functions_url PRIVATE clickhouse_common_io)
|
||||
target_include_directories(clickhouse_functions_url PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/../include) # ${CMAKE_CURRENT_BINARY_DIR}/include
|
||||
target_link_libraries(clickhouse_functions_url PRIVATE dbms)
|
||||
target_include_directories(clickhouse_functions_url PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/../include) # ${CMAKE_CURRENT_BINARY_DIR}/include
|
||||
|
||||
if (CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE" OR CMAKE_BUILD_TYPE_UC STREQUAL "RELWITHDEBINFO" OR CMAKE_BUILD_TYPE_UC STREQUAL "MINSIZEREL")
|
||||
# Won't generate debug info for files with heavy template instantiation to achieve faster linking and lower size.
|
||||
|
||||
@ -89,7 +89,7 @@ struct DecodeURLComponentImpl
|
||||
res_data.resize(res_offset);
|
||||
}
|
||||
|
||||
static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
[[noreturn]] static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
{
|
||||
throw Exception("Column of type FixedString is not supported by URL functions", ErrorCodes::ILLEGAL_COLUMN);
|
||||
}
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
include(${ClickHouse_SOURCE_DIR}/cmake/dbms_glob_sources.cmake)
|
||||
add_headers_and_sources(clickhouse_functions_array .)
|
||||
add_library(clickhouse_functions_array ${clickhouse_functions_array_sources} ${clickhouse_functions_array_headers})
|
||||
target_link_libraries(clickhouse_functions_array PRIVATE clickhouse_common_io clickhouse_functions_gatherutils)
|
||||
target_link_libraries(clickhouse_functions_array PRIVATE dbms clickhouse_functions_gatherutils)
|
||||
|
||||
if (CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE" OR CMAKE_BUILD_TYPE_UC STREQUAL "RELWITHDEBINFO" OR CMAKE_BUILD_TYPE_UC STREQUAL "MINSIZEREL")
|
||||
# Won't generate debug info for files with heavy template instantiation to achieve faster linking and lower size.
|
||||
|
||||
@ -1,5 +1,7 @@
|
||||
#include <Functions/FunctionFactory.h>
|
||||
#include <Functions/FunctionBinaryArithmetic.h>
|
||||
#include <Core/Defines.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -10,7 +12,7 @@ struct BitTestImpl
|
||||
using ResultType = UInt8;
|
||||
|
||||
template <typename Result = ResultType>
|
||||
static inline Result apply(A a, B b)
|
||||
NO_SANITIZE_UNDEFINED static inline Result apply(A a, B b)
|
||||
{
|
||||
return (typename NumberTraits::ToInteger<A>::Type(a) >> typename NumberTraits::ToInteger<B>::Type(b)) & 1;
|
||||
}
|
||||
|
||||
@ -309,7 +309,7 @@ SOFTWARE.
|
||||
res[i] = isValidUTF8(data.data() + i * n, n);
|
||||
}
|
||||
|
||||
static void array(const ColumnString::Offsets &, PaddedPODArray<UInt8> &)
|
||||
[[noreturn]] static void array(const ColumnString::Offsets &, PaddedPODArray<UInt8> &)
|
||||
{
|
||||
throw Exception("Cannot apply function isValidUTF8 to Array argument", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
|
||||
@ -48,7 +48,7 @@ struct LengthUTF8Impl
|
||||
}
|
||||
}
|
||||
|
||||
static void array(const ColumnString::Offsets &, PaddedPODArray<UInt64> &)
|
||||
[[noreturn]] static void array(const ColumnString::Offsets &, PaddedPODArray<UInt64> &)
|
||||
{
|
||||
throw Exception("Cannot apply function lengthUTF8 to Array argument", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
|
||||
}
|
||||
|
||||
@ -61,7 +61,7 @@ struct ReverseUTF8Impl
|
||||
}
|
||||
}
|
||||
|
||||
static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
[[noreturn]] static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
{
|
||||
throw Exception("Cannot apply function reverseUTF8 to fixed string.", ErrorCodes::ILLEGAL_COLUMN);
|
||||
}
|
||||
|
||||
@ -124,7 +124,7 @@ struct ToValidUTF8Impl
|
||||
write_buffer.finish();
|
||||
}
|
||||
|
||||
static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
[[noreturn]] static void vector_fixed(const ColumnString::Chars &, size_t, ColumnString::Chars &)
|
||||
{
|
||||
throw Exception("Column of type FixedString is not supported by toValidUTF8 function", ErrorCodes::ILLEGAL_COLUMN);
|
||||
}
|
||||
|
||||
@ -676,7 +676,7 @@ inline void readText(String & x, ReadBuffer & buf) { readEscapedString(x, buf);
|
||||
inline void readText(LocalDate & x, ReadBuffer & buf) { readDateText(x, buf); }
|
||||
inline void readText(LocalDateTime & x, ReadBuffer & buf) { readDateTimeText(x, buf); }
|
||||
inline void readText(UUID & x, ReadBuffer & buf) { readUUIDText(x, buf); }
|
||||
inline void readText(UInt128 &, ReadBuffer &)
|
||||
[[noreturn]] inline void readText(UInt128 &, ReadBuffer &)
|
||||
{
|
||||
/** Because UInt128 isn't a natural type, without arithmetic operator and only use as an intermediary type -for UUID-
|
||||
* it should never arrive here. But because we used the DataTypeNumber class we should have at least a definition of it.
|
||||
@ -755,7 +755,7 @@ inline void readCSV(String & x, ReadBuffer & buf, const FormatSettings::CSV & se
|
||||
inline void readCSV(LocalDate & x, ReadBuffer & buf) { readCSVSimple(x, buf); }
|
||||
inline void readCSV(LocalDateTime & x, ReadBuffer & buf) { readCSVSimple(x, buf); }
|
||||
inline void readCSV(UUID & x, ReadBuffer & buf) { readCSVSimple(x, buf); }
|
||||
inline void readCSV(UInt128 &, ReadBuffer &)
|
||||
[[noreturn]] inline void readCSV(UInt128 &, ReadBuffer &)
|
||||
{
|
||||
/** Because UInt128 isn't a natural type, without arithmetic operator and only use as an intermediary type -for UUID-
|
||||
* it should never arrive here. But because we used the DataTypeNumber class we should have at least a definition of it.
|
||||
|
||||
@ -830,7 +830,7 @@ inline void writeCSV(const String & x, WriteBuffer & buf) { writeCSVString<>(x,
|
||||
inline void writeCSV(const LocalDate & x, WriteBuffer & buf) { writeDoubleQuoted(x, buf); }
|
||||
inline void writeCSV(const LocalDateTime & x, WriteBuffer & buf) { writeDoubleQuoted(x, buf); }
|
||||
inline void writeCSV(const UUID & x, WriteBuffer & buf) { writeDoubleQuoted(x, buf); }
|
||||
inline void writeCSV(const UInt128, WriteBuffer &)
|
||||
[[noreturn]] inline void writeCSV(const UInt128, WriteBuffer &)
|
||||
{
|
||||
/** Because UInt128 isn't a natural type, without arithmetic operator and only use as an intermediary type -for UUID-
|
||||
* it should never arrive here. But because we used the DataTypeNumber class we should have at least a definition of it.
|
||||
|
||||
@ -1,10 +1,5 @@
|
||||
#if defined(__linux__) || defined(__FreeBSD__)
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <Core/Defines.h>
|
||||
|
||||
@ -16,12 +16,6 @@
|
||||
#include <typeinfo>
|
||||
#include <iostream>
|
||||
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
using namespace DB;
|
||||
|
||||
@ -1,8 +1,3 @@
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
#ifdef __clang__
|
||||
#pragma clang diagnostic ignored "-Wzero-as-null-pointer-constant"
|
||||
#pragma clang diagnostic ignored "-Wundef"
|
||||
#endif
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <stdexcept>
|
||||
|
||||
@ -18,7 +18,7 @@ void prepare2(std::string & filename, std::string & buf);
|
||||
void prepare3(std::string & filename, std::string & buf);
|
||||
void prepare4(std::string & filename, std::string & buf);
|
||||
std::string createTmpFile();
|
||||
void die(const std::string & msg);
|
||||
[[noreturn]] void die(const std::string & msg);
|
||||
void runTest(unsigned int num, const std::function<bool()> & func);
|
||||
|
||||
bool test1(const std::string & filename);
|
||||
|
||||
@ -14,7 +14,7 @@ namespace
|
||||
namespace fs = boost::filesystem;
|
||||
|
||||
void run();
|
||||
void die(const std::string & msg);
|
||||
[[noreturn]] void die(const std::string & msg);
|
||||
void runTest(unsigned int num, const std::function<bool()> & func);
|
||||
std::string createTmpFile();
|
||||
std::string generateString(size_t n);
|
||||
|
||||
@ -29,9 +29,9 @@ namespace
|
||||
/// Default shard weight.
|
||||
static constexpr UInt32 default_weight = 1;
|
||||
|
||||
inline bool isLocal(const Cluster::Address & address, const Poco::Net::SocketAddress & resolved_address, UInt16 clickhouse_port)
|
||||
inline bool isLocalImpl(const Cluster::Address & address, const Poco::Net::SocketAddress & resolved_address, UInt16 clickhouse_port)
|
||||
{
|
||||
/// If there is replica, for which:
|
||||
/// If there is replica, for which:
|
||||
/// - its port is the same that the server is listening;
|
||||
/// - its host is resolved to set of addresses, one of which is the same as one of addresses of network interfaces of the server machine*;
|
||||
/// then we must go to this shard without any inter-process communication.
|
||||
@ -48,10 +48,31 @@ inline bool isLocal(const Cluster::Address & address, const Poco::Net::SocketAdd
|
||||
|
||||
/// Implementation of Cluster::Address class
|
||||
|
||||
std::optional<Poco::Net::SocketAddress> Cluster::Address::getResolvedAddress() const
|
||||
{
|
||||
try
|
||||
{
|
||||
return DNSResolver::instance().resolveAddress(host_name, port);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
/// Failure in DNS resolution in cluster initialization is Ok.
|
||||
tryLogCurrentException("Cluster");
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
bool Cluster::Address::isLocal(UInt16 clickhouse_port) const
|
||||
{
|
||||
if (auto resolved = getResolvedAddress())
|
||||
return isLocalImpl(*this, *resolved, clickhouse_port);
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
Cluster::Address::Address(const Poco::Util::AbstractConfiguration & config, const String & config_prefix)
|
||||
{
|
||||
UInt16 clickhouse_port = static_cast<UInt16>(config.getInt("tcp_port", 0));
|
||||
|
||||
host_name = config.getString(config_prefix + ".host");
|
||||
port = static_cast<UInt16>(config.getInt(config_prefix + ".port"));
|
||||
if (config.has(config_prefix + ".user"))
|
||||
@ -60,10 +81,9 @@ Cluster::Address::Address(const Poco::Util::AbstractConfiguration & config, cons
|
||||
user = config.getString(config_prefix + ".user", "default");
|
||||
password = config.getString(config_prefix + ".password", "");
|
||||
default_database = config.getString(config_prefix + ".default_database", "");
|
||||
initially_resolved_address = DNSResolver::instance().resolveAddress(host_name, port);
|
||||
is_local = isLocal(*this, initially_resolved_address, clickhouse_port);
|
||||
secure = config.getBool(config_prefix + ".secure", false) ? Protocol::Secure::Enable : Protocol::Secure::Disable;
|
||||
compression = config.getBool(config_prefix + ".compression", true) ? Protocol::Compression::Enable : Protocol::Compression::Disable;
|
||||
is_local = isLocal(config.getInt("tcp_port", 0));
|
||||
}
|
||||
|
||||
|
||||
@ -74,9 +94,7 @@ Cluster::Address::Address(const String & host_port_, const String & user_, const
|
||||
host_name = parsed_host_port.first;
|
||||
port = parsed_host_port.second;
|
||||
secure = secure_ ? Protocol::Secure::Enable : Protocol::Secure::Disable;
|
||||
|
||||
initially_resolved_address = DNSResolver::instance().resolveAddress(parsed_host_port.first, parsed_host_port.second);
|
||||
is_local = isLocal(*this, initially_resolved_address, clickhouse_port);
|
||||
is_local = isLocal(clickhouse_port);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -60,7 +60,7 @@ public:
|
||||
/// This database is selected when no database is specified for Distributed table
|
||||
String default_database;
|
||||
/// The locality is determined at the initialization, and is not changed even if DNS is changed
|
||||
bool is_local;
|
||||
bool is_local = false;
|
||||
bool user_specified = false;
|
||||
|
||||
Protocol::Compression compression = Protocol::Compression::Enable;
|
||||
@ -84,17 +84,14 @@ public:
|
||||
String toFullString() const;
|
||||
static Address fromFullString(const String & address_full_string);
|
||||
|
||||
/// Returns initially resolved address
|
||||
Poco::Net::SocketAddress getResolvedAddress() const
|
||||
{
|
||||
return initially_resolved_address;
|
||||
}
|
||||
/// Returns resolved address if it does resolve.
|
||||
std::optional<Poco::Net::SocketAddress> getResolvedAddress() const;
|
||||
|
||||
auto tuple() const { return std::tie(host_name, port, secure, user, password, default_database); }
|
||||
bool operator==(const Address & other) const { return tuple() == other.tuple(); }
|
||||
|
||||
private:
|
||||
Poco::Net::SocketAddress initially_resolved_address;
|
||||
bool isLocal(UInt16 clickhouse_port) const;
|
||||
};
|
||||
|
||||
using Addresses = std::vector<Address>;
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user