thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-21 23:21:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel_parsing

Browse Source
This commit is contained in:
Nikita Mikhaylov 2019-11-05 12:32:26 +03:00
commit 817b52314e
314 changed files with 6849 additions and 2245 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.github/CODEOWNERS vendored
View File

@ -1,3 +1,4 @@
dbms/* @ClickHouse/core-assigner
utils/* @ClickHouse/core-assigner
docs/* @ClickHouse/docs
docs/zh/* @ClickHouse/docs-zh

70
.github/stale.yml vendored
View File

@ -1,70 +0,0 @@
# Configuration for probot-stale - https://github.com/probot/stale
# Number of days of inactivity before an Issue or Pull Request becomes stale
daysUntilStale: 60
# Number of days of inactivity before an Issue or Pull Request with the stale label is closed.
# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale.
daysUntilClose: 30
# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled)
onlyLabels: []
# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable
exemptLabels:
- bug
- feature
- memory
- performance
- prio-crit
- prio-major
- st-accepted
- st-in-progress
- st-waiting-for-fix
- segfault
- crash
# Set to true to ignore issues in a project (defaults to false)
exemptProjects: false
# Set to true to ignore issues in a milestone (defaults to false)
exemptMilestones: false
# Set to true to ignore issues with an assignee (defaults to false)
exemptAssignees: false
# Label to use when marking as stale
staleLabel: stale
# Comment to post when marking as stale. Set to `false` to disable
markComment: >
This issue has been automatically marked as stale because it has not had
recent activity. Please post a comment if this issue is still relevant to you.
Thank you for your contributions.
# Comment to post when removing the stale label.
# unmarkComment: >
# Your comment here.
# Comment to post when closing a stale Issue or Pull Request.
# closeComment: >
# Your comment here.
# Limit the number of actions per hour, from 1-30. Default is 30
limitPerRun: 30
# Limit to only `issues` or `pulls`
# only: issues
# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls':
pulls:
daysUntilStale: 365
daysUntilClose: 60
markComment: >
This pull request has been automatically marked as stale because it has not had
any activity for over a year. Please post a comment about whether you intend to continue working on it.
Thank you for your contributions.
# issues:
# exemptLabels:
# - confirmed

3
.gitignore vendored
View File

@ -245,3 +245,6 @@ website/package-lock.json
/.ccls-cache
/compile_commands.json
# Toolchains
/cmake/toolchain/*

257
CHANGELOG.md
View File

@ -1,3 +1,253 @@
## ClickHouse release v19.16.2.2, 30-10-2019
### Backward Incompatible Change
* Add missing arity validation for count/counIf.
[#7095](https://github.com/ClickHouse/ClickHouse/issues/7095)
[#7298](https://github.com/ClickHouse/ClickHouse/pull/7298) ([Vdimir](https://github.com/Vdimir))
* Remove legacy `asterisk_left_columns_only` setting (it was disabled by default).
[#7335](https://github.com/ClickHouse/ClickHouse/pull/7335) ([Artem
Zuikov](https://github.com/4ertus2))
* Format strings for Template data format are now specified in files.
[#7118](https://github.com/ClickHouse/ClickHouse/pull/7118)
([tavplubix](https://github.com/tavplubix))
### New Feature
* Introduce uniqCombined64() to calculate cardinality greater than UINT_MAX.
[#7213](https://github.com/ClickHouse/ClickHouse/pull/7213),
[#7222](https://github.com/ClickHouse/ClickHouse/pull/7222) ([Azat
Khuzhin](https://github.com/azat))
* Support Bloom filter indexes on Array columns.
[#6984](https://github.com/ClickHouse/ClickHouse/pull/6984)
([achimbab](https://github.com/achimbab))
* Add a function `getMacro(name)` that returns String with the value of corresponding `<macros>`
from server configuration. [#7240](https://github.com/ClickHouse/ClickHouse/pull/7240)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Set two configuration options for a dictionary based on an HTTP source: `credentials` and
`http-headers`. [#7092](https://github.com/ClickHouse/ClickHouse/pull/7092) ([Guillaume
Tassery](https://github.com/YiuRULE))
* Add a new ProfileEvent `Merge` that counts the number of launched background merges.
[#7093](https://github.com/ClickHouse/ClickHouse/pull/7093) ([Mikhail
Korotov](https://github.com/millb))
* Add fullHostName function that returns a fully qualified domain name.
[#7263](https://github.com/ClickHouse/ClickHouse/issues/7263)
[#7291](https://github.com/ClickHouse/ClickHouse/pull/7291) ([sundyli](https://github.com/sundy-li))
* Add function `arraySplit` and `arrayReverseSplit` which split an array by "cut off"
conditions. They are useful in time sequence handling.
[#7294](https://github.com/ClickHouse/ClickHouse/pull/7294) ([hcz](https://github.com/hczhcz))
* Add new functions that return the Array of all matched indices in multiMatch family of functions.
[#7299](https://github.com/ClickHouse/ClickHouse/pull/7299) ([Danila
Kutenin](https://github.com/danlark1))
* Add a new database engine `Lazy` that is optimized for storing a large number of small -Log
tables. [#7171](https://github.com/ClickHouse/ClickHouse/pull/7171) ([Nikita
Vasilev](https://github.com/nikvas0))
* Add aggregate functions groupBitmapAnd, -Or, -Xor for bitmap columns. [#7109](https://github.com/ClickHouse/ClickHouse/pull/7109) ([Zhichang
Yu](https://github.com/yuzhichang))
* Add aggregate function combinators -OrNull and -OrDefault, which return null
or default values when there is nothing to aggregate.
[#7331](https://github.com/ClickHouse/ClickHouse/pull/7331)
([hcz](https://github.com/hczhcz))
* Introduce CustomSeparated data format that supports custom escaping and
delimiter rules. [#7118](https://github.com/ClickHouse/ClickHouse/pull/7118)
([tavplubix](https://github.com/tavplubix))
### Bug Fix
* Fix wrong query result if it has `WHERE IN (SELECT ...)` section and `optimize_read_in_order` is
used. [#7371](https://github.com/ClickHouse/ClickHouse/pull/7371) ([Anton
Popov](https://github.com/CurtizJ))
* Disabled MariaDB authentication plugin, which depends on files outside of project.
[#7140](https://github.com/ClickHouse/ClickHouse/pull/7140) ([Yuriy
Baranov](https://github.com/yurriy))
* Fix exception `Cannot convert column ... because it is constant but values of constants are
different in source and result` which could rarely happen when functions `now()`, `today()`,
`yesterday()`, `randConstant()` are used.
[#7156](https://github.com/ClickHouse/ClickHouse/pull/7156) ([Nikolai
Kochetov](https://github.com/KochetovNicolai))
* Fixed issue of using HTTP keep alive timeout instead of TCP keep alive timeout.
[#7351](https://github.com/ClickHouse/ClickHouse/pull/7351) ([Vasily
Nemkov](https://github.com/Enmk))
* Fixed a segmentation fault in groupBitmapOr (issue [#7109](https://github.com/ClickHouse/ClickHouse/issues/7109)).
[#7289](https://github.com/ClickHouse/ClickHouse/pull/7289) ([Zhichang
Yu](https://github.com/yuzhichang))
* For materialized views the commit for Kafka is called after all data were written.
[#7175](https://github.com/ClickHouse/ClickHouse/pull/7175) ([Ivan](https://github.com/abyss7))
* Fixed wrong `duration_ms` value in `system.part_log` table. It was ten times off.
[#7172](https://github.com/ClickHouse/ClickHouse/pull/7172) ([Vladimir
Chebotarev](https://github.com/excitoon))
* A quick fix to resolve crash in LIVE VIEW table and re-enabling all LIVE VIEW tests.
[#7201](https://github.com/ClickHouse/ClickHouse/pull/7201)
([vzakaznikov](https://github.com/vzakaznikov))
* Serialize NULL values correctly in min/max indexes of MergeTree parts.
[#7234](https://github.com/ClickHouse/ClickHouse/pull/7234) ([Alexander
Kuzmenkov](https://github.com/akuzm))
* Don't put virtual columns to .sql metadata when table is created as `CREATE TABLE AS`.
[#7183](https://github.com/ClickHouse/ClickHouse/pull/7183) ([Ivan](https://github.com/abyss7))
* Fix segmentation fault in `ATTACH PART` query.
[#7185](https://github.com/ClickHouse/ClickHouse/pull/7185)
([alesapin](https://github.com/alesapin))
* Fix wrong result for some queries given by the optimization of empty IN subqueries and empty
INNER/RIGHT JOIN. [#7284](https://github.com/ClickHouse/ClickHouse/pull/7284) ([Nikolai
Kochetov](https://github.com/KochetovNicolai))
* Fixing AddressSanitizer error in the LIVE VIEW getHeader() method.
[#7271](https://github.com/ClickHouse/ClickHouse/pull/7271)
([vzakaznikov](https://github.com/vzakaznikov))
### Improvement
* Add a message in case of queue_wait_max_ms wait takes place.
[#7390](https://github.com/ClickHouse/ClickHouse/pull/7390) ([Azat
Khuzhin](https://github.com/azat))
* Made setting `s3_min_upload_part_size` table-level.
[#7059](https://github.com/ClickHouse/ClickHouse/pull/7059) ([Vladimir
Chebotarev](https://github.com/excitoon))
* Check TTL in StorageFactory. [#7304](https://github.com/ClickHouse/ClickHouse/pull/7304)
([sundyli](https://github.com/sundy-li))
* Squash left-hand blocks in partial merge join (optimization).
[#7122](https://github.com/ClickHouse/ClickHouse/pull/7122) ([Artem
Zuikov](https://github.com/4ertus2))
* Do not allow non-deterministic functions in mutations of Replicated table engines, because this
can introduce inconsistencies between replicas.
[#7247](https://github.com/ClickHouse/ClickHouse/pull/7247) ([Alexander
Kazakov](https://github.com/Akazz))
* Disable memory tracker while converting exception stack trace to string. It can prevent the loss
of error messages of type `Memory limit exceeded` on server, which caused the `Attempt to read
after eof` exception on client. [#7264](https://github.com/ClickHouse/ClickHouse/pull/7264)
([Nikolai Kochetov](https://github.com/KochetovNicolai))
* Miscellaneous format improvements. Resolves
[#6033](https://github.com/ClickHouse/ClickHouse/issues/6033),
[#2633](https://github.com/ClickHouse/ClickHouse/issues/2633),
[#6611](https://github.com/ClickHouse/ClickHouse/issues/6611),
[#6742](https://github.com/ClickHouse/ClickHouse/issues/6742)
[#7215](https://github.com/ClickHouse/ClickHouse/pull/7215)
([tavplubix](https://github.com/tavplubix))
* ClickHouse ignores values on the right side of IN operator that are not convertible to the left
side type. Make it work properly for compound types -- Array and Tuple.
[#7283](https://github.com/ClickHouse/ClickHouse/pull/7283) ([Alexander
Kuzmenkov](https://github.com/akuzm))
* Support missing inequalities for ASOF JOIN. It's possible to join less-or-equal variant and strict
greater and less variants for ASOF column in ON syntax.
[#7282](https://github.com/ClickHouse/ClickHouse/pull/7282) ([Artem
Zuikov](https://github.com/4ertus2))
* Optimize partial merge join. [#7070](https://github.com/ClickHouse/ClickHouse/pull/7070)
([Artem Zuikov](https://github.com/4ertus2))
* Do not use more then 98K of memory in uniqCombined functions.
[#7236](https://github.com/ClickHouse/ClickHouse/pull/7236),
[#7270](https://github.com/ClickHouse/ClickHouse/pull/7270) ([Azat
Khuzhin](https://github.com/azat))
* Flush parts of right-hand joining table on disk in PartialMergeJoin (if there is not enough
memory). Load data back when needed. [#7186](https://github.com/ClickHouse/ClickHouse/pull/7186)
([Artem Zuikov](https://github.com/4ertus2))
### Performance Improvement
* Speed up joinGet with const arguments by avoiding data duplication.
[#7359](https://github.com/ClickHouse/ClickHouse/pull/7359) ([Amos
Bird](https://github.com/amosbird))
* Return early if the subquery is empty.
[#7007](https://github.com/ClickHouse/ClickHouse/pull/7007) ([小路](https://github.com/nicelulu))
* Optimize parsing of SQL expression in Values.
[#6781](https://github.com/ClickHouse/ClickHouse/pull/6781)
([tavplubix](https://github.com/tavplubix))
### Build/Testing/Packaging Improvement
* Disable some contribs for cross-compilation to Mac OS.
[#7101](https://github.com/ClickHouse/ClickHouse/pull/7101) ([Ivan](https://github.com/abyss7))
* Add missing linking with PocoXML for clickhouse_common_io.
[#7200](https://github.com/ClickHouse/ClickHouse/pull/7200) ([Azat
Khuzhin](https://github.com/azat))
* Accept multiple test filter arguments in clickhouse-test.
[#7226](https://github.com/ClickHouse/ClickHouse/pull/7226) ([Alexander
Kuzmenkov](https://github.com/akuzm))
* Enable musl and jemalloc for ARM. [#7300](https://github.com/ClickHouse/ClickHouse/pull/7300)
([Amos Bird](https://github.com/amosbird))
* Added `--client-option` parameter to `clickhouse-test` to pass additional parameters to client.
[#7277](https://github.com/ClickHouse/ClickHouse/pull/7277) ([Nikolai
Kochetov](https://github.com/KochetovNicolai))
* Preserve existing configs on rpm package upgrade.
[#7103](https://github.com/ClickHouse/ClickHouse/pull/7103)
([filimonov](https://github.com/filimonov))
* Fix errors detected by PVS. [#7153](https://github.com/ClickHouse/ClickHouse/pull/7153) ([Artem
Zuikov](https://github.com/4ertus2))
* Fix build for Darwin. [#7149](https://github.com/ClickHouse/ClickHouse/pull/7149)
([Ivan](https://github.com/abyss7))
* glibc 2.29 compatibility. [#7142](https://github.com/ClickHouse/ClickHouse/pull/7142) ([Amos
Bird](https://github.com/amosbird))
* Make sure dh_clean does not touch potential source files.
[#7205](https://github.com/ClickHouse/ClickHouse/pull/7205) ([Amos
Bird](https://github.com/amosbird))
* Attempt to avoid conflict when updating from altinity rpm - it has config file packaged separately
in clickhouse-server-common. [#7073](https://github.com/ClickHouse/ClickHouse/pull/7073)
([filimonov](https://github.com/filimonov))
* Optimize some header files for faster rebuilds.
[#7212](https://github.com/ClickHouse/ClickHouse/pull/7212),
[#7231](https://github.com/ClickHouse/ClickHouse/pull/7231) ([Alexander
Kuzmenkov](https://github.com/akuzm))
* Add performance tests for Date and DateTime. [#7332](https://github.com/ClickHouse/ClickHouse/pull/7332) ([Vasily
Nemkov](https://github.com/Enmk))
* Fix some tests that contained non-deterministic mutations.
[#7132](https://github.com/ClickHouse/ClickHouse/pull/7132) ([Alexander
Kazakov](https://github.com/Akazz))
* Add build with MemorySanitizer to CI. [#7066](https://github.com/ClickHouse/ClickHouse/pull/7066)
([Alexander Kuzmenkov](https://github.com/akuzm))
* Avoid use of uninitialized values in MetricsTransmitter.
[#7158](https://github.com/ClickHouse/ClickHouse/pull/7158) ([Azat
Khuzhin](https://github.com/azat))
* Fix some issues in Fields found by MemorySanitizer.
[#7135](https://github.com/ClickHouse/ClickHouse/pull/7135),
[#7179](https://github.com/ClickHouse/ClickHouse/pull/7179) ([Alexander
Kuzmenkov](https://github.com/akuzm)), [#7376](https://github.com/ClickHouse/ClickHouse/pull/7376)
([Amos Bird](https://github.com/amosbird))
* Fix undefined behavior in murmurhash32. [#7388](https://github.com/ClickHouse/ClickHouse/pull/7388) ([Amos
Bird](https://github.com/amosbird))
* Fix undefined behavior in StoragesInfoStream. [#7384](https://github.com/ClickHouse/ClickHouse/pull/7384)
([tavplubix](https://github.com/tavplubix))
* Fixed constant expressions folding for external database engines (MySQL, ODBC, JDBC). In previous
versions it wasn't working for multiple constant expressions and was not working at all for Date,
DateTime and UUID. This fixes [#7245](https://github.com/ClickHouse/ClickHouse/issues/7245)
[#7252](https://github.com/ClickHouse/ClickHouse/pull/7252)
([alexey-milovidov](https://github.com/alexey-milovidov))
* Fixing ThreadSanitizer data race error in the LIVE VIEW when accessing no_users_thread variable.
[#7353](https://github.com/ClickHouse/ClickHouse/pull/7353)
([vzakaznikov](https://github.com/vzakaznikov))
* Get rid of malloc symbols in libcommon
[#7134](https://github.com/ClickHouse/ClickHouse/pull/7134),
[#7065](https://github.com/ClickHouse/ClickHouse/pull/7065) ([Amos
Bird](https://github.com/amosbird))
* Add global flag ENABLE_LIBRARIES for disabling all libraries.
[#7063](https://github.com/ClickHouse/ClickHouse/pull/7063)
([proller](https://github.com/proller))
### Code cleanup
* Generalize configuration repository to prepare for DDL for Dictionaries. [#7155](https://github.com/ClickHouse/ClickHouse/pull/7155)
([alesapin](https://github.com/alesapin))
* Parser for dictionaries DDL without any semantic.
[#7209](https://github.com/ClickHouse/ClickHouse/pull/7209)
([alesapin](https://github.com/alesapin))
* Split ParserCreateQuery into different smaller parsers.
[#7253](https://github.com/ClickHouse/ClickHouse/pull/7253)
([alesapin](https://github.com/alesapin))
* Small refactoring and renaming near external dictionaries.
[#7111](https://github.com/ClickHouse/ClickHouse/pull/7111)
([alesapin](https://github.com/alesapin))
* Refactor some code to prepare for role-based access control. [#7235](https://github.com/ClickHouse/ClickHouse/pull/7235) ([Vitaly
Baranov](https://github.com/vitlibar))
* Some improvements in DatabaseOrdinary code.
[#7086](https://github.com/ClickHouse/ClickHouse/pull/7086) ([Nikita
Vasilev](https://github.com/nikvas0))
* Do not use iterators in find() and emplace() methods of hash tables.
[#7026](https://github.com/ClickHouse/ClickHouse/pull/7026) ([Alexander
Kuzmenkov](https://github.com/akuzm))
* Fix getMultipleValuesFromConfig in case when parameter root is not empty. [#7374](https://github.com/ClickHouse/ClickHouse/pull/7374)
([Mikhail Korotov](https://github.com/millb))
* Remove some copy-paste (TemporaryFile and TemporaryFileStream)
[#7166](https://github.com/ClickHouse/ClickHouse/pull/7166) ([Artem
Zuikov](https://github.com/4ertus2))
* Improved code readability a little bit (`MergeTreeData::getActiveContainingPart`).
[#7361](https://github.com/ClickHouse/ClickHouse/pull/7361) ([Vladimir
Chebotarev](https://github.com/excitoon))
* Wait for all scheduled jobs, which are using local objects, if `ThreadPool::schedule(...)` throws
an exception. Rename `ThreadPool::schedule(...)` to `ThreadPool::scheduleOrThrowOnError(...)` and
fix comments to make obvious that it may throw.
[#7350](https://github.com/ClickHouse/ClickHouse/pull/7350)
([tavplubix](https://github.com/tavplubix))
## ClickHouse release 19.15.2.2, 2019-10-01
### New Feature
@ -345,6 +595,13 @@
### Security Fix
* Fix two vulnerabilities in codecs in decompression phase (malicious user can fabricate compressed data that will lead to buffer overflow in decompression). [#6670](https://github.com/ClickHouse/ClickHouse/pull/6670) ([Artem Zuikov](https://github.com/4ertus2))
## ClickHouse release 19.11.13.74, 2019-11-01
### Bug Fix
* Fixed rare crash in `ALTER MODIFY COLUMN` and vertical merge when one of merged/altered parts is empty (0 rows). [#6780](https://github.com/ClickHouse/ClickHouse/pull/6780) ([alesapin](https://github.com/alesapin))
* Manual update of `SIMDJSON`. This fixes possible flooding of stderr files with bogus json diagnostic messages. [#7548](https://github.com/ClickHouse/ClickHouse/pull/7548) ([Alexander Kazakov](https://github.com/Akazz))
* Fixed bug with `mrk` file extension for mutations ([alesapin](https://github.com/alesapin))
## ClickHouse release 19.11.12.69, 2019-10-02
### Bug Fix

6
CMakeLists.txt
View File

@ -13,7 +13,10 @@ foreach(policy
endforeach()
project(ClickHouse)
include (cmake/arch.cmake)
include (cmake/target.cmake)
include (cmake/tools.cmake)
# Ignore export() since we don't use it,
# but it gets broken with a global targets via link_libraries()
@ -26,8 +29,6 @@ set(CMAKE_LINK_DEPENDS_NO_SHARED 1) # Do not relink all depended targets on .so
set(CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Debug;Release;MinSizeRel" CACHE STRING "" FORCE)
set(CMAKE_DEBUG_POSTFIX "d" CACHE STRING "Generate debug library name with a postfix.") # To be consistent with CMakeLists from contrib libs.
include (cmake/arch.cmake)
option(ENABLE_IPO "Enable inter-procedural optimization (aka LTO)" OFF) # need cmake 3.9+
if(ENABLE_IPO)
cmake_policy(SET CMP0069 NEW)
@ -230,7 +231,6 @@ include(cmake/dbms_glob_sources.cmake)
if (OS_LINUX)
include(cmake/linux/default_libs.cmake)
elseif (OS_DARWIN)
include(cmake/darwin/sdk.cmake)
include(cmake/darwin/default_libs.cmake)
endif ()

1
cmake/arch.cmake
View File

@ -17,6 +17,7 @@ endif ()
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^(ppc64le.*|PPC64LE.*)")
set (ARCH_PPC64LE 1)
# FIXME: move this check into tools.cmake
if (COMPILER_CLANG OR (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8))
message(FATAL_ERROR "Only gcc-8 is supported for powerpc architecture")
endif ()

8
cmake/darwin/default_libs.cmake
View File

@ -11,6 +11,14 @@ message(STATUS "Default libraries: ${DEFAULT_LIBS}")
set(CMAKE_CXX_STANDARD_LIBRARIES ${DEFAULT_LIBS})
set(CMAKE_C_STANDARD_LIBRARIES ${DEFAULT_LIBS})
# Minimal supported SDK version
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mmacosx-version-min=10.14")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mmacosx-version-min=10.14")
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -mmacosx-version-min=10.14")
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -mmacosx-version-min=10.14")
# Global libraries
add_library(global-libs INTERFACE)

11
cmake/darwin/sdk.cmake
View File

@ -1,11 +0,0 @@
option (SDK_PATH "Path to the SDK to build with" "")
if (NOT EXISTS "${SDK_PATH}/SDKSettings.plist")
message (FATAL_ERROR "Wrong SDK path provided: ${SDK_PATH}")
endif ()
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -isysroot ${SDK_PATH} -mmacosx-version-min=10.14")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -isysroot ${SDK_PATH} -mmacosx-version-min=10.14")
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -isysroot ${SDK_PATH} -mmacosx-version-min=10.14")
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -isysroot ${SDK_PATH} -mmacosx-version-min=10.14")

13
cmake/darwin/toolchain-x86_64.cmake Normal file
View File

@ -0,0 +1,13 @@
set (CMAKE_SYSTEM_NAME "Darwin")
set (CMAKE_SYSTEM_PROCESSOR "x86_64")
set (CMAKE_C_COMPILER_TARGET "x86_64-apple-darwin")
set (CMAKE_CXX_COMPILER_TARGET "x86_64-apple-darwin")
set (CMAKE_OSX_SYSROOT "${CMAKE_CURRENT_LIST_DIR}/../toolchain/darwin-x86_64")
set (CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) # disable linkage check - it doesn't work in CMake
set (HAS_PRE_1970_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_PRE_1970_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)

12
cmake/find/capnp.cmake
View File

@ -4,6 +4,14 @@ if (ENABLE_CAPNP)
option (USE_INTERNAL_CAPNP_LIBRARY "Set to FALSE to use system capnproto library instead of bundled" ${NOT_UNBUNDLED})
if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/capnproto/CMakeLists.txt")
if(USE_INTERNAL_CAPNP_LIBRARY)
message(WARNING "submodule contrib/capnproto is missing. to fix try run: \n git submodule update --init --recursive")
endif()
set(MISSING_INTERNAL_CAPNP_LIBRARY 1)
set(USE_INTERNAL_CAPNP_LIBRARY 0)
endif()
# FIXME: refactor to use `add_library(… IMPORTED)` if possible.
if (NOT USE_INTERNAL_CAPNP_LIBRARY)
find_library (KJ kj)
@ -11,7 +19,7 @@ if (NOT USE_INTERNAL_CAPNP_LIBRARY)
find_library (CAPNPC capnpc)
set (CAPNP_LIBRARIES ${CAPNPC} ${CAPNP} ${KJ})
else ()
elseif(NOT MISSING_INTERNAL_CAPNP_LIBRARY)
add_subdirectory(contrib/capnproto-cmake)
set (CAPNP_LIBRARIES capnpc)
@ -23,4 +31,4 @@ endif ()
endif ()
message (STATUS "Using capnp: ${CAPNP_LIBRARIES}")
message (STATUS "Using capnp=${USE_CAPNP}: ${CAPNP_LIBRARIES}")

5
cmake/find/orc.cmake
View File

@ -1,7 +1,8 @@
option (ENABLE_ORC "Enable ORC" ${ENABLE_LIBRARIES})
if(ENABLE_ORC)
option (USE_INTERNAL_ORC_LIBRARY "Set to FALSE to use system ORC instead of bundled" ${NOT_UNBUNDLED})
include(cmake/find/snappy.cmake)
option(USE_INTERNAL_ORC_LIBRARY "Set to FALSE to use system ORC instead of bundled" ${NOT_UNBUNDLED})
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/orc/c++/include/orc/OrcFile.hh")
if(USE_INTERNAL_ORC_LIBRARY)
@ -25,7 +26,7 @@ endif ()
if (ORC_LIBRARY AND ORC_INCLUDE_DIR)
set(USE_ORC 1)
elseif(NOT MISSING_INTERNAL_ORC_LIBRARY AND ARROW_LIBRARY) # (LIBGSASL_LIBRARY AND LIBXML2_LIBRARY)
elseif(NOT MISSING_INTERNAL_ORC_LIBRARY AND ARROW_LIBRARY AND SNAPPY_LIBRARY) # (LIBGSASL_LIBRARY AND LIBXML2_LIBRARY)
set(ORC_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/orc/c++/include")
set(ORC_LIBRARY orc)
set(USE_ORC 1)

5
cmake/find/parquet.cmake
View File

@ -24,7 +24,10 @@ endif()
if(ARROW_INCLUDE_DIR AND PARQUET_INCLUDE_DIR)
elseif(NOT MISSING_INTERNAL_PARQUET_LIBRARY AND NOT OS_FREEBSD)
include(cmake/find/snappy.cmake)
set(CAN_USE_INTERNAL_PARQUET_LIBRARY 1)
if(SNAPPY_LIBRARY)
set(CAN_USE_INTERNAL_PARQUET_LIBRARY 1)
endif()
include(CheckCXXSourceCompiles)
if(NOT USE_INTERNAL_DOUBLE_CONVERSION_LIBRARY)
set(CMAKE_REQUIRED_LIBRARIES ${DOUBLE_CONVERSION_LIBRARIES})

8
cmake/find/poco.cmake
View File

@ -8,6 +8,14 @@ if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/poco/CMakeLists.txt")
set (MISSING_INTERNAL_POCO_LIBRARY 1)
endif ()
if (NOT ENABLE_LIBRARIES)
set (ENABLE_POCO_NETSSL ${ENABLE_LIBRARIES} CACHE BOOL "")
set (ENABLE_POCO_MONGODB ${ENABLE_LIBRARIES} CACHE BOOL "")
set (ENABLE_POCO_REDIS ${ENABLE_LIBRARIES} CACHE BOOL "")
set (ENABLE_POCO_ODBC ${ENABLE_LIBRARIES} CACHE BOOL "")
set (ENABLE_POCO_SQL ${ENABLE_LIBRARIES} CACHE BOOL "")
endif ()
set (POCO_COMPONENTS Net XML SQL Data)
if (NOT DEFINED ENABLE_POCO_NETSSL OR ENABLE_POCO_NETSSL)
list (APPEND POCO_COMPONENTS Crypto NetSSL)

5
cmake/find/unwind.cmake
View File

@ -4,6 +4,11 @@ if (NOT CMAKE_SYSTEM MATCHES "Linux" OR ARCH_ARM OR ARCH_32)
set (USE_UNWIND OFF)
endif ()
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libunwind/CMakeLists.txt")
message(WARNING "submodule contrib/libunwind is missing. to fix try run: \n git submodule update --init --recursive")
set (USE_UNWIND OFF)
endif ()
if (USE_UNWIND)
add_subdirectory(contrib/libunwind-cmake)
set (UNWIND_LIBRARIES unwind)

2
cmake/linux/default_libs.cmake
View File

@ -5,7 +5,7 @@ set (DEFAULT_LIBS "-nodefaultlibs")
# We need builtins from Clang's RT even without libcxx - for ubsan+int128.
# See https://bugs.llvm.org/show_bug.cgi?id=16404
if (COMPILER_CLANG)
if (COMPILER_CLANG AND NOT (CMAKE_CROSSCOMPILING AND ARCH_AARCH64))
execute_process (COMMAND ${CMAKE_CXX_COMPILER} --print-file-name=libclang_rt.builtins-${CMAKE_SYSTEM_PROCESSOR}.a OUTPUT_VARIABLE BUILTINS_LIBRARY OUTPUT_STRIP_TRAILING_WHITESPACE)
else ()
set (BUILTINS_LIBRARY "-lgcc")

25
cmake/linux/toolchain-aarch64.cmake Normal file
View File

@ -0,0 +1,25 @@
set (CMAKE_SYSTEM_NAME "Linux")
set (CMAKE_SYSTEM_PROCESSOR "aarch64")
set (CMAKE_C_COMPILER_TARGET "aarch64-linux-gnu")
set (CMAKE_CXX_COMPILER_TARGET "aarch64-linux-gnu")
set (CMAKE_ASM_COMPILER_TARGET "aarch64-linux-gnu")
set (CMAKE_SYSROOT "${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64/aarch64-linux-gnu/libc")
# We don't use compiler from toolchain because it's gcc-8, and we provide support only for gcc-9.
set (CMAKE_AR "${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64/bin/aarch64-linux-gnu-ar" CACHE FILEPATH "" FORCE)
set (CMAKE_RANLIB "${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64/bin/aarch64-linux-gnu-ranlib" CACHE FILEPATH "" FORCE)
set (CMAKE_C_FLAGS_INIT "${CMAKE_C_FLAGS} --gcc-toolchain=${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64")
set (CMAKE_CXX_FLAGS_INIT "${CMAKE_CXX_FLAGS} --gcc-toolchain=${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64")
set (CMAKE_ASM_FLAGS_INIT "${CMAKE_ASM_FLAGS} --gcc-toolchain=${CMAKE_CURRENT_LIST_DIR}/../toolchain/linux-aarch64")
set (LINKER_NAME "lld" CACHE STRING "" FORCE)
set (CMAKE_EXE_LINKER_FLAGS_INIT "-fuse-ld=lld")
set (CMAKE_SHARED_LINKER_FLAGS_INIT "-fuse-ld=lld")
set (HAS_PRE_1970_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_PRE_1970_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)

67
cmake/target.cmake
View File

@ -9,62 +9,8 @@ elseif (CMAKE_SYSTEM_NAME MATCHES "Darwin")
add_definitions(-D OS_DARWIN)
endif ()
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
set (COMPILER_GCC 1)
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
set (COMPILER_CLANG 1)
endif ()
if (COMPILER_GCC)
# Require minimum version of gcc
set (GCC_MINIMUM_VERSION 8)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS ${GCC_MINIMUM_VERSION} AND NOT CMAKE_VERSION VERSION_LESS 2.8.9)
message (FATAL_ERROR "GCC version must be at least ${GCC_MINIMUM_VERSION}. For example, if GCC ${GCC_MINIMUM_VERSION} is available under gcc-${GCC_MINIMUM_VERSION}, g++-${GCC_MINIMUM_VERSION} names, do the following: export CC=gcc-${GCC_MINIMUM_VERSION} CXX=g++-${GCC_MINIMUM_VERSION}; rm -rf CMakeCache.txt CMakeFiles; and re run cmake or ./release.")
endif ()
elseif (COMPILER_CLANG)
# Require minimum version of clang
set (CLANG_MINIMUM_VERSION 7)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS ${CLANG_MINIMUM_VERSION})
message (FATAL_ERROR "Clang version must be at least ${CLANG_MINIMUM_VERSION}.")
endif ()
else ()
message (WARNING "You are using an unsupported compiler. Compilation has only been tested with Clang 6+ and GCC 7+.")
endif ()
string(REGEX MATCH "-?[0-9]+(.[0-9]+)?$" COMPILER_POSTFIX ${CMAKE_CXX_COMPILER})
if (OS_LINUX)
find_program (LLD_PATH NAMES "lld${COMPILER_POSTFIX}" "lld")
find_program (GOLD_PATH NAMES "ld.gold" "gold")
endif()
option (LINKER_NAME "Linker name or full path")
if (NOT LINKER_NAME)
if (COMPILER_CLANG AND LLD_PATH)
set (LINKER_NAME "lld")
elseif (GOLD_PATH)
set (LINKER_NAME "gold")
endif ()
endif ()
if (LINKER_NAME)
message(STATUS "Using linker: ${LINKER_NAME} (selected from: LLD_PATH=${LLD_PATH}; GOLD_PATH=${GOLD_PATH}; COMPILER_POSTFIX=${COMPILER_POSTFIX})")
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
endif ()
if (CMAKE_CROSSCOMPILING)
if (OS_DARWIN)
set (CMAKE_SYSTEM_PROCESSOR x86_64)
set (CMAKE_C_COMPILER_TARGET x86_64-apple-darwin)
set (CMAKE_CXX_COMPILER_TARGET x86_64-apple-darwin)
set (HAS_PRE_1970_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_PRE_1970_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE "0" CACHE STRING "Result from TRY_RUN" FORCE)
set (HAS_POST_2038_EXITCODE__TRYRUN_OUTPUT "" CACHE STRING "Output from TRY_RUN" FORCE)
# FIXME: broken dependencies
set (USE_SNAPPY OFF CACHE INTERNAL "")
set (ENABLE_SSL OFF CACHE INTERNAL "")
@ -73,12 +19,19 @@ if (CMAKE_CROSSCOMPILING)
set (ENABLE_READLINE OFF CACHE INTERNAL "")
set (ENABLE_ICU OFF CACHE INTERNAL "")
set (ENABLE_FASTOPS OFF CACHE INTERNAL "")
message (STATUS "Cross-compiling for Darwin")
elseif (OS_LINUX)
if (ARCH_AARCH64)
# FIXME: broken dependencies
set (ENABLE_PROTOBUF OFF CACHE INTERNAL "")
set (ENABLE_PARQUET OFF CACHE INTERNAL "")
set (ENABLE_MYSQL OFF CACHE INTERNAL "")
endif ()
else ()
message (FATAL_ERROR "Trying to cross-compile to unsupported target: ${CMAKE_SYSTEM_NAME}!")
message (FATAL_ERROR "Trying to cross-compile to unsupported system: ${CMAKE_SYSTEM_NAME}!")
endif ()
# Don't know why but CXX_STANDARD doesn't work for cross-compilation
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17")
message (STATUS "Cross-compiling for target: ${CMAKE_CXX_COMPILE_TARGET}")
endif ()

2
cmake/toolchain/darwin-x86_64/README.txt Normal file
View File

@ -0,0 +1,2 @@
wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.14-beta4/MacOSX10.14.sdk.tar.xz
tar --strip-components=1 xJf MacOSX10.14.sdk.tar.xz

2
cmake/toolchain/linux-aarch64/README.txt Normal file
View File

@ -0,0 +1,2 @@
wget https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz?revision=2e88a73f-d233-4f96-b1f4-d8b36e9bb0b9&la=en -O gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz
tar --strip-components=1 xJf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz

41
cmake/tools.cmake Normal file
View File

@ -0,0 +1,41 @@
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
set (COMPILER_GCC 1)
elseif (CMAKE_CXX_COMPILER_ID MATCHES "Clang|AppleClang")
set (COMPILER_CLANG 1)
endif ()
if (COMPILER_GCC)
# Require minimum version of gcc
set (GCC_MINIMUM_VERSION 8)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS ${GCC_MINIMUM_VERSION} AND NOT CMAKE_VERSION VERSION_LESS 2.8.9)
message (FATAL_ERROR "GCC version must be at least ${GCC_MINIMUM_VERSION}. For example, if GCC ${GCC_MINIMUM_VERSION} is available under gcc-${GCC_MINIMUM_VERSION}, g++-${GCC_MINIMUM_VERSION} names, do the following: export CC=gcc-${GCC_MINIMUM_VERSION} CXX=g++-${GCC_MINIMUM_VERSION}; rm -rf CMakeCache.txt CMakeFiles; and re run cmake or ./release.")
endif ()
elseif (COMPILER_CLANG)
# Require minimum version of clang
set (CLANG_MINIMUM_VERSION 7)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS ${CLANG_MINIMUM_VERSION})
message (FATAL_ERROR "Clang version must be at least ${CLANG_MINIMUM_VERSION}.")
endif ()
else ()
message (WARNING "You are using an unsupported compiler. Compilation has only been tested with Clang 6+ and GCC 7+.")
endif ()
option (LINKER_NAME "Linker name or full path")
find_program (LLD_PATH NAMES "ld.lld" "lld")
find_program (GOLD_PATH NAMES "ld.gold" "gold")
if (NOT LINKER_NAME)
if (LLD_PATH)
set (LINKER_NAME "lld")
elseif (GOLD_PATH)
set (LINKER_NAME "gold")
endif ()
endif ()
if (LINKER_NAME)
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
message(STATUS "Using custom linker by name: ${LINKER_NAME}")
endif ()

8
contrib/arrow-cmake/CMakeLists.txt
View File

@ -70,6 +70,14 @@ add_custom_command(OUTPUT orc_proto.pb.h orc_proto.pb.cc
--cpp_out="${CMAKE_CURRENT_BINARY_DIR}"
"${PROTO_DIR}/orc_proto.proto")
# arrow-cmake cmake file calling orc cmake subroutine which detects certain compiler features.
# Apple Clang compiler failed to compile this code without specifying c++11 standard.
# As result these compiler features detected as absent. In result it failed to compile orc itself.
# In orc makefile there is code that sets flags, but arrow-cmake ignores these flags.
if (CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
set (CXX11_FLAGS "-std=c++0x")
endif()
include(${ClickHouse_SOURCE_DIR}/contrib/orc/cmake_modules/CheckSourceCompiles.cmake)
include(orc_check.cmake)
configure_file("${ORC_INCLUDE_DIR}/orc/orc-config.hh.in" "${ORC_BUILD_INCLUDE_DIR}/orc/orc-config.hh")

2
contrib/libcxx-cmake/CMakeLists.txt
View File

@ -44,7 +44,7 @@ target_include_directories(cxx SYSTEM BEFORE PUBLIC $<BUILD_INTERFACE:${LIBCXX_S
target_compile_definitions(cxx PRIVATE -D_LIBCPP_BUILDING_LIBRARY -DLIBCXX_BUILDING_LIBCXXABI)
target_compile_options(cxx PUBLIC -nostdinc++ -Wno-reserved-id-macro)
if (OS_DARWIN AND NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9)
if (OS_DARWIN AND (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9) AND (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 11))
target_compile_options(cxx PUBLIC -Wno-ctad-maybe-unsupported)
endif ()

5
dbms/CMakeLists.txt
View File

@ -425,6 +425,11 @@ endif()
if (USE_JEMALLOC)
dbms_target_include_directories (SYSTEM BEFORE PRIVATE ${JEMALLOC_INCLUDE_DIR}) # used in Interpreters/AsynchronousMetrics.cpp
target_include_directories (clickhouse_new_delete SYSTEM BEFORE PRIVATE ${JEMALLOC_INCLUDE_DIR})
if(NOT MAKE_STATIC_LIBRARIES AND ${JEMALLOC_LIBRARIES} MATCHES "${CMAKE_STATIC_LIBRARY_SUFFIX}$")
# mallctl in dbms/src/Interpreters/AsynchronousMetrics.cpp
target_link_libraries(clickhouse_interpreters PRIVATE ${JEMALLOC_LIBRARIES})
endif()
endif ()
dbms_target_include_directories (PUBLIC ${DBMS_INCLUDE_DIR} PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/src/Formats/include)

49
dbms/programs/client/Client.cpp
View File

@ -89,6 +89,40 @@
#define DISABLE_LINE_WRAPPING "\033[?7l"
#define ENABLE_LINE_WRAPPING "\033[?7h"
#if USE_READLINE && RL_VERSION_MAJOR >= 7
#define BRACK_PASTE_PREF "\033[200~"
#define BRACK_PASTE_SUFF "\033[201~"
#define BRACK_PASTE_LAST '~'
#define BRACK_PASTE_SLEN 6
/// Make sure we don't get ^J for the enter character.
/// This handler also bypasses some unused macro/event checkings.
static int clickhouse_rl_bracketed_paste_begin(int /* count */, int /* key */)
{
std::string buf;
buf.reserve(128);
RL_SETSTATE(RL_STATE_MOREINPUT);
SCOPE_EXIT(RL_UNSETSTATE(RL_STATE_MOREINPUT));
char c;
while ((c = rl_read_key()) >= 0)
{
if (c == '\r' || c == '\n')
c = '\n';
buf.push_back(c);
if (buf.size() >= BRACK_PASTE_SLEN && c == BRACK_PASTE_LAST && buf.substr(buf.size() - BRACK_PASTE_SLEN) == BRACK_PASTE_SUFF)
{
buf.resize(buf.size() - BRACK_PASTE_SLEN);
break;
}
}
return static_cast<size_t>(rl_insert_text(buf.c_str())) == buf.size() ? 0 : 1;
}
#endif
namespace DB
{
@ -462,6 +496,18 @@ private:
if (rl_initialize())
throw Exception("Cannot initialize readline", ErrorCodes::CANNOT_READLINE);
#if RL_VERSION_MAJOR >= 7
/// When bracketed paste mode is set, pasted text is bracketed with control sequences so
/// that the program can differentiate pasted text from typed-in text. This helps
/// clickhouse-client so that without -m flag, one can still paste multiline queries, and
/// possibly get better pasting performance. See https://cirw.in/blog/bracketed-paste for
/// more details.
rl_variable_bind("enable-bracketed-paste", "on");
/// Use our bracketed paste handler to get better user experience. See comments above.
rl_bind_keyseq(BRACK_PASTE_PREF, clickhouse_rl_bracketed_paste_begin);
#endif
auto clear_prompt_or_exit = [](int)
{
/// This is signal safe.
@ -632,7 +678,8 @@ private:
/// If the user restarts the client then after pressing the "up" button
/// every line of the query will be displayed separately.
std::string logged_query = input;
std::replace(logged_query.begin(), logged_query.end(), '\n', ' ');
if (config().has("multiline"))
std::replace(logged_query.begin(), logged_query.end(), '\n', ' ');
add_history(logged_query.c_str());
#if USE_READLINE && HAVE_READLINE_HISTORY

2
dbms/programs/performance-test/JSONString.h
View File

@ -22,7 +22,7 @@ public:
void set(const std::string & key, std::string value, bool wrap = true);
template <typename T>
std::enable_if_t<std::is_arithmetic_v<T>> set(const std::string key, T value)
std::enable_if_t<is_arithmetic_v<T>> set(const std::string key, T value)
{
set(key, std::to_string(value), /*wrap= */ false);
}

12
dbms/programs/server/CMakeLists.txt
View File

@ -10,13 +10,11 @@ set(CLICKHOUSE_SERVER_SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/TCPHandler.cpp
)
if (USE_SSL)
set(CLICKHOUSE_SERVER_SOURCES
${CLICKHOUSE_SERVER_SOURCES}
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
)
endif ()
set(CLICKHOUSE_SERVER_SOURCES
${CLICKHOUSE_SERVER_SOURCES}
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandler.cpp
${CMAKE_CURRENT_SOURCE_DIR}/MySQLHandlerFactory.cpp
)
set(CLICKHOUSE_SERVER_LINK PRIVATE clickhouse_dictionaries clickhouse_common_io clickhouse_common_config clickhouse_common_zookeeper clickhouse_parsers string_utils PUBLIC daemon PRIVATE clickhouse_storages_system clickhouse_functions clickhouse_aggregate_functions clickhouse_table_functions ${Poco_Net_LIBRARY})
if (USE_POCO_NETSSL)

85
dbms/programs/server/MySQLHandler.cpp
View File

@ -1,7 +1,6 @@
#include <Common/config.h>
#if USE_SSL
#include "MySQLHandler.h"
#include "MySQLHandler.h"
#include <limits>
#include <ext/scope_guard.h>
#include <Columns/ColumnVector.h>
@ -15,37 +14,39 @@
#include <IO/ReadBufferFromPocoSocket.h>
#include <IO/ReadBufferFromString.h>
#include <IO/WriteBufferFromPocoSocket.h>
#include <Poco/Crypto/CipherFactory.h>
#include <Poco/Crypto/RSAKey.h>
#include <Poco/Net/SecureStreamSocket.h>
#include <Poco/Net/SSLManager.h>
#include <Storages/IStorage.h>
#if USE_POCO_NETSSL
#include <Poco/Net/SecureStreamSocket.h>
#include <Poco/Net/SSLManager.h>
#include <Poco/Crypto/CipherFactory.h>
#include <Poco/Crypto/RSAKey.h>
#endif
namespace DB
{
using namespace MySQLProtocol;
#if USE_POCO_NETSSL
using Poco::Net::SecureStreamSocket;
using Poco::Net::SSLManager;
#endif
namespace ErrorCodes
{
extern const int MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES;
extern const int OPENSSL_ERROR;
extern const int SUPPORT_IS_DISABLED;
}
MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key_, RSA & private_key_, bool ssl_enabled, size_t connection_id_)
MySQLHandler::MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_,
bool ssl_enabled, size_t connection_id_)
: Poco::Net::TCPServerConnection(socket_)
, server(server_)
, log(&Poco::Logger::get("MySQLHandler"))
, connection_context(server.context())
, connection_id(connection_id_)
, public_key(public_key_)
, private_key(private_key_)
, auth_plugin(new MySQLProtocol::Authentication::Native41())
{
server_capability_flags = CLIENT_PROTOCOL_41 | CLIENT_SECURE_CONNECTION | CLIENT_PLUGIN_AUTH | CLIENT_PLUGIN_AUTH_LENENC_CLIENT_DATA | CLIENT_CONNECT_WITH_DB | CLIENT_DEPRECATE_EOF;
@ -197,21 +198,7 @@ void MySQLHandler::finishHandshake(MySQLProtocol::HandshakeResponse & packet)
if (payload_size == SSL_REQUEST_PAYLOAD_SIZE)
{
read_bytes(packet_size); /// Reading rest SSLRequest.
SSLRequest ssl_request;
ReadBufferFromMemory payload(buf, pos);
payload.ignore(PACKET_HEADER_SIZE);
ssl_request.readPayload(payload);
connection_context.mysql.client_capabilities = ssl_request.capability_flags;
connection_context.mysql.max_packet_size = ssl_request.max_packet_size ? ssl_request.max_packet_size : MAX_PACKET_LENGTH;
secure_connection = true;
ss = std::make_shared<SecureStreamSocket>(SecureStreamSocket::attach(socket(), SSLManager::instance().defaultServerContext()));
in = std::make_shared<ReadBufferFromPocoSocket>(*ss);
out = std::make_shared<WriteBufferFromPocoSocket>(*ss);
connection_context.mysql.sequence_id = 2;
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.mysql.sequence_id);
packet_sender->max_packet_size = connection_context.mysql.max_packet_size;
packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket.
finishHandshakeSSL(packet_size, buf, pos, read_bytes, packet);
}
else
{
@ -232,7 +219,9 @@ void MySQLHandler::authenticate(const String & user_name, const String & auth_pl
// For compatibility with JavaScript MySQL client, Native41 authentication plugin is used when possible (if password is specified using double SHA1). Otherwise SHA256 plugin is used.
auto user = connection_context.getUser(user_name);
if (user->authentication.getType() != DB::Authentication::DOUBLE_SHA1_PASSWORD)
auth_plugin = std::make_unique<MySQLProtocol::Authentication::Sha256Password>(public_key, private_key, log);
{
authPluginSSL();
}
try {
std::optional<String> auth_response = auth_plugin_name == auth_plugin->getName() ? std::make_optional<String>(initial_auth_response) : std::nullopt;
@ -302,5 +291,47 @@ void MySQLHandler::comQuery(ReadBuffer & payload)
packet_sender->sendPacket(OK_Packet(0x00, client_capability_flags, 0, 0, 0), true);
}
void MySQLHandler::authPluginSSL()
{
throw Exception("Compiled without SSL", ErrorCodes::SUPPORT_IS_DISABLED);
}
void MySQLHandler::finishHandshakeSSL([[maybe_unused]] size_t packet_size, [[maybe_unused]] char * buf, [[maybe_unused]] size_t pos, [[maybe_unused]] std::function<void(size_t)> read_bytes, [[maybe_unused]] MySQLProtocol::HandshakeResponse & packet)
{
throw Exception("Compiled without SSL", ErrorCodes::SUPPORT_IS_DISABLED);
}
#if USE_SSL && USE_POCO_NETSSL
MySQLHandlerSSL::MySQLHandlerSSL(IServer & server_, const Poco::Net::StreamSocket & socket_, bool ssl_enabled, size_t connection_id_, RSA & public_key_, RSA & private_key_)
: MySQLHandler(server_, socket_, ssl_enabled, connection_id_)
, public_key(public_key_)
, private_key(private_key_)
{}
void MySQLHandlerSSL::authPluginSSL()
{
auth_plugin = std::make_unique<MySQLProtocol::Authentication::Sha256Password>(public_key, private_key, log);
}
void MySQLHandlerSSL::finishHandshakeSSL(size_t packet_size, char * buf, size_t pos, std::function<void(size_t)> read_bytes, MySQLProtocol::HandshakeResponse & packet)
{
read_bytes(packet_size); /// Reading rest SSLRequest.
SSLRequest ssl_request;
ReadBufferFromMemory payload(buf, pos);
payload.ignore(PACKET_HEADER_SIZE);
ssl_request.readPayload(payload);
connection_context.mysql.client_capabilities = ssl_request.capability_flags;
connection_context.mysql.max_packet_size = ssl_request.max_packet_size ? ssl_request.max_packet_size : MAX_PACKET_LENGTH;
secure_connection = true;
ss = std::make_shared<SecureStreamSocket>(SecureStreamSocket::attach(socket(), SSLManager::instance().defaultServerContext()));
in = std::make_shared<ReadBufferFromPocoSocket>(*ss);
out = std::make_shared<WriteBufferFromPocoSocket>(*ss);
connection_context.mysql.sequence_id = 2;
packet_sender = std::make_shared<PacketSender>(*in, *out, connection_context.mysql.sequence_id);
packet_sender->max_packet_size = connection_context.mysql.max_packet_size;
packet_sender->receivePacket(packet); /// Reading HandshakeResponse from secure socket.
}
#endif
}

37
dbms/programs/server/MySQLHandler.h
View File

@ -1,13 +1,13 @@
#pragma once
#include <Common/config.h>
#if USE_SSL
#include <Poco/Net/TCPServerConnection.h>
#include <Poco/Net/SecureStreamSocket.h>
#include <Common/getFQDNOrHostName.h>
#include <Core/MySQLProtocol.h>
#include "IServer.h"
#if USE_POCO_NETSSL
#include <Poco/Net/SecureStreamSocket.h>
#endif
namespace DB
{
@ -16,7 +16,7 @@ namespace DB
class MySQLHandler : public Poco::Net::TCPServerConnection
{
public:
MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, RSA & public_key_, RSA & private_key_, bool ssl_enabled, size_t connection_id_);
MySQLHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, bool ssl_enabled, size_t connection_id_);
void run() final;
@ -34,28 +34,47 @@ private:
void authenticate(const String & user_name, const String & auth_plugin_name, const String & auth_response);
virtual void authPluginSSL();
virtual void finishHandshakeSSL(size_t packet_size, char * buf, size_t pos, std::function<void(size_t)> read_bytes, MySQLProtocol::HandshakeResponse & packet);
IServer & server;
protected:
Poco::Logger * log;
Context connection_context;
std::shared_ptr<MySQLProtocol::PacketSender> packet_sender;
private:
size_t connection_id = 0;
size_t server_capability_flags = 0;
size_t client_capability_flags = 0;
RSA & public_key;
RSA & private_key;
protected:
std::unique_ptr<MySQLProtocol::Authentication::IPlugin> auth_plugin;
std::shared_ptr<Poco::Net::SecureStreamSocket> ss;
std::shared_ptr<ReadBuffer> in;
std::shared_ptr<WriteBuffer> out;
bool secure_connection = false;
};
}
#if USE_SSL && USE_POCO_NETSSL
class MySQLHandlerSSL : public MySQLHandler
{
public:
MySQLHandlerSSL(IServer & server_, const Poco::Net::StreamSocket & socket_, bool ssl_enabled, size_t connection_id_, RSA & public_key_, RSA & private_key_);
private:
void authPluginSSL() override;
void finishHandshakeSSL(size_t packet_size, char * buf, size_t pos, std::function<void(size_t)> read_bytes, MySQLProtocol::HandshakeResponse & packet) override;
RSA & public_key;
RSA & private_key;
std::shared_ptr<Poco::Net::SecureStreamSocket> ss;
};
#endif
}

21
dbms/programs/server/MySQLHandlerFactory.cpp
View File

@ -1,7 +1,5 @@
#include "MySQLHandlerFactory.h"
#if USE_POCO_NETSSL && USE_SSL
#include <Common/OpenSSLHelpers.h>
#include <Poco/Net/SSLManager.h>
#include <Poco/Net/TCPServerConnectionFactory.h>
#include <Poco/Util/Application.h>
#include <common/logger_useful.h>
@ -9,6 +7,10 @@
#include "IServer.h"
#include "MySQLHandler.h"
#if USE_POCO_NETSSL
#include <Poco/Net/SSLManager.h>
#endif
namespace DB
{
@ -24,6 +26,8 @@ MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
: server(server_)
, log(&Logger::get("MySQLHandlerFactory"))
{
#if USE_POCO_NETSSL
try
{
Poco::Net::SSLManager::instance().defaultServerContext();
@ -33,7 +37,9 @@ MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
LOG_INFO(log, "Failed to create SSL context. SSL will be disabled. Error: " << getCurrentExceptionMessage(false));
ssl_enabled = false;
}
#endif
#if USE_SSL
/// Reading rsa keys for SHA256 authentication plugin.
try
{
@ -44,8 +50,10 @@ MySQLHandlerFactory::MySQLHandlerFactory(IServer & server_)
LOG_WARNING(log, "Failed to read RSA keys. Error: " << getCurrentExceptionMessage(false));
generateRSAKeys();
}
#endif
}
#if USE_SSL
void MySQLHandlerFactory::readRSAKeys()
{
const Poco::Util::LayeredConfiguration & config = Poco::Util::Application::instance().config();
@ -113,13 +121,18 @@ void MySQLHandlerFactory::generateRSAKeys()
if (!private_key)
throw Exception("Failed to copy RSA key. Error: " + getOpenSSLErrors(), ErrorCodes::OPENSSL_ERROR);
}
#endif
Poco::Net::TCPServerConnection * MySQLHandlerFactory::createConnection(const Poco::Net::StreamSocket & socket)
{
size_t connection_id = last_connection_id++;
LOG_TRACE(log, "MySQL connection. Id: " << connection_id << ". Address: " << socket.peerAddress().toString());
return new MySQLHandler(server, socket, *public_key, *private_key, ssl_enabled, connection_id);
#if USE_POCO_NETSSL && USE_SSL
return new MySQLHandlerSSL(server, socket, ssl_enabled, connection_id, *public_key, *private_key);
#else
return new MySQLHandler(server, socket, ssl_enabled, connection_id);
#endif
}
}
#endif

11
dbms/programs/server/MySQLHandlerFactory.h
View File

@ -1,12 +1,12 @@
#pragma once
#include <Common/config.h>
#if USE_POCO_NETSSL && USE_SSL
#include <Poco/Net/TCPServerConnectionFactory.h>
#include <atomic>
#include <openssl/rsa.h>
#include "IServer.h"
#if USE_SSL
#include <openssl/rsa.h>
#endif
namespace DB
{
@ -17,6 +17,7 @@ private:
IServer & server;
Poco::Logger * log;
#if USE_SSL
struct RSADeleter
{
void operator()(RSA * ptr) { RSA_free(ptr); }
@ -27,6 +28,9 @@ private:
RSAPtr private_key;
bool ssl_enabled = true;
#else
bool ssl_enabled = false;
#endif
std::atomic<size_t> last_connection_id = 0;
public:
@ -40,4 +44,3 @@ public:
};
}
#endif

3
dbms/programs/server/Server.cpp
View File

@ -57,7 +57,7 @@
#include "TCPHandlerFactory.h"
#include "Common/config_version.h"
#include <Common/SensitiveDataMasker.h>
#include "MySQLHandlerFactory.h"
#if defined(OS_LINUX)
#include <Common/hasLinuxCapability.h>
@ -65,7 +65,6 @@
#endif
#if USE_POCO_NETSSL
#include "MySQLHandlerFactory.h"
#include <Poco/Net/Context.h>
#include <Poco/Net/SecureServerSocket.h>
#endif

3
dbms/programs/server/TCPHandler.cpp
View File

@ -530,7 +530,8 @@ void TCPHandler::processOrdinaryQuery()
sendLogs();
}
sendData(block);
if (!block || !state.io.null_format)
sendData(block);
if (!block)
break;
}

121
dbms/src/Interpreters/SettingsConstraints.cpp → dbms/src/Access/SettingsConstraints.cpp
View File

@ -1,4 +1,4 @@
#include <Interpreters/SettingsConstraints.h>
#include <Access/SettingsConstraints.h>
#include <Core/Settings.h>
#include <Common/FieldVisitors.h>
#include <IO/WriteHelpers.h>
@ -29,22 +29,118 @@ void SettingsConstraints::clear()
}
void SettingsConstraints::setReadOnly(const String & name, bool read_only)
void SettingsConstraints::setMinValue(const StringRef & name, const Field & min_value)
{
size_t setting_index = Settings::findIndexStrict(name);
getConstraintRef(setting_index).min_value = Settings::valueToCorrespondingType(setting_index, min_value);
}
Field SettingsConstraints::getMinValue(const StringRef & name) const
{
size_t setting_index = Settings::findIndexStrict(name);
const auto * ptr = tryGetConstraint(setting_index);
if (ptr)
return ptr->min_value;
else
return {};
}
void SettingsConstraints::setMaxValue(const StringRef & name, const Field & max_value)
{
size_t setting_index = Settings::findIndexStrict(name);
getConstraintRef(setting_index).max_value = Settings::valueToCorrespondingType(setting_index, max_value);
}
Field SettingsConstraints::getMaxValue(const StringRef & name) const
{
size_t setting_index = Settings::findIndexStrict(name);
const auto * ptr = tryGetConstraint(setting_index);
if (ptr)
return ptr->max_value;
else
return {};
}
void SettingsConstraints::setReadOnly(const StringRef & name, bool read_only)
{
size_t setting_index = Settings::findIndexStrict(name);
getConstraintRef(setting_index).read_only = read_only;
}
void SettingsConstraints::setMinValue(const String & name, const Field & min_value)
bool SettingsConstraints::isReadOnly(const StringRef & name) const
{
size_t setting_index = Settings::findIndexStrict(name);
getConstraintRef(setting_index).min_value = Settings::castValueWithoutApplying(setting_index, min_value);
const auto * ptr = tryGetConstraint(setting_index);
if (ptr)
return ptr->read_only;
else
return false;
}
void SettingsConstraints::setMaxValue(const String & name, const Field & max_value)
void SettingsConstraints::set(const StringRef & name, const Field & min_value, const Field & max_value, bool read_only)
{
size_t setting_index = Settings::findIndexStrict(name);
getConstraintRef(setting_index).max_value = Settings::castValueWithoutApplying(setting_index, max_value);
auto & ref = getConstraintRef(setting_index);
ref.min_value = min_value;
ref.max_value = max_value;
ref.read_only = read_only;
}
void SettingsConstraints::get(const StringRef & name, Field & min_value, Field & max_value, bool & read_only) const
{
size_t setting_index = Settings::findIndexStrict(name);
const auto * ptr = tryGetConstraint(setting_index);
if (ptr)
{
min_value = ptr->min_value;
max_value = ptr->max_value;
read_only = ptr->read_only;
}
else
{
min_value = Field{};
max_value = Field{};
read_only = false;
}
}
void SettingsConstraints::merge(const SettingsConstraints & other)
{
for (const auto & [setting_index, other_constraint] : other.constraints_by_index)
{
auto & constraint = constraints_by_index[setting_index];
if (!other_constraint.min_value.isNull())
constraint.min_value = other_constraint.min_value;
if (!other_constraint.max_value.isNull())
constraint.max_value = other_constraint.max_value;
if (other_constraint.read_only)
constraint.read_only = true;
}
}
SettingsConstraints::Infos SettingsConstraints::getInfo() const
{
Infos result;
result.reserve(constraints_by_index.size());
for (const auto & [setting_index, constraint] : constraints_by_index)
{
result.emplace_back();
Info & info = result.back();
info.name = Settings::getName(setting_index);
info.min = constraint.min_value;
info.max = constraint.max_value;
info.read_only = constraint.read_only;
}
return result;
}
@ -55,7 +151,7 @@ void SettingsConstraints::check(const Settings & current_settings, const Setting
if (setting_index == Settings::npos)
return;
Field new_value = Settings::castValueWithoutApplying(setting_index, change.value);
Field new_value = Settings::valueToCorrespondingType(setting_index, change.value);
Field current_value = current_settings.get(setting_index);
/// Setting isn't checked if value wasn't changed.
@ -159,4 +255,15 @@ void SettingsConstraints::loadFromConfig(const String & path_to_constraints, con
}
}
bool SettingsConstraints::Constraint::operator==(const Constraint & rhs) const
{
return (read_only == rhs.read_only) && (min_value == rhs.min_value) && (max_value == rhs.max_value);
}
bool operator ==(const SettingsConstraints & lhs, const SettingsConstraints & rhs)
{
return lhs.constraints_by_index == rhs.constraints_by_index;
}
}

34
dbms/src/Interpreters/SettingsConstraints.h → dbms/src/Access/SettingsConstraints.h
View File

@ -58,10 +58,32 @@ public:
~SettingsConstraints();
void clear();
bool empty() const { return constraints_by_index.empty(); }
void setMinValue(const String & name, const Field & min_value);
void setMaxValue(const String & name, const Field & max_value);
void setReadOnly(const String & name, bool read_only);
void setMinValue(const StringRef & name, const Field & min_value);
Field getMinValue(const StringRef & name) const;
void setMaxValue(const StringRef & name, const Field & max_value);
Field getMaxValue(const StringRef & name) const;
void setReadOnly(const StringRef & name, bool read_only);
bool isReadOnly(const StringRef & name) const;
void set(const StringRef & name, const Field & min_value, const Field & max_value, bool read_only);
void get(const StringRef & name, Field & min_value, Field & max_value, bool & read_only) const;
void merge(const SettingsConstraints & other);
struct Info
{
StringRef name;
Field min;
Field max;
bool read_only = false;
};
using Infos = std::vector<Info>;
Infos getInfo() const;
void check(const Settings & current_settings, const SettingChange & change) const;
void check(const Settings & current_settings, const SettingsChanges & changes) const;
@ -74,12 +96,18 @@ public:
/// Loads the constraints from configuration file, at "path" prefix in configuration.
void loadFromConfig(const String & path, const Poco::Util::AbstractConfiguration & config);
friend bool operator ==(const SettingsConstraints & lhs, const SettingsConstraints & rhs);
friend bool operator !=(const SettingsConstraints & lhs, const SettingsConstraints & rhs) { return !(lhs == rhs); }
private:
struct Constraint
{
bool read_only = false;
Field min_value;
Field max_value;
bool operator ==(const Constraint & rhs) const;
bool operator !=(const Constraint & rhs) const { return !(*this == rhs); }
};
Constraint & getConstraintRef(size_t index);

6
dbms/src/AggregateFunctions/AggregateFunctionCount.h
View File

@ -64,6 +64,12 @@ public:
}
const char * getHeaderFilePath() const override { return __FILE__; }
/// Reset the state to specified value. This function is not the part of common interface.
void set(AggregateDataPtr place, UInt64 new_count)
{
data(place).count = new_count;
}
};

4
dbms/src/AggregateFunctions/AggregateFunctionGroupBitmap.h
View File

@ -90,6 +90,10 @@ public:
{
Data & data_lhs = this->data(place);
const Data & data_rhs = this->data(rhs);
if (!data_rhs.doneFirst)
return;
if (!data_lhs.doneFirst)
{
data_lhs.doneFirst = true;

17
dbms/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h
View File

@ -581,6 +581,23 @@ public:
return max_val;
}
/**
* Replace value
*/
void rb_replace(const UInt32 * from_vals, const UInt32 * to_vals, size_t num)
{
if (isSmall())
toLarge();
for (size_t i = 0; i < num; ++i)
{
if (from_vals[i] == to_vals[i])
continue;
bool changed = roaring_bitmap_remove_checked(rb, from_vals[i]);
if (changed)
roaring_bitmap_add(rb, to_vals[i]);
}
}
private:
/// To read and write the DB Buffer directly, migrate code from CRoaring
void db_roaring_bitmap_add_many(DB::ReadBuffer & dbBuf, roaring_bitmap_t * r, size_t n_args)

8
dbms/src/AggregateFunctions/AggregateFunctionMinMaxAny.h
View File

@ -673,15 +673,15 @@ struct AggregateFunctionAnyHeavyData : Data
};
template <typename Data, bool AllocatesMemoryInArena>
class AggregateFunctionsSingleValue final : public IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, AllocatesMemoryInArena>>
template <typename Data, bool use_arena>
class AggregateFunctionsSingleValue final : public IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, use_arena>>
{
private:
DataTypePtr & type;
public:
AggregateFunctionsSingleValue(const DataTypePtr & type_)
: IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, AllocatesMemoryInArena>>({type_}, {})
: IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, use_arena>>({type_}, {})
, type(this->argument_types[0])
{
if (StringRef(Data::name()) == StringRef("min")
@ -722,7 +722,7 @@ public:
bool allocatesMemoryInArena() const override
{
return AllocatesMemoryInArena;
return use_arena;
}
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override

16
dbms/src/AggregateFunctions/AggregateFunctionQuantile.cpp
View File

@ -16,11 +16,11 @@ namespace ErrorCodes
namespace
{
template <typename Value, bool FloatReturn> using FuncQuantile = AggregateFunctionQuantile<Value, QuantileReservoirSampler<Value>, NameQuantile, false, std::conditional_t<FloatReturn, Float64, void>, false>;
template <typename Value, bool FloatReturn> using FuncQuantiles = AggregateFunctionQuantile<Value, QuantileReservoirSampler<Value>, NameQuantiles, false, std::conditional_t<FloatReturn, Float64, void>, true>;
template <typename Value, bool float_return> using FuncQuantile = AggregateFunctionQuantile<Value, QuantileReservoirSampler<Value>, NameQuantile, false, std::conditional_t<float_return, Float64, void>, false>;
template <typename Value, bool float_return> using FuncQuantiles = AggregateFunctionQuantile<Value, QuantileReservoirSampler<Value>, NameQuantiles, false, std::conditional_t<float_return, Float64, void>, true>;
template <typename Value, bool FloatReturn> using FuncQuantileDeterministic = AggregateFunctionQuantile<Value, QuantileReservoirSamplerDeterministic<Value>, NameQuantileDeterministic, true, std::conditional_t<FloatReturn, Float64, void>, false>;
template <typename Value, bool FloatReturn> using FuncQuantilesDeterministic = AggregateFunctionQuantile<Value, QuantileReservoirSamplerDeterministic<Value>, NameQuantilesDeterministic, true, std::conditional_t<FloatReturn, Float64, void>, true>;
template <typename Value, bool float_return> using FuncQuantileDeterministic = AggregateFunctionQuantile<Value, QuantileReservoirSamplerDeterministic<Value>, NameQuantileDeterministic, true, std::conditional_t<float_return, Float64, void>, false>;
template <typename Value, bool float_return> using FuncQuantilesDeterministic = AggregateFunctionQuantile<Value, QuantileReservoirSamplerDeterministic<Value>, NameQuantilesDeterministic, true, std::conditional_t<float_return, Float64, void>, true>;
template <typename Value, bool _> using FuncQuantileExact = AggregateFunctionQuantile<Value, QuantileExact<Value>, NameQuantileExact, false, void, false>;
template <typename Value, bool _> using FuncQuantilesExact = AggregateFunctionQuantile<Value, QuantileExact<Value>, NameQuantilesExact, false, void, true>;
@ -40,11 +40,11 @@ template <typename Value, bool _> using FuncQuantilesTiming = AggregateFunctionQ
template <typename Value, bool _> using FuncQuantileTimingWeighted = AggregateFunctionQuantile<Value, QuantileTiming<Value>, NameQuantileTimingWeighted, true, Float32, false>;
template <typename Value, bool _> using FuncQuantilesTimingWeighted = AggregateFunctionQuantile<Value, QuantileTiming<Value>, NameQuantilesTimingWeighted, true, Float32, true>;
template <typename Value, bool FloatReturn> using FuncQuantileTDigest = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantileTDigest, false, std::conditional_t<FloatReturn, Float32, void>, false>;
template <typename Value, bool FloatReturn> using FuncQuantilesTDigest = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantilesTDigest, false, std::conditional_t<FloatReturn, Float32, void>, true>;
template <typename Value, bool float_return> using FuncQuantileTDigest = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantileTDigest, false, std::conditional_t<float_return, Float32, void>, false>;
template <typename Value, bool float_return> using FuncQuantilesTDigest = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantilesTDigest, false, std::conditional_t<float_return, Float32, void>, true>;
template <typename Value, bool FloatReturn> using FuncQuantileTDigestWeighted = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantileTDigestWeighted, true, std::conditional_t<FloatReturn, Float32, void>, false>;
template <typename Value, bool FloatReturn> using FuncQuantilesTDigestWeighted = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantilesTDigestWeighted, true, std::conditional_t<FloatReturn, Float32, void>, true>;
template <typename Value, bool float_return> using FuncQuantileTDigestWeighted = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantileTDigestWeighted, true, std::conditional_t<float_return, Float32, void>, false>;
template <typename Value, bool float_return> using FuncQuantilesTDigestWeighted = AggregateFunctionQuantile<Value, QuantileTDigest<Value>, NameQuantilesTDigestWeighted, true, std::conditional_t<float_return, Float32, void>, true>;
template <template <typename, bool> class Function>

2
dbms/src/AggregateFunctions/ReservoirSampler.h
View File

@ -31,7 +31,7 @@ namespace ReservoirSamplerOnEmpty
};
}
template <typename ResultType, bool IsFloatingPoint>
template <typename ResultType, bool is_float>
struct NanLikeValueConstructor
{
static ResultType getValue()

4
dbms/src/Columns/ColumnSet.h
View File

@ -1,6 +1,7 @@
#pragma once
#include <Columns/IColumnDummy.h>
#include <Core/Field.h>
namespace DB
@ -28,6 +29,9 @@ public:
ConstSetPtr getData() const { return data; }
// Used only for debugging, making it DUMPABLE
Field operator[](size_t) const override { return {}; }
private:
ConstSetPtr data;
};

2
dbms/src/Columns/ColumnVector.cpp
View File

@ -112,7 +112,7 @@ void ColumnVector<T>::getPermutation(bool reverse, size_t limit, int nan_directi
else
{
/// A case for radix sort
if constexpr (std::is_arithmetic_v<T> && !std::is_same_v<T, UInt128>)
if constexpr (is_arithmetic_v<T> && !std::is_same_v<T, UInt128>)
{
/// Thresholds on size. Lower threshold is arbitrary. Upper threshold is chosen by the type for histogram counters.
if (s >= 256 && s <= std::numeric_limits<UInt32>::max())

2
dbms/src/Common/HashTable/Hash.h
View File

@ -76,7 +76,7 @@ template <typename T, typename Enable = void>
struct DefaultHash;
template <typename T>
struct DefaultHash<T, std::enable_if_t<std::is_arithmetic_v<T>>>
struct DefaultHash<T, std::enable_if_t<is_arithmetic_v<T>>>
{
size_t operator() (T key) const
{

10
dbms/src/Common/RadixSort.h
View File

@ -165,12 +165,10 @@ struct RadixSortIntTraits
template <typename T>
using RadixSortNumTraits =
std::conditional_t<std::is_integral_v<T>,
std::conditional_t<std::is_unsigned_v<T>,
RadixSortUIntTraits<T>,
RadixSortIntTraits<T>>,
RadixSortFloatTraits<T>>;
using RadixSortNumTraits = std::conditional_t<
is_integral_v<T>,
std::conditional_t<is_unsigned_v<T>, RadixSortUIntTraits<T>, RadixSortIntTraits<T>>,
RadixSortFloatTraits<T>>;
template <typename Traits>

3
dbms/src/Common/SettingsChanges.h
View File

@ -10,6 +10,9 @@ struct SettingChange
{
String name;
Field value;
friend bool operator ==(const SettingChange & lhs, const SettingChange & rhs) { return (lhs.name == rhs.name) && (lhs.value == rhs.value); }
friend bool operator !=(const SettingChange & lhs, const SettingChange & rhs) { return !(lhs == rhs); }
};
using SettingsChanges = std::vector<SettingChange>;

2
dbms/src/Common/StackTrace.cpp
View File

@ -30,7 +30,7 @@ std::string signalToErrorMessage(int sig, const siginfo_t & info, const ucontext
else
error << "Address: " << info.si_addr;
#if defined(__x86_64__) && !defined(__FreeBSD__) && !defined(__APPLE__)
#if defined(__x86_64__) && !defined(__FreeBSD__) && !defined(__APPLE__) && !defined(__arm__)
auto err_mask = context.uc_mcontext.gregs[REG_ERR];
if ((err_mask & 0x02))
error << " Access: write.";

24
dbms/src/Common/UInt128.h
View File

@ -182,18 +182,8 @@ struct UInt256HashCRC32
struct UInt256HashCRC32 : public UInt256Hash {};
#endif
}
/// Overload hash for type casting
namespace std
{
template <> struct hash<DB::UInt128>
{
size_t operator()(const DB::UInt128 & u) const
{
return CityHash_v1_0_2::Hash128to64({u.low, u.high});
}
};
}
template <> struct is_signed<DB::UInt128>
{
@ -215,4 +205,16 @@ template <> struct is_arithmetic<DB::UInt128>
{
static constexpr bool value = false;
};
/// Overload hash for type casting
namespace std
{
template <> struct hash<DB::UInt128>
{
size_t operator()(const DB::UInt128 & u) const
{
return CityHash_v1_0_2::Hash128to64({u.low, u.high});
}
};
}

6
dbms/src/Common/tests/int_hashes_perf.cpp
View File

@ -31,7 +31,7 @@ void setAffinity()
static inline ALWAYS_INLINE UInt64 rdtsc()
{
#if __x86_64__
#if defined(__x86_64__)
UInt32 a, d;
__asm__ volatile ("rdtsc" : "=a" (a), "=d" (d));
return static_cast<UInt64>(a) | (static_cast<UInt64>(d) << 32);
@ -109,7 +109,7 @@ static inline size_t murmurMix(UInt64 x)
}
#if __x86_64__
#if defined(__x86_64__)
static inline size_t crc32Hash(UInt64 x)
{
UInt64 crc = -1ULL;
@ -309,7 +309,7 @@ int main(int argc, char ** argv)
if (!method || method == 8) test<mulShift> (n, data.data(), "7: mulShift");
if (!method || method == 9) test<tabulation>(n, data.data(), "8: tabulation");
#if __x86_64__
#if defined(__x86_64__)
if (!method || method == 10) test<crc32Hash> (n, data.data(), "9: crc32");
#endif

4
dbms/src/Compression/CompressionCodecDoubleDelta.cpp
View File

@ -109,7 +109,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
{
// Since only unsinged int has granted 2-compliment overflow handling, we are doing math here on unsigned types.
// To simplify and booletproof code, we operate enforce ValueType to be unsigned too.
static_assert(std::is_unsigned_v<ValueType>, "ValueType must be unsigned.");
static_assert(is_unsigned_v<ValueType>, "ValueType must be unsigned.");
using UnsignedDeltaType = ValueType;
// We use signed delta type to turn huge unsigned values into smaller signed:
@ -189,7 +189,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
template <typename ValueType>
void decompressDataForType(const char * source, UInt32 source_size, char * dest)
{
static_assert(std::is_unsigned_v<ValueType>, "ValueType must be unsigned.");
static_assert(is_unsigned_v<ValueType>, "ValueType must be unsigned.");
using UnsignedDeltaType = ValueType;
using SignedDeltaType = typename std::make_signed<UnsignedDeltaType>::type;

10
dbms/src/Compression/CompressionCodecT64.cpp
View File

@ -262,10 +262,10 @@ void reverseTranspose(const char * src, T * buf, UInt32 num_bits, UInt32 tail =
reverseTransposeBytes(matrix, col, buf[col]);
}
template <typename T, typename MinMaxT = std::conditional_t<std::is_signed_v<T>, Int64, UInt64>>
template <typename T, typename MinMaxT = std::conditional_t<is_signed_v<T>, Int64, UInt64>>
void restoreUpperBits(T * buf, T upper_min, T upper_max [[maybe_unused]], T sign_bit [[maybe_unused]], UInt32 tail = 64)
{
if constexpr (std::is_signed_v<T>)
if constexpr (is_signed_v<T>)
{
/// Restore some data as negatives and others as positives
if (sign_bit)
@ -334,7 +334,7 @@ using Variant = CompressionCodecT64::Variant;
template <typename T, bool full>
UInt32 compressData(const char * src, UInt32 bytes_size, char * dst)
{
using MinMaxType = std::conditional_t<std::is_signed_v<T>, Int64, UInt64>;
using MinMaxType = std::conditional_t<is_signed_v<T>, Int64, UInt64>;
static constexpr const UInt32 matrix_size = 64;
static constexpr const UInt32 header_size = 2 * sizeof(UInt64);
@ -389,7 +389,7 @@ UInt32 compressData(const char * src, UInt32 bytes_size, char * dst)
template <typename T, bool full>
void decompressData(const char * src, UInt32 bytes_size, char * dst, UInt32 uncompressed_size)
{
using MinMaxType = std::conditional_t<std::is_signed_v<T>, Int64, UInt64>;
using MinMaxType = std::conditional_t<is_signed_v<T>, Int64, UInt64>;
static constexpr const UInt32 matrix_size = 64;
static constexpr const UInt32 header_size = 2 * sizeof(UInt64);
@ -441,7 +441,7 @@ void decompressData(const char * src, UInt32 bytes_size, char * dst, UInt32 unco
if (num_bits < 64)
upper_min = UInt64(min) >> num_bits << num_bits;
if constexpr (std::is_signed_v<T>)
if constexpr (is_signed_v<T>)
{
if (min < 0 && max >= 0 && num_bits < 64)
{

2
dbms/src/Compression/tests/gtest_compressionCodec.cpp
View File

@ -441,7 +441,7 @@ auto SequentialGenerator = [](auto stride = 1)
template <typename T>
using uniform_distribution =
typename std::conditional_t<std::is_floating_point_v<T>, std::uniform_real_distribution<T>,
typename std::conditional_t<std::is_integral_v<T>, std::uniform_int_distribution<T>, void>>;
typename std::conditional_t<is_integral_v<T>, std::uniform_int_distribution<T>, void>>;
template <typename T = Int32>

15
dbms/src/Core/AccurateComparison.h
View File

@ -35,10 +35,10 @@ using DB::UInt64;
// Case 1. Is pair of floats or pair of ints or pair of uints
template <typename A, typename B>
constexpr bool is_safe_conversion = (std::is_floating_point_v<A> && std::is_floating_point_v<B>)
|| (std::is_integral_v<A> && std::is_integral_v<B> && !(std::is_signed_v<A> ^ std::is_signed_v<B>))
|| (is_integral_v<A> && is_integral_v<B> && !(is_signed_v<A> ^ is_signed_v<B>))
|| (std::is_same_v<A, DB::Int128> && std::is_same_v<B, DB::Int128>)
|| (std::is_integral_v<A> && std::is_same_v<B, DB::Int128>)
|| (std::is_same_v<A, DB::Int128> && std::is_integral_v<B>);
|| (is_integral_v<A> && std::is_same_v<B, DB::Int128>)
|| (std::is_same_v<A, DB::Int128> && is_integral_v<B>);
template <typename A, typename B>
using bool_if_safe_conversion = std::enable_if_t<is_safe_conversion<A, B>, bool>;
template <typename A, typename B>
@ -47,8 +47,8 @@ using bool_if_not_safe_conversion = std::enable_if_t<!is_safe_conversion<A, B>,
/// Case 2. Are params IntXX and UIntYY ?
template <typename TInt, typename TUInt>
constexpr bool is_any_int_vs_uint = std::is_integral_v<TInt> && std::is_integral_v<TUInt> &&
std::is_signed_v<TInt> && std::is_unsigned_v<TUInt>;
constexpr bool is_any_int_vs_uint
= is_integral_v<TInt> && is_integral_v<TUInt> && is_signed_v<TInt> && is_unsigned_v<TUInt>;
// Case 2a. Are params IntXX and UIntYY and sizeof(IntXX) >= sizeof(UIntYY) (in such case will use accurate compare)
@ -117,9 +117,8 @@ inline bool_if_gt_int_vs_uint<TInt, TUInt> equalsOpTmpl(TUInt a, TInt b)
// Case 3a. Comparison via conversion to double.
template <typename TAInt, typename TAFloat>
using bool_if_double_can_be_used = std::enable_if_t<
std::is_integral_v<TAInt> && (sizeof(TAInt) <= 4) && std::is_floating_point_v<TAFloat>,
bool>;
using bool_if_double_can_be_used
= std::enable_if_t<is_integral_v<TAInt> && (sizeof(TAInt) <= 4) && std::is_floating_point_v<TAFloat>, bool>;
template <typename TAInt, typename TAFloat>
inline bool_if_double_can_be_used<TAInt, TAFloat> greaterOpTmpl(TAInt a, TAFloat b)

4
dbms/src/Core/DecimalComparison.h
View File

@ -233,9 +233,9 @@ private:
overflow |= (A(x) != a);
if constexpr (sizeof(B) > sizeof(CompareInt))
overflow |= (B(y) != b);
if constexpr (std::is_unsigned_v<A>)
if constexpr (is_unsigned_v<A>)
overflow |= (x < 0);
if constexpr (std::is_unsigned_v<B>)
if constexpr (is_unsigned_v<B>)
overflow |= (y < 0);
if constexpr (scale_left)

2
dbms/src/Core/Field.h
View File

@ -656,7 +656,7 @@ template <> struct TypeName<AggregateFunctionStateData> { static std::string get
/// char may be signed or unsigned, and behave identically to signed char or unsigned char,
/// but they are always three different types.
/// signedness of char is different in Linux on x86 and Linux on ARM.
template <> struct NearestFieldTypeImpl<char> { using Type = std::conditional_t<std::is_signed_v<char>, Int64, UInt64>; };
template <> struct NearestFieldTypeImpl<char> { using Type = std::conditional_t<is_signed_v<char>, Int64, UInt64>; };
template <> struct NearestFieldTypeImpl<signed char> { using Type = Int64; };
template <> struct NearestFieldTypeImpl<unsigned char> { using Type = UInt64; };

5
dbms/src/Core/MySQLProtocol.cpp
View File

@ -1,7 +1,4 @@
#include "MySQLProtocol.h"
#if USE_SSL
#include <IO/WriteBuffer.h>
#include <IO/ReadBufferFromString.h>
#include <IO/WriteBufferFromString.h>
@ -104,5 +101,3 @@ size_t getLengthEncodedStringSize(const String & s)
}
}
#endif // USE_SSL

17
dbms/src/Core/MySQLProtocol.h
View File

@ -1,12 +1,5 @@
#pragma once
#include "config_core.h"
#if USE_SSL
#include <ext/scope_guard.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <random>
#include <sstream>
#include <Common/MemoryTracker.h>
@ -27,6 +20,11 @@
#include <Poco/Net/StreamSocket.h>
#include <Poco/RandomStream.h>
#include <Poco/SHA1Engine.h>
#include "config_core.h"
#if USE_SSL
#include <openssl/pem.h>
#include <openssl/rsa.h>
#endif
/// Implementation of MySQL wire protocol.
/// Works only on little-endian architecture.
@ -941,6 +939,7 @@ private:
String scramble;
};
#if USE_SSL
/// Caching SHA2 plugin is not used because it would be possible to authenticate knowing hash from users.xml.
/// https://dev.mysql.com/doc/internals/en/sha256.html
class Sha256Password : public IPlugin
@ -1001,7 +1000,6 @@ public:
if (auth_response == "\1")
{
LOG_TRACE(log, "Client requests public key.");
BIO * mem = BIO_new(BIO_s_mem());
SCOPE_EXIT(BIO_free(mem));
if (PEM_write_bio_RSA_PUBKEY(mem, &public_key) != 1)
@ -1074,10 +1072,9 @@ private:
Logger * log;
String scramble;
};
#endif
}
}
}
#endif // USE_SSL

1
dbms/src/Core/Settings.h
View File

@ -384,6 +384,7 @@ struct Settings : public SettingsCollection<Settings>
M(SettingUInt64, min_free_disk_space_for_temporary_data, 0, "The minimum disk space to keep while writing temporary data used in external sorting and aggregation.") \
\
M(SettingBool, enable_scalar_subquery_optimization, true, "If it is set to true, prevent scalar subqueries from (de)serializing large scalar values and possibly avoid running the same subquery more than once.") \
M(SettingBool, optimize_trivial_count_query, true, "Process trivial 'SELECT count() FROM table' query from metadata.") \
\
/** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \
\

36
dbms/src/Core/SettingsCollectionImpl.h
View File

@ -25,15 +25,27 @@ void SettingsCollection<Derived>::reference::setValue(const Field & value)
}
template <class Derived>
Field SettingsCollection<Derived>::castValueWithoutApplying(size_t index, const Field & value)
String SettingsCollection<Derived>::valueToString(size_t index, const Field & value)
{
return members()[index].cast_value_without_applying(value);
return members()[index].value_to_string(value);
}
template <class Derived>
Field SettingsCollection<Derived>::castValueWithoutApplying(const String & name, const Field & value)
String SettingsCollection<Derived>::valueToString(const StringRef & name, const Field & value)
{
return members().findStrict(name)->cast_value_without_applying(value);
return members().findStrict(name)->value_to_string(value);
}
template <class Derived>
Field SettingsCollection<Derived>::valueToCorrespondingType(size_t index, const Field & value)
{
return members()[index].value_to_corresponding_type(value);
}
template <class Derived>
Field SettingsCollection<Derived>::valueToCorrespondingType(const StringRef & name, const Field & value)
{
return members().findStrict(name)->value_to_corresponding_type(value);
}
template <class Derived>
@ -43,7 +55,7 @@ void SettingsCollection<Derived>::set(size_t index, const Field & value)
}
template <class Derived>
void SettingsCollection<Derived>::set(const String & name, const Field & value)
void SettingsCollection<Derived>::set(const StringRef & name, const Field & value)
{
(*this)[name].setValue(value);
}
@ -55,13 +67,13 @@ Field SettingsCollection<Derived>::get(size_t index) const
}
template <class Derived>
Field SettingsCollection<Derived>::get(const String & name) const
Field SettingsCollection<Derived>::get(const StringRef & name) const
{
return (*this)[name].getValue();
}
template <class Derived>
bool SettingsCollection<Derived>::tryGet(const String & name, Field & value) const
bool SettingsCollection<Derived>::tryGet(const StringRef & name, Field & value) const
{
auto it = find(name);
if (it == end())
@ -71,7 +83,7 @@ bool SettingsCollection<Derived>::tryGet(const String & name, Field & value) con
}
template <class Derived>
bool SettingsCollection<Derived>::tryGet(const String & name, String & value) const
bool SettingsCollection<Derived>::tryGet(const StringRef & name, String & value) const
{
auto it = find(name);
if (it == end())
@ -85,8 +97,8 @@ bool SettingsCollection<Derived>::operator ==(const Derived & rhs) const
{
for (const auto & member : members())
{
bool left_changed = member.isChanged(castToDerived());
bool right_changed = member.isChanged(rhs);
bool left_changed = member.is_changed(castToDerived());
bool right_changed = member.is_changed(rhs);
if (left_changed || right_changed)
{
if (left_changed != right_changed)
@ -105,7 +117,7 @@ SettingsChanges SettingsCollection<Derived>::changes() const
SettingsChanges found_changes;
for (const auto & member : members())
{
if (member.isChanged(castToDerived()))
if (member.is_changed(castToDerived()))
found_changes.push_back({member.name.toString(), member.get_field(castToDerived())});
}
return found_changes;
@ -130,7 +142,7 @@ template <class Derived>
void SettingsCollection<Derived>::copyChangesFrom(const Derived & src)
{
for (const auto & member : members())
if (member.isChanged(src))
if (member.is_changed(src))
member.set_field(castToDerived(), member.get_field(src));
}

8
dbms/src/Core/SettingsCommon.cpp
View File

@ -92,9 +92,9 @@ void SettingNumber<bool>::set(const String & x)
template <typename Type>
void SettingNumber<Type>::serialize(WriteBuffer & buf) const
{
if constexpr (std::is_integral_v<Type> && std::is_unsigned_v<Type>)
if constexpr (is_integral_v<Type> && is_unsigned_v<Type>)
writeVarUInt(static_cast<UInt64>(value), buf);
else if constexpr (std::is_integral_v<Type> && std::is_signed_v<Type>)
else if constexpr (is_integral_v<Type> && is_signed_v<Type>)
writeVarInt(static_cast<Int64>(value), buf);
else
{
@ -106,13 +106,13 @@ void SettingNumber<Type>::serialize(WriteBuffer & buf) const
template <typename Type>
void SettingNumber<Type>::deserialize(ReadBuffer & buf)
{
if constexpr (std::is_integral_v<Type> && std::is_unsigned_v<Type>)
if constexpr (is_integral_v<Type> && is_unsigned_v<Type>)
{
UInt64 x;
readVarUInt(x, buf);
set(static_cast<Type>(x));
}
else if constexpr (std::is_integral_v<Type> && std::is_signed_v<Type>)
else if constexpr (is_integral_v<Type> && is_signed_v<Type>)
{
Int64 x;
readVarInt(x, buf);

54
dbms/src/Core/SettingsCommon.h
View File

@ -311,8 +311,8 @@ private:
using SetFieldFunction = void (*)(Derived &, const Field &);
using SerializeFunction = void (*)(const Derived &, WriteBuffer & buf);
using DeserializeFunction = void (*)(Derived &, ReadBuffer & buf);
using CastValueWithoutApplyingFunction = Field (*)(const Field &);
using ValueToStringFunction = String (*)(const Field &);
using ValueToCorrespondingTypeFunction = Field (*)(const Field &);
struct MemberInfo
{
@ -325,9 +325,8 @@ private:
SetFieldFunction set_field;
SerializeFunction serialize;
DeserializeFunction deserialize;
CastValueWithoutApplyingFunction cast_value_without_applying;
bool isChanged(const Derived & collection) const { return is_changed(collection); }
ValueToStringFunction value_to_string;
ValueToCorrespondingTypeFunction value_to_corresponding_type;
};
class MemberInfos : private boost::noncopyable
@ -394,7 +393,7 @@ public:
const_reference(const const_reference & src) = default;
const StringRef & getName() const { return member->name; }
const StringRef & getDescription() const { return member->description; }
bool isChanged() const { return member->isChanged(*collection); }
bool isChanged() const { return member->is_changed(*collection); }
Field getValue() const;
String getValueAsString() const { return member->get_string(*collection); }
protected:
@ -457,16 +456,20 @@ public:
static StringRef getDescription(const String & name) { return members().findStrict(name)->description; }
/// Searches a setting by its name; returns `npos` if not found.
static size_t findIndex(const String & name) { return members().findIndex(name); }
static size_t findIndex(const StringRef & name) { return members().findIndex(name); }
static constexpr size_t npos = static_cast<size_t>(-1);
/// Searches a setting by its name; throws an exception if not found.
static size_t findIndexStrict(const String & name) { return members().findIndexStrict(name); }
static size_t findIndexStrict(const StringRef & name) { return members().findIndexStrict(name); }
/// Casts a value to a string according to a specified setting without actual changing this settings.
static String valueToString(size_t index, const Field & value);
static String valueToString(const StringRef & name, const Field & value);
/// Casts a value to a type according to a specified setting without actual changing this settings.
/// E.g. for SettingInt64 it casts Field to Field::Types::Int64.
static Field castValueWithoutApplying(size_t index, const Field & value);
static Field castValueWithoutApplying(const String & name, const Field & value);
static Field valueToCorrespondingType(size_t index, const Field & value);
static Field valueToCorrespondingType(const StringRef & name, const Field & value);
iterator begin() { return iterator(castToDerived(), members().begin()); }
const_iterator begin() const { return const_iterator(castToDerived(), members().begin()); }
@ -475,39 +478,39 @@ public:
/// Returns a proxy object for accessing to a setting. Throws an exception if there is not setting with such name.
reference operator[](size_t index) { return reference(castToDerived(), members()[index]); }
reference operator[](const String & name) { return reference(castToDerived(), *(members().findStrict(name))); }
reference operator[](const StringRef & name) { return reference(castToDerived(), *(members().findStrict(name))); }
const_reference operator[](size_t index) const { return const_reference(castToDerived(), members()[index]); }
const_reference operator[](const String & name) const { return const_reference(castToDerived(), *(members().findStrict(name))); }
const_reference operator[](const StringRef & name) const { return const_reference(castToDerived(), *(members().findStrict(name))); }
/// Searches a setting by its name; returns end() if not found.
iterator find(const String & name) { return iterator(castToDerived(), members().find(name)); }
const_iterator find(const String & name) const { return const_iterator(castToDerived(), members().find(name)); }
iterator find(const StringRef & name) { return iterator(castToDerived(), members().find(name)); }
const_iterator find(const StringRef & name) const { return const_iterator(castToDerived(), members().find(name)); }
/// Searches a setting by its name; throws an exception if not found.
iterator findStrict(const String & name) { return iterator(castToDerived(), members().findStrict(name)); }
const_iterator findStrict(const String & name) const { return const_iterator(castToDerived(), members().findStrict(name)); }
iterator findStrict(const StringRef & name) { return iterator(castToDerived(), members().findStrict(name)); }
const_iterator findStrict(const StringRef & name) const { return const_iterator(castToDerived(), members().findStrict(name)); }
/// Sets setting's value.
void set(size_t index, const Field & value);
void set(const String & name, const Field & value);
void set(const StringRef & name, const Field & value);
/// Sets setting's value. Read value in text form from string (for example, from configuration file or from URL parameter).
void set(size_t index, const String & value) { (*this)[index].setValue(value); }
void set(const String & name, const String & value) { (*this)[name].setValue(value); }
void set(const StringRef & name, const String & value) { (*this)[name].setValue(value); }
/// Returns value of a setting.
Field get(size_t index) const;
Field get(const String & name) const;
Field get(const StringRef & name) const;
/// Returns value of a setting converted to string.
String getAsString(size_t index) const { return (*this)[index].getValueAsString(); }
String getAsString(const String & name) const { return (*this)[name].getValueAsString(); }
String getAsString(const StringRef & name) const { return (*this)[name].getValueAsString(); }
/// Returns value of a setting; returns false if there is no setting with the specified name.
bool tryGet(const String & name, Field & value) const;
bool tryGet(const StringRef & name, Field & value) const;
/// Returns value of a setting converted to string; returns false if there is no setting with the specified name.
bool tryGet(const String & name, String & value) const;
bool tryGet(const StringRef & name, String & value) const;
/// Compares two collections of settings.
bool operator ==(const Derived & rhs) const;
@ -537,7 +540,7 @@ public:
{
for (const auto & member : members())
{
if (member.isChanged(castToDerived()))
if (member.is_changed(castToDerived()))
{
details::SettingsCollectionUtils::serializeName(member.name, buf);
member.serialize(castToDerived(), buf);
@ -600,7 +603,8 @@ public:
static void NAME##_setField(Derived & collection, const Field & value) { collection.NAME.set(value); } \
static void NAME##_serialize(const Derived & collection, WriteBuffer & buf) { collection.NAME.serialize(buf); } \
static void NAME##_deserialize(Derived & collection, ReadBuffer & buf) { collection.NAME.deserialize(buf); } \
static Field NAME##_castValueWithoutApplying(const Field & value) { TYPE temp{DEFAULT}; temp.set(value); return temp.toField(); } \
static String NAME##_valueToString(const Field & value) { TYPE temp{DEFAULT}; temp.set(value); return temp.toString(); } \
static Field NAME##_valueToCorrespondingType(const Field & value) { TYPE temp{DEFAULT}; temp.set(value); return temp.toField(); } \
#define IMPLEMENT_SETTINGS_COLLECTION_ADD_MEMBER_INFO_HELPER_(TYPE, NAME, DEFAULT, DESCRIPTION) \
@ -609,5 +613,5 @@ public:
&Functions::NAME##_getString, &Functions::NAME##_getField, \
&Functions::NAME##_setString, &Functions::NAME##_setField, \
&Functions::NAME##_serialize, &Functions::NAME##_deserialize, \
&Functions::NAME##_castValueWithoutApplying });
&Functions::NAME##_valueToString, &Functions::NAME##_valueToCorrespondingType});
}

1
dbms/src/Core/Types.h
View File

@ -3,6 +3,7 @@
#include <cstdint>
#include <string>
#include <vector>
#include <common/Types.h>
namespace DB

3
dbms/src/DataStreams/BlockIO.h
View File

@ -33,6 +33,9 @@ struct BlockIO
std::function<void(IBlockInputStream *, IBlockOutputStream *)> finish_callback;
std::function<void()> exception_callback;
/// When it is true, don't bother sending any non-empty blocks to the out stream
bool null_format = false;
/// Call these functions if you want to log the request.
void onFinish()
{

91
dbms/src/DataStreams/ExecutionSpeedLimits.cpp Normal file
View File

@ -0,0 +1,91 @@
#include <DataStreams/ExecutionSpeedLimits.h>
#include <Common/ProfileEvents.h>
#include <Common/CurrentThread.h>
#include <IO/WriteHelpers.h>
#include <common/sleep.h>
namespace ProfileEvents
{
extern const Event ThrottlerSleepMicroseconds;
}
namespace DB
{
namespace ErrorCodes
{
extern const int TOO_SLOW;
}
static void limitProgressingSpeed(size_t total_progress_size, size_t max_speed_in_seconds, UInt64 total_elapsed_microseconds)
{
/// How much time to wait for the average speed to become `max_speed_in_seconds`.
UInt64 desired_microseconds = total_progress_size * 1000000 / max_speed_in_seconds;
if (desired_microseconds > total_elapsed_microseconds)
{
UInt64 sleep_microseconds = desired_microseconds - total_elapsed_microseconds;
/// Never sleep more than one second (it should be enough to limit speed for a reasonable amount, and otherwise it's too easy to make query hang).
sleep_microseconds = std::min(UInt64(1000000), sleep_microseconds);
sleepForMicroseconds(sleep_microseconds);
ProfileEvents::increment(ProfileEvents::ThrottlerSleepMicroseconds, sleep_microseconds);
}
}
void ExecutionSpeedLimits::throttle(
size_t read_rows, size_t read_bytes,
size_t total_rows_to_read, UInt64 total_elapsed_microseconds)
{
if ((min_execution_rps != 0 || max_execution_rps != 0
|| min_execution_bps != 0 || max_execution_bps != 0
|| (total_rows_to_read != 0 && timeout_before_checking_execution_speed != 0)) &&
(static_cast<Int64>(total_elapsed_microseconds) > timeout_before_checking_execution_speed.totalMicroseconds()))
{
/// Do not count sleeps in throttlers
UInt64 throttler_sleep_microseconds = CurrentThread::getProfileEvents()[ProfileEvents::ThrottlerSleepMicroseconds];
double elapsed_seconds = 0;
if (throttler_sleep_microseconds > total_elapsed_microseconds)
elapsed_seconds = static_cast<double>(total_elapsed_microseconds - throttler_sleep_microseconds) / 1000000.0;
if (elapsed_seconds > 0)
{
auto rows_per_second = read_rows / elapsed_seconds;
if (min_execution_rps && rows_per_second < min_execution_rps)
throw Exception("Query is executing too slow: " + toString(read_rows / elapsed_seconds)
+ " rows/sec., minimum: " + toString(min_execution_rps),
ErrorCodes::TOO_SLOW);
auto bytes_per_second = read_bytes / elapsed_seconds;
if (min_execution_bps && bytes_per_second < min_execution_bps)
throw Exception("Query is executing too slow: " + toString(read_bytes / elapsed_seconds)
+ " bytes/sec., minimum: " + toString(min_execution_bps),
ErrorCodes::TOO_SLOW);
/// If the predicted execution time is longer than `max_execution_time`.
if (max_execution_time != 0 && total_rows_to_read && read_rows)
{
double estimated_execution_time_seconds = elapsed_seconds * (static_cast<double>(total_rows_to_read) / read_rows);
if (estimated_execution_time_seconds > max_execution_time.totalSeconds())
throw Exception("Estimated query execution time (" + toString(estimated_execution_time_seconds) + " seconds)"
+ " is too long. Maximum: " + toString(max_execution_time.totalSeconds())
+ ". Estimated rows to process: " + toString(total_rows_to_read),
ErrorCodes::TOO_SLOW);
}
if (max_execution_rps && rows_per_second >= max_execution_rps)
limitProgressingSpeed(read_rows, max_execution_rps, total_elapsed_microseconds);
if (max_execution_bps && bytes_per_second >= max_execution_bps)
limitProgressingSpeed(read_bytes, max_execution_bps, total_elapsed_microseconds);
}
}
}
}

29
dbms/src/DataStreams/ExecutionSpeedLimits.h Normal file
View File

@ -0,0 +1,29 @@
#pragma once
#include <Poco/Timespan.h>
#include <Core/Types.h>
namespace DB
{
/// Limits for query execution speed.
class ExecutionSpeedLimits
{
public:
/// For rows per second.
size_t min_execution_rps = 0;
size_t max_execution_rps = 0;
/// For bytes per second.
size_t min_execution_bps = 0;
size_t max_execution_bps = 0;
Poco::Timespan max_execution_time = 0;
/// Verify that the speed is not too low after the specified time has elapsed.
Poco::Timespan timeout_before_checking_execution_speed = 0;
/// Pause execution in case if speed limits were exceeded.
void throttle(size_t read_rows, size_t read_bytes, size_t total_rows_to_read, UInt64 total_elapsed_microseconds);
};
}

2
dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.cpp
View File

@ -103,7 +103,7 @@ Graphite::RollupRule GraphiteRollupSortedBlockInputStream::selectPatternForPath(
UInt32 GraphiteRollupSortedBlockInputStream::selectPrecision(const Graphite::Retentions & retentions, time_t time) const
{
static_assert(std::is_signed_v<time_t>, "time_t must be signed type");
static_assert(is_signed_v<time_t>, "time_t must be signed type");
for (const auto & retention : retentions)
{

102
dbms/src/DataStreams/IBlockInputStream.cpp
View File

@ -219,11 +219,11 @@ static bool handleOverflowMode(OverflowMode mode, const String & message, int co
bool IBlockInputStream::checkTimeLimit()
{
if (limits.max_execution_time != 0
&& info.total_stopwatch.elapsed() > static_cast<UInt64>(limits.max_execution_time.totalMicroseconds()) * 1000)
if (limits.speed_limits.max_execution_time != 0
&& info.total_stopwatch.elapsed() > static_cast<UInt64>(limits.speed_limits.max_execution_time.totalMicroseconds()) * 1000)
return handleOverflowMode(limits.timeout_overflow_mode,
"Timeout exceeded: elapsed " + toString(info.total_stopwatch.elapsedSeconds())
+ " seconds, maximum: " + toString(limits.max_execution_time.totalMicroseconds() / 1000000.0),
+ " seconds, maximum: " + toString(limits.speed_limits.max_execution_time.totalMicroseconds() / 1000000.0),
ErrorCodes::TIMEOUT_EXCEEDED);
return true;
@ -252,24 +252,6 @@ void IBlockInputStream::checkQuota(Block & block)
}
}
static void limitProgressingSpeed(size_t total_progress_size, size_t max_speed_in_seconds, UInt64 total_elapsed_microseconds)
{
/// How much time to wait for the average speed to become `max_speed_in_seconds`.
UInt64 desired_microseconds = total_progress_size * 1000000 / max_speed_in_seconds;
if (desired_microseconds > total_elapsed_microseconds)
{
UInt64 sleep_microseconds = desired_microseconds - total_elapsed_microseconds;
/// Never sleep more than one second (it should be enough to limit speed for a reasonable amount, and otherwise it's too easy to make query hang).
sleep_microseconds = std::min(UInt64(1000000), sleep_microseconds);
sleepForMicroseconds(sleep_microseconds);
ProfileEvents::increment(ProfileEvents::ThrottlerSleepMicroseconds, sleep_microseconds);
}
}
void IBlockInputStream::progressImpl(const Progress & value)
{
@ -289,40 +271,11 @@ void IBlockInputStream::progressImpl(const Progress & value)
/** Check the restrictions on the amount of data to read, the speed of the query, the quota on the amount of data to read.
* NOTE: Maybe it makes sense to have them checked directly in ProcessList?
*/
if (limits.mode == LIMITS_TOTAL
&& ((limits.size_limits.max_rows && total_rows_estimate > limits.size_limits.max_rows)
|| (limits.size_limits.max_bytes && progress.read_bytes > limits.size_limits.max_bytes)))
if (limits.mode == LIMITS_TOTAL)
{
switch (limits.size_limits.overflow_mode)
{
case OverflowMode::THROW:
{
if (limits.size_limits.max_rows && total_rows_estimate > limits.size_limits.max_rows)
throw Exception("Limit for rows to read exceeded: " + toString(total_rows_estimate)
+ " rows read (or to read), maximum: " + toString(limits.size_limits.max_rows),
ErrorCodes::TOO_MANY_ROWS);
else
throw Exception("Limit for (uncompressed) bytes to read exceeded: " + toString(progress.read_bytes)
+ " bytes read, maximum: " + toString(limits.size_limits.max_bytes),
ErrorCodes::TOO_MANY_BYTES);
}
case OverflowMode::BREAK:
{
/// For `break`, we will stop only if so many rows were actually read, and not just supposed to be read.
if ((limits.size_limits.max_rows && progress.read_rows > limits.size_limits.max_rows)
|| (limits.size_limits.max_bytes && progress.read_bytes > limits.size_limits.max_bytes))
{
cancel(false);
}
break;
}
default:
throw Exception("Logical error: unknown overflow mode", ErrorCodes::LOGICAL_ERROR);
}
if (!limits.size_limits.check(total_rows_estimate, progress.read_bytes, "rows to read",
ErrorCodes::TOO_MANY_ROWS, ErrorCodes::TOO_MANY_BYTES))
cancel(false);
}
size_t total_rows = progress.total_rows_to_read;
@ -336,46 +289,7 @@ void IBlockInputStream::progressImpl(const Progress & value)
last_profile_events_update_time = total_elapsed_microseconds;
}
if ((limits.min_execution_speed || limits.max_execution_speed || limits.min_execution_speed_bytes ||
limits.max_execution_speed_bytes || (total_rows && limits.timeout_before_checking_execution_speed != 0)) &&
(static_cast<Int64>(total_elapsed_microseconds) > limits.timeout_before_checking_execution_speed.totalMicroseconds()))
{
/// Do not count sleeps in throttlers
UInt64 throttler_sleep_microseconds = CurrentThread::getProfileEvents()[ProfileEvents::ThrottlerSleepMicroseconds];
double elapsed_seconds = (throttler_sleep_microseconds > total_elapsed_microseconds)
? 0.0 : (total_elapsed_microseconds - throttler_sleep_microseconds) / 1000000.0;
if (elapsed_seconds > 0)
{
if (limits.min_execution_speed && progress.read_rows / elapsed_seconds < limits.min_execution_speed)
throw Exception("Query is executing too slow: " + toString(progress.read_rows / elapsed_seconds)
+ " rows/sec., minimum: " + toString(limits.min_execution_speed),
ErrorCodes::TOO_SLOW);
if (limits.min_execution_speed_bytes && progress.read_bytes / elapsed_seconds < limits.min_execution_speed_bytes)
throw Exception("Query is executing too slow: " + toString(progress.read_bytes / elapsed_seconds)
+ " bytes/sec., minimum: " + toString(limits.min_execution_speed_bytes),
ErrorCodes::TOO_SLOW);
/// If the predicted execution time is longer than `max_execution_time`.
if (limits.max_execution_time != 0 && total_rows && progress.read_rows)
{
double estimated_execution_time_seconds = elapsed_seconds * (static_cast<double>(total_rows) / progress.read_rows);
if (estimated_execution_time_seconds > limits.max_execution_time.totalSeconds())
throw Exception("Estimated query execution time (" + toString(estimated_execution_time_seconds) + " seconds)"
+ " is too long. Maximum: " + toString(limits.max_execution_time.totalSeconds())
+ ". Estimated rows to process: " + toString(total_rows),
ErrorCodes::TOO_SLOW);
}
if (limits.max_execution_speed && progress.read_rows / elapsed_seconds >= limits.max_execution_speed)
limitProgressingSpeed(progress.read_rows, limits.max_execution_speed, total_elapsed_microseconds);
if (limits.max_execution_speed_bytes && progress.read_bytes / elapsed_seconds >= limits.max_execution_speed_bytes)
limitProgressingSpeed(progress.read_bytes, limits.max_execution_speed_bytes, total_elapsed_microseconds);
}
}
limits.speed_limits.throttle(progress.read_rows, progress.read_bytes, total_rows, total_elapsed_microseconds);
if (quota != nullptr && limits.mode == LIMITS_TOTAL)
{

22
dbms/src/DataStreams/IBlockInputStream.h
View File

@ -5,6 +5,7 @@
#include <DataStreams/BlockStreamProfileInfo.h>
#include <DataStreams/IBlockStream_fwd.h>
#include <DataStreams/SizeLimits.h>
#include <DataStreams/ExecutionSpeedLimits.h>
#include <IO/Progress.h>
#include <Storages/TableStructureLockHolder.h>
#include <Common/TypePromotion.h>
@ -137,7 +138,7 @@ public:
* The function takes the number of rows in the last block, the number of bytes in the last block.
* Note that the callback can be called from different threads.
*/
void setProgressCallback(const ProgressCallback & callback);
virtual void setProgressCallback(const ProgressCallback & callback);
/** In this method:
@ -162,11 +163,11 @@ public:
* Based on this information, the quota and some restrictions will be checked.
* This information will also be available in the SHOW PROCESSLIST request.
*/
void setProcessListElement(QueryStatus * elem);
virtual void setProcessListElement(QueryStatus * elem);
/** Set the approximate total number of rows to read.
*/
void addTotalRowsApprox(size_t value) { total_rows_approx += value; }
virtual void addTotalRowsApprox(size_t value) { total_rows_approx += value; }
/** Ask to abort the receipt of data as soon as possible.
@ -201,20 +202,13 @@ public:
SizeLimits size_limits;
Poco::Timespan max_execution_time = 0;
OverflowMode timeout_overflow_mode = OverflowMode::THROW;
ExecutionSpeedLimits speed_limits;
/// in rows per second
size_t min_execution_speed = 0;
size_t max_execution_speed = 0;
size_t min_execution_speed_bytes = 0;
size_t max_execution_speed_bytes = 0;
/// Verify that the speed is not too low after the specified time has elapsed.
Poco::Timespan timeout_before_checking_execution_speed = 0;
OverflowMode timeout_overflow_mode = OverflowMode::THROW;
};
/** Set limitations that checked on each block. */
void setLimits(const LocalLimits & limits_)
virtual void setLimits(const LocalLimits & limits_)
{
limits = limits_;
}
@ -227,7 +221,7 @@ public:
/** Set the quota. If you set a quota on the amount of raw data,
* then you should also set mode = LIMITS_TOTAL to LocalLimits with setLimits.
*/
void setQuota(QuotaForIntervals & quota_)
virtual void setQuota(QuotaForIntervals & quota_)
{
quota = &quota_;
}

11
dbms/src/DataStreams/SizeLimits.cpp
View File

@ -7,17 +7,17 @@
namespace DB
{
bool SizeLimits::check(UInt64 rows, UInt64 bytes, const char * what, int exception_code) const
bool SizeLimits::check(UInt64 rows, UInt64 bytes, const char * what, int too_many_rows_exception_code, int too_many_bytes_exception_code) const
{
if (overflow_mode == OverflowMode::THROW)
{
if (max_rows && rows > max_rows)
throw Exception("Limit for " + std::string(what) + " exceeded, max rows: " + formatReadableQuantity(max_rows)
+ ", current rows: " + formatReadableQuantity(rows), exception_code);
+ ", current rows: " + formatReadableQuantity(rows), too_many_rows_exception_code);
if (max_bytes && bytes > max_bytes)
throw Exception("Limit for " + std::string(what) + " exceeded, max bytes: " + formatReadableSizeWithBinarySuffix(max_bytes)
+ ", current bytes: " + formatReadableSizeWithBinarySuffix(bytes), exception_code);
+ ", current bytes: " + formatReadableSizeWithBinarySuffix(bytes), too_many_bytes_exception_code);
return true;
}
@ -34,4 +34,9 @@ bool SizeLimits::softCheck(UInt64 rows, UInt64 bytes) const
return true;
}
bool SizeLimits::check(UInt64 rows, UInt64 bytes, const char * what, int exception_code) const
{
return check(rows, bytes, what, exception_code, exception_code);
}
}

1
dbms/src/DataStreams/SizeLimits.h
View File

@ -31,6 +31,7 @@ struct SizeLimits
: max_rows(max_rows_), max_bytes(max_bytes_), overflow_mode(overflow_mode_) {}
/// Check limits. If exceeded, return false or throw an exception, depending on overflow_mode.
bool check(UInt64 rows, UInt64 bytes, const char * what, int too_many_rows_exception_code, int too_many_bytes_exception_code) const;
bool check(UInt64 rows, UInt64 bytes, const char * what, int exception_code) const;
/// Check limits. No exceptions.

8
dbms/src/DataTypes/DataTypeNumberBase.cpp
View File

@ -26,7 +26,7 @@ void DataTypeNumberBase<T>::deserializeText(IColumn & column, ReadBuffer & istr,
{
T x;
if constexpr (std::is_integral_v<T> && std::is_arithmetic_v<T>)
if constexpr (is_integral_v<T> && is_arithmetic_v<T>)
readIntTextUnsafe(x, istr);
else
readText(x, istr);
@ -68,7 +68,7 @@ void DataTypeNumberBase<T>::serializeTextJSON(const IColumn & column, size_t row
auto x = assert_cast<const ColumnVector<T> &>(column).getData()[row_num];
bool is_finite = isFinite(x);
const bool need_quote = (std::is_integral_v<T> && (sizeof(T) == 8) && settings.json.quote_64bit_integers)
const bool need_quote = (is_integral_v<T> && (sizeof(T) == 8) && settings.json.quote_64bit_integers)
|| (settings.json.quote_denormals && !is_finite);
if (need_quote)
@ -242,13 +242,13 @@ MutableColumnPtr DataTypeNumberBase<T>::createColumn() const
template <typename T>
bool DataTypeNumberBase<T>::isValueRepresentedByInteger() const
{
return std::is_integral_v<T>;
return is_integral_v<T>;
}
template <typename T>
bool DataTypeNumberBase<T>::isValueRepresentedByUnsignedInteger() const
{
return std::is_integral_v<T> && std::is_unsigned_v<T>;
return is_integral_v<T> && is_unsigned_v<T>;
}

3
dbms/src/DataTypes/DataTypeSet.h
View File

@ -17,6 +17,9 @@ public:
TypeIndex getTypeId() const override { return TypeIndex::Set; }
bool equals(const IDataType & rhs) const override { return typeid(rhs) == typeid(*this); }
bool isParametric() const override { return true; }
// Used only for debugging, making it DUMPABLE
Field getDefault() const override { return Tuple(); }
};
}

7
dbms/src/DataTypes/DataTypeTuple.cpp
View File

@ -188,6 +188,13 @@ void DataTypeTuple::deserializeText(IColumn & column, ReadBuffer & istr, const F
}
});
// Special format for one element tuple (1,)
if (1 == elems.size())
{
skipWhitespaceIfAny(istr);
// Allow both (1) and (1,)
checkChar(',', istr);
}
skipWhitespaceIfAny(istr);
assertChar(')', istr);
}

30
dbms/src/DataTypes/NumberTraits.h
View File

@ -66,7 +66,7 @@ template <> struct Construct<true, true, 8> { using Type = Float64; };
template <typename A, typename B> struct ResultOfAdditionMultiplication
{
using Type = typename Construct<
std::is_signed_v<A> || std::is_signed_v<B>,
is_signed_v<A> || is_signed_v<B>,
std::is_floating_point_v<A> || std::is_floating_point_v<B>,
nextSize(max(sizeof(A), sizeof(B)))>::Type;
};
@ -91,7 +91,7 @@ template <typename A, typename B> struct ResultOfFloatingPointDivision
template <typename A, typename B> struct ResultOfIntegerDivision
{
using Type = typename Construct<
std::is_signed_v<A> || std::is_signed_v<B>,
is_signed_v<A> || is_signed_v<B>,
false,
sizeof(A)>::Type;
};
@ -101,7 +101,7 @@ template <typename A, typename B> struct ResultOfIntegerDivision
template <typename A, typename B> struct ResultOfModulo
{
using Type = typename Construct<
std::is_signed_v<A> || std::is_signed_v<B>,
is_signed_v<A> || is_signed_v<B>,
false,
sizeof(B)>::Type;
};
@ -111,7 +111,7 @@ template <typename A> struct ResultOfNegate
using Type = typename Construct<
true,
std::is_floating_point_v<A>,
std::is_signed_v<A> ? sizeof(A) : nextSize(sizeof(A))>::Type;
is_signed_v<A> ? sizeof(A) : nextSize(sizeof(A))>::Type;
};
template <typename A> struct ResultOfAbs
@ -127,7 +127,7 @@ template <typename A> struct ResultOfAbs
template <typename A, typename B> struct ResultOfBit
{
using Type = typename Construct<
std::is_signed_v<A> || std::is_signed_v<B>,
is_signed_v<A> || is_signed_v<B>,
false,
std::is_floating_point_v<A> || std::is_floating_point_v<B> ? 8 : max(sizeof(A), sizeof(B))>::Type;
};
@ -135,7 +135,7 @@ template <typename A, typename B> struct ResultOfBit
template <typename A> struct ResultOfBitNot
{
using Type = typename Construct<
std::is_signed_v<A>,
is_signed_v<A>,
false,
sizeof(A)>::Type;
};
@ -156,13 +156,13 @@ template <typename A, typename B>
struct ResultOfIf
{
static constexpr bool has_float = std::is_floating_point_v<A> || std::is_floating_point_v<B>;
static constexpr bool has_integer = std::is_integral_v<A> || std::is_integral_v<B>;
static constexpr bool has_signed = std::is_signed_v<A> || std::is_signed_v<B>;
static constexpr bool has_unsigned = !std::is_signed_v<A> || !std::is_signed_v<B>;
static constexpr bool has_integer = is_integral_v<A> || is_integral_v<B>;
static constexpr bool has_signed = is_signed_v<A> || is_signed_v<B>;
static constexpr bool has_unsigned = !is_signed_v<A> || !is_signed_v<B>;
static constexpr size_t max_size_of_unsigned_integer = max(std::is_signed_v<A> ? 0 : sizeof(A), std::is_signed_v<B> ? 0 : sizeof(B));
static constexpr size_t max_size_of_signed_integer = max(std::is_signed_v<A> ? sizeof(A) : 0, std::is_signed_v<B> ? sizeof(B) : 0);
static constexpr size_t max_size_of_integer = max(std::is_integral_v<A> ? sizeof(A) : 0, std::is_integral_v<B> ? sizeof(B) : 0);
static constexpr size_t max_size_of_unsigned_integer = max(is_signed_v<A> ? 0 : sizeof(A), is_signed_v<B> ? 0 : sizeof(B));
static constexpr size_t max_size_of_signed_integer = max(is_signed_v<A> ? sizeof(A) : 0, is_signed_v<B> ? sizeof(B) : 0);
static constexpr size_t max_size_of_integer = max(is_integral_v<A> ? sizeof(A) : 0, is_integral_v<B> ? sizeof(B) : 0);
static constexpr size_t max_size_of_float = max(std::is_floating_point_v<A> ? sizeof(A) : 0, std::is_floating_point_v<B> ? sizeof(B) : 0);
using ConstructedType = typename Construct<has_signed, has_float,
@ -181,7 +181,7 @@ struct ResultOfIf
template <typename A> struct ToInteger
{
using Type = typename Construct<
std::is_signed_v<A>,
is_signed_v<A>,
false,
std::is_floating_point_v<A> ? 8 : sizeof(A)>::Type;
};
@ -191,9 +191,9 @@ template <typename A> struct ToInteger
// NOTE: This case is applied for 64-bit integers only (for backward compatibility), but could be used for any-bit integers
template <typename A, typename B>
constexpr bool LeastGreatestSpecialCase =
std::is_integral_v<A> && std::is_integral_v<B>
is_integral_v<A> && is_integral_v<B>
&& (8 == sizeof(A) && sizeof(A) == sizeof(B))
&& (std::is_signed_v<A> ^ std::is_signed_v<B>);
&& (is_signed_v<A> ^ is_signed_v<B>);
template <typename A, typename B>
using ResultOfLeast = std::conditional_t<LeastGreatestSpecialCase<A, B>,

4
dbms/src/Formats/FormatFactory.cpp
View File

@ -312,9 +312,7 @@ void registerOutputFormatProcessorXML(FormatFactory & factory);
void registerOutputFormatProcessorODBCDriver(FormatFactory & factory);
void registerOutputFormatProcessorODBCDriver2(FormatFactory & factory);
void registerOutputFormatProcessorNull(FormatFactory & factory);
#if USE_SSL
void registerOutputFormatProcessorMySQLWrite(FormatFactory & factory);
#endif
/// Input only formats.
void registerInputFormatProcessorCapnProto(FormatFactory & factory);
@ -366,9 +364,7 @@ FormatFactory::FormatFactory()
registerOutputFormatProcessorODBCDriver(*this);
registerOutputFormatProcessorODBCDriver2(*this);
registerOutputFormatProcessorNull(*this);
#if USE_SSL
registerOutputFormatProcessorMySQLWrite(*this);
#endif
}
FormatFactory & FormatFactory::instance()

2
dbms/src/Formats/ProtobufReader.cpp
View File

@ -759,7 +759,7 @@ private:
template<typename EnumType>
bool readEnum(EnumType & value)
{
if constexpr (!std::is_integral_v<FromType>)
if constexpr (!is_integral_v<FromType>)
cannotConvertType("Enum"); // It's not correct to convert floating point to enum.
FromType number;
if (!readField(number))

2
dbms/src/Formats/ProtobufWriter.cpp
View File

@ -526,7 +526,7 @@ public:
void writeEnum16(Int16 value) override
{
if constexpr (!std::is_integral_v<ToType>)
if constexpr (!is_integral_v<ToType>)
cannotConvertType("Enum"); // It's not correct to convert enum to floating point.
castNumericAndWriteField(value);
}

4
dbms/src/Functions/FunctionBinaryArithmetic.h
View File

@ -438,7 +438,7 @@ public:
};
template <template <typename, typename> class Op, typename Name, bool CanBeExecutedOnDefaultArguments = true>
template <template <typename, typename> class Op, typename Name, bool valid_on_default_arguments = true>
class FunctionBinaryArithmetic : public IFunction
{
const Context & context;
@ -944,7 +944,7 @@ public:
}
#endif
bool canBeExecutedOnDefaultArguments() const override { return CanBeExecutedOnDefaultArguments; }
bool canBeExecutedOnDefaultArguments() const override { return valid_on_default_arguments; }
};
}

2
dbms/src/Functions/FunctionHelpers.cpp
View File

@ -4,8 +4,10 @@
#include <Columns/ColumnString.h>
#include <Columns/ColumnFixedString.h>
#include <Columns/ColumnNullable.h>
#include <Columns/ColumnLowCardinality.h>
#include <Common/assert_cast.h>
#include <DataTypes/DataTypeNullable.h>
#include <DataTypes/DataTypeLowCardinality.h>
#include <IO/WriteHelpers.h>

2
dbms/src/Functions/FunctionUnaryArithmetic.h
View File

@ -187,7 +187,7 @@ public:
{
auto & b = static_cast<llvm::IRBuilder<> &>(builder);
auto * v = nativeCast(b, types[0], values[0](), std::make_shared<DataTypeNumber<T1>>());
result = Op<T0>::compile(b, v, std::is_signed_v<T1>);
result = Op<T0>::compile(b, v, is_signed_v<T1>);
return true;
}
return false;

1
dbms/src/Functions/FunctionsBitmap.cpp
View File

@ -14,6 +14,7 @@ void registerFunctionsBitmap(FunctionFactory & factory)
factory.registerFunction<FunctionBitmapToArray>();
factory.registerFunction<FunctionBitmapSubsetInRange>();
factory.registerFunction<FunctionBitmapSubsetLimit>();
factory.registerFunction<FunctionBitmapTransform>();
factory.registerFunction<FunctionBitmapSelfCardinality>();
factory.registerFunction<FunctionBitmapMin>();

300
dbms/src/Functions/FunctionsBitmap.h
View File

@ -33,11 +33,20 @@ namespace ErrorCodes
* Convert bitmap to integer array:
* bitmapToArray: bitmap -> integer[]
*
* Retrun the smallest value in the set:
* bitmapMin: bitmap -> integer
*
* Retrun the greatest value in the set:
* bitmapMax: bitmap -> integer
*
* Return subset in specified range (not include the range_end):
* bitmapSubsetInRange: bitmap,integer,integer -> bitmap
*
* Return subset of the smallest `limit` values in set which is no smaller than `range_start`.
* bitmapSubsetInRange: bitmap,integer,integer -> bitmap
* bitmapSubsetLimit: bitmap,integer,integer -> bitmap
*
* Transform an array of values in a bitmap to another array of values, the result is a new bitmap.
* bitmapTransform: bitmap,integer[],integer[] -> bitmap
*
* Two bitmap and calculation:
* bitmapAnd: bitmap,bitmap -> bitmap
@ -54,12 +63,6 @@ namespace ErrorCodes
* Retrun bitmap cardinality:
* bitmapCardinality: bitmap -> integer
*
* Retrun the smallest value in the set:
* bitmapMin: bitmap -> integer
*
* Retrun the greatest value in the set:
* bitmapMax: bitmap -> integer
*
* Two bitmap and calculation, return cardinality:
* bitmapAndCardinality: bitmap,bitmap -> integer
*
@ -192,7 +195,7 @@ public:
const DataTypeAggregateFunction * bitmap_type = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type && bitmap_type->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
const DataTypePtr data_type = bitmap_type->getArgumentsDataTypes()[0];
@ -246,9 +249,9 @@ private:
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateFunctionGroupBitmapData<T> & bd1
const AggregateFunctionGroupBitmapData<T> & bitmap_data_1
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(column->getData()[i]);
UInt64 count = bd1.rbs.rb_to_array(res_data);
UInt64 count = bitmap_data_1.rbs.rb_to_array(res_data);
res_offset += count;
res_offsets.emplace_back(res_offset);
}
@ -274,7 +277,7 @@ public:
const DataTypeAggregateFunction * bitmap_type = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type && bitmap_type->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto arg_type1 = typeid_cast<const DataTypeNumber<UInt32> *>(arguments[1].get());
@ -322,7 +325,7 @@ private:
{
const IColumn * columns[3];
bool is_column_const[3];
const ColumnAggregateFunction * colAggFunc;
const ColumnAggregateFunction * col_agg_func;
const PaddedPODArray<AggregateDataPtr> * container0;
const PaddedPODArray<UInt32> * container1, * container2;
@ -332,14 +335,11 @@ private:
is_column_const[i] = isColumnConst(*columns[i]);
}
if (is_column_const[0])
{
colAggFunc = typeid_cast<const ColumnAggregateFunction*>(typeid_cast<const ColumnConst*>(columns[0])->getDataColumnPtr().get());
}
col_agg_func = typeid_cast<const ColumnAggregateFunction*>(typeid_cast<const ColumnConst*>(columns[0])->getDataColumnPtr().get());
else
{
colAggFunc = typeid_cast<const ColumnAggregateFunction*>(columns[0]);
}
container0 = &colAggFunc->getData();
col_agg_func = typeid_cast<const ColumnAggregateFunction*>(columns[0]);
container0 = &col_agg_func->getData();
if (is_column_const[1])
container1 = &typeid_cast<const ColumnUInt32*>(typeid_cast<const ColumnConst*>(columns[1])->getDataColumnPtr().get())->getData();
else
@ -349,21 +349,21 @@ private:
else
container2 = &typeid_cast<const ColumnUInt32*>(columns[2])->getData();
auto col_to = ColumnAggregateFunction::create(colAggFunc->getAggregateFunction());
auto col_to = ColumnAggregateFunction::create(col_agg_func->getAggregateFunction());
col_to->reserve(input_rows_count);
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateDataPtr dataPtr0 = is_column_const[0] ? (*container0)[0] : (*container0)[i];
const AggregateFunctionGroupBitmapData<T>& bd0
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr0);
const AggregateDataPtr data_ptr_0 = is_column_const[0] ? (*container0)[0] : (*container0)[i];
const AggregateFunctionGroupBitmapData<T> & bitmap_data_0
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(data_ptr_0);
const UInt32 range_start = is_column_const[1] ? (*container1)[0] : (*container1)[i];
const UInt32 range_end = is_column_const[2] ? (*container2)[0] : (*container2)[i];
col_to->insertDefault();
AggregateFunctionGroupBitmapData<T> & bd2
AggregateFunctionGroupBitmapData<T> & bitmap_data_2
= *reinterpret_cast<AggregateFunctionGroupBitmapData<T> *>(col_to->getData()[i]);
Impl::apply(bd0, range_start, range_end, bd2);
Impl::apply(bitmap_data_0, range_start, range_end, bitmap_data_2);
}
block.getByPosition(result).column = std::move(col_to);
}
@ -374,9 +374,9 @@ struct BitmapSubsetInRangeImpl
public:
static constexpr auto name = "bitmapSubsetInRange";
template <typename T>
static void apply(const AggregateFunctionGroupBitmapData<T> & bd0, UInt32 range_start, UInt32 range_end, AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_0, UInt32 range_start, UInt32 range_end, AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
bd0.rbs.rb_range(range_start, range_end, bd2.rbs);
bitmap_data_0.rbs.rb_range(range_start, range_end, bitmap_data_2.rbs);
}
};
@ -385,15 +385,161 @@ struct BitmapSubsetLimitImpl
public:
static constexpr auto name = "bitmapSubsetLimit";
template <typename T>
static void apply(const AggregateFunctionGroupBitmapData<T> & bd0, UInt32 range_start, UInt32 range_end, AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_0, UInt32 range_start, UInt32 range_end, AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
bd0.rbs.rb_limit(range_start, range_end, bd2.rbs);
bitmap_data_0.rbs.rb_limit(range_start, range_end, bitmap_data_2.rbs);
}
};
using FunctionBitmapSubsetInRange = FunctionBitmapSubset<BitmapSubsetInRangeImpl>;
using FunctionBitmapSubsetLimit = FunctionBitmapSubset<BitmapSubsetLimitImpl>;
class FunctionBitmapTransform : public IFunction
{
public:
static constexpr auto name = "bitmapTransform";
static FunctionPtr create(const Context &) { return std::make_shared<FunctionBitmapTransform>(); }
String getName() const override { return name; }
bool isVariadic() const override { return false; }
size_t getNumberOfArguments() const override { return 3; }
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
const DataTypeAggregateFunction * bitmap_type = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type && bitmap_type->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
for (size_t i = 0; i < 2; ++i)
{
auto array_type = typeid_cast<const DataTypeArray *>(arguments[i + 1].get());
String msg(i == 0 ? "Second" : "Third");
msg += " argument for function " + getName() + " must be an UInt32 array but it has type " + arguments[i + 1]->getName() + ".";
if (!array_type)
throw Exception(msg, ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto nested_type = array_type->getNestedType();
WhichDataType which(nested_type);
if (!which.isUInt32())
throw Exception(msg, ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
return arguments[0];
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
{
const IDataType * from_type = block.getByPosition(arguments[0]).type.get();
const DataTypeAggregateFunction * aggr_type = typeid_cast<const DataTypeAggregateFunction *>(from_type);
WhichDataType which(aggr_type->getArgumentsDataTypes()[0]);
if (which.isUInt8())
executeIntType<UInt8>(block, arguments, result, input_rows_count);
else if (which.isUInt16())
executeIntType<UInt16>(block, arguments, result, input_rows_count);
else if (which.isUInt32())
executeIntType<UInt32>(block, arguments, result, input_rows_count);
else if (which.isUInt64())
executeIntType<UInt64>(block, arguments, result, input_rows_count);
else
throw Exception(
"Unexpected type " + from_type->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
private:
using ToType = UInt64;
template <typename T>
void executeIntType(
Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) const
{
const IColumn * columns[3];
bool is_column_const[3];
const ColumnAggregateFunction * col_agg_func;
const PaddedPODArray<AggregateDataPtr> * container0;
const ColumnArray * array;
for (size_t i = 0; i < 3; ++i)
{
columns[i] = block.getByPosition(arguments[i]).column.get();
is_column_const[i] = isColumnConst(*columns[i]);
}
if (is_column_const[0])
{
col_agg_func = typeid_cast<const ColumnAggregateFunction*>(typeid_cast<const ColumnConst*>(columns[0])->getDataColumnPtr().get());
}
else
{
col_agg_func = typeid_cast<const ColumnAggregateFunction*>(columns[0]);
}
container0 = &col_agg_func->getData();
if (is_column_const[1])
array = typeid_cast<const ColumnArray*>(typeid_cast<const ColumnConst*>(columns[1])->getDataColumnPtr().get());
else
{
array = typeid_cast<const ColumnArray *>(block.getByPosition(arguments[1]).column.get());
}
const ColumnArray::Offsets & from_offsets = array->getOffsets();
const ColumnVector<UInt32>::Container & from_container = typeid_cast<const ColumnVector<UInt32> *>(&array->getData())->getData();
if (is_column_const[2])
array = typeid_cast<const ColumnArray*>(typeid_cast<const ColumnConst*>(columns[2])->getDataColumnPtr().get());
else
array = typeid_cast<const ColumnArray *>(block.getByPosition(arguments[2]).column.get());
const ColumnArray::Offsets & to_offsets = array->getOffsets();
const ColumnVector<UInt32>::Container & to_container = typeid_cast<const ColumnVector<UInt32> *>(&array->getData())->getData();
auto col_to = ColumnAggregateFunction::create(col_agg_func->getAggregateFunction());
col_to->reserve(input_rows_count);
size_t from_start;
size_t from_end;
size_t to_start;
size_t to_end;
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateDataPtr data_ptr_0 = is_column_const[0] ? (*container0)[0] : (*container0)[i];
const AggregateFunctionGroupBitmapData<T> & bitmap_data_0
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(data_ptr_0);
if (is_column_const[1])
{
from_start = 0;
from_end = from_container.size();
}
else
{
from_start = i == 0 ? 0 : from_offsets[i - 1];
from_end = from_offsets[i];
}
if (is_column_const[2])
{
to_start = 0;
to_end = to_container.size();
}
else
{
to_start = i == 0 ? 0 : to_offsets[i - 1];
to_end = to_offsets[i];
}
if (from_end - from_start != to_end - to_start)
throw Exception("From array size and to array size mismatch", ErrorCodes::LOGICAL_ERROR);
col_to->insertDefault();
AggregateFunctionGroupBitmapData<T> & bitmap_data_2
= *reinterpret_cast<AggregateFunctionGroupBitmapData<T> *>(col_to->getData()[i]);
bitmap_data_2.rbs.merge(bitmap_data_0.rbs);
bitmap_data_2.rbs.rb_replace(&from_container[from_start], &to_container[to_start], from_end - from_start);
}
block.getByPosition(result).column = std::move(col_to);
}
};
template <typename Impl>
class FunctionBitmapSelfCardinalityImpl : public IFunction
{
@ -413,7 +559,7 @@ public:
auto bitmap_type = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type && bitmap_type->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
return std::make_shared<DataTypeNumber<ToType>>();
}
@ -454,9 +600,9 @@ private:
= typeid_cast<const ColumnAggregateFunction *>(block.getByPosition(arguments[0]).column.get());
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateFunctionGroupBitmapData<T> & bd
const AggregateFunctionGroupBitmapData<T> & bitmap_data
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(column->getData()[i]);
vec_to[i] = Impl::apply(bd);
vec_to[i] = Impl::apply(bitmap_data);
}
}
};
@ -466,9 +612,9 @@ struct BitmapCardinalityImpl
public:
static constexpr auto name = "bitmapCardinality";
template <typename T>
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data)
{
return bd.rbs.size();
return bitmap_data.rbs.size();
}
};
@ -477,9 +623,9 @@ struct BitmapMinImpl
public:
static constexpr auto name = "bitmapMin";
template <typename T>
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data)
{
return bd.rbs.rb_min();
return bitmap_data.rbs.rb_min();
}
};
@ -488,9 +634,9 @@ struct BitmapMaxImpl
public:
static constexpr auto name = "bitmapMax";
template <typename T>
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data)
{
return bd.rbs.rb_max();
return bitmap_data.rbs.rb_max();
}
};
@ -498,10 +644,10 @@ template <typename T>
struct BitmapAndCardinalityImpl
{
using ReturnType = UInt64;
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
// roaring_bitmap_and_cardinality( rb1, rb2 );
return bd1.rbs.rb_and_cardinality(bd2.rbs);
return bitmap_data_1.rbs.rb_and_cardinality(bitmap_data_2.rbs);
}
};
@ -510,10 +656,10 @@ template <typename T>
struct BitmapOrCardinalityImpl
{
using ReturnType = UInt64;
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
// return roaring_bitmap_or_cardinality( rb1, rb2 );
return bd1.rbs.rb_or_cardinality(bd2.rbs);
return bitmap_data_1.rbs.rb_or_cardinality(bitmap_data_2.rbs);
}
};
@ -521,10 +667,10 @@ template <typename T>
struct BitmapXorCardinalityImpl
{
using ReturnType = UInt64;
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
// return roaring_bitmap_xor_cardinality( rb1, rb2 );
return bd1.rbs.rb_xor_cardinality(bd2.rbs);
return bitmap_data_1.rbs.rb_xor_cardinality(bitmap_data_2.rbs);
}
};
@ -532,10 +678,10 @@ template <typename T>
struct BitmapAndnotCardinalityImpl
{
using ReturnType = UInt64;
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt64 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
// roaring_bitmap_andnot_cardinality( rb1, rb2 );
return bd1.rbs.rb_andnot_cardinality(bd2.rbs);
return bitmap_data_1.rbs.rb_andnot_cardinality(bitmap_data_2.rbs);
}
};
@ -543,9 +689,9 @@ template <typename T>
struct BitmapHasAllImpl
{
using ReturnType = UInt8;
static UInt8 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt8 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
return bd1.rbs.rb_is_subset(bd2.rbs);
return bitmap_data_1.rbs.rb_is_subset(bitmap_data_2.rbs);
}
};
@ -553,9 +699,9 @@ template <typename T>
struct BitmapHasAnyImpl
{
using ReturnType = UInt8;
static UInt8 apply(const AggregateFunctionGroupBitmapData<T> & bd1, const AggregateFunctionGroupBitmapData<T> & bd2)
static UInt8 apply(const AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
return bd1.rbs.rb_intersect(bd2.rbs);
return bitmap_data_1.rbs.rb_intersect(bitmap_data_2.rbs);
}
};
@ -577,7 +723,7 @@ public:
auto bitmap_type0 = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type0 && bitmap_type0->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto arg_type1 = typeid_cast<const DataTypeNumber<UInt32> *>(arguments[1].get());
if (!(arg_type1))
@ -639,11 +785,11 @@ private:
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateDataPtr dataPtr0 = is_column_const[0] ? (*container0)[0] : (*container0)[i];
const AggregateDataPtr data_ptr_0 = is_column_const[0] ? (*container0)[0] : (*container0)[i];
const UInt32 data1 = is_column_const[1] ? (*container1)[0] : (*container1)[i];
const AggregateFunctionGroupBitmapData<T>& bd0
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr0);
vec_to[i] = bd0.rbs.rb_contains(data1);
const AggregateFunctionGroupBitmapData<T> & bitmap_data_0
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(data_ptr_0);
vec_to[i] = bitmap_data_0.rbs.rb_contains(data1);
}
}
};
@ -667,13 +813,13 @@ public:
auto bitmap_type0 = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type0 && bitmap_type0->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto bitmap_type1 = typeid_cast<const DataTypeAggregateFunction *>(arguments[1].get());
if (!(bitmap_type1 && bitmap_type1->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"Second argument for function " + getName() + " must be an bitmap but it has type " + arguments[1]->getName() + ".",
"Second argument for function " + getName() + " must be a bitmap but it has type " + arguments[1]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (bitmap_type0->getArgumentsDataTypes()[0]->getTypeId() != bitmap_type1->getArgumentsDataTypes()[0]->getTypeId())
@ -736,13 +882,13 @@ private:
for (size_t i = 0; i < input_rows_count; ++i)
{
const AggregateDataPtr dataPtr0 = is_column_const[0] ? container0[0] : container0[i];
const AggregateDataPtr dataPtr1 = is_column_const[1] ? container1[0] : container1[i];
const AggregateFunctionGroupBitmapData<T> & bd1
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr0);
const AggregateFunctionGroupBitmapData<T> & bd2
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T>*>(dataPtr1);
vec_to[i] = Impl<T>::apply(bd1, bd2);
const AggregateDataPtr data_ptr_0 = is_column_const[0] ? container0[0] : container0[i];
const AggregateDataPtr data_ptr_1 = is_column_const[1] ? container1[0] : container1[i];
const AggregateFunctionGroupBitmapData<T> & bitmap_data_1
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(data_ptr_0);
const AggregateFunctionGroupBitmapData<T> & bitmap_data_2
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(data_ptr_1);
vec_to[i] = Impl<T>::apply(bitmap_data_1, bitmap_data_2);
}
}
};
@ -750,36 +896,36 @@ private:
template <typename T>
struct BitmapAndImpl
{
static void apply(AggregateFunctionGroupBitmapData<T> & toBd, const AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
toBd.rbs.rb_and(bd2.rbs);
bitmap_data_1.rbs.rb_and(bitmap_data_2.rbs);
}
};
template <typename T>
struct BitmapOrImpl
{
static void apply(AggregateFunctionGroupBitmapData<T> & toBd, const AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
toBd.rbs.rb_or(bd2.rbs);
bitmap_data_1.rbs.rb_or(bitmap_data_2.rbs);
}
};
template <typename T>
struct BitmapXorImpl
{
static void apply(AggregateFunctionGroupBitmapData<T> & toBd, const AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
toBd.rbs.rb_xor(bd2.rbs);
bitmap_data_1.rbs.rb_xor(bitmap_data_2.rbs);
}
};
template <typename T>
struct BitmapAndnotImpl
{
static void apply(AggregateFunctionGroupBitmapData<T> & toBd, const AggregateFunctionGroupBitmapData<T> & bd2)
static void apply(AggregateFunctionGroupBitmapData<T> & bitmap_data_1, const AggregateFunctionGroupBitmapData<T> & bitmap_data_2)
{
toBd.rbs.rb_andnot(bd2.rbs);
bitmap_data_1.rbs.rb_andnot(bitmap_data_2.rbs);
}
};
@ -802,13 +948,13 @@ public:
auto bitmap_type0 = typeid_cast<const DataTypeAggregateFunction *>(arguments[0].get());
if (!(bitmap_type0 && bitmap_type0->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"First argument for function " + getName() + " must be an bitmap but it has type " + arguments[0]->getName() + ".",
"First argument for function " + getName() + " must be a bitmap but it has type " + arguments[0]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto bitmap_type1 = typeid_cast<const DataTypeAggregateFunction *>(arguments[1].get());
if (!(bitmap_type1 && bitmap_type1->getFunctionName() == AggregateFunctionGroupBitmapData<UInt32>::name()))
throw Exception(
"Second argument for function " + getName() + " must be an bitmap but it has type " + arguments[1]->getName() + ".",
"Second argument for function " + getName() + " must be a bitmap but it has type " + arguments[1]->getName() + ".",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (bitmap_type0->getArgumentsDataTypes()[0]->getTypeId() != bitmap_type1->getArgumentsDataTypes()[0]->getTypeId())
@ -860,10 +1006,10 @@ private:
for (size_t i = 0; i < input_rows_count; ++i)
{
col_to->insertFrom(columns[0]->getData()[i]);
AggregateFunctionGroupBitmapData<T> & toBd = *reinterpret_cast<AggregateFunctionGroupBitmapData<T> *>(col_to->getData()[i]);
const AggregateFunctionGroupBitmapData<T> & bd2
AggregateFunctionGroupBitmapData<T> & bitmap_data_1 = *reinterpret_cast<AggregateFunctionGroupBitmapData<T> *>(col_to->getData()[i]);
const AggregateFunctionGroupBitmapData<T> & bitmap_data_2
= *reinterpret_cast<const AggregateFunctionGroupBitmapData<T> *>(columns[1]->getData()[i]);
Impl<T>::apply(toBd, bd2);
Impl<T>::apply(bitmap_data_1, bitmap_data_2);
}
block.getByPosition(result).column = std::move(col_to);
}

1
dbms/src/Functions/FunctionsCoding.cpp
View File

@ -25,6 +25,7 @@ void registerFunctionsCoding(FunctionFactory & factory)
factory.registerFunction<FunctionUUIDStringToNum>();
factory.registerFunction<FunctionHex>();
factory.registerFunction<FunctionUnhex>();
factory.registerFunction<FunctionChar>(FunctionFactory::CaseInsensitive);
factory.registerFunction<FunctionBitmaskToArray>();
factory.registerFunction<FunctionToIPv4>();
factory.registerFunction<FunctionToIPv6>();

94
dbms/src/Functions/FunctionsCoding.h
View File

@ -1276,6 +1276,100 @@ public:
}
};
class FunctionChar : public IFunction
{
public:
static constexpr auto name = "char";
static FunctionPtr create(const Context &) { return std::make_shared<FunctionChar>(); }
String getName() const override
{
return name;
}
bool isVariadic() const override { return true; }
bool isInjective(const Block &) override { return true; }
size_t getNumberOfArguments() const override { return 0; }
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
{
if (arguments.empty())
throw Exception("Number of arguments for function " + getName() + " can't be " + toString(arguments.size())
+ ", should be at least 1", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
for (const auto & arg : arguments)
{
WhichDataType which(arg);
if (!(which.isInt() || which.isUInt() || which.isFloat()))
throw Exception("Illegal type " + arg->getName() + " of argument of function " + getName()
+ ", must be Int, UInt or Float number",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
}
return std::make_shared<DataTypeString>();
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override
{
auto col_str = ColumnString::create();
ColumnString::Chars & out_vec = col_str->getChars();
ColumnString::Offsets & out_offsets = col_str->getOffsets();
const auto size_per_row = arguments.size() + 1;
out_vec.resize(size_per_row * input_rows_count);
out_offsets.resize(input_rows_count);
for (size_t row = 0; row < input_rows_count; ++row)
{
out_offsets[row] = size_per_row + out_offsets[row - 1];
out_vec[row * size_per_row + size_per_row - 1] = '\0';
}
Columns columns_holder(arguments.size());
for (size_t idx = 0; idx < arguments.size(); ++idx)
{
//partial const column
columns_holder[idx] = block.getByPosition(arguments[idx]).column->convertToFullColumnIfConst();
const IColumn * column = columns_holder[idx].get();
if (!(executeNumber<UInt8>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<UInt16>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<UInt32>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<UInt64>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Int8>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Int16>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Int32>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Int64>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Float32>(*column, out_vec, idx, input_rows_count, size_per_row)
|| executeNumber<Float64>(*column, out_vec, idx, input_rows_count, size_per_row)))
{
throw Exception{"Illegal column " + block.getByPosition(arguments[idx]).column->getName()
+ " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN};
}
}
block.getByPosition(result).column = std::move(col_str);
}
private:
template <typename T>
bool executeNumber(const IColumn & src_data, ColumnString::Chars & out_vec, const size_t & column_idx, const size_t & rows, const size_t & size_per_row)
{
const ColumnVector<T> * src_data_concrete = checkAndGetColumn<ColumnVector<T>>(&src_data);
if (!src_data_concrete)
{
return false;
}
for (size_t row = 0; row < rows; ++row)
{
out_vec[row * size_per_row + column_idx] = static_cast<char>(src_data_concrete->getInt(row));
}
return true;
}
};
class FunctionBitmaskToArray : public IFunction
{

4
dbms/src/Functions/FunctionsConversion.h
View File

@ -398,7 +398,7 @@ bool tryParseImpl(typename DataType::FieldType & x, ReadBuffer & rb, const DateL
{
if constexpr (std::is_floating_point_v<typename DataType::FieldType>)
return tryReadFloatText(x, rb);
else /*if constexpr (std::is_integral_v<typename DataType::FieldType>)*/
else /*if constexpr (is_integral_v<typename DataType::FieldType>)*/
return tryReadIntText(x, rb);
}
@ -1233,7 +1233,7 @@ struct ToNumberMonotonicity
/// Integer cases.
const bool from_is_unsigned = type.isValueRepresentedByUnsignedInteger();
const bool to_is_unsigned = std::is_unsigned_v<T>;
const bool to_is_unsigned = is_unsigned_v<T>;
const size_t size_of_from = type.getSizeOfValueInMemory();
const size_t size_of_to = sizeof(T);

2
dbms/src/Functions/FunctionsJSON.h
View File

@ -478,7 +478,7 @@ public:
if (!accurate::convertNumeric(JSONParser::getDouble(it), value))
return false;
}
else if (JSONParser::isBool(it) && std::is_integral_v<NumberType> && convert_bool_to_integer)
else if (JSONParser::isBool(it) && is_integral_v<NumberType> && convert_bool_to_integer)
value = static_cast<NumberType>(JSONParser::getBool(it));
else
return false;

38
dbms/src/Functions/FunctionsStringSimilarity.cpp
View File

@ -32,7 +32,7 @@ namespace DB
* calculation. If the right string size is big (more than 2**15 bytes),
* the strings are not similar at all and we return 1.
*/
template <size_t N, class CodePoint, bool UTF8, bool CaseInsensitive, bool Symmetric>
template <size_t N, class CodePoint, bool UTF8, bool case_insensitive, bool symmetric>
struct NgramDistanceImpl
{
using ResultType = Float32;
@ -93,7 +93,7 @@ struct NgramDistanceImpl
/// Such copying allow us to have 3 codepoints from the previous read to produce the 4-grams with them.
memcpy(code_points + (N - 1), pos, default_padding * sizeof(CodePoint));
if constexpr (CaseInsensitive)
if constexpr (case_insensitive)
{
/// We really need template lambdas with C++20 to do it inline
unrollLowering<N - 1>(code_points, std::make_index_sequence<padding_offset>());
@ -141,7 +141,7 @@ struct NgramDistanceImpl
/// And first bit of first byte if there are two bytes.
/// For ASCII it works https://catonmat.net/ascii-case-conversion-trick. For most cyrrilic letters also does.
/// For others, we don't care now. Lowering UTF is not a cheap operation.
if constexpr (CaseInsensitive)
if constexpr (case_insensitive)
{
switch (length)
{
@ -166,7 +166,7 @@ struct NgramDistanceImpl
return num;
}
template <bool SaveNgrams>
template <bool save_ngrams>
static ALWAYS_INLINE inline size_t calculateNeedleStats(
const char * data,
const size_t size,
@ -189,7 +189,7 @@ struct NgramDistanceImpl
{
++len;
UInt16 hash = hash_functor(cp + i);
if constexpr (SaveNgrams)
if constexpr (save_ngrams)
*ngram_storage++ = hash;
++ngram_stats[hash];
}
@ -199,7 +199,7 @@ struct NgramDistanceImpl
return len;
}
template <bool ReuseStats>
template <bool reuse_stats>
static ALWAYS_INLINE inline UInt64 calculateHaystackStatsAndMetric(
const char * data,
const size_t size,
@ -227,9 +227,9 @@ struct NgramDistanceImpl
/// For symmetric version we should add when we can't subtract to get symmetric difference.
if (static_cast<Int16>(ngram_stats[hash]) > 0)
--distance;
else if constexpr (Symmetric)
else if constexpr (symmetric)
++distance;
if constexpr (ReuseStats)
if constexpr (reuse_stats)
ngram_storage[ngram_cnt] = hash;
++ngram_cnt;
--ngram_stats[hash];
@ -238,7 +238,7 @@ struct NgramDistanceImpl
} while (start < end && (found = read_code_points(cp, start, end)));
/// Return the state of hash map to its initial.
if constexpr (ReuseStats)
if constexpr (reuse_stats)
{
for (size_t i = 0; i < ngram_cnt; ++i)
++ngram_stats[ngram_storage[i]];
@ -270,15 +270,15 @@ struct NgramDistanceImpl
if (data_size <= max_string_size)
{
size_t first_size = dispatchSearcher(calculateHaystackStatsAndMetric<false>, data.data(), data_size, common_stats, distance, nullptr);
/// For !Symmetric version we should not use first_size.
if constexpr (Symmetric)
/// For !symmetric version we should not use first_size.
if constexpr (symmetric)
res = distance * 1.f / std::max(first_size + second_size, size_t(1));
else
res = 1.f - distance * 1.f / std::max(second_size, size_t(1));
}
else
{
if constexpr (Symmetric)
if constexpr (symmetric)
res = 1.f;
else
res = 0.f;
@ -338,8 +338,8 @@ struct NgramDistanceImpl
/// For now, common stats is a zero array.
/// For !Symmetric version we should not use haystack_stats_size.
if constexpr (Symmetric)
/// For !symmetric version we should not use haystack_stats_size.
if constexpr (symmetric)
res[i] = distance * 1.f / std::max(haystack_stats_size + needle_stats_size, size_t(1));
else
res[i] = 1.f - distance * 1.f / std::max(needle_stats_size, size_t(1));
@ -348,7 +348,7 @@ struct NgramDistanceImpl
{
/// Strings are too big, we are assuming they are not the same. This is done because of limiting number
/// of bigrams added and not allocating too much memory.
if constexpr (Symmetric)
if constexpr (symmetric)
res[i] = 1.f;
else
res[i] = 0.f;
@ -366,7 +366,7 @@ struct NgramDistanceImpl
PaddedPODArray<Float32> & res)
{
/// For symmetric version it is better to use vector_constant
if constexpr (Symmetric)
if constexpr (symmetric)
{
vector_constant(needle_data, needle_offsets, std::move(haystack), res);
}
@ -457,8 +457,8 @@ struct NgramDistanceImpl
haystack_size, common_stats,
distance,
ngram_storage.get());
/// For !Symmetric version we should not use haystack_stats_size.
if constexpr (Symmetric)
/// For !symmetric version we should not use haystack_stats_size.
if constexpr (symmetric)
res[i] = distance * 1.f / std::max(haystack_stats_size + needle_stats_size, size_t(1));
else
res[i] = 1.f - distance * 1.f / std::max(needle_stats_size, size_t(1));
@ -466,7 +466,7 @@ struct NgramDistanceImpl
else
{
/// if the strings are too big, we say they are completely not the same
if constexpr (Symmetric)
if constexpr (symmetric)
res[i] = 1.f;
else
res[i] = 0.f;

8
dbms/src/Functions/Regexps.h
View File

@ -114,7 +114,7 @@ namespace MultiRegexps
std::map<std::pair<std::vector<String>, std::optional<UInt32>>, Regexps> storage;
};
template <bool SaveIndices, bool CompileForEditDistance>
template <bool save_indices, bool CompileForEditDistance>
inline Regexps constructRegexps(const std::vector<String> & str_patterns, std::optional<UInt32> edit_distance)
{
(void)edit_distance;
@ -165,7 +165,7 @@ namespace MultiRegexps
std::unique_ptr<unsigned int[]> ids;
/// We mark the patterns to provide the callback results.
if constexpr (SaveIndices)
if constexpr (save_indices)
{
ids.reset(new unsigned int[patterns.size()]);
for (size_t i = 0; i < patterns.size(); ++i)
@ -226,7 +226,7 @@ namespace MultiRegexps
/// Also, we use templates here because each instantiation of function
/// template has its own copy of local static variables which must not be the same
/// for different hyperscan compilations.
template <bool SaveIndices, bool CompileForEditDistance>
template <bool save_indices, bool CompileForEditDistance>
inline Regexps * get(const std::vector<StringRef> & patterns, std::optional<UInt32> edit_distance)
{
/// C++11 has thread-safe function-local statics on most modern compilers.
@ -247,7 +247,7 @@ namespace MultiRegexps
it = known_regexps.storage
.emplace(
std::pair{str_patterns, edit_distance},
constructRegexps<SaveIndices, CompileForEditDistance>(str_patterns, edit_distance))
constructRegexps<save_indices, CompileForEditDistance>(str_patterns, edit_distance))
.first;
/// If found, unlock and return the database.
lock.unlock();

4
dbms/src/Functions/abs.cpp
View File

@ -15,9 +15,9 @@ struct AbsImpl
{
if constexpr (IsDecimalNumber<A>)
return a < 0 ? A(-a) : a;
else if constexpr (std::is_integral_v<A> && std::is_signed_v<A>)
else if constexpr (is_integral_v<A> && is_signed_v<A>)
return a < 0 ? static_cast<ResultType>(~a) + 1 : a;
else if constexpr (std::is_integral_v<A> && std::is_unsigned_v<A>)
else if constexpr (is_integral_v<A> && is_unsigned_v<A>)
return static_cast<ResultType>(a);
else if constexpr (std::is_floating_point_v<A>)
return static_cast<ResultType>(std::abs(a));

96
dbms/src/Functions/array/arrayCompact.cpp Normal file
View File

@ -0,0 +1,96 @@
#include <DataTypes/DataTypesNumber.h>
#include <Columns/ColumnsNumber.h>
#include "FunctionArrayMapped.h"
#include <Functions/FunctionFactory.h>
namespace DB
{
/// arrayCompact(['a', 'a', 'b', 'b', 'a']) = ['a', 'b', 'a'] - compact arrays
namespace ErrorCodes
{
extern const int ILLEGAL_COLUMN;
}
struct ArrayCompactImpl
{
static bool useDefaultImplementationForConstants() { return true; }
static bool needBoolean() { return false; }
static bool needExpression() { return false; }
static bool needOneArray() { return false; }
static DataTypePtr getReturnType(const DataTypePtr & nested_type, const DataTypePtr & /*nested_type*/)
{
return std::make_shared<DataTypeArray>(nested_type);
}
template <typename T>
static bool executeType(const ColumnPtr & mapped, const ColumnArray & array, ColumnPtr & res_ptr)
{
const ColumnVector<T> * column = checkAndGetColumn<ColumnVector<T>>(&*mapped);
if (!column)
return false;
const IColumn::Offsets & offsets = array.getOffsets();
const typename ColumnVector<T>::Container & data = column->getData();
auto column_data = ColumnVector<T>::create(data.size());
typename ColumnVector<T>::Container & res_values = column_data->getData();
auto column_offsets = ColumnArray::ColumnOffsets::create(offsets.size());
IColumn::Offsets & res_offsets = column_offsets->getData();
size_t res_pos = 0;
size_t pos = 0;
for (size_t i = 0; i < offsets.size(); ++i)
{
if (pos < offsets[i])
{
res_values[res_pos] = data[pos];
for (++pos, ++res_pos; pos < offsets[i]; ++pos)
{
if (data[pos] != data[pos - 1])
{
res_values[res_pos++] = data[pos];
}
}
}
res_offsets[i] = res_pos;
}
for (size_t i = 0; i < data.size() - res_pos; ++i)
{
res_values.pop_back();
}
res_ptr = ColumnArray::create(std::move(column_data), std::move(column_offsets));
return true;
}
static ColumnPtr execute(const ColumnArray & array, ColumnPtr mapped)
{
ColumnPtr res;
if (executeType< UInt8 >(mapped, array, res) ||
executeType< UInt16>(mapped, array, res) ||
executeType< UInt32>(mapped, array, res) ||
executeType< UInt64>(mapped, array, res) ||
executeType< Int8 >(mapped, array, res) ||
executeType< Int16 >(mapped, array, res) ||
executeType< Int32 >(mapped, array, res) ||
executeType< Int64 >(mapped, array, res) ||
executeType<Float32>(mapped, array, res) ||
executeType<Float64>(mapped, array, res))
return res;
else
throw Exception("Unexpected column for arrayCompact: " + mapped->getName(), ErrorCodes::ILLEGAL_COLUMN);
}
};
struct NameArrayCompact { static constexpr auto name = "arrayCompact"; };
using FunctionArrayCompact = FunctionArrayMapped<ArrayCompactImpl, NameArrayCompact>;
void registerFunctionArrayCompact(FunctionFactory & factory)
{
factory.registerFunction<FunctionArrayCompact>();
}
}

126
dbms/src/Functions/array/arrayFill.cpp Normal file
View File

@ -0,0 +1,126 @@
#include <DataTypes/DataTypesNumber.h>
#include <Columns/ColumnsNumber.h>
#include "FunctionArrayMapped.h"
#include <Functions/FunctionFactory.h>
namespace DB
{
template <bool reverse>
struct ArrayFillImpl
{
static bool needBoolean() { return true; }
static bool needExpression() { return true; }
static bool needOneArray() { return false; }
static DataTypePtr getReturnType(const DataTypePtr & /*expression_return*/, const DataTypePtr & array_element)
{
return std::make_shared<DataTypeArray>(array_element);
}
static ColumnPtr execute(const ColumnArray & array, ColumnPtr mapped)
{
const ColumnUInt8 * column_fill = typeid_cast<const ColumnUInt8 *>(&*mapped);
const IColumn & in_data = array.getData();
const IColumn::Offsets & in_offsets = array.getOffsets();
auto column_data = in_data.cloneEmpty();
IColumn & out_data = *column_data.get();
if (column_fill)
{
const IColumn::Filter & fill = column_fill->getData();
size_t array_begin = 0;
size_t array_end = 0;
size_t begin = 0;
size_t end = 0;
out_data.reserve(in_data.size());
for (size_t i = 0; i < in_offsets.size(); ++i)
{
array_end = in_offsets[i] - 1;
for (; end <= array_end; ++end)
{
if (end == array_end || fill[end + 1] != fill[begin])
{
if (fill[begin])
out_data.insertRangeFrom(in_data, begin, end + 1 - begin);
else
{
if constexpr (reverse)
{
if (end == array_end)
out_data.insertManyFrom(in_data, array_end, end + 1 - begin);
else
out_data.insertManyFrom(in_data, end + 1, end + 1 - begin);
}
else
{
if (begin == array_begin)
out_data.insertManyFrom(in_data, array_begin, end + 1 - begin);
else
out_data.insertManyFrom(in_data, begin - 1, end + 1 - begin);
}
}
begin = end + 1;
}
}
array_begin = array_end + 1;
}
}
else
{
auto column_fill_const = checkAndGetColumnConst<ColumnUInt8>(&*mapped);
if (!column_fill_const)
throw Exception("Unexpected type of cut column", ErrorCodes::ILLEGAL_COLUMN);
if (column_fill_const->getValue<UInt8>())
return ColumnArray::create(
array.getDataPtr(),
array.getOffsetsPtr()
);
size_t array_begin = 0;
size_t array_end = 0;
out_data.reserve(in_data.size());
for (size_t i = 0; i < in_offsets.size(); ++i)
{
array_end = in_offsets[i] - 1;
if constexpr (reverse)
out_data.insertManyFrom(in_data, array_end, array_end + 1 - array_begin);
else
out_data.insertManyFrom(in_data, array_begin, array_end + 1 - array_begin);
array_begin = array_end + 1;
}
}
return ColumnArray::create(
std::move(column_data),
array.getOffsetsPtr()
);
}
};
struct NameArrayFill { static constexpr auto name = "arrayFill"; };
struct NameArrayReverseFill { static constexpr auto name = "arrayReverseFill"; };
using FunctionArrayFill = FunctionArrayMapped<ArrayFillImpl<false>, NameArrayFill>;
using FunctionArrayReverseFill = FunctionArrayMapped<ArrayFillImpl<true>, NameArrayReverseFill>;
void registerFunctionsArrayFill(FunctionFactory & factory)
{
factory.registerFunction<FunctionArrayFill>();
factory.registerFunction<FunctionArrayReverseFill>();
}
}

12
dbms/src/Functions/array/arraySplit.cpp
View File

@ -7,7 +7,7 @@
namespace DB
{
template <bool Reverse>
template <bool reverse>
struct ArraySplitImpl
{
static bool needBoolean() { return true; }
@ -42,13 +42,13 @@ struct ArraySplitImpl
for (size_t i = 0; i < in_offsets.size(); ++i)
{
pos += !Reverse;
for (; pos < in_offsets[i] - Reverse; ++pos)
pos += !reverse;
for (; pos < in_offsets[i] - reverse; ++pos)
{
if (cut[pos])
out_offsets_2.push_back(pos + Reverse);
out_offsets_2.push_back(pos + reverse);
}
pos += Reverse;
pos += reverse;
out_offsets_2.push_back(pos);
out_offsets_1.push_back(out_offsets_2.size());
@ -99,7 +99,7 @@ struct NameArrayReverseSplit { static constexpr auto name = "arrayReverseSplit";
using FunctionArraySplit = FunctionArrayMapped<ArraySplitImpl<false>, NameArraySplit>;
using FunctionArrayReverseSplit = FunctionArrayMapped<ArraySplitImpl<true>, NameArrayReverseSplit>;
void registerFunctionArraySplit(FunctionFactory & factory)
void registerFunctionsArraySplit(FunctionFactory & factory)
{
factory.registerFunction<FunctionArraySplit>();
factory.registerFunction<FunctionArrayReverseSplit>();

2
dbms/src/Functions/bitWrapperFunc.cpp
View File

@ -19,7 +19,7 @@ namespace DB
static inline ResultType NO_SANITIZE_UNDEFINED apply(A a)
{
if constexpr (!std::is_integral_v<A>)
if constexpr (!is_integral_v<A>)
throw DB::Exception("It's a bug! Only integer types are supported by __bitWrapperFunc.", ErrorCodes::BAD_CAST);
return a == 0 ? static_cast<ResultType>(0b10) : static_cast<ResultType >(0b1);
}

14
dbms/src/Functions/formatString.h
View File

@ -200,12 +200,12 @@ struct FormatImpl
/// res_data is result_data, res_offsets is offset result.
/// input_rows_count is the number of rows processed.
/// Precondition: data.size() == offsets.size() == fixed_string_N.size() == constant_strings.size().
template <bool HasColumnString, bool HasColumnFixedString>
template <bool has_column_string, bool has_column_fixed_string>
static inline void format(
String pattern,
const std::vector<const ColumnString::Chars *> & data,
const std::vector<const ColumnString::Offsets *> & offsets,
[[maybe_unused]] /* Because sometimes !HasColumnFixedString */ const std::vector<size_t> & fixed_string_N,
[[maybe_unused]] /* Because sometimes !has_column_fixed_string */ const std::vector<size_t> & fixed_string_N,
const std::vector<String> & constant_strings,
ColumnString::Chars & res_data,
ColumnString::Offsets & res_offsets,
@ -265,7 +265,7 @@ struct FormatImpl
memcpySmallAllowReadWriteOverflow15(res_data.data() + offset, substrings[0].data(), substrings[0].size());
offset += substrings[0].size();
/// All strings are constant, we should have substrings.size() == 1.
if constexpr (HasColumnString || HasColumnFixedString)
if constexpr (has_column_string || has_column_fixed_string)
{
for (size_t j = 1; j < substrings.size(); ++j)
{
@ -274,18 +274,18 @@ struct FormatImpl
UInt64 arg_offset = 0;
UInt64 size = 0;
if constexpr (HasColumnString)
if constexpr (has_column_string)
{
if (!HasColumnFixedString || offset_ptr)
if (!has_column_fixed_string || offset_ptr)
{
arg_offset = (*offset_ptr)[i - 1];
size = (*offset_ptr)[i] - arg_offset - 1;
}
}
if constexpr (HasColumnFixedString)
if constexpr (has_column_fixed_string)
{
if (!HasColumnString || !offset_ptr)
if (!has_column_string || !offset_ptr)
{
arg_offset = fixed_string_N[arg] * i;
size = fixed_string_N[arg];

2
dbms/src/Functions/getScalar.cpp
View File

@ -42,7 +42,7 @@ public:
DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
{
if (arguments.size() != 1 || !isString(arguments[0].type) || !isColumnConst(*arguments[0].column))
if (arguments.size() != 1 || !isString(arguments[0].type) || !arguments[0].column || !isColumnConst(*arguments[0].column))
throw Exception("Function " + getName() + " accepts one const string argument", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
auto scalar_name = assert_cast<const ColumnConst &>(*arguments[0].column).getField().get<String>();
scalar = context.getScalar(scalar_name).getByPosition(0);

6
dbms/src/Functions/in.cpp
View File

@ -73,13 +73,17 @@ public:
return std::make_shared<DataTypeUInt8>();
}
bool useDefaultImplementationForConstants() const override { return true; }
void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t /*input_rows_count*/) override
{
/// NOTE: after updating this code, check that FunctionIgnoreExceptNull returns the same type of column.
/// Second argument must be ColumnSet.
ColumnPtr column_set_ptr = block.getByPosition(arguments[1]).column;
const ColumnSet * column_set = typeid_cast<const ColumnSet *>(&*column_set_ptr);
const ColumnSet * column_set = checkAndGetColumnConstData<const ColumnSet>(column_set_ptr.get());
if (!column_set)
column_set = checkAndGetColumn<const ColumnSet>(column_set_ptr.get());
if (!column_set)
throw Exception("Second argument for function '" + getName() + "' must be Set; found " + column_set_ptr->getName(),
ErrorCodes::ILLEGAL_COLUMN);

2
dbms/src/Functions/intDiv.cpp
View File

@ -28,7 +28,7 @@ struct DivideIntegralByConstantImpl
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wsign-compare"
if (unlikely(std::is_signed_v<B> && b == -1))
if (unlikely(is_signed_v<B> && b == -1))
{
size_t size = a.size();
for (size_t i = 0; i < size; ++i)

6
dbms/src/Functions/intDiv.h
View File

@ -27,7 +27,7 @@ inline void throwIfDivisionLeadsToFPE(A a, B b)
throw Exception("Division by zero", ErrorCodes::ILLEGAL_DIVISION);
/// http://avva.livejournal.com/2548306.html
if (unlikely(std::is_signed_v<A> && std::is_signed_v<B> && a == std::numeric_limits<A>::min() && b == -1))
if (unlikely(is_signed_v<A> && is_signed_v<B> && a == std::numeric_limits<A>::min() && b == -1))
throw Exception("Division of minimal signed number by minus one", ErrorCodes::ILLEGAL_DIVISION);
}
@ -37,7 +37,7 @@ inline bool divisionLeadsToFPE(A a, B b)
if (unlikely(b == 0))
return true;
if (unlikely(std::is_signed_v<A> && std::is_signed_v<B> && a == std::numeric_limits<A>::min() && b == -1))
if (unlikely(is_signed_v<A> && is_signed_v<B> && a == std::numeric_limits<A>::min() && b == -1))
return true;
return false;
@ -58,7 +58,7 @@ struct DivideIntegralImpl
/// Otherwise overflow may occur due to integer promotion. Example: int8_t(-1) / uint64_t(2).
/// NOTE: overflow is still possible when dividing large signed number to large unsigned number or vice-versa. But it's less harmful.
if constexpr (std::is_integral_v<A> && std::is_integral_v<B> && (std::is_signed_v<A> || std::is_signed_v<B>))
if constexpr (is_integral_v<A> && is_integral_v<B> && (is_signed_v<A> || is_signed_v<B>))
return std::make_signed_t<A>(a) / std::make_signed_t<B>(b);
else
return a / b;

2
dbms/src/Functions/isValidUTF8.cpp
View File

@ -251,7 +251,7 @@ SOFTWARE.
* pos-240: | 0 0 0 | 0 0 0 | 0 1 15 |
* pos+112: | 112 113 127| >= 128 | >= 128 |
*/
tmp1 = _mm_subs_epu8(pos, _mm_set1_epi8(240));
tmp1 = _mm_subs_epu8(pos, _mm_set1_epi8(0xF0));
range2 = _mm_shuffle_epi8(df_ee_tbl, tmp1);
tmp2 = _mm_adds_epu8(pos, _mm_set1_epi8(112));
range2 = _mm_add_epi8(range2, _mm_shuffle_epi8(ef_fe_tbl, tmp2));

9
dbms/src/Functions/registerFunctionsHigherOrder.cpp
View File

@ -8,12 +8,13 @@ void registerFunctionArrayFilter(FunctionFactory &);
void registerFunctionArrayCount(FunctionFactory &);
void registerFunctionArrayExists(FunctionFactory &);
void registerFunctionArrayAll(FunctionFactory &);
void registerFunctionArrayCompact(FunctionFactory &);
void registerFunctionArraySum(FunctionFactory &);
void registerFunctionArrayFirst(FunctionFactory &);
void registerFunctionArrayFirstIndex(FunctionFactory &);
void registerFunctionArraySplit(FunctionFactory &);
void registerFunctionsArrayFill(FunctionFactory &);
void registerFunctionsArraySplit(FunctionFactory &);
void registerFunctionsArraySort(FunctionFactory &);
void registerFunctionArrayReverseSort(FunctionFactory &);
void registerFunctionArrayCumSum(FunctionFactory &);
void registerFunctionArrayCumSumNonNegative(FunctionFactory &);
void registerFunctionArrayDifference(FunctionFactory &);
@ -25,10 +26,12 @@ void registerFunctionsHigherOrder(FunctionFactory & factory)
registerFunctionArrayCount(factory);
registerFunctionArrayExists(factory);
registerFunctionArrayAll(factory);
registerFunctionArrayCompact(factory);
registerFunctionArraySum(factory);
registerFunctionArrayFirst(factory);
registerFunctionArrayFirstIndex(factory);
registerFunctionArraySplit(factory);
registerFunctionsArrayFill(factory);
registerFunctionsArraySplit(factory);
registerFunctionsArraySort(factory);
registerFunctionArrayCumSum(factory);
registerFunctionArrayCumSumNonNegative(factory);

4
dbms/src/Functions/roundToExp2.cpp
View File

@ -7,14 +7,14 @@ namespace DB
{
template <typename T>
inline std::enable_if_t<std::is_integral_v<T> && (sizeof(T) <= sizeof(UInt32)), T>
inline std::enable_if_t<is_integral_v<T> && (sizeof(T) <= sizeof(UInt32)), T>
roundDownToPowerOfTwo(T x)
{
return x <= 0 ? 0 : (T(1) << (31 - __builtin_clz(x)));
}
template <typename T>
inline std::enable_if_t<std::is_integral_v<T> && (sizeof(T) == sizeof(UInt64)), T>
inline std::enable_if_t<is_integral_v<T> && (sizeof(T) == sizeof(UInt64)), T>
roundDownToPowerOfTwo(T x)
{
return x <= 0 ? 0 : (T(1) << (63 - __builtin_clzll(x)));

Some files were not shown because too many files have changed in this diff Show More