mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-22 07:31:57 +00:00
Merge remote-tracking branch 'upstream/master' into polymorphic-parts
This commit is contained in:
CurtizJ
commit
33ae97894b
6
.gitmodules
vendored
6
.gitmodules
vendored
@ -128,6 +128,12 @@
|
||||
[submodule "contrib/icu"]
|
||||
path = contrib/icu
|
||||
url = https://github.com/unicode-org/icu.git
|
||||
[submodule "contrib/flatbuffers"]
|
||||
path = contrib/flatbuffers
|
||||
url = https://github.com/google/flatbuffers.git
|
||||
[submodule "contrib/libc-headers"]
|
||||
path = contrib/libc-headers
|
||||
url = https://github.com/ClickHouse-Extras/libc-headers.git
|
||||
[submodule "contrib/ryu"]
|
||||
path = contrib/ryu
|
||||
url = https://github.com/ClickHouse-Extras/ryu.git
|
||||
|
||||
35
CHANGELOG.md
35
CHANGELOG.md
@ -1,3 +1,38 @@
|
||||
## ClickHouse release v19.17.6.36, 2019-12-27
|
||||
|
||||
### Bug Fix
|
||||
* Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that could cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. [#8404](https://github.com/ClickHouse/ClickHouse/pull/8404) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Fixed possible server crash (`std::terminate`) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. [#8384](https://github.com/ClickHouse/ClickHouse/pull/8384) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Fixed dictionaries with source from a clickhouse `VIEW`, now reading such dictionaries doesn't cause the error `There is no query`. [#8351](https://github.com/ClickHouse/ClickHouse/pull/8351) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||
* Fixed checking if a client host is allowed by host_regexp specified in users.xml. [#8241](https://github.com/ClickHouse/ClickHouse/pull/8241), [#8342](https://github.com/ClickHouse/ClickHouse/pull/8342) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* `RENAME TABLE` for a distributed table now renames the folder containing inserted data before sending to shards. This fixes an issue with successive renames `tableA->tableB`, `tableC->tableA`. [#8306](https://github.com/ClickHouse/ClickHouse/pull/8306) ([tavplubix](https://github.com/tavplubix))
|
||||
* `range_hashed` external dictionaries created by DDL queries now allow ranges of arbitrary numeric types. [#8275](https://github.com/ClickHouse/ClickHouse/pull/8275) ([alesapin](https://github.com/alesapin))
|
||||
* Fixed `INSERT INTO table SELECT ... FROM mysql(...)` table function. [#8234](https://github.com/ClickHouse/ClickHouse/pull/8234) ([tavplubix](https://github.com/tavplubix))
|
||||
* Fixed segfault in `INSERT INTO TABLE FUNCTION file()` while inserting into a file which doesn't exist. Now in this case file would be created and then insert would be processed. [#8177](https://github.com/ClickHouse/ClickHouse/pull/8177) ([Olga Khvostikova](https://github.com/stavrolia))
|
||||
* Fixed bitmapAnd error when intersecting an aggregated bitmap and a scalar bitmap. [#8082](https://github.com/ClickHouse/ClickHouse/pull/8082) ([Yue Huang](https://github.com/moon03432))
|
||||
* Fixed segfault when `EXISTS` query was used without `TABLE` or `DICTIONARY` qualifier, just like `EXISTS t`. [#8213](https://github.com/ClickHouse/ClickHouse/pull/8213) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||
* Fixed return type for functions `rand` and `randConstant` in case of nullable argument. Now functions always return `UInt32` and never `Nullable(UInt32)`. [#8204](https://github.com/ClickHouse/ClickHouse/pull/8204) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||
* Fixed `DROP DICTIONARY IF EXISTS db.dict`, now it doesn't throw exception if `db` doesn't exist. [#8185](https://github.com/ClickHouse/ClickHouse/pull/8185) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||
* If a table wasn't completely dropped because of server crash, the server will try to restore and load it [#8176](https://github.com/ClickHouse/ClickHouse/pull/8176) ([tavplubix](https://github.com/tavplubix))
|
||||
* Fixed a trivial count query for a distributed table if there are more than two shard local table. [#8164](https://github.com/ClickHouse/ClickHouse/pull/8164) ([小路](https://github.com/nicelulu))
|
||||
* Fixed bug that lead to a data race in DB::BlockStreamProfileInfo::calculateRowsBeforeLimit() [#8143](https://github.com/ClickHouse/ClickHouse/pull/8143) ([Alexander Kazakov](https://github.com/Akazz))
|
||||
* Fixed `ALTER table MOVE part` executed immediately after merging the specified part, which could cause moving a part which the specified part merged into. Now it correctly moves the specified part. [#8104](https://github.com/ClickHouse/ClickHouse/pull/8104) ([Vladimir Chebotarev](https://github.com/excitoon))
|
||||
* Expressions for dictionaries can be specified as strings now. This is useful for calculation of attributes while extracting data from non-ClickHouse sources because it allows to use non-ClickHouse syntax for those expressions. [#8098](https://github.com/ClickHouse/ClickHouse/pull/8098) ([alesapin](https://github.com/alesapin))
|
||||
* Fixed a very rare race in `clickhouse-copier` because of an overflow in ZXid. [#8088](https://github.com/ClickHouse/ClickHouse/pull/8088) ([Ding Xiang Fei](https://github.com/dingxiangfei2009))
|
||||
* Fixed the bug when after the query failed (due to "Too many simultaneous queries" for example) it would not read external tables info, and the
|
||||
next request would interpret this info as the beginning of the next query causing an error like `Unknown packet from client`. [#8084](https://github.com/ClickHouse/ClickHouse/pull/8084) ([Azat Khuzhin](https://github.com/azat))
|
||||
* Avoid null dereference after "Unknown packet X from server" [#8071](https://github.com/ClickHouse/ClickHouse/pull/8071) ([Azat Khuzhin](https://github.com/azat))
|
||||
* Restore support of all ICU locales, add the ability to apply collations for constant expressions and add language name to system.collations table. [#8051](https://github.com/ClickHouse/ClickHouse/pull/8051) ([alesapin](https://github.com/alesapin))
|
||||
* Number of streams for read from `StorageFile` and `StorageHDFS` is now limited, to avoid exceeding the memory limit. [#7981](https://github.com/ClickHouse/ClickHouse/pull/7981) ([alesapin](https://github.com/alesapin))
|
||||
* Fixed `CHECK TABLE` query for `*MergeTree` tables without key. [#7979](https://github.com/ClickHouse/ClickHouse/pull/7979) ([alesapin](https://github.com/alesapin))
|
||||
* Removed the mutation number from a part name in case there were no mutations. This removing improved the compatibility with older versions. [#8250](https://github.com/ClickHouse/ClickHouse/pull/8250) ([alesapin](https://github.com/alesapin))
|
||||
* Fixed the bug that mutations are skipped for some attached parts due to their data_version are larger than the table mutation version. [#7812](https://github.com/ClickHouse/ClickHouse/pull/7812) ([Zhichang Yu](https://github.com/yuzhichang))
|
||||
* Allow starting the server with redundant copies of parts after moving them to another device. [#7810](https://github.com/ClickHouse/ClickHouse/pull/7810) ([Vladimir Chebotarev](https://github.com/excitoon))
|
||||
* Fixed the error "Sizes of columns doesn't match" that might appear when using aggregate function columns. [#7790](https://github.com/ClickHouse/ClickHouse/pull/7790) ([Boris Granveaud](https://github.com/bgranvea))
|
||||
* Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. And now it's possible to use TOP with LIMIT BY. [#7637](https://github.com/ClickHouse/ClickHouse/pull/7637) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov))
|
||||
* Fix dictionary reload if it has `invalidate_query`, which stopped updates and some exception on previous update tries. [#8029](https://github.com/ClickHouse/ClickHouse/pull/8029) ([alesapin](https://github.com/alesapin))
|
||||
|
||||
|
||||
## ClickHouse release v19.17.4.11, 2019-11-22
|
||||
|
||||
### Backward Incompatible Change
|
||||
|
||||
@ -176,7 +176,9 @@ if (ARCH_NATIVE)
|
||||
set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")
|
||||
endif ()
|
||||
|
||||
set (CMAKE_CXX_STANDARD 17)
|
||||
# cmake < 3.12 doesn't supoprt 20. We'll set CMAKE_CXX_FLAGS for now
|
||||
# set (CMAKE_CXX_STANDARD 20)
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++2a")
|
||||
set (CMAKE_CXX_EXTENSIONS 0) # https://cmake.org/cmake/help/latest/prop_tgt/CXX_EXTENSIONS.html#prop_tgt:CXX_EXTENSIONS
|
||||
set (CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
|
||||
@ -208,7 +210,7 @@ set (CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -O0 -g3 -ggdb3
|
||||
|
||||
if (COMPILER_CLANG)
|
||||
# Exception unwinding doesn't work in clang release build without this option
|
||||
# TODO investigate if contrib/libcxxabi is out of date
|
||||
# TODO investigate that
|
||||
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-omit-frame-pointer")
|
||||
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-omit-frame-pointer")
|
||||
endif ()
|
||||
@ -248,8 +250,16 @@ endif ()
|
||||
string (TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
|
||||
set (CMAKE_POSTFIX_VARIABLE "CMAKE_${CMAKE_BUILD_TYPE_UC}_POSTFIX")
|
||||
|
||||
if (NOT MAKE_STATIC_LIBRARIES)
|
||||
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
|
||||
if (MAKE_STATIC_LIBRARIES)
|
||||
set (CMAKE_POSITION_INDEPENDENT_CODE OFF)
|
||||
if (OS_LINUX)
|
||||
# Slightly more efficient code can be generated
|
||||
set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -fno-pie")
|
||||
set (CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} -fno-pie")
|
||||
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-no-pie")
|
||||
endif ()
|
||||
else ()
|
||||
set (CMAKE_POSITION_INDEPENDENT_CODE ON)
|
||||
endif ()
|
||||
|
||||
# Using "include-what-you-use" tool.
|
||||
|
||||
@ -15,6 +15,7 @@ set(CMAKE_C_STANDARD_LIBRARIES ${DEFAULT_LIBS})
|
||||
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mmacosx-version-min=10.14")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mmacosx-version-min=10.14")
|
||||
set (CMAKE_ASM_FLAGS "${CMAKE_ASM_FLAGS} -mmacosx-version-min=10.14")
|
||||
|
||||
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -mmacosx-version-min=10.14")
|
||||
set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -mmacosx-version-min=10.14")
|
||||
|
||||
@ -2,6 +2,7 @@ set (CMAKE_SYSTEM_NAME "Darwin")
|
||||
set (CMAKE_SYSTEM_PROCESSOR "x86_64")
|
||||
set (CMAKE_C_COMPILER_TARGET "x86_64-apple-darwin")
|
||||
set (CMAKE_CXX_COMPILER_TARGET "x86_64-apple-darwin")
|
||||
set (CMAKE_ASM_COMPILER_TARGET "x86_64-apple-darwin")
|
||||
set (CMAKE_OSX_SYSROOT "${CMAKE_CURRENT_LIST_DIR}/../toolchain/darwin-x86_64")
|
||||
|
||||
set (CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) # disable linkage check - it doesn't work in CMake
|
||||
|
||||
@ -1,4 +1,8 @@
|
||||
option(ENABLE_ICU "Enable ICU" ${ENABLE_LIBRARIES})
|
||||
if (OS_LINUX)
|
||||
option(ENABLE_ICU "Enable ICU" ${ENABLE_LIBRARIES})
|
||||
else ()
|
||||
option(ENABLE_ICU "Enable ICU" 0)
|
||||
endif ()
|
||||
|
||||
if (ENABLE_ICU)
|
||||
|
||||
|
||||
@ -54,10 +54,12 @@ elseif(NOT MISSING_INTERNAL_PARQUET_LIBRARY AND NOT OS_FREEBSD)
|
||||
endif()
|
||||
|
||||
if(${USE_STATIC_LIBRARIES})
|
||||
set(FLATBUFFERS_LIBRARY flatbuffers)
|
||||
set(ARROW_LIBRARY arrow_static)
|
||||
set(PARQUET_LIBRARY parquet_static)
|
||||
set(THRIFT_LIBRARY thrift_static)
|
||||
else()
|
||||
set(FLATBUFFERS_LIBRARY flatbuffers_shared)
|
||||
set(ARROW_LIBRARY arrow_shared)
|
||||
set(PARQUET_LIBRARY parquet_shared)
|
||||
if(USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||
@ -74,7 +76,7 @@ endif()
|
||||
endif()
|
||||
|
||||
if(USE_PARQUET)
|
||||
message(STATUS "Using Parquet: ${ARROW_LIBRARY}:${ARROW_INCLUDE_DIR} ; ${PARQUET_LIBRARY}:${PARQUET_INCLUDE_DIR} ; ${THRIFT_LIBRARY}")
|
||||
message(STATUS "Using Parquet: ${ARROW_LIBRARY}:${ARROW_INCLUDE_DIR} ; ${PARQUET_LIBRARY}:${PARQUET_INCLUDE_DIR} ; ${THRIFT_LIBRARY} ; ${FLATBUFFERS_LIBRARY}")
|
||||
else()
|
||||
message(STATUS "Building without Parquet support")
|
||||
endif()
|
||||
|
||||
@ -13,7 +13,6 @@ if (CMAKE_CROSSCOMPILING)
|
||||
if (OS_DARWIN)
|
||||
# FIXME: broken dependencies
|
||||
set (USE_SNAPPY OFF CACHE INTERNAL "")
|
||||
set (ENABLE_SSL OFF CACHE INTERNAL "")
|
||||
set (ENABLE_PROTOBUF OFF CACHE INTERNAL "")
|
||||
set (ENABLE_PARQUET OFF CACHE INTERNAL "")
|
||||
set (ENABLE_READLINE OFF CACHE INTERNAL "")
|
||||
|
||||
9
contrib/CMakeLists.txt
vendored
9
contrib/CMakeLists.txt
vendored
@ -2,10 +2,10 @@
|
||||
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w -std=c++1z")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w")
|
||||
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w -std=c++1z")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w")
|
||||
endif ()
|
||||
|
||||
set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL 1)
|
||||
@ -32,6 +32,8 @@ if (USE_INTERNAL_DOUBLE_CONVERSION_LIBRARY)
|
||||
add_subdirectory (double-conversion-cmake)
|
||||
endif ()
|
||||
|
||||
add_subdirectory (ryu-cmake)
|
||||
|
||||
if (USE_INTERNAL_CITYHASH_LIBRARY)
|
||||
add_subdirectory (cityhash102)
|
||||
endif ()
|
||||
@ -159,6 +161,8 @@ if (USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||
set (ARROW_PARQUET ON CACHE INTERNAL "")
|
||||
set (ARROW_VERBOSE_THIRDPARTY_BUILD ON CACHE INTERNAL "")
|
||||
set (ARROW_BUILD_SHARED 1 CACHE INTERNAL "")
|
||||
set (ARROW_BUILD_UTILITIES OFF CACHE INTERNAL "")
|
||||
set (ARROW_BUILD_INTEGRATION OFF CACHE INTERNAL "")
|
||||
set (ARROW_BOOST_HEADER_ONLY ON CACHE INTERNAL "")
|
||||
set (Boost_FOUND 1 CACHE INTERNAL "")
|
||||
if (MAKE_STATIC_LIBRARIES)
|
||||
@ -248,6 +252,7 @@ if (USE_EMBEDDED_COMPILER AND USE_INTERNAL_LLVM_LIBRARY)
|
||||
endif ()
|
||||
set (LLVM_ENABLE_EH 1 CACHE INTERNAL "")
|
||||
set (LLVM_ENABLE_RTTI 1 CACHE INTERNAL "")
|
||||
set (LLVM_ENABLE_PIC 0 CACHE INTERNAL "")
|
||||
set (LLVM_TARGETS_TO_BUILD "X86;AArch64" CACHE STRING "")
|
||||
add_subdirectory (llvm/llvm)
|
||||
endif ()
|
||||
|
||||
2
contrib/arrow
vendored
2
contrib/arrow
vendored
@ -1 +1 @@
|
||||
Subproject commit 87ac6fddaf21d0b4ee8b8090533ff293db0da1b4
|
||||
Subproject commit b789226ccb2124285792107c758bb3b40b3d082a
|
||||
@ -1,3 +1,7 @@
|
||||
include(ExternalProject)
|
||||
|
||||
set (CMAKE_CXX_STANDARD 17)
|
||||
|
||||
# === thrift
|
||||
|
||||
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/thrift/lib/cpp)
|
||||
@ -34,13 +38,13 @@ set(thriftcpp_SOURCES
|
||||
${LIBRARY_DIR}/src/thrift/server/TSimpleServer.cpp
|
||||
${LIBRARY_DIR}/src/thrift/server/TThreadPoolServer.cpp
|
||||
${LIBRARY_DIR}/src/thrift/server/TThreadedServer.cpp
|
||||
)
|
||||
set( thriftcpp_threads_SOURCES
|
||||
)
|
||||
set(thriftcpp_threads_SOURCES
|
||||
${LIBRARY_DIR}/src/thrift/concurrency/ThreadFactory.cpp
|
||||
${LIBRARY_DIR}/src/thrift/concurrency/Thread.cpp
|
||||
${LIBRARY_DIR}/src/thrift/concurrency/Monitor.cpp
|
||||
${LIBRARY_DIR}/src/thrift/concurrency/Mutex.cpp
|
||||
)
|
||||
)
|
||||
add_library(${THRIFT_LIBRARY} ${thriftcpp_SOURCES} ${thriftcpp_threads_SOURCES})
|
||||
set_target_properties(${THRIFT_LIBRARY} PROPERTIES CXX_STANDARD 14) # REMOVE after https://github.com/apache/thrift/pull/1641
|
||||
target_include_directories(${THRIFT_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/thrift/lib/cpp/src PRIVATE ${Boost_INCLUDE_DIRS})
|
||||
@ -70,13 +74,84 @@ add_custom_command(OUTPUT orc_proto.pb.h orc_proto.pb.cc
|
||||
--cpp_out="${CMAKE_CURRENT_BINARY_DIR}"
|
||||
"${PROTO_DIR}/orc_proto.proto")
|
||||
|
||||
|
||||
# === flatbuffers
|
||||
|
||||
##############################################################
|
||||
# fbs - Step 1: build flatbuffers lib and flatc compiler
|
||||
##############################################################
|
||||
set(FLATBUFFERS_SRC_DIR ${ClickHouse_SOURCE_DIR}/contrib/flatbuffers)
|
||||
set(FLATBUFFERS_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/flatbuffers)
|
||||
set(FLATBUFFERS_INCLUDE_DIR ${FLATBUFFERS_SRC_DIR}/include)
|
||||
set(FLATBUFFERS_COMPILER "${FLATBUFFERS_BINARY_DIR}/flatc")
|
||||
|
||||
# set flatbuffers CMake options
|
||||
if (${USE_STATIC_LIBRARIES})
|
||||
set(FLATBUFFERS_BUILD_FLATLIB ON CACHE BOOL "Enable the build of the flatbuffers library")
|
||||
set(FLATBUFFERS_BUILD_SHAREDLIB OFF CACHE BOOL "Disable the build of the flatbuffers shared library")
|
||||
else ()
|
||||
set(FLATBUFFERS_BUILD_SHAREDLIB ON CACHE BOOL "Enable the build of the flatbuffers shared library")
|
||||
set(FLATBUFFERS_BUILD_FLATLIB OFF CACHE BOOL "Disable the build of the flatbuffers library")
|
||||
endif ()
|
||||
set(FLATBUFFERS_BUILD_FLATC ON CACHE BOOL "Build flatbuffers compiler")
|
||||
set(FLATBUFFERS_BUILD_TESTS OFF CACHE BOOL "Skip flatbuffers tests")
|
||||
|
||||
add_subdirectory(${FLATBUFFERS_SRC_DIR} "${FLATBUFFERS_BINARY_DIR}")
|
||||
|
||||
###################################
|
||||
# fbs - Step 2: compile *.fbs files
|
||||
###################################
|
||||
set(ARROW_IPC_SRC_DIR ${ARROW_SRC_DIR}/arrow/ipc)
|
||||
set(ARROW_FORMAT_SRC_DIR ${ARROW_SRC_DIR}/../../format)
|
||||
|
||||
set(ARROW_GENERATED_INCLUDE_DIR ${CMAKE_CURRENT_BINARY_DIR}/arrow_gen_headers)
|
||||
set(FLATBUFFERS_COMPILED_OUT_DIR ${ARROW_GENERATED_INCLUDE_DIR}/arrow/ipc)
|
||||
|
||||
set(FBS_OUTPUT_FILES
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/File_generated.h"
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/Message_generated.h"
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/feather_generated.h"
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/Schema_generated.h"
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/SparseTensor_generated.h"
|
||||
"${FLATBUFFERS_COMPILED_OUT_DIR}/Tensor_generated.h")
|
||||
|
||||
set(FBS_SRC
|
||||
${ARROW_FORMAT_SRC_DIR}/Message.fbs
|
||||
${ARROW_FORMAT_SRC_DIR}/File.fbs
|
||||
${ARROW_FORMAT_SRC_DIR}/Schema.fbs
|
||||
${ARROW_FORMAT_SRC_DIR}/Tensor.fbs
|
||||
${ARROW_FORMAT_SRC_DIR}/SparseTensor.fbs
|
||||
${ARROW_IPC_SRC_DIR}/feather.fbs)
|
||||
|
||||
foreach (FIL ${FBS_SRC})
|
||||
get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
|
||||
list(APPEND ABS_FBS_SRC ${ABS_FIL})
|
||||
endforeach ()
|
||||
|
||||
message(STATUS "FLATBUFFERS_LIBRARY: ${FLATBUFFERS_LIBRARY}, FLATBUFFERS_COMPILER: ${FLATBUFFERS_COMPILER}")
|
||||
message(STATUS "FLATBUFFERS_COMPILED_OUT_DIR: ${FLATBUFFERS_COMPILED_OUT_DIR}")
|
||||
message(STATUS "flatc: ${FLATBUFFERS_COMPILER} -c -o ${FLATBUFFERS_COMPILED_OUT_DIR}/ ${ABS_FBS_SRC}")
|
||||
|
||||
add_custom_command(OUTPUT ${FBS_OUTPUT_FILES}
|
||||
COMMAND ${FLATBUFFERS_COMPILER}
|
||||
-c
|
||||
-o
|
||||
${FLATBUFFERS_COMPILED_OUT_DIR}/
|
||||
${ABS_FBS_SRC}
|
||||
DEPENDS flatc ${ABS_FBS_SRC}
|
||||
COMMENT "Running flatc compiler on ${ABS_FBS_SRC}"
|
||||
VERBATIM)
|
||||
|
||||
add_custom_target(metadata_fbs DEPENDS ${FBS_OUTPUT_FILES})
|
||||
add_dependencies(metadata_fbs flatc)
|
||||
|
||||
# arrow-cmake cmake file calling orc cmake subroutine which detects certain compiler features.
|
||||
# Apple Clang compiler failed to compile this code without specifying c++11 standard.
|
||||
# As result these compiler features detected as absent. In result it failed to compile orc itself.
|
||||
# In orc makefile there is code that sets flags, but arrow-cmake ignores these flags.
|
||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "AppleClang")
|
||||
set (CXX11_FLAGS "-std=c++0x")
|
||||
endif()
|
||||
set(CXX11_FLAGS "-std=c++0x")
|
||||
endif ()
|
||||
|
||||
include(${ClickHouse_SOURCE_DIR}/contrib/orc/cmake_modules/CheckSourceCompiles.cmake)
|
||||
include(orc_check.cmake)
|
||||
@ -86,6 +161,7 @@ configure_file("${ORC_SOURCE_SRC_DIR}/Adaptor.hh.in" "${ORC_BUILD_INCLUDE_DIR}/A
|
||||
|
||||
set(ORC_SRCS
|
||||
${ARROW_SRC_DIR}/arrow/adapters/orc/adapter.cc
|
||||
${ARROW_SRC_DIR}/arrow/adapters/orc/adapter_util.cc
|
||||
${ORC_SOURCE_SRC_DIR}/Exceptions.cc
|
||||
${ORC_SOURCE_SRC_DIR}/OrcFile.cc
|
||||
${ORC_SOURCE_SRC_DIR}/Reader.cc
|
||||
@ -119,11 +195,29 @@ set(ORC_SRCS
|
||||
# === arrow
|
||||
|
||||
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src/arrow)
|
||||
|
||||
configure_file("${LIBRARY_DIR}/util/config.h.cmake" "${CMAKE_CURRENT_SOURCE_DIR}/cpp/src/arrow/util/config.h")
|
||||
|
||||
# arrow/cpp/src/arrow/CMakeLists.txt
|
||||
set(ARROW_SRCS
|
||||
${LIBRARY_DIR}/array.cc
|
||||
|
||||
${LIBRARY_DIR}/buffer.cc
|
||||
${LIBRARY_DIR}/builder.cc
|
||||
${LIBRARY_DIR}/compare.cc
|
||||
${LIBRARY_DIR}/extension_type.cc
|
||||
${LIBRARY_DIR}/memory_pool.cc
|
||||
${LIBRARY_DIR}/pretty_print.cc
|
||||
${LIBRARY_DIR}/record_batch.cc
|
||||
${LIBRARY_DIR}/result.cc
|
||||
${LIBRARY_DIR}/scalar.cc
|
||||
${LIBRARY_DIR}/sparse_tensor.cc
|
||||
${LIBRARY_DIR}/status.cc
|
||||
${LIBRARY_DIR}/table_builder.cc
|
||||
${LIBRARY_DIR}/table.cc
|
||||
${LIBRARY_DIR}/tensor.cc
|
||||
${LIBRARY_DIR}/type.cc
|
||||
${LIBRARY_DIR}/visitor.cc
|
||||
|
||||
${LIBRARY_DIR}/array/builder_adaptive.cc
|
||||
${LIBRARY_DIR}/array/builder_base.cc
|
||||
${LIBRARY_DIR}/array/builder_binary.cc
|
||||
@ -131,114 +225,130 @@ set(ARROW_SRCS
|
||||
${LIBRARY_DIR}/array/builder_dict.cc
|
||||
${LIBRARY_DIR}/array/builder_nested.cc
|
||||
${LIBRARY_DIR}/array/builder_primitive.cc
|
||||
|
||||
${LIBRARY_DIR}/buffer.cc
|
||||
${LIBRARY_DIR}/compare.cc
|
||||
${LIBRARY_DIR}/memory_pool.cc
|
||||
${LIBRARY_DIR}/pretty_print.cc
|
||||
${LIBRARY_DIR}/record_batch.cc
|
||||
${LIBRARY_DIR}/status.cc
|
||||
${LIBRARY_DIR}/table.cc
|
||||
${LIBRARY_DIR}/table_builder.cc
|
||||
${LIBRARY_DIR}/tensor.cc
|
||||
${LIBRARY_DIR}/sparse_tensor.cc
|
||||
${LIBRARY_DIR}/type.cc
|
||||
${LIBRARY_DIR}/visitor.cc
|
||||
${LIBRARY_DIR}/array/builder_union.cc
|
||||
${LIBRARY_DIR}/array/concatenate.cc
|
||||
${LIBRARY_DIR}/array/dict_internal.cc
|
||||
${LIBRARY_DIR}/array/diff.cc
|
||||
|
||||
${LIBRARY_DIR}/csv/converter.cc
|
||||
${LIBRARY_DIR}/csv/chunker.cc
|
||||
${LIBRARY_DIR}/csv/column-builder.cc
|
||||
${LIBRARY_DIR}/csv/column_builder.cc
|
||||
${LIBRARY_DIR}/csv/options.cc
|
||||
${LIBRARY_DIR}/csv/parser.cc
|
||||
${LIBRARY_DIR}/csv/reader.cc
|
||||
|
||||
${LIBRARY_DIR}/ipc/dictionary.cc
|
||||
${LIBRARY_DIR}/ipc/feather.cc
|
||||
${LIBRARY_DIR}/ipc/message.cc
|
||||
${LIBRARY_DIR}/ipc/metadata_internal.cc
|
||||
${LIBRARY_DIR}/ipc/options.cc
|
||||
${LIBRARY_DIR}/ipc/reader.cc
|
||||
${LIBRARY_DIR}/ipc/writer.cc
|
||||
|
||||
${LIBRARY_DIR}/io/buffered.cc
|
||||
${LIBRARY_DIR}/io/compressed.cc
|
||||
${LIBRARY_DIR}/io/file.cc
|
||||
${LIBRARY_DIR}/io/interfaces.cc
|
||||
${LIBRARY_DIR}/io/memory.cc
|
||||
${LIBRARY_DIR}/io/readahead.cc
|
||||
${LIBRARY_DIR}/io/slow.cc
|
||||
|
||||
${LIBRARY_DIR}/util/bit-util.cc
|
||||
${LIBRARY_DIR}/util/basic_decimal.cc
|
||||
${LIBRARY_DIR}/util/bit_util.cc
|
||||
${LIBRARY_DIR}/util/compression.cc
|
||||
${LIBRARY_DIR}/util/cpu-info.cc
|
||||
${LIBRARY_DIR}/util/compression_lz4.cc
|
||||
${LIBRARY_DIR}/util/compression_snappy.cc
|
||||
${LIBRARY_DIR}/util/compression_zlib.cc
|
||||
${LIBRARY_DIR}/util/compression_zstd.cc
|
||||
${LIBRARY_DIR}/util/cpu_info.cc
|
||||
${LIBRARY_DIR}/util/decimal.cc
|
||||
${LIBRARY_DIR}/util/int-util.cc
|
||||
${LIBRARY_DIR}/util/io-util.cc
|
||||
${LIBRARY_DIR}/util/logging.cc
|
||||
${LIBRARY_DIR}/util/int_util.cc
|
||||
${LIBRARY_DIR}/util/io_util.cc
|
||||
${LIBRARY_DIR}/util/key_value_metadata.cc
|
||||
${LIBRARY_DIR}/util/task-group.cc
|
||||
${LIBRARY_DIR}/util/thread-pool.cc
|
||||
${LIBRARY_DIR}/util/logging.cc
|
||||
${LIBRARY_DIR}/util/memory.cc
|
||||
${LIBRARY_DIR}/util/string_builder.cc
|
||||
${LIBRARY_DIR}/util/string.cc
|
||||
${LIBRARY_DIR}/util/task_group.cc
|
||||
${LIBRARY_DIR}/util/thread_pool.cc
|
||||
${LIBRARY_DIR}/util/trie.cc
|
||||
${LIBRARY_DIR}/util/utf8.cc
|
||||
|
||||
${LIBRARY_DIR}/vendored/base64.cpp
|
||||
${ORC_SRCS}
|
||||
)
|
||||
)
|
||||
|
||||
set(ARROW_SRCS ${ARROW_SRCS}
|
||||
${LIBRARY_DIR}/compute/context.cc
|
||||
${LIBRARY_DIR}/compute/kernels/boolean.cc
|
||||
${LIBRARY_DIR}/compute/kernels/cast.cc
|
||||
${LIBRARY_DIR}/compute/kernels/hash.cc
|
||||
${LIBRARY_DIR}/compute/kernels/util-internal.cc
|
||||
)
|
||||
${LIBRARY_DIR}/compute/kernels/util_internal.cc
|
||||
)
|
||||
|
||||
if (LZ4_INCLUDE_DIR AND LZ4_LIBRARY)
|
||||
set(ARROW_WITH_LZ4 1)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if(SNAPPY_INCLUDE_DIR AND SNAPPY_LIBRARY)
|
||||
if (SNAPPY_INCLUDE_DIR AND SNAPPY_LIBRARY)
|
||||
set(ARROW_WITH_SNAPPY 1)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if(ZLIB_INCLUDE_DIR AND ZLIB_LIBRARIES)
|
||||
if (ZLIB_INCLUDE_DIR AND ZLIB_LIBRARIES)
|
||||
set(ARROW_WITH_ZLIB 1)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (ZSTD_INCLUDE_DIR AND ZSTD_LIBRARY)
|
||||
set(ARROW_WITH_ZSTD 1)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (ARROW_WITH_LZ4)
|
||||
add_definitions(-DARROW_WITH_LZ4)
|
||||
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_lz4.cc ${ARROW_SRCS})
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (ARROW_WITH_SNAPPY)
|
||||
add_definitions(-DARROW_WITH_SNAPPY)
|
||||
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_snappy.cc ${ARROW_SRCS})
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (ARROW_WITH_ZLIB)
|
||||
add_definitions(-DARROW_WITH_ZLIB)
|
||||
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_zlib.cc ${ARROW_SRCS})
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
if (ARROW_WITH_ZSTD)
|
||||
add_definitions(-DARROW_WITH_ZSTD)
|
||||
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_zstd.cc ${ARROW_SRCS})
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
|
||||
add_library(${ARROW_LIBRARY} ${ARROW_SRCS})
|
||||
|
||||
# Arrow dependencies
|
||||
add_dependencies(${ARROW_LIBRARY} ${FLATBUFFERS_LIBRARY} metadata_fbs)
|
||||
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE boost_system_internal boost_filesystem_internal boost_regex_internal)
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${FLATBUFFERS_LIBRARY})
|
||||
|
||||
if (USE_INTERNAL_PROTOBUF_LIBRARY)
|
||||
add_dependencies(${ARROW_LIBRARY} protoc)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
target_include_directories(${ARROW_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/cpp/src ${Boost_INCLUDE_DIRS})
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${DOUBLE_CONVERSION_LIBRARIES} ${Protobuf_LIBRARY})
|
||||
if (ARROW_WITH_LZ4)
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${LZ4_LIBRARY})
|
||||
endif()
|
||||
endif ()
|
||||
if (ARROW_WITH_SNAPPY)
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${SNAPPY_LIBRARY})
|
||||
endif()
|
||||
endif ()
|
||||
if (ARROW_WITH_ZLIB)
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${ZLIB_LIBRARIES})
|
||||
endif()
|
||||
endif ()
|
||||
if (ARROW_WITH_ZSTD)
|
||||
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${ZSTD_LIBRARY})
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ORC_INCLUDE_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ORC_SOURCE_SRC_DIR})
|
||||
@ -248,6 +358,8 @@ target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ORC_BUILD_SRC_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ORC_BUILD_INCLUDE_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ORC_ADDITION_SOURCE_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ARROW_SRC_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${FLATBUFFERS_INCLUDE_DIR})
|
||||
target_include_directories(${ARROW_LIBRARY} PRIVATE SYSTEM ${ARROW_GENERATED_INCLUDE_DIR})
|
||||
|
||||
# === parquet
|
||||
|
||||
@ -255,45 +367,47 @@ set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src/parquet)
|
||||
# arrow/cpp/src/parquet/CMakeLists.txt
|
||||
set(PARQUET_SRCS
|
||||
${LIBRARY_DIR}/arrow/reader.cc
|
||||
${LIBRARY_DIR}/arrow/record_reader.cc
|
||||
${LIBRARY_DIR}/arrow/reader_internal.cc
|
||||
${LIBRARY_DIR}/arrow/schema.cc
|
||||
${LIBRARY_DIR}/arrow/writer.cc
|
||||
${LIBRARY_DIR}/bloom_filter.cc
|
||||
${LIBRARY_DIR}/column_reader.cc
|
||||
${LIBRARY_DIR}/column_scanner.cc
|
||||
${LIBRARY_DIR}/column_writer.cc
|
||||
${LIBRARY_DIR}/deprecated_io.cc
|
||||
${LIBRARY_DIR}/encoding.cc
|
||||
${LIBRARY_DIR}/file_reader.cc
|
||||
${LIBRARY_DIR}/file_writer.cc
|
||||
${LIBRARY_DIR}/metadata.cc
|
||||
${LIBRARY_DIR}/murmur3.cc
|
||||
${LIBRARY_DIR}/platform.cc
|
||||
${LIBRARY_DIR}/printer.cc
|
||||
${LIBRARY_DIR}/properties.cc
|
||||
${LIBRARY_DIR}/schema.cc
|
||||
${LIBRARY_DIR}/statistics.cc
|
||||
${LIBRARY_DIR}/types.cc
|
||||
${LIBRARY_DIR}/util/comparison.cc
|
||||
${LIBRARY_DIR}/util/memory.cc
|
||||
)
|
||||
)
|
||||
#list(TRANSFORM PARQUET_SRCS PREPEND ${LIBRARY_DIR}/) # cmake 3.12
|
||||
list(APPEND PARQUET_SRCS
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/cpp/src/parquet/parquet_constants.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/cpp/src/parquet/parquet_types.cpp
|
||||
)
|
||||
)
|
||||
add_library(${PARQUET_LIBRARY} ${PARQUET_SRCS})
|
||||
target_include_directories(${PARQUET_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src ${CMAKE_CURRENT_SOURCE_DIR}/cpp/src)
|
||||
include(${ClickHouse_SOURCE_DIR}/contrib/thrift/build/cmake/ConfigureChecks.cmake) # makes config.h
|
||||
target_link_libraries(${PARQUET_LIBRARY} PUBLIC ${ARROW_LIBRARY} PRIVATE ${THRIFT_LIBRARY} ${Boost_REGEX_LIBRARY})
|
||||
target_include_directories(${PARQUET_LIBRARY} PRIVATE ${Boost_INCLUDE_DIRS})
|
||||
|
||||
if(SANITIZE STREQUAL "undefined")
|
||||
if (SANITIZE STREQUAL "undefined")
|
||||
target_compile_options(${PARQUET_LIBRARY} PRIVATE -fno-sanitize=undefined)
|
||||
target_compile_options(${ARROW_LIBRARY} PRIVATE -fno-sanitize=undefined)
|
||||
endif()
|
||||
endif ()
|
||||
|
||||
# === tools
|
||||
|
||||
set(TOOLS_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/tools/parquet)
|
||||
set(PARQUET_TOOLS parquet-dump-schema parquet-reader parquet-scan)
|
||||
foreach(TOOL ${PARQUET_TOOLS})
|
||||
set(PARQUET_TOOLS parquet_dump_schema parquet_reader parquet_scan)
|
||||
foreach (TOOL ${PARQUET_TOOLS})
|
||||
add_executable(${TOOL} ${TOOLS_DIR}/${TOOL}.cc)
|
||||
target_link_libraries(${TOOL} PRIVATE ${PARQUET_LIBRARY})
|
||||
endforeach()
|
||||
endforeach ()
|
||||
|
||||
24
contrib/arrow-cmake/cpp/src/arrow/util/config.h
Normal file
24
contrib/arrow-cmake/cpp/src/arrow/util/config.h
Normal file
@ -0,0 +1,24 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#define ARROW_VERSION_MAJOR
|
||||
#define ARROW_VERSION_MINOR
|
||||
#define ARROW_VERSION_PATCH
|
||||
#define ARROW_VERSION ((ARROW_VERSION_MAJOR * 1000) + ARROW_VERSION_MINOR) * 1000 + ARROW_VERSION_PATCH
|
||||
|
||||
/* #undef DOUBLE_CONVERSION_HAS_CASE_INSENSIBILITY */
|
||||
/* #undef GRPCPP_PP_INCLUDE */
|
||||
@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Autogenerated by Thrift Compiler (0.11.0)
|
||||
* Autogenerated by Thrift Compiler (0.12.0)
|
||||
*
|
||||
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
|
||||
* @generated
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Autogenerated by Thrift Compiler (0.11.0)
|
||||
* Autogenerated by Thrift Compiler (0.12.0)
|
||||
*
|
||||
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
|
||||
* @generated
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Autogenerated by Thrift Compiler (0.11.0)
|
||||
* Autogenerated by Thrift Compiler (0.12.0)
|
||||
*
|
||||
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
|
||||
* @generated
|
||||
@ -17,7 +17,7 @@
|
||||
|
||||
#include <thrift/stdcxx.h>
|
||||
|
||||
#include "parquet/util/windows_compatibility.h"
|
||||
#include "parquet/windows_compatibility.h"
|
||||
|
||||
namespace parquet { namespace format {
|
||||
|
||||
@ -161,6 +161,8 @@ class MilliSeconds;
|
||||
|
||||
class MicroSeconds;
|
||||
|
||||
class NanoSeconds;
|
||||
|
||||
class TimeUnit;
|
||||
|
||||
class TimestampType;
|
||||
@ -215,14 +217,14 @@ class OffsetIndex;
|
||||
|
||||
class ColumnIndex;
|
||||
|
||||
class FileMetaData;
|
||||
|
||||
class AesGcmV1;
|
||||
|
||||
class AesGcmCtrV1;
|
||||
|
||||
class EncryptionAlgorithm;
|
||||
|
||||
class FileMetaData;
|
||||
|
||||
class FileCryptoMetaData;
|
||||
|
||||
typedef struct _Statistics__isset {
|
||||
@ -629,10 +631,42 @@ void swap(MicroSeconds &a, MicroSeconds &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const MicroSeconds& obj);
|
||||
|
||||
|
||||
class NanoSeconds : public virtual ::apache::thrift::TBase {
|
||||
public:
|
||||
|
||||
NanoSeconds(const NanoSeconds&);
|
||||
NanoSeconds& operator=(const NanoSeconds&);
|
||||
NanoSeconds() {
|
||||
}
|
||||
|
||||
virtual ~NanoSeconds() throw();
|
||||
|
||||
bool operator == (const NanoSeconds & /* rhs */) const
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool operator != (const NanoSeconds &rhs) const {
|
||||
return !(*this == rhs);
|
||||
}
|
||||
|
||||
bool operator < (const NanoSeconds & ) const;
|
||||
|
||||
uint32_t read(::apache::thrift::protocol::TProtocol* iprot);
|
||||
uint32_t write(::apache::thrift::protocol::TProtocol* oprot) const;
|
||||
|
||||
virtual void printTo(std::ostream& out) const;
|
||||
};
|
||||
|
||||
void swap(NanoSeconds &a, NanoSeconds &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const NanoSeconds& obj);
|
||||
|
||||
typedef struct _TimeUnit__isset {
|
||||
_TimeUnit__isset() : MILLIS(false), MICROS(false) {}
|
||||
_TimeUnit__isset() : MILLIS(false), MICROS(false), NANOS(false) {}
|
||||
bool MILLIS :1;
|
||||
bool MICROS :1;
|
||||
bool NANOS :1;
|
||||
} _TimeUnit__isset;
|
||||
|
||||
class TimeUnit : public virtual ::apache::thrift::TBase {
|
||||
@ -646,6 +680,7 @@ class TimeUnit : public virtual ::apache::thrift::TBase {
|
||||
virtual ~TimeUnit() throw();
|
||||
MilliSeconds MILLIS;
|
||||
MicroSeconds MICROS;
|
||||
NanoSeconds NANOS;
|
||||
|
||||
_TimeUnit__isset __isset;
|
||||
|
||||
@ -653,6 +688,8 @@ class TimeUnit : public virtual ::apache::thrift::TBase {
|
||||
|
||||
void __set_MICROS(const MicroSeconds& val);
|
||||
|
||||
void __set_NANOS(const NanoSeconds& val);
|
||||
|
||||
bool operator == (const TimeUnit & rhs) const
|
||||
{
|
||||
if (__isset.MILLIS != rhs.__isset.MILLIS)
|
||||
@ -663,6 +700,10 @@ class TimeUnit : public virtual ::apache::thrift::TBase {
|
||||
return false;
|
||||
else if (__isset.MICROS && !(MICROS == rhs.MICROS))
|
||||
return false;
|
||||
if (__isset.NANOS != rhs.__isset.NANOS)
|
||||
return false;
|
||||
else if (__isset.NANOS && !(NANOS == rhs.NANOS))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
bool operator != (const TimeUnit &rhs) const {
|
||||
@ -867,7 +908,7 @@ void swap(BsonType &a, BsonType &b);
|
||||
std::ostream& operator<<(std::ostream& out, const BsonType& obj);
|
||||
|
||||
typedef struct _LogicalType__isset {
|
||||
_LogicalType__isset() : STRING(false), MAP(false), LIST(false), ENUM(false), DECIMAL(false), DATE(false), TIME(false), TIMESTAMP(false), INTEGER(false), UNKNOWN(false), JSON(false), BSON(false) {}
|
||||
_LogicalType__isset() : STRING(false), MAP(false), LIST(false), ENUM(false), DECIMAL(false), DATE(false), TIME(false), TIMESTAMP(false), INTEGER(false), UNKNOWN(false), JSON(false), BSON(false), UUID(false) {}
|
||||
bool STRING :1;
|
||||
bool MAP :1;
|
||||
bool LIST :1;
|
||||
@ -880,6 +921,7 @@ typedef struct _LogicalType__isset {
|
||||
bool UNKNOWN :1;
|
||||
bool JSON :1;
|
||||
bool BSON :1;
|
||||
bool UUID :1;
|
||||
} _LogicalType__isset;
|
||||
|
||||
class LogicalType : public virtual ::apache::thrift::TBase {
|
||||
@ -903,6 +945,7 @@ class LogicalType : public virtual ::apache::thrift::TBase {
|
||||
NullType UNKNOWN;
|
||||
JsonType JSON;
|
||||
BsonType BSON;
|
||||
UUIDType UUID;
|
||||
|
||||
_LogicalType__isset __isset;
|
||||
|
||||
@ -930,6 +973,8 @@ class LogicalType : public virtual ::apache::thrift::TBase {
|
||||
|
||||
void __set_BSON(const BsonType& val);
|
||||
|
||||
void __set_UUID(const UUIDType& val);
|
||||
|
||||
bool operator == (const LogicalType & rhs) const
|
||||
{
|
||||
if (__isset.STRING != rhs.__isset.STRING)
|
||||
@ -980,6 +1025,10 @@ class LogicalType : public virtual ::apache::thrift::TBase {
|
||||
return false;
|
||||
else if (__isset.BSON && !(BSON == rhs.BSON))
|
||||
return false;
|
||||
if (__isset.UUID != rhs.__isset.UUID)
|
||||
return false;
|
||||
else if (__isset.UUID && !(UUID == rhs.UUID))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
bool operator != (const LogicalType &rhs) const {
|
||||
@ -1722,8 +1771,8 @@ void swap(EncryptionWithFooterKey &a, EncryptionWithFooterKey &b);
|
||||
std::ostream& operator<<(std::ostream& out, const EncryptionWithFooterKey& obj);
|
||||
|
||||
typedef struct _EncryptionWithColumnKey__isset {
|
||||
_EncryptionWithColumnKey__isset() : column_key_metadata(false) {}
|
||||
bool column_key_metadata :1;
|
||||
_EncryptionWithColumnKey__isset() : key_metadata(false) {}
|
||||
bool key_metadata :1;
|
||||
} _EncryptionWithColumnKey__isset;
|
||||
|
||||
class EncryptionWithColumnKey : public virtual ::apache::thrift::TBase {
|
||||
@ -1731,26 +1780,26 @@ class EncryptionWithColumnKey : public virtual ::apache::thrift::TBase {
|
||||
|
||||
EncryptionWithColumnKey(const EncryptionWithColumnKey&);
|
||||
EncryptionWithColumnKey& operator=(const EncryptionWithColumnKey&);
|
||||
EncryptionWithColumnKey() : column_key_metadata() {
|
||||
EncryptionWithColumnKey() : key_metadata() {
|
||||
}
|
||||
|
||||
virtual ~EncryptionWithColumnKey() throw();
|
||||
std::vector<std::string> path_in_schema;
|
||||
std::string column_key_metadata;
|
||||
std::string key_metadata;
|
||||
|
||||
_EncryptionWithColumnKey__isset __isset;
|
||||
|
||||
void __set_path_in_schema(const std::vector<std::string> & val);
|
||||
|
||||
void __set_column_key_metadata(const std::string& val);
|
||||
void __set_key_metadata(const std::string& val);
|
||||
|
||||
bool operator == (const EncryptionWithColumnKey & rhs) const
|
||||
{
|
||||
if (!(path_in_schema == rhs.path_in_schema))
|
||||
return false;
|
||||
if (__isset.column_key_metadata != rhs.__isset.column_key_metadata)
|
||||
if (__isset.key_metadata != rhs.__isset.key_metadata)
|
||||
return false;
|
||||
else if (__isset.column_key_metadata && !(column_key_metadata == rhs.column_key_metadata))
|
||||
else if (__isset.key_metadata && !(key_metadata == rhs.key_metadata))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
@ -1823,14 +1872,15 @@ void swap(ColumnCryptoMetaData &a, ColumnCryptoMetaData &b);
|
||||
std::ostream& operator<<(std::ostream& out, const ColumnCryptoMetaData& obj);
|
||||
|
||||
typedef struct _ColumnChunk__isset {
|
||||
_ColumnChunk__isset() : file_path(false), meta_data(false), offset_index_offset(false), offset_index_length(false), column_index_offset(false), column_index_length(false), crypto_meta_data(false) {}
|
||||
_ColumnChunk__isset() : file_path(false), meta_data(false), offset_index_offset(false), offset_index_length(false), column_index_offset(false), column_index_length(false), crypto_metadata(false), encrypted_column_metadata(false) {}
|
||||
bool file_path :1;
|
||||
bool meta_data :1;
|
||||
bool offset_index_offset :1;
|
||||
bool offset_index_length :1;
|
||||
bool column_index_offset :1;
|
||||
bool column_index_length :1;
|
||||
bool crypto_meta_data :1;
|
||||
bool crypto_metadata :1;
|
||||
bool encrypted_column_metadata :1;
|
||||
} _ColumnChunk__isset;
|
||||
|
||||
class ColumnChunk : public virtual ::apache::thrift::TBase {
|
||||
@ -1838,7 +1888,7 @@ class ColumnChunk : public virtual ::apache::thrift::TBase {
|
||||
|
||||
ColumnChunk(const ColumnChunk&);
|
||||
ColumnChunk& operator=(const ColumnChunk&);
|
||||
ColumnChunk() : file_path(), file_offset(0), offset_index_offset(0), offset_index_length(0), column_index_offset(0), column_index_length(0) {
|
||||
ColumnChunk() : file_path(), file_offset(0), offset_index_offset(0), offset_index_length(0), column_index_offset(0), column_index_length(0), encrypted_column_metadata() {
|
||||
}
|
||||
|
||||
virtual ~ColumnChunk() throw();
|
||||
@ -1849,7 +1899,8 @@ class ColumnChunk : public virtual ::apache::thrift::TBase {
|
||||
int32_t offset_index_length;
|
||||
int64_t column_index_offset;
|
||||
int32_t column_index_length;
|
||||
ColumnCryptoMetaData crypto_meta_data;
|
||||
ColumnCryptoMetaData crypto_metadata;
|
||||
std::string encrypted_column_metadata;
|
||||
|
||||
_ColumnChunk__isset __isset;
|
||||
|
||||
@ -1867,7 +1918,9 @@ class ColumnChunk : public virtual ::apache::thrift::TBase {
|
||||
|
||||
void __set_column_index_length(const int32_t val);
|
||||
|
||||
void __set_crypto_meta_data(const ColumnCryptoMetaData& val);
|
||||
void __set_crypto_metadata(const ColumnCryptoMetaData& val);
|
||||
|
||||
void __set_encrypted_column_metadata(const std::string& val);
|
||||
|
||||
bool operator == (const ColumnChunk & rhs) const
|
||||
{
|
||||
@ -1897,9 +1950,13 @@ class ColumnChunk : public virtual ::apache::thrift::TBase {
|
||||
return false;
|
||||
else if (__isset.column_index_length && !(column_index_length == rhs.column_index_length))
|
||||
return false;
|
||||
if (__isset.crypto_meta_data != rhs.__isset.crypto_meta_data)
|
||||
if (__isset.crypto_metadata != rhs.__isset.crypto_metadata)
|
||||
return false;
|
||||
else if (__isset.crypto_meta_data && !(crypto_meta_data == rhs.crypto_meta_data))
|
||||
else if (__isset.crypto_metadata && !(crypto_metadata == rhs.crypto_metadata))
|
||||
return false;
|
||||
if (__isset.encrypted_column_metadata != rhs.__isset.encrypted_column_metadata)
|
||||
return false;
|
||||
else if (__isset.encrypted_column_metadata && !(encrypted_column_metadata == rhs.encrypted_column_metadata))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
@ -1920,10 +1977,11 @@ void swap(ColumnChunk &a, ColumnChunk &b);
|
||||
std::ostream& operator<<(std::ostream& out, const ColumnChunk& obj);
|
||||
|
||||
typedef struct _RowGroup__isset {
|
||||
_RowGroup__isset() : sorting_columns(false), file_offset(false), total_compressed_size(false) {}
|
||||
_RowGroup__isset() : sorting_columns(false), file_offset(false), total_compressed_size(false), ordinal(false) {}
|
||||
bool sorting_columns :1;
|
||||
bool file_offset :1;
|
||||
bool total_compressed_size :1;
|
||||
bool ordinal :1;
|
||||
} _RowGroup__isset;
|
||||
|
||||
class RowGroup : public virtual ::apache::thrift::TBase {
|
||||
@ -1931,7 +1989,7 @@ class RowGroup : public virtual ::apache::thrift::TBase {
|
||||
|
||||
RowGroup(const RowGroup&);
|
||||
RowGroup& operator=(const RowGroup&);
|
||||
RowGroup() : total_byte_size(0), num_rows(0), file_offset(0), total_compressed_size(0) {
|
||||
RowGroup() : total_byte_size(0), num_rows(0), file_offset(0), total_compressed_size(0), ordinal(0) {
|
||||
}
|
||||
|
||||
virtual ~RowGroup() throw();
|
||||
@ -1941,6 +1999,7 @@ class RowGroup : public virtual ::apache::thrift::TBase {
|
||||
std::vector<SortingColumn> sorting_columns;
|
||||
int64_t file_offset;
|
||||
int64_t total_compressed_size;
|
||||
int16_t ordinal;
|
||||
|
||||
_RowGroup__isset __isset;
|
||||
|
||||
@ -1956,6 +2015,8 @@ class RowGroup : public virtual ::apache::thrift::TBase {
|
||||
|
||||
void __set_total_compressed_size(const int64_t val);
|
||||
|
||||
void __set_ordinal(const int16_t val);
|
||||
|
||||
bool operator == (const RowGroup & rhs) const
|
||||
{
|
||||
if (!(columns == rhs.columns))
|
||||
@ -1976,6 +2037,10 @@ class RowGroup : public virtual ::apache::thrift::TBase {
|
||||
return false;
|
||||
else if (__isset.total_compressed_size && !(total_compressed_size == rhs.total_compressed_size))
|
||||
return false;
|
||||
if (__isset.ordinal != rhs.__isset.ordinal)
|
||||
return false;
|
||||
else if (__isset.ordinal && !(ordinal == rhs.ordinal))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
bool operator != (const RowGroup &rhs) const {
|
||||
@ -2215,90 +2280,11 @@ void swap(ColumnIndex &a, ColumnIndex &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const ColumnIndex& obj);
|
||||
|
||||
typedef struct _FileMetaData__isset {
|
||||
_FileMetaData__isset() : key_value_metadata(false), created_by(false), column_orders(false) {}
|
||||
bool key_value_metadata :1;
|
||||
bool created_by :1;
|
||||
bool column_orders :1;
|
||||
} _FileMetaData__isset;
|
||||
|
||||
class FileMetaData : public virtual ::apache::thrift::TBase {
|
||||
public:
|
||||
|
||||
FileMetaData(const FileMetaData&);
|
||||
FileMetaData& operator=(const FileMetaData&);
|
||||
FileMetaData() : version(0), num_rows(0), created_by() {
|
||||
}
|
||||
|
||||
virtual ~FileMetaData() throw();
|
||||
int32_t version;
|
||||
std::vector<SchemaElement> schema;
|
||||
int64_t num_rows;
|
||||
std::vector<RowGroup> row_groups;
|
||||
std::vector<KeyValue> key_value_metadata;
|
||||
std::string created_by;
|
||||
std::vector<ColumnOrder> column_orders;
|
||||
|
||||
_FileMetaData__isset __isset;
|
||||
|
||||
void __set_version(const int32_t val);
|
||||
|
||||
void __set_schema(const std::vector<SchemaElement> & val);
|
||||
|
||||
void __set_num_rows(const int64_t val);
|
||||
|
||||
void __set_row_groups(const std::vector<RowGroup> & val);
|
||||
|
||||
void __set_key_value_metadata(const std::vector<KeyValue> & val);
|
||||
|
||||
void __set_created_by(const std::string& val);
|
||||
|
||||
void __set_column_orders(const std::vector<ColumnOrder> & val);
|
||||
|
||||
bool operator == (const FileMetaData & rhs) const
|
||||
{
|
||||
if (!(version == rhs.version))
|
||||
return false;
|
||||
if (!(schema == rhs.schema))
|
||||
return false;
|
||||
if (!(num_rows == rhs.num_rows))
|
||||
return false;
|
||||
if (!(row_groups == rhs.row_groups))
|
||||
return false;
|
||||
if (__isset.key_value_metadata != rhs.__isset.key_value_metadata)
|
||||
return false;
|
||||
else if (__isset.key_value_metadata && !(key_value_metadata == rhs.key_value_metadata))
|
||||
return false;
|
||||
if (__isset.created_by != rhs.__isset.created_by)
|
||||
return false;
|
||||
else if (__isset.created_by && !(created_by == rhs.created_by))
|
||||
return false;
|
||||
if (__isset.column_orders != rhs.__isset.column_orders)
|
||||
return false;
|
||||
else if (__isset.column_orders && !(column_orders == rhs.column_orders))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
bool operator != (const FileMetaData &rhs) const {
|
||||
return !(*this == rhs);
|
||||
}
|
||||
|
||||
bool operator < (const FileMetaData & ) const;
|
||||
|
||||
uint32_t read(::apache::thrift::protocol::TProtocol* iprot);
|
||||
uint32_t write(::apache::thrift::protocol::TProtocol* oprot) const;
|
||||
|
||||
virtual void printTo(std::ostream& out) const;
|
||||
};
|
||||
|
||||
void swap(FileMetaData &a, FileMetaData &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const FileMetaData& obj);
|
||||
|
||||
typedef struct _AesGcmV1__isset {
|
||||
_AesGcmV1__isset() : aad_metadata(false), iv_prefix(false) {}
|
||||
bool aad_metadata :1;
|
||||
bool iv_prefix :1;
|
||||
_AesGcmV1__isset() : aad_prefix(false), aad_file_unique(false), supply_aad_prefix(false) {}
|
||||
bool aad_prefix :1;
|
||||
bool aad_file_unique :1;
|
||||
bool supply_aad_prefix :1;
|
||||
} _AesGcmV1__isset;
|
||||
|
||||
class AesGcmV1 : public virtual ::apache::thrift::TBase {
|
||||
@ -2306,28 +2292,35 @@ class AesGcmV1 : public virtual ::apache::thrift::TBase {
|
||||
|
||||
AesGcmV1(const AesGcmV1&);
|
||||
AesGcmV1& operator=(const AesGcmV1&);
|
||||
AesGcmV1() : aad_metadata(), iv_prefix() {
|
||||
AesGcmV1() : aad_prefix(), aad_file_unique(), supply_aad_prefix(0) {
|
||||
}
|
||||
|
||||
virtual ~AesGcmV1() throw();
|
||||
std::string aad_metadata;
|
||||
std::string iv_prefix;
|
||||
std::string aad_prefix;
|
||||
std::string aad_file_unique;
|
||||
bool supply_aad_prefix;
|
||||
|
||||
_AesGcmV1__isset __isset;
|
||||
|
||||
void __set_aad_metadata(const std::string& val);
|
||||
void __set_aad_prefix(const std::string& val);
|
||||
|
||||
void __set_iv_prefix(const std::string& val);
|
||||
void __set_aad_file_unique(const std::string& val);
|
||||
|
||||
void __set_supply_aad_prefix(const bool val);
|
||||
|
||||
bool operator == (const AesGcmV1 & rhs) const
|
||||
{
|
||||
if (__isset.aad_metadata != rhs.__isset.aad_metadata)
|
||||
if (__isset.aad_prefix != rhs.__isset.aad_prefix)
|
||||
return false;
|
||||
else if (__isset.aad_metadata && !(aad_metadata == rhs.aad_metadata))
|
||||
else if (__isset.aad_prefix && !(aad_prefix == rhs.aad_prefix))
|
||||
return false;
|
||||
if (__isset.iv_prefix != rhs.__isset.iv_prefix)
|
||||
if (__isset.aad_file_unique != rhs.__isset.aad_file_unique)
|
||||
return false;
|
||||
else if (__isset.iv_prefix && !(iv_prefix == rhs.iv_prefix))
|
||||
else if (__isset.aad_file_unique && !(aad_file_unique == rhs.aad_file_unique))
|
||||
return false;
|
||||
if (__isset.supply_aad_prefix != rhs.__isset.supply_aad_prefix)
|
||||
return false;
|
||||
else if (__isset.supply_aad_prefix && !(supply_aad_prefix == rhs.supply_aad_prefix))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
@ -2348,10 +2341,10 @@ void swap(AesGcmV1 &a, AesGcmV1 &b);
|
||||
std::ostream& operator<<(std::ostream& out, const AesGcmV1& obj);
|
||||
|
||||
typedef struct _AesGcmCtrV1__isset {
|
||||
_AesGcmCtrV1__isset() : aad_metadata(false), gcm_iv_prefix(false), ctr_iv_prefix(false) {}
|
||||
bool aad_metadata :1;
|
||||
bool gcm_iv_prefix :1;
|
||||
bool ctr_iv_prefix :1;
|
||||
_AesGcmCtrV1__isset() : aad_prefix(false), aad_file_unique(false), supply_aad_prefix(false) {}
|
||||
bool aad_prefix :1;
|
||||
bool aad_file_unique :1;
|
||||
bool supply_aad_prefix :1;
|
||||
} _AesGcmCtrV1__isset;
|
||||
|
||||
class AesGcmCtrV1 : public virtual ::apache::thrift::TBase {
|
||||
@ -2359,35 +2352,35 @@ class AesGcmCtrV1 : public virtual ::apache::thrift::TBase {
|
||||
|
||||
AesGcmCtrV1(const AesGcmCtrV1&);
|
||||
AesGcmCtrV1& operator=(const AesGcmCtrV1&);
|
||||
AesGcmCtrV1() : aad_metadata(), gcm_iv_prefix(), ctr_iv_prefix() {
|
||||
AesGcmCtrV1() : aad_prefix(), aad_file_unique(), supply_aad_prefix(0) {
|
||||
}
|
||||
|
||||
virtual ~AesGcmCtrV1() throw();
|
||||
std::string aad_metadata;
|
||||
std::string gcm_iv_prefix;
|
||||
std::string ctr_iv_prefix;
|
||||
std::string aad_prefix;
|
||||
std::string aad_file_unique;
|
||||
bool supply_aad_prefix;
|
||||
|
||||
_AesGcmCtrV1__isset __isset;
|
||||
|
||||
void __set_aad_metadata(const std::string& val);
|
||||
void __set_aad_prefix(const std::string& val);
|
||||
|
||||
void __set_gcm_iv_prefix(const std::string& val);
|
||||
void __set_aad_file_unique(const std::string& val);
|
||||
|
||||
void __set_ctr_iv_prefix(const std::string& val);
|
||||
void __set_supply_aad_prefix(const bool val);
|
||||
|
||||
bool operator == (const AesGcmCtrV1 & rhs) const
|
||||
{
|
||||
if (__isset.aad_metadata != rhs.__isset.aad_metadata)
|
||||
if (__isset.aad_prefix != rhs.__isset.aad_prefix)
|
||||
return false;
|
||||
else if (__isset.aad_metadata && !(aad_metadata == rhs.aad_metadata))
|
||||
else if (__isset.aad_prefix && !(aad_prefix == rhs.aad_prefix))
|
||||
return false;
|
||||
if (__isset.gcm_iv_prefix != rhs.__isset.gcm_iv_prefix)
|
||||
if (__isset.aad_file_unique != rhs.__isset.aad_file_unique)
|
||||
return false;
|
||||
else if (__isset.gcm_iv_prefix && !(gcm_iv_prefix == rhs.gcm_iv_prefix))
|
||||
else if (__isset.aad_file_unique && !(aad_file_unique == rhs.aad_file_unique))
|
||||
return false;
|
||||
if (__isset.ctr_iv_prefix != rhs.__isset.ctr_iv_prefix)
|
||||
if (__isset.supply_aad_prefix != rhs.__isset.supply_aad_prefix)
|
||||
return false;
|
||||
else if (__isset.ctr_iv_prefix && !(ctr_iv_prefix == rhs.ctr_iv_prefix))
|
||||
else if (__isset.supply_aad_prefix && !(supply_aad_prefix == rhs.supply_aad_prefix))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
@ -2459,9 +2452,105 @@ void swap(EncryptionAlgorithm &a, EncryptionAlgorithm &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const EncryptionAlgorithm& obj);
|
||||
|
||||
typedef struct _FileMetaData__isset {
|
||||
_FileMetaData__isset() : key_value_metadata(false), created_by(false), column_orders(false), encryption_algorithm(false), footer_signing_key_metadata(false) {}
|
||||
bool key_value_metadata :1;
|
||||
bool created_by :1;
|
||||
bool column_orders :1;
|
||||
bool encryption_algorithm :1;
|
||||
bool footer_signing_key_metadata :1;
|
||||
} _FileMetaData__isset;
|
||||
|
||||
class FileMetaData : public virtual ::apache::thrift::TBase {
|
||||
public:
|
||||
|
||||
FileMetaData(const FileMetaData&);
|
||||
FileMetaData& operator=(const FileMetaData&);
|
||||
FileMetaData() : version(0), num_rows(0), created_by(), footer_signing_key_metadata() {
|
||||
}
|
||||
|
||||
virtual ~FileMetaData() throw();
|
||||
int32_t version;
|
||||
std::vector<SchemaElement> schema;
|
||||
int64_t num_rows;
|
||||
std::vector<RowGroup> row_groups;
|
||||
std::vector<KeyValue> key_value_metadata;
|
||||
std::string created_by;
|
||||
std::vector<ColumnOrder> column_orders;
|
||||
EncryptionAlgorithm encryption_algorithm;
|
||||
std::string footer_signing_key_metadata;
|
||||
|
||||
_FileMetaData__isset __isset;
|
||||
|
||||
void __set_version(const int32_t val);
|
||||
|
||||
void __set_schema(const std::vector<SchemaElement> & val);
|
||||
|
||||
void __set_num_rows(const int64_t val);
|
||||
|
||||
void __set_row_groups(const std::vector<RowGroup> & val);
|
||||
|
||||
void __set_key_value_metadata(const std::vector<KeyValue> & val);
|
||||
|
||||
void __set_created_by(const std::string& val);
|
||||
|
||||
void __set_column_orders(const std::vector<ColumnOrder> & val);
|
||||
|
||||
void __set_encryption_algorithm(const EncryptionAlgorithm& val);
|
||||
|
||||
void __set_footer_signing_key_metadata(const std::string& val);
|
||||
|
||||
bool operator == (const FileMetaData & rhs) const
|
||||
{
|
||||
if (!(version == rhs.version))
|
||||
return false;
|
||||
if (!(schema == rhs.schema))
|
||||
return false;
|
||||
if (!(num_rows == rhs.num_rows))
|
||||
return false;
|
||||
if (!(row_groups == rhs.row_groups))
|
||||
return false;
|
||||
if (__isset.key_value_metadata != rhs.__isset.key_value_metadata)
|
||||
return false;
|
||||
else if (__isset.key_value_metadata && !(key_value_metadata == rhs.key_value_metadata))
|
||||
return false;
|
||||
if (__isset.created_by != rhs.__isset.created_by)
|
||||
return false;
|
||||
else if (__isset.created_by && !(created_by == rhs.created_by))
|
||||
return false;
|
||||
if (__isset.column_orders != rhs.__isset.column_orders)
|
||||
return false;
|
||||
else if (__isset.column_orders && !(column_orders == rhs.column_orders))
|
||||
return false;
|
||||
if (__isset.encryption_algorithm != rhs.__isset.encryption_algorithm)
|
||||
return false;
|
||||
else if (__isset.encryption_algorithm && !(encryption_algorithm == rhs.encryption_algorithm))
|
||||
return false;
|
||||
if (__isset.footer_signing_key_metadata != rhs.__isset.footer_signing_key_metadata)
|
||||
return false;
|
||||
else if (__isset.footer_signing_key_metadata && !(footer_signing_key_metadata == rhs.footer_signing_key_metadata))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
bool operator != (const FileMetaData &rhs) const {
|
||||
return !(*this == rhs);
|
||||
}
|
||||
|
||||
bool operator < (const FileMetaData & ) const;
|
||||
|
||||
uint32_t read(::apache::thrift::protocol::TProtocol* iprot);
|
||||
uint32_t write(::apache::thrift::protocol::TProtocol* oprot) const;
|
||||
|
||||
virtual void printTo(std::ostream& out) const;
|
||||
};
|
||||
|
||||
void swap(FileMetaData &a, FileMetaData &b);
|
||||
|
||||
std::ostream& operator<<(std::ostream& out, const FileMetaData& obj);
|
||||
|
||||
typedef struct _FileCryptoMetaData__isset {
|
||||
_FileCryptoMetaData__isset() : footer_key_metadata(false) {}
|
||||
bool footer_key_metadata :1;
|
||||
_FileCryptoMetaData__isset() : key_metadata(false) {}
|
||||
bool key_metadata :1;
|
||||
} _FileCryptoMetaData__isset;
|
||||
|
||||
class FileCryptoMetaData : public virtual ::apache::thrift::TBase {
|
||||
@ -2469,36 +2558,26 @@ class FileCryptoMetaData : public virtual ::apache::thrift::TBase {
|
||||
|
||||
FileCryptoMetaData(const FileCryptoMetaData&);
|
||||
FileCryptoMetaData& operator=(const FileCryptoMetaData&);
|
||||
FileCryptoMetaData() : encrypted_footer(0), footer_key_metadata(), footer_offset(0) {
|
||||
FileCryptoMetaData() : key_metadata() {
|
||||
}
|
||||
|
||||
virtual ~FileCryptoMetaData() throw();
|
||||
EncryptionAlgorithm encryption_algorithm;
|
||||
bool encrypted_footer;
|
||||
std::string footer_key_metadata;
|
||||
int64_t footer_offset;
|
||||
std::string key_metadata;
|
||||
|
||||
_FileCryptoMetaData__isset __isset;
|
||||
|
||||
void __set_encryption_algorithm(const EncryptionAlgorithm& val);
|
||||
|
||||
void __set_encrypted_footer(const bool val);
|
||||
|
||||
void __set_footer_key_metadata(const std::string& val);
|
||||
|
||||
void __set_footer_offset(const int64_t val);
|
||||
void __set_key_metadata(const std::string& val);
|
||||
|
||||
bool operator == (const FileCryptoMetaData & rhs) const
|
||||
{
|
||||
if (!(encryption_algorithm == rhs.encryption_algorithm))
|
||||
return false;
|
||||
if (!(encrypted_footer == rhs.encrypted_footer))
|
||||
if (__isset.key_metadata != rhs.__isset.key_metadata)
|
||||
return false;
|
||||
if (__isset.footer_key_metadata != rhs.__isset.footer_key_metadata)
|
||||
return false;
|
||||
else if (__isset.footer_key_metadata && !(footer_key_metadata == rhs.footer_key_metadata))
|
||||
return false;
|
||||
if (!(footer_offset == rhs.footer_offset))
|
||||
else if (__isset.key_metadata && !(key_metadata == rhs.key_metadata))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -40,30 +40,30 @@ file(GLOB AWS_CORE_SOURCES
|
||||
"${AWS_CORE_LIBRARY_DIR}/source/utils/stream/*.cpp"
|
||||
"${AWS_CORE_LIBRARY_DIR}/source/utils/threading/*.cpp"
|
||||
"${AWS_CORE_LIBRARY_DIR}/source/utils/xml/*.cpp"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB AWS_S3_SOURCES
|
||||
"${AWS_S3_LIBRARY_DIR}/source/*.cpp"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB AWS_S3_MODEL_SOURCES
|
||||
"${AWS_S3_LIBRARY_DIR}/source/model/*.cpp"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB AWS_EVENT_STREAM_SOURCES
|
||||
"${AWS_EVENT_STREAM_LIBRARY_DIR}/source/*.c"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB AWS_COMMON_SOURCES
|
||||
"${AWS_COMMON_LIBRARY_DIR}/source/*.c"
|
||||
"${AWS_COMMON_LIBRARY_DIR}/source/posix/*.c"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB AWS_CHECKSUMS_SOURCES
|
||||
"${AWS_CHECKSUMS_LIBRARY_DIR}/source/*.c"
|
||||
"${AWS_CHECKSUMS_LIBRARY_DIR}/source/intel/*.c"
|
||||
"${AWS_CHECKSUMS_LIBRARY_DIR}/source/arm/*.c"
|
||||
)
|
||||
)
|
||||
|
||||
file(GLOB S3_UNIFIED_SRC
|
||||
${AWS_EVENT_STREAM_SOURCES}
|
||||
@ -71,7 +71,7 @@ file(GLOB S3_UNIFIED_SRC
|
||||
${AWS_S3_SOURCES}
|
||||
${AWS_S3_MODEL_SOURCES}
|
||||
${AWS_CORE_SOURCES}
|
||||
)
|
||||
)
|
||||
|
||||
set(S3_INCLUDES
|
||||
"${CMAKE_CURRENT_SOURCE_DIR}/include/"
|
||||
@ -80,14 +80,13 @@ set(S3_INCLUDES
|
||||
"${AWS_S3_LIBRARY_DIR}/include/"
|
||||
"${AWS_CORE_LIBRARY_DIR}/include/"
|
||||
"${CMAKE_CURRENT_BINARY_DIR}/include/"
|
||||
)
|
||||
)
|
||||
|
||||
add_library(aws_s3_checksums ${AWS_CHECKSUMS_SOURCES})
|
||||
target_include_directories(aws_s3_checksums PUBLIC "${AWS_CHECKSUMS_LIBRARY_DIR}/include/")
|
||||
if(CMAKE_BUILD_TYPE STREQUAL "" OR CMAKE_BUILD_TYPE STREQUAL "Debug")
|
||||
target_compile_definitions(aws_s3_checksums PRIVATE "-DDEBUG_BUILD")
|
||||
endif()
|
||||
set_target_properties(aws_s3_checksums PROPERTIES COMPILE_OPTIONS -fPIC)
|
||||
set_target_properties(aws_s3_checksums PROPERTIES LINKER_LANGUAGE C)
|
||||
set_property(TARGET aws_s3_checksums PROPERTY C_STANDARD 99)
|
||||
|
||||
|
||||
2
contrib/base64
vendored
2
contrib/base64
vendored
@ -1 +1 @@
|
||||
Subproject commit a27c565d1b6c676beaf297fe503c4518185666f7
|
||||
Subproject commit 32a87227aa254e2f02332c0ed8bc375522414fa3
|
||||
@ -37,7 +37,8 @@ add_library(base64
|
||||
${LIBRARY_DIR}/lib/arch/ssse3/codec.c
|
||||
|
||||
${LIBRARY_DIR}/lib/codecs.h
|
||||
${CMAKE_CURRENT_BINARY_DIR}/config.h)
|
||||
${CMAKE_CURRENT_BINARY_DIR}/config.h
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/tables.h)
|
||||
|
||||
if(HAVE_AVX)
|
||||
set_source_files_properties(${LIBRARY_DIR}/lib/arch/avx/codec.c PROPERTIES COMPILE_FLAGS -mavx)
|
||||
@ -55,4 +56,4 @@ if(HAVE_SSSE3)
|
||||
set_source_files_properties(${LIBRARY_DIR}/lib/arch/ssse3/codec.c PROPERTIES COMPILE_FLAGS -mssse3)
|
||||
endif()
|
||||
|
||||
target_include_directories(base64 PRIVATE ${LIBRARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR})
|
||||
target_include_directories(base64 PRIVATE ${LIBRARY_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
|
||||
|
||||
5
contrib/base64-cmake/README.md
Normal file
5
contrib/base64-cmake/README.md
Normal file
@ -0,0 +1,5 @@
|
||||
`tables.h` file was generated by the base64 library with it's makefile:
|
||||
```
|
||||
cc -std=c99 -O3 -Wall -Wextra -pedantic -o lib/table_generator lib/table_generator.c
|
||||
./lib/table_generator > lib/tables.h
|
||||
```
|
||||
393
contrib/base64-cmake/tables.h
Normal file
393
contrib/base64-cmake/tables.h
Normal file
@ -0,0 +1,393 @@
|
||||
#include <stdint.h>
|
||||
#define CHAR62 '+'
|
||||
#define CHAR63 '/'
|
||||
#define CHARPAD '='
|
||||
|
||||
|
||||
#if BASE64_LITTLE_ENDIAN
|
||||
|
||||
|
||||
/* SPECIAL DECODE TABLES FOR LITTLE ENDIAN (INTEL) CPUS */
|
||||
|
||||
const uint32_t base64_table_dec_d0[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x000000f8, 0xffffffff, 0xffffffff, 0xffffffff, 0x000000fc,
|
||||
0x000000d0, 0x000000d4, 0x000000d8, 0x000000dc, 0x000000e0, 0x000000e4,
|
||||
0x000000e8, 0x000000ec, 0x000000f0, 0x000000f4, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00000004, 0x00000008, 0x0000000c, 0x00000010, 0x00000014, 0x00000018,
|
||||
0x0000001c, 0x00000020, 0x00000024, 0x00000028, 0x0000002c, 0x00000030,
|
||||
0x00000034, 0x00000038, 0x0000003c, 0x00000040, 0x00000044, 0x00000048,
|
||||
0x0000004c, 0x00000050, 0x00000054, 0x00000058, 0x0000005c, 0x00000060,
|
||||
0x00000064, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00000068, 0x0000006c, 0x00000070, 0x00000074, 0x00000078,
|
||||
0x0000007c, 0x00000080, 0x00000084, 0x00000088, 0x0000008c, 0x00000090,
|
||||
0x00000094, 0x00000098, 0x0000009c, 0x000000a0, 0x000000a4, 0x000000a8,
|
||||
0x000000ac, 0x000000b0, 0x000000b4, 0x000000b8, 0x000000bc, 0x000000c0,
|
||||
0x000000c4, 0x000000c8, 0x000000cc, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d1[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x0000e003, 0xffffffff, 0xffffffff, 0xffffffff, 0x0000f003,
|
||||
0x00004003, 0x00005003, 0x00006003, 0x00007003, 0x00008003, 0x00009003,
|
||||
0x0000a003, 0x0000b003, 0x0000c003, 0x0000d003, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00001000, 0x00002000, 0x00003000, 0x00004000, 0x00005000, 0x00006000,
|
||||
0x00007000, 0x00008000, 0x00009000, 0x0000a000, 0x0000b000, 0x0000c000,
|
||||
0x0000d000, 0x0000e000, 0x0000f000, 0x00000001, 0x00001001, 0x00002001,
|
||||
0x00003001, 0x00004001, 0x00005001, 0x00006001, 0x00007001, 0x00008001,
|
||||
0x00009001, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x0000a001, 0x0000b001, 0x0000c001, 0x0000d001, 0x0000e001,
|
||||
0x0000f001, 0x00000002, 0x00001002, 0x00002002, 0x00003002, 0x00004002,
|
||||
0x00005002, 0x00006002, 0x00007002, 0x00008002, 0x00009002, 0x0000a002,
|
||||
0x0000b002, 0x0000c002, 0x0000d002, 0x0000e002, 0x0000f002, 0x00000003,
|
||||
0x00001003, 0x00002003, 0x00003003, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d2[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00800f00, 0xffffffff, 0xffffffff, 0xffffffff, 0x00c00f00,
|
||||
0x00000d00, 0x00400d00, 0x00800d00, 0x00c00d00, 0x00000e00, 0x00400e00,
|
||||
0x00800e00, 0x00c00e00, 0x00000f00, 0x00400f00, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00400000, 0x00800000, 0x00c00000, 0x00000100, 0x00400100, 0x00800100,
|
||||
0x00c00100, 0x00000200, 0x00400200, 0x00800200, 0x00c00200, 0x00000300,
|
||||
0x00400300, 0x00800300, 0x00c00300, 0x00000400, 0x00400400, 0x00800400,
|
||||
0x00c00400, 0x00000500, 0x00400500, 0x00800500, 0x00c00500, 0x00000600,
|
||||
0x00400600, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00800600, 0x00c00600, 0x00000700, 0x00400700, 0x00800700,
|
||||
0x00c00700, 0x00000800, 0x00400800, 0x00800800, 0x00c00800, 0x00000900,
|
||||
0x00400900, 0x00800900, 0x00c00900, 0x00000a00, 0x00400a00, 0x00800a00,
|
||||
0x00c00a00, 0x00000b00, 0x00400b00, 0x00800b00, 0x00c00b00, 0x00000c00,
|
||||
0x00400c00, 0x00800c00, 0x00c00c00, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d3[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x003e0000, 0xffffffff, 0xffffffff, 0xffffffff, 0x003f0000,
|
||||
0x00340000, 0x00350000, 0x00360000, 0x00370000, 0x00380000, 0x00390000,
|
||||
0x003a0000, 0x003b0000, 0x003c0000, 0x003d0000, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00010000, 0x00020000, 0x00030000, 0x00040000, 0x00050000, 0x00060000,
|
||||
0x00070000, 0x00080000, 0x00090000, 0x000a0000, 0x000b0000, 0x000c0000,
|
||||
0x000d0000, 0x000e0000, 0x000f0000, 0x00100000, 0x00110000, 0x00120000,
|
||||
0x00130000, 0x00140000, 0x00150000, 0x00160000, 0x00170000, 0x00180000,
|
||||
0x00190000, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x001a0000, 0x001b0000, 0x001c0000, 0x001d0000, 0x001e0000,
|
||||
0x001f0000, 0x00200000, 0x00210000, 0x00220000, 0x00230000, 0x00240000,
|
||||
0x00250000, 0x00260000, 0x00270000, 0x00280000, 0x00290000, 0x002a0000,
|
||||
0x002b0000, 0x002c0000, 0x002d0000, 0x002e0000, 0x002f0000, 0x00300000,
|
||||
0x00310000, 0x00320000, 0x00330000, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
#else
|
||||
|
||||
|
||||
/* SPECIAL DECODE TABLES FOR BIG ENDIAN (IBM/MOTOROLA/SUN) CPUS */
|
||||
|
||||
const uint32_t base64_table_dec_d0[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xf8000000, 0xffffffff, 0xffffffff, 0xffffffff, 0xfc000000,
|
||||
0xd0000000, 0xd4000000, 0xd8000000, 0xdc000000, 0xe0000000, 0xe4000000,
|
||||
0xe8000000, 0xec000000, 0xf0000000, 0xf4000000, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x04000000, 0x08000000, 0x0c000000, 0x10000000, 0x14000000, 0x18000000,
|
||||
0x1c000000, 0x20000000, 0x24000000, 0x28000000, 0x2c000000, 0x30000000,
|
||||
0x34000000, 0x38000000, 0x3c000000, 0x40000000, 0x44000000, 0x48000000,
|
||||
0x4c000000, 0x50000000, 0x54000000, 0x58000000, 0x5c000000, 0x60000000,
|
||||
0x64000000, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x68000000, 0x6c000000, 0x70000000, 0x74000000, 0x78000000,
|
||||
0x7c000000, 0x80000000, 0x84000000, 0x88000000, 0x8c000000, 0x90000000,
|
||||
0x94000000, 0x98000000, 0x9c000000, 0xa0000000, 0xa4000000, 0xa8000000,
|
||||
0xac000000, 0xb0000000, 0xb4000000, 0xb8000000, 0xbc000000, 0xc0000000,
|
||||
0xc4000000, 0xc8000000, 0xcc000000, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d1[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x03e00000, 0xffffffff, 0xffffffff, 0xffffffff, 0x03f00000,
|
||||
0x03400000, 0x03500000, 0x03600000, 0x03700000, 0x03800000, 0x03900000,
|
||||
0x03a00000, 0x03b00000, 0x03c00000, 0x03d00000, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00100000, 0x00200000, 0x00300000, 0x00400000, 0x00500000, 0x00600000,
|
||||
0x00700000, 0x00800000, 0x00900000, 0x00a00000, 0x00b00000, 0x00c00000,
|
||||
0x00d00000, 0x00e00000, 0x00f00000, 0x01000000, 0x01100000, 0x01200000,
|
||||
0x01300000, 0x01400000, 0x01500000, 0x01600000, 0x01700000, 0x01800000,
|
||||
0x01900000, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x01a00000, 0x01b00000, 0x01c00000, 0x01d00000, 0x01e00000,
|
||||
0x01f00000, 0x02000000, 0x02100000, 0x02200000, 0x02300000, 0x02400000,
|
||||
0x02500000, 0x02600000, 0x02700000, 0x02800000, 0x02900000, 0x02a00000,
|
||||
0x02b00000, 0x02c00000, 0x02d00000, 0x02e00000, 0x02f00000, 0x03000000,
|
||||
0x03100000, 0x03200000, 0x03300000, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d2[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x000f8000, 0xffffffff, 0xffffffff, 0xffffffff, 0x000fc000,
|
||||
0x000d0000, 0x000d4000, 0x000d8000, 0x000dc000, 0x000e0000, 0x000e4000,
|
||||
0x000e8000, 0x000ec000, 0x000f0000, 0x000f4000, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00004000, 0x00008000, 0x0000c000, 0x00010000, 0x00014000, 0x00018000,
|
||||
0x0001c000, 0x00020000, 0x00024000, 0x00028000, 0x0002c000, 0x00030000,
|
||||
0x00034000, 0x00038000, 0x0003c000, 0x00040000, 0x00044000, 0x00048000,
|
||||
0x0004c000, 0x00050000, 0x00054000, 0x00058000, 0x0005c000, 0x00060000,
|
||||
0x00064000, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00068000, 0x0006c000, 0x00070000, 0x00074000, 0x00078000,
|
||||
0x0007c000, 0x00080000, 0x00084000, 0x00088000, 0x0008c000, 0x00090000,
|
||||
0x00094000, 0x00098000, 0x0009c000, 0x000a0000, 0x000a4000, 0x000a8000,
|
||||
0x000ac000, 0x000b0000, 0x000b4000, 0x000b8000, 0x000bc000, 0x000c0000,
|
||||
0x000c4000, 0x000c8000, 0x000cc000, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
const uint32_t base64_table_dec_d3[256] = {
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00003e00, 0xffffffff, 0xffffffff, 0xffffffff, 0x00003f00,
|
||||
0x00003400, 0x00003500, 0x00003600, 0x00003700, 0x00003800, 0x00003900,
|
||||
0x00003a00, 0x00003b00, 0x00003c00, 0x00003d00, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000,
|
||||
0x00000100, 0x00000200, 0x00000300, 0x00000400, 0x00000500, 0x00000600,
|
||||
0x00000700, 0x00000800, 0x00000900, 0x00000a00, 0x00000b00, 0x00000c00,
|
||||
0x00000d00, 0x00000e00, 0x00000f00, 0x00001000, 0x00001100, 0x00001200,
|
||||
0x00001300, 0x00001400, 0x00001500, 0x00001600, 0x00001700, 0x00001800,
|
||||
0x00001900, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0x00001a00, 0x00001b00, 0x00001c00, 0x00001d00, 0x00001e00,
|
||||
0x00001f00, 0x00002000, 0x00002100, 0x00002200, 0x00002300, 0x00002400,
|
||||
0x00002500, 0x00002600, 0x00002700, 0x00002800, 0x00002900, 0x00002a00,
|
||||
0x00002b00, 0x00002c00, 0x00002d00, 0x00002e00, 0x00002f00, 0x00003000,
|
||||
0x00003100, 0x00003200, 0x00003300, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff,
|
||||
0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
|
||||
};
|
||||
|
||||
|
||||
#endif
|
||||
@ -1,5 +1,7 @@
|
||||
set (CAPNPROTO_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/capnproto/c++/src)
|
||||
|
||||
set (CMAKE_CXX_STANDARD 17)
|
||||
|
||||
set (KJ_SRCS
|
||||
${CAPNPROTO_SOURCE_DIR}/kj/array.c++
|
||||
${CAPNPROTO_SOURCE_DIR}/kj/common.c++
|
||||
|
||||
@ -65,11 +65,6 @@ if(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_CLANG)
|
||||
endif()
|
||||
endif()
|
||||
|
||||
# For debug libs and exes, add "-d" postfix
|
||||
if(NOT DEFINED CMAKE_DEBUG_POSTFIX)
|
||||
set(CMAKE_DEBUG_POSTFIX "-d")
|
||||
endif()
|
||||
|
||||
# initialize CURL_LIBS
|
||||
set(CURL_LIBS "")
|
||||
|
||||
@ -115,8 +110,6 @@ if(ENABLE_IPV6 AND NOT WIN32)
|
||||
endif()
|
||||
endif()
|
||||
|
||||
curl_nroff_check()
|
||||
|
||||
# We need ansi c-flags, especially on HP
|
||||
set(CMAKE_C_FLAGS "${CMAKE_ANSI_CFLAGS} ${CMAKE_C_FLAGS}")
|
||||
set(CMAKE_REQUIRED_FLAGS ${CMAKE_ANSI_CFLAGS})
|
||||
@ -132,21 +125,21 @@ include(CheckCSourceCompiles)
|
||||
|
||||
if(ENABLE_THREADED_RESOLVER)
|
||||
find_package(Threads REQUIRED)
|
||||
if(WIN32)
|
||||
set(USE_THREADS_WIN32 ON)
|
||||
else()
|
||||
set(USE_THREADS_POSIX ${CMAKE_USE_PTHREADS_INIT})
|
||||
set(HAVE_PTHREAD_H ${CMAKE_USE_PTHREADS_INIT})
|
||||
endif()
|
||||
set(CURL_LIBS ${CURL_LIBS} ${CMAKE_THREAD_LIBS_INIT})
|
||||
endif()
|
||||
|
||||
# Check for all needed libraries
|
||||
check_library_exists_concat("${CMAKE_DL_LIBS}" dlopen HAVE_LIBDL)
|
||||
check_library_exists_concat("socket" connect HAVE_LIBSOCKET)
|
||||
check_library_exists("c" gethostbyname "" NOT_NEED_LIBNSL)
|
||||
|
||||
check_function_exists(gethostname HAVE_GETHOSTNAME)
|
||||
# We don't want any plugin loading at runtime. It is harmful.
|
||||
#check_library_exists_concat("${CMAKE_DL_LIBS}" dlopen HAVE_LIBDL)
|
||||
|
||||
# This is unneeded.
|
||||
#check_library_exists_concat("socket" connect HAVE_LIBSOCKET)
|
||||
|
||||
set (NOT_NEED_LIBNSL 1)
|
||||
set (gethostname HAVE_GETHOSTNAME 1)
|
||||
|
||||
# From cmake/find/ssl.cmake
|
||||
if (OPENSSL_FOUND)
|
||||
@ -167,10 +160,12 @@ if (OPENSSL_FOUND)
|
||||
endif()
|
||||
|
||||
# Check for idn
|
||||
check_library_exists_concat("idn2" idn2_lookup_ul HAVE_LIBIDN2)
|
||||
# No, we don't need that.
|
||||
# check_library_exists_concat("idn2" idn2_lookup_ul HAVE_LIBIDN2)
|
||||
|
||||
# Check for symbol dlopen (same as HAVE_LIBDL)
|
||||
check_library_exists("${CURL_LIBS}" dlopen "" HAVE_DLOPEN)
|
||||
# We don't want any plugin loading at runtime. It is harmful.
|
||||
# check_library_exists("${CURL_LIBS}" dlopen "" HAVE_DLOPEN)
|
||||
|
||||
# From /cmake/find/zlib.cmake
|
||||
if (ZLIB_FOUND)
|
||||
@ -181,7 +176,7 @@ if (ZLIB_FOUND)
|
||||
list(APPEND CURL_LIBS ${ZLIB_LIBRARIES})
|
||||
endif()
|
||||
|
||||
option(ENABLE_UNIX_SOCKETS "Define if you want Unix domain sockets support" ON)
|
||||
option(ENABLE_UNIX_SOCKETS "Define if you want Unix domain sockets support" OFF)
|
||||
if(ENABLE_UNIX_SOCKETS)
|
||||
include(CheckStructHasMember)
|
||||
check_struct_has_member("struct sockaddr_un" sun_path "sys/un.h" USE_UNIX_SOCKETS)
|
||||
@ -217,14 +212,14 @@ check_include_file_concat("sys/utime.h" HAVE_SYS_UTIME_H)
|
||||
check_include_file_concat("sys/xattr.h" HAVE_SYS_XATTR_H)
|
||||
check_include_file_concat("alloca.h" HAVE_ALLOCA_H)
|
||||
check_include_file_concat("arpa/inet.h" HAVE_ARPA_INET_H)
|
||||
check_include_file_concat("arpa/tftp.h" HAVE_ARPA_TFTP_H)
|
||||
#check_include_file_concat("arpa/tftp.h" HAVE_ARPA_TFTP_H)
|
||||
check_include_file_concat("assert.h" HAVE_ASSERT_H)
|
||||
check_include_file_concat("crypto.h" HAVE_CRYPTO_H)
|
||||
check_include_file_concat("des.h" HAVE_DES_H)
|
||||
check_include_file_concat("err.h" HAVE_ERR_H)
|
||||
check_include_file_concat("errno.h" HAVE_ERRNO_H)
|
||||
check_include_file_concat("fcntl.h" HAVE_FCNTL_H)
|
||||
check_include_file_concat("idn2.h" HAVE_IDN2_H)
|
||||
#check_include_file_concat("idn2.h" HAVE_IDN2_H)
|
||||
check_include_file_concat("ifaddrs.h" HAVE_IFADDRS_H)
|
||||
check_include_file_concat("io.h" HAVE_IO_H)
|
||||
check_include_file_concat("krb.h" HAVE_KRB_H)
|
||||
@ -259,7 +254,7 @@ check_include_file_concat("x509.h" HAVE_X509_H)
|
||||
|
||||
check_include_file_concat("process.h" HAVE_PROCESS_H)
|
||||
check_include_file_concat("stddef.h" HAVE_STDDEF_H)
|
||||
check_include_file_concat("dlfcn.h" HAVE_DLFCN_H)
|
||||
#check_include_file_concat("dlfcn.h" HAVE_DLFCN_H)
|
||||
check_include_file_concat("malloc.h" HAVE_MALLOC_H)
|
||||
check_include_file_concat("memory.h" HAVE_MEMORY_H)
|
||||
check_include_file_concat("netinet/if_ether.h" HAVE_NETINET_IF_ETHER_H)
|
||||
@ -276,30 +271,11 @@ check_type_size("int" SIZEOF_INT)
|
||||
check_type_size("__int64" SIZEOF___INT64)
|
||||
check_type_size("long double" SIZEOF_LONG_DOUBLE)
|
||||
check_type_size("time_t" SIZEOF_TIME_T)
|
||||
if(NOT HAVE_SIZEOF_SSIZE_T)
|
||||
if(SIZEOF_LONG EQUAL SIZEOF_SIZE_T)
|
||||
set(ssize_t long)
|
||||
endif()
|
||||
if(NOT ssize_t AND SIZEOF___INT64 EQUAL SIZEOF_SIZE_T)
|
||||
set(ssize_t __int64)
|
||||
endif()
|
||||
endif()
|
||||
# off_t is sized later, after the HAVE_FILE_OFFSET_BITS test
|
||||
|
||||
if(HAVE_SIZEOF_LONG_LONG)
|
||||
set(HAVE_LONGLONG 1)
|
||||
set(HAVE_LL 1)
|
||||
endif()
|
||||
set(HAVE_LONGLONG 1)
|
||||
set(HAVE_LL 1)
|
||||
|
||||
find_file(RANDOM_FILE urandom /dev)
|
||||
mark_as_advanced(RANDOM_FILE)
|
||||
|
||||
# Check for some functions that are used
|
||||
if(HAVE_LIBWS2_32)
|
||||
set(CMAKE_REQUIRED_LIBRARIES ws2_32)
|
||||
elseif(HAVE_LIBSOCKET)
|
||||
set(CMAKE_REQUIRED_LIBRARIES socket)
|
||||
endif()
|
||||
set(RANDOM_FILE /dev/urandom)
|
||||
|
||||
check_symbol_exists(basename "${CURL_INCLUDES}" HAVE_BASENAME)
|
||||
check_symbol_exists(socket "${CURL_INCLUDES}" HAVE_SOCKET)
|
||||
@ -311,18 +287,15 @@ check_symbol_exists(strtok_r "${CURL_INCLUDES}" HAVE_STRTOK_R)
|
||||
check_symbol_exists(strftime "${CURL_INCLUDES}" HAVE_STRFTIME)
|
||||
check_symbol_exists(uname "${CURL_INCLUDES}" HAVE_UNAME)
|
||||
check_symbol_exists(strcasecmp "${CURL_INCLUDES}" HAVE_STRCASECMP)
|
||||
check_symbol_exists(stricmp "${CURL_INCLUDES}" HAVE_STRICMP)
|
||||
check_symbol_exists(strcmpi "${CURL_INCLUDES}" HAVE_STRCMPI)
|
||||
check_symbol_exists(strncmpi "${CURL_INCLUDES}" HAVE_STRNCMPI)
|
||||
#check_symbol_exists(stricmp "${CURL_INCLUDES}" HAVE_STRICMP)
|
||||
#check_symbol_exists(strcmpi "${CURL_INCLUDES}" HAVE_STRCMPI)
|
||||
#check_symbol_exists(strncmpi "${CURL_INCLUDES}" HAVE_STRNCMPI)
|
||||
check_symbol_exists(alarm "${CURL_INCLUDES}" HAVE_ALARM)
|
||||
if(NOT HAVE_STRNCMPI)
|
||||
set(HAVE_STRCMPI)
|
||||
endif()
|
||||
check_symbol_exists(gethostbyaddr "${CURL_INCLUDES}" HAVE_GETHOSTBYADDR)
|
||||
#check_symbol_exists(gethostbyaddr "${CURL_INCLUDES}" HAVE_GETHOSTBYADDR)
|
||||
check_symbol_exists(gethostbyaddr_r "${CURL_INCLUDES}" HAVE_GETHOSTBYADDR_R)
|
||||
check_symbol_exists(gettimeofday "${CURL_INCLUDES}" HAVE_GETTIMEOFDAY)
|
||||
check_symbol_exists(inet_addr "${CURL_INCLUDES}" HAVE_INET_ADDR)
|
||||
check_symbol_exists(inet_ntoa "${CURL_INCLUDES}" HAVE_INET_NTOA)
|
||||
#check_symbol_exists(inet_ntoa "${CURL_INCLUDES}" HAVE_INET_NTOA)
|
||||
check_symbol_exists(inet_ntoa_r "${CURL_INCLUDES}" HAVE_INET_NTOA_R)
|
||||
check_symbol_exists(tcsetattr "${CURL_INCLUDES}" HAVE_TCSETATTR)
|
||||
check_symbol_exists(tcgetattr "${CURL_INCLUDES}" HAVE_TCGETATTR)
|
||||
@ -331,8 +304,8 @@ check_symbol_exists(closesocket "${CURL_INCLUDES}" HAVE_CLOSESOCKET)
|
||||
check_symbol_exists(setvbuf "${CURL_INCLUDES}" HAVE_SETVBUF)
|
||||
check_symbol_exists(sigsetjmp "${CURL_INCLUDES}" HAVE_SIGSETJMP)
|
||||
check_symbol_exists(getpass_r "${CURL_INCLUDES}" HAVE_GETPASS_R)
|
||||
check_symbol_exists(strlcat "${CURL_INCLUDES}" HAVE_STRLCAT)
|
||||
check_symbol_exists(getpwuid "${CURL_INCLUDES}" HAVE_GETPWUID)
|
||||
#check_symbol_exists(strlcat "${CURL_INCLUDES}" HAVE_STRLCAT)
|
||||
#check_symbol_exists(getpwuid "${CURL_INCLUDES}" HAVE_GETPWUID)
|
||||
check_symbol_exists(getpwuid_r "${CURL_INCLUDES}" HAVE_GETPWUID_R)
|
||||
check_symbol_exists(geteuid "${CURL_INCLUDES}" HAVE_GETEUID)
|
||||
check_symbol_exists(usleep "${CURL_INCLUDES}" HAVE_USLEEP)
|
||||
@ -340,17 +313,15 @@ check_symbol_exists(utime "${CURL_INCLUDES}" HAVE_UTIME)
|
||||
check_symbol_exists(gmtime_r "${CURL_INCLUDES}" HAVE_GMTIME_R)
|
||||
check_symbol_exists(localtime_r "${CURL_INCLUDES}" HAVE_LOCALTIME_R)
|
||||
|
||||
check_symbol_exists(gethostbyname "${CURL_INCLUDES}" HAVE_GETHOSTBYNAME)
|
||||
#check_symbol_exists(gethostbyname "${CURL_INCLUDES}" HAVE_GETHOSTBYNAME)
|
||||
check_symbol_exists(gethostbyname_r "${CURL_INCLUDES}" HAVE_GETHOSTBYNAME_R)
|
||||
|
||||
check_symbol_exists(signal "${CURL_INCLUDES}" HAVE_SIGNAL_FUNC)
|
||||
check_symbol_exists(SIGALRM "${CURL_INCLUDES}" HAVE_SIGNAL_MACRO)
|
||||
if(HAVE_SIGNAL_FUNC AND HAVE_SIGNAL_MACRO)
|
||||
set(HAVE_SIGNAL 1)
|
||||
endif()
|
||||
set(HAVE_SIGNAL 1)
|
||||
check_symbol_exists(uname "${CURL_INCLUDES}" HAVE_UNAME)
|
||||
check_symbol_exists(strtoll "${CURL_INCLUDES}" HAVE_STRTOLL)
|
||||
check_symbol_exists(_strtoi64 "${CURL_INCLUDES}" HAVE__STRTOI64)
|
||||
#check_symbol_exists(_strtoi64 "${CURL_INCLUDES}" HAVE__STRTOI64)
|
||||
check_symbol_exists(strerror_r "${CURL_INCLUDES}" HAVE_STRERROR_R)
|
||||
check_symbol_exists(siginterrupt "${CURL_INCLUDES}" HAVE_SIGINTERRUPT)
|
||||
check_symbol_exists(perror "${CURL_INCLUDES}" HAVE_PERROR)
|
||||
|
||||
1
contrib/flatbuffers
vendored
Submodule
1
contrib/flatbuffers
vendored
Submodule
@ -0,0 +1 @@
|
||||
Subproject commit bf9eb67ab9371755c6bcece13cadc7693bcbf264
|
||||
@ -1,6 +1,8 @@
|
||||
set(ICU_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/icu/icu4c/source)
|
||||
set(ICUDATA_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/icudata/)
|
||||
|
||||
set (CMAKE_CXX_STANDARD 17)
|
||||
|
||||
# These lists of sources were generated from build log of the original ICU build system (configure + make).
|
||||
|
||||
set(ICUUC_SOURCES
|
||||
|
||||
2
contrib/libc-headers
vendored
2
contrib/libc-headers
vendored
@ -1 +1 @@
|
||||
Subproject commit cd82fd9d8eefe50a47a0adf7c617c3ea7d558d11
|
||||
Subproject commit 9676d2645a713e679dc981ffd84dee99fcd68b8e
|
||||
2
contrib/libcxx
vendored
2
contrib/libcxx
vendored
@ -1 +1 @@
|
||||
Subproject commit f7c63235238a71b7e0563fab8c7c5ec1b54831f6
|
||||
Subproject commit a8c453300879d0bf255f9d5959d42e2c8aac1bfb
|
||||
@ -47,6 +47,11 @@ add_library(cxx ${SRCS})
|
||||
target_include_directories(cxx SYSTEM BEFORE PUBLIC $<BUILD_INTERFACE:${LIBCXX_SOURCE_DIR}/include>)
|
||||
target_compile_definitions(cxx PRIVATE -D_LIBCPP_BUILDING_LIBRARY -DLIBCXX_BUILDING_LIBCXXABI)
|
||||
|
||||
# Enable capturing stack traces for all exceptions.
|
||||
if (USE_UNWIND)
|
||||
target_compile_definitions(cxx PUBLIC -DSTD_EXCEPTION_HAS_STACK_TRACE=1)
|
||||
endif ()
|
||||
|
||||
target_compile_options(cxx PUBLIC $<$<COMPILE_LANGUAGE:CXX>:-nostdinc++>)
|
||||
|
||||
check_cxx_compiler_flag(-Wreserved-id-macro HAVE_WARNING_RESERVED_ID_MACRO)
|
||||
|
||||
@ -32,6 +32,11 @@ target_compile_definitions(cxxabi PRIVATE -D_LIBCPP_BUILDING_LIBRARY)
|
||||
target_compile_options(cxxabi PRIVATE -nostdinc++ -fno-sanitize=undefined -Wno-macro-redefined) # If we don't disable UBSan, infinite recursion happens in dynamic_cast.
|
||||
target_link_libraries(cxxabi PUBLIC ${EXCEPTION_HANDLING_LIBRARY})
|
||||
|
||||
# Enable capturing stack traces for all exceptions.
|
||||
if (USE_UNWIND)
|
||||
target_compile_definitions(cxxabi PUBLIC -DSTD_EXCEPTION_HAS_STACK_TRACE=1)
|
||||
endif ()
|
||||
|
||||
install(
|
||||
TARGETS cxxabi
|
||||
EXPORT global
|
||||
|
||||
@ -7,11 +7,15 @@ ELSE(CMAKE_SYSTEM_NAME STREQUAL "Linux")
|
||||
ENDIF(CMAKE_SYSTEM_NAME STREQUAL "Linux")
|
||||
|
||||
IF(CMAKE_COMPILER_IS_GNUCXX)
|
||||
EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpfullversion OUTPUT_VARIABLE GCC_COMPILER_VERSION)
|
||||
|
||||
IF (NOT GCC_COMPILER_VERSION)
|
||||
EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_COMPILER_VERSION)
|
||||
|
||||
IF (NOT GCC_COMPILER_VERSION)
|
||||
MESSAGE(FATAL_ERROR "Cannot get gcc version")
|
||||
ENDIF (NOT GCC_COMPILER_VERSION)
|
||||
ENDIF (NOT GCC_COMPILER_VERSION)
|
||||
|
||||
STRING(REGEX MATCHALL "[0-9]+" GCC_COMPILER_VERSION ${GCC_COMPILER_VERSION})
|
||||
|
||||
|
||||
@ -1,16 +1,6 @@
|
||||
set(OPENSSL_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/openssl)
|
||||
set(OPENSSL_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/openssl)
|
||||
|
||||
#file(READ ${CMAKE_CURRENT_SOURCE_DIR}/${OPENSSL_SOURCE_DIR}/ssl/VERSION SSL_VERSION)
|
||||
#string(STRIP ${SSL_VERSION} SSL_VERSION)
|
||||
#string(REPLACE ":" "." SSL_VERSION ${SSL_VERSION})
|
||||
#string(REGEX REPLACE "\\..*" "" SSL_MAJOR_VERSION ${SSL_VERSION})
|
||||
|
||||
#file(READ ${CMAKE_CURRENT_SOURCE_DIR}/${OPENSSL_SOURCE_DIR}/crypto/VERSION CRYPTO_VERSION)
|
||||
#string(STRIP ${CRYPTO_VERSION} CRYPTO_VERSION)
|
||||
#string(REPLACE ":" "." CRYPTO_VERSION ${CRYPTO_VERSION})
|
||||
#string(REGEX REPLACE "\\..*" "" CRYPTO_MAJOR_VERSION ${CRYPTO_VERSION})
|
||||
|
||||
set(OPENSSLDIR "/etc/ssl" CACHE PATH "Set the default openssl directory")
|
||||
set(OPENSSL_ENGINESDIR "/usr/lib/engines-3" CACHE PATH "Set the default openssl directory for engines")
|
||||
set(OPENSSL_MODULESDIR "/usr/local/lib/ossl-modules" CACHE PATH "Set the default openssl directory for modules")
|
||||
@ -27,19 +17,27 @@ elseif(ARCH_AARCH64)
|
||||
endif()
|
||||
|
||||
enable_language(ASM)
|
||||
|
||||
if (COMPILER_CLANG)
|
||||
add_definitions(-Wno-unused-command-line-argument)
|
||||
endif ()
|
||||
|
||||
if (ARCH_AMD64)
|
||||
if (OS_DARWIN)
|
||||
set (OPENSSL_SYSTEM "macosx")
|
||||
endif ()
|
||||
|
||||
macro(perl_generate_asm FILE_IN FILE_OUT)
|
||||
get_filename_component(DIRNAME ${FILE_OUT} DIRECTORY)
|
||||
file(MAKE_DIRECTORY ${DIRNAME})
|
||||
add_custom_command(OUTPUT ${FILE_OUT}
|
||||
COMMAND /usr/bin/env perl ${FILE_IN} ${FILE_OUT}
|
||||
COMMAND /usr/bin/env perl ${FILE_IN} ${OPENSSL_SYSTEM} ${FILE_OUT}
|
||||
# ASM code has broken unwind tables (CFI), strip them.
|
||||
# Otherwise asynchronous unwind (that we use for query profiler)
|
||||
# will lead to segfault while trying to interpret wrong "CFA expression".
|
||||
COMMAND sed -i -e '/^\.cfi_/d' ${FILE_OUT})
|
||||
endmacro()
|
||||
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aes-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aes-x86_64.s)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesni-mb-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesni-mb-x86_64.s)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesni-sha1-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesni-sha1-x86_64.s)
|
||||
@ -70,12 +68,17 @@ if (ARCH_AMD64)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha256-x86_64.s) # This is not a mistake
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha512-x86_64.s)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/whrlpool/asm/wp-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/whrlpool/wp-x86_64.s)
|
||||
|
||||
elseif (ARCH_AARCH64)
|
||||
|
||||
macro(perl_generate_asm FILE_IN FILE_OUT)
|
||||
get_filename_component(DIRNAME ${FILE_OUT} DIRECTORY)
|
||||
file(MAKE_DIRECTORY ${DIRNAME})
|
||||
add_custom_command(OUTPUT ${FILE_OUT}
|
||||
COMMAND /usr/bin/env perl ${FILE_IN} "linux64" ${FILE_OUT})
|
||||
# Hope that the ASM code for AArch64 doesn't have broken CFI. Otherwise, add the same sed as for x86_64.
|
||||
endmacro()
|
||||
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesv8-armx.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesv8-armx.S)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/vpaes-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/aes/vpaes-armv8.S)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/bn/asm/armv8-mont.pl ${OPENSSL_BINARY_DIR}/crypto/bn/armv8-mont.S)
|
||||
@ -88,6 +91,7 @@ elseif (ARCH_AARCH64)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha1-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha1-armv8.S)
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha256-armv8.S) # This is not a mistake
|
||||
perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha512-armv8.S)
|
||||
|
||||
endif ()
|
||||
|
||||
set(CRYPTO_SRCS
|
||||
|
||||
1
contrib/ryu
vendored
Submodule
1
contrib/ryu
vendored
Submodule
@ -0,0 +1 @@
|
||||
Subproject commit 5b4a853534b47438b4d97935370f6b2397137c2b
|
||||
10
contrib/ryu-cmake/CMakeLists.txt
Normal file
10
contrib/ryu-cmake/CMakeLists.txt
Normal file
@ -0,0 +1,10 @@
|
||||
SET(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/ryu)
|
||||
|
||||
add_library(ryu
|
||||
${LIBRARY_DIR}/ryu/d2fixed.c
|
||||
${LIBRARY_DIR}/ryu/d2s.c
|
||||
${LIBRARY_DIR}/ryu/f2s.c
|
||||
${LIBRARY_DIR}/ryu/generic_128.c
|
||||
)
|
||||
|
||||
target_include_directories(ryu SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}")
|
||||
2
contrib/zlib-ng
vendored
2
contrib/zlib-ng
vendored
@ -1 +1 @@
|
||||
Subproject commit 5673222fbd37ea89afb2ea73096f9bf5ec68ea31
|
||||
Subproject commit bba56a73be249514acfbc7d49aa2a68994dad8ab
|
||||
@ -186,8 +186,8 @@ endif()
|
||||
list (APPEND clickhouse_common_io_sources ${CONFIG_BUILD})
|
||||
list (APPEND clickhouse_common_io_headers ${CONFIG_VERSION} ${CONFIG_COMMON})
|
||||
|
||||
list (APPEND dbms_sources src/Functions/IFunction.cpp src/Functions/FunctionFactory.cpp src/Functions/FunctionHelpers.cpp)
|
||||
list (APPEND dbms_headers src/Functions/IFunctionImpl.h src/Functions/FunctionFactory.h src/Functions/FunctionHelpers.h)
|
||||
list (APPEND dbms_sources src/Functions/IFunction.cpp src/Functions/FunctionFactory.cpp src/Functions/FunctionHelpers.cpp src/Functions/extractTimeZoneFromFunctionArguments.cpp)
|
||||
list (APPEND dbms_headers src/Functions/IFunctionImpl.h src/Functions/FunctionFactory.h src/Functions/FunctionHelpers.h src/Functions/extractTimeZoneFromFunctionArguments.h)
|
||||
|
||||
list (APPEND dbms_sources
|
||||
src/AggregateFunctions/AggregateFunctionFactory.cpp
|
||||
@ -330,6 +330,7 @@ target_link_libraries (clickhouse_common_io
|
||||
${LINK_LIBRARIES_ONLY_ON_X86_64}
|
||||
PUBLIC
|
||||
${DOUBLE_CONVERSION_LIBRARIES}
|
||||
ryu
|
||||
PUBLIC
|
||||
${Poco_Net_LIBRARY}
|
||||
${Poco_Util_LIBRARY}
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
# This strings autochanged from release_lib.sh:
|
||||
set(VERSION_REVISION 54430)
|
||||
set(VERSION_MAJOR 19)
|
||||
set(VERSION_MINOR 19)
|
||||
set(VERSION_REVISION 54431)
|
||||
set(VERSION_MAJOR 20)
|
||||
set(VERSION_MINOR 1)
|
||||
set(VERSION_PATCH 1)
|
||||
set(VERSION_GITHASH 8bd9709d1dec3366e35d2efeab213435857f67a9)
|
||||
set(VERSION_DESCRIBE v19.19.1.1-prestable)
|
||||
set(VERSION_STRING 19.19.1.1)
|
||||
set(VERSION_GITHASH 51d4c8a53be94504e3607b2232e12e5ef7a8ec28)
|
||||
set(VERSION_DESCRIBE v20.1.1.1-prestable)
|
||||
set(VERSION_STRING 20.1.1.1)
|
||||
# end of autochange
|
||||
|
||||
set(VERSION_EXTRA "" CACHE STRING "")
|
||||
|
||||
@ -300,7 +300,7 @@ private:
|
||||
&& std::string::npos == embedded_stack_trace_pos)
|
||||
{
|
||||
std::cerr << "Stack trace:" << std::endl
|
||||
<< e.getStackTrace().toString();
|
||||
<< e.getStackTraceString();
|
||||
}
|
||||
|
||||
/// If exception code isn't zero, we should return non-zero return code anyway.
|
||||
@ -327,6 +327,78 @@ private:
|
||||
|| (now.month() == 1 && now.day() <= 5);
|
||||
}
|
||||
|
||||
bool isChineseNewYearMode(const String & local_tz)
|
||||
{
|
||||
/// Days of Dec. 20 in Chinese calendar starting from year 2019 to year 2105
|
||||
static constexpr UInt16 chineseNewYearIndicators[]
|
||||
= {18275, 18659, 19014, 19368, 19752, 20107, 20491, 20845, 21199, 21583, 21937, 22292, 22676, 23030, 23414, 23768, 24122, 24506,
|
||||
24860, 25215, 25599, 25954, 26308, 26692, 27046, 27430, 27784, 28138, 28522, 28877, 29232, 29616, 29970, 30354, 30708, 31062,
|
||||
31446, 31800, 32155, 32539, 32894, 33248, 33632, 33986, 34369, 34724, 35078, 35462, 35817, 36171, 36555, 36909, 37293, 37647,
|
||||
38002, 38386, 38740, 39095, 39479, 39833, 40187, 40571, 40925, 41309, 41664, 42018, 42402, 42757, 43111, 43495, 43849, 44233,
|
||||
44587, 44942, 45326, 45680, 46035, 46418, 46772, 47126, 47510, 47865, 48249, 48604, 48958, 49342};
|
||||
static constexpr size_t N = sizeof(chineseNewYearIndicators) / sizeof(chineseNewYearIndicators[0]);
|
||||
|
||||
/// All time zone names are acquired from https://www.iana.org/time-zones
|
||||
static constexpr const char * chineseNewYearTimeZoneIndicators[] = {
|
||||
/// Time zones celebrating Chinese new year.
|
||||
"Asia/Shanghai",
|
||||
"Asia/Chongqing",
|
||||
"Asia/Harbin",
|
||||
"Asia/Urumqi",
|
||||
"Asia/Hong_Kong",
|
||||
"Asia/Chungking",
|
||||
"Asia/Macao",
|
||||
"Asia/Macau",
|
||||
"Asia/Taipei",
|
||||
"Asia/Singapore",
|
||||
|
||||
/// Time zones celebrating Chinese new year but with different festival names. Let's not print the message for now.
|
||||
// "Asia/Brunei",
|
||||
// "Asia/Ho_Chi_Minh",
|
||||
// "Asia/Hovd",
|
||||
// "Asia/Jakarta",
|
||||
// "Asia/Jayapura",
|
||||
// "Asia/Kashgar",
|
||||
// "Asia/Kuala_Lumpur",
|
||||
// "Asia/Kuching",
|
||||
// "Asia/Makassar",
|
||||
// "Asia/Pontianak",
|
||||
// "Asia/Pyongyang",
|
||||
// "Asia/Saigon",
|
||||
// "Asia/Seoul",
|
||||
// "Asia/Ujung_Pandang",
|
||||
// "Asia/Ulaanbaatar",
|
||||
// "Asia/Ulan_Bator",
|
||||
};
|
||||
static constexpr size_t M = sizeof(chineseNewYearTimeZoneIndicators) / sizeof(chineseNewYearTimeZoneIndicators[0]);
|
||||
|
||||
time_t current_time = time(nullptr);
|
||||
|
||||
if (chineseNewYearTimeZoneIndicators + M
|
||||
== std::find_if(chineseNewYearTimeZoneIndicators, chineseNewYearTimeZoneIndicators + M, [&local_tz](const char * tz)
|
||||
{
|
||||
return tz == local_tz;
|
||||
}))
|
||||
return false;
|
||||
|
||||
/// It's bad to be intrusive.
|
||||
if (current_time % 3 != 0)
|
||||
return false;
|
||||
|
||||
auto days = DateLUT::instance().toDayNum(current_time).toUnderType();
|
||||
for (auto i = 0ul; i < N; ++i)
|
||||
{
|
||||
auto d = chineseNewYearIndicators[i];
|
||||
|
||||
/// Let's celebrate until Lantern Festival
|
||||
if (d <= days && d + 25u >= days)
|
||||
return true;
|
||||
else if (d > days)
|
||||
return false;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
int mainImpl()
|
||||
{
|
||||
UseSSL use_ssl;
|
||||
@ -374,7 +446,7 @@ private:
|
||||
connect();
|
||||
|
||||
/// Initialize DateLUT here to avoid counting time spent here as query execution time.
|
||||
DateLUT::instance();
|
||||
const auto local_tz = DateLUT::instance().getTimeZone();
|
||||
if (!context.getSettingsRef().use_client_time_zone)
|
||||
{
|
||||
const auto & time_zone = connection->getServerTimezone(connection_parameters.timeouts);
|
||||
@ -540,7 +612,12 @@ private:
|
||||
|
||||
loop();
|
||||
|
||||
std::cout << (isNewYearMode() ? "Happy new year." : "Bye.") << std::endl;
|
||||
if (isNewYearMode())
|
||||
std::cout << "Happy new year." << std::endl;
|
||||
else if (isChineseNewYearMode(local_tz))
|
||||
std::cout << "Happy Chinese new year. 春节快乐!" << std::endl;
|
||||
else
|
||||
std::cout << "Bye." << std::endl;
|
||||
return 0;
|
||||
}
|
||||
else
|
||||
@ -714,7 +791,7 @@ private:
|
||||
|
||||
if (config().getBool("stacktrace", false))
|
||||
std::cerr << "Stack trace:" << std::endl
|
||||
<< e.getStackTrace().toString() << std::endl;
|
||||
<< e.getStackTraceString() << std::endl;
|
||||
|
||||
std::cerr << std::endl;
|
||||
|
||||
|
||||
@ -76,7 +76,7 @@ void LocalServer::initialize(Poco::Util::Application & self)
|
||||
if (config().has("logger") || config().has("logger.level") || config().has("logger.log"))
|
||||
{
|
||||
// sensitive data rules are not used here
|
||||
buildLoggers(config(), logger());
|
||||
buildLoggers(config(), logger(), self.commandName());
|
||||
}
|
||||
else
|
||||
{
|
||||
|
||||
@ -115,7 +115,7 @@ void ODBCHandler::handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Ne
|
||||
catch (const Exception & ex)
|
||||
{
|
||||
process_error("Invalid 'columns' parameter in request body '" + ex.message() + "'");
|
||||
LOG_WARNING(log, ex.getStackTrace().toString());
|
||||
LOG_WARNING(log, ex.getStackTraceString());
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@ -124,7 +124,7 @@ void ODBCBridge::initialize(Application & self)
|
||||
|
||||
config().setString("logger", "ODBCBridge");
|
||||
|
||||
buildLoggers(config(), logger());
|
||||
buildLoggers(config(), logger(), self.commandName());
|
||||
|
||||
log = &logger();
|
||||
hostname = config().getString("listen-host", "localhost");
|
||||
|
||||
@ -85,16 +85,6 @@ bool PerformanceTest::checkPreconditions() const
|
||||
|
||||
for (const std::string & precondition : preconditions)
|
||||
{
|
||||
if (precondition == "flush_disk_cache")
|
||||
{
|
||||
if (system(
|
||||
"(>&2 echo 'Flushing disk cache...') && (sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches') && (>&2 echo 'Flushed.')"))
|
||||
{
|
||||
LOG_WARNING(log, "Failed to flush disk cache");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (precondition == "ram_size")
|
||||
{
|
||||
size_t ram_size_needed = config->getUInt64("preconditions.ram_size");
|
||||
@ -337,7 +327,7 @@ void PerformanceTest::runQueries(
|
||||
{
|
||||
statistics.exception = "Code: " + std::to_string(e.code()) + ", e.displayText() = " + e.displayText();
|
||||
LOG_WARNING(log, "Code: " << e.code() << ", e.displayText() = " << e.displayText()
|
||||
<< ", Stack trace:\n\n" << e.getStackTrace().toString());
|
||||
<< ", Stack trace:\n\n" << e.getStackTraceString());
|
||||
}
|
||||
|
||||
if (!statistics.got_SIGINT)
|
||||
|
||||
@ -45,21 +45,11 @@ namespace fs = std::filesystem;
|
||||
|
||||
PerformanceTestInfo::PerformanceTestInfo(
|
||||
XMLConfigurationPtr config,
|
||||
const std::string & profiles_file_,
|
||||
const Settings & global_settings_)
|
||||
: profiles_file(profiles_file_)
|
||||
, settings(global_settings_)
|
||||
: settings(global_settings_)
|
||||
{
|
||||
path = config->getString("path");
|
||||
test_name = fs::path(path).stem().string();
|
||||
if (config->has("main_metric"))
|
||||
{
|
||||
Strings main_metrics;
|
||||
config->keys("main_metric", main_metrics);
|
||||
if (main_metrics.size())
|
||||
main_metric = main_metrics[0];
|
||||
}
|
||||
|
||||
applySettings(config);
|
||||
extractQueries(config);
|
||||
extractAuxiliaryQueries(config);
|
||||
@ -75,38 +65,8 @@ void PerformanceTestInfo::applySettings(XMLConfigurationPtr config)
|
||||
SettingsChanges settings_to_apply;
|
||||
Strings config_settings;
|
||||
config->keys("settings", config_settings);
|
||||
|
||||
auto settings_contain = [&config_settings] (const std::string & setting)
|
||||
{
|
||||
auto position = std::find(config_settings.begin(), config_settings.end(), setting);
|
||||
return position != config_settings.end();
|
||||
|
||||
};
|
||||
/// Preprocess configuration file
|
||||
if (settings_contain("profile"))
|
||||
{
|
||||
if (!profiles_file.empty())
|
||||
{
|
||||
std::string profile_name = config->getString("settings.profile");
|
||||
XMLConfigurationPtr profiles_config(new XMLConfiguration(profiles_file));
|
||||
|
||||
Strings profile_settings;
|
||||
profiles_config->keys("profiles." + profile_name, profile_settings);
|
||||
|
||||
extractSettings(profiles_config, "profiles." + profile_name, profile_settings, settings_to_apply);
|
||||
}
|
||||
}
|
||||
|
||||
extractSettings(config, "settings", config_settings, settings_to_apply);
|
||||
settings.applyChanges(settings_to_apply);
|
||||
|
||||
if (settings_contain("average_rows_speed_precision"))
|
||||
TestStats::avg_rows_speed_precision =
|
||||
config->getDouble("settings.average_rows_speed_precision");
|
||||
|
||||
if (settings_contain("average_bytes_speed_precision"))
|
||||
TestStats::avg_bytes_speed_precision =
|
||||
config->getDouble("settings.average_bytes_speed_precision");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -26,15 +26,13 @@ using StringToVector = std::map<std::string, Strings>;
|
||||
class PerformanceTestInfo
|
||||
{
|
||||
public:
|
||||
PerformanceTestInfo(XMLConfigurationPtr config, const std::string & profiles_file_, const Settings & global_settings_);
|
||||
PerformanceTestInfo(XMLConfigurationPtr config, const Settings & global_settings_);
|
||||
|
||||
std::string test_name;
|
||||
std::string path;
|
||||
std::string main_metric;
|
||||
|
||||
Strings queries;
|
||||
|
||||
std::string profiles_file;
|
||||
Settings settings;
|
||||
ExecutionType exec_type;
|
||||
StringToVector substitutions;
|
||||
|
||||
@ -64,7 +64,6 @@ public:
|
||||
const std::string & password_,
|
||||
const Settings & cmd_settings,
|
||||
const bool lite_output_,
|
||||
const std::string & profiles_file_,
|
||||
Strings && input_files_,
|
||||
Strings && tests_tags_,
|
||||
Strings && skip_tags_,
|
||||
@ -86,7 +85,6 @@ public:
|
||||
, skip_names_regexp(std::move(skip_names_regexp_))
|
||||
, query_indexes(query_indexes_)
|
||||
, lite_output(lite_output_)
|
||||
, profiles_file(profiles_file_)
|
||||
, input_files(input_files_)
|
||||
, log(&Poco::Logger::get("PerformanceTestSuite"))
|
||||
{
|
||||
@ -139,7 +137,6 @@ private:
|
||||
using XMLConfigurationPtr = Poco::AutoPtr<XMLConfiguration>;
|
||||
|
||||
bool lite_output;
|
||||
std::string profiles_file;
|
||||
|
||||
Strings input_files;
|
||||
std::vector<XMLConfigurationPtr> tests_configurations;
|
||||
@ -197,7 +194,7 @@ private:
|
||||
|
||||
std::pair<std::string, bool> runTest(XMLConfigurationPtr & test_config)
|
||||
{
|
||||
PerformanceTestInfo info(test_config, profiles_file, global_context.getSettingsRef());
|
||||
PerformanceTestInfo info(test_config, global_context.getSettingsRef());
|
||||
LOG_INFO(log, "Config for test '" << info.test_name << "' parsed");
|
||||
PerformanceTest current(test_config, connection, timeouts, interrupt_listener, info, global_context, query_indexes[info.path]);
|
||||
|
||||
@ -332,7 +329,6 @@ try
|
||||
desc.add_options()
|
||||
("help", "produce help message")
|
||||
("lite", "use lite version of output")
|
||||
("profiles-file", value<std::string>()->default_value(""), "Specify a file with global profiles")
|
||||
("host,h", value<std::string>()->default_value("localhost"), "")
|
||||
("port", value<UInt16>()->default_value(9000), "")
|
||||
("secure,s", "Use TLS connection")
|
||||
@ -401,7 +397,6 @@ try
|
||||
options["password"].as<std::string>(),
|
||||
cmd_settings,
|
||||
options.count("lite") > 0,
|
||||
options["profiles-file"].as<std::string>(),
|
||||
std::move(input_files),
|
||||
std::move(tests_tags),
|
||||
std::move(skip_tags),
|
||||
|
||||
@ -17,23 +17,25 @@ namespace DB
|
||||
|
||||
namespace
|
||||
{
|
||||
const std::regex QUOTE_REGEX{"\""};
|
||||
std::string getMainMetric(const PerformanceTestInfo & test_info)
|
||||
{
|
||||
std::string main_metric;
|
||||
if (test_info.main_metric.empty())
|
||||
if (test_info.exec_type == ExecutionType::Loop)
|
||||
main_metric = "min_time";
|
||||
return "min_time";
|
||||
else
|
||||
main_metric = "rows_per_second";
|
||||
else
|
||||
main_metric = test_info.main_metric;
|
||||
return main_metric;
|
||||
return "rows_per_second";
|
||||
}
|
||||
|
||||
bool isASCIIString(const std::string & str)
|
||||
{
|
||||
return std::all_of(str.begin(), str.end(), isASCII);
|
||||
}
|
||||
|
||||
String jsonString(const String & str, FormatSettings & settings)
|
||||
{
|
||||
WriteBufferFromOwnString buffer;
|
||||
writeJSONString(str, buffer, settings);
|
||||
return std::move(buffer.str());
|
||||
}
|
||||
}
|
||||
|
||||
ReportBuilder::ReportBuilder(const std::string & server_version_)
|
||||
@ -55,6 +57,8 @@ std::string ReportBuilder::buildFullReport(
|
||||
std::vector<TestStats> & stats,
|
||||
const std::vector<std::size_t> & queries_to_run) const
|
||||
{
|
||||
FormatSettings settings;
|
||||
|
||||
JSONString json_output;
|
||||
|
||||
json_output.set("hostname", hostname);
|
||||
@ -65,22 +69,18 @@ std::string ReportBuilder::buildFullReport(
|
||||
json_output.set("time", getCurrentTime());
|
||||
json_output.set("test_name", test_info.test_name);
|
||||
json_output.set("path", test_info.path);
|
||||
json_output.set("main_metric", getMainMetric(test_info));
|
||||
|
||||
if (test_info.substitutions.size())
|
||||
if (!test_info.substitutions.empty())
|
||||
{
|
||||
JSONString json_parameters(2); /// here, 2 is the size of \t padding
|
||||
|
||||
for (auto it = test_info.substitutions.begin(); it != test_info.substitutions.end(); ++it)
|
||||
for (auto & [parameter, values] : test_info.substitutions)
|
||||
{
|
||||
std::string parameter = it->first;
|
||||
Strings values = it->second;
|
||||
|
||||
std::ostringstream array_string;
|
||||
array_string << "[";
|
||||
for (size_t i = 0; i != values.size(); ++i)
|
||||
{
|
||||
array_string << '"' << std::regex_replace(values[i], QUOTE_REGEX, "\\\"") << '"';
|
||||
array_string << jsonString(values[i], settings);
|
||||
if (i != values.size() - 1)
|
||||
{
|
||||
array_string << ", ";
|
||||
@ -110,13 +110,12 @@ std::string ReportBuilder::buildFullReport(
|
||||
|
||||
JSONString runJSON;
|
||||
|
||||
auto query = std::regex_replace(test_info.queries[query_index], QUOTE_REGEX, "\\\"");
|
||||
runJSON.set("query", query);
|
||||
runJSON.set("query", jsonString(test_info.queries[query_index], settings), false);
|
||||
runJSON.set("query_index", query_index);
|
||||
if (!statistics.exception.empty())
|
||||
{
|
||||
if (isASCIIString(statistics.exception))
|
||||
runJSON.set("exception", std::regex_replace(statistics.exception, QUOTE_REGEX, "\\\""));
|
||||
runJSON.set("exception", jsonString(statistics.exception, settings), false);
|
||||
else
|
||||
runJSON.set("exception", "Some exception occured with non ASCII message. This may produce invalid JSON. Try reproduce locally.");
|
||||
}
|
||||
@ -183,7 +182,7 @@ std::string ReportBuilder::buildCompactReport(
|
||||
std::vector<TestStats> & stats,
|
||||
const std::vector<std::size_t> & queries_to_run) const
|
||||
{
|
||||
|
||||
FormatSettings settings;
|
||||
std::ostringstream output;
|
||||
|
||||
for (size_t query_index = 0; query_index < test_info.queries.size(); ++query_index)
|
||||
@ -194,7 +193,7 @@ std::string ReportBuilder::buildCompactReport(
|
||||
for (size_t number_of_launch = 0; number_of_launch < test_info.times_to_run; ++number_of_launch)
|
||||
{
|
||||
if (test_info.queries.size() > 1)
|
||||
output << "query \"" << test_info.queries[query_index] << "\", ";
|
||||
output << "query " << jsonString(test_info.queries[query_index], settings) << ", ";
|
||||
|
||||
output << "run " << std::to_string(number_of_launch + 1) << ": ";
|
||||
|
||||
|
||||
@ -20,8 +20,6 @@
|
||||
#include <Compression/CompressedReadBuffer.h>
|
||||
#include <Compression/CompressedWriteBuffer.h>
|
||||
#include <IO/ReadBufferFromIStream.h>
|
||||
#include <IO/ZlibInflatingReadBuffer.h>
|
||||
#include <IO/BrotliReadBuffer.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/WriteBufferFromHTTPServerResponse.h>
|
||||
@ -300,31 +298,23 @@ void HTTPHandler::processQuery(
|
||||
|
||||
/// The client can pass a HTTP header indicating supported compression method (gzip or deflate).
|
||||
String http_response_compression_methods = request.get("Accept-Encoding", "");
|
||||
bool client_supports_http_compression = false;
|
||||
CompressionMethod http_response_compression_method {};
|
||||
CompressionMethod http_response_compression_method = CompressionMethod::None;
|
||||
|
||||
if (!http_response_compression_methods.empty())
|
||||
{
|
||||
/// If client supports brotli - it's preferred.
|
||||
/// Both gzip and deflate are supported. If the client supports both, gzip is preferred.
|
||||
/// NOTE parsing of the list of methods is slightly incorrect.
|
||||
if (std::string::npos != http_response_compression_methods.find("gzip"))
|
||||
{
|
||||
client_supports_http_compression = true;
|
||||
|
||||
if (std::string::npos != http_response_compression_methods.find("br"))
|
||||
http_response_compression_method = CompressionMethod::Brotli;
|
||||
else if (std::string::npos != http_response_compression_methods.find("gzip"))
|
||||
http_response_compression_method = CompressionMethod::Gzip;
|
||||
}
|
||||
else if (std::string::npos != http_response_compression_methods.find("deflate"))
|
||||
{
|
||||
client_supports_http_compression = true;
|
||||
http_response_compression_method = CompressionMethod::Zlib;
|
||||
}
|
||||
#if USE_BROTLI
|
||||
else if (http_response_compression_methods == "br")
|
||||
{
|
||||
client_supports_http_compression = true;
|
||||
http_response_compression_method = CompressionMethod::Brotli;
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
bool client_supports_http_compression = http_response_compression_method != CompressionMethod::None;
|
||||
|
||||
/// Client can pass a 'compress' flag in the query string. In this case the query result is
|
||||
/// compressed using internal algorithm. This is not reflected in HTTP headers.
|
||||
@ -344,8 +334,8 @@ void HTTPHandler::processQuery(
|
||||
unsigned keep_alive_timeout = config.getUInt("keep_alive_timeout", 10);
|
||||
|
||||
used_output.out = std::make_shared<WriteBufferFromHTTPServerResponse>(
|
||||
request, response, keep_alive_timeout,
|
||||
client_supports_http_compression, http_response_compression_method, buffer_size_http);
|
||||
request, response, keep_alive_timeout, client_supports_http_compression, http_response_compression_method);
|
||||
|
||||
if (internal_compression)
|
||||
used_output.out_maybe_compressed = std::make_shared<CompressedWriteBuffer>(*used_output.out);
|
||||
else
|
||||
@ -400,32 +390,9 @@ void HTTPHandler::processQuery(
|
||||
std::unique_ptr<ReadBuffer> in_post_raw = std::make_unique<ReadBufferFromIStream>(istr);
|
||||
|
||||
/// Request body can be compressed using algorithm specified in the Content-Encoding header.
|
||||
std::unique_ptr<ReadBuffer> in_post;
|
||||
String http_request_compression_method_str = request.get("Content-Encoding", "");
|
||||
if (!http_request_compression_method_str.empty())
|
||||
{
|
||||
if (http_request_compression_method_str == "gzip")
|
||||
{
|
||||
in_post = std::make_unique<ZlibInflatingReadBuffer>(std::move(in_post_raw), CompressionMethod::Gzip);
|
||||
}
|
||||
else if (http_request_compression_method_str == "deflate")
|
||||
{
|
||||
in_post = std::make_unique<ZlibInflatingReadBuffer>(std::move(in_post_raw), CompressionMethod::Zlib);
|
||||
}
|
||||
#if USE_BROTLI
|
||||
else if (http_request_compression_method_str == "br")
|
||||
{
|
||||
in_post = std::make_unique<BrotliReadBuffer>(std::move(in_post_raw));
|
||||
}
|
||||
#endif
|
||||
else
|
||||
{
|
||||
throw Exception("Unknown Content-Encoding of HTTP request: " + http_request_compression_method_str,
|
||||
ErrorCodes::UNKNOWN_COMPRESSION_METHOD);
|
||||
}
|
||||
}
|
||||
else
|
||||
in_post = std::move(in_post_raw);
|
||||
std::unique_ptr<ReadBuffer> in_post = wrapReadBufferWithCompressionMethod(
|
||||
std::make_unique<ReadBufferFromIStream>(istr), chooseCompressionMethod({}, http_request_compression_method_str));
|
||||
|
||||
/// The data can also be compressed using incompatible internal algorithm. This is indicated by
|
||||
/// 'decompress' query parameter.
|
||||
|
||||
@ -298,7 +298,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
|
||||
global_context->shutdown();
|
||||
|
||||
LOG_DEBUG(log, "Shutted down storages.");
|
||||
LOG_DEBUG(log, "Shut down storages.");
|
||||
|
||||
/** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available.
|
||||
* At this moment, no one could own shared part of Context.
|
||||
@ -947,6 +947,7 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
});
|
||||
|
||||
/// try to load dictionaries immediately, throw on error and die
|
||||
ext::scope_guard dictionaries_xmls, models_xmls;
|
||||
try
|
||||
{
|
||||
if (!config().getBool("dictionaries_lazy_load", true))
|
||||
@ -954,12 +955,10 @@ int Server::main(const std::vector<std::string> & /*args*/)
|
||||
global_context->tryCreateEmbeddedDictionaries();
|
||||
global_context->getExternalDictionariesLoader().enableAlwaysLoadEverything(true);
|
||||
}
|
||||
|
||||
auto dictionaries_repository = std::make_unique<ExternalLoaderXMLConfigRepository>(config(), "dictionaries_config");
|
||||
global_context->getExternalDictionariesLoader().addConfigRepository("", std::move(dictionaries_repository));
|
||||
|
||||
auto models_repository = std::make_unique<ExternalLoaderXMLConfigRepository>(config(), "models_config");
|
||||
global_context->getExternalModelsLoader().addConfigRepository("", std::move(models_repository));
|
||||
dictionaries_xmls = global_context->getExternalDictionariesLoader().addConfigRepository(
|
||||
std::make_unique<ExternalLoaderXMLConfigRepository>(config(), "dictionaries_config"));
|
||||
models_xmls = global_context->getExternalModelsLoader().addConfigRepository(
|
||||
std::make_unique<ExternalLoaderXMLConfigRepository>(config(), "models_config"));
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
|
||||
@ -112,7 +112,7 @@ void TCPHandler::runImpl()
|
||||
{
|
||||
Exception e("Database " + backQuote(default_database) + " doesn't exist", ErrorCodes::UNKNOWN_DATABASE);
|
||||
LOG_ERROR(log, "Code: " << e.code() << ", e.displayText() = " << e.displayText()
|
||||
<< ", Stack trace:\n\n" << e.getStackTrace().toString());
|
||||
<< ", Stack trace:\n\n" << e.getStackTraceString());
|
||||
sendException(e, connection_context.getSettingsRef().calculate_text_stack_trace);
|
||||
return;
|
||||
}
|
||||
@ -158,7 +158,7 @@ void TCPHandler::runImpl()
|
||||
/** An exception during the execution of request (it must be sent over the network to the client).
|
||||
* The client will be able to accept it, if it did not happen while sending another packet and the client has not disconnected yet.
|
||||
*/
|
||||
std::unique_ptr<Exception> exception;
|
||||
std::optional<DB::Exception> exception;
|
||||
bool network_error = false;
|
||||
|
||||
bool send_exception_with_stack_trace = connection_context.getSettingsRef().calculate_text_stack_trace;
|
||||
@ -280,7 +280,7 @@ void TCPHandler::runImpl()
|
||||
catch (const Exception & e)
|
||||
{
|
||||
state.io.onException();
|
||||
exception.reset(e.clone());
|
||||
exception.emplace(e);
|
||||
|
||||
if (e.code() == ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT)
|
||||
throw;
|
||||
@ -298,22 +298,22 @@ void TCPHandler::runImpl()
|
||||
* We will try to send exception to the client in any case - see below.
|
||||
*/
|
||||
state.io.onException();
|
||||
exception = std::make_unique<Exception>(e.displayText(), ErrorCodes::POCO_EXCEPTION);
|
||||
exception.emplace(Exception::CreateFromPoco, e);
|
||||
}
|
||||
catch (const Poco::Exception & e)
|
||||
{
|
||||
state.io.onException();
|
||||
exception = std::make_unique<Exception>(e.displayText(), ErrorCodes::POCO_EXCEPTION);
|
||||
exception.emplace(Exception::CreateFromPoco, e);
|
||||
}
|
||||
catch (const std::exception & e)
|
||||
{
|
||||
state.io.onException();
|
||||
exception = std::make_unique<Exception>(e.what(), ErrorCodes::STD_EXCEPTION);
|
||||
exception.emplace(Exception::CreateFromSTD, e);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
state.io.onException();
|
||||
exception = std::make_unique<Exception>("Unknown exception", ErrorCodes::UNKNOWN_EXCEPTION);
|
||||
exception.emplace("Unknown exception", ErrorCodes::UNKNOWN_EXCEPTION);
|
||||
}
|
||||
|
||||
try
|
||||
@ -546,7 +546,7 @@ void TCPHandler::processOrdinaryQueryWithProcessors(size_t num_threads)
|
||||
auto & pipeline = state.io.pipeline;
|
||||
|
||||
if (pipeline.getMaxThreads())
|
||||
num_threads = pipeline.getMaxThreads();
|
||||
num_threads = std::min(num_threads, pipeline.getMaxThreads());
|
||||
|
||||
/// Send header-block, to allow client to prepare output format for data to send.
|
||||
{
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
<yandex>
|
||||
<logger>
|
||||
<console>true</console>
|
||||
<log remove="remove"/>
|
||||
<errorlog remove="remove"/>
|
||||
</logger>
|
||||
|
||||
@ -84,25 +84,17 @@
|
||||
<!-- Quota for user. -->
|
||||
<quota>default</quota>
|
||||
|
||||
<!-- For testing the table filters -->
|
||||
<databases>
|
||||
<!-- Example of row level security policy. -->
|
||||
<!-- <databases>
|
||||
<test>
|
||||
<!-- Simple expression filter -->
|
||||
<filtered_table1>
|
||||
<filter>a = 1</filter>
|
||||
</filtered_table1>
|
||||
|
||||
<!-- Complex expression filter -->
|
||||
<filtered_table2>
|
||||
<filter>a + b < 1 or c - d > 5</filter>
|
||||
</filtered_table2>
|
||||
|
||||
<!-- Filter with ALIAS column -->
|
||||
<filtered_table3>
|
||||
<filter>c = 1</filter>
|
||||
</filtered_table3>
|
||||
</test>
|
||||
</databases>
|
||||
</databases> -->
|
||||
</default>
|
||||
|
||||
<!-- Example of user with readonly access. -->
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
#include <Access/MemoryAccessStorage.h>
|
||||
#include <Access/UsersConfigAccessStorage.h>
|
||||
#include <Access/QuotaContextFactory.h>
|
||||
#include <Access/RowPolicyContextFactory.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -21,7 +22,8 @@ namespace
|
||||
|
||||
AccessControlManager::AccessControlManager()
|
||||
: MultipleAccessStorage(createStorages()),
|
||||
quota_context_factory(std::make_unique<QuotaContextFactory>(*this))
|
||||
quota_context_factory(std::make_unique<QuotaContextFactory>(*this)),
|
||||
row_policy_context_factory(std::make_unique<RowPolicyContextFactory>(*this))
|
||||
{
|
||||
}
|
||||
|
||||
@ -49,4 +51,11 @@ std::vector<QuotaUsageInfo> AccessControlManager::getQuotaUsageInfo() const
|
||||
{
|
||||
return quota_context_factory->getUsageInfo();
|
||||
}
|
||||
|
||||
|
||||
std::shared_ptr<RowPolicyContext> AccessControlManager::getRowPolicyContext(const String & user_name) const
|
||||
{
|
||||
return row_policy_context_factory->createContext(user_name);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -22,6 +22,8 @@ namespace DB
|
||||
class QuotaContext;
|
||||
class QuotaContextFactory;
|
||||
struct QuotaUsageInfo;
|
||||
class RowPolicyContext;
|
||||
class RowPolicyContextFactory;
|
||||
|
||||
|
||||
/// Manages access control entities.
|
||||
@ -38,8 +40,11 @@ public:
|
||||
|
||||
std::vector<QuotaUsageInfo> getQuotaUsageInfo() const;
|
||||
|
||||
std::shared_ptr<RowPolicyContext> getRowPolicyContext(const String & user_name) const;
|
||||
|
||||
private:
|
||||
std::unique_ptr<QuotaContextFactory> quota_context_factory;
|
||||
std::unique_ptr<RowPolicyContextFactory> row_policy_context_factory;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#include <Access/IAccessEntity.h>
|
||||
#include <Access/Quota.h>
|
||||
#include <Access/RowPolicy.h>
|
||||
#include <common/demangle.h>
|
||||
|
||||
|
||||
@ -9,6 +10,8 @@ String IAccessEntity::getTypeName(std::type_index type)
|
||||
{
|
||||
if (type == typeid(Quota))
|
||||
return "Quota";
|
||||
if (type == typeid(RowPolicy))
|
||||
return "Row policy";
|
||||
return demangle(type.name());
|
||||
}
|
||||
|
||||
|
||||
111
dbms/src/Access/RowPolicy.cpp
Normal file
111
dbms/src/Access/RowPolicy.cpp
Normal file
@ -0,0 +1,111 @@
|
||||
#include <Access/RowPolicy.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Common/quoteString.h>
|
||||
#include <boost/range/algorithm/equal.hpp>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace
|
||||
{
|
||||
void generateFullNameImpl(const String & database_, const String & table_name_, const String & policy_name_, String & full_name_)
|
||||
{
|
||||
full_name_.clear();
|
||||
full_name_.reserve(database_.length() + table_name_.length() + policy_name_.length() + 6);
|
||||
full_name_ += backQuoteIfNeed(policy_name_);
|
||||
full_name_ += " ON ";
|
||||
if (!database_.empty())
|
||||
{
|
||||
full_name_ += backQuoteIfNeed(database_);
|
||||
full_name_ += '.';
|
||||
}
|
||||
full_name_ += backQuoteIfNeed(table_name_);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
String RowPolicy::FullNameParts::getFullName() const
|
||||
{
|
||||
String full_name;
|
||||
generateFullNameImpl(database, table_name, policy_name, full_name);
|
||||
return full_name;
|
||||
}
|
||||
|
||||
|
||||
String RowPolicy::FullNameParts::getFullName(const Context & context) const
|
||||
{
|
||||
String full_name;
|
||||
generateFullNameImpl(database.empty() ? context.getCurrentDatabase() : database, table_name, policy_name, full_name);
|
||||
return full_name;
|
||||
}
|
||||
|
||||
|
||||
void RowPolicy::setDatabase(const String & database_)
|
||||
{
|
||||
database = database_;
|
||||
generateFullNameImpl(database, table_name, policy_name, full_name);
|
||||
}
|
||||
|
||||
|
||||
void RowPolicy::setTableName(const String & table_name_)
|
||||
{
|
||||
table_name = table_name_;
|
||||
generateFullNameImpl(database, table_name, policy_name, full_name);
|
||||
}
|
||||
|
||||
|
||||
void RowPolicy::setName(const String & policy_name_)
|
||||
{
|
||||
policy_name = policy_name_;
|
||||
generateFullNameImpl(database, table_name, policy_name, full_name);
|
||||
}
|
||||
|
||||
|
||||
void RowPolicy::setFullName(const String & database_, const String & table_name_, const String & policy_name_)
|
||||
{
|
||||
database = database_;
|
||||
table_name = table_name_;
|
||||
policy_name = policy_name_;
|
||||
generateFullNameImpl(database, table_name, policy_name, full_name);
|
||||
}
|
||||
|
||||
|
||||
bool RowPolicy::equal(const IAccessEntity & other) const
|
||||
{
|
||||
if (!IAccessEntity::equal(other))
|
||||
return false;
|
||||
const auto & other_policy = typeid_cast<const RowPolicy &>(other);
|
||||
return (database == other_policy.database) && (table_name == other_policy.table_name) && (policy_name == other_policy.policy_name)
|
||||
&& boost::range::equal(conditions, other_policy.conditions) && restrictive == other_policy.restrictive
|
||||
&& (roles == other_policy.roles) && (all_roles == other_policy.all_roles) && (except_roles == other_policy.except_roles);
|
||||
}
|
||||
|
||||
|
||||
const char * RowPolicy::conditionIndexToString(ConditionIndex index)
|
||||
{
|
||||
switch (index)
|
||||
{
|
||||
case SELECT_FILTER: return "SELECT_FILTER";
|
||||
case INSERT_CHECK: return "INSERT_CHECK";
|
||||
case UPDATE_FILTER: return "UPDATE_FILTER";
|
||||
case UPDATE_CHECK: return "UPDATE_CHECK";
|
||||
case DELETE_FILTER: return "DELETE_FILTER";
|
||||
}
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
|
||||
const char * RowPolicy::conditionIndexToColumnName(ConditionIndex index)
|
||||
{
|
||||
switch (index)
|
||||
{
|
||||
case SELECT_FILTER: return "select_filter";
|
||||
case INSERT_CHECK: return "insert_check";
|
||||
case UPDATE_FILTER: return "update_filter";
|
||||
case UPDATE_CHECK: return "update_check";
|
||||
case DELETE_FILTER: return "delete_filter";
|
||||
}
|
||||
__builtin_unreachable();
|
||||
}
|
||||
|
||||
}
|
||||
81
dbms/src/Access/RowPolicy.h
Normal file
81
dbms/src/Access/RowPolicy.h
Normal file
@ -0,0 +1,81 @@
|
||||
#pragma once
|
||||
|
||||
#include <Access/IAccessEntity.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
class Context;
|
||||
|
||||
|
||||
/** Represents a row level security policy for a table.
|
||||
*/
|
||||
struct RowPolicy : public IAccessEntity
|
||||
{
|
||||
void setDatabase(const String & database_);
|
||||
void setTableName(const String & table_name_);
|
||||
void setName(const String & policy_name_) override;
|
||||
void setFullName(const String & database_, const String & table_name_, const String & policy_name_);
|
||||
|
||||
String getDatabase() const { return database; }
|
||||
String getTableName() const { return table_name; }
|
||||
String getName() const override { return policy_name; }
|
||||
|
||||
struct FullNameParts
|
||||
{
|
||||
String database;
|
||||
String table_name;
|
||||
String policy_name;
|
||||
String getFullName() const;
|
||||
String getFullName(const Context & context) const;
|
||||
};
|
||||
|
||||
/// Filter is a SQL conditional expression used to figure out which rows should be visible
|
||||
/// for user or available for modification. If the expression returns NULL or false for some rows
|
||||
/// those rows are silently suppressed.
|
||||
/// Check is a SQL condition expression used to check whether a row can be written into
|
||||
/// the table. If the expression returns NULL or false an exception is thrown.
|
||||
/// If a conditional expression here is empty it means no filtering is applied.
|
||||
enum ConditionIndex
|
||||
{
|
||||
SELECT_FILTER,
|
||||
INSERT_CHECK,
|
||||
UPDATE_FILTER,
|
||||
UPDATE_CHECK,
|
||||
DELETE_FILTER,
|
||||
};
|
||||
static constexpr size_t MAX_CONDITION_INDEX = 5;
|
||||
static const char * conditionIndexToString(ConditionIndex index);
|
||||
static const char * conditionIndexToColumnName(ConditionIndex index);
|
||||
|
||||
String conditions[MAX_CONDITION_INDEX];
|
||||
|
||||
/// Sets that the policy is permissive.
|
||||
/// A row is only accessible if at least one of the permissive policies passes,
|
||||
/// in addition to all the restrictive policies.
|
||||
void setPermissive(bool permissive_ = true) { setRestrictive(!permissive_); }
|
||||
bool isPermissive() const { return !isRestrictive(); }
|
||||
|
||||
/// Sets that the policy is restrictive.
|
||||
/// A row is only accessible if at least one of the permissive policies passes,
|
||||
/// in addition to all the restrictive policies.
|
||||
void setRestrictive(bool restrictive_ = true) { restrictive = restrictive_; }
|
||||
bool isRestrictive() const { return restrictive; }
|
||||
|
||||
bool equal(const IAccessEntity & other) const override;
|
||||
std::shared_ptr<IAccessEntity> clone() const override { return cloneImpl<RowPolicy>(); }
|
||||
|
||||
/// Which roles or users should use this quota.
|
||||
Strings roles;
|
||||
bool all_roles = false;
|
||||
Strings except_roles;
|
||||
|
||||
private:
|
||||
String database;
|
||||
String table_name;
|
||||
String policy_name;
|
||||
bool restrictive = false;
|
||||
};
|
||||
|
||||
using RowPolicyPtr = std::shared_ptr<const RowPolicy>;
|
||||
}
|
||||
59
dbms/src/Access/RowPolicyContext.cpp
Normal file
59
dbms/src/Access/RowPolicyContext.cpp
Normal file
@ -0,0 +1,59 @@
|
||||
#include <Access/RowPolicyContext.h>
|
||||
#include <boost/range/adaptor/map.hpp>
|
||||
#include <boost/range/algorithm/copy.hpp>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
size_t RowPolicyContext::Hash::operator()(const DatabaseAndTableNameRef & database_and_table_name) const
|
||||
{
|
||||
return std::hash<StringRef>{}(database_and_table_name.first) - std::hash<StringRef>{}(database_and_table_name.second);
|
||||
}
|
||||
|
||||
|
||||
RowPolicyContext::RowPolicyContext()
|
||||
: atomic_map_of_mixed_conditions(std::make_shared<MapOfMixedConditions>())
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
RowPolicyContext::~RowPolicyContext() = default;
|
||||
|
||||
|
||||
RowPolicyContext::RowPolicyContext(const String & user_name_)
|
||||
: user_name(user_name_)
|
||||
{}
|
||||
|
||||
|
||||
ASTPtr RowPolicyContext::getCondition(const String & database, const String & table_name, ConditionIndex index) const
|
||||
{
|
||||
/// We don't lock `mutex` here.
|
||||
auto map_of_mixed_conditions = std::atomic_load(&atomic_map_of_mixed_conditions);
|
||||
auto it = map_of_mixed_conditions->find({database, table_name});
|
||||
if (it == map_of_mixed_conditions->end())
|
||||
return {};
|
||||
return it->second.mixed_conditions[index];
|
||||
}
|
||||
|
||||
|
||||
std::vector<UUID> RowPolicyContext::getCurrentPolicyIDs() const
|
||||
{
|
||||
/// We don't lock `mutex` here.
|
||||
auto map_of_mixed_conditions = std::atomic_load(&atomic_map_of_mixed_conditions);
|
||||
std::vector<UUID> policy_ids;
|
||||
for (const auto & mixed_conditions : *map_of_mixed_conditions | boost::adaptors::map_values)
|
||||
boost::range::copy(mixed_conditions.policy_ids, std::back_inserter(policy_ids));
|
||||
return policy_ids;
|
||||
}
|
||||
|
||||
|
||||
std::vector<UUID> RowPolicyContext::getCurrentPolicyIDs(const String & database, const String & table_name) const
|
||||
{
|
||||
/// We don't lock `mutex` here.
|
||||
auto map_of_mixed_conditions = std::atomic_load(&atomic_map_of_mixed_conditions);
|
||||
auto it = map_of_mixed_conditions->find({database, table_name});
|
||||
if (it == map_of_mixed_conditions->end())
|
||||
return {};
|
||||
return it->second.policy_ids;
|
||||
}
|
||||
}
|
||||
66
dbms/src/Access/RowPolicyContext.h
Normal file
66
dbms/src/Access/RowPolicyContext.h
Normal file
@ -0,0 +1,66 @@
|
||||
#pragma once
|
||||
|
||||
#include <Access/RowPolicy.h>
|
||||
#include <Core/Types.h>
|
||||
#include <Core/UUID.h>
|
||||
#include <common/StringRef.h>
|
||||
#include <memory>
|
||||
#include <unordered_map>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
class IAST;
|
||||
using ASTPtr = std::shared_ptr<IAST>;
|
||||
|
||||
|
||||
/// Provides fast access to row policies' conditions for a specific user and tables.
|
||||
class RowPolicyContext
|
||||
{
|
||||
public:
|
||||
/// Default constructor makes a row policy usage context which restricts nothing.
|
||||
RowPolicyContext();
|
||||
|
||||
~RowPolicyContext();
|
||||
|
||||
using ConditionIndex = RowPolicy::ConditionIndex;
|
||||
|
||||
/// Returns prepared filter for a specific table and operations.
|
||||
/// The function can return nullptr, that means there is no filters applied.
|
||||
/// The returned filter can be a combination of the filters defined by multiple row policies.
|
||||
ASTPtr getCondition(const String & database, const String & table_name, ConditionIndex index) const;
|
||||
|
||||
/// Returns IDs of all the policies used by the current user.
|
||||
std::vector<UUID> getCurrentPolicyIDs() const;
|
||||
|
||||
/// Returns IDs of the policies used by a concrete table.
|
||||
std::vector<UUID> getCurrentPolicyIDs(const String & database, const String & table_name) const;
|
||||
|
||||
private:
|
||||
friend class RowPolicyContextFactory;
|
||||
friend struct ext::shared_ptr_helper<RowPolicyContext>;
|
||||
RowPolicyContext(const String & user_name_); /// RowPolicyContext should be created by RowPolicyContextFactory.
|
||||
|
||||
using DatabaseAndTableName = std::pair<String, String>;
|
||||
using DatabaseAndTableNameRef = std::pair<StringRef, StringRef>;
|
||||
struct Hash
|
||||
{
|
||||
size_t operator()(const DatabaseAndTableNameRef & database_and_table_name) const;
|
||||
};
|
||||
static constexpr size_t MAX_CONDITION_INDEX = RowPolicy::MAX_CONDITION_INDEX;
|
||||
using ParsedConditions = std::array<ASTPtr, MAX_CONDITION_INDEX>;
|
||||
struct MixedConditions
|
||||
{
|
||||
std::unique_ptr<DatabaseAndTableName> database_and_table_name_keeper;
|
||||
ParsedConditions mixed_conditions;
|
||||
std::vector<UUID> policy_ids;
|
||||
};
|
||||
using MapOfMixedConditions = std::unordered_map<DatabaseAndTableNameRef, MixedConditions, Hash>;
|
||||
|
||||
const String user_name;
|
||||
std::shared_ptr<const MapOfMixedConditions> atomic_map_of_mixed_conditions; /// Changed atomically, not protected by `mutex`.
|
||||
};
|
||||
|
||||
|
||||
using RowPolicyContextPtr = std::shared_ptr<RowPolicyContext>;
|
||||
}
|
||||
314
dbms/src/Access/RowPolicyContextFactory.cpp
Normal file
314
dbms/src/Access/RowPolicyContextFactory.cpp
Normal file
@ -0,0 +1,314 @@
|
||||
#include <Access/RowPolicyContextFactory.h>
|
||||
#include <Access/RowPolicyContext.h>
|
||||
#include <Access/AccessControlManager.h>
|
||||
#include <Parsers/ASTLiteral.h>
|
||||
#include <Parsers/ASTFunction.h>
|
||||
#include <Parsers/ExpressionListParsers.h>
|
||||
#include <Parsers/parseQuery.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/quoteString.h>
|
||||
#include <ext/range.h>
|
||||
#include <boost/range/algorithm/copy.hpp>
|
||||
#include <boost/range/algorithm_ext/erase.hpp>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace
|
||||
{
|
||||
bool tryGetLiteralBool(const IAST & ast, bool & value)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (const ASTLiteral * literal = ast.as<ASTLiteral>())
|
||||
{
|
||||
value = !literal->value.isNull() && applyVisitor(FieldVisitorConvertToNumber<bool>(), literal->value);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
ASTPtr applyFunctionAND(ASTs arguments)
|
||||
{
|
||||
bool const_arguments = true;
|
||||
boost::range::remove_erase_if(arguments, [&](const ASTPtr & argument) -> bool
|
||||
{
|
||||
bool b;
|
||||
if (!tryGetLiteralBool(*argument, b))
|
||||
return false;
|
||||
const_arguments &= b;
|
||||
return true;
|
||||
});
|
||||
|
||||
if (!const_arguments)
|
||||
return std::make_shared<ASTLiteral>(Field{UInt8(0)});
|
||||
if (arguments.empty())
|
||||
return std::make_shared<ASTLiteral>(Field{UInt8(1)});
|
||||
if (arguments.size() == 1)
|
||||
return arguments[0];
|
||||
|
||||
auto function = std::make_shared<ASTFunction>();
|
||||
auto exp_list = std::make_shared<ASTExpressionList>();
|
||||
function->name = "and";
|
||||
function->arguments = exp_list;
|
||||
function->children.push_back(exp_list);
|
||||
exp_list->children = std::move(arguments);
|
||||
return function;
|
||||
}
|
||||
|
||||
|
||||
ASTPtr applyFunctionOR(ASTs arguments)
|
||||
{
|
||||
bool const_arguments = false;
|
||||
boost::range::remove_erase_if(arguments, [&](const ASTPtr & argument) -> bool
|
||||
{
|
||||
bool b;
|
||||
if (!tryGetLiteralBool(*argument, b))
|
||||
return false;
|
||||
const_arguments |= b;
|
||||
return true;
|
||||
});
|
||||
|
||||
if (const_arguments)
|
||||
return std::make_shared<ASTLiteral>(Field{UInt8(1)});
|
||||
if (arguments.empty())
|
||||
return std::make_shared<ASTLiteral>(Field{UInt8(0)});
|
||||
if (arguments.size() == 1)
|
||||
return arguments[0];
|
||||
|
||||
auto function = std::make_shared<ASTFunction>();
|
||||
auto exp_list = std::make_shared<ASTExpressionList>();
|
||||
function->name = "or";
|
||||
function->arguments = exp_list;
|
||||
function->children.push_back(exp_list);
|
||||
exp_list->children = std::move(arguments);
|
||||
return function;
|
||||
}
|
||||
|
||||
|
||||
using ConditionIndex = RowPolicy::ConditionIndex;
|
||||
static constexpr size_t MAX_CONDITION_INDEX = RowPolicy::MAX_CONDITION_INDEX;
|
||||
|
||||
|
||||
/// Accumulates conditions from multiple row policies and joins them using the AND logical operation.
|
||||
class ConditionsMixer
|
||||
{
|
||||
public:
|
||||
void add(const ASTPtr & condition, bool is_restrictive)
|
||||
{
|
||||
if (!condition)
|
||||
return;
|
||||
|
||||
if (is_restrictive)
|
||||
restrictions.push_back(condition);
|
||||
else
|
||||
permissions.push_back(condition);
|
||||
}
|
||||
|
||||
ASTPtr getResult() &&
|
||||
{
|
||||
/// Process permissive conditions.
|
||||
if (!permissions.empty())
|
||||
restrictions.push_back(applyFunctionOR(std::move(permissions)));
|
||||
|
||||
/// Process restrictive conditions.
|
||||
if (!restrictions.empty())
|
||||
return applyFunctionAND(std::move(restrictions));
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
private:
|
||||
ASTs permissions;
|
||||
ASTs restrictions;
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::PolicyInfo::setPolicy(const RowPolicyPtr & policy_)
|
||||
{
|
||||
policy = policy_;
|
||||
|
||||
boost::range::copy(policy->roles, std::inserter(roles, roles.end()));
|
||||
all_roles = policy->all_roles;
|
||||
boost::range::copy(policy->except_roles, std::inserter(except_roles, except_roles.end()));
|
||||
|
||||
for (auto index : ext::range_with_static_cast<ConditionIndex>(0, MAX_CONDITION_INDEX))
|
||||
{
|
||||
const String & condition = policy->conditions[index];
|
||||
auto previous_range = std::pair(std::begin(policy->conditions), std::begin(policy->conditions) + index);
|
||||
auto previous_it = std::find(previous_range.first, previous_range.second, condition);
|
||||
if (previous_it != previous_range.second)
|
||||
{
|
||||
/// The condition is already parsed before.
|
||||
parsed_conditions[index] = parsed_conditions[previous_it - previous_range.first];
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Try to parse the condition.
|
||||
try
|
||||
{
|
||||
ParserExpression parser;
|
||||
parsed_conditions[index] = parseQuery(parser, condition, 0);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(
|
||||
&Poco::Logger::get("RowPolicy"),
|
||||
String("Could not parse the condition ") + RowPolicy::conditionIndexToString(index) + " of row policy "
|
||||
+ backQuote(policy->getFullName()));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
bool RowPolicyContextFactory::PolicyInfo::canUseWithContext(const RowPolicyContext & context) const
|
||||
{
|
||||
if (roles.count(context.user_name))
|
||||
return true;
|
||||
|
||||
if (all_roles && !except_roles.count(context.user_name))
|
||||
return true;
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
RowPolicyContextFactory::RowPolicyContextFactory(const AccessControlManager & access_control_manager_)
|
||||
: access_control_manager(access_control_manager_)
|
||||
{
|
||||
}
|
||||
|
||||
RowPolicyContextFactory::~RowPolicyContextFactory() = default;
|
||||
|
||||
|
||||
RowPolicyContextPtr RowPolicyContextFactory::createContext(const String & user_name)
|
||||
{
|
||||
std::lock_guard lock{mutex};
|
||||
ensureAllRowPoliciesRead();
|
||||
auto context = ext::shared_ptr_helper<RowPolicyContext>::create(user_name);
|
||||
contexts.push_back(context);
|
||||
mixConditionsForContext(*context);
|
||||
return context;
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::ensureAllRowPoliciesRead()
|
||||
{
|
||||
/// `mutex` is already locked.
|
||||
if (all_policies_read)
|
||||
return;
|
||||
all_policies_read = true;
|
||||
|
||||
subscription = access_control_manager.subscribeForChanges<RowPolicy>(
|
||||
[&](const UUID & id, const AccessEntityPtr & entity)
|
||||
{
|
||||
if (entity)
|
||||
rowPolicyAddedOrChanged(id, typeid_cast<RowPolicyPtr>(entity));
|
||||
else
|
||||
rowPolicyRemoved(id);
|
||||
});
|
||||
|
||||
for (const UUID & id : access_control_manager.findAll<RowPolicy>())
|
||||
{
|
||||
auto quota = access_control_manager.tryRead<RowPolicy>(id);
|
||||
if (quota)
|
||||
all_policies.emplace(id, PolicyInfo(quota));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::rowPolicyAddedOrChanged(const UUID & policy_id, const RowPolicyPtr & new_policy)
|
||||
{
|
||||
std::lock_guard lock{mutex};
|
||||
auto it = all_policies.find(policy_id);
|
||||
if (it == all_policies.end())
|
||||
{
|
||||
it = all_policies.emplace(policy_id, PolicyInfo(new_policy)).first;
|
||||
}
|
||||
else
|
||||
{
|
||||
if (it->second.policy == new_policy)
|
||||
return;
|
||||
}
|
||||
|
||||
auto & info = it->second;
|
||||
info.setPolicy(new_policy);
|
||||
mixConditionsForAllContexts();
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::rowPolicyRemoved(const UUID & policy_id)
|
||||
{
|
||||
std::lock_guard lock{mutex};
|
||||
all_policies.erase(policy_id);
|
||||
mixConditionsForAllContexts();
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::mixConditionsForAllContexts()
|
||||
{
|
||||
/// `mutex` is already locked.
|
||||
boost::range::remove_erase_if(
|
||||
contexts,
|
||||
[&](const std::weak_ptr<RowPolicyContext> & weak)
|
||||
{
|
||||
auto context = weak.lock();
|
||||
if (!context)
|
||||
return true; // remove from the `contexts` list.
|
||||
mixConditionsForContext(*context);
|
||||
return false; // keep in the `contexts` list.
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
void RowPolicyContextFactory::mixConditionsForContext(RowPolicyContext & context)
|
||||
{
|
||||
/// `mutex` is already locked.
|
||||
struct Mixers
|
||||
{
|
||||
ConditionsMixer mixers[MAX_CONDITION_INDEX];
|
||||
std::vector<UUID> policy_ids;
|
||||
};
|
||||
using MapOfMixedConditions = RowPolicyContext::MapOfMixedConditions;
|
||||
using DatabaseAndTableName = RowPolicyContext::DatabaseAndTableName;
|
||||
using DatabaseAndTableNameRef = RowPolicyContext::DatabaseAndTableNameRef;
|
||||
using Hash = RowPolicyContext::Hash;
|
||||
|
||||
std::unordered_map<DatabaseAndTableName, Mixers, Hash> map_of_mixers;
|
||||
|
||||
for (const auto & [policy_id, info] : all_policies)
|
||||
{
|
||||
if (info.canUseWithContext(context))
|
||||
{
|
||||
const auto & policy = *info.policy;
|
||||
auto & mixers = map_of_mixers[std::pair{policy.getDatabase(), policy.getTableName()}];
|
||||
mixers.policy_ids.push_back(policy_id);
|
||||
for (auto index : ext::range(0, MAX_CONDITION_INDEX))
|
||||
mixers.mixers[index].add(info.parsed_conditions[index], policy.isRestrictive());
|
||||
}
|
||||
}
|
||||
|
||||
auto map_of_mixed_conditions = std::make_shared<MapOfMixedConditions>();
|
||||
for (auto & [database_and_table_name, mixers] : map_of_mixers)
|
||||
{
|
||||
auto database_and_table_name_keeper = std::make_unique<DatabaseAndTableName>();
|
||||
database_and_table_name_keeper->first = database_and_table_name.first;
|
||||
database_and_table_name_keeper->second = database_and_table_name.second;
|
||||
auto & mixed_conditions = (*map_of_mixed_conditions)[DatabaseAndTableNameRef{database_and_table_name_keeper->first,
|
||||
database_and_table_name_keeper->second}];
|
||||
mixed_conditions.database_and_table_name_keeper = std::move(database_and_table_name_keeper);
|
||||
mixed_conditions.policy_ids = std::move(mixers.policy_ids);
|
||||
for (auto index : ext::range(0, MAX_CONDITION_INDEX))
|
||||
mixed_conditions.mixed_conditions[index] = std::move(mixers.mixers[index]).getResult();
|
||||
}
|
||||
|
||||
std::atomic_store(&context.atomic_map_of_mixed_conditions, std::shared_ptr<const MapOfMixedConditions>{map_of_mixed_conditions});
|
||||
}
|
||||
|
||||
}
|
||||
54
dbms/src/Access/RowPolicyContextFactory.h
Normal file
54
dbms/src/Access/RowPolicyContextFactory.h
Normal file
@ -0,0 +1,54 @@
|
||||
#pragma once
|
||||
|
||||
#include <Access/RowPolicyContext.h>
|
||||
#include <Access/IAccessStorage.h>
|
||||
#include <mutex>
|
||||
#include <unordered_map>
|
||||
#include <unordered_set>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
class AccessControlManager;
|
||||
|
||||
|
||||
/// Stores read and parsed row policies.
|
||||
class RowPolicyContextFactory
|
||||
{
|
||||
public:
|
||||
RowPolicyContextFactory(const AccessControlManager & access_control_manager_);
|
||||
~RowPolicyContextFactory();
|
||||
|
||||
RowPolicyContextPtr createContext(const String & user_name);
|
||||
|
||||
private:
|
||||
using ParsedConditions = RowPolicyContext::ParsedConditions;
|
||||
|
||||
struct PolicyInfo
|
||||
{
|
||||
PolicyInfo(const RowPolicyPtr & policy_) { setPolicy(policy_); }
|
||||
void setPolicy(const RowPolicyPtr & policy_);
|
||||
bool canUseWithContext(const RowPolicyContext & context) const;
|
||||
|
||||
RowPolicyPtr policy;
|
||||
std::unordered_set<String> roles;
|
||||
bool all_roles = false;
|
||||
std::unordered_set<String> except_roles;
|
||||
ParsedConditions parsed_conditions;
|
||||
};
|
||||
|
||||
void ensureAllRowPoliciesRead();
|
||||
void rowPolicyAddedOrChanged(const UUID & policy_id, const RowPolicyPtr & new_policy);
|
||||
void rowPolicyRemoved(const UUID & policy_id);
|
||||
void mixConditionsForAllContexts();
|
||||
void mixConditionsForContext(RowPolicyContext & context);
|
||||
|
||||
const AccessControlManager & access_control_manager;
|
||||
std::unordered_map<UUID, PolicyInfo> all_policies;
|
||||
bool all_policies_read = false;
|
||||
IAccessStorage::SubscriptionPtr subscription;
|
||||
std::vector<std::weak_ptr<RowPolicyContext>> contexts;
|
||||
std::mutex mutex;
|
||||
};
|
||||
|
||||
}
|
||||
@ -217,9 +217,18 @@ const SettingsConstraints::Constraint * SettingsConstraints::tryGetConstraint(si
|
||||
|
||||
void SettingsConstraints::setProfile(const String & profile_name, const Poco::Util::AbstractConfiguration & config)
|
||||
{
|
||||
String parent_profile = "profiles." + profile_name + ".profile";
|
||||
if (config.has(parent_profile))
|
||||
setProfile(parent_profile, config); // Inheritance of one profile from another.
|
||||
String elem = "profiles." + profile_name;
|
||||
|
||||
Poco::Util::AbstractConfiguration::Keys config_keys;
|
||||
config.keys(elem, config_keys);
|
||||
|
||||
for (const std::string & key : config_keys)
|
||||
{
|
||||
if (key == "profile" || 0 == key.compare(0, strlen("profile["), "profile[")) /// Inheritance of profiles from the current one.
|
||||
setProfile(config.getString(elem + "." + key), config);
|
||||
else
|
||||
continue;
|
||||
}
|
||||
|
||||
String path_to_constraints = "profiles." + profile_name + ".constraints";
|
||||
if (config.has(path_to_constraints))
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#include <Access/UsersConfigAccessStorage.h>
|
||||
#include <Access/Quota.h>
|
||||
#include <Access/RowPolicy.h>
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
#include <Common/quoteString.h>
|
||||
#include <Poco/Util/AbstractConfiguration.h>
|
||||
@ -15,6 +16,8 @@ namespace
|
||||
{
|
||||
if (type == typeid(Quota))
|
||||
return 'Q';
|
||||
if (type == typeid(RowPolicy))
|
||||
return 'P';
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -112,6 +115,57 @@ namespace
|
||||
}
|
||||
return quotas;
|
||||
}
|
||||
|
||||
|
||||
std::vector<AccessEntityPtr> parseRowPolicies(const Poco::Util::AbstractConfiguration & config, Poco::Logger * log)
|
||||
{
|
||||
std::vector<AccessEntityPtr> policies;
|
||||
Poco::Util::AbstractConfiguration::Keys user_names;
|
||||
config.keys("users", user_names);
|
||||
|
||||
for (const String & user_name : user_names)
|
||||
{
|
||||
const String databases_config = "users." + user_name + ".databases";
|
||||
if (config.has(databases_config))
|
||||
{
|
||||
Poco::Util::AbstractConfiguration::Keys databases;
|
||||
config.keys(databases_config, databases);
|
||||
|
||||
/// Read tables within databases
|
||||
for (const String & database : databases)
|
||||
{
|
||||
const String database_config = databases_config + "." + database;
|
||||
Poco::Util::AbstractConfiguration::Keys table_names;
|
||||
config.keys(database_config, table_names);
|
||||
|
||||
/// Read table properties
|
||||
for (const String & table_name : table_names)
|
||||
{
|
||||
const auto filter_config = database_config + "." + table_name + ".filter";
|
||||
if (config.has(filter_config))
|
||||
{
|
||||
try
|
||||
{
|
||||
auto policy = std::make_shared<RowPolicy>();
|
||||
policy->setFullName(database, table_name, user_name);
|
||||
policy->conditions[RowPolicy::SELECT_FILTER] = config.getString(filter_config);
|
||||
policy->roles.push_back(user_name);
|
||||
policies.push_back(policy);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(
|
||||
log,
|
||||
"Could not parse row policy " + backQuote(user_name) + " on table " + backQuoteIfNeed(database) + "."
|
||||
+ backQuoteIfNeed(table_name));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return policies;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -128,6 +182,8 @@ void UsersConfigAccessStorage::loadFromConfig(const Poco::Util::AbstractConfigur
|
||||
std::vector<std::pair<UUID, AccessEntityPtr>> all_entities;
|
||||
for (const auto & entity : parseQuotas(config, getLogger()))
|
||||
all_entities.emplace_back(generateID(*entity), entity);
|
||||
for (const auto & entity : parseRowPolicies(config, getLogger()))
|
||||
all_entities.emplace_back(generateID(*entity), entity);
|
||||
memory_storage.setAll(all_entities);
|
||||
}
|
||||
|
||||
|
||||
119
dbms/src/AggregateFunctions/AggregateFunctionAggThrow.cpp
Normal file
119
dbms/src/AggregateFunctions/AggregateFunctionAggThrow.cpp
Normal file
@ -0,0 +1,119 @@
|
||||
#include <memory>
|
||||
#include <random>
|
||||
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <Common/thread_local_rng.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
#include <AggregateFunctions/IAggregateFunction.h>
|
||||
#include <AggregateFunctions/AggregateFunctionFactory.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int AGGREGATE_FUNCTION_THROW;
|
||||
extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
struct AggregateFunctionThrowData
|
||||
{
|
||||
bool allocated;
|
||||
|
||||
AggregateFunctionThrowData() : allocated(true) {}
|
||||
~AggregateFunctionThrowData()
|
||||
{
|
||||
volatile bool * allocated_ptr = &allocated;
|
||||
|
||||
if (*allocated_ptr)
|
||||
*allocated_ptr = false;
|
||||
else
|
||||
abort();
|
||||
}
|
||||
};
|
||||
|
||||
/** Throw on creation with probability specified in parameter.
|
||||
* It will check correct destruction of the state.
|
||||
* This is intended to check for exception safety.
|
||||
*/
|
||||
class AggregateFunctionThrow final : public IAggregateFunctionDataHelper<AggregateFunctionThrowData, AggregateFunctionThrow>
|
||||
{
|
||||
private:
|
||||
Float64 throw_probability;
|
||||
|
||||
public:
|
||||
AggregateFunctionThrow(const DataTypes & argument_types_, const Array & parameters_, Float64 throw_probability_)
|
||||
: IAggregateFunctionDataHelper(argument_types_, parameters_), throw_probability(throw_probability_) {}
|
||||
|
||||
String getName() const override
|
||||
{
|
||||
return "aggThrow";
|
||||
}
|
||||
|
||||
DataTypePtr getReturnType() const override
|
||||
{
|
||||
return std::make_shared<DataTypeUInt8>();
|
||||
}
|
||||
|
||||
void create(AggregateDataPtr place) const override
|
||||
{
|
||||
if (std::uniform_real_distribution<>(0.0, 1.0)(thread_local_rng) <= throw_probability)
|
||||
throw Exception("Aggregate function " + getName() + " has thrown exception successfully", ErrorCodes::AGGREGATE_FUNCTION_THROW);
|
||||
|
||||
new (place) Data;
|
||||
}
|
||||
|
||||
void destroy(AggregateDataPtr place) const noexcept override
|
||||
{
|
||||
data(place).~Data();
|
||||
}
|
||||
|
||||
void add(AggregateDataPtr, const IColumn **, size_t, Arena *) const override
|
||||
{
|
||||
}
|
||||
|
||||
void merge(AggregateDataPtr, ConstAggregateDataPtr, Arena *) const override
|
||||
{
|
||||
}
|
||||
|
||||
void serialize(ConstAggregateDataPtr, WriteBuffer & buf) const override
|
||||
{
|
||||
char c = 0;
|
||||
buf.write(c);
|
||||
}
|
||||
|
||||
void deserialize(AggregateDataPtr, ReadBuffer & buf, Arena *) const override
|
||||
{
|
||||
char c = 0;
|
||||
buf.read(c);
|
||||
}
|
||||
|
||||
void insertResultInto(ConstAggregateDataPtr, IColumn & to) const override
|
||||
{
|
||||
to.insertDefault();
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
void registerAggregateFunctionAggThrow(AggregateFunctionFactory & factory)
|
||||
{
|
||||
factory.registerFunction("aggThrow", [](const std::string & name, const DataTypes & argument_types, const Array & parameters)
|
||||
{
|
||||
Float64 throw_probability = 1.0;
|
||||
if (parameters.size() == 1)
|
||||
throw_probability = parameters[0].safeGet<Float64>();
|
||||
else if (parameters.size() > 1)
|
||||
throw Exception("Aggregate function " + name + " cannot have more than one parameter", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
|
||||
return std::make_shared<AggregateFunctionThrow>(argument_types, parameters, throw_probability);
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -24,11 +24,16 @@ struct AggregateFunctionArgMinMaxData
|
||||
|
||||
ResultData result; // the argument at which the minimum/maximum value is reached.
|
||||
ValueData value; // value for which the minimum/maximum is calculated.
|
||||
|
||||
static bool allocatesMemoryInArena()
|
||||
{
|
||||
return ResultData::allocatesMemoryInArena() || ValueData::allocatesMemoryInArena();
|
||||
}
|
||||
};
|
||||
|
||||
/// Returns the first arg value found for the minimum/maximum value. Example: argMax(arg, value).
|
||||
template <typename Data, bool AllocatesMemoryInArena>
|
||||
class AggregateFunctionArgMinMax final : public IAggregateFunctionDataHelper<Data, AggregateFunctionArgMinMax<Data, AllocatesMemoryInArena>>
|
||||
template <typename Data>
|
||||
class AggregateFunctionArgMinMax final : public IAggregateFunctionDataHelper<Data, AggregateFunctionArgMinMax<Data>>
|
||||
{
|
||||
private:
|
||||
const DataTypePtr & type_res;
|
||||
@ -36,7 +41,7 @@ private:
|
||||
|
||||
public:
|
||||
AggregateFunctionArgMinMax(const DataTypePtr & type_res_, const DataTypePtr & type_val_)
|
||||
: IAggregateFunctionDataHelper<Data, AggregateFunctionArgMinMax<Data, AllocatesMemoryInArena>>({type_res_, type_val_}, {}),
|
||||
: IAggregateFunctionDataHelper<Data, AggregateFunctionArgMinMax<Data>>({type_res_, type_val_}, {}),
|
||||
type_res(this->argument_types[0]), type_val(this->argument_types[1])
|
||||
{
|
||||
if (!type_val->isComparable())
|
||||
@ -77,7 +82,7 @@ public:
|
||||
|
||||
bool allocatesMemoryInArena() const override
|
||||
{
|
||||
return AllocatesMemoryInArena;
|
||||
return Data::allocatesMemoryInArena();
|
||||
}
|
||||
|
||||
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
|
||||
|
||||
@ -129,9 +129,9 @@ public:
|
||||
|
||||
void add(AggregateDataPtr place, const IColumn ** columns, const size_t row_num, Arena *) const override
|
||||
{
|
||||
/// TODO Inefficient.
|
||||
const auto x = applyVisitor(FieldVisitorConvertToNumber<Float64>(), (*columns[0])[row_num]);
|
||||
const auto y = applyVisitor(FieldVisitorConvertToNumber<Float64>(), (*columns[1])[row_num]);
|
||||
/// NOTE Slightly inefficient.
|
||||
const auto x = columns[0]->getFloat64(row_num);
|
||||
const auto y = columns[1]->getFloat64(row_num);
|
||||
data(place).add(x, y);
|
||||
}
|
||||
|
||||
|
||||
@ -166,6 +166,11 @@ public:
|
||||
{
|
||||
return has() && assert_cast<const ColVecType &>(column).getData()[row_num] == value;
|
||||
}
|
||||
|
||||
static bool allocatesMemoryInArena()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -384,6 +389,11 @@ public:
|
||||
{
|
||||
return has() && assert_cast<const ColumnString &>(column).getDataAtWithTerminatingZero(row_num) == getStringRef();
|
||||
}
|
||||
|
||||
static bool allocatesMemoryInArena()
|
||||
{
|
||||
return true;
|
||||
}
|
||||
};
|
||||
|
||||
static_assert(
|
||||
@ -555,6 +565,11 @@ public:
|
||||
{
|
||||
return has() && to.value == value;
|
||||
}
|
||||
|
||||
static bool allocatesMemoryInArena()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -675,15 +690,15 @@ struct AggregateFunctionAnyHeavyData : Data
|
||||
};
|
||||
|
||||
|
||||
template <typename Data, bool use_arena>
|
||||
class AggregateFunctionsSingleValue final : public IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, use_arena>>
|
||||
template <typename Data>
|
||||
class AggregateFunctionsSingleValue final : public IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data>>
|
||||
{
|
||||
private:
|
||||
DataTypePtr & type;
|
||||
|
||||
public:
|
||||
AggregateFunctionsSingleValue(const DataTypePtr & type_)
|
||||
: IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data, use_arena>>({type_}, {})
|
||||
: IAggregateFunctionDataHelper<Data, AggregateFunctionsSingleValue<Data>>({type_}, {})
|
||||
, type(this->argument_types[0])
|
||||
{
|
||||
if (StringRef(Data::name()) == StringRef("min")
|
||||
@ -724,7 +739,7 @@ public:
|
||||
|
||||
bool allocatesMemoryInArena() const override
|
||||
{
|
||||
return use_arena;
|
||||
return Data::allocatesMemoryInArena();
|
||||
}
|
||||
|
||||
void insertResultInto(ConstAggregateDataPtr place, IColumn & to) const override
|
||||
|
||||
@ -100,8 +100,19 @@ public:
|
||||
void create(AggregateDataPtr place) const override
|
||||
{
|
||||
for (size_t i = 0; i < total; ++i)
|
||||
{
|
||||
try
|
||||
{
|
||||
nested_function->create(place + i * size_of_data);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
for (size_t j = 0; j < i; ++j)
|
||||
nested_function->destroy(place + j * size_of_data);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void destroy(AggregateDataPtr place) const noexcept override
|
||||
{
|
||||
|
||||
@ -23,13 +23,13 @@ inline void assertNoParameters(const std::string & name, const Array & parameter
|
||||
inline void assertUnary(const std::string & name, const DataTypes & argument_types)
|
||||
{
|
||||
if (argument_types.size() != 1)
|
||||
throw Exception("Aggregate function " + name + " require single argument", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
throw Exception("Aggregate function " + name + " requires single argument", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
}
|
||||
|
||||
inline void assertBinary(const std::string & name, const DataTypes & argument_types)
|
||||
{
|
||||
if (argument_types.size() != 2)
|
||||
throw Exception("Aggregate function " + name + " require two arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
throw Exception("Aggregate function " + name + " requires two arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
|
||||
}
|
||||
|
||||
template<std::size_t maximal_arity>
|
||||
|
||||
@ -13,8 +13,8 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// min, max, any, anyLast
|
||||
template <template <typename, bool> class AggregateFunctionTemplate, template <typename> class Data>
|
||||
/// min, max, any, anyLast, anyHeavy, etc...
|
||||
template <template <typename> class AggregateFunctionTemplate, template <typename> class Data>
|
||||
static IAggregateFunction * createAggregateFunctionSingleValue(const String & name, const DataTypes & argument_types, const Array & parameters)
|
||||
{
|
||||
assertNoParameters(name, parameters);
|
||||
@ -24,26 +24,26 @@ static IAggregateFunction * createAggregateFunctionSingleValue(const String & na
|
||||
|
||||
WhichDataType which(argument_type);
|
||||
#define DISPATCH(TYPE) \
|
||||
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<Data<SingleValueDataFixed<TYPE>>, false>(argument_type);
|
||||
if (which.idx == TypeIndex::TYPE) return new AggregateFunctionTemplate<Data<SingleValueDataFixed<TYPE>>>(argument_type);
|
||||
FOR_NUMERIC_TYPES(DISPATCH)
|
||||
#undef DISPATCH
|
||||
|
||||
if (which.idx == TypeIndex::Date)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDate::FieldType>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDate::FieldType>>>(argument_type);
|
||||
if (which.idx == TypeIndex::DateTime)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDateTime::FieldType>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DataTypeDateTime::FieldType>>>(argument_type);
|
||||
if (which.idx == TypeIndex::DateTime64)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DateTime64>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DateTime64>>>(argument_type);
|
||||
if (which.idx == TypeIndex::Decimal32)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal32>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal32>>>(argument_type);
|
||||
if (which.idx == TypeIndex::Decimal64)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal64>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal64>>>(argument_type);
|
||||
if (which.idx == TypeIndex::Decimal128)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal128>>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Decimal128>>>(argument_type);
|
||||
if (which.idx == TypeIndex::String)
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataString>, true>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataString>>(argument_type);
|
||||
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataGeneric>, false>(argument_type);
|
||||
return new AggregateFunctionTemplate<Data<SingleValueDataGeneric>>(argument_type);
|
||||
}
|
||||
|
||||
|
||||
@ -52,28 +52,29 @@ template <template <typename> class MinMaxData, typename ResData>
|
||||
static IAggregateFunction * createAggregateFunctionArgMinMaxSecond(const DataTypePtr & res_type, const DataTypePtr & val_type)
|
||||
{
|
||||
WhichDataType which(val_type);
|
||||
|
||||
#define DISPATCH(TYPE) \
|
||||
if (which.idx == TypeIndex::TYPE) \
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<TYPE>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<TYPE>>>>(res_type, val_type);
|
||||
FOR_NUMERIC_TYPES(DISPATCH)
|
||||
#undef DISPATCH
|
||||
|
||||
if (which.idx == TypeIndex::Date)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDate::FieldType>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDate::FieldType>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::DateTime)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDateTime::FieldType>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DataTypeDateTime::FieldType>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::DateTime64)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DateTime64>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<DateTime64>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::Decimal32)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal32>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal32>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::Decimal64)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal64>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal64>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::Decimal128)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal128>>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataFixed<Decimal128>>>>(res_type, val_type);
|
||||
if (which.idx == TypeIndex::String)
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataString>>, true>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataString>>>(res_type, val_type);
|
||||
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataGeneric>>, false>(res_type, val_type);
|
||||
return new AggregateFunctionArgMinMax<AggregateFunctionArgMinMaxData<ResData, MinMaxData<SingleValueDataGeneric>>>(res_type, val_type);
|
||||
}
|
||||
|
||||
template <template <typename> class MinMaxData>
|
||||
|
||||
@ -131,9 +131,7 @@ public:
|
||||
/** Contains a loop with calls to "add" function. You can collect arguments into array "places"
|
||||
* and do a single call to "addBatch" for devirtualization and inlining.
|
||||
*/
|
||||
virtual void
|
||||
addBatch(size_t batch_size, AggregateDataPtr * places, size_t place_offset, const IColumn ** columns, Arena * arena)
|
||||
const = 0;
|
||||
virtual void addBatch(size_t batch_size, AggregateDataPtr * places, size_t place_offset, const IColumn ** columns, Arena * arena) const = 0;
|
||||
|
||||
/** The same for single place.
|
||||
*/
|
||||
@ -144,9 +142,8 @@ public:
|
||||
* -Array combinator. It might also be used generally to break data dependency when array
|
||||
* "places" contains a large number of same values consecutively.
|
||||
*/
|
||||
virtual void
|
||||
addBatchArray(size_t batch_size, AggregateDataPtr * places, size_t place_offset, const IColumn ** columns, const UInt64 * offsets, Arena * arena)
|
||||
const = 0;
|
||||
virtual void addBatchArray(
|
||||
size_t batch_size, AggregateDataPtr * places, size_t place_offset, const IColumn ** columns, const UInt64 * offsets, Arena * arena) const = 0;
|
||||
|
||||
const DataTypes & getArgumentTypes() const { return argument_types; }
|
||||
const Array & getParameters() const { return parameters; }
|
||||
|
||||
@ -42,6 +42,7 @@ void registerAggregateFunctions()
|
||||
registerAggregateFunctionSimpleLinearRegression(factory);
|
||||
registerAggregateFunctionMoving(factory);
|
||||
registerAggregateFunctionCategoricalIV(factory);
|
||||
registerAggregateFunctionAggThrow(factory);
|
||||
}
|
||||
|
||||
{
|
||||
|
||||
@ -34,6 +34,7 @@ void registerAggregateFunctionEntropy(AggregateFunctionFactory &);
|
||||
void registerAggregateFunctionSimpleLinearRegression(AggregateFunctionFactory &);
|
||||
void registerAggregateFunctionMoving(AggregateFunctionFactory &);
|
||||
void registerAggregateFunctionCategoricalIV(AggregateFunctionFactory &);
|
||||
void registerAggregateFunctionAggThrow(AggregateFunctionFactory &);
|
||||
|
||||
class AggregateFunctionCombinatorFactory;
|
||||
void registerAggregateFunctionCombinatorIf(AggregateFunctionCombinatorFactory &);
|
||||
|
||||
@ -212,21 +212,23 @@ public:
|
||||
Float64 getFloat64(size_t n) const override;
|
||||
Float32 getFloat32(size_t n) const override;
|
||||
|
||||
UInt64 getUInt(size_t n) const override
|
||||
/// Out of range conversion is permitted.
|
||||
UInt64 NO_SANITIZE_UNDEFINED getUInt(size_t n) const override
|
||||
{
|
||||
return UInt64(data[n]);
|
||||
}
|
||||
|
||||
/// Out of range conversion is permitted.
|
||||
Int64 NO_SANITIZE_UNDEFINED getInt(size_t n) const override
|
||||
{
|
||||
return Int64(data[n]);
|
||||
}
|
||||
|
||||
bool getBool(size_t n) const override
|
||||
{
|
||||
return bool(data[n]);
|
||||
}
|
||||
|
||||
Int64 getInt(size_t n) const override
|
||||
{
|
||||
return Int64(data[n]);
|
||||
}
|
||||
|
||||
void insert(const Field & x) override
|
||||
{
|
||||
data.push_back(DB::get<NearestFieldType<T>>(x));
|
||||
|
||||
@ -127,8 +127,8 @@ public:
|
||||
uint64_t line = 0;
|
||||
};
|
||||
|
||||
/**
|
||||
* Find the file and line number information corresponding to address.
|
||||
/** Find the file and line number information corresponding to address.
|
||||
* The address must be physical - offset in object file without offset in virtual memory where the object is loaded.
|
||||
*/
|
||||
bool findAddress(uintptr_t address, LocationInfo & info, LocationInfoMode mode) const;
|
||||
|
||||
|
||||
@ -138,7 +138,6 @@ namespace ErrorCodes
|
||||
extern const int FUNCTION_IS_SPECIAL = 129;
|
||||
extern const int CANNOT_READ_ARRAY_FROM_TEXT = 130;
|
||||
extern const int TOO_LARGE_STRING_SIZE = 131;
|
||||
extern const int CANNOT_CREATE_TABLE_FROM_METADATA = 132;
|
||||
extern const int AGGREGATE_FUNCTION_DOESNT_ALLOW_PARAMETERS = 133;
|
||||
extern const int PARAMETERS_TO_AGGREGATE_FUNCTIONS_MUST_BE_LITERALS = 134;
|
||||
extern const int ZERO_ARRAY_OR_TUPLE_INDEX = 135;
|
||||
@ -474,10 +473,10 @@ namespace ErrorCodes
|
||||
extern const int NOT_ENOUGH_PRIVILEGES = 497;
|
||||
extern const int LIMIT_BY_WITH_TIES_IS_NOT_SUPPORTED = 498;
|
||||
extern const int S3_ERROR = 499;
|
||||
extern const int CANNOT_CREATE_DICTIONARY_FROM_METADATA = 500;
|
||||
extern const int CANNOT_CREATE_DATABASE = 501;
|
||||
extern const int CANNOT_SIGQUEUE = 502;
|
||||
extern const int UNKNOWN_PART_TYPE = 503;
|
||||
extern const int AGGREGATE_FUNCTION_THROW = 503;
|
||||
extern const int UNKNOWN_PART_TYPE = 504;
|
||||
|
||||
extern const int KEEPER_EXCEPTION = 999;
|
||||
extern const int POCO_EXCEPTION = 1000;
|
||||
|
||||
@ -25,6 +25,55 @@ namespace ErrorCodes
|
||||
extern const int NOT_IMPLEMENTED;
|
||||
}
|
||||
|
||||
|
||||
Exception::Exception()
|
||||
{
|
||||
}
|
||||
|
||||
Exception::Exception(const std::string & msg, int code)
|
||||
: Poco::Exception(msg, code)
|
||||
{
|
||||
}
|
||||
|
||||
Exception::Exception(CreateFromPocoTag, const Poco::Exception & exc)
|
||||
: Poco::Exception(exc.displayText(), ErrorCodes::POCO_EXCEPTION)
|
||||
{
|
||||
#ifdef STD_EXCEPTION_HAS_STACK_TRACE
|
||||
set_stack_trace(exc.get_stack_trace_frames(), exc.get_stack_trace_size());
|
||||
#endif
|
||||
}
|
||||
|
||||
Exception::Exception(CreateFromSTDTag, const std::exception & exc)
|
||||
: Poco::Exception(String(typeid(exc).name()) + ": " + String(exc.what()), ErrorCodes::STD_EXCEPTION)
|
||||
{
|
||||
#ifdef STD_EXCEPTION_HAS_STACK_TRACE
|
||||
set_stack_trace(exc.get_stack_trace_frames(), exc.get_stack_trace_size());
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
std::string getExceptionStackTraceString(const std::exception & e)
|
||||
{
|
||||
#ifdef STD_EXCEPTION_HAS_STACK_TRACE
|
||||
return StackTrace::toString(e.get_stack_trace_frames(), 0, e.get_stack_trace_size());
|
||||
#else
|
||||
if (const auto * db_exception = dynamic_cast<const Exception *>(&e))
|
||||
return db_exception->getStackTraceString();
|
||||
return {};
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
std::string Exception::getStackTraceString() const
|
||||
{
|
||||
#ifdef STD_EXCEPTION_HAS_STACK_TRACE
|
||||
return StackTrace::toString(get_stack_trace_frames(), 0, get_stack_trace_size());
|
||||
#else
|
||||
return trace.toString();
|
||||
#endif
|
||||
}
|
||||
|
||||
|
||||
std::string errnoToString(int code, int e)
|
||||
{
|
||||
const size_t buf_size = 128;
|
||||
@ -141,6 +190,7 @@ std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded
|
||||
{
|
||||
stream << "Poco::Exception. Code: " << ErrorCodes::POCO_EXCEPTION << ", e.code() = " << e.code()
|
||||
<< ", e.displayText() = " << e.displayText()
|
||||
<< (with_stacktrace ? getExceptionStackTraceString(e) : "")
|
||||
<< (with_extra_info ? getExtraExceptionInfo(e) : "")
|
||||
<< " (version " << VERSION_STRING << VERSION_OFFICIAL;
|
||||
}
|
||||
@ -157,6 +207,7 @@ std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded
|
||||
name += " (demangling status: " + toString(status) + ")";
|
||||
|
||||
stream << "std::exception. Code: " << ErrorCodes::STD_EXCEPTION << ", type: " << name << ", e.what() = " << e.what()
|
||||
<< (with_stacktrace ? getExceptionStackTraceString(e) : "")
|
||||
<< (with_extra_info ? getExtraExceptionInfo(e) : "")
|
||||
<< ", version = " << VERSION_STRING << VERSION_OFFICIAL;
|
||||
}
|
||||
@ -261,7 +312,7 @@ std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool
|
||||
stream << "Code: " << e.code() << ", e.displayText() = " << text;
|
||||
|
||||
if (with_stacktrace && !has_embedded_stack_trace)
|
||||
stream << ", Stack trace (when copying this message, always include the lines below):\n\n" << e.getStackTrace().toString();
|
||||
stream << ", Stack trace (when copying this message, always include the lines below):\n\n" << e.getStackTraceString();
|
||||
}
|
||||
catch (...) {}
|
||||
|
||||
|
||||
@ -22,13 +22,14 @@ namespace ErrorCodes
|
||||
class Exception : public Poco::Exception
|
||||
{
|
||||
public:
|
||||
Exception() {} /// For deferred initialization.
|
||||
Exception(const std::string & msg, int code) : Poco::Exception(msg, code) {}
|
||||
Exception(const std::string & msg, const Exception & nested_exception, int code)
|
||||
: Poco::Exception(msg, nested_exception, code), trace(nested_exception.trace) {}
|
||||
Exception();
|
||||
Exception(const std::string & msg, int code);
|
||||
|
||||
enum CreateFromPocoTag { CreateFromPoco };
|
||||
Exception(CreateFromPocoTag, const Poco::Exception & exc) : Poco::Exception(exc.displayText(), ErrorCodes::POCO_EXCEPTION) {}
|
||||
enum CreateFromSTDTag { CreateFromSTD };
|
||||
|
||||
Exception(CreateFromPocoTag, const Poco::Exception & exc);
|
||||
Exception(CreateFromSTDTag, const std::exception & exc);
|
||||
|
||||
Exception * clone() const override { return new Exception(*this); }
|
||||
void rethrow() const override { throw *this; }
|
||||
@ -38,15 +39,20 @@ public:
|
||||
/// Add something to the existing message.
|
||||
void addMessage(const std::string & arg) { extendedMessage(arg); }
|
||||
|
||||
const StackTrace & getStackTrace() const { return trace; }
|
||||
std::string getStackTraceString() const;
|
||||
|
||||
private:
|
||||
#ifndef STD_EXCEPTION_HAS_STACK_TRACE
|
||||
StackTrace trace;
|
||||
#endif
|
||||
|
||||
const char * className() const throw() override { return "DB::Exception"; }
|
||||
};
|
||||
|
||||
|
||||
std::string getExceptionStackTraceString(const std::exception & e);
|
||||
|
||||
|
||||
/// Contains an additional member `saved_errno`. See the throwFromErrno function.
|
||||
class ErrnoException : public Exception
|
||||
{
|
||||
|
||||
@ -23,11 +23,10 @@ struct TrivialWeightFunction
|
||||
};
|
||||
|
||||
|
||||
/// Thread-safe cache that evicts entries which are not used for a long time or are expired.
|
||||
/// Thread-safe cache that evicts entries which are not used for a long time.
|
||||
/// WeightFunction is a functor that takes Mapped as a parameter and returns "weight" (approximate size)
|
||||
/// of that value.
|
||||
/// Cache starts to evict entries when their total weight exceeds max_size and when expiration time of these
|
||||
/// entries is due.
|
||||
/// Cache starts to evict entries when their total weight exceeds max_size.
|
||||
/// Value weight should not change after insertion.
|
||||
template <typename TKey, typename TMapped, typename HashFunction = std::hash<TKey>, typename WeightFunction = TrivialWeightFunction<TMapped>>
|
||||
class LRUCache
|
||||
@ -36,15 +35,13 @@ public:
|
||||
using Key = TKey;
|
||||
using Mapped = TMapped;
|
||||
using MappedPtr = std::shared_ptr<Mapped>;
|
||||
using Delay = std::chrono::seconds;
|
||||
|
||||
private:
|
||||
using Clock = std::chrono::steady_clock;
|
||||
using Timestamp = Clock::time_point;
|
||||
|
||||
public:
|
||||
LRUCache(size_t max_size_, const Delay & expiration_delay_ = Delay::zero())
|
||||
: max_size(std::max(static_cast<size_t>(1), max_size_)), expiration_delay(expiration_delay_) {}
|
||||
LRUCache(size_t max_size_)
|
||||
: max_size(std::max(static_cast<size_t>(1), max_size_)) {}
|
||||
|
||||
MappedPtr get(const Key & key)
|
||||
{
|
||||
@ -167,16 +164,9 @@ protected:
|
||||
|
||||
struct Cell
|
||||
{
|
||||
bool expired(const Timestamp & last_timestamp, const Delay & delay) const
|
||||
{
|
||||
return (delay == Delay::zero()) ||
|
||||
((last_timestamp > timestamp) && ((last_timestamp - timestamp) > delay));
|
||||
}
|
||||
|
||||
MappedPtr value;
|
||||
size_t size;
|
||||
LRUQueueIterator queue_iterator;
|
||||
Timestamp timestamp;
|
||||
};
|
||||
|
||||
using Cells = std::unordered_map<Key, Cell, HashFunction>;
|
||||
@ -257,7 +247,6 @@ private:
|
||||
/// Total weight of values.
|
||||
size_t current_size = 0;
|
||||
const size_t max_size;
|
||||
const Delay expiration_delay;
|
||||
|
||||
std::atomic<size_t> hits {0};
|
||||
std::atomic<size_t> misses {0};
|
||||
@ -273,7 +262,6 @@ private:
|
||||
}
|
||||
|
||||
Cell & cell = it->second;
|
||||
updateCellTimestamp(cell);
|
||||
|
||||
/// Move the key to the end of the queue. The iterator remains valid.
|
||||
queue.splice(queue.end(), queue, cell.queue_iterator);
|
||||
@ -303,18 +291,11 @@ private:
|
||||
cell.value = mapped;
|
||||
cell.size = cell.value ? weight_function(*cell.value) : 0;
|
||||
current_size += cell.size;
|
||||
updateCellTimestamp(cell);
|
||||
|
||||
removeOverflow(cell.timestamp);
|
||||
removeOverflow();
|
||||
}
|
||||
|
||||
void updateCellTimestamp(Cell & cell)
|
||||
{
|
||||
if (expiration_delay != Delay::zero())
|
||||
cell.timestamp = Clock::now();
|
||||
}
|
||||
|
||||
void removeOverflow(const Timestamp & last_timestamp)
|
||||
void removeOverflow()
|
||||
{
|
||||
size_t current_weight_lost = 0;
|
||||
size_t queue_size = cells.size();
|
||||
@ -330,8 +311,6 @@ private:
|
||||
}
|
||||
|
||||
const auto & cell = it->second;
|
||||
if (!cell.expired(last_timestamp, expiration_delay))
|
||||
break;
|
||||
|
||||
current_size -= cell.size;
|
||||
current_weight_lost += cell.size;
|
||||
|
||||
@ -37,6 +37,8 @@
|
||||
M(CreatedReadBufferOrdinary, "") \
|
||||
M(CreatedReadBufferAIO, "") \
|
||||
M(CreatedReadBufferAIOFailed, "") \
|
||||
M(CreatedReadBufferMMap, "") \
|
||||
M(CreatedReadBufferMMapFailed, "") \
|
||||
M(CreatedWriteBufferOrdinary, "") \
|
||||
M(CreatedWriteBufferAIO, "") \
|
||||
M(CreatedWriteBufferAIOFailed, "") \
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
#include <Common/Elf.h>
|
||||
#include <Common/SymbolIndex.h>
|
||||
#include <Common/config.h>
|
||||
#include <Common/MemorySanitizer.h>
|
||||
#include <common/SimpleCache.h>
|
||||
#include <common/demangle.h>
|
||||
#include <Core/Defines.h>
|
||||
@ -226,6 +227,7 @@ void StackTrace::tryCapture()
|
||||
size = 0;
|
||||
#if USE_UNWIND
|
||||
size = unw_backtrace(frames.data(), capacity);
|
||||
__msan_unpoison(frames.data(), size * sizeof(frames[0]));
|
||||
#endif
|
||||
}
|
||||
|
||||
@ -258,10 +260,14 @@ static void toStringEveryLineImpl(const StackTrace::Frames & frames, size_t offs
|
||||
|
||||
for (size_t i = offset; i < size; ++i)
|
||||
{
|
||||
const void * addr = frames[i];
|
||||
const void * virtual_addr = frames[i];
|
||||
auto object = symbol_index.findObject(virtual_addr);
|
||||
uintptr_t virtual_offset = object ? uintptr_t(object->address_begin) : 0;
|
||||
const void * physical_addr = reinterpret_cast<const void *>(uintptr_t(virtual_addr) - virtual_offset);
|
||||
|
||||
out << i << ". " << addr << " ";
|
||||
auto symbol = symbol_index.findSymbol(addr);
|
||||
out << i << ". " << physical_addr << " ";
|
||||
|
||||
auto symbol = symbol_index.findSymbol(virtual_addr);
|
||||
if (symbol)
|
||||
{
|
||||
int status = 0;
|
||||
@ -272,18 +278,17 @@ static void toStringEveryLineImpl(const StackTrace::Frames & frames, size_t offs
|
||||
|
||||
out << " ";
|
||||
|
||||
if (auto object = symbol_index.findObject(addr))
|
||||
if (object)
|
||||
{
|
||||
if (std::filesystem::exists(object->name))
|
||||
{
|
||||
auto dwarf_it = dwarfs.try_emplace(object->name, *object->elf).first;
|
||||
|
||||
DB::Dwarf::LocationInfo location;
|
||||
if (dwarf_it->second.findAddress(uintptr_t(addr) - uintptr_t(object->address_begin), location, DB::Dwarf::LocationInfoMode::FAST))
|
||||
if (dwarf_it->second.findAddress(uintptr_t(physical_addr), location, DB::Dwarf::LocationInfoMode::FAST))
|
||||
out << location.file.toString() << ":" << location.line;
|
||||
else
|
||||
out << object->name;
|
||||
}
|
||||
out << " in " << object->name;
|
||||
}
|
||||
else
|
||||
out << "?";
|
||||
@ -325,3 +330,15 @@ std::string StackTrace::toString() const
|
||||
static SimpleCache<decltype(toStringImpl), &toStringImpl> func_cached;
|
||||
return func_cached(frames, offset, size);
|
||||
}
|
||||
|
||||
std::string StackTrace::toString(void ** frames_, size_t offset, size_t size)
|
||||
{
|
||||
__msan_unpoison(frames_, size * sizeof(*frames_));
|
||||
|
||||
StackTrace::Frames frames_copy{};
|
||||
for (size_t i = 0; i < size; ++i)
|
||||
frames_copy[i] = frames_[i];
|
||||
|
||||
static SimpleCache<decltype(toStringImpl), &toStringImpl> func_cached;
|
||||
return func_cached(frames_copy, offset, size);
|
||||
}
|
||||
|
||||
@ -41,6 +41,8 @@ public:
|
||||
const Frames & getFrames() const;
|
||||
std::string toString() const;
|
||||
|
||||
static std::string toString(void ** frames, size_t offset, size_t size);
|
||||
|
||||
void toStringEveryLine(std::function<void(const std::string &)> callback) const;
|
||||
|
||||
protected:
|
||||
|
||||
@ -38,6 +38,7 @@ public:
|
||||
std::unique_ptr<Elf> elf;
|
||||
};
|
||||
|
||||
/// Address in virtual memory should be passed. These addresses include offset where the object is loaded in memory.
|
||||
const Symbol * findSymbol(const void * address) const;
|
||||
const Object * findObject(const void * address) const;
|
||||
|
||||
|
||||
@ -13,9 +13,6 @@ target_link_libraries (sip_hash_perf PRIVATE clickhouse_common_io)
|
||||
add_executable (auto_array auto_array.cpp)
|
||||
target_link_libraries (auto_array PRIVATE clickhouse_common_io)
|
||||
|
||||
add_executable (lru_cache lru_cache.cpp)
|
||||
target_link_libraries (lru_cache PRIVATE clickhouse_common_io)
|
||||
|
||||
add_executable (hash_table hash_table.cpp)
|
||||
target_link_libraries (hash_table PRIVATE clickhouse_common_io)
|
||||
|
||||
|
||||
@ -1,317 +0,0 @@
|
||||
#include <Common/LRUCache.h>
|
||||
#include <Common/Exception.h>
|
||||
|
||||
#include <iostream>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
void run();
|
||||
void runTest(unsigned int num, const std::function<bool()> & func);
|
||||
bool test1();
|
||||
bool test2();
|
||||
bool test_concurrent();
|
||||
|
||||
#define ASSERT_CHECK(cond, res) \
|
||||
do \
|
||||
{ \
|
||||
if (!(cond)) \
|
||||
{ \
|
||||
std::cout << __FILE__ << ":" << __LINE__ << ":" \
|
||||
<< "Assertion " << #cond << " failed.\n"; \
|
||||
if ((res)) { (res) = false; } \
|
||||
} \
|
||||
} \
|
||||
while (0)
|
||||
|
||||
void run()
|
||||
{
|
||||
const std::vector<std::function<bool()>> tests =
|
||||
{
|
||||
test1,
|
||||
test2,
|
||||
test_concurrent
|
||||
};
|
||||
|
||||
unsigned int num = 0;
|
||||
for (const auto & test : tests)
|
||||
{
|
||||
++num;
|
||||
runTest(num, test);
|
||||
}
|
||||
}
|
||||
|
||||
void runTest(unsigned int num, const std::function<bool()> & func)
|
||||
{
|
||||
bool ok;
|
||||
|
||||
try
|
||||
{
|
||||
ok = func();
|
||||
}
|
||||
catch (const DB::Exception & ex)
|
||||
{
|
||||
ok = false;
|
||||
std::cout << "Caught exception " << ex.displayText() << "\n";
|
||||
}
|
||||
catch (const std::exception & ex)
|
||||
{
|
||||
ok = false;
|
||||
std::cout << "Caught exception " << ex.what() << "\n";
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
ok = false;
|
||||
std::cout << "Caught unhandled exception\n";
|
||||
}
|
||||
|
||||
if (ok)
|
||||
std::cout << "Test " << num << " passed\n";
|
||||
else
|
||||
std::cout << "Test " << num << " failed\n";
|
||||
}
|
||||
|
||||
struct Weight
|
||||
{
|
||||
size_t operator()(const std::string & s) const
|
||||
{
|
||||
return s.size();
|
||||
}
|
||||
};
|
||||
|
||||
bool test1()
|
||||
{
|
||||
using Cache = DB::LRUCache<std::string, std::string, std::hash<std::string>, Weight>;
|
||||
using MappedPtr = Cache::MappedPtr;
|
||||
|
||||
auto ptr = [](const std::string & s)
|
||||
{
|
||||
return MappedPtr(new std::string(s));
|
||||
};
|
||||
|
||||
Cache cache(10);
|
||||
|
||||
bool res = true;
|
||||
|
||||
ASSERT_CHECK(!cache.get("asd"), res);
|
||||
|
||||
cache.set("asd", ptr("qwe"));
|
||||
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
|
||||
cache.set("zxcv", ptr("12345"));
|
||||
cache.set("01234567891234567", ptr("--"));
|
||||
|
||||
ASSERT_CHECK((*cache.get("zxcv") == "12345"), res);
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res);
|
||||
ASSERT_CHECK(!cache.get("123x"), res);
|
||||
|
||||
cache.set("321x", ptr("+"));
|
||||
|
||||
ASSERT_CHECK(!cache.get("zxcv"), res);
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res);
|
||||
ASSERT_CHECK(!cache.get("123x"), res);
|
||||
ASSERT_CHECK((*cache.get("321x") == "+"), res);
|
||||
|
||||
ASSERT_CHECK((cache.weight() == 6), res);
|
||||
ASSERT_CHECK((cache.count() == 3), res);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
bool test2()
|
||||
{
|
||||
using namespace std::literals;
|
||||
using Cache = DB::LRUCache<std::string, std::string, std::hash<std::string>, Weight>;
|
||||
using MappedPtr = Cache::MappedPtr;
|
||||
|
||||
auto ptr = [](const std::string & s)
|
||||
{
|
||||
return MappedPtr(new std::string(s));
|
||||
};
|
||||
|
||||
Cache cache(10, 3s);
|
||||
|
||||
bool res = true;
|
||||
|
||||
ASSERT_CHECK(!cache.get("asd"), res);
|
||||
|
||||
cache.set("asd", ptr("qwe"));
|
||||
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
|
||||
cache.set("zxcv", ptr("12345"));
|
||||
cache.set("01234567891234567", ptr("--"));
|
||||
|
||||
ASSERT_CHECK((*cache.get("zxcv") == "12345"), res);
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res);
|
||||
ASSERT_CHECK(!cache.get("123x"), res);
|
||||
|
||||
cache.set("321x", ptr("+"));
|
||||
|
||||
ASSERT_CHECK((cache.get("zxcv")), res);
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res);
|
||||
ASSERT_CHECK(!cache.get("123x"), res);
|
||||
ASSERT_CHECK((*cache.get("321x") == "+"), res);
|
||||
|
||||
ASSERT_CHECK((cache.weight() == 11), res);
|
||||
ASSERT_CHECK((cache.count() == 4), res);
|
||||
|
||||
std::this_thread::sleep_for(5s);
|
||||
|
||||
cache.set("123x", ptr("2769"));
|
||||
|
||||
ASSERT_CHECK(!cache.get("zxcv"), res);
|
||||
ASSERT_CHECK((*cache.get("asd") == "qwe"), res);
|
||||
ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res);
|
||||
ASSERT_CHECK((*cache.get("321x") == "+"), res);
|
||||
|
||||
ASSERT_CHECK((cache.weight() == 10), res);
|
||||
ASSERT_CHECK((cache.count() == 4), res);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
bool test_concurrent()
|
||||
{
|
||||
using namespace std::literals;
|
||||
|
||||
using Cache = DB::LRUCache<std::string, std::string, std::hash<std::string>, Weight>;
|
||||
Cache cache(2);
|
||||
|
||||
bool res = true;
|
||||
|
||||
auto load_func = [](const std::string & result, std::chrono::seconds sleep_for, bool throw_exc)
|
||||
{
|
||||
std::this_thread::sleep_for(sleep_for);
|
||||
if (throw_exc)
|
||||
throw std::runtime_error("Exception!");
|
||||
return std::make_shared<std::string>(result);
|
||||
};
|
||||
|
||||
/// Case 1: Both threads are able to load the value.
|
||||
|
||||
std::pair<Cache::MappedPtr, bool> result1;
|
||||
std::thread thread1([&]()
|
||||
{
|
||||
result1 = cache.getOrSet("key", [&]() { return load_func("val1", 1s, false); });
|
||||
});
|
||||
|
||||
std::pair<Cache::MappedPtr, bool> result2;
|
||||
std::thread thread2([&]()
|
||||
{
|
||||
result2 = cache.getOrSet("key", [&]() { return load_func("val2", 1s, false); });
|
||||
});
|
||||
|
||||
thread1.join();
|
||||
thread2.join();
|
||||
|
||||
ASSERT_CHECK((result1.first == result2.first), res);
|
||||
ASSERT_CHECK((result1.second != result2.second), res);
|
||||
|
||||
/// Case 2: One thread throws an exception during loading.
|
||||
|
||||
cache.reset();
|
||||
|
||||
bool thrown = false;
|
||||
thread1 = std::thread([&]()
|
||||
{
|
||||
try
|
||||
{
|
||||
cache.getOrSet("key", [&]() { return load_func("val1", 2s, true); });
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
thrown = true;
|
||||
}
|
||||
});
|
||||
|
||||
thread2 = std::thread([&]()
|
||||
{
|
||||
std::this_thread::sleep_for(1s);
|
||||
result2 = cache.getOrSet("key", [&]() { return load_func("val2", 1s, false); });
|
||||
});
|
||||
|
||||
thread1.join();
|
||||
thread2.join();
|
||||
|
||||
ASSERT_CHECK((thrown == true), res);
|
||||
ASSERT_CHECK((result2.second == true), res);
|
||||
ASSERT_CHECK((result2.first.get() == cache.get("key").get()), res);
|
||||
ASSERT_CHECK((*result2.first == "val2"), res);
|
||||
|
||||
/// Case 3: All threads throw an exception.
|
||||
|
||||
cache.reset();
|
||||
|
||||
bool thrown1 = false;
|
||||
thread1 = std::thread([&]()
|
||||
{
|
||||
try
|
||||
{
|
||||
cache.getOrSet("key", [&]() { return load_func("val1", 1s, true); });
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
thrown1 = true;
|
||||
}
|
||||
});
|
||||
|
||||
bool thrown2 = false;
|
||||
thread2 = std::thread([&]()
|
||||
{
|
||||
try
|
||||
{
|
||||
cache.getOrSet("key", [&]() { return load_func("val1", 1s, true); });
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
thrown2 = true;
|
||||
}
|
||||
});
|
||||
|
||||
thread1.join();
|
||||
thread2.join();
|
||||
|
||||
ASSERT_CHECK((thrown1 == true), res);
|
||||
ASSERT_CHECK((thrown2 == true), res);
|
||||
ASSERT_CHECK((cache.get("key") == nullptr), res);
|
||||
|
||||
/// Case 4: Concurrent reset.
|
||||
|
||||
cache.reset();
|
||||
|
||||
thread1 = std::thread([&]()
|
||||
{
|
||||
result1 = cache.getOrSet("key", [&]() { return load_func("val1", 2s, false); });
|
||||
});
|
||||
|
||||
std::this_thread::sleep_for(1s);
|
||||
cache.reset();
|
||||
|
||||
thread1.join();
|
||||
|
||||
ASSERT_CHECK((result1.second == true), res);
|
||||
ASSERT_CHECK((*result1.first == "val1"), res);
|
||||
ASSERT_CHECK((cache.get("key") == nullptr), res);
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
int main()
|
||||
{
|
||||
run();
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -49,7 +49,7 @@ int main(int argc, char ** argv)
|
||||
Dwarf dwarf(*object->elf);
|
||||
|
||||
Dwarf::LocationInfo location;
|
||||
if (dwarf.findAddress(uintptr_t(address), location, Dwarf::LocationInfoMode::FAST))
|
||||
if (dwarf.findAddress(uintptr_t(address) - uintptr_t(info.dli_fbase), location, Dwarf::LocationInfoMode::FAST))
|
||||
std::cerr << location.file.toString() << ":" << location.line << "\n";
|
||||
else
|
||||
std::cerr << "Dwarf: Not found\n";
|
||||
|
||||
@ -19,7 +19,7 @@ void CachedCompressedReadBuffer::initInput()
|
||||
{
|
||||
if (!file_in)
|
||||
{
|
||||
file_in = createReadBufferFromFileBase(path, estimated_size, aio_threshold, buf_size);
|
||||
file_in = createReadBufferFromFileBase(path, estimated_size, aio_threshold, mmap_threshold, buf_size);
|
||||
compressed_in = file_in.get();
|
||||
|
||||
if (profile_callback)
|
||||
@ -73,10 +73,11 @@ bool CachedCompressedReadBuffer::nextImpl()
|
||||
|
||||
|
||||
CachedCompressedReadBuffer::CachedCompressedReadBuffer(
|
||||
const std::string & path_, UncompressedCache * cache_, size_t estimated_size_, size_t aio_threshold_,
|
||||
const std::string & path_, UncompressedCache * cache_,
|
||||
size_t estimated_size_, size_t aio_threshold_, size_t mmap_threshold_,
|
||||
size_t buf_size_)
|
||||
: ReadBuffer(nullptr, 0), path(path_), cache(cache_), buf_size(buf_size_), estimated_size(estimated_size_),
|
||||
aio_threshold(aio_threshold_), file_pos(0)
|
||||
aio_threshold(aio_threshold_), mmap_threshold(mmap_threshold_), file_pos(0)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
@ -26,6 +26,7 @@ private:
|
||||
size_t buf_size;
|
||||
size_t estimated_size;
|
||||
size_t aio_threshold;
|
||||
size_t mmap_threshold;
|
||||
|
||||
std::unique_ptr<ReadBufferFromFileBase> file_in;
|
||||
size_t file_pos;
|
||||
@ -42,7 +43,8 @@ private:
|
||||
|
||||
public:
|
||||
CachedCompressedReadBuffer(
|
||||
const std::string & path_, UncompressedCache * cache_, size_t estimated_size_, size_t aio_threshold_,
|
||||
const std::string & path_, UncompressedCache * cache_,
|
||||
size_t estimated_size_, size_t aio_threshold_, size_t mmap_threshold_,
|
||||
size_t buf_size_ = DBMS_DEFAULT_BUFFER_SIZE);
|
||||
|
||||
|
||||
|
||||
@ -32,6 +32,7 @@ namespace ErrorCodes
|
||||
extern const int TOO_LARGE_SIZE_COMPRESSED;
|
||||
extern const int CHECKSUM_DOESNT_MATCH;
|
||||
extern const int CANNOT_DECOMPRESS;
|
||||
extern const int CORRUPTED_DATA;
|
||||
}
|
||||
|
||||
using Checksum = CityHash_v1_0_2::uint128;
|
||||
@ -124,6 +125,10 @@ size_t CompressedReadBufferBase::readCompressedData(size_t & size_decompressed,
|
||||
+ ". Most likely corrupted data.",
|
||||
ErrorCodes::TOO_LARGE_SIZE_COMPRESSED);
|
||||
|
||||
if (size_compressed_without_checksum < header_size)
|
||||
throw Exception("Can't decompress data: the compressed data size (" + toString(size_compressed_without_checksum)
|
||||
+ ", this should include header size) is less than the header size (" + toString(header_size) + ")", ErrorCodes::CORRUPTED_DATA);
|
||||
|
||||
ProfileEvents::increment(ProfileEvents::ReadCompressedBytes, size_compressed_without_checksum + sizeof(Checksum));
|
||||
|
||||
/// Is whole compressed block located in 'compressed_in->' buffer?
|
||||
|
||||
@ -33,9 +33,9 @@ bool CompressedReadBufferFromFile::nextImpl()
|
||||
|
||||
|
||||
CompressedReadBufferFromFile::CompressedReadBufferFromFile(
|
||||
const std::string & path, size_t estimated_size, size_t aio_threshold, size_t buf_size)
|
||||
const std::string & path, size_t estimated_size, size_t aio_threshold, size_t mmap_threshold, size_t buf_size)
|
||||
: BufferWithOwnMemory<ReadBuffer>(0),
|
||||
p_file_in(createReadBufferFromFileBase(path, estimated_size, aio_threshold, buf_size)),
|
||||
p_file_in(createReadBufferFromFileBase(path, estimated_size, aio_threshold, mmap_threshold, buf_size)),
|
||||
file_in(*p_file_in)
|
||||
{
|
||||
compressed_in = &file_in;
|
||||
|
||||
@ -30,7 +30,7 @@ private:
|
||||
|
||||
public:
|
||||
CompressedReadBufferFromFile(
|
||||
const std::string & path, size_t estimated_size, size_t aio_threshold, size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE);
|
||||
const std::string & path, size_t estimated_size, size_t aio_threshold, size_t mmap_threshold, size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE);
|
||||
|
||||
void seek(size_t offset_in_compressed_file, size_t offset_in_decompressed_block);
|
||||
|
||||
|
||||
@ -26,7 +26,7 @@ extern const int CANNOT_DECOMPRESS;
|
||||
namespace
|
||||
{
|
||||
|
||||
Int64 getMaxValueForByteSize(UInt8 byte_size)
|
||||
inline Int64 getMaxValueForByteSize(Int8 byte_size)
|
||||
{
|
||||
switch (byte_size)
|
||||
{
|
||||
@ -51,11 +51,56 @@ struct WriteSpec
|
||||
const UInt8 data_bits;
|
||||
};
|
||||
|
||||
const std::array<UInt8, 5> DELTA_SIZES{7, 9, 12, 32, 64};
|
||||
// delta size prefix and data lengths based on few high bits peeked from binary stream
|
||||
static const WriteSpec WRITE_SPEC_LUT[32] = {
|
||||
// 0b0 - 1-bit prefix, no data to read
|
||||
/* 00000 */ {1, 0b0, 0},
|
||||
/* 00001 */ {1, 0b0, 0},
|
||||
/* 00010 */ {1, 0b0, 0},
|
||||
/* 00011 */ {1, 0b0, 0},
|
||||
/* 00100 */ {1, 0b0, 0},
|
||||
/* 00101 */ {1, 0b0, 0},
|
||||
/* 00110 */ {1, 0b0, 0},
|
||||
/* 00111 */ {1, 0b0, 0},
|
||||
/* 01000 */ {1, 0b0, 0},
|
||||
/* 01001 */ {1, 0b0, 0},
|
||||
/* 01010 */ {1, 0b0, 0},
|
||||
/* 01011 */ {1, 0b0, 0},
|
||||
/* 01100 */ {1, 0b0, 0},
|
||||
/* 01101 */ {1, 0b0, 0},
|
||||
/* 01110 */ {1, 0b0, 0},
|
||||
/* 01111 */ {1, 0b0, 0},
|
||||
|
||||
// 0b10 - 2 bit prefix, 7 bits of data
|
||||
/* 10000 */ {2, 0b10, 7},
|
||||
/* 10001 */ {2, 0b10, 7},
|
||||
/* 10010 */ {2, 0b10, 7},
|
||||
/* 10011 */ {2, 0b10, 7},
|
||||
/* 10100 */ {2, 0b10, 7},
|
||||
/* 10101 */ {2, 0b10, 7},
|
||||
/* 10110 */ {2, 0b10, 7},
|
||||
/* 10111 */ {2, 0b10, 7},
|
||||
|
||||
// 0b110 - 3 bit prefix, 9 bits of data
|
||||
/* 11000 */ {3, 0b110, 9},
|
||||
/* 11001 */ {3, 0b110, 9},
|
||||
/* 11010 */ {3, 0b110, 9},
|
||||
/* 11011 */ {3, 0b110, 9},
|
||||
|
||||
// 0b1110 - 4 bit prefix, 12 bits of data
|
||||
/* 11100 */ {4, 0b1110, 12},
|
||||
/* 11101 */ {4, 0b1110, 12},
|
||||
|
||||
// 5-bit prefixes
|
||||
/* 11110 */ {5, 0b11110, 32},
|
||||
/* 11111 */ {5, 0b11111, 64},
|
||||
};
|
||||
|
||||
|
||||
template <typename T>
|
||||
WriteSpec getDeltaWriteSpec(const T & value)
|
||||
{
|
||||
// TODO: to speed up things a bit by counting number of leading zeroes instead of doing lots of comparisons
|
||||
if (value > -63 && value < 64)
|
||||
{
|
||||
return WriteSpec{2, 0b10, 7};
|
||||
@ -107,14 +152,15 @@ UInt32 getCompressedDataSize(UInt8 data_bytes_size, UInt32 uncompressed_size)
|
||||
template <typename ValueType>
|
||||
UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
{
|
||||
// Since only unsinged int has granted 2-compliment overflow handling, we are doing math here on unsigned types.
|
||||
// To simplify and booletproof code, we operate enforce ValueType to be unsigned too.
|
||||
// Since only unsinged int has granted 2-complement overflow handling,
|
||||
// we are doing math here only on unsigned types.
|
||||
// To simplify and booletproof code, we enforce ValueType to be unsigned too.
|
||||
static_assert(is_unsigned_v<ValueType>, "ValueType must be unsigned.");
|
||||
using UnsignedDeltaType = ValueType;
|
||||
|
||||
// We use signed delta type to turn huge unsigned values into smaller signed:
|
||||
// ffffffff => -1
|
||||
using SignedDeltaType = typename std::make_signed<UnsignedDeltaType>::type;
|
||||
using SignedDeltaType = typename std::make_signed_t<UnsignedDeltaType>;
|
||||
|
||||
if (source_size % sizeof(ValueType) != 0)
|
||||
throw Exception("Cannot compress, data size " + toString(source_size)
|
||||
@ -149,8 +195,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
prev_value = curr_value;
|
||||
}
|
||||
|
||||
WriteBuffer buffer(dest, getCompressedDataSize(sizeof(ValueType), source_size - sizeof(ValueType)*2));
|
||||
BitWriter writer(buffer);
|
||||
BitWriter writer(dest, getCompressedDataSize(sizeof(ValueType), source_size - sizeof(ValueType)*2));
|
||||
|
||||
int item = 2;
|
||||
for (; source < source_end; source += sizeof(ValueType), ++item)
|
||||
@ -170,7 +215,8 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
else
|
||||
{
|
||||
const SignedDeltaType signed_dd = static_cast<SignedDeltaType>(double_delta);
|
||||
const auto sign = std::signbit(signed_dd);
|
||||
const auto sign = signed_dd < 0;
|
||||
|
||||
// -1 shirnks dd down to fit into number of bits, and there can't be 0, so it is OK.
|
||||
const auto abs_value = static_cast<UnsignedDeltaType>(std::abs(signed_dd) - 1);
|
||||
const auto write_spec = getDeltaWriteSpec(signed_dd);
|
||||
@ -183,7 +229,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
|
||||
writer.flush();
|
||||
|
||||
return sizeof(items_count) + sizeof(prev_value) + sizeof(prev_delta) + buffer.count();
|
||||
return sizeof(items_count) + sizeof(prev_value) + sizeof(prev_delta) + writer.count() / 8;
|
||||
}
|
||||
|
||||
template <typename ValueType>
|
||||
@ -220,35 +266,28 @@ void decompressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
dest += sizeof(prev_value);
|
||||
}
|
||||
|
||||
ReadBufferFromMemory buffer(source, source_size - sizeof(prev_value) - sizeof(prev_delta) - sizeof(items_count));
|
||||
BitReader reader(buffer);
|
||||
BitReader reader(source, source_size - sizeof(prev_value) - sizeof(prev_delta) - sizeof(items_count));
|
||||
|
||||
// since data is tightly packed, up to 1 bit per value, and last byte is padded with zeroes,
|
||||
// we have to keep track of items to avoid reading more that there is.
|
||||
for (UInt32 items_read = 2; items_read < items_count && !reader.eof(); ++items_read)
|
||||
{
|
||||
UnsignedDeltaType double_delta = 0;
|
||||
if (reader.readBit() == 1)
|
||||
{
|
||||
UInt8 i = 0;
|
||||
for (; i < sizeof(DELTA_SIZES) - 1; ++i)
|
||||
{
|
||||
const auto next_bit = reader.readBit();
|
||||
if (next_bit == 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
static_assert(sizeof(WRITE_SPEC_LUT)/sizeof(WRITE_SPEC_LUT[0]) == 32); // 5-bit prefix lookup table
|
||||
const auto write_spec = WRITE_SPEC_LUT[reader.peekByte() >> (8 - 5)]; // only 5 high bits of peeked byte value
|
||||
|
||||
reader.skipBufferedBits(write_spec.prefix_bits); // discard the prefix value, since we've already used it
|
||||
if (write_spec.data_bits != 0)
|
||||
{
|
||||
const UInt8 sign = reader.readBit();
|
||||
SignedDeltaType signed_dd = static_cast<SignedDeltaType>(reader.readBits(DELTA_SIZES[i] - 1) + 1);
|
||||
SignedDeltaType signed_dd = static_cast<SignedDeltaType>(reader.readBits(write_spec.data_bits - 1) + 1);
|
||||
if (sign)
|
||||
{
|
||||
signed_dd *= -1;
|
||||
}
|
||||
double_delta = static_cast<UnsignedDeltaType>(signed_dd);
|
||||
}
|
||||
// else if first bit is zero, no need to read more data.
|
||||
|
||||
const UnsignedDeltaType delta = double_delta + prev_delta;
|
||||
const ValueType curr_value = prev_value + delta;
|
||||
|
||||
@ -5,6 +5,92 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/** DoubleDelta column codec implementation.
|
||||
*
|
||||
* Based on Gorilla paper: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf, which was extended
|
||||
* to support 64bit types. The drawback is 1 extra bit for 32-byte wide deltas: 5-bit prefix
|
||||
* instead of 4-bit prefix.
|
||||
*
|
||||
* This codec is best used against monotonic integer sequences with constant (or almost contant)
|
||||
* stride, like event timestamp for some monitoring application.
|
||||
*
|
||||
* Given input sequence a: [a0, a1, ... an]:
|
||||
*
|
||||
* First, write number of items (sizeof(int32)*8 bits): n
|
||||
* Then write first item as is (sizeof(a[0])*8 bits): a[0]
|
||||
* Second item is written as delta (sizeof(a[0])*8 bits): a[1] - a[0]
|
||||
* Loop over remaining items and calculate double delta:
|
||||
* double_delta = a[i] - 2 * a[i - 1] + a[i - 2]
|
||||
* Write it in compact binary form with `BitWriter`
|
||||
* if double_delta == 0:
|
||||
* write 1bit: 0
|
||||
* else if -63 < double_delta < 64:
|
||||
* write 2 bit prefix: 10
|
||||
* write sign bit (1 if signed): x
|
||||
* write 7-1 bits of abs(double_delta - 1): xxxxxx
|
||||
* else if -255 < double_delta < 256:
|
||||
* write 3 bit prefix: 110
|
||||
* write sign bit (1 if signed): x
|
||||
* write 9-1 bits of abs(double_delta - 1): xxxxxxxx
|
||||
* else if -2047 < double_delta < 2048:
|
||||
* write 4 bit prefix: 1110
|
||||
* write sign bit (1 if signed): x
|
||||
* write 12-1 bits of abs(double_delta - 1): xxxxxxxxxxx
|
||||
* else if double_delta fits into 32-bit int:
|
||||
* write 5 bit prefix: 11110
|
||||
* write sign bit (1 if signed): x
|
||||
* write 32-1 bits of abs(double_delta - 1): xxxxxxxxxxx...
|
||||
* else
|
||||
* write 5 bit prefix: 11111
|
||||
* write sign bit (1 if signed): x
|
||||
* write 64-1 bits of abs(double_delta - 1): xxxxxxxxxxx...
|
||||
*
|
||||
* @example sequence of UInt8 values [1, 2, 3, 4, 5, 6, 7, 8, 9 10] is encoded as (codec header is ommited):
|
||||
*
|
||||
* .- 4-byte little-endian sequence length (10 == 0xa)
|
||||
* | .- 1 byte (sizeof(UInt8) a[0] : 0x01
|
||||
* | | .- 1 byte of delta: a[1] - a[0] = 2 - 1 = 1 : 0x01
|
||||
* | | | .- 8 zero bits since double delta for remaining 8 elements was 0 : 0x00
|
||||
* v_______________v___v___v___
|
||||
* \x0a\x00\x00\x00\x01\x01\x00
|
||||
*
|
||||
* @example sequence of Int16 values [-10, 10, -20, 20, -40, 40] is encoded as:
|
||||
*
|
||||
* .- 4-byte little endian sequence length = 6 : 0x00000006
|
||||
* | .- 2 bytes (sizeof(Int16) a[0] as UInt16 = -10 : 0xfff6
|
||||
* | | .- 2 bytes of delta: a[1] - a[0] = 10 - (-10) = 20 : 0x0014
|
||||
* | | | .- 4 encoded double deltas (see below)
|
||||
* v_______________ v______ v______ v______________________
|
||||
* \x06\x00\x00\x00\xf6\xff\x14\x00\xb8\xe2\x2e\xb1\xe4\x58
|
||||
*
|
||||
* 4 binary encoded double deltas (\xb8\xe2\x2e\xb1\xe4\x58):
|
||||
* double_delta (DD) = -20 - 2 * 10 + (-10) = -50
|
||||
* .- 2-bit prefix : 0b10
|
||||
* | .- sign-bit : 0b1
|
||||
* | |.- abs(DD - 1) = 49 : 0b110001
|
||||
* | ||
|
||||
* | || DD = 20 - 2 * (-20) + 10 = 70
|
||||
* | || .- 3-bit prefix : 0b110
|
||||
* | || | .- sign bit : 0b0
|
||||
* | || | |.- abs(DD - 1) = 69 : 0b1000101
|
||||
* | || | ||
|
||||
* | || | || DD = -40 - 2 * 20 + (-20) = -100
|
||||
* | || | || .- 3-bit prefix : 0b110
|
||||
* | || | || | .- sign-bit : 0b0
|
||||
* | || | || | |.- abs(DD - 1) = 99 : 0b1100011
|
||||
* | || | || | ||
|
||||
* | || | || | || DD = 40 - 2 * (-40) + 20 = 140
|
||||
* | || | || | || .- 3-bit prefix : 0b110
|
||||
* | || | || | || | .- sign bit : 0b0
|
||||
* | || | || | || | |.- abs(DD - 1) = 139 : 0b10001011
|
||||
* | || | || | || | ||
|
||||
* V_vv______V__vv________V____vv_______V__vv________,- padding bits
|
||||
* 10111000 11100010 00101110 10110001 11100100 01011000
|
||||
*
|
||||
* Please also see unit tests for:
|
||||
* * Examples on what output `BitWriter` produces on predefined input.
|
||||
* * Compatibility tests solidifying encoded binary output on set of predefined sequences.
|
||||
*/
|
||||
class CompressionCodecDoubleDelta : public ICompressionCodec
|
||||
{
|
||||
public:
|
||||
|
||||
@ -112,8 +112,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest,
|
||||
dest += sizeof(prev_value);
|
||||
}
|
||||
|
||||
WriteBuffer buffer(dest, dest_end - dest);
|
||||
BitWriter writer(buffer);
|
||||
BitWriter writer(dest, dest_end - dest);
|
||||
|
||||
while (source < source_end)
|
||||
{
|
||||
@ -148,7 +147,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest,
|
||||
|
||||
writer.flush();
|
||||
|
||||
return sizeof(items_count) + sizeof(prev_value) + buffer.count();
|
||||
return sizeof(items_count) + sizeof(prev_value) + writer.count() / 8;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -174,8 +173,7 @@ void decompressDataForType(const char * source, UInt32 source_size, char * dest)
|
||||
dest += sizeof(prev_value);
|
||||
}
|
||||
|
||||
ReadBufferFromMemory buffer(source, source_size - sizeof(items_count) - sizeof(prev_value));
|
||||
BitReader reader(buffer);
|
||||
BitReader reader(source, source_size - sizeof(items_count) - sizeof(prev_value));
|
||||
|
||||
binary_value_info prev_xored_info{0, 0, 0};
|
||||
|
||||
|
||||
@ -5,6 +5,89 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/** Gorilla column codec implementation.
|
||||
*
|
||||
* Based on Gorilla paper: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
|
||||
*
|
||||
* This codec is best used against monotonic floating sequences, like CPU usage percentage
|
||||
* or any other gauge.
|
||||
*
|
||||
* Given input sequence a: [a0, a1, ... an]
|
||||
*
|
||||
* First, write number of items (sizeof(int32)*8 bits): n
|
||||
* Then write first item as is (sizeof(a[0])*8 bits): a[0]
|
||||
* Loop over remaining items and calculate xor_diff:
|
||||
* xor_diff = a[i] ^ a[i - 1] (e.g. 00000011'10110100)
|
||||
* Write it in compact binary form with `BitWriter`
|
||||
* if xor_diff == 0:
|
||||
* write 1 bit: 0
|
||||
* else:
|
||||
* calculate leading zero bits (lzb)
|
||||
* and trailing zero bits (tzb) of xor_diff,
|
||||
* compare to lzb and tzb of previous xor_diff
|
||||
* (X = sizeof(a[i]) * 8, e.g. X = 16, lzb = 6, tzb = 2)
|
||||
* if lzb >= prev_lzb && tzb >= prev_tzb:
|
||||
* (e.g. prev_lzb=4, prev_tzb=1)
|
||||
* write 2 bit prefix: 0b10
|
||||
* write xor_diff >> prev_tzb (X - prev_lzb - prev_tzb bits):0b00111011010
|
||||
* (where X = sizeof(a[i]) * 8, e.g. 16)
|
||||
* else:
|
||||
* write 2 bit prefix: 0b11
|
||||
* write 5 bits of lzb: 0b00110
|
||||
* write 6 bits of (X - lzb - tzb)=(16-6-2)=8: 0b001000
|
||||
* write (X - lzb - tzb) non-zero bits of xor_diff: 0b11101101
|
||||
* prev_lzb = lzb
|
||||
* prev_tzb = tzb
|
||||
*
|
||||
* @example sequence of Float32 values [0.1, 0.1, 0.11, 0.2, 0.1] is encoded as:
|
||||
*
|
||||
* .- 4-byte little endian sequence length: 5 : 0x00000005
|
||||
* | .- 4 byte (sizeof(Float32) a[0] as UInt32 : -10 : 0xcdcccc3d
|
||||
* | | .- 4 encoded xor diffs (see below)
|
||||
* v_______________ v______________ v__________________________________________________
|
||||
* \x05\x00\x00\x00\xcd\xcc\xcc\x3d\x6a\x5a\xd8\xb6\x3c\xcd\x75\xb1\x6c\x77\x00\x00\x00
|
||||
*
|
||||
* 4 binary encoded xor diffs (\x6a\x5a\xd8\xb6\x3c\xcd\x75\xb1\x6c\x77\x00\x00\x00):
|
||||
*
|
||||
* ...........................................
|
||||
* a[i-1] = 00111101110011001100110011001101
|
||||
* a[i] = 00111101110011001100110011001101
|
||||
* xor_diff = 00000000000000000000000000000000
|
||||
* .- 1-bit prefix : 0b0
|
||||
* |
|
||||
* | ...........................................
|
||||
* | a[i-1] = 00111101110011001100110011001101
|
||||
* ! a[i] = 00111101111000010100011110101110
|
||||
* | xor_diff = 00000000001011011000101101100011
|
||||
* | lzb = 10
|
||||
* | tzb = 0
|
||||
* |.- 2-bit prefix : 0b11
|
||||
* || .- lzb (10) : 0b1010
|
||||
* || | .- data length (32-10-0): 22 : 0b010110
|
||||
* || | | .- data : 0b1011011000101101100011
|
||||
* || | | |
|
||||
* || | | | ...........................................
|
||||
* || | | | a[i-1] = 00111101111000010100011110101110
|
||||
* || | | | a[i] = 00111110010011001100110011001101
|
||||
* || | | | xor_diff = 00000011101011011000101101100011
|
||||
* || | | | .- 2-bit prefix : 0b11
|
||||
* || | | | | .- lzb = 6 : 0b00110
|
||||
* || | | | | | .- data length = (32 - 6) = 26 : 0b011010
|
||||
* || | | | | | | .- data : 0b11101011011000101101100011
|
||||
* || | | | | | | |
|
||||
* || | | | | | | | ...........................................
|
||||
* || | | | | | | | a[i-1] = 00111110010011001100110011001101
|
||||
* || | | | | | | | a[i] = 00111101110011001100110011001101
|
||||
* || | | | | | | | xor_diff = 00000011100000000000000000000000
|
||||
* || | | | | | | | .- 2-bit prefix : 0b10
|
||||
* || | | | | | | | | .- data : 0b11100000000000000000000000
|
||||
* VV_v____ v_____v________________________V_v_____v______v____________________________V_v_____________________________
|
||||
* 01101010 01011010 11011000 10110110 00111100 11001101 01110101 10110001 01101100 01110111 00000000 00000000 00000000
|
||||
*
|
||||
* Please also see unit tests for:
|
||||
* * Examples on what output `BitWriter` produces on predefined input.
|
||||
* * Compatibility tests solidifying encoded binary output on set of predefined sequences.
|
||||
*/
|
||||
class CompressionCodecGorilla : public ICompressionCodec
|
||||
{
|
||||
public:
|
||||
|
||||
@ -5,6 +5,8 @@
|
||||
#include <Compression/CompressionFactory.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Common/hex.h>
|
||||
#include <sstream>
|
||||
|
||||
@ -31,16 +33,15 @@ UInt8 CompressionCodecMultiple::getMethodByte() const
|
||||
|
||||
String CompressionCodecMultiple::getCodecDesc() const
|
||||
{
|
||||
std::ostringstream ss;
|
||||
for (size_t idx = 0; idx < codecs.size(); idx++)
|
||||
WriteBufferFromOwnString out;
|
||||
for (size_t idx = 0; idx < codecs.size(); ++idx)
|
||||
{
|
||||
if (idx != 0)
|
||||
ss << ',' << ' ';
|
||||
out << ", ";
|
||||
|
||||
const auto codec = codecs[idx];
|
||||
ss << codec->getCodecDesc();
|
||||
out << codecs[idx]->getCodecDesc();
|
||||
}
|
||||
return ss.str();
|
||||
return out.str();
|
||||
}
|
||||
|
||||
UInt32 CompressionCodecMultiple::getMaxCompressedDataSize(UInt32 uncompressed_size) const
|
||||
@ -55,7 +56,6 @@ UInt32 CompressionCodecMultiple::getMaxCompressedDataSize(UInt32 uncompressed_si
|
||||
|
||||
UInt32 CompressionCodecMultiple::doCompressData(const char * source, UInt32 source_size, char * dest) const
|
||||
{
|
||||
|
||||
PODArray<char> compressed_buf;
|
||||
PODArray<char> uncompressed_buf(source, source + source_size);
|
||||
|
||||
|
||||
@ -8,6 +8,7 @@
|
||||
#include <Common/typeid_cast.h>
|
||||
#include <Compression/CompressionFactory.h>
|
||||
|
||||
|
||||
namespace ProfileEvents
|
||||
{
|
||||
extern const Event ReadCompressedBytes;
|
||||
@ -25,6 +26,7 @@ namespace ErrorCodes
|
||||
extern const int UNKNOWN_COMPRESSION_METHOD;
|
||||
extern const int CANNOT_DECOMPRESS;
|
||||
extern const int SEEK_POSITION_OUT_OF_BOUND;
|
||||
extern const int CORRUPTED_DATA;
|
||||
}
|
||||
|
||||
|
||||
@ -42,11 +44,16 @@ UInt32 ICompressionCodec::compress(const char * source, UInt32 source_size, char
|
||||
|
||||
UInt32 ICompressionCodec::decompress(const char * source, UInt32 source_size, char * dest) const
|
||||
{
|
||||
UInt8 header_size = getHeaderSize();
|
||||
|
||||
if (source_size < header_size)
|
||||
throw Exception("Can't decompress data: the compressed data size (" + toString(source_size)
|
||||
+ ", this should include header size) is less than the header size (" + toString(header_size) + ")", ErrorCodes::CORRUPTED_DATA);
|
||||
|
||||
UInt8 method = source[0];
|
||||
if (method != getMethodByte())
|
||||
throw Exception("Can't decompress data with codec byte " + toString(method) + " from codec with byte " + toString(method), ErrorCodes::CANNOT_DECOMPRESS);
|
||||
|
||||
UInt8 header_size = getHeaderSize();
|
||||
UInt32 decompressed_size = readDecompressedBlockSize(source);
|
||||
doDecompressData(&source[header_size], source_size - header_size, dest, decompressed_size);
|
||||
|
||||
|
||||
@ -33,10 +33,10 @@ public:
|
||||
virtual String getCodecDesc() const = 0;
|
||||
|
||||
/// Compressed bytes from uncompressed source to dest. Dest should preallocate memory
|
||||
virtual UInt32 compress(const char * source, UInt32 source_size, char * dest) const;
|
||||
UInt32 compress(const char * source, UInt32 source_size, char * dest) const;
|
||||
|
||||
/// Decompress bytes from compressed source to dest. Dest should preallocate memory
|
||||
virtual UInt32 decompress(const char * source, UInt32 source_size, char * dest) const;
|
||||
UInt32 decompress(const char * source, UInt32 source_size, char * dest) const;
|
||||
|
||||
/// Number of bytes, that will be used to compress uncompressed_size bytes with current codec
|
||||
virtual UInt32 getCompressedReserveSize(UInt32 uncompressed_size) const { return getHeaderSize() + getMaxCompressedDataSize(uncompressed_size); }
|
||||
@ -45,7 +45,7 @@ public:
|
||||
virtual UInt32 getAdditionalSizeAtTheEndOfBuffer() const { return 0; }
|
||||
|
||||
/// Size of header in compressed data on disk
|
||||
static UInt8 getHeaderSize() { return COMPRESSED_BLOCK_HEADER_SIZE; }
|
||||
static constexpr UInt8 getHeaderSize() { return COMPRESSED_BLOCK_HEADER_SIZE; }
|
||||
|
||||
/// Read size of compressed block from compressed source
|
||||
static UInt32 readCompressedBlockSize(const char * source);
|
||||
|
||||
@ -32,7 +32,7 @@ int main(int argc, char ** argv)
|
||||
|
||||
{
|
||||
Stopwatch watch;
|
||||
CachedCompressedReadBuffer in(path, &cache, 0, 0);
|
||||
CachedCompressedReadBuffer in(path, &cache, 0, 0, 0);
|
||||
WriteBufferFromFile out("/dev/null");
|
||||
copyData(in, out);
|
||||
|
||||
@ -44,7 +44,7 @@ int main(int argc, char ** argv)
|
||||
|
||||
{
|
||||
Stopwatch watch;
|
||||
CachedCompressedReadBuffer in(path, &cache, 0, 0);
|
||||
CachedCompressedReadBuffer in(path, &cache, 0, 0, 0);
|
||||
WriteBufferFromFile out("/dev/null");
|
||||
copyData(in, out);
|
||||
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
#include <Compression/CompressionFactory.h>
|
||||
|
||||
#include <Common/PODArray.h>
|
||||
#include <Common/Stopwatch.h>
|
||||
#include <Core/Types.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/IDataType.h>
|
||||
@ -62,6 +63,32 @@ std::vector<T> operator+(std::vector<T> && left, std::vector<T> && right)
|
||||
namespace
|
||||
{
|
||||
|
||||
template <typename T>
|
||||
struct AsHexStringHelper
|
||||
{
|
||||
const T & container;
|
||||
};
|
||||
|
||||
template <typename T>
|
||||
std::ostream & operator << (std::ostream & ostr, const AsHexStringHelper<T> & helper)
|
||||
{
|
||||
ostr << std::hex;
|
||||
for (const auto & e : helper.container)
|
||||
{
|
||||
ostr << "\\x" << std::setw(2) << std::setfill('0') << (static_cast<unsigned int>(e) & 0xFF);
|
||||
}
|
||||
|
||||
return ostr;
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
AsHexStringHelper<T> AsHexString(const T & container)
|
||||
{
|
||||
static_assert (sizeof(container[0]) == 1 && std::is_pod<std::decay_t<decltype(container[0])>>::value, "Only works on containers of byte-size PODs.");
|
||||
|
||||
return AsHexStringHelper<T>{container};
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
std::string bin(const T & value, size_t bits = sizeof(T)*8)
|
||||
{
|
||||
@ -113,10 +140,71 @@ DataTypePtr makeDataType()
|
||||
|
||||
#undef MAKE_DATA_TYPE
|
||||
|
||||
assert(false && "unsupported size");
|
||||
assert(false && "unknown datatype");
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
template <typename T, typename Container>
|
||||
class BinaryDataAsSequenceOfValuesIterator
|
||||
{
|
||||
const Container & container;
|
||||
const void * data;
|
||||
const void * data_end;
|
||||
|
||||
T current_value;
|
||||
|
||||
public:
|
||||
using Self = BinaryDataAsSequenceOfValuesIterator<T, Container>;
|
||||
|
||||
explicit BinaryDataAsSequenceOfValuesIterator(const Container & container_)
|
||||
: container(container_),
|
||||
data(&container[0]),
|
||||
data_end(reinterpret_cast<const char *>(data) + container.size()),
|
||||
current_value(T{})
|
||||
{
|
||||
static_assert(sizeof(container[0]) == 1 && std::is_pod<std::decay_t<decltype(container[0])>>::value, "Only works on containers of byte-size PODs.");
|
||||
read();
|
||||
}
|
||||
|
||||
const T & operator*() const
|
||||
{
|
||||
return current_value;
|
||||
}
|
||||
|
||||
size_t ItemsLeft() const
|
||||
{
|
||||
return reinterpret_cast<const char *>(data_end) - reinterpret_cast<const char *>(data);
|
||||
}
|
||||
|
||||
Self & operator++()
|
||||
{
|
||||
read();
|
||||
return *this;
|
||||
}
|
||||
|
||||
operator bool() const
|
||||
{
|
||||
return ItemsLeft() > 0;
|
||||
}
|
||||
|
||||
private:
|
||||
void read()
|
||||
{
|
||||
if (!*this)
|
||||
{
|
||||
throw std::runtime_error("No more data to read");
|
||||
}
|
||||
|
||||
current_value = unalignedLoad<T>(data);
|
||||
data = reinterpret_cast<const char *>(data) + sizeof(T);
|
||||
}
|
||||
};
|
||||
|
||||
template <typename T, typename Container>
|
||||
BinaryDataAsSequenceOfValuesIterator<T, Container> AsSequenceOf(const Container & container)
|
||||
{
|
||||
return BinaryDataAsSequenceOfValuesIterator<T, Container>(container);
|
||||
}
|
||||
|
||||
template <typename T, typename ContainerLeft, typename ContainerRight>
|
||||
::testing::AssertionResult EqualByteContainersAs(const ContainerLeft & left, const ContainerRight & right)
|
||||
@ -126,9 +214,6 @@ template <typename T, typename ContainerLeft, typename ContainerRight>
|
||||
|
||||
::testing::AssertionResult result = ::testing::AssertionSuccess();
|
||||
|
||||
ReadBufferFromMemory left_read_buffer(left.data(), left.size());
|
||||
ReadBufferFromMemory right_read_buffer(right.data(), right.size());
|
||||
|
||||
const auto l_size = left.size() / sizeof(T);
|
||||
const auto r_size = right.size() / sizeof(T);
|
||||
const auto size = std::min(l_size, r_size);
|
||||
@ -137,16 +222,25 @@ template <typename T, typename ContainerLeft, typename ContainerRight>
|
||||
{
|
||||
result = ::testing::AssertionFailure() << "size mismatch" << " expected: " << l_size << " got:" << r_size;
|
||||
}
|
||||
if (l_size == 0 || r_size == 0)
|
||||
{
|
||||
return result;
|
||||
}
|
||||
|
||||
auto l = AsSequenceOf<T>(left);
|
||||
auto r = AsSequenceOf<T>(right);
|
||||
|
||||
const auto MAX_MISMATCHING_ITEMS = 5;
|
||||
int mismatching_items = 0;
|
||||
for (int i = 0; i < size; ++i)
|
||||
{
|
||||
T left_value{};
|
||||
left_read_buffer.readStrict(reinterpret_cast<char*>(&left_value), sizeof(left_value));
|
||||
size_t i = 0;
|
||||
|
||||
T right_value{};
|
||||
right_read_buffer.readStrict(reinterpret_cast<char*>(&right_value), sizeof(right_value));
|
||||
while (l && r)
|
||||
{
|
||||
const auto left_value = *l;
|
||||
const auto right_value = *r;
|
||||
++l;
|
||||
++r;
|
||||
++i;
|
||||
|
||||
if (left_value != right_value)
|
||||
{
|
||||
@ -157,25 +251,47 @@ template <typename T, typename ContainerLeft, typename ContainerRight>
|
||||
|
||||
if (++mismatching_items <= MAX_MISMATCHING_ITEMS)
|
||||
{
|
||||
result << "mismatching " << sizeof(T) << "-byte item #" << i
|
||||
result << "\nmismatching " << sizeof(T) << "-byte item #" << i
|
||||
<< "\nexpected: " << bin(left_value) << " (0x" << std::hex << left_value << ")"
|
||||
<< "\ngot : " << bin(right_value) << " (0x" << std::hex << right_value << ")"
|
||||
<< std::endl;
|
||||
<< "\ngot : " << bin(right_value) << " (0x" << std::hex << right_value << ")";
|
||||
if (mismatching_items == MAX_MISMATCHING_ITEMS)
|
||||
{
|
||||
result << "..." << std::endl;
|
||||
result << "\n..." << std::endl;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (mismatching_items > 0)
|
||||
{
|
||||
result << "\ntotal mismatching items:" << mismatching_items << " of " << size;
|
||||
result << "total mismatching items:" << mismatching_items << " of " << size;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
template <typename ContainerLeft, typename ContainerRight>
|
||||
::testing::AssertionResult EqualByteContainers(UInt8 element_size, const ContainerLeft & left, const ContainerRight & right)
|
||||
{
|
||||
switch (element_size)
|
||||
{
|
||||
case 1:
|
||||
return EqualByteContainersAs<UInt8>(left, right);
|
||||
break;
|
||||
case 2:
|
||||
return EqualByteContainersAs<UInt16>(left, right);
|
||||
break;
|
||||
case 4:
|
||||
return EqualByteContainersAs<UInt32>(left, right);
|
||||
break;
|
||||
case 8:
|
||||
return EqualByteContainersAs<UInt64>(left, right);
|
||||
break;
|
||||
default:
|
||||
assert(false && "Invalid element_size");
|
||||
return ::testing::AssertionFailure() << "Invalid element_size: " << element_size;
|
||||
}
|
||||
}
|
||||
|
||||
struct Codec
|
||||
{
|
||||
std::string codec_statement;
|
||||
@ -214,20 +330,23 @@ struct CodecTestSequence
|
||||
CodecTestSequence & operator=(const CodecTestSequence &) = default;
|
||||
CodecTestSequence(CodecTestSequence &&) = default;
|
||||
CodecTestSequence & operator=(CodecTestSequence &&) = default;
|
||||
|
||||
CodecTestSequence & append(const CodecTestSequence & other)
|
||||
{
|
||||
assert(data_type->equals(*other.data_type));
|
||||
|
||||
serialized_data.insert(serialized_data.end(), other.serialized_data.begin(), other.serialized_data.end());
|
||||
if (!name.empty())
|
||||
name += " + ";
|
||||
name += other.name;
|
||||
|
||||
return *this;
|
||||
}
|
||||
};
|
||||
|
||||
CodecTestSequence operator+(CodecTestSequence && left, CodecTestSequence && right)
|
||||
CodecTestSequence operator+(CodecTestSequence && left, const CodecTestSequence & right)
|
||||
{
|
||||
assert(left.data_type->equals(*right.data_type));
|
||||
|
||||
std::vector<char> data(std::move(left.serialized_data));
|
||||
data.insert(data.end(), right.serialized_data.begin(), right.serialized_data.end());
|
||||
|
||||
return CodecTestSequence{
|
||||
left.name + " + " + right.name,
|
||||
std::move(data),
|
||||
std::move(left.data_type)
|
||||
};
|
||||
return left.append(right);
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -288,17 +407,22 @@ CodecTestSequence makeSeq(Args && ... args)
|
||||
};
|
||||
}
|
||||
|
||||
template <typename T, typename Generator>
|
||||
CodecTestSequence generateSeq(Generator gen, const char* gen_name, size_t Begin = 0, size_t End = 10000)
|
||||
template <typename T, typename Generator, typename B = int, typename E = int>
|
||||
CodecTestSequence generateSeq(Generator gen, const char* gen_name, B Begin = 0, E End = 10000)
|
||||
{
|
||||
assert (End >= Begin);
|
||||
|
||||
const auto direction = std::signbit(End - Begin) ? -1 : 1;
|
||||
std::vector<char> data(sizeof(T) * (End - Begin));
|
||||
char * write_pos = data.data();
|
||||
|
||||
for (size_t i = Begin; i < End; ++i)
|
||||
for (auto i = Begin; i < End; i += direction)
|
||||
{
|
||||
const T v = gen(static_cast<T>(i));
|
||||
|
||||
// if constexpr (debug_log_items)
|
||||
// {
|
||||
// std::cerr << "#" << i << " " << type_name<T>() << "(" << sizeof(T) << " bytes) : " << v << std::endl;
|
||||
// }
|
||||
|
||||
unalignedStore<T>(write_pos, v);
|
||||
write_pos += sizeof(v);
|
||||
}
|
||||
@ -310,21 +434,47 @@ CodecTestSequence generateSeq(Generator gen, const char* gen_name, size_t Begin
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
class CodecTest : public ::testing::TestWithParam<std::tuple<Codec, CodecTestSequence>>
|
||||
struct NoOpTimer
|
||||
{
|
||||
public:
|
||||
enum MakeCodecParam
|
||||
{
|
||||
CODEC_WITH_DATA_TYPE,
|
||||
CODEC_WITHOUT_DATA_TYPE,
|
||||
};
|
||||
void start() {}
|
||||
void report(const char*) {}
|
||||
};
|
||||
|
||||
CompressionCodecPtr makeCodec(MakeCodecParam with_data_type) const
|
||||
struct StopwatchTimer
|
||||
{
|
||||
explicit StopwatchTimer(clockid_t clock_type, size_t estimated_marks = 32)
|
||||
: stopwatch(clock_type)
|
||||
{
|
||||
const auto & codec_string = std::get<0>(GetParam()).codec_statement;
|
||||
const auto & data_type = with_data_type == CODEC_WITH_DATA_TYPE ? std::get<1>(GetParam()).data_type : nullptr;
|
||||
results.reserve(estimated_marks);
|
||||
}
|
||||
|
||||
void start()
|
||||
{
|
||||
stopwatch.restart();
|
||||
}
|
||||
|
||||
void report(const char * mark)
|
||||
{
|
||||
results.emplace_back(mark, stopwatch.elapsed());
|
||||
}
|
||||
|
||||
void stop()
|
||||
{
|
||||
stopwatch.stop();
|
||||
}
|
||||
|
||||
const std::vector<std::tuple<const char*, UInt64>> & getResults() const
|
||||
{
|
||||
return results;
|
||||
}
|
||||
|
||||
private:
|
||||
Stopwatch stopwatch;
|
||||
std::vector<std::tuple<const char*, UInt64>> results;
|
||||
};
|
||||
|
||||
CompressionCodecPtr makeCodec(const std::string & codec_string, const DataTypePtr data_type)
|
||||
{
|
||||
const std::string codec_statement = "(" + codec_string + ")";
|
||||
Tokens tokens(codec_statement.begin().base(), codec_statement.end().base());
|
||||
IParser::Pos token_iterator(tokens);
|
||||
@ -336,51 +486,66 @@ public:
|
||||
parser.parse(token_iterator, codec_ast, expected);
|
||||
|
||||
return CompressionCodecFactory::instance().get(codec_ast, data_type);
|
||||
}
|
||||
}
|
||||
|
||||
void testTranscoding(ICompressionCodec & codec)
|
||||
{
|
||||
const auto & test_sequence = std::get<1>(GetParam());
|
||||
template <typename Timer>
|
||||
void testTranscoding(Timer & timer, ICompressionCodec & codec, const CodecTestSequence & test_sequence, std::optional<double> expected_compression_ratio = std::optional<double>{})
|
||||
{
|
||||
const auto & source_data = test_sequence.serialized_data;
|
||||
|
||||
const UInt32 encoded_max_size = codec.getCompressedReserveSize(source_data.size());
|
||||
PODArray<char> encoded(encoded_max_size);
|
||||
|
||||
timer.start();
|
||||
|
||||
const UInt32 encoded_size = codec.compress(source_data.data(), source_data.size(), encoded.data());
|
||||
timer.report("encoding");
|
||||
|
||||
encoded.resize(encoded_size);
|
||||
|
||||
PODArray<char> decoded(source_data.size());
|
||||
|
||||
timer.start();
|
||||
const UInt32 decoded_size = codec.decompress(encoded.data(), encoded.size(), decoded.data());
|
||||
timer.report("decoding");
|
||||
|
||||
decoded.resize(decoded_size);
|
||||
|
||||
switch (test_sequence.data_type->getSizeOfValueInMemory())
|
||||
{
|
||||
case 1:
|
||||
ASSERT_TRUE(EqualByteContainersAs<UInt8>(source_data, decoded));
|
||||
break;
|
||||
case 2:
|
||||
ASSERT_TRUE(EqualByteContainersAs<UInt16>(source_data, decoded));
|
||||
break;
|
||||
case 4:
|
||||
ASSERT_TRUE(EqualByteContainersAs<UInt32>(source_data, decoded));
|
||||
break;
|
||||
case 8:
|
||||
ASSERT_TRUE(EqualByteContainersAs<UInt64>(source_data, decoded));
|
||||
break;
|
||||
default:
|
||||
FAIL() << "Invalid test sequence data type: " << test_sequence.data_type->getName();
|
||||
}
|
||||
ASSERT_TRUE(EqualByteContainers(test_sequence.data_type->getSizeOfValueInMemory(), source_data, decoded));
|
||||
|
||||
const auto header_size = codec.getHeaderSize();
|
||||
const auto compression_ratio = (encoded_size - header_size) / (source_data.size() * 1.0);
|
||||
|
||||
const auto & codec_spec = std::get<0>(GetParam());
|
||||
if (codec_spec.expected_compression_ratio)
|
||||
if (expected_compression_ratio)
|
||||
{
|
||||
ASSERT_LE(compression_ratio, *codec_spec.expected_compression_ratio)
|
||||
ASSERT_LE(compression_ratio, *expected_compression_ratio)
|
||||
<< "\n\tdecoded size: " << source_data.size()
|
||||
<< "\n\tencoded size: " << encoded_size
|
||||
<< "(no header: " << encoded_size - header_size << ")";
|
||||
}
|
||||
}
|
||||
|
||||
class CodecTest : public ::testing::TestWithParam<std::tuple<Codec, CodecTestSequence>>
|
||||
{
|
||||
public:
|
||||
enum MakeCodecParam
|
||||
{
|
||||
CODEC_WITH_DATA_TYPE,
|
||||
CODEC_WITHOUT_DATA_TYPE,
|
||||
};
|
||||
|
||||
CompressionCodecPtr makeCodec(MakeCodecParam with_data_type)
|
||||
{
|
||||
const auto & codec_string = std::get<0>(GetParam()).codec_statement;
|
||||
const auto & data_type = with_data_type == CODEC_WITH_DATA_TYPE ? std::get<1>(GetParam()).data_type : nullptr;
|
||||
|
||||
return ::makeCodec(codec_string, data_type);
|
||||
}
|
||||
|
||||
void testTranscoding(ICompressionCodec & codec)
|
||||
{
|
||||
NoOpTimer timer;
|
||||
::testTranscoding(timer, codec, std::get<1>(GetParam()), std::get<0>(GetParam()).expected_compression_ratio);
|
||||
}
|
||||
};
|
||||
|
||||
@ -396,10 +561,121 @@ TEST_P(CodecTest, TranscodingWithoutDataType)
|
||||
testTranscoding(*codec);
|
||||
}
|
||||
|
||||
// Param is tuple-of-tuple to simplify instantiating with values, since typically group of cases test only one codec.
|
||||
class CodecTest_Compatibility : public ::testing::TestWithParam<std::tuple<Codec, std::tuple<CodecTestSequence, std::string>>>
|
||||
{};
|
||||
|
||||
// Check that iput sequence when encoded matches the encoded string binary.
|
||||
TEST_P(CodecTest_Compatibility, Encoding)
|
||||
{
|
||||
const auto & codec_spec = std::get<0>(GetParam());
|
||||
const auto & [data_sequence, expected] = std::get<1>(GetParam());
|
||||
const auto codec = makeCodec(codec_spec.codec_statement, data_sequence.data_type);
|
||||
|
||||
const auto & source_data = data_sequence.serialized_data;
|
||||
|
||||
// Just encode the data with codec
|
||||
const UInt32 encoded_max_size = codec->getCompressedReserveSize(source_data.size());
|
||||
PODArray<char> encoded(encoded_max_size);
|
||||
|
||||
const UInt32 encoded_size = codec->compress(source_data.data(), source_data.size(), encoded.data());
|
||||
encoded.resize(encoded_size);
|
||||
SCOPED_TRACE(::testing::Message("encoded: ") << AsHexString(encoded));
|
||||
|
||||
ASSERT_TRUE(EqualByteContainersAs<UInt8>(expected, encoded));
|
||||
}
|
||||
|
||||
// Check that binary string is exactly decoded into input sequence.
|
||||
TEST_P(CodecTest_Compatibility, Decoding)
|
||||
{
|
||||
const auto & codec_spec = std::get<0>(GetParam());
|
||||
const auto & [expected, encoded_data] = std::get<1>(GetParam());
|
||||
const auto codec = makeCodec(codec_spec.codec_statement, expected.data_type);
|
||||
|
||||
PODArray<char> decoded(expected.serialized_data.size());
|
||||
const UInt32 decoded_size = codec->decompress(encoded_data.c_str(), encoded_data.size(), decoded.data());
|
||||
decoded.resize(decoded_size);
|
||||
|
||||
ASSERT_TRUE(EqualByteContainers(expected.data_type->getSizeOfValueInMemory(), expected.serialized_data, decoded));
|
||||
}
|
||||
|
||||
class CodecTest_Performance : public ::testing::TestWithParam<std::tuple<Codec, CodecTestSequence>>
|
||||
{};
|
||||
|
||||
TEST_P(CodecTest_Performance, TranscodingWithDataType)
|
||||
{
|
||||
const auto & [codec_spec, test_seq] = GetParam();
|
||||
const auto codec = ::makeCodec(codec_spec.codec_statement, test_seq.data_type);
|
||||
|
||||
const auto runs = 10;
|
||||
std::map<std::string, std::vector<UInt64>> results;
|
||||
|
||||
for (size_t i = 0; i < runs; ++i)
|
||||
{
|
||||
StopwatchTimer timer{CLOCK_THREAD_CPUTIME_ID};
|
||||
::testTranscoding(timer, *codec, test_seq);
|
||||
timer.stop();
|
||||
|
||||
for (const auto & [label, value] : timer.getResults())
|
||||
{
|
||||
results[label].push_back(value);
|
||||
}
|
||||
}
|
||||
|
||||
auto computeMeanAndStdDev = [](const auto & values)
|
||||
{
|
||||
double mean{};
|
||||
|
||||
if (values.size() < 2)
|
||||
return std::make_tuple(mean, double{});
|
||||
|
||||
using ValueType = typename std::decay_t<decltype(values)>::value_type;
|
||||
std::vector<ValueType> tmp_v(std::begin(values), std::end(values));
|
||||
std::sort(tmp_v.begin(), tmp_v.end());
|
||||
|
||||
// remove min and max
|
||||
tmp_v.erase(tmp_v.begin());
|
||||
tmp_v.erase(tmp_v.end() - 1);
|
||||
|
||||
for (const auto & v : tmp_v)
|
||||
{
|
||||
mean += v;
|
||||
}
|
||||
|
||||
mean = mean / tmp_v.size();
|
||||
double std_dev = 0.0;
|
||||
for (const auto & v : tmp_v)
|
||||
{
|
||||
const auto d = (v - mean);
|
||||
std_dev += (d * d);
|
||||
}
|
||||
std_dev = std::sqrt(std_dev / tmp_v.size());
|
||||
|
||||
return std::make_tuple(mean, std_dev);
|
||||
};
|
||||
|
||||
std::cerr << codec_spec.codec_statement
|
||||
<< " " << test_seq.data_type->getName()
|
||||
<< " (" << test_seq.serialized_data.size() << " bytes, "
|
||||
<< std::hex << CityHash_v1_0_2::CityHash64(test_seq.serialized_data.data(), test_seq.serialized_data.size()) << std::dec
|
||||
<< ", average of " << runs << " runs, μs)";
|
||||
|
||||
for (const auto & k : {"encoding", "decoding"})
|
||||
{
|
||||
const auto & values = results[k];
|
||||
const auto & [mean, std_dev] = computeMeanAndStdDev(values);
|
||||
// Ensure that Coefficient of variation is reasonably low, otherwise these numbers are meaningless
|
||||
EXPECT_GT(0.05, std_dev / mean);
|
||||
std::cerr << "\t" << std::fixed << std::setprecision(1) << mean / 1000.0;
|
||||
}
|
||||
|
||||
std::cerr << std::endl;
|
||||
}
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////////////////
|
||||
// Here we use generators to produce test payload for codecs.
|
||||
// Generator is a callable that can produce infinite number of values,
|
||||
// output value MUST be of the same type input value.
|
||||
// output value MUST be of the same type as input value.
|
||||
///////////////////////////////////////////////////////////////////////////////////////////////////
|
||||
|
||||
auto SameValueGenerator = [](auto value)
|
||||
@ -543,6 +819,23 @@ std::vector<CodecTestSequence> generatePyramidOfSequences(const size_t sequences
|
||||
return sequences;
|
||||
};
|
||||
|
||||
// Just as if all sequences from generatePyramidOfSequences were appended to one-by-one to the first one.
|
||||
template <typename T, typename Generator>
|
||||
CodecTestSequence generatePyramidSequence(const size_t sequences_count, Generator && generator, const char* generator_name)
|
||||
{
|
||||
CodecTestSequence sequence;
|
||||
sequence.data_type = makeDataType<T>();
|
||||
sequence.serialized_data.reserve(sequences_count * sequences_count * sizeof(T));
|
||||
|
||||
for (size_t i = 1; i < sequences_count; ++i)
|
||||
{
|
||||
std::string name = generator_name + std::string(" from 0 to ") + std::to_string(i);
|
||||
sequence.append(generateSeq<T>(std::forward<decltype(generator)>(generator), name.c_str(), 0, i));
|
||||
}
|
||||
|
||||
return sequence;
|
||||
};
|
||||
|
||||
|
||||
// helper macro to produce human-friendly sequence name from generator
|
||||
#define G(generator) generator, #generator
|
||||
@ -609,7 +902,7 @@ INSTANTIATE_TEST_CASE_P(SameValueInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(SameValueGenerator(1000))),
|
||||
generateSeq<Int8>(G(SameValueGenerator(1000))),
|
||||
generateSeq<Int16 >(G(SameValueGenerator(1000))),
|
||||
generateSeq<Int32 >(G(SameValueGenerator(1000))),
|
||||
generateSeq<Int64 >(G(SameValueGenerator(1000))),
|
||||
@ -626,7 +919,7 @@ INSTANTIATE_TEST_CASE_P(SameNegativeValueInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(SameValueGenerator(-1000))),
|
||||
generateSeq<Int8>(G(SameValueGenerator(-1000))),
|
||||
generateSeq<Int16 >(G(SameValueGenerator(-1000))),
|
||||
generateSeq<Int32 >(G(SameValueGenerator(-1000))),
|
||||
generateSeq<Int64 >(G(SameValueGenerator(-1000))),
|
||||
@ -671,7 +964,7 @@ INSTANTIATE_TEST_CASE_P(SequentialInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(SequentialGenerator(1))),
|
||||
generateSeq<Int8>(G(SequentialGenerator(1))),
|
||||
generateSeq<Int16 >(G(SequentialGenerator(1))),
|
||||
generateSeq<Int32 >(G(SequentialGenerator(1))),
|
||||
generateSeq<Int64 >(G(SequentialGenerator(1))),
|
||||
@ -690,7 +983,7 @@ INSTANTIATE_TEST_CASE_P(SequentialReverseInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(SequentialGenerator(-1))),
|
||||
generateSeq<Int8>(G(SequentialGenerator(-1))),
|
||||
generateSeq<Int16 >(G(SequentialGenerator(-1))),
|
||||
generateSeq<Int32 >(G(SequentialGenerator(-1))),
|
||||
generateSeq<Int64 >(G(SequentialGenerator(-1))),
|
||||
@ -735,10 +1028,10 @@ INSTANTIATE_TEST_CASE_P(MonotonicInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int16 >(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int32 >(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int64 >(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int8>(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int16>(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int32>(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<Int64>(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<UInt8 >(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<UInt16>(G(MonotonicGenerator(1, 5))),
|
||||
generateSeq<UInt32>(G(MonotonicGenerator(1, 5))),
|
||||
@ -752,11 +1045,11 @@ INSTANTIATE_TEST_CASE_P(MonotonicReverseInt,
|
||||
::testing::Combine(
|
||||
DefaultCodecsToTest,
|
||||
::testing::Values(
|
||||
generateSeq<Int8 >(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int16 >(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int32 >(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int64 >(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<UInt8 >(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int8>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int16>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int32>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<Int64>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<UInt8>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<UInt16>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<UInt32>(G(MonotonicGenerator(-1, 5))),
|
||||
generateSeq<UInt64>(G(MonotonicGenerator(-1, 5)))
|
||||
@ -862,4 +1155,191 @@ INSTANTIATE_TEST_CASE_P(OverflowFloat,
|
||||
),
|
||||
);
|
||||
|
||||
template <typename ValueType>
|
||||
auto DDCompatibilityTestSequence()
|
||||
{
|
||||
// Generates sequences with double delta in given range.
|
||||
auto ddGenerator = [prev_delta = static_cast<Int64>(0), prev = static_cast<Int64>(0)](auto dd) mutable
|
||||
{
|
||||
const auto curr = dd + prev + prev_delta;
|
||||
prev = curr;
|
||||
prev_delta = dd + prev_delta;
|
||||
return curr;
|
||||
};
|
||||
|
||||
auto ret = generateSeq<ValueType>(G(SameValueGenerator(42)), 0, 3);
|
||||
|
||||
// These values are from DoubleDelta paper (and implementation) and represent points at which DD encoded length is changed.
|
||||
// DD value less that this point is encoded in shorter binary form (bigger - longer binary).
|
||||
const Int64 dd_corner_points[] = {-63, 64, -255, 256, -2047, 2048, std::numeric_limits<Int32>::min(), std::numeric_limits<Int32>::max()};
|
||||
for (const auto & p : dd_corner_points)
|
||||
{
|
||||
if (std::abs(p) > std::numeric_limits<ValueType>::max())
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
// - 4 is to allow DD value to settle before transitioning through important point,
|
||||
// since DD depends on 2 previous values of data, + 2 is arbitrary.
|
||||
ret.append(generateSeq<ValueType>(G(ddGenerator), p - 4, p + 2));
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
#define BIN_STR(x) std::string{x, sizeof(x) - 1}
|
||||
|
||||
INSTANTIATE_TEST_CASE_P(DoubleDelta,
|
||||
CodecTest_Compatibility,
|
||||
::testing::Combine(
|
||||
::testing::Values(Codec("DoubleDelta")),
|
||||
::testing::ValuesIn(std::initializer_list<std::tuple<CodecTestSequence, std::string>>{
|
||||
{
|
||||
DDCompatibilityTestSequence<Int8>(),
|
||||
BIN_STR("\x94\x21\x00\x00\x00\x0f\x00\x00\x00\x01\x00\x0f\x00\x00\x00\x2a\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xb1\xaa\xf4\xf6\x7d\x87\xf8\x80")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<UInt8>(),
|
||||
BIN_STR("\x94\x27\x00\x00\x00\x15\x00\x00\x00\x01\x00\x15\x00\x00\x00\x2a\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xb1\xaa\xf4\xf6\x7d\x87\xf8\x81\x8e\xd0\xca\x02\x01\x01")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<Int16>(),
|
||||
BIN_STR("\x94\x70\x00\x00\x00\x4e\x00\x00\x00\x02\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x40\x00\x0f\xf2\x78\x00\x01\x7f\x83\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<UInt16>(),
|
||||
BIN_STR("\x94\x70\x00\x00\x00\x4e\x00\x00\x00\x02\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x40\x00\x0f\xf2\x78\x00\x01\x7f\x83\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<Int32>(),
|
||||
BIN_STR("\x94\x74\x00\x00\x00\x9c\x00\x00\x00\x04\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<UInt32>(),
|
||||
BIN_STR("\x94\xb5\x00\x00\x00\xcc\x00\x00\x00\x04\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xf3\xff\xf9\x41\xaf\xbf\xff\xd6\x0c\xfc\xff\xff\xff\xfb\xf0\x00\x00\x00\x07\xff\xff\xff\xef\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfa\x69\x74\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf8\x00\x00\x00\x07\xff\xff\xff\xf0")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<Int64>(),
|
||||
BIN_STR("\x94\xd4\x00\x00\x00\x98\x01\x00\x00\x08\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xfc\x00\x00\x00\x04\x00\x06\xbe\x4f\xbf\xff\xd6\x0c\xff\x00\x00\x00\x01\x00\x00\x00\x03\xf8\x00\x00\x00\x08\x00\x00\x00\x0f\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfb\xe0\x00\x00\x01\xc0\x00\x00\x06\x9f\x80\x00\x00\x0a\x00\x00\x00\x34\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf0\x00\x00\x00\x07\xff\xff\xff\xf0")
|
||||
},
|
||||
{
|
||||
DDCompatibilityTestSequence<UInt64>(),
|
||||
BIN_STR("\x94\xd4\x00\x00\x00\x98\x01\x00\x00\x08\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xfc\x00\x00\x00\x04\x00\x06\xbe\x4f\xbf\xff\xd6\x0c\xff\x00\x00\x00\x01\x00\x00\x00\x03\xf8\x00\x00\x00\x08\x00\x00\x00\x0f\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfb\xe0\x00\x00\x01\xc0\x00\x00\x06\x9f\x80\x00\x00\x0a\x00\x00\x00\x34\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf0\x00\x00\x00\x07\xff\xff\xff\xf0")
|
||||
},
|
||||
})
|
||||
),
|
||||
);
|
||||
|
||||
template <typename ValueType>
|
||||
auto DDperformanceTestSequence()
|
||||
{
|
||||
const auto times = 100'000;
|
||||
return DDCompatibilityTestSequence<ValueType>() * times // average case
|
||||
+ generateSeq<ValueType>(G(MinMaxGenerator()), 0, times) // worst
|
||||
+ generateSeq<ValueType>(G(SameValueGenerator(42)), 0, times); // best
|
||||
}
|
||||
|
||||
// prime numbers in ascending order with some random repitions hit all the cases of Gorilla.
|
||||
auto PrimesWithMultiplierGenerator = [](int multiplier = 1)
|
||||
{
|
||||
return [multiplier](auto i)
|
||||
{
|
||||
static const int vals[] = {
|
||||
2, 3, 5, 7, 11, 11, 13, 17, 19, 23, 29, 29, 31, 37, 41, 43,
|
||||
47, 47, 53, 59, 61, 61, 67, 71, 73, 79, 83, 89, 89, 97, 101, 103,
|
||||
107, 107, 109, 113, 113, 127, 127, 127
|
||||
};
|
||||
static const size_t count = sizeof(vals)/sizeof(vals[0]);
|
||||
|
||||
using T = decltype(i);
|
||||
return static_cast<T>(vals[i % count] * static_cast<T>(multiplier));
|
||||
};
|
||||
};
|
||||
|
||||
template <typename ValueType>
|
||||
auto GCompatibilityTestSequence()
|
||||
{
|
||||
// Also multiply result by some factor to test large values on types that can hold those.
|
||||
return generateSeq<ValueType>(G(PrimesWithMultiplierGenerator(intExp10(sizeof(ValueType)))), 0, 42);
|
||||
}
|
||||
|
||||
INSTANTIATE_TEST_CASE_P(Gorilla,
|
||||
CodecTest_Compatibility,
|
||||
::testing::Combine(
|
||||
::testing::Values(Codec("Gorilla")),
|
||||
::testing::ValuesIn(std::initializer_list<std::tuple<CodecTestSequence, std::string>>{
|
||||
{
|
||||
GCompatibilityTestSequence<Int8>(),
|
||||
BIN_STR("\x95\x35\x00\x00\x00\x2a\x00\x00\x00\x01\x00\x2a\x00\x00\x00\x14\xe1\xdd\x25\xe5\x7b\x29\x86\xee\x2a\x16\x5a\xc5\x0b\x23\x75\x1b\x3c\xb1\x97\x8b\x5f\xcb\x43\xd9\xc5\x48\xab\x23\xaf\x62\x93\x71\x4a\x73\x0f\xc6\x0a")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<UInt8>(),
|
||||
BIN_STR("\x95\x35\x00\x00\x00\x2a\x00\x00\x00\x01\x00\x2a\x00\x00\x00\x14\xe1\xdd\x25\xe5\x7b\x29\x86\xee\x2a\x16\x5a\xc5\x0b\x23\x75\x1b\x3c\xb1\x97\x8b\x5f\xcb\x43\xd9\xc5\x48\xab\x23\xaf\x62\x93\x71\x4a\x73\x0f\xc6\x0a")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<Int16>(),
|
||||
BIN_STR("\x95\x52\x00\x00\x00\x54\x00\x00\x00\x02\x00\x2a\x00\x00\x00\xc8\x00\xdc\xfe\x66\xdb\x1f\x4e\xa7\xde\xdc\xd5\xec\x6e\xf7\x37\x3a\x23\xe7\x63\xf5\x6a\x8e\x99\x37\x34\xf9\xf8\x2e\x76\x35\x2d\x51\xbb\x3b\xc3\x6d\x13\xbf\x86\x53\x9e\x25\xe4\xaf\xaf\x63\xd5\x6a\x6e\x76\x35\x3a\x27\xd3\x0f\x91\xae\x6b\x33\x57\x6e\x64\xcc\x55\x81\xe4")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<UInt16>(),
|
||||
BIN_STR("\x95\x52\x00\x00\x00\x54\x00\x00\x00\x02\x00\x2a\x00\x00\x00\xc8\x00\xdc\xfe\x66\xdb\x1f\x4e\xa7\xde\xdc\xd5\xec\x6e\xf7\x37\x3a\x23\xe7\x63\xf5\x6a\x8e\x99\x37\x34\xf9\xf8\x2e\x76\x35\x2d\x51\xbb\x3b\xc3\x6d\x13\xbf\x86\x53\x9e\x25\xe4\xaf\xaf\x63\xd5\x6a\x6e\x76\x35\x3a\x27\xd3\x0f\x91\xae\x6b\x33\x57\x6e\x64\xcc\x55\x81\xe4")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<Int32>(),
|
||||
BIN_STR("\x95\x65\x00\x00\x00\xa8\x00\x00\x00\x04\x00\x2a\x00\x00\x00\x20\x4e\x00\x00\xe4\x57\x63\xc0\xbb\x67\xbc\xce\x91\x97\x99\x15\x9e\xe3\x36\x3f\x89\x5f\x8e\xf2\xec\x8e\xd3\xbf\x75\x43\x58\xc4\x7e\xcf\x93\x43\x38\xc6\x91\x36\x1f\xe7\xb6\x11\x6f\x02\x73\x46\xef\xe0\xec\x50\xfb\x79\xcb\x9c\x14\xfa\x13\xea\x8d\x66\x43\x48\xa0\xde\x3a\xcf\xff\x26\xe0\x5f\x93\xde\x5e\x7f\x6e\x36\x5e\xe6\xb4\x66\x5d\xb0\x0e\xc4")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<UInt32>(),
|
||||
BIN_STR("\x95\x65\x00\x00\x00\xa8\x00\x00\x00\x04\x00\x2a\x00\x00\x00\x20\x4e\x00\x00\xe4\x57\x63\xc0\xbb\x67\xbc\xce\x91\x97\x99\x15\x9e\xe3\x36\x3f\x89\x5f\x8e\xf2\xec\x8e\xd3\xbf\x75\x43\x58\xc4\x7e\xcf\x93\x43\x38\xc6\x91\x36\x1f\xe7\xb6\x11\x6f\x02\x73\x46\xef\xe0\xec\x50\xfb\x79\xcb\x9c\x14\xfa\x13\xea\x8d\x66\x43\x48\xa0\xde\x3a\xcf\xff\x26\xe0\x5f\x93\xde\x5e\x7f\x6e\x36\x5e\xe6\xb4\x66\x5d\xb0\x0e\xc4")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<Int64>(),
|
||||
BIN_STR("\x95\x91\x00\x00\x00\x50\x01\x00\x00\x08\x00\x2a\x00\x00\x00\x00\xc2\xeb\x0b\x00\x00\x00\x00\xe3\x2b\xa0\xa6\x19\x85\x98\xdc\x45\x74\x74\x43\xc2\x57\x41\x4c\x6e\x42\x79\xd9\x8f\x88\xa5\x05\xf3\xf1\x94\xa3\x62\x1e\x02\xdf\x05\x10\xf1\x15\x97\x35\x2a\x50\x71\x0f\x09\x6c\x89\xf7\x65\x1d\x11\xb7\xcc\x7d\x0b\x70\xc1\x86\x88\x48\x47\x87\xb6\x32\x26\xa7\x86\x87\x88\xd3\x93\x3d\xfc\x28\x68\x85\x05\x0b\x13\xc6\x5f\xd4\x70\xe1\x5e\x76\xf1\x9f\xf3\x33\x2a\x14\x14\x5e\x40\xc1\x5c\x28\x3f\xec\x43\x03\x05\x11\x91\xe8\xeb\x8e\x0a\x0e\x27\x21\x55\xcb\x39\xbc\x6a\xff\x11\x5d\x81\xa0\xa6\x10")
|
||||
},
|
||||
{
|
||||
GCompatibilityTestSequence<UInt64>(),
|
||||
BIN_STR("\x95\x91\x00\x00\x00\x50\x01\x00\x00\x08\x00\x2a\x00\x00\x00\x00\xc2\xeb\x0b\x00\x00\x00\x00\xe3\x2b\xa0\xa6\x19\x85\x98\xdc\x45\x74\x74\x43\xc2\x57\x41\x4c\x6e\x42\x79\xd9\x8f\x88\xa5\x05\xf3\xf1\x94\xa3\x62\x1e\x02\xdf\x05\x10\xf1\x15\x97\x35\x2a\x50\x71\x0f\x09\x6c\x89\xf7\x65\x1d\x11\xb7\xcc\x7d\x0b\x70\xc1\x86\x88\x48\x47\x87\xb6\x32\x26\xa7\x86\x87\x88\xd3\x93\x3d\xfc\x28\x68\x85\x05\x0b\x13\xc6\x5f\xd4\x70\xe1\x5e\x76\xf1\x9f\xf3\x33\x2a\x14\x14\x5e\x40\xc1\x5c\x28\x3f\xec\x43\x03\x05\x11\x91\xe8\xeb\x8e\x0a\x0e\x27\x21\x55\xcb\x39\xbc\x6a\xff\x11\x5d\x81\xa0\xa6\x10")
|
||||
},
|
||||
})
|
||||
),
|
||||
);
|
||||
|
||||
// These 'tests' try to measure performance of encoding and decoding and hence only make sence to be run locally,
|
||||
// also they require pretty big data to run agains and generating this data slows down startup of unit test process.
|
||||
// So un-comment only at your discretion.
|
||||
|
||||
//INSTANTIATE_TEST_CASE_P(DoubleDelta,
|
||||
// CodecTest_Performance,
|
||||
// ::testing::Combine(
|
||||
// ::testing::Values(Codec("DoubleDelta")),
|
||||
// ::testing::Values(
|
||||
// DDperformanceTestSequence<Int8 >(),
|
||||
// DDperformanceTestSequence<UInt8 >(),
|
||||
// DDperformanceTestSequence<Int16 >(),
|
||||
// DDperformanceTestSequence<UInt16>(),
|
||||
// DDperformanceTestSequence<Int32 >(),
|
||||
// DDperformanceTestSequence<UInt32>(),
|
||||
// DDperformanceTestSequence<Int64 >(),
|
||||
// DDperformanceTestSequence<UInt64>()
|
||||
// )
|
||||
// ),
|
||||
//);
|
||||
|
||||
//INSTANTIATE_TEST_CASE_P(Gorilla,
|
||||
// CodecTest_Performance,
|
||||
// ::testing::Combine(
|
||||
// ::testing::Values(Codec("Gorilla")),
|
||||
// ::testing::Values(
|
||||
// generatePyramidSequence<Int8 >(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<UInt8 >(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<Int16 >(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<UInt16>(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<Int32 >(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<UInt32>(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<Int64 >(42, G(PrimesWithMultiplierGenerator())) * 6'000,
|
||||
// generatePyramidSequence<UInt64>(42, G(PrimesWithMultiplierGenerator())) * 6'000
|
||||
// )
|
||||
// ),
|
||||
//);
|
||||
|
||||
}
|
||||
|
||||
@ -37,7 +37,7 @@ void Settings::setProfile(const String & profile_name, const Poco::Util::Abstrac
|
||||
{
|
||||
if (key == "constraints")
|
||||
continue;
|
||||
if (key == "profile") /// Inheritance of one profile from another.
|
||||
if (key == "profile" || 0 == key.compare(0, strlen("profile["), "profile[")) /// Inheritance of profiles from the current one.
|
||||
setProfile(config.getString(elem + "." + key), config);
|
||||
else
|
||||
set(key, config.getString(elem + "." + key));
|
||||
|
||||
@ -127,12 +127,11 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingUInt64, optimize_min_equality_disjunction_chain_length, 3, "The minimum length of the expression `expr = x1 OR ... expr = xN` for optimization ", 0) \
|
||||
\
|
||||
M(SettingUInt64, min_bytes_to_use_direct_io, 0, "The minimum number of bytes for reading the data with O_DIRECT option during SELECT queries execution. 0 - disabled.", 0) \
|
||||
M(SettingUInt64, min_bytes_to_use_mmap_io, 0, "The minimum number of bytes for reading the data with mmap option during SELECT queries execution. 0 - disabled.", 0) \
|
||||
\
|
||||
M(SettingBool, force_index_by_date, 0, "Throw an exception if there is a partition key in a table, and it is not used.", 0) \
|
||||
M(SettingBool, force_primary_key, 0, "Throw an exception if there is primary key in a table, and it is not used.", 0) \
|
||||
\
|
||||
M(SettingUInt64, mark_cache_min_lifetime, 10000, "If the maximum size of mark_cache is exceeded, delete only records older than mark_cache_min_lifetime seconds.", 0) \
|
||||
\
|
||||
M(SettingFloat, max_streams_to_max_threads_ratio, 1, "Allows you to use more sources than the number of threads - to more evenly distribute work across threads. It is assumed that this is a temporary solution, since it will be possible in the future to make the number of sources equal to the number of threads, but for each source to dynamically select available work for itself.", 0) \
|
||||
M(SettingFloat, max_streams_multiplier_for_merge_tables, 5, "Ask more streams when reading from Merge table. Streams will be spread across tables that Merge table will use. This allows more even distribution of work across threads and especially helpful when merged tables differ in size.", 0) \
|
||||
\
|
||||
@ -358,11 +357,8 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingBool, enable_unaligned_array_join, false, "Allow ARRAY JOIN with multiple arrays that have different sizes. When this settings is enabled, arrays will be resized to the longest one.", 0) \
|
||||
M(SettingBool, optimize_read_in_order, true, "Enable ORDER BY optimization for reading data in corresponding order in MergeTree tables.", 0) \
|
||||
M(SettingBool, low_cardinality_allow_in_native_format, true, "Use LowCardinality type in Native format. Otherwise, convert LowCardinality columns to ordinary for select query, and convert ordinary columns to required LowCardinality for insert query.", 0) \
|
||||
M(SettingBool, allow_experimental_multiple_joins_emulation, true, "Emulate multiple joins using subselects", 0) \
|
||||
M(SettingBool, allow_experimental_cross_to_join_conversion, true, "Convert CROSS JOIN to INNER JOIN if possible", 0) \
|
||||
M(SettingBool, cancel_http_readonly_queries_on_client_close, false, "Cancel HTTP readonly queries when a client closes the connection without waiting for response.", 0) \
|
||||
M(SettingBool, external_table_functions_use_nulls, true, "If it is set to true, external table functions will implicitly use Nullable type if needed. Otherwise NULLs will be substituted with default values. Currently supported only by 'mysql' and 'odbc' table functions.", 0) \
|
||||
M(SettingBool, allow_experimental_data_skipping_indices, false, "If it is set to true, data skipping indices can be used in CREATE TABLE/ALTER TABLE queries.", 0) \
|
||||
\
|
||||
M(SettingBool, experimental_use_processors, false, "Use processors pipeline.", 0) \
|
||||
\
|
||||
@ -386,14 +382,18 @@ struct Settings : public SettingsCollection<Settings>
|
||||
M(SettingBool, enable_scalar_subquery_optimization, true, "If it is set to true, prevent scalar subqueries from (de)serializing large scalar values and possibly avoid running the same subquery more than once.", 0) \
|
||||
M(SettingBool, optimize_trivial_count_query, true, "Process trivial 'SELECT count() FROM table' query from metadata.", 0) \
|
||||
M(SettingUInt64, mutations_sync, 0, "Wait for synchronous execution of ALTER TABLE UPDATE/DELETE queries (mutations). 0 - execute asynchronously. 1 - wait current server. 2 - wait all replicas if they exist.", 0) \
|
||||
M(SettingBool, optimize_if_chain_to_miltiif, false, "Replace if(cond1, then1, if(cond2, ...)) chains to multiIf. Currently it's not beneficial for numeric types.", 0) \
|
||||
\
|
||||
/** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \
|
||||
\
|
||||
M(SettingBool, allow_experimental_low_cardinality_type, true, "Obsolete setting, does nothing. Will be removed after 2019-08-13", 0) \
|
||||
M(SettingBool, compile, false, "Obsolete setting, does nothing. Will be removed after 2020-03-13", 0) \
|
||||
M(SettingBool, compile, false, "Whether query compilation is enabled. Will be removed after 2020-03-13", 0) \
|
||||
M(SettingUInt64, min_count_to_compile, 0, "Obsolete setting, does nothing. Will be removed after 2020-03-13", 0) \
|
||||
M(SettingBool, allow_experimental_multiple_joins_emulation, true, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \
|
||||
M(SettingBool, allow_experimental_cross_to_join_conversion, true, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \
|
||||
M(SettingBool, allow_experimental_data_skipping_indices, true, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \
|
||||
M(SettingBool, merge_tree_uniform_read_distribution, true, "Obsolete setting, does nothing. Will be removed after 2020-05-20", 0) \
|
||||
|
||||
M(SettingUInt64, mark_cache_min_lifetime, 0, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \
|
||||
|
||||
DECLARE_SETTINGS_COLLECTION(LIST_OF_SETTINGS)
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user