thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 00:22:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
2b3db1d33c
ClickHouse/programs/library-bridge/CMakeLists.txt

36 lines
1.3 KiB
CMake
Raw Normal View History

Better naming for stuff related to splitted debug symbols The previous name was slightly misleading, e.g. it is not about "intalling stripped binaries" but about splitting debug symbols from the binary.
2022-06-30 21:35:44 +00:00
include(${ClickHouse_SOURCE_DIR}/cmake/split_debug_symbols.cmake)
Add ability to strip binaries in cmake
2022-03-10 21:23:28 +00:00
First version
2021-03-05 09:38:00 +00:00
set (CLICKHOUSE_LIBRARY_BRIDGE_SOURCES
feat: implement catboost in library-bridge This commit moves the catboost model evaluation out of the server process into the library-bridge binary. This serves two goals: On the one hand, crashes / memory corruptions of the catboost library no longer affect the server. On the other hand, we can forbid loading dynamic libraries in the server (catboost was the last consumer of this functionality), thus improving security. SQL syntax: SELECT catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction, ACTION AS target FROM amazon_train LIMIT 10 Required configuration: <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path> *** Implementation Details *** The internal protocol between the server and the library-bridge is simple: - HTTP GET on path "/extdict_ping": A ping, used during the handshake to check if the library-bridge runs. - HTTP POST on path "extdict_request" (1) Send a "catboost_GetTreeCount" request from the server to the bridge, containing a library path (e.g /home/user/libcatboost.so) and a model path (e.g. /home/user/model.bin). Rirst, this unloads the catboost library handler associated to the model path (if it was loaded), then loads the catboost library handler associated to the model path, then executes GetTreeCount() on the library handler and finally sends the result back to the server. Step (1) is called once by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The library path handler is unloaded in the beginning because it contains state which may no longer be valid if the user runs catboost("/path/to/model.bin", ...) more than once and if "model.bin" was updated in between. (2) Send "catboost_Evaluate" from the server to the bridge, containing the model path and the features to run the interference on. Step (2) is called multiple times (once per chunk) by the server from function FunctionCatBoostEvaluate::executeImpl(). The library handler for the given model path is expected to be already loaded by Step (1). Fixes #27870
2022-08-05 07:53:06 +00:00
CatBoostLibraryHandler.cpp
CatBoostLibraryHandlerFactory.cpp
Prepare library-bridge for catboost integration - Rename generic file and identifier names in library-bridge to something more dictionary-specific. This is needed because later on, catboost will be integrated into library-bridge. - Also: Some smaller fixes like typos and un-inlining non-performance critical code. - The logic remains unchanged in this commit.
2022-08-03 11:19:13 +00:00
ExternalDictionaryLibraryAPI.cpp
ExternalDictionaryLibraryHandler.cpp
ExternalDictionaryLibraryHandlerFactory.cpp
First version
2021-03-05 09:38:00 +00:00
LibraryBridge.cpp
Prepare library-bridge for catboost integration - Rename generic file and identifier names in library-bridge to something more dictionary-specific. This is needed because later on, catboost will be integrated into library-bridge. - Also: Some smaller fixes like typos and un-inlining non-performance critical code. - The logic remains unchanged in this commit.
2022-08-03 11:19:13 +00:00
LibraryBridgeHandlerFactory.cpp
LibraryBridgeHandlers.cpp
Remove dlopen
2022-09-17 01:02:34 +00:00
SharedLibrary.cpp
Prepare library-bridge for catboost integration - Rename generic file and identifier names in library-bridge to something more dictionary-specific. This is needed because later on, catboost will be integrated into library-bridge. - Also: Some smaller fixes like typos and un-inlining non-performance critical code. - The logic remains unchanged in this commit.
2022-08-03 11:19:13 +00:00
library-bridge.cpp
First version
2021-03-05 09:38:00 +00:00
)
if (OS_LINUX)
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--no-export-dynamic")
endif ()
Prefix overridden add_executable() command with "clickhouse_" A simple HelloWorld program with zero includes except iostream triggers a build of ca. 2000 source files. The reason is that ClickHouse's top-level CMakeLists.txt overrides "add_executable()" to link all binaries against "clickhouse_new_delete". This links against "clickhouse_common_io", which in turn has lots of 3rd party library dependencies ... Without linking "clickhouse_new_delete", the number of compiled files for "HelloWorld" goes down to ca. 70. As an example, the self-extracting-executable needs none of its current dependencies but other programs may also benefit. In order to restore access to the original "add_executable()", the overriding version is now prefixed. There is precedence for a "clickhouse_" prefix (as opposed to "ch_"), for example "clickhouse_split_debug_symbols". In general prefixing makes sense also because overriding CMake commands relies on undocumented behavior and is considered not-so-great practice (*). (*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
2022-07-11 17:19:15 +00:00
clickhouse_add_executable(clickhouse-library-bridge ${CLICKHOUSE_LIBRARY_BRIDGE_SOURCES})
First version
2021-03-05 09:38:00 +00:00
target_link_libraries(clickhouse-library-bridge PRIVATE
Fix
2021-04-07 07:16:50 +00:00
daemon
First version
2021-03-05 09:38:00 +00:00
dbms
Fix split build finally
2021-03-12 21:12:34 +00:00
bridge
First version
2021-03-05 09:38:00 +00:00
)
set_target_properties(clickhouse-library-bridge PROPERTIES RUNTIME_OUTPUT_DIRECTORY ..)
Better naming for stuff related to splitted debug symbols The previous name was slightly misleading, e.g. it is not about "intalling stripped binaries" but about splitting debug symbols from the binary.
2022-06-30 21:35:44 +00:00
if (SPLIT_DEBUG_SYMBOLS)
clickhouse_split_debug_symbols(TARGET clickhouse-library-bridge DESTINATION_DIR ${CMAKE_CURRENT_BINARY_DIR}/../${SPLITTED_DEBUG_SYMBOLS_DIR} BINARY_PATH ../clickhouse-library-bridge)
Fix strip bug
2022-03-16 11:53:15 +00:00
else()
Better naming for stuff related to splitted debug symbols The previous name was slightly misleading, e.g. it is not about "intalling stripped binaries" but about splitting debug symbols from the binary.
2022-06-30 21:35:44 +00:00
clickhouse_make_empty_debug_info_for_nfpm(TARGET clickhouse-library-bridge DESTINATION_DIR ${CMAKE_CURRENT_BINARY_DIR}/../${SPLITTED_DEBUG_SYMBOLS_DIR})
Add ability to strip binaries in cmake
2022-03-10 21:23:28 +00:00
install(TARGETS clickhouse-library-bridge RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse)
endif()
Reference in New Issue Copy Permalink