ClickHouse/.gitignore
Robert Schulze 6b2b3c1eb3
feat: implement catboost in library-bridge
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
2022-08-29 20:26:45 +00:00

156 lines
1.7 KiB
Plaintext

# emacs files
*~
*\#
.tramp_history
# vim cache files
*.swp
# auto generated files
*.logrt
/build
/build_*
/build-*
/tests/venv
/obj-x86_64-linux-gnu/
# logs
*.log
*.stderr
*.stdout
/docs/build
/docs/publish
/docs/edit
/docs/website
/docs/venv
/docs/tools/venv
/docs/tools/translate/venv
/docs/tools/translate/output.md
/docs/en/single.md
/docs/ru/single.md
/docs/zh/single.md
/docs/ja/single.md
/docs/fa/single.md
# callgrind files
callgrind.out.*
# ignore kdevelop files
*.kdev4
*.kdev_include_paths
# ignore sublime project files
*.sublime-*
# Qt Creator files
*.user
# ignore perf output
*/perf.data
# ignore build files
CMakeCache.txt
CMakeFiles
Makefile
cmake_install.cmake
CTestTestfile.cmake
*.a
*.o
*.so
*.dll
*.lib
*.dylib
cmake-build-*
# Python cache
*.pyc
__pycache__
*.pytest_cache
test.cpp
CPackConfig.cmake
CPackSourceConfig.cmake
*-preprocessed.xml
core
!core/
vgcore*
*.deb
*.build
*.upload
*.changes
build-stamp
configure-stamp
*.bin
*.mrk
*.mrk2
*.mrk3
.dupload.conf
# Netbeans project files
nbproject/*
# JetBrains project files
.idea
# Microsoft Visual Studio Code
.vscode
config-preprocessed.xml
# Protobuf
*.pb.cpp
*.pb.h
# Ignore symlink to private repository
/private
# Gulp dependencies used to minify website
node_modules
public
website/docs
website/presentations
website/package-lock.json
.DS_Store
*/.DS_Store
# cquery cache
/.cquery-cache
# ccls cache
/.ccls-cache
# clangd cache
/.clangd
/.cache
/compile_commands.json
# Toolchains
/cmake/toolchain/*
# ANTLR extension cache
.antlr
# ANTLR generated files
/src/Parsers/New/*.interp
/src/Parsers/New/*.tokens
/src/Parsers/New/ClickHouseParserBaseVisitor.*
# pytest-profiling
/prof
*.iml
# data store
/programs/server/data
/programs/server/metadata
/programs/server/store