Commit Graph

87 Commits

Author SHA1 Message Date
Azat Khuzhin
0b258dda4e Reproducible builds for Rust
From now on cargo will not download anything from the internet during
builds. This step had been moved for docker image builds (via cargo
vendor).

And now cargo inside docker.io/clickhouse/binary-builder will not use
any crates from the internet, so we don't need to add --offline for
cargo commands in cmake (corrosion_import_crate()).

Also the docker build command had been adjusted to allow following
symlinks inside build context, by using tar, this is required for Rust
packages.

Note, that to make proper Cargo.lock that could be vendored I did the
following:
- per-project locks had been removed (since there is no automatic way to
  sync the workspace Cargo.lock with per-project Cargo.lock, since cargo
  update/generate-lockfile will use only per-project Cargo.toml files
  apparently, -Z minimal-versions does not helps either)
- and to generate Cargo.lock with less changes I've pinned version in
  the Cargo.toml strictly, i.e. not 'foo = "0.1"' but 'foo = "=0.1"'
  then the Cargo.lock for workspace had been generated and afterwards
  I've reverted this part.

Plus I have to update the dependencies afterwards, since otherwise there
are conflicts with dependencies for std library. Non trivial.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-22 22:46:22 +02:00
Azat Khuzhin
1dbb86a045 Add _gen to ignore (generated by pytest)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-17 17:17:21 +02:00
Azat Khuzhin
87b5ab2fcd Enable few slow clang-tidy checks for clangd
In  the readability-identifier-naming had been disabled for
clang-tidy builds, but the code base is already uses this notations, so
at least, let's enable it for clangd, so that developers who are using
editor/IDE with clangd will be warned.

FWIW clangd does not think that this check is slow [1], but I guess this
file hadn't been updated for quite a while.

  [1]: https://github.com/llvm/llvm-project/blob/main/clang-tools-extra/clangd/TidyFastChecks.inc

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-13 14:08:25 +02:00
Azat Khuzhin
238d44783b Fix flakiness of expect tests for clickhouse-client by avoiding history overlap
Yes, all writes to the history file is done under flock, *but*, before
writing the history file there is sort(), and so if you will run the
following tests the 01300_client_save_history_when_terminated_long will fail:

    $ /src/tests/clickhouse-test --print-time -j2 01180_client_syntax_errors 01300_client_save_history_when_terminated_long 0001_select

And it has nothing todo with timeouts:

    expect: does "" (spawn_id exp8) match glob pattern "for the history"? no
    f8f1dbfdaaca :) select (1, 2
    expect: does "\u001b[1Gf8f1dbfdaaca :) select \u001b[0;22;33m(\u001b[0;22;32m1\u001b[0;1m,\u001b[0m \u001b[0;22;32m2\u001b[0m\u001b[J" (spawn_id exp8) match glob pattern "for the history"? no

    expect: does "\u001b[1Gf8f1dbfdaaca :) select \u001b[0;22;33m(\u001b[0;22;32m1\u001b[0;1m,\u001b[0m \u001b[0;22;32m2\u001b[0m\u001b[J\u001b[29G" (spawn_id exp8) match glob pattern "for the history"? no
    expect: timed out

The "select (1, 2" is from 01180_client_syntax_errors

And use real file only when the history should be preserved across runs
(i.e. there are multiple invocations of clickhouse-client)

CI: https://s3.amazonaws.com/clickhouse-test-reports/0/1adfbac19fed9813725d8b1df14e617b58a45d20/stateless_tests__asan__[2/2].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-20 09:51:08 +01:00
Nikolay Degterinsky
bd4bc2df71
Update .gitignore 2023-01-18 01:49:52 +01:00
Azat Khuzhin
82aaad61aa Integrate skim into the client/local
Note, that it can the fail the client if the skim itself will fail,
however I haven't seen it panicd, so let's try.

P.S. about adding USE_SKIM into configure header instead of just compile
option for target, it is better, because it allows not to recompile lots
of C++ headers, since we have to add skim library as PUBLIC. But anyway
this will be resolved in a different way, but separatelly.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-14 20:57:41 +01:00
Azat Khuzhin
dab6919411 Add rust target folder to gitignore
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-14 20:57:22 +01:00
Azat Khuzhin
67fa185611
Revert "Builtin skim" 2022-12-14 17:17:19 +03:00
Alexey Milovidov
c93b7741b3
Merge branch 'master' into builtin-skim 2022-12-14 02:30:21 +03:00
Alexander Tokmakov
eaa7096c4e add *.generated-expect to gitignore 2022-12-12 14:05:37 +01:00
Azat Khuzhin
de58e9c02d Integrate skim into the client/local
Note, that it can the fail the client if the skim itself will fail,
however I haven't seen it panicd, so let's try.

P.S. about adding USE_SKIM into configure header instead of just compile
option for target, it is better, because it allows not to recompile lots
of C++ headers, since we have to add skim library as PUBLIC.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-11 15:52:00 +01:00
Azat Khuzhin
f8a070acbe Add rust target folder to gitignore
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-12-11 15:51:37 +01:00
Azat Khuzhin
0b48a2ed5e tests: add *.debuglog to gitignore
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-30 19:36:17 +01:00
Yatsishin Ilya
8e77180968 Merge remote-tracking branch 'origin/master' into revive-sqlancer 2022-10-26 08:31:00 +00:00
Yatsishin Ilya
3794d21089 ignore tests tmp files 2022-10-26 08:30:57 +00:00
Azat Khuzhin
5016597e58 Add arch packages to .gitignore
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-18 18:33:48 +03:00
Robert Schulze
60f9f6855d
feat: implement catboost in library-bridge
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes 
2022-09-08 09:01:32 +00:00
Robert Schulze
912663b719
Revert "Move CatBoost evaluation into clickhouse-library-bridge" 2022-08-31 20:54:43 +02:00
Robert Schulze
6b2b3c1eb3
feat: implement catboost in library-bridge
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes 
2022-08-29 20:26:45 +00:00
rfraposa
06ac99b1e7 Add cmake page back to docs && fix /settings/settings in /zh 2022-04-24 16:47:19 -06:00
Yatsishin Ilya
f59c7f5254 Add more build paths to .gitignore 2022-01-18 15:25:48 +00:00
Alexander Tokmakov
571dd3acfb fix style check 2021-09-21 10:28:33 +03:00
Azat Khuzhin
24c8968c80 Add *.log/*.stderr/*.stdout into gitignore 2021-06-08 09:14:47 +03:00
Nikolai Kochetov
77233bbdbb Ignore cmake-in-clickhouse 2021-04-28 18:10:12 +03:00
tison
c809af5dc2 ignore data store files 2021-02-17 12:58:17 +08:00
Ivan
315ff4f0d9
ANTLR4 Grammar for ClickHouse and new parser () 2020-12-04 05:15:44 +03:00
qianmoQ
e740bd40f1 fix document for index.md and distinctive-features.md 2020-11-24 20:36:19 +08:00
Ivan Lezhankin
7ea393ada8 Fix build without libraries 2020-10-10 23:41:27 +03:00
vladimir golovchenko
cb153d2605 Updated gitignore-files. 2020-08-06 18:05:32 -07:00
Ivan Blinkov
4ef322274d Add integration test 2020-05-29 22:53:16 +03:00
Alexey Milovidov
2986fcd93e Remove outdated contents from gitignore 2020-05-05 19:38:40 +03:00
Ivan Blinkov
98769778f4
Turkish docs translation stub () 2020-04-15 16:56:49 +03:00
Ivan Blinkov
765dd7c495
Update some docs translations () 2020-04-04 12:15:31 +03:00
Ivan Lezhankin
06446b4f08 dbms/ → src/ 2020-04-03 18:14:31 +03:00
Ivan
97f2a2213e
Move all folders inside /dbms one level up ()
* Move some code outside dbms/src folder
* Fix paths
2020-04-02 02:51:21 +03:00
Ivan Blinkov
03aa7894d9
Draft of docs translation helper ()
* replace exit with assert in test_single_page

* improve save_raw_single_page docs option

* More grammar fixes

* "Built from" link in new tab

* fix mistype

* Example of include in docs

* add anchor to meeting form

* Draft of translation helper

* WIP on translation helper

* Replace some fa docs content with machine translation
2020-03-19 20:49:27 +03:00
Ivan Blinkov
5abe3ac3f1
Switch docs to python3 and update MkDocs to 1.1 ()
+ some grammar and css fixes
2020-03-18 16:02:32 +03:00
Ivan Blinkov
242a1a85d4 adjust .gitignore 2020-02-14 12:34:18 +03:00
Ivan Blinkov
18538f5c65 Domain change in docs 2020-01-30 13:34:55 +03:00
Ivan Lezhankin
e63ef08af8 Update gitignore 2020-01-14 16:30:06 +03:00
Ivan
4f2f5cca84
Add support for cross-compiling to the CPU architecture AARCH64 () 2019-10-30 10:01:53 +03:00
Alexander Tokmakov
93c672aa0b delete BlockInputStreamFromRowInputStream 2019-08-27 21:29:56 +03:00
Nikita Mikhaylov
b8f99255ae better gitignore with mrk2 2019-07-04 21:30:01 +03:00
Ivan Lezhankin
92769a2460 Don't update "arrow" on client-side for nothing 2019-02-07 16:47:16 +03:00
Alexander GQ Gerasiov
afa9e8d4ea .gitignore: Move debian/ specific entries to debian/.gitignore
Signed-off-by: Alexander GQ Gerasiov <gq@cs.msu.su>
2019-01-21 01:26:50 +03:00
Ivan Lezhankin
77daa519ff Update librdkafka to v1.0.0-RC5 2019-01-14 14:15:57 +03:00
Ivan Lezhankin
6df757c6f7 Refactor constant folding and make it reusable for primary_key_expr 2018-12-17 17:59:01 +03:00
Ivan Lezhankin
935615a647 Reimplement FREEZE command. 2018-11-12 15:26:14 +03:00
proller
a7437b93a9 Commited StorageSystemContributors.generated.cpp ()
* CLICKHOUSE-4085 system.contributors

* fi

* Fix random

* Commited StorageSystemContributors.generated.cpp

* fix

* Update CMakeLists.txt
2018-11-01 17:05:37 +03:00
abyss7
d538f70679 Fix build and tests on Fedora ()
* Fix some tests and build on Fedora 28

* Update contrib/ssl

* Try `sudo` first, then without `sudo`.
2018-10-30 17:05:44 +03:00