Commit Graph

20525 Commits

Author SHA1 Message Date
Robert Schulze
56eece40ec
Merge pull request #40736 from LevyCory/add-offset-to-formatDateTime
Add timezone offset support  to `formatDateTime`
2022-09-01 09:50:17 +02:00
Nikolay Degterinsky
e98ceb2575
Merge pull request #40251 from ClibMouse/Structured-logging-custom-keys
Feature Improvement: Custom Key Names for Structured Logging Support
2022-09-01 00:36:23 +02:00
Robert Schulze
912663b719
Revert "Move CatBoost evaluation into clickhouse-library-bridge" 2022-08-31 20:54:43 +02:00
Kruglov Pavel
86516d3bb4
Merge pull request #40740 from amosbird/row-policy-index-fix-1
Use index when row_policy_filter is always false
2022-08-31 18:46:14 +02:00
Kruglov Pavel
b7ea529307
Merge pull request #40746 from Avogar/fix-csv-schema-inference
Fix possible error Attempt to read after eof in CSV schema inference
2022-08-31 18:29:01 +02:00
Robert Schulze
ca01286028
Merge pull request #39629 from ClickHouse/catboost-bridge
Move CatBoost evaluation into clickhouse-library-bridge
2022-08-31 16:16:11 +02:00
Anton Popov
5a3e24c4e4
Merge pull request #40483 from canhld94/ch_canh_fix_json
Fix columnObject crash with array of variadic dimension elems
2022-08-31 16:07:58 +02:00
root
ea517de61a style check correction 2022-08-30 23:20:19 -03:00
root
f02f7ce99c addressed minor change requests 2022-08-30 23:20:19 -03:00
root
84a617675d style check correction in test.py 2022-08-30 23:20:19 -03:00
root
3ced3c9933 style check correction in test.py 2022-08-30 23:20:19 -03:00
root
1d29a494db Modified test code to cover various scenarios for custom keys 2022-08-30 23:20:19 -03:00
root
d5db88fa27 Structured-logging-custom-keys PR init 2022-08-30 23:20:19 -03:00
Cory Levy
4a9ccc6496 Fix timezone offset tests 2022-08-30 13:42:12 -04:00
Alexander Tokmakov
6fdfb964d0
Revert "Add Annoy index" 2022-08-30 15:10:10 +03:00
Vladimir C
7266c007b7
Merge pull request #39131 from amosbird/cte-global-with-fix1
Fix unused unknown columns introduced by WITH statement
2022-08-30 13:52:19 +02:00
Alexander Tokmakov
022f440ad0
Merge pull request #40769 from arthurpassos/caresptrresolver-hosts-patch
Fix CaresPTRResolver not reading hosts file
2022-08-30 14:35:10 +03:00
alesapin
2588901bc9
Merge pull request #40780 from ClickHouse/add_cache_to_zero_copy_test
Add cache to zero copy replication test
2022-08-30 12:42:05 +02:00
Kseniia Sumarokova
c88db2ef97
Merge pull request #40751 from kssenii/fix-mysql-timeouts
Fix issue with mysql db / table function timeouts
2022-08-30 11:59:01 +02:00
Robert Schulze
cc4225109f
Merge pull request #37215 from Vector-Similarity-Search-for-ClickHouse/annoy-2
Test failures are unrelated, merging.
2022-08-30 09:25:57 +02:00
Robert Schulze
8ba9f6dd5d
fix: delete obsolete test 02413_model_evaluate_smoke 2022-08-30 07:21:05 +00:00
Cory Levy
cd371110ad Fix timezone offset formatting 2022-08-29 21:01:27 -04:00
alesapin
ac41de73df Add cache to zero copy replication test 2022-08-30 00:17:01 +02:00
Kseniia Sumarokova
8001560082
Fix test 2022-08-30 00:14:00 +02:00
Cory Levy
fe24bd9a7f Add tests 2022-08-29 16:47:33 -04:00
Robert Schulze
4ae7586a2d
fix: test 02415_all_new_functions_must_be_documented
I am happy to move the documentation into the source code but currently
we'd need to maintain it twice (source code + traditional markdown docs)
because we don't render the docs from the source code.
2022-08-29 20:27:06 +00:00
Robert Schulze
35a37c91f8
chore: incorporate review feedback 2022-08-29 20:27:06 +00:00
Robert Schulze
3aec76d281
fix: test 02117_show_create_table_system 2022-08-29 20:27:06 +00:00
robot-clickhouse
64fa077148
style: fix style 2022-08-29 20:27:06 +00:00
Robert Schulze
4d511332c4
chore: delete obsolete modelEvaluate() function
- superseded by catboostEvaluate() which no longer uses the internal
  repository for external models

- also removed was statement SYSTEM RELOAD MODELS and the monitoring view
  SYSTEM.SYSTEMMODELS
2022-08-29 20:27:06 +00:00
Robert Schulze
6b2b3c1eb3
feat: implement catboost in library-bridge
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
2022-08-29 20:26:45 +00:00
Vitaly Baranov
33f72fb011
Merge pull request #40060 from ClickHouse/vitlibar-increase-timeout-for-test_concurrent_backups
Increase timeout for test_concurrent_backups
2022-08-29 22:25:56 +02:00
Arthur Passos
dd49b44abb Fix host_regexp hosts file tst 2022-08-29 15:58:18 -03:00
Arthur Passos
961365c7a4 Fix CaresPTRResolver not reading hosts file 2022-08-29 15:11:39 -03:00
Kseniia Sumarokova
c5c48e44ea
Merge branch 'master' into fix-mysql-timeouts 2022-08-29 19:33:29 +02:00
Filatenkov Artur
d73f661732
Merge branch 'master' into annoy-2 2022-08-29 17:33:13 +03:00
kssenii
0a6c4b9265 Fix 2022-08-29 16:20:53 +02:00
Vladimir C
5cbe7e0846
Merge pull request #40548 from ClickHouse/vdimir/warn-suppress-40330
Add config option warning_supress_regexp
2022-08-29 14:02:00 +02:00
Alexander Tokmakov
8d90d30d37
Merge pull request #40589 from ClickHouse/remove_wrong_code_from_mutations
Remove wrong code for skipping mutations in MergeTree
2022-08-29 14:18:59 +03:00
avogar
85fa372ad7 Fix possible error Attempt to read after eof in CSV schema inference 2022-08-29 11:18:53 +00:00
Alexander Tokmakov
eda0582ec0
Merge pull request #40641 from ClickHouse/fix_startup_of_dropped_replica
Do not try to strartup dropped replica
2022-08-29 14:15:15 +03:00
Vitaly Baranov
2bec3d3a7c Increase timeout for test_concurrent_backups 2022-08-29 13:13:43 +02:00
alesapin
7ce0afc0df
Merge pull request #40670 from Avogar/kafka
Add setting to disable limit on kafka_num_consumers
2022-08-29 10:53:35 +02:00
Amos Bird
15a69bce84
Use index when row_policy_filter is always false 2022-08-29 16:44:32 +08:00
Alexey Milovidov
b72fceb441
Merge pull request #40708 from lesandie/test_s3_table_functions
Added integration test for s3 table function
2022-08-29 03:49:53 +03:00
Alexey Milovidov
18eaf7d0dc
Merge pull request #40721 from ClickHouse/enable_zero_copy_replication_in_ci
Enable zero-copy replication in CI
2022-08-29 03:49:23 +03:00
Alexey Milovidov
71f6c52c2d
Merge pull request #40727 from amosbird/column-transformer-fix1
Correct format of APPLY transformer param
2022-08-29 03:48:04 +03:00
Alexey Milovidov
82ef85e713
Merge pull request #40722 from kssenii/fix-test-02382
Fix flaky test
2022-08-28 23:34:10 +03:00
Diego Nieto (lesandie)
3d50dbea34 Black reformatting 2022-08-28 20:05:39 +02:00
Alexey Milovidov
a82723b5d9
Merge pull request #40719 from ClickHouse/kssenii-patch-4
Update 02313_filesystem_cache_seeks.queries
2022-08-28 21:04:48 +03:00