kssenii
559c696230
Fix
2022-09-14 20:35:28 +02:00
mateng0915
372a0b7794
added events into s3 integration test
2022-09-14 19:19:47 +08:00
alesapin
51a302a70f
Merge branch 'master' into revert-40968-s3-sharding-2
2022-09-14 11:53:14 +02:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown
...
Close sessions on Keeper shutdown
2022-09-13 19:09:34 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage
...
KeeperMap storage engine
2022-09-13 14:15:09 +02:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze
...
Fix SYSTEM UNFREEZE for ordinary database
2022-09-13 14:55:11 +03:00
alesapin
9af591a328
Revert "Sharding s3 key names (2)"
2022-09-13 12:29:02 +02:00
Antonio Andelic
45cde90219
Fix test
2022-09-13 09:51:31 +00:00
Sergei Trifonov
9cd78585c3
fix tests
2022-09-13 00:12:40 +02:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS
...
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
a mechanism to update a model explicitly. It also saves potentially
unnecessary reloads of a model from disk after it's initial load.
To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
was changed from eager to lazy. This means that both statements
previously immedately reloaded the specified/all models, whereas now
the statements only trigger an unload and the first call to
catboostEvaluate() does the actual load.
- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
fields removed. The view was not documented in the past and for now it
remains undocumented. The commit is thus not considered a breach of
ClickHouse's public interface.
2022-09-12 19:33:02 +00:00
Azat Khuzhin
b698a4ff65
Apply changes to http handlers on fly without server restart
...
This has been implemented by simply restarting http servers in case of
http_handlers directive in configuration xml had been changed.
But, for this I have to change the handlers interface to accept
configuration separatelly, since the configuration that contains in the
server is the configuration with which server had been started.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Antonio Andelic
2022-09-12 17:34:51 +02:00
Antonio Andelic
deee0d639f
Address PR comments
2022-09-12 14:43:52 +00:00
Sergei Trifonov
fe88a7991d
Merge branch 'master' into readonly-settings-allow
2022-09-12 16:20:40 +02:00
Antonio Andelic
1cd43d8e5c
Merge branch 'master' into keeper-storage
2022-09-12 13:42:51 +00:00
Antonio Andelic
e985ee3354
Close sessions on Keeper shutdown
2022-09-12 12:22:48 +00:00
kssenii
02c3d8b0a0
Fix
2022-09-11 16:42:56 +02:00
Alexey Milovidov
5802c2fdd2
Merge pull request #40713 from ClickHouse/remove-useless-line
...
Remove one line from XML, because we do not care
2022-09-11 09:53:30 +03:00
Alexey Milovidov
a5388affb8
Merge pull request #41023 from ClickHouse/remove-trash-3
...
Remove trash from config
2022-09-11 09:52:45 +03:00
Alexey Milovidov
91338ea771
Merge branch 'master' into remove-useless-line
2022-09-11 02:03:39 +03:00
Azat Khuzhin
46eafb4732
tests: skip test_send_crash_reports under ASan
...
Since it also does not fits into timeouts [1]:
2022-09-08 21:16:20 [ 377 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestsendcrashreports_node_1', 'bash', '-c', 'pkill -SEGV clickhouse'] (cluster.py:95, run_and_check)
...
2022-09-08 21:16:22 [ 377 ] DEBUG : run container_id:roottestsendcrashreports_node_1 detach:False nothrow:False cmd: ['cat', '/result.txt'] (cluster.py:1744, exec_in_container)
...
2022-09-08 21:16:36 [ 377 ] DEBUG : Stdout:INITIAL_STATE (cluster.py:103, run_and_check)
And server logs:
2022.09.08 21:16:21.112076 [ 228 ] {} <Fatal> BaseDaemon: ########################################
2022.09.08 21:16:21.112170 [ 228 ] {} <Fatal> BaseDaemon: (version 22.9.1.1, build id: 0F7336E0A4D64134C51C8365DADCB78A9B39AA3B) (from thread 1) (no query) Received signal Segmentation fault (11)
2022.09.08 21:16:21.112244 [ 228 ] {} <Fatal> BaseDaemon: Address: 0xde Access: read. Unknown si_code.
2022.09.08 21:16:21.112321 [ 228 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fbe21d09376 0x40a4f71a 0xe293a4b 0xdc5c51a 0x38dee227 0xdc326a0 0x38e319d9 0xdc2b4e2 0xdc25fdb 0x7fbe21b2c083 0xdb636ae
2022.09.08 21:16:21.112419 [ 228 ] {} <Fatal> BaseDaemon: 3. pthread_cond_wait @ 0x7fbe21d09376 in ?
2022.09.08 21:16:21.122914 [ 228 ] {} <Fatal> BaseDaemon: 4. ./build_docker/../contrib/libcxx/src/condition_variable.cpp:0: std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) @ 0x40a4f71a in /usr/bin/clickhouse
2022.09.08 21:16:21.233016 [ 228 ] {} <Fatal> BaseDaemon: 5.1. inlined from ./build_docker/../contrib/libcxx/include/atomic:952: unsigned long std::__1::__cxx_atomic_load<unsigned long>(std::__1::__cxx_atomic_base_impl<unsigned long> const*, std::__1::memory_order)
2022.09.08 21:16:21.233135 [ 228 ] {} <Fatal> BaseDaemon: 5.2. inlined from ../contrib/libcxx/include/atomic:1582: std::__1::__atomic_base<unsigned long, false>::load(std::__1::memory_order) const
2022.09.08 21:16:21.233183 [ 228 ] {} <Fatal> BaseDaemon: 5.3. inlined from ../contrib/libcxx/include/atomic:1586: std::__1::__atomic_base<unsigned long, false>::operator unsigned long() const
2022.09.08 21:16:21.233234 [ 228 ] {} <Fatal> BaseDaemon: 5.4. inlined from ../src/Daemon/BaseDaemon.cpp:967: operator()
2022.09.08 21:16:21.233303 [ 228 ] {} <Fatal> BaseDaemon: 5.5. inlined from ../contrib/libcxx/include/__mutex_base:402: void std::__1::condition_variable::wait<BaseDaemon::waitForTerminationRequest()::$_0>(std::__1::unique_lock<std::__1::mutex>&, BaseDaemon::waitForTerminationRequest()::$_0)
2022.09.08 21:16:21.233334 [ 228 ] {} <Fatal> BaseDaemon: 5. ../src/Daemon/BaseDaemon.cpp:967: BaseDaemon::waitForTerminationRequest() @ 0xe293a4b in /usr/bin/clickhouse
2022.09.08 21:16:21.350675 [ 228 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../programs/server/Server.cpp:0: DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xdc5c51a in /usr/bin/clickhouse
2022.09.08 21:16:21.394092 [ 228 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../contrib/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x38dee227 in /usr/bin/clickhouse
2022.09.08 21:16:21.654195 [ 228 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../programs/server/Server.cpp:466: DB::Server::run() @ 0xdc326a0 in /usr/bin/clickhouse
2022.09.08 21:16:21.666991 [ 228 ] {} <Fatal> BaseDaemon: 9. ./build_docker/../contrib/poco/Util/src/ServerApplication.cpp:0: Poco::Util::ServerApplication::run(int, char**) @ 0x38e319d9 in /usr/bin/clickhouse
2022.09.08 21:16:21.916078 [ 228 ] {} <Fatal> BaseDaemon: 10. ./build_docker/../programs/server/Server.cpp:0: mainEntryClickHouseServer(int, char**) @ 0xdc2b4e2 in /usr/bin/clickhouse
2022.09.08 21:16:21.929922 [ 228 ] {} <Fatal> BaseDaemon: 11. ./build_docker/../programs/main.cpp:0: main @ 0xdc25fdb in /usr/bin/clickhouse
2022.09.08 21:16:21.929981 [ 228 ] {} <Fatal> BaseDaemon: 12. __libc_start_main @ 0x7fbe21b2c083 in ?
2022.09.08 21:16:30.357032 [ 228 ] {} <Fatal> BaseDaemon: 13. _start @ 0xdb636ae in /usr/bin/clickhouse
2022.09.08 21:16:31.383233 [ 228 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read. (calculated checksum: 6200AC7C1270DC293DF3302E1C64399B)
...
2022.09.08 21:16:40.564453 [ 228 ] {} <Information> SentryWriter: Sending crash report
[1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/a0b85eaca8d4003c9fbc4571b30830d30f1984e9/integration_tests__asan__[3/3].html
Though another option is to increase waiting time.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-10 21:38:35 +02:00
robot-clickhouse
5bc2845e49
Automatic style fix
2022-09-10 03:17:34 +00:00
Alexey Milovidov
f2227e125f
Fix test
2022-09-10 05:11:09 +02:00
Alexey Milovidov
61278c81e4
Merge branch 'master' into remove-useless-line
2022-09-10 05:07:10 +02:00
robot-clickhouse
79ccce23e9
Automatic style fix
2022-09-09 20:09:54 +00:00
Igor Nikonov
dbfb448290
Fix: integration test, standalone keeper mode
...
There is logic regarding which keeper binary use to start keeper cluster in an integration test
There 2 options:
(1) standalone keeper binary (expected binary name clickhouse-keeper)
(2) clickhouse binary with keeper inside
Fixed:
- option (1) didn't work since docker_compose_keeper.yaml didn't create
target clickhouse-keeper at all
- if clickhouse-keeper existed, option (1) was taken but
clickhouse-keeper could be just a link to clickhouse binary (the link
is created always during build if cmake option BUILD_STANDALONE_KEEPER is OFF)
2022-09-09 14:51:34 +00:00
robot-clickhouse
aeac3916ef
Automatic style fix
2022-09-09 12:27:13 +00:00
Antonio Andelic
32483aeec8
Merge branch 'master' into keeper-storage
2022-09-09 12:24:01 +00:00
Sergei Trifonov
76eb001e78
add test for changeable_in_readonly + fix SQL in docs
2022-09-09 14:01:29 +02:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db
2022-09-09 14:24:44 +03:00
Vitaly Baranov
10629a66e5
Fix black.
2022-09-08 20:58:51 +02:00
Vitaly Baranov
e48cd2b6f6
Add more test cases.
2022-09-08 18:14:25 +02:00
robot-clickhouse
807b09f6b0
Automatic style fix
2022-09-08 15:52:00 +00:00
Sergei Trifonov
f92b6a6176
Merge branch 'master' into readonly-settings-allow
2022-09-08 17:44:32 +02:00
Sergei Trifonov
62541ab764
fix more tests and clang tidy build
2022-09-08 17:43:09 +02:00
Antonio Andelic
b99996961d
Address PR comments
2022-09-08 12:45:36 +00:00
Vitaly Baranov
122009a2bd
Use table lock if database is ordinary and zero-copy-replication is enabled.
2022-09-08 13:54:59 +02:00
Robert Schulze
c16707ff00
chore: delete obsolete modelEvaluate() function + SYSTEM.MODELS view
...
- The deleted function modelEvaluate() was superseded by
catboostEvaluate().
- Also delete the external model repository, as modelEvaluate() was it's
last user. Additionally remove the system view SYSTEM.MODELS for
inspecting the repository.
- SYSTEM RELOAD MODELS is also obsolete. HOWEVER, it was retained and
made a no-op instead of deleted.
Why?
The reason is that RBAC in distributed setups works by storing
privileges (granted and revoked) as plain SQL statements in Keeper.
Nodes read these statements at startup and parse them. If a privilege
for SYSTEM RELOAD MODELS exists but parser doesn't recognize it
nodes would fail to come up.
Considered but rejected alternatives:
- Ignore SYSTEM RELOAD MODELS during parsing RBAC privileges and
return an error for regular SYSTEM RELOAD MODELS SQL. Special-case
of no-op behavior, too brittle.
- Remove SYSTEM RELOAD MODELS manually from Keeper via command-line
manipulation of Keeper nodes or via SQL by dropping the privileges.
Needs user intervention during upgrade.
2022-09-08 09:10:11 +00:00
Robert Schulze
60f9f6855d
feat: implement catboost in library-bridge
...
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.
SQL syntax:
SELECT
catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
ACTION AS target
FROM amazon_train
LIMIT 10
Required configuration:
<catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>
*** Implementation Details ***
The internal protocol between the server and the library-bridge is
simple:
- HTTP GET on path "/extdict_ping":
A ping, used during the handshake to check if the library-bridge runs.
- HTTP POST on path "extdict_request"
(1) Send a "catboost_GetTreeCount" request from the server to the
bridge, containing a library path (e.g /home/user/libcatboost.so) and
a model path (e.g. /home/user/model.bin). Rirst, this unloads the
catboost library handler associated to the model path (if it was
loaded), then loads the catboost library handler associated to the
model path, then executes GetTreeCount() on the library handler and
finally sends the result back to the server. Step (1) is called once
by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
library path handler is unloaded in the beginning because it contains
state which may no longer be valid if the user runs
catboost("/path/to/model.bin", ...) more than once and if "model.bin"
was updated in between.
(2) Send "catboost_Evaluate" from the server to the bridge, containing
the model path and the features to run the interference on. Step (2)
is called multiple times (once per chunk) by the server from function
FunctionCatBoostEvaluate::executeImpl(). The library handler for the
given model path is expected to be already loaded by Step (1).
Fixes #27870
2022-09-08 09:01:32 +00:00
Vitaly Baranov
31ed722572
Merge pull request #41044 from vitlibar/more-conventional-conversion-yaml-to-xmk
...
More conventional conversion yaml to xml
2022-09-07 13:46:32 +02:00
Vitaly Baranov
9c847ceec9
No hardlinks while making backup of MergeTree in atomic database.
2022-09-07 11:44:50 +02:00
Vitaly Baranov
63e992d52d
Edit test configs.
2022-09-06 17:09:26 +02:00
Sergei Trifonov
6217559175
Merge branch 'master' into readonly-settings-allow
2022-09-06 16:10:48 +02:00
Antonio Andelic
5eb034a920
Merge branch 'master' into start-embedded-keeper-async
2022-09-06 12:14:47 +00:00
Antonio Andelic
4a68bfef39
Fix tests with async Keeper start
2022-09-06 12:12:17 +00:00
Sergei Trifonov
f77809ddbc
Merge pull request #40900 from ClickHouse/s3-detailed-metrics
...
S3 detailed metrics
2022-09-06 13:20:57 +02:00
Anton Ivashkin
e91d22a5b7
Merge branch 'fix-ordinary-system-unfreeze' of github.com:PolyProgrammist/ClickHouse into fix-ordinary-system-unfreeze
2022-09-06 12:18:05 +03:00
Anton Ivashkin
5e7cf1f238
Fix test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]
2022-09-06 12:14:40 +03:00
Alexey Milovidov
59bd94dcf8
Merge pull request #40975 from vitlibar/fix-access-rights-for-describe
...
Fix access rights for DESCRIBE TABLE
2022-09-06 02:38:34 +03:00
Alexey Milovidov
4c30dbc905
Merge pull request #40968 from ClickHouse/s3-sharding-2
...
Sharding s3 key names (2)
2022-09-05 22:47:25 +03:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze
2022-09-05 17:10:59 +03:00
Sergei Trifonov
1c247308f1
Merge branch 'master' into s3-detailed-metrics
2022-09-05 15:55:41 +02:00
Sergei Trifonov
225773634c
fix integration test for profile events s3
2022-09-05 15:54:28 +02:00
Kseniia Sumarokova
cff1dd6a8e
Merge branch 'master' into move-strange-test
2022-09-05 15:19:57 +02:00
Antonio Andelic
e07539d1b9
Merge branch 'master' into keeper-storage
2022-09-05 09:54:13 +00:00
Antonio Andelic
3a0581e990
Merge pull request #40543 from Lloyd-Pottiger/feat/support-read-only-for-embeddedrocksdb
...
Add read-only support for EmbeddedRocksDB
2022-09-05 09:31:07 +02:00
Alexey Milovidov
cda8052df0
Fix trash
2022-09-05 03:46:10 +02:00
Alexey Milovidov
e70d526896
Fix trash
2022-09-05 03:35:43 +02:00
Alexey Milovidov
bff96b0af6
Merge branch 'master' into fix-access-rights-for-describe
2022-09-05 04:34:57 +03:00
Alexey Milovidov
cc50aa39a2
Merge branch 's3-sharding-2' of github.com:ClickHouse/ClickHouse into s3-sharding-2
2022-09-05 02:25:01 +02:00
Alexey Milovidov
79e3723272
Merge branch 'master' into s3-sharding-2
2022-09-05 02:24:35 +02:00
robot-clickhouse
89d40b6f70
Automatic style fix
2022-09-05 00:23:36 +00:00
Alexey Milovidov
a0b5567d3e
Move strange test
2022-09-05 02:16:48 +02:00
Alexey Milovidov
007680d93f
Preparation
2022-09-05 02:13:51 +02:00
Vitaly Baranov
6c164905ca
Add test.
2022-09-04 18:57:51 +02:00
Alexey Milovidov
c4adc9ed8f
Remove trash
2022-09-04 04:28:08 +02:00
Alexey Milovidov
13a129bee7
Merge branch 'master' into remove-useless-line
2022-09-04 03:24:40 +02:00
Lloyd-Pottiger
ec68ed8a93
fix test
...
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
2022-09-03 10:44:07 +08:00
Sergei Trifonov
005b2588d8
Merge branch 'master' into readonly-settings-allow
2022-09-02 16:37:43 +02:00
Lloyd-Pottiger
b76af9284e
style check
...
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
2022-09-02 19:29:57 +08:00
Lloyd-Pottiger
f499381f57
move functional test to integration test
...
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
2022-09-02 18:45:37 +08:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage
2022-09-02 12:13:07 +02:00
Lloyd-Pottiger
5a6b2106b5
Merge branch 'master' of github.com:ClickHouse/ClickHouse into feat/support-read-only-for-embeddedrocksdb
2022-09-02 18:10:00 +08:00
robot-clickhouse
46f74fcbaf
Automatic style fix
2022-09-02 07:49:09 +00:00
Antonio Andelic
b02bf49939
Use path prefix
2022-09-02 07:34:26 +00:00
Lloyd-Pottiger
59dccd6e49
fix test
...
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
2022-09-02 11:14:49 +08:00
Sergei Trifonov
7fec55eea4
work in progress
2022-09-02 04:12:05 +02:00
Frank Chen
9d63cbe811
Merge 'origin/master' into tracing_context_propagation to resolve conflicts
2022-09-01 23:18:59 +08:00
Vitaly Baranov
6dcca686cb
Merge pull request #40827 from vitlibar/fix-incremental-backups-for-log-family
...
Fix incremental backups for Log family.
2022-09-01 13:13:32 +02:00
Sema Checherinda
7b59fdc042
Merge pull request #40779 from CheSema/detached-parts-metric
...
Metric for the number of detached parts
2022-09-01 12:24:42 +02:00
Vitaly Baranov
007ae0e6cc
Fix incremental backups for Log family.
2022-08-31 12:57:28 +02:00
Antonio Andelic
9dd1a9859d
Merge branch 'master' into keeper-storage
2022-08-31 10:12:11 +00:00
root
ea517de61a
style check correction
2022-08-30 23:20:19 -03:00
root
f02f7ce99c
addressed minor change requests
2022-08-30 23:20:19 -03:00
root
84a617675d
style check correction in test.py
2022-08-30 23:20:19 -03:00
root
3ced3c9933
style check correction in test.py
2022-08-30 23:20:19 -03:00
root
1d29a494db
Modified test code to cover various scenarios for custom keys
2022-08-30 23:20:19 -03:00
root
d5db88fa27
Structured-logging-custom-keys PR init
2022-08-30 23:20:19 -03:00
Sema Checherinda
9bb4f556c3
work on remarks on the review
2022-08-30 20:18:35 +02:00
Antonio Andelic
3c098bd0cb
Add integration test for KeeperMap
2022-08-30 13:41:57 +00:00
Alexander Tokmakov
022f440ad0
Merge pull request #40769 from arthurpassos/caresptrresolver-hosts-patch
...
Fix CaresPTRResolver not reading hosts file
2022-08-30 14:35:10 +03:00
alesapin
2588901bc9
Merge pull request #40780 from ClickHouse/add_cache_to_zero_copy_test
...
Add cache to zero copy replication test
2022-08-30 12:42:05 +02:00
Kseniia Sumarokova
c88db2ef97
Merge pull request #40751 from kssenii/fix-mysql-timeouts
...
Fix issue with mysql db / table function timeouts
2022-08-30 11:59:01 +02:00
Frank Chen
f17d56b528
Merge branch 'master' into tracing_context_propagation
2022-08-30 14:24:36 +08:00
Sema Checherinda
e436b4f4cc
Metric for the number of detached parts
2022-08-30 02:12:57 +02:00
alesapin
ac41de73df
Add cache to zero copy replication test
2022-08-30 00:17:01 +02:00
Kseniia Sumarokova
8001560082
Fix test
2022-08-30 00:14:00 +02:00
Vitaly Baranov
33f72fb011
Merge pull request #40060 from ClickHouse/vitlibar-increase-timeout-for-test_concurrent_backups
...
Increase timeout for test_concurrent_backups
2022-08-29 22:25:56 +02:00
Arthur Passos
dd49b44abb
Fix host_regexp hosts file tst
2022-08-29 15:58:18 -03:00
Arthur Passos
961365c7a4
Fix CaresPTRResolver not reading hosts file
2022-08-29 15:11:39 -03:00