Commit Graph

277 Commits

Author SHA1 Message Date
alesapin
eba60ff38f Fix restorable schema 2022-05-13 17:00:47 +02:00
kssenii
18dd74be0c Remove unneede changes 2022-05-11 10:26:30 +02:00
Kseniia Sumarokova
e3d2d72b39
Merge pull request #36802 from KinderRiven/local_cache_trace
system table used for statistical query of remote file access
2022-05-02 00:16:04 +02:00
Nikita Mikhaylov
93a65463d0
Added SYSTEM SYNC DATABASE query (#35944) 2022-05-01 15:40:18 +02:00
KinderRiven
d595df1604 fix 2022-05-01 19:45:16 +08:00
KinderRiven
29f0471caf add cache log 2022-05-01 19:45:16 +08:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone"
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"

About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.

About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.

Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
2022-04-30 19:40:28 +02:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3
Backup for replicated tables and other improvements
2022-04-28 09:23:53 +02:00
kssenii
a76a35d3ce Better drop cache 2022-04-25 19:20:25 +02:00
Vitaly Baranov
030f3e488c Add shard_index and replica_index to params of executeDDLQueryOnCluster(). 2022-04-25 16:34:33 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
2022-04-08 12:06:26 +02:00
Alexander Tokmakov
7f54e7b422 Merge branch 'master' into mvcc_prototype 2022-04-07 15:14:06 +02:00
Azat Khuzhin
14538f6456 Add system.processors_profile_log
This is the system table that will contain Processors level profiling.

v2: one entry per Processor, not 3 (PortFull/NeedData/work())
v3: us over ms
v4: Enable processors_profile_log table by default

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-04-04 13:19:41 +03:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Vitaly Baranov
cc38fe3fb6 Fix checking access for the SYSTEM command. 2022-01-25 18:54:59 +07:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Alexander Tokmakov
e1a41fc694 add system log for introspection 2022-01-14 17:03:00 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
vdimir
94bb7cba62
cluster discovery wip 2021-12-27 15:26:05 +03:00
Stig Bakken
b91f21d658 Don't use setInternalQuery() when forwarding drop calls to database 2021-11-18 11:46:51 +01:00
Stig Bakken
ff46e8bb51 Drop support for DatabaseOrdinary in MaterializedMySQL
1. Dropped support for DatabaseOrdinary for MaterializeMySQL. It
   is marked as experimental, and dropping support makes the code
   more maintaible, and speeds up integration tests by 50%.

2. Get rid of thread name logic for StorageMaterializeMySQL wrapping,
   use setInternalQuery instead (similar to MaterializedPostgreSQL).
2021-11-18 11:46:51 +01:00
Nikolay Degterinsky
0d58c5231f Better 2021-11-11 13:28:18 +00:00
Nikolay Degterinsky
7b5954b736 Merge branch 'master' into query_parameters 2021-11-11 11:43:00 +00:00
Vitaly Baranov
3ed7f8f0b3 Move access-rights' source files needed for parser to a separate target. 2021-11-01 19:13:49 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
Nikolay Degterinsky
d76976e272 Merge branch 'master' into query_parameters 2021-10-18 17:34:20 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
Nikolay Degterinsky
2b7e1e2949 Add parmeters to SYSTEM queries 2021-10-12 21:11:00 +03:00
tavplubix
8adfb9b593
Stop ThreadFuzzer before hung check (#29167)
* stop ThreadFuzzer before hung check

* fix

* fix
2021-09-20 17:23:10 +03:00
Maksim Kita
85a4d4bb50 Added user defined executable functions to system.functions 2021-09-17 18:43:00 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum
Improving CH type system with concepts
2021-09-12 15:26:29 +03:00
mergify[bot]
c02953b59a
Merge branch 'master' into reloadsymbolcache 2021-09-08 23:25:53 +00:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
Amos Bird
163a416366
Refactor 2021-09-02 19:29:14 +08:00
Vasily Nemkov
c902afddde Added system.session_log table
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
2021-08-30 18:28:28 +03:00
Amos Bird
a2256b1307
Dedicated Mark/Uncompressed cache for skip indices 2021-08-23 15:27:31 +08:00
Raúl Marín
4a09c54a04 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-07-28 10:38:47 +02:00
Raúl Marín
efe1938437 WIP: Slow improvements 2021-07-27 16:26:29 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00
Raúl Marín
bc1ccd2d62 Flush materialization log on FLUSH_LOGS 2021-07-27 16:23:46 +02:00
mergify[bot]
2929cf3ba8
Merge branch 'master' into system_zookeeper_log 2021-07-26 08:40:55 +00:00
Alexey Milovidov
f62fc8c43d Fix another cause of flakiness 2021-07-21 04:05:56 +03:00
Alexander Tokmakov
4165ba2a01 add system.zookeeper_log 2021-07-09 17:05:35 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652)
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Azat Khuzhin
23981ffd0e Introduce IStorage::flushAndShutdown() 2021-05-16 13:23:53 +03:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
Pavel Kovalenko
0050dbb75b Apply disk settings on config reload. 2021-04-29 23:32:19 +03:00
Pavel Kovalenko
10cf881a08 Add SYSTEM RESTART DISK command. 2021-04-20 22:23:54 +03:00
Maksim Kita
abf2a87ce7 Added system query reload model 2021-04-17 20:59:08 +03:00
Alexander Tokmakov
ebf0a3119e fix data_type_default_nullable on attach 2021-04-16 13:29:33 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297)
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Maksim Kita
8be263062d
Merge pull request #21910 from kitaisreal/table-function-dictionary
Added table function dictionary
2021-03-22 16:08:56 +03:00
Maksim Kita
849a980644 Updated ExternalDictionariesLoader interface with context parameter 2021-03-20 18:02:09 +03:00
Maksim Kita
32831e37ba Added table function dictionary 2021-03-19 15:47:27 +03:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Vasily Nemkov
6af4d06dff Added Grant, Revoke and System query_kind for system.query_log
Also some tests to ensure that all queries are logged
2021-03-02 12:27:39 +02:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Azat Khuzhin
809fa7e4cc Sync SYSTEM FLUSH DISTRIBUTED with TRUNCATE 2021-02-10 23:10:37 +03:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
Azat Khuzhin
39cfffd4d2 Add log message for list of tables on RESTART REPLICAS 2021-01-10 12:44:56 +03:00
alexey-milovidov
98bd7fb653
Merge branch 'master' into better-kill 2021-01-08 07:27:00 +03:00
Alexey Milovidov
9cbb9c6d7c Add SYSTEM SUSPEND command #15979, continued 2021-01-07 22:22:41 +03:00
Alexey Milovidov
ffa5bbd5fd Add SYSTEM SUSPEND command #15979 2021-01-07 22:19:33 +03:00
Alexey Milovidov
32d8785a26 Better kill #18847 2021-01-07 21:36:38 +03:00
alexey-milovidov
a7324f8b4c
Update InterpreterSystemQuery.cpp 2020-12-08 00:12:09 +03:00
Amos Bird
08675028e5
fix build 2020-12-08 01:40:49 +08:00
Amos Bird
310918b06a
reload symbols and fix build-id 2020-11-30 22:30:55 +08:00
Alexander Tokmakov
0731bfb7be Merge branch 'master' into database_replicated 2020-11-12 21:04:18 +03:00
Alexander Tokmakov
cbcdee0cf9 split DDLWorker.cpp 2020-11-03 16:47:26 +03:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
Alexander Kuzmenkov
5cbf645de4 Merge remote-tracking branch 'origin/master' into tmp 2020-10-22 19:48:02 +03:00
Alexander Kuzmenkov
145e2b012f cleanup 2020-10-22 19:47:20 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956)
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-10-20 18:10:24 +03:00
alesapin
2b2a55a081 Add action lock only if table exists 2020-10-15 22:23:46 +03:00
alesapin
4c1f5cab13 Sleep less in background task 2020-10-15 19:10:22 +03:00
alexey-milovidov
7b53a0ef33 Revert "Added allow_merges option for volumes in multi-disk configuration (#13402)"
This reverts commit 1e2616542a.
2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration (#13402) 2020-08-21 12:04:13 +03:00
Alexander Kuzmenkov
05ad9b9fff opentelemetry wip 2020-08-20 23:59:58 +03:00
Alexander Tokmakov
f6949b2f47 use dictionary ids 2020-07-15 22:25:31 +03:00
Alexander Tokmakov
445acd0294 Merge branch 'master' into database_atomic_improvements 2020-07-09 16:02:54 +03:00
Alexander Tokmakov
7a5d532c7b implement rename database for atomic 2020-07-07 15:11:58 +03:00
Vitaly Baranov
c39eb8f71b Fix partial revokes (complex cases). 2020-06-30 18:47:02 +03:00
Alexander Tokmakov
7b5de16da9 minor improvements 2020-06-23 22:15:01 +03:00
amudong
2a51286527 Optimize code 2020-06-23 16:22:53 +08:00
amudong
0101947403 rewrite static StorageReplicatedMergeTree::dropReplica 2020-06-23 12:12:30 +08:00
amudong
7723dc4935 add drop relica database and the whole replica
fix removeReplicaByZKPath

fix bug: add zkpath empty judge

fix: rewrite code

delete useless code.

fix:ast fromat

fix bug

add test_drop_replica

add drop_replica doc

add drop databse checkAccess

refactor dropReplica

update tests

add static method StorageReplicatedMergeTree::dropReplicaByZkPath

update doc and delete useless code

fix conflict

fix doc

fix doc

fix StorageReplicatedMergeTree::dropReplica

fix bug

delete useless code
2020-06-23 12:12:30 +08:00
amudong
2e6a3eff8e fix: can not drop local table and replace ALTER with SYSTEM
fix test_drop_replica

fix drop replica '/path/to/zk/' ending in '/' and update doc
2020-06-23 12:12:30 +08:00
sundy-li
0a4af8f0a7 add SYSTEM DROP REPLICA 2020-06-23 12:12:30 +08:00
Azat Khuzhin
103eb17107 Force table creation on SYSTEM FLUSH LOGS (v2)
After this patch SYSTEM FLUSH LOGS will force system.*_log table
creatoins (only for enabled tables of course).

This will avoid such hacks like:

   set log_queries=1;
   select 1;
   system flush logs;

This is the second version of the patch, since first has a deadlock [1]
(reported by @alesapin). This version does not use separate lock and do
not issue CREATE TABLE from the query execution thread, instead it
activate flushing thread to do so, hence this should be the same as if
the table will be created during flush.

  [1]: https://gist.github.com/alesapin/d915d1deaa27d49aa31223daded02be2

Fixes: #11563
Cc: @alesapin
Cc: @nikitamikhaylov
2020-06-11 23:12:23 +03:00
Alexander Kuzmenkov
e4be52f35f Add system.asynchronous_metric_log 2020-06-10 22:17:54 +03:00
Alexander Tokmakov
0cf8015f4a use DNSResolver to check if user is allowed to connect 2020-06-07 00:02:29 +03:00
Alexey Milovidov
a6bf4b5d70 Fix error with Lazy database 2020-06-02 05:06:16 +03:00
Alexey Milovidov
bdfa552844 Fix nullptr dereference in SYSTEM query and Lazy database 2020-06-02 04:53:32 +03:00
Alexander Tokmakov
c071e0a473 add context to DatabaseCatalog methods 2020-05-29 02:01:18 +03:00
Alexander Tokmakov
3cff070f89 add context to IDatabase methods 2020-05-28 23:10:45 +03:00
Alexey Milovidov
7e1813825b Return old names of macros 2020-05-24 01:24:01 +03:00
Alexey Milovidov
241f8c5431 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" \+ [^+]+ \+ "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" \+ ([^+]+) \+ "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/' 2020-05-23 20:10:21 +03:00
Alexey Milovidov
8d2e80a5e2 find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+"\)' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+, "[^"]+")\)/\1_FORMATTED(\2)/' 2020-05-23 19:42:39 +03:00
Alexey Milovidov
99c18c5a09 Check for suspicious codecs #4966 2020-05-04 03:11:49 +03:00
Alexander Tokmakov
9ca144cab9 drop access cache on DROP DNS CACHE 2020-05-01 01:29:47 +03:00
Alexander Tokmakov
f413061e0d remove Context from some IDatabase methods 2020-04-23 19:51:48 +03:00
Alexander Tokmakov
04d6b59ac0 Merge branch 'master' into database_atomic 2020-04-23 17:31:37 +03:00
Alexey Milovidov
0a7edce036 Checkpoint 2020-04-22 09:01:33 +03:00
Alexander Tokmakov
b29bddac12 Merge branch 'master' into database_atomic 2020-04-20 14:09:09 +03:00
Ivan Lezhankin
e230632645 Changes required for auto-sync with Arcadia 2020-04-16 15:31:57 +03:00
Alexander Tokmakov
5e6d4b9449 Merge branch 'master' into database_atomic 2020-04-12 16:35:44 +03:00
Alexander Tokmakov
f0edb65e33 add test 2020-04-11 02:02:15 +03:00
Alexander Kazakov
497df3086f Merge branch 'master' into timed_rwlock
Change-Id: I620bfde2121ff013773b001d514b40b1e796a58b
2020-04-10 11:38:20 +03:00
Alexander Kazakov
26dd6140b2 Added new config settings to control timeouts
* "lock_acquire_timeout" controls for how long a query will continue to
acquire each lock on its argument tables
 * "lock_acquire_timeout_for_background_operations" is a per-table
setting for storages of *MergeTree family
2020-04-09 21:10:27 +03:00
Alexander Tokmakov
dd1590830b Merge branch 'master' into database_atomic 2020-04-08 22:00:46 +03:00
Vitaly Baranov
b4d7ef390c Rename some access types: add ALTER and SYSTEM prefixes. 2020-04-07 23:20:38 +03:00
Alexander Tokmakov
a2101c9b1c better code 2020-04-07 02:22:44 +03:00
Alexander Tokmakov
08bae4668d Merge branch 'master' into database_atomic 2020-04-06 16:18:07 +03:00
Ivan Lezhankin
06446b4f08 dbms/ → src/ 2020-04-03 18:14:31 +03:00