Commit Graph

203 Commits

Author SHA1 Message Date
KinderRiven
9d83b93e88 fix rebase 2022-08-11 00:11:40 +08:00
kssenii
0ff5609738 Finish support for persistent files 2022-08-10 16:54:59 +02:00
avogar
1cfb5b7cf1 Fix build 2022-08-08 12:17:18 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
Vadim Volodin
85bf022258 review zookeeper and partition matching 2022-07-21 16:57:28 +03:00
Alexander Tokmakov
6e77155dc8 fix 2022-07-20 23:11:35 +02:00
kssenii
708b93a03b Merge master 2022-06-25 00:08:24 +02:00
kssenii
6fbd49f554 Merge master 2022-06-23 21:40:01 +02:00
kssenii
468c98ff42 Better 2022-06-23 17:46:27 +02:00
kssenii
dce1c76270 Fix 2022-06-22 18:00:25 +02:00
kssenii
90ce0b44f0 Fixes 2022-06-21 21:03:18 +02:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache
Extract some diff from pr #36171
2022-06-17 12:07:31 +02:00
kssenii
797482f132 Better 2022-06-16 16:40:29 +02:00
kssenii
416c6008ee Merge master 2022-06-15 16:50:25 +02:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.

__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage
Separate object storage operations from disks
2022-05-27 13:59:50 +02:00
Alexander Tokmakov
e8f33fb0d9 fix flaky tests 2022-05-26 14:17:05 +02:00
kssenii
5e3410f60a Better 2022-05-16 22:09:11 +02:00
kssenii
646cf4cc95 Better 2022-05-14 14:26:04 +02:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
alesapin
eba60ff38f Fix restorable schema 2022-05-13 17:00:47 +02:00
kssenii
18dd74be0c Remove unneede changes 2022-05-11 10:26:30 +02:00
Kseniia Sumarokova
e3d2d72b39
Merge pull request #36802 from KinderRiven/local_cache_trace
system table used for statistical query of remote file access
2022-05-02 00:16:04 +02:00
Nikita Mikhaylov
93a65463d0
Added SYSTEM SYNC DATABASE query (#35944) 2022-05-01 15:40:18 +02:00
KinderRiven
d595df1604 fix 2022-05-01 19:45:16 +08:00
KinderRiven
29f0471caf add cache log 2022-05-01 19:45:16 +08:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone"
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"

About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.

About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.

Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
2022-04-30 19:40:28 +02:00
Vitaly Baranov
641a5f5e35
Merge pull request #36198 from vitlibar/backup-improvements-3
Backup for replicated tables and other improvements
2022-04-28 09:23:53 +02:00
kssenii
a76a35d3ce Better drop cache 2022-04-25 19:20:25 +02:00
Vitaly Baranov
030f3e488c Add shard_index and replica_index to params of executeDDLQueryOnCluster(). 2022-04-25 16:34:33 +02:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
2022-04-08 12:06:26 +02:00
Alexander Tokmakov
7f54e7b422 Merge branch 'master' into mvcc_prototype 2022-04-07 15:14:06 +02:00
Azat Khuzhin
14538f6456 Add system.processors_profile_log
This is the system table that will contain Processors level profiling.

v2: one entry per Processor, not 3 (PortFull/NeedData/work())
v3: us over ms
v4: Enable processors_profile_log table by default

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-04-04 13:19:41 +03:00
kssenii
0fc92fe2aa Get rid of all "remote" mentions 2022-03-30 17:15:29 +02:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Alexander Tokmakov
fb9b2d5326 Merge branch 'master' into mvcc_prototype 2022-01-28 21:18:36 +03:00
Vitaly Baranov
cc38fe3fb6 Fix checking access for the SYSTEM command. 2022-01-25 18:54:59 +07:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Alexander Tokmakov
e1a41fc694 add system log for introspection 2022-01-14 17:03:00 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
vdimir
94bb7cba62
cluster discovery wip 2021-12-27 15:26:05 +03:00
Stig Bakken
b91f21d658 Don't use setInternalQuery() when forwarding drop calls to database 2021-11-18 11:46:51 +01:00
Stig Bakken
ff46e8bb51 Drop support for DatabaseOrdinary in MaterializedMySQL
1. Dropped support for DatabaseOrdinary for MaterializeMySQL. It
   is marked as experimental, and dropping support makes the code
   more maintaible, and speeds up integration tests by 50%.

2. Get rid of thread name logic for StorageMaterializeMySQL wrapping,
   use setInternalQuery instead (similar to MaterializedPostgreSQL).
2021-11-18 11:46:51 +01:00
Nikolay Degterinsky
0d58c5231f Better 2021-11-11 13:28:18 +00:00
Nikolay Degterinsky
7b5954b736 Merge branch 'master' into query_parameters 2021-11-11 11:43:00 +00:00
Vitaly Baranov
3ed7f8f0b3 Move access-rights' source files needed for parser to a separate target. 2021-11-01 19:13:49 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
Nikolay Degterinsky
d76976e272 Merge branch 'master' into query_parameters 2021-10-18 17:34:20 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
Nikolay Degterinsky
2b7e1e2949 Add parmeters to SYSTEM queries 2021-10-12 21:11:00 +03:00
tavplubix
8adfb9b593
Stop ThreadFuzzer before hung check (#29167)
* stop ThreadFuzzer before hung check

* fix

* fix
2021-09-20 17:23:10 +03:00
Maksim Kita
85a4d4bb50 Added user defined executable functions to system.functions 2021-09-17 18:43:00 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum
Improving CH type system with concepts
2021-09-12 15:26:29 +03:00
mergify[bot]
c02953b59a
Merge branch 'master' into reloadsymbolcache 2021-09-08 23:25:53 +00:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
Amos Bird
163a416366
Refactor 2021-09-02 19:29:14 +08:00
Vasily Nemkov
c902afddde Added system.session_log table
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
2021-08-30 18:28:28 +03:00
Amos Bird
a2256b1307
Dedicated Mark/Uncompressed cache for skip indices 2021-08-23 15:27:31 +08:00
Raúl Marín
4a09c54a04 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-07-28 10:38:47 +02:00
Raúl Marín
efe1938437 WIP: Slow improvements 2021-07-27 16:26:29 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00
Raúl Marín
bc1ccd2d62 Flush materialization log on FLUSH_LOGS 2021-07-27 16:23:46 +02:00
mergify[bot]
2929cf3ba8
Merge branch 'master' into system_zookeeper_log 2021-07-26 08:40:55 +00:00
Alexey Milovidov
f62fc8c43d Fix another cause of flakiness 2021-07-21 04:05:56 +03:00
Alexander Tokmakov
4165ba2a01 add system.zookeeper_log 2021-07-09 17:05:35 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652)
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
2021-06-20 11:24:43 +03:00
tavplubix
e9ff0b6d70
Merge pull request #23657 from kssenii/poco-file-to-std-fs
Poco::File to std::filesystem
2021-05-31 23:17:02 +03:00
Nikolai Kochetov
afc1fe7f3d Make ContextPtr const by default. 2021-05-31 17:49:02 +03:00
kssenii
3dee003f9b Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs 2021-05-20 19:20:09 +03:00
Azat Khuzhin
23981ffd0e Introduce IStorage::flushAndShutdown() 2021-05-16 13:23:53 +03:00
kssenii
ab1a05a1f4 Poco::Path to fs::path, less concatination 2021-05-09 14:59:49 +03:00
Pavel Kovalenko
0050dbb75b Apply disk settings on config reload. 2021-04-29 23:32:19 +03:00
Pavel Kovalenko
10cf881a08 Add SYSTEM RESTART DISK command. 2021-04-20 22:23:54 +03:00
Maksim Kita
abf2a87ce7 Added system query reload model 2021-04-17 20:59:08 +03:00
Alexander Tokmakov
ebf0a3119e fix data_type_default_nullable on attach 2021-04-16 13:29:33 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297)
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
Maksim Kita
8be263062d
Merge pull request #21910 from kitaisreal/table-function-dictionary
Added table function dictionary
2021-03-22 16:08:56 +03:00
Maksim Kita
849a980644 Updated ExternalDictionariesLoader interface with context parameter 2021-03-20 18:02:09 +03:00
Maksim Kita
32831e37ba Added table function dictionary 2021-03-19 15:47:27 +03:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Vasily Nemkov
6af4d06dff Added Grant, Revoke and System query_kind for system.query_log
Also some tests to ensure that all queries are logged
2021-03-02 12:27:39 +02:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Azat Khuzhin
809fa7e4cc Sync SYSTEM FLUSH DISTRIBUTED with TRUNCATE 2021-02-10 23:10:37 +03:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
Azat Khuzhin
39cfffd4d2 Add log message for list of tables on RESTART REPLICAS 2021-01-10 12:44:56 +03:00
alexey-milovidov
98bd7fb653
Merge branch 'master' into better-kill 2021-01-08 07:27:00 +03:00
Alexey Milovidov
9cbb9c6d7c Add SYSTEM SUSPEND command #15979, continued 2021-01-07 22:22:41 +03:00