In case you have different roles for the same user on multiple clusters,
ON CLUSTER query can help to overcome some limitations.
Consider the following example:
- cluster_with_data, dev_user (readonly=2)
- stage_cluster, dev_user (readonly=0)
So when you will execute the following query from stage_cluster, it will
be successfully executed, since ON CLUSTER queries has different system
profile:
DROP DATABASE default ON CLUSTER cluster_with_data
This is not 100% safe, but at least something.
Note, that right now only ON CLUSTER query it self is supported, but
separate clusters are not (i.e. GRANT CLUSTER some_cluster_name TO
default), since right now grants sticked to database+.
v2: on_cluster_queries_require_cluster_grant
v3: fix test and process flags as bit mask
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This is the system table that will contain Processors level profiling.
v2: one entry per Processor, not 3 (PortFull/NeedData/work())
v3: us over ms
v4: Enable processors_profile_log table by default
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.
Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL
Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
* Add a codec Encrypted() for encrypting columns on disk
While this is implemented as a compression codec, it does not actually compress data. It instead encrypts data on disk. The key is obtained by executing a user-specified command at the server startup, or if it's not specified the codec refuses to process any data. For now the only supported cipher is 'AES-128-GCM-SIV'.
* master: (694 commits)
Fix integration test test_storage_kafka failed error
Fix test 00163_column_oriented_formats failed error
Read ORC file by stripe to reduce memory cost
Function toDateTime decimal overflow ubsan fix
Revert "[RFC] Fix memory tracking with min_bytes_to_use_mmap_io"
Zlib use attribute constructor for functable initialization
Translate to Russian (clickhouse-client documentation)
Simple key dictionary primary key wrong order fix
Disable hedged requests
Added integration test
Revert "Function `arrayFold` for folding over array with accumulator"
Fix documentation for the GRANT command.
Added system query reload model
Make function `unhex` case insensitive for compatibility
Improve documentation for CREATE ROW POLICY command #2.
Add exception message
Fix tidy
Fix waiting for all connections closed on shutdown.
Disable postgresql_port in perf tests
Mark 01605_adaptive_granularity_block_borders as long
...
* master: (860 commits)
Update version_date.tsv after release 21.2.8.31
Update version_date.tsv after release 21.3.5.42
Fixed typos
Add metric to track how much time is spend during waiting for Buffer layer lock
Safer SCOPE_EXIT
Add SCOPE_EXIT_SAFE/SCOPE_EXIT_MEMORY_SAFE helpers
Lock MEMORY_LIMIT_EXCEEDED in ThreadStatus::detachQuery()
Update CHANGELOG.md
Reset timeouts to default
Add Third party service info
Disable table function view in expression (#21465)
fix test 01702_system_query_log
Remove strange fsync on coordination logs rotation
add test
MemoryStorage sync comments and code
Fix typos
Support alter setting
Handle not plain where tree in StorageMerge modifySelect
Updated test
Change Aggregatingmergetree to AggregatingMergeTree in docs
...
Refactor some config parsing code
Rename some arguments to better reflect their meaning
Add documentation for user_dn_detection section and user_dn placeholder in config.xml and in docs
If you push data via Buffer engine then all your queries will be done
from one user, however this is not always desired behavior, since this
will not allow to limit queries with max_concurrent_queries_for_user and
similar.
* master: (605 commits)
DOCSUP-4710: Added support numeric parameters in number and string data types (#18696)
DOCSUP-5604: Edit and translate to Russian (#18929)
Update version_date.tsv after release 21.1.2.15
Usability improvement of clickhouse-test
Update jit_large_requests.xml
Update README.md
Update images.json
Make symbolizers available in fuzzer Docker image
Update Dragonbox
Speed up aggregate function sum
Fix MSan report in Kerberos library
Fix MSan error in rocksdb #19213
Add more Fuzzer tasks
Fixes
Update comment for curl dependency for aws
Disable curl for mariadb-connector-c (it is not required)
Fix TSan
Skip test for ANTLR
DistributedBlockOutputStream: add more comments
DistributedBlockOutputStream: Remove superfluous brackets for string construction
...
* master: (620 commits)
Add test for some possible ambiguities in syntax
Update PushingToViewsBlockOutputStream.h
[For #18707] MySQL compatibility: support DIV and MOD operators
Mark another flaky test
Remove some headers
Mark some TestFlows as flaky
Fix error
Fix errors
One more test
Arcadia does not support distributed queries
Add a test for #14974
Added a test from #15641
More robust stateful test
Update tests
Remove bad code in HashJoin
Update test
Don't allow conversion between UUID and numeric types
Remove pink screen with confusing questions about Kerberos
Do not throw from Parser
Fix the unexpected behaviour of show tables when antlr parser enabled (#18431)
...
# Conflicts:
# programs/server/config.xml
# src/Access/Authentication.cpp
# src/Access/Authentication.h
v2: Add a note that top_level_domains_lists aren not applied w/o restart
v3: Remove ExtractFirstSignificantSubdomain{Default,Custom}Lookup.h headers
v4: TLDListsHolder: remove FIXME for dense_hash_map (this is not significant)
* master: (207 commits)
Update RadixSort.h
rerun tests to be sure
Update date_time_short perf test for toUnixTimestamp(Date())
update test
remove comments
better
fix tests
style
update copy pasted test
better
comments
better merge
new interface for the function
better
Fix comments
Add missing file
Make the code less bad
initial
test added
style
...
* master: (70 commits)
Update documentation-issue.md
Add an option to use existing tables to perf.py
DOCSUP-4280: Update the SELECT query (#17231)
DOCSUP-3584 edit and translate (#17176)
Fixed flaky test_storage_s3::test_custom_auth_headers
Update 01560_merge_distributed_join.sql
Minor improvements
Slightly more correct
Auto version update to [20.13.1.1] [54444]
Auto version update to [20.12.1.5236] [54443]
Update roadmap
Add favicon; add loading indicator
Fix race condition; history and sharing capabilities
Update bitmap-functions.md
Fix exception message
Use default value for read-only flag in metadata for Disk3.
ISSUES-16605 try fix review comment
trigger CI
ISSUES-16605 try fix integration failure
ISSUES-16605 try fix integration test failure
...
* master: (50 commits)
Update documentation-issue.md
Add an option to use existing tables to perf.py
DOCSUP-4280: Update the SELECT query (#17231)
DOCSUP-3584 edit and translate (#17176)
Fixed flaky test_storage_s3::test_custom_auth_headers
Update 01560_merge_distributed_join.sql
Minor improvements
Slightly more correct
Auto version update to [20.13.1.1] [54444]
Auto version update to [20.12.1.5236] [54443]
Update roadmap
Add favicon; add loading indicator
Fix race condition; history and sharing capabilities
Update bitmap-functions.md
Fix exception message
Use default value for read-only flag in metadata for Disk3.
ISSUES-16605 try fix review comment
trigger CI
ISSUES-16605 try fix integration failure
ISSUES-16605 try fix integration test failure
...
* master: (375 commits)
Update type-conversion-functions.md
Update maxmap.md
Update maxmap.md
Update maxmap.md
Update single_fixed_string_groupby.xml
Alter remove column properties and TTLs (#14742)
better fixed string group by support
Fix incorrect key condition of fixed strings.
constant output order
more tests for #14646
Maybe fix MSan report in base64
Proper exception message for wrong number of arguments of CAST
Added a test
Fix buffer overflow in "bar" function
Update convertMySQLDataType.cpp
Fix clang-tidy
Remove obsolete code from performance test
Slightly better code
Even more
Even more
...
# Conflicts:
# src/Interpreters/Context.cpp
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:
<remote_servers>
<logs>
<shard>
<secret>foobar</secret> <!-- empty -- works as before -->
...
</shard>
</logs>
</remote_servers>
And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).
v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
Add distributed_ddl.pool_size to control maximum parallel to handle
distributed DDL.
Also:
- convert Exception constructors to fmt-like
- use sleepFor* over std::this_thread::sleep_for()
This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).
This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.
