This stores Access Entities in ZooKeeper and replicates them across an entire cluster.
This can be enabled by using the following configuration :
<user_directories>
<replicated>
<zookeeper_path>/clickhouse/access/</zookeeper_path>
</replicated>
</user_directories>
Add an ability to reset custom setting to default and remove it from table's metadata.
This will allow to rollback the change without knowing the system/config's default.
Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>
* initial commit: add setting and stub
* typo
* added test stub
* fix
* wip merging new integration test and code proto
* adding steps interpreters
* adding firstly proposed solution (moving parts etc)
* added checking zookeeper path existence
* fixing the include
* fixing and sorting includes
* fixing outdated struct
* fix the name
* added ast ptr as level of indirection
* fix ref
* updating the changes
* working on test stub
* fix iterator -> reference
* revert rocksdb submodule update
* fixed show privileges test
* updated the test stub
* replaced rand() with thread_local_rng(), updated the tests
updated the test
fixed test config path
test fix
removed error messages
fixed the test
updated the test
fixed string literal
fixed literal
typo: =
* fixed the empty replica error message
* updated the test and the code with logs
* updated the possible test cases, updated
* added the code/test milestone comments
* updated the test (added more testcases)
* replaced native assert with CH one
* individual replicas recursive delete fix
* updated the AS db.name AST
* two small logging fixes
* manually generated AST fixes
* Updated the test, added the possible algo change
* Some thoughts about optimizing the solution:
ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH
* fix
* Removed the replica sync in test as it's invalid
* Some test tweaks
* tmp
* Rewrote the algo by using the executeQuery instead of
hand-crafting the ASTPtr.
Two questions still active.
* tr: logging active parts
* Extracted the parts moving algo into a separate helper function
* Fixed the test data and the queries slightly
* Replaced query to system.parts to direct invocation,
started building the test that breaks on various parts.
* Added the case for tables when at least one replica is alive
* Updated the test to test replicas restoration by detaching/attaching
* Altered the test to check restoration without replica restart
* Added the tables swap in the start if the server failed last time
* Hotfix when only /replicas/replica... path was deleted
* Restore ZK paths while creating a replicated MergeTree table
* Updated the docs, fixed the algo for individual replicas restoration case
* Initial parts table storage fix, tests sync fix
* Reverted individual replica restoration to general algo
* Slightly optimised getDataParts
* Trying another solution with parts detaching
* Rewrote algo without any steps, added ON CLUSTER support
* Attaching parts from other replica on restoration
* Getting part checksums from ZK
* Removed ON CLUSTER, finished working solution
* Multiple small changes after review
* Fixing parallel test
* Supporting rewritten form on cluster
* Test fix
* Moar logging
* Using source replica as checksum provider
* improve test, remove some code from parser
* Trying solution with move to detached + forget
* Moving all parts (not only Committed) to detached
* Edited docs for RESTORE REPLICA
* Re-merging
* minor fixes
Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
TODO (suggested by Nikolai)
1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies
3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)
Also need to figure out how prewhere works for projections, and
row_filter_policies.
wip
* master: (694 commits)
Fix integration test test_storage_kafka failed error
Fix test 00163_column_oriented_formats failed error
Read ORC file by stripe to reduce memory cost
Function toDateTime decimal overflow ubsan fix
Revert "[RFC] Fix memory tracking with min_bytes_to_use_mmap_io"
Zlib use attribute constructor for functable initialization
Translate to Russian (clickhouse-client documentation)
Simple key dictionary primary key wrong order fix
Disable hedged requests
Added integration test
Revert "Function `arrayFold` for folding over array with accumulator"
Fix documentation for the GRANT command.
Added system query reload model
Make function `unhex` case insensitive for compatibility
Improve documentation for CREATE ROW POLICY command #2.
Add exception message
Fix tidy
Fix waiting for all connections closed on shutdown.
Disable postgresql_port in perf tests
Mark 01605_adaptive_granularity_block_borders as long
...
* master: (860 commits)
Update version_date.tsv after release 21.2.8.31
Update version_date.tsv after release 21.3.5.42
Fixed typos
Add metric to track how much time is spend during waiting for Buffer layer lock
Safer SCOPE_EXIT
Add SCOPE_EXIT_SAFE/SCOPE_EXIT_MEMORY_SAFE helpers
Lock MEMORY_LIMIT_EXCEEDED in ThreadStatus::detachQuery()
Update CHANGELOG.md
Reset timeouts to default
Add Third party service info
Disable table function view in expression (#21465)
fix test 01702_system_query_log
Remove strange fsync on coordination logs rotation
add test
MemoryStorage sync comments and code
Fix typos
Support alter setting
Handle not plain where tree in StorageMerge modifySelect
Updated test
Change Aggregatingmergetree to AggregatingMergeTree in docs
...
Refactor some config parsing code
Rename some arguments to better reflect their meaning
Add documentation for user_dn_detection section and user_dn placeholder in config.xml and in docs
* master: (605 commits)
DOCSUP-4710: Added support numeric parameters in number and string data types (#18696)
DOCSUP-5604: Edit and translate to Russian (#18929)
Update version_date.tsv after release 21.1.2.15
Usability improvement of clickhouse-test
Update jit_large_requests.xml
Update README.md
Update images.json
Make symbolizers available in fuzzer Docker image
Update Dragonbox
Speed up aggregate function sum
Fix MSan report in Kerberos library
Fix MSan error in rocksdb #19213
Add more Fuzzer tasks
Fixes
Update comment for curl dependency for aws
Disable curl for mariadb-connector-c (it is not required)
Fix TSan
Skip test for ANTLR
DistributedBlockOutputStream: add more comments
DistributedBlockOutputStream: Remove superfluous brackets for string construction
...
* master: (620 commits)
Add test for some possible ambiguities in syntax
Update PushingToViewsBlockOutputStream.h
[For #18707] MySQL compatibility: support DIV and MOD operators
Mark another flaky test
Remove some headers
Mark some TestFlows as flaky
Fix error
Fix errors
One more test
Arcadia does not support distributed queries
Add a test for #14974
Added a test from #15641
More robust stateful test
Update tests
Remove bad code in HashJoin
Update test
Don't allow conversion between UUID and numeric types
Remove pink screen with confusing questions about Kerberos
Do not throw from Parser
Fix the unexpected behaviour of show tables when antlr parser enabled (#18431)
...
# Conflicts:
# programs/server/config.xml
# src/Access/Authentication.cpp
# src/Access/Authentication.h
* master: (159 commits)
Review fix.
Update version_date.tsv after release 20.8.7.15
wrong translation
Update version_date.tsv after release 20.9.6.14
Update version_date.tsv after release 20.10.5.10
Update version_date.tsv after release 20.11.4.13
Improvements in coverage images
Fixed a problem with the translation of the document
final_parallel
final_parallel
DOCSUP-4162: Document the system.replicated_fetches system table (#16900)
Update settings.md
Update settings.md
Less verbose logging when fetch is impossible
Don't add tons of client coverage files in stateful tests with coverage
More compatible watches in TestKeeper
Trying to make read_in_order_many_parts more stable
trigger CI
Update version_date.tsv after release 20.6.10.2
Update visibleWidth.cpp
...
* master: (224 commits)
Update in.md
Add a test for embedded configs
Allow to run without /proc/self/maps
Update adopters.md
More convenient
Enable embedded configs for AArch64
More results
More handy
More handy
Added a comment
Adjustments
Adjustments
Adjustments
Added new results
Step 1: make adding hardware benchmark results more convenient
Revert "Avoid deadlocks in Log/TinyLog"
Fix MSan report in QueryLog
add some disabled tests from arcadia to skip_list.json
Update skip_list.json
Bump CI. [2]
...
# Conflicts:
# docker/test/testflows/runner/Dockerfile
* master: (375 commits)
Update type-conversion-functions.md
Update maxmap.md
Update maxmap.md
Update maxmap.md
Update single_fixed_string_groupby.xml
Alter remove column properties and TTLs (#14742)
better fixed string group by support
Fix incorrect key condition of fixed strings.
constant output order
more tests for #14646
Maybe fix MSan report in base64
Proper exception message for wrong number of arguments of CAST
Added a test
Fix buffer overflow in "bar" function
Update convertMySQLDataType.cpp
Fix clang-tidy
Remove obsolete code from performance test
Slightly better code
Even more
Even more
...
# Conflicts:
# src/Interpreters/Context.cpp
* master: (414 commits)
Add .reference
Update 01460_mark_inclusion_search_crash.sql
Fix bug in mark inclusion search.
cosmetic fixes for performance report
cosmetic fixes in performance report
Add more docs about functional tests
Add ability to specify Default codec for columns (#14049)
better
done
Disable force TTL on optimise
Update CMakeLists.txt
Update custom parts of storage AST only if it has extended definition
DOCSUP-712: Documented the ttl_only_drop_parts setting (#13823)
Small fixes
gix comment and useDefault*(), add tests for nullables
Fix a build for old some OS with old find
Update hdfs.md
Fix 01085_max_distributed_connections flackiness
Fix 00974_distributed_join_on flackiness (by allow retries and hide logs)
Update adopters.md
...
* master: (43 commits)
Publish list of tests that failed the concurrent fast test
fix test
Use gnu++2a instead of c++2a for unbundled build to fix numeric_limits<__int128>
fix test
Update 01453_fixsed_string_sort.sql
Added test.
Fix ColumnString::updatePermutationWithCollation.
Add support for extended precision integers and decimals (#13097)
remove retries
Check that ya.make files are auto-generated
Fix "Arcadia"
ISSUES-4006 trigger CI again
fixed typo arrayCompact
Stratify nans comparison in arrayCompact function
Remove even more useless code
Remove useless code around zkutil
Fix 00956_sensitive_data_masking flackiness
Update docker/test/stress/run.sh
add exclusive DDLGuard for database
Fix handling embedded config.
...
