After #36425 there was a lot of confusions/problems with configuring pools - when the message was confusing, and settings need to be ajusted in several places.
See some examples in #44251, #43351, #47900, #46515.
The commit includes the following changes:
1) Introduced a unified mechanism for reading pool sizes from the configuration file(s). Previously, pool sizes were read from the Context.cpp with fallbacks to profiles, whereas main_config_reloader in Server.cpp read them directly without fallbacks.
2) Corrected the data type for background_merges_mutations_concurrency_ratio. It should be float instead of int.
3) Refactored the default values for settings. Previously, they were defined in multiple places throughout the codebase, but they are now defined in one place (or two, to be exact: Settings.h and ServerSettings.h).
4) Improved documentation, including the correct message in system.settings.
Additionally make the code more conform with #46550.
* Remove obsolete code for archive suffixes.
* Simplify backup coordination, stop using it for restoring.
* Build all file infos before writing to backup. Decrease number of znodes.
* Split long values before writing to ZooKeeper.
* Use separate mutexes for unrelated activities.
* Make test test_disallow_concurrency less flaky.
* Add comments and test for backup_keeper_value_max_size.
Those settings can be applied only at start, so it is better to have
them in configuration file instead of user profile settings.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
ClickHouse reads table data in blocks of 'max_block_size' rows. Due to
filtering, aggregation, etc., result blocks are typically much smaller
than 'max_block_size' but there are also cases where they are much
bigger. Setting 'query_cache_squash_partial_results' (enabled by
default) now controls if result blocks are squashed (if they are tiny)
or split (if they are large) into blocks of 'max_block_size' size before
insertion into the query result cache. This reduces performance of
writes into the query cache but improves compressability of cache
entries and provides more natural block granularity when query results
are later served from the query cache.
Entries in the query cache are now also compressed by default. This
reduces the overall memory consumption at the cost of slower writes into
/ reads from the query cache. To disable compression, use setting
'query_cache_compress_entries'.
Prevous implementation (DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET)
accepts the salt from the client, which make it useless.
Reimplement the protocol to send the salt by the server and use it in
the client instead.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
adds a new "virtual" parameter `__request_body`
which can be used in http rules, see below
tremendously helps in producing arbitrary REST interfaces
for example:
```xml
<http_handlers>
<rule>
<methods>POST</methods>
<url>/ingest</url>
<handler>
<type>predefined_query_handler</type>
<query>INSERT INTO events(id, data) SELECT {id:UInt32}, {__request_body:String}</query>
</handler>
</rule>
</http_handlers>
```
This can be useful in the following scenarious:
- you want to duplicate the data to another table and you don't care
about the errors
- you want to duplicate system.*_log to another server, you are adding
materialized view that will push to Distributed table, but you don't
want to miss original blocks in the local system.*_log
- you want to push some data to a 3d party service, using i.e. URL
engine.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree
* Add new test for the new ReplacingMergeTree engine
* Limit sign value to -1/1
* Replace sign column(Int8) by is_deleted(UInt8)
* Add keyword 'CLEANUP' when OPTIMIZE
* Cleanup uniquely when it's a replacingMergeTree
* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'
* Cleanup data flagged as 'is_deleted'
* Fix merge when optimize and add a test
* Fix OPTIMIZE and INSERT + add tests
* New fix for cleanup at the merge
* Cleanup debug logs
* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'
* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'
* Add Replicated tests
* Disable tag 'long' for our test and cleanup some white spaces
* Update tests
* Fix tests and remove additional useless whitespace
* Fix replica test
* Style clean && add condition check for is_deleted values
* clean_deleted_rows settings is nom an enum
* Add valid default value to the clean_deleted_rows settings
* Update cleanup checkers to use the enum and fix typos in the test
* Fix submodule contrib/AMQP-CPP pointer
* Add missing messages in test reference and remove a print with non derterministic order
* fix replica test reference
* Fix edge case
* Fix a typo for the spell checker
* Fix reference
* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup
* Change tests file name and update number
* This should fix the ReplacingMergeTree parameter set
* Fix replicated parameters
* Disable allow_deprecated_syntax_for_merge_tree for our new column
* Fix a test
* Remove non deterministic order print in the test
* Test on replicas
* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe
* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"
This reverts commit b65d64c05e.
* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise
* Test a fix for replicated log information error
* Try to add sync to have consistent results
* Change path of replicas that should cause one issue and add few prints in case it's not that
* Get cleanup info on replicas only if information found
* Fix style issues
* Try to avoid replication error 'cannot select parts...' and and replica read/write field order
* Cleanup according to PR reviews
and add tests on error raised.
* Update src/Storages/MergeTree/registerStorageMergeTree.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
* Select ... FINAL don't show rows with is_deleted = true
* Update and fix SELECT ... FINAL merge parameter
* Remove is_deleted rows only on the version inserted when merge
* Fix (master) updates issues
* Revert changes that should not be commited
* Add changes according to review
* Revert changes that should not be commited - part 2
---------
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
Allow configuring storage as `SETTINGS disk='<disk_name>'` (instead of `storage_policy`) and explicit disk creation `SETTINGS disk=disk(<disk_configuration>)`
