mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-09-19 16:20:50 +00:00
Merge branch 'master' into vdimir/ineq_join_v2
This commit is contained in:
vdimir
commit
7617155901
1
.github/PULL_REQUEST_TEMPLATE.md
vendored
1
.github/PULL_REQUEST_TEMPLATE.md
vendored
@ -56,6 +56,7 @@ At a minimum, the following information should be added (but add more as needed)
|
||||
- [ ] <!---ci_include_asan--> All with ASAN
|
||||
- [ ] <!---ci_include_tsan--> All with TSAN
|
||||
- [ ] <!---ci_include_analyzer--> All with Analyzer
|
||||
- [ ] <!---ci_include_azure --> All with Azure
|
||||
- [ ] <!---ci_include_KEYWORD--> Add your option here
|
||||
|
||||
#### Exclude tests:
|
||||
|
||||
2
.github/workflows/debug.yml
vendored
2
.github/workflows/debug.yml
vendored
@ -8,4 +8,4 @@ jobs:
|
||||
DebugInfo:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: hmarr/debug-action@a701ed95a46e6f2fb0df25e1a558c16356fae35a
|
||||
- uses: hmarr/debug-action@f7318c783045ac39ed9bb497e22ce835fdafbfe6
|
||||
|
||||
2
.github/workflows/master.yml
vendored
2
.github/workflows/master.yml
vendored
@ -16,7 +16,7 @@ jobs:
|
||||
data: ${{ steps.runconfig.outputs.CI_DATA }}
|
||||
steps:
|
||||
- name: DebugInfo
|
||||
uses: hmarr/debug-action@a701ed95a46e6f2fb0df25e1a558c16356fae35a
|
||||
uses: hmarr/debug-action@f7318c783045ac39ed9bb497e22ce835fdafbfe6
|
||||
- name: Check out repository code
|
||||
uses: ClickHouse/checkout@v1
|
||||
with:
|
||||
|
||||
2
.github/workflows/pull_request.yml
vendored
2
.github/workflows/pull_request.yml
vendored
@ -22,7 +22,7 @@ jobs:
|
||||
data: ${{ steps.runconfig.outputs.CI_DATA }}
|
||||
steps:
|
||||
- name: DebugInfo
|
||||
uses: hmarr/debug-action@a701ed95a46e6f2fb0df25e1a558c16356fae35a
|
||||
uses: hmarr/debug-action@f7318c783045ac39ed9bb497e22ce835fdafbfe6
|
||||
- name: Check out repository code
|
||||
uses: ClickHouse/checkout@v1
|
||||
with:
|
||||
|
||||
2
.github/workflows/reusable_simple_job.yml
vendored
2
.github/workflows/reusable_simple_job.yml
vendored
@ -63,7 +63,7 @@ jobs:
|
||||
GITHUB_JOB_OVERRIDDEN: ${{inputs.test_name}}
|
||||
steps:
|
||||
- name: DebugInfo
|
||||
uses: hmarr/debug-action@a701ed95a46e6f2fb0df25e1a558c16356fae35a
|
||||
uses: hmarr/debug-action@f7318c783045ac39ed9bb497e22ce835fdafbfe6
|
||||
- name: Check out repository code
|
||||
uses: ClickHouse/checkout@v1
|
||||
with:
|
||||
|
||||
163
CHANGELOG.md
163
CHANGELOG.md
@ -1,4 +1,5 @@
|
||||
### Table of Contents
|
||||
**[ClickHouse release v24.4, 2024-04-30](#244)**<br/>
|
||||
**[ClickHouse release v24.3 LTS, 2024-03-26](#243)**<br/>
|
||||
**[ClickHouse release v24.2, 2024-02-29](#242)**<br/>
|
||||
**[ClickHouse release v24.1, 2024-01-30](#241)**<br/>
|
||||
@ -6,6 +7,167 @@
|
||||
|
||||
# 2024 Changelog
|
||||
|
||||
### <a id="244"></a> ClickHouse release 24.4 LTS, 2024-04-30
|
||||
|
||||
#### Upgrade Notes
|
||||
* `clickhouse-odbc-bridge` and `clickhouse-library-bridge` are now separate packages. This closes [#61677](https://github.com/ClickHouse/ClickHouse/issues/61677). [#62114](https://github.com/ClickHouse/ClickHouse/pull/62114) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Don't allow to set max_parallel_replicas (for the experimental parallel reading from replicas) to `0` as it doesn't make sense. Closes [#60140](https://github.com/ClickHouse/ClickHouse/issues/60140). [#61201](https://github.com/ClickHouse/ClickHouse/pull/61201) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Remove support for `INSERT WATCH` query (part of the deprecated `LIVE VIEW` feature). [#62382](https://github.com/ClickHouse/ClickHouse/pull/62382) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Removed the `optimize_monotonous_functions_in_order_by` setting. [#63004](https://github.com/ClickHouse/ClickHouse/pull/63004) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Remove experimental tag from the `Replicated` database engine. Now it is in Beta stage. [#62937](https://github.com/ClickHouse/ClickHouse/pull/62937) ([Justin de Guzman](https://github.com/justindeguzman)).
|
||||
|
||||
#### New Feature
|
||||
* Support recursive CTEs. [#62074](https://github.com/ClickHouse/ClickHouse/pull/62074) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Support `QUALIFY` clause. Closes [#47819](https://github.com/ClickHouse/ClickHouse/issues/47819). [#62619](https://github.com/ClickHouse/ClickHouse/pull/62619) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Table engines are grantable now, and it won't affect existing users behavior. [#60117](https://github.com/ClickHouse/ClickHouse/pull/60117) ([jsc0218](https://github.com/jsc0218)).
|
||||
* Added a rewritable S3 disk which supports INSERT operations and does not require locally stored metadata. [#61116](https://github.com/ClickHouse/ClickHouse/pull/61116) ([Julia Kartseva](https://github.com/jkartseva)). The main use case is for system tables.
|
||||
* The syntax highlighting while typing in the client will work on the syntax level (previously, it worked on the lexer level). [#62123](https://github.com/ClickHouse/ClickHouse/pull/62123) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Supports dropping multiple tables at the same time like `DROP TABLE a, b, c`;. [#58705](https://github.com/ClickHouse/ClickHouse/pull/58705) ([zhongyuankai](https://github.com/zhongyuankai)).
|
||||
* Modifying memory table settings through `ALTER MODIFY SETTING` is now supported. Example: `ALTER TABLE memory MODIFY SETTING min_rows_to_keep = 100, max_rows_to_keep = 1000;`. [#62039](https://github.com/ClickHouse/ClickHouse/pull/62039) ([zhongyuankai](https://github.com/zhongyuankai)).
|
||||
* Added `role` query parameter to the HTTP interface. It works similarly to `SET ROLE x`, applying the role before the statement is executed. This allows for overcoming the limitation of the HTTP interface, as multiple statements are not allowed, and it is not possible to send both `SET ROLE x` and the statement itself at the same time. It is possible to set multiple roles that way, e.g., `?role=x&role=y`, which will be an equivalent of `SET ROLE x, y`. [#62669](https://github.com/ClickHouse/ClickHouse/pull/62669) ([Serge Klochkov](https://github.com/slvrtrn)).
|
||||
* Add `SYSTEM UNLOAD PRIMARY KEY` to free up memory usage for a table's primary key. [#62738](https://github.com/ClickHouse/ClickHouse/pull/62738) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
* Added `value1`, `value2`, ..., `value10` columns to `system.text_log`. These columns contain values that were used to format the message. [#59619](https://github.com/ClickHouse/ClickHouse/pull/59619) ([Alexey Katsman](https://github.com/alexkats)).
|
||||
* Added persistent virtual column `_block_offset` which stores original number of row in block that was assigned at insert. Persistence of column `_block_offset` can be enabled by the MergeTree setting `enable_block_offset_column`. Added virtual column`_part_data_version` which contains either min block number or mutation version of part. Persistent virtual column `_block_number` is not considered experimental anymore. [#60676](https://github.com/ClickHouse/ClickHouse/pull/60676) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Add a setting `input_format_json_throw_on_bad_escape_sequence`, disabling it allows saving bad escape sequences in JSON input formats. [#61889](https://github.com/ClickHouse/ClickHouse/pull/61889) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
|
||||
#### Performance Improvement
|
||||
* JOIN filter push down improvements using equivalent sets. [#61216](https://github.com/ClickHouse/ClickHouse/pull/61216) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Convert OUTER JOIN to INNER JOIN optimization if the filter after JOIN always filters default values. Optimization can be controlled with setting `query_plan_convert_outer_join_to_inner_join`, enabled by default. [#62907](https://github.com/ClickHouse/ClickHouse/pull/62907) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Improvement for AWS S3. Client has to send header 'Keep-Alive: timeout=X' to the server. If a client receives a response from the server with that header, client has to use the value from the server. Also for a client it is better not to use a connection which is nearly expired in order to avoid connection close race. [#62249](https://github.com/ClickHouse/ClickHouse/pull/62249) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Reduce overhead of the mutations for SELECTs (v2). [#60856](https://github.com/ClickHouse/ClickHouse/pull/60856) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* More frequently invoked functions in PODArray are now force-inlined. [#61144](https://github.com/ClickHouse/ClickHouse/pull/61144) ([李扬](https://github.com/taiyang-li)).
|
||||
* Speed up parsing of JSON by skipping the rest of the object when all required columns are read. [#62210](https://github.com/ClickHouse/ClickHouse/pull/62210) ([lgbo](https://github.com/lgbo-ustc)).
|
||||
* Improve trivial insert select from files in file/s3/hdfs/url/... table functions. Add separate max_parsing_threads setting to control the number of threads used in parallel parsing. [#62404](https://github.com/ClickHouse/ClickHouse/pull/62404) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Functions `to_utc_timestamp` and `from_utc_timestamp` are now about 2x faster. [#62583](https://github.com/ClickHouse/ClickHouse/pull/62583) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* Functions `parseDateTimeOrNull`, `parseDateTimeOrZero`, `parseDateTimeInJodaSyntaxOrNull` and `parseDateTimeInJodaSyntaxOrZero` now run significantly faster (10x - 1000x) when the input contains mostly non-parseable values. [#62634](https://github.com/ClickHouse/ClickHouse/pull/62634) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
* SELECTs against `system.query_cache` are now noticeably faster when the query cache contains lots of entries (e.g. more than 100.000). [#62671](https://github.com/ClickHouse/ClickHouse/pull/62671) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Less contention in filesystem cache (part 3): execute removal from filesystem without lock on space reservation attempt. [#61163](https://github.com/ClickHouse/ClickHouse/pull/61163) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Speed up dynamic resize of filesystem cache. [#61723](https://github.com/ClickHouse/ClickHouse/pull/61723) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Dictionary source with `INVALIDATE_QUERY` is not reloaded twice on startup. [#62050](https://github.com/ClickHouse/ClickHouse/pull/62050) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix an issue where when a redundant `= 1` or `= 0` is added after a boolean expression involving the primary key, the primary index is not used. For example, both `SELECT * FROM <table> WHERE <primary-key> IN (<value>) = 1` and `SELECT * FROM <table> WHERE <primary-key> NOT IN (<value>) = 0` will both perform a full table scan, when the primary index can be used. [#62142](https://github.com/ClickHouse/ClickHouse/pull/62142) ([josh-hildred](https://github.com/josh-hildred)).
|
||||
* Return stream of chunks from `system.remote_data_paths` instead of accumulating the whole result in one big chunk. This allows to consume less memory, show intermediate progress and cancel the query. [#62613](https://github.com/ClickHouse/ClickHouse/pull/62613) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
|
||||
#### Experimental Feature
|
||||
* Support parallel write buffer for Azure Blob Storage managed by setting `azure_allow_parallel_part_upload`. [#62534](https://github.com/ClickHouse/ClickHouse/pull/62534) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Userspace page cache works with static web storage (`disk(type = web)`) now. Use client setting `use_page_cache_for_disks_without_file_cache=1` to enable. [#61911](https://github.com/ClickHouse/ClickHouse/pull/61911) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* Don't treat Bool and number variants as suspicious in the `Variant` type. [#61999](https://github.com/ClickHouse/ClickHouse/pull/61999) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Implement better conversion from String to `Variant` using parsing. [#62005](https://github.com/ClickHouse/ClickHouse/pull/62005) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support `Variant` in JSONExtract functions. [#62014](https://github.com/ClickHouse/ClickHouse/pull/62014) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Mark type `Variant` as comparable so it can be used in primary key. [#62693](https://github.com/ClickHouse/ClickHouse/pull/62693) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
|
||||
#### Improvement
|
||||
* For convenience purpose, `SELECT * FROM numbers() `will work in the same way as `SELECT * FROM system.numbers` - without a limit. [#61969](https://github.com/ClickHouse/ClickHouse/pull/61969) ([YenchangChan](https://github.com/YenchangChan)).
|
||||
* Introduce separate consumer/producer tags for the Kafka configuration. This avoids warnings from librdkafka (a bad C library with a lot of bugs) that consumer properties were specified for producer instances and vice versa (e.g. `Configuration property session.timeout.ms is a consumer property and will be ignored by this producer instance`). Closes: [#58983](https://github.com/ClickHouse/ClickHouse/issues/58983). [#58956](https://github.com/ClickHouse/ClickHouse/pull/58956) ([Aleksandr Musorin](https://github.com/AVMusorin)).
|
||||
* Functions `date_diff` and `age` now calculate their result at nanosecond instead of microsecond precision. They now also offer `nanosecond` (or `nanoseconds` or `ns`) as a possible value for the `unit` parameter. [#61409](https://github.com/ClickHouse/ClickHouse/pull/61409) ([Austin Kothig](https://github.com/kothiga)).
|
||||
* Added nano-, micro-, milliseconds unit for `date_trunc`. [#62335](https://github.com/ClickHouse/ClickHouse/pull/62335) ([Misz606](https://github.com/Misz606)).
|
||||
* Reload certificate chain during certificate reload. [#61671](https://github.com/ClickHouse/ClickHouse/pull/61671) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)).
|
||||
* Try to prevent an error [#60432](https://github.com/ClickHouse/ClickHouse/issues/60432) by not allowing a table to be attached if there is an active replica for that replica path. [#61876](https://github.com/ClickHouse/ClickHouse/pull/61876) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Implement support for `input` for `clickhouse-local`. [#61923](https://github.com/ClickHouse/ClickHouse/pull/61923) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* `Join` table engine with strictness `ANY` is consistent after reload. When several rows with the same key are inserted, the first one will have higher priority (before, it was chosen randomly upon table loading). close [#51027](https://github.com/ClickHouse/ClickHouse/issues/51027). [#61972](https://github.com/ClickHouse/ClickHouse/pull/61972) ([vdimir](https://github.com/vdimir)).
|
||||
* Automatically infer Nullable column types from Apache Arrow schema. [#61984](https://github.com/ClickHouse/ClickHouse/pull/61984) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Allow to cancel parallel merge of aggregate states during aggregation. Example: `uniqExact`. [#61992](https://github.com/ClickHouse/ClickHouse/pull/61992) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Use `system.keywords` to fill in the suggestions and also use them in the all places internally. [#62000](https://github.com/ClickHouse/ClickHouse/pull/62000) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* `OPTIMIZE FINAL` for `ReplicatedMergeTree` now will wait for currently active merges to finish and then reattempt to schedule a final merge. This will put it more in line with ordinary `MergeTree` behaviour. [#62067](https://github.com/ClickHouse/ClickHouse/pull/62067) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* While read data from a hive text file, it would use the first line of hive text file to resize of number of input fields, and sometimes the fields number of first line is not matched with the hive table defined , such as the hive table is defined to have 3 columns, like `test_tbl(a Int32, b Int32, c Int32)`, but the first line of text file only has 2 fields, and in this suitation, the input fields will be resized to 2, and if the next line of the text file has 3 fields, then the third field can not be read but set a default value 0, which is not right. [#62086](https://github.com/ClickHouse/ClickHouse/pull/62086) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* `CREATE AS` copies the table's comment. [#62117](https://github.com/ClickHouse/ClickHouse/pull/62117) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
* Add query progress to table zookeeper. [#62152](https://github.com/ClickHouse/ClickHouse/pull/62152) ([JackyWoo](https://github.com/JackyWoo)).
|
||||
* Add ability to turn on trace collector (Real and CPU) server-wide. [#62189](https://github.com/ClickHouse/ClickHouse/pull/62189) ([alesapin](https://github.com/alesapin)).
|
||||
* Added setting `lightweight_deletes_sync` (default value: 2 - wait all replicas synchronously). It is similar to setting `mutations_sync` but affects only behaviour of lightweight deletes. [#62195](https://github.com/ClickHouse/ClickHouse/pull/62195) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Distinguish booleans and integers while parsing values for custom settings: `SET custom_a = true; SET custom_b = 1;`. [#62206](https://github.com/ClickHouse/ClickHouse/pull/62206) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Support S3 access through AWS Private Link Interface endpoints. Closes [#60021](https://github.com/ClickHouse/ClickHouse/issues/60021), [#31074](https://github.com/ClickHouse/ClickHouse/issues/31074) and [#53761](https://github.com/ClickHouse/ClickHouse/issues/53761). [#62208](https://github.com/ClickHouse/ClickHouse/pull/62208) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Do not create a directory for UDF in clickhouse-client if it does not exist. This closes [#59597](https://github.com/ClickHouse/ClickHouse/issues/59597). [#62366](https://github.com/ClickHouse/ClickHouse/pull/62366) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* The query cache now no longer caches results of queries against system tables (`system.*`, `information_schema.*`, `INFORMATION_SCHEMA.*`). [#62376](https://github.com/ClickHouse/ClickHouse/pull/62376) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* `MOVE PARTITION TO TABLE` query can be delayed or can throw `TOO_MANY_PARTS` exception to avoid exceeding limits on the part count. The same settings and limits are applied as for the`INSERT` query (see `max_parts_in_total`, `parts_to_delay_insert`, `parts_to_throw_insert`, `inactive_parts_to_throw_insert`, `inactive_parts_to_delay_insert`, `max_avg_part_size_for_too_many_parts`, `min_delay_to_insert_ms` and `max_delay_to_insert` settings). [#62420](https://github.com/ClickHouse/ClickHouse/pull/62420) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Changed the default installation directory on macOS from `/usr/bin` to `/usr/local/bin`. This is necessary because Apple's System Integrity Protection introduced with macOS El Capitan (2015) prevents writing into `/usr/bin`, even with `sudo`. [#62489](https://github.com/ClickHouse/ClickHouse/pull/62489) ([haohang](https://github.com/yokofly)).
|
||||
* Make transform always return the first match. [#62518](https://github.com/ClickHouse/ClickHouse/pull/62518) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Added the missing `hostname` column to system table `blob_storage_log`. [#62456](https://github.com/ClickHouse/ClickHouse/pull/62456) ([Jayme Bird](https://github.com/jaymebrd)).
|
||||
* For consistency with other system tables, `system.backup_log` now has a column `event_time`. [#62541](https://github.com/ClickHouse/ClickHouse/pull/62541) ([Jayme Bird](https://github.com/jaymebrd)).
|
||||
* Table `system.backup_log` now has the "default" sorting key which is `event_date, event_time`, the same as for other `_log` table engines. [#62667](https://github.com/ClickHouse/ClickHouse/pull/62667) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Avoid evaluating table DEFAULT expressions while executing `RESTORE`. [#62601](https://github.com/ClickHouse/ClickHouse/pull/62601) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* S3 storage and backups also need the same default keep alive settings as s3 disk. [#62648](https://github.com/ClickHouse/ClickHouse/pull/62648) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Add librdkafka's (that infamous C library, which has a lot of bugs) client identifier to log messages to be able to differentiate log messages from different consumers of a single table. [#62813](https://github.com/ClickHouse/ClickHouse/pull/62813) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Allow special macros `{uuid}` and `{database}` in a Replicated database ZooKeeper path. [#62818](https://github.com/ClickHouse/ClickHouse/pull/62818) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Allow quota key with different auth scheme in HTTP requests. [#62842](https://github.com/ClickHouse/ClickHouse/pull/62842) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Reduce the verbosity of command line argument `--help` in `clickhouse client` and `clickhouse local`. The previous output is now generated by `--help --verbose`. [#62973](https://github.com/ClickHouse/ClickHouse/pull/62973) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* `log_bin_use_v1_row_events` was removed in MySQL 8.3, and we adjust the experimental `MaterializedMySQL` engine for it [#60479](https://github.com/ClickHouse/ClickHouse/issues/60479). [#63101](https://github.com/ClickHouse/ClickHouse/pull/63101) ([Eugene Klimov](https://github.com/Slach)). Author: Nikolay Yankin.
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
* Vendor in Rust dependencies, so the Rust code (that we use for minor features for hype and lulz) can be built in a sane way, similarly to C++. [#62297](https://github.com/ClickHouse/ClickHouse/pull/62297) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* ClickHouse now uses OpenSSL 3.2 instead of BoringSSL. [#59870](https://github.com/ClickHouse/ClickHouse/pull/59870) ([Robert Schulze](https://github.com/rschu1ze)). Note that OpenSSL has generally worse engineering culture (such as non-zero number of sanitizer reports, that we had to patch, a complex build system with generated files, etc.) but has better compatibility.
|
||||
* Ignore DROP queries in stress test with 1/2 probability, use TRUNCATE instead of ignoring DROP in upgrade check for Memory/JOIN tables. [#61476](https://github.com/ClickHouse/ClickHouse/pull/61476) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Remove from the Keeper Docker image the volumes at /etc/clickhouse-keeper and /var/log/clickhouse-keeper. [#61683](https://github.com/ClickHouse/ClickHouse/pull/61683) ([Tristan](https://github.com/Tristan971)).
|
||||
* Add tests for all issues which are no longer relevant with Analyzer being enabled by default. Closes: [#55794](https://github.com/ClickHouse/ClickHouse/issues/55794) Closes: [#49472](https://github.com/ClickHouse/ClickHouse/issues/49472) Closes: [#44414](https://github.com/ClickHouse/ClickHouse/issues/44414) Closes: [#13843](https://github.com/ClickHouse/ClickHouse/issues/13843) Closes: [#55803](https://github.com/ClickHouse/ClickHouse/issues/55803) Closes: [#48308](https://github.com/ClickHouse/ClickHouse/issues/48308) Closes: [#45535](https://github.com/ClickHouse/ClickHouse/issues/45535) Closes: [#44365](https://github.com/ClickHouse/ClickHouse/issues/44365) Closes: [#44153](https://github.com/ClickHouse/ClickHouse/issues/44153) Closes: [#42399](https://github.com/ClickHouse/ClickHouse/issues/42399) Closes: [#27115](https://github.com/ClickHouse/ClickHouse/issues/27115) Closes: [#23162](https://github.com/ClickHouse/ClickHouse/issues/23162) Closes: [#15395](https://github.com/ClickHouse/ClickHouse/issues/15395) Closes: [#15411](https://github.com/ClickHouse/ClickHouse/issues/15411) Closes: [#14978](https://github.com/ClickHouse/ClickHouse/issues/14978) Closes: [#17319](https://github.com/ClickHouse/ClickHouse/issues/17319) Closes: [#11813](https://github.com/ClickHouse/ClickHouse/issues/11813) Closes: [#13210](https://github.com/ClickHouse/ClickHouse/issues/13210) Closes: [#23053](https://github.com/ClickHouse/ClickHouse/issues/23053) Closes: [#37729](https://github.com/ClickHouse/ClickHouse/issues/37729) Closes: [#32639](https://github.com/ClickHouse/ClickHouse/issues/32639) Closes: [#9954](https://github.com/ClickHouse/ClickHouse/issues/9954) Closes: [#41964](https://github.com/ClickHouse/ClickHouse/issues/41964) Closes: [#54317](https://github.com/ClickHouse/ClickHouse/issues/54317) Closes: [#7520](https://github.com/ClickHouse/ClickHouse/issues/7520) Closes: [#36973](https://github.com/ClickHouse/ClickHouse/issues/36973) Closes: [#40955](https://github.com/ClickHouse/ClickHouse/issues/40955) Closes: [#19687](https://github.com/ClickHouse/ClickHouse/issues/19687) Closes: [#23104](https://github.com/ClickHouse/ClickHouse/issues/23104) Closes: [#21584](https://github.com/ClickHouse/ClickHouse/issues/21584) Closes: [#23344](https://github.com/ClickHouse/ClickHouse/issues/23344) Closes: [#22627](https://github.com/ClickHouse/ClickHouse/issues/22627) Closes: [#10276](https://github.com/ClickHouse/ClickHouse/issues/10276) Closes: [#19687](https://github.com/ClickHouse/ClickHouse/issues/19687) Closes: [#4567](https://github.com/ClickHouse/ClickHouse/issues/4567) Closes: [#17710](https://github.com/ClickHouse/ClickHouse/issues/17710) Closes: [#11068](https://github.com/ClickHouse/ClickHouse/issues/11068) Closes: [#24395](https://github.com/ClickHouse/ClickHouse/issues/24395) Closes: [#23416](https://github.com/ClickHouse/ClickHouse/issues/23416) Closes: [#23162](https://github.com/ClickHouse/ClickHouse/issues/23162) Closes: [#25655](https://github.com/ClickHouse/ClickHouse/issues/25655) Closes: [#11757](https://github.com/ClickHouse/ClickHouse/issues/11757) Closes: [#6571](https://github.com/ClickHouse/ClickHouse/issues/6571) Closes: [#4432](https://github.com/ClickHouse/ClickHouse/issues/4432) Closes: [#8259](https://github.com/ClickHouse/ClickHouse/issues/8259) Closes: [#9233](https://github.com/ClickHouse/ClickHouse/issues/9233) Closes: [#14699](https://github.com/ClickHouse/ClickHouse/issues/14699) Closes: [#27068](https://github.com/ClickHouse/ClickHouse/issues/27068) Closes: [#28687](https://github.com/ClickHouse/ClickHouse/issues/28687) Closes: [#28777](https://github.com/ClickHouse/ClickHouse/issues/28777) Closes: [#29734](https://github.com/ClickHouse/ClickHouse/issues/29734) Closes: [#61238](https://github.com/ClickHouse/ClickHouse/issues/61238) Closes: [#33825](https://github.com/ClickHouse/ClickHouse/issues/33825) Closes: [#35608](https://github.com/ClickHouse/ClickHouse/issues/35608) Closes: [#29838](https://github.com/ClickHouse/ClickHouse/issues/29838) Closes: [#35652](https://github.com/ClickHouse/ClickHouse/issues/35652) Closes: [#36189](https://github.com/ClickHouse/ClickHouse/issues/36189) Closes: [#39634](https://github.com/ClickHouse/ClickHouse/issues/39634) Closes: [#47432](https://github.com/ClickHouse/ClickHouse/issues/47432) Closes: [#54910](https://github.com/ClickHouse/ClickHouse/issues/54910) Closes: [#57321](https://github.com/ClickHouse/ClickHouse/issues/57321) Closes: [#59154](https://github.com/ClickHouse/ClickHouse/issues/59154) Closes: [#61014](https://github.com/ClickHouse/ClickHouse/issues/61014) Closes: [#61950](https://github.com/ClickHouse/ClickHouse/issues/61950) Closes: [#55647](https://github.com/ClickHouse/ClickHouse/issues/55647) Closes: [#61947](https://github.com/ClickHouse/ClickHouse/issues/61947). [#62185](https://github.com/ClickHouse/ClickHouse/pull/62185) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Add more tests from issues which are no longer relevant or fixed by analyzer. Closes: [#58985](https://github.com/ClickHouse/ClickHouse/issues/58985) Closes: [#59549](https://github.com/ClickHouse/ClickHouse/issues/59549) Closes: [#36963](https://github.com/ClickHouse/ClickHouse/issues/36963) Closes: [#39453](https://github.com/ClickHouse/ClickHouse/issues/39453) Closes: [#56521](https://github.com/ClickHouse/ClickHouse/issues/56521) Closes: [#47552](https://github.com/ClickHouse/ClickHouse/issues/47552) Closes: [#56503](https://github.com/ClickHouse/ClickHouse/issues/56503) Closes: [#59101](https://github.com/ClickHouse/ClickHouse/issues/59101) Closes: [#50271](https://github.com/ClickHouse/ClickHouse/issues/50271) Closes: [#54954](https://github.com/ClickHouse/ClickHouse/issues/54954) Closes: [#56466](https://github.com/ClickHouse/ClickHouse/issues/56466) Closes: [#11000](https://github.com/ClickHouse/ClickHouse/issues/11000) Closes: [#10894](https://github.com/ClickHouse/ClickHouse/issues/10894) Closes: https://github.com/ClickHouse/ClickHouse/issues/448 Closes: [#8030](https://github.com/ClickHouse/ClickHouse/issues/8030) Closes: [#32139](https://github.com/ClickHouse/ClickHouse/issues/32139) Closes: [#47288](https://github.com/ClickHouse/ClickHouse/issues/47288) Closes: [#50705](https://github.com/ClickHouse/ClickHouse/issues/50705) Closes: [#54511](https://github.com/ClickHouse/ClickHouse/issues/54511) Closes: [#55466](https://github.com/ClickHouse/ClickHouse/issues/55466) Closes: [#58500](https://github.com/ClickHouse/ClickHouse/issues/58500) Closes: [#39923](https://github.com/ClickHouse/ClickHouse/issues/39923) Closes: [#39855](https://github.com/ClickHouse/ClickHouse/issues/39855) Closes: [#4596](https://github.com/ClickHouse/ClickHouse/issues/4596) Closes: [#47422](https://github.com/ClickHouse/ClickHouse/issues/47422) Closes: [#33000](https://github.com/ClickHouse/ClickHouse/issues/33000) Closes: [#14739](https://github.com/ClickHouse/ClickHouse/issues/14739) Closes: [#44039](https://github.com/ClickHouse/ClickHouse/issues/44039) Closes: [#8547](https://github.com/ClickHouse/ClickHouse/issues/8547) Closes: [#22923](https://github.com/ClickHouse/ClickHouse/issues/22923) Closes: [#23865](https://github.com/ClickHouse/ClickHouse/issues/23865) Closes: [#29748](https://github.com/ClickHouse/ClickHouse/issues/29748) Closes: [#4222](https://github.com/ClickHouse/ClickHouse/issues/4222). [#62457](https://github.com/ClickHouse/ClickHouse/pull/62457) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Fixed build errors when OpenSSL is linked dynamically (note: this is generally unsupported and only required for IBM's s390x platforms). [#62888](https://github.com/ClickHouse/ClickHouse/pull/62888) ([Harry Lee](https://github.com/HarryLeeIBM)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
* Fix logical-error when undoing quorum insert transaction. [#61953](https://github.com/ClickHouse/ClickHouse/pull/61953) ([Han Fei](https://github.com/hanfei1991)).
|
||||
* Fix parser error when using COUNT(*) with FILTER clause [#61357](https://github.com/ClickHouse/ClickHouse/pull/61357) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix logical error in `group_by_use_nulls` + grouping sets + analyzer + materialize/constant [#61567](https://github.com/ClickHouse/ClickHouse/pull/61567) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Cancel merges before removing moved parts [#61610](https://github.com/ClickHouse/ClickHouse/pull/61610) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Fix abort in Apache Arrow [#61720](https://github.com/ClickHouse/ClickHouse/pull/61720) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Search for `convert_to_replicated` flag at the correct path corresponding to the specific disk [#61769](https://github.com/ClickHouse/ClickHouse/pull/61769) ([Kirill](https://github.com/kirillgarbar)).
|
||||
* Fix possible connections data-race for distributed_foreground_insert/distributed_background_insert_batch [#61867](https://github.com/ClickHouse/ClickHouse/pull/61867) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Mark CANNOT_PARSE_ESCAPE_SEQUENCE error as parse error to be able to skip it in row input formats [#61883](https://github.com/ClickHouse/ClickHouse/pull/61883) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix writing exception message in output format in HTTP when http_wait_end_of_query is used [#61951](https://github.com/ClickHouse/ClickHouse/pull/61951) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Proper fix for LowCardinality together with JSONExtact functions [#61957](https://github.com/ClickHouse/ClickHouse/pull/61957) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Crash in Engine Merge if Row Policy does not have expression [#61971](https://github.com/ClickHouse/ClickHouse/pull/61971) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* Fix WriteBufferAzureBlobStorage destructor uncaught exception [#61988](https://github.com/ClickHouse/ClickHouse/pull/61988) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Fix CREATE TABLE without columns definition for ReplicatedMergeTree [#62040](https://github.com/ClickHouse/ClickHouse/pull/62040) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix optimize_skip_unused_shards_rewrite_in for composite sharding key [#62047](https://github.com/ClickHouse/ClickHouse/pull/62047) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* ReadWriteBufferFromHTTP set right header host when redirected [#62068](https://github.com/ClickHouse/ClickHouse/pull/62068) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Fix external table cannot parse data type Bool [#62115](https://github.com/ClickHouse/ClickHouse/pull/62115) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Analyzer: Fix query parameter resolution [#62186](https://github.com/ClickHouse/ClickHouse/pull/62186) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix restoring parts while readonly [#62207](https://github.com/ClickHouse/ClickHouse/pull/62207) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Fix crash in index definition containing SQL UDF [#62225](https://github.com/ClickHouse/ClickHouse/pull/62225) ([vdimir](https://github.com/vdimir)).
|
||||
* Fixing NULL random seed for generateRandom with analyzer. [#62248](https://github.com/ClickHouse/ClickHouse/pull/62248) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Correctly handle const columns in Distinct Transfom [#62250](https://github.com/ClickHouse/ClickHouse/pull/62250) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix Parts Splitter for queries with the FINAL modifier [#62268](https://github.com/ClickHouse/ClickHouse/pull/62268) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Analyzer: Fix alias to parametrized view resolution [#62274](https://github.com/ClickHouse/ClickHouse/pull/62274) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Analyzer: Fix name resolution from parent scopes [#62281](https://github.com/ClickHouse/ClickHouse/pull/62281) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix argMax with nullable non native numeric column [#62285](https://github.com/ClickHouse/ClickHouse/pull/62285) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix BACKUP and RESTORE of a materialized view in Ordinary database [#62295](https://github.com/ClickHouse/ClickHouse/pull/62295) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Fix data race on scalars in Context [#62305](https://github.com/ClickHouse/ClickHouse/pull/62305) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix primary key in materialized view [#62319](https://github.com/ClickHouse/ClickHouse/pull/62319) ([Murat Khairulin](https://github.com/mxwell)).
|
||||
* Do not build multithread insert pipeline for tables without support [#62333](https://github.com/ClickHouse/ClickHouse/pull/62333) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix analyzer with positional arguments in distributed query [#62362](https://github.com/ClickHouse/ClickHouse/pull/62362) ([flynn](https://github.com/ucasfl)).
|
||||
* Fix filter pushdown from additional_table_filters in Merge engine in analyzer [#62398](https://github.com/ClickHouse/ClickHouse/pull/62398) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix GLOBAL IN table queries with analyzer. [#62409](https://github.com/ClickHouse/ClickHouse/pull/62409) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Respect settings truncate_on_insert/create_new_file_on_insert in s3/hdfs/azure engines during partitioned write [#62425](https://github.com/ClickHouse/ClickHouse/pull/62425) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix backup restore path for AzureBlobStorage [#62447](https://github.com/ClickHouse/ClickHouse/pull/62447) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Fix SimpleSquashingChunksTransform [#62451](https://github.com/ClickHouse/ClickHouse/pull/62451) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix capture of nested lambda. [#62462](https://github.com/ClickHouse/ClickHouse/pull/62462) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Avoid crash when reading protobuf with recursive types [#62506](https://github.com/ClickHouse/ClickHouse/pull/62506) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix a bug moving one partition from one to itself [#62524](https://github.com/ClickHouse/ClickHouse/pull/62524) ([helifu](https://github.com/helifu)).
|
||||
* Fix scalar subquery in LIMIT [#62567](https://github.com/ClickHouse/ClickHouse/pull/62567) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix segfault in the experimental and unsupported Hive engine, which we don't like anyway [#62578](https://github.com/ClickHouse/ClickHouse/pull/62578) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix memory leak in groupArraySorted [#62597](https://github.com/ClickHouse/ClickHouse/pull/62597) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix crash in largestTriangleThreeBuckets [#62646](https://github.com/ClickHouse/ClickHouse/pull/62646) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix tumble\[Start,End\] and hop\[Start,End\] for bigger resolutions [#62705](https://github.com/ClickHouse/ClickHouse/pull/62705) ([Jordi Villar](https://github.com/jrdi)).
|
||||
* Fix argMin/argMax combinator state [#62708](https://github.com/ClickHouse/ClickHouse/pull/62708) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix temporary data in cache failing because of cache lock contention optimization [#62715](https://github.com/ClickHouse/ClickHouse/pull/62715) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix crash in function `mergeTreeIndex` [#62762](https://github.com/ClickHouse/ClickHouse/pull/62762) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* fix: update: nested materialized columns: size check fixes [#62773](https://github.com/ClickHouse/ClickHouse/pull/62773) ([Eliot Hautefeuille](https://github.com/hileef)).
|
||||
* Fix FINAL modifier is not respected in CTE with analyzer [#62811](https://github.com/ClickHouse/ClickHouse/pull/62811) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix crash in function `formatRow` with `JSON` format and HTTP interface [#62840](https://github.com/ClickHouse/ClickHouse/pull/62840) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Azure: fix building final url from endpoint object [#62850](https://github.com/ClickHouse/ClickHouse/pull/62850) ([Daniel Pozo Escalona](https://github.com/danipozo)).
|
||||
* Fix GCD codec [#62853](https://github.com/ClickHouse/ClickHouse/pull/62853) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix LowCardinality(Nullable) key in hyperrectangle [#62866](https://github.com/ClickHouse/ClickHouse/pull/62866) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix fromUnixtimestamp in joda syntax while the input value beyond UInt32 [#62901](https://github.com/ClickHouse/ClickHouse/pull/62901) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* Disable optimize_rewrite_aggregate_function_with_if for sum(nullable) [#62912](https://github.com/ClickHouse/ClickHouse/pull/62912) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix PREWHERE for StorageBuffer with different source table column types. [#62916](https://github.com/ClickHouse/ClickHouse/pull/62916) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix temporary data in cache incorrectly processing failure of cache key directory creation [#62925](https://github.com/ClickHouse/ClickHouse/pull/62925) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* gRPC: fix crash on IPv6 peer connection [#62978](https://github.com/ClickHouse/ClickHouse/pull/62978) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Fix possible CHECKSUM_DOESNT_MATCH (and others) during replicated fetches [#62987](https://github.com/ClickHouse/ClickHouse/pull/62987) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix terminate with uncaught exception in temporary data in cache [#62998](https://github.com/ClickHouse/ClickHouse/pull/62998) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix optimize_rewrite_aggregate_function_with_if implicit cast [#62999](https://github.com/ClickHouse/ClickHouse/pull/62999) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix unhandled exception in ~RestorerFromBackup [#63040](https://github.com/ClickHouse/ClickHouse/pull/63040) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Do not remove server constants from GROUP BY key for secondary query. [#63047](https://github.com/ClickHouse/ClickHouse/pull/63047) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix incorrect judgement of of monotonicity of function abs [#63097](https://github.com/ClickHouse/ClickHouse/pull/63097) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Set server name for SSL handshake in MongoDB engine [#63122](https://github.com/ClickHouse/ClickHouse/pull/63122) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Use user specified db instead of "config" for MongoDB wire protocol version check [#63126](https://github.com/ClickHouse/ClickHouse/pull/63126) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
|
||||
|
||||
### <a id="243"></a> ClickHouse release 24.3 LTS, 2024-03-27
|
||||
|
||||
#### Upgrade Notes
|
||||
@ -185,6 +347,7 @@
|
||||
* Add sanity check for number of threads and block sizes. [#60138](https://github.com/ClickHouse/ClickHouse/pull/60138) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Don't infer floats in exponential notation by default. Add a setting `input_format_try_infer_exponent_floats` that will restore previous behaviour (disabled by default). Closes [#59476](https://github.com/ClickHouse/ClickHouse/issues/59476). [#59500](https://github.com/ClickHouse/ClickHouse/pull/59500) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Allow alter operations to be surrounded by parenthesis. The emission of parentheses can be controlled by the `format_alter_operations_with_parentheses` config. By default, in formatted queries the parentheses are emitted as we store the formatted alter operations in some places as metadata (e.g.: mutations). The new syntax clarifies some of the queries where alter operations end in a list. E.g.: `ALTER TABLE x MODIFY TTL date GROUP BY a, b, DROP COLUMN c` cannot be parsed properly with the old syntax. In the new syntax the query `ALTER TABLE x (MODIFY TTL date GROUP BY a, b), (DROP COLUMN c)` is obvious. Older versions are not able to read the new syntax, therefore using the new syntax might cause issues if newer and older version of ClickHouse are mixed in a single cluster. [#59532](https://github.com/ClickHouse/ClickHouse/pull/59532) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Fix for the materialized view security issue, which allowed a user to insert into a table without required grants for that. Fix validates that the user has permission to insert not only into a materialized view but also into all underlying tables. This means that some queries, which worked before, now can fail with `Not enough privileges`. To address this problem, the release introduces a new feature of SQL security for views https://clickhouse.com/docs/en/sql-reference/statements/create/view#sql_security. [#54901](https://github.com/ClickHouse/ClickHouse/pull/54901) [#60439](https://github.com/ClickHouse/ClickHouse/pull/60439) ([pufit](https://github.com/pufit)).
|
||||
|
||||
#### New Feature
|
||||
* Added new syntax which allows to specify definer user in View/Materialized View. This allows to execute selects/inserts from views without explicit grants for underlying tables. So, a View will encapsulate the grants. [#54901](https://github.com/ClickHouse/ClickHouse/pull/54901) [#60439](https://github.com/ClickHouse/ClickHouse/pull/60439) ([pufit](https://github.com/pufit)).
|
||||
|
||||
@ -13,9 +13,10 @@ The following versions of ClickHouse server are currently being supported with s
|
||||
|
||||
| Version | Supported |
|
||||
|:-|:-|
|
||||
| 24.4 | ✔️ |
|
||||
| 24.3 | ✔️ |
|
||||
| 24.2 | ✔️ |
|
||||
| 24.1 | ✔️ |
|
||||
| 24.1 | ❌ |
|
||||
| 23.* | ❌ |
|
||||
| 23.8 | ✔️ |
|
||||
| 23.7 | ❌ |
|
||||
|
||||
@ -2,11 +2,11 @@
|
||||
|
||||
# NOTE: has nothing common with DBMS_TCP_PROTOCOL_VERSION,
|
||||
# only DBMS_TCP_PROTOCOL_VERSION should be incremented on protocol changes.
|
||||
SET(VERSION_REVISION 54485)
|
||||
SET(VERSION_REVISION 54486)
|
||||
SET(VERSION_MAJOR 24)
|
||||
SET(VERSION_MINOR 4)
|
||||
SET(VERSION_MINOR 5)
|
||||
SET(VERSION_PATCH 1)
|
||||
SET(VERSION_GITHASH 2c5c589a882ceec35439650337b92db3e76f0081)
|
||||
SET(VERSION_DESCRIBE v24.4.1.1-testing)
|
||||
SET(VERSION_STRING 24.4.1.1)
|
||||
SET(VERSION_GITHASH 6d4b31322d168356c8b10c43b4cef157c82337ff)
|
||||
SET(VERSION_DESCRIBE v24.5.1.1-testing)
|
||||
SET(VERSION_STRING 24.5.1.1)
|
||||
# end of autochange
|
||||
|
||||
2
contrib/openssl
vendored
2
contrib/openssl
vendored
@ -1 +1 @@
|
||||
Subproject commit 417f9d2825799769708d99917d0465574c36f79a
|
||||
Subproject commit f7b8721dfc66abb147f24ca07b9c9d1d64f40f71
|
||||
@ -34,7 +34,7 @@ RUN arch=${TARGETARCH:-amd64} \
|
||||
# lts / testing / prestable / etc
|
||||
ARG REPO_CHANNEL="stable"
|
||||
ARG REPOSITORY="https://packages.clickhouse.com/tgz/${REPO_CHANNEL}"
|
||||
ARG VERSION="24.3.2.23"
|
||||
ARG VERSION="24.4.1.2088"
|
||||
ARG PACKAGES="clickhouse-keeper"
|
||||
ARG DIRECT_DOWNLOAD_URLS=""
|
||||
|
||||
|
||||
@ -32,7 +32,7 @@ RUN arch=${TARGETARCH:-amd64} \
|
||||
# lts / testing / prestable / etc
|
||||
ARG REPO_CHANNEL="stable"
|
||||
ARG REPOSITORY="https://packages.clickhouse.com/tgz/${REPO_CHANNEL}"
|
||||
ARG VERSION="24.3.2.23"
|
||||
ARG VERSION="24.4.1.2088"
|
||||
ARG PACKAGES="clickhouse-client clickhouse-server clickhouse-common-static"
|
||||
ARG DIRECT_DOWNLOAD_URLS=""
|
||||
|
||||
|
||||
@ -27,7 +27,7 @@ RUN sed -i "s|http://archive.ubuntu.com|${apt_archive}|g" /etc/apt/sources.list
|
||||
|
||||
ARG REPO_CHANNEL="stable"
|
||||
ARG REPOSITORY="deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb ${REPO_CHANNEL} main"
|
||||
ARG VERSION="24.3.2.23"
|
||||
ARG VERSION="24.4.1.2088"
|
||||
ARG PACKAGES="clickhouse-client clickhouse-server clickhouse-common-static"
|
||||
|
||||
# set non-empty deb_location_url url to create a docker image
|
||||
|
||||
60
docs/changelogs/v24.2.3.70-stable.md
Normal file
60
docs/changelogs/v24.2.3.70-stable.md
Normal file
@ -0,0 +1,60 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: 2024
|
||||
---
|
||||
|
||||
# 2024 Changelog
|
||||
|
||||
### ClickHouse release v24.2.3.70-stable (8a7b8c7afb6) FIXME as compared to v24.2.2.71-stable (9293d361e72)
|
||||
|
||||
#### Performance Improvement
|
||||
* Backported in [#61630](https://github.com/ClickHouse/ClickHouse/issues/61630): Optimized function `dotProduct` to omit unnecessary and expensive memory copies. [#60928](https://github.com/ClickHouse/ClickHouse/pull/60928) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
* Backported in [#62011](https://github.com/ClickHouse/ClickHouse/issues/62011): Remove from the Keeper Docker image the volumes at /etc/clickhouse-keeper and /var/log/clickhouse-keeper. [#61683](https://github.com/ClickHouse/ClickHouse/pull/61683) ([Tristan](https://github.com/Tristan971)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
|
||||
* Fix possible incorrect result of aggregate function `uniqExact` [#61257](https://github.com/ClickHouse/ClickHouse/pull/61257) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix ATTACH query with external ON CLUSTER [#61365](https://github.com/ClickHouse/ClickHouse/pull/61365) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix consecutive keys optimization for nullable keys [#61393](https://github.com/ClickHouse/ClickHouse/pull/61393) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* fix issue of actions dag split [#61458](https://github.com/ClickHouse/ClickHouse/pull/61458) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Disable async_insert_use_adaptive_busy_timeout correctly with compatibility settings [#61468](https://github.com/ClickHouse/ClickHouse/pull/61468) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix bug when reading system.parts using UUID (issue 61220). [#61479](https://github.com/ClickHouse/ClickHouse/pull/61479) ([Dan Wu](https://github.com/wudanzy)).
|
||||
* Fix ALTER QUERY MODIFY SQL SECURITY [#61480](https://github.com/ClickHouse/ClickHouse/pull/61480) ([pufit](https://github.com/pufit)).
|
||||
* Fix client `-s` argument [#61530](https://github.com/ClickHouse/ClickHouse/pull/61530) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Fix string search with const position [#61547](https://github.com/ClickHouse/ClickHouse/pull/61547) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Cancel merges before removing moved parts [#61610](https://github.com/ClickHouse/ClickHouse/pull/61610) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Fix crash in `multiSearchAllPositionsCaseInsensitiveUTF8` for incorrect UTF-8 [#61749](https://github.com/ClickHouse/ClickHouse/pull/61749) ([pufit](https://github.com/pufit)).
|
||||
* Mark CANNOT_PARSE_ESCAPE_SEQUENCE error as parse error to be able to skip it in row input formats [#61883](https://github.com/ClickHouse/ClickHouse/pull/61883) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Crash in Engine Merge if Row Policy does not have expression [#61971](https://github.com/ClickHouse/ClickHouse/pull/61971) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* Fix data race on scalars in Context [#62305](https://github.com/ClickHouse/ClickHouse/pull/62305) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Try to fix segfault in Hive engine [#62578](https://github.com/ClickHouse/ClickHouse/pull/62578) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix memory leak in groupArraySorted [#62597](https://github.com/ClickHouse/ClickHouse/pull/62597) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix GCD codec [#62853](https://github.com/ClickHouse/ClickHouse/pull/62853) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix temporary data in cache incorrectly processing failure of cache key directory creation [#62925](https://github.com/ClickHouse/ClickHouse/pull/62925) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix incorrect judgement of of monotonicity of function abs [#63097](https://github.com/ClickHouse/ClickHouse/pull/63097) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Make sanity check of settings worse [#63119](https://github.com/ClickHouse/ClickHouse/pull/63119) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Set server name for SSL handshake in MongoDB engine [#63122](https://github.com/ClickHouse/ClickHouse/pull/63122) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Format SQL security option only in `CREATE VIEW` queries. [#63136](https://github.com/ClickHouse/ClickHouse/pull/63136) ([pufit](https://github.com/pufit)).
|
||||
|
||||
#### CI Fix or Improvement (changelog entry is not required)
|
||||
|
||||
* Backported in [#61430](https://github.com/ClickHouse/ClickHouse/issues/61430):. [#61374](https://github.com/ClickHouse/ClickHouse/pull/61374) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#61490](https://github.com/ClickHouse/ClickHouse/issues/61490): ... [#61441](https://github.com/ClickHouse/ClickHouse/pull/61441) ([Max K.](https://github.com/maxknv)).
|
||||
* Backported in [#61638](https://github.com/ClickHouse/ClickHouse/issues/61638):. [#61592](https://github.com/ClickHouse/ClickHouse/pull/61592) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#61896](https://github.com/ClickHouse/ClickHouse/issues/61896): ... [#61877](https://github.com/ClickHouse/ClickHouse/pull/61877) ([Max K.](https://github.com/maxknv)).
|
||||
* Backported in [#62055](https://github.com/ClickHouse/ClickHouse/issues/62055): ... [#62044](https://github.com/ClickHouse/ClickHouse/pull/62044) ([Max K.](https://github.com/maxknv)).
|
||||
* Backported in [#62203](https://github.com/ClickHouse/ClickHouse/issues/62203):. [#62190](https://github.com/ClickHouse/ClickHouse/pull/62190) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Backported in [#62800](https://github.com/ClickHouse/ClickHouse/issues/62800): We won't fail the job when GH fails to retrieve the job ID and URLs. [#62651](https://github.com/ClickHouse/ClickHouse/pull/62651) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#62970](https://github.com/ClickHouse/ClickHouse/issues/62970):. [#62932](https://github.com/ClickHouse/ClickHouse/pull/62932) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Backported in [#63115](https://github.com/ClickHouse/ClickHouse/issues/63115): ... [#63108](https://github.com/ClickHouse/ClickHouse/pull/63108) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
|
||||
#### NO CL ENTRY
|
||||
|
||||
* NO CL ENTRY: 'Revert "Backport [#61479](https://github.com/ClickHouse/ClickHouse/issues/61479) to 24.2: Fix bug when reading system.parts using UUID (issue 61220)."'. [#61776](https://github.com/ClickHouse/ClickHouse/pull/61776) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
|
||||
#### NOT FOR CHANGELOG / INSIGNIFICANT
|
||||
|
||||
* Throw on query timeout in ZooKeeperRetries [#60922](https://github.com/ClickHouse/ClickHouse/pull/60922) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
|
||||
76
docs/changelogs/v24.3.3.102-lts.md
Normal file
76
docs/changelogs/v24.3.3.102-lts.md
Normal file
@ -0,0 +1,76 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: 2024
|
||||
---
|
||||
|
||||
# 2024 Changelog
|
||||
|
||||
### ClickHouse release v24.3.3.102-lts (7e7f3bdd9be) FIXME as compared to v24.3.2.23-lts (8b7d910960c)
|
||||
|
||||
#### Improvement
|
||||
* Backported in [#62188](https://github.com/ClickHouse/ClickHouse/issues/62188): StorageJoin with strictness `ANY` is consistent after reload. When several rows with the same key are inserted, the first one will have higher priority (before, it was chosen randomly upon table loading). close [#51027](https://github.com/ClickHouse/ClickHouse/issues/51027). [#61972](https://github.com/ClickHouse/ClickHouse/pull/61972) ([vdimir](https://github.com/vdimir)).
|
||||
* Backported in [#62443](https://github.com/ClickHouse/ClickHouse/issues/62443): Client has to send header 'Keep-Alive: timeout=X' to the server. If a client receives a response from the server with that header, client has to use the value from the server. Also for a client it is better not to use a connection which is nearly expired in order to avoid connection close race. [#62249](https://github.com/ClickHouse/ClickHouse/pull/62249) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Backported in [#62666](https://github.com/ClickHouse/ClickHouse/issues/62666): S3 storage and backups also need the same default keep alive settings as s3 disk. [#62648](https://github.com/ClickHouse/ClickHouse/pull/62648) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
* Backported in [#62013](https://github.com/ClickHouse/ClickHouse/issues/62013): Remove from the Keeper Docker image the volumes at /etc/clickhouse-keeper and /var/log/clickhouse-keeper. [#61683](https://github.com/ClickHouse/ClickHouse/pull/61683) ([Tristan](https://github.com/Tristan971)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
|
||||
* Cancel merges before removing moved parts [#61610](https://github.com/ClickHouse/ClickHouse/pull/61610) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Mark CANNOT_PARSE_ESCAPE_SEQUENCE error as parse error to be able to skip it in row input formats [#61883](https://github.com/ClickHouse/ClickHouse/pull/61883) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Crash in Engine Merge if Row Policy does not have expression [#61971](https://github.com/ClickHouse/ClickHouse/pull/61971) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* ReadWriteBufferFromHTTP set right header host when redirected [#62068](https://github.com/ClickHouse/ClickHouse/pull/62068) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Analyzer: Fix query parameter resolution [#62186](https://github.com/ClickHouse/ClickHouse/pull/62186) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fixing NULL random seed for generateRandom with analyzer. [#62248](https://github.com/ClickHouse/ClickHouse/pull/62248) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix PartsSplitter [#62268](https://github.com/ClickHouse/ClickHouse/pull/62268) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Analyzer: Fix alias to parametrized view resolution [#62274](https://github.com/ClickHouse/ClickHouse/pull/62274) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Analyzer: Fix name resolution from parent scopes [#62281](https://github.com/ClickHouse/ClickHouse/pull/62281) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix argMax with nullable non native numeric column [#62285](https://github.com/ClickHouse/ClickHouse/pull/62285) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix data race on scalars in Context [#62305](https://github.com/ClickHouse/ClickHouse/pull/62305) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix analyzer with positional arguments in distributed query [#62362](https://github.com/ClickHouse/ClickHouse/pull/62362) ([flynn](https://github.com/ucasfl)).
|
||||
* Fix filter pushdown from additional_table_filters in Merge engine in analyzer [#62398](https://github.com/ClickHouse/ClickHouse/pull/62398) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix GLOBAL IN table queries with analyzer. [#62409](https://github.com/ClickHouse/ClickHouse/pull/62409) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix scalar subquery in LIMIT [#62567](https://github.com/ClickHouse/ClickHouse/pull/62567) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Try to fix segfault in Hive engine [#62578](https://github.com/ClickHouse/ClickHouse/pull/62578) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix memory leak in groupArraySorted [#62597](https://github.com/ClickHouse/ClickHouse/pull/62597) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix argMin/argMax combinator state [#62708](https://github.com/ClickHouse/ClickHouse/pull/62708) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix temporary data in cache failing because of cache lock contention optimization [#62715](https://github.com/ClickHouse/ClickHouse/pull/62715) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix FINAL modifier is not respected in CTE with analyzer [#62811](https://github.com/ClickHouse/ClickHouse/pull/62811) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix crash in function `formatRow` with `JSON` format and HTTP interface [#62840](https://github.com/ClickHouse/ClickHouse/pull/62840) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix GCD codec [#62853](https://github.com/ClickHouse/ClickHouse/pull/62853) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Disable optimize_rewrite_aggregate_function_with_if for sum(nullable) [#62912](https://github.com/ClickHouse/ClickHouse/pull/62912) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix temporary data in cache incorrectly processing failure of cache key directory creation [#62925](https://github.com/ClickHouse/ClickHouse/pull/62925) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix optimize_rewrite_aggregate_function_with_if implicit cast [#62999](https://github.com/ClickHouse/ClickHouse/pull/62999) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Do not remove server constants from GROUP BY key for secondary query. [#63047](https://github.com/ClickHouse/ClickHouse/pull/63047) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix incorrect judgement of of monotonicity of function abs [#63097](https://github.com/ClickHouse/ClickHouse/pull/63097) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Set server name for SSL handshake in MongoDB engine [#63122](https://github.com/ClickHouse/ClickHouse/pull/63122) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Use user specified db instead of "config" for MongoDB wire protocol version check [#63126](https://github.com/ClickHouse/ClickHouse/pull/63126) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Format SQL security option only in `CREATE VIEW` queries. [#63136](https://github.com/ClickHouse/ClickHouse/pull/63136) ([pufit](https://github.com/pufit)).
|
||||
|
||||
#### CI Fix or Improvement (changelog entry is not required)
|
||||
|
||||
* Backported in [#62802](https://github.com/ClickHouse/ClickHouse/issues/62802): We won't fail the job when GH fails to retrieve the job ID and URLs. [#62651](https://github.com/ClickHouse/ClickHouse/pull/62651) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#62834](https://github.com/ClickHouse/ClickHouse/issues/62834): Add `isort` config fo the first-party imports; fail build reports on non-success statuses. [#62786](https://github.com/ClickHouse/ClickHouse/pull/62786) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#62879](https://github.com/ClickHouse/ClickHouse/issues/62879):. [#62835](https://github.com/ClickHouse/ClickHouse/pull/62835) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Backported in [#62971](https://github.com/ClickHouse/ClickHouse/issues/62971):. [#62932](https://github.com/ClickHouse/ClickHouse/pull/62932) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Backported in [#63064](https://github.com/ClickHouse/ClickHouse/issues/63064): ... [#63035](https://github.com/ClickHouse/ClickHouse/pull/63035) ([Aleksei Filatov](https://github.com/aalexfvk)).
|
||||
* Backported in [#63117](https://github.com/ClickHouse/ClickHouse/issues/63117): ... [#63108](https://github.com/ClickHouse/ClickHouse/pull/63108) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
|
||||
#### NO CL CATEGORY
|
||||
|
||||
* Backported in [#62572](https://github.com/ClickHouse/ClickHouse/issues/62572):. [#62549](https://github.com/ClickHouse/ClickHouse/pull/62549) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
|
||||
#### NOT FOR CHANGELOG / INSIGNIFICANT
|
||||
|
||||
* Fix another logical error in group_by_use_nulls. [#62236](https://github.com/ClickHouse/ClickHouse/pull/62236) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix lambda(tuple(x), x + 1) syntax in analyzer [#62253](https://github.com/ClickHouse/ClickHouse/pull/62253) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix __actionName, add tests for internal functions direct call [#62287](https://github.com/ClickHouse/ClickHouse/pull/62287) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix optimize_uniq_to_count when only prefix of key is matched [#62325](https://github.com/ClickHouse/ClickHouse/pull/62325) ([vdimir](https://github.com/vdimir)).
|
||||
* Analyzer: Fix PREWHERE with lambda functions [#62336](https://github.com/ClickHouse/ClickHouse/pull/62336) ([vdimir](https://github.com/vdimir)).
|
||||
* Use function isNotDistinctFrom only in join key [#62387](https://github.com/ClickHouse/ClickHouse/pull/62387) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix integration-tests logs compression [#62556](https://github.com/ClickHouse/ClickHouse/pull/62556) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Try to fix if_transform_strings_to_enum performance test [#62558](https://github.com/ClickHouse/ClickHouse/pull/62558) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix shellcheck style checking and issues [#62761](https://github.com/ClickHouse/ClickHouse/pull/62761) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Fix flaky 03128_argMin_combinator_projection [#62965](https://github.com/ClickHouse/ClickHouse/pull/62965) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
|
||||
403
docs/changelogs/v24.4.1.2088-stable.md
Normal file
403
docs/changelogs/v24.4.1.2088-stable.md
Normal file
@ -0,0 +1,403 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: 2024
|
||||
---
|
||||
|
||||
# 2024 Changelog
|
||||
|
||||
### ClickHouse release v24.4.1.2088-stable (6d4b31322d1) FIXME as compared to v24.3.1.2672-lts (2c5c589a882)
|
||||
|
||||
#### Backward Incompatible Change
|
||||
* Don't allow to set max_parallel_replicas to 0 as it doesn't make sense. Setting it to 0 could lead to unexpected logical errors. Closes [#60140](https://github.com/ClickHouse/ClickHouse/issues/60140). [#61201](https://github.com/ClickHouse/ClickHouse/pull/61201) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* `clickhouse-odbc-bridge` and `clickhouse-library-bridge` are separate packages. This closes [#61677](https://github.com/ClickHouse/ClickHouse/issues/61677). [#62114](https://github.com/ClickHouse/ClickHouse/pull/62114) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Remove support for INSERT WATCH query (part of the experimental LIVE VIEW feature). [#62382](https://github.com/ClickHouse/ClickHouse/pull/62382) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Remove optimize_monotonous_functions_in_order_by setting. [#63004](https://github.com/ClickHouse/ClickHouse/pull/63004) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
|
||||
#### New Feature
|
||||
* Supports dropping multiple tables at the same time like `drop table a,b,c`;. [#58705](https://github.com/ClickHouse/ClickHouse/pull/58705) ([zhongyuankai](https://github.com/zhongyuankai)).
|
||||
* Table engine is grantable now, and it won't affect existing users behavior. [#60117](https://github.com/ClickHouse/ClickHouse/pull/60117) ([jsc0218](https://github.com/jsc0218)).
|
||||
* Added a rewritable S3 disk which supports INSERT operations and does not require locally stored metadata. [#61116](https://github.com/ClickHouse/ClickHouse/pull/61116) ([Julia Kartseva](https://github.com/jkartseva)).
|
||||
* For convenience purpose, `SELECT * FROM numbers() `will work in the same way as `SELECT * FROM system.numbers` - without a limit. [#61969](https://github.com/ClickHouse/ClickHouse/pull/61969) ([YenchangChan](https://github.com/YenchangChan)).
|
||||
* Modifying memory table settings through `ALTER MODIFY SETTING` is now supported. ``` ALTER TABLE memory MODIFY SETTING min_rows_to_keep = 100, max_rows_to_keep = 1000; ```. [#62039](https://github.com/ClickHouse/ClickHouse/pull/62039) ([zhongyuankai](https://github.com/zhongyuankai)).
|
||||
* Analyzer support recursive CTEs. [#62074](https://github.com/ClickHouse/ClickHouse/pull/62074) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Analyzer support QUALIFY clause. Closes [#47819](https://github.com/ClickHouse/ClickHouse/issues/47819). [#62619](https://github.com/ClickHouse/ClickHouse/pull/62619) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Added `role` query parameter to the HTTP interface. It works similarly to `SET ROLE x`, applying the role before the statement is executed. This allows for overcoming the limitation of the HTTP interface, as multiple statements are not allowed, and it is not possible to send both `SET ROLE x` and the statement itself at the same time. It is possible to set multiple roles that way, e.g., `?role=x&role=y`, which will be an equivalent of `SET ROLE x, y`. [#62669](https://github.com/ClickHouse/ClickHouse/pull/62669) ([Serge Klochkov](https://github.com/slvrtrn)).
|
||||
* Add `SYSTEM UNLOAD PRIMARY KEY`. [#62738](https://github.com/ClickHouse/ClickHouse/pull/62738) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
|
||||
#### Performance Improvement
|
||||
* Reduce overhead of the mutations for SELECTs (v2). [#60856](https://github.com/ClickHouse/ClickHouse/pull/60856) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* More frequently invoked functions in PODArray are now force-inlined. [#61144](https://github.com/ClickHouse/ClickHouse/pull/61144) ([李扬](https://github.com/taiyang-li)).
|
||||
* JOIN filter push down improvements using equivalent sets. [#61216](https://github.com/ClickHouse/ClickHouse/pull/61216) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Enabled fast Parquet encoder by default (output_format_parquet_use_custom_encoder). [#62088](https://github.com/ClickHouse/ClickHouse/pull/62088) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* ... When all required fields are read, skip all remaining fields directly which can save a lot of comparison. [#62210](https://github.com/ClickHouse/ClickHouse/pull/62210) ([lgbo](https://github.com/lgbo-ustc)).
|
||||
* Functions `splitByChar` and `splitByRegexp` were speed up significantly. [#62392](https://github.com/ClickHouse/ClickHouse/pull/62392) ([李扬](https://github.com/taiyang-li)).
|
||||
* Improve trivial insert select from files in file/s3/hdfs/url/... table functions. Add separate max_parsing_threads setting to control the number of threads used in parallel parsing. [#62404](https://github.com/ClickHouse/ClickHouse/pull/62404) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support parallel write buffer for AzureBlobStorage managed by setting `azure_allow_parallel_part_upload`. [#62534](https://github.com/ClickHouse/ClickHouse/pull/62534) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Functions `to_utc_timestamp` and `from_utc_timestamp` are now about 2x faster. [#62583](https://github.com/ClickHouse/ClickHouse/pull/62583) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* Functions `parseDateTimeOrNull`, `parseDateTimeOrZero`, `parseDateTimeInJodaSyntaxOrNull` and `parseDateTimeInJodaSyntaxOrZero` now run significantly faster (10x - 1000x) when the input contains mostly non-parseable values. [#62634](https://github.com/ClickHouse/ClickHouse/pull/62634) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
* SELECTs against `system.query_cache` are now noticeably faster when the query cache contains lots of entries (e.g. more than 100.000). [#62671](https://github.com/ClickHouse/ClickHouse/pull/62671) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* QueryPlan convert OUTER JOIN to INNER JOIN optimization if filter after JOIN always filters default values. Optimization can be controlled with setting `query_plan_convert_outer_join_to_inner_join`, enabled by default. [#62907](https://github.com/ClickHouse/ClickHouse/pull/62907) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Enable optimize_rewrite_sum_if_to_count_if by default. [#62929](https://github.com/ClickHouse/ClickHouse/pull/62929) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
|
||||
#### Improvement
|
||||
* Introduce separate consumer/producer tags for the Kafka configuration. This avoids warnings from librdkafka that consumer properties were specified for producer instances and vice versa (e.g. `Configuration property session.timeout.ms is a consumer property and will be ignored by this producer instance`). Closes: [#58983](https://github.com/ClickHouse/ClickHouse/issues/58983). [#58956](https://github.com/ClickHouse/ClickHouse/pull/58956) ([Aleksandr Musorin](https://github.com/AVMusorin)).
|
||||
* Added `value1`, `value2`, ..., `value10` columns to `system.text_log`. These columns contain values that were used to format the message. [#59619](https://github.com/ClickHouse/ClickHouse/pull/59619) ([Alexey Katsman](https://github.com/alexkats)).
|
||||
* Add a setting `first_day_of_week` which affects the first day of the week considered by functions `toStartOfInterval(..., INTERVAL ... WEEK)`. This allows for consistency with function `toStartOfWeek` which defaults to Sunday as the first day of the week. [#60598](https://github.com/ClickHouse/ClickHouse/pull/60598) ([Jordi Villar](https://github.com/jrdi)).
|
||||
* Added persistent virtual column `_block_offset` which stores original number of row in block that was assigned at insert. Persistence of column `_block_offset` can be enabled by setting `enable_block_offset_column`. Added virtual column`_part_data_version` which contains either min block number or mutation version of part. Persistent virtual column `_block_number` is not considered experimental anymore. [#60676](https://github.com/ClickHouse/ClickHouse/pull/60676) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Less contention in filesystem cache (part 3): execute removal from filesystem without lock on space reservation attempt. [#61163](https://github.com/ClickHouse/ClickHouse/pull/61163) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Functions `date_diff` and `age` now calculate their result at nanosecond instead of microsecond precision. They now also offer `nanosecond` (or `nanoseconds` or `ns`) as a possible value for the `unit` parameter. [#61409](https://github.com/ClickHouse/ClickHouse/pull/61409) ([Austin Kothig](https://github.com/kothiga)).
|
||||
* Now marks are not loaded for wide parts during merges. [#61551](https://github.com/ClickHouse/ClickHouse/pull/61551) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Reload certificate chain during certificate reload. [#61671](https://github.com/ClickHouse/ClickHouse/pull/61671) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)).
|
||||
* Speed up dynamic resize of filesystem cache. [#61723](https://github.com/ClickHouse/ClickHouse/pull/61723) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Add `TRUNCATE ALL TABLES`. [#61862](https://github.com/ClickHouse/ClickHouse/pull/61862) ([豪肥肥](https://github.com/HowePa)).
|
||||
* Try to prevent [#60432](https://github.com/ClickHouse/ClickHouse/issues/60432) by not allowing a table to be attached if there is an active replica for that replica path. [#61876](https://github.com/ClickHouse/ClickHouse/pull/61876) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Add a setting `input_format_json_throw_on_bad_escape_sequence`, disabling it allows saving bad escape sequences in JSON input formats. [#61889](https://github.com/ClickHouse/ClickHouse/pull/61889) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Userspace page cache works with static web storage (`disk(type = web)`) now. Use client setting `use_page_cache_for_disks_without_file_cache=1` to enable. [#61911](https://github.com/ClickHouse/ClickHouse/pull/61911) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* Implement input() for clickhouse-local. [#61923](https://github.com/ClickHouse/ClickHouse/pull/61923) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix logical-error when undoing quorum insert transaction. [#61953](https://github.com/ClickHouse/ClickHouse/pull/61953) ([Han Fei](https://github.com/hanfei1991)).
|
||||
* StorageJoin with strictness `ANY` is consistent after reload. When several rows with the same key are inserted, the first one will have higher priority (before, it was chosen randomly upon table loading). close [#51027](https://github.com/ClickHouse/ClickHouse/issues/51027). [#61972](https://github.com/ClickHouse/ClickHouse/pull/61972) ([vdimir](https://github.com/vdimir)).
|
||||
* Automatically infer Nullable column types from Apache Arrow schema. [#61984](https://github.com/ClickHouse/ClickHouse/pull/61984) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Allow to cancel parallel merge of aggregate states during aggregation. Example: `uniqExact`. [#61992](https://github.com/ClickHouse/ClickHouse/pull/61992) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Don't treat Bool and number variants as suspicious in Variant type. [#61999](https://github.com/ClickHouse/ClickHouse/pull/61999) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Use `system.keywords` to fill in the suggestions and also use them in the all places internally. [#62000](https://github.com/ClickHouse/ClickHouse/pull/62000) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Implement better conversion from String to Variant using parsing. [#62005](https://github.com/ClickHouse/ClickHouse/pull/62005) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support Variant in JSONExtract functions. [#62014](https://github.com/ClickHouse/ClickHouse/pull/62014) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Dictionary source with `INVALIDATE_QUERY` is not reloaded twice on startup. [#62050](https://github.com/ClickHouse/ClickHouse/pull/62050) ([vdimir](https://github.com/vdimir)).
|
||||
* `OPTIMIZE FINAL` for `ReplicatedMergeTree` now will wait for currently active merges to finish and then reattempt to schedule a final merge. This will put it more in line with ordinary `MergeTree` behaviour. [#62067](https://github.com/ClickHouse/ClickHouse/pull/62067) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* While read data from a hive text file, it would use the first line of hive text file to resize of number of input fields, and sometimes the fields number of first line is not matched with the hive table defined , such as the hive table is defined to have 3 columns, like `test_tbl(a Int32, b Int32, c Int32)`, but the first line of text file only has 2 fields, and in this suitation, the input fields will be resized to 2, and if the next line of the text file has 3 fields, then the third field can not be read but set a default value 0, which is not right. [#62086](https://github.com/ClickHouse/ClickHouse/pull/62086) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* CREATE AS copies the comment. [#62117](https://github.com/ClickHouse/ClickHouse/pull/62117) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
* The syntax highlighting while typing in the client will work on the syntax level (previously, it worked on the lexer level). [#62123](https://github.com/ClickHouse/ClickHouse/pull/62123) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix an issue where when a redundant `= 1` or `= 0` is added after a boolean expression involving the primary key, the primary index is not used. For example, both `SELECT * FROM <table> WHERE <primary-key> IN (<value>) = 1` and `SELECT * FROM <table> WHERE <primary-key> NOT IN (<value>) = 0` will both perform a full table scan, when the primary index can be used. [#62142](https://github.com/ClickHouse/ClickHouse/pull/62142) ([josh-hildred](https://github.com/josh-hildred)).
|
||||
* Add query progress to table zookeeper. [#62152](https://github.com/ClickHouse/ClickHouse/pull/62152) ([JackyWoo](https://github.com/JackyWoo)).
|
||||
* Add ability to turn on trace collector (Real and CPU) server-wide. [#62189](https://github.com/ClickHouse/ClickHouse/pull/62189) ([alesapin](https://github.com/alesapin)).
|
||||
* Added setting `lightweight_deletes_sync` (default value: 2 - wait all replicas synchronously). It is similar to setting `mutations_sync` but affects only behaviour of lightweight deletes. [#62195](https://github.com/ClickHouse/ClickHouse/pull/62195) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Distinguish booleans and integers while parsing values for custom settings: ``` SET custom_a = true; SET custom_b = 1; ```. [#62206](https://github.com/ClickHouse/ClickHouse/pull/62206) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Support S3 access through AWS Private Link Interface endpoints. Closes [#60021](https://github.com/ClickHouse/ClickHouse/issues/60021), [#31074](https://github.com/ClickHouse/ClickHouse/issues/31074) and [#53761](https://github.com/ClickHouse/ClickHouse/issues/53761). [#62208](https://github.com/ClickHouse/ClickHouse/pull/62208) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Client has to send header 'Keep-Alive: timeout=X' to the server. If a client receives a response from the server with that header, client has to use the value from the server. Also for a client it is better not to use a connection which is nearly expired in order to avoid connection close race. [#62249](https://github.com/ClickHouse/ClickHouse/pull/62249) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Added nano- micro- milliseconds unit for date_trunc. [#62335](https://github.com/ClickHouse/ClickHouse/pull/62335) ([Misz606](https://github.com/Misz606)).
|
||||
* Do not create a directory for UDF in clickhouse-client if it does not exist. This closes [#59597](https://github.com/ClickHouse/ClickHouse/issues/59597). [#62366](https://github.com/ClickHouse/ClickHouse/pull/62366) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* The query cache now no longer caches results of queries against system tables (`system.*`, `information_schema.*`, `INFORMATION_SCHEMA.*`). [#62376](https://github.com/ClickHouse/ClickHouse/pull/62376) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* `MOVE PARTITION TO TABLE` query can be delayed or can throw `TOO_MANY_PARTS` exception to avoid exceeding limits on the part count. The same settings and limits are applied as for the`INSERT` query (see `max_parts_in_total`, `parts_to_delay_insert`, `parts_to_throw_insert`, `inactive_parts_to_throw_insert`, `inactive_parts_to_delay_insert`, `max_avg_part_size_for_too_many_parts`, `min_delay_to_insert_ms` and `max_delay_to_insert` settings). [#62420](https://github.com/ClickHouse/ClickHouse/pull/62420) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Added the missing `hostname` column to system table `blob_storage_log`. [#62456](https://github.com/ClickHouse/ClickHouse/pull/62456) ([Jayme Bird](https://github.com/jaymebrd)).
|
||||
* Changed the default installation directory on macOS from `/usr/bin` to `/usr/local/bin`. This is necessary because Apple's System Integrity Protection introduced with macOS El Capitan (2015) prevents writing into `/usr/bin`, even with `sudo`. [#62489](https://github.com/ClickHouse/ClickHouse/pull/62489) ([haohang](https://github.com/yokofly)).
|
||||
* Make transform always return the first match. [#62518](https://github.com/ClickHouse/ClickHouse/pull/62518) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* For consistency with other system tables, `system.backup_log` now has a column `event_time`. [#62541](https://github.com/ClickHouse/ClickHouse/pull/62541) ([Jayme Bird](https://github.com/jaymebrd)).
|
||||
* Avoid evaluating table DEFAULT expressions while executing `RESTORE`. [#62601](https://github.com/ClickHouse/ClickHouse/pull/62601) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Return stream of chunks from `system.remote_data_paths` instead of accumulating the whole result in one big chunk. This allows to consume less memory, show intermediate progress and cancel the query. [#62613](https://github.com/ClickHouse/ClickHouse/pull/62613) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* S3 storage and backups also need the same default keep alive settings as s3 disk. [#62648](https://github.com/ClickHouse/ClickHouse/pull/62648) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Table `system.backup_log` now has the "default" sorting key which is `event_date, event_time`, the same as for other `_log` table engines. [#62667](https://github.com/ClickHouse/ClickHouse/pull/62667) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Mark type Variant as comparable so it can be used in primary key. [#62693](https://github.com/ClickHouse/ClickHouse/pull/62693) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add librdkafka's client identifier to log messages to be able to differentiate log messages from different consumers of a single table. [#62813](https://github.com/ClickHouse/ClickHouse/pull/62813) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Allow special macros {uuid} and {database} in a Replicated database ZooKeeper path. [#62818](https://github.com/ClickHouse/ClickHouse/pull/62818) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Allow quota key with different auth scheme in HTTP requests. [#62842](https://github.com/ClickHouse/ClickHouse/pull/62842) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Remove experimental tag from Replicated database engine. Now it is in Beta stage. [#62937](https://github.com/ClickHouse/ClickHouse/pull/62937) ([Justin de Guzman](https://github.com/justindeguzman)).
|
||||
* Reduce the verbosity of command line argument `--help` in `clickhouse client` and `clickhouse local`. The previous output is now generated by `--help --verbose`. [#62973](https://github.com/ClickHouse/ClickHouse/pull/62973) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Close session if [user's `valid_until`](https://clickhouse.com/docs/en/sql-reference/statements/create/user#valid-until-clause) is reached. [#63046](https://github.com/ClickHouse/ClickHouse/pull/63046) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* `log_bin_use_v1_row_events` was removed in MySQL 8.3, fix [#60479](https://github.com/ClickHouse/ClickHouse/issues/60479). [#63101](https://github.com/ClickHouse/ClickHouse/pull/63101) ([Eugene Klimov](https://github.com/Slach)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
* Ignore DROP queries in stress test with 1/2 probability, use TRUNCATE instead of ignoring DROP in upgrade check for Memory/JOIN tables. [#61476](https://github.com/ClickHouse/ClickHouse/pull/61476) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Remove from the Keeper Docker image the volumes at /etc/clickhouse-keeper and /var/log/clickhouse-keeper. [#61683](https://github.com/ClickHouse/ClickHouse/pull/61683) ([Tristan](https://github.com/Tristan971)).
|
||||
* Timeout was updated in https://github.com/ClickHouse/ClickHouse/pull/45765, but exception message was not. [#62139](https://github.com/ClickHouse/ClickHouse/pull/62139) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Add tests for all issues which are no longer relevant with Analyzer being enabled by default. Closes: [#55794](https://github.com/ClickHouse/ClickHouse/issues/55794) Closes: [#49472](https://github.com/ClickHouse/ClickHouse/issues/49472) Closes: [#44414](https://github.com/ClickHouse/ClickHouse/issues/44414) Closes: [#13843](https://github.com/ClickHouse/ClickHouse/issues/13843) Closes: [#55803](https://github.com/ClickHouse/ClickHouse/issues/55803) Closes: [#48308](https://github.com/ClickHouse/ClickHouse/issues/48308) Closes: [#45535](https://github.com/ClickHouse/ClickHouse/issues/45535) Closes: [#44365](https://github.com/ClickHouse/ClickHouse/issues/44365) Closes: [#44153](https://github.com/ClickHouse/ClickHouse/issues/44153) Closes: [#42399](https://github.com/ClickHouse/ClickHouse/issues/42399) Closes: [#27115](https://github.com/ClickHouse/ClickHouse/issues/27115) Closes: [#23162](https://github.com/ClickHouse/ClickHouse/issues/23162) Closes: [#15395](https://github.com/ClickHouse/ClickHouse/issues/15395) Closes: [#15411](https://github.com/ClickHouse/ClickHouse/issues/15411) Closes: [#14978](https://github.com/ClickHouse/ClickHouse/issues/14978) Closes: [#17319](https://github.com/ClickHouse/ClickHouse/issues/17319) Closes: [#11813](https://github.com/ClickHouse/ClickHouse/issues/11813) Closes: [#13210](https://github.com/ClickHouse/ClickHouse/issues/13210) Closes: [#23053](https://github.com/ClickHouse/ClickHouse/issues/23053) Closes: [#37729](https://github.com/ClickHouse/ClickHouse/issues/37729) Closes: [#32639](https://github.com/ClickHouse/ClickHouse/issues/32639) Closes: [#9954](https://github.com/ClickHouse/ClickHouse/issues/9954) Closes: [#41964](https://github.com/ClickHouse/ClickHouse/issues/41964) Closes: [#54317](https://github.com/ClickHouse/ClickHouse/issues/54317) Closes: [#7520](https://github.com/ClickHouse/ClickHouse/issues/7520) Closes: [#36973](https://github.com/ClickHouse/ClickHouse/issues/36973) Closes: [#40955](https://github.com/ClickHouse/ClickHouse/issues/40955) Closes: [#19687](https://github.com/ClickHouse/ClickHouse/issues/19687) Closes: [#23104](https://github.com/ClickHouse/ClickHouse/issues/23104) Closes: [#21584](https://github.com/ClickHouse/ClickHouse/issues/21584) Closes: [#23344](https://github.com/ClickHouse/ClickHouse/issues/23344) Closes: [#22627](https://github.com/ClickHouse/ClickHouse/issues/22627) Closes: [#10276](https://github.com/ClickHouse/ClickHouse/issues/10276) Closes: [#19687](https://github.com/ClickHouse/ClickHouse/issues/19687) Closes: [#4567](https://github.com/ClickHouse/ClickHouse/issues/4567) Closes: [#17710](https://github.com/ClickHouse/ClickHouse/issues/17710) Closes: [#11068](https://github.com/ClickHouse/ClickHouse/issues/11068) Closes: [#24395](https://github.com/ClickHouse/ClickHouse/issues/24395) Closes: [#23416](https://github.com/ClickHouse/ClickHouse/issues/23416) Closes: [#23162](https://github.com/ClickHouse/ClickHouse/issues/23162) Closes: [#25655](https://github.com/ClickHouse/ClickHouse/issues/25655) Closes: [#11757](https://github.com/ClickHouse/ClickHouse/issues/11757) Closes: [#6571](https://github.com/ClickHouse/ClickHouse/issues/6571) Closes: [#4432](https://github.com/ClickHouse/ClickHouse/issues/4432) Closes: [#8259](https://github.com/ClickHouse/ClickHouse/issues/8259) Closes: [#9233](https://github.com/ClickHouse/ClickHouse/issues/9233) Closes: [#14699](https://github.com/ClickHouse/ClickHouse/issues/14699) Closes: [#27068](https://github.com/ClickHouse/ClickHouse/issues/27068) Closes: [#28687](https://github.com/ClickHouse/ClickHouse/issues/28687) Closes: [#28777](https://github.com/ClickHouse/ClickHouse/issues/28777) Closes: [#29734](https://github.com/ClickHouse/ClickHouse/issues/29734) Closes: [#61238](https://github.com/ClickHouse/ClickHouse/issues/61238) Closes: [#33825](https://github.com/ClickHouse/ClickHouse/issues/33825) Closes: [#35608](https://github.com/ClickHouse/ClickHouse/issues/35608) Closes: [#29838](https://github.com/ClickHouse/ClickHouse/issues/29838) Closes: [#35652](https://github.com/ClickHouse/ClickHouse/issues/35652) Closes: [#36189](https://github.com/ClickHouse/ClickHouse/issues/36189) Closes: [#39634](https://github.com/ClickHouse/ClickHouse/issues/39634) Closes: [#47432](https://github.com/ClickHouse/ClickHouse/issues/47432) Closes: [#54910](https://github.com/ClickHouse/ClickHouse/issues/54910) Closes: [#57321](https://github.com/ClickHouse/ClickHouse/issues/57321) Closes: [#59154](https://github.com/ClickHouse/ClickHouse/issues/59154) Closes: [#61014](https://github.com/ClickHouse/ClickHouse/issues/61014) Closes: [#61950](https://github.com/ClickHouse/ClickHouse/issues/61950) Closes: [#55647](https://github.com/ClickHouse/ClickHouse/issues/55647) Closes: [#61947](https://github.com/ClickHouse/ClickHouse/issues/61947). [#62185](https://github.com/ClickHouse/ClickHouse/pull/62185) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Vendor in rust dependencies. [#62297](https://github.com/ClickHouse/ClickHouse/pull/62297) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Add more tests from issues which are no longer relevant or fixed by analyzer. Closes: [#58985](https://github.com/ClickHouse/ClickHouse/issues/58985) Closes: [#59549](https://github.com/ClickHouse/ClickHouse/issues/59549) Closes: [#36963](https://github.com/ClickHouse/ClickHouse/issues/36963) Closes: [#39453](https://github.com/ClickHouse/ClickHouse/issues/39453) Closes: [#56521](https://github.com/ClickHouse/ClickHouse/issues/56521) Closes: [#47552](https://github.com/ClickHouse/ClickHouse/issues/47552) Closes: [#56503](https://github.com/ClickHouse/ClickHouse/issues/56503) Closes: [#59101](https://github.com/ClickHouse/ClickHouse/issues/59101) Closes: [#50271](https://github.com/ClickHouse/ClickHouse/issues/50271) Closes: [#54954](https://github.com/ClickHouse/ClickHouse/issues/54954) Closes: [#56466](https://github.com/ClickHouse/ClickHouse/issues/56466) Closes: [#11000](https://github.com/ClickHouse/ClickHouse/issues/11000) Closes: [#10894](https://github.com/ClickHouse/ClickHouse/issues/10894) Closes: https://github.com/ClickHouse/ClickHouse/issues/448 Closes: [#8030](https://github.com/ClickHouse/ClickHouse/issues/8030) Closes: [#32139](https://github.com/ClickHouse/ClickHouse/issues/32139) Closes: [#47288](https://github.com/ClickHouse/ClickHouse/issues/47288) Closes: [#50705](https://github.com/ClickHouse/ClickHouse/issues/50705) Closes: [#54511](https://github.com/ClickHouse/ClickHouse/issues/54511) Closes: [#55466](https://github.com/ClickHouse/ClickHouse/issues/55466) Closes: [#58500](https://github.com/ClickHouse/ClickHouse/issues/58500) Closes: [#39923](https://github.com/ClickHouse/ClickHouse/issues/39923) Closes: [#39855](https://github.com/ClickHouse/ClickHouse/issues/39855) Closes: [#4596](https://github.com/ClickHouse/ClickHouse/issues/4596) Closes: [#47422](https://github.com/ClickHouse/ClickHouse/issues/47422) Closes: [#33000](https://github.com/ClickHouse/ClickHouse/issues/33000) Closes: [#14739](https://github.com/ClickHouse/ClickHouse/issues/14739) Closes: [#44039](https://github.com/ClickHouse/ClickHouse/issues/44039) Closes: [#8547](https://github.com/ClickHouse/ClickHouse/issues/8547) Closes: [#22923](https://github.com/ClickHouse/ClickHouse/issues/22923) Closes: [#23865](https://github.com/ClickHouse/ClickHouse/issues/23865) Closes: [#29748](https://github.com/ClickHouse/ClickHouse/issues/29748) Closes: [#4222](https://github.com/ClickHouse/ClickHouse/issues/4222). [#62457](https://github.com/ClickHouse/ClickHouse/pull/62457) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Fixed build errors when OpenSSL is linked dynamically (note: this is generally unsupported and only required for s390x platforms). [#62888](https://github.com/ClickHouse/ClickHouse/pull/62888) ([Harry Lee](https://github.com/HarryLeeIBM)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
|
||||
* Fix parser error when using COUNT(*) with FILTER clause [#61357](https://github.com/ClickHouse/ClickHouse/pull/61357) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix logical error in group_by_use_nulls + grouping set + analyzer + materialize/constant [#61567](https://github.com/ClickHouse/ClickHouse/pull/61567) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Cancel merges before removing moved parts [#61610](https://github.com/ClickHouse/ClickHouse/pull/61610) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Try to fix abort in arrow [#61720](https://github.com/ClickHouse/ClickHouse/pull/61720) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Search for convert_to_replicated flag at the correct path [#61769](https://github.com/ClickHouse/ClickHouse/pull/61769) ([Kirill](https://github.com/kirillgarbar)).
|
||||
* Fix possible connections data-race for distributed_foreground_insert/distributed_background_insert_batch [#61867](https://github.com/ClickHouse/ClickHouse/pull/61867) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Mark CANNOT_PARSE_ESCAPE_SEQUENCE error as parse error to be able to skip it in row input formats [#61883](https://github.com/ClickHouse/ClickHouse/pull/61883) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix writing exception message in output format in HTTP when http_wait_end_of_query is used [#61951](https://github.com/ClickHouse/ClickHouse/pull/61951) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Proper fix for LowCardinality together with JSONExtact functions [#61957](https://github.com/ClickHouse/ClickHouse/pull/61957) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Crash in Engine Merge if Row Policy does not have expression [#61971](https://github.com/ClickHouse/ClickHouse/pull/61971) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* Fix WriteBufferAzureBlobStorage destructor uncaught exception [#61988](https://github.com/ClickHouse/ClickHouse/pull/61988) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Fix CREATE TABLE w/o columns definition for ReplicatedMergeTree [#62040](https://github.com/ClickHouse/ClickHouse/pull/62040) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix optimize_skip_unused_shards_rewrite_in for composite sharding key [#62047](https://github.com/ClickHouse/ClickHouse/pull/62047) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* ReadWriteBufferFromHTTP set right header host when redirected [#62068](https://github.com/ClickHouse/ClickHouse/pull/62068) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Fix external table cannot parse data type Bool [#62115](https://github.com/ClickHouse/ClickHouse/pull/62115) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Revert "Merge pull request [#61564](https://github.com/ClickHouse/ClickHouse/issues/61564) from liuneng1994/optimize_in_single_value" [#62135](https://github.com/ClickHouse/ClickHouse/pull/62135) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Add test for [#35215](https://github.com/ClickHouse/ClickHouse/issues/35215) [#62180](https://github.com/ClickHouse/ClickHouse/pull/62180) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Analyzer: Fix query parameter resolution [#62186](https://github.com/ClickHouse/ClickHouse/pull/62186) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix restoring parts while readonly [#62207](https://github.com/ClickHouse/ClickHouse/pull/62207) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Fix crash in index definition containing sql udf [#62225](https://github.com/ClickHouse/ClickHouse/pull/62225) ([vdimir](https://github.com/vdimir)).
|
||||
* Fixing NULL random seed for generateRandom with analyzer. [#62248](https://github.com/ClickHouse/ClickHouse/pull/62248) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Correctly handle const columns in DistinctTransfom [#62250](https://github.com/ClickHouse/ClickHouse/pull/62250) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix PartsSplitter [#62268](https://github.com/ClickHouse/ClickHouse/pull/62268) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Analyzer: Fix alias to parametrized view resolution [#62274](https://github.com/ClickHouse/ClickHouse/pull/62274) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Analyzer: Fix name resolution from parent scopes [#62281](https://github.com/ClickHouse/ClickHouse/pull/62281) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix argMax with nullable non native numeric column [#62285](https://github.com/ClickHouse/ClickHouse/pull/62285) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix BACKUP and RESTORE of a materialized view in Ordinary database [#62295](https://github.com/ClickHouse/ClickHouse/pull/62295) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Fix data race on scalars in Context [#62305](https://github.com/ClickHouse/ClickHouse/pull/62305) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix primary key in materialized view [#62319](https://github.com/ClickHouse/ClickHouse/pull/62319) ([Murat Khairulin](https://github.com/mxwell)).
|
||||
* Do not build multithread insert pipeline for tables without support [#62333](https://github.com/ClickHouse/ClickHouse/pull/62333) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix analyzer with positional arguments in distributed query [#62362](https://github.com/ClickHouse/ClickHouse/pull/62362) ([flynn](https://github.com/ucasfl)).
|
||||
* Fix filter pushdown from additional_table_filters in Merge engine in analyzer [#62398](https://github.com/ClickHouse/ClickHouse/pull/62398) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix GLOBAL IN table queries with analyzer. [#62409](https://github.com/ClickHouse/ClickHouse/pull/62409) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Respect settings truncate_on_insert/create_new_file_on_insert in s3/hdfs/azure engines during partitioned write [#62425](https://github.com/ClickHouse/ClickHouse/pull/62425) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix backup restore path for AzureBlobStorage [#62447](https://github.com/ClickHouse/ClickHouse/pull/62447) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).
|
||||
* Fix SimpleSquashingChunksTransform [#62451](https://github.com/ClickHouse/ClickHouse/pull/62451) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix capture of nested lambda. [#62462](https://github.com/ClickHouse/ClickHouse/pull/62462) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix validation of special MergeTree columns [#62498](https://github.com/ClickHouse/ClickHouse/pull/62498) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Avoid crash when reading protobuf with recursive types [#62506](https://github.com/ClickHouse/ClickHouse/pull/62506) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix a bug moving one partition from one to itself [#62524](https://github.com/ClickHouse/ClickHouse/pull/62524) ([helifu](https://github.com/helifu)).
|
||||
* Fix scalar subquery in LIMIT [#62567](https://github.com/ClickHouse/ClickHouse/pull/62567) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Try to fix segfault in Hive engine [#62578](https://github.com/ClickHouse/ClickHouse/pull/62578) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix memory leak in groupArraySorted [#62597](https://github.com/ClickHouse/ClickHouse/pull/62597) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix crash in largestTriangleThreeBuckets [#62646](https://github.com/ClickHouse/ClickHouse/pull/62646) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix tumble[Start,End] and hop[Start,End] for bigger resolutions [#62705](https://github.com/ClickHouse/ClickHouse/pull/62705) ([Jordi Villar](https://github.com/jrdi)).
|
||||
* Fix argMin/argMax combinator state [#62708](https://github.com/ClickHouse/ClickHouse/pull/62708) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix temporary data in cache failing because of cache lock contention optimization [#62715](https://github.com/ClickHouse/ClickHouse/pull/62715) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix crash in function `mergeTreeIndex` [#62762](https://github.com/ClickHouse/ClickHouse/pull/62762) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* fix: update: nested materialized columns: size check fixes [#62773](https://github.com/ClickHouse/ClickHouse/pull/62773) ([Eliot Hautefeuille](https://github.com/hileef)).
|
||||
* Fix FINAL modifier is not respected in CTE with analyzer [#62811](https://github.com/ClickHouse/ClickHouse/pull/62811) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix crash in function `formatRow` with `JSON` format and HTTP interface [#62840](https://github.com/ClickHouse/ClickHouse/pull/62840) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Azure: fix building final url from endpoint object [#62850](https://github.com/ClickHouse/ClickHouse/pull/62850) ([Daniel Pozo Escalona](https://github.com/danipozo)).
|
||||
* Fix GCD codec [#62853](https://github.com/ClickHouse/ClickHouse/pull/62853) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix LowCardinality(Nullable) key in hyperrectangle [#62866](https://github.com/ClickHouse/ClickHouse/pull/62866) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix fromUnixtimestamp in joda syntax while the input value beyond UInt32 [#62901](https://github.com/ClickHouse/ClickHouse/pull/62901) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* Disable optimize_rewrite_aggregate_function_with_if for sum(nullable) [#62912](https://github.com/ClickHouse/ClickHouse/pull/62912) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix PREWHERE for StorageBuffer with different source table column types. [#62916](https://github.com/ClickHouse/ClickHouse/pull/62916) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix temporary data in cache incorrectly processing failure of cache key directory creation [#62925](https://github.com/ClickHouse/ClickHouse/pull/62925) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* gRPC: fix crash on IPv6 peer connection [#62978](https://github.com/ClickHouse/ClickHouse/pull/62978) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Fix possible CHECKSUM_DOESNT_MATCH (and others) during replicated fetches [#62987](https://github.com/ClickHouse/ClickHouse/pull/62987) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix terminate with uncaught exception in temporary data in cache [#62998](https://github.com/ClickHouse/ClickHouse/pull/62998) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix optimize_rewrite_aggregate_function_with_if implicit cast [#62999](https://github.com/ClickHouse/ClickHouse/pull/62999) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix unhandled exception in ~RestorerFromBackup [#63040](https://github.com/ClickHouse/ClickHouse/pull/63040) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Do not remove server constants from GROUP BY key for secondary query. [#63047](https://github.com/ClickHouse/ClickHouse/pull/63047) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix incorrect judgement of of monotonicity of function abs [#63097](https://github.com/ClickHouse/ClickHouse/pull/63097) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Make sanity check of settings worse [#63119](https://github.com/ClickHouse/ClickHouse/pull/63119) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Set server name for SSL handshake in MongoDB engine [#63122](https://github.com/ClickHouse/ClickHouse/pull/63122) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Use user specified db instead of "config" for MongoDB wire protocol version check [#63126](https://github.com/ClickHouse/ClickHouse/pull/63126) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Format SQL security option only in `CREATE VIEW` queries. [#63136](https://github.com/ClickHouse/ClickHouse/pull/63136) ([pufit](https://github.com/pufit)).
|
||||
|
||||
#### CI Fix or Improvement (changelog entry is not required)
|
||||
|
||||
* ... [#62044](https://github.com/ClickHouse/ClickHouse/pull/62044) ([Max K.](https://github.com/maxknv)).
|
||||
* We won't fail the job when GH fails to retrieve the job ID and URLs. [#62651](https://github.com/ClickHouse/ClickHouse/pull/62651) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Decouple some work from https://github.com/ClickHouse/ClickHouse/pull/61464 to simplify sync. [#62739](https://github.com/ClickHouse/ClickHouse/pull/62739) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Add `isort` config fo the first-party imports; fail build reports on non-success statuses. [#62786](https://github.com/ClickHouse/ClickHouse/pull/62786) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Move all Labels around to have it in a single place. [#62919](https://github.com/ClickHouse/ClickHouse/pull/62919) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* ... [#63035](https://github.com/ClickHouse/ClickHouse/pull/63035) ([Aleksei Filatov](https://github.com/aalexfvk)).
|
||||
* ... [#63108](https://github.com/ClickHouse/ClickHouse/pull/63108) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
|
||||
#### NO CL ENTRY
|
||||
|
||||
* NO CL ENTRY: 'Revert "Revert "Updated format settings references in the docs (datetime.md)""'. [#61442](https://github.com/ClickHouse/ClickHouse/pull/61442) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* NO CL ENTRY: 'Write `binary version -> commit hash` mapping to CI database (in private)'. [#61544](https://github.com/ClickHouse/ClickHouse/pull/61544) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* NO CL ENTRY: 'Fix flaky tests 2 (stateless, integration) '. [#61869](https://github.com/ClickHouse/ClickHouse/pull/61869) ([Nikita Fomichev](https://github.com/fm4v)).
|
||||
* NO CL ENTRY: 'Fix PR [#60656](https://github.com/ClickHouse/ClickHouse/issues/60656) for install check tests'. [#61910](https://github.com/ClickHouse/ClickHouse/pull/61910) ([Chun-Sheng, Li](https://github.com/peter279k)).
|
||||
* NO CL ENTRY: '00002_log_and_exception_messages_formatting: exclude one more format string'. [#62190](https://github.com/ClickHouse/ClickHouse/pull/62190) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* NO CL ENTRY: 'Revert "Resubmit 'Update invalidate_query_response on dictionary startup'"'. [#62230](https://github.com/ClickHouse/ClickHouse/pull/62230) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* NO CL ENTRY: 'Fix contributor name vulnerability'. [#62357](https://github.com/ClickHouse/ClickHouse/pull/62357) ([Anita Hammer](https://github.com/anitahammer)).
|
||||
* NO CL ENTRY: 'Revert "Rich syntax highlighting in the client"'. [#62508](https://github.com/ClickHouse/ClickHouse/pull/62508) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* NO CL ENTRY: 'Revert "Revert "Rich syntax highlighting in the client""'. [#62512](https://github.com/ClickHouse/ClickHouse/pull/62512) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* NO CL ENTRY: 'Revert "[feature]: allow to attach parts from a different disk"'. [#62549](https://github.com/ClickHouse/ClickHouse/pull/62549) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* NO CL ENTRY: 'Revert "More optimal loading of marks"'. [#62577](https://github.com/ClickHouse/ClickHouse/pull/62577) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* NO CL ENTRY: 'Revert "Speed up `splitByRegexp`"'. [#62692](https://github.com/ClickHouse/ClickHouse/pull/62692) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* NO CL ENTRY: 'Get rid of merge_commit in style check autofix'. [#62835](https://github.com/ClickHouse/ClickHouse/pull/62835) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* NO CL ENTRY: 'Revert "CI: add FT to MQ remove Style from master"'. [#62927](https://github.com/ClickHouse/ClickHouse/pull/62927) ([Max K.](https://github.com/maxknv)).
|
||||
* NO CL ENTRY: 'Unflake 02813_func_now_and_alias'. [#62932](https://github.com/ClickHouse/ClickHouse/pull/62932) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* NO CL ENTRY: 'Revert "Enable custom parquet encoder by default"'. [#63153](https://github.com/ClickHouse/ClickHouse/pull/63153) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
|
||||
#### NOT FOR CHANGELOG / INSIGNIFICANT
|
||||
|
||||
* Update protobuf to v25.1 [#58020](https://github.com/ClickHouse/ClickHouse/pull/58020) ([Mikhail Koviazin](https://github.com/mkmkme)).
|
||||
* boringssl --> OpenSSL 3.2 [#59870](https://github.com/ClickHouse/ClickHouse/pull/59870) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Enable all access control improvements by default (even without config.xml) [#60153](https://github.com/ClickHouse/ClickHouse/pull/60153) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Change back how receive_timeout is handled for INSERTs [#60302](https://github.com/ClickHouse/ClickHouse/pull/60302) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Context getGlobalTemporaryVolume use shared lock [#60997](https://github.com/ClickHouse/ClickHouse/pull/60997) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Do nothing in `waitForOutdatedPartsToBeLoaded()` if loading is not required [#61232](https://github.com/ClickHouse/ClickHouse/pull/61232) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Fix db iterator wait during async metrics collection [#61534](https://github.com/ClickHouse/ClickHouse/pull/61534) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Fix 02943_rmt_alter_metadata_merge_checksum_mismatch flakiness [#61594](https://github.com/ClickHouse/ClickHouse/pull/61594) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Stream rows when reading from system.replicas [#61784](https://github.com/ClickHouse/ClickHouse/pull/61784) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Skip more sanity checks for secondary create queries [#61799](https://github.com/ClickHouse/ClickHouse/pull/61799) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Fix 00002_log_and_exception_messages_formatting [#61882](https://github.com/ClickHouse/ClickHouse/pull/61882) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Add test for [#53352](https://github.com/ClickHouse/ClickHouse/issues/53352) [#61886](https://github.com/ClickHouse/ClickHouse/pull/61886) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Test: tuple elimination with analyzer [#61887](https://github.com/ClickHouse/ClickHouse/pull/61887) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Fix performance test `aggregating_merge_tree_simple_aggregate_function_string` [#61931](https://github.com/ClickHouse/ClickHouse/pull/61931) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Fix some crashes with analyzer and group_by_use_nulls. [#61933](https://github.com/ClickHouse/ClickHouse/pull/61933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* fix a race in refreshable view [#61936](https://github.com/ClickHouse/ClickHouse/pull/61936) ([Han Fei](https://github.com/hanfei1991)).
|
||||
* Follow up to [#60452](https://github.com/ClickHouse/ClickHouse/issues/60452) [#61954](https://github.com/ClickHouse/ClickHouse/pull/61954) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Update 02916_move_partition_inactive_replica.sql [#61955](https://github.com/ClickHouse/ClickHouse/pull/61955) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Check for "SYSTEM STOP MERGES" primarily for MERGE_PARTS/MUTATE_PART [#61976](https://github.com/ClickHouse/ClickHouse/pull/61976) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* CI: failover for job_url request from gh [#61986](https://github.com/ClickHouse/ClickHouse/pull/61986) ([Max K.](https://github.com/maxknv)).
|
||||
* CI: remove unnecessary job url for Mark release ready [#61991](https://github.com/ClickHouse/ClickHouse/pull/61991) ([Max K.](https://github.com/maxknv)).
|
||||
* Update version after release [#61994](https://github.com/ClickHouse/ClickHouse/pull/61994) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Update version_date.tsv and changelogs after v24.3.1.2672-lts [#61996](https://github.com/ClickHouse/ClickHouse/pull/61996) ([robot-clickhouse](https://github.com/robot-clickhouse)).
|
||||
* [RFC] Send LOGICAL_ERRORs to sentry [#61997](https://github.com/ClickHouse/ClickHouse/pull/61997) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix scalars create as select [#61998](https://github.com/ClickHouse/ClickHouse/pull/61998) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix clickhouse-test [#62016](https://github.com/ClickHouse/ClickHouse/pull/62016) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix logs saving in DatabaseReplicated tests [#62019](https://github.com/ClickHouse/ClickHouse/pull/62019) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* fix npy big endianness [#62020](https://github.com/ClickHouse/ClickHouse/pull/62020) ([豪肥肥](https://github.com/HowePa)).
|
||||
* Update analyzer_tech_debt.txt [#62035](https://github.com/ClickHouse/ClickHouse/pull/62035) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Add analyzer pattern to 00002_log_and_exception_messages_formatting [#62038](https://github.com/ClickHouse/ClickHouse/pull/62038) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Fix clickhouse-test in case of missing .reference file [#62041](https://github.com/ClickHouse/ClickHouse/pull/62041) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix optimize_arithmetic_operations_in_aggregate_functions [#62046](https://github.com/ClickHouse/ClickHouse/pull/62046) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Update DatabaseOnDisk.cpp [#62049](https://github.com/ClickHouse/ClickHouse/pull/62049) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Ignore IfChainToMultiIfPass if returned type changed. [#62059](https://github.com/ClickHouse/ClickHouse/pull/62059) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Support more that 255 replicas in system table [#62064](https://github.com/ClickHouse/ClickHouse/pull/62064) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Fix stress tests for analyzer due to experimental WINDOW VIEW (by disabling it) [#62065](https://github.com/ClickHouse/ClickHouse/pull/62065) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix type for ConvertInToEqualPass [#62066](https://github.com/ClickHouse/ClickHouse/pull/62066) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Remove Dead Code [#62082](https://github.com/ClickHouse/ClickHouse/pull/62082) ([jsc0218](https://github.com/jsc0218)).
|
||||
* Revert output Pretty in tty [#62090](https://github.com/ClickHouse/ClickHouse/pull/62090) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* More than 255 replicas in ReplicatedTableStatus [#62127](https://github.com/ClickHouse/ClickHouse/pull/62127) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Fix upgrade check [#62136](https://github.com/ClickHouse/ClickHouse/pull/62136) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix 0320_long_values_pretty_are_not_cut_if_single [#62150](https://github.com/ClickHouse/ClickHouse/pull/62150) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Update NuRaft [#62156](https://github.com/ClickHouse/ClickHouse/pull/62156) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Unify lightweight mutation control [#62159](https://github.com/ClickHouse/ClickHouse/pull/62159) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Add some logging [#62160](https://github.com/ClickHouse/ClickHouse/pull/62160) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix xml part in documentation [#62169](https://github.com/ClickHouse/ClickHouse/pull/62169) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Remove a few nested include dependencies [#62170](https://github.com/ClickHouse/ClickHouse/pull/62170) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* User specific S3 endpoint for Backup/Restore on cluster [#62175](https://github.com/ClickHouse/ClickHouse/pull/62175) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Bump `double-conversion` submodule [#62177](https://github.com/ClickHouse/ClickHouse/pull/62177) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Fix `retention` docs [#62182](https://github.com/ClickHouse/ClickHouse/pull/62182) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Fix 02503_insert_storage_snapshot [#62194](https://github.com/ClickHouse/ClickHouse/pull/62194) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Use ClickHouse threads in NuRaft [#62196](https://github.com/ClickHouse/ClickHouse/pull/62196) ([alesapin](https://github.com/alesapin)).
|
||||
* Unflake and speed up `01676_clickhouse_client_autocomplete` [#62209](https://github.com/ClickHouse/ClickHouse/pull/62209) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Fix build with clang-19 (master) [#62212](https://github.com/ClickHouse/ClickHouse/pull/62212) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add more documentation to the release script [#62213](https://github.com/ClickHouse/ClickHouse/pull/62213) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Update version_date.tsv and changelogs after v24.3.2.23-lts [#62214](https://github.com/ClickHouse/ClickHouse/pull/62214) ([robot-clickhouse](https://github.com/robot-clickhouse)).
|
||||
* Unlimited output_format_pretty_max_value_width for --pager [#62221](https://github.com/ClickHouse/ClickHouse/pull/62221) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Include table name in paranoid checks [#62232](https://github.com/ClickHouse/ClickHouse/pull/62232) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix another logical error in group_by_use_nulls. [#62236](https://github.com/ClickHouse/ClickHouse/pull/62236) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Remove reverted PR from 24.3 changelog [#62251](https://github.com/ClickHouse/ClickHouse/pull/62251) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix lambda(tuple(x), x + 1) syntax in analyzer [#62253](https://github.com/ClickHouse/ClickHouse/pull/62253) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix s3-style link mapper for gcs [#62257](https://github.com/ClickHouse/ClickHouse/pull/62257) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Disable 02980_dist_insert_readonly_replica for SMT [#62260](https://github.com/ClickHouse/ClickHouse/pull/62260) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Fix logical error from fs cache in stress test [#62261](https://github.com/ClickHouse/ClickHouse/pull/62261) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Remove more nested includes [#62264](https://github.com/ClickHouse/ClickHouse/pull/62264) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Don't access static members through instance [#62265](https://github.com/ClickHouse/ClickHouse/pull/62265) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add fault injection for "Cannot allocate thread" [#62266](https://github.com/ClickHouse/ClickHouse/pull/62266) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Analyzer: limit maximal size of column in constant folding [#62273](https://github.com/ClickHouse/ClickHouse/pull/62273) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix __actionName, add tests for internal functions direct call [#62287](https://github.com/ClickHouse/ClickHouse/pull/62287) ([vdimir](https://github.com/vdimir)).
|
||||
* Fix `mortonEncode` `use-of-uninitialized-value` [#62288](https://github.com/ClickHouse/ClickHouse/pull/62288) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Add local address to network exception messages [#62300](https://github.com/ClickHouse/ClickHouse/pull/62300) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Better handling of errors from azure storage [#62306](https://github.com/ClickHouse/ClickHouse/pull/62306) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Cleanup SSH-based authentication code [#62307](https://github.com/ClickHouse/ClickHouse/pull/62307) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Fix data race in LocalServer [#62309](https://github.com/ClickHouse/ClickHouse/pull/62309) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix for postprocess script: print correct count for frame [#62317](https://github.com/ClickHouse/ClickHouse/pull/62317) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Use DETACHED_DIR_NAME everywhere [#62318](https://github.com/ClickHouse/ClickHouse/pull/62318) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix small typo in Dictionary source loader [#62320](https://github.com/ClickHouse/ClickHouse/pull/62320) ([Sean Haynes](https://github.com/seandhaynes)).
|
||||
* Fix optimize_uniq_to_count when only prefix of key is matched [#62325](https://github.com/ClickHouse/ClickHouse/pull/62325) ([vdimir](https://github.com/vdimir)).
|
||||
* More complex locking in `StackTrace::toString` [#62332](https://github.com/ClickHouse/ClickHouse/pull/62332) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Analyzer: Fix PREWHERE with lambda functions [#62336](https://github.com/ClickHouse/ClickHouse/pull/62336) ([vdimir](https://github.com/vdimir)).
|
||||
* Reduce log levels for ReadWriteBufferFromHTTP retries [#62348](https://github.com/ClickHouse/ClickHouse/pull/62348) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* dhparams are not enabled by default [#62365](https://github.com/ClickHouse/ClickHouse/pull/62365) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Disable window view with analyzer properly [#62367](https://github.com/ClickHouse/ClickHouse/pull/62367) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Don't access static members through instance, pt. II [#62375](https://github.com/ClickHouse/ClickHouse/pull/62375) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Use function isNotDistinctFrom only in join key [#62387](https://github.com/ClickHouse/ClickHouse/pull/62387) ([vdimir](https://github.com/vdimir)).
|
||||
* CI: fix for docs only pr [#62396](https://github.com/ClickHouse/ClickHouse/pull/62396) ([Max K.](https://github.com/maxknv)).
|
||||
* Fix one phony case [#62397](https://github.com/ClickHouse/ClickHouse/pull/62397) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* CI: test merge queue [#62403](https://github.com/ClickHouse/ClickHouse/pull/62403) ([Max K.](https://github.com/maxknv)).
|
||||
* Add part name to check part exception message [#62408](https://github.com/ClickHouse/ClickHouse/pull/62408) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* CI: disable finish check for mq [#62410](https://github.com/ClickHouse/ClickHouse/pull/62410) ([Max K.](https://github.com/maxknv)).
|
||||
* Fix logical error 'numbers_storage.step != UInt64{0}' [#62413](https://github.com/ClickHouse/ClickHouse/pull/62413) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Don't check overflow in arrayDotProduct in undefined sanitizer [#62417](https://github.com/ClickHouse/ClickHouse/pull/62417) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Avoid uncaught exception for onFault handler [#62418](https://github.com/ClickHouse/ClickHouse/pull/62418) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Update StorageFileLog.cpp [#62421](https://github.com/ClickHouse/ClickHouse/pull/62421) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Support for a tiny feature in stateless tests image [#62427](https://github.com/ClickHouse/ClickHouse/pull/62427) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* OptimizeGroupByInjectiveFunctionsPass remove unused constant [#62433](https://github.com/ClickHouse/ClickHouse/pull/62433) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Perf script update path in documentation [#62439](https://github.com/ClickHouse/ClickHouse/pull/62439) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Fix completion of available ClickHouse tools [#62446](https://github.com/ClickHouse/ClickHouse/pull/62446) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Use shared mutex for global stacktrace cache [#62453](https://github.com/ClickHouse/ClickHouse/pull/62453) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Keeper logging fixes [#62455](https://github.com/ClickHouse/ClickHouse/pull/62455) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Add profile events for azure disk [#62458](https://github.com/ClickHouse/ClickHouse/pull/62458) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* CI: gh runner version 2.315.0 [#62461](https://github.com/ClickHouse/ClickHouse/pull/62461) ([Max K.](https://github.com/maxknv)).
|
||||
* Fix clang-tidy build [#62478](https://github.com/ClickHouse/ClickHouse/pull/62478) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Fix random clang tidy warning [#62480](https://github.com/ClickHouse/ClickHouse/pull/62480) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Disable external sort in 01592_long_window_functions1 [#62487](https://github.com/ClickHouse/ClickHouse/pull/62487) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* CI: merge sync pr on push to master [#62488](https://github.com/ClickHouse/ClickHouse/pull/62488) ([Max K.](https://github.com/maxknv)).
|
||||
* Don't allow the fuzzer to change allow_experimental_analyzer [#62500](https://github.com/ClickHouse/ClickHouse/pull/62500) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Update comment in 02911_support_alias_column_in_indices.sql [#62503](https://github.com/ClickHouse/ClickHouse/pull/62503) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add test for [#26674](https://github.com/ClickHouse/ClickHouse/issues/26674) [#62504](https://github.com/ClickHouse/ClickHouse/pull/62504) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Add test for Bug 37909 [#62509](https://github.com/ClickHouse/ClickHouse/pull/62509) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add test for bug [#33446](https://github.com/ClickHouse/ClickHouse/issues/33446) [#62511](https://github.com/ClickHouse/ClickHouse/pull/62511) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Fix upgrade test. Again [#62513](https://github.com/ClickHouse/ClickHouse/pull/62513) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Blind fix for a flaky test [#62516](https://github.com/ClickHouse/ClickHouse/pull/62516) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Add asserts to COW example programs [#62543](https://github.com/ClickHouse/ClickHouse/pull/62543) ([Tomer Shafir](https://github.com/tomershafir)).
|
||||
* CI: respect Sync status in the MQ [#62550](https://github.com/ClickHouse/ClickHouse/pull/62550) ([Max K.](https://github.com/maxknv)).
|
||||
* Fix assertion in stress test [#62551](https://github.com/ClickHouse/ClickHouse/pull/62551) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix flaky 03093_bug37909_query_does_not_finish [#62553](https://github.com/ClickHouse/ClickHouse/pull/62553) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add test for issue 24607 [#62554](https://github.com/ClickHouse/ClickHouse/pull/62554) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Follow up to [#61723](https://github.com/ClickHouse/ClickHouse/issues/61723) [#62555](https://github.com/ClickHouse/ClickHouse/pull/62555) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix integration-tests logs compression [#62556](https://github.com/ClickHouse/ClickHouse/pull/62556) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Try to fix if_transform_strings_to_enum performance test [#62558](https://github.com/ClickHouse/ClickHouse/pull/62558) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Always use new analyzer in performance tests [#62564](https://github.com/ClickHouse/ClickHouse/pull/62564) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* CI: Add tests with Azure storage [#62565](https://github.com/ClickHouse/ClickHouse/pull/62565) ([Max K.](https://github.com/maxknv)).
|
||||
* CI: fix for sync check status in mq [#62568](https://github.com/ClickHouse/ClickHouse/pull/62568) ([Max K.](https://github.com/maxknv)).
|
||||
* Remove mentions of clean_deleted_rows from the documentation [#62570](https://github.com/ClickHouse/ClickHouse/pull/62570) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Try to fix Bugfix validation job [#62579](https://github.com/ClickHouse/ClickHouse/pull/62579) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* CI: add FT to MQ remove Style from master [#62588](https://github.com/ClickHouse/ClickHouse/pull/62588) ([Max K.](https://github.com/maxknv)).
|
||||
* CI: MQ sync status check fix [#62591](https://github.com/ClickHouse/ClickHouse/pull/62591) ([Max K.](https://github.com/maxknv)).
|
||||
* Better retries in azure sdk [#62608](https://github.com/ClickHouse/ClickHouse/pull/62608) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix: msan in UUIDStringToNum [#62610](https://github.com/ClickHouse/ClickHouse/pull/62610) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Fix a typo and grammar in `intersect` [#62622](https://github.com/ClickHouse/ClickHouse/pull/62622) ([Josh Rodriguez](https://github.com/jrodshua)).
|
||||
* JOIN filter push down right stream filled crash fix [#62624](https://github.com/ClickHouse/ClickHouse/pull/62624) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* HashedDictionaryParallelLoader exception safe constructor [#62625](https://github.com/ClickHouse/ClickHouse/pull/62625) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Fix 02366_kql_summarize [#62642](https://github.com/ClickHouse/ClickHouse/pull/62642) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Disable 02581_share_big_sets_between_mutation_tasks under sanitizers [#62645](https://github.com/ClickHouse/ClickHouse/pull/62645) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Don't allow relative paths when installing [#62658](https://github.com/ClickHouse/ClickHouse/pull/62658) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Update TransactionLog.cpp [#62663](https://github.com/ClickHouse/ClickHouse/pull/62663) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Disable aggregation-by-partitions optimisation with parallel replicas [#62697](https://github.com/ClickHouse/ClickHouse/pull/62697) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix build when `$CC` isn't set [#62700](https://github.com/ClickHouse/ClickHouse/pull/62700) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Bump Azure to 1.8.0 [#62702](https://github.com/ClickHouse/ClickHouse/pull/62702) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Fix --client-option for $CLICKHOUSE_CLIENT in .sh tests [#62710](https://github.com/ClickHouse/ClickHouse/pull/62710) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Bump Azure to v1.10 [#62712](https://github.com/ClickHouse/ClickHouse/pull/62712) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Bump Azure to v1.11 [#62713](https://github.com/ClickHouse/ClickHouse/pull/62713) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* `Trunc` docs fix [#62720](https://github.com/ClickHouse/ClickHouse/pull/62720) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Unify a batch of tests [#62723](https://github.com/ClickHouse/ClickHouse/pull/62723) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix typo in exception explanation [#62740](https://github.com/ClickHouse/ClickHouse/pull/62740) ([Igor Markelov](https://github.com/ElderlyPassionFruit)).
|
||||
* Block cannot allocate thread fault in noexcept functions in `MergeTreeTransaction` [#62751](https://github.com/ClickHouse/ClickHouse/pull/62751) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Log profile events send timings [#62752](https://github.com/ClickHouse/ClickHouse/pull/62752) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Follow-up to [#62700](https://github.com/ClickHouse/ClickHouse/issues/62700): Fix build when `$CC` isn't set [#62754](https://github.com/ClickHouse/ClickHouse/pull/62754) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Analyzer: Fix exception message [#62755](https://github.com/ClickHouse/ClickHouse/pull/62755) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Fix shellcheck style checking and issues [#62761](https://github.com/ClickHouse/ClickHouse/pull/62761) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Fix taking full part if part contains less than 'limit' rows [#62812](https://github.com/ClickHouse/ClickHouse/pull/62812) ([Artur Malchanau](https://github.com/Hexta)).
|
||||
* TableEngineGrant: undo breaking change [#62828](https://github.com/ClickHouse/ClickHouse/pull/62828) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Fix typo [#62836](https://github.com/ClickHouse/ClickHouse/pull/62836) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Revert "Add test for bug [#33446](https://github.com/ClickHouse/ClickHouse/issues/33446)" [#62844](https://github.com/ClickHouse/ClickHouse/pull/62844) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* SYSTEM DROP uninitialized cache fix [#62868](https://github.com/ClickHouse/ClickHouse/pull/62868) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* PlannerJoins remove unused comments [#62874](https://github.com/ClickHouse/ClickHouse/pull/62874) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Add test for bug 33446 [#62880](https://github.com/ClickHouse/ClickHouse/pull/62880) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Build kererberized_hadoop image by downloading commons-daemon via https [#62886](https://github.com/ClickHouse/ClickHouse/pull/62886) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* Update run.sh [#62889](https://github.com/ClickHouse/ClickHouse/pull/62889) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix build failed on clang-18 [#62899](https://github.com/ClickHouse/ClickHouse/pull/62899) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
* Fix parsing of nested proto messages [#62906](https://github.com/ClickHouse/ClickHouse/pull/62906) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix `00993_system_parts_race_condition_drop_zookeeper` [#62908](https://github.com/ClickHouse/ClickHouse/pull/62908) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix 03013_forbid_attach_table_if_active_replica_already_exists for private [#62909](https://github.com/ClickHouse/ClickHouse/pull/62909) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix 03015_optimize_final_rmt in private [#62911](https://github.com/ClickHouse/ClickHouse/pull/62911) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Add some functions to zookeeper client [#62920](https://github.com/ClickHouse/ClickHouse/pull/62920) ([alesapin](https://github.com/alesapin)).
|
||||
* Fix build on Mac using clang-18 [#62954](https://github.com/ClickHouse/ClickHouse/pull/62954) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Reapply: CI add FT to MQ remove Style from master [#62963](https://github.com/ClickHouse/ClickHouse/pull/62963) ([Max K.](https://github.com/maxknv)).
|
||||
* Fix flaky 03128_argMin_combinator_projection [#62965](https://github.com/ClickHouse/ClickHouse/pull/62965) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Better exception message [#62967](https://github.com/ClickHouse/ClickHouse/pull/62967) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix race in `executeJob` when updating exception message [#62972](https://github.com/ClickHouse/ClickHouse/pull/62972) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Remove incorrect assertion from DatabaseReplicated [#63000](https://github.com/ClickHouse/ClickHouse/pull/63000) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Update version_date.tsv and changelogs after v23.8.13.25-lts [#63014](https://github.com/ClickHouse/ClickHouse/pull/63014) ([robot-clickhouse](https://github.com/robot-clickhouse)).
|
||||
* JIT sort description crash fix [#63024](https://github.com/ClickHouse/ClickHouse/pull/63024) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* CI: fix ci config to run FT in MQ [#63025](https://github.com/ClickHouse/ClickHouse/pull/63025) ([Max K.](https://github.com/maxknv)).
|
||||
* Add test for [#42769](https://github.com/ClickHouse/ClickHouse/issues/42769) [#63033](https://github.com/ClickHouse/ClickHouse/pull/63033) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix suppressions for librdkafka data-race for statistics code [#63039](https://github.com/ClickHouse/ClickHouse/pull/63039) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Enable 03015_optimize_final_rmt for SMT [#63042](https://github.com/ClickHouse/ClickHouse/pull/63042) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* CI: fix job config for MQ [#63045](https://github.com/ClickHouse/ClickHouse/pull/63045) ([Max K.](https://github.com/maxknv)).
|
||||
* Unfork and update curl to 8.7.1 [#63048](https://github.com/ClickHouse/ClickHouse/pull/63048) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Fix integration tests with old analyzer (and fix some leftovers of enabling it) [#63069](https://github.com/ClickHouse/ClickHouse/pull/63069) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Get back test for old inter-server mode (DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET non-v2) [#63070](https://github.com/ClickHouse/ClickHouse/pull/63070) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix "invalid escape sequence" in clickhouse-test [#63073](https://github.com/ClickHouse/ClickHouse/pull/63073) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix stateful tests [#63077](https://github.com/ClickHouse/ClickHouse/pull/63077) ([alesapin](https://github.com/alesapin)).
|
||||
* Better highlighting of keywords [#63079](https://github.com/ClickHouse/ClickHouse/pull/63079) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix race in OpenSSL X509 store [#63109](https://github.com/ClickHouse/ClickHouse/pull/63109) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Azure always green [#63120](https://github.com/ClickHouse/ClickHouse/pull/63120) ([alesapin](https://github.com/alesapin)).
|
||||
* Fix flaky `03094_grouparraysorted_memory` [#63121](https://github.com/ClickHouse/ClickHouse/pull/63121) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* test for [#56564](https://github.com/ClickHouse/ClickHouse/issues/56564) [#63124](https://github.com/ClickHouse/ClickHouse/pull/63124) ([Denny Crane](https://github.com/den-crane)).
|
||||
* Recursive CTE data race fix [#63125](https://github.com/ClickHouse/ClickHouse/pull/63125) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* add test for [#55360](https://github.com/ClickHouse/ClickHouse/issues/55360) [#63127](https://github.com/ClickHouse/ClickHouse/pull/63127) ([flynn](https://github.com/ucasfl)).
|
||||
* add tests for [#47217](https://github.com/ClickHouse/ClickHouse/issues/47217), [#55965](https://github.com/ClickHouse/ClickHouse/issues/55965) [#63128](https://github.com/ClickHouse/ClickHouse/pull/63128) ([Denny Crane](https://github.com/den-crane)).
|
||||
* Revert "Merge pull request [#60598](https://github.com/ClickHouse/ClickHouse/issues/60598) from jrdi/week-default-mode" [#63157](https://github.com/ClickHouse/ClickHouse/pull/63157) ([Jordi Villar](https://github.com/jrdi)).
|
||||
|
||||
@ -1,19 +1,19 @@
|
||||
---
|
||||
slug: /en/engines/table-engines/mergetree-family/invertedindexes
|
||||
sidebar_label: Inverted Indexes
|
||||
sidebar_label: Full-text Indexes
|
||||
description: Quickly find search terms in text.
|
||||
keywords: [full-text search, text search, inverted, index, indices]
|
||||
---
|
||||
|
||||
# Full-text Search using Inverted Indexes [experimental]
|
||||
# Full-text Search using Full-text Indexes [experimental]
|
||||
|
||||
Inverted indexes are an experimental type of [secondary indexes](/docs/en/engines/table-engines/mergetree-family/mergetree.md/#available-types-of-indices) which provide fast text search
|
||||
Full-text indexes are an experimental type of [secondary indexes](/docs/en/engines/table-engines/mergetree-family/mergetree.md/#available-types-of-indices) which provide fast text search
|
||||
capabilities for [String](/docs/en/sql-reference/data-types/string.md) or [FixedString](/docs/en/sql-reference/data-types/fixedstring.md)
|
||||
columns. The main idea of an inverted index is to store a mapping from "terms" to the rows which contain these terms. "Terms" are
|
||||
columns. The main idea of a full-text index is to store a mapping from "terms" to the rows which contain these terms. "Terms" are
|
||||
tokenized cells of the string column. For example, the string cell "I will be a little late" is by default tokenized into six terms "I", "will",
|
||||
"be", "a", "little" and "late". Another kind of tokenizer is n-grams. For example, the result of 3-gram tokenization will be 21 terms "I w",
|
||||
" wi", "wil", "ill", "ll ", "l b", " be" etc. The more fine-granular the input strings are tokenized, the bigger but also the more
|
||||
useful the resulting inverted index will be.
|
||||
useful the resulting full-text index will be.
|
||||
|
||||
<div class='vimeo-container'>
|
||||
<iframe src="//www.youtube.com/embed/O_MnyUkrIq8"
|
||||
@ -28,26 +28,26 @@ useful the resulting inverted index will be.
|
||||
</div>
|
||||
|
||||
:::note
|
||||
Inverted indexes are experimental and should not be used in production environments yet. They may change in the future in backward-incompatible
|
||||
Full-text indexes are experimental and should not be used in production environments yet. They may change in the future in backward-incompatible
|
||||
ways, for example with respect to their DDL/DQL syntax or performance/compression characteristics.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
To use inverted indexes, first enable them in the configuration:
|
||||
To use full-text indexes, first enable them in the configuration:
|
||||
|
||||
```sql
|
||||
SET allow_experimental_inverted_index = true;
|
||||
```
|
||||
|
||||
An inverted index can be defined on a string column using the following syntax
|
||||
An full-text index can be defined on a string column using the following syntax
|
||||
|
||||
``` sql
|
||||
CREATE TABLE tab
|
||||
(
|
||||
`key` UInt64,
|
||||
`str` String,
|
||||
INDEX inv_idx(str) TYPE inverted(0) GRANULARITY 1
|
||||
INDEX inv_idx(str) TYPE full_text(0) GRANULARITY 1
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
ORDER BY key
|
||||
@ -55,20 +55,20 @@ ORDER BY key
|
||||
|
||||
where `N` specifies the tokenizer:
|
||||
|
||||
- `inverted(0)` (or shorter: `inverted()`) set the tokenizer to "tokens", i.e. split strings along spaces,
|
||||
- `inverted(N)` with `N` between 2 and 8 sets the tokenizer to "ngrams(N)"
|
||||
- `full_text(0)` (or shorter: `full_text()`) set the tokenizer to "tokens", i.e. split strings along spaces,
|
||||
- `full_text(N)` with `N` between 2 and 8 sets the tokenizer to "ngrams(N)"
|
||||
|
||||
The maximum rows per postings list can be specified as the second parameter. This parameter can be used to control postings list sizes to avoid generating huge postings list files. The following variants exist:
|
||||
|
||||
- `inverted(ngrams, max_rows_per_postings_list)`: Use given max_rows_per_postings_list (assuming it is not 0)
|
||||
- `inverted(ngrams, 0)`: No limitation of maximum rows per postings list
|
||||
- `inverted(ngrams)`: Use a default maximum rows which is 64K.
|
||||
- `full_text(ngrams, max_rows_per_postings_list)`: Use given max_rows_per_postings_list (assuming it is not 0)
|
||||
- `full_text(ngrams, 0)`: No limitation of maximum rows per postings list
|
||||
- `full_text(ngrams)`: Use a default maximum rows which is 64K.
|
||||
|
||||
Being a type of skipping index, inverted indexes can be dropped or added to a column after table creation:
|
||||
Being a type of skipping index, full-text indexes can be dropped or added to a column after table creation:
|
||||
|
||||
``` sql
|
||||
ALTER TABLE tab DROP INDEX inv_idx;
|
||||
ALTER TABLE tab ADD INDEX inv_idx(s) TYPE inverted(2);
|
||||
ALTER TABLE tab ADD INDEX inv_idx(s) TYPE full_text(2);
|
||||
```
|
||||
|
||||
To use the index, no special functions or syntax are required. Typical string search predicates automatically leverage the index. As
|
||||
@ -83,9 +83,9 @@ SELECT * from tab WHERE multiSearchAny(str, ['Hello', 'World']);
|
||||
SELECT * from tab WHERE hasToken(str, 'Hello');
|
||||
```
|
||||
|
||||
The inverted index also works on columns of type `Array(String)`, `Array(FixedString)`, `Map(String)` and `Map(String)`.
|
||||
The full-text index also works on columns of type `Array(String)`, `Array(FixedString)`, `Map(String)` and `Map(String)`.
|
||||
|
||||
Like for other secondary indices, each column part has its own inverted index. Furthermore, each inverted index is internally divided into
|
||||
Like for other secondary indices, each column part has its own full-text index. Furthermore, each full-text index is internally divided into
|
||||
"segments". The existence and size of the segments are generally transparent to users but the segment size determines the memory consumption
|
||||
during index construction (e.g. when two parts are merged). Configuration parameter "max_digestion_size_per_segment" (default: 256 MB)
|
||||
controls the amount of data read consumed from the underlying column before a new segment is created. Incrementing the parameter raises the
|
||||
@ -94,7 +94,7 @@ average to evaluate a query.
|
||||
|
||||
## Full-text search of the Hacker News dataset
|
||||
|
||||
Let's look at the performance improvements of inverted indexes on a large dataset with lots of text. We will use 28.7M rows of comments on the popular Hacker News website. Here is the table without an inverted index:
|
||||
Let's look at the performance improvements of full-text indexes on a large dataset with lots of text. We will use 28.7M rows of comments on the popular Hacker News website. Here is the table without an full-text index:
|
||||
|
||||
```sql
|
||||
CREATE TABLE hackernews (
|
||||
@ -162,11 +162,11 @@ Notice it takes 3 seconds to execute the query:
|
||||
1 row in set. Elapsed: 3.001 sec. Processed 28.74 million rows, 9.75 GB (9.58 million rows/s., 3.25 GB/s.)
|
||||
```
|
||||
|
||||
We will use `ALTER TABLE` and add an inverted index on the lowercase of the `comment` column, then materialize it (which can take a while - wait for it to materialize):
|
||||
We will use `ALTER TABLE` and add an full-text index on the lowercase of the `comment` column, then materialize it (which can take a while - wait for it to materialize):
|
||||
|
||||
```sql
|
||||
ALTER TABLE hackernews
|
||||
ADD INDEX comment_lowercase(lower(comment)) TYPE inverted;
|
||||
ADD INDEX comment_lowercase(lower(comment)) TYPE full_text;
|
||||
|
||||
ALTER TABLE hackernews MATERIALIZE INDEX comment_lowercase;
|
||||
```
|
||||
@ -204,9 +204,9 @@ WHERE hasToken(lower(comment), 'avx') AND hasToken(lower(comment), 'sve');
|
||||
```
|
||||
|
||||
:::note
|
||||
Unlike other secondary indices, inverted indexes (for now) map to row numbers (row ids) instead of granule ids. The reason for this design
|
||||
Unlike other secondary indices, full-text indexes (for now) map to row numbers (row ids) instead of granule ids. The reason for this design
|
||||
is performance. In practice, users often search for multiple terms at once. For example, filter predicate `WHERE s LIKE '%little%' OR s LIKE
|
||||
'%big%'` can be evaluated directly using an inverted index by forming the union of the row id lists for terms "little" and "big". This also
|
||||
'%big%'` can be evaluated directly using an full-text index by forming the union of the row id lists for terms "little" and "big". This also
|
||||
means that the parameter `GRANULARITY` supplied to index creation has no meaning (it may be removed from the syntax in the future).
|
||||
:::
|
||||
|
||||
|
||||
@ -287,9 +287,9 @@ The number of columns in the primary key is not explicitly limited. Depending on
|
||||
|
||||
A long primary key will negatively affect the insert performance and memory consumption, but extra columns in the primary key do not affect ClickHouse performance during `SELECT` queries.
|
||||
|
||||
You can create a table without a primary key using the `ORDER BY tuple()` syntax. In this case, ClickHouse stores data in the order of inserting. If you want to save data order when inserting data by `INSERT ... SELECT` queries, set [max_insert_threads = 1](/docs/en/operations/settings/settings.md/#settings-max-insert-threads).
|
||||
You can create a table without a primary key using the `ORDER BY tuple()` syntax. In this case, ClickHouse stores data in the order of inserting. If you want to save data order when inserting data by `INSERT ... SELECT` queries, set [max_insert_threads = 1](/docs/en/operations/settings/settings.md/#max-insert-threads).
|
||||
|
||||
To select data in the initial order, use [single-threaded](/docs/en/operations/settings/settings.md/#settings-max_threads) `SELECT` queries.
|
||||
To select data in the initial order, use [single-threaded](/docs/en/operations/settings/settings.md/#max_threads) `SELECT` queries.
|
||||
|
||||
### Choosing a Primary Key that Differs from the Sorting Key {#choosing-a-primary-key-that-differs-from-the-sorting-key}
|
||||
|
||||
@ -344,7 +344,7 @@ In the example below, the index can’t be used.
|
||||
SELECT count() FROM table WHERE CounterID = 34 OR URL LIKE '%upyachka%'
|
||||
```
|
||||
|
||||
To check whether ClickHouse can use the index when running a query, use the settings [force_index_by_date](/docs/en/operations/settings/settings.md/#settings-force_index_by_date) and [force_primary_key](/docs/en/operations/settings/settings.md/#force-primary-key).
|
||||
To check whether ClickHouse can use the index when running a query, use the settings [force_index_by_date](/docs/en/operations/settings/settings.md/#force_index_by_date) and [force_primary_key](/docs/en/operations/settings/settings.md/#force-primary-key).
|
||||
|
||||
The key for partitioning by month allows reading only those data blocks which contain dates from the proper range. In this case, the data block may contain data for many dates (up to an entire month). Within a block, data is sorted by primary key, which might not contain the date as the first column. Because of this, using a query with only a date condition that does not specify the primary key prefix will cause more data to be read than for a single date.
|
||||
|
||||
@ -769,6 +769,7 @@ In addition to local block devices, ClickHouse supports these storage types:
|
||||
- [`web` for read-only from web](#web-storage)
|
||||
- [`cache` for local caching](/docs/en/operations/storing-data.md/#using-local-cache)
|
||||
- [`s3_plain` for backups to S3](/docs/en/operations/backup#backuprestore-using-an-s3-disk)
|
||||
- [`s3_plain_rewritable` for immutable, non-replicated tables in S3](/docs/en/operations/storing-data.md#s3-plain-rewritable-storage)
|
||||
|
||||
## Using Multiple Block Devices for Data Storage {#table_engine-mergetree-multiple-volumes}
|
||||
|
||||
|
||||
@ -113,7 +113,7 @@ You can specify any existing ZooKeeper cluster and the system will use a directo
|
||||
|
||||
If ZooKeeper is not set in the config file, you can’t create replicated tables, and any existing replicated tables will be read-only.
|
||||
|
||||
ZooKeeper is not used in `SELECT` queries because replication does not affect the performance of `SELECT` and queries run just as fast as they do for non-replicated tables. When querying distributed replicated tables, ClickHouse behavior is controlled by the settings [max_replica_delay_for_distributed_queries](/docs/en/operations/settings/settings.md/#settings-max_replica_delay_for_distributed_queries) and [fallback_to_stale_replicas_for_distributed_queries](/docs/en/operations/settings/settings.md/#settings-fallback_to_stale_replicas_for_distributed_queries).
|
||||
ZooKeeper is not used in `SELECT` queries because replication does not affect the performance of `SELECT` and queries run just as fast as they do for non-replicated tables. When querying distributed replicated tables, ClickHouse behavior is controlled by the settings [max_replica_delay_for_distributed_queries](/docs/en/operations/settings/settings.md/#max_replica_delay_for_distributed_queries) and [fallback_to_stale_replicas_for_distributed_queries](/docs/en/operations/settings/settings.md/#fallback_to_stale_replicas_for_distributed_queries).
|
||||
|
||||
For each `INSERT` query, approximately ten entries are added to ZooKeeper through several transactions. (To be more precise, this is for each inserted block of data; an INSERT query contains one block or one block per `max_insert_block_size = 1048576` rows.) This leads to slightly longer latencies for `INSERT` compared to non-replicated tables. But if you follow the recommendations to insert data in batches of no more than one `INSERT` per second, it does not create any problems. The entire ClickHouse cluster used for coordinating one ZooKeeper cluster has a total of several hundred `INSERTs` per second. The throughput on data inserts (the number of rows per second) is just as high as for non-replicated data.
|
||||
|
||||
|
||||
@ -83,7 +83,7 @@ When creating a table, the following settings are applied:
|
||||
|
||||
#### join_any_take_last_row
|
||||
|
||||
[join_any_take_last_row](/docs/en/operations/settings/settings.md/#settings-join_any_take_last_row)
|
||||
[join_any_take_last_row](/docs/en/operations/settings/settings.md/#join_any_take_last_row)
|
||||
#### join_use_nulls
|
||||
|
||||
#### persistent
|
||||
|
||||
2
docs/en/interfaces/third-party/gui.md
vendored
2
docs/en/interfaces/third-party/gui.md
vendored
@ -172,7 +172,7 @@ Features:
|
||||
|
||||
### ClickVisual {#clickvisual}
|
||||
|
||||
[ClickVisual](https://clickvisual.gocn.vip/) ClickVisual is a lightweight open source log query, analysis and alarm visualization platform.
|
||||
[ClickVisual](https://clickvisual.net/) ClickVisual is a lightweight open source log query, analysis and alarm visualization platform.
|
||||
|
||||
Features:
|
||||
|
||||
|
||||
@ -4373,17 +4373,6 @@ Possible values:
|
||||
|
||||
Default value: `ignore`.
|
||||
|
||||
## first_day_of_week
|
||||
|
||||
The first day of the week assumed by [`toStartOfInterval`](../../sql-reference/functions/date-time-functions.md#toStartOfInterval) function when using weeks as unit.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Monday - Week starts on Monday
|
||||
- Sunday - Week starts on Sunday
|
||||
|
||||
Default value: 'Monday'.
|
||||
|
||||
## optimize_move_to_prewhere {#optimize_move_to_prewhere}
|
||||
|
||||
Enables or disables automatic [PREWHERE](../../sql-reference/statements/select/prewhere.md) optimization in [SELECT](../../sql-reference/statements/select/index.md) queries.
|
||||
|
||||
@ -28,7 +28,7 @@ Starting from 24.1 clickhouse version, it is possible to use a new configuration
|
||||
It requires to specify:
|
||||
1. `type` equal to `object_storage`
|
||||
2. `object_storage_type`, equal to one of `s3`, `azure_blob_storage` (or just `azure` from `24.3`), `hdfs`, `local_blob_storage` (or just `local` from `24.3`), `web`.
|
||||
Optionally, `metadata_type` can be specified (it is equal to `local` by default), but it can also be set to `plain`, `web`.
|
||||
Optionally, `metadata_type` can be specified (it is equal to `local` by default), but it can also be set to `plain`, `web` and, starting from `24.4`, `plain_rewritable`.
|
||||
Usage of `plain` metadata type is described in [plain storage section](/docs/en/operations/storing-data.md/#storing-data-on-webserver), `web` metadata type can be used only with `web` object storage type, `local` metadata type stores metadata files locally (each metadata files contains mapping to files in object storage and some additional meta information about them).
|
||||
|
||||
E.g. configuration option
|
||||
@ -341,6 +341,36 @@ Configuration:
|
||||
</s3_plain>
|
||||
```
|
||||
|
||||
### Using S3 Plain Rewritable Storage {#s3-plain-rewritable-storage}
|
||||
A new disk type `s3_plain_rewritable` was introduced in `24.4`.
|
||||
Similar to the `s3_plain` disk type, it does not require additional storage for metadata files; instead, metadata is stored in S3.
|
||||
Unlike `s3_plain` disk type, `s3_plain_rewritable` allows executing merges and supports INSERT operations.
|
||||
[Mutations](/docs/en/sql-reference/statements/alter#mutations) and replication of tables are not supported.
|
||||
|
||||
A use case for this disk type are non-replicated `MergeTree` tables. Although the `s3` disk type is suitable for non-replicated
|
||||
MergeTree tables, you may opt for the `s3_plain_rewritable` disk type if you do not require local metadata for the table and are
|
||||
willing to accept a limited set of operations. This could be useful, for example, for system tables.
|
||||
|
||||
Configuration:
|
||||
``` xml
|
||||
<s3_plain_rewritable>
|
||||
<type>s3_plain_rewritable</type>
|
||||
<endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
|
||||
<use_environment_credentials>1</use_environment_credentials>
|
||||
</s3_plain_rewritable>
|
||||
```
|
||||
|
||||
is equal to
|
||||
``` xml
|
||||
<s3_plain_rewritable>
|
||||
<type>object_storage</type>
|
||||
<object_storage_type>s3</object_storage_type>
|
||||
<metadata_type>plain_rewritable</metadata_type>
|
||||
<endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint>
|
||||
<use_environment_credentials>1</use_environment_credentials>
|
||||
</s3_plain_rewritable>
|
||||
```
|
||||
|
||||
### Using Azure Blob Storage {#azure-blob-storage}
|
||||
|
||||
`MergeTree` family table engines can store data to [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) using a disk with type `azure_blob_storage`.
|
||||
|
||||
@ -505,9 +505,117 @@ HAVING uniqUpTo(4)(UserID) >= 5
|
||||
|
||||
`uniqUpTo(4)(UserID)` calculates the number of unique `UserID` values for each `SearchPhrase`, but it only counts up to 4 unique values. If there are more than 4 unique `UserID` values for a `SearchPhrase`, the function returns 5 (4 + 1). The `HAVING` clause then filters out the `SearchPhrase` values for which the number of unique `UserID` values is less than 5. This will give you a list of search keywords that were used by at least 5 unique users.
|
||||
|
||||
## sumMapFiltered(keys_to_keep)(keys, values)
|
||||
## sumMapFiltered
|
||||
|
||||
Same behavior as [sumMap](../../sql-reference/aggregate-functions/reference/summap.md#agg_functions-summap) except that an array of keys is passed as a parameter. This can be especially useful when working with a high cardinality of keys.
|
||||
This function behaves the same as [sumMap](../../sql-reference/aggregate-functions/reference/summap.md#agg_functions-summap) except that it also accepts an array of keys to filter with as a parameter. This can be especially useful when working with a high cardinality of keys.
|
||||
|
||||
**Syntax**
|
||||
|
||||
`sumMapFiltered(keys_to_keep)(keys, values)`
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `keys_to_keep`: [Array](../data-types/array.md) of keys to filter with.
|
||||
- `keys`: [Array](../data-types/array.md) of keys.
|
||||
- `values`: [Array](../data-types/array.md) of values.
|
||||
|
||||
**Returned Value**
|
||||
|
||||
- Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
CREATE TABLE sum_map
|
||||
(
|
||||
`date` Date,
|
||||
`timeslot` DateTime,
|
||||
`statusMap` Nested(status UInt16, requests UInt64)
|
||||
)
|
||||
ENGINE = Log
|
||||
|
||||
INSERT INTO sum_map VALUES
|
||||
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10]);
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT sumMapFiltered([1, 4, 8])(statusMap.status, statusMap.requests) FROM sum_map;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌─sumMapFiltered([1, 4, 8])(statusMap.status, statusMap.requests)─┐
|
||||
1. │ ([1,4,8],[10,20,10]) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## sumMapFilteredWithOverflow
|
||||
|
||||
This function behaves the same as [sumMap](../../sql-reference/aggregate-functions/reference/summap.md#agg_functions-summap) except that it also accepts an array of keys to filter with as a parameter. This can be especially useful when working with a high cardinality of keys. It differs from the [sumMapFiltered](#summapfiltered) function in that it does summation with overflow - i.e. returns the same data type for the summation as the argument data type.
|
||||
|
||||
**Syntax**
|
||||
|
||||
`sumMapFilteredWithOverflow(keys_to_keep)(keys, values)`
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `keys_to_keep`: [Array](../data-types/array.md) of keys to filter with.
|
||||
- `keys`: [Array](../data-types/array.md) of keys.
|
||||
- `values`: [Array](../data-types/array.md) of values.
|
||||
|
||||
**Returned Value**
|
||||
|
||||
- Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
|
||||
**Example**
|
||||
|
||||
In this example we create a table `sum_map`, insert some data into it and then use both `sumMapFilteredWithOverflow` and `sumMapFiltered` and the `toTypeName` function for comparison of the result. Where `requests` was of type `UInt8` in the created table, `sumMapFiltered` has promoted the type of the summed values to `UInt64` to avoid overflow whereas `sumMapFilteredWithOverflow` has kept the type as `UInt8` which is not large enough to store the result - i.e. overflow has occurred.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
CREATE TABLE sum_map
|
||||
(
|
||||
`date` Date,
|
||||
`timeslot` DateTime,
|
||||
`statusMap` Nested(status UInt8, requests UInt8)
|
||||
)
|
||||
ENGINE = Log
|
||||
|
||||
INSERT INTO sum_map VALUES
|
||||
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10]);
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT sumMapFilteredWithOverflow([1, 4, 8])(statusMap.status, statusMap.requests) as summap_overflow, toTypeName(summap_overflow) FROM sum_map;
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT sumMapFiltered([1, 4, 8])(statusMap.status, statusMap.requests) as summap, toTypeName(summap) FROM sum_map;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌─sum──────────────────┬─toTypeName(sum)───────────────────┐
|
||||
1. │ ([1,4,8],[10,20,10]) │ Tuple(Array(UInt8), Array(UInt8)) │
|
||||
└──────────────────────┴───────────────────────────────────┘
|
||||
```
|
||||
|
||||
```response
|
||||
┌─summap───────────────┬─toTypeName(summap)─────────────────┐
|
||||
1. │ ([1,4,8],[10,20,10]) │ Tuple(Array(UInt8), Array(UInt64)) │
|
||||
└──────────────────────┴────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## sequenceNextNode
|
||||
|
||||
|
||||
@ -16,7 +16,9 @@ Standard aggregate functions:
|
||||
- [avg](/docs/en/sql-reference/aggregate-functions/reference/avg.md)
|
||||
- [any](/docs/en/sql-reference/aggregate-functions/reference/any.md)
|
||||
- [stddevPop](/docs/en/sql-reference/aggregate-functions/reference/stddevpop.md)
|
||||
- [stddevPopStable](/docs/en/sql-reference/aggregate-functions/reference/stddevpopstable.md)
|
||||
- [stddevSamp](/docs/en/sql-reference/aggregate-functions/reference/stddevsamp.md)
|
||||
- [stddevSampStable](/docs/en/sql-reference/aggregate-functions/reference/stddevsampstable.md)
|
||||
- [varPop](/docs/en/sql-reference/aggregate-functions/reference/varpop.md)
|
||||
- [varSamp](/docs/en/sql-reference/aggregate-functions/reference/varsamp.md)
|
||||
- [corr](./corr.md)
|
||||
@ -65,6 +67,9 @@ ClickHouse-specific aggregate functions:
|
||||
- [groupBitmapXor](/docs/en/sql-reference/aggregate-functions/reference/groupbitmapxor.md)
|
||||
- [sumWithOverflow](/docs/en/sql-reference/aggregate-functions/reference/sumwithoverflow.md)
|
||||
- [sumMap](/docs/en/sql-reference/aggregate-functions/reference/summap.md)
|
||||
- [sumMapWithOverflow](/docs/en/sql-reference/aggregate-functions/reference/summapwithoverflow.md)
|
||||
- [sumMapFiltered](/docs/en/sql-reference/aggregate-functions/parametric-functions.md/#summapfiltered)
|
||||
- [sumMapFilteredWithOverflow](/docs/en/sql-reference/aggregate-functions/parametric-functions.md/#summapfilteredwithoverflow)
|
||||
- [minMap](/docs/en/sql-reference/aggregate-functions/reference/minmap.md)
|
||||
- [maxMap](/docs/en/sql-reference/aggregate-functions/reference/maxmap.md)
|
||||
- [skewSamp](/docs/en/sql-reference/aggregate-functions/reference/skewsamp.md)
|
||||
|
||||
@ -7,10 +7,50 @@ sidebar_position: 30
|
||||
|
||||
The result is equal to the square root of [varPop](../../../sql-reference/aggregate-functions/reference/varpop.md).
|
||||
|
||||
Alias:
|
||||
- `STD`
|
||||
- `STDDEV_POP`
|
||||
Aliases: `STD`, `STDDEV_POP`.
|
||||
|
||||
:::note
|
||||
This function uses a numerically unstable algorithm. If you need [numerical stability](https://en.wikipedia.org/wiki/Numerical_stability) in calculations, use the `stddevPopStable` function. It works slower but provides a lower computational error.
|
||||
:::
|
||||
This function uses a numerically unstable algorithm. If you need [numerical stability](https://en.wikipedia.org/wiki/Numerical_stability) in calculations, use the [`stddevPopStable`](../reference/stddevpopstable.md) function. It works slower but provides a lower computational error.
|
||||
:::
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
stddevPop(x)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `x`: Population of values to find the standard deviation of. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Square root of standard deviation of `x`. [Float64](../../data-types/float.md).
|
||||
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS test_data;
|
||||
CREATE TABLE test_data
|
||||
(
|
||||
population UInt8,
|
||||
)
|
||||
ENGINE = Log;
|
||||
|
||||
INSERT INTO test_data VALUES (3),(3),(3),(4),(4),(5),(5),(7),(11),(15);
|
||||
|
||||
SELECT
|
||||
stddevPop(population) AS stddev
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌────────────stddev─┐
|
||||
│ 3.794733192202055 │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
@ -0,0 +1,49 @@
|
||||
---
|
||||
slug: /en/sql-reference/aggregate-functions/reference/stddevpopstable
|
||||
sidebar_position: 30
|
||||
---
|
||||
|
||||
# stddevPopStable
|
||||
|
||||
The result is equal to the square root of [varPop](../../../sql-reference/aggregate-functions/reference/varpop.md). Unlike [`stddevPop`](../reference/stddevpop.md), this function uses a numerically stable algorithm. It works slower but provides a lower computational error.
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
stddevPopStable(x)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `x`: Population of values to find the standard deviation of. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Square root of standard deviation of `x`. [Float64](../../data-types/float.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS test_data;
|
||||
CREATE TABLE test_data
|
||||
(
|
||||
population Float64,
|
||||
)
|
||||
ENGINE = Log;
|
||||
|
||||
INSERT INTO test_data SELECT randUniform(5.5, 10) FROM numbers(1000000)
|
||||
|
||||
SELECT
|
||||
stddevPopStable(population) AS stddev
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌─────────────stddev─┐
|
||||
│ 1.2999977786592576 │
|
||||
└────────────────────┘
|
||||
```
|
||||
@ -10,5 +10,46 @@ The result is equal to the square root of [varSamp](../../../sql-reference/aggre
|
||||
Alias: `STDDEV_SAMP`.
|
||||
|
||||
:::note
|
||||
This function uses a numerically unstable algorithm. If you need [numerical stability](https://en.wikipedia.org/wiki/Numerical_stability) in calculations, use the `stddevSampStable` function. It works slower but provides a lower computational error.
|
||||
:::
|
||||
This function uses a numerically unstable algorithm. If you need [numerical stability](https://en.wikipedia.org/wiki/Numerical_stability) in calculations, use the [`stddevSampStable`](../reference/stddevsampstable.md) function. It works slower but provides a lower computational error.
|
||||
:::
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
stddevSamp(x)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `x`: Values for which to find the square root of sample variance. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Square root of sample variance of `x`. [Float64](../../data-types/float.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS test_data;
|
||||
CREATE TABLE test_data
|
||||
(
|
||||
population UInt8,
|
||||
)
|
||||
ENGINE = Log;
|
||||
|
||||
INSERT INTO test_data VALUES (3),(3),(3),(4),(4),(5),(5),(7),(11),(15);
|
||||
|
||||
SELECT
|
||||
stddevSamp(population)
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌─stddevSamp(population)─┐
|
||||
│ 4 │
|
||||
└────────────────────────┘
|
||||
```
|
||||
@ -0,0 +1,49 @@
|
||||
---
|
||||
slug: /en/sql-reference/aggregate-functions/reference/stddevsampstable
|
||||
sidebar_position: 31
|
||||
---
|
||||
|
||||
# stddevSampStable
|
||||
|
||||
The result is equal to the square root of [varSamp](../../../sql-reference/aggregate-functions/reference/varsamp.md). Unlike [`stddevSamp`](../reference/stddevsamp.md) This function uses a numerically stable algorithm. It works slower but provides a lower computational error.
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
stddevSampStable(x)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
|
||||
- `x`: Values for which to find the square root of sample variance. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Square root of sample variance of `x`. [Float64](../../data-types/float.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS test_data;
|
||||
CREATE TABLE test_data
|
||||
(
|
||||
population UInt8,
|
||||
)
|
||||
ENGINE = Log;
|
||||
|
||||
INSERT INTO test_data VALUES (3),(3),(3),(4),(4),(5),(5),(7),(11),(15);
|
||||
|
||||
SELECT
|
||||
stddevSampStable(population)
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
┌─stddevSampStable(population)─┐
|
||||
│ 4 │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
@ -7,6 +7,56 @@ sidebar_position: 4
|
||||
|
||||
Calculates the sum. Only works for numbers.
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
sum(num)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
- `num`: Column of numeric values. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
- The sum of the values. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Example**
|
||||
|
||||
First we create a table `employees` and insert some fictional employee data into it.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
CREATE TABLE employees
|
||||
(
|
||||
`id` UInt32,
|
||||
`name` String,
|
||||
`salary` UInt32
|
||||
)
|
||||
ENGINE = Log
|
||||
```
|
||||
|
||||
```sql
|
||||
INSERT INTO employees VALUES
|
||||
(87432, 'John Smith', 45680),
|
||||
(59018, 'Jane Smith', 72350),
|
||||
(20376, 'Ivan Ivanovich', 58900),
|
||||
(71245, 'Anastasia Ivanovna', 89210);
|
||||
```
|
||||
|
||||
We query for the total amount of the employee salaries using the `sum` function.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT sum(salary) FROM employees;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
|
||||
```response
|
||||
┌─sum(salary)─┐
|
||||
1. │ 266140 │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
@ -5,21 +5,35 @@ sidebar_position: 141
|
||||
|
||||
# sumMap
|
||||
|
||||
Syntax: `sumMap(key <Array>, value <Array>)` [Array type](../../data-types/array.md) or `sumMap(Tuple(key <Array>, value <Array>))` [Tuple type](../../data-types/tuple.md).
|
||||
Totals a `value` array according to the keys specified in the `key` array. Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys without overflow.
|
||||
|
||||
Arguments:
|
||||
**Syntax**
|
||||
|
||||
- `sumMap(key <Array>, value <Array>)` [Array type](../../data-types/array.md).
|
||||
- `sumMap(Tuple(key <Array>, value <Array>))` [Tuple type](../../data-types/tuple.md).
|
||||
|
||||
Alias: `sumMappedArrays`.
|
||||
|
||||
Totals the `value` array according to the keys specified in the `key` array.
|
||||
**Arguments**
|
||||
|
||||
Passing tuple of keys and values arrays is a synonym to passing two arrays of keys and values.
|
||||
- `key`: [Array](../../data-types/array.md) of keys.
|
||||
- `value`: [Array](../../data-types/array.md) of values.
|
||||
|
||||
Passing a tuple of key and value arrays is a synonym to passing separately an array of keys and an array of values.
|
||||
|
||||
:::note
|
||||
The number of elements in `key` and `value` must be the same for each row that is totaled.
|
||||
:::
|
||||
|
||||
Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
**Returned Value**
|
||||
|
||||
Example:
|
||||
- Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
|
||||
**Example**
|
||||
|
||||
First we create a table called `sum_map`, and insert some data into it. Arrays of keys and values are stored separately as a column called `statusMap` of [Nested](../../data-types/nested-data-structures/index.md) type, and together as a column called `statusMapTuple` of [tuple](../../data-types/tuple.md) type to illustrate the use of the two different syntaxes of this function described above.
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
CREATE TABLE sum_map(
|
||||
@ -31,13 +45,20 @@ CREATE TABLE sum_map(
|
||||
),
|
||||
statusMapTuple Tuple(Array(Int32), Array(Int32))
|
||||
) ENGINE = Log;
|
||||
|
||||
```
|
||||
```sql
|
||||
INSERT INTO sum_map VALUES
|
||||
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10], ([1, 2, 3], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10], ([3, 4, 5], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10], ([4, 5, 6], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10], ([6, 7, 8], [10, 10, 10]));
|
||||
```
|
||||
|
||||
Next, we query the table using the `sumMap` function, making use of both array and tuple type syntaxes:
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
timeslot,
|
||||
sumMap(statusMap.status, statusMap.requests),
|
||||
@ -46,6 +67,8 @@ FROM sum_map
|
||||
GROUP BY timeslot
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌────────────timeslot─┬─sumMap(statusMap.status, statusMap.requests)─┬─sumMap(statusMapTuple)─────────┐
|
||||
│ 2000-01-01 00:00:00 │ ([1,2,3,4,5],[10,10,20,10,10]) │ ([1,2,3,4,5],[10,10,20,10,10]) │
|
||||
@ -54,5 +77,6 @@ GROUP BY timeslot
|
||||
```
|
||||
|
||||
**See Also**
|
||||
|
||||
- [-Map combinator for Map datatype](../combinators.md#-map)
|
||||
|
||||
- [Map combinator for Map datatype](../combinators.md#-map)
|
||||
- [sumMapWithOverflow](../reference/summapwithoverflow.md)
|
||||
|
||||
@ -0,0 +1,92 @@
|
||||
---
|
||||
slug: /en/sql-reference/aggregate-functions/reference/summapwithoverflow
|
||||
sidebar_position: 141
|
||||
---
|
||||
|
||||
# sumMapWithOverflow
|
||||
|
||||
Totals a `value` array according to the keys specified in the `key` array. Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
It differs from the [sumMap](../reference/summap.md) function in that it does summation with overflow - i.e. returns the same data type for the summation as the argument data type.
|
||||
|
||||
**Syntax**
|
||||
|
||||
- `sumMapWithOverflow(key <Array>, value <Array>)` [Array type](../../data-types/array.md).
|
||||
- `sumMapWithOverflow(Tuple(key <Array>, value <Array>))` [Tuple type](../../data-types/tuple.md).
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `key`: [Array](../../data-types/array.md) of keys.
|
||||
- `value`: [Array](../../data-types/array.md) of values.
|
||||
|
||||
Passing a tuple of key and value arrays is a synonym to passing separately an array of keys and an array of values.
|
||||
|
||||
:::note
|
||||
The number of elements in `key` and `value` must be the same for each row that is totaled.
|
||||
:::
|
||||
|
||||
**Returned Value**
|
||||
|
||||
- Returns a tuple of two arrays: keys in sorted order, and values summed for the corresponding keys.
|
||||
|
||||
**Example**
|
||||
|
||||
First we create a table called `sum_map`, and insert some data into it. Arrays of keys and values are stored separately as a column called `statusMap` of [Nested](../../data-types/nested-data-structures/index.md) type, and together as a column called `statusMapTuple` of [tuple](../../data-types/tuple.md) type to illustrate the use of the two different syntaxes of this function described above.
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
CREATE TABLE sum_map(
|
||||
date Date,
|
||||
timeslot DateTime,
|
||||
statusMap Nested(
|
||||
status UInt8,
|
||||
requests UInt8
|
||||
),
|
||||
statusMapTuple Tuple(Array(Int8), Array(Int8))
|
||||
) ENGINE = Log;
|
||||
```
|
||||
```sql
|
||||
INSERT INTO sum_map VALUES
|
||||
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10], ([1, 2, 3], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10], ([3, 4, 5], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10], ([4, 5, 6], [10, 10, 10])),
|
||||
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10], ([6, 7, 8], [10, 10, 10]));
|
||||
```
|
||||
|
||||
If we query the table using the `sumMap`, `sumMapWithOverflow` with the array type syntax, and `toTypeName` functions then we can see that
|
||||
for the `sumMapWithOverflow` function, the data type of the summed values array is the same as the argument type, both `UInt8` (i.e. summation was done with overflow). For `sumMap` the data type of the summed values arrays has changed from `UInt8` to `UInt64` such that overflow does not occur.
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
timeslot,
|
||||
toTypeName(sumMap(statusMap.status, statusMap.requests)),
|
||||
toTypeName(sumMapWithOverflow(statusMap.status, statusMap.requests)),
|
||||
FROM sum_map
|
||||
GROUP BY timeslot
|
||||
```
|
||||
|
||||
Equivalently we could have used the tuple syntax with for the same result.
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
timeslot,
|
||||
toTypeName(sumMap(statusMapTuple)),
|
||||
toTypeName(sumMapWithOverflow(statusMapTuple)),
|
||||
FROM sum_map
|
||||
GROUP BY timeslot
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌────────────timeslot─┬─toTypeName(sumMap(statusMap.status, statusMap.requests))─┬─toTypeName(sumMapWithOverflow(statusMap.status, statusMap.requests))─┐
|
||||
1. │ 2000-01-01 00:01:00 │ Tuple(Array(UInt8), Array(UInt64)) │ Tuple(Array(UInt8), Array(UInt8)) │
|
||||
2. │ 2000-01-01 00:00:00 │ Tuple(Array(UInt8), Array(UInt64)) │ Tuple(Array(UInt8), Array(UInt8)) │
|
||||
└─────────────────────┴──────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**See Also**
|
||||
|
||||
- [sumMap](../reference/summap.md)
|
||||
@ -8,3 +8,64 @@ sidebar_position: 140
|
||||
Computes the sum of the numbers, using the same data type for the result as for the input parameters. If the sum exceeds the maximum value for this data type, it is calculated with overflow.
|
||||
|
||||
Only works for numbers.
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
sumWithOverflow(num)
|
||||
```
|
||||
|
||||
**Parameters**
|
||||
- `num`: Column of numeric values. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
- The sum of the values. [(U)Int*](../../data-types/int-uint.md), [Float*](../../data-types/float.md), [Decimal*](../../data-types/decimal.md).
|
||||
|
||||
**Example**
|
||||
|
||||
First we create a table `employees` and insert some fictional employee data into it. For this example we will select `salary` as `UInt16` such that a sum of these values may produce an overflow.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
CREATE TABLE employees
|
||||
(
|
||||
`id` UInt32,
|
||||
`name` String,
|
||||
`monthly_salary` UInt16
|
||||
)
|
||||
ENGINE = Log
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
sum(monthly_salary) AS no_overflow,
|
||||
sumWithOverflow(monthly_salary) AS overflow,
|
||||
toTypeName(no_overflow),
|
||||
toTypeName(overflow)
|
||||
FROM employees
|
||||
```
|
||||
|
||||
We query for the total amount of the employee salaries using the `sum` and `sumWithOverflow` functions and show their types using the `toTypeName` function.
|
||||
For the `sum` function the resulting type is `UInt64`, big enough to contain the sum, whilst for `sumWithOverflow` the resulting type remains as `UInt16`.
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
sum(monthly_salary) AS no_overflow,
|
||||
sumWithOverflow(monthly_salary) AS overflow,
|
||||
toTypeName(no_overflow),
|
||||
toTypeName(overflow),
|
||||
FROM employees;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
|
||||
```response
|
||||
┌─no_overflow─┬─overflow─┬─toTypeName(no_overflow)─┬─toTypeName(overflow)─┐
|
||||
1. │ 118700 │ 53164 │ UInt64 │ UInt16 │
|
||||
└─────────────┴──────────┴─────────────────────────┴──────────────────────┘
|
||||
```
|
||||
@ -31,6 +31,8 @@ This function uses a numerically unstable algorithm. If you need numerical stabi
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
DROP TABLE IF EXISTS test_data;
|
||||
CREATE TABLE test_data
|
||||
@ -47,6 +49,8 @@ SELECT
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
3
|
||||
```
|
||||
@ -94,6 +98,8 @@ SELECT
|
||||
FROM test_data;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
0.5999999999999999
|
||||
```
|
||||
|
||||
@ -1052,7 +1052,7 @@ toStartOfWeek(t[, mode[, timezone]])
|
||||
**Arguments**
|
||||
|
||||
- `t` - a [Date](../data-types/date.md), [Date32](../data-types/date32.md), [DateTime](../data-types/datetime.md) or [DateTime64](../data-types/datetime64.md)
|
||||
- `mode` - determines the first day of the week as described in the [toWeek()](date-time-functions#toweek) function. Default: 0
|
||||
- `mode` - determines the first day of the week as described in the [toWeek()](date-time-functions#toweek) function
|
||||
- `timezone` - Optional parameter, it behaves like any other conversion function
|
||||
|
||||
**Returned value**
|
||||
@ -1413,7 +1413,7 @@ toStartOfFifteenMinutes(toDateTime('2023-04-21 10:20:00')): 2023-04-21 10:15:00
|
||||
toStartOfFifteenMinutes(toDateTime('2023-04-21 10:23:00')): 2023-04-21 10:15:00
|
||||
```
|
||||
|
||||
## toStartOfInterval
|
||||
## toStartOfInterval(date_or_date_with_time, INTERVAL x unit \[, time_zone\])
|
||||
|
||||
This function generalizes other `toStartOf*()` functions. For example,
|
||||
- `toStartOfInterval(t, INTERVAL 1 year)` returns the same as `toStartOfYear(t)`,
|
||||
@ -1440,8 +1440,6 @@ The calculation is performed relative to specific points in time:
|
||||
(*) hour intervals are special: the calculation is always performed relative to 00:00:00 (midnight) of the current day. As a result, only
|
||||
hour values between 1 and 23 are useful.
|
||||
|
||||
If unit `week` was specified, `toStartOfInterval` assumes by default that weeks start on Monday. You can change this behavior with setting [`first_day_of_week`](../../operations/settings/settings.md/#first-day-of-week)
|
||||
|
||||
**See Also**
|
||||
|
||||
- [date_trunc](#date_trunc)
|
||||
@ -1675,7 +1673,7 @@ Like [fromDaysSinceYearZero](#fromDaysSinceYearZero) but returns a [Date32](../.
|
||||
Returns the `unit` component of the difference between `startdate` and `enddate`. The difference is calculated using a precision of 1 nanosecond.
|
||||
E.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for `day` unit, 0 months for `month` unit, 0 years for `year` unit.
|
||||
|
||||
For an alternative to `age`, see function `date_diff`.
|
||||
For an alternative to `age`, see function `date\_diff`.
|
||||
|
||||
**Syntax**
|
||||
|
||||
@ -1749,9 +1747,9 @@ Result:
|
||||
Returns the count of the specified `unit` boundaries crossed between the `startdate` and the `enddate`.
|
||||
The difference is calculated using relative units, e.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for unit `day` (see [toRelativeDayNum](#torelativedaynum)), 1 month for unit `month` (see [toRelativeMonthNum](#torelativemonthnum)) and 1 year for unit `year` (see [toRelativeYearNum](#torelativeyearnum)).
|
||||
|
||||
If unit `week` was specified, `date_diff` assumes that weeks start on Monday. Note that this behavior is different from that of function `toWeek()` in which weeks start by default on Sunday.
|
||||
If unit `week` was specified, `date\_diff` assumes that weeks start on Monday. Note that this behavior is different from that of function `toWeek()` in which weeks start by default on Sunday.
|
||||
|
||||
For an alternative to `date_diff`, see function `age`.
|
||||
For an alternative to `date\_diff`, see function `age`.
|
||||
|
||||
**Syntax**
|
||||
|
||||
@ -2885,7 +2883,7 @@ Result:
|
||||
|
||||
## fromUnixTimestamp
|
||||
|
||||
This function converts a Unix timestamp to a calendar date and a time of a day.
|
||||
This function converts a Unix timestamp to a calendar date and a time of a day.
|
||||
|
||||
It can be called in two ways:
|
||||
|
||||
|
||||
@ -4,48 +4,72 @@ sidebar_label: SVG
|
||||
title: "Functions for Generating SVG images from Geo data"
|
||||
---
|
||||
|
||||
## Syntax
|
||||
## Svg
|
||||
|
||||
Returns a string of select SVG element tags from Geo data.
|
||||
|
||||
**Syntax**
|
||||
|
||||
``` sql
|
||||
SVG(geometry,[style])
|
||||
Svg(geometry,[style])
|
||||
```
|
||||
|
||||
### Parameters
|
||||
Aliases: `SVG`, `svg`
|
||||
|
||||
- `geometry` — Geo data
|
||||
- `style` — Optional style name
|
||||
**Parameters**
|
||||
|
||||
### Returned value
|
||||
- `geometry` — Geo data. [Geo](../../data-types/geo).
|
||||
- `style` — Optional style name. [String](../../data-types/string).
|
||||
|
||||
**Returned value**
|
||||
|
||||
- The SVG representation of the geometry:
|
||||
- SVG circle
|
||||
- SVG polygon
|
||||
- SVG path
|
||||
|
||||
Type: String
|
||||
Type: [String](../../data-types/string)
|
||||
|
||||
## Examples
|
||||
**Examples**
|
||||
|
||||
**Circle**
|
||||
|
||||
Query:
|
||||
|
||||
### Circle
|
||||
```sql
|
||||
SELECT SVG((0., 0.))
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
<circle cx="0" cy="0" r="5" style=""/>
|
||||
```
|
||||
|
||||
### Polygon
|
||||
**Polygon**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT SVG([(0., 0.), (10, 0), (10, 10), (0, 10)])
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
<polygon points="0,0 0,10 10,10 10,0 0,0" style=""/>
|
||||
```
|
||||
|
||||
### Path
|
||||
**Path**
|
||||
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT SVG([[(0., 0.), (10, 0), (10, 10), (0, 10)], [(4., 4.), (5, 4), (5, 5), (4, 5)]])
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```response
|
||||
<g fill-rule="evenodd"><path d="M 0,0 L 0,10 L 10,10 L 10,0 L 0,0M 4,4 L 5,4 L 5,5 L 4,5 L 4,4 z " style=""/></g>
|
||||
```
|
||||
|
||||
@ -1147,13 +1147,13 @@ tryBase58Decode(encoded)
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT tryBase58Decode('3dc8KtHrwM') as res;
|
||||
SELECT tryBase58Decode('3dc8KtHrwM') as res, tryBase58Decode('invalid') as res_invalid;
|
||||
```
|
||||
|
||||
```response
|
||||
┌─res─────┐
|
||||
│ Encoded │
|
||||
└─────────┘
|
||||
┌─res─────┬─res_invalid─┐
|
||||
│ Encoded │ │
|
||||
└─────────┴─────────────┘
|
||||
```
|
||||
|
||||
## base64Encode
|
||||
@ -1187,13 +1187,13 @@ tryBase64Decode(encoded)
|
||||
Query:
|
||||
|
||||
```sql
|
||||
SELECT tryBase64Decode('RW5jb2RlZA==') as res;
|
||||
SELECT tryBase64Decode('RW5jb2RlZA==') as res, tryBase64Decode('invalid') as res_invalid;
|
||||
```
|
||||
|
||||
```response
|
||||
┌─res─────┐
|
||||
│ Encoded │
|
||||
└─────────┘
|
||||
┌─res─────┬─res_invalid─┐
|
||||
│ Encoded │ │
|
||||
└─────────┴─────────────┘
|
||||
```
|
||||
|
||||
## endsWith {#endswith}
|
||||
|
||||
@ -532,3 +532,15 @@ If there's a refresh in progress for the given view, interrupt and cancel it. Ot
|
||||
```sql
|
||||
SYSTEM CANCEL VIEW [db.]name
|
||||
```
|
||||
|
||||
### SYSTEM UNLOAD PRIMARY KEY
|
||||
|
||||
Unload the primary keys for the given table or for all tables.
|
||||
|
||||
```sql
|
||||
SYSTEM UNLOAD PRIMARY KEY [db.]name
|
||||
```
|
||||
|
||||
```sql
|
||||
SYSTEM UNLOAD PRIMARY KEY
|
||||
```
|
||||
2
docs/ru/interfaces/third-party/gui.md
vendored
2
docs/ru/interfaces/third-party/gui.md
vendored
@ -150,7 +150,7 @@ sidebar_label: "Визуальные интерфейсы от сторонни
|
||||
|
||||
### ClickVisual {#clickvisual}
|
||||
|
||||
[ClickVisual](https://clickvisual.gocn.vip/) ClickVisual — это легкодоступная платформа для запросов, анализа и уведомлений. Документация на китайском.
|
||||
[ClickVisual](https://clickvisual.net/) ClickVisual — это легкодоступная платформа для запросов, анализа и уведомлений. Документация на китайском.
|
||||
|
||||
Основные возможности:
|
||||
|
||||
|
||||
2
docs/zh/interfaces/third-party/gui.md
vendored
2
docs/zh/interfaces/third-party/gui.md
vendored
@ -93,7 +93,7 @@ ClickHouse Web 界面 [Tabix](https://github.com/tabixio/tabix).
|
||||
|
||||
### ClickVisual {#clickvisual}
|
||||
|
||||
[ClickVisual](https://clickvisual.gocn.vip/) ClickVisual是一款轻量级的开源日志查询、分析、报警的可视化平台！
|
||||
[ClickVisual](https://clickvisual.net/) ClickVisual是一款轻量级的开源日志查询、分析、报警的可视化平台！
|
||||
|
||||
特征：
|
||||
|
||||
|
||||
@ -121,9 +121,12 @@ if (BUILD_STANDALONE_KEEPER)
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/DiskType.cpp
|
||||
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/IObjectStorage.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataOperationsHolder.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFromPlainObjectStorageOperations.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFromPlainObjectStorage.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFromPlainRewritableObjectStorage.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFromDisk.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataFromDiskTransactionState.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageTransactionState.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/DiskObjectStorageMetadata.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/DiskObjectStorage.cpp
|
||||
@ -137,6 +140,7 @@ if (BUILD_STANDALONE_KEEPER)
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/S3/S3Capabilities.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/S3/diskSettings.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/S3/DiskS3Utils.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/ObjectStorageFactory.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/MetadataStorageFactory.cpp
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../../src/Disks/ObjectStorages/RegisterDiskObjectStorage.cpp
|
||||
|
||||
@ -481,7 +481,6 @@ try
|
||||
registerFormats();
|
||||
|
||||
processConfig();
|
||||
adjustSettings();
|
||||
initTTYBuffer(toProgressOption(config().getString("progress", "default")));
|
||||
ASTAlterCommand::setFormatAlterCommandsWithParentheses(true);
|
||||
|
||||
@ -689,6 +688,7 @@ void LocalServer::processConfig()
|
||||
/// NOTE: it is important to apply any overrides before
|
||||
/// setDefaultProfiles() calls since it will copy current context (i.e.
|
||||
/// there is separate context for Buffer tables).
|
||||
adjustSettings();
|
||||
applySettingsOverridesForLocal(global_context);
|
||||
applyCmdOptions(global_context);
|
||||
|
||||
|
||||
@ -1573,8 +1573,11 @@ try
|
||||
|
||||
global_context->reloadQueryMaskingRulesIfChanged(config);
|
||||
|
||||
std::lock_guard lock(servers_lock);
|
||||
updateServers(*config, server_pool, async_metrics, servers, servers_to_start_before_tables);
|
||||
if (global_context->isServerCompletelyStarted())
|
||||
{
|
||||
std::lock_guard lock(servers_lock);
|
||||
updateServers(*config, server_pool, async_metrics, servers, servers_to_start_before_tables);
|
||||
}
|
||||
}
|
||||
|
||||
global_context->updateStorageConfiguration(*config);
|
||||
|
||||
@ -210,6 +210,7 @@ enum class AccessType
|
||||
M(SYSTEM_FAILPOINT, "SYSTEM ENABLE FAILPOINT, SYSTEM DISABLE FAILPOINT, SYSTEM WAIT FAILPOINT", GLOBAL, SYSTEM) \
|
||||
M(SYSTEM_LISTEN, "SYSTEM START LISTEN, SYSTEM STOP LISTEN", GLOBAL, SYSTEM) \
|
||||
M(SYSTEM_JEMALLOC, "SYSTEM JEMALLOC PURGE, SYSTEM JEMALLOC ENABLE PROFILE, SYSTEM JEMALLOC DISABLE PROFILE, SYSTEM JEMALLOC FLUSH PROFILE", GLOBAL, SYSTEM) \
|
||||
M(SYSTEM_UNLOAD_PRIMARY_KEY, "SYSTEM UNLOAD PRIMARY KEY", TABLE, SYSTEM) \
|
||||
M(SYSTEM, "", GROUP, ALL) /* allows to execute SYSTEM {SHUTDOWN|RELOAD CONFIG|...} */ \
|
||||
\
|
||||
M(dictGet, "dictHas, dictGetHierarchy, dictIsIn", DICTIONARY, ALL) /* allows to execute functions dictGet(), dictHas(), dictGetHierarchy(), dictIsIn() */\
|
||||
|
||||
@ -53,8 +53,9 @@ TEST(AccessRights, Union)
|
||||
"SHOW ROW POLICIES, SYSTEM MERGES, SYSTEM TTL MERGES, SYSTEM FETCHES, "
|
||||
"SYSTEM MOVES, SYSTEM PULLING REPLICATION LOG, SYSTEM CLEANUP, SYSTEM VIEWS, SYSTEM SENDS, SYSTEM REPLICATION QUEUES, SYSTEM VIRTUAL PARTS UPDATE, "
|
||||
"SYSTEM DROP REPLICA, SYSTEM SYNC REPLICA, SYSTEM RESTART REPLICA, "
|
||||
"SYSTEM RESTORE REPLICA, SYSTEM WAIT LOADING PARTS, SYSTEM SYNC DATABASE REPLICA, SYSTEM FLUSH DISTRIBUTED, dictGet ON db1.*, "
|
||||
"GRANT TABLE ENGINE ON db1, GRANT SET DEFINER ON db1, GRANT NAMED COLLECTION ADMIN ON db1");
|
||||
"SYSTEM RESTORE REPLICA, SYSTEM WAIT LOADING PARTS, SYSTEM SYNC DATABASE REPLICA, SYSTEM FLUSH DISTRIBUTED, "
|
||||
"SYSTEM UNLOAD PRIMARY KEY, dictGet ON db1.*, GRANT TABLE ENGINE ON db1, "
|
||||
"GRANT SET DEFINER ON db1, GRANT NAMED COLLECTION ADMIN ON db1");
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -10,9 +10,10 @@
|
||||
|
||||
#include <Interpreters/Context.h>
|
||||
|
||||
#include <Analyzer/InDepthQueryTreeVisitor.h>
|
||||
#include <Analyzer/ConstantNode.h>
|
||||
#include <Analyzer/FunctionNode.h>
|
||||
#include <Analyzer/InDepthQueryTreeVisitor.h>
|
||||
#include <Analyzer/Utils.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -52,17 +53,24 @@ public:
|
||||
const auto & second_const_value = second_const_node->getValue();
|
||||
if (second_const_value.isNull()
|
||||
|| (lower_name == "sum" && isInt64OrUInt64FieldType(second_const_value.getType()) && second_const_value.get<UInt64>() == 0
|
||||
&& !function_node->getResultType()->isNullable()))
|
||||
&& !if_node->getResultType()->isNullable()))
|
||||
{
|
||||
/// avg(if(cond, a, null)) -> avgIf(a, cond)
|
||||
/// avg(if(cond, nullable_a, null)) -> avgIfOrNull(a, cond)
|
||||
|
||||
/// avg(if(cond, a, null)) -> avgIf(a::ResultTypeIf, cond)
|
||||
/// avg(if(cond, nullable_a, null)) -> avgIf(nullable_a, cond)
|
||||
/// sum(if(cond, a, 0)) -> sumIf(a, cond)
|
||||
/// sum(if(cond, nullable_a, 0)) **is not** equivalent to sumIfOrNull(cond, nullable_a) as
|
||||
/// it changes the output when no rows pass the condition (from 0 to NULL)
|
||||
function_arguments_nodes.resize(2);
|
||||
function_arguments_nodes[0] = std::move(if_arguments_nodes[1]);
|
||||
function_arguments_nodes[1] = std::move(if_arguments_nodes[0]);
|
||||
|
||||
QueryTreeNodes new_arguments{2};
|
||||
|
||||
/// We need to preserve the output type from if()
|
||||
if (if_arguments_nodes[1]->getResultType()->getName() != if_node->getResultType()->getName())
|
||||
new_arguments[0] = createCastFunction(std::move(if_arguments_nodes[1]), if_node->getResultType(), getContext());
|
||||
else
|
||||
new_arguments[0] = std::move(if_arguments_nodes[1]);
|
||||
|
||||
new_arguments[1] = std::move(if_arguments_nodes[0]);
|
||||
function_arguments_nodes = std::move(new_arguments);
|
||||
resolveAsAggregateFunctionWithIf(
|
||||
*function_node, {function_arguments_nodes[0]->getResultType(), function_arguments_nodes[1]->getResultType()});
|
||||
}
|
||||
@ -72,21 +80,27 @@ public:
|
||||
const auto & first_const_value = first_const_node->getValue();
|
||||
if (first_const_value.isNull()
|
||||
|| (lower_name == "sum" && isInt64OrUInt64FieldType(first_const_value.getType()) && first_const_value.get<UInt64>() == 0
|
||||
&& !function_node->getResultType()->isNullable()))
|
||||
&& !if_node->getResultType()->isNullable()))
|
||||
{
|
||||
/// avg(if(cond, null, a) -> avgIfOrNullable(a, !cond))
|
||||
|
||||
/// avg(if(cond, null, a) -> avgIf(a::ResultTypeIf, !cond))
|
||||
/// sum(if(cond, 0, a) -> sumIf(a, !cond))
|
||||
/// sum(if(cond, 0, nullable_a) **is not** sumIf(a, !cond)) -> Same as above
|
||||
|
||||
QueryTreeNodes new_arguments{2};
|
||||
|
||||
if (if_arguments_nodes[2]->getResultType()->getName() != if_node->getResultType()->getName())
|
||||
new_arguments[0] = createCastFunction(std::move(if_arguments_nodes[2]), if_node->getResultType(), getContext());
|
||||
else
|
||||
new_arguments[0] = std::move(if_arguments_nodes[2]);
|
||||
|
||||
auto not_function = std::make_shared<FunctionNode>("not");
|
||||
auto & not_function_arguments = not_function->getArguments().getNodes();
|
||||
not_function_arguments.push_back(std::move(if_arguments_nodes[0]));
|
||||
not_function->resolveAsFunction(
|
||||
FunctionFactory::instance().get("not", getContext())->build(not_function->getArgumentColumns()));
|
||||
new_arguments[1] = std::move(not_function);
|
||||
|
||||
function_arguments_nodes.resize(2);
|
||||
function_arguments_nodes[0] = std::move(if_arguments_nodes[2]);
|
||||
function_arguments_nodes[1] = std::move(not_function);
|
||||
function_arguments_nodes = std::move(new_arguments);
|
||||
resolveAsAggregateFunctionWithIf(
|
||||
*function_node, {function_arguments_nodes[0]->getResultType(), function_arguments_nodes[1]->getResultType()});
|
||||
}
|
||||
@ -98,13 +112,9 @@ private:
|
||||
{
|
||||
auto result_type = function_node.getResultType();
|
||||
|
||||
std::string suffix = "If";
|
||||
if (result_type->isNullable())
|
||||
suffix = "OrNullIf";
|
||||
|
||||
AggregateFunctionProperties properties;
|
||||
auto aggregate_function = AggregateFunctionFactory::instance().get(
|
||||
function_node.getFunctionName() + suffix,
|
||||
function_node.getFunctionName() + "If",
|
||||
function_node.getNullsAction(),
|
||||
argument_types,
|
||||
function_node.getAggregateFunction()->getParameters(),
|
||||
|
||||
@ -258,8 +258,6 @@ void addQueryTreePasses(QueryTreePassManager & manager, bool only_analyze)
|
||||
manager.addPass(std::make_unique<RewriteSumFunctionWithSumAndCountPass>());
|
||||
manager.addPass(std::make_unique<CountDistinctPass>());
|
||||
manager.addPass(std::make_unique<UniqToCountPass>());
|
||||
manager.addPass(std::make_unique<RewriteAggregateFunctionWithIfPass>());
|
||||
manager.addPass(std::make_unique<SumIfToCountIfPass>());
|
||||
manager.addPass(std::make_unique<RewriteArrayExistsToHasPass>());
|
||||
manager.addPass(std::make_unique<NormalizeCountVariantsPass>());
|
||||
|
||||
@ -276,9 +274,12 @@ void addQueryTreePasses(QueryTreePassManager & manager, bool only_analyze)
|
||||
manager.addPass(std::make_unique<OptimizeGroupByFunctionKeysPass>());
|
||||
manager.addPass(std::make_unique<OptimizeGroupByInjectiveFunctionsPass>());
|
||||
|
||||
/// The order here is important as we want to keep collapsing in order
|
||||
manager.addPass(std::make_unique<MultiIfToIfPass>());
|
||||
manager.addPass(std::make_unique<IfConstantConditionPass>());
|
||||
manager.addPass(std::make_unique<IfChainToMultiIfPass>());
|
||||
manager.addPass(std::make_unique<RewriteAggregateFunctionWithIfPass>());
|
||||
manager.addPass(std::make_unique<SumIfToCountIfPass>());
|
||||
|
||||
manager.addPass(std::make_unique<ComparisonTupleEliminationPass>());
|
||||
|
||||
|
||||
@ -125,7 +125,7 @@ void highlight(const String & query, std::vector<replxx::Replxx::Color> & colors
|
||||
|
||||
const char * begin = query.data();
|
||||
const char * end = begin + query.size();
|
||||
Tokens tokens(begin, end, 1000, true);
|
||||
Tokens tokens(begin, end, 10000, true);
|
||||
IParser::Pos token_iterator(tokens, static_cast<uint32_t>(1000), static_cast<uint32_t>(10000));
|
||||
Expected expected;
|
||||
expected.enable_highlighting = true;
|
||||
|
||||
@ -1048,20 +1048,16 @@ public:
|
||||
|
||||
template <typename Date>
|
||||
requires std::is_same_v<Date, DayNum> || std::is_same_v<Date, ExtendedDayNum>
|
||||
auto toStartOfWeekInterval(Date d, UInt64 weeks, UInt8 week_mode) const
|
||||
auto toStartOfWeekInterval(Date d, UInt64 weeks) const
|
||||
{
|
||||
if (weeks == 1)
|
||||
return toFirstDayNumOfWeek(d, week_mode);
|
||||
|
||||
bool monday_first_mode = week_mode & static_cast<UInt8>(WeekModeFlag::MONDAY_FIRST);
|
||||
// January 1st 1970 was Thursday so we need this 4-days offset to make weeks start on Monday, or
|
||||
// 3 days to start on Sunday.
|
||||
auto offset = monday_first_mode ? 4 : 3;
|
||||
return toFirstDayNumOfWeek(d);
|
||||
UInt64 days = weeks * 7;
|
||||
// January 1st 1970 was Thursday so we need this 4-days offset to make weeks start on Monday.
|
||||
if constexpr (std::is_same_v<Date, DayNum>)
|
||||
return DayNum(offset + (d - offset) / days * days);
|
||||
return DayNum(4 + (d - 4) / days * days);
|
||||
else
|
||||
return ExtendedDayNum(static_cast<Int32>(offset + (d - offset) / days * days));
|
||||
return ExtendedDayNum(static_cast<Int32>(4 + (d - 4) / days * days));
|
||||
}
|
||||
|
||||
template <typename Date>
|
||||
|
||||
@ -598,6 +598,7 @@
|
||||
M(717, EXPERIMENTAL_FEATURE_ERROR) \
|
||||
M(718, TOO_SLOW_PARSING) \
|
||||
M(719, QUERY_CACHE_USED_WITH_SYSTEM_TABLE) \
|
||||
M(720, USER_EXPIRED) \
|
||||
\
|
||||
M(900, DISTRIBUTED_CACHE_ERROR) \
|
||||
M(901, CANNOT_USE_DISTRIBUTED_CACHE) \
|
||||
|
||||
@ -14,10 +14,7 @@ public:
|
||||
, re_gen(key_template)
|
||||
{
|
||||
}
|
||||
DB::ObjectStorageKey generate(const String &) const override
|
||||
{
|
||||
return DB::ObjectStorageKey::createAsAbsolute(re_gen.generate());
|
||||
}
|
||||
DB::ObjectStorageKey generate(const String &, bool) const override { return DB::ObjectStorageKey::createAsAbsolute(re_gen.generate()); }
|
||||
|
||||
private:
|
||||
String key_template;
|
||||
@ -32,7 +29,7 @@ public:
|
||||
: key_prefix(std::move(key_prefix_))
|
||||
{}
|
||||
|
||||
DB::ObjectStorageKey generate(const String &) const override
|
||||
DB::ObjectStorageKey generate(const String &, bool) const override
|
||||
{
|
||||
/// Path to store the new S3 object.
|
||||
|
||||
@ -63,7 +60,7 @@ public:
|
||||
: key_prefix(std::move(key_prefix_))
|
||||
{}
|
||||
|
||||
DB::ObjectStorageKey generate(const String & path) const override
|
||||
DB::ObjectStorageKey generate(const String & path, bool) const override
|
||||
{
|
||||
return DB::ObjectStorageKey::createAsRelative(key_prefix, path);
|
||||
}
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
#pragma once
|
||||
|
||||
#include "ObjectStorageKey.h"
|
||||
#include <memory>
|
||||
#include "ObjectStorageKey.h"
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -9,8 +9,9 @@ namespace DB
|
||||
class IObjectStorageKeysGenerator
|
||||
{
|
||||
public:
|
||||
virtual ObjectStorageKey generate(const String & path) const = 0;
|
||||
virtual ~IObjectStorageKeysGenerator() = default;
|
||||
|
||||
virtual ObjectStorageKey generate(const String & path, bool is_directory) const = 0;
|
||||
};
|
||||
|
||||
using ObjectStorageKeysGeneratorPtr = std::shared_ptr<IObjectStorageKeysGenerator>;
|
||||
|
||||
@ -50,6 +50,7 @@ class IColumn;
|
||||
M(MaxThreads, max_threads, 0, "The maximum number of threads to execute the request. By default, it is determined automatically.", 0) \
|
||||
M(Bool, use_concurrency_control, true, "Respect the server's concurrency control (see the `concurrent_threads_soft_limit_num` and `concurrent_threads_soft_limit_ratio_to_cores` global server settings). If disabled, it allows using a larger number of threads even if the server is overloaded (not recommended for normal usage, and needed mostly for tests).", 0) \
|
||||
M(MaxThreads, max_download_threads, 4, "The maximum number of threads to download data (e.g. for URL engine).", 0) \
|
||||
M(MaxThreads, max_parsing_threads, 0, "The maximum number of threads to parse data in input formats that support parallel parsing. By default, it is determined automatically", 0) \
|
||||
M(UInt64, max_download_buffer_size, 10*1024*1024, "The maximal size of buffer for parallel downloading (e.g. for URL engine) per each thread.", 0) \
|
||||
M(UInt64, max_read_buffer_size, DBMS_DEFAULT_BUFFER_SIZE, "The maximum size of the buffer to read from the filesystem.", 0) \
|
||||
M(UInt64, max_read_buffer_size_local_fs, 128*1024, "The maximum size of the buffer to read from local filesystem. If set to 0 then max_read_buffer_size will be used.", 0) \
|
||||
@ -234,7 +235,7 @@ class IColumn;
|
||||
M(Bool, do_not_merge_across_partitions_select_final, false, "Merge parts only in one partition in select final", 0) \
|
||||
M(Bool, split_parts_ranges_into_intersecting_and_non_intersecting_final, true, "Split parts ranges into intersecting and non intersecting during FINAL optimization", 0) \
|
||||
M(Bool, split_intersecting_parts_ranges_into_layers_final, true, "Split intersecting parts ranges into layers during FINAL optimization", 0) \
|
||||
M(Bool, allow_experimental_inverted_index, false, "If it is set to true, allow to use experimental inverted index.", 0) \
|
||||
M(Bool, allow_experimental_inverted_index, false, "If it is set to true, allow to use experimental fulltext (inverted) index.", 0) \
|
||||
\
|
||||
M(UInt64, mysql_max_rows_to_insert, 65536, "The maximum number of rows in MySQL batch insertion of the MySQL storage engine", 0) \
|
||||
M(Bool, mysql_map_string_to_text_in_show_columns, true, "If enabled, String type will be mapped to TEXT in SHOW [FULL] COLUMNS, BLOB otherwise. Has an effect only when the connection is made through the MySQL wire protocol.", 0) \
|
||||
@ -551,7 +552,6 @@ class IColumn;
|
||||
M(Bool, formatdatetime_parsedatetime_m_is_month_name, true, "Formatter '%M' in functions 'formatDateTime()' and 'parseDateTime()' print/parse the month name instead of minutes.", 0) \
|
||||
M(Bool, parsedatetime_parse_without_leading_zeros, true, "Formatters '%c', '%l' and '%k' in function 'parseDateTime()' parse months and hours without leading zeros.", 0) \
|
||||
M(Bool, formatdatetime_format_without_leading_zeros, false, "Formatters '%c', '%l' and '%k' in function 'formatDateTime()' print months and hours without leading zeros.", 0) \
|
||||
M(FirstDayOfWeek, first_day_of_week, FirstDayOfWeek::Monday, "The first day of the week (Monday or Sunday) used by date/time functions (default: Monday).", 0) \
|
||||
\
|
||||
M(UInt64, max_partitions_per_insert_block, 100, "Limit maximum number of partitions in single INSERTed block. Zero means unlimited. Throw exception if the block contains too many partitions. This setting is a safety threshold, because using large number of partitions is a common misconception.", 0) \
|
||||
M(Bool, throw_on_max_partitions_per_insert_block, true, "Used with max_partitions_per_insert_block. If true (default), an exception will be thrown when max_partitions_per_insert_block is reached. If false, details of the insert query reaching this limit with the number of partitions will be logged. This can be useful if you're trying to understand the impact on users when changing max_partitions_per_insert_block.", 0) \
|
||||
@ -683,7 +683,7 @@ class IColumn;
|
||||
M(Bool, query_cache_share_between_users, false, "Allow other users to read entry in the query cache", 0) \
|
||||
M(Bool, enable_sharing_sets_for_mutations, true, "Allow sharing set objects build for IN subqueries between different tasks of the same mutation. This reduces memory usage and CPU consumption", 0) \
|
||||
\
|
||||
M(Bool, optimize_rewrite_sum_if_to_count_if, false, "Rewrite sumIf() and sum(if()) function countIf() function when logically equivalent", 0) \
|
||||
M(Bool, optimize_rewrite_sum_if_to_count_if, true, "Rewrite sumIf() and sum(if()) function countIf() function when logically equivalent", 0) \
|
||||
M(Bool, optimize_rewrite_aggregate_function_with_if, true, "Rewrite aggregate functions with if expression as argument when logically equivalent. For example, avg(if(cond, col, null)) can be rewritten to avgIf(cond, col)", 0) \
|
||||
M(Bool, optimize_rewrite_array_exists_to_has, false, "Rewrite arrayExists() functions to has() when logically equivalent. For example, arrayExists(x -> x = 1, arr) can be rewritten to has(arr, 1)", 0) \
|
||||
M(UInt64, insert_shard_id, 0, "If non zero, when insert into a distributed table, the data will be inserted into the shard `insert_shard_id` synchronously. Possible values range from 1 to `shards_number` of corresponding distributed table", 0) \
|
||||
|
||||
@ -86,14 +86,15 @@ namespace SettingsChangesHistory
|
||||
static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> settings_changes_history =
|
||||
{
|
||||
{"24.4", {{"input_format_json_throw_on_bad_escape_sequence", true, true, "Allow to save JSON strings with bad escape sequences"},
|
||||
{"max_parsing_threads", 0, 0, "Add a separate setting to control number of threads in parallel parsing from files"},
|
||||
{"ignore_drop_queries_probability", 0, 0, "Allow to ignore drop queries in server with specified probability for testing purposes"},
|
||||
{"lightweight_deletes_sync", 2, 2, "The same as 'mutation_sync', but controls only execution of lightweight deletes"},
|
||||
{"query_cache_system_table_handling", "save", "throw", "The query cache no longer caches results of queries against system tables"},
|
||||
{"input_format_json_ignore_unnecessary_fields", false, true, "Ignore unnecessary fields and not parse them. Enabling this may not throw exceptions on json strings of invalid format or with duplicated fields"},
|
||||
{"input_format_hive_text_allow_variable_number_of_columns", false, true, "Ignore extra columns in Hive Text input (if file has more columns than expected) and treat missing fields in Hive Text input as default values."},
|
||||
{"first_day_of_week", "Monday", "Monday", "Added a setting for the first day of the week for date/time functions"},
|
||||
{"allow_experimental_database_replicated", false, true, "Database engine Replicated is now in Beta stage"},
|
||||
{"temporary_data_in_cache_reserve_space_wait_lock_timeout_milliseconds", (10 * 60 * 1000), (10 * 60 * 1000), "Wait time to lock cache for sapce reservation in temporary data in filesystem cache"},
|
||||
{"optimize_rewrite_sum_if_to_count_if", false, true, "Only available for the analyzer, where it works correctly"},
|
||||
{"azure_allow_parallel_part_upload", "true", "true", "Use multiple threads for azure multipart upload."},
|
||||
{"allow_experimental_join_condition", false, false, "Support join with inequal conditions which involve columns from both left and right table. e.g. t1.y < t2.y."},
|
||||
{"max_recursive_cte_evaluation_depth", DBMS_RECURSIVE_CTE_MAX_EVALUATION_DEPTH, DBMS_RECURSIVE_CTE_MAX_EVALUATION_DEPTH, "Maximum limit on recursive CTE evaluation depth"},
|
||||
|
||||
@ -229,8 +229,4 @@ IMPLEMENT_SETTING_ENUM(SQLSecurityType, ErrorCodes::BAD_ARGUMENTS,
|
||||
{{"DEFINER", SQLSecurityType::DEFINER},
|
||||
{"INVOKER", SQLSecurityType::INVOKER},
|
||||
{"NONE", SQLSecurityType::NONE}})
|
||||
|
||||
IMPLEMENT_SETTING_ENUM(FirstDayOfWeek, ErrorCodes::BAD_ARGUMENTS,
|
||||
{{"Monday", FirstDayOfWeek::Monday},
|
||||
{"Sunday", FirstDayOfWeek::Sunday}})
|
||||
}
|
||||
|
||||
@ -370,12 +370,4 @@ DECLARE_SETTING_ENUM(SchemaInferenceMode)
|
||||
DECLARE_SETTING_ENUM_WITH_RENAME(DateTimeOverflowBehavior, FormatSettings::DateTimeOverflowBehavior)
|
||||
|
||||
DECLARE_SETTING_ENUM(SQLSecurityType)
|
||||
|
||||
enum class FirstDayOfWeek
|
||||
{
|
||||
Monday,
|
||||
Sunday
|
||||
};
|
||||
|
||||
DECLARE_SETTING_ENUM(FirstDayOfWeek)
|
||||
}
|
||||
|
||||
@ -48,11 +48,6 @@ bool queryProfilerWorks() { return false; }
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int INVALID_SETTING_VALUE;
|
||||
}
|
||||
|
||||
/// Update some settings defaults to avoid some known issues.
|
||||
void applySettingsQuirks(Settings & settings, LoggerPtr log)
|
||||
{
|
||||
@ -95,7 +90,7 @@ void applySettingsQuirks(Settings & settings, LoggerPtr log)
|
||||
}
|
||||
}
|
||||
|
||||
void doSettingsSanityCheck(const Settings & current_settings)
|
||||
void doSettingsSanityCheckClamp(Settings & current_settings, LoggerPtr log)
|
||||
{
|
||||
auto getCurrentValue = [¤t_settings](const std::string_view name) -> Field
|
||||
{
|
||||
@ -106,8 +101,13 @@ void doSettingsSanityCheck(const Settings & current_settings)
|
||||
};
|
||||
|
||||
UInt64 max_threads = getCurrentValue("max_threads").get<UInt64>();
|
||||
if (max_threads > getNumberOfPhysicalCPUCores() * 65536)
|
||||
throw Exception(ErrorCodes::INVALID_SETTING_VALUE, "Sanity check: Too many threads requested ({})", max_threads);
|
||||
UInt64 max_threads_max_value = 256 * getNumberOfPhysicalCPUCores();
|
||||
if (max_threads > max_threads_max_value)
|
||||
{
|
||||
if (log)
|
||||
LOG_WARNING(log, "Sanity check: Too many threads requested ({}). Reduced to {}", max_threads, max_threads_max_value);
|
||||
current_settings.set("max_threads", max_threads_max_value);
|
||||
}
|
||||
|
||||
constexpr UInt64 max_sane_block_rows_size = 4294967296; // 2^32
|
||||
std::unordered_set<String> block_rows_settings{
|
||||
@ -122,7 +122,11 @@ void doSettingsSanityCheck(const Settings & current_settings)
|
||||
{
|
||||
auto block_size = getCurrentValue(setting).get<UInt64>();
|
||||
if (block_size > max_sane_block_rows_size)
|
||||
throw Exception(ErrorCodes::INVALID_SETTING_VALUE, "Sanity check: '{}' value is too high ({})", setting, block_size);
|
||||
{
|
||||
if (log)
|
||||
LOG_WARNING(log, "Sanity check: '{}' value is too high ({}). Reduced to {}", setting, block_size, max_sane_block_rows_size);
|
||||
current_settings.set(setting, max_sane_block_rows_size);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -10,6 +10,6 @@ struct Settings;
|
||||
/// Update some settings defaults to avoid some known issues.
|
||||
void applySettingsQuirks(Settings & settings, LoggerPtr log = nullptr);
|
||||
|
||||
/// Verify that some settings have sane values. Throws if not
|
||||
void doSettingsSanityCheck(const Settings & settings);
|
||||
/// Verify that some settings have sane values. Alters the value to a reasonable one if not
|
||||
void doSettingsSanityCheckClamp(Settings & settings, LoggerPtr log);
|
||||
}
|
||||
|
||||
@ -151,8 +151,7 @@ static void checkMySQLVariables(const mysqlxx::Pool::Entry & connection, const S
|
||||
{"log_bin", "ON"},
|
||||
{"binlog_format", "ROW"},
|
||||
{"binlog_row_image", "FULL"},
|
||||
{"default_authentication_plugin", "mysql_native_password"},
|
||||
{"log_bin_use_v1_row_events", "OFF"}
|
||||
{"default_authentication_plugin", "mysql_native_password"}
|
||||
};
|
||||
|
||||
QueryPipeline pipeline(std::move(variables_input));
|
||||
|
||||
@ -16,6 +16,8 @@ MetadataStorageType metadataTypeFromString(const String & type)
|
||||
return MetadataStorageType::Local;
|
||||
if (check_type == "plain")
|
||||
return MetadataStorageType::Plain;
|
||||
if (check_type == "plain_rewritable")
|
||||
return MetadataStorageType::PlainRewritable;
|
||||
if (check_type == "web")
|
||||
return MetadataStorageType::StaticWeb;
|
||||
|
||||
|
||||
@ -28,6 +28,7 @@ enum class MetadataStorageType
|
||||
None,
|
||||
Local,
|
||||
Plain,
|
||||
PlainRewritable,
|
||||
StaticWeb,
|
||||
};
|
||||
|
||||
|
||||
@ -363,6 +363,8 @@ public:
|
||||
|
||||
virtual bool isWriteOnce() const { return false; }
|
||||
|
||||
virtual bool supportsHardLinks() const { return true; }
|
||||
|
||||
/// Check if disk is broken. Broken disks will have 0 space and cannot be used.
|
||||
virtual bool isBroken() const { return false; }
|
||||
|
||||
|
||||
72
src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.cpp
Normal file
72
src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.cpp
Normal file
@ -0,0 +1,72 @@
|
||||
#include "CommonPathPrefixKeyGenerator.h"
|
||||
|
||||
#include <Common/getRandomASCIIString.h>
|
||||
|
||||
#include <deque>
|
||||
#include <filesystem>
|
||||
#include <tuple>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
CommonPathPrefixKeyGenerator::CommonPathPrefixKeyGenerator(
|
||||
String key_prefix_, SharedMutex & shared_mutex_, std::weak_ptr<PathMap> path_map_)
|
||||
: storage_key_prefix(key_prefix_), shared_mutex(shared_mutex_), path_map(std::move(path_map_))
|
||||
{
|
||||
}
|
||||
|
||||
ObjectStorageKey CommonPathPrefixKeyGenerator::generate(const String & path, bool is_directory) const
|
||||
{
|
||||
const auto & [object_key_prefix, suffix_parts] = getLongestObjectKeyPrefix(path);
|
||||
|
||||
auto key = std::filesystem::path(object_key_prefix.empty() ? storage_key_prefix : object_key_prefix);
|
||||
|
||||
/// The longest prefix is the same as path, meaning that the path is already mapped.
|
||||
if (suffix_parts.empty())
|
||||
return ObjectStorageKey::createAsRelative(std::move(key));
|
||||
|
||||
/// File and top-level directory paths are mapped as is.
|
||||
if (!is_directory || object_key_prefix.empty())
|
||||
for (const auto & part : suffix_parts)
|
||||
key /= part;
|
||||
/// Replace the last part of the directory path with a pseudorandom suffix.
|
||||
else
|
||||
{
|
||||
for (size_t i = 0; i + 1 < suffix_parts.size(); ++i)
|
||||
key /= suffix_parts[i];
|
||||
|
||||
constexpr size_t part_size = 16;
|
||||
key /= getRandomASCIIString(part_size);
|
||||
}
|
||||
|
||||
return ObjectStorageKey::createAsRelative(key);
|
||||
}
|
||||
|
||||
std::tuple<std::string, std::vector<std::string>> CommonPathPrefixKeyGenerator::getLongestObjectKeyPrefix(const std::string & path) const
|
||||
{
|
||||
std::filesystem::path p(path);
|
||||
std::deque<std::string> dq;
|
||||
|
||||
std::shared_lock lock(shared_mutex);
|
||||
|
||||
auto ptr = path_map.lock();
|
||||
|
||||
while (p != p.root_path())
|
||||
{
|
||||
auto it = ptr->find(p / "");
|
||||
if (it != ptr->end())
|
||||
{
|
||||
std::vector<std::string> vec(std::make_move_iterator(dq.begin()), std::make_move_iterator(dq.end()));
|
||||
return std::make_tuple(it->second, std::move(vec));
|
||||
}
|
||||
|
||||
if (!p.filename().empty())
|
||||
dq.push_front(p.filename());
|
||||
|
||||
p = p.parent_path();
|
||||
}
|
||||
|
||||
return {std::string(), std::vector<std::string>(std::make_move_iterator(dq.begin()), std::make_move_iterator(dq.end()))};
|
||||
}
|
||||
|
||||
}
|
||||
41
src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.h
Normal file
41
src/Disks/ObjectStorages/CommonPathPrefixKeyGenerator.h
Normal file
@ -0,0 +1,41 @@
|
||||
#pragma once
|
||||
|
||||
#include <Common/ObjectStorageKeyGenerator.h>
|
||||
#include <Common/SharedMutex.h>
|
||||
|
||||
#include <filesystem>
|
||||
#include <map>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/// Object storage key generator used specifically with the
|
||||
/// MetadataStorageFromPlainObjectStorage if multiple writes are allowed.
|
||||
|
||||
/// It searches for the local (metadata) path in a pre-loaded path map.
|
||||
/// If no such path exists, it searches for the parent path, until it is found
|
||||
/// or no parent path exists.
|
||||
///
|
||||
/// The key generator ensures that the original directory hierarchy is
|
||||
/// preserved, which is required for the MergeTree family.
|
||||
class CommonPathPrefixKeyGenerator : public IObjectStorageKeysGenerator
|
||||
{
|
||||
public:
|
||||
/// Local to remote path map. Leverages filesystem::path comparator for paths.
|
||||
using PathMap = std::map<std::filesystem::path, std::string>;
|
||||
|
||||
explicit CommonPathPrefixKeyGenerator(String key_prefix_, SharedMutex & shared_mutex_, std::weak_ptr<PathMap> path_map_);
|
||||
|
||||
ObjectStorageKey generate(const String & path, bool is_directory) const override;
|
||||
|
||||
private:
|
||||
/// Longest key prefix and unresolved parts of the source path.
|
||||
std::tuple<std::string, std::vector<String>> getLongestObjectKeyPrefix(const String & path) const;
|
||||
|
||||
const String storage_key_prefix;
|
||||
|
||||
SharedMutex & shared_mutex;
|
||||
std::weak_ptr<PathMap> path_map;
|
||||
};
|
||||
|
||||
}
|
||||
@ -112,20 +112,21 @@ size_t DiskObjectStorage::getFileSize(const String & path) const
|
||||
return metadata_storage->getFileSize(path);
|
||||
}
|
||||
|
||||
void DiskObjectStorage::moveDirectory(const String & from_path, const String & to_path)
|
||||
{
|
||||
if (send_metadata)
|
||||
sendMoveMetadata(from_path, to_path);
|
||||
|
||||
auto transaction = createObjectStorageTransaction();
|
||||
transaction->moveDirectory(from_path, to_path);
|
||||
transaction->commit();
|
||||
}
|
||||
|
||||
void DiskObjectStorage::moveFile(const String & from_path, const String & to_path, bool should_send_metadata)
|
||||
{
|
||||
|
||||
if (should_send_metadata)
|
||||
{
|
||||
auto revision = metadata_helper->revision_counter + 1;
|
||||
metadata_helper->revision_counter += 1;
|
||||
|
||||
const ObjectAttributes object_metadata {
|
||||
{"from_path", from_path},
|
||||
{"to_path", to_path}
|
||||
};
|
||||
metadata_helper->createFileOperationObject("rename", revision, object_metadata);
|
||||
}
|
||||
sendMoveMetadata(from_path, to_path);
|
||||
|
||||
auto transaction = createObjectStorageTransaction();
|
||||
transaction->moveFile(from_path, to_path);
|
||||
@ -409,6 +410,15 @@ bool DiskObjectStorage::tryReserve(UInt64 bytes)
|
||||
|
||||
return false;
|
||||
}
|
||||
void DiskObjectStorage::sendMoveMetadata(const String & from_path, const String & to_path)
|
||||
{
|
||||
chassert(send_metadata);
|
||||
auto revision = metadata_helper->revision_counter + 1;
|
||||
metadata_helper->revision_counter += 1;
|
||||
|
||||
const ObjectAttributes object_metadata{{"from_path", from_path}, {"to_path", to_path}};
|
||||
metadata_helper->createFileOperationObject("rename", revision, object_metadata);
|
||||
}
|
||||
|
||||
bool DiskObjectStorage::supportsCache() const
|
||||
{
|
||||
@ -425,6 +435,11 @@ bool DiskObjectStorage::isWriteOnce() const
|
||||
return object_storage->isWriteOnce();
|
||||
}
|
||||
|
||||
bool DiskObjectStorage::supportsHardLinks() const
|
||||
{
|
||||
return !isWriteOnce() && !object_storage->isPlain();
|
||||
}
|
||||
|
||||
DiskObjectStoragePtr DiskObjectStorage::createDiskObjectStorage()
|
||||
{
|
||||
const auto config_prefix = "storage_configuration.disks." + name;
|
||||
|
||||
@ -112,7 +112,7 @@ public:
|
||||
|
||||
void clearDirectory(const String & path) override;
|
||||
|
||||
void moveDirectory(const String & from_path, const String & to_path) override { moveFile(from_path, to_path); }
|
||||
void moveDirectory(const String & from_path, const String & to_path) override;
|
||||
|
||||
void removeDirectory(const String & path) override;
|
||||
|
||||
@ -183,6 +183,8 @@ public:
|
||||
/// MergeTree table on this disk.
|
||||
bool isWriteOnce() const override;
|
||||
|
||||
bool supportsHardLinks() const override;
|
||||
|
||||
/// Get structure of object storage this disk works with. Examples:
|
||||
/// DiskObjectStorage(S3ObjectStorage)
|
||||
/// DiskObjectStorage(CachedObjectStorage(S3ObjectStorage))
|
||||
@ -228,6 +230,7 @@ private:
|
||||
std::mutex reservation_mutex;
|
||||
|
||||
bool tryReserve(UInt64 bytes);
|
||||
void sendMoveMetadata(const String & from_path, const String & to_path);
|
||||
|
||||
const bool send_metadata;
|
||||
|
||||
|
||||
@ -507,7 +507,7 @@ struct CopyFileObjectStorageOperation final : public IDiskObjectStorageOperation
|
||||
std::string to_path;
|
||||
|
||||
StoredObjects created_objects;
|
||||
IObjectStorage& destination_object_storage;
|
||||
IObjectStorage & destination_object_storage;
|
||||
|
||||
CopyFileObjectStorageOperation(
|
||||
IObjectStorage & object_storage_,
|
||||
@ -714,7 +714,7 @@ std::unique_ptr<WriteBufferFromFileBase> DiskObjectStorageTransaction::writeFile
|
||||
{
|
||||
/// Otherwise we will produce lost blobs which nobody points to
|
||||
/// WriteOnce storages are not affected by the issue
|
||||
if (!tx->object_storage.isWriteOnce() && tx->metadata_storage.exists(path))
|
||||
if (!tx->object_storage.isPlain() && tx->metadata_storage.exists(path))
|
||||
tx->object_storage.removeObjectsIfExist(tx->metadata_storage.getStorageObjects(path));
|
||||
|
||||
tx->metadata_transaction->createMetadataFile(path, key_, count);
|
||||
@ -747,10 +747,9 @@ std::unique_ptr<WriteBufferFromFileBase> DiskObjectStorageTransaction::writeFile
|
||||
{
|
||||
/// Otherwise we will produce lost blobs which nobody points to
|
||||
/// WriteOnce storages are not affected by the issue
|
||||
if (!object_storage_tx->object_storage.isWriteOnce() && object_storage_tx->metadata_storage.exists(path))
|
||||
if (!object_storage_tx->object_storage.isPlain() && object_storage_tx->metadata_storage.exists(path))
|
||||
{
|
||||
object_storage_tx->object_storage.removeObjectsIfExist(
|
||||
object_storage_tx->metadata_storage.getStorageObjects(path));
|
||||
object_storage_tx->object_storage.removeObjectsIfExist(object_storage_tx->metadata_storage.getStorageObjects(path));
|
||||
}
|
||||
|
||||
tx->createMetadataFile(path, key_, count);
|
||||
@ -877,14 +876,14 @@ void DiskObjectStorageTransaction::createFile(const std::string & path)
|
||||
|
||||
void DiskObjectStorageTransaction::copyFile(const std::string & from_file_path, const std::string & to_file_path, const ReadSettings & read_settings, const WriteSettings & write_settings)
|
||||
{
|
||||
operations_to_execute.emplace_back(
|
||||
std::make_unique<CopyFileObjectStorageOperation>(object_storage, metadata_storage, object_storage, read_settings, write_settings, from_file_path, to_file_path));
|
||||
operations_to_execute.emplace_back(std::make_unique<CopyFileObjectStorageOperation>(
|
||||
object_storage, metadata_storage, object_storage, read_settings, write_settings, from_file_path, to_file_path));
|
||||
}
|
||||
|
||||
void MultipleDisksObjectStorageTransaction::copyFile(const std::string & from_file_path, const std::string & to_file_path, const ReadSettings & read_settings, const WriteSettings & write_settings)
|
||||
{
|
||||
operations_to_execute.emplace_back(
|
||||
std::make_unique<CopyFileObjectStorageOperation>(object_storage, metadata_storage, destination_object_storage, read_settings, write_settings, from_file_path, to_file_path));
|
||||
operations_to_execute.emplace_back(std::make_unique<CopyFileObjectStorageOperation>(
|
||||
object_storage, metadata_storage, destination_object_storage, read_settings, write_settings, from_file_path, to_file_path));
|
||||
}
|
||||
|
||||
void DiskObjectStorageTransaction::commit()
|
||||
|
||||
19
src/Disks/ObjectStorages/IMetadataOperation.h
Normal file
19
src/Disks/ObjectStorages/IMetadataOperation.h
Normal file
@ -0,0 +1,19 @@
|
||||
#pragma once
|
||||
|
||||
#include <mutex>
|
||||
#include <Common/SharedMutex.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
struct IMetadataOperation
|
||||
{
|
||||
virtual void execute(std::unique_lock<SharedMutex> & metadata_lock) = 0;
|
||||
virtual void undo(std::unique_lock<SharedMutex> & metadata_lock) = 0;
|
||||
virtual void finalize() { }
|
||||
virtual ~IMetadataOperation() = default;
|
||||
};
|
||||
|
||||
using MetadataOperationPtr = std::unique_ptr<IMetadataOperation>;
|
||||
|
||||
}
|
||||
@ -145,7 +145,7 @@ public:
|
||||
|
||||
virtual ~IMetadataTransaction() = default;
|
||||
|
||||
private:
|
||||
protected:
|
||||
[[noreturn]] static void throwNotImplemented()
|
||||
{
|
||||
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Operation is not implemented");
|
||||
@ -229,7 +229,7 @@ public:
|
||||
/// object_storage_path is absolute.
|
||||
virtual StoredObjects getStorageObjects(const std::string & path) const = 0;
|
||||
|
||||
private:
|
||||
protected:
|
||||
[[noreturn]] static void throwNotImplemented()
|
||||
{
|
||||
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Operation is not implemented");
|
||||
|
||||
@ -1,11 +1,12 @@
|
||||
#include <Disks/ObjectStorages/IObjectStorage.h>
|
||||
#include <Disks/IO/ThreadPoolRemoteFSReader.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Disks/ObjectStorages/IObjectStorage.h>
|
||||
#include <Disks/ObjectStorages/ObjectStorageIterator.h>
|
||||
#include <IO/ReadBufferFromFileBase.h>
|
||||
#include <IO/WriteBufferFromFileBase.h>
|
||||
#include <IO/copyData.h>
|
||||
#include <IO/ReadBufferFromFileBase.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Disks/ObjectStorages/ObjectStorageIterator.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/ObjectStorageKeyGenerator.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -83,6 +83,9 @@ using ObjectKeysWithMetadata = std::vector<ObjectKeyWithMetadata>;
|
||||
class IObjectStorageIterator;
|
||||
using ObjectStorageIteratorPtr = std::shared_ptr<IObjectStorageIterator>;
|
||||
|
||||
class IObjectStorageKeysGenerator;
|
||||
using ObjectStorageKeysGeneratorPtr = std::shared_ptr<IObjectStorageKeysGenerator>;
|
||||
|
||||
/// Base class for all object storages which implement some subset of ordinary filesystem operations.
|
||||
///
|
||||
/// Examples of object storages are S3, Azure Blob Storage, HDFS.
|
||||
@ -208,6 +211,12 @@ public:
|
||||
/// Path can be generated either independently or based on `path`.
|
||||
virtual ObjectStorageKey generateObjectKeyForPath(const std::string & path) const = 0;
|
||||
|
||||
/// Object key prefix for local paths in the directory 'path'.
|
||||
virtual ObjectStorageKey generateObjectKeyPrefixForDirectoryPath(const std::string & /* path */) const
|
||||
{
|
||||
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Method 'generateObjectKeyPrefixForDirectoryPath' is not implemented");
|
||||
}
|
||||
|
||||
/// Get unique id for passed absolute path in object storage.
|
||||
virtual std::string getUniqueId(const std::string & path) const { return path; }
|
||||
|
||||
@ -226,6 +235,8 @@ public:
|
||||
|
||||
virtual WriteSettings patchSettings(const WriteSettings & write_settings) const;
|
||||
|
||||
virtual void setKeysGenerator(ObjectStorageKeysGeneratorPtr) { }
|
||||
|
||||
#if USE_AZURE_BLOB_STORAGE
|
||||
virtual std::shared_ptr<const Azure::Storage::Blobs::BlobContainerClient> getAzureBlobStorageClient()
|
||||
{
|
||||
|
||||
@ -1,23 +0,0 @@
|
||||
#include <base/defines.h>
|
||||
#include <Disks/ObjectStorages/MetadataFromDiskTransactionState.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
std::string toString(MetadataFromDiskTransactionState state)
|
||||
{
|
||||
switch (state)
|
||||
{
|
||||
case MetadataFromDiskTransactionState::PREPARING:
|
||||
return "PREPARING";
|
||||
case MetadataFromDiskTransactionState::FAILED:
|
||||
return "FAILED";
|
||||
case MetadataFromDiskTransactionState::COMMITTED:
|
||||
return "COMMITTED";
|
||||
case MetadataFromDiskTransactionState::PARTIALLY_ROLLED_BACK:
|
||||
return "PARTIALLY_ROLLED_BACK";
|
||||
}
|
||||
UNREACHABLE();
|
||||
}
|
||||
|
||||
}
|
||||
93
src/Disks/ObjectStorages/MetadataOperationsHolder.cpp
Normal file
93
src/Disks/ObjectStorages/MetadataOperationsHolder.cpp
Normal file
@ -0,0 +1,93 @@
|
||||
#include "MetadataOperationsHolder.h"
|
||||
|
||||
#include <Common/Exception.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int FS_METADATA_ERROR;
|
||||
}
|
||||
|
||||
void MetadataOperationsHolder::rollback(std::unique_lock<SharedMutex> & lock, size_t until_pos)
|
||||
{
|
||||
/// Otherwise everything is alright
|
||||
if (state == MetadataStorageTransactionState::FAILED)
|
||||
{
|
||||
for (int64_t i = until_pos; i >= 0; --i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->undo(lock);
|
||||
}
|
||||
catch (Exception & ex)
|
||||
{
|
||||
state = MetadataStorageTransactionState::PARTIALLY_ROLLED_BACK;
|
||||
ex.addMessage(fmt::format("While rolling back operation #{}", i));
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Nothing to do, transaction committed or not even started to commit
|
||||
}
|
||||
}
|
||||
|
||||
void MetadataOperationsHolder::addOperation(MetadataOperationPtr && operation)
|
||||
{
|
||||
if (state != MetadataStorageTransactionState::PREPARING)
|
||||
throw Exception(
|
||||
ErrorCodes::FS_METADATA_ERROR,
|
||||
"Cannot add operations to transaction in {} state, it should be in {} state",
|
||||
toString(state),
|
||||
toString(MetadataStorageTransactionState::PREPARING));
|
||||
|
||||
operations.emplace_back(std::move(operation));
|
||||
}
|
||||
|
||||
void MetadataOperationsHolder::commitImpl(SharedMutex & metadata_mutex)
|
||||
{
|
||||
if (state != MetadataStorageTransactionState::PREPARING)
|
||||
throw Exception(
|
||||
ErrorCodes::FS_METADATA_ERROR,
|
||||
"Cannot commit transaction in {} state, it should be in {} state",
|
||||
toString(state),
|
||||
toString(MetadataStorageTransactionState::PREPARING));
|
||||
|
||||
{
|
||||
std::unique_lock lock(metadata_mutex);
|
||||
for (size_t i = 0; i < operations.size(); ++i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->execute(lock);
|
||||
}
|
||||
catch (Exception & ex)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
ex.addMessage(fmt::format("While committing metadata operation #{}", i));
|
||||
state = MetadataStorageTransactionState::FAILED;
|
||||
rollback(lock, i);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Do it in "best effort" mode
|
||||
for (size_t i = 0; i < operations.size(); ++i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->finalize();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__, fmt::format("Failed to finalize operation #{}", i));
|
||||
}
|
||||
}
|
||||
|
||||
state = MetadataStorageTransactionState::COMMITTED;
|
||||
}
|
||||
}
|
||||
29
src/Disks/ObjectStorages/MetadataOperationsHolder.h
Normal file
29
src/Disks/ObjectStorages/MetadataOperationsHolder.h
Normal file
@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
|
||||
#include <mutex>
|
||||
#include <Disks/ObjectStorages/IMetadataOperation.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageTransactionState.h>
|
||||
#include <Common/SharedMutex.h>
|
||||
|
||||
/**
|
||||
* Implementations for transactional operations with metadata used by MetadataStorageFromDisk
|
||||
* and MetadataStorageFromPlainObjectStorage.
|
||||
*/
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class MetadataOperationsHolder
|
||||
{
|
||||
private:
|
||||
std::vector<MetadataOperationPtr> operations;
|
||||
MetadataStorageTransactionState state{MetadataStorageTransactionState::PREPARING};
|
||||
|
||||
void rollback(std::unique_lock<SharedMutex> & lock, size_t until_pos);
|
||||
|
||||
protected:
|
||||
void addOperation(MetadataOperationPtr && operation);
|
||||
void commitImpl(SharedMutex & metadata_mutex);
|
||||
};
|
||||
|
||||
}
|
||||
@ -1,6 +1,7 @@
|
||||
#include <Disks/ObjectStorages/MetadataStorageFactory.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromDisk.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainObjectStorage.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainRewritableObjectStorage.h>
|
||||
#ifndef CLICKHOUSE_KEEPER_STANDALONE_BUILD
|
||||
#include <Disks/ObjectStorages/Web/MetadataStorageFromStaticFilesWebServer.h>
|
||||
#endif
|
||||
@ -118,6 +119,20 @@ void registerPlainMetadataStorage(MetadataStorageFactory & factory)
|
||||
});
|
||||
}
|
||||
|
||||
void registerPlainRewritableMetadataStorage(MetadataStorageFactory & factory)
|
||||
{
|
||||
factory.registerMetadataStorageType(
|
||||
"plain_rewritable",
|
||||
[](const std::string & /* name */,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
ObjectStoragePtr object_storage) -> MetadataStoragePtr
|
||||
{
|
||||
auto key_compatibility_prefix = getObjectKeyCompatiblePrefix(*object_storage, config, config_prefix);
|
||||
return std::make_shared<MetadataStorageFromPlainRewritableObjectStorage>(object_storage, key_compatibility_prefix);
|
||||
});
|
||||
}
|
||||
|
||||
#ifndef CLICKHOUSE_KEEPER_STANDALONE_BUILD
|
||||
void registerMetadataStorageFromStaticFilesWebServer(MetadataStorageFactory & factory)
|
||||
{
|
||||
@ -137,6 +152,7 @@ void registerMetadataStorages()
|
||||
auto & factory = MetadataStorageFactory::instance();
|
||||
registerMetadataStorageFromDisk(factory);
|
||||
registerPlainMetadataStorage(factory);
|
||||
registerPlainRewritableMetadataStorage(factory);
|
||||
#ifndef CLICKHOUSE_KEEPER_STANDALONE_BUILD
|
||||
registerMetadataStorageFromStaticFilesWebServer(factory);
|
||||
#endif
|
||||
|
||||
@ -10,14 +10,8 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int FS_METADATA_ERROR;
|
||||
}
|
||||
|
||||
MetadataStorageFromDisk::MetadataStorageFromDisk(DiskPtr disk_, String compatible_key_prefix_)
|
||||
: disk(disk_)
|
||||
, compatible_key_prefix(compatible_key_prefix_)
|
||||
: disk(disk_), compatible_key_prefix(compatible_key_prefix_)
|
||||
{
|
||||
}
|
||||
|
||||
@ -158,83 +152,9 @@ const IMetadataStorage & MetadataStorageFromDiskTransaction::getStorageForNonTra
|
||||
return metadata_storage;
|
||||
}
|
||||
|
||||
void MetadataStorageFromDiskTransaction::addOperation(MetadataOperationPtr && operation)
|
||||
{
|
||||
if (state != MetadataFromDiskTransactionState::PREPARING)
|
||||
throw Exception(
|
||||
ErrorCodes::FS_METADATA_ERROR,
|
||||
"Cannot add operations to transaction in {} state, it should be in {} state",
|
||||
toString(state), toString(MetadataFromDiskTransactionState::PREPARING));
|
||||
|
||||
operations.emplace_back(std::move(operation));
|
||||
}
|
||||
|
||||
void MetadataStorageFromDiskTransaction::commit()
|
||||
{
|
||||
if (state != MetadataFromDiskTransactionState::PREPARING)
|
||||
throw Exception(
|
||||
ErrorCodes::FS_METADATA_ERROR,
|
||||
"Cannot commit transaction in {} state, it should be in {} state",
|
||||
toString(state), toString(MetadataFromDiskTransactionState::PREPARING));
|
||||
|
||||
{
|
||||
std::unique_lock lock(metadata_storage.metadata_mutex);
|
||||
for (size_t i = 0; i < operations.size(); ++i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->execute(lock);
|
||||
}
|
||||
catch (Exception & ex)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
ex.addMessage(fmt::format("While committing metadata operation #{}", i));
|
||||
state = MetadataFromDiskTransactionState::FAILED;
|
||||
rollback(i);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Do it in "best effort" mode
|
||||
for (size_t i = 0; i < operations.size(); ++i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->finalize();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__, fmt::format("Failed to finalize operation #{}", i));
|
||||
}
|
||||
}
|
||||
|
||||
state = MetadataFromDiskTransactionState::COMMITTED;
|
||||
}
|
||||
|
||||
void MetadataStorageFromDiskTransaction::rollback(size_t until_pos)
|
||||
{
|
||||
/// Otherwise everything is alright
|
||||
if (state == MetadataFromDiskTransactionState::FAILED)
|
||||
{
|
||||
for (int64_t i = until_pos; i >= 0; --i)
|
||||
{
|
||||
try
|
||||
{
|
||||
operations[i]->undo();
|
||||
}
|
||||
catch (Exception & ex)
|
||||
{
|
||||
state = MetadataFromDiskTransactionState::PARTIALLY_ROLLED_BACK;
|
||||
ex.addMessage(fmt::format("While rolling back operation #{}", i));
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Nothing to do, transaction committed or not even started to commit
|
||||
}
|
||||
MetadataOperationsHolder::commitImpl(metadata_storage.metadata_mutex);
|
||||
}
|
||||
|
||||
void MetadataStorageFromDiskTransaction::writeStringToFile(
|
||||
|
||||
@ -5,8 +5,9 @@
|
||||
|
||||
#include <Disks/IDisk.h>
|
||||
#include <Disks/ObjectStorages/DiskObjectStorageMetadata.h>
|
||||
#include <Disks/ObjectStorages/MetadataFromDiskTransactionState.h>
|
||||
#include <Disks/ObjectStorages/MetadataOperationsHolder.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageTransactionState.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -74,18 +75,11 @@ public:
|
||||
DiskObjectStorageMetadataPtr readMetadataUnlocked(const std::string & path, std::shared_lock<SharedMutex> & lock) const;
|
||||
};
|
||||
|
||||
class MetadataStorageFromDiskTransaction final : public IMetadataTransaction
|
||||
class MetadataStorageFromDiskTransaction final : public IMetadataTransaction, private MetadataOperationsHolder
|
||||
{
|
||||
private:
|
||||
const MetadataStorageFromDisk & metadata_storage;
|
||||
|
||||
std::vector<MetadataOperationPtr> operations;
|
||||
MetadataFromDiskTransactionState state{MetadataFromDiskTransactionState::PREPARING};
|
||||
|
||||
void addOperation(MetadataOperationPtr && operation);
|
||||
|
||||
void rollback(size_t until_pos);
|
||||
|
||||
public:
|
||||
explicit MetadataStorageFromDiskTransaction(const MetadataStorageFromDisk & metadata_storage_)
|
||||
: metadata_storage(metadata_storage_)
|
||||
@ -135,7 +129,6 @@ public:
|
||||
|
||||
UnlinkMetadataFileOperationOutcomePtr unlinkMetadata(const std::string & path) override;
|
||||
|
||||
|
||||
};
|
||||
|
||||
|
||||
|
||||
@ -32,7 +32,7 @@ void SetLastModifiedOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.setLastModified(path, new_timestamp);
|
||||
}
|
||||
|
||||
void SetLastModifiedOperation::undo()
|
||||
void SetLastModifiedOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.setLastModified(path, old_timestamp);
|
||||
}
|
||||
@ -50,7 +50,7 @@ void ChmodOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.chmod(path, mode);
|
||||
}
|
||||
|
||||
void ChmodOperation::undo()
|
||||
void ChmodOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.chmod(path, old_mode);
|
||||
}
|
||||
@ -68,7 +68,7 @@ void UnlinkFileOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.removeFile(path);
|
||||
}
|
||||
|
||||
void UnlinkFileOperation::undo()
|
||||
void UnlinkFileOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
auto buf = disk.writeFile(path);
|
||||
writeString(prev_data, *buf);
|
||||
@ -86,7 +86,7 @@ void CreateDirectoryOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.createDirectory(path);
|
||||
}
|
||||
|
||||
void CreateDirectoryOperation::undo()
|
||||
void CreateDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.removeDirectory(path);
|
||||
}
|
||||
@ -112,7 +112,7 @@ void CreateDirectoryRecursiveOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.createDirectory(path_to_create);
|
||||
}
|
||||
|
||||
void CreateDirectoryRecursiveOperation::undo()
|
||||
void CreateDirectoryRecursiveOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
for (const auto & path_created : paths_created)
|
||||
disk.removeDirectory(path_created);
|
||||
@ -129,7 +129,7 @@ void RemoveDirectoryOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.removeDirectory(path);
|
||||
}
|
||||
|
||||
void RemoveDirectoryOperation::undo()
|
||||
void RemoveDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.createDirectory(path);
|
||||
}
|
||||
@ -149,7 +149,7 @@ void RemoveRecursiveOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.moveDirectory(path, temp_path);
|
||||
}
|
||||
|
||||
void RemoveRecursiveOperation::undo()
|
||||
void RemoveRecursiveOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
if (disk.isFile(temp_path))
|
||||
disk.moveFile(temp_path, path);
|
||||
@ -187,10 +187,10 @@ void CreateHardlinkOperation::execute(std::unique_lock<SharedMutex> & lock)
|
||||
disk.createHardLink(path_from, path_to);
|
||||
}
|
||||
|
||||
void CreateHardlinkOperation::undo()
|
||||
void CreateHardlinkOperation::undo(std::unique_lock<SharedMutex> & lock)
|
||||
{
|
||||
if (write_operation)
|
||||
write_operation->undo();
|
||||
write_operation->undo(lock);
|
||||
disk.removeFile(path_to);
|
||||
}
|
||||
|
||||
@ -206,7 +206,7 @@ void MoveFileOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.moveFile(path_from, path_to);
|
||||
}
|
||||
|
||||
void MoveFileOperation::undo()
|
||||
void MoveFileOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.moveFile(path_to, path_from);
|
||||
}
|
||||
@ -223,7 +223,7 @@ void MoveDirectoryOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.moveDirectory(path_from, path_to);
|
||||
}
|
||||
|
||||
void MoveDirectoryOperation::undo()
|
||||
void MoveDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.moveDirectory(path_to, path_from);
|
||||
}
|
||||
@ -244,7 +244,7 @@ void ReplaceFileOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
disk.replaceFile(path_from, path_to);
|
||||
}
|
||||
|
||||
void ReplaceFileOperation::undo()
|
||||
void ReplaceFileOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
disk.moveFile(path_to, path_from);
|
||||
disk.moveFile(temp_path_to, path_to);
|
||||
@ -275,7 +275,7 @@ void WriteFileOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
buf->finalize();
|
||||
}
|
||||
|
||||
void WriteFileOperation::undo()
|
||||
void WriteFileOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
if (!existed)
|
||||
{
|
||||
@ -303,10 +303,10 @@ void AddBlobOperation::execute(std::unique_lock<SharedMutex> & metadata_lock)
|
||||
write_operation->execute(metadata_lock);
|
||||
}
|
||||
|
||||
void AddBlobOperation::undo()
|
||||
void AddBlobOperation::undo(std::unique_lock<SharedMutex> & lock)
|
||||
{
|
||||
if (write_operation)
|
||||
write_operation->undo();
|
||||
write_operation->undo(lock);
|
||||
}
|
||||
|
||||
void UnlinkMetadataFileOperation::execute(std::unique_lock<SharedMutex> & metadata_lock)
|
||||
@ -325,17 +325,17 @@ void UnlinkMetadataFileOperation::execute(std::unique_lock<SharedMutex> & metada
|
||||
unlink_operation->execute(metadata_lock);
|
||||
}
|
||||
|
||||
void UnlinkMetadataFileOperation::undo()
|
||||
void UnlinkMetadataFileOperation::undo(std::unique_lock<SharedMutex> & lock)
|
||||
{
|
||||
/// Operations MUST be reverted in the reversed order, so
|
||||
/// when we apply operation #1 (write) and operation #2 (unlink)
|
||||
/// we should revert #2 and only after it #1. Otherwise #1 will overwrite
|
||||
/// file with incorrect data.
|
||||
if (unlink_operation)
|
||||
unlink_operation->undo();
|
||||
unlink_operation->undo(lock);
|
||||
|
||||
if (write_operation)
|
||||
write_operation->undo();
|
||||
write_operation->undo(lock);
|
||||
|
||||
/// Update outcome to reflect the fact that we have restored the file.
|
||||
outcome->num_hardlinks++;
|
||||
@ -349,10 +349,10 @@ void SetReadonlyFileOperation::execute(std::unique_lock<SharedMutex> & metadata_
|
||||
write_operation->execute(metadata_lock);
|
||||
}
|
||||
|
||||
void SetReadonlyFileOperation::undo()
|
||||
void SetReadonlyFileOperation::undo(std::unique_lock<SharedMutex> & lock)
|
||||
{
|
||||
if (write_operation)
|
||||
write_operation->undo();
|
||||
write_operation->undo(lock);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
#pragma once
|
||||
|
||||
#include <Common/SharedMutex.h>
|
||||
#include <Disks/ObjectStorages/IMetadataOperation.h>
|
||||
#include <Disks/ObjectStorages/IMetadataStorage.h>
|
||||
|
||||
#include <numeric>
|
||||
@ -14,24 +14,13 @@ class IDisk;
|
||||
* Implementations for transactional operations with metadata used by MetadataStorageFromDisk.
|
||||
*/
|
||||
|
||||
struct IMetadataOperation
|
||||
{
|
||||
virtual void execute(std::unique_lock<SharedMutex> & metadata_lock) = 0;
|
||||
virtual void undo() = 0;
|
||||
virtual void finalize() {}
|
||||
virtual ~IMetadataOperation() = default;
|
||||
};
|
||||
|
||||
using MetadataOperationPtr = std::unique_ptr<IMetadataOperation>;
|
||||
|
||||
|
||||
struct SetLastModifiedOperation final : public IMetadataOperation
|
||||
{
|
||||
SetLastModifiedOperation(const std::string & path_, Poco::Timestamp new_timestamp_, IDisk & disk_);
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -46,7 +35,7 @@ struct ChmodOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -62,7 +51,7 @@ struct UnlinkFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -77,7 +66,7 @@ struct CreateDirectoryOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -91,7 +80,7 @@ struct CreateDirectoryRecursiveOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -106,7 +95,7 @@ struct RemoveDirectoryOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -119,7 +108,7 @@ struct RemoveRecursiveOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void finalize() override;
|
||||
|
||||
@ -135,7 +124,8 @@ struct WriteFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
IDisk & disk;
|
||||
@ -154,7 +144,7 @@ struct CreateHardlinkOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path_from;
|
||||
@ -171,7 +161,7 @@ struct MoveFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path_from;
|
||||
@ -186,7 +176,7 @@ struct MoveDirectoryOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path_from;
|
||||
@ -201,7 +191,7 @@ struct ReplaceFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void finalize() override;
|
||||
|
||||
@ -229,7 +219,7 @@ struct AddBlobOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -257,7 +247,7 @@ struct UnlinkMetadataFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
@ -282,7 +272,7 @@ struct SetReadonlyFileOperation final : public IMetadataOperation
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo() override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
private:
|
||||
std::string path;
|
||||
|
||||
@ -1,18 +1,27 @@
|
||||
#include "MetadataStorageFromPlainObjectStorage.h"
|
||||
#include <Disks/IDisk.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainObjectStorageOperations.h>
|
||||
#include <Disks/ObjectStorages/StaticDirectoryIterator.h>
|
||||
#include <Common/filesystemHelpers.h>
|
||||
#include <Common/logger_useful.h>
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
|
||||
#include <Common/filesystemHelpers.h>
|
||||
|
||||
#include <filesystem>
|
||||
#include <tuple>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
MetadataStorageFromPlainObjectStorage::MetadataStorageFromPlainObjectStorage(
|
||||
ObjectStoragePtr object_storage_,
|
||||
String storage_path_prefix_)
|
||||
namespace
|
||||
{
|
||||
|
||||
std::filesystem::path normalizeDirectoryPath(const std::filesystem::path & path)
|
||||
{
|
||||
return path / "";
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
MetadataStorageFromPlainObjectStorage::MetadataStorageFromPlainObjectStorage(ObjectStoragePtr object_storage_, String storage_path_prefix_)
|
||||
: object_storage(object_storage_)
|
||||
, storage_path_prefix(std::move(storage_path_prefix_))
|
||||
{
|
||||
@ -20,7 +29,7 @@ MetadataStorageFromPlainObjectStorage::MetadataStorageFromPlainObjectStorage(
|
||||
|
||||
MetadataTransactionPtr MetadataStorageFromPlainObjectStorage::createTransaction()
|
||||
{
|
||||
return std::make_shared<MetadataStorageFromPlainObjectStorageTransaction>(*this);
|
||||
return std::make_shared<MetadataStorageFromPlainObjectStorageTransaction>(*this, object_storage);
|
||||
}
|
||||
|
||||
const std::string & MetadataStorageFromPlainObjectStorage::getPath() const
|
||||
@ -44,10 +53,9 @@ bool MetadataStorageFromPlainObjectStorage::isFile(const std::string & path) con
|
||||
|
||||
bool MetadataStorageFromPlainObjectStorage::isDirectory(const std::string & path) const
|
||||
{
|
||||
auto object_key = object_storage->generateObjectKeyForPath(path);
|
||||
std::string directory = object_key.serialize();
|
||||
if (!directory.ends_with('/'))
|
||||
directory += '/';
|
||||
auto key_prefix = object_storage->generateObjectKeyForPath(path).serialize();
|
||||
auto directory = std::filesystem::path(std::move(key_prefix)) / "";
|
||||
|
||||
return object_storage->existsOrHasAnyChild(directory);
|
||||
}
|
||||
|
||||
@ -62,33 +70,16 @@ uint64_t MetadataStorageFromPlainObjectStorage::getFileSize(const String & path)
|
||||
|
||||
std::vector<std::string> MetadataStorageFromPlainObjectStorage::listDirectory(const std::string & path) const
|
||||
{
|
||||
auto object_key = object_storage->generateObjectKeyForPath(path);
|
||||
auto key_prefix = object_storage->generateObjectKeyForPath(path).serialize();
|
||||
|
||||
RelativePathsWithMetadata files;
|
||||
std::string abs_key = object_key.serialize();
|
||||
std::string abs_key = key_prefix;
|
||||
if (!abs_key.ends_with('/'))
|
||||
abs_key += '/';
|
||||
|
||||
object_storage->listObjects(abs_key, files, 0);
|
||||
|
||||
std::vector<std::string> result;
|
||||
for (const auto & path_size : files)
|
||||
{
|
||||
result.push_back(path_size.relative_path);
|
||||
}
|
||||
|
||||
std::unordered_set<std::string> duplicates_filter;
|
||||
for (auto & row : result)
|
||||
{
|
||||
chassert(row.starts_with(abs_key));
|
||||
row.erase(0, abs_key.size());
|
||||
auto slash_pos = row.find_first_of('/');
|
||||
if (slash_pos != std::string::npos)
|
||||
row.erase(slash_pos, row.size() - slash_pos);
|
||||
duplicates_filter.insert(row);
|
||||
}
|
||||
|
||||
return std::vector<std::string>(duplicates_filter.begin(), duplicates_filter.end());
|
||||
return getDirectChildrenOnDisk(abs_key, files, path);
|
||||
}
|
||||
|
||||
DirectoryIteratorPtr MetadataStorageFromPlainObjectStorage::iterateDirectory(const std::string & path) const
|
||||
@ -108,6 +99,25 @@ StoredObjects MetadataStorageFromPlainObjectStorage::getStorageObjects(const std
|
||||
return {StoredObject(object_key.serialize(), path, object_size)};
|
||||
}
|
||||
|
||||
std::vector<std::string> MetadataStorageFromPlainObjectStorage::getDirectChildrenOnDisk(
|
||||
const std::string & storage_key, const RelativePathsWithMetadata & remote_paths, const std::string & /* local_path */) const
|
||||
{
|
||||
std::unordered_set<std::string> duplicates_filter;
|
||||
for (const auto & elem : remote_paths)
|
||||
{
|
||||
const auto & path = elem.relative_path;
|
||||
chassert(path.find(storage_key) == 0);
|
||||
const auto child_pos = storage_key.size();
|
||||
/// string::npos is ok.
|
||||
const auto slash_pos = path.find('/', child_pos);
|
||||
if (slash_pos == std::string::npos)
|
||||
duplicates_filter.emplace(path.substr(child_pos));
|
||||
else
|
||||
duplicates_filter.emplace(path.substr(child_pos, slash_pos - child_pos));
|
||||
}
|
||||
return std::vector<std::string>(std::make_move_iterator(duplicates_filter.begin()), std::make_move_iterator(duplicates_filter.end()));
|
||||
}
|
||||
|
||||
const IMetadataStorage & MetadataStorageFromPlainObjectStorageTransaction::getStorageForNonTransactionalReads() const
|
||||
{
|
||||
return metadata_storage;
|
||||
@ -122,18 +132,44 @@ void MetadataStorageFromPlainObjectStorageTransaction::unlinkFile(const std::str
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::removeDirectory(const std::string & path)
|
||||
{
|
||||
for (auto it = metadata_storage.iterateDirectory(path); it->isValid(); it->next())
|
||||
metadata_storage.object_storage->removeObject(StoredObject(it->path()));
|
||||
if (metadata_storage.object_storage->isWriteOnce())
|
||||
{
|
||||
for (auto it = metadata_storage.iterateDirectory(path); it->isValid(); it->next())
|
||||
metadata_storage.object_storage->removeObject(StoredObject(it->path()));
|
||||
}
|
||||
else
|
||||
{
|
||||
addOperation(std::make_unique<MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation>(
|
||||
normalizeDirectoryPath(path), *metadata_storage.getPathMap(), object_storage));
|
||||
}
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::createDirectory(const std::string &)
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::createDirectory(const std::string & path)
|
||||
{
|
||||
/// Noop. It is an Object Storage not a filesystem.
|
||||
if (metadata_storage.object_storage->isWriteOnce())
|
||||
return;
|
||||
|
||||
auto normalized_path = normalizeDirectoryPath(path);
|
||||
auto key_prefix = object_storage->generateObjectKeyPrefixForDirectoryPath(normalized_path).serialize();
|
||||
auto op = std::make_unique<MetadataStorageFromPlainObjectStorageCreateDirectoryOperation>(
|
||||
std::move(normalized_path), std::move(key_prefix), *metadata_storage.getPathMap(), object_storage);
|
||||
addOperation(std::move(op));
|
||||
}
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::createDirectoryRecursive(const std::string &)
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::createDirectoryRecursive(const std::string & path)
|
||||
{
|
||||
/// Noop. It is an Object Storage not a filesystem.
|
||||
return createDirectory(path);
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::moveDirectory(const std::string & path_from, const std::string & path_to)
|
||||
{
|
||||
if (metadata_storage.object_storage->isWriteOnce())
|
||||
throwNotImplemented();
|
||||
|
||||
addOperation(std::make_unique<MetadataStorageFromPlainObjectStorageMoveDirectoryOperation>(
|
||||
normalizeDirectoryPath(path_from), normalizeDirectoryPath(path_to), *metadata_storage.getPathMap(), object_storage));
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::addBlobToMetadata(
|
||||
const std::string &, ObjectStorageKey /* object_key */, uint64_t /* size_in_bytes */)
|
||||
{
|
||||
@ -146,4 +182,8 @@ UnlinkMetadataFileOperationOutcomePtr MetadataStorageFromPlainObjectStorageTrans
|
||||
return std::make_shared<UnlinkMetadataFileOperationOutcome>(UnlinkMetadataFileOperationOutcome{0});
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageTransaction::commit()
|
||||
{
|
||||
MetadataOperationsHolder::commitImpl(metadata_storage.metadata_mutex);
|
||||
}
|
||||
}
|
||||
|
||||
@ -2,9 +2,10 @@
|
||||
|
||||
#include <Disks/IDisk.h>
|
||||
#include <Disks/ObjectStorages/IMetadataStorage.h>
|
||||
#include <Disks/ObjectStorages/MetadataFromDiskTransactionState.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromDiskTransactionOperations.h>
|
||||
#include <Disks/ObjectStorages/MetadataOperationsHolder.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageTransactionState.h>
|
||||
|
||||
#include <map>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
@ -23,14 +24,21 @@ using UnlinkMetadataFileOperationOutcomePtr = std::shared_ptr<UnlinkMetadataFile
|
||||
/// It is used to allow BACKUP/RESTORE to ObjectStorage (S3/...) with the same
|
||||
/// structure as on disk MergeTree, and does not requires metadata from local
|
||||
/// disk to restore.
|
||||
class MetadataStorageFromPlainObjectStorage final : public IMetadataStorage
|
||||
class MetadataStorageFromPlainObjectStorage : public IMetadataStorage
|
||||
{
|
||||
public:
|
||||
/// Local path prefixes mapped to storage key prefixes.
|
||||
using PathMap = std::map<std::filesystem::path, std::string>;
|
||||
|
||||
private:
|
||||
friend class MetadataStorageFromPlainObjectStorageTransaction;
|
||||
|
||||
protected:
|
||||
ObjectStoragePtr object_storage;
|
||||
String storage_path_prefix;
|
||||
|
||||
mutable SharedMutex metadata_mutex;
|
||||
|
||||
public:
|
||||
MetadataStorageFromPlainObjectStorage(ObjectStoragePtr object_storage_, String storage_path_prefix_);
|
||||
|
||||
@ -69,23 +77,37 @@ public:
|
||||
|
||||
bool supportsChmod() const override { return false; }
|
||||
bool supportsStat() const override { return false; }
|
||||
|
||||
protected:
|
||||
virtual std::shared_ptr<PathMap> getPathMap() const { throwNotImplemented(); }
|
||||
|
||||
virtual std::vector<std::string> getDirectChildrenOnDisk(
|
||||
const std::string & storage_key, const RelativePathsWithMetadata & remote_paths, const std::string & local_path) const;
|
||||
};
|
||||
|
||||
class MetadataStorageFromPlainObjectStorageTransaction final : public IMetadataTransaction
|
||||
class MetadataStorageFromPlainObjectStorageTransaction final : public IMetadataTransaction, private MetadataOperationsHolder
|
||||
{
|
||||
private:
|
||||
const MetadataStorageFromPlainObjectStorage & metadata_storage;
|
||||
MetadataStorageFromPlainObjectStorage & metadata_storage;
|
||||
ObjectStoragePtr object_storage;
|
||||
|
||||
std::vector<MetadataOperationPtr> operations;
|
||||
|
||||
public:
|
||||
explicit MetadataStorageFromPlainObjectStorageTransaction(const MetadataStorageFromPlainObjectStorage & metadata_storage_)
|
||||
: metadata_storage(metadata_storage_)
|
||||
explicit MetadataStorageFromPlainObjectStorageTransaction(
|
||||
MetadataStorageFromPlainObjectStorage & metadata_storage_, ObjectStoragePtr object_storage_)
|
||||
: metadata_storage(metadata_storage_), object_storage(object_storage_)
|
||||
{}
|
||||
|
||||
const IMetadataStorage & getStorageForNonTransactionalReads() const override;
|
||||
|
||||
void addBlobToMetadata(const std::string & path, ObjectStorageKey object_key, uint64_t size_in_bytes) override;
|
||||
|
||||
void setLastModified(const String &, const Poco::Timestamp &) override
|
||||
{
|
||||
/// Noop
|
||||
}
|
||||
|
||||
void createEmptyMetadataFile(const std::string & /* path */) override
|
||||
{
|
||||
/// No metadata, no need to create anything.
|
||||
@ -100,17 +122,15 @@ public:
|
||||
|
||||
void createDirectoryRecursive(const std::string & path) override;
|
||||
|
||||
void moveDirectory(const std::string & path_from, const std::string & path_to) override;
|
||||
|
||||
void unlinkFile(const std::string & path) override;
|
||||
void removeDirectory(const std::string & path) override;
|
||||
|
||||
UnlinkMetadataFileOperationOutcomePtr unlinkMetadata(const std::string & path) override;
|
||||
|
||||
void commit() override
|
||||
{
|
||||
/// TODO: rewrite with transactions
|
||||
}
|
||||
void commit() override;
|
||||
|
||||
bool supportsChmod() const override { return false; }
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -0,0 +1,190 @@
|
||||
#include "MetadataStorageFromPlainObjectStorageOperations.h"
|
||||
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/logger_useful.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int FILE_DOESNT_EXIST;
|
||||
extern const int FILE_ALREADY_EXISTS;
|
||||
extern const int INCORRECT_DATA;
|
||||
};
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
constexpr auto PREFIX_PATH_FILE_NAME = "prefix.path";
|
||||
|
||||
}
|
||||
|
||||
MetadataStorageFromPlainObjectStorageCreateDirectoryOperation::MetadataStorageFromPlainObjectStorageCreateDirectoryOperation(
|
||||
std::filesystem::path && path_,
|
||||
std::string && key_prefix_,
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map_,
|
||||
ObjectStoragePtr object_storage_)
|
||||
: path(std::move(path_)), key_prefix(key_prefix_), path_map(path_map_), object_storage(object_storage_)
|
||||
{
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageCreateDirectoryOperation::execute(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
if (path_map.contains(path))
|
||||
return;
|
||||
|
||||
LOG_TRACE(getLogger("MetadataStorageFromPlainObjectStorageCreateDirectoryOperation"), "Creating metadata for directory '{}'", path);
|
||||
|
||||
auto object_key = ObjectStorageKey::createAsRelative(key_prefix, PREFIX_PATH_FILE_NAME);
|
||||
|
||||
auto object = StoredObject(object_key.serialize(), path / PREFIX_PATH_FILE_NAME);
|
||||
auto buf = object_storage->writeObject(
|
||||
object,
|
||||
WriteMode::Rewrite,
|
||||
/* object_attributes */ std::nullopt,
|
||||
/* buf_size */ DBMS_DEFAULT_BUFFER_SIZE,
|
||||
/* settings */ {});
|
||||
|
||||
write_created = true;
|
||||
|
||||
[[maybe_unused]] auto result = path_map.emplace(path, std::move(key_prefix));
|
||||
chassert(result.second);
|
||||
|
||||
writeString(path.string(), *buf);
|
||||
buf->finalize();
|
||||
|
||||
write_finalized = true;
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageCreateDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
auto object_key = ObjectStorageKey::createAsRelative(key_prefix, PREFIX_PATH_FILE_NAME);
|
||||
if (write_finalized)
|
||||
{
|
||||
path_map.erase(path);
|
||||
object_storage->removeObject(StoredObject(object_key.serialize(), path / PREFIX_PATH_FILE_NAME));
|
||||
}
|
||||
else if (write_created)
|
||||
object_storage->removeObjectIfExists(StoredObject(object_key.serialize(), path / PREFIX_PATH_FILE_NAME));
|
||||
}
|
||||
|
||||
MetadataStorageFromPlainObjectStorageMoveDirectoryOperation::MetadataStorageFromPlainObjectStorageMoveDirectoryOperation(
|
||||
std::filesystem::path && path_from_,
|
||||
std::filesystem::path && path_to_,
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map_,
|
||||
ObjectStoragePtr object_storage_)
|
||||
: path_from(std::move(path_from_)), path_to(std::move(path_to_)), path_map(path_map_), object_storage(object_storage_)
|
||||
{
|
||||
}
|
||||
|
||||
std::unique_ptr<WriteBufferFromFileBase> MetadataStorageFromPlainObjectStorageMoveDirectoryOperation::createWriteBuf(
|
||||
const std::filesystem::path & expected_path, const std::filesystem::path & new_path, bool validate_content)
|
||||
{
|
||||
auto expected_it = path_map.find(expected_path);
|
||||
if (expected_it == path_map.end())
|
||||
throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "Metadata object for the expected (source) path '{}' does not exist", expected_path);
|
||||
|
||||
if (path_map.contains(new_path))
|
||||
throw Exception(ErrorCodes::FILE_ALREADY_EXISTS, "Metadata object for the new (destination) path '{}' already exists", new_path);
|
||||
|
||||
auto object_key = ObjectStorageKey::createAsRelative(expected_it->second, PREFIX_PATH_FILE_NAME);
|
||||
|
||||
auto object = StoredObject(object_key.serialize(), expected_path / PREFIX_PATH_FILE_NAME);
|
||||
|
||||
if (validate_content)
|
||||
{
|
||||
std::string data;
|
||||
auto read_buf = object_storage->readObject(object);
|
||||
readStringUntilEOF(data, *read_buf);
|
||||
if (data != path_from)
|
||||
throw Exception(
|
||||
ErrorCodes::INCORRECT_DATA,
|
||||
"Incorrect data for object key {}, expected {}, got {}",
|
||||
object_key.serialize(),
|
||||
expected_path,
|
||||
data);
|
||||
}
|
||||
|
||||
auto write_buf = object_storage->writeObject(
|
||||
object,
|
||||
WriteMode::Rewrite,
|
||||
/* object_attributes */ std::nullopt,
|
||||
/*buf_size*/ DBMS_DEFAULT_BUFFER_SIZE,
|
||||
/*settings*/ {});
|
||||
|
||||
return write_buf;
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageMoveDirectoryOperation::execute(std::unique_lock<SharedMutex> & /* metadata_lock */)
|
||||
{
|
||||
LOG_TRACE(
|
||||
getLogger("MetadataStorageFromPlainObjectStorageMoveDirectoryOperation"), "Moving directory '{}' to '{}'", path_from, path_to);
|
||||
|
||||
auto write_buf = createWriteBuf(path_from, path_to, /* validate_content */ true);
|
||||
write_created = true;
|
||||
writeString(path_to.string(), *write_buf);
|
||||
write_buf->finalize();
|
||||
|
||||
[[maybe_unused]] auto result = path_map.emplace(path_to, path_map.extract(path_from).mapped());
|
||||
chassert(result.second);
|
||||
|
||||
write_finalized = true;
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageMoveDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
if (write_finalized)
|
||||
path_map.emplace(path_from, path_map.extract(path_to).mapped());
|
||||
|
||||
if (write_created)
|
||||
{
|
||||
auto write_buf = createWriteBuf(path_to, path_from, /* verify_content */ false);
|
||||
writeString(path_from.string(), *write_buf);
|
||||
write_buf->finalize();
|
||||
}
|
||||
}
|
||||
|
||||
MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation::MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation(
|
||||
std::filesystem::path && path_, MetadataStorageFromPlainObjectStorage::PathMap & path_map_, ObjectStoragePtr object_storage_)
|
||||
: path(std::move(path_)), path_map(path_map_), object_storage(object_storage_)
|
||||
{
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation::execute(std::unique_lock<SharedMutex> & /* metadata_lock */)
|
||||
{
|
||||
auto path_it = path_map.find(path);
|
||||
if (path_it == path_map.end())
|
||||
return;
|
||||
|
||||
LOG_TRACE(getLogger("MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation"), "Removing directory '{}'", path);
|
||||
|
||||
key_prefix = path_it->second;
|
||||
auto object_key = ObjectStorageKey::createAsRelative(key_prefix, PREFIX_PATH_FILE_NAME);
|
||||
auto object = StoredObject(object_key.serialize(), path / PREFIX_PATH_FILE_NAME);
|
||||
object_storage->removeObject(object);
|
||||
path_map.erase(path_it);
|
||||
}
|
||||
|
||||
void MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation::undo(std::unique_lock<SharedMutex> &)
|
||||
{
|
||||
if (!removed)
|
||||
return;
|
||||
|
||||
auto object_key = ObjectStorageKey::createAsRelative(key_prefix, PREFIX_PATH_FILE_NAME);
|
||||
auto object = StoredObject(object_key.serialize(), path / PREFIX_PATH_FILE_NAME);
|
||||
auto buf = object_storage->writeObject(
|
||||
object,
|
||||
WriteMode::Rewrite,
|
||||
/* object_attributes */ std::nullopt,
|
||||
/* buf_size */ DBMS_DEFAULT_BUFFER_SIZE,
|
||||
/* settings */ {});
|
||||
writeString(path.string(), *buf);
|
||||
buf->finalize();
|
||||
|
||||
path_map.emplace(path, std::move(key_prefix));
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,80 @@
|
||||
#pragma once
|
||||
|
||||
#include <Disks/ObjectStorages/IMetadataOperation.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainObjectStorage.h>
|
||||
|
||||
#include <filesystem>
|
||||
#include <map>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class MetadataStorageFromPlainObjectStorageCreateDirectoryOperation final : public IMetadataOperation
|
||||
{
|
||||
private:
|
||||
std::filesystem::path path;
|
||||
std::string key_prefix;
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map;
|
||||
ObjectStoragePtr object_storage;
|
||||
|
||||
bool write_created = false;
|
||||
bool write_finalized = false;
|
||||
|
||||
public:
|
||||
// Assuming that paths are normalized.
|
||||
MetadataStorageFromPlainObjectStorageCreateDirectoryOperation(
|
||||
std::filesystem::path && path_,
|
||||
std::string && key_prefix_,
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map_,
|
||||
ObjectStoragePtr object_storage_);
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
};
|
||||
|
||||
class MetadataStorageFromPlainObjectStorageMoveDirectoryOperation final : public IMetadataOperation
|
||||
{
|
||||
private:
|
||||
std::filesystem::path path_from;
|
||||
std::filesystem::path path_to;
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map;
|
||||
ObjectStoragePtr object_storage;
|
||||
|
||||
bool write_created = false;
|
||||
bool write_finalized = false;
|
||||
|
||||
std::unique_ptr<WriteBufferFromFileBase>
|
||||
createWriteBuf(const std::filesystem::path & expected_path, const std::filesystem::path & new_path, bool validate_content);
|
||||
|
||||
public:
|
||||
MetadataStorageFromPlainObjectStorageMoveDirectoryOperation(
|
||||
std::filesystem::path && path_from_,
|
||||
std::filesystem::path && path_to_,
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map_,
|
||||
ObjectStoragePtr object_storage_);
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
};
|
||||
|
||||
class MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation final : public IMetadataOperation
|
||||
{
|
||||
private:
|
||||
std::filesystem::path path;
|
||||
|
||||
MetadataStorageFromPlainObjectStorage::PathMap & path_map;
|
||||
ObjectStoragePtr object_storage;
|
||||
|
||||
std::string key_prefix;
|
||||
bool removed = false;
|
||||
|
||||
public:
|
||||
MetadataStorageFromPlainObjectStorageRemoveDirectoryOperation(
|
||||
std::filesystem::path && path_, MetadataStorageFromPlainObjectStorage::PathMap & path_map_, ObjectStoragePtr object_storage_);
|
||||
|
||||
void execute(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
void undo(std::unique_lock<SharedMutex> & metadata_lock) override;
|
||||
};
|
||||
|
||||
}
|
||||
@ -0,0 +1,143 @@
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainRewritableObjectStorage.h>
|
||||
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <Common/ErrorCodes.h>
|
||||
#include <Common/logger_useful.h>
|
||||
#include "CommonPathPrefixKeyGenerator.h"
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
constexpr auto PREFIX_PATH_FILE_NAME = "prefix.path";
|
||||
|
||||
MetadataStorageFromPlainObjectStorage::PathMap loadPathPrefixMap(const std::string & root, ObjectStoragePtr object_storage)
|
||||
{
|
||||
MetadataStorageFromPlainObjectStorage::PathMap result;
|
||||
|
||||
RelativePathsWithMetadata files;
|
||||
object_storage->listObjects(root, files, 0);
|
||||
for (const auto & file : files)
|
||||
{
|
||||
auto remote_path = std::filesystem::path(file.relative_path);
|
||||
if (remote_path.filename() != PREFIX_PATH_FILE_NAME)
|
||||
continue;
|
||||
|
||||
StoredObject object{file.relative_path};
|
||||
|
||||
auto read_buf = object_storage->readObject(object);
|
||||
String local_path;
|
||||
readStringUntilEOF(local_path, *read_buf);
|
||||
|
||||
chassert(remote_path.has_parent_path());
|
||||
auto res = result.emplace(local_path, remote_path.parent_path());
|
||||
|
||||
/// This can happen if table replication is enabled, then the same local path is written
|
||||
/// in `prefix.path` of each replica.
|
||||
/// TODO: should replicated tables (e.g., RMT) be explicitly disallowed?
|
||||
if (!res.second)
|
||||
LOG_WARNING(
|
||||
getLogger("MetadataStorageFromPlainObjectStorage"),
|
||||
"The local path '{}' is already mapped to a remote path '{}', ignoring: '{}'",
|
||||
local_path,
|
||||
res.first->second,
|
||||
remote_path.parent_path().string());
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
std::vector<std::string> getDirectChildrenOnRewritableDisk(
|
||||
const std::string & storage_key,
|
||||
const RelativePathsWithMetadata & remote_paths,
|
||||
const std::string & local_path,
|
||||
const MetadataStorageFromPlainObjectStorage::PathMap & local_path_prefixes,
|
||||
SharedMutex & shared_mutex)
|
||||
{
|
||||
using PathMap = MetadataStorageFromPlainObjectStorage::PathMap;
|
||||
|
||||
std::unordered_set<std::string> duplicates_filter;
|
||||
|
||||
/// Map remote paths into local subdirectories.
|
||||
std::unordered_map<PathMap::mapped_type, PathMap::key_type> remote_to_local_subdir;
|
||||
|
||||
{
|
||||
std::shared_lock lock(shared_mutex);
|
||||
auto end_it = local_path_prefixes.end();
|
||||
for (auto it = local_path_prefixes.lower_bound(local_path); it != end_it; ++it)
|
||||
{
|
||||
const auto & [k, v] = std::make_tuple(it->first.string(), it->second);
|
||||
if (!k.starts_with(local_path))
|
||||
break;
|
||||
|
||||
auto slash_num = count(k.begin() + local_path.size(), k.end(), '/');
|
||||
if (slash_num != 1)
|
||||
continue;
|
||||
|
||||
chassert(k.back() == '/');
|
||||
remote_to_local_subdir.emplace(v, std::string(k.begin() + local_path.size(), k.end() - 1));
|
||||
}
|
||||
}
|
||||
|
||||
auto skip_list = std::set<std::string>{PREFIX_PATH_FILE_NAME};
|
||||
for (const auto & elem : remote_paths)
|
||||
{
|
||||
const auto & path = elem.relative_path;
|
||||
chassert(path.find(storage_key) == 0);
|
||||
const auto child_pos = storage_key.size();
|
||||
|
||||
auto slash_pos = path.find('/', child_pos);
|
||||
|
||||
if (slash_pos == std::string::npos)
|
||||
{
|
||||
/// File names.
|
||||
auto filename = path.substr(child_pos);
|
||||
if (!skip_list.contains(filename))
|
||||
duplicates_filter.emplace(std::move(filename));
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Subdirectories.
|
||||
auto it = remote_to_local_subdir.find(path.substr(0, slash_pos));
|
||||
/// Mapped subdirectories.
|
||||
if (it != remote_to_local_subdir.end())
|
||||
duplicates_filter.emplace(it->second);
|
||||
/// The remote subdirectory name is the same as the local subdirectory.
|
||||
else
|
||||
duplicates_filter.emplace(path.substr(child_pos, slash_pos - child_pos));
|
||||
}
|
||||
}
|
||||
|
||||
return std::vector<std::string>(std::make_move_iterator(duplicates_filter.begin()), std::make_move_iterator(duplicates_filter.end()));
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
MetadataStorageFromPlainRewritableObjectStorage::MetadataStorageFromPlainRewritableObjectStorage(
|
||||
ObjectStoragePtr object_storage_, String storage_path_prefix_)
|
||||
: MetadataStorageFromPlainObjectStorage(object_storage_, storage_path_prefix_)
|
||||
, path_map(std::make_shared<PathMap>(loadPathPrefixMap(object_storage->getCommonKeyPrefix(), object_storage)))
|
||||
{
|
||||
if (object_storage->isWriteOnce())
|
||||
throw Exception(
|
||||
ErrorCodes::LOGICAL_ERROR,
|
||||
"MetadataStorageFromPlainRewritableObjectStorage is not compatible with write-once storage '{}'",
|
||||
object_storage->getName());
|
||||
|
||||
auto keys_gen = std::make_shared<CommonPathPrefixKeyGenerator>(object_storage->getCommonKeyPrefix(), metadata_mutex, path_map);
|
||||
object_storage->setKeysGenerator(keys_gen);
|
||||
}
|
||||
|
||||
std::vector<std::string> MetadataStorageFromPlainRewritableObjectStorage::getDirectChildrenOnDisk(
|
||||
const std::string & storage_key, const RelativePathsWithMetadata & remote_paths, const std::string & local_path) const
|
||||
{
|
||||
return getDirectChildrenOnRewritableDisk(storage_key, remote_paths, local_path, *getPathMap(), metadata_mutex);
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,26 @@
|
||||
#pragma once
|
||||
|
||||
#include <Disks/ObjectStorages/MetadataStorageFromPlainObjectStorage.h>
|
||||
|
||||
#include <memory>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class MetadataStorageFromPlainRewritableObjectStorage final : public MetadataStorageFromPlainObjectStorage
|
||||
{
|
||||
private:
|
||||
std::shared_ptr<PathMap> path_map;
|
||||
|
||||
public:
|
||||
MetadataStorageFromPlainRewritableObjectStorage(ObjectStoragePtr object_storage_, String storage_path_prefix_);
|
||||
|
||||
MetadataStorageType getType() const override { return MetadataStorageType::PlainRewritable; }
|
||||
|
||||
protected:
|
||||
std::shared_ptr<PathMap> getPathMap() const override { return path_map; }
|
||||
std::vector<std::string> getDirectChildrenOnDisk(
|
||||
const std::string & storage_key, const RelativePathsWithMetadata & remote_paths, const std::string & local_path) const override;
|
||||
};
|
||||
|
||||
}
|
||||
23
src/Disks/ObjectStorages/MetadataStorageTransactionState.cpp
Normal file
23
src/Disks/ObjectStorages/MetadataStorageTransactionState.cpp
Normal file
@ -0,0 +1,23 @@
|
||||
#include <Disks/ObjectStorages/MetadataStorageTransactionState.h>
|
||||
#include <base/defines.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
std::string toString(MetadataStorageTransactionState state)
|
||||
{
|
||||
switch (state)
|
||||
{
|
||||
case MetadataStorageTransactionState::PREPARING:
|
||||
return "PREPARING";
|
||||
case MetadataStorageTransactionState::FAILED:
|
||||
return "FAILED";
|
||||
case MetadataStorageTransactionState::COMMITTED:
|
||||
return "COMMITTED";
|
||||
case MetadataStorageTransactionState::PARTIALLY_ROLLED_BACK:
|
||||
return "PARTIALLY_ROLLED_BACK";
|
||||
}
|
||||
UNREACHABLE();
|
||||
}
|
||||
|
||||
}
|
||||
@ -4,7 +4,7 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
enum class MetadataFromDiskTransactionState
|
||||
enum class MetadataStorageTransactionState
|
||||
{
|
||||
PREPARING,
|
||||
FAILED,
|
||||
@ -12,6 +12,5 @@ enum class MetadataFromDiskTransactionState
|
||||
PARTIALLY_ROLLED_BACK,
|
||||
};
|
||||
|
||||
std::string toString(MetadataFromDiskTransactionState state);
|
||||
|
||||
std::string toString(MetadataStorageTransactionState state);
|
||||
}
|
||||
@ -1,9 +1,11 @@
|
||||
#include "config.h"
|
||||
#include <utility>
|
||||
#include <Disks/ObjectStorages/ObjectStorageFactory.h>
|
||||
#include "Disks/DiskType.h"
|
||||
#include "config.h"
|
||||
#if USE_AWS_S3
|
||||
#include <Disks/ObjectStorages/S3/DiskS3Utils.h>
|
||||
#include <Disks/ObjectStorages/S3/S3ObjectStorage.h>
|
||||
#include <Disks/ObjectStorages/S3/diskSettings.h>
|
||||
#include <Disks/ObjectStorages/S3/DiskS3Utils.h>
|
||||
#endif
|
||||
#if USE_HDFS && !defined(CLICKHOUSE_KEEPER_STANDALONE_BUILD)
|
||||
#include <Disks/ObjectStorages/HDFS/HDFSObjectStorage.h>
|
||||
@ -20,6 +22,7 @@
|
||||
#endif
|
||||
#include <Disks/ObjectStorages/MetadataStorageFactory.h>
|
||||
#include <Disks/ObjectStorages/PlainObjectStorage.h>
|
||||
#include <Disks/ObjectStorages/PlainRewritableObjectStorage.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Common/Macros.h>
|
||||
|
||||
@ -35,36 +38,50 @@ namespace ErrorCodes
|
||||
extern const int UNKNOWN_ELEMENT_IN_CONFIG;
|
||||
extern const int BAD_ARGUMENTS;
|
||||
extern const int LOGICAL_ERROR;
|
||||
extern const int NOT_IMPLEMENTED;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
bool isPlainStorage(
|
||||
ObjectStorageType type,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix)
|
||||
{
|
||||
auto compatibility_hint = MetadataStorageFactory::getCompatibilityMetadataTypeHint(type);
|
||||
auto metadata_type = MetadataStorageFactory::getMetadataType(config, config_prefix, compatibility_hint);
|
||||
return metadataTypeFromString(metadata_type) == MetadataStorageType::Plain;
|
||||
}
|
||||
|
||||
template <typename BaseObjectStorage, class ...Args>
|
||||
ObjectStoragePtr createObjectStorage(
|
||||
ObjectStorageType type,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
Args && ...args)
|
||||
bool isCompatibleWithMetadataStorage(
|
||||
ObjectStorageType storage_type,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
MetadataStorageType target_metadata_type)
|
||||
{
|
||||
auto compatibility_hint = MetadataStorageFactory::getCompatibilityMetadataTypeHint(storage_type);
|
||||
auto metadata_type = MetadataStorageFactory::getMetadataType(config, config_prefix, compatibility_hint);
|
||||
return metadataTypeFromString(metadata_type) == target_metadata_type;
|
||||
}
|
||||
|
||||
bool isPlainStorage(ObjectStorageType type, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix)
|
||||
{
|
||||
return isCompatibleWithMetadataStorage(type, config, config_prefix, MetadataStorageType::Plain);
|
||||
}
|
||||
|
||||
bool isPlainRewritableStorage(ObjectStorageType type, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix)
|
||||
{
|
||||
return isCompatibleWithMetadataStorage(type, config, config_prefix, MetadataStorageType::PlainRewritable);
|
||||
}
|
||||
|
||||
template <typename BaseObjectStorage, class... Args>
|
||||
ObjectStoragePtr createObjectStorage(
|
||||
ObjectStorageType type, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, Args &&... args)
|
||||
{
|
||||
if (isPlainStorage(type, config, config_prefix))
|
||||
return std::make_shared<PlainObjectStorage<BaseObjectStorage>>(std::forward<Args>(args)...);
|
||||
else if (isPlainRewritableStorage(type, config, config_prefix))
|
||||
{
|
||||
if (isPlainStorage(type, config, config_prefix))
|
||||
{
|
||||
return std::make_shared<PlainObjectStorage<BaseObjectStorage>>(std::forward<Args>(args)...);
|
||||
}
|
||||
else
|
||||
{
|
||||
return std::make_shared<BaseObjectStorage>(std::forward<Args>(args)...);
|
||||
}
|
||||
/// TODO(jkartseva@): Test support for generic disk type
|
||||
if (type != ObjectStorageType::S3)
|
||||
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "plain_rewritable metadata storage support is implemented only for S3");
|
||||
|
||||
return std::make_shared<PlainRewritableObjectStorage<BaseObjectStorage>>(std::forward<Args>(args)...);
|
||||
}
|
||||
else
|
||||
return std::make_shared<BaseObjectStorage>(std::forward<Args>(args)...);
|
||||
}
|
||||
}
|
||||
|
||||
ObjectStorageFactory & ObjectStorageFactory::instance()
|
||||
@ -76,10 +93,7 @@ ObjectStorageFactory & ObjectStorageFactory::instance()
|
||||
void ObjectStorageFactory::registerObjectStorageType(const std::string & type, Creator creator)
|
||||
{
|
||||
if (!registry.emplace(type, creator).second)
|
||||
{
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR,
|
||||
"ObjectStorageFactory: the metadata type '{}' is not unique", type);
|
||||
}
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "ObjectStorageFactory: the metadata type '{}' is not unique", type);
|
||||
}
|
||||
|
||||
ObjectStoragePtr ObjectStorageFactory::create(
|
||||
@ -91,13 +105,9 @@ ObjectStoragePtr ObjectStorageFactory::create(
|
||||
{
|
||||
std::string type;
|
||||
if (config.has(config_prefix + ".object_storage_type"))
|
||||
{
|
||||
type = config.getString(config_prefix + ".object_storage_type");
|
||||
}
|
||||
else if (config.has(config_prefix + ".type")) /// .type -- for compatibility.
|
||||
{
|
||||
type = config.getString(config_prefix + ".type");
|
||||
}
|
||||
else
|
||||
{
|
||||
throw Exception(ErrorCodes::NO_ELEMENTS_IN_CONFIG, "Expected `object_storage_type` in config");
|
||||
@ -210,31 +220,66 @@ void registerS3PlainObjectStorage(ObjectStorageFactory & factory)
|
||||
return object_storage;
|
||||
});
|
||||
}
|
||||
|
||||
void registerS3PlainRewritableObjectStorage(ObjectStorageFactory & factory)
|
||||
{
|
||||
static constexpr auto disk_type = "s3_plain_rewritable";
|
||||
|
||||
factory.registerObjectStorageType(
|
||||
disk_type,
|
||||
[](const std::string & name,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
const ContextPtr & context,
|
||||
bool skip_access_check) -> ObjectStoragePtr
|
||||
{
|
||||
/// send_metadata changes the filenames (includes revision), while
|
||||
/// s3_plain_rewritable does not support file renaming.
|
||||
if (config.getBool(config_prefix + ".send_metadata", false))
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "s3_plain_rewritable does not supports send_metadata");
|
||||
|
||||
auto uri = getS3URI(config, config_prefix, context);
|
||||
auto s3_capabilities = getCapabilitiesFromConfig(config, config_prefix);
|
||||
auto settings = getSettings(config, config_prefix, context);
|
||||
auto client = getClient(config, config_prefix, context, *settings);
|
||||
auto key_generator = getKeyGenerator(uri, config, config_prefix);
|
||||
|
||||
auto object_storage = std::make_shared<PlainRewritableObjectStorage<S3ObjectStorage>>(
|
||||
std::move(client), std::move(settings), uri, s3_capabilities, key_generator, name);
|
||||
|
||||
/// NOTE: should we still perform this check for clickhouse-disks?
|
||||
if (!skip_access_check)
|
||||
checkS3Capabilities(*dynamic_cast<S3ObjectStorage *>(object_storage.get()), s3_capabilities, name);
|
||||
|
||||
return object_storage;
|
||||
});
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
#if USE_HDFS && !defined(CLICKHOUSE_KEEPER_STANDALONE_BUILD)
|
||||
void registerHDFSObjectStorage(ObjectStorageFactory & factory)
|
||||
{
|
||||
factory.registerObjectStorageType("hdfs", [](
|
||||
const std::string & /* name */,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
const ContextPtr & context,
|
||||
bool /* skip_access_check */) -> ObjectStoragePtr
|
||||
{
|
||||
auto uri = context->getMacros()->expand(config.getString(config_prefix + ".endpoint"));
|
||||
checkHDFSURL(uri);
|
||||
if (uri.back() != '/')
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "HDFS path must ends with '/', but '{}' doesn't.", uri);
|
||||
factory.registerObjectStorageType(
|
||||
"hdfs",
|
||||
[](const std::string & /* name */,
|
||||
const Poco::Util::AbstractConfiguration & config,
|
||||
const std::string & config_prefix,
|
||||
const ContextPtr & context,
|
||||
bool /* skip_access_check */) -> ObjectStoragePtr
|
||||
{
|
||||
auto uri = context->getMacros()->expand(config.getString(config_prefix + ".endpoint"));
|
||||
checkHDFSURL(uri);
|
||||
if (uri.back() != '/')
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "HDFS path must ends with '/', but '{}' doesn't.", uri);
|
||||
|
||||
std::unique_ptr<HDFSObjectStorageSettings> settings = std::make_unique<HDFSObjectStorageSettings>(
|
||||
config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024),
|
||||
config.getInt(config_prefix + ".objects_chunk_size_to_delete", 1000),
|
||||
context->getSettingsRef().hdfs_replication
|
||||
);
|
||||
std::unique_ptr<HDFSObjectStorageSettings> settings = std::make_unique<HDFSObjectStorageSettings>(
|
||||
config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024),
|
||||
config.getInt(config_prefix + ".objects_chunk_size_to_delete", 1000),
|
||||
context->getSettingsRef().hdfs_replication);
|
||||
|
||||
return createObjectStorage<HDFSObjectStorage>(ObjectStorageType::HDFS, config, config_prefix, uri, std::move(settings), config);
|
||||
});
|
||||
return createObjectStorage<HDFSObjectStorage>(ObjectStorageType::HDFS, config, config_prefix, uri, std::move(settings), config);
|
||||
});
|
||||
}
|
||||
#endif
|
||||
|
||||
@ -317,6 +362,7 @@ void registerObjectStorages()
|
||||
#if USE_AWS_S3
|
||||
registerS3ObjectStorage(factory);
|
||||
registerS3PlainObjectStorage(factory);
|
||||
registerS3PlainRewritableObjectStorage(factory);
|
||||
#endif
|
||||
|
||||
#if USE_HDFS && !defined(CLICKHOUSE_KEEPER_STANDALONE_BUILD)
|
||||
|
||||
24
src/Disks/ObjectStorages/PlainRewritableObjectStorage.h
Normal file
24
src/Disks/ObjectStorages/PlainRewritableObjectStorage.h
Normal file
@ -0,0 +1,24 @@
|
||||
#pragma once
|
||||
|
||||
#include <Disks/ObjectStorages/IObjectStorage.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
template <typename BaseObjectStorage>
|
||||
class PlainRewritableObjectStorage : public BaseObjectStorage
|
||||
{
|
||||
public:
|
||||
template <class... Args>
|
||||
explicit PlainRewritableObjectStorage(Args &&... args) : BaseObjectStorage(std::forward<Args>(args)...)
|
||||
{
|
||||
}
|
||||
|
||||
std::string getName() const override { return "PlainRewritable" + BaseObjectStorage::getName(); }
|
||||
|
||||
bool isWriteOnce() const override { return false; }
|
||||
|
||||
bool isPlain() const override { return true; }
|
||||
};
|
||||
|
||||
}
|
||||
@ -29,7 +29,10 @@ void registerDiskObjectStorage(DiskFactory & factory, bool global_skip_access_ch
|
||||
if (!config.has(config_prefix + ".metadata_type"))
|
||||
{
|
||||
if (object_storage->isPlain())
|
||||
compatibility_metadata_type_hint = "plain";
|
||||
if (object_storage->isWriteOnce())
|
||||
compatibility_metadata_type_hint = "plain";
|
||||
else
|
||||
compatibility_metadata_type_hint = "plain_rewritable";
|
||||
else
|
||||
compatibility_metadata_type_hint = MetadataStorageFactory::getCompatibilityMetadataTypeHint(object_storage->getType());
|
||||
}
|
||||
@ -53,6 +56,7 @@ void registerDiskObjectStorage(DiskFactory & factory, bool global_skip_access_ch
|
||||
#if USE_AWS_S3
|
||||
factory.registerDiskType("s3", creator); /// For compatibility
|
||||
factory.registerDiskType("s3_plain", creator); /// For compatibility
|
||||
factory.registerDiskType("s3_plain_rewritable", creator); // For compatibility
|
||||
#endif
|
||||
#if USE_HDFS
|
||||
factory.registerDiskType("hdfs", creator); /// For compatibility
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
#include <Disks/ObjectStorages/S3/S3ObjectStorage.h>
|
||||
#include "Common/ObjectStorageKey.h"
|
||||
|
||||
#if USE_AWS_S3
|
||||
|
||||
@ -568,10 +569,17 @@ ObjectStorageKey S3ObjectStorage::generateObjectKeyForPath(const std::string & p
|
||||
{
|
||||
if (!key_generator)
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Key generator is not set");
|
||||
return key_generator->generate(path);
|
||||
|
||||
return key_generator->generate(path, /* is_directory */ false);
|
||||
}
|
||||
|
||||
ObjectStorageKey S3ObjectStorage::generateObjectKeyPrefixForDirectoryPath(const std::string & path) const
|
||||
{
|
||||
if (!key_generator)
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Key generator is not set");
|
||||
|
||||
return key_generator->generate(path, /* is_directory */ true);
|
||||
}
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
@ -43,8 +43,6 @@ struct S3ObjectStorageSettings
|
||||
class S3ObjectStorage : public IObjectStorage
|
||||
{
|
||||
private:
|
||||
friend class S3PlainObjectStorage;
|
||||
|
||||
S3ObjectStorage(
|
||||
const char * logger_name,
|
||||
std::unique_ptr<S3::Client> && client_,
|
||||
@ -54,11 +52,11 @@ private:
|
||||
ObjectStorageKeysGeneratorPtr key_generator_,
|
||||
const String & disk_name_)
|
||||
: uri(uri_)
|
||||
, key_generator(std::move(key_generator_))
|
||||
, disk_name(disk_name_)
|
||||
, client(std::move(client_))
|
||||
, s3_settings(std::move(s3_settings_))
|
||||
, s3_capabilities(s3_capabilities_)
|
||||
, key_generator(std::move(key_generator_))
|
||||
, log(getLogger(logger_name))
|
||||
{
|
||||
}
|
||||
@ -161,9 +159,12 @@ public:
|
||||
bool supportParallelWrite() const override { return true; }
|
||||
|
||||
ObjectStorageKey generateObjectKeyForPath(const std::string & path) const override;
|
||||
ObjectStorageKey generateObjectKeyPrefixForDirectoryPath(const std::string & path) const override;
|
||||
|
||||
bool isReadOnly() const override { return s3_settings.get()->read_only; }
|
||||
|
||||
void setKeysGenerator(ObjectStorageKeysGeneratorPtr gen) override { key_generator = gen; }
|
||||
|
||||
private:
|
||||
void setNewSettings(std::unique_ptr<S3ObjectStorageSettings> && s3_settings_);
|
||||
|
||||
@ -172,13 +173,14 @@ private:
|
||||
|
||||
const S3::URI uri;
|
||||
|
||||
ObjectStorageKeysGeneratorPtr key_generator;
|
||||
std::string disk_name;
|
||||
|
||||
MultiVersion<S3::Client> client;
|
||||
MultiVersion<S3ObjectStorageSettings> s3_settings;
|
||||
S3Capabilities s3_capabilities;
|
||||
|
||||
ObjectStorageKeysGeneratorPtr key_generator;
|
||||
|
||||
LoggerPtr log;
|
||||
};
|
||||
|
||||
|
||||
@ -1,9 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#include <Disks/ObjectStorages/IMetadataStorage.h>
|
||||
#include <Disks/ObjectStorages/MetadataFromDiskTransactionState.h>
|
||||
#include <Disks/ObjectStorages/Web/WebObjectStorage.h>
|
||||
#include <Disks/IDisk.h>
|
||||
#include <Disks/ObjectStorages/IMetadataStorage.h>
|
||||
#include <Disks/ObjectStorages/MetadataStorageTransactionState.h>
|
||||
#include <Disks/ObjectStorages/Web/WebObjectStorage.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -305,7 +305,7 @@ InputFormatPtr FormatFactory::getInput(
|
||||
|
||||
auto format_settings = _format_settings ? *_format_settings : getFormatSettings(context);
|
||||
const Settings & settings = context->getSettingsRef();
|
||||
size_t max_parsing_threads = _max_parsing_threads.value_or(settings.max_threads);
|
||||
size_t max_parsing_threads = _max_parsing_threads.value_or(settings.max_parsing_threads);
|
||||
size_t max_download_threads = _max_download_threads.value_or(settings.max_download_threads);
|
||||
|
||||
RowInputFormatParams row_input_format_params;
|
||||
|
||||
@ -478,19 +478,19 @@ static constexpr auto TO_START_OF_INTERVAL_NAME = "toStartOfInterval";
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Nanosecond>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateTimeIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 nanoseconds, Int64 scale_multiplier, UInt8, const DateLUTImpl &)
|
||||
static Int64 execute(Int64 t, Int64 nanoseconds, const DateLUTImpl &, Int64 scale_multiplier)
|
||||
{
|
||||
if (scale_multiplier < 1000000000)
|
||||
{
|
||||
@ -513,19 +513,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Nanosecond>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Microsecond>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateTimeIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 microseconds, Int64 scale_multiplier, UInt8, const DateLUTImpl &)
|
||||
static Int64 execute(Int64 t, Int64 microseconds, const DateLUTImpl &, Int64 scale_multiplier)
|
||||
{
|
||||
if (scale_multiplier < 1000000)
|
||||
{
|
||||
@ -556,19 +556,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Microsecond>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Millisecond>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateTimeIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 milliseconds, Int64 scale_multiplier, UInt8, const DateLUTImpl &)
|
||||
static Int64 execute(Int64 t, Int64 milliseconds, const DateLUTImpl &, Int64 scale_multiplier)
|
||||
{
|
||||
if (scale_multiplier < 1000)
|
||||
{
|
||||
@ -599,19 +599,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Millisecond>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Second>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32 t, Int64 seconds, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(UInt32 t, Int64 seconds, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfSecondInterval(t, seconds);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 seconds, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static Int64 execute(Int64 t, Int64 seconds, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfSecondInterval(t / scale_multiplier, seconds);
|
||||
}
|
||||
@ -620,19 +620,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Second>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Minute>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32 t, Int64 minutes, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(UInt32 t, Int64 minutes, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfMinuteInterval(t, minutes);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 minutes, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static Int64 execute(Int64 t, Int64 minutes, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfMinuteInterval(t / scale_multiplier, minutes);
|
||||
}
|
||||
@ -641,19 +641,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Minute>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Hour>
|
||||
{
|
||||
static UInt32 execute(UInt16, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(UInt16, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDateIsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(Int32, Int64, Int64, UInt8, const DateLUTImpl &)
|
||||
static UInt32 execute(Int32, Int64, const DateLUTImpl &, Int64)
|
||||
{
|
||||
throwDate32IsNotSupported(TO_START_OF_INTERVAL_NAME);
|
||||
}
|
||||
static UInt32 execute(UInt32 t, Int64 hours, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(UInt32 t, Int64 hours, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfHourInterval(t, hours);
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 hours, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static Int64 execute(Int64 t, Int64 hours, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfHourInterval(t / scale_multiplier, hours);
|
||||
}
|
||||
@ -662,19 +662,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Hour>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Day>
|
||||
{
|
||||
static UInt32 execute(UInt16 d, Int64 days, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(UInt16 d, Int64 days, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return static_cast<UInt32>(time_zone.toStartOfDayInterval(ExtendedDayNum(d), days));
|
||||
}
|
||||
static UInt32 execute(Int32 d, Int64 days, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(Int32 d, Int64 days, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return static_cast<UInt32>(time_zone.toStartOfDayInterval(ExtendedDayNum(d), days));
|
||||
}
|
||||
static UInt32 execute(UInt32 t, Int64 days, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt32 execute(UInt32 t, Int64 days, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return static_cast<UInt32>(time_zone.toStartOfDayInterval(time_zone.toDayNum(t), days));
|
||||
}
|
||||
static Int64 execute(Int64 t, Int64 days, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static Int64 execute(Int64 t, Int64 days, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfDayInterval(time_zone.toDayNum(t / scale_multiplier), days);
|
||||
}
|
||||
@ -683,40 +683,40 @@ struct ToStartOfInterval<IntervalKind::Kind::Day>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Week>
|
||||
{
|
||||
static UInt16 execute(UInt16 d, Int64 weeks, Int64, UInt8 week_mode, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt16 d, Int64 weeks, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfWeekInterval(DayNum(d), weeks, week_mode);
|
||||
return time_zone.toStartOfWeekInterval(DayNum(d), weeks);
|
||||
}
|
||||
static UInt16 execute(Int32 d, Int64 weeks, Int64, UInt8 week_mode, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int32 d, Int64 weeks, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfWeekInterval(ExtendedDayNum(d), weeks, week_mode);
|
||||
return time_zone.toStartOfWeekInterval(ExtendedDayNum(d), weeks);
|
||||
}
|
||||
static UInt16 execute(UInt32 t, Int64 weeks, Int64, UInt8 week_mode, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt32 t, Int64 weeks, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfWeekInterval(time_zone.toDayNum(t), weeks, week_mode);
|
||||
return time_zone.toStartOfWeekInterval(time_zone.toDayNum(t), weeks);
|
||||
}
|
||||
static UInt16 execute(Int64 t, Int64 weeks, Int64 scale_multiplier, UInt8 week_mode, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int64 t, Int64 weeks, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfWeekInterval(time_zone.toDayNum(t / scale_multiplier), weeks, week_mode);
|
||||
return time_zone.toStartOfWeekInterval(time_zone.toDayNum(t / scale_multiplier), weeks);
|
||||
}
|
||||
};
|
||||
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Month>
|
||||
{
|
||||
static UInt16 execute(UInt16 d, Int64 months, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt16 d, Int64 months, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfMonthInterval(DayNum(d), months);
|
||||
}
|
||||
static UInt16 execute(Int32 d, Int64 months, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int32 d, Int64 months, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfMonthInterval(ExtendedDayNum(d), months);
|
||||
}
|
||||
static UInt16 execute(UInt32 t, Int64 months, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt32 t, Int64 months, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfMonthInterval(time_zone.toDayNum(t), months);
|
||||
}
|
||||
static UInt16 execute(Int64 t, Int64 months, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int64 t, Int64 months, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfMonthInterval(time_zone.toDayNum(t / scale_multiplier), months);
|
||||
}
|
||||
@ -725,19 +725,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Month>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Quarter>
|
||||
{
|
||||
static UInt16 execute(UInt16 d, Int64 quarters, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt16 d, Int64 quarters, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfQuarterInterval(DayNum(d), quarters);
|
||||
}
|
||||
static UInt16 execute(Int32 d, Int64 quarters, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int32 d, Int64 quarters, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfQuarterInterval(ExtendedDayNum(d), quarters);
|
||||
}
|
||||
static UInt16 execute(UInt32 t, Int64 quarters, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt32 t, Int64 quarters, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfQuarterInterval(time_zone.toDayNum(t), quarters);
|
||||
}
|
||||
static UInt16 execute(Int64 t, Int64 quarters, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int64 t, Int64 quarters, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfQuarterInterval(time_zone.toDayNum(t / scale_multiplier), quarters);
|
||||
}
|
||||
@ -746,19 +746,19 @@ struct ToStartOfInterval<IntervalKind::Kind::Quarter>
|
||||
template <>
|
||||
struct ToStartOfInterval<IntervalKind::Kind::Year>
|
||||
{
|
||||
static UInt16 execute(UInt16 d, Int64 years, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt16 d, Int64 years, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfYearInterval(DayNum(d), years);
|
||||
}
|
||||
static UInt16 execute(Int32 d, Int64 years, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int32 d, Int64 years, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfYearInterval(ExtendedDayNum(d), years);
|
||||
}
|
||||
static UInt16 execute(UInt32 t, Int64 years, Int64, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(UInt32 t, Int64 years, const DateLUTImpl & time_zone, Int64)
|
||||
{
|
||||
return time_zone.toStartOfYearInterval(time_zone.toDayNum(t), years);
|
||||
}
|
||||
static UInt16 execute(Int64 t, Int64 years, Int64 scale_multiplier, UInt8, const DateLUTImpl & time_zone)
|
||||
static UInt16 execute(Int64 t, Int64 years, const DateLUTImpl & time_zone, Int64 scale_multiplier)
|
||||
{
|
||||
return time_zone.toStartOfYearInterval(time_zone.toDayNum(t / scale_multiplier), years);
|
||||
}
|
||||
|
||||
@ -130,6 +130,7 @@ UUIDSerializer::Variant parseVariant(const DB::ColumnsWithTypeAndName & argument
|
||||
|
||||
const auto representation = static_cast<magic_enum::underlying_type_t<UUIDSerializer::Variant>>(arguments[1].column->getInt(0));
|
||||
const auto as_enum = magic_enum::enum_cast<UUIDSerializer::Variant>(representation);
|
||||
|
||||
if (!as_enum)
|
||||
throw DB::Exception(DB::ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Expected UUID variant, got {}", representation);
|
||||
|
||||
@ -170,6 +171,7 @@ public:
|
||||
}
|
||||
|
||||
bool useDefaultImplementationForConstants() const override { return true; }
|
||||
ColumnNumbers getArgumentsThatAreAlwaysConstant() const override { return {1}; }
|
||||
|
||||
ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t /*input_rows_count*/) const override
|
||||
{
|
||||
@ -248,6 +250,7 @@ public:
|
||||
}
|
||||
|
||||
bool useDefaultImplementationForConstants() const override { return true; }
|
||||
ColumnNumbers getArgumentsThatAreAlwaysConstant() const override { return {1}; }
|
||||
|
||||
ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t /*input_rows_count*/) const override
|
||||
{
|
||||
|
||||
@ -39,17 +39,9 @@ struct ToStartOfTransform;
|
||||
TRANSFORM_DATE(Year)
|
||||
TRANSFORM_DATE(Quarter)
|
||||
TRANSFORM_DATE(Month)
|
||||
TRANSFORM_DATE(Week)
|
||||
#undef TRANSFORM_DATE
|
||||
|
||||
template <>
|
||||
struct ToStartOfTransform<IntervalKind::Kind::Week>
|
||||
{
|
||||
static auto execute(UInt32 t, UInt64 delta, const DateLUTImpl & time_zone)
|
||||
{
|
||||
return time_zone.toStartOfWeekInterval(time_zone.toDayNum(t), delta, /*week_mode*/ 1);
|
||||
}
|
||||
};
|
||||
|
||||
template <>
|
||||
struct ToStartOfTransform<IntervalKind::Kind::Day>
|
||||
{
|
||||
|
||||
@ -45,7 +45,7 @@ template <> struct FunctionUnaryArithmeticMonotonicity<NameAbs>
|
||||
if ((left_float < 0 && right_float > 0) || (left_float > 0 && right_float < 0))
|
||||
return {};
|
||||
|
||||
return { .is_monotonic = true, .is_positive = left_float > 0, .is_strict = true, };
|
||||
return { .is_monotonic = true, .is_positive = std::min(left_float, right_float) >= 0, .is_strict = true, };
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -2,7 +2,6 @@
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <Common/DateLUTImpl.h>
|
||||
#include <Common/IntervalKind.h>
|
||||
#include <Core/SettingsEnums.h>
|
||||
#include <DataTypes/DataTypeDate.h>
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
#include <DataTypes/DataTypeDateTime64.h>
|
||||
@ -10,7 +9,6 @@
|
||||
#include <Functions/DateTimeTransforms.h>
|
||||
#include <Functions/FunctionFactory.h>
|
||||
#include <Functions/IFunction.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
|
||||
|
||||
@ -28,13 +26,9 @@ namespace ErrorCodes
|
||||
class FunctionToStartOfInterval : public IFunction
|
||||
{
|
||||
public:
|
||||
static constexpr auto name = "toStartOfInterval";
|
||||
static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionToStartOfInterval>(); }
|
||||
|
||||
static FunctionPtr create(ContextPtr context) { return std::make_shared<FunctionToStartOfInterval>(context); }
|
||||
explicit FunctionToStartOfInterval(ContextPtr context)
|
||||
: first_day_of_week(context->getSettingsRef().first_day_of_week)
|
||||
{
|
||||
}
|
||||
static constexpr auto name = "toStartOfInterval";
|
||||
String getName() const override { return name; }
|
||||
bool isVariadic() const override { return true; }
|
||||
size_t getNumberOfArguments() const override { return 0; }
|
||||
@ -253,16 +247,13 @@ private:
|
||||
auto & result_data = col_to->getData();
|
||||
result_data.resize(size);
|
||||
|
||||
const Int64 scale_multiplier = DecimalUtils::scaleMultiplier<DateTime64>(scale);
|
||||
const UInt8 week_mode = (first_day_of_week == FirstDayOfWeek::Monday) ? 1 : 0;
|
||||
Int64 scale_multiplier = DecimalUtils::scaleMultiplier<DateTime64>(scale);
|
||||
|
||||
for (size_t i = 0; i != size; ++i)
|
||||
result_data[i] = static_cast<ResultFieldType>(ToStartOfInterval<unit>::execute(time_data[i], num_units, scale_multiplier, week_mode, time_zone));
|
||||
result_data[i] = static_cast<ResultFieldType>(ToStartOfInterval<unit>::execute(time_data[i], num_units, time_zone, scale_multiplier));
|
||||
|
||||
return result_col;
|
||||
}
|
||||
|
||||
const FirstDayOfWeek first_day_of_week;
|
||||
};
|
||||
|
||||
REGISTER_FUNCTION(ToStartOfInterval)
|
||||
|
||||
@ -1374,18 +1374,18 @@ std::shared_ptr<const EnabledRolesInfo> Context::getRolesInfo() const
|
||||
namespace
|
||||
{
|
||||
ALWAYS_INLINE inline void
|
||||
contextSanityCheckWithLock(const Context & context, const Settings & settings, const std::lock_guard<ContextSharedMutex> &)
|
||||
contextSanityClampSettingsWithLock(const Context & context, Settings & settings, const std::lock_guard<ContextSharedMutex> &)
|
||||
{
|
||||
const auto type = context.getApplicationType();
|
||||
if (type == Context::ApplicationType::LOCAL || type == Context::ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
ALWAYS_INLINE inline void contextSanityCheck(const Context & context, const Settings & settings)
|
||||
ALWAYS_INLINE inline void contextSanityClampSettings(const Context & context, Settings & settings)
|
||||
{
|
||||
const auto type = context.getApplicationType();
|
||||
if (type == Context::ApplicationType::LOCAL || type == Context::ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
}
|
||||
|
||||
@ -1498,7 +1498,7 @@ void Context::setCurrentProfilesWithLock(const SettingsProfilesInfo & profiles_i
|
||||
checkSettingsConstraintsWithLock(profiles_info.settings, SettingSource::PROFILE);
|
||||
applySettingsChangesWithLock(profiles_info.settings, lock);
|
||||
settings_constraints_and_current_profiles = profiles_info.getConstraintsAndProfileIDs(settings_constraints_and_current_profiles);
|
||||
contextSanityCheckWithLock(*this, settings, lock);
|
||||
contextSanityClampSettingsWithLock(*this, settings, lock);
|
||||
}
|
||||
|
||||
void Context::setCurrentProfile(const String & profile_name, bool check_constraints)
|
||||
@ -2101,7 +2101,7 @@ void Context::setSettings(const Settings & settings_)
|
||||
std::lock_guard lock(mutex);
|
||||
settings = settings_;
|
||||
need_recalculate_access = true;
|
||||
contextSanityCheck(*this, settings);
|
||||
contextSanityClampSettings(*this, settings);
|
||||
}
|
||||
|
||||
void Context::setSettingWithLock(std::string_view name, const String & value, const std::lock_guard<ContextSharedMutex> & lock)
|
||||
@ -2114,7 +2114,7 @@ void Context::setSettingWithLock(std::string_view name, const String & value, co
|
||||
settings.set(name, value);
|
||||
if (ContextAccessParams::dependsOnSettingName(name))
|
||||
need_recalculate_access = true;
|
||||
contextSanityCheckWithLock(*this, settings, lock);
|
||||
contextSanityClampSettingsWithLock(*this, settings, lock);
|
||||
}
|
||||
|
||||
void Context::setSettingWithLock(std::string_view name, const Field & value, const std::lock_guard<ContextSharedMutex> & lock)
|
||||
@ -2134,7 +2134,7 @@ void Context::applySettingChangeWithLock(const SettingChange & change, const std
|
||||
try
|
||||
{
|
||||
setSettingWithLock(change.name, change.value, lock);
|
||||
contextSanityCheckWithLock(*this, settings, lock);
|
||||
contextSanityClampSettingsWithLock(*this, settings, lock);
|
||||
}
|
||||
catch (Exception & e)
|
||||
{
|
||||
@ -2162,7 +2162,7 @@ void Context::setSetting(std::string_view name, const Field & value)
|
||||
{
|
||||
std::lock_guard lock(mutex);
|
||||
setSettingWithLock(name, value, lock);
|
||||
contextSanityCheckWithLock(*this, settings, lock);
|
||||
contextSanityClampSettingsWithLock(*this, settings, lock);
|
||||
}
|
||||
|
||||
void Context::applySettingChange(const SettingChange & change)
|
||||
@ -2187,39 +2187,39 @@ void Context::applySettingsChanges(const SettingsChanges & changes)
|
||||
applySettingsChangesWithLock(changes, lock);
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingsProfileElements & profile_elements, SettingSource source) const
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingsProfileElements & profile_elements, SettingSource source)
|
||||
{
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(settings, profile_elements, source);
|
||||
if (getApplicationType() == ApplicationType::LOCAL || getApplicationType() == ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingChange & change, SettingSource source) const
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingChange & change, SettingSource source)
|
||||
{
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(settings, change, source);
|
||||
if (getApplicationType() == ApplicationType::LOCAL || getApplicationType() == ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingsChanges & changes, SettingSource source) const
|
||||
void Context::checkSettingsConstraintsWithLock(const SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(settings, changes, source);
|
||||
if (getApplicationType() == ApplicationType::LOCAL || getApplicationType() == ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source) const
|
||||
void Context::checkSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(settings, changes, source);
|
||||
if (getApplicationType() == ApplicationType::LOCAL || getApplicationType() == ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::clampToSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source) const
|
||||
void Context::clampToSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.clamp(settings, changes, source);
|
||||
if (getApplicationType() == ApplicationType::LOCAL || getApplicationType() == ApplicationType::SERVER)
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::checkMergeTreeSettingsConstraintsWithLock(const MergeTreeSettings & merge_tree_settings, const SettingsChanges & changes) const
|
||||
@ -2227,32 +2227,32 @@ void Context::checkMergeTreeSettingsConstraintsWithLock(const MergeTreeSettings
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(merge_tree_settings, changes);
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraints(const SettingsProfileElements & profile_elements, SettingSource source) const
|
||||
void Context::checkSettingsConstraints(const SettingsProfileElements & profile_elements, SettingSource source)
|
||||
{
|
||||
SharedLockGuard lock(mutex);
|
||||
checkSettingsConstraintsWithLock(profile_elements, source);
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraints(const SettingChange & change, SettingSource source) const
|
||||
void Context::checkSettingsConstraints(const SettingChange & change, SettingSource source)
|
||||
{
|
||||
SharedLockGuard lock(mutex);
|
||||
checkSettingsConstraintsWithLock(change, source);
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraints(const SettingsChanges & changes, SettingSource source) const
|
||||
void Context::checkSettingsConstraints(const SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
SharedLockGuard lock(mutex);
|
||||
getSettingsConstraintsAndCurrentProfilesWithLock()->constraints.check(settings, changes, source);
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
}
|
||||
|
||||
void Context::checkSettingsConstraints(SettingsChanges & changes, SettingSource source) const
|
||||
void Context::checkSettingsConstraints(SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
SharedLockGuard lock(mutex);
|
||||
checkSettingsConstraintsWithLock(changes, source);
|
||||
}
|
||||
|
||||
void Context::clampToSettingsConstraints(SettingsChanges & changes, SettingSource source) const
|
||||
void Context::clampToSettingsConstraints(SettingsChanges & changes, SettingSource source)
|
||||
{
|
||||
SharedLockGuard lock(mutex);
|
||||
clampToSettingsConstraintsWithLock(changes, source);
|
||||
@ -4484,7 +4484,7 @@ void Context::setDefaultProfiles(const Poco::Util::AbstractConfiguration & confi
|
||||
setCurrentProfile(shared->system_profile_name);
|
||||
|
||||
applySettingsQuirks(settings, getLogger("SettingsQuirks"));
|
||||
doSettingsSanityCheck(settings);
|
||||
doSettingsSanityCheckClamp(settings, getLogger("SettingsSanity"));
|
||||
|
||||
shared->buffer_profile_name = config.getString("buffer_profile", shared->system_profile_name);
|
||||
buffer_context = Context::createCopy(shared_from_this());
|
||||
|
||||
@ -315,6 +315,7 @@ protected:
|
||||
/// This parameter can be set by the HTTP client to tune the behavior of output formats for compatibility.
|
||||
UInt64 client_protocol_version = 0;
|
||||
|
||||
public:
|
||||
/// Record entities accessed by current query, and store this information in system.query_log.
|
||||
struct QueryAccessInfo
|
||||
{
|
||||
@ -339,8 +340,10 @@ protected:
|
||||
return *this;
|
||||
}
|
||||
|
||||
void swap(QueryAccessInfo & rhs) noexcept
|
||||
void swap(QueryAccessInfo & rhs) noexcept TSA_NO_THREAD_SAFETY_ANALYSIS
|
||||
{
|
||||
/// TSA_NO_THREAD_SAFETY_ANALYSIS because it doesn't support scoped_lock
|
||||
std::scoped_lock lck{mutex, rhs.mutex};
|
||||
std::swap(databases, rhs.databases);
|
||||
std::swap(tables, rhs.tables);
|
||||
std::swap(columns, rhs.columns);
|
||||
@ -351,19 +354,21 @@ protected:
|
||||
|
||||
/// To prevent a race between copy-constructor and other uses of this structure.
|
||||
mutable std::mutex mutex{};
|
||||
std::set<std::string> databases{};
|
||||
std::set<std::string> tables{};
|
||||
std::set<std::string> columns{};
|
||||
std::set<std::string> partitions{};
|
||||
std::set<std::string> projections{};
|
||||
std::set<std::string> views{};
|
||||
std::set<std::string> databases TSA_GUARDED_BY(mutex){};
|
||||
std::set<std::string> tables TSA_GUARDED_BY(mutex){};
|
||||
std::set<std::string> columns TSA_GUARDED_BY(mutex){};
|
||||
std::set<std::string> partitions TSA_GUARDED_BY(mutex){};
|
||||
std::set<std::string> projections TSA_GUARDED_BY(mutex){};
|
||||
std::set<std::string> views TSA_GUARDED_BY(mutex){};
|
||||
};
|
||||
using QueryAccessInfoPtr = std::shared_ptr<QueryAccessInfo>;
|
||||
|
||||
protected:
|
||||
/// In some situations, we want to be able to transfer the access info from children back to parents (e.g. definers context).
|
||||
/// Therefore, query_access_info must be a pointer.
|
||||
QueryAccessInfoPtr query_access_info;
|
||||
|
||||
public:
|
||||
/// Record names of created objects of factories (for testing, etc)
|
||||
struct QueryFactoriesInfo
|
||||
{
|
||||
@ -385,19 +390,20 @@ protected:
|
||||
|
||||
QueryFactoriesInfo(QueryFactoriesInfo && rhs) = delete;
|
||||
|
||||
std::unordered_set<std::string> aggregate_functions;
|
||||
std::unordered_set<std::string> aggregate_function_combinators;
|
||||
std::unordered_set<std::string> database_engines;
|
||||
std::unordered_set<std::string> data_type_families;
|
||||
std::unordered_set<std::string> dictionaries;
|
||||
std::unordered_set<std::string> formats;
|
||||
std::unordered_set<std::string> functions;
|
||||
std::unordered_set<std::string> storages;
|
||||
std::unordered_set<std::string> table_functions;
|
||||
std::unordered_set<std::string> aggregate_functions TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> aggregate_function_combinators TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> database_engines TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> data_type_families TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> dictionaries TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> formats TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> functions TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> storages TSA_GUARDED_BY(mutex);
|
||||
std::unordered_set<std::string> table_functions TSA_GUARDED_BY(mutex);
|
||||
|
||||
mutable std::mutex mutex;
|
||||
};
|
||||
|
||||
protected:
|
||||
/// Needs to be changed while having const context in factories methods
|
||||
mutable QueryFactoriesInfo query_factories_info;
|
||||
/// Query metrics for reading data asynchronously with IAsynchronousReader.
|
||||
@ -783,11 +789,11 @@ public:
|
||||
void applySettingsChanges(const SettingsChanges & changes);
|
||||
|
||||
/// Checks the constraints.
|
||||
void checkSettingsConstraints(const SettingsProfileElements & profile_elements, SettingSource source) const;
|
||||
void checkSettingsConstraints(const SettingChange & change, SettingSource source) const;
|
||||
void checkSettingsConstraints(const SettingsChanges & changes, SettingSource source) const;
|
||||
void checkSettingsConstraints(SettingsChanges & changes, SettingSource source) const;
|
||||
void clampToSettingsConstraints(SettingsChanges & changes, SettingSource source) const;
|
||||
void checkSettingsConstraints(const SettingsProfileElements & profile_elements, SettingSource source);
|
||||
void checkSettingsConstraints(const SettingChange & change, SettingSource source);
|
||||
void checkSettingsConstraints(const SettingsChanges & changes, SettingSource source);
|
||||
void checkSettingsConstraints(SettingsChanges & changes, SettingSource source);
|
||||
void clampToSettingsConstraints(SettingsChanges & changes, SettingSource source);
|
||||
void checkMergeTreeSettingsConstraints(const MergeTreeSettings & merge_tree_settings, const SettingsChanges & changes) const;
|
||||
|
||||
/// Reset settings to default value
|
||||
@ -1293,15 +1299,15 @@ private:
|
||||
|
||||
void setCurrentDatabaseWithLock(const String & name, const std::lock_guard<ContextSharedMutex> & lock);
|
||||
|
||||
void checkSettingsConstraintsWithLock(const SettingsProfileElements & profile_elements, SettingSource source) const;
|
||||
void checkSettingsConstraintsWithLock(const SettingsProfileElements & profile_elements, SettingSource source);
|
||||
|
||||
void checkSettingsConstraintsWithLock(const SettingChange & change, SettingSource source) const;
|
||||
void checkSettingsConstraintsWithLock(const SettingChange & change, SettingSource source);
|
||||
|
||||
void checkSettingsConstraintsWithLock(const SettingsChanges & changes, SettingSource source) const;
|
||||
void checkSettingsConstraintsWithLock(const SettingsChanges & changes, SettingSource source);
|
||||
|
||||
void checkSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source) const;
|
||||
void checkSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source);
|
||||
|
||||
void clampToSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source) const;
|
||||
void clampToSettingsConstraintsWithLock(SettingsChanges & changes, SettingSource source);
|
||||
|
||||
void checkMergeTreeSettingsConstraintsWithLock(const MergeTreeSettings & merge_tree_settings, const SettingsChanges & changes) const;
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user