mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-27 10:02:01 +00:00
Merge branch 'master' into kssenii-patch-6
This commit is contained in:
Kseniia Sumarokova
GitHub
commit
d211ec943b
132
CHANGELOG.md
132
CHANGELOG.md
@ -1,4 +1,5 @@
|
||||
### Table of Contents
|
||||
**[ClickHouse release v22.12, 2022-12-15](#2212)**<br/>
|
||||
**[ClickHouse release v22.11, 2022-11-17](#2211)**<br/>
|
||||
**[ClickHouse release v22.10, 2022-10-25](#2210)**<br/>
|
||||
**[ClickHouse release v22.9, 2022-09-22](#229)**<br/>
|
||||
@ -12,6 +13,121 @@
|
||||
**[ClickHouse release v22.1, 2022-01-18](#221)**<br/>
|
||||
**[Changelog for 2021](https://clickhouse.com/docs/en/whats-new/changelog/2021/)**<br/>
|
||||
|
||||
# 2022 Changelog
|
||||
|
||||
### <a id="2212"></a> ClickHouse release 22.12, 2022-12-15
|
||||
|
||||
#### Upgrade Notes
|
||||
* Fixed backward incompatibility in (de)serialization of states of `min`, `max`, `any*`, `argMin`, `argMax` aggregate functions with `String` argument. The incompatibility affects 22.9, 22.10 and 22.11 branches (fixed since 22.9.6, 22.10.4 and 22.11.2 correspondingly). Some minor releases of 22.3, 22.7 and 22.8 branches are also affected: 22.3.13...22.3.14 (fixed since 22.3.15), 22.8.6...22.8.9 (fixed since 22.8.10), 22.7.6 and newer (will not be fixed in 22.7, we recommend upgrading from 22.7.* to 22.8.10 or newer). This release note does not concern users that have never used affected versions. Incompatible versions append an extra `'\0'` to strings when reading states of the aggregate functions mentioned above. For example, if an older version saved state of `anyState('foobar')` to `state_column` then the incompatible version will print `'foobar\0'` on `anyMerge(state_column)`. Also incompatible versions write states of the aggregate functions without trailing `'\0'`. Newer versions (that have the fix) can correctly read data written by all versions including incompatible versions, except one corner case. If an incompatible version saved a state with a string that actually ends with null character, then newer version will trim trailing `'\0'` when reading state of affected aggregate function. For example, if an incompatible version saved state of `anyState('abrac\0dabra\0')` to `state_column` then newer versions will print `'abrac\0dabra'` on `anyMerge(state_column)`. The issue also affects distributed queries when an incompatible version works in a cluster together with older or newer versions. [#43038](https://github.com/ClickHouse/ClickHouse/pull/43038) ([Alexander Tokmakov](https://github.com/tavplubix), [Raúl Marín](https://github.com/Algunenano)). Note: all the official ClickHouse builds already include the patches. This is not necessarily true for unofficial third-party builds that should be avoided.
|
||||
|
||||
#### New Feature
|
||||
* Add `BSONEachRow` input/output format. In this format, ClickHouse formats/parses each row as a separate BSON document and each column is formatted/parsed as a single BSON field with the column name as the key. [#42033](https://github.com/ClickHouse/ClickHouse/pull/42033) ([mark-polokhov](https://github.com/mark-polokhov)).
|
||||
* Add `grace_hash` JOIN algorithm, it can be enabled with `SET join_algorithm = 'grace_hash'`. [#38191](https://github.com/ClickHouse/ClickHouse/pull/38191) ([BigRedEye](https://github.com/BigRedEye), [Vladimir C](https://github.com/vdimir)).
|
||||
* Allow configuring password complexity rules and checks for creating and changing users. [#43719](https://github.com/ClickHouse/ClickHouse/pull/43719) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Mask sensitive information in logs; mask secret parts in the output of queries `SHOW CREATE TABLE` and `SELECT FROM system.tables`. Also resolves [#41418](https://github.com/ClickHouse/ClickHouse/issues/41418). [#43227](https://github.com/ClickHouse/ClickHouse/pull/43227) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Add `GROUP BY ALL` syntax: [#37631](https://github.com/ClickHouse/ClickHouse/issues/37631). [#42265](https://github.com/ClickHouse/ClickHouse/pull/42265) ([刘陶峰](https://github.com/taofengliu)).
|
||||
* Add `FROM table SELECT column` syntax. [#41095](https://github.com/ClickHouse/ClickHouse/pull/41095) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Added function `concatWithSeparator` and `concat_ws` as an alias for Spark SQL compatibility. A function `concatWithSeparatorAssumeInjective` added as a variant to enable GROUP BY optimization, similarly to `concatAssumeInjective`. [#43749](https://github.com/ClickHouse/ClickHouse/pull/43749) ([李扬](https://github.com/taiyang-li)).
|
||||
* Added `multiplyDecimal` and `divideDecimal` functions for decimal operations with fixed precision. [#42438](https://github.com/ClickHouse/ClickHouse/pull/42438) ([Andrey Zvonov](https://github.com/zvonand)).
|
||||
* Added `system.moves` table with list of currently moving parts. [#42660](https://github.com/ClickHouse/ClickHouse/pull/42660) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Add support for embedded Prometheus endpoint for ClickHouse Keeper. [#43087](https://github.com/ClickHouse/ClickHouse/pull/43087) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Support numeric literals with `_` as the separator, for example, `1_000_000`. [#43925](https://github.com/ClickHouse/ClickHouse/pull/43925) ([jh0x](https://github.com/jh0x)).
|
||||
* Added possibility to use an array as a second parameter for `cutURLParameter` function. It will cut multiple parameters. Close [#6827](https://github.com/ClickHouse/ClickHouse/issues/6827). [#43788](https://github.com/ClickHouse/ClickHouse/pull/43788) ([Roman Vasin](https://github.com/rvasin)).
|
||||
* Add a column with the expression of the index in the `system.data_skipping_indices` table. [#43308](https://github.com/ClickHouse/ClickHouse/pull/43308) ([Guillaume Tassery](https://github.com/YiuRULE)).
|
||||
* Add column `engine_full` to system table `databases` so that users can access the entire engine definition of a database via system tables. [#43468](https://github.com/ClickHouse/ClickHouse/pull/43468) ([凌涛](https://github.com/lingtaolf)).
|
||||
* New hash function [xxh3](https://github.com/Cyan4973/xxHash) added. Also, the performance of `xxHash32` and `xxHash64` are improved on ARM thanks to a library update. [#43411](https://github.com/ClickHouse/ClickHouse/pull/43411) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Added support to define constraints for merge tree settings. For example you can forbid overriding the `storage_policy` by users. [#43903](https://github.com/ClickHouse/ClickHouse/pull/43903) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Add a new setting `input_format_json_read_objects_as_strings` that allows the parsing of nested JSON objects into Strings in all JSON input formats. This setting is disabled by default. [#44052](https://github.com/ClickHouse/ClickHouse/pull/44052) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
|
||||
#### Experimental Feature
|
||||
* Support deduplication for asynchronous inserts. Before this change, async inserts did not support deduplication, because multiple small inserts coexisted in one inserted batch. Closes [#38075](https://github.com/ClickHouse/ClickHouse/issues/38075). [#43304](https://github.com/ClickHouse/ClickHouse/pull/43304) ([Han Fei](https://github.com/hanfei1991)).
|
||||
* Add support for cosine distance for the experimental Annoy (vector similarity search) index. [#42778](https://github.com/ClickHouse/ClickHouse/pull/42778) ([Filatenkov Artur](https://github.com/FArthur-cmd)).
|
||||
* Add `CREATE / ALTER / DROP NAMED COLLECTION` queries. [#43252](https://github.com/ClickHouse/ClickHouse/pull/43252) ([Kseniia Sumarokova](https://github.com/kssenii)). This feature is under development and the queries are not effective as of version 22.12. This changelog entry is added only to avoid confusion. Restrict default access to named collections to the user defined in config. This requires that `show_named_collections = 1` is set to be able to see them. [#43325](https://github.com/ClickHouse/ClickHouse/pull/43325) ([Kseniia Sumarokova](https://github.com/kssenii)). The `system.named_collections` table is introduced [#43147](https://github.com/ClickHouse/ClickHouse/pull/43147) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
|
||||
#### Performance Improvement
|
||||
* Add settings `max_streams_for_merge_tree_reading` and `allow_asynchronous_read_from_io_pool_for_merge_tree`. Setting `max_streams_for_merge_tree_reading` limits the number of reading streams for MergeTree tables. Setting `allow_asynchronous_read_from_io_pool_for_merge_tree` enables a background I/O pool to read from `MergeTree` tables. This may increase performance for I/O bound queries if used together with `max_streams_to_max_threads_ratio` or `max_streams_for_merge_tree_reading`. [#43260](https://github.com/ClickHouse/ClickHouse/pull/43260) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). This improves performance up to 100 times in case of high latency storage, low number of CPU and high number of data parts.
|
||||
* Settings `merge_tree_min_rows_for_concurrent_read_for_remote_filesystem/merge_tree_min_bytes_for_concurrent_read_for_remote_filesystem` did not respect adaptive granularity. Fat rows did not decrease the number of read rows (as it was done for `merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read`, which could lead to high memory usage when using remote filesystems. [#43965](https://github.com/ClickHouse/ClickHouse/pull/43965) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Optimized the number of list requests to ZooKeeper or ClickHouse Keeper when selecting a part to merge. Previously it could produce thousands of requests in some cases. Fixes [#43647](https://github.com/ClickHouse/ClickHouse/issues/43647). [#43675](https://github.com/ClickHouse/ClickHouse/pull/43675) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Optimization is getting skipped now if `max_size_to_preallocate_for_aggregation` has too small a value. The default value of this setting increased to `10^8`. [#43945](https://github.com/ClickHouse/ClickHouse/pull/43945) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Speed-up server shutdown by avoiding cleaning up of old data parts. Because it is unnecessary after https://github.com/ClickHouse/ClickHouse/pull/41145. [#43760](https://github.com/ClickHouse/ClickHouse/pull/43760) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* Merging on initiator now uses the same memory bound approach as merging of local aggregation results if `enable_memory_bound_merging_of_aggregation_results` is set. [#40879](https://github.com/ClickHouse/ClickHouse/pull/40879) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Keeper improvement: try syncing logs to disk in parallel with replication. [#43450](https://github.com/ClickHouse/ClickHouse/pull/43450) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Keeper improvement: requests are batched more often. The batching can be controlled with the new setting `max_requests_quick_batch_size`. [#43686](https://github.com/ClickHouse/ClickHouse/pull/43686) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
|
||||
#### Improvement
|
||||
* Implement referential dependencies and use them to create tables in the correct order while restoring from a backup. [#43834](https://github.com/ClickHouse/ClickHouse/pull/43834) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Substitute UDFs in `CREATE` query to avoid failures during loading at startup. Additionally, UDFs can now be used as `DEFAULT` expressions for columns. [#43539](https://github.com/ClickHouse/ClickHouse/pull/43539) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Change how the following queries delete parts: TRUNCATE TABLE, ALTER TABLE DROP PART, ALTER TABLE DROP PARTITION. Now, these queries make empty parts which cover the old parts. This makes the TRUNCATE query work without a followedexclusive lock which means concurrent reads aren't locked. Also achieved durability in all those queries. If the request succeeds, then no resurrected parts appear later. Note that atomicity is achieved only with transaction scope. [#41145](https://github.com/ClickHouse/ClickHouse/pull/41145) ([Sema Checherinda](https://github.com/CheSema)).
|
||||
* `SET param_x` query no longer requires manual string serialization for the value of the parameter. For example, query `SET param_a = '[\'a\', \'b\']'` can now be written like `SET param_a = ['a', 'b']`. [#41874](https://github.com/ClickHouse/ClickHouse/pull/41874) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Show read rows in the progress indication while reading from STDIN from client. Closes [#43423](https://github.com/ClickHouse/ClickHouse/issues/43423). [#43442](https://github.com/ClickHouse/ClickHouse/pull/43442) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Show progress bar while reading from s3 table function / engine. [#43454](https://github.com/ClickHouse/ClickHouse/pull/43454) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* `filesystemAvailable` and related functions support one optional argument with disk name, and change `filesystemFree` to `filesystemUnreserved`. Closes [#35076](https://github.com/ClickHouse/ClickHouse/issues/35076). [#42064](https://github.com/ClickHouse/ClickHouse/pull/42064) ([flynn](https://github.com/ucasfl)).
|
||||
* Integration with LDAP: increased the default value of search_limit to 256, and added LDAP server config option to change that to an arbitrary value. Closes: [#42276](https://github.com/ClickHouse/ClickHouse/issues/42276). [#42461](https://github.com/ClickHouse/ClickHouse/pull/42461) ([Vasily Nemkov](https://github.com/Enmk)).
|
||||
* Allow the removal of sensitive information (see the `query_masking_rules` in the configuration file) from the exception messages as well. Resolves [#41418](https://github.com/ClickHouse/ClickHouse/issues/41418). [#42940](https://github.com/ClickHouse/ClickHouse/pull/42940) ([filimonov](https://github.com/filimonov)).
|
||||
* Support queries like `SHOW FULL TABLES ...` for MySQL compatibility. [#43910](https://github.com/ClickHouse/ClickHouse/pull/43910) ([Filatenkov Artur](https://github.com/FArthur-cmd)).
|
||||
* Keeper improvement: Add 4lw command `rqld` which can manually assign a node as leader. [#43026](https://github.com/ClickHouse/ClickHouse/pull/43026) ([JackyWoo](https://github.com/JackyWoo)).
|
||||
* Apply connection timeout settings for Distributed async INSERT from the query. [#43156](https://github.com/ClickHouse/ClickHouse/pull/43156) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* The `unhex` function now supports `FixedString` arguments. [issue42369](https://github.com/ClickHouse/ClickHouse/issues/42369). [#43207](https://github.com/ClickHouse/ClickHouse/pull/43207) ([DR](https://github.com/freedomDR)).
|
||||
* Priority is given to deleting completely expired parts according to the TTL rules, see [#42869](https://github.com/ClickHouse/ClickHouse/issues/42869). [#43222](https://github.com/ClickHouse/ClickHouse/pull/43222) ([zhongyuankai](https://github.com/zhongyuankai)).
|
||||
* More precise and reactive CPU load indication in clickhouse-client. [#43307](https://github.com/ClickHouse/ClickHouse/pull/43307) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Support reading of subcolumns of nested types from storage `S3` and table function `s3` with formats `Parquet`, `Arrow` and `ORC`. [#43329](https://github.com/ClickHouse/ClickHouse/pull/43329) ([chen](https://github.com/xiedeyantu)).
|
||||
* Add `table_uuid` column to the `system.parts` table. [#43404](https://github.com/ClickHouse/ClickHouse/pull/43404) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Added client option to display the number of locally processed rows in non-interactive mode (`--print-num-processed-rows`). [#43407](https://github.com/ClickHouse/ClickHouse/pull/43407) ([jh0x](https://github.com/jh0x)).
|
||||
* Implement `aggregation-in-order` optimization on top of a query plan. It is enabled by default (but works only together with `optimize_aggregation_in_order`, which is disabled by default). Set `query_plan_aggregation_in_order = 0` to use the previous AST-based version. [#43592](https://github.com/ClickHouse/ClickHouse/pull/43592) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Allow to collect profile events with `trace_type = 'ProfileEvent'` to `system.trace_log` on each increment with current stack, profile event name and value of the increment. It can be enabled by the setting `trace_profile_events` and used to investigate performance of queries. [#43639](https://github.com/ClickHouse/ClickHouse/pull/43639) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Add a new setting `input_format_max_binary_string_size` to limit string size in RowBinary format. [#43842](https://github.com/ClickHouse/ClickHouse/pull/43842) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* When ClickHouse requests a remote HTTP server, and it returns an error, the numeric HTTP code was not displayed correctly in the exception message. Closes [#43919](https://github.com/ClickHouse/ClickHouse/issues/43919). [#43920](https://github.com/ClickHouse/ClickHouse/pull/43920) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Correctly report errors in queries even when multiple JOINs optimization is taking place. [#43583](https://github.com/ClickHouse/ClickHouse/pull/43583) ([Salvatore](https://github.com/tbsal)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
|
||||
* Systemd integration now correctly notifies systemd that the service is really started and is ready to serve requests. [#43400](https://github.com/ClickHouse/ClickHouse/pull/43400) ([Коренберг Марк](https://github.com/socketpair)).
|
||||
* Added the option to build ClickHouse with OpenSSL using the [OpenSSL FIPS Module](https://www.openssl.org/docs/man3.0/man7/fips_module.html). This build type has not been tested to validate security and is not supported. [#43991](https://github.com/ClickHouse/ClickHouse/pull/43991) ([Boris Kuschel](https://github.com/bkuschel)).
|
||||
* Upgrade to the new `DeflateQpl` compression codec which has been implemented in a previous PR (details: https://github.com/ClickHouse/ClickHouse/pull/39494). This patch improves codec on below aspects: 1. QPL v0.2.0 to QPL v0.3.0 [Intel® Query Processing Library (QPL)](https://github.com/intel/qpl) 2. Improve CMake file for fixing QPL build issues for QPL v0.3.0. 3. Link the QPL library with libaccel-config at build time instead of runtime loading on QPL v0.2.0 (dlopen) 4. Fixed log print issue in CompressionCodecDeflateQpl.cpp. [#44024](https://github.com/ClickHouse/ClickHouse/pull/44024) ([jasperzhu](https://github.com/jinjunzh)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in official stable or prestable release)
|
||||
|
||||
* Fixed bug which could lead to deadlock while using asynchronous inserts. [#43233](https://github.com/ClickHouse/ClickHouse/pull/43233) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix some incorrect logic in AST level optimization `optimize_normalize_count_variants`. [#43873](https://github.com/ClickHouse/ClickHouse/pull/43873) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix a case when mutations are not making progress when checksums do not match between replicas (e.g. caused by a change in data format on an upgrade). [#36877](https://github.com/ClickHouse/ClickHouse/pull/36877) ([nvartolomei](https://github.com/nvartolomei)).
|
||||
* Fix the `skip_unavailable_shards` optimization which did not work with the `hdfsCluster` table function. [#43236](https://github.com/ClickHouse/ClickHouse/pull/43236) ([chen](https://github.com/xiedeyantu)).
|
||||
* Fix `s3` support for the `?` wildcard. Closes [#42731](https://github.com/ClickHouse/ClickHouse/issues/42731). [#43253](https://github.com/ClickHouse/ClickHouse/pull/43253) ([chen](https://github.com/xiedeyantu)).
|
||||
* Fix functions `arrayFirstOrNull` and `arrayLastOrNull` or null when the array contains `Nullable` elements. [#43274](https://github.com/ClickHouse/ClickHouse/pull/43274) ([Duc Canh Le](https://github.com/canhld94)).
|
||||
* Fix incorrect `UserTimeMicroseconds`/`SystemTimeMicroseconds` accounting related to Kafka tables. [#42791](https://github.com/ClickHouse/ClickHouse/pull/42791) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Do not suppress exceptions in `web` disks. Fix retries for the `web` disk. [#42800](https://github.com/ClickHouse/ClickHouse/pull/42800) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fixed (logical) race condition between inserts and dropping materialized views. A race condition happened when a Materialized View was dropped at the same time as an INSERT, where the MVs were present as a dependency of the insert at the begining of the execution, but the table has been dropped by the time the insert chain tries to access it, producing either an `UNKNOWN_TABLE` or `TABLE_IS_DROPPED` exception, and stopping the insertion. After this change, we avoid these exceptions and just continue with the insert if the dependency is gone. [#43161](https://github.com/ClickHouse/ClickHouse/pull/43161) ([AlfVII](https://github.com/AlfVII)).
|
||||
* Fix undefined behavior in the `quantiles` function, which might lead to uninitialized memory. Found by fuzzer. This closes [#44066](https://github.com/ClickHouse/ClickHouse/issues/44066). [#44067](https://github.com/ClickHouse/ClickHouse/pull/44067) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Additional check on zero uncompressed size is added to `CompressionCodecDelta`. [#43255](https://github.com/ClickHouse/ClickHouse/pull/43255) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Flatten arrays from Parquet to avoid an issue with inconsistent data in arrays. These incorrect files can be generated by Apache Iceberg. [#43297](https://github.com/ClickHouse/ClickHouse/pull/43297) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Fix bad cast from `LowCardinality` column when using short circuit function execution. [#43311](https://github.com/ClickHouse/ClickHouse/pull/43311) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fixed queries with `SAMPLE BY` with prewhere optimization on tables using `Merge` engine. [#43315](https://github.com/ClickHouse/ClickHouse/pull/43315) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Check and compare the content of the `format_version` file in `MergeTreeData` so that tables can be loaded even if the storage policy was changed. [#43328](https://github.com/ClickHouse/ClickHouse/pull/43328) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix possible (very unlikely) "No column to rollback" logical error during INSERT into `Buffer` tables. [#43336](https://github.com/ClickHouse/ClickHouse/pull/43336) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix a bug that allowed the parser to parse an unlimited amount of round brackets into one function if `allow_function_parameters` is set. [#43350](https://github.com/ClickHouse/ClickHouse/pull/43350) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* `MaterializeMySQL` (experimental feature) support DDL： `drop table t1, t2` and compatible with most of MySQL DROP DDL. [#43366](https://github.com/ClickHouse/ClickHouse/pull/43366) ([zzsmdfj](https://github.com/zzsmdfj)).
|
||||
* `session_log` (experimental feature): Fixed the inability to log in (because of failure to create the session_log entry) in a very rare case of messed up setting profiles. [#42641](https://github.com/ClickHouse/ClickHouse/pull/42641) ([Vasily Nemkov](https://github.com/Enmk)).
|
||||
* Fix possible `Cannot create non-empty column with type Nothing` in functions `if`/`multiIf`. Closes [#43356](https://github.com/ClickHouse/ClickHouse/issues/43356). [#43368](https://github.com/ClickHouse/ClickHouse/pull/43368) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix a bug when a row level filter uses the default value of a column. [#43387](https://github.com/ClickHouse/ClickHouse/pull/43387) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Query with `DISTINCT` + `LIMIT BY` + `LIMIT` can return fewer rows than expected. Fixes [#43377](https://github.com/ClickHouse/ClickHouse/issues/43377). [#43410](https://github.com/ClickHouse/ClickHouse/pull/43410) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Fix `sumMap` for `Nullable(Decimal(...))`. [#43414](https://github.com/ClickHouse/ClickHouse/pull/43414) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix `date_diff` for hour/minute on macOS. Close [#42742](https://github.com/ClickHouse/ClickHouse/issues/42742). [#43466](https://github.com/ClickHouse/ClickHouse/pull/43466) ([zzsmdfj](https://github.com/zzsmdfj)).
|
||||
* Fix incorrect memory accounting because of merges/mutations. [#43516](https://github.com/ClickHouse/ClickHouse/pull/43516) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fixed primary key analysis with conditions involving `toString(enum)`. [#43596](https://github.com/ClickHouse/ClickHouse/pull/43596) ([Nikita Taranov](https://github.com/nickitat)). This error has been found by @tisonkun.
|
||||
* Ensure consistency when `clickhouse-copier` updates status and `attach_is_done` in Keeper after partition attach is done. [#43602](https://github.com/ClickHouse/ClickHouse/pull/43602) ([lzydmxy](https://github.com/lzydmxy)).
|
||||
* During the recovery of a lost replica of a `Replicated` database (experimental feature), there could a situation where we need to atomically swap two table names (use EXCHANGE). Previously we tried to use two RENAME queries, which was obviously failing and moreover, failed the whole recovery process of the database replica. [#43628](https://github.com/ClickHouse/ClickHouse/pull/43628) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Fix the case when the `s3Cluster` function throws `NOT_FOUND_COLUMN_IN_BLOCK` error. Closes [#43534](https://github.com/ClickHouse/ClickHouse/issues/43534). [#43629](https://github.com/ClickHouse/ClickHouse/pull/43629) ([chen](https://github.com/xiedeyantu)).
|
||||
* Fix possible logical error `Array sizes mismatched` while parsing JSON object with arrays with same key names but with different nesting level. Closes [#43569](https://github.com/ClickHouse/ClickHouse/issues/43569). [#43693](https://github.com/ClickHouse/ClickHouse/pull/43693) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fixed possible exception in the case of distributed `GROUP BY` with an `ALIAS` column among aggregation keys. [#43709](https://github.com/ClickHouse/ClickHouse/pull/43709) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix bug which can lead to broken projections if zero-copy replication (experimental feature) is enabled and used. [#43764](https://github.com/ClickHouse/ClickHouse/pull/43764) ([alesapin](https://github.com/alesapin)).
|
||||
* Fix using multipart upload for very large S3 objects in AWS S3. [#43824](https://github.com/ClickHouse/ClickHouse/pull/43824) ([ianton-ru](https://github.com/ianton-ru)).
|
||||
* Fixed `ALTER ... RESET SETTING` with `ON CLUSTER`. It could have been applied to one replica only. Fixes [#43843](https://github.com/ClickHouse/ClickHouse/issues/43843). [#43848](https://github.com/ClickHouse/ClickHouse/pull/43848) ([Elena Torró](https://github.com/elenatorro)).
|
||||
* Fix a logical error in JOIN with `Join` table engine at right hand side, if `USING` is being used. [#43963](https://github.com/ClickHouse/ClickHouse/pull/43963) ([Vladimir C](https://github.com/vdimir)). Fix a bug with wrong order of keys in `Join` table engine. [#44012](https://github.com/ClickHouse/ClickHouse/pull/44012) ([Vladimir C](https://github.com/vdimir)).
|
||||
* Keeper fix: throw if the interserver port for Raft is already in use. [#43984](https://github.com/ClickHouse/ClickHouse/pull/43984) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix ORDER BY positional argument (example: `ORDER BY 1, 2`) in case of unneeded columns pruning from subqueries. Closes [#43964](https://github.com/ClickHouse/ClickHouse/issues/43964). [#43987](https://github.com/ClickHouse/ClickHouse/pull/43987) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fixed exception when a subquery contains HAVING but doesn't contain an actual aggregation. [#44051](https://github.com/ClickHouse/ClickHouse/pull/44051) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Fix race in s3 multipart upload. This race could cause the error `Part number must be an integer between 1 and 10000, inclusive. (S3_ERROR)` while restoring from a backup. [#44065](https://github.com/ClickHouse/ClickHouse/pull/44065) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
|
||||
|
||||
### <a id="2211"></a> ClickHouse release 22.11, 2022-11-17
|
||||
|
||||
#### Backward Incompatible Change
|
||||
@ -534,30 +650,30 @@
|
||||
* Add counters (ProfileEvents) for cases when query complexity limitation has been set and has reached (a separate counter for `overflow_mode` = `break` and `throw`). For example, if you have set up `max_rows_to_read` with `read_overflow_mode = 'break'`, looking at the value of `OverflowBreak` counter will allow distinguishing incomplete results. [#40205](https://github.com/ClickHouse/ClickHouse/pull/40205) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix memory accounting in case of "Memory limit exceeded" errors (previously [peak] memory usage was takes failed allocations into account). [#40249](https://github.com/ClickHouse/ClickHouse/pull/40249) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Add metrics for filesystem cache: `FilesystemCacheSize` and `FilesystemCacheElements`. [#40260](https://github.com/ClickHouse/ClickHouse/pull/40260) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Support hadoop secure RPC transfer (hadoop.rpc.protection=privacy and hadoop.rpc.protection=integrity). [#39411](https://github.com/ClickHouse/ClickHouse/pull/39411) ([michael1589](https://github.com/michael1589)).
|
||||
* Support Hadoop secure RPC transfer (hadoop.rpc.protection=privacy and hadoop.rpc.protection=integrity). [#39411](https://github.com/ClickHouse/ClickHouse/pull/39411) ([michael1589](https://github.com/michael1589)).
|
||||
* Avoid continuously growing memory consumption of pattern cache when using functions multi(Fuzzy)Match(Any|AllIndices|AnyIndex)(). [#40264](https://github.com/ClickHouse/ClickHouse/pull/40264) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add cache for schema inference for file/s3/hdfs/url table functions. Now, schema inference will be performed only on the first query to the file, all subsequent queries to the same file will use the schema from cache if data wasn't changed. Add system table system.schema_inference_cache with all current schemas in cache and system queries SYSTEM DROP SCHEMA CACHE [FOR FILE/S3/HDFS/URL] to drop schemas from cache. [#38286](https://github.com/ClickHouse/ClickHouse/pull/38286) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add cache for schema inference for file/s3/hdfs/url table functions. Now, schema inference will be performed only on the first query to the file, all subsequent queries to the same file will use the schema from the cache if data has not changed. Add system table system.schema_inference_cache with all current schemas in cache and system queries SYSTEM DROP SCHEMA CACHE [FOR FILE/S3/HDFS/URL] to drop schemas from cache. [#38286](https://github.com/ClickHouse/ClickHouse/pull/38286) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add support for LARGE_BINARY/LARGE_STRING with Arrow (Closes [#32401](https://github.com/ClickHouse/ClickHouse/issues/32401)). [#40293](https://github.com/ClickHouse/ClickHouse/pull/40293) ([Josh Taylor](https://github.com/joshuataylor)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
|
||||
* [ClickFiddle](https://fiddle.clickhouse.com/): A new tool for testing ClickHouse versions in read/write mode (**Igor Baliuk**).
|
||||
* ClickHouse binary is made self-extracting [#35775](https://github.com/ClickHouse/ClickHouse/pull/35775) ([Yakov Olkhovskiy, Arthur Filatenkov](https://github.com/yakov-olkhovskiy)).
|
||||
* Update tzdata to 2022b to support the new timezone changes. See https://github.com/google/cctz/pull/226. Chile's 2022 DST start is delayed from September 4 to September 11. Iran plans to stop observing DST permanently, after it falls back on 2022-09-21. There are corrections of the historical time zone of Asia/Tehran in the year 1977: Iran adopted standard time in 1935, not 1946. In 1977 it observed DST from 03-21 23:00 to 10-20 24:00; its 1978 transitions were on 03-24 and 08-05, not 03-20 and 10-20; and its spring 1979 transition was on 05-27, not 03-21 (https://data.iana.org/time-zones/tzdb/NEWS). ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Former packages used to install systemd.service file to `/etc`. The files there are marked as `conf` and are not cleaned out, and not updated automatically. This PR cleans them out. [#39323](https://github.com/ClickHouse/ClickHouse/pull/39323) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Update `tzdata` to 2022b to support the new timezone changes. See https://github.com/google/cctz/pull/226. Chile's 2022 DST start is delayed from September 4 to September 11. Iran plans to stop observing DST permanently after it falls back on 2022-09-21. There are corrections to the historical time zone of Asia/Tehran in the year 1977: Iran adopted standard time in 1935, not 1946. In 1977 it observed DST from 03-21 23:00 to 10-20 24:00; its 1978 transitions were on 03-24 and 08-05, not 03-20 and 10-20; and its spring 1979 transition was on 05-27, not 03-21 (https://data.iana.org/time-zones/tzdb/NEWS). ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Former packages used to install systemd.service file to `/etc`. The files there are marked as `conf` and are not cleaned out, and are not updated automatically. This PR cleans them out. [#39323](https://github.com/ClickHouse/ClickHouse/pull/39323) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Ensure LSan is effective. [#39430](https://github.com/ClickHouse/ClickHouse/pull/39430) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* TSAN has issues with clang-14 (https://github.com/google/sanitizers/issues/1552, https://github.com/google/sanitizers/issues/1540), so here we build the TSAN binaries with clang-15. [#39450](https://github.com/ClickHouse/ClickHouse/pull/39450) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
|
||||
* Remove the option to build ClickHouse tools as separate executable programs. This fixes [#37847](https://github.com/ClickHouse/ClickHouse/issues/37847). [#39520](https://github.com/ClickHouse/ClickHouse/pull/39520) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Small preparations for build on s390x (which is big-endian). [#39627](https://github.com/ClickHouse/ClickHouse/pull/39627) ([Harry Lee](https://github.com/HarryLeeIBM)). [#39656](https://github.com/ClickHouse/ClickHouse/pull/39656) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issue in BitHelpers for s390x. [#39656](https://github.com/ClickHouse/ClickHouse/pull/39656) ([Harry Lee](https://github.com/HarryLeeIBM)). Implement a piece of code related to SipHash for s390x architecture (which is not supported by ClickHouse). [#39732](https://github.com/ClickHouse/ClickHouse/pull/39732) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed an Endian issue in Coordination snapshot code for s390x architecture (which is not supported by ClickHouse). [#39931](https://github.com/ClickHouse/ClickHouse/pull/39931) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issues in Codec code for s390x architecture (which is not supported by ClickHouse). [#40008](https://github.com/ClickHouse/ClickHouse/pull/40008) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issues in reading/writing BigEndian binary data in ReadHelpers and WriteHelpers code for s390x architecture (which is not supported by ClickHouse). [#40179](https://github.com/ClickHouse/ClickHouse/pull/40179) ([Harry Lee](https://github.com/HarryLeeIBM)).
|
||||
* Small preparations for build on s390x (which is big-endian). [#39627](https://github.com/ClickHouse/ClickHouse/pull/39627) ([Harry Lee](https://github.com/HarryLeeIBM)). [#39656](https://github.com/ClickHouse/ClickHouse/pull/39656) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issue in BitHelpers for s390x. [#39656](https://github.com/ClickHouse/ClickHouse/pull/39656) ([Harry Lee](https://github.com/HarryLeeIBM)). Implement a piece of code related to SipHash for s390x architecture (which is not supported by ClickHouse). [#39732](https://github.com/ClickHouse/ClickHouse/pull/39732) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed an Endian issue in the Coordination snapshot code for s390x architecture (which is not supported by ClickHouse). [#39931](https://github.com/ClickHouse/ClickHouse/pull/39931) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issues in Codec code for s390x architecture (which is not supported by ClickHouse). [#40008](https://github.com/ClickHouse/ClickHouse/pull/40008) ([Harry Lee](https://github.com/HarryLeeIBM)). Fixed Endian issues in reading/writing BigEndian binary data in ReadHelpers and WriteHelpers code for s390x architecture (which is not supported by ClickHouse). [#40179](https://github.com/ClickHouse/ClickHouse/pull/40179) ([Harry Lee](https://github.com/HarryLeeIBM)).
|
||||
* Support build with `clang-16` (trunk). This closes [#39949](https://github.com/ClickHouse/ClickHouse/issues/39949). [#40181](https://github.com/ClickHouse/ClickHouse/pull/40181) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Prepare RISC-V 64 build to run in CI. This is for [#40141](https://github.com/ClickHouse/ClickHouse/issues/40141). [#40197](https://github.com/ClickHouse/ClickHouse/pull/40197) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Simplified function registration macro interface (`FUNCTION_REGISTER*`) to eliminate the step to add and call an extern function in the registerFunctions.cpp, it also makes incremental builds of a new function faster. [#38615](https://github.com/ClickHouse/ClickHouse/pull/38615) ([Li Yin](https://github.com/liyinsg)).
|
||||
* Docker: Now entrypoint.sh in docker image creates and executes chown for all folders it found in config for multidisk setup [#17717](https://github.com/ClickHouse/ClickHouse/issues/17717). [#39121](https://github.com/ClickHouse/ClickHouse/pull/39121) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Docker: Now entrypoint.sh in docker image creates and executes chown for all folders it finds in the config for multidisk setup [#17717](https://github.com/ClickHouse/ClickHouse/issues/17717). [#39121](https://github.com/ClickHouse/ClickHouse/pull/39121) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
|
||||
#### Bug Fix
|
||||
|
||||
* Fix possible segfault in `CapnProto` input format. This bug was found and send through ClickHouse bug-bounty [program](https://github.com/ClickHouse/ClickHouse/issues/38986) by *kiojj*. [#40241](https://github.com/ClickHouse/ClickHouse/pull/40241) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix a very rare case of incorrect behavior of array subscript operator. This closes [#28720](https://github.com/ClickHouse/ClickHouse/issues/28720). [#40185](https://github.com/ClickHouse/ClickHouse/pull/40185) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix possible segfault in `CapnProto` input format. This bug was found and sent in through the ClickHouse bug-bounty [program](https://github.com/ClickHouse/ClickHouse/issues/38986) by *kiojj*. [#40241](https://github.com/ClickHouse/ClickHouse/pull/40241) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix a very rare case of incorrect behavior of the array subscript operator. This closes [#28720](https://github.com/ClickHouse/ClickHouse/issues/28720). [#40185](https://github.com/ClickHouse/ClickHouse/pull/40185) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix insufficient argument check for encryption functions (found by query fuzzer). This closes [#39987](https://github.com/ClickHouse/ClickHouse/issues/39987). [#40194](https://github.com/ClickHouse/ClickHouse/pull/40194) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix the case when the order of columns can be incorrect if the `IN` operator is used with a table with `ENGINE = Set` containing multiple columns. This fixes [#13014](https://github.com/ClickHouse/ClickHouse/issues/13014). [#40225](https://github.com/ClickHouse/ClickHouse/pull/40225) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix seeking while reading from encrypted disk. This PR fixes [#38381](https://github.com/ClickHouse/ClickHouse/issues/38381). [#39687](https://github.com/ClickHouse/ClickHouse/pull/39687) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
|
||||
@ -16,6 +16,6 @@ ClickHouse® is an open-source column-oriented database management system that a
|
||||
* [Contacts](https://clickhouse.com/company/contact) can help to get your questions answered if there are any.
|
||||
|
||||
## Upcoming events
|
||||
* [**v22.12 Release Webinar**](https://clickhouse.com/company/events/v22-12-release-webinar) Original creator, co-founder, and CTO of ClickHouse Alexey Milovidov will walk us through the highlights of the release, provide live demos, and share vision into what is coming in the roadmap.
|
||||
* [**v22.12 Release Webinar**](https://clickhouse.com/company/events/v22-12-release-webinar) 22.12 is the ClickHouse Christmas release. There are plenty of gifts (a new JOIN algorithm among them) and we adopted something from MongoDB. Original creator, co-founder, and CTO of ClickHouse Alexey Milovidov will walk us through the highlights of the release.
|
||||
* [**ClickHouse Meetup at the CHEQ office in Tel Aviv**](https://www.meetup.com/clickhouse-tel-aviv-user-group/events/289599423/) - Jan 16 - We are very excited to be holding our next in-person ClickHouse meetup at the CHEQ office in Tel Aviv! Hear from CHEQ, ServiceNow and Contentsquare, as well as a deep dive presentation from ClickHouse CTO Alexey Milovidov. Join us for a fun evening of talks, food and discussion!
|
||||
* **ClickHouse Meetup in Seattle* - Keep an eye on this space as we will be announcing a January meetup in Seattle soon!
|
||||
* [**ClickHouse Meetup at Microsoft Office in Seattle**](https://www.meetup.com/clickhouse-seattle-user-group/events/290310025/) - Jan 18 - Keep an eye on this space as we will be announcing speakers soon!
|
||||
|
||||
@ -127,23 +127,24 @@ EOL
|
||||
|
||||
function stop()
|
||||
{
|
||||
local max_tries=""
|
||||
if [ -n "$1" ]
|
||||
then
|
||||
max_tries="--max-tries $1"
|
||||
fi
|
||||
|
||||
local pid
|
||||
# Preserve the pid, since the server can hung after the PID will be deleted.
|
||||
pid="$(cat /var/run/clickhouse-server/clickhouse-server.pid)"
|
||||
|
||||
clickhouse stop $max_tries --do-not-kill && return
|
||||
|
||||
if [ -n "$1" ]
|
||||
then
|
||||
# temporarily disable it in BC check
|
||||
clickhouse stop --force
|
||||
return
|
||||
fi
|
||||
|
||||
# We failed to stop the server with SIGTERM. Maybe it hang, let's collect stacktraces.
|
||||
kill -TERM "$(pidof gdb)" ||:
|
||||
sleep 5
|
||||
echo "thread apply all backtrace (on stop)" >> /test_output/gdb.log
|
||||
gdb -batch -ex 'thread apply all backtrace' -p "$pid" | ts '%Y-%m-%d %H:%M:%S' >> /test_output/gdb.log
|
||||
timeout 30m gdb -batch -ex 'thread apply all backtrace' -p "$pid" | ts '%Y-%m-%d %H:%M:%S' >> /test_output/gdb.log
|
||||
clickhouse stop --force
|
||||
}
|
||||
|
||||
@ -431,7 +432,7 @@ else
|
||||
|
||||
clickhouse-client --query="SELECT 'Tables count:', count() FROM system.tables"
|
||||
|
||||
stop 180
|
||||
stop 1
|
||||
mv /var/log/clickhouse-server/clickhouse-server.log /var/log/clickhouse-server/clickhouse-server.backward.stress.log
|
||||
|
||||
# Start new server
|
||||
|
||||
@ -34,7 +34,7 @@ For a description of request parameters, see [request description](../../../sql-
|
||||
`columns` - a tuple with the names of columns where values will be summarized. Optional parameter.
|
||||
The columns must be of a numeric type and must not be in the primary key.
|
||||
|
||||
If `columns` not specified, ClickHouse summarizes the values in all columns with a numeric data type that are not in the primary key.
|
||||
If `columns` is not specified, ClickHouse summarizes the values in all columns with a numeric data type that are not in the primary key.
|
||||
|
||||
### Query clauses
|
||||
|
||||
|
||||
@ -92,7 +92,7 @@ Code: 452, e.displayText() = DB::Exception: Setting force_index_by_date should n
|
||||
**Note:** the `default` profile has special handling: all the constraints defined for the `default` profile become the default constraints, so they restrict all the users until they’re overridden explicitly for these users.
|
||||
|
||||
## Constraints on Merge Tree Settings
|
||||
It is possible to set constraints for [merge tree settings](merge-tree-settings.md). There constraints are applied when table with merge tree engine is created or its storage settings are altered. Name of merge tree setting must be prepended by `merge_tree_` prefix when referenced in `<constraint>` section.
|
||||
It is possible to set constraints for [merge tree settings](merge-tree-settings.md). These constraints are applied when table with merge tree engine is created or its storage settings are altered. Name of merge tree setting must be prepended by `merge_tree_` prefix when referenced in `<constraints>` section.
|
||||
|
||||
**Example:** Forbid to create new tables with explicitly specified `storage_policy`
|
||||
|
||||
|
||||
@ -13,6 +13,7 @@ Columns:
|
||||
- `metadata_path` ([String](../../sql-reference/data-types/enum.md)) — Metadata path.
|
||||
- `uuid` ([UUID](../../sql-reference/data-types/uuid.md)) — Database UUID.
|
||||
- `comment` ([String](../../sql-reference/data-types/enum.md)) — Database comment.
|
||||
- `engine_full` ([String](../../sql-reference/data-types/enum.md)) — Parameters of the database engine.
|
||||
|
||||
The `name` column from this system table is used for implementing the `SHOW DATABASES` query.
|
||||
|
||||
@ -31,10 +32,12 @@ SELECT * FROM system.databases;
|
||||
```
|
||||
|
||||
``` text
|
||||
┌─name───────────────┬─engine─┬─data_path──────────────────┬─metadata_path───────────────────────────────────────────────────────┬─uuid─────────────────────────────────┬─comment─┐
|
||||
│ INFORMATION_SCHEMA │ Memory │ /var/lib/clickhouse/ │ │ 00000000-0000-0000-0000-000000000000 │ │
|
||||
│ default │ Atomic │ /var/lib/clickhouse/store/ │ /var/lib/clickhouse/store/d31/d317b4bd-3595-4386-81ee-c2334694128a/ │ 24363899-31d7-42a0-a436-389931d752a0 │ │
|
||||
│ information_schema │ Memory │ /var/lib/clickhouse/ │ │ 00000000-0000-0000-0000-000000000000 │ │
|
||||
│ system │ Atomic │ /var/lib/clickhouse/store/ │ /var/lib/clickhouse/store/1d1/1d1c869d-e465-4b1b-a51f-be033436ebf9/ │ 03e9f3d1-cc88-4a49-83e9-f3d1cc881a49 │ │
|
||||
└────────────────────┴────────┴────────────────────────────┴─────────────────────────────────────────────────────────────────────┴──────────────────────────────────────┴─────────┘
|
||||
┌─name────────────────┬─engine─────┬─data_path────────────────────┬─metadata_path─────────────────────────────────────────────────────────┬─uuid─────────────────────────────────┬─engine_full────────────────────────────────────────────┬─comment─┐
|
||||
│ INFORMATION_SCHEMA │ Memory │ /data/clickhouse_data/ │ │ 00000000-0000-0000-0000-000000000000 │ Memory │ │
|
||||
│ default │ Atomic │ /data/clickhouse_data/store/ │ /data/clickhouse_data/store/f97/f97a3ceb-2e8a-4912-a043-c536e826a4d4/ │ f97a3ceb-2e8a-4912-a043-c536e826a4d4 │ Atomic │ │
|
||||
│ information_schema │ Memory │ /data/clickhouse_data/ │ │ 00000000-0000-0000-0000-000000000000 │ Memory │ │
|
||||
│ replicated_database │ Replicated │ /data/clickhouse_data/store/ │ /data/clickhouse_data/store/da8/da85bb71-102b-4f69-9aad-f8d6c403905e/ │ da85bb71-102b-4f69-9aad-f8d6c403905e │ Replicated('some/path/database', 'shard1', 'replica1') │ │
|
||||
│ system │ Atomic │ /data/clickhouse_data/store/ │ /data/clickhouse_data/store/b57/b5770419-ac7a-4b67-8229-524122024076/ │ b5770419-ac7a-4b67-8229-524122024076 │ Atomic │ │
|
||||
└─────────────────────┴────────────┴──────────────────────────────┴───────────────────────────────────────────────────────────────────────┴──────────────────────────────────────┴────────────────────────────────────────────────────────┴─────────┘
|
||||
|
||||
```
|
||||
|
||||
@ -410,35 +410,35 @@ Converts a date with time to a certain fixed date, while preserving the time.
|
||||
|
||||
## toRelativeYearNum
|
||||
|
||||
Converts a date or date with time to the number of the year, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the year, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeQuarterNum
|
||||
|
||||
Converts a date or date with time to the number of the quarter, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the quarter, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeMonthNum
|
||||
|
||||
Converts a date or date with time to the number of the month, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the month, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeWeekNum
|
||||
|
||||
Converts a date or date with time to the number of the week, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the week, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeDayNum
|
||||
|
||||
Converts a date or date with time to the number of the day, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the day, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeHourNum
|
||||
|
||||
Converts a date or date with time to the number of the hour, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the hour, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeMinuteNum
|
||||
|
||||
Converts a date or date with time to the number of the minute, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the minute, starting from a certain fixed point in the past.
|
||||
|
||||
## toRelativeSecondNum
|
||||
|
||||
Converts a date or date with time to the number of the second, starting from a certain fixed point in the past.
|
||||
Converts a date with time or date to the number of the second, starting from a certain fixed point in the past.
|
||||
|
||||
## toISOYear
|
||||
|
||||
@ -517,154 +517,6 @@ SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(d
|
||||
└────────────┴───────────┴───────────┴───────────┘
|
||||
```
|
||||
|
||||
## age
|
||||
|
||||
Returns the `unit` component of the difference between `startdate` and `enddate`. The difference is calculated using a precision of 1 second.
|
||||
E.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for `day` unit, 0 months for `month` unit, 0 years for `year` unit.
|
||||
|
||||
|
||||
**Syntax**
|
||||
|
||||
``` sql
|
||||
age('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `unit` — The type of interval for result. [String](../../sql-reference/data-types/string.md).
|
||||
Possible values:
|
||||
|
||||
- `second` (possible abbreviations: `ss`, `s`)
|
||||
- `minute` (possible abbreviations: `mi`, `n`)
|
||||
- `hour` (possible abbreviations: `hh`, `h`)
|
||||
- `day` (possible abbreviations: `dd`, `d`)
|
||||
- `week` (possible abbreviations: `wk`, `ww`)
|
||||
- `month` (possible abbreviations: `mm`, `m`)
|
||||
- `quarter` (possible abbreviations: `qq`, `q`)
|
||||
- `year` (possible abbreviations: `yyyy`, `yy`)
|
||||
|
||||
- `startdate` — The first time value to subtract (the subtrahend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — The second time value to subtract from (the minuend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (optional). If specified, it is applied to both `startdate` and `enddate`. If not specified, timezones of `startdate` and `enddate` are used. If they are not the same, the result is unspecified. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Difference between `enddate` and `startdate` expressed in `unit`.
|
||||
|
||||
Type: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌─age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 24 │
|
||||
└───────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
toDate('2022-01-01') AS e,
|
||||
toDate('2021-12-29') AS s,
|
||||
age('day', s, e) AS day_age,
|
||||
age('month', s, e) AS month__age,
|
||||
age('year', s, e) AS year_age;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌──────────e─┬──────────s─┬─day_age─┬─month__age─┬─year_age─┐
|
||||
│ 2022-01-01 │ 2021-12-29 │ 3 │ 0 │ 0 │
|
||||
└────────────┴────────────┴─────────┴────────────┴──────────┘
|
||||
```
|
||||
|
||||
|
||||
## date\_diff
|
||||
|
||||
Returns the count of the specified `unit` boundaries crossed between the `startdate` and `enddate`.
|
||||
The difference is calculated using relative units, e.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for day unit (see [toRelativeDayNum](#torelativedaynum)), 1 month for month unit (see [toRelativeMonthNum](#torelativemonthnum)), 1 year for year unit (see [toRelativeYearNum](#torelativeyearnum)).
|
||||
|
||||
**Syntax**
|
||||
|
||||
``` sql
|
||||
date_diff('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
Aliases: `dateDiff`, `DATE_DIFF`.
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `unit` — The type of interval for result. [String](../../sql-reference/data-types/string.md).
|
||||
Possible values:
|
||||
|
||||
- `second` (possible abbreviations: `ss`, `s`)
|
||||
- `minute` (possible abbreviations: `mi`, `n`)
|
||||
- `hour` (possible abbreviations: `hh`, `h`)
|
||||
- `day` (possible abbreviations: `dd`, `d`)
|
||||
- `week` (possible abbreviations: `wk`, `ww`)
|
||||
- `month` (possible abbreviations: `mm`, `m`)
|
||||
- `quarter` (possible abbreviations: `qq`, `q`)
|
||||
- `year` (possible abbreviations: `yyyy`, `yy`)
|
||||
|
||||
- `startdate` — The first time value to subtract (the subtrahend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — The second time value to subtract from (the minuend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (optional). If specified, it is applied to both `startdate` and `enddate`. If not specified, timezones of `startdate` and `enddate` are used. If they are not the same, the result is unspecified. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Difference between `enddate` and `startdate` expressed in `unit`.
|
||||
|
||||
Type: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 25 │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
toDate('2022-01-01') AS e,
|
||||
toDate('2021-12-29') AS s,
|
||||
dateDiff('day', s, e) AS day_diff,
|
||||
dateDiff('month', s, e) AS month__diff,
|
||||
dateDiff('year', s, e) AS year_diff;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌──────────e─┬──────────s─┬─day_diff─┬─month__diff─┬─year_diff─┐
|
||||
│ 2022-01-01 │ 2021-12-29 │ 3 │ 1 │ 1 │
|
||||
└────────────┴────────────┴──────────┴─────────────┴───────────┘
|
||||
```
|
||||
|
||||
## date\_trunc
|
||||
|
||||
Truncates date and time data to the specified part of date.
|
||||
@ -785,6 +637,80 @@ Result:
|
||||
└───────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## date\_diff
|
||||
|
||||
Returns the difference between two dates or dates with time values.
|
||||
The difference is calculated using relative units, e.g. the difference between `2022-01-01` and `2021-12-29` is 3 days for day unit (see [toRelativeDayNum](#torelativedaynum)), 1 month for month unit (see [toRelativeMonthNum](#torelativemonthnum)), 1 year for year unit (see [toRelativeYearNum](#torelativeyearnum)).
|
||||
|
||||
**Syntax**
|
||||
|
||||
``` sql
|
||||
date_diff('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
Aliases: `dateDiff`, `DATE_DIFF`.
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `unit` — The type of interval for result. [String](../../sql-reference/data-types/string.md).
|
||||
Possible values:
|
||||
|
||||
- `second`
|
||||
- `minute`
|
||||
- `hour`
|
||||
- `day`
|
||||
- `week`
|
||||
- `month`
|
||||
- `quarter`
|
||||
- `year`
|
||||
|
||||
- `startdate` — The first time value to subtract (the subtrahend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — The second time value to subtract from (the minuend). [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (optional). If specified, it is applied to both `startdate` and `enddate`. If not specified, timezones of `startdate` and `enddate` are used. If they are not the same, the result is unspecified. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
Difference between `enddate` and `startdate` expressed in `unit`.
|
||||
|
||||
Type: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 25 │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
toDate('2022-01-01') AS e,
|
||||
toDate('2021-12-29') AS s,
|
||||
dateDiff('day', s, e) AS day_diff,
|
||||
dateDiff('month', s, e) AS month__diff,
|
||||
dateDiff('year', s, e) AS year_diff;
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌──────────e─┬──────────s─┬─day_diff─┬─month__diff─┬─year_diff─┐
|
||||
│ 2022-01-01 │ 2021-12-29 │ 3 │ 1 │ 1 │
|
||||
└────────────┴────────────┴──────────┴─────────────┴───────────┘
|
||||
```
|
||||
|
||||
## date\_sub
|
||||
|
||||
Subtracts the time interval or date interval from the provided date or date with time.
|
||||
|
||||
@ -1159,4 +1159,40 @@ If s is empty, the result is 0. If the first character is not an ASCII character
|
||||
|
||||
|
||||
|
||||
## concatWithSeparator
|
||||
|
||||
Returns the concatenation strings separated by string separator. If any of the argument values is `NULL`, the function returns `NULL`.
|
||||
|
||||
**Syntax**
|
||||
|
||||
``` sql
|
||||
concatWithSeparator(sep, expr1, expr2, expr3...)
|
||||
```
|
||||
|
||||
**Arguments**
|
||||
- sep — separator. Const [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md).
|
||||
- exprN — expression to be concatenated. [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md).
|
||||
|
||||
**Returned values**
|
||||
- The concatenated String.
|
||||
|
||||
**Example**
|
||||
|
||||
Query:
|
||||
|
||||
``` sql
|
||||
SELECT concatWithSeparator('a', '1', '2', '3', '4')
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
``` text
|
||||
┌─concatWithSeparator('a', '1', '2', '3', '4')─┐
|
||||
│ 1a2a3a4 │
|
||||
└───────────────────────────────────┘
|
||||
```
|
||||
|
||||
## concatWithSeparatorAssumeInjective
|
||||
Same as concatWithSeparator, the difference is that you need to ensure that concatWithSeparator(sep, expr1, expr2, expr3...) → result is injective, it will be used for optimization of GROUP BY.
|
||||
|
||||
The function is named “injective” if it always returns different result for different values of arguments. In other words: different arguments never yield identical result.
|
||||
|
||||
@ -77,8 +77,9 @@ Numeric literal tries to be parsed:
|
||||
|
||||
Literal value has the smallest type that the value fits in.
|
||||
For example, 1 is parsed as `UInt8`, but 256 is parsed as `UInt16`. For more information, see [Data types](../sql-reference/data-types/index.md).
|
||||
Underscores `_` inside numeric literals are ignored and can be used for better readability.
|
||||
|
||||
Examples: `1`, `18446744073709551615`, `0xDEADBEEF`, `01`, `0.1`, `1e100`, `-1e-100`, `inf`, `nan`.
|
||||
Examples: `1`, `10_000_000`, `0xffff_ffff`, `18446744073709551615`, `0xDEADBEEF`, `01`, `0.1`, `1e100`, `-1e-100`, `inf`, `nan`.
|
||||
|
||||
### String
|
||||
|
||||
|
||||
@ -424,23 +424,23 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d
|
||||
|
||||
## toRelativeYearNum {#torelativeyearnum}
|
||||
|
||||
Переводит дату или дату-с-временем в номер года, начиная с некоторого фиксированного момента в прошлом.
|
||||
Переводит дату-с-временем или дату в номер года, начиная с некоторого фиксированного момента в прошлом.
|
||||
|
||||
## toRelativeQuarterNum {#torelativequarternum}
|
||||
|
||||
Переводит дату или дату-с-временем в номер квартала, начиная с некоторого фиксированного момента в прошлом.
|
||||
Переводит дату-с-временем или дату в номер квартала, начиная с некоторого фиксированного момента в прошлом.
|
||||
|
||||
## toRelativeMonthNum {#torelativemonthnum}
|
||||
|
||||
Переводит дату или дату-с-временем в номер месяца, начиная с некоторого фиксированного момента в прошлом.
|
||||
Переводит дату-с-временем или дату в номер месяца, начиная с некоторого фиксированного момента в прошлом.
|
||||
|
||||
## toRelativeWeekNum {#torelativeweeknum}
|
||||
|
||||
Переводит дату или дату-с-временем в номер недели, начиная с некоторого фиксированного момента в прошлом.
|
||||
Переводит дату-с-временем или дату в номер недели, начиная с некоторого фиксированного момента в прошлом.
|
||||
|
||||
## toRelativeDayNum {#torelativedaynum}
|
||||
|
||||
Переводит дату или дату-с-временем в номер дня, начиная с некоторого фиксированного момента в прошлом.
|
||||
Переводит дату-с-временем или дату в номер дня, начиная с некоторого фиксированного момента в прошлом.
|
||||
|
||||
## toRelativeHourNum {#torelativehournum}
|
||||
|
||||
@ -456,7 +456,7 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d
|
||||
|
||||
## toISOYear {#toisoyear}
|
||||
|
||||
Переводит дату или дату-с-временем в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого января.
|
||||
Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого января.
|
||||
|
||||
**Пример**
|
||||
|
||||
@ -479,7 +479,7 @@ SELECT
|
||||
|
||||
## toISOWeek {#toisoweek}
|
||||
|
||||
Переводит дату или дату-с-временем в число типа UInt8, содержащее номер ISO недели.
|
||||
Переводит дату-с-временем или дату в число типа UInt8, содержащее номер ISO недели.
|
||||
Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года - это неделя с четырьмя или более днями в этом году.
|
||||
|
||||
1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это последняя неделя 2016 года.
|
||||
@ -503,7 +503,7 @@ SELECT
|
||||
```
|
||||
|
||||
## toWeek(date\[, mode\]\[, timezone\]) {#toweek}
|
||||
Переводит дату или дату-с-временем в число UInt8, содержащее номер недели. Второй аргументам mode задает режим, начинается ли неделя с воскресенья или с понедельника и должно ли возвращаемое значение находиться в диапазоне от 0 до 53 или от 1 до 53. Если аргумент mode опущен, то используется режим 0.
|
||||
Переводит дату-с-временем или дату в число UInt8, содержащее номер недели. Второй аргументам mode задает режим, начинается ли неделя с воскресенья или с понедельника и должно ли возвращаемое значение находиться в диапазоне от 0 до 53 или от 1 до 53. Если аргумент mode опущен, то используется режим 0.
|
||||
|
||||
`toISOWeek() ` эквивалентно `toWeek(date,3)`.
|
||||
|
||||
@ -569,132 +569,6 @@ SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(d
|
||||
└────────────┴───────────┴───────────┴───────────┘
|
||||
```
|
||||
|
||||
## age
|
||||
|
||||
Вычисляет компонент `unit` разницы между `startdate` и `enddate`. Разница вычисляется с точностью в 1 секунду.
|
||||
Например, разница между `2021-12-29` и `2022-01-01` 3 дня для единицы `day`, 0 месяцев для единицы `month`, 0 лет для единицы `year`.
|
||||
|
||||
**Синтаксис**
|
||||
|
||||
``` sql
|
||||
age('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
**Аргументы**
|
||||
|
||||
- `unit` — единица измерения времени, в которой будет выражено возвращаемое значение функции. [String](../../sql-reference/data-types/string.md).
|
||||
Возможные значения:
|
||||
|
||||
- `second` (возможные сокращения: `ss`, `s`)
|
||||
- `minute` (возможные сокращения: `mi`, `n`)
|
||||
- `hour` (возможные сокращения: `hh`, `h`)
|
||||
- `day` (возможные сокращения: `dd`, `d`)
|
||||
- `week` (возможные сокращения: `wk`, `ww`)
|
||||
- `month` (возможные сокращения: `mm`, `m`)
|
||||
- `quarter` (возможные сокращения: `qq`, `q`)
|
||||
- `year` (возможные сокращения: `yyyy`, `yy`)
|
||||
|
||||
- `startdate` — первая дата или дата со временем, которая вычитается из `enddate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — вторая дата или дата со временем, из которой вычитается `startdate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [часовой пояс](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (необязательно). Если этот аргумент указан, то он применяется как для `startdate`, так и для `enddate`. Если этот аргумент не указан, то используются часовые пояса аргументов `startdate` и `enddate`. Если часовые пояса аргументов `startdate` и `enddate` не совпадают, то результат не определен. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Возвращаемое значение**
|
||||
|
||||
Разница между `enddate` и `startdate`, выраженная в `unit`.
|
||||
|
||||
Тип: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Пример**
|
||||
|
||||
Запрос:
|
||||
|
||||
``` sql
|
||||
SELECT age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Результат:
|
||||
|
||||
``` text
|
||||
┌─age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 24 │
|
||||
└───────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Запрос:
|
||||
|
||||
``` sql
|
||||
SELECT
|
||||
toDate('2022-01-01') AS e,
|
||||
toDate('2021-12-29') AS s,
|
||||
age('day', s, e) AS day_age,
|
||||
age('month', s, e) AS month__age,
|
||||
age('year', s, e) AS year_age;
|
||||
```
|
||||
|
||||
Результат:
|
||||
|
||||
``` text
|
||||
┌──────────e─┬──────────s─┬─day_age─┬─month__age─┬─year_age─┐
|
||||
│ 2022-01-01 │ 2021-12-29 │ 3 │ 0 │ 0 │
|
||||
└────────────┴────────────┴─────────┴────────────┴──────────┘
|
||||
```
|
||||
|
||||
## date\_diff {#date_diff}
|
||||
|
||||
Вычисляет разницу указанных границ `unit` пересекаемых между `startdate` и `enddate`.
|
||||
|
||||
**Синтаксис**
|
||||
|
||||
``` sql
|
||||
date_diff('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
Синонимы: `dateDiff`, `DATE_DIFF`.
|
||||
|
||||
**Аргументы**
|
||||
|
||||
- `unit` — единица измерения времени, в которой будет выражено возвращаемое значение функции. [String](../../sql-reference/data-types/string.md).
|
||||
Возможные значения:
|
||||
|
||||
- `second` (возможные сокращения: `ss`, `s`)
|
||||
- `minute` (возможные сокращения: `mi`, `n`)
|
||||
- `hour` (возможные сокращения: `hh`, `h`)
|
||||
- `day` (возможные сокращения: `dd`, `d`)
|
||||
- `week` (возможные сокращения: `wk`, `ww`)
|
||||
- `month` (возможные сокращения: `mm`, `m`)
|
||||
- `quarter` (возможные сокращения: `qq`, `q`)
|
||||
- `year` (возможные сокращения: `yyyy`, `yy`)
|
||||
|
||||
- `startdate` — первая дата или дата со временем, которая вычитается из `enddate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — вторая дата или дата со временем, из которой вычитается `startdate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [часовой пояс](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (необязательно). Если этот аргумент указан, то он применяется как для `startdate`, так и для `enddate`. Если этот аргумент не указан, то используются часовые пояса аргументов `startdate` и `enddate`. Если часовые пояса аргументов `startdate` и `enddate` не совпадают, то результат не определен. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Возвращаемое значение**
|
||||
|
||||
Разница между `enddate` и `startdate`, выраженная в `unit`.
|
||||
|
||||
Тип: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Пример**
|
||||
|
||||
Запрос:
|
||||
|
||||
``` sql
|
||||
SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Результат:
|
||||
|
||||
``` text
|
||||
┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 25 │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## date_trunc {#date_trunc}
|
||||
|
||||
Отсекает от даты и времени части, меньшие чем указанная часть.
|
||||
@ -815,6 +689,60 @@ SELECT date_add(YEAR, 3, toDate('2018-01-01'));
|
||||
└───────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## date\_diff {#date_diff}
|
||||
|
||||
Вычисляет разницу между двумя значениями дат или дат со временем.
|
||||
|
||||
**Синтаксис**
|
||||
|
||||
``` sql
|
||||
date_diff('unit', startdate, enddate, [timezone])
|
||||
```
|
||||
|
||||
Синонимы: `dateDiff`, `DATE_DIFF`.
|
||||
|
||||
**Аргументы**
|
||||
|
||||
- `unit` — единица измерения времени, в которой будет выражено возвращаемое значение функции. [String](../../sql-reference/data-types/string.md).
|
||||
Возможные значения:
|
||||
|
||||
- `second`
|
||||
- `minute`
|
||||
- `hour`
|
||||
- `day`
|
||||
- `week`
|
||||
- `month`
|
||||
- `quarter`
|
||||
- `year`
|
||||
|
||||
- `startdate` — первая дата или дата со временем, которая вычитается из `enddate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `enddate` — вторая дата или дата со временем, из которой вычитается `startdate`. [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md).
|
||||
|
||||
- `timezone` — [часовой пояс](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (необязательно). Если этот аргумент указан, то он применяется как для `startdate`, так и для `enddate`. Если этот аргумент не указан, то используются часовые пояса аргументов `startdate` и `enddate`. Если часовые пояса аргументов `startdate` и `enddate` не совпадают, то результат не определен. [String](../../sql-reference/data-types/string.md).
|
||||
|
||||
**Возвращаемое значение**
|
||||
|
||||
Разница между `enddate` и `startdate`, выраженная в `unit`.
|
||||
|
||||
Тип: [Int](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Пример**
|
||||
|
||||
Запрос:
|
||||
|
||||
``` sql
|
||||
SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'));
|
||||
```
|
||||
|
||||
Результат:
|
||||
|
||||
``` text
|
||||
┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐
|
||||
│ 25 │
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## date\_sub {#date_sub}
|
||||
|
||||
Вычитает интервал времени или даты из указанной даты или даты со временем.
|
||||
|
||||
@ -16,6 +16,8 @@
|
||||
|
||||
#include <base/find_symbols.h>
|
||||
|
||||
#include <Access/AccessControl.h>
|
||||
|
||||
#include "config_version.h"
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/formatReadable.h>
|
||||
@ -258,6 +260,10 @@ try
|
||||
if (is_interactive && !config().has("no-warnings"))
|
||||
showWarnings();
|
||||
|
||||
/// Set user password complexity rules
|
||||
auto & access_control = global_context->getAccessControl();
|
||||
access_control.setPasswordComplexityRules(connection->getPasswordComplexityRules());
|
||||
|
||||
if (is_interactive && !delayed_interactive)
|
||||
{
|
||||
runInteractive();
|
||||
|
||||
@ -466,6 +466,30 @@
|
||||
<allow_no_password>1</allow_no_password>
|
||||
<allow_implicit_no_password>1</allow_implicit_no_password>
|
||||
|
||||
<!-- Complexity requirements for user passwords. -->
|
||||
<!-- <password_complexity>
|
||||
<rule>
|
||||
<pattern>.{12}</pattern>
|
||||
<message>be at least 12 characters long</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{N}</pattern>
|
||||
<message>contain at least 1 numeric character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{Ll}</pattern>
|
||||
<message>contain at least 1 lowercase character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{Lu}</pattern>
|
||||
<message>contain at least 1 uppercase character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>[^\p{L}\p{N}]</pattern>
|
||||
<message>contain at least 1 special character</message>
|
||||
</rule>
|
||||
</password_complexity> -->
|
||||
|
||||
<!-- Policy from the <storage_configuration> for the temporary files.
|
||||
If not set <tmp_path> is used, otherwise <tmp_path> is ignored.
|
||||
|
||||
|
||||
@ -27,6 +27,7 @@
|
||||
#include <boost/algorithm/string/join.hpp>
|
||||
#include <boost/algorithm/string/split.hpp>
|
||||
#include <boost/algorithm/string/trim.hpp>
|
||||
#include <re2/re2.h>
|
||||
#include <filesystem>
|
||||
#include <mutex>
|
||||
|
||||
@ -38,6 +39,8 @@ namespace ErrorCodes
|
||||
extern const int UNKNOWN_ELEMENT_IN_CONFIG;
|
||||
extern const int UNKNOWN_SETTING;
|
||||
extern const int AUTHENTICATION_FAILED;
|
||||
extern const int CANNOT_COMPILE_REGEXP;
|
||||
extern const int BAD_ARGUMENTS;
|
||||
}
|
||||
|
||||
namespace
|
||||
@ -140,6 +143,109 @@ private:
|
||||
};
|
||||
|
||||
|
||||
class AccessControl::PasswordComplexityRules
|
||||
{
|
||||
public:
|
||||
void setPasswordComplexityRulesFromConfig(const Poco::Util::AbstractConfiguration & config_)
|
||||
{
|
||||
std::lock_guard lock{mutex};
|
||||
|
||||
rules.clear();
|
||||
|
||||
if (config_.has("password_complexity"))
|
||||
{
|
||||
Poco::Util::AbstractConfiguration::Keys password_complexity;
|
||||
config_.keys("password_complexity", password_complexity);
|
||||
|
||||
for (const auto & key : password_complexity)
|
||||
{
|
||||
if (key == "rule" || key.starts_with("rule["))
|
||||
{
|
||||
String pattern(config_.getString("password_complexity." + key + ".pattern"));
|
||||
String message(config_.getString("password_complexity." + key + ".message"));
|
||||
|
||||
auto matcher = std::make_unique<RE2>(pattern, RE2::Quiet);
|
||||
if (!matcher->ok())
|
||||
throw Exception(ErrorCodes::CANNOT_COMPILE_REGEXP,
|
||||
"Password complexity pattern {} cannot be compiled: {}",
|
||||
pattern, matcher->error());
|
||||
|
||||
rules.push_back({std::move(matcher), std::move(pattern), std::move(message)});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void setPasswordComplexityRules(const std::vector<std::pair<String, String>> & rules_)
|
||||
{
|
||||
Rules new_rules;
|
||||
|
||||
for (const auto & [original_pattern, exception_message] : rules_)
|
||||
{
|
||||
auto matcher = std::make_unique<RE2>(original_pattern, RE2::Quiet);
|
||||
if (!matcher->ok())
|
||||
throw Exception(ErrorCodes::CANNOT_COMPILE_REGEXP,
|
||||
"Password complexity pattern {} cannot be compiled: {}",
|
||||
original_pattern, matcher->error());
|
||||
|
||||
new_rules.push_back({std::move(matcher), original_pattern, exception_message});
|
||||
}
|
||||
|
||||
std::lock_guard lock{mutex};
|
||||
rules = std::move(new_rules);
|
||||
}
|
||||
|
||||
void checkPasswordComplexityRules(const String & password_) const
|
||||
{
|
||||
String exception_text;
|
||||
bool failed = false;
|
||||
|
||||
std::lock_guard lock{mutex};
|
||||
for (const auto & rule : rules)
|
||||
{
|
||||
if (!RE2::PartialMatch(password_, *rule.matcher))
|
||||
{
|
||||
failed = true;
|
||||
|
||||
if (!exception_text.empty())
|
||||
exception_text += ", ";
|
||||
|
||||
exception_text += rule.exception_message;
|
||||
}
|
||||
}
|
||||
|
||||
if (failed)
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid password. The password should: {}", exception_text);
|
||||
}
|
||||
|
||||
std::vector<std::pair<String, String>> getPasswordComplexityRules()
|
||||
{
|
||||
std::vector<std::pair<String, String>> result;
|
||||
|
||||
std::lock_guard lock{mutex};
|
||||
result.reserve(rules.size());
|
||||
|
||||
for (const auto & rule : rules)
|
||||
result.push_back({rule.original_pattern, rule.exception_message});
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private:
|
||||
struct Rule
|
||||
{
|
||||
std::unique_ptr<RE2> matcher;
|
||||
String original_pattern;
|
||||
String exception_message;
|
||||
};
|
||||
|
||||
using Rules = std::vector<Rule>;
|
||||
|
||||
Rules rules TSA_GUARDED_BY(mutex);
|
||||
mutable std::mutex mutex;
|
||||
};
|
||||
|
||||
|
||||
AccessControl::AccessControl()
|
||||
: MultipleAccessStorage("user directories"),
|
||||
context_access_cache(std::make_unique<ContextAccessCache>(*this)),
|
||||
@ -149,7 +255,8 @@ AccessControl::AccessControl()
|
||||
settings_profiles_cache(std::make_unique<SettingsProfilesCache>(*this)),
|
||||
external_authenticators(std::make_unique<ExternalAuthenticators>()),
|
||||
custom_settings_prefixes(std::make_unique<CustomSettingsPrefixes>()),
|
||||
changes_notifier(std::make_unique<AccessChangesNotifier>())
|
||||
changes_notifier(std::make_unique<AccessChangesNotifier>()),
|
||||
password_rules(std::make_unique<PasswordComplexityRules>())
|
||||
{
|
||||
}
|
||||
|
||||
@ -166,6 +273,7 @@ void AccessControl::setUpFromMainConfig(const Poco::Util::AbstractConfiguration
|
||||
setImplicitNoPasswordAllowed(config_.getBool("allow_implicit_no_password", true));
|
||||
setNoPasswordAllowed(config_.getBool("allow_no_password", true));
|
||||
setPlaintextPasswordAllowed(config_.getBool("allow_plaintext_password", true));
|
||||
setPasswordComplexityRulesFromConfig(config_);
|
||||
|
||||
/// Optional improvements in access control system.
|
||||
/// The default values are false because we need to be compatible with earlier access configurations
|
||||
@ -543,6 +651,26 @@ bool AccessControl::isPlaintextPasswordAllowed() const
|
||||
return allow_plaintext_password;
|
||||
}
|
||||
|
||||
void AccessControl::setPasswordComplexityRulesFromConfig(const Poco::Util::AbstractConfiguration & config_)
|
||||
{
|
||||
password_rules->setPasswordComplexityRulesFromConfig(config_);
|
||||
}
|
||||
|
||||
void AccessControl::setPasswordComplexityRules(const std::vector<std::pair<String, String>> & rules_)
|
||||
{
|
||||
password_rules->setPasswordComplexityRules(rules_);
|
||||
}
|
||||
|
||||
void AccessControl::checkPasswordComplexityRules(const String & password_) const
|
||||
{
|
||||
password_rules->checkPasswordComplexityRules(password_);
|
||||
}
|
||||
|
||||
std::vector<std::pair<String, String>> AccessControl::getPasswordComplexityRules() const
|
||||
{
|
||||
return password_rules->getPasswordComplexityRules();
|
||||
}
|
||||
|
||||
|
||||
std::shared_ptr<const ContextAccess> AccessControl::getContextAccess(
|
||||
const UUID & user_id,
|
||||
|
||||
@ -147,6 +147,13 @@ public:
|
||||
void setPlaintextPasswordAllowed(const bool allow_plaintext_password_);
|
||||
bool isPlaintextPasswordAllowed() const;
|
||||
|
||||
/// Check complexity requirements for plaintext passwords
|
||||
|
||||
void setPasswordComplexityRulesFromConfig(const Poco::Util::AbstractConfiguration & config_);
|
||||
void setPasswordComplexityRules(const std::vector<std::pair<String, String>> & rules_);
|
||||
void checkPasswordComplexityRules(const String & password_) const;
|
||||
std::vector<std::pair<String, String>> getPasswordComplexityRules() const;
|
||||
|
||||
/// Enables logic that users without permissive row policies can still read rows using a SELECT query.
|
||||
/// For example, if there two users A, B and a row policy is defined only for A, then
|
||||
/// if this setting is true the user B will see all rows, and if this setting is false the user B will see no rows.
|
||||
@ -212,6 +219,7 @@ public:
|
||||
private:
|
||||
class ContextAccessCache;
|
||||
class CustomSettingsPrefixes;
|
||||
class PasswordComplexityRules;
|
||||
|
||||
std::optional<UUID> insertImpl(const AccessEntityPtr & entity, bool replace_if_exists, bool throw_if_exists) override;
|
||||
bool removeImpl(const UUID & id, bool throw_if_not_exists) override;
|
||||
@ -225,6 +233,7 @@ private:
|
||||
std::unique_ptr<ExternalAuthenticators> external_authenticators;
|
||||
std::unique_ptr<CustomSettingsPrefixes> custom_settings_prefixes;
|
||||
std::unique_ptr<AccessChangesNotifier> changes_notifier;
|
||||
std::unique_ptr<PasswordComplexityRules> password_rules;
|
||||
std::atomic_bool allow_plaintext_password = true;
|
||||
std::atomic_bool allow_no_password = true;
|
||||
std::atomic_bool allow_implicit_no_password = true;
|
||||
|
||||
197
src/Analyzer/Passes/IfTransformStringsToEnumPass.cpp
Normal file
197
src/Analyzer/Passes/IfTransformStringsToEnumPass.cpp
Normal file
@ -0,0 +1,197 @@
|
||||
#include <Analyzer/Passes/IfTransformStringsToEnumPass.h>
|
||||
|
||||
#include <Analyzer/ConstantNode.h>
|
||||
#include <Analyzer/FunctionNode.h>
|
||||
#include <Analyzer/IQueryTreeNode.h>
|
||||
#include <Analyzer/InDepthQueryTreeVisitor.h>
|
||||
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypeEnum.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/IDataType.h>
|
||||
|
||||
#include <Functions/FunctionFactory.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
/// We place strings in ascending order here under the assumption it could speed up String to Enum conversion.
|
||||
template <typename EnumType>
|
||||
auto getDataEnumType(const std::set<std::string> & string_values)
|
||||
{
|
||||
using EnumValues = typename EnumType::Values;
|
||||
EnumValues enum_values;
|
||||
enum_values.reserve(string_values.size());
|
||||
|
||||
size_t number = 1;
|
||||
for (const auto & value : string_values)
|
||||
enum_values.emplace_back(value, number++);
|
||||
|
||||
return std::make_shared<EnumType>(std::move(enum_values));
|
||||
}
|
||||
|
||||
DataTypePtr getEnumType(const std::set<std::string> & string_values)
|
||||
{
|
||||
if (string_values.size() >= 255)
|
||||
return getDataEnumType<DataTypeEnum16>(string_values);
|
||||
else
|
||||
return getDataEnumType<DataTypeEnum8>(string_values);
|
||||
}
|
||||
|
||||
QueryTreeNodePtr createCastFunction(QueryTreeNodePtr from, DataTypePtr result_type, ContextPtr context)
|
||||
{
|
||||
auto enum_literal = std::make_shared<ConstantValue>(result_type->getName(), std::make_shared<DataTypeString>());
|
||||
auto enum_literal_node = std::make_shared<ConstantNode>(std::move(enum_literal));
|
||||
|
||||
auto cast_function = FunctionFactory::instance().get("_CAST", std::move(context));
|
||||
QueryTreeNodes arguments{std::move(from), std::move(enum_literal_node)};
|
||||
|
||||
auto function_node = std::make_shared<FunctionNode>("_CAST");
|
||||
function_node->resolveAsFunction(std::move(cast_function), std::move(result_type));
|
||||
function_node->getArguments().getNodes() = std::move(arguments);
|
||||
|
||||
return function_node;
|
||||
}
|
||||
|
||||
/// if(arg1, arg2, arg3) will be transformed to if(arg1, _CAST(arg2, Enum...), _CAST(arg3, Enum...))
|
||||
/// where Enum is generated based on the possible values stored in string_values
|
||||
void changeIfArguments(
|
||||
QueryTreeNodePtr & first, QueryTreeNodePtr & second, const std::set<std::string> & string_values, const ContextPtr & context)
|
||||
{
|
||||
auto result_type = getEnumType(string_values);
|
||||
|
||||
first = createCastFunction(first, result_type, context);
|
||||
second = createCastFunction(second, result_type, context);
|
||||
}
|
||||
|
||||
/// transform(value, array_from, array_to, default_value) will be transformed to transform(value, array_from, _CAST(array_to, Array(Enum...)), _CAST(default_value, Enum...))
|
||||
/// where Enum is generated based on the possible values stored in string_values
|
||||
void changeTransformArguments(
|
||||
QueryTreeNodePtr & array_to,
|
||||
QueryTreeNodePtr & default_value,
|
||||
const std::set<std::string> & string_values,
|
||||
const ContextPtr & context)
|
||||
{
|
||||
auto result_type = getEnumType(string_values);
|
||||
|
||||
array_to = createCastFunction(array_to, std::make_shared<DataTypeArray>(result_type), context);
|
||||
default_value = createCastFunction(default_value, std::move(result_type), context);
|
||||
}
|
||||
|
||||
void wrapIntoToString(FunctionNode & function_node, QueryTreeNodePtr arg, ContextPtr context)
|
||||
{
|
||||
assert(isString(function_node.getResultType()));
|
||||
|
||||
auto to_string_function = FunctionFactory::instance().get("toString", std::move(context));
|
||||
QueryTreeNodes arguments{std::move(arg)};
|
||||
|
||||
function_node.resolveAsFunction(std::move(to_string_function), std::make_shared<DataTypeString>());
|
||||
function_node.getArguments().getNodes() = std::move(arguments);
|
||||
}
|
||||
|
||||
class ConvertStringsToEnumVisitor : public InDepthQueryTreeVisitor<ConvertStringsToEnumVisitor>
|
||||
{
|
||||
public:
|
||||
explicit ConvertStringsToEnumVisitor(ContextPtr context_)
|
||||
: context(std::move(context_))
|
||||
{
|
||||
}
|
||||
|
||||
void visitImpl(QueryTreeNodePtr & node)
|
||||
{
|
||||
auto * function_node = node->as<FunctionNode>();
|
||||
|
||||
if (!function_node)
|
||||
return;
|
||||
|
||||
/// to preserve return type (String) of the current function_node, we wrap the newly
|
||||
/// generated function nodes into toString
|
||||
|
||||
std::string_view function_name = function_node->getFunctionName();
|
||||
if (function_name == "if")
|
||||
{
|
||||
if (function_node->getArguments().getNodes().size() != 3)
|
||||
return;
|
||||
|
||||
auto modified_if_node = function_node->clone();
|
||||
auto & argument_nodes = modified_if_node->as<FunctionNode>()->getArguments().getNodes();
|
||||
|
||||
const auto * first_literal = argument_nodes[1]->as<ConstantNode>();
|
||||
const auto * second_literal = argument_nodes[2]->as<ConstantNode>();
|
||||
|
||||
if (!first_literal || !second_literal)
|
||||
return;
|
||||
|
||||
if (!isString(first_literal->getResultType()) || !isString(second_literal->getResultType()))
|
||||
return;
|
||||
|
||||
std::set<std::string> string_values;
|
||||
string_values.insert(first_literal->getValue().get<std::string>());
|
||||
string_values.insert(second_literal->getValue().get<std::string>());
|
||||
|
||||
changeIfArguments(argument_nodes[1], argument_nodes[2], string_values, context);
|
||||
wrapIntoToString(*function_node, std::move(modified_if_node), context);
|
||||
return;
|
||||
}
|
||||
|
||||
if (function_name == "transform")
|
||||
{
|
||||
if (function_node->getArguments().getNodes().size() != 4)
|
||||
return;
|
||||
|
||||
auto modified_transform_node = function_node->clone();
|
||||
auto & argument_nodes = modified_transform_node->as<FunctionNode>()->getArguments().getNodes();
|
||||
|
||||
if (!isString(function_node->getResultType()))
|
||||
return;
|
||||
|
||||
const auto * literal_to = argument_nodes[2]->as<ConstantNode>();
|
||||
const auto * literal_default = argument_nodes[3]->as<ConstantNode>();
|
||||
|
||||
if (!literal_to || !literal_default)
|
||||
return;
|
||||
|
||||
if (!isArray(literal_to->getResultType()) || !isString(literal_default->getResultType()))
|
||||
return;
|
||||
|
||||
auto array_to = literal_to->getValue().get<Array>();
|
||||
|
||||
if (array_to.empty())
|
||||
return;
|
||||
|
||||
if (!std::all_of(
|
||||
array_to.begin(),
|
||||
array_to.end(),
|
||||
[](const auto & field) { return field.getType() == Field::Types::Which::String; }))
|
||||
return;
|
||||
|
||||
/// collect possible string values
|
||||
std::set<std::string> string_values;
|
||||
|
||||
for (const auto & value : array_to)
|
||||
string_values.insert(value.get<std::string>());
|
||||
|
||||
string_values.insert(literal_default->getValue().get<std::string>());
|
||||
|
||||
changeTransformArguments(argument_nodes[2], argument_nodes[3], string_values, context);
|
||||
wrapIntoToString(*function_node, std::move(modified_transform_node), context);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
private:
|
||||
ContextPtr context;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
void IfTransformStringsToEnumPass::run(QueryTreeNodePtr query, ContextPtr context)
|
||||
{
|
||||
ConvertStringsToEnumVisitor visitor(context);
|
||||
visitor.visit(query);
|
||||
}
|
||||
|
||||
}
|
||||
39
src/Analyzer/Passes/IfTransformStringsToEnumPass.h
Normal file
39
src/Analyzer/Passes/IfTransformStringsToEnumPass.h
Normal file
@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
#include <Analyzer/IQueryTreePass.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
/**
|
||||
* This pass replaces string-type arguments in If and Transform to enum.
|
||||
*
|
||||
* E.g.
|
||||
* -------------------------------
|
||||
* SELECT if(number > 5, 'a', 'b')
|
||||
* FROM system.numbers;
|
||||
*
|
||||
* will be transformed into
|
||||
*
|
||||
* SELECT if(number > 5, _CAST('a', 'Enum8(\'a\' = 1, \'b\' = 2)'), _CAST('b', 'Enum8(\'a\' = 1, \'b\' = 2)'))
|
||||
* FROM system.numbers;
|
||||
* -------------------------------
|
||||
* SELECT transform(number, [2, 4], ['a', 'b'], 'c') FROM system.numbers;
|
||||
*
|
||||
* will be transformed into
|
||||
*
|
||||
* SELECT transform(number, [2, 4], _CAST(['a', 'b'], 'Array(Enum8(\'a\' = 1, \'b\' = 2, \'c\' = 3)'), _CAST('c', 'Enum8(\'a\' = 1, \'b\' = 2, \'c\' = 3)'))
|
||||
* FROM system.numbers;
|
||||
* -------------------------------
|
||||
*/
|
||||
class IfTransformStringsToEnumPass final : public IQueryTreePass
|
||||
{
|
||||
public:
|
||||
String getName() override { return "IfTransformStringsToEnumPass"; }
|
||||
|
||||
String getDescription() override { return "Replaces string-type arguments in If and Transform to enum"; }
|
||||
|
||||
void run(QueryTreeNodePtr query_tree_node, ContextPtr context) override;
|
||||
};
|
||||
|
||||
}
|
||||
@ -14,6 +14,7 @@
|
||||
#include <Analyzer/Passes/UniqInjectiveFunctionsEliminationPass.h>
|
||||
#include <Analyzer/Passes/OrderByLimitByDuplicateEliminationPass.h>
|
||||
#include <Analyzer/Passes/FuseFunctionsPass.h>
|
||||
#include <Analyzer/Passes/IfTransformStringsToEnumPass.h>
|
||||
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
@ -77,7 +78,6 @@ public:
|
||||
* TODO: Support setting optimize_duplicate_order_by_and_distinct.
|
||||
* TODO: Support setting optimize_redundant_functions_in_order_by.
|
||||
* TODO: Support setting optimize_monotonous_functions_in_order_by.
|
||||
* TODO: Support setting optimize_if_transform_strings_to_enum.
|
||||
* TODO: Support settings.optimize_or_like_chain.
|
||||
* TODO: Add optimizations based on function semantics. Example: SELECT * FROM test_table WHERE id != id. (id is not nullable column).
|
||||
*/
|
||||
@ -193,6 +193,9 @@ void addQueryTreePasses(QueryTreePassManager & manager)
|
||||
|
||||
if (settings.optimize_syntax_fuse_functions)
|
||||
manager.addPass(std::make_unique<FuseFunctionsPass>());
|
||||
|
||||
if (settings.optimize_if_transform_strings_to_enum)
|
||||
manager.addPass(std::make_unique<IfTransformStringsToEnumPass>());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -29,6 +29,7 @@ public:
|
||||
virtual UInt64 getFileSize(const String & file_name) = 0;
|
||||
virtual bool fileContentsEqual(const String & file_name, const String & expected_file_contents) = 0;
|
||||
virtual std::unique_ptr<WriteBuffer> writeFile(const String & file_name) = 0;
|
||||
virtual void removeFile(const String & file_name) = 0;
|
||||
virtual void removeFiles(const Strings & file_names) = 0;
|
||||
virtual DataSourceDescription getDataSourceDescription() const = 0;
|
||||
virtual void copyFileThroughBuffer(std::unique_ptr<SeekableReadBuffer> && source, const String & file_name);
|
||||
|
||||
@ -75,6 +75,13 @@ std::unique_ptr<WriteBuffer> BackupWriterDisk::writeFile(const String & file_nam
|
||||
return disk->writeFile(file_path);
|
||||
}
|
||||
|
||||
void BackupWriterDisk::removeFile(const String & file_name)
|
||||
{
|
||||
disk->removeFileIfExists(path / file_name);
|
||||
if (disk->isDirectory(path) && disk->isDirectoryEmpty(path))
|
||||
disk->removeDirectory(path);
|
||||
}
|
||||
|
||||
void BackupWriterDisk::removeFiles(const Strings & file_names)
|
||||
{
|
||||
for (const auto & file_name : file_names)
|
||||
|
||||
@ -34,6 +34,7 @@ public:
|
||||
UInt64 getFileSize(const String & file_name) override;
|
||||
bool fileContentsEqual(const String & file_name, const String & expected_file_contents) override;
|
||||
std::unique_ptr<WriteBuffer> writeFile(const String & file_name) override;
|
||||
void removeFile(const String & file_name) override;
|
||||
void removeFiles(const Strings & file_names) override;
|
||||
DataSourceDescription getDataSourceDescription() const override;
|
||||
|
||||
|
||||
@ -72,6 +72,13 @@ std::unique_ptr<WriteBuffer> BackupWriterFile::writeFile(const String & file_nam
|
||||
return std::make_unique<WriteBufferFromFile>(file_path);
|
||||
}
|
||||
|
||||
void BackupWriterFile::removeFile(const String & file_name)
|
||||
{
|
||||
fs::remove(path / file_name);
|
||||
if (fs::is_directory(path) && fs::is_empty(path))
|
||||
fs::remove(path);
|
||||
}
|
||||
|
||||
void BackupWriterFile::removeFiles(const Strings & file_names)
|
||||
{
|
||||
for (const auto & file_name : file_names)
|
||||
|
||||
@ -31,6 +31,7 @@ public:
|
||||
UInt64 getFileSize(const String & file_name) override;
|
||||
bool fileContentsEqual(const String & file_name, const String & expected_file_contents) override;
|
||||
std::unique_ptr<WriteBuffer> writeFile(const String & file_name) override;
|
||||
void removeFile(const String & file_name) override;
|
||||
void removeFiles(const Strings & file_names) override;
|
||||
DataSourceDescription getDataSourceDescription() const override;
|
||||
bool supportNativeCopy(DataSourceDescription data_source_description) const override;
|
||||
|
||||
@ -372,7 +372,48 @@ std::unique_ptr<WriteBuffer> BackupWriterS3::writeFile(const String & file_name)
|
||||
threadPoolCallbackRunner<void>(IOThreadPool::get(), "BackupWriterS3"));
|
||||
}
|
||||
|
||||
void BackupWriterS3::removeFile(const String & file_name)
|
||||
{
|
||||
Aws::S3::Model::DeleteObjectRequest request;
|
||||
request.SetBucket(s3_uri.bucket);
|
||||
request.SetKey(fs::path(s3_uri.key) / file_name);
|
||||
auto outcome = client->DeleteObject(request);
|
||||
if (!outcome.IsSuccess())
|
||||
throw Exception(outcome.GetError().GetMessage(), ErrorCodes::S3_ERROR);
|
||||
}
|
||||
|
||||
void BackupWriterS3::removeFiles(const Strings & file_names)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!supports_batch_delete.has_value() || supports_batch_delete.value() == true)
|
||||
{
|
||||
removeFilesBatch(file_names);
|
||||
supports_batch_delete = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
for (const auto & file_name : file_names)
|
||||
removeFile(file_name);
|
||||
}
|
||||
}
|
||||
catch (const Exception &)
|
||||
{
|
||||
if (!supports_batch_delete.has_value())
|
||||
{
|
||||
supports_batch_delete = false;
|
||||
LOG_TRACE(log, "DeleteObjects is not supported. Retrying with plain DeleteObject.");
|
||||
|
||||
for (const auto & file_name : file_names)
|
||||
removeFile(file_name);
|
||||
}
|
||||
else
|
||||
throw;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
void BackupWriterS3::removeFilesBatch(const Strings & file_names)
|
||||
{
|
||||
/// One call of DeleteObjects() cannot remove more than 1000 keys.
|
||||
size_t chunk_size_limit = 1000;
|
||||
|
||||
@ -54,6 +54,7 @@ public:
|
||||
UInt64 getFileSize(const String & file_name) override;
|
||||
bool fileContentsEqual(const String & file_name, const String & expected_file_contents) override;
|
||||
std::unique_ptr<WriteBuffer> writeFile(const String & file_name) override;
|
||||
void removeFile(const String & file_name) override;
|
||||
void removeFiles(const Strings & file_names) override;
|
||||
|
||||
DataSourceDescription getDataSourceDescription() const override;
|
||||
@ -79,11 +80,14 @@ private:
|
||||
const Aws::S3::Model::HeadObjectResult & head,
|
||||
const std::optional<ObjectAttributes> & metadata = std::nullopt) const;
|
||||
|
||||
void removeFilesBatch(const Strings & file_names);
|
||||
|
||||
S3::URI s3_uri;
|
||||
std::shared_ptr<Aws::S3::S3Client> client;
|
||||
ReadSettings read_settings;
|
||||
S3Settings::RequestSettings request_settings;
|
||||
Poco::Logger * log;
|
||||
std::optional<bool> supports_batch_delete;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -506,7 +506,7 @@ void BackupImpl::removeLockFile()
|
||||
return; /// Internal backup must not remove the lock file (it's still used by the initiator).
|
||||
|
||||
if (checkLockFile(false))
|
||||
writer->removeFiles({lock_file_name});
|
||||
writer->removeFile(lock_file_name);
|
||||
}
|
||||
|
||||
Strings BackupImpl::listFiles(const String & directory, bool recursive) const
|
||||
|
||||
@ -181,6 +181,7 @@ OperationID BackupsWorker::startMakingBackup(const ASTPtr & query, const Context
|
||||
/// For ON CLUSTER queries we will need to change some settings.
|
||||
/// For ASYNC queries we have to clone the context anyway.
|
||||
context_in_use = mutable_context = Context::createCopy(context);
|
||||
mutable_context->makeQueryContext();
|
||||
}
|
||||
|
||||
if (backup_settings.async)
|
||||
@ -400,6 +401,7 @@ OperationID BackupsWorker::startRestoring(const ASTPtr & query, ContextMutablePt
|
||||
/// For ON CLUSTER queries we will need to change some settings.
|
||||
/// For ASYNC queries we have to clone the context anyway.
|
||||
context_in_use = Context::createCopy(context);
|
||||
context_in_use->makeQueryContext();
|
||||
}
|
||||
|
||||
if (restore_settings.async)
|
||||
|
||||
@ -346,7 +346,7 @@ void RestorerFromBackup::findTableInBackup(const QualifiedTableName & table_name
|
||||
res_table_info.has_data = backup->hasFiles(data_path_in_backup);
|
||||
res_table_info.data_path_in_backup = data_path_in_backup;
|
||||
|
||||
tables_dependencies.addDependencies(table_name, getDependenciesFromCreateQuery(context->getGlobalContext(), table_name, create_table_query));
|
||||
tables_dependencies.addDependencies(table_name, getDependenciesFromCreateQuery(context, table_name, create_table_query));
|
||||
|
||||
if (partitions)
|
||||
{
|
||||
@ -674,6 +674,7 @@ void RestorerFromBackup::removeUnresolvedDependencies()
|
||||
void RestorerFromBackup::createTables()
|
||||
{
|
||||
/// We need to create tables considering their dependencies.
|
||||
tables_dependencies.log();
|
||||
auto tables_to_create = tables_dependencies.getTablesSortedByDependency();
|
||||
for (const auto & table_id : tables_to_create)
|
||||
{
|
||||
|
||||
@ -22,6 +22,7 @@
|
||||
#include <Core/Block.h>
|
||||
#include <Core/Protocol.h>
|
||||
#include <Formats/FormatFactory.h>
|
||||
#include <Access/AccessControl.h>
|
||||
|
||||
#include "config_version.h"
|
||||
|
||||
@ -43,6 +44,7 @@
|
||||
#include <Parsers/ASTInsertQuery.h>
|
||||
#include <Parsers/ASTCreateQuery.h>
|
||||
#include <Parsers/ASTCreateFunctionQuery.h>
|
||||
#include <Parsers/Access/ASTCreateUserQuery.h>
|
||||
#include <Parsers/ASTDropQuery.h>
|
||||
#include <Parsers/ASTSetQuery.h>
|
||||
#include <Parsers/ASTUseQuery.h>
|
||||
@ -1562,6 +1564,15 @@ void ClientBase::processParsedSingleQuery(const String & full_query, const Strin
|
||||
updateLoggerLevel(logs_level_field->safeGet<String>());
|
||||
}
|
||||
|
||||
if (const auto * create_user_query = parsed_query->as<ASTCreateUserQuery>())
|
||||
{
|
||||
if (!create_user_query->attach && create_user_query->temporary_password_for_checks)
|
||||
{
|
||||
global_context->getAccessControl().checkPasswordComplexityRules(create_user_query->temporary_password_for_checks.value());
|
||||
create_user_query->temporary_password_for_checks.reset();
|
||||
}
|
||||
}
|
||||
|
||||
processed_rows = 0;
|
||||
written_first_block = false;
|
||||
progress_indication.resetProgress();
|
||||
|
||||
@ -309,6 +309,21 @@ void Connection::receiveHello()
|
||||
readVarUInt(server_version_patch, *in);
|
||||
else
|
||||
server_version_patch = server_revision;
|
||||
|
||||
if (server_revision >= DBMS_MIN_PROTOCOL_VERSION_WITH_PASSWORD_COMPLEXITY_RULES)
|
||||
{
|
||||
UInt64 rules_size;
|
||||
readVarUInt(rules_size, *in);
|
||||
password_complexity_rules.reserve(rules_size);
|
||||
|
||||
for (size_t i = 0; i < rules_size; ++i)
|
||||
{
|
||||
String original_pattern, exception_message;
|
||||
readStringBinary(original_pattern, *in);
|
||||
readStringBinary(exception_message, *in);
|
||||
password_complexity_rules.push_back({std::move(original_pattern), std::move(exception_message)});
|
||||
}
|
||||
}
|
||||
}
|
||||
else if (packet_type == Protocol::Server::Exception)
|
||||
receiveException()->rethrow();
|
||||
|
||||
@ -93,6 +93,8 @@ public:
|
||||
|
||||
Protocol::Compression getCompression() const { return compression; }
|
||||
|
||||
std::vector<std::pair<String, String>> getPasswordComplexityRules() const override { return password_complexity_rules; }
|
||||
|
||||
void sendQuery(
|
||||
const ConnectionTimeouts & timeouts,
|
||||
const String & query,
|
||||
@ -207,6 +209,8 @@ private:
|
||||
*/
|
||||
ThrottlerPtr throttler;
|
||||
|
||||
std::vector<std::pair<String, String>> password_complexity_rules;
|
||||
|
||||
/// From where to read query execution result.
|
||||
std::shared_ptr<ReadBuffer> maybe_compressed_in;
|
||||
std::unique_ptr<NativeReader> block_in;
|
||||
|
||||
@ -82,6 +82,8 @@ public:
|
||||
|
||||
virtual const String & getDescription() const = 0;
|
||||
|
||||
virtual std::vector<std::pair<String, String>> getPasswordComplexityRules() const = 0;
|
||||
|
||||
/// If last flag is true, you need to call sendExternalTablesData after.
|
||||
virtual void sendQuery(
|
||||
const ConnectionTimeouts & timeouts,
|
||||
|
||||
@ -91,6 +91,8 @@ public:
|
||||
|
||||
const String & getDescription() const override { return description; }
|
||||
|
||||
std::vector<std::pair<String, String>> getPasswordComplexityRules() const override { return {}; }
|
||||
|
||||
void sendQuery(
|
||||
const ConnectionTimeouts & timeouts,
|
||||
const String & query,
|
||||
|
||||
@ -103,6 +103,7 @@
|
||||
M(S3Requests, "S3 requests") \
|
||||
M(KeeperAliveConnections, "Number of alive connections") \
|
||||
M(KeeperOutstandingRequets, "Number of outstanding requests") \
|
||||
M(ThreadsInOvercommitTracker, "Number of waiting threads inside of OvercommitTracker") \
|
||||
|
||||
namespace CurrentMetrics
|
||||
{
|
||||
|
||||
@ -1204,11 +1204,6 @@ public:
|
||||
return res;
|
||||
}
|
||||
|
||||
template <typename DateOrTime>
|
||||
inline DateTimeComponents toDateTimeComponents(DateOrTime v) const
|
||||
{
|
||||
return toDateTimeComponents(lut[toLUTIndex(v)].date);
|
||||
}
|
||||
|
||||
inline UInt64 toNumYYYYMMDDhhmmss(Time t) const
|
||||
{
|
||||
|
||||
@ -3,8 +3,13 @@
|
||||
#include <chrono>
|
||||
#include <mutex>
|
||||
#include <Common/ProfileEvents.h>
|
||||
#include <Common/CurrentMetrics.h>
|
||||
#include <Interpreters/ProcessList.h>
|
||||
|
||||
namespace CurrentMetrics
|
||||
{
|
||||
extern const Metric ThreadsInOvercommitTracker;
|
||||
}
|
||||
|
||||
namespace ProfileEvents
|
||||
{
|
||||
@ -32,6 +37,8 @@ OvercommitResult OvercommitTracker::needToStopQuery(MemoryTracker * tracker, Int
|
||||
|
||||
if (OvercommitTrackerBlockerInThread::isBlocked())
|
||||
return OvercommitResult::NONE;
|
||||
|
||||
CurrentMetrics::Increment metric_increment(CurrentMetrics::ThreadsInOvercommitTracker);
|
||||
// NOTE: Do not change the order of locks
|
||||
//
|
||||
// global mutex must be acquired before overcommit_m, because
|
||||
|
||||
@ -123,16 +123,13 @@ void ProgressIndication::writeFinalProgress()
|
||||
if (progress.read_rows < 1000)
|
||||
return;
|
||||
|
||||
UInt64 processed_rows = progress.read_rows + progress.written_rows;
|
||||
UInt64 processed_bytes = progress.read_bytes + progress.written_bytes;
|
||||
|
||||
std::cout << "Processed " << formatReadableQuantity(processed_rows) << " rows, "
|
||||
<< formatReadableSizeWithDecimalSuffix(processed_bytes);
|
||||
std::cout << "Processed " << formatReadableQuantity(progress.read_rows) << " rows, "
|
||||
<< formatReadableSizeWithDecimalSuffix(progress.read_bytes);
|
||||
|
||||
UInt64 elapsed_ns = getElapsedNanoseconds();
|
||||
if (elapsed_ns)
|
||||
std::cout << " (" << formatReadableQuantity(processed_rows * 1000000000.0 / elapsed_ns) << " rows/s., "
|
||||
<< formatReadableSizeWithDecimalSuffix(processed_bytes * 1000000000.0 / elapsed_ns) << "/s.)";
|
||||
std::cout << " (" << formatReadableQuantity(progress.read_rows * 1000000000.0 / elapsed_ns) << " rows/s., "
|
||||
<< formatReadableSizeWithDecimalSuffix(progress.read_bytes * 1000000000.0 / elapsed_ns) << "/s.)";
|
||||
else
|
||||
std::cout << ". ";
|
||||
}
|
||||
@ -167,18 +164,16 @@ void ProgressIndication::writeProgress(WriteBufferFromFileDescriptor & message)
|
||||
|
||||
size_t prefix_size = message.count();
|
||||
|
||||
UInt64 processed_rows = progress.read_rows + progress.written_rows;
|
||||
UInt64 processed_bytes = progress.read_bytes + progress.written_bytes;
|
||||
message << indicator << " Progress: ";
|
||||
message
|
||||
<< formatReadableQuantity(processed_rows) << " rows, "
|
||||
<< formatReadableSizeWithDecimalSuffix(processed_bytes);
|
||||
<< formatReadableQuantity(progress.read_rows) << " rows, "
|
||||
<< formatReadableSizeWithDecimalSuffix(progress.read_bytes);
|
||||
|
||||
UInt64 elapsed_ns = getElapsedNanoseconds();
|
||||
if (elapsed_ns)
|
||||
message << " ("
|
||||
<< formatReadableQuantity(processed_rows * 1000000000.0 / elapsed_ns) << " rows/s., "
|
||||
<< formatReadableSizeWithDecimalSuffix(processed_bytes * 1000000000.0 / elapsed_ns) << "/s.) ";
|
||||
<< formatReadableQuantity(progress.read_rows * 1000000000.0 / elapsed_ns) << " rows/s., "
|
||||
<< formatReadableSizeWithDecimalSuffix(progress.read_bytes * 1000000000.0 / elapsed_ns) << "/s.) ";
|
||||
else
|
||||
message << ". ";
|
||||
|
||||
|
||||
@ -52,7 +52,7 @@
|
||||
/// NOTE: DBMS_TCP_PROTOCOL_VERSION has nothing common with VERSION_REVISION,
|
||||
/// later is just a number for server version (one number instead of commit SHA)
|
||||
/// for simplicity (sometimes it may be more convenient in some use cases).
|
||||
#define DBMS_TCP_PROTOCOL_VERSION 54460
|
||||
#define DBMS_TCP_PROTOCOL_VERSION 54461
|
||||
|
||||
#define DBMS_MIN_PROTOCOL_VERSION_WITH_INITIAL_QUERY_START_TIME 54449
|
||||
|
||||
@ -68,3 +68,5 @@
|
||||
|
||||
/// The server will send query elapsed run time in the Progress packet.
|
||||
#define DBMS_MIN_PROTOCOL_VERSION_WITH_SERVER_QUERY_TIME_IN_PROGRESS 54460
|
||||
|
||||
#define DBMS_MIN_PROTOCOL_VERSION_WITH_PASSWORD_COMPLEXITY_RULES 54461
|
||||
|
||||
@ -222,6 +222,7 @@ static constexpr UInt64 operator""_GiB(unsigned long long value)
|
||||
M(UInt64, max_concurrent_queries_for_user, 0, "The maximum number of concurrent requests per user.", 0) \
|
||||
\
|
||||
M(Bool, insert_deduplicate, true, "For INSERT queries in the replicated table, specifies that deduplication of insertings blocks should be performed", 0) \
|
||||
M(Bool, async_insert_deduplicate, false, "For async INSERT queries in the replicated table, specifies that deduplication of insertings blocks should be performed", 0) \
|
||||
\
|
||||
M(UInt64Auto, insert_quorum, 0, "For INSERT queries in the replicated table, wait writing for the specified number of replicas and linearize the addition of the data. 0 - disabled, 'auto' - use majority", 0) \
|
||||
M(Milliseconds, insert_quorum_timeout, 600000, "If the quorum of replicas did not meet in specified time (in milliseconds), exception will be thrown and insertion is aborted.", 0) \
|
||||
|
||||
@ -1,12 +1,17 @@
|
||||
#include <Databases/DDLDependencyVisitor.h>
|
||||
#include <Dictionaries/getDictionaryConfigurationFromAST.h>
|
||||
#include <Interpreters/Cluster.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Interpreters/InDepthNodeVisitor.h>
|
||||
#include <Interpreters/evaluateConstantExpression.h>
|
||||
#include <Interpreters/getClusterName.h>
|
||||
#include <Parsers/ASTCreateQuery.h>
|
||||
#include <Parsers/ASTFunction.h>
|
||||
#include <Parsers/ASTIdentifier.h>
|
||||
#include <Parsers/ASTLiteral.h>
|
||||
#include <Parsers/ASTSelectWithUnionQuery.h>
|
||||
#include <Parsers/ASTTablesInSelectQuery.h>
|
||||
#include <Common/KnownObjectNames.h>
|
||||
#include <Poco/String.h>
|
||||
|
||||
|
||||
@ -15,225 +20,401 @@ namespace DB
|
||||
|
||||
namespace
|
||||
{
|
||||
/// CREATE TABLE or CREATE DICTIONARY or CREATE VIEW or CREATE TEMPORARY TABLE or CREATE DATABASE query.
|
||||
void visitCreateQuery(const ASTCreateQuery & create, DDLDependencyVisitor::Data & data)
|
||||
/// Data for DDLDependencyVisitor.
|
||||
/// Used to visits ASTCreateQuery and extracts the names of all tables explicitly referenced in the create query.
|
||||
class DDLDependencyVisitorData
|
||||
{
|
||||
QualifiedTableName to_table{create.to_table_id.database_name, create.to_table_id.table_name};
|
||||
if (!to_table.table.empty())
|
||||
public:
|
||||
DDLDependencyVisitorData(const ContextPtr & context_, const QualifiedTableName & table_name_, const ASTPtr & ast_)
|
||||
: create_query(ast_), table_name(table_name_), current_database(context_->getCurrentDatabase()), context(context_)
|
||||
{
|
||||
/// TO target_table (for materialized views)
|
||||
if (to_table.database.empty())
|
||||
to_table.database = data.default_database;
|
||||
data.dependencies.emplace(to_table);
|
||||
}
|
||||
|
||||
QualifiedTableName as_table{create.as_database, create.as_table};
|
||||
if (!as_table.table.empty())
|
||||
/// Acquire the result of visiting the create query.
|
||||
TableNamesSet getDependencies() &&
|
||||
{
|
||||
/// AS table_name
|
||||
if (as_table.database.empty())
|
||||
as_table.database = data.default_database;
|
||||
data.dependencies.emplace(as_table);
|
||||
}
|
||||
}
|
||||
|
||||
/// ASTTableExpression represents a reference to a table in SELECT query.
|
||||
/// DDLDependencyVisitor should handle ASTTableExpression because some CREATE queries can contain SELECT queries after AS
|
||||
/// (for example, CREATE VIEW).
|
||||
void visitTableExpression(const ASTTableExpression & expr, DDLDependencyVisitor::Data & data)
|
||||
{
|
||||
if (!expr.database_and_table_name)
|
||||
return;
|
||||
|
||||
const ASTIdentifier * identifier = dynamic_cast<const ASTIdentifier *>(expr.database_and_table_name.get());
|
||||
if (!identifier)
|
||||
return;
|
||||
|
||||
auto table_identifier = identifier->createTable();
|
||||
if (!table_identifier)
|
||||
return;
|
||||
|
||||
QualifiedTableName qualified_name{table_identifier->getDatabaseName(), table_identifier->shortName()};
|
||||
if (qualified_name.table.empty())
|
||||
return;
|
||||
|
||||
if (qualified_name.database.empty())
|
||||
{
|
||||
/// It can be table/dictionary from default database or XML dictionary, but we cannot distinguish it here.
|
||||
qualified_name.database = data.default_database;
|
||||
dependencies.erase(table_name);
|
||||
return std::move(dependencies);
|
||||
}
|
||||
|
||||
data.dependencies.emplace(qualified_name);
|
||||
}
|
||||
bool needChildVisit(const ASTPtr & child) const { return !skip_asts.contains(child.get()); }
|
||||
|
||||
/// Extracts a table name with optional database written in the form db_name.table_name (as identifier) or 'db_name.table_name' (as string).
|
||||
void extractQualifiedTableNameFromArgument(const ASTFunction & function, DDLDependencyVisitor::Data & data, size_t arg_idx)
|
||||
{
|
||||
/// Just ignore incorrect arguments, proper exception will be thrown later
|
||||
if (!function.arguments || function.arguments->children.size() <= arg_idx)
|
||||
return;
|
||||
|
||||
QualifiedTableName qualified_name;
|
||||
|
||||
const auto * expr_list = function.arguments->as<ASTExpressionList>();
|
||||
if (!expr_list)
|
||||
return;
|
||||
|
||||
const auto * arg = expr_list->children[arg_idx].get();
|
||||
if (const auto * literal = arg->as<ASTLiteral>())
|
||||
void visit(const ASTPtr & ast)
|
||||
{
|
||||
if (literal->value.getType() != Field::Types::String)
|
||||
if (auto * create = ast->as<ASTCreateQuery>())
|
||||
{
|
||||
visitCreateQuery(*create);
|
||||
}
|
||||
else if (auto * dictionary = ast->as<ASTDictionary>())
|
||||
{
|
||||
visitDictionaryDef(*dictionary);
|
||||
}
|
||||
else if (auto * expr = ast->as<ASTTableExpression>())
|
||||
{
|
||||
visitTableExpression(*expr);
|
||||
}
|
||||
else if (const auto * function = ast->as<ASTFunction>())
|
||||
{
|
||||
if (function->kind == ASTFunction::Kind::TABLE_ENGINE)
|
||||
visitTableEngine(*function);
|
||||
else
|
||||
visitFunction(*function);
|
||||
}
|
||||
}
|
||||
|
||||
private:
|
||||
ASTPtr create_query;
|
||||
std::unordered_set<const IAST *> skip_asts;
|
||||
QualifiedTableName table_name;
|
||||
String current_database;
|
||||
ContextPtr context;
|
||||
TableNamesSet dependencies;
|
||||
|
||||
/// CREATE TABLE or CREATE DICTIONARY or CREATE VIEW or CREATE TEMPORARY TABLE or CREATE DATABASE query.
|
||||
void visitCreateQuery(const ASTCreateQuery & create)
|
||||
{
|
||||
QualifiedTableName to_table{create.to_table_id.database_name, create.to_table_id.table_name};
|
||||
if (!to_table.table.empty())
|
||||
{
|
||||
/// TO target_table (for materialized views)
|
||||
if (to_table.database.empty())
|
||||
to_table.database = current_database;
|
||||
dependencies.emplace(to_table);
|
||||
}
|
||||
|
||||
QualifiedTableName as_table{create.as_database, create.as_table};
|
||||
if (!as_table.table.empty())
|
||||
{
|
||||
/// AS table_name
|
||||
if (as_table.database.empty())
|
||||
as_table.database = current_database;
|
||||
dependencies.emplace(as_table);
|
||||
}
|
||||
}
|
||||
|
||||
/// The definition of a dictionary: SOURCE(CLICKHOUSE(...)) LAYOUT(...) LIFETIME(...)
|
||||
void visitDictionaryDef(const ASTDictionary & dictionary)
|
||||
{
|
||||
if (!dictionary.source || dictionary.source->name != "clickhouse" || !dictionary.source->elements)
|
||||
return;
|
||||
|
||||
auto maybe_qualified_name = QualifiedTableName::tryParseFromString(literal->value.get<String>());
|
||||
/// Just return if name if invalid
|
||||
if (!maybe_qualified_name)
|
||||
auto config = getDictionaryConfigurationFromAST(create_query->as<ASTCreateQuery &>(), context);
|
||||
auto info = getInfoIfClickHouseDictionarySource(config, context);
|
||||
|
||||
/// We consider only dependencies on local tables.
|
||||
if (!info || !info->is_local)
|
||||
return;
|
||||
|
||||
qualified_name = std::move(*maybe_qualified_name);
|
||||
if (info->table_name.database.empty())
|
||||
info->table_name.database = current_database;
|
||||
dependencies.emplace(std::move(info->table_name));
|
||||
}
|
||||
else if (const auto * identifier = dynamic_cast<const ASTIdentifier *>(arg))
|
||||
|
||||
/// ASTTableExpression represents a reference to a table in SELECT query.
|
||||
/// DDLDependencyVisitor should handle ASTTableExpression because some CREATE queries can contain SELECT queries after AS
|
||||
/// (for example, CREATE VIEW).
|
||||
void visitTableExpression(const ASTTableExpression & expr)
|
||||
{
|
||||
/// ASTIdentifier or ASTTableIdentifier
|
||||
if (!expr.database_and_table_name)
|
||||
return;
|
||||
|
||||
const ASTIdentifier * identifier = dynamic_cast<const ASTIdentifier *>(expr.database_and_table_name.get());
|
||||
if (!identifier)
|
||||
return;
|
||||
|
||||
auto table_identifier = identifier->createTable();
|
||||
/// Just return if table identified is invalid
|
||||
if (!table_identifier)
|
||||
return;
|
||||
|
||||
qualified_name.database = table_identifier->getDatabaseName();
|
||||
qualified_name.table = table_identifier->shortName();
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Just return because we don't validate AST in this function.
|
||||
return;
|
||||
QualifiedTableName qualified_name{table_identifier->getDatabaseName(), table_identifier->shortName()};
|
||||
if (qualified_name.table.empty())
|
||||
return;
|
||||
|
||||
if (qualified_name.database.empty())
|
||||
{
|
||||
/// It can be table/dictionary from default database or XML dictionary, but we cannot distinguish it here.
|
||||
qualified_name.database = current_database;
|
||||
}
|
||||
|
||||
dependencies.emplace(qualified_name);
|
||||
}
|
||||
|
||||
if (qualified_name.database.empty())
|
||||
/// Finds dependencies of a table engine.
|
||||
void visitTableEngine(const ASTFunction & table_engine)
|
||||
{
|
||||
/// It can be table/dictionary from default database or XML dictionary, but we cannot distinguish it here.
|
||||
qualified_name.database = data.default_database;
|
||||
}
|
||||
data.dependencies.emplace(std::move(qualified_name));
|
||||
}
|
||||
/// Dictionary(db_name.dictionary_name)
|
||||
if (table_engine.name == "Dictionary")
|
||||
addQualifiedNameFromArgument(table_engine, 0);
|
||||
|
||||
/// Extracts a table name with database written in the form 'db_name', 'table_name' (two strings).
|
||||
void extractDatabaseAndTableNameFromArguments(const ASTFunction & function, DDLDependencyVisitor::Data & data, size_t database_arg_idx, size_t table_arg_idx)
|
||||
/// Buffer('db_name', 'dest_table_name')
|
||||
if (table_engine.name == "Buffer")
|
||||
addDatabaseAndTableNameFromArguments(table_engine, 0, 1);
|
||||
|
||||
/// Distributed(cluster_name, db_name, table_name, ...)
|
||||
if (table_engine.name == "Distributed")
|
||||
visitDistributedTableEngine(table_engine);
|
||||
}
|
||||
|
||||
/// Distributed(cluster_name, database_name, table_name, ...)
|
||||
void visitDistributedTableEngine(const ASTFunction & table_engine)
|
||||
{
|
||||
/// We consider only dependencies on local tables.
|
||||
bool has_local_replicas = false;
|
||||
|
||||
if (auto cluster_name = tryGetClusterNameFromArgument(table_engine, 0))
|
||||
{
|
||||
auto cluster = context->tryGetCluster(*cluster_name);
|
||||
if (cluster && cluster->getLocalShardCount())
|
||||
has_local_replicas = true;
|
||||
}
|
||||
|
||||
if (has_local_replicas)
|
||||
addDatabaseAndTableNameFromArguments(table_engine, 1, 2);
|
||||
}
|
||||
|
||||
/// Finds dependencies of a function.
|
||||
void visitFunction(const ASTFunction & function)
|
||||
{
|
||||
if (function.name == "joinGet" || function.name == "dictHas" || function.name == "dictIsIn" || function.name.starts_with("dictGet"))
|
||||
{
|
||||
/// dictGet('dict_name', attr_names, id_expr)
|
||||
/// dictHas('dict_name', id_expr)
|
||||
/// joinGet(join_storage_table_name, `value_column`, join_keys)
|
||||
addQualifiedNameFromArgument(function, 0);
|
||||
}
|
||||
else if (function.name == "in" || function.name == "notIn" || function.name == "globalIn" || function.name == "globalNotIn")
|
||||
{
|
||||
/// in(x, table_name) - function for evaluating (x IN table_name)
|
||||
addQualifiedNameFromArgument(function, 1);
|
||||
}
|
||||
else if (function.name == "dictionary")
|
||||
{
|
||||
/// dictionary(dict_name)
|
||||
addQualifiedNameFromArgument(function, 0);
|
||||
}
|
||||
else if (function.name == "remote" || function.name == "remoteSecure")
|
||||
{
|
||||
visitRemoteFunction(function, /* is_cluster_function= */ false);
|
||||
}
|
||||
else if (function.name == "cluster" || function.name == "clusterAllReplicas")
|
||||
{
|
||||
visitRemoteFunction(function, /* is_cluster_function= */ true);
|
||||
}
|
||||
}
|
||||
|
||||
/// remote('addresses_expr', db_name.table_name, ...)
|
||||
/// remote('addresses_expr', 'db_name', 'table_name', ...)
|
||||
/// remote('addresses_expr', table_function(), ...)
|
||||
/// cluster('cluster_name', db_name.table_name, ...)
|
||||
/// cluster('cluster_name', 'db_name', 'table_name', ...)
|
||||
/// cluster('cluster_name', table_function(), ...)
|
||||
void visitRemoteFunction(const ASTFunction & function, bool is_cluster_function)
|
||||
{
|
||||
/// We consider dependencies on local tables only.
|
||||
bool has_local_replicas = false;
|
||||
|
||||
if (is_cluster_function)
|
||||
{
|
||||
if (auto cluster_name = tryGetClusterNameFromArgument(function, 0))
|
||||
{
|
||||
if (auto cluster = context->tryGetCluster(*cluster_name))
|
||||
{
|
||||
if (cluster->getLocalShardCount())
|
||||
has_local_replicas = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
/// remote() and remoteSecure() are not fully supported. To properly support them we would need to check the first
|
||||
/// argument to decide whether the host & port pattern specified in the first argument contains the local host or not
|
||||
/// which is not trivial. For now we just always assume that the host & port pattern doesn't contain the local host.
|
||||
}
|
||||
|
||||
if (!function.arguments)
|
||||
return;
|
||||
|
||||
ASTs & args = function.arguments->children;
|
||||
if (args.size() < 2)
|
||||
return;
|
||||
|
||||
const ASTFunction * table_function = nullptr;
|
||||
if (const auto * second_arg_as_function = args[1]->as<ASTFunction>();
|
||||
second_arg_as_function && KnownTableFunctionNames::instance().exists(second_arg_as_function->name))
|
||||
{
|
||||
table_function = second_arg_as_function;
|
||||
}
|
||||
|
||||
if (has_local_replicas && !table_function)
|
||||
{
|
||||
auto maybe_qualified_name = tryGetQualifiedNameFromArgument(function, 1, /* apply_current_database= */ false);
|
||||
if (!maybe_qualified_name)
|
||||
return;
|
||||
auto & qualified_name = *maybe_qualified_name;
|
||||
if (qualified_name.database.empty())
|
||||
{
|
||||
auto table = tryGetStringFromArgument(function, 2);
|
||||
if (!table)
|
||||
return;
|
||||
qualified_name.database = std::move(qualified_name.table);
|
||||
qualified_name.table = std::move(table).value();
|
||||
}
|
||||
dependencies.insert(qualified_name);
|
||||
}
|
||||
|
||||
if (!has_local_replicas && table_function)
|
||||
{
|
||||
/// `table function` will be executed remotely, so we won't check it or its arguments for dependencies.
|
||||
skip_asts.emplace(table_function);
|
||||
}
|
||||
}
|
||||
|
||||
/// Gets an argument as a string, evaluates constants if necessary.
|
||||
std::optional<String> tryGetStringFromArgument(const ASTFunction & function, size_t arg_idx) const
|
||||
{
|
||||
if (!function.arguments)
|
||||
return {};
|
||||
|
||||
const ASTs & args = function.arguments->children;
|
||||
if (arg_idx >= args.size())
|
||||
return {};
|
||||
|
||||
const auto & arg = args[arg_idx];
|
||||
ASTPtr evaluated;
|
||||
|
||||
try
|
||||
{
|
||||
evaluated = evaluateConstantExpressionOrIdentifierAsLiteral(arg, context);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
return {};
|
||||
}
|
||||
|
||||
const auto * literal = evaluated->as<ASTLiteral>();
|
||||
if (!literal || (literal->value.getType() != Field::Types::String))
|
||||
return {};
|
||||
return literal->value.safeGet<String>();
|
||||
}
|
||||
|
||||
/// Gets an argument as a qualified table name.
|
||||
/// Accepts forms db_name.table_name (as an identifier) and 'db_name.table_name' (as a string).
|
||||
/// The function doesn't replace an empty database name with the current_database (the caller must do that).
|
||||
std::optional<QualifiedTableName>
|
||||
tryGetQualifiedNameFromArgument(const ASTFunction & function, size_t arg_idx, bool apply_current_database = true) const
|
||||
{
|
||||
if (!function.arguments)
|
||||
return {};
|
||||
|
||||
const ASTs & args = function.arguments->children;
|
||||
if (arg_idx >= args.size())
|
||||
return {};
|
||||
|
||||
const auto & arg = args[arg_idx];
|
||||
QualifiedTableName qualified_name;
|
||||
|
||||
if (const auto * identifier = dynamic_cast<const ASTIdentifier *>(arg.get()))
|
||||
{
|
||||
/// ASTIdentifier or ASTTableIdentifier
|
||||
auto table_identifier = identifier->createTable();
|
||||
if (!table_identifier)
|
||||
return {};
|
||||
|
||||
qualified_name.database = table_identifier->getDatabaseName();
|
||||
qualified_name.table = table_identifier->shortName();
|
||||
}
|
||||
else
|
||||
{
|
||||
auto qualified_name_as_string = tryGetStringFromArgument(function, arg_idx);
|
||||
if (!qualified_name_as_string)
|
||||
return {};
|
||||
|
||||
auto maybe_qualified_name = QualifiedTableName::tryParseFromString(*qualified_name_as_string);
|
||||
if (!maybe_qualified_name)
|
||||
return {};
|
||||
|
||||
qualified_name = std::move(maybe_qualified_name).value();
|
||||
}
|
||||
|
||||
if (qualified_name.database.empty() && apply_current_database)
|
||||
qualified_name.database = current_database;
|
||||
|
||||
return qualified_name;
|
||||
}
|
||||
|
||||
/// Adds a qualified table name from an argument to the collection of dependencies.
|
||||
/// Accepts forms db_name.table_name (as an identifier) and 'db_name.table_name' (as a string).
|
||||
void addQualifiedNameFromArgument(const ASTFunction & function, size_t arg_idx)
|
||||
{
|
||||
if (auto qualified_name = tryGetQualifiedNameFromArgument(function, arg_idx))
|
||||
dependencies.emplace(std::move(qualified_name).value());
|
||||
}
|
||||
|
||||
/// Returns a database name and a table name extracted from two separate arguments.
|
||||
std::optional<QualifiedTableName> tryGetDatabaseAndTableNameFromArguments(
|
||||
const ASTFunction & function, size_t database_arg_idx, size_t table_arg_idx, bool apply_current_database = true) const
|
||||
{
|
||||
auto database = tryGetStringFromArgument(function, database_arg_idx);
|
||||
if (!database)
|
||||
return {};
|
||||
|
||||
auto table = tryGetStringFromArgument(function, table_arg_idx);
|
||||
if (!table)
|
||||
return {};
|
||||
|
||||
QualifiedTableName qualified_name;
|
||||
qualified_name.database = std::move(database).value();
|
||||
qualified_name.table = std::move(table).value();
|
||||
|
||||
if (qualified_name.database.empty() && apply_current_database)
|
||||
qualified_name.database = current_database;
|
||||
|
||||
return qualified_name;
|
||||
}
|
||||
|
||||
/// Adds a database name and a table name from two separate arguments to the collection of dependencies.
|
||||
void addDatabaseAndTableNameFromArguments(const ASTFunction & function, size_t database_arg_idx, size_t table_arg_idx)
|
||||
{
|
||||
if (auto qualified_name = tryGetDatabaseAndTableNameFromArguments(function, database_arg_idx, table_arg_idx))
|
||||
dependencies.emplace(std::move(qualified_name).value());
|
||||
}
|
||||
|
||||
std::optional<String> tryGetClusterNameFromArgument(const ASTFunction & function, size_t arg_idx) const
|
||||
{
|
||||
if (!function.arguments)
|
||||
return {};
|
||||
|
||||
ASTs & args = function.arguments->children;
|
||||
if (arg_idx >= args.size())
|
||||
return {};
|
||||
|
||||
auto cluster_name = ::DB::tryGetClusterName(*args[arg_idx]);
|
||||
if (cluster_name)
|
||||
return cluster_name;
|
||||
|
||||
return tryGetStringFromArgument(function, arg_idx);
|
||||
}
|
||||
};
|
||||
|
||||
/// Visits ASTCreateQuery and extracts the names of all tables explicitly referenced in the create query.
|
||||
class DDLDependencyVisitor
|
||||
{
|
||||
/// Just ignore incorrect arguments, proper exception will be thrown later
|
||||
if (!function.arguments || (function.arguments->children.size() <= database_arg_idx)
|
||||
|| (function.arguments->children.size() <= table_arg_idx))
|
||||
return;
|
||||
public:
|
||||
using Data = DDLDependencyVisitorData;
|
||||
using Visitor = ConstInDepthNodeVisitor<DDLDependencyVisitor, /* top_to_bottom= */ true, /* need_child_accept_data= */ true>;
|
||||
|
||||
const auto * expr_list = function.arguments->as<ASTExpressionList>();
|
||||
if (!expr_list)
|
||||
return;
|
||||
|
||||
const auto * database_literal = expr_list->children[database_arg_idx]->as<ASTLiteral>();
|
||||
const auto * table_name_literal = expr_list->children[table_arg_idx]->as<ASTLiteral>();
|
||||
|
||||
if (!database_literal || !table_name_literal || (database_literal->value.getType() != Field::Types::String)
|
||||
|| (table_name_literal->value.getType() != Field::Types::String))
|
||||
return;
|
||||
|
||||
QualifiedTableName qualified_name{database_literal->value.get<String>(), table_name_literal->value.get<String>()};
|
||||
if (qualified_name.table.empty())
|
||||
return;
|
||||
|
||||
if (qualified_name.database.empty())
|
||||
qualified_name.database = data.default_database;
|
||||
|
||||
data.dependencies.emplace(qualified_name);
|
||||
}
|
||||
|
||||
void visitFunction(const ASTFunction & function, DDLDependencyVisitor::Data & data)
|
||||
{
|
||||
if (function.name == "joinGet" || function.name == "dictHas" || function.name == "dictIsIn" || function.name.starts_with("dictGet"))
|
||||
{
|
||||
/// dictGet('dict_name', attr_names, id_expr)
|
||||
/// dictHas('dict_name', id_expr)
|
||||
/// joinGet(join_storage_table_name, `value_column`, join_keys)
|
||||
extractQualifiedTableNameFromArgument(function, data, 0);
|
||||
}
|
||||
else if (function.name == "in" || function.name == "notIn" || function.name == "globalIn" || function.name == "globalNotIn")
|
||||
{
|
||||
/// in(x, table_name) - function for evaluating (x IN table_name)
|
||||
extractQualifiedTableNameFromArgument(function, data, 1);
|
||||
}
|
||||
else if (function.name == "dictionary")
|
||||
{
|
||||
/// dictionary(dict_name)
|
||||
extractQualifiedTableNameFromArgument(function, data, 0);
|
||||
}
|
||||
}
|
||||
|
||||
void visitTableEngine(const ASTFunction & table_engine, DDLDependencyVisitor::Data & data)
|
||||
{
|
||||
if (table_engine.name == "Dictionary")
|
||||
extractQualifiedTableNameFromArgument(table_engine, data, 0);
|
||||
|
||||
if (table_engine.name == "Buffer")
|
||||
extractDatabaseAndTableNameFromArguments(table_engine, data, 0, 1);
|
||||
}
|
||||
|
||||
void visitDictionaryDef(const ASTDictionary & dictionary, DDLDependencyVisitor::Data & data)
|
||||
{
|
||||
if (!dictionary.source || dictionary.source->name != "clickhouse" || !dictionary.source->elements)
|
||||
return;
|
||||
|
||||
auto config = getDictionaryConfigurationFromAST(data.create_query->as<ASTCreateQuery &>(), data.global_context);
|
||||
auto info = getInfoIfClickHouseDictionarySource(config, data.global_context);
|
||||
|
||||
if (!info || !info->is_local)
|
||||
return;
|
||||
|
||||
if (info->table_name.database.empty())
|
||||
info->table_name.database = data.default_database;
|
||||
data.dependencies.emplace(std::move(info->table_name));
|
||||
}
|
||||
static bool needChildVisit(const ASTPtr &, const ASTPtr & child, const Data & data) { return data.needChildVisit(child); }
|
||||
static void visit(const ASTPtr & ast, Data & data) { data.visit(ast); }
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
TableNamesSet getDependenciesFromCreateQuery(const ContextPtr & global_context, const QualifiedTableName & table_name, const ASTPtr & ast)
|
||||
TableNamesSet getDependenciesFromCreateQuery(const ContextPtr & context, const QualifiedTableName & table_name, const ASTPtr & ast)
|
||||
{
|
||||
assert(global_context == global_context->getGlobalContext());
|
||||
DDLDependencyVisitor::Data data;
|
||||
data.table_name = table_name;
|
||||
data.default_database = global_context->getCurrentDatabase();
|
||||
data.create_query = ast;
|
||||
data.global_context = global_context;
|
||||
DDLDependencyVisitor::Data data{context, table_name, ast};
|
||||
DDLDependencyVisitor::Visitor visitor{data};
|
||||
visitor.visit(ast);
|
||||
data.dependencies.erase(data.table_name);
|
||||
return data.dependencies;
|
||||
}
|
||||
|
||||
void DDLDependencyVisitor::visit(const ASTPtr & ast, Data & data)
|
||||
{
|
||||
if (auto * create = ast->as<ASTCreateQuery>())
|
||||
{
|
||||
visitCreateQuery(*create, data);
|
||||
}
|
||||
else if (auto * dictionary = ast->as<ASTDictionary>())
|
||||
{
|
||||
visitDictionaryDef(*dictionary, data);
|
||||
}
|
||||
else if (auto * expr = ast->as<ASTTableExpression>())
|
||||
{
|
||||
visitTableExpression(*expr, data);
|
||||
}
|
||||
else if (const auto * function = ast->as<ASTFunction>())
|
||||
{
|
||||
if (function->kind == ASTFunction::Kind::TABLE_ENGINE)
|
||||
visitTableEngine(*function, data);
|
||||
else
|
||||
visitFunction(*function, data);
|
||||
}
|
||||
}
|
||||
|
||||
bool DDLDependencyVisitor::needChildVisit(const ASTPtr &, const ASTPtr &)
|
||||
{
|
||||
return true;
|
||||
return std::move(data).getDependencies();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -1,8 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#include <Parsers/IAST_fwd.h>
|
||||
#include <Interpreters/InDepthNodeVisitor.h>
|
||||
#include <Core/QualifiedTableName.h>
|
||||
#include <Interpreters/Context_fwd.h>
|
||||
#include <Parsers/IAST_fwd.h>
|
||||
#include <unordered_set>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -12,25 +13,6 @@ using TableNamesSet = std::unordered_set<QualifiedTableName>;
|
||||
/// Returns a list of all tables explicitly referenced in the create query of a specified table.
|
||||
/// For example, a column default expression can use dictGet() and thus reference a dictionary.
|
||||
/// Does not validate AST, works a best-effort way.
|
||||
TableNamesSet getDependenciesFromCreateQuery(const ContextPtr & global_context, const QualifiedTableName & table_name, const ASTPtr & ast);
|
||||
|
||||
/// Visits ASTCreateQuery and extracts the names of all tables explicitly referenced in the create query.
|
||||
class DDLDependencyVisitor
|
||||
{
|
||||
public:
|
||||
struct Data
|
||||
{
|
||||
ASTPtr create_query;
|
||||
QualifiedTableName table_name;
|
||||
String default_database;
|
||||
ContextPtr global_context;
|
||||
TableNamesSet dependencies;
|
||||
};
|
||||
|
||||
using Visitor = ConstInDepthNodeVisitor<DDLDependencyVisitor, /* top_to_bottom= */ true>;
|
||||
|
||||
static void visit(const ASTPtr & ast, Data & data);
|
||||
static bool needChildVisit(const ASTPtr & node, const ASTPtr & child);
|
||||
};
|
||||
TableNamesSet getDependenciesFromCreateQuery(const ContextPtr & context, const QualifiedTableName & table_name, const ASTPtr & ast);
|
||||
|
||||
}
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#include <Databases/TablesDependencyGraph.h>
|
||||
#include <Common/logger_useful.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <boost/range/adaptor/reversed.hpp>
|
||||
|
||||
|
||||
@ -9,12 +10,13 @@ namespace DB
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int INFINITE_LOOP;
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
|
||||
namespace
|
||||
{
|
||||
constexpr const size_t CYCLIC_LEVEL = static_cast<size_t>(-2);
|
||||
constexpr const size_t CYCLIC_LEVEL = std::numeric_limits<size_t>::max();
|
||||
}
|
||||
|
||||
|
||||
@ -40,7 +42,7 @@ TablesDependencyGraph::TablesDependencyGraph(TablesDependencyGraph && src) noexc
|
||||
|
||||
TablesDependencyGraph & TablesDependencyGraph::operator=(const TablesDependencyGraph & src)
|
||||
{
|
||||
if (&src != this)
|
||||
if (this != &src)
|
||||
{
|
||||
nodes = src.nodes;
|
||||
nodes_by_database_and_table_names = src.nodes_by_database_and_table_names;
|
||||
@ -54,11 +56,14 @@ TablesDependencyGraph & TablesDependencyGraph::operator=(const TablesDependencyG
|
||||
|
||||
TablesDependencyGraph & TablesDependencyGraph::operator=(TablesDependencyGraph && src) noexcept
|
||||
{
|
||||
nodes = std::exchange(src.nodes, decltype(nodes){});
|
||||
nodes_by_database_and_table_names = std::exchange(src.nodes_by_database_and_table_names, decltype(nodes_by_database_and_table_names){});
|
||||
nodes_by_uuid = std::exchange(src.nodes_by_uuid, decltype(nodes_by_uuid){});
|
||||
levels_calculated = std::exchange(src.levels_calculated, false);
|
||||
nodes_sorted_by_level_lazy = std::exchange(src.nodes_sorted_by_level_lazy, decltype(nodes_sorted_by_level_lazy){});
|
||||
if (this != &src)
|
||||
{
|
||||
nodes = std::exchange(src.nodes, decltype(nodes){});
|
||||
nodes_by_database_and_table_names = std::exchange(src.nodes_by_database_and_table_names, decltype(nodes_by_database_and_table_names){});
|
||||
nodes_by_uuid = std::exchange(src.nodes_by_uuid, decltype(nodes_by_uuid){});
|
||||
levels_calculated = std::exchange(src.levels_calculated, false);
|
||||
nodes_sorted_by_level_lazy = std::exchange(src.nodes_sorted_by_level_lazy, decltype(nodes_sorted_by_level_lazy){});
|
||||
}
|
||||
return *this;
|
||||
}
|
||||
|
||||
@ -89,11 +94,13 @@ void TablesDependencyGraph::addDependency(const StorageID & table_id, const Stor
|
||||
auto * table_node = addOrUpdateNode(table_id);
|
||||
auto * dependency_node = addOrUpdateNode(dependency);
|
||||
|
||||
if (table_node->dependencies.contains(dependency_node))
|
||||
return; /// Already have this dependency.
|
||||
bool inserted = table_node->dependencies.insert(dependency_node).second;
|
||||
if (!inserted)
|
||||
return; /// Not inserted because we already had this dependency.
|
||||
|
||||
table_node->dependencies.insert(dependency_node);
|
||||
dependency_node->dependents.insert(table_node);
|
||||
/// `dependency_node` must be updated too.
|
||||
[[maybe_unused]] bool inserted_to_set = dependency_node->dependents.insert(table_node).second;
|
||||
chassert(inserted_to_set);
|
||||
|
||||
setNeedRecalculateLevels();
|
||||
}
|
||||
@ -126,13 +133,19 @@ void TablesDependencyGraph::addDependencies(const StorageID & table_id, const st
|
||||
for (auto * dependency_node : old_dependency_nodes)
|
||||
{
|
||||
if (!new_dependency_nodes.contains(dependency_node))
|
||||
dependency_node->dependents.erase(table_node);
|
||||
{
|
||||
[[maybe_unused]] bool removed_from_set = dependency_node->dependents.erase(table_node);
|
||||
chassert(removed_from_set);
|
||||
}
|
||||
}
|
||||
|
||||
for (auto * dependency_node : new_dependency_nodes)
|
||||
{
|
||||
if (!old_dependency_nodes.contains(dependency_node))
|
||||
dependency_node->dependents.insert(table_node);
|
||||
{
|
||||
[[maybe_unused]] bool inserted_to_set = dependency_node->dependents.insert(table_node).second;
|
||||
chassert(inserted_to_set);
|
||||
}
|
||||
}
|
||||
|
||||
table_node->dependencies = std::move(new_dependency_nodes);
|
||||
@ -167,21 +180,28 @@ bool TablesDependencyGraph::removeDependency(const StorageID & table_id, const S
|
||||
|
||||
auto dependency_it = table_node->dependencies.find(dependency_node);
|
||||
if (dependency_it == table_node->dependencies.end())
|
||||
return false;
|
||||
return false; /// No such dependency, nothing to remove.
|
||||
|
||||
table_node->dependencies.erase(dependency_it);
|
||||
dependency_node->dependents.erase(table_node);
|
||||
bool table_node_removed = false;
|
||||
|
||||
/// `dependency_node` must be updated too.
|
||||
[[maybe_unused]] bool removed_from_set = dependency_node->dependents.erase(table_node);
|
||||
chassert(removed_from_set);
|
||||
|
||||
if (remove_isolated_tables && dependency_node->dependencies.empty() && dependency_node->dependents.empty())
|
||||
{
|
||||
/// The dependency table has no dependencies and no dependents now, so we will remove it from the graph.
|
||||
removeNode(dependency_node);
|
||||
if (table_node == dependency_node)
|
||||
table_node_removed = true;
|
||||
}
|
||||
|
||||
if (remove_isolated_tables && !table_node_removed && table_node->dependencies.empty() && table_node->dependents.empty())
|
||||
{
|
||||
/// The table `table_id` has no dependencies and no dependents now, so we will remove it from the graph.
|
||||
removeNode(table_node);
|
||||
}
|
||||
|
||||
setNeedRecalculateLevels();
|
||||
return true;
|
||||
@ -203,19 +223,28 @@ std::vector<StorageID> TablesDependencyGraph::removeDependencies(const StorageID
|
||||
|
||||
for (auto * dependency_node : dependency_nodes)
|
||||
{
|
||||
/// We're gathering the list of dependencies the table `table_id` had in the graph to return from the function.
|
||||
dependencies.emplace_back(dependency_node->storage_id);
|
||||
dependency_node->dependents.erase(table_node);
|
||||
|
||||
/// Update `dependency_node`.
|
||||
[[maybe_unused]] bool removed_from_set = dependency_node->dependents.erase(table_node);
|
||||
chassert(removed_from_set);
|
||||
|
||||
if (remove_isolated_tables && dependency_node->dependencies.empty() && dependency_node->dependents.empty())
|
||||
{
|
||||
/// The dependency table has no dependencies and no dependents now, so we will remove it from the graph.
|
||||
removeNode(dependency_node);
|
||||
if (table_node == dependency_node)
|
||||
table_node_removed = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (remove_isolated_tables && !table_node_removed && table_node->dependencies.empty() && table_node->dependents.empty())
|
||||
chassert(table_node->dependencies.empty());
|
||||
if (remove_isolated_tables && !table_node_removed && table_node->dependents.empty())
|
||||
{
|
||||
/// The table `table_id` has no dependencies and no dependents now, so we will remove it from the graph.
|
||||
removeNode(table_node);
|
||||
}
|
||||
|
||||
setNeedRecalculateLevels();
|
||||
return dependencies;
|
||||
@ -251,7 +280,12 @@ TablesDependencyGraph::Node * TablesDependencyGraph::findNode(const StorageID &
|
||||
{
|
||||
auto * node = it->second;
|
||||
if (table_id.hasUUID() && node->storage_id.hasUUID() && (table_id.uuid != node->storage_id.uuid))
|
||||
return nullptr; /// UUID is different, it's not the node we're looking for.
|
||||
{
|
||||
/// We found a table with specified database and table names in the graph, but surprisingly it has a different UUID.
|
||||
/// Maybe an "EXCHANGE TABLES" command has been executed somehow without changing the graph?
|
||||
LOG_WARNING(getLogger(), "Found table {} in the graph with unexpected UUID {}", table_id, node->storage_id.uuid);
|
||||
return nullptr; /// Act like it's not found.
|
||||
}
|
||||
return node; /// Found by table name.
|
||||
}
|
||||
}
|
||||
@ -268,7 +302,8 @@ TablesDependencyGraph::Node * TablesDependencyGraph::addOrUpdateNode(const Stora
|
||||
if (table_id.hasUUID() && !node->storage_id.hasUUID())
|
||||
{
|
||||
node->storage_id.uuid = table_id.uuid;
|
||||
nodes_by_uuid.emplace(node->storage_id.uuid, node);
|
||||
[[maybe_unused]] bool inserted_to_map = nodes_by_uuid.emplace(node->storage_id.uuid, node).second;
|
||||
chassert(inserted_to_map);
|
||||
}
|
||||
|
||||
if (!table_id.table_name.empty() && ((table_id.table_name != node->storage_id.table_name) || (table_id.database_name != node->storage_id.database_name)))
|
||||
@ -283,7 +318,8 @@ TablesDependencyGraph::Node * TablesDependencyGraph::addOrUpdateNode(const Stora
|
||||
nodes_by_database_and_table_names.erase(node->storage_id);
|
||||
node->storage_id.database_name = table_id.database_name;
|
||||
node->storage_id.table_name = table_id.table_name;
|
||||
nodes_by_database_and_table_names.emplace(node->storage_id, node);
|
||||
[[maybe_unused]] bool inserted_to_map = nodes_by_database_and_table_names.emplace(node->storage_id, node).second;
|
||||
chassert(inserted_to_map);
|
||||
}
|
||||
}
|
||||
else
|
||||
@ -303,9 +339,15 @@ TablesDependencyGraph::Node * TablesDependencyGraph::addOrUpdateNode(const Stora
|
||||
nodes.insert(node_ptr);
|
||||
node = node_ptr.get();
|
||||
if (table_id.hasUUID())
|
||||
nodes_by_uuid.emplace(table_id.uuid, node);
|
||||
{
|
||||
[[maybe_unused]] bool inserted_to_map = nodes_by_uuid.emplace(table_id.uuid, node).second;
|
||||
chassert(inserted_to_map);
|
||||
}
|
||||
if (!table_id.table_name.empty())
|
||||
nodes_by_database_and_table_names.emplace(table_id, node);
|
||||
{
|
||||
[[maybe_unused]] bool inserted_to_map = nodes_by_database_and_table_names.emplace(table_id, node).second;
|
||||
chassert(inserted_to_map);
|
||||
}
|
||||
}
|
||||
return node;
|
||||
}
|
||||
@ -313,22 +355,39 @@ TablesDependencyGraph::Node * TablesDependencyGraph::addOrUpdateNode(const Stora
|
||||
|
||||
void TablesDependencyGraph::removeNode(Node * node)
|
||||
{
|
||||
chassert(node);
|
||||
auto dependency_nodes = std::move(node->dependencies);
|
||||
auto dependent_nodes = std::move(node->dependents);
|
||||
|
||||
if (node->storage_id.hasUUID())
|
||||
nodes_by_uuid.erase(node->storage_id.uuid);
|
||||
{
|
||||
[[maybe_unused]] bool removed_from_map = nodes_by_uuid.erase(node->storage_id.uuid);
|
||||
chassert(removed_from_map);
|
||||
}
|
||||
|
||||
if (!node->storage_id.table_name.empty())
|
||||
nodes_by_database_and_table_names.erase(node->storage_id);
|
||||
{
|
||||
[[maybe_unused]]bool removed_from_map = nodes_by_database_and_table_names.erase(node->storage_id);
|
||||
chassert(removed_from_map);
|
||||
}
|
||||
|
||||
for (auto * dependency_node : dependency_nodes)
|
||||
dependency_node->dependents.erase(node);
|
||||
{
|
||||
[[maybe_unused]] bool removed_from_set = dependency_node->dependents.erase(node);
|
||||
chassert(removed_from_set);
|
||||
}
|
||||
|
||||
for (auto * dependent_node : dependent_nodes)
|
||||
dependent_node->dependencies.erase(node);
|
||||
{
|
||||
[[maybe_unused]] bool removed_from_set = dependent_node->dependencies.erase(node);
|
||||
chassert(removed_from_set);
|
||||
}
|
||||
|
||||
nodes.erase(node->shared_from_this());
|
||||
auto it = nodes.find(node);
|
||||
chassert(it != nodes.end());
|
||||
nodes.erase(it);
|
||||
|
||||
nodes_sorted_by_level_lazy.clear();
|
||||
}
|
||||
|
||||
|
||||
@ -533,7 +592,7 @@ String TablesDependencyGraph::describeCyclicDependencies() const
|
||||
}
|
||||
|
||||
|
||||
void TablesDependencyGraph::setNeedRecalculateLevels()
|
||||
void TablesDependencyGraph::setNeedRecalculateLevels() const
|
||||
{
|
||||
levels_calculated = false;
|
||||
nodes_sorted_by_level_lazy.clear();
|
||||
@ -546,49 +605,73 @@ void TablesDependencyGraph::calculateLevels() const
|
||||
return;
|
||||
levels_calculated = true;
|
||||
|
||||
/// First find tables with no dependencies, add them to `nodes_sorted_by_level_lazy`.
|
||||
/// Then remove those tables from the dependency graph (we imitate that removing by decrementing `num_dependencies_to_count`),
|
||||
/// and find which tables have no dependencies now.
|
||||
/// Repeat until we have tables with no dependencies.
|
||||
/// In the end we expect all nodes from `nodes` to be added to `nodes_sorted_by_level_lazy`.
|
||||
/// If some nodes are still not added to `nodes_sorted_by_level_lazy` in the end then there is a cyclic dependency.
|
||||
/// Complexity: O(V + E)
|
||||
|
||||
nodes_sorted_by_level_lazy.clear();
|
||||
nodes_sorted_by_level_lazy.reserve(nodes.size());
|
||||
|
||||
std::unordered_set<const Node *> nodes_to_process;
|
||||
for (const auto & node_ptr : nodes)
|
||||
nodes_to_process.emplace(node_ptr.get());
|
||||
|
||||
size_t current_level = 0;
|
||||
|
||||
while (!nodes_to_process.empty())
|
||||
/// Find tables with no dependencies.
|
||||
for (const auto & node_ptr : nodes)
|
||||
{
|
||||
size_t old_num_sorted = nodes_sorted_by_level_lazy.size();
|
||||
|
||||
for (auto it = nodes_to_process.begin(); it != nodes_to_process.end();)
|
||||
const Node * node = node_ptr.get();
|
||||
node->num_dependencies_to_count = node->dependencies.size();
|
||||
if (!node->num_dependencies_to_count)
|
||||
{
|
||||
const auto * current_node = *(it++);
|
||||
bool has_dependencies = false;
|
||||
for (const auto * dependency : current_node->dependencies)
|
||||
{
|
||||
if (nodes_to_process.contains(dependency))
|
||||
has_dependencies = true;
|
||||
}
|
||||
node->level = current_level;
|
||||
nodes_sorted_by_level_lazy.emplace_back(node);
|
||||
}
|
||||
}
|
||||
|
||||
if (!has_dependencies)
|
||||
size_t num_nodes_without_dependencies = nodes_sorted_by_level_lazy.size();
|
||||
++current_level;
|
||||
|
||||
while (num_nodes_without_dependencies)
|
||||
{
|
||||
size_t begin = nodes_sorted_by_level_lazy.size() - num_nodes_without_dependencies;
|
||||
size_t end = nodes_sorted_by_level_lazy.size();
|
||||
|
||||
/// Decrement number of dependencies for each dependent table.
|
||||
for (size_t i = begin; i != end; ++i)
|
||||
{
|
||||
const Node * current_node = nodes_sorted_by_level_lazy[i];
|
||||
for (const Node * dependent_node : current_node->dependents)
|
||||
{
|
||||
current_node->level = current_level;
|
||||
nodes_sorted_by_level_lazy.emplace_back(current_node);
|
||||
if (!dependent_node->num_dependencies_to_count)
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "{}: Trying to decrement 0 dependencies counter for {}. It's a bug", name_for_logging, dependent_node->storage_id);
|
||||
|
||||
if (!--dependent_node->num_dependencies_to_count)
|
||||
{
|
||||
dependent_node->level = current_level;
|
||||
nodes_sorted_by_level_lazy.emplace_back(dependent_node);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (nodes_sorted_by_level_lazy.size() == old_num_sorted)
|
||||
break;
|
||||
|
||||
for (size_t i = old_num_sorted; i != nodes_sorted_by_level_lazy.size(); ++i)
|
||||
nodes_to_process.erase(nodes_sorted_by_level_lazy[i]);
|
||||
if (nodes_sorted_by_level_lazy.size() > nodes.size())
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "{}: Some tables were found more than once while passing through the dependency graph. It's a bug", name_for_logging);
|
||||
|
||||
num_nodes_without_dependencies = nodes_sorted_by_level_lazy.size() - end;
|
||||
++current_level;
|
||||
}
|
||||
|
||||
for (const auto * node_with_cyclic_dependencies : nodes_to_process)
|
||||
if (nodes_sorted_by_level_lazy.size() < nodes.size())
|
||||
{
|
||||
node_with_cyclic_dependencies->level = CYCLIC_LEVEL;
|
||||
nodes_sorted_by_level_lazy.emplace_back(node_with_cyclic_dependencies);
|
||||
for (const auto & node_ptr : nodes)
|
||||
{
|
||||
const Node * node = node_ptr.get();
|
||||
if (node->num_dependencies_to_count)
|
||||
{
|
||||
node->level = CYCLIC_LEVEL;
|
||||
nodes_sorted_by_level_lazy.emplace_back(node);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -630,7 +713,7 @@ std::vector<std::vector<StorageID>> TablesDependencyGraph::getTablesSortedByDepe
|
||||
|
||||
void TablesDependencyGraph::log() const
|
||||
{
|
||||
if (empty())
|
||||
if (nodes.empty())
|
||||
{
|
||||
LOG_TEST(getLogger(), "No tables");
|
||||
return;
|
||||
|
||||
@ -20,11 +20,11 @@ using TableNamesSet = std::unordered_set<QualifiedTableName>;
|
||||
///
|
||||
/// This class is used to represent various types of table-table dependencies:
|
||||
/// 1. View dependencies: "source_table -> materialized_view".
|
||||
/// Data inserted to a source table is also inserted to corresponding materialized views.
|
||||
/// Data inserted to a source table is also inserted to corresponding materialized views.
|
||||
/// 2. Loading dependencies: specify in which order tables must be loaded during startup.
|
||||
/// For example a dictionary should be loaded after it's source table and it's written in the graph as "dictionary -> source_table".
|
||||
/// For example a dictionary should be loaded after it's source table and it's written in the graph as "dictionary -> source_table".
|
||||
/// 3. Referential dependencies: "table -> all tables mentioned in its definition".
|
||||
/// Referential dependencies are checked to decide if it's safe to drop a table (it can be unsafe if the table is used by another table).
|
||||
/// Referential dependencies are checked to decide if it's safe to drop a table (it can be unsafe if the table is used by another table).
|
||||
///
|
||||
/// WARNING: This class doesn't have an embedded mutex, so it must be synchronized outside.
|
||||
class TablesDependencyGraph
|
||||
@ -98,8 +98,8 @@ public:
|
||||
/// Cyclic dependencies are dependencies like "A->A" or "A->B->C->D->A".
|
||||
void checkNoCyclicDependencies() const;
|
||||
bool hasCyclicDependencies() const;
|
||||
std::vector<StorageID> getTablesWithCyclicDependencies() const;
|
||||
String describeCyclicDependencies() const;
|
||||
std::vector<StorageID> getTablesWithCyclicDependencies() const;
|
||||
|
||||
/// Returns a list of tables sorted by their dependencies:
|
||||
/// tables without dependencies first, then
|
||||
@ -113,8 +113,12 @@ public:
|
||||
/// Outputs information about this graph as a bunch of logging messages.
|
||||
void log() const;
|
||||
|
||||
/// Calculates levels - this is required for checking cyclic dependencies, to sort tables by dependency, and to log the graph.
|
||||
/// This function is called automatically by the functions which need it, but can be invoked directly.
|
||||
void calculateLevels() const;
|
||||
|
||||
private:
|
||||
struct Node : public std::enable_shared_from_this<Node>
|
||||
struct Node
|
||||
{
|
||||
StorageID storage_id;
|
||||
|
||||
@ -128,28 +132,38 @@ private:
|
||||
/// Calculated lazily.
|
||||
mutable size_t level = 0;
|
||||
|
||||
/// Number of dependencies left, used only while we're calculating levels.
|
||||
mutable size_t num_dependencies_to_count = 0;
|
||||
|
||||
explicit Node(const StorageID & storage_id_) : storage_id(storage_id_) {}
|
||||
};
|
||||
|
||||
using NodeSharedPtr = std::shared_ptr<Node>;
|
||||
|
||||
struct LessByLevel
|
||||
struct Hash
|
||||
{
|
||||
bool operator()(const Node * left, const Node * right) { return left->level < right->level; }
|
||||
using is_transparent = void;
|
||||
size_t operator()(const Node * node) const { return std::hash<const Node *>{}(node); }
|
||||
size_t operator()(const NodeSharedPtr & node_ptr) const { return operator()(node_ptr.get()); }
|
||||
};
|
||||
|
||||
std::unordered_set<NodeSharedPtr> nodes;
|
||||
struct Equal
|
||||
{
|
||||
using is_transparent = void;
|
||||
size_t operator()(const NodeSharedPtr & left, const Node * right) const { return left.get() == right; }
|
||||
size_t operator()(const NodeSharedPtr & left, const NodeSharedPtr & right) const { return left == right; }
|
||||
};
|
||||
|
||||
std::unordered_set<NodeSharedPtr, Hash, Equal> nodes;
|
||||
|
||||
/// Nodes can be found either by UUID or by database name & table name. That's why we need two maps here.
|
||||
std::unordered_map<StorageID, Node *, StorageID::DatabaseAndTableNameHash, StorageID::DatabaseAndTableNameEqual> nodes_by_database_and_table_names;
|
||||
std::unordered_map<UUID, Node *> nodes_by_uuid;
|
||||
|
||||
/// This is set if both `level` inside each node and `nodes_sorted_by_level_lazy` are calculated.
|
||||
mutable bool levels_calculated = false;
|
||||
|
||||
/// Nodes sorted by their level. Calculated lazily.
|
||||
using NodesSortedByLevel = std::vector<const Node *>;
|
||||
mutable NodesSortedByLevel nodes_sorted_by_level_lazy;
|
||||
mutable bool levels_calculated = false;
|
||||
|
||||
const String name_for_logging;
|
||||
mutable Poco::Logger * logger = nullptr;
|
||||
@ -161,8 +175,7 @@ private:
|
||||
static std::vector<StorageID> getDependencies(const Node & node);
|
||||
static std::vector<StorageID> getDependents(const Node & node);
|
||||
|
||||
void setNeedRecalculateLevels();
|
||||
void calculateLevels() const;
|
||||
void setNeedRecalculateLevels() const;
|
||||
const NodesSortedByLevel & getNodesSortedByLevel() const;
|
||||
|
||||
Poco::Logger * getLogger() const;
|
||||
|
||||
@ -624,20 +624,21 @@ getInfoIfClickHouseDictionarySource(DictionaryConfigurationPtr & config, Context
|
||||
{
|
||||
ClickHouseDictionarySourceInfo info;
|
||||
|
||||
String host = config->getString("dictionary.source.clickhouse.host", "");
|
||||
UInt16 port = config->getUInt("dictionary.source.clickhouse.port", 0);
|
||||
bool secure = config->getBool("dictionary.source.clickhouse.secure", false);
|
||||
UInt16 default_port = secure ? global_context->getTCPPortSecure().value_or(0) : global_context->getTCPPort();
|
||||
|
||||
String host = config->getString("dictionary.source.clickhouse.host", "localhost");
|
||||
UInt16 port = config->getUInt("dictionary.source.clickhouse.port", default_port);
|
||||
String database = config->getString("dictionary.source.clickhouse.db", "");
|
||||
String table = config->getString("dictionary.source.clickhouse.table", "");
|
||||
bool secure = config->getBool("dictionary.source.clickhouse.secure", false);
|
||||
|
||||
if (host.empty() || port == 0 || table.empty())
|
||||
if (table.empty())
|
||||
return {};
|
||||
|
||||
info.table_name = {database, table};
|
||||
|
||||
try
|
||||
{
|
||||
UInt16 default_port = secure ? global_context->getTCPPortSecure().value_or(0) : global_context->getTCPPort();
|
||||
if (isLocalAddress({host, port}, default_port))
|
||||
info.is_local = true;
|
||||
}
|
||||
|
||||
@ -1343,30 +1343,6 @@ struct ToYYYYMMDDhhmmssImpl
|
||||
using FactorTransform = ZeroTransform;
|
||||
};
|
||||
|
||||
struct ToDateTimeComponentsImpl
|
||||
{
|
||||
static constexpr auto name = "toDateTimeComponents";
|
||||
|
||||
static inline DateLUTImpl::DateTimeComponents execute(Int64 t, const DateLUTImpl & time_zone)
|
||||
{
|
||||
return time_zone.toDateTimeComponents(t);
|
||||
}
|
||||
static inline DateLUTImpl::DateTimeComponents execute(UInt32 t, const DateLUTImpl & time_zone)
|
||||
{
|
||||
return time_zone.toDateTimeComponents(static_cast<DateLUTImpl::Time>(t));
|
||||
}
|
||||
static inline DateLUTImpl::DateTimeComponents execute(Int32 d, const DateLUTImpl & time_zone)
|
||||
{
|
||||
return time_zone.toDateTimeComponents(ExtendedDayNum(d));
|
||||
}
|
||||
static inline DateLUTImpl::DateTimeComponents execute(UInt16 d, const DateLUTImpl & time_zone)
|
||||
{
|
||||
return time_zone.toDateTimeComponents(DayNum(d));
|
||||
}
|
||||
|
||||
using FactorTransform = ZeroTransform;
|
||||
};
|
||||
|
||||
|
||||
template <typename FromType, typename ToType, typename Transform, bool is_extended_result = false>
|
||||
struct Transformer
|
||||
|
||||
@ -106,7 +106,7 @@ public:
|
||||
|
||||
DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
|
||||
{
|
||||
if (arguments.size() < 1 || arguments.size() > 2)
|
||||
if (arguments.empty() || arguments.size() > 2)
|
||||
throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, "Function {} takes one or two arguments", name);
|
||||
|
||||
if (!isInteger(arguments[0].type))
|
||||
@ -126,7 +126,7 @@ public:
|
||||
const auto & col = *src.column;
|
||||
|
||||
if (!checkAndGetColumn<ColumnVector<T>>(col))
|
||||
return 0;
|
||||
return false;
|
||||
|
||||
auto & result_data = result_column->getData();
|
||||
|
||||
@ -135,7 +135,7 @@ public:
|
||||
for (size_t i = 0; i < input_rows_count; ++i)
|
||||
result_data[i] = source_data[i];
|
||||
|
||||
return 1;
|
||||
return true;
|
||||
}
|
||||
|
||||
ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
|
||||
|
||||
@ -1,17 +0,0 @@
|
||||
#include <Functions/FunctionsDecimalArithmetics.h>
|
||||
#include <Functions/FunctionFactory.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
REGISTER_FUNCTION(DivideDecimals)
|
||||
{
|
||||
factory.registerFunction<FunctionsDecimalArithmetics<DivideDecimalsImpl>>(Documentation(
|
||||
"Decimal division with given precision. Slower than simple `divide`, but has controlled precision and no sound overflows"));
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(MultiplyDecimals)
|
||||
{
|
||||
factory.registerFunction<FunctionsDecimalArithmetics<MultiplyDecimalsImpl>>(Documentation(
|
||||
"Decimal multiplication with given precision. Slower than simple `divide`, but has controlled precision and no sound overflows"));
|
||||
}
|
||||
}
|
||||
@ -1,4 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include <type_traits>
|
||||
#include <Core/AccurateComparison.h>
|
||||
|
||||
@ -23,7 +24,6 @@ namespace ErrorCodes
|
||||
extern const int ILLEGAL_COLUMN;
|
||||
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
|
||||
extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
|
||||
extern const int ILLEGAL_DIVISION;
|
||||
}
|
||||
|
||||
|
||||
@ -140,91 +140,6 @@ struct DecimalOpHelpers
|
||||
};
|
||||
|
||||
|
||||
struct DivideDecimalsImpl
|
||||
{
|
||||
static constexpr auto name = "divideDecimal";
|
||||
|
||||
template <typename FirstType, typename SecondType>
|
||||
static inline Decimal256
|
||||
execute(FirstType a, SecondType b, UInt16 scale_a, UInt16 scale_b, UInt16 result_scale)
|
||||
{
|
||||
if (b.value == 0)
|
||||
throw DB::Exception("Division by zero", ErrorCodes::ILLEGAL_DIVISION);
|
||||
if (a.value == 0)
|
||||
return Decimal256(0);
|
||||
|
||||
Int256 sign_a = a.value < 0 ? -1 : 1;
|
||||
Int256 sign_b = b.value < 0 ? -1 : 1;
|
||||
|
||||
std::vector<UInt8> a_digits = DecimalOpHelpers::toDigits(a.value * sign_a);
|
||||
|
||||
while (scale_a < scale_b + result_scale)
|
||||
{
|
||||
a_digits.push_back(0);
|
||||
++scale_a;
|
||||
}
|
||||
|
||||
while (scale_a > scale_b + result_scale && !a_digits.empty())
|
||||
{
|
||||
a_digits.pop_back();
|
||||
--scale_a;
|
||||
}
|
||||
|
||||
if (a_digits.empty())
|
||||
return Decimal256(0);
|
||||
|
||||
std::vector<UInt8> divided = DecimalOpHelpers::divide(a_digits, b.value * sign_b);
|
||||
|
||||
if (divided.size() > DecimalUtils::max_precision<Decimal256>)
|
||||
throw DB::Exception("Numeric overflow: result bigger that Decimal256", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
return Decimal256(sign_a * sign_b * DecimalOpHelpers::fromDigits(divided));
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
struct MultiplyDecimalsImpl
|
||||
{
|
||||
static constexpr auto name = "multiplyDecimal";
|
||||
|
||||
template <typename FirstType, typename SecondType>
|
||||
static inline Decimal256
|
||||
execute(FirstType a, SecondType b, UInt16 scale_a, UInt16 scale_b, UInt16 result_scale)
|
||||
{
|
||||
if (a.value == 0 || b.value == 0)
|
||||
return Decimal256(0);
|
||||
|
||||
Int256 sign_a = a.value < 0 ? -1 : 1;
|
||||
Int256 sign_b = b.value < 0 ? -1 : 1;
|
||||
|
||||
std::vector<UInt8> a_digits = DecimalOpHelpers::toDigits(a.value * sign_a);
|
||||
std::vector<UInt8> b_digits = DecimalOpHelpers::toDigits(b.value * sign_b);
|
||||
|
||||
std::vector<UInt8> multiplied = DecimalOpHelpers::multiply(a_digits, b_digits);
|
||||
|
||||
UInt16 product_scale = scale_a + scale_b;
|
||||
while (product_scale < result_scale)
|
||||
{
|
||||
multiplied.push_back(0);
|
||||
++product_scale;
|
||||
}
|
||||
|
||||
while (product_scale > result_scale&& !multiplied.empty())
|
||||
{
|
||||
multiplied.pop_back();
|
||||
--product_scale;
|
||||
}
|
||||
|
||||
if (multiplied.empty())
|
||||
return Decimal256(0);
|
||||
|
||||
if (multiplied.size() > DecimalUtils::max_precision<Decimal256>)
|
||||
throw DB::Exception("Numeric overflow: result bigger that Decimal256", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
|
||||
return Decimal256(sign_a * sign_b * DecimalOpHelpers::fromDigits(multiplied));
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
template <typename ResultType, typename Transform>
|
||||
struct Processor
|
||||
{
|
||||
@ -388,11 +303,12 @@ public:
|
||||
}
|
||||
|
||||
private:
|
||||
//long resolver to call proper templated func
|
||||
// long resolver to call proper templated func
|
||||
ColumnPtr resolveOverload(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type) const
|
||||
{
|
||||
WhichDataType which_dividend(arguments[0].type.get());
|
||||
WhichDataType which_divisor(arguments[1].type.get());
|
||||
|
||||
if (which_dividend.isDecimal32())
|
||||
{
|
||||
using DividendType = DataTypeDecimal32;
|
||||
@ -454,4 +370,3 @@ private:
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
|
||||
@ -48,10 +48,6 @@ public:
|
||||
: scale_multiplier(DecimalUtils::scaleMultiplier<DateTime64::NativeType>(scale_))
|
||||
{}
|
||||
|
||||
TransformDateTime64(DateTime64::NativeType scale_multiplier_ = 1) /// NOLINT(google-explicit-constructor)
|
||||
: scale_multiplier(scale_multiplier_)
|
||||
{}
|
||||
|
||||
template <typename ... Args>
|
||||
inline auto NO_SANITIZE_UNDEFINED execute(const DateTime64 & t, Args && ... args) const
|
||||
{
|
||||
@ -131,8 +127,6 @@ public:
|
||||
return wrapped_transform.executeExtendedResult(t, std::forward<Args>(args)...);
|
||||
}
|
||||
|
||||
DateTime64::NativeType getScaleMultiplier() const { return scale_multiplier; }
|
||||
|
||||
private:
|
||||
DateTime64::NativeType scale_multiplier = 1;
|
||||
Transform wrapped_transform = {};
|
||||
|
||||
178
src/Functions/concatWithSeparator.cpp
Normal file
178
src/Functions/concatWithSeparator.cpp
Normal file
@ -0,0 +1,178 @@
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Columns/ColumnFixedString.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <Functions/FunctionFactory.h>
|
||||
#include <Functions/FunctionHelpers.h>
|
||||
#include <Functions/IFunction.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <base/map.h>
|
||||
#include <base/range.h>
|
||||
|
||||
#include "formatString.h"
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
|
||||
extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
|
||||
extern const int ILLEGAL_COLUMN;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
template <typename Name, bool is_injective>
|
||||
class ConcatWithSeparatorImpl : public IFunction
|
||||
{
|
||||
public:
|
||||
static constexpr auto name = Name::name;
|
||||
explicit ConcatWithSeparatorImpl(ContextPtr context_) : context(context_) {}
|
||||
|
||||
static FunctionPtr create(ContextPtr context) { return std::make_shared<ConcatWithSeparatorImpl>(context); }
|
||||
|
||||
String getName() const override { return name; }
|
||||
|
||||
bool isVariadic() const override { return true; }
|
||||
|
||||
size_t getNumberOfArguments() const override { return 0; }
|
||||
|
||||
bool isInjective(const ColumnsWithTypeAndName &) const override { return is_injective; }
|
||||
|
||||
bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo & /*arguments*/) const override { return true; }
|
||||
|
||||
DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
|
||||
{
|
||||
if (arguments.empty())
|
||||
throw Exception(
|
||||
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
|
||||
"Number of arguments for function {} doesn't match: passed {}, should be at least 1",
|
||||
getName(),
|
||||
arguments.size());
|
||||
|
||||
for (const auto arg_idx : collections::range(0, arguments.size()))
|
||||
{
|
||||
const auto * arg = arguments[arg_idx].get();
|
||||
if (!isStringOrFixedString(arg))
|
||||
throw Exception(
|
||||
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
|
||||
"Illegal type {} of argument {} of function {}",
|
||||
arg->getName(),
|
||||
arg_idx + 1,
|
||||
getName());
|
||||
}
|
||||
|
||||
return std::make_shared<DataTypeString>();
|
||||
}
|
||||
|
||||
ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & result_type, size_t input_rows_count) const override
|
||||
{
|
||||
assert(!arguments.empty());
|
||||
if (arguments.size() == 1)
|
||||
return result_type->createColumnConstWithDefaultValue(input_rows_count);
|
||||
|
||||
auto c_res = ColumnString::create();
|
||||
c_res->reserve(input_rows_count);
|
||||
const ColumnConst * col_sep = checkAndGetColumnConstStringOrFixedString(arguments[0].column.get());
|
||||
if (!col_sep)
|
||||
throw Exception(
|
||||
ErrorCodes::ILLEGAL_COLUMN,
|
||||
"Illegal column {} of first argument of function {}. Must be a constant String.",
|
||||
arguments[0].column->getName(),
|
||||
getName());
|
||||
String sep_str = col_sep->getValue<String>();
|
||||
|
||||
const size_t num_exprs = arguments.size() - 1;
|
||||
const size_t num_args = 2 * num_exprs - 1;
|
||||
|
||||
std::vector<const ColumnString::Chars *> data(num_args);
|
||||
std::vector<const ColumnString::Offsets *> offsets(num_args);
|
||||
std::vector<size_t> fixed_string_sizes(num_args);
|
||||
std::vector<std::optional<String>> constant_strings(num_args);
|
||||
|
||||
bool has_column_string = false;
|
||||
bool has_column_fixed_string = false;
|
||||
|
||||
for (size_t i = 0; i < num_exprs; ++i)
|
||||
{
|
||||
if (i != 0)
|
||||
constant_strings[2 * i - 1] = sep_str;
|
||||
|
||||
const ColumnPtr & column = arguments[i + 1].column;
|
||||
if (const ColumnString * col = checkAndGetColumn<ColumnString>(column.get()))
|
||||
{
|
||||
has_column_string = true;
|
||||
data[2 * i] = &col->getChars();
|
||||
offsets[2 * i] = &col->getOffsets();
|
||||
}
|
||||
else if (const ColumnFixedString * fixed_col = checkAndGetColumn<ColumnFixedString>(column.get()))
|
||||
{
|
||||
has_column_fixed_string = true;
|
||||
data[2 * i] = &fixed_col->getChars();
|
||||
fixed_string_sizes[2 * i] = fixed_col->getN();
|
||||
}
|
||||
else if (const ColumnConst * const_col = checkAndGetColumnConstStringOrFixedString(column.get()))
|
||||
constant_strings[2 * i] = const_col->getValue<String>();
|
||||
else
|
||||
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
|
||||
"Illegal column {} of argument of function {}", column->getName(), getName());
|
||||
}
|
||||
|
||||
String pattern;
|
||||
pattern.reserve(num_args * 2);
|
||||
for (size_t i = 0; i < num_args; ++i)
|
||||
pattern += "{}";
|
||||
|
||||
FormatImpl::formatExecute(
|
||||
has_column_string,
|
||||
has_column_fixed_string,
|
||||
std::move(pattern),
|
||||
data,
|
||||
offsets,
|
||||
fixed_string_sizes,
|
||||
constant_strings,
|
||||
c_res->getChars(),
|
||||
c_res->getOffsets(),
|
||||
input_rows_count);
|
||||
return std::move(c_res);
|
||||
}
|
||||
|
||||
private:
|
||||
ContextWeakPtr context;
|
||||
};
|
||||
|
||||
struct NameConcatWithSeparator
|
||||
{
|
||||
static constexpr auto name = "concatWithSeparator";
|
||||
};
|
||||
struct NameConcatWithSeparatorAssumeInjective
|
||||
{
|
||||
static constexpr auto name = "concatWithSeparatorAssumeInjective";
|
||||
};
|
||||
|
||||
using FunctionConcatWithSeparator = ConcatWithSeparatorImpl<NameConcatWithSeparator, false>;
|
||||
using FunctionConcatWithSeparatorAssumeInjective = ConcatWithSeparatorImpl<NameConcatWithSeparatorAssumeInjective, true>;
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(ConcatWithSeparator)
|
||||
{
|
||||
factory.registerFunction<FunctionConcatWithSeparator>({
|
||||
R"(
|
||||
Returns the concatenation strings separated by string separator. Syntax: concatWithSeparator(sep, expr1, expr2, expr3...)
|
||||
)",
|
||||
Documentation::Examples{{"concatWithSeparator", "SELECT concatWithSeparator('a', '1', '2', '3')"}},
|
||||
Documentation::Categories{"String"}});
|
||||
|
||||
factory.registerFunction<FunctionConcatWithSeparatorAssumeInjective>({
|
||||
R"(
|
||||
Same as concatWithSeparator, the difference is that you need to ensure that concatWithSeparator(sep, expr1, expr2, expr3...) → result is injective, it will be used for optimization of GROUP BY.
|
||||
|
||||
The function is named “injective” if it always returns different result for different values of arguments. In other words: different arguments never yield identical result.
|
||||
)",
|
||||
Documentation::Examples{{"concatWithSeparatorAssumeInjective", "SELECT concatWithSeparatorAssumeInjective('a', '1', '2', '3')"}},
|
||||
Documentation::Categories{"String"}});
|
||||
|
||||
/// Compatibility with Spark:
|
||||
factory.registerAlias("concat_ws", "concatWithSeparator", FunctionFactory::CaseInsensitive);
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,7 +1,6 @@
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
#include <DataTypes/DataTypeDateTime64.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <Common/IntervalKind.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Columns/ColumnsDateTime.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
@ -35,7 +34,6 @@ namespace ErrorCodes
|
||||
namespace
|
||||
{
|
||||
|
||||
template <bool is_diff>
|
||||
class DateDiffImpl
|
||||
{
|
||||
public:
|
||||
@ -167,92 +165,8 @@ public:
|
||||
template <typename TransformX, typename TransformY, typename T1, typename T2>
|
||||
Int64 calculate(const TransformX & transform_x, const TransformY & transform_y, T1 x, T2 y, const DateLUTImpl & timezone_x, const DateLUTImpl & timezone_y) const
|
||||
{
|
||||
if constexpr (is_diff)
|
||||
return static_cast<Int64>(transform_y.execute(y, timezone_y))
|
||||
return static_cast<Int64>(transform_y.execute(y, timezone_y))
|
||||
- static_cast<Int64>(transform_x.execute(x, timezone_x));
|
||||
else
|
||||
{
|
||||
auto res = static_cast<Int64>(transform_y.execute(y, timezone_y))
|
||||
- static_cast<Int64>(transform_x.execute(x, timezone_x));
|
||||
DateLUTImpl::DateTimeComponents a_comp;
|
||||
DateLUTImpl::DateTimeComponents b_comp;
|
||||
Int64 adjust_value;
|
||||
auto x_seconds = TransformDateTime64<ToRelativeSecondNumImpl<ResultPrecision::Extended>>(transform_x.getScaleMultiplier()).execute(x, timezone_x);
|
||||
auto y_seconds = TransformDateTime64<ToRelativeSecondNumImpl<ResultPrecision::Extended>>(transform_y.getScaleMultiplier()).execute(y, timezone_y);
|
||||
if (x_seconds <= y_seconds)
|
||||
{
|
||||
a_comp = TransformDateTime64<ToDateTimeComponentsImpl>(transform_x.getScaleMultiplier()).execute(x, timezone_x);
|
||||
b_comp = TransformDateTime64<ToDateTimeComponentsImpl>(transform_y.getScaleMultiplier()).execute(y, timezone_y);
|
||||
adjust_value = -1;
|
||||
}
|
||||
else
|
||||
{
|
||||
a_comp = TransformDateTime64<ToDateTimeComponentsImpl>(transform_y.getScaleMultiplier()).execute(y, timezone_y);
|
||||
b_comp = TransformDateTime64<ToDateTimeComponentsImpl>(transform_x.getScaleMultiplier()).execute(x, timezone_x);
|
||||
adjust_value = 1;
|
||||
}
|
||||
|
||||
if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeYearNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
if ((a_comp.date.month > b_comp.date.month)
|
||||
|| ((a_comp.date.month == b_comp.date.month) && ((a_comp.date.day > b_comp.date.day)
|
||||
|| ((a_comp.date.day == b_comp.date.day) && ((a_comp.time.hour > b_comp.time.hour)
|
||||
|| ((a_comp.time.hour == b_comp.time.hour) && ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second))))
|
||||
)))))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeQuarterNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
auto x_month_in_quarter = (a_comp.date.month - 1) % 3;
|
||||
auto y_month_in_quarter = (b_comp.date.month - 1) % 3;
|
||||
if ((x_month_in_quarter > y_month_in_quarter)
|
||||
|| ((x_month_in_quarter == y_month_in_quarter) && ((a_comp.date.day > b_comp.date.day)
|
||||
|| ((a_comp.date.day == b_comp.date.day) && ((a_comp.time.hour > b_comp.time.hour)
|
||||
|| ((a_comp.time.hour == b_comp.time.hour) && ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second))))
|
||||
)))))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeMonthNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
if ((a_comp.date.day > b_comp.date.day)
|
||||
|| ((a_comp.date.day == b_comp.date.day) && ((a_comp.time.hour > b_comp.time.hour)
|
||||
|| ((a_comp.time.hour == b_comp.time.hour) && ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second))))
|
||||
)))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeWeekNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
auto x_day_of_week = TransformDateTime64<ToDayOfWeekImpl>(transform_x.getScaleMultiplier()).execute(x, timezone_x);
|
||||
auto y_day_of_week = TransformDateTime64<ToDayOfWeekImpl>(transform_y.getScaleMultiplier()).execute(y, timezone_y);
|
||||
if ((x_day_of_week > y_day_of_week)
|
||||
|| ((x_day_of_week == y_day_of_week) && (a_comp.time.hour > b_comp.time.hour))
|
||||
|| ((a_comp.time.hour == b_comp.time.hour) && ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second)))))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeDayNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
if ((a_comp.time.hour > b_comp.time.hour)
|
||||
|| ((a_comp.time.hour == b_comp.time.hour) && ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second)))))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeHourNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
if ((a_comp.time.minute > b_comp.time.minute)
|
||||
|| ((a_comp.time.minute == b_comp.time.minute) && (a_comp.time.second > b_comp.time.second)))
|
||||
res += adjust_value;
|
||||
}
|
||||
else if constexpr (std::is_same_v<TransformX, TransformDateTime64<ToRelativeMinuteNumImpl<ResultPrecision::Extended>>>)
|
||||
{
|
||||
if (a_comp.time.second > b_comp.time.second)
|
||||
res += adjust_value;
|
||||
}
|
||||
return res;
|
||||
}
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
@ -279,8 +193,7 @@ private:
|
||||
|
||||
|
||||
/** dateDiff('unit', t1, t2, [timezone])
|
||||
* age('unit', t1, t2, [timezone])
|
||||
* t1 and t2 can be Date, Date32, DateTime or DateTime64
|
||||
* t1 and t2 can be Date or DateTime
|
||||
*
|
||||
* If timezone is specified, it applied to both arguments.
|
||||
* If not, timezones from datatypes t1 and t2 are used.
|
||||
@ -288,11 +201,10 @@ private:
|
||||
*
|
||||
* Timezone matters because days can have different length.
|
||||
*/
|
||||
template <bool is_relative>
|
||||
class FunctionDateDiff : public IFunction
|
||||
{
|
||||
public:
|
||||
static constexpr auto name = is_relative ? "dateDiff" : "age";
|
||||
static constexpr auto name = "dateDiff";
|
||||
static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionDateDiff>(); }
|
||||
|
||||
String getName() const override
|
||||
@ -358,21 +270,21 @@ public:
|
||||
const auto & timezone_y = extractTimeZoneFromFunctionArguments(arguments, 3, 2);
|
||||
|
||||
if (unit == "year" || unit == "yy" || unit == "yyyy")
|
||||
impl.template dispatchForColumns<ToRelativeYearNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeYearNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "quarter" || unit == "qq" || unit == "q")
|
||||
impl.template dispatchForColumns<ToRelativeQuarterNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeQuarterNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "month" || unit == "mm" || unit == "m")
|
||||
impl.template dispatchForColumns<ToRelativeMonthNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeMonthNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "week" || unit == "wk" || unit == "ww")
|
||||
impl.template dispatchForColumns<ToRelativeWeekNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeWeekNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "day" || unit == "dd" || unit == "d")
|
||||
impl.template dispatchForColumns<ToRelativeDayNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeDayNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "hour" || unit == "hh" || unit == "h")
|
||||
impl.template dispatchForColumns<ToRelativeHourNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeHourNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "minute" || unit == "mi" || unit == "n")
|
||||
impl.template dispatchForColumns<ToRelativeMinuteNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeMinuteNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else if (unit == "second" || unit == "ss" || unit == "s")
|
||||
impl.template dispatchForColumns<ToRelativeSecondNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
impl.dispatchForColumns<ToRelativeSecondNumImpl<ResultPrecision::Extended>>(x, y, timezone_x, timezone_y, res->getData());
|
||||
else
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS,
|
||||
"Function {} does not support '{}' unit", getName(), unit);
|
||||
@ -380,7 +292,7 @@ public:
|
||||
return res;
|
||||
}
|
||||
private:
|
||||
DateDiffImpl<is_relative> impl{name};
|
||||
DateDiffImpl impl{name};
|
||||
};
|
||||
|
||||
|
||||
@ -440,14 +352,14 @@ public:
|
||||
return res;
|
||||
}
|
||||
private:
|
||||
DateDiffImpl<true> impl{name};
|
||||
DateDiffImpl impl{name};
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(DateDiff)
|
||||
{
|
||||
factory.registerFunction<FunctionDateDiff<true>>({}, FunctionFactory::CaseInsensitive);
|
||||
factory.registerFunction<FunctionDateDiff>({}, FunctionFactory::CaseInsensitive);
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(TimeDiff)
|
||||
@ -464,9 +376,4 @@ Example:
|
||||
Documentation::Categories{"Dates and Times"}}, FunctionFactory::CaseInsensitive);
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(Age)
|
||||
{
|
||||
factory.registerFunction<FunctionDateDiff<false>>({}, FunctionFactory::CaseInsensitive);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
126
src/Functions/divideDecimal.cpp
Normal file
126
src/Functions/divideDecimal.cpp
Normal file
@ -0,0 +1,126 @@
|
||||
#include <Functions/FunctionsDecimalArithmetics.h>
|
||||
#include <Functions/FunctionFactory.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int DECIMAL_OVERFLOW;
|
||||
extern const int ILLEGAL_DIVISION;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
struct DivideDecimalsImpl
|
||||
{
|
||||
static constexpr auto name = "divideDecimal";
|
||||
|
||||
template <typename FirstType, typename SecondType>
|
||||
static inline Decimal256
|
||||
execute(FirstType a, SecondType b, UInt16 scale_a, UInt16 scale_b, UInt16 result_scale)
|
||||
{
|
||||
if (b.value == 0)
|
||||
throw DB::Exception("Division by zero", ErrorCodes::ILLEGAL_DIVISION);
|
||||
if (a.value == 0)
|
||||
return Decimal256(0);
|
||||
|
||||
Int256 sign_a = a.value < 0 ? -1 : 1;
|
||||
Int256 sign_b = b.value < 0 ? -1 : 1;
|
||||
|
||||
std::vector<UInt8> a_digits = DecimalOpHelpers::toDigits(a.value * sign_a);
|
||||
|
||||
while (scale_a < scale_b + result_scale)
|
||||
{
|
||||
a_digits.push_back(0);
|
||||
++scale_a;
|
||||
}
|
||||
|
||||
while (scale_a > scale_b + result_scale && !a_digits.empty())
|
||||
{
|
||||
a_digits.pop_back();
|
||||
--scale_a;
|
||||
}
|
||||
|
||||
if (a_digits.empty())
|
||||
return Decimal256(0);
|
||||
|
||||
std::vector<UInt8> divided = DecimalOpHelpers::divide(a_digits, b.value * sign_b);
|
||||
|
||||
if (divided.size() > DecimalUtils::max_precision<Decimal256>)
|
||||
throw DB::Exception("Numeric overflow: result bigger that Decimal256", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
return Decimal256(sign_a * sign_b * DecimalOpHelpers::fromDigits(divided));
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(DivideDecimals)
|
||||
{
|
||||
factory.registerFunction<FunctionsDecimalArithmetics<DivideDecimalsImpl>>(Documentation(
|
||||
R"(
|
||||
Performs division on two decimals. Result value will be of type [Decimal256](../../sql-reference/data-types/decimal.md).
|
||||
Result scale can be explicitly specified by `result_scale` argument (const Integer in range `[0, 76]`). If not specified, the result scale is the max scale of given arguments.
|
||||
|
||||
:::note
|
||||
These function work significantly slower than usual `divide`.
|
||||
In case you don't really need controlled precision and/or need fast computation, consider using [divide](#divide).
|
||||
:::
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
divideDecimal(a, b[, result_scale])
|
||||
```
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `a` — First value: [Decimal](../../sql-reference/data-types/decimal.md).
|
||||
- `b` — Second value: [Decimal](../../sql-reference/data-types/decimal.md).
|
||||
- `result_scale` — Scale of result: [Int/UInt](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
- The result of division with given scale.
|
||||
|
||||
Type: [Decimal256](../../sql-reference/data-types/decimal.md).
|
||||
|
||||
**Example**
|
||||
|
||||
```text
|
||||
┌─divideDecimal(toDecimal256(-12, 0), toDecimal32(2.1, 1), 10)─┐
|
||||
│ -5.7142857142 │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Difference from regular division:**
|
||||
```sql
|
||||
SELECT toDecimal64(-12, 1) / toDecimal32(2.1, 1);
|
||||
SELECT toDecimal64(-12, 1) as a, toDecimal32(2.1, 1) as b, divideDecimal(a, b, 1), divideDecimal(a, b, 5);
|
||||
```
|
||||
|
||||
```text
|
||||
┌─divide(toDecimal64(-12, 1), toDecimal32(2.1, 1))─┐
|
||||
│ -5.7 │
|
||||
└──────────────────────────────────────────────────┘
|
||||
┌───a─┬───b─┬─divideDecimal(toDecimal64(-12, 1), toDecimal32(2.1, 1), 1)─┬─divideDecimal(toDecimal64(-12, 1), toDecimal32(2.1, 1), 5)─┐
|
||||
│ -12 │ 2.1 │ -5.7 │ -5.71428 │
|
||||
└─────┴─────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT toDecimal64(-12, 0) / toDecimal32(2.1, 1);
|
||||
SELECT toDecimal64(-12, 0) as a, toDecimal32(2.1, 1) as b, divideDecimal(a, b, 1), divideDecimal(a, b, 5);
|
||||
```
|
||||
|
||||
```text
|
||||
DB::Exception: Decimal result's scale is less than argument's one: While processing toDecimal64(-12, 0) / toDecimal32(2.1, 1). (ARGUMENT_OUT_OF_BOUND)
|
||||
┌───a─┬───b─┬─divideDecimal(toDecimal64(-12, 0), toDecimal32(2.1, 1), 1)─┬─divideDecimal(toDecimal64(-12, 0), toDecimal32(2.1, 1), 5)─┐
|
||||
│ -12 │ 2.1 │ -5.7 │ -5.71428 │
|
||||
└─────┴─────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
)"));
|
||||
}
|
||||
|
||||
}
|
||||
134
src/Functions/multiplyDecimal.cpp
Normal file
134
src/Functions/multiplyDecimal.cpp
Normal file
@ -0,0 +1,134 @@
|
||||
#include <Functions/FunctionsDecimalArithmetics.h>
|
||||
#include <Functions/FunctionFactory.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int DECIMAL_OVERFLOW;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
struct MultiplyDecimalsImpl
|
||||
{
|
||||
static constexpr auto name = "multiplyDecimal";
|
||||
|
||||
template <typename FirstType, typename SecondType>
|
||||
static inline Decimal256
|
||||
execute(FirstType a, SecondType b, UInt16 scale_a, UInt16 scale_b, UInt16 result_scale)
|
||||
{
|
||||
if (a.value == 0 || b.value == 0)
|
||||
return Decimal256(0);
|
||||
|
||||
Int256 sign_a = a.value < 0 ? -1 : 1;
|
||||
Int256 sign_b = b.value < 0 ? -1 : 1;
|
||||
|
||||
std::vector<UInt8> a_digits = DecimalOpHelpers::toDigits(a.value * sign_a);
|
||||
std::vector<UInt8> b_digits = DecimalOpHelpers::toDigits(b.value * sign_b);
|
||||
|
||||
std::vector<UInt8> multiplied = DecimalOpHelpers::multiply(a_digits, b_digits);
|
||||
|
||||
UInt16 product_scale = scale_a + scale_b;
|
||||
while (product_scale < result_scale)
|
||||
{
|
||||
multiplied.push_back(0);
|
||||
++product_scale;
|
||||
}
|
||||
|
||||
while (product_scale > result_scale&& !multiplied.empty())
|
||||
{
|
||||
multiplied.pop_back();
|
||||
--product_scale;
|
||||
}
|
||||
|
||||
if (multiplied.empty())
|
||||
return Decimal256(0);
|
||||
|
||||
if (multiplied.size() > DecimalUtils::max_precision<Decimal256>)
|
||||
throw DB::Exception("Numeric overflow: result bigger that Decimal256", ErrorCodes::DECIMAL_OVERFLOW);
|
||||
|
||||
return Decimal256(sign_a * sign_b * DecimalOpHelpers::fromDigits(multiplied));
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
REGISTER_FUNCTION(MultiplyDecimals)
|
||||
{
|
||||
factory.registerFunction<FunctionsDecimalArithmetics<MultiplyDecimalsImpl>>(Documentation(
|
||||
R"(
|
||||
Performs multiplication on two decimals. Result value will be of type [Decimal256](../../sql-reference/data-types/decimal.md).
|
||||
Result scale can be explicitly specified by `result_scale` argument (const Integer in range `[0, 76]`). If not specified, the result scale is the max scale of given arguments.
|
||||
|
||||
:::note
|
||||
These functions work significantly slower than usual `multiply`.
|
||||
In case you don't really need controlled precision and/or need fast computation, consider using [multiply](#multiply)
|
||||
:::
|
||||
|
||||
**Syntax**
|
||||
|
||||
```sql
|
||||
multiplyDecimal(a, b[, result_scale])
|
||||
```
|
||||
|
||||
**Arguments**
|
||||
|
||||
- `a` — First value: [Decimal](../../sql-reference/data-types/decimal.md).
|
||||
- `b` — Second value: [Decimal](../../sql-reference/data-types/decimal.md).
|
||||
- `result_scale` — Scale of result: [Int/UInt](../../sql-reference/data-types/int-uint.md).
|
||||
|
||||
**Returned value**
|
||||
|
||||
- The result of multiplication with given scale.
|
||||
|
||||
Type: [Decimal256](../../sql-reference/data-types/decimal.md).
|
||||
|
||||
**Example**
|
||||
|
||||
```text
|
||||
┌─multiplyDecimal(toDecimal256(-12, 0), toDecimal32(-2.1, 1), 1)─┐
|
||||
│ 25.2 │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Difference from regular multiplication:**
|
||||
```sql
|
||||
SELECT toDecimal64(-12.647, 3) * toDecimal32(2.1239, 4);
|
||||
SELECT toDecimal64(-12.647, 3) as a, toDecimal32(2.1239, 4) as b, multiplyDecimal(a, b);
|
||||
```
|
||||
|
||||
```text
|
||||
┌─multiply(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
|
||||
│ -26.8609633 │
|
||||
└───────────────────────────────────────────────────────────┘
|
||||
┌─multiplyDecimal(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
|
||||
│ -26.8609 │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
toDecimal64(-12.647987876, 9) AS a,
|
||||
toDecimal64(123.967645643, 9) AS b,
|
||||
multiplyDecimal(a, b);
|
||||
SELECT
|
||||
toDecimal64(-12.647987876, 9) AS a,
|
||||
toDecimal64(123.967645643, 9) AS b,
|
||||
a * b;
|
||||
```
|
||||
|
||||
```text
|
||||
┌─────────────a─┬─────────────b─┬─multiplyDecimal(toDecimal64(-12.647987876, 9), toDecimal64(123.967645643, 9))─┐
|
||||
│ -12.647987876 │ 123.967645643 │ -1567.941279108 │
|
||||
└───────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────┘
|
||||
Received exception from server (version 22.11.1):
|
||||
Code: 407. DB::Exception: Received from localhost:9000. DB::Exception: Decimal math overflow: While processing toDecimal64(-12.647987876, 9) AS a, toDecimal64(123.967645643, 9) AS b, a * b. (DECIMAL_OVERFLOW)
|
||||
```
|
||||
)"));
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -108,6 +108,12 @@ BlockIO InterpreterCreateUserQuery::execute()
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS,
|
||||
"Authentication type NO_PASSWORD must be explicitly specified, check the setting allow_implicit_no_password in the server configuration");
|
||||
|
||||
if (!query.attach && query.temporary_password_for_checks)
|
||||
{
|
||||
access_control.checkPasswordComplexityRules(query.temporary_password_for_checks.value());
|
||||
query.temporary_password_for_checks.reset();
|
||||
}
|
||||
|
||||
std::optional<RolesOrUsersSet> default_roles_from_query;
|
||||
if (query.default_roles)
|
||||
{
|
||||
|
||||
@ -141,7 +141,7 @@ void ConvertStringsToEnumMatcher::visit(ASTFunction & function_node, Data & data
|
||||
|
||||
if (function_node.name == "if")
|
||||
{
|
||||
if (function_node.arguments->children.size() != 2)
|
||||
if (function_node.arguments->children.size() != 3)
|
||||
return;
|
||||
|
||||
const ASTLiteral * literal1 = function_node.arguments->children[1]->as<ASTLiteral>();
|
||||
|
||||
@ -1126,6 +1126,7 @@ void DatabaseCatalog::cleanupStoreDirectoryTask()
|
||||
continue;
|
||||
|
||||
size_t affected_dirs = 0;
|
||||
size_t checked_dirs = 0;
|
||||
for (auto it = disk->iterateDirectory("store"); it->isValid(); it->next())
|
||||
{
|
||||
String prefix = it->name();
|
||||
@ -1135,6 +1136,7 @@ void DatabaseCatalog::cleanupStoreDirectoryTask()
|
||||
if (!expected_prefix_dir)
|
||||
{
|
||||
LOG_WARNING(log, "Found invalid directory {} on disk {}, will try to remove it", it->path(), disk_name);
|
||||
checked_dirs += 1;
|
||||
affected_dirs += maybeRemoveDirectory(disk_name, disk, it->path());
|
||||
continue;
|
||||
}
|
||||
@ -1150,6 +1152,7 @@ void DatabaseCatalog::cleanupStoreDirectoryTask()
|
||||
if (!expected_dir)
|
||||
{
|
||||
LOG_WARNING(log, "Found invalid directory {} on disk {}, will try to remove it", jt->path(), disk_name);
|
||||
checked_dirs += 1;
|
||||
affected_dirs += maybeRemoveDirectory(disk_name, disk, jt->path());
|
||||
continue;
|
||||
}
|
||||
@ -1161,6 +1164,7 @@ void DatabaseCatalog::cleanupStoreDirectoryTask()
|
||||
/// so it looks safe enough to remove directory if we don't have uuid mapping for it.
|
||||
/// No table or database using this directory should concurrently appear,
|
||||
/// because creation of new table would fail with "directory already exists".
|
||||
checked_dirs += 1;
|
||||
affected_dirs += maybeRemoveDirectory(disk_name, disk, jt->path());
|
||||
}
|
||||
}
|
||||
@ -1168,7 +1172,7 @@ void DatabaseCatalog::cleanupStoreDirectoryTask()
|
||||
|
||||
if (affected_dirs)
|
||||
LOG_INFO(log, "Cleaned up {} directories from store/ on disk {}", affected_dirs, disk_name);
|
||||
else
|
||||
if (checked_dirs == 0)
|
||||
LOG_TEST(log, "Nothing to clean up from store/ on disk {}", disk_name);
|
||||
}
|
||||
|
||||
|
||||
@ -225,7 +225,7 @@ HashJoin::HashJoin(std::shared_ptr<TableJoin> table_join_, const Block & right_s
|
||||
, right_sample_block(right_sample_block_)
|
||||
, log(&Poco::Logger::get("HashJoin"))
|
||||
{
|
||||
LOG_DEBUG(log, "Datatype: {}, kind: {}, strictness: {}", data->type, kind, strictness);
|
||||
LOG_DEBUG(log, "Datatype: {}, kind: {}, strictness: {}, right header: {}", data->type, kind, strictness, right_sample_block.dumpStructure());
|
||||
LOG_DEBUG(log, "Keys: {}", TableJoin::formatClauses(table_join->getClauses(), true));
|
||||
|
||||
if (isCrossOrComma(kind))
|
||||
|
||||
@ -672,6 +672,11 @@ String TableJoin::renamedRightColumnName(const String & name) const
|
||||
return name;
|
||||
}
|
||||
|
||||
void TableJoin::setRename(const String & from, const String & to)
|
||||
{
|
||||
renames[from] = to;
|
||||
}
|
||||
|
||||
void TableJoin::addKey(const String & left_name, const String & right_name, const ASTPtr & left_ast, const ASTPtr & right_ast)
|
||||
{
|
||||
clauses.back().key_names_left.emplace_back(left_name);
|
||||
|
||||
@ -334,6 +334,7 @@ public:
|
||||
Block getRequiredRightKeys(const Block & right_table_keys, std::vector<String> & keys_sources) const;
|
||||
|
||||
String renamedRightColumnName(const String & name) const;
|
||||
void setRename(const String & from, const String & to);
|
||||
|
||||
void resetKeys();
|
||||
void resetToCross();
|
||||
|
||||
@ -18,18 +18,31 @@ namespace ErrorCodes
|
||||
|
||||
|
||||
std::string getClusterName(const IAST & node)
|
||||
{
|
||||
auto name = tryGetClusterName(node);
|
||||
if (!name)
|
||||
throw Exception("Illegal expression instead of cluster name.", ErrorCodes::BAD_ARGUMENTS);
|
||||
return std::move(name).value();
|
||||
}
|
||||
|
||||
|
||||
std::optional<std::string> tryGetClusterName(const IAST & node)
|
||||
{
|
||||
if (const auto * ast_id = node.as<ASTIdentifier>())
|
||||
return ast_id->name();
|
||||
|
||||
if (const auto * ast_lit = node.as<ASTLiteral>())
|
||||
return checkAndGetLiteralArgument<String>(*ast_lit, "cluster_name");
|
||||
{
|
||||
if (ast_lit->value.getType() != Field::Types::String)
|
||||
return {};
|
||||
return ast_lit->value.safeGet<String>();
|
||||
}
|
||||
|
||||
/// A hack to support hyphens in cluster names.
|
||||
if (const auto * ast_func = node.as<ASTFunction>())
|
||||
{
|
||||
if (ast_func->name != "minus" || !ast_func->arguments || ast_func->arguments->children.size() < 2)
|
||||
throw Exception("Illegal expression instead of cluster name.", ErrorCodes::BAD_ARGUMENTS);
|
||||
return {};
|
||||
|
||||
String name;
|
||||
for (const auto & arg : ast_func->arguments->children)
|
||||
@ -43,7 +56,7 @@ std::string getClusterName(const IAST & node)
|
||||
return name;
|
||||
}
|
||||
|
||||
throw Exception("Illegal expression instead of cluster name.", ErrorCodes::BAD_ARGUMENTS);
|
||||
return {};
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -15,6 +15,7 @@ namespace DB
|
||||
* Therefore, consider this case separately.
|
||||
*/
|
||||
std::string getClusterName(const IAST & node);
|
||||
std::optional<std::string> tryGetClusterName(const IAST & node);
|
||||
|
||||
std::string getClusterNameAndMakeLiteral(ASTPtr & node);
|
||||
|
||||
|
||||
@ -46,6 +46,8 @@ public:
|
||||
|
||||
std::optional<AuthenticationData> auth_data;
|
||||
|
||||
mutable std::optional<String> temporary_password_for_checks;
|
||||
|
||||
std::optional<AllowedClientHosts> hosts;
|
||||
std::optional<AllowedClientHosts> add_hosts;
|
||||
std::optional<AllowedClientHosts> remove_hosts;
|
||||
|
||||
@ -51,7 +51,7 @@ namespace
|
||||
}
|
||||
|
||||
|
||||
bool parseAuthenticationData(IParserBase::Pos & pos, Expected & expected, AuthenticationData & auth_data)
|
||||
bool parseAuthenticationData(IParserBase::Pos & pos, Expected & expected, AuthenticationData & auth_data, std::optional<String> & temporary_password_for_checks)
|
||||
{
|
||||
return IParserBase::wrapParseImpl(pos, [&]
|
||||
{
|
||||
@ -165,6 +165,10 @@ namespace
|
||||
common_names.insert(ast_child->as<const ASTLiteral &>().value.safeGet<String>());
|
||||
}
|
||||
|
||||
/// Save password separately for future complexity rules check
|
||||
if (expect_password)
|
||||
temporary_password_for_checks = value;
|
||||
|
||||
auth_data = AuthenticationData{*type};
|
||||
if (auth_data.getType() == AuthenticationType::SHA256_PASSWORD)
|
||||
{
|
||||
@ -438,6 +442,7 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
|
||||
std::optional<String> new_name;
|
||||
std::optional<AuthenticationData> auth_data;
|
||||
std::optional<String> temporary_password_for_checks;
|
||||
std::optional<AllowedClientHosts> hosts;
|
||||
std::optional<AllowedClientHosts> add_hosts;
|
||||
std::optional<AllowedClientHosts> remove_hosts;
|
||||
@ -452,9 +457,11 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
if (!auth_data)
|
||||
{
|
||||
AuthenticationData new_auth_data;
|
||||
if (parseAuthenticationData(pos, expected, new_auth_data))
|
||||
std::optional<String> new_temporary_password_for_checks;
|
||||
if (parseAuthenticationData(pos, expected, new_auth_data, new_temporary_password_for_checks))
|
||||
{
|
||||
auth_data = std::move(new_auth_data);
|
||||
temporary_password_for_checks = std::move(new_temporary_password_for_checks);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
@ -539,6 +546,7 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
query->names = std::move(names);
|
||||
query->new_name = std::move(new_name);
|
||||
query->auth_data = std::move(auth_data);
|
||||
query->temporary_password_for_checks = std::move(temporary_password_for_checks);
|
||||
query->hosts = std::move(hosts);
|
||||
query->add_hosts = std::move(add_hosts);
|
||||
query->remove_hosts = std::move(remove_hosts);
|
||||
|
||||
@ -45,8 +45,9 @@ namespace DB
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LOGICAL_ERROR;
|
||||
extern const int INCOMPATIBLE_TYPE_OF_JOIN;
|
||||
extern const int INVALID_JOIN_ON_EXPRESSION;
|
||||
extern const int LOGICAL_ERROR;
|
||||
extern const int NOT_IMPLEMENTED;
|
||||
}
|
||||
|
||||
@ -671,9 +672,23 @@ std::shared_ptr<IJoin> chooseJoinAlgorithm(std::shared_ptr<TableJoin> & table_jo
|
||||
{
|
||||
trySetStorageInTableJoin(right_table_expression, table_join);
|
||||
|
||||
auto & right_table_expression_data = planner_context->getTableExpressionDataOrThrow(right_table_expression);
|
||||
|
||||
/// JOIN with JOIN engine.
|
||||
if (auto storage = table_join->getStorageJoin())
|
||||
{
|
||||
for (const auto & result_column : right_table_expression_header)
|
||||
{
|
||||
const auto * source_column_name = right_table_expression_data.getColumnNameOrNull(result_column.name);
|
||||
if (!source_column_name)
|
||||
throw Exception(ErrorCodes::INCOMPATIBLE_TYPE_OF_JOIN,
|
||||
"JOIN with 'Join' table engine should be performed by storage keys [{}], but column '{}' was found",
|
||||
fmt::join(storage->getKeyNames(), ", "), result_column.name);
|
||||
|
||||
table_join->setRename(*source_column_name, result_column.name);
|
||||
}
|
||||
return storage->getJoinLocked(table_join, planner_context->getQueryContext());
|
||||
}
|
||||
|
||||
/** JOIN with constant.
|
||||
* Example: SELECT * FROM test_table AS t1 INNER JOIN test_table AS t2 ON 1;
|
||||
|
||||
@ -36,6 +36,7 @@
|
||||
#include <Storages/MergeTree/MergeTreeDataPartUUID.h>
|
||||
#include <Storages/StorageS3Cluster.h>
|
||||
#include <Core/ExternalTable.h>
|
||||
#include <Access/AccessControl.h>
|
||||
#include <Access/Credentials.h>
|
||||
#include <Storages/ColumnDefault.h>
|
||||
#include <DataTypes/DataTypeLowCardinality.h>
|
||||
@ -1193,6 +1194,17 @@ void TCPHandler::sendHello()
|
||||
writeStringBinary(server_display_name, *out);
|
||||
if (client_tcp_protocol_version >= DBMS_MIN_REVISION_WITH_VERSION_PATCH)
|
||||
writeVarUInt(DBMS_VERSION_PATCH, *out);
|
||||
if (client_tcp_protocol_version >= DBMS_MIN_PROTOCOL_VERSION_WITH_PASSWORD_COMPLEXITY_RULES)
|
||||
{
|
||||
auto rules = server.context()->getAccessControl().getPasswordComplexityRules();
|
||||
|
||||
writeVarUInt(rules.size(), *out);
|
||||
for (const auto & [original_pattern, exception_message] : rules)
|
||||
{
|
||||
writeStringBinary(original_pattern, *out);
|
||||
writeStringBinary(exception_message, *out);
|
||||
}
|
||||
}
|
||||
out->next();
|
||||
}
|
||||
|
||||
|
||||
@ -794,8 +794,6 @@ void Fetcher::downloadBasePartOrProjectionPartToDiskRemoteMeta(
|
||||
/// NOTE The is_cancelled flag also makes sense to check every time you read over the network,
|
||||
/// performing a poll with a not very large timeout.
|
||||
/// And now we check it only between read chunks (in the `copyData` function).
|
||||
data_part_storage->removeSharedRecursive(true);
|
||||
data_part_storage->commitTransaction();
|
||||
throw Exception("Fetching of part was cancelled", ErrorCodes::ABORTED);
|
||||
}
|
||||
|
||||
@ -855,7 +853,6 @@ void Fetcher::downloadBaseOrProjectionPartToDisk(
|
||||
/// NOTE The is_cancelled flag also makes sense to check every time you read over the network,
|
||||
/// performing a poll with a not very large timeout.
|
||||
/// And now we check it only between read chunks (in the `copyData` function).
|
||||
data_part_storage->removeRecursive();
|
||||
throw Exception("Fetching of part was cancelled", ErrorCodes::ABORTED);
|
||||
}
|
||||
|
||||
@ -934,22 +931,36 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk(
|
||||
|
||||
CurrentMetrics::Increment metric_increment{CurrentMetrics::ReplicatedFetch};
|
||||
|
||||
for (size_t i = 0; i < projections; ++i)
|
||||
try
|
||||
{
|
||||
String projection_name;
|
||||
readStringBinary(projection_name, in);
|
||||
MergeTreeData::DataPart::Checksums projection_checksum;
|
||||
for (size_t i = 0; i < projections; ++i)
|
||||
{
|
||||
String projection_name;
|
||||
readStringBinary(projection_name, in);
|
||||
MergeTreeData::DataPart::Checksums projection_checksum;
|
||||
|
||||
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
|
||||
projection_part_storage->createDirectories();
|
||||
downloadBaseOrProjectionPartToDisk(
|
||||
replica_path, projection_part_storage, sync, in, projection_checksum, throttler);
|
||||
checksums.addFile(
|
||||
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
|
||||
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
|
||||
projection_part_storage->createDirectories();
|
||||
downloadBaseOrProjectionPartToDisk(
|
||||
replica_path, projection_part_storage, sync, in, projection_checksum, throttler);
|
||||
checksums.addFile(
|
||||
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
|
||||
}
|
||||
|
||||
// Download the base part
|
||||
downloadBaseOrProjectionPartToDisk(replica_path, data_part_storage, sync, in, checksums, throttler);
|
||||
}
|
||||
catch (const Exception & e)
|
||||
{
|
||||
/// Remove the whole part directory if fetch of base
|
||||
/// part or fetch of any projection was stopped.
|
||||
if (e.code() == ErrorCodes::ABORTED)
|
||||
{
|
||||
data_part_storage->removeRecursive();
|
||||
data_part_storage->commitTransaction();
|
||||
}
|
||||
throw;
|
||||
}
|
||||
|
||||
// Download the base part
|
||||
downloadBaseOrProjectionPartToDisk(replica_path, data_part_storage, sync, in, checksums, throttler);
|
||||
|
||||
assertEOF(in);
|
||||
data_part_storage->commitTransaction();
|
||||
@ -1007,23 +1018,37 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDiskRemoteMeta(
|
||||
|
||||
data_part_storage->createDirectories();
|
||||
|
||||
for (size_t i = 0; i < projections; ++i)
|
||||
try
|
||||
{
|
||||
String projection_name;
|
||||
readStringBinary(projection_name, in);
|
||||
MergeTreeData::DataPart::Checksums projection_checksum;
|
||||
for (size_t i = 0; i < projections; ++i)
|
||||
{
|
||||
String projection_name;
|
||||
readStringBinary(projection_name, in);
|
||||
MergeTreeData::DataPart::Checksums projection_checksum;
|
||||
|
||||
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
|
||||
projection_part_storage->createDirectories();
|
||||
downloadBasePartOrProjectionPartToDiskRemoteMeta(
|
||||
replica_path, projection_part_storage, in, projection_checksum, throttler);
|
||||
|
||||
checksums.addFile(
|
||||
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
|
||||
}
|
||||
|
||||
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
|
||||
projection_part_storage->createDirectories();
|
||||
downloadBasePartOrProjectionPartToDiskRemoteMeta(
|
||||
replica_path, projection_part_storage, in, projection_checksum, throttler);
|
||||
|
||||
checksums.addFile(
|
||||
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
|
||||
replica_path, data_part_storage, in, checksums, throttler);
|
||||
}
|
||||
catch (const Exception & e)
|
||||
{
|
||||
if (e.code() == ErrorCodes::ABORTED)
|
||||
{
|
||||
/// Remove the whole part directory if fetch of base
|
||||
/// part or fetch of any projection was stopped.
|
||||
data_part_storage->removeSharedRecursive(true);
|
||||
data_part_storage->commitTransaction();
|
||||
}
|
||||
throw;
|
||||
}
|
||||
|
||||
downloadBasePartOrProjectionPartToDiskRemoteMeta(
|
||||
replica_path, data_part_storage, in, checksums, throttler);
|
||||
|
||||
assertEOF(in);
|
||||
MergeTreeData::MutableDataPartPtr new_data_part;
|
||||
|
||||
@ -2600,7 +2600,17 @@ void MergeTreeData::checkAlterIsPossible(const AlterCommands & commands, Context
|
||||
}
|
||||
}
|
||||
|
||||
dropped_columns.emplace(command.column_name);
|
||||
if (old_metadata.columns.has(command.column_name))
|
||||
{
|
||||
dropped_columns.emplace(command.column_name);
|
||||
}
|
||||
else
|
||||
{
|
||||
const auto & nested = old_metadata.columns.getNested(command.column_name);
|
||||
for (const auto & nested_column : nested)
|
||||
dropped_columns.emplace(nested_column.name);
|
||||
}
|
||||
|
||||
}
|
||||
else if (command.type == AlterCommand::RESET_SETTING)
|
||||
{
|
||||
@ -3884,9 +3894,9 @@ MergeTreeData::DataPartsVector MergeTreeData::getVisibleDataPartsVectorInPartiti
|
||||
return res;
|
||||
}
|
||||
|
||||
MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const MergeTreePartInfo & part_info, const MergeTreeData::DataPartStates & valid_states)
|
||||
MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const MergeTreePartInfo & part_info, const MergeTreeData::DataPartStates & valid_states, DataPartsLock * acquired_lock)
|
||||
{
|
||||
auto lock = lockParts();
|
||||
auto lock = (acquired_lock) ? DataPartsLock() : lockParts();
|
||||
|
||||
auto it = data_parts_by_info.find(part_info);
|
||||
if (it == data_parts_by_info.end())
|
||||
@ -3899,9 +3909,9 @@ MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const MergeTreePartInf
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const String & part_name, const MergeTreeData::DataPartStates & valid_states)
|
||||
MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const String & part_name, const MergeTreeData::DataPartStates & valid_states, DataPartsLock * acquired_lock)
|
||||
{
|
||||
return getPartIfExists(MergeTreePartInfo::fromPartName(part_name, format_version), valid_states);
|
||||
return getPartIfExists(MergeTreePartInfo::fromPartName(part_name, format_version), valid_states, acquired_lock);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -514,8 +514,8 @@ public:
|
||||
DataPartsVector getDataPartsVectorInPartitionForInternalUsage(const DataPartStates & affordable_states, const String & partition_id, DataPartsLock * acquired_lock = nullptr) const;
|
||||
|
||||
/// Returns the part with the given name and state or nullptr if no such part.
|
||||
DataPartPtr getPartIfExists(const String & part_name, const DataPartStates & valid_states);
|
||||
DataPartPtr getPartIfExists(const MergeTreePartInfo & part_info, const DataPartStates & valid_states);
|
||||
DataPartPtr getPartIfExists(const String & part_name, const DataPartStates & valid_states, DataPartsLock * acquired_lock = nullptr);
|
||||
DataPartPtr getPartIfExists(const MergeTreePartInfo & part_info, const DataPartStates & valid_states, DataPartsLock * acquired_lock = nullptr);
|
||||
|
||||
/// Total size of active parts in bytes.
|
||||
size_t getTotalActiveSizeInBytes() const;
|
||||
|
||||
@ -172,7 +172,7 @@ ColumnWithTypeAndName RPNBuilderTreeNode::getConstantColumn() const
|
||||
|
||||
if (ast_node)
|
||||
{
|
||||
const auto * literal = assert_cast<const ASTLiteral *>(ast_node);
|
||||
const auto * literal = typeid_cast<const ASTLiteral *>(ast_node);
|
||||
if (literal)
|
||||
{
|
||||
result.type = applyVisitor(FieldToDataType(), literal->value);
|
||||
|
||||
@ -1193,7 +1193,7 @@ bool ReplicatedMergeTreeQueue::isCoveredByFuturePartsImpl(const LogEntry & entry
|
||||
const LogEntry & another_entry = *entry_for_same_part_it->second;
|
||||
out_reason = fmt::format(
|
||||
"Not executing log entry {} of type {} for part {} "
|
||||
"because another log entry {} of type {} for the same part ({}) is being processed. This shouldn't happen often.",
|
||||
"because another log entry {} of type {} for the same part ({}) is being processed.",
|
||||
entry.znode_name, entry.type, entry.new_part_name,
|
||||
another_entry.znode_name, another_entry.type, another_entry.new_part_name);
|
||||
LOG_INFO(log, fmt::runtime(out_reason));
|
||||
|
||||
@ -386,8 +386,13 @@ void ReplicatedMergeTreeRestartingThread::setReadonly(bool on_shutdown)
|
||||
CurrentMetrics::add(CurrentMetrics::ReadonlyReplica);
|
||||
|
||||
/// Replica was already readonly, but we should decrement the metric, because we are detaching/dropping table.
|
||||
if (on_shutdown)
|
||||
/// if first pass wasn't done we don't have to decrement because it wasn't incremented in the first place
|
||||
/// the task should be deactivated if it's full shutdown so no race is present
|
||||
if (!first_time && on_shutdown)
|
||||
{
|
||||
CurrentMetrics::sub(CurrentMetrics::ReadonlyReplica);
|
||||
assert(CurrentMetrics::get(CurrentMetrics::ReadonlyReplica) >= 0);
|
||||
}
|
||||
}
|
||||
|
||||
void ReplicatedMergeTreeRestartingThread::setNotReadonly()
|
||||
@ -397,7 +402,10 @@ void ReplicatedMergeTreeRestartingThread::setNotReadonly()
|
||||
/// because we don't want to change this metric if replication is started successfully.
|
||||
/// So we should not decrement it when replica stopped being readonly on startup.
|
||||
if (storage.is_readonly.compare_exchange_strong(old_val, false) && !first_time)
|
||||
{
|
||||
CurrentMetrics::sub(CurrentMetrics::ReadonlyReplica);
|
||||
assert(CurrentMetrics::get(CurrentMetrics::ReadonlyReplica) >= 0);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -174,6 +174,9 @@ HashJoinPtr StorageJoin::getJoinLocked(std::shared_ptr<TableJoin> analyzed_join,
|
||||
"Table {} needs the same join_use_nulls setting as present in LEFT or FULL JOIN",
|
||||
getStorageID().getNameForLogs());
|
||||
|
||||
if (analyzed_join->getClauses().size() != 1)
|
||||
throw Exception(ErrorCodes::INCOMPATIBLE_TYPE_OF_JOIN, "JOIN keys should match to the Join engine keys [{}]", fmt::join(getKeyNames(), ", "));
|
||||
|
||||
const auto & join_on = analyzed_join->getOnlyClause();
|
||||
if (join_on.on_filter_condition_left || join_on.on_filter_condition_right)
|
||||
throw Exception(ErrorCodes::INCOMPATIBLE_TYPE_OF_JOIN, "ON section of JOIN with filter conditions is not implemented");
|
||||
@ -211,9 +214,9 @@ HashJoinPtr StorageJoin::getJoinLocked(std::shared_ptr<TableJoin> analyzed_join,
|
||||
left_key_names_resorted.push_back(key_names_left[key_position]);
|
||||
}
|
||||
|
||||
/// Set names qualifiers: table.column -> column
|
||||
/// It's required because storage join stores non-qualified names
|
||||
/// Qualifies will be added by join implementation (HashJoin)
|
||||
/// Set qualified identifiers to original names (table.column -> column).
|
||||
/// It's required because storage join stores non-qualified names.
|
||||
/// Qualifies will be added by join implementation (TableJoin contains a rename mapping).
|
||||
analyzed_join->setRightKeys(key_names);
|
||||
analyzed_join->setLeftKeys(left_key_names_resorted);
|
||||
|
||||
|
||||
@ -1370,19 +1370,21 @@ MergeTreeDataPartPtr StorageMergeTree::outdatePart(MergeTreeTransaction * txn, c
|
||||
{
|
||||
/// Forcefully stop merges and make part outdated
|
||||
auto merge_blocker = stopMergesAndWait();
|
||||
auto part = getPartIfExists(part_name, {MergeTreeDataPartState::Active});
|
||||
auto parts_lock = lockParts();
|
||||
auto part = getPartIfExists(part_name, {MergeTreeDataPartState::Active}, &parts_lock);
|
||||
if (!part)
|
||||
throw Exception(ErrorCodes::NO_SUCH_DATA_PART, "Part {} not found, won't try to drop it.", part_name);
|
||||
|
||||
removePartsFromWorkingSet(txn, {part}, true);
|
||||
removePartsFromWorkingSet(txn, {part}, true, &parts_lock);
|
||||
return part;
|
||||
}
|
||||
else
|
||||
{
|
||||
/// Wait merges selector
|
||||
std::unique_lock lock(currently_processing_in_background_mutex);
|
||||
auto parts_lock = lockParts();
|
||||
|
||||
auto part = getPartIfExists(part_name, {MergeTreeDataPartState::Active});
|
||||
auto part = getPartIfExists(part_name, {MergeTreeDataPartState::Active}, &parts_lock);
|
||||
/// It's okay, part was already removed
|
||||
if (!part)
|
||||
return nullptr;
|
||||
@ -1392,7 +1394,7 @@ MergeTreeDataPartPtr StorageMergeTree::outdatePart(MergeTreeTransaction * txn, c
|
||||
if (currently_merging_mutating_parts.contains(part))
|
||||
return nullptr;
|
||||
|
||||
removePartsFromWorkingSet(txn, {part}, true);
|
||||
removePartsFromWorkingSet(txn, {part}, true, &parts_lock);
|
||||
return part;
|
||||
}
|
||||
}
|
||||
|
||||
@ -633,6 +633,8 @@ void StorageReplicatedMergeTree::createNewZooKeeperNodes()
|
||||
futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/pinned_part_uuids", getPinnedPartUUIDs()->toString(), zkutil::CreateMode::Persistent));
|
||||
/// For ALTER PARTITION with multi-leaders
|
||||
futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/alter_partition_version", String(), zkutil::CreateMode::Persistent));
|
||||
/// For deduplication of async inserts
|
||||
futures.push_back(zookeeper->asyncTryCreateNoThrow(zookeeper_path + "/async_blocks", String(), zkutil::CreateMode::Persistent));
|
||||
|
||||
/// As for now, "/temp" node must exist, but we want to be able to remove it in future
|
||||
if (zookeeper->exists(zookeeper_path + "/temp"))
|
||||
@ -4535,7 +4537,7 @@ SinkToStoragePtr StorageReplicatedMergeTree::write(const ASTPtr & /*query*/, con
|
||||
const auto storage_settings_ptr = getSettings();
|
||||
const Settings & query_settings = local_context->getSettingsRef();
|
||||
bool deduplicate = storage_settings_ptr->replicated_deduplication_window != 0 && query_settings.insert_deduplicate;
|
||||
bool async_deduplicate = query_settings.async_insert && storage_settings_ptr->replicated_deduplication_window_for_async_inserts != 0 && query_settings.insert_deduplicate;
|
||||
bool async_deduplicate = query_settings.async_insert && query_settings.async_insert_deduplicate && storage_settings_ptr->replicated_deduplication_window_for_async_inserts != 0 && query_settings.insert_deduplicate;
|
||||
if (async_deduplicate)
|
||||
return std::make_shared<ReplicatedMergeTreeSinkWithAsyncDeduplicate>(
|
||||
*this, metadata_snapshot, query_settings.insert_quorum.valueOr(0),
|
||||
@ -6562,7 +6564,7 @@ void StorageReplicatedMergeTree::getClearBlocksInPartitionOpsImpl(
|
||||
{
|
||||
Strings blocks;
|
||||
if (Coordination::Error::ZOK != zookeeper.tryGetChildren(fs::path(zookeeper_path) / blocks_dir_name, blocks))
|
||||
throw Exception(zookeeper_path + "/" + blocks_dir_name + "blocks doesn't exist", ErrorCodes::NOT_FOUND_NODE);
|
||||
throw Exception(ErrorCodes::NOT_FOUND_NODE, "Node {}/{} doesn't exist", zookeeper_path, blocks_dir_name);
|
||||
|
||||
String partition_prefix = partition_id + "_";
|
||||
Strings paths_to_get;
|
||||
|
||||
@ -4,6 +4,8 @@
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Access/ContextAccess.h>
|
||||
#include <Storages/System/StorageSystemDatabases.h>
|
||||
#include <Parsers/ASTCreateQuery.h>
|
||||
#include <Common/logger_useful.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
@ -17,6 +19,7 @@ NamesAndTypesList StorageSystemDatabases::getNamesAndTypes()
|
||||
{"data_path", std::make_shared<DataTypeString>()},
|
||||
{"metadata_path", std::make_shared<DataTypeString>()},
|
||||
{"uuid", std::make_shared<DataTypeUUID>()},
|
||||
{"engine_full", std::make_shared<DataTypeString>()},
|
||||
{"comment", std::make_shared<DataTypeString>()}
|
||||
};
|
||||
}
|
||||
@ -28,6 +31,43 @@ NamesAndAliases StorageSystemDatabases::getNamesAndAliases()
|
||||
};
|
||||
}
|
||||
|
||||
static String getEngineFull(const DatabasePtr & database)
|
||||
{
|
||||
DDLGuardPtr guard;
|
||||
while (true)
|
||||
{
|
||||
String name = database->getDatabaseName();
|
||||
guard = DatabaseCatalog::instance().getDDLGuard(name, "");
|
||||
|
||||
/// Ensure that the database was not renamed before we acquired the lock
|
||||
auto locked_database = DatabaseCatalog::instance().tryGetDatabase(name);
|
||||
|
||||
if (locked_database.get() == database.get())
|
||||
break;
|
||||
|
||||
/// Database was dropped
|
||||
if (!locked_database && name == database->getDatabaseName())
|
||||
return {};
|
||||
|
||||
guard.reset();
|
||||
LOG_TRACE(&Poco::Logger::get("StorageSystemDatabases"), "Failed to lock database {} ({}), will retry", name, database->getUUID());
|
||||
}
|
||||
|
||||
ASTPtr ast = database->getCreateDatabaseQuery();
|
||||
auto * ast_create = ast->as<ASTCreateQuery>();
|
||||
|
||||
if (!ast_create || !ast_create->storage)
|
||||
return {};
|
||||
|
||||
String engine_full = ast_create->storage->formatWithSecretsHidden();
|
||||
static const char * const extra_head = " ENGINE = ";
|
||||
|
||||
if (startsWith(engine_full, extra_head))
|
||||
engine_full = engine_full.substr(strlen(extra_head));
|
||||
|
||||
return engine_full;
|
||||
}
|
||||
|
||||
void StorageSystemDatabases::fillData(MutableColumns & res_columns, ContextPtr context, const SelectQueryInfo &) const
|
||||
{
|
||||
const auto access = context->getAccess();
|
||||
@ -47,7 +87,8 @@ void StorageSystemDatabases::fillData(MutableColumns & res_columns, ContextPtr c
|
||||
res_columns[2]->insert(context->getPath() + database->getDataPath());
|
||||
res_columns[3]->insert(database->getMetadataPath());
|
||||
res_columns[4]->insert(database->getUUID());
|
||||
res_columns[5]->insert(database->getDatabaseComment());
|
||||
res_columns[5]->insert(getEngineFull(database));
|
||||
res_columns[6]->insert(database->getDatabaseComment());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -145,7 +145,7 @@ if __name__ == "__main__":
|
||||
)
|
||||
logging.info("Going to run func tests: %s", run_command)
|
||||
|
||||
with TeePopen(run_command, run_log_path) as process:
|
||||
with TeePopen(run_command, run_log_path, timeout=60 * 150) as process:
|
||||
retcode = process.wait()
|
||||
if retcode == 0:
|
||||
logging.info("Run successfully")
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
from io import TextIOWrapper
|
||||
from subprocess import Popen, PIPE, STDOUT
|
||||
from threading import Thread
|
||||
from time import sleep
|
||||
@ -14,15 +15,23 @@ import sys
|
||||
# it finishes. stderr and stdout will be redirected both to specified file and
|
||||
# stdout.
|
||||
class TeePopen:
|
||||
# pylint: disable=W0102
|
||||
def __init__(self, command, log_file, env=os.environ.copy(), timeout=None):
|
||||
def __init__(
|
||||
self,
|
||||
command: str,
|
||||
log_file: str,
|
||||
env: Optional[dict] = None,
|
||||
timeout: Optional[int] = None,
|
||||
):
|
||||
self.command = command
|
||||
self.log_file = log_file
|
||||
self.env = env
|
||||
self._log_file_name = log_file
|
||||
self._log_file = None # type: Optional[TextIOWrapper]
|
||||
self.env = env or os.environ.copy()
|
||||
self._process = None # type: Optional[Popen]
|
||||
self.timeout = timeout
|
||||
|
||||
def _check_timeout(self):
|
||||
def _check_timeout(self) -> None:
|
||||
if self.timeout is None:
|
||||
return
|
||||
sleep(self.timeout)
|
||||
while self.process.poll() is None:
|
||||
logging.warning(
|
||||
@ -33,7 +42,7 @@ class TeePopen:
|
||||
os.killpg(self.process.pid, 9)
|
||||
sleep(10)
|
||||
|
||||
def __enter__(self):
|
||||
def __enter__(self) -> "TeePopen":
|
||||
self.process = Popen(
|
||||
self.command,
|
||||
shell=True,
|
||||
@ -44,25 +53,21 @@ class TeePopen:
|
||||
stdout=PIPE,
|
||||
bufsize=1,
|
||||
)
|
||||
self.log_file = open(self.log_file, "w", encoding="utf-8")
|
||||
if self.timeout is not None and self.timeout > 0:
|
||||
t = Thread(target=self._check_timeout)
|
||||
t.daemon = True # does not block the program from exit
|
||||
t.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, t, value, traceback):
|
||||
for line in self.process.stdout: # type: ignore
|
||||
sys.stdout.write(line)
|
||||
self.log_file.write(line)
|
||||
|
||||
self.process.wait()
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
self.wait()
|
||||
self.log_file.close()
|
||||
|
||||
def wait(self):
|
||||
for line in self.process.stdout: # type: ignore
|
||||
sys.stdout.write(line)
|
||||
self.log_file.write(line)
|
||||
if self.process.stdout is not None:
|
||||
for line in self.process.stdout:
|
||||
sys.stdout.write(line)
|
||||
self.log_file.write(line)
|
||||
|
||||
return self.process.wait()
|
||||
|
||||
@ -75,3 +80,9 @@ class TeePopen:
|
||||
@process.setter
|
||||
def process(self, process: Popen) -> None:
|
||||
self._process = process
|
||||
|
||||
@property
|
||||
def log_file(self) -> TextIOWrapper:
|
||||
if self._log_file is None:
|
||||
self._log_file = open(self._log_file_name, "w", encoding="utf-8")
|
||||
return self._log_file
|
||||
|
||||
@ -1180,6 +1180,9 @@ def test_tables_dependency():
|
||||
t4 = random_table_names[3]
|
||||
t5 = random_table_names[4]
|
||||
t6 = random_table_names[5]
|
||||
t7 = random_table_names[6]
|
||||
t8 = random_table_names[7]
|
||||
t9 = random_table_names[8]
|
||||
|
||||
# Create a materialized view and a dictionary with a local table as source.
|
||||
instance.query(
|
||||
@ -1193,7 +1196,7 @@ def test_tables_dependency():
|
||||
instance.query(f"CREATE MATERIALIZED VIEW {t3} TO {t2} AS SELECT x, y FROM {t1}")
|
||||
|
||||
instance.query(
|
||||
f"CREATE DICTIONARY {t4} (x Int64, y String) PRIMARY KEY x SOURCE(CLICKHOUSE(HOST 'localhost' PORT tcpPort() TABLE '{t1.split('.')[1]}' DB '{t1.split('.')[0]}')) LAYOUT(FLAT()) LIFETIME(0)"
|
||||
f"CREATE DICTIONARY {t4} (x Int64, y String) PRIMARY KEY x SOURCE(CLICKHOUSE(HOST 'localhost' PORT tcpPort() TABLE '{t1.split('.')[1]}' DB '{t1.split('.')[0]}')) LAYOUT(FLAT()) LIFETIME(4)"
|
||||
)
|
||||
|
||||
instance.query(f"CREATE TABLE {t5} AS dictionary({t4})")
|
||||
@ -1202,12 +1205,25 @@ def test_tables_dependency():
|
||||
f"CREATE TABLE {t6}(x Int64, y String DEFAULT dictGet({t4}, 'y', x)) ENGINE=MergeTree ORDER BY tuple()"
|
||||
)
|
||||
|
||||
instance.query(f"CREATE VIEW {t7} AS SELECT sum(x) FROM (SELECT x FROM {t6})")
|
||||
|
||||
instance.query(
|
||||
f"CREATE TABLE {t8} AS {t2} ENGINE = Buffer({t2.split('.')[0]}, {t2.split('.')[1]}, 16, 10, 100, 10000, 1000000, 10000000, 100000000)"
|
||||
)
|
||||
|
||||
instance.query(
|
||||
f"CREATE DICTIONARY {t9} (x Int64, y String) PRIMARY KEY x SOURCE(CLICKHOUSE(TABLE '{t1.split('.')[1]}' DB '{t1.split('.')[0]}')) LAYOUT(FLAT()) LIFETIME(9)"
|
||||
)
|
||||
|
||||
# Make backup.
|
||||
backup_name = new_backup_name()
|
||||
instance.query(f"BACKUP DATABASE test, DATABASE test2 TO {backup_name}")
|
||||
|
||||
# Drop everything in reversive order.
|
||||
def drop():
|
||||
instance.query(f"DROP DICTIONARY {t9}")
|
||||
instance.query(f"DROP TABLE {t8} NO DELAY")
|
||||
instance.query(f"DROP TABLE {t7} NO DELAY")
|
||||
instance.query(f"DROP TABLE {t6} NO DELAY")
|
||||
instance.query(f"DROP TABLE {t5} NO DELAY")
|
||||
instance.query(f"DROP DICTIONARY {t4}")
|
||||
@ -1219,11 +1235,36 @@ def test_tables_dependency():
|
||||
|
||||
drop()
|
||||
|
||||
# Restore everything and check.
|
||||
# Restore everything.
|
||||
instance.query(f"RESTORE ALL FROM {backup_name}")
|
||||
|
||||
# Check everything is restored.
|
||||
assert instance.query(
|
||||
"SELECT concat(database, '.', name) AS c FROM system.tables WHERE database IN ['test', 'test2'] ORDER BY c"
|
||||
) == TSV(sorted([t1, t2, t3, t4, t5, t6]))
|
||||
) == TSV(sorted([t1, t2, t3, t4, t5, t6, t7, t8, t9]))
|
||||
|
||||
# Check logs.
|
||||
instance.query("SYSTEM FLUSH LOGS")
|
||||
expect_in_logs = [
|
||||
f"Table {t1} has no dependencies (level 0)",
|
||||
f"Table {t2} has no dependencies (level 0)",
|
||||
(
|
||||
f"Table {t3} has 2 dependencies: {t1}, {t2} (level 1)",
|
||||
f"Table {t3} has 2 dependencies: {t2}, {t1} (level 1)",
|
||||
),
|
||||
f"Table {t4} has 1 dependencies: {t1} (level 1)",
|
||||
f"Table {t5} has 1 dependencies: {t4} (level 2)",
|
||||
f"Table {t6} has 1 dependencies: {t4} (level 2)",
|
||||
f"Table {t7} has 1 dependencies: {t6} (level 3)",
|
||||
f"Table {t8} has 1 dependencies: {t2} (level 1)",
|
||||
f"Table {t9} has 1 dependencies: {t1} (level 1)",
|
||||
]
|
||||
for expect in expect_in_logs:
|
||||
assert any(
|
||||
[
|
||||
instance.contains_in_log(f"RestorerFromBackup: {x}")
|
||||
for x in tuple(expect)
|
||||
]
|
||||
)
|
||||
|
||||
drop()
|
||||
|
||||
@ -796,6 +796,84 @@ def test_mutation():
|
||||
node1.query(f"RESTORE TABLE tbl ON CLUSTER 'cluster' FROM {backup_name}")
|
||||
|
||||
|
||||
def test_tables_dependency():
|
||||
node1.query("CREATE DATABASE mydb ON CLUSTER 'cluster3'")
|
||||
|
||||
node1.query(
|
||||
"CREATE TABLE mydb.src ON CLUSTER 'cluster' (x Int64, y String) ENGINE=MergeTree ORDER BY tuple()"
|
||||
)
|
||||
|
||||
node1.query(
|
||||
"CREATE DICTIONARY mydb.dict ON CLUSTER 'cluster' (x Int64, y String) PRIMARY KEY x "
|
||||
"SOURCE(CLICKHOUSE(HOST 'localhost' PORT tcpPort() DB 'mydb' TABLE 'src')) LAYOUT(FLAT()) LIFETIME(0)"
|
||||
)
|
||||
|
||||
node1.query(
|
||||
"CREATE TABLE mydb.dist1 (x Int64) ENGINE=Distributed('cluster', 'mydb', 'src')"
|
||||
)
|
||||
|
||||
node3.query(
|
||||
"CREATE TABLE mydb.dist2 (x Int64) ENGINE=Distributed(cluster, 'mydb', 'src')"
|
||||
)
|
||||
|
||||
node1.query("CREATE TABLE mydb.clusterfunc1 AS cluster('cluster', 'mydb.src')")
|
||||
node1.query("CREATE TABLE mydb.clusterfunc2 AS cluster(cluster, mydb.src)")
|
||||
node1.query("CREATE TABLE mydb.clusterfunc3 AS cluster(cluster, 'mydb', 'src')")
|
||||
node1.query(
|
||||
"CREATE TABLE mydb.clusterfunc4 AS cluster(cluster, dictionary(mydb.dict))"
|
||||
)
|
||||
node1.query(
|
||||
"CREATE TABLE mydb.clusterfunc5 AS clusterAllReplicas(cluster, dictionary(mydb.dict))"
|
||||
)
|
||||
|
||||
node3.query("CREATE TABLE mydb.clusterfunc6 AS cluster('cluster', 'mydb.src')")
|
||||
node3.query("CREATE TABLE mydb.clusterfunc7 AS cluster(cluster, mydb.src)")
|
||||
node3.query("CREATE TABLE mydb.clusterfunc8 AS cluster(cluster, 'mydb', 'src')")
|
||||
node3.query(
|
||||
"CREATE TABLE mydb.clusterfunc9 AS cluster(cluster, dictionary(mydb.dict))"
|
||||
)
|
||||
node3.query(
|
||||
"CREATE TABLE mydb.clusterfunc10 AS clusterAllReplicas(cluster, dictionary(mydb.dict))"
|
||||
)
|
||||
|
||||
backup_name = new_backup_name()
|
||||
node3.query(f"BACKUP DATABASE mydb ON CLUSTER 'cluster3' TO {backup_name}")
|
||||
|
||||
node3.query("DROP DATABASE mydb")
|
||||
|
||||
node3.query(f"RESTORE DATABASE mydb ON CLUSTER 'cluster3' FROM {backup_name}")
|
||||
|
||||
node3.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster3'")
|
||||
expect_in_logs_1 = [
|
||||
"Table mydb.src has no dependencies (level 0)",
|
||||
"Table mydb.dict has 1 dependencies: mydb.src (level 1)",
|
||||
"Table mydb.dist1 has 1 dependencies: mydb.src (level 1)",
|
||||
"Table mydb.clusterfunc1 has 1 dependencies: mydb.src (level 1)",
|
||||
"Table mydb.clusterfunc2 has 1 dependencies: mydb.src (level 1)",
|
||||
"Table mydb.clusterfunc3 has 1 dependencies: mydb.src (level 1)",
|
||||
"Table mydb.clusterfunc4 has 1 dependencies: mydb.dict (level 2)",
|
||||
"Table mydb.clusterfunc5 has 1 dependencies: mydb.dict (level 2)",
|
||||
]
|
||||
expect_in_logs_2 = [
|
||||
"Table mydb.src has no dependencies (level 0)",
|
||||
"Table mydb.dict has 1 dependencies: mydb.src (level 1)",
|
||||
]
|
||||
expect_in_logs_3 = [
|
||||
"Table mydb.dist2 has no dependencies (level 0)",
|
||||
"Table mydb.clusterfunc6 has no dependencies (level 0)",
|
||||
"Table mydb.clusterfunc7 has no dependencies (level 0)",
|
||||
"Table mydb.clusterfunc8 has no dependencies (level 0)",
|
||||
"Table mydb.clusterfunc9 has no dependencies (level 0)",
|
||||
"Table mydb.clusterfunc10 has no dependencies (level 0)",
|
||||
]
|
||||
for expect in expect_in_logs_1:
|
||||
assert node1.contains_in_log(f"RestorerFromBackup: {expect}")
|
||||
for expect in expect_in_logs_2:
|
||||
assert node2.contains_in_log(f"RestorerFromBackup: {expect}")
|
||||
for expect in expect_in_logs_3:
|
||||
assert node3.contains_in_log(f"RestorerFromBackup: {expect}")
|
||||
|
||||
|
||||
def test_get_error_from_other_host():
|
||||
node1.query("CREATE TABLE tbl (`x` UInt8) ENGINE = MergeTree ORDER BY x")
|
||||
node1.query("INSERT INTO tbl VALUES (3)")
|
||||
|
||||
@ -211,8 +211,8 @@ def test_attach_detach_partition(cluster):
|
||||
|
||||
node.query("ALTER TABLE hdfs_test DETACH PARTITION '2020-01-03'")
|
||||
assert node.query("SELECT count(*) FROM hdfs_test FORMAT Values") == "(4096)"
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
wait_for_delete_empty_parts(node, "hdfs_test")
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
|
||||
hdfs_objects = fs.listdir("/clickhouse")
|
||||
assert len(hdfs_objects) == FILES_OVERHEAD + FILES_OVERHEAD_PER_PART_WIDE * 2
|
||||
@ -225,8 +225,8 @@ def test_attach_detach_partition(cluster):
|
||||
|
||||
node.query("ALTER TABLE hdfs_test DROP PARTITION '2020-01-03'")
|
||||
assert node.query("SELECT count(*) FROM hdfs_test FORMAT Values") == "(4096)"
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
wait_for_delete_empty_parts(node, "hdfs_test")
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
|
||||
hdfs_objects = fs.listdir("/clickhouse")
|
||||
assert len(hdfs_objects) == FILES_OVERHEAD + FILES_OVERHEAD_PER_PART_WIDE
|
||||
@ -237,8 +237,8 @@ def test_attach_detach_partition(cluster):
|
||||
settings={"allow_drop_detached": 1},
|
||||
)
|
||||
assert node.query("SELECT count(*) FROM hdfs_test FORMAT Values") == "(0)"
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
wait_for_delete_empty_parts(node, "hdfs_test")
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
|
||||
hdfs_objects = fs.listdir("/clickhouse")
|
||||
assert len(hdfs_objects) == FILES_OVERHEAD
|
||||
@ -305,8 +305,8 @@ def test_table_manipulations(cluster):
|
||||
|
||||
node.query("TRUNCATE TABLE hdfs_test")
|
||||
assert node.query("SELECT count(*) FROM hdfs_test FORMAT Values") == "(0)"
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
wait_for_delete_empty_parts(node, "hdfs_test")
|
||||
wait_for_delete_inactive_parts(node, "hdfs_test")
|
||||
|
||||
hdfs_objects = fs.listdir("/clickhouse")
|
||||
assert len(hdfs_objects) == FILES_OVERHEAD
|
||||
|
||||
@ -323,8 +323,8 @@ def test_attach_detach_partition(cluster, node_name):
|
||||
)
|
||||
|
||||
node.query("ALTER TABLE s3_test DETACH PARTITION '2020-01-03'")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
assert node.query("SELECT count(*) FROM s3_test FORMAT Values") == "(4096)"
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/", recursive=True)))
|
||||
@ -339,8 +339,8 @@ def test_attach_detach_partition(cluster, node_name):
|
||||
)
|
||||
|
||||
node.query("ALTER TABLE s3_test DROP PARTITION '2020-01-03'")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
assert node.query("SELECT count(*) FROM s3_test FORMAT Values") == "(4096)"
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/", recursive=True)))
|
||||
@ -348,8 +348,8 @@ def test_attach_detach_partition(cluster, node_name):
|
||||
)
|
||||
|
||||
node.query("ALTER TABLE s3_test DETACH PARTITION '2020-01-04'")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
assert node.query("SELECT count(*) FROM s3_test FORMAT Values") == "(0)"
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/")))
|
||||
@ -431,8 +431,8 @@ def test_table_manipulations(cluster, node_name):
|
||||
)
|
||||
|
||||
node.query("TRUNCATE TABLE s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
assert node.query("SELECT count(*) FROM s3_test FORMAT Values") == "(0)"
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/", recursive=True)))
|
||||
@ -546,8 +546,8 @@ def test_freeze_unfreeze(cluster, node_name):
|
||||
node.query("ALTER TABLE s3_test FREEZE WITH NAME 'backup2'")
|
||||
|
||||
node.query("TRUNCATE TABLE s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/", recursive=True)))
|
||||
== FILES_OVERHEAD + FILES_OVERHEAD_PER_PART_WIDE * 2
|
||||
@ -586,8 +586,8 @@ def test_freeze_system_unfreeze(cluster, node_name):
|
||||
node.query("ALTER TABLE s3_test_removed FREEZE WITH NAME 'backup3'")
|
||||
|
||||
node.query("TRUNCATE TABLE s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
wait_for_delete_empty_parts(node, "s3_test")
|
||||
wait_for_delete_inactive_parts(node, "s3_test")
|
||||
node.query("DROP TABLE s3_test_removed NO DELAY")
|
||||
assert (
|
||||
len(list(minio.list_objects(cluster.minio_bucket, "data/", recursive=True)))
|
||||
|
||||
@ -201,8 +201,8 @@ def attach_check_all_parts_table(started_cluster):
|
||||
def test_attach_check_all_parts(attach_check_all_parts_table):
|
||||
q("ALTER TABLE test.attach_partition DETACH PARTITION 0")
|
||||
|
||||
wait_for_delete_inactive_parts(instance, "test.attach_partition")
|
||||
wait_for_delete_empty_parts(instance, "test.attach_partition")
|
||||
wait_for_delete_inactive_parts(instance, "test.attach_partition")
|
||||
|
||||
path_to_detached = path_to_data + "data/test/attach_partition/detached/"
|
||||
instance.exec_in_container(["mkdir", "{}".format(path_to_detached + "0_5_5_0")])
|
||||
|
||||
@ -0,0 +1,25 @@
|
||||
<clickhouse>
|
||||
<password_complexity>
|
||||
<rule>
|
||||
<pattern>.{12}</pattern>
|
||||
<message>be at least 12 characters long</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{N}</pattern>
|
||||
<message>contain at least 1 numeric character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{Ll}</pattern>
|
||||
<message>contain at least 1 lowercase character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>\p{Lu}</pattern>
|
||||
<message>contain at least 1 uppercase character</message>
|
||||
</rule>
|
||||
<rule>
|
||||
<pattern>[^\p{L}\p{N}]</pattern>
|
||||
<message>contain at least 1 special character</message>
|
||||
</rule>
|
||||
</password_complexity>
|
||||
</clickhouse>
|
||||
|
||||
42
tests/integration/test_password_constraints/test.py
Normal file
42
tests/integration/test_password_constraints/test.py
Normal file
@ -0,0 +1,42 @@
|
||||
import pytest
|
||||
|
||||
from helpers.cluster import ClickHouseCluster
|
||||
|
||||
cluster = ClickHouseCluster(__file__)
|
||||
|
||||
node = cluster.add_instance("node", main_configs=["configs/complexity_rules.xml"])
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def start_cluster():
|
||||
try:
|
||||
cluster.start()
|
||||
yield cluster
|
||||
finally:
|
||||
cluster.shutdown()
|
||||
|
||||
|
||||
def test_complexity_rules(start_cluster):
|
||||
|
||||
error_message = "DB::Exception: Invalid password. The password should: be at least 12 characters long, contain at least 1 numeric character, contain at least 1 lowercase character, contain at least 1 uppercase character, contain at least 1 special character"
|
||||
assert error_message in node.query_and_get_error(
|
||||
"CREATE USER u_1 IDENTIFIED WITH plaintext_password BY ''"
|
||||
)
|
||||
|
||||
error_message = "DB::Exception: Invalid password. The password should: contain at least 1 lowercase character, contain at least 1 uppercase character, contain at least 1 special character"
|
||||
assert error_message in node.query_and_get_error(
|
||||
"CREATE USER u_2 IDENTIFIED WITH sha256_password BY '000000000000'"
|
||||
)
|
||||
|
||||
error_message = "DB::Exception: Invalid password. The password should: contain at least 1 uppercase character, contain at least 1 special character"
|
||||
assert error_message in node.query_and_get_error(
|
||||
"CREATE USER u_3 IDENTIFIED WITH double_sha1_password BY 'a00000000000'"
|
||||
)
|
||||
|
||||
error_message = "DB::Exception: Invalid password. The password should: contain at least 1 special character"
|
||||
assert error_message in node.query_and_get_error(
|
||||
"CREATE USER u_4 IDENTIFIED WITH plaintext_password BY 'aA0000000000'"
|
||||
)
|
||||
|
||||
node.query("CREATE USER u_5 IDENTIFIED WITH plaintext_password BY 'aA!000000000'")
|
||||
node.query("DROP USER u_5")
|
||||
@ -112,11 +112,15 @@ def get_pgsql_client(cluster, port):
|
||||
time.sleep(0.1)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def get_grpc_channel(cluster, port):
|
||||
host_port = cluster.get_instance_ip("instance") + f":{port}"
|
||||
channel = grpc.insecure_channel(host_port)
|
||||
grpc.channel_ready_future(channel).result(timeout=10)
|
||||
return channel
|
||||
try:
|
||||
yield channel
|
||||
finally:
|
||||
channel.close()
|
||||
|
||||
|
||||
def grpc_query(channel, query_text):
|
||||
@ -238,16 +242,17 @@ def test_change_postgresql_port(cluster, zk):
|
||||
|
||||
def test_change_grpc_port(cluster, zk):
|
||||
with default_client(cluster, zk) as client:
|
||||
grpc_channel = get_grpc_channel(cluster, port=9100)
|
||||
assert grpc_query(grpc_channel, "SELECT 1") == "1\n"
|
||||
with sync_loaded_config(client.query):
|
||||
zk.set("/clickhouse/ports/grpc", b"9090")
|
||||
with pytest.raises(
|
||||
grpc._channel._InactiveRpcError, match="StatusCode.UNAVAILABLE"
|
||||
):
|
||||
grpc_query(grpc_channel, "SELECT 1")
|
||||
grpc_channel_on_new_port = get_grpc_channel(cluster, port=9090)
|
||||
assert grpc_query(grpc_channel_on_new_port, "SELECT 1") == "1\n"
|
||||
with get_grpc_channel(cluster, port=9100) as grpc_channel:
|
||||
assert grpc_query(grpc_channel, "SELECT 1") == "1\n"
|
||||
with sync_loaded_config(client.query):
|
||||
zk.set("/clickhouse/ports/grpc", b"9090")
|
||||
with pytest.raises(
|
||||
grpc._channel._InactiveRpcError, match="StatusCode.UNAVAILABLE"
|
||||
):
|
||||
grpc_query(grpc_channel, "SELECT 1")
|
||||
|
||||
with get_grpc_channel(cluster, port=9090) as grpc_channel_on_new_port:
|
||||
assert grpc_query(grpc_channel_on_new_port, "SELECT 1") == "1\n"
|
||||
|
||||
|
||||
def test_remove_tcp_port(cluster, zk):
|
||||
@ -292,14 +297,14 @@ def test_remove_postgresql_port(cluster, zk):
|
||||
|
||||
def test_remove_grpc_port(cluster, zk):
|
||||
with default_client(cluster, zk) as client:
|
||||
grpc_channel = get_grpc_channel(cluster, port=9100)
|
||||
assert grpc_query(grpc_channel, "SELECT 1") == "1\n"
|
||||
with sync_loaded_config(client.query):
|
||||
zk.delete("/clickhouse/ports/grpc")
|
||||
with pytest.raises(
|
||||
grpc._channel._InactiveRpcError, match="StatusCode.UNAVAILABLE"
|
||||
):
|
||||
grpc_query(grpc_channel, "SELECT 1")
|
||||
with get_grpc_channel(cluster, port=9100) as grpc_channel:
|
||||
assert grpc_query(grpc_channel, "SELECT 1") == "1\n"
|
||||
with sync_loaded_config(client.query):
|
||||
zk.delete("/clickhouse/ports/grpc")
|
||||
with pytest.raises(
|
||||
grpc._channel._InactiveRpcError, match="StatusCode.UNAVAILABLE"
|
||||
):
|
||||
grpc_query(grpc_channel, "SELECT 1")
|
||||
|
||||
|
||||
def test_change_listen_host(cluster, zk):
|
||||
|
||||
@ -449,8 +449,8 @@ def test_ttl_empty_parts(started_cluster):
|
||||
assert node1.query("SELECT count() FROM test_ttl_empty_parts") == "3000\n"
|
||||
|
||||
# Wait for cleanup thread
|
||||
wait_for_delete_inactive_parts(node1, "test_ttl_empty_parts")
|
||||
wait_for_delete_empty_parts(node1, "test_ttl_empty_parts")
|
||||
wait_for_delete_inactive_parts(node1, "test_ttl_empty_parts")
|
||||
|
||||
assert (
|
||||
node1.query(
|
||||
|
||||
@ -65,7 +65,7 @@ $CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (1, '0', 1);"
|
||||
$CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (1, '1', 1);"
|
||||
$CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (2, '0', 1);"
|
||||
|
||||
query_with_retry "ALTER TABLE src MOVE PARTITION 1 TO TABLE dst;" &>-
|
||||
query_with_retry "ALTER TABLE src MOVE PARTITION 1 TO TABLE dst;" &>/dev/null
|
||||
$CLICKHOUSE_CLIENT --query="SYSTEM SYNC REPLICA dst;"
|
||||
|
||||
$CLICKHOUSE_CLIENT --query="SELECT count(), sum(d) FROM src;"
|
||||
@ -85,7 +85,7 @@ $CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (1, '0', 1);"
|
||||
$CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (1, '1', 1);"
|
||||
$CLICKHOUSE_CLIENT --query="INSERT INTO src VALUES (2, '0', 1);"
|
||||
|
||||
query_with_retry "ALTER TABLE src MOVE PARTITION 1 TO TABLE dst;" &>-
|
||||
query_with_retry "ALTER TABLE src MOVE PARTITION 1 TO TABLE dst;" &>/dev/null
|
||||
$CLICKHOUSE_CLIENT --query="SYSTEM SYNC REPLICA dst;"
|
||||
|
||||
$CLICKHOUSE_CLIENT --query="SELECT count(), sum(d) FROM src;"
|
||||
|
||||
@ -39,7 +39,7 @@ RENAME TABLE test_01155_ordinary.mv1 TO test_01155_atomic.mv1;
|
||||
RENAME TABLE test_01155_ordinary.mv2 TO test_01155_atomic.mv2;
|
||||
RENAME TABLE test_01155_ordinary.dst TO test_01155_atomic.dst;
|
||||
RENAME TABLE test_01155_ordinary.src TO test_01155_atomic.src;
|
||||
SET check_table_dependencies=0; -- Otherwise we'll get error "test_01155_atomic.dict depends on test_01155_ordinary.dist" in the next line.
|
||||
SET check_table_dependencies=0; -- Otherwise we'll get error "test_01155_ordinary.dict depends on test_01155_ordinary.dist" in the next line.
|
||||
RENAME TABLE test_01155_ordinary.dist TO test_01155_atomic.dist;
|
||||
SET check_table_dependencies=1;
|
||||
RENAME DICTIONARY test_01155_ordinary.dict TO test_01155_atomic.dict;
|
||||
@ -65,7 +65,7 @@ SELECT dictGet('test_01155_ordinary.dict', 'x', 'after renaming database');
|
||||
SELECT database, substr(name, 1, 10) FROM system.tables WHERE database like 'test_01155_%';
|
||||
|
||||
-- Move tables back
|
||||
SET check_table_dependencies=0; -- Otherwise we'll get error "test_01155_atomic.dict depends on test_01155_ordinary.dist" in the next line.
|
||||
SET check_table_dependencies=0; -- Otherwise we'll get error "test_01155_ordinary.dict depends on test_01155_ordinary.dist" in the next line.
|
||||
RENAME DATABASE test_01155_ordinary TO test_01155_atomic;
|
||||
SET check_table_dependencies=1;
|
||||
|
||||
|
||||
@ -21,7 +21,7 @@ censor.net
|
||||
censor.net
|
||||
censor.net
|
||||
censor.net
|
||||
SELECT if(number > 5, \'censor.net\', \'google\')
|
||||
SELECT if(number > 5, _CAST(\'censor.net\', \'Enum8(\\\'censor.net\\\' = 1, \\\'google\\\' = 2)\'), _CAST(\'google\', \'Enum8(\\\'censor.net\\\' = 1, \\\'google\\\' = 2)\'))
|
||||
FROM system.numbers
|
||||
LIMIT 10
|
||||
other
|
||||
|
||||
@ -34,8 +34,8 @@ DROP VIEW IF EXISTS test_view_different_db;
|
||||
CREATE VIEW test_view_different_db AS SELECT id, value, dictGet('2025_test_db.test_dictionary', 'value', id) FROM 2025_test_db.view_table;
|
||||
SELECT * FROM test_view_different_db;
|
||||
|
||||
DROP TABLE 2025_test_db.test_table;
|
||||
DROP DICTIONARY 2025_test_db.test_dictionary;
|
||||
DROP TABLE 2025_test_db.test_table;
|
||||
DROP TABLE 2025_test_db.view_table;
|
||||
|
||||
DROP VIEW test_view_different_db;
|
||||
|
||||
@ -128,6 +128,7 @@ CREATE TABLE system.databases
|
||||
`data_path` String,
|
||||
`metadata_path` String,
|
||||
`uuid` UUID,
|
||||
`engine_full` String,
|
||||
`comment` String,
|
||||
`database` String
|
||||
)
|
||||
|
||||
@ -1,4 +1,7 @@
|
||||
-- Tags: no-fasttest
|
||||
-- Tags: no-fasttest, no-parallel
|
||||
|
||||
SYSTEM DROP FILESYSTEM CACHE 's3_cache/';
|
||||
SYSTEM DROP FILESYSTEM CACHE 's3_cache_2/';
|
||||
|
||||
DESCRIBE FILESYSTEM CACHE 's3_cache';
|
||||
DESCRIBE FILESYSTEM CACHE 's3_cache_2';
|
||||
|
||||
@ -0,0 +1,6 @@
|
||||
1000
|
||||
1.000001
|
||||
100.00001
|
||||
305419896
|
||||
610839792
|
||||
583
|
||||
@ -0,0 +1,6 @@
|
||||
select 1_000;
|
||||
select 1.00_00_01;
|
||||
select 1.000_0001e2;
|
||||
select 0x12_34_56_78;
|
||||
select 0x12_34_56_78p1;
|
||||
select 0b0010_0100_0111;
|
||||
@ -82,7 +82,6 @@ addYears
|
||||
addressToLine
|
||||
addressToLineWithInlines
|
||||
addressToSymbol
|
||||
age
|
||||
alphaTokens
|
||||
and
|
||||
appendTrailingCharIfAbsent
|
||||
|
||||
@ -1,76 +0,0 @@
|
||||
Various intervals
|
||||
-1
|
||||
0
|
||||
0
|
||||
-7
|
||||
-3
|
||||
0
|
||||
-23
|
||||
-11
|
||||
0
|
||||
-103
|
||||
-52
|
||||
0
|
||||
-730
|
||||
-364
|
||||
1
|
||||
-17520
|
||||
-8736
|
||||
24
|
||||
-1051200
|
||||
-524160
|
||||
1440
|
||||
-63072000
|
||||
-31449600
|
||||
86400
|
||||
DateTime arguments
|
||||
0
|
||||
23
|
||||
1439
|
||||
86399
|
||||
Date and DateTime arguments
|
||||
-63072000
|
||||
-31449600
|
||||
86400
|
||||
Constant and non-constant arguments
|
||||
-1051200
|
||||
-524160
|
||||
1440
|
||||
Case insensitive
|
||||
-10
|
||||
Dependance of timezones
|
||||
0
|
||||
0
|
||||
1
|
||||
25
|
||||
1500
|
||||
90000
|
||||
0
|
||||
0
|
||||
1
|
||||
24
|
||||
1440
|
||||
86400
|
||||
0
|
||||
0
|
||||
1
|
||||
25
|
||||
1500
|
||||
90000
|
||||
0
|
||||
0
|
||||
1
|
||||
24
|
||||
1440
|
||||
86400
|
||||
Additional test
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
@ -1,82 +0,0 @@
|
||||
SELECT 'Various intervals';
|
||||
|
||||
SELECT age('year', toDate('2017-12-31'), toDate('2016-01-01'));
|
||||
SELECT age('year', toDate('2017-12-31'), toDate('2017-01-01'));
|
||||
SELECT age('year', toDate('2017-12-31'), toDate('2018-01-01'));
|
||||
SELECT age('quarter', toDate('2017-12-31'), toDate('2016-01-01'));
|
||||
SELECT age('quarter', toDate('2017-12-31'), toDate('2017-01-01'));
|
||||
SELECT age('quarter', toDate('2017-12-31'), toDate('2018-01-01'));
|
||||
SELECT age('month', toDate('2017-12-31'), toDate('2016-01-01'));
|
||||
SELECT age('month', toDate('2017-12-31'), toDate('2017-01-01'));
|
||||
SELECT age('month', toDate('2017-12-31'), toDate('2018-01-01'));
|
||||
SELECT age('week', toDate('2017-12-31'), toDate('2016-01-01'));
|
||||
SELECT age('week', toDate('2017-12-31'), toDate('2017-01-01'));
|
||||
SELECT age('week', toDate('2017-12-31'), toDate('2018-01-01'));
|
||||
SELECT age('day', toDate('2017-12-31'), toDate('2016-01-01'));
|
||||
SELECT age('day', toDate('2017-12-31'), toDate('2017-01-01'));
|
||||
SELECT age('day', toDate('2017-12-31'), toDate('2018-01-01'));
|
||||
SELECT age('hour', toDate('2017-12-31'), toDate('2016-01-01'), 'UTC');
|
||||
SELECT age('hour', toDate('2017-12-31'), toDate('2017-01-01'), 'UTC');
|
||||
SELECT age('hour', toDate('2017-12-31'), toDate('2018-01-01'), 'UTC');
|
||||
SELECT age('minute', toDate('2017-12-31'), toDate('2016-01-01'), 'UTC');
|
||||
SELECT age('minute', toDate('2017-12-31'), toDate('2017-01-01'), 'UTC');
|
||||
SELECT age('minute', toDate('2017-12-31'), toDate('2018-01-01'), 'UTC');
|
||||
SELECT age('second', toDate('2017-12-31'), toDate('2016-01-01'), 'UTC');
|
||||
SELECT age('second', toDate('2017-12-31'), toDate('2017-01-01'), 'UTC');
|
||||
SELECT age('second', toDate('2017-12-31'), toDate('2018-01-01'), 'UTC');
|
||||
|
||||
SELECT 'DateTime arguments';
|
||||
SELECT age('day', toDateTime('2016-01-01 00:00:01', 'UTC'), toDateTime('2016-01-02 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDateTime('2016-01-01 00:00:01', 'UTC'), toDateTime('2016-01-02 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDateTime('2016-01-01 00:00:01', 'UTC'), toDateTime('2016-01-02 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('second', toDateTime('2016-01-01 00:00:01', 'UTC'), toDateTime('2016-01-02 00:00:00', 'UTC'), 'UTC');
|
||||
|
||||
SELECT 'Date and DateTime arguments';
|
||||
|
||||
SELECT age('second', toDate('2017-12-31'), toDateTime('2016-01-01 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('second', toDateTime('2017-12-31 00:00:00', 'UTC'), toDate('2017-01-01'), 'UTC');
|
||||
SELECT age('second', toDateTime('2017-12-31 00:00:00', 'UTC'), toDateTime('2018-01-01 00:00:00', 'UTC'));
|
||||
|
||||
SELECT 'Constant and non-constant arguments';
|
||||
|
||||
SELECT age('minute', materialize(toDate('2017-12-31')), toDate('2016-01-01'), 'UTC');
|
||||
SELECT age('minute', toDate('2017-12-31'), materialize(toDate('2017-01-01')), 'UTC');
|
||||
SELECT age('minute', materialize(toDate('2017-12-31')), materialize(toDate('2018-01-01')), 'UTC');
|
||||
|
||||
SELECT 'Case insensitive';
|
||||
|
||||
SELECT age('year', today(), today() - INTERVAL 10 YEAR);
|
||||
|
||||
SELECT 'Dependance of timezones';
|
||||
|
||||
SELECT age('month', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
SELECT age('week', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
SELECT age('day', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
SELECT age('hour', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
SELECT age('minute', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
SELECT age('second', toDate('2014-10-26'), toDate('2014-10-27'), 'Asia/Istanbul');
|
||||
|
||||
SELECT age('month', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
SELECT age('week', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
SELECT age('day', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
SELECT age('hour', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
SELECT age('minute', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
SELECT age('second', toDate('2014-10-26'), toDate('2014-10-27'), 'UTC');
|
||||
|
||||
SELECT age('month', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
SELECT age('week', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
SELECT age('day', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
SELECT age('hour', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
SELECT age('minute', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
SELECT age('second', toDateTime('2014-10-26 00:00:00', 'Asia/Istanbul'), toDateTime('2014-10-27 00:00:00', 'Asia/Istanbul'));
|
||||
|
||||
SELECT age('month', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
SELECT age('week', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
SELECT age('day', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
SELECT age('hour', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
SELECT age('minute', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
SELECT age('second', toDateTime('2014-10-26 00:00:00', 'UTC'), toDateTime('2014-10-27 00:00:00', 'UTC'));
|
||||
|
||||
SELECT 'Additional test';
|
||||
|
||||
SELECT number = age('month', now() - INTERVAL number MONTH, now()) FROM system.numbers LIMIT 10;
|
||||
@ -1,169 +0,0 @@
|
||||
-- { echo }
|
||||
|
||||
-- Date32 vs Date32
|
||||
SELECT age('second', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-08', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDate32('1927-01-01', 'UTC'), toDate32('1927-02-01', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDate32('1927-01-01', 'UTC'), toDate32('1927-04-01', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDate32('1927-01-01', 'UTC'), toDate32('1928-01-01', 'UTC'), 'UTC');
|
||||
1
|
||||
-- With DateTime64
|
||||
-- Date32 vs DateTime64
|
||||
SELECT age('second', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-08 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-02-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-04-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDate32('1927-01-01', 'UTC'), toDateTime64('1928-01-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
-- DateTime64 vs Date32
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-08', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-02-01', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-04-01', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1928-01-01', 'UTC'), 'UTC');
|
||||
1
|
||||
-- With DateTime
|
||||
-- Date32 vs DateTime
|
||||
SELECT age('second', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-25 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDate32('2015-08-18', 'UTC'), toDateTime('2015-09-18 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDate32('2015-08-18', 'UTC'), toDateTime('2015-11-18 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDate32('2015-08-18', 'UTC'), toDateTime('2016-08-18 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
-- DateTime vs Date32
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-25', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-09-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-11-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2016-08-18', 'UTC'), 'UTC');
|
||||
1
|
||||
-- With Date
|
||||
-- Date32 vs Date
|
||||
SELECT age('second', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDate32('2015-08-18', 'UTC'), toDate('2015-08-25', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDate32('2015-08-18', 'UTC'), toDate('2015-09-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDate32('2015-08-18', 'UTC'), toDate('2015-11-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDate32('2015-08-18', 'UTC'), toDate('2016-08-18', 'UTC'), 'UTC');
|
||||
1
|
||||
-- Date vs Date32
|
||||
SELECT age('second', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
86400
|
||||
SELECT age('minute', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1440
|
||||
SELECT age('hour', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
24
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('week', toDate('2015-08-18', 'UTC'), toDate32('2015-08-25', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('month', toDate('2015-08-18', 'UTC'), toDate32('2015-09-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('quarter', toDate('2015-08-18', 'UTC'), toDate32('2015-11-18', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('year', toDate('2015-08-18', 'UTC'), toDate32('2016-08-18', 'UTC'), 'UTC');
|
||||
1
|
||||
-- Const vs non-const columns
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), materialize(toDateTime64('1927-01-02 00:00:00', 3, 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), materialize(toDateTime('2015-08-19 00:00:00', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDateTime('2015-08-18 00:00:00', 'UTC'), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), materialize(toDate('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
-- Non-const vs const columns
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDateTime64('1927-01-01 00:00:00', 3, 'UTC')), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
1
|
||||
-- Non-const vs non-const columns
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), materialize(toDateTime64('1927-01-02 00:00:00', 3, 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDateTime64('1927-01-01 00:00:00', 3, 'UTC')), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), materialize(toDateTime('2015-08-19 00:00:00', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), materialize(toDate('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
1
|
||||
@ -1,101 +0,0 @@
|
||||
-- { echo }
|
||||
|
||||
-- Date32 vs Date32
|
||||
SELECT age('second', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDate32('1927-01-01', 'UTC'), toDate32('1927-01-08', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDate32('1927-01-01', 'UTC'), toDate32('1927-02-01', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDate32('1927-01-01', 'UTC'), toDate32('1927-04-01', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDate32('1927-01-01', 'UTC'), toDate32('1928-01-01', 'UTC'), 'UTC');
|
||||
|
||||
-- With DateTime64
|
||||
-- Date32 vs DateTime64
|
||||
SELECT age('second', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('week', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-01-08 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('month', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-02-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDate32('1927-01-01', 'UTC'), toDateTime64('1927-04-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('year', toDate32('1927-01-01', 'UTC'), toDateTime64('1928-01-01 00:00:00', 3, 'UTC'), 'UTC');
|
||||
|
||||
-- DateTime64 vs Date32
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-01-08', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-02-01', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1927-04-01', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), toDate32('1928-01-01', 'UTC'), 'UTC');
|
||||
|
||||
-- With DateTime
|
||||
-- Date32 vs DateTime
|
||||
SELECT age('second', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDate32('2015-08-18', 'UTC'), toDateTime('2015-08-25 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDate32('2015-08-18', 'UTC'), toDateTime('2015-09-18 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDate32('2015-08-18', 'UTC'), toDateTime('2015-11-18 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDate32('2015-08-18', 'UTC'), toDateTime('2016-08-18 00:00:00', 'UTC'), 'UTC');
|
||||
|
||||
-- DateTime vs Date32
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-08-25', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-09-18', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2015-11-18', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDateTime('2015-08-18 00:00:00', 'UTC'), toDate32('2016-08-18', 'UTC'), 'UTC');
|
||||
|
||||
-- With Date
|
||||
-- Date32 vs Date
|
||||
SELECT age('second', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDate32('2015-08-18', 'UTC'), toDate('2015-08-25', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDate32('2015-08-18', 'UTC'), toDate('2015-09-18', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDate32('2015-08-18', 'UTC'), toDate('2015-11-18', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDate32('2015-08-18', 'UTC'), toDate('2016-08-18', 'UTC'), 'UTC');
|
||||
|
||||
-- Date vs Date32
|
||||
SELECT age('second', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('minute', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('hour', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('week', toDate('2015-08-18', 'UTC'), toDate32('2015-08-25', 'UTC'), 'UTC');
|
||||
SELECT age('month', toDate('2015-08-18', 'UTC'), toDate32('2015-09-18', 'UTC'), 'UTC');
|
||||
SELECT age('quarter', toDate('2015-08-18', 'UTC'), toDate32('2015-11-18', 'UTC'), 'UTC');
|
||||
SELECT age('year', toDate('2015-08-18', 'UTC'), toDate32('2016-08-18', 'UTC'), 'UTC');
|
||||
|
||||
-- Const vs non-const columns
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
SELECT age('day', toDate32('1927-01-01', 'UTC'), materialize(toDateTime64('1927-01-02 00:00:00', 3, 'UTC')), 'UTC');
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 3, 'UTC'), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), materialize(toDateTime('2015-08-19 00:00:00', 'UTC')), 'UTC');
|
||||
SELECT age('day', toDateTime('2015-08-18 00:00:00', 'UTC'), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
SELECT age('day', toDate32('2015-08-18', 'UTC'), materialize(toDate('2015-08-19', 'UTC')), 'UTC');
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
|
||||
-- Non-const vs const columns
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDateTime64('1927-01-01 00:00:00', 3, 'UTC')), toDate32('1927-01-02', 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), toDateTime('2015-08-19 00:00:00', 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), toDate('2015-08-19', 'UTC'), 'UTC');
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), toDate32('2015-08-19', 'UTC'), 'UTC');
|
||||
|
||||
-- Non-const vs non-const columns
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('1927-01-01', 'UTC')), materialize(toDateTime64('1927-01-02 00:00:00', 3, 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDateTime64('1927-01-01 00:00:00', 3, 'UTC')), materialize(toDate32('1927-01-02', 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), materialize(toDateTime('2015-08-19 00:00:00', 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDate32('2015-08-18', 'UTC')), materialize(toDate('2015-08-19', 'UTC')), 'UTC');
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), materialize(toDate32('2015-08-19', 'UTC')), 'UTC');
|
||||
@ -1,113 +0,0 @@
|
||||
-- { echo }
|
||||
|
||||
-- DateTime64 vs DateTime64 same scale
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 00:00:10', 0, 'UTC'));
|
||||
10
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 00:10:00', 0, 'UTC'));
|
||||
600
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 01:00:00', 0, 'UTC'));
|
||||
3600
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 01:10:10', 0, 'UTC'));
|
||||
4210
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 00:10:00', 0, 'UTC'));
|
||||
10
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 10:00:00', 0, 'UTC'));
|
||||
600
|
||||
SELECT age('hour', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-01 10:00:00', 0, 'UTC'));
|
||||
10
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-01-02 00:00:00', 0, 'UTC'));
|
||||
1
|
||||
SELECT age('month', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1927-02-01 00:00:00', 0, 'UTC'));
|
||||
1
|
||||
SELECT age('year', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), toDateTime64('1928-01-01 00:00:00', 0, 'UTC'));
|
||||
1
|
||||
-- DateTime64 vs DateTime64 different scale
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 00:00:10', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 00:10:00', 3, 'UTC'));
|
||||
600
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 01:00:00', 3, 'UTC'));
|
||||
3600
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 01:10:10', 3, 'UTC'));
|
||||
4210
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 00:10:00', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('minute', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 10:00:00', 3, 'UTC'));
|
||||
600
|
||||
SELECT age('hour', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-01 10:00:00', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('day', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-01-02 00:00:00', 3, 'UTC'));
|
||||
1
|
||||
SELECT age('month', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1927-02-01 00:00:00', 3, 'UTC'));
|
||||
1
|
||||
SELECT age('year', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), toDateTime64('1928-01-01 00:00:00', 3, 'UTC'));
|
||||
1
|
||||
-- With DateTime
|
||||
-- DateTime64 vs DateTime
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDateTime('2015-08-18 00:00:00', 'UTC'));
|
||||
0
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDateTime('2015-08-18 00:00:10', 'UTC'));
|
||||
10
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDateTime('2015-08-18 00:10:00', 'UTC'));
|
||||
600
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDateTime('2015-08-18 01:00:00', 'UTC'));
|
||||
3600
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDateTime('2015-08-18 01:10:10', 'UTC'));
|
||||
4210
|
||||
-- DateTime vs DateTime64
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDateTime64('2015-08-18 00:00:00', 3, 'UTC'));
|
||||
0
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDateTime64('2015-08-18 00:00:10', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDateTime64('2015-08-18 00:10:00', 3, 'UTC'));
|
||||
600
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDateTime64('2015-08-18 01:00:00', 3, 'UTC'));
|
||||
3600
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), toDateTime64('2015-08-18 01:10:10', 3, 'UTC'));
|
||||
4210
|
||||
-- With Date
|
||||
-- DateTime64 vs Date
|
||||
SELECT age('day', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), toDate('2015-08-19', 'UTC'));
|
||||
1
|
||||
-- Date vs DateTime64
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), toDateTime64('2015-08-19 00:00:00', 3, 'UTC'));
|
||||
1
|
||||
-- Same thing but const vs non-const columns
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 0, 'UTC'), materialize(toDateTime64('1927-01-01 00:00:10', 0, 'UTC')));
|
||||
10
|
||||
SELECT age('second', toDateTime64('1927-01-01 00:00:00', 6, 'UTC'), materialize(toDateTime64('1927-01-01 00:00:10', 3, 'UTC')));
|
||||
10
|
||||
SELECT age('second', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), materialize(toDateTime('2015-08-18 00:00:10', 'UTC')));
|
||||
10
|
||||
SELECT age('second', toDateTime('2015-08-18 00:00:00', 'UTC'), materialize(toDateTime64('2015-08-18 00:00:10', 3, 'UTC')));
|
||||
10
|
||||
SELECT age('day', toDateTime64('2015-08-18 00:00:00', 0, 'UTC'), materialize(toDate('2015-08-19', 'UTC')));
|
||||
1
|
||||
SELECT age('day', toDate('2015-08-18', 'UTC'), materialize(toDateTime64('2015-08-19 00:00:00', 3, 'UTC')));
|
||||
1
|
||||
-- Same thing but non-const vs const columns
|
||||
SELECT age('second', materialize(toDateTime64('1927-01-01 00:00:00', 0, 'UTC')), toDateTime64('1927-01-01 00:00:10', 0, 'UTC'));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime64('1927-01-01 00:00:00', 6, 'UTC')), toDateTime64('1927-01-01 00:00:10', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime64('2015-08-18 00:00:00', 0, 'UTC')), toDateTime('2015-08-18 00:00:10', 'UTC'));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), toDateTime64('2015-08-18 00:00:10', 3, 'UTC'));
|
||||
10
|
||||
SELECT age('day', materialize(toDateTime64('2015-08-18 00:00:00', 0, 'UTC')), toDate('2015-08-19', 'UTC'));
|
||||
1
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), toDateTime64('2015-08-19 00:00:00', 3, 'UTC'));
|
||||
1
|
||||
-- Same thing but non-const vs non-const columns
|
||||
SELECT age('second', materialize(toDateTime64('1927-01-01 00:00:00', 0, 'UTC')), materialize(toDateTime64('1927-01-01 00:00:10', 0, 'UTC')));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime64('1927-01-01 00:00:00', 6, 'UTC')), materialize(toDateTime64('1927-01-01 00:00:10', 3, 'UTC')));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime64('2015-08-18 00:00:00', 0, 'UTC')), materialize(toDateTime('2015-08-18 00:00:10', 'UTC')));
|
||||
10
|
||||
SELECT age('second', materialize(toDateTime('2015-08-18 00:00:00', 'UTC')), materialize(toDateTime64('2015-08-18 00:00:10', 3, 'UTC')));
|
||||
10
|
||||
SELECT age('day', materialize(toDateTime64('2015-08-18 00:00:00', 0, 'UTC')), materialize(toDate('2015-08-19', 'UTC')));
|
||||
1
|
||||
SELECT age('day', materialize(toDate('2015-08-18', 'UTC')), materialize(toDateTime64('2015-08-19 00:00:00', 3, 'UTC')));
|
||||
1
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user