mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-11 17:02:25 +00:00
Merge branch 'master' into ch-disks-remove
This commit is contained in:
Vladimir Cherkasov
GitHub
commit
3352a7b4af
160
CHANGELOG.md
160
CHANGELOG.md
@ -1,4 +1,5 @@
|
||||
### Table of Contents
|
||||
**[ClickHouse release v24.10, 2024-10-31](#2410)**<br/>
|
||||
**[ClickHouse release v24.9, 2024-09-26](#249)**<br/>
|
||||
**[ClickHouse release v24.8 LTS, 2024-08-20](#248)**<br/>
|
||||
**[ClickHouse release v24.7, 2024-07-30](#247)**<br/>
|
||||
@ -12,6 +13,165 @@
|
||||
|
||||
# 2024 Changelog
|
||||
|
||||
### <a id="2410"></a> ClickHouse release 24.10, 2024-10-31
|
||||
|
||||
#### Backward Incompatible Change
|
||||
* Allow to write `SETTINGS` before `FORMAT` in a chain of queries with `UNION` when subqueries are inside parentheses. This closes [#39712](https://github.com/ClickHouse/ClickHouse/issues/39712). Change the behavior when a query has the SETTINGS clause specified twice in a sequence. The closest SETTINGS clause will have a preference for the corresponding subquery. In the previous versions, the outermost SETTINGS clause could take a preference over the inner one. [#68614](https://github.com/ClickHouse/ClickHouse/pull/68614) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Reordering of filter conditions from `[PRE]WHERE` clause is now allowed by default. It could be disabled by setting `allow_reorder_prewhere_conditions` to `false`. [#70657](https://github.com/ClickHouse/ClickHouse/pull/70657) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Remove the `idxd-config` library, which has an incompatible license. This also removes the experimental Intel DeflateQPL codec. [#70987](https://github.com/ClickHouse/ClickHouse/pull/70987) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
|
||||
#### New Feature
|
||||
* Allow to grant access to the wildcard prefixes. `GRANT SELECT ON db.table_pefix_* TO user`. [#65311](https://github.com/ClickHouse/ClickHouse/pull/65311) ([pufit](https://github.com/pufit)).
|
||||
* If you press space bar during query runtime, the client will display a real-time table with detailed metrics. You can enable it globally with the new `--progress-table` option in clickhouse-client; a new `--enable-progress-table-toggle` is associated with the `--progress-table` option, and toggles the rendering of the progress table by pressing the control key (Space). [#63689](https://github.com/ClickHouse/ClickHouse/pull/63689) ([Maria Khristenko](https://github.com/mariaKhr)), [#70423](https://github.com/ClickHouse/ClickHouse/pull/70423) ([Julia Kartseva](https://github.com/jkartseva)).
|
||||
* Allow to cache read files for object storage table engines and data lakes using hash from ETag + file path as cache key. [#70135](https://github.com/ClickHouse/ClickHouse/pull/70135) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Support creating a table with a query: `CREATE TABLE ... CLONE AS ...`. It clones the source table's schema and then attaches all partitions to the newly created table. This feature is only supported with tables of the `MergeTree` family Closes [#65015](https://github.com/ClickHouse/ClickHouse/issues/65015). [#69091](https://github.com/ClickHouse/ClickHouse/pull/69091) ([tuanpach](https://github.com/tuanpach)).
|
||||
* Add a new system table, `system.query_metric_log` which contains history of memory and metric values from table system.events for individual queries, periodically flushed to disk. [#66532](https://github.com/ClickHouse/ClickHouse/pull/66532) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
* A simple SELECT query can be written with implicit SELECT to enable calculator-style expressions, e.g., `ch "1 + 2"`. This is controlled by a new setting, `implicit_select`. [#68502](https://github.com/ClickHouse/ClickHouse/pull/68502) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Support the `--copy` mode for clickhouse local as a shortcut for format conversion [#68503](https://github.com/ClickHouse/ClickHouse/issues/68503). [#68583](https://github.com/ClickHouse/ClickHouse/pull/68583) ([Denis Hananein](https://github.com/denis-hananein)).
|
||||
* Add a builtin HTML page for visualizing merges which is available at the `/merges` path. [#70821](https://github.com/ClickHouse/ClickHouse/pull/70821) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add support for `arrayUnion` function. [#68989](https://github.com/ClickHouse/ClickHouse/pull/68989) ([Peter Nguyen](https://github.com/petern48)).
|
||||
* Allow parametrised SQL aliases. [#50665](https://github.com/ClickHouse/ClickHouse/pull/50665) ([Anton Kozlov](https://github.com/tonickkozlov)).
|
||||
* A new aggregate function `quantileExactWeightedInterpolated`, which is a interpolated version based on quantileExactWeighted. Some people may wonder why we need a new `quantileExactWeightedInterpolated` since we already have `quantileExactInterpolatedWeighted`. The reason is the new one is more accurate than the old one. This is for spark compatibility. [#69619](https://github.com/ClickHouse/ClickHouse/pull/69619) ([李扬](https://github.com/taiyang-li)).
|
||||
* A new function `arrayElementOrNull`. It returns `NULL` if the array index is out of range or a Map key not found. [#69646](https://github.com/ClickHouse/ClickHouse/pull/69646) ([李扬](https://github.com/taiyang-li)).
|
||||
* Allows users to specify regular expressions through new `message_regexp` and `message_regexp_negative` fields in the `config.xml` file to filter out logging. The logging is applied to the formatted un-colored text for the most intuitive developer experience. [#69657](https://github.com/ClickHouse/ClickHouse/pull/69657) ([Peter Nguyen](https://github.com/petern48)).
|
||||
* Added `RIPEMD160` function, which computes the RIPEMD-160 cryptographic hash of a string. Example: `SELECT HEX(RIPEMD160('The quick brown fox jumps over the lazy dog'))` returns `37F332F68DB77BD9D7EDD4969571AD671CF9DD3B`. [#70087](https://github.com/ClickHouse/ClickHouse/pull/70087) ([Dergousov Maxim](https://github.com/m7kss1)).
|
||||
* Support reading `Iceberg` tables on `HDFS`. [#70268](https://github.com/ClickHouse/ClickHouse/pull/70268) ([flynn](https://github.com/ucasfl)).
|
||||
* Support for CTE in the form of `WITH ... INSERT`, as previously we only supported `INSERT ... WITH ...`. [#70593](https://github.com/ClickHouse/ClickHouse/pull/70593) ([Shichao Jin](https://github.com/jsc0218)).
|
||||
* MongoDB integration: support for all MongoDB types, support for WHERE and ORDER BY statements on MongoDB side, restriction for expressions unsupported by MongoDB. Note that the new inegration is disabled by default, to use it, please set `<use_legacy_mongodb_integration>` to `false` in server config. [#63279](https://github.com/ClickHouse/ClickHouse/pull/63279) ([Kirill Nikiforov](https://github.com/allmazz)).
|
||||
* A new function `getSettingOrDefault` added to return the default value and avoid exception if a custom setting is not found in the current profile. [#69917](https://github.com/ClickHouse/ClickHouse/pull/69917) ([Shankar](https://github.com/shiyer7474)).
|
||||
|
||||
#### Experimental feature
|
||||
* Refreshable materialized views are production ready. [#70550](https://github.com/ClickHouse/ClickHouse/pull/70550) ([Michael Kolupaev](https://github.com/al13n321)). Refreshable materialized views are now supported in Replicated databases. [#60669](https://github.com/ClickHouse/ClickHouse/pull/60669) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* Parallel replicas are moved from experimental to beta. Reworked settings that control the behavior of parallel replicas algorithms. A quick recap: ClickHouse has four different algorithms for parallel reading involving multiple replicas, which is reflected in the setting `parallel_replicas_mode`, the default value for it is `read_tasks` Additionally, the toggle-switch setting `enable_parallel_replicas` has been added. [#63151](https://github.com/ClickHouse/ClickHouse/pull/63151) ([Alexey Milovidov](https://github.com/alexey-milovidov)), ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Support for the `Dynamic` type in most functions by executing them on internal types inside `Dynamic`. [#69691](https://github.com/ClickHouse/ClickHouse/pull/69691) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Allow to read/write the `JSON` type as a binary string in `RowBinary` format under settings `input_format_binary_read_json_as_string/output_format_binary_write_json_as_string`. [#70288](https://github.com/ClickHouse/ClickHouse/pull/70288) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Allow to serialize/deserialize `JSON` column as single String column in the Native format. For output use setting `output_format_native_write_json_as_string`. For input, use serialization version `1` before the column data. [#70312](https://github.com/ClickHouse/ClickHouse/pull/70312) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Introduced a special (experimental) mode of a merge selector for MergeTree tables which makes it more aggressive for the partitions that are close to the limit by the number of parts. It is controlled by the `merge_selector_use_blurry_base` MergeTree-level setting. [#70645](https://github.com/ClickHouse/ClickHouse/pull/70645) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Implement generic ser/de between Avro's `Union` and ClickHouse's `Variant` types. Resolves [#69713](https://github.com/ClickHouse/ClickHouse/issues/69713). [#69712](https://github.com/ClickHouse/ClickHouse/pull/69712) ([Jiří Kozlovský](https://github.com/jirislav)).
|
||||
|
||||
#### Performance Improvement
|
||||
* Refactor `IDisk` and `IObjectStorage` for better performance. Tables from `plain` and `plain_rewritable` object storages will initialize faster. [#68146](https://github.com/ClickHouse/ClickHouse/pull/68146) ([Alexey Milovidov](https://github.com/alexey-milovidov), [Julia Kartseva](https://github.com/jkartseva)). Do not call the LIST object storage API when determining if a file or directory exists on the plain rewritable disk, as it can be cost-inefficient. [#70852](https://github.com/ClickHouse/ClickHouse/pull/70852) ([Julia Kartseva](https://github.com/jkartseva)). Reduce the number of object storage HEAD API requests in the plain_rewritable disk. [#70915](https://github.com/ClickHouse/ClickHouse/pull/70915) ([Julia Kartseva](https://github.com/jkartseva)).
|
||||
* Added an ability to parse data directly into sparse columns. [#69828](https://github.com/ClickHouse/ClickHouse/pull/69828) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Improved performance of parsing formats with high number of missed values (e.g. `JSONEachRow`). [#69875](https://github.com/ClickHouse/ClickHouse/pull/69875) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Supports parallel reading of parquet row groups and prefetching of row groups in single-threaded mode. [#69862](https://github.com/ClickHouse/ClickHouse/pull/69862) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
* Support minmax index for `pointInPolygon`. [#62085](https://github.com/ClickHouse/ClickHouse/pull/62085) ([JackyWoo](https://github.com/JackyWoo)).
|
||||
* Use bloom filters when reading Parquet files. [#62966](https://github.com/ClickHouse/ClickHouse/pull/62966) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Lock-free parts rename to avoid INSERT affect SELECT (due to parts lock) (under normal circumstances with `fsync_part_directory`, QPS of SELECT with INSERT in parallel, increased 2x, under heavy load the effect is even bigger). Note, this only includes `ReplicatedMergeTree` for now. [#64955](https://github.com/ClickHouse/ClickHouse/pull/64955) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Respect `ttl_only_drop_parts` on `materialize ttl`; only read necessary columns to recalculate TTL and drop parts by replacing them with an empty one. [#65488](https://github.com/ClickHouse/ClickHouse/pull/65488) ([Andrey Zvonov](https://github.com/zvonand)).
|
||||
* Optimized thread creation in the ThreadPool to minimize lock contention. Thread creation is now performed outside of the critical section to avoid delays in job scheduling and thread management under high load conditions. This leads to a much more responsive ClickHouse under heavy concurrent load. [#68694](https://github.com/ClickHouse/ClickHouse/pull/68694) ([filimonov](https://github.com/filimonov)).
|
||||
* Enable reading `LowCardinality` string columns from `ORC`. [#69481](https://github.com/ClickHouse/ClickHouse/pull/69481) ([李扬](https://github.com/taiyang-li)).
|
||||
* Use `LowCardinality` for `ProfileEvents` in system logs such as `part_log`, `query_views_log`, `filesystem_cache_log`. [#70152](https://github.com/ClickHouse/ClickHouse/pull/70152) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Improve performance of `fromUnixTimestamp`/`toUnixTimestamp` functions. [#71042](https://github.com/ClickHouse/ClickHouse/pull/71042) ([kevinyhzou](https://github.com/KevinyhZou)).
|
||||
* Don't disable nonblocking read from page cache for the entire server when reading from a blocking I/O. This was leading to a poorer performance when a single filesystem (e.g., tmpfs) didn't support the `preadv2` syscall while others do. [#70299](https://github.com/ClickHouse/ClickHouse/pull/70299) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* `ALTER TABLE .. REPLACE PARTITION` doesn't wait anymore for mutations/merges that happen in other partitions. [#59138](https://github.com/ClickHouse/ClickHouse/pull/59138) ([Vasily Nemkov](https://github.com/Enmk)).
|
||||
* Don't do validation when synchronizing ACL from Keeper. It's validating during creation. It shouldn't matter that much, but there are installations with tens of thousands or even more user created, and the unnecessary hash validation can take a long time to finish during server startup (it synchronizes everything from keeper). [#70644](https://github.com/ClickHouse/ClickHouse/pull/70644) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
|
||||
#### Improvement
|
||||
* `CREATE TABLE AS` will copy `PRIMARY KEY`, `ORDER BY`, and similar clauses (of `MergeTree` tables). [#69739](https://github.com/ClickHouse/ClickHouse/pull/69739) ([sakulali](https://github.com/sakulali)).
|
||||
* Support 64-bit XID in Keeper. It can be enabled with the `use_xid_64` configuration value. [#69908](https://github.com/ClickHouse/ClickHouse/pull/69908) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Command-line arguments for Bool settings are set to true when no value is provided for the argument (e.g. `clickhouse-client --optimize_aggregation_in_order --query "SELECT 1"`). [#70459](https://github.com/ClickHouse/ClickHouse/pull/70459) ([davidtsuk](https://github.com/davidtsuk)).
|
||||
* Added user-level settings `min_free_disk_bytes_to_throw_insert` and `min_free_disk_ratio_to_throw_insert` to prevent insertions on disks that are almost full. [#69755](https://github.com/ClickHouse/ClickHouse/pull/69755) ([Marco Vilas Boas](https://github.com/marco-vb)).
|
||||
* Embedded documentation for settings will be strictly more detailed and complete than the documentation on the website. This is the first step before making the website documentation always auto-generated from the source code. This has long-standing implications: - it will be guaranteed to have every setting; - there is no chance of having default values obsolete; - we can generate this documentation for each ClickHouse version; - the documentation can be displayed by the server itself even without Internet access. Generate the docs on the website from the source code. [#70289](https://github.com/ClickHouse/ClickHouse/pull/70289) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Allow empty needle in the function `replace`, the same behavior with PostgreSQL. [#69918](https://github.com/ClickHouse/ClickHouse/pull/69918) ([zhanglistar](https://github.com/zhanglistar)).
|
||||
* Allow empty needle in functions `replaceRegexp*`. [#70053](https://github.com/ClickHouse/ClickHouse/pull/70053) ([zhanglistar](https://github.com/zhanglistar)).
|
||||
* Symbolic links for tables in the `data/database_name/` directory are created for the actual paths to the table's data, depending on the storage policy, instead of the `store/...` directory on the default disk. [#61777](https://github.com/ClickHouse/ClickHouse/pull/61777) ([Kirill](https://github.com/kirillgarbar)).
|
||||
* While parsing an `Enum` field from `JSON`, a string containing an integer will be interpreted as the corresponding `Enum` element. This closes [#65119](https://github.com/ClickHouse/ClickHouse/issues/65119). [#66801](https://github.com/ClickHouse/ClickHouse/pull/66801) ([scanhex12](https://github.com/scanhex12)).
|
||||
* Allow `TRIM` -ing `LEADING` or `TRAILING` empty string as a no-op. Closes [#67792](https://github.com/ClickHouse/ClickHouse/issues/67792). [#68455](https://github.com/ClickHouse/ClickHouse/pull/68455) ([Peter Nguyen](https://github.com/petern48)).
|
||||
* Improve compatibility of `cast(timestamp as String)` with Spark. [#69179](https://github.com/ClickHouse/ClickHouse/pull/69179) ([Wenzheng Liu](https://github.com/lwz9103)).
|
||||
* Always use the new analyzer to calculate constant expressions when `enable_analyzer` is set to `true`. Support calculation of `executable` table function arguments without using `SELECT` query for constant expressions. [#69292](https://github.com/ClickHouse/ClickHouse/pull/69292) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Add a setting `enable_secure_identifiers` to disallow identifiers with special characters. [#69411](https://github.com/ClickHouse/ClickHouse/pull/69411) ([tuanpach](https://github.com/tuanpach)).
|
||||
* Add `show_create_query_identifier_quoting_rule` to define identifier quoting behavior in the `SHOW CREATE TABLE` query result. Possible values: - `user_display`: When the identifiers is a keyword. - `when_necessary`: When the identifiers is one of `{"distinct", "all", "table"}` and when it could lead to ambiguity: column names, dictionary attribute names. - `always`: Always quote identifiers. [#69448](https://github.com/ClickHouse/ClickHouse/pull/69448) ([tuanpach](https://github.com/tuanpach)).
|
||||
* Improve restoring of access entities' dependencies [#69563](https://github.com/ClickHouse/ClickHouse/pull/69563) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* If you run `clickhouse-client` or other CLI application, and it starts up slowly due to an overloaded server, and you start typing your query, such as `SELECT`, the previous versions will display the remaining of the terminal echo contents before printing the greetings message, such as `SELECTClickHouse local version 24.10.1.1.` instead of `ClickHouse local version 24.10.1.1.`. Now it is fixed. This closes [#31696](https://github.com/ClickHouse/ClickHouse/issues/31696). [#69856](https://github.com/ClickHouse/ClickHouse/pull/69856) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add new column `readonly_duration` to the `system.replicas` table. Needed to be able to distinguish actual readonly replicas from sentinel ones in alerts. [#69871](https://github.com/ClickHouse/ClickHouse/pull/69871) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)).
|
||||
* Change the type of `join_output_by_rowlist_perkey_rows_threshold` setting type to unsigned integer. [#69886](https://github.com/ClickHouse/ClickHouse/pull/69886) ([kevinyhzou](https://github.com/KevinyhZou)).
|
||||
* Enhance OpenTelemetry span logging to include query settings. [#70011](https://github.com/ClickHouse/ClickHouse/pull/70011) ([sharathks118](https://github.com/sharathks118)).
|
||||
* Add diagnostic info about higher-order array functions if lambda result type is unexpected. [#70093](https://github.com/ClickHouse/ClickHouse/pull/70093) ([ttanay](https://github.com/ttanay)).
|
||||
* Keeper improvement: less locking during cluster changes. [#70275](https://github.com/ClickHouse/ClickHouse/pull/70275) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Add `WITH IMPLICIT` and `FINAL` keywords to the `SHOW GRANTS` command. Fix a minor bug with implicit grants: [#70094](https://github.com/ClickHouse/ClickHouse/issues/70094). [#70293](https://github.com/ClickHouse/ClickHouse/pull/70293) ([pufit](https://github.com/pufit)).
|
||||
* Respect `compatibility` for MergeTree settings. The `compatibility` value is taken from the `default` profile on server startup, and default MergeTree settings are changed accordingly. Further changes of the `compatibility` setting do not affect MergeTree settings. [#70322](https://github.com/ClickHouse/ClickHouse/pull/70322) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Avoid spamming the logs with large HTTP response bodies in case of errors during inter-server communication. [#70487](https://github.com/ClickHouse/ClickHouse/pull/70487) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Added a new setting `max_parts_to_move` to control the maximum number of parts that can be moved at once. [#70520](https://github.com/ClickHouse/ClickHouse/pull/70520) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Limit the frequency of certain log messages. [#70601](https://github.com/ClickHouse/ClickHouse/pull/70601) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* `CHECK TABLE` with `PART` qualifier was incorrectly formatted in the client. [#70660](https://github.com/ClickHouse/ClickHouse/pull/70660) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Support writing the column index and the offset index using parquet native writer. [#70669](https://github.com/ClickHouse/ClickHouse/pull/70669) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
* Support parsing `DateTime64` for microsecond and timezone in joda syntax ("joda" is a popular Java library for date and time, and the "joda syntax" is that library's style). [#70737](https://github.com/ClickHouse/ClickHouse/pull/70737) ([kevinyhzou](https://github.com/KevinyhZou)).
|
||||
* Changed an approach to figure out if a cloud storage supports [batch delete](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) or not. [#70786](https://github.com/ClickHouse/ClickHouse/pull/70786) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Support for Parquet page v2 in the native reader. [#70807](https://github.com/ClickHouse/ClickHouse/pull/70807) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* A check if table has both `storage_policy` and `disk` set. A check if a new storage policy is compatible with an old one when using `disk` setting is added. [#70839](https://github.com/ClickHouse/ClickHouse/pull/70839) ([Kirill](https://github.com/kirillgarbar)).
|
||||
* Add `system.s3_queue_settings` and `system.azure_queue_settings`. [#70841](https://github.com/ClickHouse/ClickHouse/pull/70841) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Functions `base58Encode` and `base58Decode` now accept arguments of type `FixedString`. Example: `SELECT base58Encode(toFixedString('plaintext', 9));`. [#70846](https://github.com/ClickHouse/ClickHouse/pull/70846) ([Faizan Patel](https://github.com/faizan2786)).
|
||||
* Add the `partition` column to every entry type of the part log. Previously, it was set only for some entries. This closes [#70819](https://github.com/ClickHouse/ClickHouse/issues/70819). [#70848](https://github.com/ClickHouse/ClickHouse/pull/70848) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add `MergeStart` and `MutateStart` events into `system.part_log` which helps with merges analysis and visualization. [#70850](https://github.com/ClickHouse/ClickHouse/pull/70850) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add a profile event about the number of merged source parts. It allows the monitoring of the fanout of the merge tree in production. [#70908](https://github.com/ClickHouse/ClickHouse/pull/70908) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Background downloads to the filesystem cache were enabled back. [#70929](https://github.com/ClickHouse/ClickHouse/pull/70929) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
* Add a new merge selector algorithm, named `Trivial`, for professional usage only. It is worse than the `Simple` merge selector. [#70969](https://github.com/ClickHouse/ClickHouse/pull/70969) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Support for atomic `CREATE OR REPLACE VIEW`. [#70536](https://github.com/ClickHouse/ClickHouse/pull/70536) ([tuanpach](https://github.com/tuanpach))
|
||||
* Added `strict_once` mode to aggregate function `windowFunnel` to avoid counting one event several times in case it matches multiple conditions, close [#21835](https://github.com/ClickHouse/ClickHouse/issues/21835). [#69738](https://github.com/ClickHouse/ClickHouse/pull/69738) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
* Apply configuration updates in global context object. It fixes issues like [#62308](https://github.com/ClickHouse/ClickHouse/issues/62308). [#62944](https://github.com/ClickHouse/ClickHouse/pull/62944) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix `ReadSettings` not using user set values, because defaults were only used. [#65625](https://github.com/ClickHouse/ClickHouse/pull/65625) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix type mismatch issue in `sumMapFiltered` when using signed arguments. [#58408](https://github.com/ClickHouse/ClickHouse/pull/58408) ([Chen768959](https://github.com/Chen768959)).
|
||||
* Fix toHour-like conversion functions' monotonicity when optional time zone argument is passed. [#60264](https://github.com/ClickHouse/ClickHouse/pull/60264) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Relax `supportsPrewhere` check for `Merge` tables. This fixes [#61064](https://github.com/ClickHouse/ClickHouse/issues/61064). It was hardened unnecessarily in [#60082](https://github.com/ClickHouse/ClickHouse/issues/60082). [#61091](https://github.com/ClickHouse/ClickHouse/pull/61091) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix `use_concurrency_control` setting handling for proper `concurrent_threads_soft_limit_num` limit enforcing. This enables concurrency control by default because previously it was broken. [#61473](https://github.com/ClickHouse/ClickHouse/pull/61473) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Fix incorrect `JOIN ON` section optimization in case of `IS NULL` check under any other function (like `NOT`) that may lead to wrong results. Closes [#67915](https://github.com/ClickHouse/ClickHouse/issues/67915). [#68049](https://github.com/ClickHouse/ClickHouse/pull/68049) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Prevent `ALTER` queries that would make the `CREATE` query of tables invalid. [#68574](https://github.com/ClickHouse/ClickHouse/pull/68574) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)).
|
||||
* Fix inconsistent AST formatting for `negate` (`-`) and `NOT` functions with tuples and arrays. [#68600](https://github.com/ClickHouse/ClickHouse/pull/68600) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Fix insertion of incomplete type into `Dynamic` during deserialization. It could lead to `Parameter out of bound` errors. [#69291](https://github.com/ClickHouse/ClickHouse/pull/69291) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Zero-copy replication, which is experimental and should not be used in production: fix inf loop after `restore replica` in the replicated merge tree with zero copy. [#69293](https://github.com/CljmnickHouse/ClickHouse/pull/69293) ([MikhailBurdukov](https://github.com/MikhailBurdukov)).
|
||||
* Return back default value of `processing_threads_num` as number of cpu cores in storage `S3Queue`. [#69384](https://github.com/ClickHouse/ClickHouse/pull/69384) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Bypass try/catch flow when de/serializing nested repeated protobuf to nested columns (fixes [#41971](https://github.com/ClickHouse/ClickHouse/issues/41971)). [#69556](https://github.com/ClickHouse/ClickHouse/pull/69556) ([Eliot Hautefeuille](https://github.com/hileef)).
|
||||
* Fix crash during insertion into FixedString column in PostgreSQL engine. [#69584](https://github.com/ClickHouse/ClickHouse/pull/69584) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix crash when executing `create view t as (with recursive 42 as ttt select ttt);`. [#69676](https://github.com/ClickHouse/ClickHouse/pull/69676) ([Han Fei](https://github.com/hanfei1991)).
|
||||
* Fixed `maxMapState` throwing 'Bad get' if value type is DateTime64. [#69787](https://github.com/ClickHouse/ClickHouse/pull/69787) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* Fix `getSubcolumn` with `LowCardinality` columns by overriding `useDefaultImplementationForLowCardinalityColumns` to return `true`. [#69831](https://github.com/ClickHouse/ClickHouse/pull/69831) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)).
|
||||
* Fix permanent blocked distributed sends if a DROP of distributed table failed. [#69843](https://github.com/ClickHouse/ClickHouse/pull/69843) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix non-cancellable queries containing WITH FILL with NaN keys. This closes [#69261](https://github.com/ClickHouse/ClickHouse/issues/69261). [#69845](https://github.com/ClickHouse/ClickHouse/pull/69845) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix analyzer default with old compatibility value. [#69895](https://github.com/ClickHouse/ClickHouse/pull/69895) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Don't check dependencies during CREATE OR REPLACE VIEW during DROP of old table. Previously CREATE OR REPLACE query failed when there are dependent tables of the recreated view. [#69907](https://github.com/ClickHouse/ClickHouse/pull/69907) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Something for Decimal. Fixes [#69730](https://github.com/ClickHouse/ClickHouse/issues/69730). [#69978](https://github.com/ClickHouse/ClickHouse/pull/69978) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Now DEFINER/INVOKER will work with parameterized views. [#69984](https://github.com/ClickHouse/ClickHouse/pull/69984) ([pufit](https://github.com/pufit)).
|
||||
* Fix parsing for view's definers. [#69985](https://github.com/ClickHouse/ClickHouse/pull/69985) ([pufit](https://github.com/pufit)).
|
||||
* Fixed a bug when the timezone could change the result of the query with a `Date` or `Date32` arguments. [#70036](https://github.com/ClickHouse/ClickHouse/pull/70036) ([Yarik Briukhovetskyi](https://github.com/yariks5s)).
|
||||
* Fixes `Block structure mismatch` for queries with nested views and `WHERE` condition. Fixes [#66209](https://github.com/ClickHouse/ClickHouse/issues/66209). [#70054](https://github.com/ClickHouse/ClickHouse/pull/70054) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Avoid reusing columns among different named tuples when evaluating `tuple` functions. This fixes [#70022](https://github.com/ClickHouse/ClickHouse/issues/70022). [#70103](https://github.com/ClickHouse/ClickHouse/pull/70103) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix wrong LOGICAL_ERROR when replacing literals in ranges. [#70122](https://github.com/ClickHouse/ClickHouse/pull/70122) ([Pablo Marcos](https://github.com/pamarcos)).
|
||||
* Check for Nullable(Nothing) type during ALTER TABLE MODIFY COLUMN/QUERY to prevent tables with such data type. [#70123](https://github.com/ClickHouse/ClickHouse/pull/70123) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Proper error message for illegal query `JOIN ... ON *` , close [#68650](https://github.com/ClickHouse/ClickHouse/issues/68650). [#70124](https://github.com/ClickHouse/ClickHouse/pull/70124) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Fix wrong result with skipping index. [#70127](https://github.com/ClickHouse/ClickHouse/pull/70127) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix data race in ColumnObject/ColumnTuple decompress method that could lead to heap use after free. [#70137](https://github.com/ClickHouse/ClickHouse/pull/70137) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix possible hung in ALTER COLUMN with Dynamic type. [#70144](https://github.com/ClickHouse/ClickHouse/pull/70144) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Now ClickHouse will consider more errors as retriable and will not mark data parts as broken in case of such errors. [#70145](https://github.com/ClickHouse/ClickHouse/pull/70145) ([alesapin](https://github.com/alesapin)).
|
||||
* Use correct `max_types` parameter during Dynamic type creation for JSON subcolumn. [#70147](https://github.com/ClickHouse/ClickHouse/pull/70147) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix the password being displayed in `system.query_log` for users with bcrypt password authentication method. [#70148](https://github.com/ClickHouse/ClickHouse/pull/70148) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Fix event counter for the native interface (InterfaceNativeSendBytes). [#70153](https://github.com/ClickHouse/ClickHouse/pull/70153) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
|
||||
* Fix possible crash related to JSON columns. [#70172](https://github.com/ClickHouse/ClickHouse/pull/70172) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix multiple issues with arrayMin and arrayMax. [#70207](https://github.com/ClickHouse/ClickHouse/pull/70207) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Respect setting allow_simdjson in the JSON type parser. [#70218](https://github.com/ClickHouse/ClickHouse/pull/70218) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix a null pointer dereference on creating a materialized view with two selects and an `INTERSECT`, e.g. `CREATE MATERIALIZED VIEW v0 AS (SELECT 1) INTERSECT (SELECT 1);`. [#70264](https://github.com/ClickHouse/ClickHouse/pull/70264) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Don't modify global settings with startup scripts. Previously, changing a setting in a startup script would change it globally. [#70310](https://github.com/ClickHouse/ClickHouse/pull/70310) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix ALTER of `Dynamic` type with reducing max_types parameter that could lead to server crash. [#70328](https://github.com/ClickHouse/ClickHouse/pull/70328) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Fix crash when using WITH FILL incorrectly. [#70338](https://github.com/ClickHouse/ClickHouse/pull/70338) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix possible use-after-free in `SYSTEM DROP FORMAT SCHEMA CACHE FOR Protobuf`. [#70358](https://github.com/ClickHouse/ClickHouse/pull/70358) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix crash during GROUP BY JSON sub-object subcolumn. [#70374](https://github.com/ClickHouse/ClickHouse/pull/70374) ([Pavel Kruglov](https://github.com/Avogar)).
|
||||
* Don't prefetch parts for vertical merges if part has no rows. [#70452](https://github.com/ClickHouse/ClickHouse/pull/70452) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix crash in WHERE with lambda functions. [#70464](https://github.com/ClickHouse/ClickHouse/pull/70464) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix table creation with `CREATE ... AS table_function(...)` with database `Replicated` and unavailable table function source on secondary replica. [#70511](https://github.com/ClickHouse/ClickHouse/pull/70511) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Ignore all output on async insert with `wait_for_async_insert=1`. Closes [#62644](https://github.com/ClickHouse/ClickHouse/issues/62644). [#70530](https://github.com/ClickHouse/ClickHouse/pull/70530) ([Konstantin Bogdanov](https://github.com/thevar1able)).
|
||||
* Ignore frozen_metadata.txt while traversing shadow directory from system.remote_data_paths. [#70590](https://github.com/ClickHouse/ClickHouse/pull/70590) ([Aleksei Filatov](https://github.com/aalexfvk)).
|
||||
* Fix creation of stateful window functions on misaligned memory. [#70631](https://github.com/ClickHouse/ClickHouse/pull/70631) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fixed rare crashes in `SELECT`-s and merges after adding a column of `Array` type with non-empty default expression. [#70695](https://github.com/ClickHouse/ClickHouse/pull/70695) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Insert into table function s3 will respect query settings. [#70696](https://github.com/ClickHouse/ClickHouse/pull/70696) ([Vladimir Cherkasov](https://github.com/vdimir)).
|
||||
* Fix infinite recursion when inferring a protobuf schema when skipping unsupported fields is enabled. [#70697](https://github.com/ClickHouse/ClickHouse/pull/70697) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Disable enable_named_columns_in_function_tuple by default. [#70833](https://github.com/ClickHouse/ClickHouse/pull/70833) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix S3Queue table engine setting processing_threads_num not being effective in case it was deduced from the number of cpu cores on the server. [#70837](https://github.com/ClickHouse/ClickHouse/pull/70837) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Normalize named tuple arguments in aggregation states. This fixes [#69732](https://github.com/ClickHouse/ClickHouse/issues/69732) . [#70853](https://github.com/ClickHouse/ClickHouse/pull/70853) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix a logical error due to negative zeros in the two-level hash table. This closes [#70973](https://github.com/ClickHouse/ClickHouse/issues/70973). [#70979](https://github.com/ClickHouse/ClickHouse/pull/70979) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix `limit by`, `limit with ties` for distributed and parallel replicas. [#70880](https://github.com/ClickHouse/ClickHouse/pull/70880) ([Nikita Taranov](https://github.com/nickitat)).
|
||||
|
||||
|
||||
### <a id="249"></a> ClickHouse release 24.9, 2024-09-26
|
||||
|
||||
#### Backward Incompatible Change
|
||||
|
||||
@ -48,6 +48,7 @@ Upcoming meetups
|
||||
* [Paris Meetup](https://www.meetup.com/clickhouse-france-user-group/events/303096434) - November 26
|
||||
* [Amsterdam Meetup](https://www.meetup.com/clickhouse-netherlands-user-group/events/303638814) - December 3
|
||||
* [New York Meetup](https://www.meetup.com/clickhouse-new-york-user-group/events/304268174) - December 9
|
||||
* [San Francisco Meetup](https://www.meetup.com/clickhouse-silicon-valley-meetup-group/events/304286951/) - December 12
|
||||
|
||||
Recently completed meetups
|
||||
|
||||
|
||||
@ -28,7 +28,7 @@ COPY requirements.txt /
|
||||
RUN pip3 install --no-cache-dir -r requirements.txt
|
||||
|
||||
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen en_US.UTF-8
|
||||
ENV LC_ALL en_US.UTF-8
|
||||
ENV LC_ALL=en_US.UTF-8
|
||||
|
||||
# Architecture of the image when BuildKit/buildx is used
|
||||
ARG TARGETARCH
|
||||
|
||||
@ -12,6 +12,7 @@ charset-normalizer==3.3.2
|
||||
click==8.1.7
|
||||
codespell==2.2.1
|
||||
cryptography==43.0.1
|
||||
datacompy==0.7.3
|
||||
Deprecated==1.2.14
|
||||
dill==0.3.8
|
||||
flake8==4.0.1
|
||||
@ -23,6 +24,7 @@ mccabe==0.6.1
|
||||
multidict==6.0.5
|
||||
mypy==1.8.0
|
||||
mypy-extensions==1.0.0
|
||||
pandas==2.2.3
|
||||
packaging==24.1
|
||||
pathspec==0.9.0
|
||||
pip==24.1.1
|
||||
|
||||
@ -290,6 +290,7 @@ The following settings can be specified in configuration file for given endpoint
|
||||
- `expiration_window_seconds` — Grace period for checking if expiration-based credentials have expired. Optional, default value is `120`.
|
||||
- `no_sign_request` - Ignore all the credentials so requests are not signed. Useful for accessing public buckets.
|
||||
- `header` — Adds specified HTTP header to a request to given endpoint. Optional, can be specified multiple times.
|
||||
- `access_header` - Adds specified HTTP header to a request to given endpoint, in cases where there are no other credentials from another source.
|
||||
- `server_side_encryption_customer_key_base64` — If specified, required headers for accessing S3 objects with SSE-C encryption will be set. Optional.
|
||||
- `server_side_encryption_kms_key_id` - If specified, required headers for accessing S3 objects with [SSE-KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html) will be set. If an empty string is specified, the AWS managed S3 key will be used. Optional.
|
||||
- `server_side_encryption_kms_encryption_context` - If specified alongside `server_side_encryption_kms_key_id`, the given encryption context header for SSE-KMS will be set. Optional.
|
||||
@ -320,6 +321,32 @@ The following settings can be specified in configuration file for given endpoint
|
||||
</s3>
|
||||
```
|
||||
|
||||
## Working with archives
|
||||
|
||||
Suppose that we have several archive files with following URIs on S3:
|
||||
|
||||
- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-10.csv.zip'
|
||||
- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-11.csv.zip'
|
||||
- 'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-12.csv.zip'
|
||||
|
||||
Extracting data from these archives is possible using ::. Globs can be used both in the url part as well as in the part after :: (responsible for the name of a file inside the archive).
|
||||
|
||||
``` sql
|
||||
SELECT *
|
||||
FROM s3(
|
||||
'https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-1{0..2}.csv.zip :: *.csv'
|
||||
);
|
||||
```
|
||||
|
||||
:::note
|
||||
ClickHouse supports three archive formats:
|
||||
ZIP
|
||||
TAR
|
||||
7Z
|
||||
While ZIP and TAR archives can be accessed from any supported storage location, 7Z archives can only be read from the local filesystem where ClickHouse is installed.
|
||||
:::
|
||||
|
||||
|
||||
## Accessing public buckets
|
||||
|
||||
ClickHouse tries to fetch credentials from many different types of sources.
|
||||
|
||||
@ -18,6 +18,11 @@ Columns:
|
||||
- `1` — Current user can’t change the setting.
|
||||
- `type` ([String](../../sql-reference/data-types/string.md)) — Setting type (implementation specific string value).
|
||||
- `is_obsolete` ([UInt8](../../sql-reference/data-types/int-uint.md#uint-ranges)) - Shows whether a setting is obsolete.
|
||||
- `tier` ([Enum8](../../sql-reference/data-types/enum.md)) — Support level for this feature. ClickHouse features are organized in tiers, varying depending on the current status of their development and the expectations one might have when using them. Values:
|
||||
- `'Production'` — The feature is stable, safe to use and does not have issues interacting with other **production** features. .
|
||||
- `'Beta'` — The feature is stable and safe. The outcome of using it together with other features is unknown and correctness is not guaranteed. Testing and reports are welcome.
|
||||
- `'Experimental'` — The feature is under development. Only intended for developers and ClickHouse enthusiasts. The feature might or might not work and could be removed at any time.
|
||||
- `'Obsolete'` — No longer supported. Either it is already removed or it will be removed in future releases.
|
||||
|
||||
**Example**
|
||||
```sql
|
||||
|
||||
@ -18,6 +18,11 @@ Columns:
|
||||
- `1` — Current user can’t change the setting.
|
||||
- `default` ([String](../../sql-reference/data-types/string.md)) — Setting default value.
|
||||
- `is_obsolete` ([UInt8](../../sql-reference/data-types/int-uint.md#uint-ranges)) - Shows whether a setting is obsolete.
|
||||
- `tier` ([Enum8](../../sql-reference/data-types/enum.md)) — Support level for this feature. ClickHouse features are organized in tiers, varying depending on the current status of their development and the expectations one might have when using them. Values:
|
||||
- `'Production'` — The feature is stable, safe to use and does not have issues interacting with other **production** features. .
|
||||
- `'Beta'` — The feature is stable and safe. The outcome of using it together with other features is unknown and correctness is not guaranteed. Testing and reports are welcome.
|
||||
- `'Experimental'` — The feature is under development. Only intended for developers and ClickHouse enthusiasts. The feature might or might not work and could be removed at any time.
|
||||
- `'Obsolete'` — No longer supported. Either it is already removed or it will be removed in future releases.
|
||||
|
||||
**Example**
|
||||
|
||||
@ -26,19 +31,99 @@ The following example shows how to get information about settings which name con
|
||||
``` sql
|
||||
SELECT *
|
||||
FROM system.settings
|
||||
WHERE name LIKE '%min_i%'
|
||||
WHERE name LIKE '%min_insert_block_size_%'
|
||||
FORMAT Vertical
|
||||
```
|
||||
|
||||
``` text
|
||||
┌─name───────────────────────────────────────────────_─value─────_─changed─_─description───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────_─min──_─max──_─readonly─_─type─────────_─default───_─alias_for─_─is_obsolete─┐
|
||||
│ min_insert_block_size_rows │ 1048449 │ 0 │ Squash blocks passed to INSERT query to specified size in rows, if blocks are not big enough. │ ____ │ ____ │ 0 │ UInt64 │ 1048449 │ │ 0 │
|
||||
│ min_insert_block_size_bytes │ 268402944 │ 0 │ Squash blocks passed to INSERT query to specified size in bytes, if blocks are not big enough. │ ____ │ ____ │ 0 │ UInt64 │ 268402944 │ │ 0 │
|
||||
│ min_insert_block_size_rows_for_materialized_views │ 0 │ 0 │ Like min_insert_block_size_rows, but applied only during pushing to MATERIALIZED VIEW (default: min_insert_block_size_rows) │ ____ │ ____ │ 0 │ UInt64 │ 0 │ │ 0 │
|
||||
│ min_insert_block_size_bytes_for_materialized_views │ 0 │ 0 │ Like min_insert_block_size_bytes, but applied only during pushing to MATERIALIZED VIEW (default: min_insert_block_size_bytes) │ ____ │ ____ │ 0 │ UInt64 │ 0 │ │ 0 │
|
||||
│ read_backoff_min_interval_between_events_ms │ 1000 │ 0 │ Settings to reduce the number of threads in case of slow reads. Do not pay attention to the event, if the previous one has passed less than a certain amount of time. │ ____ │ ____ │ 0 │ Milliseconds │ 1000 │ │ 0 │
|
||||
└────────────────────────────────────────────────────┴───────────┴─────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
──────────────────────────────────────────────────────┴──────┴──────┴──────────┴──────────────┴───────────┴───────────┴─────────────┘
|
||||
```
|
||||
Row 1:
|
||||
──────
|
||||
name: min_insert_block_size_rows
|
||||
value: 1048449
|
||||
changed: 0
|
||||
description: Sets the minimum number of rows in the block that can be inserted into a table by an `INSERT` query. Smaller-sized blocks are squashed into bigger ones.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Positive integer.
|
||||
- 0 — Squashing disabled.
|
||||
min: ᴺᵁᴸᴸ
|
||||
max: ᴺᵁᴸᴸ
|
||||
readonly: 0
|
||||
type: UInt64
|
||||
default: 1048449
|
||||
alias_for:
|
||||
is_obsolete: 0
|
||||
tier: Production
|
||||
|
||||
Row 2:
|
||||
──────
|
||||
name: min_insert_block_size_bytes
|
||||
value: 268402944
|
||||
changed: 0
|
||||
description: Sets the minimum number of bytes in the block which can be inserted into a table by an `INSERT` query. Smaller-sized blocks are squashed into bigger ones.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Positive integer.
|
||||
- 0 — Squashing disabled.
|
||||
min: ᴺᵁᴸᴸ
|
||||
max: ᴺᵁᴸᴸ
|
||||
readonly: 0
|
||||
type: UInt64
|
||||
default: 268402944
|
||||
alias_for:
|
||||
is_obsolete: 0
|
||||
tier: Production
|
||||
|
||||
Row 3:
|
||||
──────
|
||||
name: min_insert_block_size_rows_for_materialized_views
|
||||
value: 0
|
||||
changed: 0
|
||||
description: Sets the minimum number of rows in the block which can be inserted into a table by an `INSERT` query. Smaller-sized blocks are squashed into bigger ones. This setting is applied only for blocks inserted into [materialized view](../../sql-reference/statements/create/view.md). By adjusting this setting, you control blocks squashing while pushing to materialized view and avoid excessive memory usage.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Any positive integer.
|
||||
- 0 — Squashing disabled.
|
||||
|
||||
**See Also**
|
||||
|
||||
- [min_insert_block_size_rows](#min-insert-block-size-rows)
|
||||
min: ᴺᵁᴸᴸ
|
||||
max: ᴺᵁᴸᴸ
|
||||
readonly: 0
|
||||
type: UInt64
|
||||
default: 0
|
||||
alias_for:
|
||||
is_obsolete: 0
|
||||
tier: Production
|
||||
|
||||
Row 4:
|
||||
──────
|
||||
name: min_insert_block_size_bytes_for_materialized_views
|
||||
value: 0
|
||||
changed: 0
|
||||
description: Sets the minimum number of bytes in the block which can be inserted into a table by an `INSERT` query. Smaller-sized blocks are squashed into bigger ones. This setting is applied only for blocks inserted into [materialized view](../../sql-reference/statements/create/view.md). By adjusting this setting, you control blocks squashing while pushing to materialized view and avoid excessive memory usage.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Any positive integer.
|
||||
- 0 — Squashing disabled.
|
||||
|
||||
**See also**
|
||||
|
||||
- [min_insert_block_size_bytes](#min-insert-block-size-bytes)
|
||||
min: ᴺᵁᴸᴸ
|
||||
max: ᴺᵁᴸᴸ
|
||||
readonly: 0
|
||||
type: UInt64
|
||||
default: 0
|
||||
alias_for:
|
||||
is_obsolete: 0
|
||||
tier: Production
|
||||
```
|
||||
|
||||
Using of `WHERE changed` can be useful, for example, when you want to check:
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@ Syntax:
|
||||
``` sql
|
||||
ALTER USER [IF EXISTS] name1 [RENAME TO new_name |, name2 [,...]]
|
||||
[ON CLUSTER cluster_name]
|
||||
[NOT IDENTIFIED | RESET AUTHENTICATION METHODS TO NEW | {IDENTIFIED | ADD IDENTIFIED} {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']}
|
||||
[NOT IDENTIFIED | RESET AUTHENTICATION METHODS TO NEW | {IDENTIFIED | ADD IDENTIFIED} {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']} [VALID UNTIL datetime]
|
||||
[, {[{plaintext_password | sha256_password | sha256_hash | ...}] BY {'password' | 'hash'}} | {ldap SERVER 'server_name'} | {...} | ... [,...]]]
|
||||
[[ADD | DROP] HOST {LOCAL | NAME 'name' | REGEXP 'name_regexp' | IP 'address' | LIKE 'pattern'} [,...] | ANY | NONE]
|
||||
[VALID UNTIL datetime]
|
||||
@ -91,3 +91,15 @@ Reset authentication methods and keep the most recent added one:
|
||||
``` sql
|
||||
ALTER USER user1 RESET AUTHENTICATION METHODS TO NEW
|
||||
```
|
||||
|
||||
## VALID UNTIL Clause
|
||||
|
||||
Allows you to specify the expiration date and, optionally, the time for an authentication method. It accepts a string as a parameter. It is recommended to use the `YYYY-MM-DD [hh:mm:ss] [timezone]` format for datetime. By default, this parameter equals `'infinity'`.
|
||||
The `VALID UNTIL` clause can only be specified along with an authentication method, except for the case where no authentication method has been specified in the query. In this scenario, the `VALID UNTIL` clause will be applied to all existing authentication methods.
|
||||
|
||||
Examples:
|
||||
|
||||
- `ALTER USER name1 VALID UNTIL '2025-01-01'`
|
||||
- `ALTER USER name1 VALID UNTIL '2025-01-01 12:00:00 UTC'`
|
||||
- `ALTER USER name1 VALID UNTIL 'infinity'`
|
||||
- `ALTER USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID UNTIL'2025-01-01''`
|
||||
|
||||
@ -11,7 +11,7 @@ Syntax:
|
||||
|
||||
``` sql
|
||||
CREATE USER [IF NOT EXISTS | OR REPLACE] name1 [, name2 [,...]] [ON CLUSTER cluster_name]
|
||||
[NOT IDENTIFIED | IDENTIFIED {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']}
|
||||
[NOT IDENTIFIED | IDENTIFIED {[WITH {plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | WITH NO_PASSWORD | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']} | {WITH ssl_certificate CN 'common_name' | SAN 'TYPE:subject_alt_name'} | {WITH ssh_key BY KEY 'public_key' TYPE 'ssh-rsa|...'} | {WITH http SERVER 'server_name' [SCHEME 'Basic']} [VALID UNTIL datetime]
|
||||
[, {[{plaintext_password | sha256_password | sha256_hash | ...}] BY {'password' | 'hash'}} | {ldap SERVER 'server_name'} | {...} | ... [,...]]]
|
||||
[HOST {LOCAL | NAME 'name' | REGEXP 'name_regexp' | IP 'address' | LIKE 'pattern'} [,...] | ANY | NONE]
|
||||
[VALID UNTIL datetime]
|
||||
@ -178,7 +178,8 @@ ClickHouse treats `user_name@'address'` as a username as a whole. Thus, technica
|
||||
|
||||
## VALID UNTIL Clause
|
||||
|
||||
Allows you to specify the expiration date and, optionally, the time for user credentials. It accepts a string as a parameter. It is recommended to use the `YYYY-MM-DD [hh:mm:ss] [timezone]` format for datetime. By default, this parameter equals `'infinity'`.
|
||||
Allows you to specify the expiration date and, optionally, the time for an authentication method. It accepts a string as a parameter. It is recommended to use the `YYYY-MM-DD [hh:mm:ss] [timezone]` format for datetime. By default, this parameter equals `'infinity'`.
|
||||
The `VALID UNTIL` clause can only be specified along with an authentication method, except for the case where no authentication method has been specified in the query. In this scenario, the `VALID UNTIL` clause will be applied to all existing authentication methods.
|
||||
|
||||
Examples:
|
||||
|
||||
@ -186,6 +187,7 @@ Examples:
|
||||
- `CREATE USER name1 VALID UNTIL '2025-01-01 12:00:00 UTC'`
|
||||
- `CREATE USER name1 VALID UNTIL 'infinity'`
|
||||
- ```CREATE USER name1 VALID UNTIL '2025-01-01 12:00:00 `Asia/Tokyo`'```
|
||||
- `CREATE USER name1 IDENTIFIED WITH plaintext_password BY 'no_expiration', bcrypt_password BY 'expiration_set' VALID UNTIL '2025-01-01''`
|
||||
|
||||
## GRANTEES Clause
|
||||
|
||||
|
||||
@ -83,7 +83,7 @@ The presence of long-running or incomplete mutations often indicates that a Clic
|
||||
- Or manually kill some of these mutations by sending a `KILL` command.
|
||||
|
||||
``` sql
|
||||
KILL MUTATION [ON CLUSTER cluster]
|
||||
KILL MUTATION
|
||||
WHERE <where expression to SELECT FROM system.mutations query>
|
||||
[TEST]
|
||||
[FORMAT format]
|
||||
@ -135,7 +135,6 @@ KILL MUTATION WHERE database = 'default' AND table = 'table'
|
||||
-- Cancel the specific mutation:
|
||||
KILL MUTATION WHERE database = 'default' AND table = 'table' AND mutation_id = 'mutation_3.txt'
|
||||
```
|
||||
:::tip If you are killing a mutation in ClickHouse Cloud or in a self-managed cluster, then be sure to use the ```ON CLUSTER [cluster-name]``` option, in order to ensure the mutation is killed on all replicas:::
|
||||
|
||||
The query is useful when a mutation is stuck and cannot finish (e.g. if some function in the mutation query throws an exception when applied to the data contained in the table).
|
||||
|
||||
|
||||
@ -284,6 +284,14 @@ FROM s3(
|
||||
);
|
||||
```
|
||||
|
||||
:::note
|
||||
ClickHouse supports three archive formats:
|
||||
ZIP
|
||||
TAR
|
||||
7Z
|
||||
While ZIP and TAR archives can be accessed from any supported storage location, 7Z archives can only be read from the local filesystem where ClickHouse is installed.
|
||||
:::
|
||||
|
||||
|
||||
## Virtual Columns {#virtual-columns}
|
||||
|
||||
|
||||
@ -138,6 +138,7 @@ CREATE TABLE table_with_asterisk (name String, value UInt32)
|
||||
- `use_insecure_imds_request` — признак использования менее безопасного соединения при выполнении запроса к IMDS при получении учётных данных из метаданных Amazon EC2. Значение по умолчанию — `false`.
|
||||
- `region` — название региона S3.
|
||||
- `header` — добавляет указанный HTTP-заголовок к запросу на заданную точку приема запроса. Может быть определен несколько раз.
|
||||
- `access_header` - добавляет указанный HTTP-заголовок к запросу на заданную точку приема запроса, в случая если не указаны другие способы авторизации.
|
||||
- `server_side_encryption_customer_key_base64` — устанавливает необходимые заголовки для доступа к объектам S3 с шифрованием SSE-C.
|
||||
- `single_read_retries` — Максимальное количество попыток запроса при единичном чтении. Значение по умолчанию — `4`.
|
||||
|
||||
|
||||
@ -590,6 +590,7 @@ try
|
||||
|
||||
#if USE_SSL
|
||||
CertificateReloader::instance().tryLoad(*config);
|
||||
CertificateReloader::instance().tryLoadClient(*config);
|
||||
#endif
|
||||
});
|
||||
|
||||
|
||||
@ -2341,6 +2341,7 @@ try
|
||||
|
||||
#if USE_SSL
|
||||
CertificateReloader::instance().tryLoad(config());
|
||||
CertificateReloader::instance().tryLoadClient(config());
|
||||
#endif
|
||||
|
||||
/// Must be done after initialization of `servers`, because async_metrics will access `servers` variable from its thread.
|
||||
|
||||
@ -1,12 +1,16 @@
|
||||
#include <Access/AccessControl.h>
|
||||
#include <Access/AuthenticationData.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Interpreters/Access/getValidUntilFromAST.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Interpreters/evaluateConstantExpression.h>
|
||||
#include <Parsers/ASTExpressionList.h>
|
||||
#include <Parsers/ASTLiteral.h>
|
||||
#include <Parsers/Access/ASTPublicSSHKey.h>
|
||||
#include <Storages/checkAndGetLiteralArgument.h>
|
||||
#include <IO/parseDateTimeBestEffort.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
|
||||
#include <Common/OpenSSLHelpers.h>
|
||||
#include <Poco/SHA1Engine.h>
|
||||
@ -113,7 +117,8 @@ bool operator ==(const AuthenticationData & lhs, const AuthenticationData & rhs)
|
||||
&& (lhs.ssh_keys == rhs.ssh_keys)
|
||||
#endif
|
||||
&& (lhs.http_auth_scheme == rhs.http_auth_scheme)
|
||||
&& (lhs.http_auth_server_name == rhs.http_auth_server_name);
|
||||
&& (lhs.http_auth_server_name == rhs.http_auth_server_name)
|
||||
&& (lhs.valid_until == rhs.valid_until);
|
||||
}
|
||||
|
||||
|
||||
@ -384,14 +389,34 @@ std::shared_ptr<ASTAuthenticationData> AuthenticationData::toAST() const
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "AST: Unexpected authentication type {}", toString(auth_type));
|
||||
}
|
||||
|
||||
|
||||
if (valid_until)
|
||||
{
|
||||
WriteBufferFromOwnString out;
|
||||
writeDateTimeText(valid_until, out);
|
||||
|
||||
node->valid_until = std::make_shared<ASTLiteral>(out.str());
|
||||
}
|
||||
|
||||
return node;
|
||||
}
|
||||
|
||||
|
||||
AuthenticationData AuthenticationData::fromAST(const ASTAuthenticationData & query, ContextPtr context, bool validate)
|
||||
{
|
||||
time_t valid_until = 0;
|
||||
|
||||
if (query.valid_until)
|
||||
{
|
||||
valid_until = getValidUntilFromAST(query.valid_until, context);
|
||||
}
|
||||
|
||||
if (query.type && query.type == AuthenticationType::NO_PASSWORD)
|
||||
return AuthenticationData();
|
||||
{
|
||||
AuthenticationData auth_data;
|
||||
auth_data.setValidUntil(valid_until);
|
||||
return auth_data;
|
||||
}
|
||||
|
||||
/// For this type of authentication we have ASTPublicSSHKey as children for ASTAuthenticationData
|
||||
if (query.type && query.type == AuthenticationType::SSH_KEY)
|
||||
@ -418,6 +443,7 @@ AuthenticationData AuthenticationData::fromAST(const ASTAuthenticationData & que
|
||||
}
|
||||
|
||||
auth_data.setSSHKeys(std::move(keys));
|
||||
auth_data.setValidUntil(valid_until);
|
||||
return auth_data;
|
||||
#else
|
||||
throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "SSH is disabled, because ClickHouse is built without libssh");
|
||||
@ -451,6 +477,8 @@ AuthenticationData AuthenticationData::fromAST(const ASTAuthenticationData & que
|
||||
|
||||
AuthenticationData auth_data(current_type);
|
||||
|
||||
auth_data.setValidUntil(valid_until);
|
||||
|
||||
if (validate)
|
||||
context->getAccessControl().checkPasswordComplexityRules(value);
|
||||
|
||||
@ -494,6 +522,7 @@ AuthenticationData AuthenticationData::fromAST(const ASTAuthenticationData & que
|
||||
}
|
||||
|
||||
AuthenticationData auth_data(*query.type);
|
||||
auth_data.setValidUntil(valid_until);
|
||||
|
||||
if (query.contains_hash)
|
||||
{
|
||||
|
||||
@ -74,6 +74,9 @@ public:
|
||||
const String & getHTTPAuthenticationServerName() const { return http_auth_server_name; }
|
||||
void setHTTPAuthenticationServerName(const String & name) { http_auth_server_name = name; }
|
||||
|
||||
time_t getValidUntil() const { return valid_until; }
|
||||
void setValidUntil(time_t valid_until_) { valid_until = valid_until_; }
|
||||
|
||||
friend bool operator ==(const AuthenticationData & lhs, const AuthenticationData & rhs);
|
||||
friend bool operator !=(const AuthenticationData & lhs, const AuthenticationData & rhs) { return !(lhs == rhs); }
|
||||
|
||||
@ -106,6 +109,7 @@ private:
|
||||
/// HTTP authentication properties
|
||||
String http_auth_server_name;
|
||||
HTTPAuthenticationScheme http_auth_scheme = HTTPAuthenticationScheme::BASIC;
|
||||
time_t valid_until = 0;
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -554,7 +554,7 @@ std::optional<AuthResult> IAccessStorage::authenticateImpl(
|
||||
continue;
|
||||
}
|
||||
|
||||
if (areCredentialsValid(user->getName(), user->valid_until, auth_method, credentials, external_authenticators, auth_result.settings))
|
||||
if (areCredentialsValid(user->getName(), auth_method, credentials, external_authenticators, auth_result.settings))
|
||||
{
|
||||
auth_result.authentication_data = auth_method;
|
||||
return auth_result;
|
||||
@ -579,7 +579,6 @@ std::optional<AuthResult> IAccessStorage::authenticateImpl(
|
||||
|
||||
bool IAccessStorage::areCredentialsValid(
|
||||
const std::string & user_name,
|
||||
time_t valid_until,
|
||||
const AuthenticationData & authentication_method,
|
||||
const Credentials & credentials,
|
||||
const ExternalAuthenticators & external_authenticators,
|
||||
@ -591,6 +590,7 @@ bool IAccessStorage::areCredentialsValid(
|
||||
if (credentials.getUserName() != user_name)
|
||||
return false;
|
||||
|
||||
auto valid_until = authentication_method.getValidUntil();
|
||||
if (valid_until)
|
||||
{
|
||||
const time_t now = std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
|
||||
|
||||
@ -236,7 +236,6 @@ protected:
|
||||
bool allow_plaintext_password) const;
|
||||
virtual bool areCredentialsValid(
|
||||
const std::string & user_name,
|
||||
time_t valid_until,
|
||||
const AuthenticationData & authentication_method,
|
||||
const Credentials & credentials,
|
||||
const ExternalAuthenticators & external_authenticators,
|
||||
|
||||
@ -19,8 +19,7 @@ bool User::equal(const IAccessEntity & other) const
|
||||
return (authentication_methods == other_user.authentication_methods)
|
||||
&& (allowed_client_hosts == other_user.allowed_client_hosts)
|
||||
&& (access == other_user.access) && (granted_roles == other_user.granted_roles) && (default_roles == other_user.default_roles)
|
||||
&& (settings == other_user.settings) && (grantees == other_user.grantees) && (default_database == other_user.default_database)
|
||||
&& (valid_until == other_user.valid_until);
|
||||
&& (settings == other_user.settings) && (grantees == other_user.grantees) && (default_database == other_user.default_database);
|
||||
}
|
||||
|
||||
void User::setName(const String & name_)
|
||||
@ -88,7 +87,6 @@ void User::clearAllExceptDependencies()
|
||||
access = {};
|
||||
settings.removeSettingsKeepProfiles();
|
||||
default_database = {};
|
||||
valid_until = 0;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -23,7 +23,6 @@ struct User : public IAccessEntity
|
||||
SettingsProfileElements settings;
|
||||
RolesOrUsersSet grantees = RolesOrUsersSet::AllTag{};
|
||||
String default_database;
|
||||
time_t valid_until = 0;
|
||||
|
||||
bool equal(const IAccessEntity & other) const override;
|
||||
std::shared_ptr<IAccessEntity> clone() const override { return cloneImpl<User>(); }
|
||||
|
||||
@ -227,8 +227,13 @@ void QueryAnalyzer::resolveConstantExpression(QueryTreeNodePtr & node, const Que
|
||||
scope.context = context;
|
||||
|
||||
auto node_type = node->getNodeType();
|
||||
if (node_type == QueryTreeNodeType::QUERY || node_type == QueryTreeNodeType::UNION)
|
||||
{
|
||||
evaluateScalarSubqueryIfNeeded(node, scope);
|
||||
return;
|
||||
}
|
||||
|
||||
if (table_expression && node_type != QueryTreeNodeType::QUERY && node_type != QueryTreeNodeType::UNION)
|
||||
if (table_expression)
|
||||
{
|
||||
scope.expression_join_tree_node = table_expression;
|
||||
validateTableExpressionModifiers(scope.expression_join_tree_node, scope);
|
||||
|
||||
@ -418,7 +418,7 @@ void ClientApplicationBase::init(int argc, char ** argv)
|
||||
UInt64 max_client_memory_usage_int = parseWithSizeSuffix<UInt64>(max_client_memory_usage.c_str(), max_client_memory_usage.length());
|
||||
|
||||
total_memory_tracker.setHardLimit(max_client_memory_usage_int);
|
||||
total_memory_tracker.setDescription("(total)");
|
||||
total_memory_tracker.setDescription("Global");
|
||||
total_memory_tracker.setMetric(CurrentMetrics::MemoryTracking);
|
||||
}
|
||||
|
||||
|
||||
@ -1454,8 +1454,22 @@ void ClientBase::resetOutput()
|
||||
|
||||
/// Order is important: format, compression, file
|
||||
|
||||
if (output_format)
|
||||
output_format->finalize();
|
||||
try
|
||||
{
|
||||
if (output_format)
|
||||
output_format->finalize();
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
/// We need to make sure we continue resetting output_format (will stop threads on parallel output)
|
||||
/// as well as cleaning other output related setup
|
||||
if (!have_error)
|
||||
{
|
||||
client_exception
|
||||
= std::make_unique<Exception>(getCurrentExceptionMessageAndPattern(print_stack_trace), getCurrentExceptionCode());
|
||||
have_error = true;
|
||||
}
|
||||
}
|
||||
output_format.reset();
|
||||
|
||||
logs_out_stream.reset();
|
||||
|
||||
@ -68,15 +68,15 @@ inline std::string_view toDescription(OvercommitResult result)
|
||||
case OvercommitResult::NONE:
|
||||
return "";
|
||||
case OvercommitResult::DISABLED:
|
||||
return "Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero.";
|
||||
return "Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero";
|
||||
case OvercommitResult::MEMORY_FREED:
|
||||
throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "OvercommitResult::MEMORY_FREED shouldn't be asked for description");
|
||||
case OvercommitResult::SELECTED:
|
||||
return "Query was selected to stop by OvercommitTracker.";
|
||||
return "Query was selected to stop by OvercommitTracker";
|
||||
case OvercommitResult::TIMEOUTED:
|
||||
return "Waiting timeout for memory to be freed is reached.";
|
||||
return "Waiting timeout for memory to be freed is reached";
|
||||
case OvercommitResult::NOT_ENOUGH_FREED:
|
||||
return "Memory overcommit has freed not enough memory.";
|
||||
return "Memory overcommit has not freed enough memory";
|
||||
}
|
||||
}
|
||||
|
||||
@ -150,15 +150,23 @@ void MemoryTracker::logPeakMemoryUsage()
|
||||
auto peak_bytes = peak.load(std::memory_order::relaxed);
|
||||
if (peak_bytes < 128 * 1024)
|
||||
return;
|
||||
LOG_DEBUG(getLogger("MemoryTracker"),
|
||||
"Peak memory usage{}: {}.", (description ? " " + std::string(description) : ""), ReadableSize(peak_bytes));
|
||||
LOG_DEBUG(
|
||||
getLogger("MemoryTracker"),
|
||||
"{}{} memory usage: {}.",
|
||||
description ? std::string(description) : "",
|
||||
description ? " peak" : "Peak",
|
||||
ReadableSize(peak_bytes));
|
||||
}
|
||||
|
||||
void MemoryTracker::logMemoryUsage(Int64 current) const
|
||||
{
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
LOG_DEBUG(getLogger("MemoryTracker"),
|
||||
"Current memory usage{}: {}.", (description ? " " + std::string(description) : ""), ReadableSize(current));
|
||||
LOG_DEBUG(
|
||||
getLogger("MemoryTracker"),
|
||||
"{}{} memory usage: {}.",
|
||||
description ? std::string(description) : "",
|
||||
description ? " current" : "Current",
|
||||
ReadableSize(current));
|
||||
}
|
||||
|
||||
void MemoryTracker::injectFault() const
|
||||
@ -178,9 +186,9 @@ void MemoryTracker::injectFault() const
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
throw DB::Exception(
|
||||
DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"Memory tracker{}{}: fault injected (at specific point)",
|
||||
description ? " " : "",
|
||||
description ? description : "");
|
||||
"{}{}: fault injected (at specific point)",
|
||||
description ? description : "",
|
||||
description ? " memory tracker" : "Memory tracker");
|
||||
}
|
||||
|
||||
void MemoryTracker::debugLogBigAllocationWithoutCheck(Int64 size [[maybe_unused]])
|
||||
@ -282,9 +290,9 @@ AllocationTrace MemoryTracker::allocImpl(Int64 size, bool throw_if_memory_exceed
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
throw DB::Exception(
|
||||
DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"Memory tracker{}{}: fault injected. Would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
|
||||
description ? " " : "",
|
||||
"{}{}: fault injected. Would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
|
||||
description ? description : "",
|
||||
description ? " memory tracker" : "Memory tracker",
|
||||
formatReadableSizeWithBinarySuffix(will_be),
|
||||
size,
|
||||
formatReadableSizeWithBinarySuffix(current_hard_limit));
|
||||
@ -305,6 +313,8 @@ AllocationTrace MemoryTracker::allocImpl(Int64 size, bool throw_if_memory_exceed
|
||||
|
||||
if (overcommit_result != OvercommitResult::MEMORY_FREED)
|
||||
{
|
||||
bool overcommit_result_ignore
|
||||
= overcommit_result == OvercommitResult::NONE || overcommit_result == OvercommitResult::DISABLED;
|
||||
/// Revert
|
||||
amount.fetch_sub(size, std::memory_order_relaxed);
|
||||
rss.fetch_sub(size, std::memory_order_relaxed);
|
||||
@ -314,18 +324,18 @@ AllocationTrace MemoryTracker::allocImpl(Int64 size, bool throw_if_memory_exceed
|
||||
ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
throw DB::Exception(
|
||||
DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"Memory limit{}{} exceeded: "
|
||||
"would use {} (attempt to allocate chunk of {} bytes), current RSS {}, maximum: {}."
|
||||
"{}{}",
|
||||
description ? " " : "",
|
||||
description ? description : "",
|
||||
formatReadableSizeWithBinarySuffix(will_be),
|
||||
size,
|
||||
formatReadableSizeWithBinarySuffix(rss.load(std::memory_order_relaxed)),
|
||||
formatReadableSizeWithBinarySuffix(current_hard_limit),
|
||||
overcommit_result == OvercommitResult::NONE ? "" : " OvercommitTracker decision: ",
|
||||
toDescription(overcommit_result));
|
||||
DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"{}{} exceeded: "
|
||||
"would use {} (attempt to allocate chunk of {} bytes), current RSS {}, maximum: {}."
|
||||
"{}{}",
|
||||
description ? description : "",
|
||||
description ? " memory limit" : "Memory limit",
|
||||
formatReadableSizeWithBinarySuffix(will_be),
|
||||
size,
|
||||
formatReadableSizeWithBinarySuffix(rss.load(std::memory_order_relaxed)),
|
||||
formatReadableSizeWithBinarySuffix(current_hard_limit),
|
||||
overcommit_result_ignore ? "" : " OvercommitTracker decision: ",
|
||||
overcommit_result_ignore ? "" : toDescription(overcommit_result));
|
||||
}
|
||||
|
||||
// If OvercommitTracker::needToStopQuery returned false, it guarantees that enough memory is freed.
|
||||
|
||||
@ -78,7 +78,7 @@ ThreadStatus::ThreadStatus(bool check_current_thread_on_destruction_)
|
||||
|
||||
last_rusage = std::make_unique<RUsageCounters>();
|
||||
|
||||
memory_tracker.setDescription("(for thread)");
|
||||
memory_tracker.setDescription("Thread");
|
||||
log = getLogger("ThreadStatus");
|
||||
|
||||
current_thread = this;
|
||||
|
||||
@ -8,6 +8,7 @@ namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int INCORRECT_DATA;
|
||||
extern const int UNKNOWN_SETTING;
|
||||
}
|
||||
|
||||
@ -31,11 +32,19 @@ void BaseSettingsHelpers::writeFlags(Flags flags, WriteBuffer & out)
|
||||
}
|
||||
|
||||
|
||||
BaseSettingsHelpers::Flags BaseSettingsHelpers::readFlags(ReadBuffer & in)
|
||||
UInt64 BaseSettingsHelpers::readFlags(ReadBuffer & in)
|
||||
{
|
||||
UInt64 res;
|
||||
readVarUInt(res, in);
|
||||
return static_cast<Flags>(res);
|
||||
return res;
|
||||
}
|
||||
|
||||
SettingsTierType BaseSettingsHelpers::getTier(UInt64 flags)
|
||||
{
|
||||
int8_t tier = static_cast<int8_t>(flags & Flags::TIER);
|
||||
if (tier > SettingsTierType::BETA)
|
||||
throw Exception(ErrorCodes::INCORRECT_DATA, "Unknown tier value: '{}'", tier);
|
||||
return SettingsTierType{tier};
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
|
||||
#include <unordered_map>
|
||||
#include <Core/SettingsFields.h>
|
||||
#include <Core/SettingsTierType.h>
|
||||
#include <Core/SettingsWriteFormat.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <base/range.h>
|
||||
@ -21,6 +22,27 @@ namespace DB
|
||||
class ReadBuffer;
|
||||
class WriteBuffer;
|
||||
|
||||
struct BaseSettingsHelpers
|
||||
{
|
||||
[[noreturn]] static void throwSettingNotFound(std::string_view name);
|
||||
static void warningSettingNotFound(std::string_view name);
|
||||
|
||||
static void writeString(std::string_view str, WriteBuffer & out);
|
||||
static String readString(ReadBuffer & in);
|
||||
|
||||
enum Flags : UInt64
|
||||
{
|
||||
IMPORTANT = 0x01,
|
||||
CUSTOM = 0x02,
|
||||
TIER = 0x0c, /// 0b1100 == 2 bits
|
||||
/// If adding new flags, consider first if Tier might need more bits
|
||||
};
|
||||
|
||||
static SettingsTierType getTier(UInt64 flags);
|
||||
static void writeFlags(Flags flags, WriteBuffer & out);
|
||||
static UInt64 readFlags(ReadBuffer & in);
|
||||
};
|
||||
|
||||
/** Template class to define collections of settings.
|
||||
* If you create a new setting, please also add it to ./utils/check-style/check-settings-style
|
||||
* for validation
|
||||
@ -138,7 +160,7 @@ public:
|
||||
const char * getTypeName() const;
|
||||
const char * getDescription() const;
|
||||
bool isCustom() const;
|
||||
bool isObsolete() const;
|
||||
SettingsTierType getTier() const;
|
||||
|
||||
bool operator==(const SettingFieldRef & other) const { return (getName() == other.getName()) && (getValue() == other.getValue()); }
|
||||
bool operator!=(const SettingFieldRef & other) const { return !(*this == other); }
|
||||
@ -225,24 +247,6 @@ private:
|
||||
std::conditional_t<Traits::allow_custom_settings, CustomSettingMap, boost::blank> custom_settings_map;
|
||||
};
|
||||
|
||||
struct BaseSettingsHelpers
|
||||
{
|
||||
[[noreturn]] static void throwSettingNotFound(std::string_view name);
|
||||
static void warningSettingNotFound(std::string_view name);
|
||||
|
||||
static void writeString(std::string_view str, WriteBuffer & out);
|
||||
static String readString(ReadBuffer & in);
|
||||
|
||||
enum Flags : UInt64
|
||||
{
|
||||
IMPORTANT = 0x01,
|
||||
CUSTOM = 0x02,
|
||||
OBSOLETE = 0x04,
|
||||
};
|
||||
static void writeFlags(Flags flags, WriteBuffer & out);
|
||||
static Flags readFlags(ReadBuffer & in);
|
||||
};
|
||||
|
||||
template <typename TTraits>
|
||||
void BaseSettings<TTraits>::set(std::string_view name, const Field & value)
|
||||
{
|
||||
@ -477,7 +481,7 @@ void BaseSettings<TTraits>::read(ReadBuffer & in, SettingsWriteFormat format)
|
||||
size_t index = accessor.find(name);
|
||||
|
||||
using Flags = BaseSettingsHelpers::Flags;
|
||||
Flags flags{0};
|
||||
UInt64 flags{0};
|
||||
if (format >= SettingsWriteFormat::STRINGS_WITH_FLAGS)
|
||||
flags = BaseSettingsHelpers::readFlags(in);
|
||||
bool is_important = (flags & Flags::IMPORTANT);
|
||||
@ -797,14 +801,14 @@ bool BaseSettings<TTraits>::SettingFieldRef::isCustom() const
|
||||
}
|
||||
|
||||
template <typename TTraits>
|
||||
bool BaseSettings<TTraits>::SettingFieldRef::isObsolete() const
|
||||
SettingsTierType BaseSettings<TTraits>::SettingFieldRef::getTier() const
|
||||
{
|
||||
if constexpr (Traits::allow_custom_settings)
|
||||
{
|
||||
if (custom_setting)
|
||||
return false;
|
||||
return SettingsTierType::PRODUCTION;
|
||||
}
|
||||
return accessor->isObsolete(index);
|
||||
return accessor->getTier(index);
|
||||
}
|
||||
|
||||
using AliasMap = std::unordered_map<std::string_view, std::string_view>;
|
||||
@ -835,8 +839,8 @@ using AliasMap = std::unordered_map<std::string_view, std::string_view>;
|
||||
const String & getName(size_t index) const { return field_infos[index].name; } \
|
||||
const char * getTypeName(size_t index) const { return field_infos[index].type; } \
|
||||
const char * getDescription(size_t index) const { return field_infos[index].description; } \
|
||||
bool isImportant(size_t index) const { return field_infos[index].is_important; } \
|
||||
bool isObsolete(size_t index) const { return field_infos[index].is_obsolete; } \
|
||||
bool isImportant(size_t index) const { return field_infos[index].flags & BaseSettingsHelpers::Flags::IMPORTANT; } \
|
||||
SettingsTierType getTier(size_t index) const { return BaseSettingsHelpers::getTier(field_infos[index].flags); } \
|
||||
Field castValueUtil(size_t index, const Field & value) const { return field_infos[index].cast_value_util_function(value); } \
|
||||
String valueToStringUtil(size_t index, const Field & value) const { return field_infos[index].value_to_string_util_function(value); } \
|
||||
Field stringToValueUtil(size_t index, const String & str) const { return field_infos[index].string_to_value_util_function(str); } \
|
||||
@ -856,8 +860,7 @@ using AliasMap = std::unordered_map<std::string_view, std::string_view>;
|
||||
String name; \
|
||||
const char * type; \
|
||||
const char * description; \
|
||||
bool is_important; \
|
||||
bool is_obsolete; \
|
||||
UInt64 flags; \
|
||||
Field (*cast_value_util_function)(const Field &); \
|
||||
String (*value_to_string_util_function)(const Field &); \
|
||||
Field (*string_to_value_util_function)(const String &); \
|
||||
@ -968,8 +971,8 @@ struct DefineAliases
|
||||
/// NOLINTNEXTLINE
|
||||
#define IMPLEMENT_SETTINGS_TRAITS_(TYPE, NAME, DEFAULT, DESCRIPTION, FLAGS) \
|
||||
res.field_infos.emplace_back( \
|
||||
FieldInfo{#NAME, #TYPE, DESCRIPTION, (FLAGS) & IMPORTANT, \
|
||||
static_cast<bool>((FLAGS) & BaseSettingsHelpers::Flags::OBSOLETE), \
|
||||
FieldInfo{#NAME, #TYPE, DESCRIPTION, \
|
||||
static_cast<UInt64>(FLAGS), \
|
||||
[](const Field & value) -> Field { return static_cast<Field>(SettingField##TYPE{value}); }, \
|
||||
[](const Field & value) -> String { return SettingField##TYPE{value}.toString(); }, \
|
||||
[](const String & str) -> Field { SettingField##TYPE temp; temp.parseFromString(str); return static_cast<Field>(temp); }, \
|
||||
|
||||
@ -192,6 +192,13 @@ namespace DB
|
||||
DECLARE(UInt64, parts_killer_pool_size, 128, "Threads for cleanup of shared merge tree outdated threads. Only available in ClickHouse Cloud", 0) \
|
||||
DECLARE(UInt64, keeper_multiread_batch_size, 10'000, "Maximum size of batch for MultiRead request to [Zoo]Keeper that support batching. If set to 0, batching is disabled. Available only in ClickHouse Cloud.", 0) \
|
||||
DECLARE(Bool, use_legacy_mongodb_integration, true, "Use the legacy MongoDB integration implementation. Note: it's highly recommended to set this option to false, since legacy implementation will be removed in the future. Please submit any issues you encounter with the new implementation.", 0) \
|
||||
\
|
||||
DECLARE(UInt64, prefetch_threadpool_pool_size, 100, "Size of background pool for prefetches for remote object storages", 0) \
|
||||
DECLARE(UInt64, prefetch_threadpool_queue_size, 1000000, "Number of tasks which is possible to push into prefetches pool", 0) \
|
||||
DECLARE(UInt64, load_marks_threadpool_pool_size, 50, "Size of background pool for marks loading", 0) \
|
||||
DECLARE(UInt64, load_marks_threadpool_queue_size, 1000000, "Number of tasks which is possible to push into prefetches pool", 0) \
|
||||
DECLARE(UInt64, threadpool_writer_pool_size, 100, "Size of background pool for write requests to object storages", 0) \
|
||||
DECLARE(UInt64, threadpool_writer_queue_size, 1000000, "Number of tasks which is possible to push into background pool for write requests to object storages", 0)
|
||||
|
||||
/// If you add a setting which can be updated at runtime, please update 'changeable_settings' map in dumpToSystemServerSettingsColumns below
|
||||
|
||||
@ -339,7 +346,7 @@ void ServerSettings::dumpToSystemServerSettingsColumns(ServerSettingColumnsParam
|
||||
res_columns[4]->insert(setting.getDescription());
|
||||
res_columns[5]->insert(setting.getTypeName());
|
||||
res_columns[6]->insert(is_changeable ? changeable_settings_it->second.second : ChangeableWithoutRestart::No);
|
||||
res_columns[7]->insert(setting.isObsolete());
|
||||
res_columns[7]->insert(setting.getTier() == SettingsTierType::OBSOLETE);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,7 +1,5 @@
|
||||
#include <Columns/ColumnArray.h>
|
||||
#include <Columns/ColumnMap.h>
|
||||
#include <Core/BaseSettings.h>
|
||||
#include <Core/BaseSettingsFwdMacros.h>
|
||||
#include <Core/BaseSettingsFwdMacrosImpl.h>
|
||||
#include <Core/BaseSettingsProgramOptions.h>
|
||||
#include <Core/DistributedCacheProtocol.h>
|
||||
@ -40,10 +38,17 @@ namespace ErrorCodes
|
||||
* Note: as an alternative, we could implement settings to be completely dynamic in the form of the map: String -> Field,
|
||||
* but we are not going to do it, because settings are used everywhere as static struct fields.
|
||||
*
|
||||
* `flags` can be either 0 or IMPORTANT.
|
||||
* A setting is "IMPORTANT" if it affects the results of queries and can't be ignored by older versions.
|
||||
* `flags` can include a Tier (BETA | EXPERIMENTAL) and an optional bitwise AND with IMPORTANT.
|
||||
* The default (0) means a PRODUCTION ready setting
|
||||
*
|
||||
* When adding new or changing existing settings add them to the settings changes history in SettingsChangesHistory.h
|
||||
* A setting is "IMPORTANT" if it affects the results of queries and can't be ignored by older versions.
|
||||
* Tiers:
|
||||
* EXPERIMENTAL: The feature is in active development stage. Mostly for developers or for ClickHouse enthusiasts.
|
||||
* BETA: There are no known bugs problems in the functionality, but the outcome of using it together with other
|
||||
* features/components is unknown and correctness is not guaranteed.
|
||||
* PRODUCTION (Default): The feature is safe to use along with other features from the PRODUCTION tier.
|
||||
*
|
||||
* When adding new or changing existing settings add them to the settings changes history in SettingsChangesHistory.cpp
|
||||
* for tracking settings changes in different versions and for special `compatibility` settings to work correctly.
|
||||
*/
|
||||
|
||||
@ -437,6 +442,9 @@ Enables or disables creating a new file on each insert in azure engine tables
|
||||
)", 0) \
|
||||
DECLARE(Bool, s3_check_objects_after_upload, false, R"(
|
||||
Check each uploaded object to s3 with head request to be sure that upload was successful
|
||||
)", 0) \
|
||||
DECLARE(Bool, azure_check_objects_after_upload, false, R"(
|
||||
Check each uploaded object in azure blob storage to be sure that upload was successful
|
||||
)", 0) \
|
||||
DECLARE(Bool, s3_allow_parallel_part_upload, true, R"(
|
||||
Use multiple threads for s3 multipart upload. It may lead to slightly higher memory usage
|
||||
@ -4451,9 +4459,8 @@ Optimize GROUP BY when all keys in block are constant
|
||||
DECLARE(Bool, legacy_column_name_of_tuple_literal, false, R"(
|
||||
List all names of element of large tuple literals in their column names instead of hash. This settings exists only for compatibility reasons. It makes sense to set to 'true', while doing rolling update of cluster from version lower than 21.7 to higher.
|
||||
)", 0) \
|
||||
DECLARE(Bool, enable_named_columns_in_function_tuple, true, R"(
|
||||
DECLARE(Bool, enable_named_columns_in_function_tuple, false, R"(
|
||||
Generate named tuples in function tuple() when all names are unique and can be treated as unquoted identifiers.
|
||||
Beware that this setting might currently result in broken queries. It's not recommended to use in production
|
||||
)", 0) \
|
||||
\
|
||||
DECLARE(Bool, query_plan_enable_optimizations, true, R"(
|
||||
@ -5105,6 +5112,9 @@ Only in ClickHouse Cloud. A maximum number of unacknowledged in-flight packets i
|
||||
)", 0) \
|
||||
DECLARE(UInt64, distributed_cache_data_packet_ack_window, DistributedCache::ACK_DATA_PACKET_WINDOW, R"(
|
||||
Only in ClickHouse Cloud. A window for sending ACK for DataPacket sequence in a single distributed cache read request
|
||||
)", 0) \
|
||||
DECLARE(Bool, distributed_cache_discard_connection_if_unread_data, true, R"(
|
||||
Only in ClickHouse Cloud. Discard connection if some data is unread.
|
||||
)", 0) \
|
||||
\
|
||||
DECLARE(Bool, parallelize_output_from_storages, true, R"(
|
||||
@ -5504,90 +5514,102 @@ For testing purposes. Replaces all external table functions to Null to not initi
|
||||
DECLARE(Bool, restore_replace_external_dictionary_source_to_null, false, R"(
|
||||
Replace external dictionary sources to Null on restore. Useful for testing purposes
|
||||
)", 0) \
|
||||
DECLARE(Bool, create_if_not_exists, false, R"(
|
||||
Enable `IF NOT EXISTS` for `CREATE` statement by default. If either this setting or `IF NOT EXISTS` is specified and a table with the provided name already exists, no exception will be thrown.
|
||||
)", 0) \
|
||||
DECLARE(Bool, enforce_strict_identifier_format, false, R"(
|
||||
If enabled, only allow identifiers containing alphanumeric characters and underscores.
|
||||
)", 0) \
|
||||
DECLARE(Bool, mongodb_throw_on_unsupported_query, true, R"(
|
||||
If enabled, MongoDB tables will return an error when a MongoDB query cannot be built. Otherwise, ClickHouse reads the full table and processes it locally. This option does not apply to the legacy implementation or when 'allow_experimental_analyzer=0'.
|
||||
)", 0) \
|
||||
\
|
||||
/* ###################################### */ \
|
||||
/* ######## EXPERIMENTAL FEATURES ####### */ \
|
||||
/* ###################################### */ \
|
||||
DECLARE(Bool, allow_experimental_materialized_postgresql_table, false, R"(
|
||||
Allows to use the MaterializedPostgreSQL table engine. Disabled by default, because this feature is experimental
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_funnel_functions, false, R"(
|
||||
Enable experimental functions for funnel analysis.
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_nlp_functions, false, R"(
|
||||
Enable experimental functions for natural language processing.
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_hash_functions, false, R"(
|
||||
Enable experimental hash functions
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_object_type, false, R"(
|
||||
Allow Object and JSON data types
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_time_series_table, false, R"(
|
||||
Allows creation of tables with the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine.
|
||||
/* Parallel replicas */ \
|
||||
DECLARE(UInt64, allow_experimental_parallel_reading_from_replicas, 0, R"(
|
||||
Use up to `max_parallel_replicas` the number of replicas from each shard for SELECT query execution. Reading is parallelized and coordinated dynamically. 0 - disabled, 1 - enabled, silently disable them in case of failure, 2 - enabled, throw an exception in case of failure
|
||||
)", BETA) ALIAS(enable_parallel_replicas) \
|
||||
DECLARE(NonZeroUInt64, max_parallel_replicas, 1, R"(
|
||||
The maximum number of replicas for each shard when executing a query.
|
||||
|
||||
Possible values:
|
||||
|
||||
- 0 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is disabled.
|
||||
- 1 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is enabled.
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_vector_similarity_index, false, R"(
|
||||
Allow experimental vector similarity index
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_variant_type, false, R"(
|
||||
Allows creation of experimental [Variant](../../sql-reference/data-types/variant.md).
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_dynamic_type, false, R"(
|
||||
Allow Dynamic data type
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_json_type, false, R"(
|
||||
Allow JSON data type
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_codecs, false, R"(
|
||||
If it is set to true, allow to specify experimental compression codecs (but we don't have those yet and this option does nothing).
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_shared_set_join, true, R"(
|
||||
Only in ClickHouse Cloud. Allow to create ShareSet and SharedJoin
|
||||
)", 0) \
|
||||
DECLARE(UInt64, max_limit_for_ann_queries, 1'000'000, R"(
|
||||
SELECT queries with LIMIT bigger than this setting cannot use vector similarity indexes. Helps to prevent memory overflows in vector similarity indexes.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, hnsw_candidate_list_size_for_search, 256, R"(
|
||||
The size of the dynamic candidate list when searching the vector similarity index, also known as 'ef_search'.
|
||||
)", 0) \
|
||||
DECLARE(Bool, throw_on_unsupported_query_inside_transaction, true, R"(
|
||||
Throw exception if unsupported query is used inside transaction
|
||||
)", 0) \
|
||||
DECLARE(TransactionsWaitCSNMode, wait_changes_become_visible_after_commit_mode, TransactionsWaitCSNMode::WAIT_UNKNOWN, R"(
|
||||
Wait for committed changes to become actually visible in the latest snapshot
|
||||
)", 0) \
|
||||
DECLARE(Bool, implicit_transaction, false, R"(
|
||||
If enabled and not already inside a transaction, wraps the query inside a full transaction (begin + commit or rollback)
|
||||
)", 0) \
|
||||
DECLARE(UInt64, grace_hash_join_initial_buckets, 1, R"(
|
||||
Initial number of grace hash join buckets
|
||||
)", 0) \
|
||||
DECLARE(UInt64, grace_hash_join_max_buckets, 1024, R"(
|
||||
Limit on the number of grace hash join buckets
|
||||
)", 0) \
|
||||
DECLARE(UInt64, join_to_sort_minimum_perkey_rows, 40, R"(
|
||||
The lower limit of per-key average rows in the right table to determine whether to rerange the right table by key in left or inner join. This setting ensures that the optimization is not applied for sparse table keys
|
||||
)", 0) \
|
||||
DECLARE(UInt64, join_to_sort_maximum_table_rows, 10000, R"(
|
||||
The maximum number of rows in the right table to determine whether to rerange the right table by key in left or inner join.
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_join_right_table_sorting, false, R"(
|
||||
If it is set to true, and the conditions of `join_to_sort_minimum_perkey_rows` and `join_to_sort_maximum_table_rows` are met, rerange the right table by key to improve the performance in left or inner hash join.
|
||||
- Positive integer.
|
||||
|
||||
**Additional Info**
|
||||
|
||||
This options will produce different results depending on the settings used.
|
||||
|
||||
:::note
|
||||
This setting will produce incorrect results when joins or subqueries are involved, and all tables don't meet certain requirements. See [Distributed Subqueries and max_parallel_replicas](../../sql-reference/operators/in.md/#max_parallel_replica-subqueries) for more details.
|
||||
:::
|
||||
|
||||
### Parallel processing using `SAMPLE` key
|
||||
|
||||
A query may be processed faster if it is executed on several servers in parallel. But the query performance may degrade in the following cases:
|
||||
|
||||
- The position of the sampling key in the partitioning key does not allow efficient range scans.
|
||||
- Adding a sampling key to the table makes filtering by other columns less efficient.
|
||||
- The sampling key is an expression that is expensive to calculate.
|
||||
- The cluster latency distribution has a long tail, so that querying more servers increases the query overall latency.
|
||||
|
||||
### Parallel processing using [parallel_replicas_custom_key](#parallel_replicas_custom_key)
|
||||
|
||||
This setting is useful for any replicated table.
|
||||
)", 0) \
|
||||
DECLARE(ParallelReplicasMode, parallel_replicas_mode, ParallelReplicasMode::READ_TASKS, R"(
|
||||
Type of filter to use with custom key for parallel replicas. default - use modulo operation on the custom key, range - use range filter on custom key using all possible values for the value type of custom key.
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replicas_count, 0, R"(
|
||||
This is internal setting that should not be used directly and represents an implementation detail of the 'parallel replicas' mode. This setting will be automatically set up by the initiator server for distributed queries to the number of parallel replicas participating in query processing.
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replica_offset, 0, R"(
|
||||
This is internal setting that should not be used directly and represents an implementation detail of the 'parallel replicas' mode. This setting will be automatically set up by the initiator server for distributed queries to the index of the replica participating in query processing among parallel replicas.
|
||||
)", BETA) \
|
||||
DECLARE(String, parallel_replicas_custom_key, "", R"(
|
||||
An arbitrary integer expression that can be used to split work between replicas for a specific table.
|
||||
The value can be any integer expression.
|
||||
|
||||
Simple expressions using primary keys are preferred.
|
||||
|
||||
If the setting is used on a cluster that consists of a single shard with multiple replicas, those replicas will be converted into virtual shards.
|
||||
Otherwise, it will behave same as for `SAMPLE` key, it will use multiple replicas of each shard.
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replicas_custom_key_range_lower, 0, R"(
|
||||
Allows the filter type `range` to split the work evenly between replicas based on the custom range `[parallel_replicas_custom_key_range_lower, INT_MAX]`.
|
||||
|
||||
When used in conjunction with [parallel_replicas_custom_key_range_upper](#parallel_replicas_custom_key_range_upper), it lets the filter evenly split the work over replicas for the range `[parallel_replicas_custom_key_range_lower, parallel_replicas_custom_key_range_upper]`.
|
||||
|
||||
Note: This setting will not cause any additional data to be filtered during query processing, rather it changes the points at which the range filter breaks up the range `[0, INT_MAX]` for parallel processing.
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replicas_custom_key_range_upper, 0, R"(
|
||||
Allows the filter type `range` to split the work evenly between replicas based on the custom range `[0, parallel_replicas_custom_key_range_upper]`. A value of 0 disables the upper bound, setting it the max value of the custom key expression.
|
||||
|
||||
When used in conjunction with [parallel_replicas_custom_key_range_lower](#parallel_replicas_custom_key_range_lower), it lets the filter evenly split the work over replicas for the range `[parallel_replicas_custom_key_range_lower, parallel_replicas_custom_key_range_upper]`.
|
||||
|
||||
Note: This setting will not cause any additional data to be filtered during query processing, rather it changes the points at which the range filter breaks up the range `[0, INT_MAX]` for parallel processing
|
||||
)", BETA) \
|
||||
DECLARE(String, cluster_for_parallel_replicas, "", R"(
|
||||
Cluster for a shard in which current server is located
|
||||
)", BETA) \
|
||||
DECLARE(Bool, parallel_replicas_allow_in_with_subquery, true, R"(
|
||||
If true, subquery for IN will be executed on every follower replica.
|
||||
)", BETA) \
|
||||
DECLARE(Float, parallel_replicas_single_task_marks_count_multiplier, 2, R"(
|
||||
A multiplier which will be added during calculation for minimal number of marks to retrieve from coordinator. This will be applied only for remote replicas.
|
||||
)", BETA) \
|
||||
DECLARE(Bool, parallel_replicas_for_non_replicated_merge_tree, false, R"(
|
||||
If true, ClickHouse will use parallel replicas algorithm also for non-replicated MergeTree tables
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replicas_min_number_of_rows_per_replica, 0, R"(
|
||||
Limit the number of replicas used in a query to (estimated rows to read / min_number_of_rows_per_replica). The max is still limited by 'max_parallel_replicas'
|
||||
)", BETA) \
|
||||
DECLARE(Bool, parallel_replicas_prefer_local_join, true, R"(
|
||||
If true, and JOIN can be executed with parallel replicas algorithm, and all storages of right JOIN part are *MergeTree, local JOIN will be used instead of GLOBAL JOIN.
|
||||
)", BETA) \
|
||||
DECLARE(UInt64, parallel_replicas_mark_segment_size, 0, R"(
|
||||
Parts virtually divided into segments to be distributed between replicas for parallel reading. This setting controls the size of these segments. Not recommended to change until you're absolutely sure in what you're doing. Value should be in range [128; 16384]
|
||||
)", BETA) \
|
||||
DECLARE(Bool, parallel_replicas_local_plan, false, R"(
|
||||
Build local plan for local replica
|
||||
)", BETA) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_analyzer, true, R"(
|
||||
Allow new query analyzer.
|
||||
)", IMPORTANT | BETA) ALIAS(enable_analyzer) \
|
||||
DECLARE(Bool, analyzer_compatibility_join_using_top_level_identifier, false, R"(
|
||||
Force to resolve identifier in JOIN USING from projection (for example, in `SELECT a + 1 AS b FROM t1 JOIN t2 USING (b)` join will be performed by `t1.a + 1 = t2.b`, rather then `t1.b = t2.b`).
|
||||
)", BETA) \
|
||||
\
|
||||
DECLARE(Timezone, session_timezone, "", R"(
|
||||
Sets the implicit time zone of the current session or query.
|
||||
The implicit time zone is the time zone applied to values of type DateTime/DateTime64 which have no explicitly specified time zone.
|
||||
@ -5647,126 +5669,121 @@ This happens due to different parsing pipelines:
|
||||
**See also**
|
||||
|
||||
- [timezone](../server-configuration-parameters/settings.md#timezone)
|
||||
)", BETA) \
|
||||
DECLARE(Bool, create_if_not_exists, false, R"(
|
||||
Enable `IF NOT EXISTS` for `CREATE` statement by default. If either this setting or `IF NOT EXISTS` is specified and a table with the provided name already exists, no exception will be thrown.
|
||||
)", 0) \
|
||||
DECLARE(Bool, enforce_strict_identifier_format, false, R"(
|
||||
If enabled, only allow identifiers containing alphanumeric characters and underscores.
|
||||
)", 0) \
|
||||
DECLARE(Bool, mongodb_throw_on_unsupported_query, true, R"(
|
||||
If enabled, MongoDB tables will return an error when a MongoDB query cannot be built. Otherwise, ClickHouse reads the full table and processes it locally. This option does not apply to the legacy implementation or when 'allow_experimental_analyzer=0'.
|
||||
)", 0) \
|
||||
DECLARE(Bool, implicit_select, false, R"(
|
||||
Allow writing simple SELECT queries without the leading SELECT keyword, which makes it simple for calculator-style usage, e.g. `1 + 2` becomes a valid query.
|
||||
)", 0) \
|
||||
DECLARE(Bool, use_hive_partitioning, false, R"(
|
||||
When enabled, ClickHouse will detect Hive-style partitioning in path (`/name=value/`) in file-like table engines [File](../../engines/table-engines/special/file.md#hive-style-partitioning)/[S3](../../engines/table-engines/integrations/s3.md#hive-style-partitioning)/[URL](../../engines/table-engines/special/url.md#hive-style-partitioning)/[HDFS](../../engines/table-engines/integrations/hdfs.md#hive-style-partitioning)/[AzureBlobStorage](../../engines/table-engines/integrations/azureBlobStorage.md#hive-style-partitioning) and will allow to use partition columns as virtual columns in the query. These virtual columns will have the same names as in the partitioned path, but starting with `_`.
|
||||
)", 0)\
|
||||
\
|
||||
DECLARE(Bool, allow_statistics_optimize, false, R"(
|
||||
Allows using statistics to optimize queries
|
||||
)", 0) ALIAS(allow_statistic_optimize) \
|
||||
DECLARE(Bool, allow_experimental_statistics, false, R"(
|
||||
Allows defining columns with [statistics](../../engines/table-engines/mergetree-family/mergetree.md#table_engine-mergetree-creating-a-table) and [manipulate statistics](../../engines/table-engines/mergetree-family/mergetree.md#column-statistics).
|
||||
)", 0) ALIAS(allow_experimental_statistic) \
|
||||
\
|
||||
/* Parallel replicas */ \
|
||||
DECLARE(UInt64, allow_experimental_parallel_reading_from_replicas, 0, R"(
|
||||
Use up to `max_parallel_replicas` the number of replicas from each shard for SELECT query execution. Reading is parallelized and coordinated dynamically. 0 - disabled, 1 - enabled, silently disable them in case of failure, 2 - enabled, throw an exception in case of failure
|
||||
)", 0) ALIAS(enable_parallel_replicas) \
|
||||
DECLARE(NonZeroUInt64, max_parallel_replicas, 1, R"(
|
||||
The maximum number of replicas for each shard when executing a query.
|
||||
/* ####################################################### */ \
|
||||
/* ########### START OF EXPERIMENTAL FEATURES ############ */ \
|
||||
/* ## ADD PRODUCTION / BETA FEATURES BEFORE THIS BLOCK ## */ \
|
||||
/* ####################################################### */ \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_materialized_postgresql_table, false, R"(
|
||||
Allows to use the MaterializedPostgreSQL table engine. Disabled by default, because this feature is experimental
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_funnel_functions, false, R"(
|
||||
Enable experimental functions for funnel analysis.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_nlp_functions, false, R"(
|
||||
Enable experimental functions for natural language processing.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_hash_functions, false, R"(
|
||||
Enable experimental hash functions
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_object_type, false, R"(
|
||||
Allow Object and JSON data types
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_time_series_table, false, R"(
|
||||
Allows creation of tables with the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine.
|
||||
|
||||
Possible values:
|
||||
|
||||
- Positive integer.
|
||||
|
||||
**Additional Info**
|
||||
|
||||
This options will produce different results depending on the settings used.
|
||||
|
||||
:::note
|
||||
This setting will produce incorrect results when joins or subqueries are involved, and all tables don't meet certain requirements. See [Distributed Subqueries and max_parallel_replicas](../../sql-reference/operators/in.md/#max_parallel_replica-subqueries) for more details.
|
||||
:::
|
||||
|
||||
### Parallel processing using `SAMPLE` key
|
||||
|
||||
A query may be processed faster if it is executed on several servers in parallel. But the query performance may degrade in the following cases:
|
||||
|
||||
- The position of the sampling key in the partitioning key does not allow efficient range scans.
|
||||
- Adding a sampling key to the table makes filtering by other columns less efficient.
|
||||
- The sampling key is an expression that is expensive to calculate.
|
||||
- The cluster latency distribution has a long tail, so that querying more servers increases the query overall latency.
|
||||
|
||||
### Parallel processing using [parallel_replicas_custom_key](#parallel_replicas_custom_key)
|
||||
|
||||
This setting is useful for any replicated table.
|
||||
)", 0) \
|
||||
DECLARE(ParallelReplicasMode, parallel_replicas_mode, ParallelReplicasMode::READ_TASKS, R"(
|
||||
Type of filter to use with custom key for parallel replicas. default - use modulo operation on the custom key, range - use range filter on custom key using all possible values for the value type of custom key.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replicas_count, 0, R"(
|
||||
This is internal setting that should not be used directly and represents an implementation detail of the 'parallel replicas' mode. This setting will be automatically set up by the initiator server for distributed queries to the number of parallel replicas participating in query processing.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replica_offset, 0, R"(
|
||||
This is internal setting that should not be used directly and represents an implementation detail of the 'parallel replicas' mode. This setting will be automatically set up by the initiator server for distributed queries to the index of the replica participating in query processing among parallel replicas.
|
||||
)", 0) \
|
||||
DECLARE(String, parallel_replicas_custom_key, "", R"(
|
||||
An arbitrary integer expression that can be used to split work between replicas for a specific table.
|
||||
The value can be any integer expression.
|
||||
|
||||
Simple expressions using primary keys are preferred.
|
||||
|
||||
If the setting is used on a cluster that consists of a single shard with multiple replicas, those replicas will be converted into virtual shards.
|
||||
Otherwise, it will behave same as for `SAMPLE` key, it will use multiple replicas of each shard.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replicas_custom_key_range_lower, 0, R"(
|
||||
Allows the filter type `range` to split the work evenly between replicas based on the custom range `[parallel_replicas_custom_key_range_lower, INT_MAX]`.
|
||||
|
||||
When used in conjunction with [parallel_replicas_custom_key_range_upper](#parallel_replicas_custom_key_range_upper), it lets the filter evenly split the work over replicas for the range `[parallel_replicas_custom_key_range_lower, parallel_replicas_custom_key_range_upper]`.
|
||||
|
||||
Note: This setting will not cause any additional data to be filtered during query processing, rather it changes the points at which the range filter breaks up the range `[0, INT_MAX]` for parallel processing.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replicas_custom_key_range_upper, 0, R"(
|
||||
Allows the filter type `range` to split the work evenly between replicas based on the custom range `[0, parallel_replicas_custom_key_range_upper]`. A value of 0 disables the upper bound, setting it the max value of the custom key expression.
|
||||
|
||||
When used in conjunction with [parallel_replicas_custom_key_range_lower](#parallel_replicas_custom_key_range_lower), it lets the filter evenly split the work over replicas for the range `[parallel_replicas_custom_key_range_lower, parallel_replicas_custom_key_range_upper]`.
|
||||
|
||||
Note: This setting will not cause any additional data to be filtered during query processing, rather it changes the points at which the range filter breaks up the range `[0, INT_MAX]` for parallel processing
|
||||
)", 0) \
|
||||
DECLARE(String, cluster_for_parallel_replicas, "", R"(
|
||||
Cluster for a shard in which current server is located
|
||||
)", 0) \
|
||||
DECLARE(Bool, parallel_replicas_allow_in_with_subquery, true, R"(
|
||||
If true, subquery for IN will be executed on every follower replica.
|
||||
)", 0) \
|
||||
DECLARE(Float, parallel_replicas_single_task_marks_count_multiplier, 2, R"(
|
||||
A multiplier which will be added during calculation for minimal number of marks to retrieve from coordinator. This will be applied only for remote replicas.
|
||||
)", 0) \
|
||||
DECLARE(Bool, parallel_replicas_for_non_replicated_merge_tree, false, R"(
|
||||
If true, ClickHouse will use parallel replicas algorithm also for non-replicated MergeTree tables
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replicas_min_number_of_rows_per_replica, 0, R"(
|
||||
Limit the number of replicas used in a query to (estimated rows to read / min_number_of_rows_per_replica). The max is still limited by 'max_parallel_replicas'
|
||||
)", 0) \
|
||||
DECLARE(Bool, parallel_replicas_prefer_local_join, true, R"(
|
||||
If true, and JOIN can be executed with parallel replicas algorithm, and all storages of right JOIN part are *MergeTree, local JOIN will be used instead of GLOBAL JOIN.
|
||||
)", 0) \
|
||||
DECLARE(UInt64, parallel_replicas_mark_segment_size, 0, R"(
|
||||
Parts virtually divided into segments to be distributed between replicas for parallel reading. This setting controls the size of these segments. Not recommended to change until you're absolutely sure in what you're doing. Value should be in range [128; 16384]
|
||||
- 0 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is disabled.
|
||||
- 1 — the [TimeSeries](../../engines/table-engines/integrations/time-series.md) table engine is enabled.
|
||||
)", 0) \
|
||||
DECLARE(Bool, allow_experimental_vector_similarity_index, false, R"(
|
||||
Allow experimental vector similarity index
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_variant_type, false, R"(
|
||||
Allows creation of experimental [Variant](../../sql-reference/data-types/variant.md).
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_dynamic_type, false, R"(
|
||||
Allow Dynamic data type
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_json_type, false, R"(
|
||||
Allow JSON data type
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_codecs, false, R"(
|
||||
If it is set to true, allow to specify experimental compression codecs (but we don't have those yet and this option does nothing).
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_shared_set_join, true, R"(
|
||||
Only in ClickHouse Cloud. Allow to create ShareSet and SharedJoin
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, max_limit_for_ann_queries, 1'000'000, R"(
|
||||
SELECT queries with LIMIT bigger than this setting cannot use vector similarity indexes. Helps to prevent memory overflows in vector similarity indexes.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, hnsw_candidate_list_size_for_search, 256, R"(
|
||||
The size of the dynamic candidate list when searching the vector similarity index, also known as 'ef_search'.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, throw_on_unsupported_query_inside_transaction, true, R"(
|
||||
Throw exception if unsupported query is used inside transaction
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(TransactionsWaitCSNMode, wait_changes_become_visible_after_commit_mode, TransactionsWaitCSNMode::WAIT_UNKNOWN, R"(
|
||||
Wait for committed changes to become actually visible in the latest snapshot
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, implicit_transaction, false, R"(
|
||||
If enabled and not already inside a transaction, wraps the query inside a full transaction (begin + commit or rollback)
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, grace_hash_join_initial_buckets, 1, R"(
|
||||
Initial number of grace hash join buckets
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, grace_hash_join_max_buckets, 1024, R"(
|
||||
Limit on the number of grace hash join buckets
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, join_to_sort_minimum_perkey_rows, 40, R"(
|
||||
The lower limit of per-key average rows in the right table to determine whether to rerange the right table by key in left or inner join. This setting ensures that the optimization is not applied for sparse table keys
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, join_to_sort_maximum_table_rows, 10000, R"(
|
||||
The maximum number of rows in the right table to determine whether to rerange the right table by key in left or inner join.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_join_right_table_sorting, false, R"(
|
||||
If it is set to true, and the conditions of `join_to_sort_minimum_perkey_rows` and `join_to_sort_maximum_table_rows` are met, rerange the right table by key to improve the performance in left or inner hash join.
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, use_hive_partitioning, false, R"(
|
||||
When enabled, ClickHouse will detect Hive-style partitioning in path (`/name=value/`) in file-like table engines [File](../../engines/table-engines/special/file.md#hive-style-partitioning)/[S3](../../engines/table-engines/integrations/s3.md#hive-style-partitioning)/[URL](../../engines/table-engines/special/url.md#hive-style-partitioning)/[HDFS](../../engines/table-engines/integrations/hdfs.md#hive-style-partitioning)/[AzureBlobStorage](../../engines/table-engines/integrations/azureBlobStorage.md#hive-style-partitioning) and will allow to use partition columns as virtual columns in the query. These virtual columns will have the same names as in the partitioned path, but starting with `_`.
|
||||
)", EXPERIMENTAL)\
|
||||
\
|
||||
DECLARE(Bool, allow_statistics_optimize, false, R"(
|
||||
Allows using statistics to optimize queries
|
||||
)", EXPERIMENTAL) ALIAS(allow_statistic_optimize) \
|
||||
DECLARE(Bool, allow_experimental_statistics, false, R"(
|
||||
Allows defining columns with [statistics](../../engines/table-engines/mergetree-family/mergetree.md#table_engine-mergetree-creating-a-table) and [manipulate statistics](../../engines/table-engines/mergetree-family/mergetree.md#column-statistics).
|
||||
)", EXPERIMENTAL) ALIAS(allow_experimental_statistic) \
|
||||
\
|
||||
DECLARE(Bool, allow_archive_path_syntax, true, R"(
|
||||
File/S3 engines/table function will parse paths with '::' as '\\<archive\\> :: \\<file\\>' if archive has correct extension
|
||||
)", 0) \
|
||||
DECLARE(Bool, parallel_replicas_local_plan, false, R"(
|
||||
Build local plan for local replica
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_inverted_index, false, R"(
|
||||
If it is set to true, allow to use experimental inverted index.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_full_text_index, false, R"(
|
||||
If it is set to true, allow to use experimental full-text index.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_join_condition, false, R"(
|
||||
Support join with inequal conditions which involve columns from both left and right table. e.g. t1.y < t2.y.
|
||||
)", 0) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_analyzer, true, R"(
|
||||
Allow new query analyzer.
|
||||
)", IMPORTANT) ALIAS(enable_analyzer) \
|
||||
DECLARE(Bool, analyzer_compatibility_join_using_top_level_identifier, false, R"(
|
||||
Force to resolve identifier in JOIN USING from projection (for example, in `SELECT a + 1 AS b FROM t1 JOIN t2 USING (b)` join will be performed by `t1.a + 1 = t2.b`, rather then `t1.b = t2.b`).
|
||||
)", 0) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_live_view, false, R"(
|
||||
@ -5779,43 +5796,43 @@ Possible values:
|
||||
)", 0) \
|
||||
DECLARE(Seconds, live_view_heartbeat_interval, 15, R"(
|
||||
The heartbeat interval in seconds to indicate live query is alive.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, max_live_view_insert_blocks_before_refresh, 64, R"(
|
||||
Limit maximum number of inserted blocks after which mergeable blocks are dropped and query is re-executed.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_window_view, false, R"(
|
||||
Enable WINDOW VIEW. Not mature enough.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Seconds, window_view_clean_interval, 60, R"(
|
||||
The clean interval of window view in seconds to free outdated data.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Seconds, window_view_heartbeat_interval, 15, R"(
|
||||
The heartbeat interval in seconds to indicate watch query is alive.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Seconds, wait_for_window_view_fire_signal_timeout, 10, R"(
|
||||
Timeout for waiting for window view fire signal in event time processing
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
DECLARE(Bool, stop_refreshable_materialized_views_on_startup, false, R"(
|
||||
On server startup, prevent scheduling of refreshable materialized views, as if with SYSTEM STOP VIEWS. You can manually start them with SYSTEM START VIEWS or SYSTEM START VIEW \\<name\\> afterwards. Also applies to newly created views. Has no effect on non-refreshable materialized views.
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
DECLARE(Bool, allow_experimental_database_materialized_mysql, false, R"(
|
||||
Allow to create database with Engine=MaterializedMySQL(...).
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_database_materialized_postgresql, false, R"(
|
||||
Allow to create database with Engine=MaterializedPostgreSQL(...).
|
||||
)", 0) \
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
/** Experimental feature for moving data between shards. */ \
|
||||
DECLARE(Bool, allow_experimental_query_deduplication, false, R"(
|
||||
Experimental data deduplication for SELECT queries based on part UUIDs
|
||||
)", 0) \
|
||||
DECLARE(Bool, implicit_select, false, R"(
|
||||
Allow writing simple SELECT queries without the leading SELECT keyword, which makes it simple for calculator-style usage, e.g. `1 + 2` becomes a valid query.
|
||||
)", 0)
|
||||
|
||||
)", EXPERIMENTAL) \
|
||||
\
|
||||
/* ####################################################### */ \
|
||||
/* ############ END OF EXPERIMENTAL FEATURES ############# */ \
|
||||
/* ####################################################### */ \
|
||||
|
||||
// End of COMMON_SETTINGS
|
||||
// Please add settings related to formats in Core/FormatFactorySettings.h, move obsolete settings to OBSOLETE_SETTINGS and obsolete format settings to OBSOLETE_FORMAT_SETTINGS.
|
||||
@ -5894,13 +5911,14 @@ Allow writing simple SELECT queries without the leading SELECT keyword, which ma
|
||||
/** The section above is for obsolete settings. Do not add anything there. */
|
||||
#endif /// __CLION_IDE__
|
||||
|
||||
|
||||
#define LIST_OF_SETTINGS(M, ALIAS) \
|
||||
COMMON_SETTINGS(M, ALIAS) \
|
||||
OBSOLETE_SETTINGS(M, ALIAS) \
|
||||
FORMAT_FACTORY_SETTINGS(M, ALIAS) \
|
||||
OBSOLETE_FORMAT_SETTINGS(M, ALIAS) \
|
||||
|
||||
// clang-format on
|
||||
|
||||
DECLARE_SETTINGS_TRAITS_ALLOW_CUSTOM_SETTINGS(SettingsTraits, LIST_OF_SETTINGS)
|
||||
IMPLEMENT_SETTINGS_TRAITS(SettingsTraits, LIST_OF_SETTINGS)
|
||||
|
||||
@ -6008,7 +6026,7 @@ void SettingsImpl::checkNoSettingNamesAtTopLevel(const Poco::Util::AbstractConfi
|
||||
{
|
||||
const auto & name = setting.getName();
|
||||
bool should_skip_check = name == "max_table_size_to_drop" || name == "max_partition_size_to_drop";
|
||||
if (config.has(name) && !setting.isObsolete() && !should_skip_check)
|
||||
if (config.has(name) && (setting.getTier() != SettingsTierType::OBSOLETE) && !should_skip_check)
|
||||
{
|
||||
throw Exception(ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG, "A setting '{}' appeared at top level in config {}."
|
||||
" But it is user-level setting that should be located in users.xml inside <profiles> section for specific profile."
|
||||
@ -6184,7 +6202,7 @@ std::vector<std::string_view> Settings::getChangedAndObsoleteNames() const
|
||||
std::vector<std::string_view> setting_names;
|
||||
for (const auto & setting : impl->allChanged())
|
||||
{
|
||||
if (setting.isObsolete())
|
||||
if (setting.getTier() == SettingsTierType::OBSOLETE)
|
||||
setting_names.emplace_back(setting.getName());
|
||||
}
|
||||
return setting_names;
|
||||
@ -6233,7 +6251,8 @@ void Settings::dumpToSystemSettingsColumns(MutableColumnsAndConstraints & params
|
||||
res_columns[6]->insert(writability == SettingConstraintWritability::CONST);
|
||||
res_columns[7]->insert(setting.getTypeName());
|
||||
res_columns[8]->insert(setting.getDefaultValueString());
|
||||
res_columns[10]->insert(setting.isObsolete());
|
||||
res_columns[10]->insert(setting.getTier() == SettingsTierType::OBSOLETE);
|
||||
res_columns[11]->insert(setting.getTier());
|
||||
};
|
||||
|
||||
const auto & settings_to_aliases = SettingsImpl::Traits::settingsToAliases();
|
||||
|
||||
@ -65,6 +65,7 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
|
||||
{"24.11",
|
||||
{
|
||||
{"object_storage_remove_recursive_file_limit", 0, 1000, "Added new setting to limit number of files stored in memory while removing from object storage. Zero value means unlimited."},
|
||||
{"distributed_cache_discard_connection_if_unread_data", true, true, "New setting"},
|
||||
}
|
||||
},
|
||||
{"24.10",
|
||||
@ -88,7 +89,7 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
|
||||
{"input_format_binary_read_json_as_string", false, false, "Add new setting to read values of JSON type as JSON string in RowBinary input format"},
|
||||
{"min_free_disk_bytes_to_perform_insert", 0, 0, "New setting."},
|
||||
{"min_free_disk_ratio_to_perform_insert", 0.0, 0.0, "New setting."},
|
||||
{"enable_named_columns_in_function_tuple", false, true, "Re-enable the setting since all known bugs are fixed"},
|
||||
{"enable_named_columns_in_function_tuple", false, false, "Disabled pending usability improvements"},
|
||||
{"cloud_mode_database_engine", 1, 1, "A setting for ClickHouse Cloud"},
|
||||
{"allow_experimental_shared_set_join", 1, 1, "A setting for ClickHouse Cloud"},
|
||||
{"read_through_distributed_cache", 0, 0, "A setting for ClickHouse Cloud"},
|
||||
@ -112,7 +113,8 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
|
||||
{"hnsw_candidate_list_size_for_search", 64, 256, "New setting. Previously, the value was optionally specified in CREATE INDEX and 64 by default."},
|
||||
{"allow_reorder_prewhere_conditions", false, true, "New setting"},
|
||||
{"input_format_parquet_bloom_filter_push_down", false, true, "When reading Parquet files, skip whole row groups based on the WHERE/PREWHERE expressions and bloom filter in the Parquet metadata."},
|
||||
{"date_time_64_output_format_cut_trailing_zeros_align_to_groups_of_thousands", false, false, "Dynamically trim the trailing zeros of datetime64 values to adjust the output scale to (0, 3, 6), corresponding to 'seconds', 'milliseconds', and 'microseconds'."}
|
||||
{"date_time_64_output_format_cut_trailing_zeros_align_to_groups_of_thousands", false, false, "Dynamically trim the trailing zeros of datetime64 values to adjust the output scale to (0, 3, 6), corresponding to 'seconds', 'milliseconds', and 'microseconds'."},
|
||||
{"azure_check_objects_after_upload", false, false, "Check each uploaded object in azure blob storage to be sure that upload was successful"},
|
||||
}
|
||||
},
|
||||
{"24.9",
|
||||
|
||||
@ -2,8 +2,8 @@
|
||||
|
||||
// clang-format off
|
||||
#define MAKE_OBSOLETE(M, TYPE, NAME, DEFAULT) \
|
||||
M(TYPE, NAME, DEFAULT, "Obsolete setting, does nothing.", BaseSettingsHelpers::Flags::OBSOLETE)
|
||||
M(TYPE, NAME, DEFAULT, "Obsolete setting, does nothing.", SettingsTierType::OBSOLETE)
|
||||
|
||||
/// NOTE: ServerSettings::loadSettingsFromConfig() should be updated to include this settings
|
||||
#define MAKE_DEPRECATED_BY_SERVER_CONFIG(M, TYPE, NAME, DEFAULT) \
|
||||
M(TYPE, NAME, DEFAULT, "User-level setting is deprecated, and it must be defined in the server configuration instead.", BaseSettingsHelpers::Flags::OBSOLETE)
|
||||
M(TYPE, NAME, DEFAULT, "User-level setting is deprecated, and it must be defined in the server configuration instead.", SettingsTierType::OBSOLETE)
|
||||
|
||||
19
src/Core/SettingsTierType.cpp
Normal file
19
src/Core/SettingsTierType.cpp
Normal file
@ -0,0 +1,19 @@
|
||||
#include <Core/SettingsTierType.h>
|
||||
#include <DataTypes/DataTypeEnum.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
std::shared_ptr<DataTypeEnum8> getSettingsTierEnum()
|
||||
{
|
||||
return std::make_shared<DataTypeEnum8>(
|
||||
DataTypeEnum8::Values
|
||||
{
|
||||
{"Production", static_cast<Int8>(SettingsTierType::PRODUCTION)},
|
||||
{"Obsolete", static_cast<Int8>(SettingsTierType::OBSOLETE)},
|
||||
{"Experimental", static_cast<Int8>(SettingsTierType::EXPERIMENTAL)},
|
||||
{"Beta", static_cast<Int8>(SettingsTierType::BETA)}
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
26
src/Core/SettingsTierType.h
Normal file
26
src/Core/SettingsTierType.h
Normal file
@ -0,0 +1,26 @@
|
||||
#pragma once
|
||||
|
||||
#include <Core/Types.h>
|
||||
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
template <typename Type>
|
||||
class DataTypeEnum;
|
||||
using DataTypeEnum8 = DataTypeEnum<Int8>;
|
||||
|
||||
// Make it signed for compatibility with DataTypeEnum8
|
||||
enum SettingsTierType : int8_t
|
||||
{
|
||||
PRODUCTION = 0b0000,
|
||||
OBSOLETE = 0b0100,
|
||||
EXPERIMENTAL = 0b1000,
|
||||
BETA = 0b1100
|
||||
};
|
||||
|
||||
std::shared_ptr<DataTypeEnum8> getSettingsTierEnum();
|
||||
|
||||
}
|
||||
@ -161,7 +161,7 @@ String getNameForSubstreamPath(
|
||||
String stream_name,
|
||||
SubstreamIterator begin,
|
||||
SubstreamIterator end,
|
||||
bool escape_tuple_delimiter)
|
||||
bool escape_for_file_name)
|
||||
{
|
||||
using Substream = ISerialization::Substream;
|
||||
|
||||
@ -186,7 +186,7 @@ String getNameForSubstreamPath(
|
||||
/// Because nested data may be represented not by Array of Tuple,
|
||||
/// but by separate Array columns with names in a form of a.b,
|
||||
/// and name is encoded as a whole.
|
||||
if (it->type == Substream::TupleElement && escape_tuple_delimiter)
|
||||
if (it->type == Substream::TupleElement && escape_for_file_name)
|
||||
stream_name += escapeForFileName(substream_name);
|
||||
else
|
||||
stream_name += substream_name;
|
||||
@ -206,7 +206,7 @@ String getNameForSubstreamPath(
|
||||
else if (it->type == SubstreamType::ObjectSharedData)
|
||||
stream_name += ".object_shared_data";
|
||||
else if (it->type == SubstreamType::ObjectTypedPath || it->type == SubstreamType::ObjectDynamicPath)
|
||||
stream_name += "." + it->object_path_name;
|
||||
stream_name += "." + (escape_for_file_name ? escapeForFileName(it->object_path_name) : it->object_path_name);
|
||||
}
|
||||
|
||||
return stream_name;
|
||||
@ -434,6 +434,14 @@ bool ISerialization::isDynamicSubcolumn(const DB::ISerialization::SubstreamPath
|
||||
return false;
|
||||
}
|
||||
|
||||
bool ISerialization::isLowCardinalityDictionarySubcolumn(const DB::ISerialization::SubstreamPath & path)
|
||||
{
|
||||
if (path.empty())
|
||||
return false;
|
||||
|
||||
return path[path.size() - 1].type == SubstreamType::DictionaryKeys;
|
||||
}
|
||||
|
||||
ISerialization::SubstreamData ISerialization::createFromPath(const SubstreamPath & path, size_t prefix_len)
|
||||
{
|
||||
assert(prefix_len <= path.size());
|
||||
|
||||
@ -463,6 +463,8 @@ public:
|
||||
/// Returns true if stream with specified path corresponds to dynamic subcolumn.
|
||||
static bool isDynamicSubcolumn(const SubstreamPath & path, size_t prefix_len);
|
||||
|
||||
static bool isLowCardinalityDictionarySubcolumn(const SubstreamPath & path);
|
||||
|
||||
protected:
|
||||
template <typename State, typename StatePtr>
|
||||
State * checkAndGetState(const StatePtr & state) const;
|
||||
|
||||
@ -54,7 +54,7 @@ void SerializationLowCardinality::enumerateStreams(
|
||||
.withSerializationInfo(data.serialization_info);
|
||||
|
||||
settings.path.back().data = dict_data;
|
||||
dict_inner_serialization->enumerateStreams(settings, callback, dict_data);
|
||||
callback(settings.path);
|
||||
|
||||
settings.path.back() = Substream::DictionaryIndexes;
|
||||
settings.path.back().data = data;
|
||||
|
||||
@ -365,7 +365,7 @@ AsynchronousBoundedReadBuffer::~AsynchronousBoundedReadBuffer()
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
tryLogCurrentException(log);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -30,6 +30,7 @@ namespace DB
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int AZURE_BLOB_STORAGE_ERROR;
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
@ -72,6 +73,7 @@ WriteBufferFromAzureBlobStorage::WriteBufferFromAzureBlobStorage(
|
||||
std::move(schedule_),
|
||||
settings_->max_inflight_parts_for_one_file,
|
||||
limited_log))
|
||||
, check_objects_after_upload(settings_->check_objects_after_upload)
|
||||
{
|
||||
allocateBuffer();
|
||||
}
|
||||
@ -178,6 +180,24 @@ void WriteBufferFromAzureBlobStorage::finalizeImpl()
|
||||
execWithRetry([&](){ block_blob_client.CommitBlockList(block_ids); }, max_unexpected_write_error_retries);
|
||||
LOG_TRACE(log, "Committed {} blocks for blob `{}`", block_ids.size(), blob_path);
|
||||
}
|
||||
|
||||
if (check_objects_after_upload)
|
||||
{
|
||||
try
|
||||
{
|
||||
auto blob_client = blob_container_client->GetBlobClient(blob_path);
|
||||
blob_client.GetProperties();
|
||||
}
|
||||
catch (const Azure::Storage::StorageException & e)
|
||||
{
|
||||
if (e.StatusCode == Azure::Core::Http::HttpStatusCode::NotFound)
|
||||
throw Exception(
|
||||
ErrorCodes::AZURE_BLOB_STORAGE_ERROR,
|
||||
"Object {} not uploaded to azure blob storage, it's a bug in Azure Blob Storage or its API.",
|
||||
blob_path);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void WriteBufferFromAzureBlobStorage::nextImpl()
|
||||
|
||||
@ -90,6 +90,7 @@ private:
|
||||
size_t hidden_size = 0;
|
||||
|
||||
std::unique_ptr<TaskTracker> task_tracker;
|
||||
bool check_objects_after_upload = false;
|
||||
|
||||
std::deque<PartData> detached_part_data;
|
||||
};
|
||||
|
||||
@ -39,6 +39,7 @@ namespace Setting
|
||||
extern const SettingsUInt64 azure_sdk_max_retries;
|
||||
extern const SettingsUInt64 azure_sdk_retry_initial_backoff_ms;
|
||||
extern const SettingsUInt64 azure_sdk_retry_max_backoff_ms;
|
||||
extern const SettingsBool azure_check_objects_after_upload;
|
||||
}
|
||||
|
||||
namespace ErrorCodes
|
||||
@ -352,6 +353,7 @@ std::unique_ptr<RequestSettings> getRequestSettings(const Settings & query_setti
|
||||
settings->sdk_max_retries = query_settings[Setting::azure_sdk_max_retries];
|
||||
settings->sdk_retry_initial_backoff_ms = query_settings[Setting::azure_sdk_retry_initial_backoff_ms];
|
||||
settings->sdk_retry_max_backoff_ms = query_settings[Setting::azure_sdk_retry_max_backoff_ms];
|
||||
settings->check_objects_after_upload = query_settings[Setting::azure_check_objects_after_upload];
|
||||
|
||||
return settings;
|
||||
}
|
||||
@ -389,6 +391,8 @@ std::unique_ptr<RequestSettings> getRequestSettings(const Poco::Util::AbstractCo
|
||||
settings->sdk_retry_initial_backoff_ms = config.getUInt64(config_prefix + ".retry_initial_backoff_ms", settings_ref[Setting::azure_sdk_retry_initial_backoff_ms]);
|
||||
settings->sdk_retry_max_backoff_ms = config.getUInt64(config_prefix + ".retry_max_backoff_ms", settings_ref[Setting::azure_sdk_retry_max_backoff_ms]);
|
||||
|
||||
settings->check_objects_after_upload = config.getBool(config_prefix + ".check_objects_after_upload", settings_ref[Setting::azure_check_objects_after_upload]);
|
||||
|
||||
if (config.has(config_prefix + ".curl_ip_resolve"))
|
||||
{
|
||||
using CurlOptions = Azure::Core::Http::CurlTransportOptions;
|
||||
|
||||
@ -50,6 +50,7 @@ struct RequestSettings
|
||||
size_t sdk_retry_initial_backoff_ms = 10;
|
||||
size_t sdk_retry_max_backoff_ms = 1000;
|
||||
bool use_native_copy = false;
|
||||
bool check_objects_after_upload = false;
|
||||
|
||||
using CurlOptions = Azure::Core::Http::CurlTransportOptions;
|
||||
CurlOptions::CurlOptIPResolve curl_ip_resolve = CurlOptions::CURL_IPRESOLVE_WHATEVER;
|
||||
|
||||
@ -103,15 +103,15 @@ std::unique_ptr<WriteBufferFromFileBase> HDFSObjectStorage::writeObject( /// NOL
|
||||
ErrorCodes::UNSUPPORTED_METHOD,
|
||||
"HDFS API doesn't support custom attributes/metadata for stored objects");
|
||||
|
||||
std::string path = object.remote_path;
|
||||
if (path.starts_with("/"))
|
||||
path = path.substr(1);
|
||||
if (!path.starts_with(url))
|
||||
path = fs::path(url) / path;
|
||||
|
||||
auto path = extractObjectKeyFromURL(object);
|
||||
/// Single O_WRONLY in libhdfs adds O_TRUNC
|
||||
return std::make_unique<WriteBufferFromHDFS>(
|
||||
path, config, settings->replication, patchSettings(write_settings), buf_size,
|
||||
url_without_path,
|
||||
fs::path(data_directory) / path,
|
||||
config,
|
||||
settings->replication,
|
||||
patchSettings(write_settings),
|
||||
buf_size,
|
||||
mode == WriteMode::Rewrite ? O_WRONLY : O_WRONLY | O_APPEND);
|
||||
}
|
||||
|
||||
|
||||
@ -177,7 +177,7 @@ std::unique_ptr<S3::Client> getClient(
|
||||
auth_settings[S3AuthSetting::secret_access_key],
|
||||
auth_settings[S3AuthSetting::server_side_encryption_customer_key_base64],
|
||||
auth_settings.server_side_encryption_kms_config,
|
||||
auth_settings.headers,
|
||||
auth_settings.getHeaders(),
|
||||
credentials_configuration,
|
||||
auth_settings[S3AuthSetting::session_token]);
|
||||
}
|
||||
|
||||
@ -1171,7 +1171,7 @@ public:
|
||||
|
||||
if (left_tuple && right_tuple)
|
||||
{
|
||||
auto func = FunctionToOverloadResolverAdaptor(std::make_shared<FunctionComparison<Op, Name>>(check_decimal_overflow));
|
||||
auto func = std::make_shared<FunctionToOverloadResolverAdaptor>(std::make_shared<FunctionComparison<Op, Name>>(check_decimal_overflow));
|
||||
|
||||
bool has_nullable = false;
|
||||
bool has_null = false;
|
||||
@ -1181,7 +1181,7 @@ public:
|
||||
{
|
||||
ColumnsWithTypeAndName args = {{nullptr, left_tuple->getElements()[i], ""},
|
||||
{nullptr, right_tuple->getElements()[i], ""}};
|
||||
auto element_type = func.build(args)->getResultType();
|
||||
auto element_type = func->build(args)->getResultType();
|
||||
has_nullable = has_nullable || element_type->isNullable();
|
||||
has_null = has_null || element_type->onlyNull();
|
||||
}
|
||||
|
||||
@ -349,19 +349,14 @@ struct MapKeyLikeAdapter
|
||||
}
|
||||
};
|
||||
|
||||
struct FunctionIdentityMap : public FunctionIdentity
|
||||
{
|
||||
bool useDefaultImplementationForLowCardinalityColumns() const override { return false; }
|
||||
};
|
||||
|
||||
struct NameMapConcat { static constexpr auto name = "mapConcat"; };
|
||||
using FunctionMapConcat = FunctionMapToArrayAdapter<FunctionArrayConcat, MapToNestedAdapter<NameMapConcat>, NameMapConcat>;
|
||||
|
||||
struct NameMapKeys { static constexpr auto name = "mapKeys"; };
|
||||
using FunctionMapKeys = FunctionMapToArrayAdapter<FunctionIdentityMap, MapToSubcolumnAdapter<NameMapKeys, 0>, NameMapKeys>;
|
||||
using FunctionMapKeys = FunctionMapToArrayAdapter<FunctionIdentity, MapToSubcolumnAdapter<NameMapKeys, 0>, NameMapKeys>;
|
||||
|
||||
struct NameMapValues { static constexpr auto name = "mapValues"; };
|
||||
using FunctionMapValues = FunctionMapToArrayAdapter<FunctionIdentityMap, MapToSubcolumnAdapter<NameMapValues, 1>, NameMapValues>;
|
||||
using FunctionMapValues = FunctionMapToArrayAdapter<FunctionIdentity, MapToSubcolumnAdapter<NameMapValues, 1>, NameMapValues>;
|
||||
|
||||
struct NameMapContains { static constexpr auto name = "mapContains"; };
|
||||
using FunctionMapContains = FunctionMapToArrayAdapter<FunctionArrayIndex<HasAction, NameMapContains>, MapToSubcolumnAdapter<NameMapContains, 0>, NameMapContains>;
|
||||
|
||||
@ -211,7 +211,7 @@ namespace
|
||||
ColumnsWithTypeAndName args = arguments;
|
||||
args[0].column = args[0].column->cloneResized(input_rows_count)->convertToFullColumnIfConst();
|
||||
|
||||
auto impl = FunctionToOverloadResolverAdaptor(std::make_shared<FunctionTransform>()).build(args);
|
||||
auto impl = std::make_shared<FunctionToOverloadResolverAdaptor>(std::make_shared<FunctionTransform>())->build(args);
|
||||
|
||||
return impl->execute(args, result_type, input_rows_count);
|
||||
}
|
||||
|
||||
@ -43,7 +43,10 @@ std::shared_ptr<IArchiveReader> createArchiveReader(
|
||||
else if (hasSupported7zExtension(path_to_archive))
|
||||
{
|
||||
#if USE_LIBARCHIVE
|
||||
return std::make_shared<SevenZipArchiveReader>(path_to_archive);
|
||||
if (archive_read_function)
|
||||
throw Exception(ErrorCodes::CANNOT_UNPACK_ARCHIVE, "7z archive supports only local files reading");
|
||||
else
|
||||
return std::make_shared<SevenZipArchiveReader>(path_to_archive);
|
||||
#else
|
||||
throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "libarchive library is disabled");
|
||||
#endif
|
||||
|
||||
@ -1,3 +1,5 @@
|
||||
#include <IO/ReadBufferFromFile.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/S3/Credentials.h>
|
||||
#include <Common/Exception.h>
|
||||
|
||||
@ -693,6 +695,7 @@ S3CredentialsProviderChain::S3CredentialsProviderChain(
|
||||
static const char AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI[] = "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI";
|
||||
static const char AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI[] = "AWS_CONTAINER_CREDENTIALS_FULL_URI";
|
||||
static const char AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN[] = "AWS_CONTAINER_AUTHORIZATION_TOKEN";
|
||||
static const char AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN_PATH[] = "AWS_CONTAINER_AUTHORIZATION_TOKEN_PATH";
|
||||
static const char AWS_EC2_METADATA_DISABLED[] = "AWS_EC2_METADATA_DISABLED";
|
||||
|
||||
/// The only difference from DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain()
|
||||
@ -750,7 +753,22 @@ S3CredentialsProviderChain::S3CredentialsProviderChain(
|
||||
}
|
||||
else if (!absolute_uri.empty())
|
||||
{
|
||||
const auto token = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN);
|
||||
auto token = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN);
|
||||
const auto token_path = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN_PATH);
|
||||
|
||||
if (!token_path.empty())
|
||||
{
|
||||
LOG_INFO(logger, "The environment variable value {} is {}", AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN_PATH, token_path);
|
||||
|
||||
String token_from_file;
|
||||
|
||||
ReadBufferFromFile in(token_path);
|
||||
readStringUntilEOF(token_from_file, in);
|
||||
Poco::trimInPlace(token_from_file);
|
||||
|
||||
token = token_from_file;
|
||||
}
|
||||
|
||||
AddProvider(std::make_shared<Aws::Auth::TaskRoleCredentialsProvider>(absolute_uri.c_str(), token.c_str()));
|
||||
|
||||
/// DO NOT log the value of the authorization token for security purposes.
|
||||
|
||||
@ -105,7 +105,9 @@ S3AuthSettings::S3AuthSettings(
|
||||
}
|
||||
}
|
||||
|
||||
headers = getHTTPHeaders(config_prefix, config);
|
||||
headers = getHTTPHeaders(config_prefix, config, "header");
|
||||
access_headers = getHTTPHeaders(config_prefix, config, "access_header");
|
||||
|
||||
server_side_encryption_kms_config = getSSEKMSConfig(config_prefix, config);
|
||||
|
||||
Poco::Util::AbstractConfiguration::Keys keys;
|
||||
@ -119,6 +121,7 @@ S3AuthSettings::S3AuthSettings(
|
||||
|
||||
S3AuthSettings::S3AuthSettings(const S3AuthSettings & settings)
|
||||
: headers(settings.headers)
|
||||
, access_headers(settings.access_headers)
|
||||
, users(settings.users)
|
||||
, server_side_encryption_kms_config(settings.server_side_encryption_kms_config)
|
||||
, impl(std::make_unique<S3AuthSettingsImpl>(*settings.impl))
|
||||
@ -127,6 +130,7 @@ S3AuthSettings::S3AuthSettings(const S3AuthSettings & settings)
|
||||
|
||||
S3AuthSettings::S3AuthSettings(S3AuthSettings && settings) noexcept
|
||||
: headers(std::move(settings.headers))
|
||||
, access_headers(std::move(settings.access_headers))
|
||||
, users(std::move(settings.users))
|
||||
, server_side_encryption_kms_config(std::move(settings.server_side_encryption_kms_config))
|
||||
, impl(std::make_unique<S3AuthSettingsImpl>(std::move(*settings.impl)))
|
||||
@ -145,6 +149,7 @@ S3AUTH_SETTINGS_SUPPORTED_TYPES(S3AuthSettings, IMPLEMENT_SETTING_SUBSCRIPT_OPER
|
||||
S3AuthSettings & S3AuthSettings::operator=(S3AuthSettings && settings) noexcept
|
||||
{
|
||||
headers = std::move(settings.headers);
|
||||
access_headers = std::move(settings.access_headers);
|
||||
users = std::move(settings.users);
|
||||
server_side_encryption_kms_config = std::move(settings.server_side_encryption_kms_config);
|
||||
*impl = std::move(*settings.impl);
|
||||
@ -157,6 +162,9 @@ bool S3AuthSettings::operator==(const S3AuthSettings & right)
|
||||
if (headers != right.headers)
|
||||
return false;
|
||||
|
||||
if (access_headers != right.access_headers)
|
||||
return false;
|
||||
|
||||
if (users != right.users)
|
||||
return false;
|
||||
|
||||
@ -196,6 +204,9 @@ void S3AuthSettings::updateIfChanged(const S3AuthSettings & settings)
|
||||
if (!settings.headers.empty())
|
||||
headers = settings.headers;
|
||||
|
||||
if (!settings.access_headers.empty())
|
||||
access_headers = settings.access_headers;
|
||||
|
||||
if (!settings.users.empty())
|
||||
users.insert(settings.users.begin(), settings.users.end());
|
||||
|
||||
@ -205,6 +216,17 @@ void S3AuthSettings::updateIfChanged(const S3AuthSettings & settings)
|
||||
server_side_encryption_kms_config = settings.server_side_encryption_kms_config;
|
||||
}
|
||||
|
||||
HTTPHeaderEntries S3AuthSettings::getHeaders() const
|
||||
{
|
||||
bool auth_settings_is_default = !impl->isChanged("access_key_id");
|
||||
if (access_headers.empty() || !auth_settings_is_default)
|
||||
return headers;
|
||||
|
||||
HTTPHeaderEntries result(headers);
|
||||
result.insert(result.end(), access_headers.begin(), access_headers.end());
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@ -55,8 +55,11 @@ struct S3AuthSettings
|
||||
bool hasUpdates(const S3AuthSettings & other) const;
|
||||
void updateIfChanged(const S3AuthSettings & settings);
|
||||
bool canBeUsedByUser(const String & user) const { return users.empty() || users.contains(user); }
|
||||
HTTPHeaderEntries getHeaders() const;
|
||||
|
||||
HTTPHeaderEntries headers;
|
||||
HTTPHeaderEntries access_headers;
|
||||
|
||||
std::unordered_set<std::string> users;
|
||||
ServerSideEncryptionKMSConfig server_side_encryption_kms_config;
|
||||
|
||||
|
||||
@ -74,14 +74,14 @@ namespace ErrorCodes
|
||||
namespace S3
|
||||
{
|
||||
|
||||
HTTPHeaderEntries getHTTPHeaders(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config)
|
||||
HTTPHeaderEntries getHTTPHeaders(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config, const std::string header_key)
|
||||
{
|
||||
HTTPHeaderEntries headers;
|
||||
Poco::Util::AbstractConfiguration::Keys subconfig_keys;
|
||||
config.keys(config_elem, subconfig_keys);
|
||||
for (const std::string & subkey : subconfig_keys)
|
||||
{
|
||||
if (subkey.starts_with("header"))
|
||||
if (subkey.starts_with(header_key))
|
||||
{
|
||||
auto header_str = config.getString(config_elem + "." + subkey);
|
||||
auto delimiter = header_str.find(':');
|
||||
|
||||
@ -69,7 +69,7 @@ struct ProxyConfigurationResolver;
|
||||
namespace S3
|
||||
{
|
||||
|
||||
HTTPHeaderEntries getHTTPHeaders(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config);
|
||||
HTTPHeaderEntries getHTTPHeaders(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config, std::string header_key = "header");
|
||||
ServerSideEncryptionKMSConfig getSSEKMSConfig(const std::string & config_elem, const Poco::Util::AbstractConfiguration & config);
|
||||
|
||||
}
|
||||
|
||||
@ -8,6 +8,7 @@
|
||||
#include <Common/logger_useful.h>
|
||||
#include <Core/ServerSettings.h>
|
||||
#include <Interpreters/Access/InterpreterSetRoleQuery.h>
|
||||
#include <Interpreters/Access/getValidUntilFromAST.h>
|
||||
#include <Interpreters/Context.h>
|
||||
#include <Interpreters/executeDDLQueryOnCluster.h>
|
||||
#include <Interpreters/removeOnClusterClauseIfNeeded.h>
|
||||
@ -44,7 +45,7 @@ namespace
|
||||
const std::optional<RolesOrUsersSet> & override_default_roles,
|
||||
const std::optional<SettingsProfileElements> & override_settings,
|
||||
const std::optional<RolesOrUsersSet> & override_grantees,
|
||||
const std::optional<time_t> & valid_until,
|
||||
const std::optional<time_t> & global_valid_until,
|
||||
bool reset_authentication_methods,
|
||||
bool replace_authentication_methods,
|
||||
bool allow_implicit_no_password,
|
||||
@ -105,12 +106,20 @@ namespace
|
||||
user.authentication_methods.emplace_back(authentication_method);
|
||||
}
|
||||
|
||||
bool has_no_password_authentication_method = std::find_if(user.authentication_methods.begin(),
|
||||
user.authentication_methods.end(),
|
||||
[](const AuthenticationData & authentication_data)
|
||||
{
|
||||
return authentication_data.getType() == AuthenticationType::NO_PASSWORD;
|
||||
}) != user.authentication_methods.end();
|
||||
bool has_no_password_authentication_method = false;
|
||||
|
||||
for (auto & authentication_method : user.authentication_methods)
|
||||
{
|
||||
if (global_valid_until)
|
||||
{
|
||||
authentication_method.setValidUntil(*global_valid_until);
|
||||
}
|
||||
|
||||
if (authentication_method.getType() == AuthenticationType::NO_PASSWORD)
|
||||
{
|
||||
has_no_password_authentication_method = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (has_no_password_authentication_method && user.authentication_methods.size() > 1)
|
||||
{
|
||||
@ -133,9 +142,6 @@ namespace
|
||||
}
|
||||
}
|
||||
|
||||
if (valid_until)
|
||||
user.valid_until = *valid_until;
|
||||
|
||||
if (override_name && !override_name->host_pattern.empty())
|
||||
{
|
||||
user.allowed_client_hosts = AllowedClientHosts{};
|
||||
@ -175,34 +181,6 @@ namespace
|
||||
else if (query.grantees)
|
||||
user.grantees = *query.grantees;
|
||||
}
|
||||
|
||||
time_t getValidUntilFromAST(ASTPtr valid_until, ContextPtr context)
|
||||
{
|
||||
if (context)
|
||||
valid_until = evaluateConstantExpressionAsLiteral(valid_until, context);
|
||||
|
||||
const String valid_until_str = checkAndGetLiteralArgument<String>(valid_until, "valid_until");
|
||||
|
||||
if (valid_until_str == "infinity")
|
||||
return 0;
|
||||
|
||||
time_t time = 0;
|
||||
ReadBufferFromString in(valid_until_str);
|
||||
|
||||
if (context)
|
||||
{
|
||||
const auto & time_zone = DateLUT::instance("");
|
||||
const auto & utc_time_zone = DateLUT::instance("UTC");
|
||||
|
||||
parseDateTimeBestEffort(time, in, time_zone, utc_time_zone);
|
||||
}
|
||||
else
|
||||
{
|
||||
readDateTimeText(time, in);
|
||||
}
|
||||
|
||||
return time;
|
||||
}
|
||||
}
|
||||
|
||||
BlockIO InterpreterCreateUserQuery::execute()
|
||||
@ -226,9 +204,9 @@ BlockIO InterpreterCreateUserQuery::execute()
|
||||
}
|
||||
}
|
||||
|
||||
std::optional<time_t> valid_until;
|
||||
if (query.valid_until)
|
||||
valid_until = getValidUntilFromAST(query.valid_until, getContext());
|
||||
std::optional<time_t> global_valid_until;
|
||||
if (query.global_valid_until)
|
||||
global_valid_until = getValidUntilFromAST(query.global_valid_until, getContext());
|
||||
|
||||
std::optional<RolesOrUsersSet> default_roles_from_query;
|
||||
if (query.default_roles)
|
||||
@ -274,7 +252,7 @@ BlockIO InterpreterCreateUserQuery::execute()
|
||||
auto updated_user = typeid_cast<std::shared_ptr<User>>(entity->clone());
|
||||
updateUserFromQueryImpl(
|
||||
*updated_user, query, authentication_methods, {}, default_roles_from_query, settings_from_query, grantees_from_query,
|
||||
valid_until, query.reset_authentication_methods_to_new, query.replace_authentication_methods,
|
||||
global_valid_until, query.reset_authentication_methods_to_new, query.replace_authentication_methods,
|
||||
implicit_no_password_allowed, no_password_allowed,
|
||||
plaintext_password_allowed, getContext()->getServerSettings()[ServerSetting::max_authentication_methods_per_user]);
|
||||
return updated_user;
|
||||
@ -296,7 +274,7 @@ BlockIO InterpreterCreateUserQuery::execute()
|
||||
auto new_user = std::make_shared<User>();
|
||||
updateUserFromQueryImpl(
|
||||
*new_user, query, authentication_methods, name, default_roles_from_query, settings_from_query, RolesOrUsersSet::AllTag{},
|
||||
valid_until, query.reset_authentication_methods_to_new, query.replace_authentication_methods,
|
||||
global_valid_until, query.reset_authentication_methods_to_new, query.replace_authentication_methods,
|
||||
implicit_no_password_allowed, no_password_allowed,
|
||||
plaintext_password_allowed, getContext()->getServerSettings()[ServerSetting::max_authentication_methods_per_user]);
|
||||
new_users.emplace_back(std::move(new_user));
|
||||
@ -351,9 +329,9 @@ void InterpreterCreateUserQuery::updateUserFromQuery(
|
||||
}
|
||||
}
|
||||
|
||||
std::optional<time_t> valid_until;
|
||||
if (query.valid_until)
|
||||
valid_until = getValidUntilFromAST(query.valid_until, {});
|
||||
std::optional<time_t> global_valid_until;
|
||||
if (query.global_valid_until)
|
||||
global_valid_until = getValidUntilFromAST(query.global_valid_until, {});
|
||||
|
||||
updateUserFromQueryImpl(
|
||||
user,
|
||||
@ -363,7 +341,7 @@ void InterpreterCreateUserQuery::updateUserFromQuery(
|
||||
{},
|
||||
{},
|
||||
{},
|
||||
valid_until,
|
||||
global_valid_until,
|
||||
query.reset_authentication_methods_to_new,
|
||||
query.replace_authentication_methods,
|
||||
allow_no_password,
|
||||
|
||||
@ -69,13 +69,6 @@ namespace
|
||||
query->authentication_methods.push_back(authentication_method.toAST());
|
||||
}
|
||||
|
||||
if (user.valid_until)
|
||||
{
|
||||
WriteBufferFromOwnString out;
|
||||
writeDateTimeText(user.valid_until, out);
|
||||
query->valid_until = std::make_shared<ASTLiteral>(out.str());
|
||||
}
|
||||
|
||||
if (!user.settings.empty())
|
||||
{
|
||||
if (attach_mode)
|
||||
|
||||
37
src/Interpreters/Access/getValidUntilFromAST.cpp
Normal file
37
src/Interpreters/Access/getValidUntilFromAST.cpp
Normal file
@ -0,0 +1,37 @@
|
||||
#include <Interpreters/Access/getValidUntilFromAST.h>
|
||||
#include <Interpreters/evaluateConstantExpression.h>
|
||||
#include <IO/parseDateTimeBestEffort.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <Storages/checkAndGetLiteralArgument.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
time_t getValidUntilFromAST(ASTPtr valid_until, ContextPtr context)
|
||||
{
|
||||
if (context)
|
||||
valid_until = evaluateConstantExpressionAsLiteral(valid_until, context);
|
||||
|
||||
const String valid_until_str = checkAndGetLiteralArgument<String>(valid_until, "valid_until");
|
||||
|
||||
if (valid_until_str == "infinity")
|
||||
return 0;
|
||||
|
||||
time_t time = 0;
|
||||
ReadBufferFromString in(valid_until_str);
|
||||
|
||||
if (context)
|
||||
{
|
||||
const auto & time_zone = DateLUT::instance("");
|
||||
const auto & utc_time_zone = DateLUT::instance("UTC");
|
||||
|
||||
parseDateTimeBestEffort(time, in, time_zone, utc_time_zone);
|
||||
}
|
||||
else
|
||||
{
|
||||
readDateTimeText(time, in);
|
||||
}
|
||||
|
||||
return time;
|
||||
}
|
||||
}
|
||||
9
src/Interpreters/Access/getValidUntilFromAST.h
Normal file
9
src/Interpreters/Access/getValidUntilFromAST.h
Normal file
@ -0,0 +1,9 @@
|
||||
#pragma once
|
||||
|
||||
#include <Parsers/IAST.h>
|
||||
#include <Interpreters/Context_fwd.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
time_t getValidUntilFromAST(ASTPtr valid_until, ContextPtr context);
|
||||
}
|
||||
@ -1120,6 +1120,13 @@ Chunk AsynchronousInsertQueue::processPreprocessedEntries(
|
||||
"Expected entry with data kind Preprocessed. Got: {}", entry->chunk.getDataKind());
|
||||
|
||||
Block block_to_insert = *block;
|
||||
if (block_to_insert.rows() == 0)
|
||||
{
|
||||
add_to_async_insert_log(entry, /*parsing_exception=*/ "", block_to_insert.rows(), block_to_insert.bytes());
|
||||
entry->resetChunk();
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!isCompatibleHeader(block_to_insert, header))
|
||||
convertBlockToHeader(block_to_insert, header);
|
||||
|
||||
|
||||
@ -273,6 +273,13 @@ namespace ServerSetting
|
||||
extern const ServerSettingsUInt64 max_replicated_sends_network_bandwidth_for_server;
|
||||
extern const ServerSettingsUInt64 tables_loader_background_pool_size;
|
||||
extern const ServerSettingsUInt64 tables_loader_foreground_pool_size;
|
||||
extern const ServerSettingsUInt64 prefetch_threadpool_pool_size;
|
||||
extern const ServerSettingsUInt64 prefetch_threadpool_queue_size;
|
||||
extern const ServerSettingsUInt64 load_marks_threadpool_pool_size;
|
||||
extern const ServerSettingsUInt64 load_marks_threadpool_queue_size;
|
||||
extern const ServerSettingsUInt64 threadpool_writer_pool_size;
|
||||
extern const ServerSettingsUInt64 threadpool_writer_queue_size;
|
||||
|
||||
}
|
||||
|
||||
namespace ErrorCodes
|
||||
@ -3215,9 +3222,8 @@ void Context::clearMarkCache() const
|
||||
ThreadPool & Context::getLoadMarksThreadpool() const
|
||||
{
|
||||
callOnce(shared->load_marks_threadpool_initialized, [&] {
|
||||
const auto & config = getConfigRef();
|
||||
auto pool_size = config.getUInt(".load_marks_threadpool_pool_size", 50);
|
||||
auto queue_size = config.getUInt(".load_marks_threadpool_queue_size", 1000000);
|
||||
auto pool_size = shared->server_settings[ServerSetting::load_marks_threadpool_pool_size];
|
||||
auto queue_size = shared->server_settings[ServerSetting::load_marks_threadpool_queue_size];
|
||||
shared->load_marks_threadpool = std::make_unique<ThreadPool>(
|
||||
CurrentMetrics::MarksLoaderThreads, CurrentMetrics::MarksLoaderThreadsActive, CurrentMetrics::MarksLoaderThreadsScheduled, pool_size, pool_size, queue_size);
|
||||
});
|
||||
@ -3410,9 +3416,9 @@ AsynchronousMetrics * Context::getAsynchronousMetrics() const
|
||||
ThreadPool & Context::getPrefetchThreadpool() const
|
||||
{
|
||||
callOnce(shared->prefetch_threadpool_initialized, [&] {
|
||||
const auto & config = getConfigRef();
|
||||
auto pool_size = config.getUInt(".prefetch_threadpool_pool_size", 100);
|
||||
auto queue_size = config.getUInt(".prefetch_threadpool_queue_size", 1000000);
|
||||
auto pool_size = shared->server_settings[ServerSetting::prefetch_threadpool_pool_size];
|
||||
auto queue_size = shared->server_settings[ServerSetting::prefetch_threadpool_queue_size];
|
||||
|
||||
shared->prefetch_threadpool = std::make_unique<ThreadPool>(
|
||||
CurrentMetrics::IOPrefetchThreads, CurrentMetrics::IOPrefetchThreadsActive, CurrentMetrics::IOPrefetchThreadsScheduled, pool_size, pool_size, queue_size);
|
||||
});
|
||||
@ -3422,8 +3428,7 @@ ThreadPool & Context::getPrefetchThreadpool() const
|
||||
|
||||
size_t Context::getPrefetchThreadpoolSize() const
|
||||
{
|
||||
const auto & config = getConfigRef();
|
||||
return config.getUInt(".prefetch_threadpool_pool_size", 100);
|
||||
return shared->server_settings[ServerSetting::prefetch_threadpool_pool_size];
|
||||
}
|
||||
|
||||
ThreadPool & Context::getBuildVectorSimilarityIndexThreadPool() const
|
||||
@ -5696,9 +5701,8 @@ IOUringReader & Context::getIOUringReader() const
|
||||
ThreadPool & Context::getThreadPoolWriter() const
|
||||
{
|
||||
callOnce(shared->threadpool_writer_initialized, [&] {
|
||||
const auto & config = getConfigRef();
|
||||
auto pool_size = config.getUInt(".threadpool_writer_pool_size", 100);
|
||||
auto queue_size = config.getUInt(".threadpool_writer_queue_size", 1000000);
|
||||
auto pool_size = shared->server_settings[ServerSetting::threadpool_writer_pool_size];
|
||||
auto queue_size = shared->server_settings[ServerSetting::threadpool_writer_queue_size];
|
||||
|
||||
shared->threadpool_writer = std::make_unique<ThreadPool>(
|
||||
CurrentMetrics::IOWriterThreads, CurrentMetrics::IOWriterThreadsActive, CurrentMetrics::IOWriterThreadsScheduled, pool_size, pool_size, queue_size);
|
||||
|
||||
@ -1025,7 +1025,7 @@ void InterpreterSystemQuery::dropReplica(ASTSystemQuery & query)
|
||||
{
|
||||
ReplicatedTableStatus status;
|
||||
storage_replicated->getStatus(status);
|
||||
if (status.zookeeper_info.path == query.replica_zk_path)
|
||||
if (status.replica_path == remote_replica_path)
|
||||
throw Exception(ErrorCodes::TABLE_WAS_NOT_DROPPED,
|
||||
"There is a local table {}, which has the same table path in ZooKeeper. "
|
||||
"Please check the path in query. "
|
||||
|
||||
@ -70,6 +70,15 @@ void MetricLog::stepFunction(const std::chrono::system_clock::time_point current
|
||||
{
|
||||
const ProfileEvents::Count new_value = ProfileEvents::global_counters[i].load(std::memory_order_relaxed);
|
||||
auto & old_value = prev_profile_events[i];
|
||||
|
||||
/// Profile event counters are supposed to be monotonic. However, at least the `NetworkReceiveBytes` can be inaccurate.
|
||||
/// So, since in the future the counter should always have a bigger value than in the past, we skip this event.
|
||||
/// It can be reproduced with the following integration tests:
|
||||
/// - test_hedged_requests/test.py::test_receive_timeout2
|
||||
/// - test_secure_socket::test
|
||||
if (new_value < old_value)
|
||||
continue;
|
||||
|
||||
elem.profile_events[i] = new_value - old_value;
|
||||
old_value = new_value;
|
||||
}
|
||||
|
||||
@ -276,7 +276,7 @@ ProcessList::insert(const String & query_, const IAST * ast, ContextMutablePtr q
|
||||
thread_group->performance_counters.setTraceProfileEvents(settings[Setting::trace_profile_events]);
|
||||
}
|
||||
|
||||
thread_group->memory_tracker.setDescription("(for query)");
|
||||
thread_group->memory_tracker.setDescription("Query");
|
||||
if (settings[Setting::memory_tracker_fault_probability] > 0.0)
|
||||
thread_group->memory_tracker.setFaultProbability(settings[Setting::memory_tracker_fault_probability]);
|
||||
|
||||
@ -311,7 +311,7 @@ ProcessList::insert(const String & query_, const IAST * ast, ContextMutablePtr q
|
||||
/// Track memory usage for all simultaneously running queries from single user.
|
||||
user_process_list.user_memory_tracker.setOrRaiseHardLimit(settings[Setting::max_memory_usage_for_user]);
|
||||
user_process_list.user_memory_tracker.setSoftLimit(settings[Setting::memory_overcommit_ratio_denominator_for_user]);
|
||||
user_process_list.user_memory_tracker.setDescription("(for user)");
|
||||
user_process_list.user_memory_tracker.setDescription("User");
|
||||
|
||||
if (!total_network_throttler && settings[Setting::max_network_bandwidth_for_all_users])
|
||||
{
|
||||
|
||||
@ -15,6 +15,7 @@
|
||||
#include <Parsers/parseQuery.h>
|
||||
|
||||
#include <chrono>
|
||||
#include <fmt/chrono.h>
|
||||
#include <mutex>
|
||||
|
||||
|
||||
@ -86,11 +87,11 @@ void QueryMetricLog::shutdown()
|
||||
Base::shutdown();
|
||||
}
|
||||
|
||||
void QueryMetricLog::startQuery(const String & query_id, TimePoint query_start_time, UInt64 interval_milliseconds)
|
||||
void QueryMetricLog::startQuery(const String & query_id, TimePoint start_time, UInt64 interval_milliseconds)
|
||||
{
|
||||
QueryMetricLogStatus status;
|
||||
status.interval_milliseconds = interval_milliseconds;
|
||||
status.next_collect_time = query_start_time + std::chrono::milliseconds(interval_milliseconds);
|
||||
status.next_collect_time = start_time + std::chrono::milliseconds(interval_milliseconds);
|
||||
|
||||
auto context = getContext();
|
||||
const auto & process_list = context->getProcessList();
|
||||
@ -99,24 +100,21 @@ void QueryMetricLog::startQuery(const String & query_id, TimePoint query_start_t
|
||||
const auto query_info = process_list.getQueryInfo(query_id, false, true, false);
|
||||
if (!query_info)
|
||||
{
|
||||
LOG_TRACE(logger, "Query {} is not running anymore, so we couldn't get its QueryInfo", query_id);
|
||||
LOG_TRACE(logger, "Query {} is not running anymore, so we couldn't get its QueryStatusInfo", query_id);
|
||||
return;
|
||||
}
|
||||
|
||||
auto elem = createLogMetricElement(query_id, *query_info, current_time);
|
||||
if (elem)
|
||||
add(std::move(elem.value()));
|
||||
else
|
||||
LOG_TRACE(logger, "Query {} finished already while this collecting task was running", query_id);
|
||||
});
|
||||
|
||||
status.task->scheduleAfter(interval_milliseconds);
|
||||
|
||||
std::lock_guard lock(queries_mutex);
|
||||
status.task->scheduleAfter(interval_milliseconds);
|
||||
queries.emplace(query_id, std::move(status));
|
||||
}
|
||||
|
||||
void QueryMetricLog::finishQuery(const String & query_id, QueryStatusInfoPtr query_info)
|
||||
void QueryMetricLog::finishQuery(const String & query_id, TimePoint finish_time, QueryStatusInfoPtr query_info)
|
||||
{
|
||||
std::unique_lock lock(queries_mutex);
|
||||
auto it = queries.find(query_id);
|
||||
@ -128,7 +126,7 @@ void QueryMetricLog::finishQuery(const String & query_id, QueryStatusInfoPtr que
|
||||
|
||||
if (query_info)
|
||||
{
|
||||
auto elem = createLogMetricElement(query_id, *query_info, std::chrono::system_clock::now(), false);
|
||||
auto elem = createLogMetricElement(query_id, *query_info, finish_time, false);
|
||||
if (elem)
|
||||
add(std::move(elem.value()));
|
||||
}
|
||||
@ -137,57 +135,84 @@ void QueryMetricLog::finishQuery(const String & query_id, QueryStatusInfoPtr que
|
||||
/// deactivating the task, which happens automatically on its destructor. Thus, we cannot
|
||||
/// deactivate/destroy the task while it's running. Now, the task locks `queries_mutex` to
|
||||
/// prevent concurrent edition of the queries. In short, the mutex order is: exec_mutex ->
|
||||
/// queries_mutex. Thus, to prevent a deadblock we need to make sure that we always lock them in
|
||||
/// queries_mutex. So, to prevent a deadblock we need to make sure that we always lock them in
|
||||
/// that order.
|
||||
{
|
||||
/// Take ownership of the task so that we can destroy it in this scope after unlocking `queries_lock`.
|
||||
/// Take ownership of the task so that we can destroy it in this scope after unlocking `queries_mutex`.
|
||||
auto task = std::move(it->second.task);
|
||||
|
||||
/// Build an empty task for the old task to make sure it does not lock any mutex on its destruction.
|
||||
it->second.task = {};
|
||||
|
||||
queries.erase(query_id);
|
||||
|
||||
/// Ensure `queries_mutex` is unlocked before calling task's destructor at the end of this
|
||||
/// scope which will lock `exec_mutex`.
|
||||
lock.unlock();
|
||||
}
|
||||
|
||||
lock.lock();
|
||||
queries.erase(query_id);
|
||||
}
|
||||
|
||||
std::optional<QueryMetricLogElement> QueryMetricLog::createLogMetricElement(const String & query_id, const QueryStatusInfo & query_info, TimePoint current_time, bool schedule_next)
|
||||
std::optional<QueryMetricLogElement> QueryMetricLog::createLogMetricElement(const String & query_id, const QueryStatusInfo & query_info, TimePoint query_info_time, bool schedule_next)
|
||||
{
|
||||
LOG_DEBUG(logger, "Collecting query_metric_log for query {}. Schedule next: {}", query_id, schedule_next);
|
||||
std::lock_guard lock(queries_mutex);
|
||||
/// fmtlib supports subsecond formatting in 10.0.0. We're in 9.1.0, so we need to add the milliseconds ourselves.
|
||||
auto seconds = std::chrono::time_point_cast<std::chrono::seconds>(query_info_time);
|
||||
auto microseconds = std::chrono::duration_cast<std::chrono::microseconds>(query_info_time - seconds).count();
|
||||
LOG_DEBUG(logger, "Collecting query_metric_log for query {} with QueryStatusInfo from {:%Y.%m.%d %H:%M:%S}.{:06}. Schedule next: {}", query_id, seconds, microseconds, schedule_next);
|
||||
|
||||
std::unique_lock lock(queries_mutex);
|
||||
auto query_status_it = queries.find(query_id);
|
||||
|
||||
/// The query might have finished while the scheduled task is running.
|
||||
if (query_status_it == queries.end())
|
||||
{
|
||||
lock.unlock();
|
||||
LOG_TRACE(logger, "Query {} finished already while this collecting task was running", query_id);
|
||||
return {};
|
||||
}
|
||||
|
||||
auto & query_status = query_status_it->second;
|
||||
if (query_info_time <= query_status.last_collect_time)
|
||||
{
|
||||
lock.unlock();
|
||||
LOG_TRACE(logger, "Query {} has a more recent metrics collected. Skipping this one", query_id);
|
||||
return {};
|
||||
}
|
||||
|
||||
query_status.last_collect_time = query_info_time;
|
||||
|
||||
QueryMetricLogElement elem;
|
||||
elem.event_time = timeInSeconds(current_time);
|
||||
elem.event_time_microseconds = timeInMicroseconds(current_time);
|
||||
elem.event_time = timeInSeconds(query_info_time);
|
||||
elem.event_time_microseconds = timeInMicroseconds(query_info_time);
|
||||
elem.query_id = query_status_it->first;
|
||||
elem.memory_usage = query_info.memory_usage > 0 ? query_info.memory_usage : 0;
|
||||
elem.peak_memory_usage = query_info.peak_memory_usage > 0 ? query_info.peak_memory_usage : 0;
|
||||
|
||||
auto & query_status = query_status_it->second;
|
||||
if (query_info.profile_counters)
|
||||
{
|
||||
for (ProfileEvents::Event i = ProfileEvents::Event(0), end = ProfileEvents::end(); i < end; ++i)
|
||||
{
|
||||
const auto & new_value = (*(query_info.profile_counters))[i];
|
||||
elem.profile_events[i] = new_value - query_status.last_profile_events[i];
|
||||
query_status.last_profile_events[i] = new_value;
|
||||
auto & old_value = query_status.last_profile_events[i];
|
||||
|
||||
/// Profile event counters are supposed to be monotonic. However, at least the `NetworkReceiveBytes` can be inaccurate.
|
||||
/// So, since in the future the counter should always have a bigger value than in the past, we skip this event.
|
||||
/// It can be reproduced with the following integration tests:
|
||||
/// - test_hedged_requests/test.py::test_receive_timeout2
|
||||
/// - test_secure_socket::test
|
||||
if (new_value < old_value)
|
||||
continue;
|
||||
|
||||
elem.profile_events[i] = new_value - old_value;
|
||||
old_value = new_value;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
elem.profile_events = query_status.last_profile_events;
|
||||
LOG_TRACE(logger, "Query {} has no profile counters", query_id);
|
||||
elem.profile_events = std::vector<ProfileEvents::Count>(ProfileEvents::end());
|
||||
}
|
||||
|
||||
if (query_status.task && schedule_next)
|
||||
if (schedule_next)
|
||||
{
|
||||
query_status.next_collect_time += std::chrono::milliseconds(query_status.interval_milliseconds);
|
||||
const auto wait_time = std::chrono::duration_cast<std::chrono::milliseconds>(query_status.next_collect_time - std::chrono::system_clock::now()).count();
|
||||
|
||||
@ -37,6 +37,7 @@ struct QueryMetricLogElement
|
||||
struct QueryMetricLogStatus
|
||||
{
|
||||
UInt64 interval_milliseconds;
|
||||
std::chrono::system_clock::time_point last_collect_time;
|
||||
std::chrono::system_clock::time_point next_collect_time;
|
||||
std::vector<ProfileEvents::Count> last_profile_events = std::vector<ProfileEvents::Count>(ProfileEvents::end());
|
||||
BackgroundSchedulePool::TaskHolder task;
|
||||
@ -52,11 +53,11 @@ public:
|
||||
void shutdown() final;
|
||||
|
||||
// Both startQuery and finishQuery are called from the thread that executes the query
|
||||
void startQuery(const String & query_id, TimePoint query_start_time, UInt64 interval_milliseconds);
|
||||
void finishQuery(const String & query_id, QueryStatusInfoPtr query_info = nullptr);
|
||||
void startQuery(const String & query_id, TimePoint start_time, UInt64 interval_milliseconds);
|
||||
void finishQuery(const String & query_id, TimePoint finish_time, QueryStatusInfoPtr query_info = nullptr);
|
||||
|
||||
private:
|
||||
std::optional<QueryMetricLogElement> createLogMetricElement(const String & query_id, const QueryStatusInfo & query_info, TimePoint current_time, bool schedule_next = true);
|
||||
std::optional<QueryMetricLogElement> createLogMetricElement(const String & query_id, const QueryStatusInfo & query_info, TimePoint query_info_time, bool schedule_next = true);
|
||||
|
||||
std::recursive_mutex queries_mutex;
|
||||
std::unordered_map<String, QueryMetricLogStatus> queries;
|
||||
|
||||
@ -383,12 +383,12 @@ void Session::authenticate(const Credentials & credentials_, const Poco::Net::So
|
||||
|
||||
void Session::checkIfUserIsStillValid()
|
||||
{
|
||||
if (user && user->valid_until)
|
||||
if (const auto valid_until = user_authenticated_with.getValidUntil())
|
||||
{
|
||||
const time_t now = std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
|
||||
|
||||
if (now > user->valid_until)
|
||||
throw Exception(ErrorCodes::USER_EXPIRED, "User expired");
|
||||
if (now > valid_until)
|
||||
throw Exception(ErrorCodes::USER_EXPIRED, "Authentication method used has expired");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -6,7 +6,9 @@
|
||||
#include <Columns/ColumnTuple.h>
|
||||
|
||||
#include <Common/typeid_cast.h>
|
||||
#include <Columns/ColumnDecimal.h>
|
||||
|
||||
#include <DataTypes/DataTypeDateTime64.h>
|
||||
#include <DataTypes/DataTypeTuple.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
|
||||
@ -278,6 +280,108 @@ void Set::checkIsCreated() const
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Trying to use set before it has been built.");
|
||||
}
|
||||
|
||||
ColumnUInt8::Ptr checkDateTimePrecision(const ColumnWithTypeAndName & column_to_cast)
|
||||
{
|
||||
// Handle nullable columns
|
||||
const ColumnNullable * original_nullable_column = typeid_cast<const ColumnNullable *>(column_to_cast.column.get());
|
||||
const IColumn * original_nested_column = original_nullable_column
|
||||
? &original_nullable_column->getNestedColumn()
|
||||
: column_to_cast.column.get();
|
||||
|
||||
// Check if the original column is of ColumnDecimal<DateTime64> type
|
||||
const auto * original_decimal_column = typeid_cast<const ColumnDecimal<DateTime64> *>(original_nested_column);
|
||||
if (!original_decimal_column)
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Expected ColumnDecimal for DateTime64");
|
||||
|
||||
// Get the data array from the original column
|
||||
const auto & original_data = original_decimal_column->getData();
|
||||
size_t vec_res_size = original_data.size();
|
||||
|
||||
// Prepare the precision null map
|
||||
auto precision_null_map_column = ColumnUInt8::create(vec_res_size, 0);
|
||||
NullMap & precision_null_map = precision_null_map_column->getData();
|
||||
|
||||
// Determine which rows should be null based on precision loss
|
||||
const auto * datetime64_type = assert_cast<const DataTypeDateTime64 *>(column_to_cast.type.get());
|
||||
auto scale = datetime64_type->getScale();
|
||||
if (scale >= 1)
|
||||
{
|
||||
Int64 scale_multiplier = common::exp10_i32(scale);
|
||||
for (size_t row = 0; row < vec_res_size; ++row)
|
||||
{
|
||||
Int64 value = original_data[row];
|
||||
if (value % scale_multiplier != 0)
|
||||
precision_null_map[row] = 1; // Mark as null due to precision loss
|
||||
else
|
||||
precision_null_map[row] = 0;
|
||||
}
|
||||
}
|
||||
|
||||
return precision_null_map_column;
|
||||
}
|
||||
|
||||
ColumnPtr mergeNullMaps(const ColumnPtr & null_map_column1, const ColumnUInt8::Ptr & null_map_column2)
|
||||
{
|
||||
if (!null_map_column1)
|
||||
return null_map_column2;
|
||||
if (!null_map_column2)
|
||||
return null_map_column1;
|
||||
|
||||
const auto & null_map1 = assert_cast<const ColumnUInt8 &>(*null_map_column1).getData();
|
||||
const auto & null_map2 = (*null_map_column2).getData();
|
||||
|
||||
size_t size = null_map1.size();
|
||||
if (size != null_map2.size())
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Null maps have different sizes");
|
||||
|
||||
auto merged_null_map_column = ColumnUInt8::create(size);
|
||||
auto & merged_null_map = merged_null_map_column->getData();
|
||||
|
||||
for (size_t i = 0; i < size; ++i)
|
||||
merged_null_map[i] = null_map1[i] || null_map2[i];
|
||||
|
||||
return merged_null_map_column;
|
||||
}
|
||||
|
||||
void Set::processDateTime64Column(
|
||||
const ColumnWithTypeAndName & column_to_cast,
|
||||
ColumnPtr & result,
|
||||
ColumnPtr & null_map_holder,
|
||||
ConstNullMapPtr & null_map) const
|
||||
{
|
||||
// Check for sub-second precision and create a null map
|
||||
ColumnUInt8::Ptr filtered_null_map_column = checkDateTimePrecision(column_to_cast);
|
||||
|
||||
// Extract existing null map and nested column from the result
|
||||
const ColumnNullable * result_nullable_column = typeid_cast<const ColumnNullable *>(result.get());
|
||||
const IColumn * nested_result_column = result_nullable_column
|
||||
? &result_nullable_column->getNestedColumn()
|
||||
: result.get();
|
||||
|
||||
ColumnPtr existing_null_map_column = result_nullable_column
|
||||
? result_nullable_column->getNullMapColumnPtr()
|
||||
: nullptr;
|
||||
|
||||
if (transform_null_in)
|
||||
{
|
||||
if (!null_map_holder)
|
||||
null_map_holder = filtered_null_map_column;
|
||||
else
|
||||
null_map_holder = mergeNullMaps(null_map_holder, filtered_null_map_column);
|
||||
|
||||
const ColumnUInt8 * null_map_column = checkAndGetColumn<ColumnUInt8>(null_map_holder.get());
|
||||
if (!null_map_column)
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Null map must be ColumnUInt8");
|
||||
|
||||
null_map = &null_map_column->getData();
|
||||
}
|
||||
else
|
||||
{
|
||||
ColumnPtr merged_null_map_column = mergeNullMaps(existing_null_map_column, filtered_null_map_column);
|
||||
result = ColumnNullable::create(nested_result_column->getPtr(), merged_null_map_column);
|
||||
}
|
||||
}
|
||||
|
||||
ColumnPtr Set::execute(const ColumnsWithTypeAndName & columns, bool negative) const
|
||||
{
|
||||
size_t num_key_columns = columns.size();
|
||||
@ -314,6 +418,10 @@ ColumnPtr Set::execute(const ColumnsWithTypeAndName & columns, bool negative) co
|
||||
Columns materialized_columns;
|
||||
materialized_columns.reserve(num_key_columns);
|
||||
|
||||
/// We will check existence in Set only for keys whose components do not contain any NULL value.
|
||||
ConstNullMapPtr null_map{};
|
||||
ColumnPtr null_map_holder;
|
||||
|
||||
for (size_t i = 0; i < num_key_columns; ++i)
|
||||
{
|
||||
ColumnPtr result;
|
||||
@ -331,13 +439,17 @@ ColumnPtr Set::execute(const ColumnsWithTypeAndName & columns, bool negative) co
|
||||
result = castColumnAccurate(column_to_cast, data_types[i], cast_cache.get());
|
||||
}
|
||||
|
||||
materialized_columns.emplace_back() = result;
|
||||
key_columns.emplace_back() = materialized_columns.back().get();
|
||||
// If the original column is DateTime64, check for sub-second precision
|
||||
if (isDateTime64(column_to_cast.column->getDataType()))
|
||||
{
|
||||
processDateTime64Column(column_to_cast, result, null_map_holder, null_map);
|
||||
}
|
||||
|
||||
// Append the result to materialized columns
|
||||
materialized_columns.emplace_back(std::move(result));
|
||||
key_columns.emplace_back(materialized_columns.back().get());
|
||||
}
|
||||
|
||||
/// We will check existence in Set only for keys whose components do not contain any NULL value.
|
||||
ConstNullMapPtr null_map{};
|
||||
ColumnPtr null_map_holder;
|
||||
if (!transform_null_in)
|
||||
null_map_holder = extractNestedColumnsAndNullMap(key_columns, null_map);
|
||||
|
||||
|
||||
@ -61,6 +61,8 @@ public:
|
||||
|
||||
void checkIsCreated() const;
|
||||
|
||||
void processDateTime64Column(const ColumnWithTypeAndName & column_to_cast, ColumnPtr & result, ColumnPtr & null_map_holder, ConstNullMapPtr & null_map) const;
|
||||
|
||||
/** For columns of 'block', check belonging of corresponding rows to the set.
|
||||
* Return UInt8 column with the result.
|
||||
*/
|
||||
|
||||
@ -119,7 +119,7 @@ void ThreadGroup::unlinkThread()
|
||||
ThreadGroupPtr ThreadGroup::createForQuery(ContextPtr query_context_, std::function<void()> fatal_error_callback_)
|
||||
{
|
||||
auto group = std::make_shared<ThreadGroup>(query_context_, std::move(fatal_error_callback_));
|
||||
group->memory_tracker.setDescription("(for query)");
|
||||
group->memory_tracker.setDescription("Query");
|
||||
return group;
|
||||
}
|
||||
|
||||
@ -127,7 +127,7 @@ ThreadGroupPtr ThreadGroup::createForBackgroundProcess(ContextPtr storage_contex
|
||||
{
|
||||
auto group = std::make_shared<ThreadGroup>(storage_context);
|
||||
|
||||
group->memory_tracker.setDescription("background process to apply mutate/merge in table");
|
||||
group->memory_tracker.setDescription("Background process (mutate/merge)");
|
||||
/// However settings from storage context have to be applied
|
||||
const Settings & settings = storage_context->getSettingsRef();
|
||||
group->memory_tracker.setProfilerStep(settings[Setting::memory_profiler_step]);
|
||||
@ -384,7 +384,7 @@ void ThreadStatus::initPerformanceCounters()
|
||||
/// TODO: make separate query_thread_performance_counters and thread_performance_counters
|
||||
performance_counters.resetCounters();
|
||||
memory_tracker.resetCounters();
|
||||
memory_tracker.setDescription("(for thread)");
|
||||
memory_tracker.setDescription("Thread");
|
||||
|
||||
query_start_time.setUp();
|
||||
|
||||
|
||||
@ -81,6 +81,7 @@
|
||||
#include <base/EnumReflection.h>
|
||||
#include <base/demangle.h>
|
||||
|
||||
#include <chrono>
|
||||
#include <memory>
|
||||
#include <random>
|
||||
|
||||
@ -460,7 +461,7 @@ QueryLogElement logQueryStart(
|
||||
return elem;
|
||||
}
|
||||
|
||||
void logQueryMetricLogFinish(ContextPtr context, bool internal, String query_id, QueryStatusInfoPtr info)
|
||||
void logQueryMetricLogFinish(ContextPtr context, bool internal, String query_id, std::chrono::system_clock::time_point finish_time, QueryStatusInfoPtr info)
|
||||
{
|
||||
if (auto query_metric_log = context->getQueryMetricLog(); query_metric_log && !internal)
|
||||
{
|
||||
@ -475,11 +476,11 @@ void logQueryMetricLogFinish(ContextPtr context, bool internal, String query_id,
|
||||
/// to query the final state in query_log.
|
||||
auto collect_on_finish = info->elapsed_microseconds > interval_milliseconds * 1000;
|
||||
auto query_info = collect_on_finish ? info : nullptr;
|
||||
query_metric_log->finishQuery(query_id, query_info);
|
||||
query_metric_log->finishQuery(query_id, finish_time, query_info);
|
||||
}
|
||||
else
|
||||
{
|
||||
query_metric_log->finishQuery(query_id, nullptr);
|
||||
query_metric_log->finishQuery(query_id, finish_time, nullptr);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -503,6 +504,7 @@ void logQueryFinish(
|
||||
/// Update performance counters before logging to query_log
|
||||
CurrentThread::finalizePerformanceCounters();
|
||||
|
||||
auto time_now = std::chrono::system_clock::now();
|
||||
QueryStatusInfo info = process_list_elem->getInfo(true, settings[Setting::log_profile_events]);
|
||||
elem.type = QueryLogElementType::QUERY_FINISH;
|
||||
|
||||
@ -597,7 +599,7 @@ void logQueryFinish(
|
||||
}
|
||||
}
|
||||
|
||||
logQueryMetricLogFinish(context, internal, elem.client_info.current_query_id, std::make_shared<QueryStatusInfo>(info));
|
||||
logQueryMetricLogFinish(context, internal, elem.client_info.current_query_id, time_now, std::make_shared<QueryStatusInfo>(info));
|
||||
}
|
||||
|
||||
if (query_span)
|
||||
@ -697,7 +699,7 @@ void logQueryException(
|
||||
query_span->finish();
|
||||
}
|
||||
|
||||
logQueryMetricLogFinish(context, internal, elem.client_info.current_query_id, info);
|
||||
logQueryMetricLogFinish(context, internal, elem.client_info.current_query_id, time_now, info);
|
||||
}
|
||||
|
||||
void logExceptionBeforeStart(
|
||||
@ -796,7 +798,7 @@ void logExceptionBeforeStart(
|
||||
}
|
||||
}
|
||||
|
||||
logQueryMetricLogFinish(context, false, elem.client_info.current_query_id, nullptr);
|
||||
logQueryMetricLogFinish(context, false, elem.client_info.current_query_id, std::chrono::system_clock::now(), nullptr);
|
||||
}
|
||||
|
||||
void validateAnalyzerSettings(ASTPtr ast, bool context_value)
|
||||
|
||||
@ -14,6 +14,15 @@ namespace ErrorCodes
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
void formatValidUntil(const IAST & valid_until, const IAST::FormatSettings & settings)
|
||||
{
|
||||
settings.ostr << (settings.hilite ? IAST::hilite_keyword : "") << " VALID UNTIL " << (settings.hilite ? IAST::hilite_none : "");
|
||||
valid_until.format(settings);
|
||||
}
|
||||
}
|
||||
|
||||
std::optional<String> ASTAuthenticationData::getPassword() const
|
||||
{
|
||||
if (contains_password)
|
||||
@ -46,6 +55,12 @@ void ASTAuthenticationData::formatImpl(const FormatSettings & settings, FormatSt
|
||||
{
|
||||
settings.ostr << (settings.hilite ? IAST::hilite_keyword : "") << " no_password"
|
||||
<< (settings.hilite ? IAST::hilite_none : "");
|
||||
|
||||
if (valid_until)
|
||||
{
|
||||
formatValidUntil(*valid_until, settings);
|
||||
}
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
@ -205,6 +220,11 @@ void ASTAuthenticationData::formatImpl(const FormatSettings & settings, FormatSt
|
||||
children[1]->format(settings);
|
||||
}
|
||||
|
||||
if (valid_until)
|
||||
{
|
||||
formatValidUntil(*valid_until, settings);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
bool ASTAuthenticationData::hasSecretParts() const
|
||||
|
||||
@ -41,6 +41,7 @@ public:
|
||||
|
||||
bool contains_password = false;
|
||||
bool contains_hash = false;
|
||||
ASTPtr valid_until;
|
||||
|
||||
protected:
|
||||
void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override;
|
||||
|
||||
@ -260,8 +260,10 @@ void ASTCreateUserQuery::formatImpl(const FormatSettings & format, FormatState &
|
||||
formatAuthenticationData(authentication_methods, format);
|
||||
}
|
||||
|
||||
if (valid_until)
|
||||
formatValidUntil(*valid_until, format);
|
||||
if (global_valid_until)
|
||||
{
|
||||
formatValidUntil(*global_valid_until, format);
|
||||
}
|
||||
|
||||
if (hosts)
|
||||
formatHosts(nullptr, *hosts, format);
|
||||
|
||||
@ -62,7 +62,7 @@ public:
|
||||
|
||||
std::shared_ptr<ASTDatabaseOrNone> default_database;
|
||||
|
||||
ASTPtr valid_until;
|
||||
ASTPtr global_valid_until;
|
||||
|
||||
String getID(char) const override;
|
||||
ASTPtr clone() const override;
|
||||
|
||||
@ -43,6 +43,19 @@ namespace
|
||||
});
|
||||
}
|
||||
|
||||
bool parseValidUntil(IParserBase::Pos & pos, Expected & expected, ASTPtr & valid_until)
|
||||
{
|
||||
return IParserBase::wrapParseImpl(pos, [&]
|
||||
{
|
||||
if (!ParserKeyword{Keyword::VALID_UNTIL}.ignore(pos, expected))
|
||||
return false;
|
||||
|
||||
ParserStringAndSubstitution until_p;
|
||||
|
||||
return until_p.parse(pos, valid_until, expected);
|
||||
});
|
||||
}
|
||||
|
||||
bool parseAuthenticationData(
|
||||
IParserBase::Pos & pos,
|
||||
Expected & expected,
|
||||
@ -223,6 +236,8 @@ namespace
|
||||
if (http_auth_scheme)
|
||||
auth_data->children.push_back(std::move(http_auth_scheme));
|
||||
|
||||
parseValidUntil(pos, expected, auth_data->valid_until);
|
||||
|
||||
return true;
|
||||
});
|
||||
}
|
||||
@ -283,6 +298,8 @@ namespace
|
||||
authentication_methods.emplace_back(std::make_shared<ASTAuthenticationData>());
|
||||
authentication_methods.back()->type = AuthenticationType::NO_PASSWORD;
|
||||
|
||||
parseValidUntil(pos, expected, authentication_methods.back()->valid_until);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -471,19 +488,6 @@ namespace
|
||||
});
|
||||
}
|
||||
|
||||
bool parseValidUntil(IParserBase::Pos & pos, Expected & expected, ASTPtr & valid_until)
|
||||
{
|
||||
return IParserBase::wrapParseImpl(pos, [&]
|
||||
{
|
||||
if (!ParserKeyword{Keyword::VALID_UNTIL}.ignore(pos, expected))
|
||||
return false;
|
||||
|
||||
ParserStringAndSubstitution until_p;
|
||||
|
||||
return until_p.parse(pos, valid_until, expected);
|
||||
});
|
||||
}
|
||||
|
||||
bool parseAddIdentifiedWith(IParserBase::Pos & pos, Expected & expected, std::vector<std::shared_ptr<ASTAuthenticationData>> & auth_data)
|
||||
{
|
||||
return IParserBase::wrapParseImpl(pos, [&]
|
||||
@ -554,7 +558,7 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
std::shared_ptr<ASTSettingsProfileElements> settings;
|
||||
std::shared_ptr<ASTRolesOrUsersSet> grantees;
|
||||
std::shared_ptr<ASTDatabaseOrNone> default_database;
|
||||
ASTPtr valid_until;
|
||||
ASTPtr global_valid_until;
|
||||
String cluster;
|
||||
String storage_name;
|
||||
bool reset_authentication_methods_to_new = false;
|
||||
@ -568,20 +572,27 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
{
|
||||
parsed_identified_with = parseIdentifiedOrNotIdentified(pos, expected, auth_data);
|
||||
|
||||
if (!parsed_identified_with && alter)
|
||||
if (parsed_identified_with)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
else if (alter)
|
||||
{
|
||||
parsed_add_identified_with = parseAddIdentifiedWith(pos, expected, auth_data);
|
||||
if (parsed_add_identified_with)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!reset_authentication_methods_to_new && alter && auth_data.empty())
|
||||
{
|
||||
reset_authentication_methods_to_new = parseResetAuthenticationMethods(pos, expected);
|
||||
}
|
||||
|
||||
if (!valid_until)
|
||||
{
|
||||
parseValidUntil(pos, expected, valid_until);
|
||||
if (reset_authentication_methods_to_new)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
AllowedClientHosts new_hosts;
|
||||
@ -640,6 +651,14 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
if (storage_name.empty() && ParserKeyword{Keyword::IN}.ignore(pos, expected) && parseAccessStorageName(pos, expected, storage_name))
|
||||
continue;
|
||||
|
||||
if (auth_data.empty() && !global_valid_until)
|
||||
{
|
||||
if (parseValidUntil(pos, expected, global_valid_until))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
@ -674,7 +693,7 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
query->settings = std::move(settings);
|
||||
query->grantees = std::move(grantees);
|
||||
query->default_database = std::move(default_database);
|
||||
query->valid_until = std::move(valid_until);
|
||||
query->global_valid_until = std::move(global_valid_until);
|
||||
query->storage_name = std::move(storage_name);
|
||||
query->reset_authentication_methods_to_new = reset_authentication_methods_to_new;
|
||||
query->add_identified_with = parsed_add_identified_with;
|
||||
@ -685,8 +704,8 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
|
||||
query->children.push_back(authentication_method);
|
||||
}
|
||||
|
||||
if (query->valid_until)
|
||||
query->children.push_back(query->valid_until);
|
||||
if (query->global_valid_until)
|
||||
query->children.push_back(query->global_valid_until);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -15,16 +15,17 @@ namespace DB
|
||||
class NativeInputFormat final : public IInputFormat
|
||||
{
|
||||
public:
|
||||
NativeInputFormat(ReadBuffer & buf, const Block & header_, const FormatSettings & settings)
|
||||
NativeInputFormat(ReadBuffer & buf, const Block & header_, const FormatSettings & settings_)
|
||||
: IInputFormat(header_, &buf)
|
||||
, reader(std::make_unique<NativeReader>(
|
||||
buf,
|
||||
header_,
|
||||
0,
|
||||
settings,
|
||||
settings.defaults_for_omitted_fields ? &block_missing_values : nullptr))
|
||||
settings_,
|
||||
settings_.defaults_for_omitted_fields ? &block_missing_values : nullptr))
|
||||
, header(header_)
|
||||
, block_missing_values(header.columns())
|
||||
, settings(settings_)
|
||||
{
|
||||
}
|
||||
|
||||
@ -55,7 +56,7 @@ public:
|
||||
|
||||
void setReadBuffer(ReadBuffer & in_) override
|
||||
{
|
||||
reader = std::make_unique<NativeReader>(in_, header, 0);
|
||||
reader = std::make_unique<NativeReader>(in_, header, 0, settings, settings.defaults_for_omitted_fields ? &block_missing_values : nullptr);
|
||||
IInputFormat::setReadBuffer(in_);
|
||||
}
|
||||
|
||||
@ -67,6 +68,7 @@ private:
|
||||
std::unique_ptr<NativeReader> reader;
|
||||
Block header;
|
||||
BlockMissingValues block_missing_values;
|
||||
const FormatSettings settings;
|
||||
size_t approx_bytes_read_for_chunk = 0;
|
||||
};
|
||||
|
||||
|
||||
@ -296,6 +296,31 @@ void ParquetPlainValuesReader<ColumnString>::readBatch(
|
||||
);
|
||||
}
|
||||
|
||||
template <typename TColumn>
|
||||
void ParquetBitPlainReader<TColumn>::readBatch(
|
||||
MutableColumnPtr & col_ptr, LazyNullMap & null_map, UInt32 num_values)
|
||||
{
|
||||
auto cursor = col_ptr->size();
|
||||
auto * column_data = getResizedPrimitiveData(*assert_cast<TColumn *>(col_ptr.get()), cursor + num_values);
|
||||
|
||||
def_level_reader->visitNullableValues(
|
||||
cursor,
|
||||
num_values,
|
||||
max_def_level,
|
||||
null_map,
|
||||
/* individual_visitor */ [&](size_t nest_cursor)
|
||||
{
|
||||
uint8_t byte;
|
||||
bit_reader->GetValue(1, &byte);
|
||||
column_data[nest_cursor] = byte;
|
||||
},
|
||||
/* repeated_visitor */ [&](size_t nest_cursor, UInt32 count)
|
||||
{
|
||||
bit_reader->GetBatch(1, &column_data[nest_cursor], count);
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
template <>
|
||||
void ParquetPlainValuesReader<ColumnDecimal<DateTime64>, ParquetReaderTypes::TimestampInt96>::readBatch(
|
||||
@ -561,6 +586,9 @@ template class ParquetPlainValuesReader<ColumnDecimal<Decimal32>>;
|
||||
template class ParquetPlainValuesReader<ColumnDecimal<Decimal64>>;
|
||||
template class ParquetPlainValuesReader<ColumnDecimal<DateTime64>>;
|
||||
template class ParquetPlainValuesReader<ColumnString>;
|
||||
template class ParquetPlainValuesReader<ColumnUInt8>;
|
||||
|
||||
template class ParquetBitPlainReader<ColumnUInt8>;
|
||||
|
||||
template class ParquetFixedLenPlainReader<ColumnDecimal<Decimal128>>;
|
||||
template class ParquetFixedLenPlainReader<ColumnDecimal<Decimal256>>;
|
||||
@ -569,6 +597,7 @@ template class ParquetRleLCReader<ColumnUInt8>;
|
||||
template class ParquetRleLCReader<ColumnUInt16>;
|
||||
template class ParquetRleLCReader<ColumnUInt32>;
|
||||
|
||||
template class ParquetRleDictReader<ColumnUInt8>;
|
||||
template class ParquetRleDictReader<ColumnInt32>;
|
||||
template class ParquetRleDictReader<ColumnUInt32>;
|
||||
template class ParquetRleDictReader<ColumnInt64>;
|
||||
|
||||
@ -172,6 +172,27 @@ private:
|
||||
ParquetDataBuffer plain_data_buffer;
|
||||
};
|
||||
|
||||
template <typename TColumn>
|
||||
class ParquetBitPlainReader : public ParquetDataValuesReader
|
||||
{
|
||||
public:
|
||||
ParquetBitPlainReader(
|
||||
Int32 max_def_level_,
|
||||
std::unique_ptr<RleValuesReader> def_level_reader_,
|
||||
std::unique_ptr<arrow::bit_util::BitReader> bit_reader_)
|
||||
: max_def_level(max_def_level_)
|
||||
, def_level_reader(std::move(def_level_reader_))
|
||||
, bit_reader(std::move(bit_reader_))
|
||||
{}
|
||||
|
||||
void readBatch(MutableColumnPtr & col_ptr, LazyNullMap & null_map, UInt32 num_values) override;
|
||||
|
||||
private:
|
||||
Int32 max_def_level;
|
||||
std::unique_ptr<RleValuesReader> def_level_reader;
|
||||
std::unique_ptr<arrow::bit_util::BitReader> bit_reader;
|
||||
};
|
||||
|
||||
/**
|
||||
* The data and definition level encoding are same as ParquetPlainValuesReader.
|
||||
* But the element size is const and bigger than primitive data type.
|
||||
|
||||
@ -425,16 +425,29 @@ void ParquetLeafColReader<TColumn>::initDataReader(
|
||||
degradeDictionary();
|
||||
}
|
||||
|
||||
ParquetDataBuffer parquet_buffer = [&]()
|
||||
if (col_descriptor.physical_type() == parquet::Type::BOOLEAN)
|
||||
{
|
||||
if constexpr (!std::is_same_v<ColumnDecimal<DateTime64>, TColumn>)
|
||||
return ParquetDataBuffer(buffer, max_size);
|
||||
if constexpr (std::is_same_v<TColumn, ColumnUInt8>)
|
||||
{
|
||||
auto bit_reader = std::make_unique<arrow::bit_util::BitReader>(buffer, max_size);
|
||||
data_values_reader = std::make_unique<ParquetBitPlainReader<ColumnUInt8>>(col_descriptor.max_definition_level(),
|
||||
std::move(def_level_reader),
|
||||
std::move(bit_reader));
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
ParquetDataBuffer parquet_buffer = [&]()
|
||||
{
|
||||
if constexpr (!std::is_same_v<ColumnDecimal<DateTime64>, TColumn>)
|
||||
return ParquetDataBuffer(buffer, max_size);
|
||||
|
||||
auto scale = assert_cast<const DataTypeDateTime64 &>(*base_data_type).getScale();
|
||||
return ParquetDataBuffer(buffer, max_size, scale);
|
||||
}();
|
||||
data_values_reader = createPlainReader<TColumn>(
|
||||
col_descriptor, std::move(def_level_reader), std::move(parquet_buffer));
|
||||
auto scale = assert_cast<const DataTypeDateTime64 &>(*base_data_type).getScale();
|
||||
return ParquetDataBuffer(buffer, max_size, scale);
|
||||
}();
|
||||
data_values_reader = createPlainReader<TColumn>(
|
||||
col_descriptor, std::move(def_level_reader), std::move(parquet_buffer));
|
||||
}
|
||||
break;
|
||||
}
|
||||
case parquet::Encoding::RLE_DICTIONARY:
|
||||
@ -612,6 +625,12 @@ std::unique_ptr<ParquetDataValuesReader> ParquetLeafColReader<TColumn>::createDi
|
||||
});
|
||||
return res;
|
||||
}
|
||||
|
||||
if (col_descriptor.physical_type() == parquet::Type::type::BOOLEAN)
|
||||
{
|
||||
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Dictionary encoding for booleans is not supported");
|
||||
}
|
||||
|
||||
return std::make_unique<ParquetRleDictReader<TColumn>>(
|
||||
col_descriptor.max_definition_level(),
|
||||
std::move(def_level_reader),
|
||||
@ -620,6 +639,7 @@ std::unique_ptr<ParquetDataValuesReader> ParquetLeafColReader<TColumn>::createDi
|
||||
}
|
||||
|
||||
|
||||
template class ParquetLeafColReader<ColumnUInt8>;
|
||||
template class ParquetLeafColReader<ColumnInt32>;
|
||||
template class ParquetLeafColReader<ColumnUInt32>;
|
||||
template class ParquetLeafColReader<ColumnInt64>;
|
||||
|
||||
@ -263,7 +263,7 @@ std::unique_ptr<ParquetColumnReader> ColReaderFactory::makeReader()
|
||||
switch (col_descriptor.physical_type())
|
||||
{
|
||||
case parquet::Type::BOOLEAN:
|
||||
break;
|
||||
return makeLeafReader<DataTypeUInt8>();
|
||||
case parquet::Type::INT32:
|
||||
return fromInt32();
|
||||
case parquet::Type::INT64:
|
||||
|
||||
@ -752,7 +752,8 @@ std::optional<String> optimizeUseAggregateProjections(QueryPlan::Node & node, Qu
|
||||
Pipe pipe(std::make_shared<SourceFromSingleChunk>(std::move(block_with_count)));
|
||||
projection_reading = std::make_unique<ReadFromPreparedSource>(std::move(pipe));
|
||||
|
||||
selected_projection_name = "Optimized trivial count";
|
||||
/// Use @minmax_count_projection name as it goes through the same optimization.
|
||||
selected_projection_name = metadata->minmax_count_projection->name;
|
||||
has_ordinary_parts = reading->getAnalyzedResult() != nullptr;
|
||||
}
|
||||
else
|
||||
|
||||
@ -91,6 +91,12 @@ void CertificateReloader::tryLoad(const Poco::Util::AbstractConfiguration & conf
|
||||
}
|
||||
|
||||
|
||||
void CertificateReloader::tryLoadClient(const Poco::Util::AbstractConfiguration & config)
|
||||
{
|
||||
tryLoad(config, nullptr, Poco::Net::SSLManager::CFG_CLIENT_PREFIX);
|
||||
}
|
||||
|
||||
|
||||
void CertificateReloader::tryLoad(const Poco::Util::AbstractConfiguration & config, SSL_CTX * ctx, const std::string & prefix)
|
||||
{
|
||||
std::lock_guard lock{data_mutex};
|
||||
@ -107,7 +113,12 @@ std::list<CertificateReloader::MultiData>::iterator CertificateReloader::findOrI
|
||||
else
|
||||
{
|
||||
if (!ctx)
|
||||
ctx = Poco::Net::SSLManager::instance().defaultServerContext()->sslContext();
|
||||
{
|
||||
if (prefix == Poco::Net::SSLManager::CFG_CLIENT_PREFIX)
|
||||
ctx = Poco::Net::SSLManager::instance().defaultClientContext()->sslContext();
|
||||
else
|
||||
ctx = Poco::Net::SSLManager::instance().defaultServerContext()->sslContext();
|
||||
}
|
||||
data.push_back(MultiData(ctx));
|
||||
--it;
|
||||
data_index[prefix] = it;
|
||||
|
||||
@ -77,6 +77,9 @@ public:
|
||||
/// Handle configuration reload for default path
|
||||
void tryLoad(const Poco::Util::AbstractConfiguration & config);
|
||||
|
||||
/// Handle configuration reload client for default path
|
||||
void tryLoadClient(const Poco::Util::AbstractConfiguration & config);
|
||||
|
||||
/// Handle configuration reload
|
||||
void tryLoad(const Poco::Util::AbstractConfiguration & config, SSL_CTX * ctx, const std::string & prefix);
|
||||
|
||||
|
||||
@ -262,7 +262,7 @@ MergeTreeReaderWide::FileStreams::iterator MergeTreeReaderWide::addStream(const
|
||||
/*num_columns_in_mark=*/ 1);
|
||||
|
||||
auto stream_settings = settings;
|
||||
stream_settings.is_low_cardinality_dictionary = substream_path.size() > 1 && substream_path[substream_path.size() - 2].type == ISerialization::Substream::Type::DictionaryKeys;
|
||||
stream_settings.is_low_cardinality_dictionary = ISerialization::isLowCardinalityDictionarySubcolumn(substream_path);
|
||||
|
||||
auto create_stream = [&]<typename Stream>()
|
||||
{
|
||||
|
||||
@ -30,10 +30,11 @@ namespace ErrorCodes
|
||||
extern const int BAD_ARGUMENTS;
|
||||
}
|
||||
|
||||
// clang-format off
|
||||
|
||||
/** These settings represent fine tunes for internal details of MergeTree storages
|
||||
* and should not be changed by the user without a reason.
|
||||
*/
|
||||
|
||||
#define MERGE_TREE_SETTINGS(DECLARE, ALIAS) \
|
||||
DECLARE(UInt64, min_compress_block_size, 0, "When granule is written, compress the data in buffer if the size of pending uncompressed data is larger or equal than the specified threshold. If this setting is not set, the corresponding global setting is used.", 0) \
|
||||
DECLARE(UInt64, max_compress_block_size, 0, "Compress the pending uncompressed data in buffer if its size is larger or equal than the specified threshold. Block of data will be compressed even if the current granule is not finished. If this setting is not set, the corresponding global setting is used.", 0) \
|
||||
@ -88,7 +89,7 @@ namespace ErrorCodes
|
||||
DECLARE(UInt64, min_age_to_force_merge_seconds, 0, "If all parts in a certain range are older than this value, range will be always eligible for merging. Set to 0 to disable.", 0) \
|
||||
DECLARE(Bool, min_age_to_force_merge_on_partition_only, false, "Whether min_age_to_force_merge_seconds should be applied only on the entire partition and not on subset.", false) \
|
||||
DECLARE(UInt64, number_of_free_entries_in_pool_to_execute_optimize_entire_partition, 25, "When there is less than specified number of free entries in pool, do not try to execute optimize entire partition with a merge (this merge is created when set min_age_to_force_merge_seconds > 0 and min_age_to_force_merge_on_partition_only = true). This is to leave free threads for regular merges and avoid \"Too many parts\"", 0) \
|
||||
DECLARE(Bool, remove_rolled_back_parts_immediately, 1, "Setting for an incomplete experimental feature.", 0) \
|
||||
DECLARE(Bool, remove_rolled_back_parts_immediately, 1, "Setting for an incomplete experimental feature.", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, replicated_max_mutations_in_one_entry, 10000, "Max number of mutation commands that can be merged together and executed in one MUTATE_PART entry (0 means unlimited)", 0) \
|
||||
DECLARE(UInt64, number_of_mutations_to_delay, 500, "If table has at least that many unfinished mutations, artificially slow down mutations of table. Disabled if set to 0", 0) \
|
||||
DECLARE(UInt64, number_of_mutations_to_throw, 1000, "If table has at least that many unfinished mutations, throw 'Too many mutations' exception. Disabled if set to 0", 0) \
|
||||
@ -98,7 +99,7 @@ namespace ErrorCodes
|
||||
DECLARE(String, merge_workload, "", "Name of workload to be used to access resources for merges", 0) \
|
||||
DECLARE(String, mutation_workload, "", "Name of workload to be used to access resources for mutations", 0) \
|
||||
DECLARE(Milliseconds, background_task_preferred_step_execution_time_ms, 50, "Target time to execution of one step of merge or mutation. Can be exceeded if one step takes longer time", 0) \
|
||||
DECLARE(MergeSelectorAlgorithm, merge_selector_algorithm, MergeSelectorAlgorithm::SIMPLE, "The algorithm to select parts for merges assignment", 0) \
|
||||
DECLARE(MergeSelectorAlgorithm, merge_selector_algorithm, MergeSelectorAlgorithm::SIMPLE, "The algorithm to select parts for merges assignment", EXPERIMENTAL) \
|
||||
\
|
||||
/** Inserts settings. */ \
|
||||
DECLARE(UInt64, parts_to_delay_insert, 1000, "If table contains at least that many active parts in single partition, artificially slow down insert into table. Disabled if set to 0", 0) \
|
||||
@ -214,14 +215,14 @@ namespace ErrorCodes
|
||||
DECLARE(Bool, enable_block_offset_column, false, "Enable persisting column _block_offset for each row.", 0) \
|
||||
\
|
||||
/** Experimental/work in progress feature. Unsafe for production. */ \
|
||||
DECLARE(UInt64, part_moves_between_shards_enable, 0, "Experimental/Incomplete feature to move parts between shards. Does not take into account sharding expressions.", 0) \
|
||||
DECLARE(UInt64, part_moves_between_shards_delay_seconds, 30, "Time to wait before/after moving parts between shards.", 0) \
|
||||
DECLARE(Bool, allow_remote_fs_zero_copy_replication, false, "Don't use this setting in production, because it is not ready.", 0) \
|
||||
DECLARE(String, remote_fs_zero_copy_zookeeper_path, "/clickhouse/zero_copy", "ZooKeeper path for zero-copy table-independent info.", 0) \
|
||||
DECLARE(Bool, remote_fs_zero_copy_path_compatible_mode, false, "Run zero-copy in compatible mode during conversion process.", 0) \
|
||||
DECLARE(Bool, cache_populated_by_fetch, false, "Only available in ClickHouse Cloud", 0) \
|
||||
DECLARE(Bool, force_read_through_cache_for_merges, false, "Force read-through filesystem cache for merges", 0) \
|
||||
DECLARE(Bool, allow_experimental_replacing_merge_with_cleanup, false, "Allow experimental CLEANUP merges for ReplacingMergeTree with is_deleted column.", 0) \
|
||||
DECLARE(UInt64, part_moves_between_shards_enable, 0, "Experimental/Incomplete feature to move parts between shards. Does not take into account sharding expressions.", EXPERIMENTAL) \
|
||||
DECLARE(UInt64, part_moves_between_shards_delay_seconds, 30, "Time to wait before/after moving parts between shards.", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_remote_fs_zero_copy_replication, false, "Don't use this setting in production, because it is not ready.", BETA) \
|
||||
DECLARE(String, remote_fs_zero_copy_zookeeper_path, "/clickhouse/zero_copy", "ZooKeeper path for zero-copy table-independent info.", EXPERIMENTAL) \
|
||||
DECLARE(Bool, remote_fs_zero_copy_path_compatible_mode, false, "Run zero-copy in compatible mode during conversion process.", EXPERIMENTAL) \
|
||||
DECLARE(Bool, cache_populated_by_fetch, false, "Only available in ClickHouse Cloud", EXPERIMENTAL) \
|
||||
DECLARE(Bool, force_read_through_cache_for_merges, false, "Force read-through filesystem cache for merges", EXPERIMENTAL) \
|
||||
DECLARE(Bool, allow_experimental_replacing_merge_with_cleanup, false, "Allow experimental CLEANUP merges for ReplacingMergeTree with is_deleted column.", EXPERIMENTAL) \
|
||||
\
|
||||
/** Compress marks and primary key. */ \
|
||||
DECLARE(Bool, compress_marks, true, "Marks support compression, reduce mark file size and speed up network transmission.", 0) \
|
||||
@ -240,7 +241,7 @@ namespace ErrorCodes
|
||||
DECLARE(DeduplicateMergeProjectionMode, deduplicate_merge_projection_mode, DeduplicateMergeProjectionMode::THROW, "Whether to allow create projection for the table with non-classic MergeTree. Ignore option is purely for compatibility which might result in incorrect answer. Otherwise, if allowed, what is the action when merge, drop or rebuild.", 0) \
|
||||
|
||||
#define MAKE_OBSOLETE_MERGE_TREE_SETTING(M, TYPE, NAME, DEFAULT) \
|
||||
M(TYPE, NAME, DEFAULT, "Obsolete setting, does nothing.", BaseSettingsHelpers::Flags::OBSOLETE)
|
||||
M(TYPE, NAME, DEFAULT, "Obsolete setting, does nothing.", SettingsTierType::OBSOLETE)
|
||||
|
||||
#define OBSOLETE_MERGE_TREE_SETTINGS(M, ALIAS) \
|
||||
/** Obsolete settings that do nothing but left for compatibility reasons. */ \
|
||||
@ -278,8 +279,9 @@ namespace ErrorCodes
|
||||
MERGE_TREE_SETTINGS(M, ALIAS) \
|
||||
OBSOLETE_MERGE_TREE_SETTINGS(M, ALIAS)
|
||||
|
||||
DECLARE_SETTINGS_TRAITS(MergeTreeSettingsTraits, LIST_OF_MERGE_TREE_SETTINGS)
|
||||
// clang-format on
|
||||
|
||||
DECLARE_SETTINGS_TRAITS(MergeTreeSettingsTraits, LIST_OF_MERGE_TREE_SETTINGS)
|
||||
|
||||
/** Settings for the MergeTree family of engines.
|
||||
* Could be loaded from config or from a CREATE TABLE query (SETTINGS clause).
|
||||
@ -650,7 +652,8 @@ void MergeTreeSettings::dumpToSystemMergeTreeSettingsColumns(MutableColumnsAndCo
|
||||
res_columns[5]->insert(max);
|
||||
res_columns[6]->insert(writability == SettingConstraintWritability::CONST);
|
||||
res_columns[7]->insert(setting.getTypeName());
|
||||
res_columns[8]->insert(setting.isObsolete());
|
||||
res_columns[8]->insert(setting.getTier() == SettingsTierType::OBSOLETE);
|
||||
res_columns[9]->insert(setting.getTier());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -29,6 +29,7 @@ extern const int CANNOT_FSYNC;
|
||||
struct WriteBufferFromHDFS::WriteBufferFromHDFSImpl
|
||||
{
|
||||
std::string hdfs_uri;
|
||||
std::string hdfs_file_path;
|
||||
hdfsFile fout;
|
||||
HDFSBuilderWrapper builder;
|
||||
HDFSFSPtr fs;
|
||||
@ -36,25 +37,24 @@ struct WriteBufferFromHDFS::WriteBufferFromHDFSImpl
|
||||
|
||||
WriteBufferFromHDFSImpl(
|
||||
const std::string & hdfs_uri_,
|
||||
const std::string & hdfs_file_path_,
|
||||
const Poco::Util::AbstractConfiguration & config_,
|
||||
int replication_,
|
||||
const WriteSettings & write_settings_,
|
||||
int flags)
|
||||
: hdfs_uri(hdfs_uri_)
|
||||
, hdfs_file_path(hdfs_file_path_)
|
||||
, builder(createHDFSBuilder(hdfs_uri, config_))
|
||||
, fs(createHDFSFS(builder.get()))
|
||||
, write_settings(write_settings_)
|
||||
{
|
||||
const size_t begin_of_path = hdfs_uri.find('/', hdfs_uri.find("//") + 2);
|
||||
const String path = hdfs_uri.substr(begin_of_path);
|
||||
|
||||
/// O_WRONLY meaning create or overwrite i.e., implies O_TRUNCAT here
|
||||
fout = hdfsOpenFile(fs.get(), path.c_str(), flags, 0, replication_, 0);
|
||||
fout = hdfsOpenFile(fs.get(), hdfs_file_path.c_str(), flags, 0, replication_, 0);
|
||||
|
||||
if (fout == nullptr)
|
||||
{
|
||||
throw Exception(ErrorCodes::CANNOT_OPEN_FILE, "Unable to open HDFS file: {} ({}) error: {}",
|
||||
path, hdfs_uri, std::string(hdfsGetLastError()));
|
||||
hdfs_file_path, hdfs_uri, std::string(hdfsGetLastError()));
|
||||
}
|
||||
}
|
||||
|
||||
@ -71,7 +71,7 @@ struct WriteBufferFromHDFS::WriteBufferFromHDFSImpl
|
||||
rlock.unlock(std::max(0, bytes_written));
|
||||
|
||||
if (bytes_written < 0)
|
||||
throw Exception(ErrorCodes::NETWORK_ERROR, "Fail to write HDFS file: {} {}", hdfs_uri, std::string(hdfsGetLastError()));
|
||||
throw Exception(ErrorCodes::NETWORK_ERROR, "Fail to write HDFS file: {}, hdfs_uri: {}, {}", hdfs_file_path, hdfs_uri, std::string(hdfsGetLastError()));
|
||||
|
||||
if (write_settings.remote_throttler)
|
||||
write_settings.remote_throttler->add(bytes_written, ProfileEvents::RemoteWriteThrottlerBytes, ProfileEvents::RemoteWriteThrottlerSleepMicroseconds);
|
||||
@ -83,20 +83,21 @@ struct WriteBufferFromHDFS::WriteBufferFromHDFSImpl
|
||||
{
|
||||
int result = hdfsSync(fs.get(), fout);
|
||||
if (result < 0)
|
||||
throw ErrnoException(ErrorCodes::CANNOT_FSYNC, "Cannot HDFS sync {} {}", hdfs_uri, std::string(hdfsGetLastError()));
|
||||
throw ErrnoException(ErrorCodes::CANNOT_FSYNC, "Cannot HDFS sync {}, hdfs_url: {}, {}", hdfs_file_path, hdfs_uri, std::string(hdfsGetLastError()));
|
||||
}
|
||||
};
|
||||
|
||||
WriteBufferFromHDFS::WriteBufferFromHDFS(
|
||||
const std::string & hdfs_name_,
|
||||
const std::string & hdfs_uri_,
|
||||
const std::string & hdfs_file_path_,
|
||||
const Poco::Util::AbstractConfiguration & config_,
|
||||
int replication_,
|
||||
const WriteSettings & write_settings_,
|
||||
size_t buf_size_,
|
||||
int flags_)
|
||||
: WriteBufferFromFileBase(buf_size_, nullptr, 0)
|
||||
, impl(std::make_unique<WriteBufferFromHDFSImpl>(hdfs_name_, config_, replication_, write_settings_, flags_))
|
||||
, filename(hdfs_name_)
|
||||
, impl(std::make_unique<WriteBufferFromHDFSImpl>(hdfs_uri_, hdfs_file_path_, config_, replication_, write_settings_, flags_))
|
||||
, filename(hdfs_file_path_)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
@ -22,7 +22,8 @@ class WriteBufferFromHDFS final : public WriteBufferFromFileBase
|
||||
|
||||
public:
|
||||
WriteBufferFromHDFS(
|
||||
const String & hdfs_name_,
|
||||
const String & hdfs_uri_,
|
||||
const String & hdfs_file_path_,
|
||||
const Poco::Util::AbstractConfiguration & config_,
|
||||
int replication_,
|
||||
const WriteSettings & write_settings_ = {},
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
#include <Core/Settings.h>
|
||||
#include <Core/SettingsTierType.h>
|
||||
#include <DataTypes/DataTypeEnum.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
@ -30,6 +31,14 @@ ColumnsDescription SystemMergeTreeSettings<replicated>::getColumnsDescription()
|
||||
},
|
||||
{"type", std::make_shared<DataTypeString>(), "Setting type (implementation specific string value)."},
|
||||
{"is_obsolete", std::make_shared<DataTypeUInt8>(), "Shows whether a setting is obsolete."},
|
||||
{"tier", getSettingsTierEnum(), R"(
|
||||
Support level for this feature. ClickHouse features are organized in tiers, varying depending on the current status of their
|
||||
development and the expectations one might have when using them:
|
||||
* PRODUCTION: The feature is stable, safe to use and does not have issues interacting with other PRODUCTION features.
|
||||
* BETA: The feature is stable and safe. The outcome of using it together with other features is unknown and correctness is not guaranteed. Testing and reports are welcome.
|
||||
* EXPERIMENTAL: The feature is under development. Only intended for developers and ClickHouse enthusiasts. The feature might or might not work and could be removed at any time.
|
||||
* OBSOLETE: No longer supported. Either it is already removed or it will be removed in future releases.
|
||||
)"},
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@ -2,6 +2,8 @@
|
||||
|
||||
#include <Access/SettingsConstraintsAndProfileIDs.h>
|
||||
#include <Core/Settings.h>
|
||||
#include <Core/SettingsTierType.h>
|
||||
#include <DataTypes/DataTypeEnum.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
@ -34,6 +36,14 @@ ColumnsDescription StorageSystemSettings::getColumnsDescription()
|
||||
{"default", std::make_shared<DataTypeString>(), "Setting default value."},
|
||||
{"alias_for", std::make_shared<DataTypeString>(), "Flag that shows whether this name is an alias to another setting."},
|
||||
{"is_obsolete", std::make_shared<DataTypeUInt8>(), "Shows whether a setting is obsolete."},
|
||||
{"tier", getSettingsTierEnum(), R"(
|
||||
Support level for this feature. ClickHouse features are organized in tiers, varying depending on the current status of their
|
||||
development and the expectations one might have when using them:
|
||||
* PRODUCTION: The feature is stable, safe to use and does not have issues interacting with other PRODUCTION features.
|
||||
* BETA: The feature is stable and safe. The outcome of using it together with other features is unknown and correctness is not guaranteed. Testing and reports are welcome.
|
||||
* EXPERIMENTAL: The feature is under development. Only intended for developers and ClickHouse enthusiasts. The feature might or might not work and could be removed at any time.
|
||||
* OBSOLETE: No longer supported. Either it is already removed or it will be removed in future releases.
|
||||
)"},
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@ -97,9 +97,9 @@ class CI:
|
||||
),
|
||||
runner_type=Runners.BUILDER_ARM,
|
||||
),
|
||||
BuildNames.PACKAGE_AARCH64_ASAN: CommonJobConfigs.BUILD.with_properties(
|
||||
BuildNames.PACKAGE_ARM_ASAN: CommonJobConfigs.BUILD.with_properties(
|
||||
build_config=BuildConfig(
|
||||
name=BuildNames.PACKAGE_AARCH64_ASAN,
|
||||
name=BuildNames.PACKAGE_ARM_ASAN,
|
||||
compiler="clang-18-aarch64",
|
||||
sanitizer="address",
|
||||
package_type="deb",
|
||||
@ -283,6 +283,10 @@ class CI:
|
||||
JobNames.STATEFUL_TEST_ASAN: CommonJobConfigs.STATEFUL_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_ASAN]
|
||||
),
|
||||
JobNames.STATEFUL_TEST_ARM_ASAN: CommonJobConfigs.STATEFUL_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_ARM_ASAN],
|
||||
runner_type=Runners.FUNC_TESTER_ARM,
|
||||
),
|
||||
JobNames.STATEFUL_TEST_TSAN: CommonJobConfigs.STATEFUL_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_TSAN]
|
||||
),
|
||||
@ -331,6 +335,11 @@ class CI:
|
||||
JobNames.STATELESS_TEST_ASAN: CommonJobConfigs.STATELESS_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_ASAN], num_batches=2
|
||||
),
|
||||
JobNames.STATELESS_TEST_ARM_ASAN: CommonJobConfigs.STATELESS_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_ARM_ASAN],
|
||||
num_batches=2,
|
||||
runner_type=Runners.FUNC_TESTER_ARM,
|
||||
),
|
||||
JobNames.STATELESS_TEST_TSAN: CommonJobConfigs.STATELESS_TEST.with_properties(
|
||||
required_builds=[BuildNames.PACKAGE_TSAN], num_batches=4
|
||||
),
|
||||
|
||||
@ -106,7 +106,7 @@ class BuildNames(metaclass=WithIter):
|
||||
PACKAGE_MSAN = "package_msan"
|
||||
PACKAGE_DEBUG = "package_debug"
|
||||
PACKAGE_AARCH64 = "package_aarch64"
|
||||
PACKAGE_AARCH64_ASAN = "package_aarch64_asan"
|
||||
PACKAGE_ARM_ASAN = "package_aarch64_asan"
|
||||
PACKAGE_RELEASE_COVERAGE = "package_release_coverage"
|
||||
BINARY_RELEASE = "binary_release"
|
||||
BINARY_TIDY = "binary_tidy"
|
||||
@ -141,6 +141,7 @@ class JobNames(metaclass=WithIter):
|
||||
STATELESS_TEST_RELEASE_COVERAGE = "Stateless tests (coverage)"
|
||||
STATELESS_TEST_AARCH64 = "Stateless tests (aarch64)"
|
||||
STATELESS_TEST_ASAN = "Stateless tests (asan)"
|
||||
STATELESS_TEST_ARM_ASAN = "Stateless tests (aarch64, asan)"
|
||||
STATELESS_TEST_TSAN = "Stateless tests (tsan)"
|
||||
STATELESS_TEST_MSAN = "Stateless tests (msan)"
|
||||
STATELESS_TEST_UBSAN = "Stateless tests (ubsan)"
|
||||
@ -157,6 +158,7 @@ class JobNames(metaclass=WithIter):
|
||||
STATEFUL_TEST_RELEASE_COVERAGE = "Stateful tests (coverage)"
|
||||
STATEFUL_TEST_AARCH64 = "Stateful tests (aarch64)"
|
||||
STATEFUL_TEST_ASAN = "Stateful tests (asan)"
|
||||
STATEFUL_TEST_ARM_ASAN = "Stateful tests (aarch64, asan)"
|
||||
STATEFUL_TEST_TSAN = "Stateful tests (tsan)"
|
||||
STATEFUL_TEST_MSAN = "Stateful tests (msan)"
|
||||
STATEFUL_TEST_UBSAN = "Stateful tests (ubsan)"
|
||||
@ -241,7 +243,7 @@ class StatusNames(metaclass=WithIter):
|
||||
# mergeable status
|
||||
MERGEABLE = "Mergeable Check"
|
||||
# status of a sync pr
|
||||
SYNC = "Cloud fork sync (only for ClickHouse Inc. employees)"
|
||||
SYNC = "CH Inc sync"
|
||||
# PR formatting check status
|
||||
PR_CHECK = "PR Check"
|
||||
|
||||
@ -632,6 +634,8 @@ REQUIRED_CHECKS = [
|
||||
JobNames.STATEFUL_TEST_RELEASE,
|
||||
JobNames.STATELESS_TEST_RELEASE,
|
||||
JobNames.STATELESS_TEST_ASAN,
|
||||
JobNames.STATELESS_TEST_ARM_ASAN,
|
||||
JobNames.STATEFUL_TEST_ARM_ASAN,
|
||||
JobNames.STATELESS_TEST_FLAKY_ASAN,
|
||||
JobNames.STATEFUL_TEST_ASAN,
|
||||
JobNames.STYLE_CHECK,
|
||||
|
||||
@ -36,7 +36,7 @@ class TestCIConfig(unittest.TestCase):
|
||||
elif "binary_" in job.lower() or "package_" in job.lower():
|
||||
if job.lower() in (
|
||||
CI.BuildNames.PACKAGE_AARCH64,
|
||||
CI.BuildNames.PACKAGE_AARCH64_ASAN,
|
||||
CI.BuildNames.PACKAGE_ARM_ASAN,
|
||||
):
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].runner_type in (CI.Runners.BUILDER_ARM,),
|
||||
@ -95,69 +95,39 @@ class TestCIConfig(unittest.TestCase):
|
||||
self.assertTrue(CI.JOB_CONFIGS[job].required_builds is None)
|
||||
else:
|
||||
self.assertTrue(CI.JOB_CONFIGS[job].build_config is None)
|
||||
if "asan" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_ASAN,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
if "asan" in job and "aarch" in job:
|
||||
expected_builds = [CI.BuildNames.PACKAGE_ARM_ASAN]
|
||||
elif "asan" in job:
|
||||
expected_builds = [CI.BuildNames.PACKAGE_ASAN]
|
||||
elif "msan" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_MSAN,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_MSAN]
|
||||
elif "tsan" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_TSAN,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_TSAN]
|
||||
elif "ubsan" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_UBSAN,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_UBSAN]
|
||||
elif "debug" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_DEBUG,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_DEBUG]
|
||||
elif job in (
|
||||
"Unit tests (release)",
|
||||
"ClickHouse Keeper Jepsen",
|
||||
"ClickHouse Server Jepsen",
|
||||
):
|
||||
expected_builds = [CI.BuildNames.BINARY_RELEASE]
|
||||
elif "release" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
in (
|
||||
CI.BuildNames.PACKAGE_RELEASE,
|
||||
CI.BuildNames.BINARY_RELEASE,
|
||||
),
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_RELEASE]
|
||||
elif "coverage" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_RELEASE_COVERAGE,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_RELEASE_COVERAGE]
|
||||
elif "aarch" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_AARCH64,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_AARCH64]
|
||||
elif "amd64" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_RELEASE,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_RELEASE]
|
||||
elif "uzzer" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0] == CI.BuildNames.FUZZERS,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.FUZZERS]
|
||||
elif "Docker" in job:
|
||||
expected_builds = [
|
||||
CI.BuildNames.PACKAGE_RELEASE,
|
||||
CI.BuildNames.PACKAGE_AARCH64,
|
||||
]
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
in (
|
||||
@ -167,20 +137,12 @@ class TestCIConfig(unittest.TestCase):
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
elif "SQLTest" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
== CI.BuildNames.PACKAGE_RELEASE,
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [CI.BuildNames.PACKAGE_RELEASE]
|
||||
elif "Jepsen" in job:
|
||||
self.assertTrue(
|
||||
CI.JOB_CONFIGS[job].required_builds[0]
|
||||
in (
|
||||
CI.BuildNames.PACKAGE_RELEASE,
|
||||
CI.BuildNames.BINARY_RELEASE,
|
||||
),
|
||||
f"Job [{job}] probably has wrong required build [{CI.JOB_CONFIGS[job].required_builds[0]}] in JobConfig",
|
||||
)
|
||||
expected_builds = [
|
||||
CI.BuildNames.PACKAGE_RELEASE,
|
||||
CI.BuildNames.BINARY_RELEASE,
|
||||
]
|
||||
elif job in (
|
||||
CI.JobNames.STYLE_CHECK,
|
||||
CI.JobNames.FAST_TEST,
|
||||
@ -188,9 +150,16 @@ class TestCIConfig(unittest.TestCase):
|
||||
CI.JobNames.DOCS_CHECK,
|
||||
CI.JobNames.BUGFIX_VALIDATE,
|
||||
):
|
||||
self.assertTrue(CI.JOB_CONFIGS[job].required_builds is None)
|
||||
expected_builds = []
|
||||
else:
|
||||
print(f"Job [{job}] required build not checked")
|
||||
assert False
|
||||
|
||||
self.assertCountEqual(
|
||||
expected_builds,
|
||||
CI.JOB_CONFIGS[job].required_builds or [],
|
||||
f"Required builds are not valid for job [{job}]",
|
||||
)
|
||||
|
||||
def test_job_stage_config(self):
|
||||
"""
|
||||
|
||||
@ -263,8 +263,12 @@ function check_logs_for_critical_errors()
|
||||
# Remove file logical_errors.txt if it's empty
|
||||
[ -s /test_output/logical_errors.txt ] || rm /test_output/logical_errors.txt
|
||||
|
||||
# No such key errors (ignore a.myext which is used in 02724_database_s3.sh and does not exist)
|
||||
rg --text "Code: 499.*The specified key does not exist" /var/log/clickhouse-server/clickhouse-server*.log | grep -v "a.myext" > /test_output/no_such_key_errors.txt \
|
||||
# ignore:
|
||||
# - a.myext which is used in 02724_database_s3.sh and does not exist
|
||||
# - "DistributedCacheTCPHandler" and "caller id: None:DistribCache" because they happen inside distributed cache server
|
||||
# - "ReadBufferFromDistributedCache", "AsynchronousBoundedReadBuffer", "ReadBufferFromS3", "ReadBufferFromAzureBlobStorage"
|
||||
# exceptions printed internally by a buffer, exception will be rethrown and handled correctly
|
||||
rg --text "Code: 499.*The specified key does not exist" /var/log/clickhouse-server/clickhouse-server*.log | grep -v -e "a.myext" -e "DistributedCacheTCPHandler" -e "ReadBufferFromDistributedCache" -e "ReadBufferFromS3" -e "ReadBufferFromAzureBlobStorage" -e "AsynchronousBoundedReadBuffer" -e "caller id: None:DistribCache" > /test_output/no_such_key_errors.txt \
|
||||
&& echo -e "S3_ERROR No such key thrown (see clickhouse-server.log or no_such_key_errors.txt)$FAIL$(trim_server_logs no_such_key_errors.txt)" >> /test_output/test_results.tsv \
|
||||
|| echo -e "No lost s3 keys$OK" >> /test_output/test_results.tsv
|
||||
|
||||
|
||||
@ -78,7 +78,7 @@ def test_distributed_background_insert_split_batch_on_failure_OFF(started_cluste
|
||||
with pytest.raises(
|
||||
QueryRuntimeException,
|
||||
# no DOTALL in pytest.raises, use '(.|\n)'
|
||||
match=r"DB::Exception: Received from.*Memory limit \(for query\) exceeded: (.|\n)*While sending a batch",
|
||||
match=r"DB::Exception: Received from.*Query memory limit exceeded: (.|\n)*While sending a batch",
|
||||
):
|
||||
node2.query("system flush distributed dist")
|
||||
assert int(node2.query("select count() from dist_data")) == 0
|
||||
|
||||
@ -9,6 +9,7 @@ def fill_nodes(nodes, shard):
|
||||
for node in nodes:
|
||||
node.query(
|
||||
"""
|
||||
DROP DATABASE IF EXISTS test SYNC;
|
||||
CREATE DATABASE test;
|
||||
|
||||
CREATE TABLE test.test_table(date Date, id UInt32)
|
||||
@ -21,6 +22,7 @@ def fill_nodes(nodes, shard):
|
||||
|
||||
node.query(
|
||||
"""
|
||||
DROP DATABASE IF EXISTS test1 SYNC;
|
||||
CREATE DATABASE test1;
|
||||
|
||||
CREATE TABLE test1.test_table(date Date, id UInt32)
|
||||
@ -33,6 +35,7 @@ def fill_nodes(nodes, shard):
|
||||
|
||||
node.query(
|
||||
"""
|
||||
DROP DATABASE IF EXISTS test2 SYNC;
|
||||
CREATE DATABASE test2;
|
||||
|
||||
CREATE TABLE test2.test_table(date Date, id UInt32)
|
||||
@ -45,7 +48,8 @@ def fill_nodes(nodes, shard):
|
||||
|
||||
node.query(
|
||||
"""
|
||||
CREATE DATABASE test3;
|
||||
DROP DATABASE IF EXISTS test3 SYNC;
|
||||
CREATE DATABASE test3;
|
||||
|
||||
CREATE TABLE test3.test_table(date Date, id UInt32)
|
||||
ENGINE = ReplicatedMergeTree('/clickhouse/tables/test3/{shard}/replicated/test_table', '{replica}') ORDER BY id PARTITION BY toYYYYMM(date)
|
||||
@ -57,6 +61,7 @@ def fill_nodes(nodes, shard):
|
||||
|
||||
node.query(
|
||||
"""
|
||||
DROP DATABASE IF EXISTS test4 SYNC;
|
||||
CREATE DATABASE test4;
|
||||
|
||||
CREATE TABLE test4.test_table(date Date, id UInt32)
|
||||
@ -84,9 +89,6 @@ node_1_3 = cluster.add_instance(
|
||||
def start_cluster():
|
||||
try:
|
||||
cluster.start()
|
||||
|
||||
fill_nodes([node_1_1, node_1_2], 1)
|
||||
|
||||
yield cluster
|
||||
|
||||
except Exception as ex:
|
||||
@ -102,6 +104,8 @@ def check_exists(zk, path):
|
||||
|
||||
|
||||
def test_drop_replica(start_cluster):
|
||||
fill_nodes([node_1_1, node_1_2], 1)
|
||||
|
||||
node_1_1.query(
|
||||
"INSERT INTO test.test_table SELECT number, toString(number) FROM numbers(100)"
|
||||
)
|
||||
@ -142,11 +146,7 @@ def test_drop_replica(start_cluster):
|
||||
shard=1
|
||||
)
|
||||
)
|
||||
assert "There is a local table" in node_1_2.query_and_get_error(
|
||||
"SYSTEM DROP REPLICA 'node_1_1' FROM ZKPATH '/clickhouse/tables/test/{shard}/replicated/test_table'".format(
|
||||
shard=1
|
||||
)
|
||||
)
|
||||
|
||||
assert "There is a local table" in node_1_1.query_and_get_error(
|
||||
"SYSTEM DROP REPLICA 'node_1_1' FROM ZKPATH '/clickhouse/tables/test/{shard}/replicated/test_table'".format(
|
||||
shard=1
|
||||
@ -222,11 +222,22 @@ def test_drop_replica(start_cluster):
|
||||
)
|
||||
assert exists_replica_1_1 == None
|
||||
|
||||
node_1_2.query("SYSTEM DROP REPLICA 'node_1_1'")
|
||||
exists_replica_1_1 = check_exists(
|
||||
node_1_1.query("ATTACH DATABASE test4")
|
||||
|
||||
node_1_2.query("DETACH TABLE test4.test_table")
|
||||
node_1_1.query(
|
||||
"SYSTEM DROP REPLICA 'node_1_2' FROM ZKPATH '/clickhouse/tables/test4/{shard}/replicated/test_table'".format(
|
||||
shard=1
|
||||
)
|
||||
)
|
||||
exists_replica_1_2 = check_exists(
|
||||
zk,
|
||||
"/clickhouse/tables/test4/{shard}/replicated/test_table/replicas/{replica}".format(
|
||||
shard=1, replica="node_1_1"
|
||||
shard=1, replica="node_1_2"
|
||||
),
|
||||
)
|
||||
assert exists_replica_1_1 == None
|
||||
assert exists_replica_1_2 == None
|
||||
|
||||
node_1_1.query("ATTACH DATABASE test")
|
||||
for i in range(1, 4):
|
||||
node_1_1.query("ATTACH DATABASE test{}".format(i))
|
||||
|
||||
@ -364,7 +364,7 @@ def test_logs():
|
||||
)
|
||||
assert query in logs
|
||||
assert "Read 1000000 rows" in logs
|
||||
assert "Peak memory usage" in logs
|
||||
assert "Query peak memory usage" in logs
|
||||
|
||||
|
||||
def test_progress():
|
||||
|
||||
@ -68,7 +68,8 @@ def get_memory_usage_from_client_output_and_close(client_output):
|
||||
for line in client_output:
|
||||
print(f"'{line}'\n")
|
||||
if not peek_memory_usage_str_found:
|
||||
peek_memory_usage_str_found = "Peak memory usage" in line
|
||||
# Can be both Peak/peak
|
||||
peek_memory_usage_str_found = "eak memory usage" in line
|
||||
|
||||
if peek_memory_usage_str_found:
|
||||
search_obj = re.search(r"[+-]?[0-9]+\.[0-9]+", line)
|
||||
@ -97,9 +98,7 @@ def test_clickhouse_client_max_peak_memory_usage_distributed(started_cluster):
|
||||
|
||||
peak_memory_usage = get_memory_usage_from_client_output_and_close(client_output)
|
||||
assert peak_memory_usage
|
||||
assert shard_2.contains_in_log(
|
||||
f"Peak memory usage (for query): {peak_memory_usage}"
|
||||
)
|
||||
assert shard_2.contains_in_log(f"Query peak memory usage: {peak_memory_usage}")
|
||||
|
||||
|
||||
def test_clickhouse_client_max_peak_memory_single_node(started_cluster):
|
||||
@ -118,6 +117,4 @@ def test_clickhouse_client_max_peak_memory_single_node(started_cluster):
|
||||
|
||||
peak_memory_usage = get_memory_usage_from_client_output_and_close(client_output)
|
||||
assert peak_memory_usage
|
||||
assert shard_1.contains_in_log(
|
||||
f"Peak memory usage (for query): {peak_memory_usage}"
|
||||
)
|
||||
assert shard_1.contains_in_log(f"Query peak memory usage: {peak_memory_usage}")
|
||||
|
||||
@ -0,0 +1,23 @@
|
||||
<clickhouse>
|
||||
<remote_servers>
|
||||
<test_cluster>
|
||||
<shard>
|
||||
<replica>
|
||||
<host>node1</host>
|
||||
<port>9000</port>
|
||||
</replica>
|
||||
|
||||
<replica>
|
||||
<host>node2</host>
|
||||
<port>9000</port>
|
||||
</replica>
|
||||
|
||||
<replica>
|
||||
<host>node3</host>
|
||||
<port>9000</port>
|
||||
</replica>
|
||||
|
||||
</shard>
|
||||
</test_cluster>
|
||||
</remote_servers>
|
||||
</clickhouse>
|
||||
@ -0,0 +1,20 @@
|
||||
<clickhouse>
|
||||
<zookeeper>
|
||||
<node index="1">
|
||||
<host>zoo1</host>
|
||||
<port>2281</port>
|
||||
<secure>1</secure>
|
||||
</node>
|
||||
<node index="2">
|
||||
<host>zoo2</host>
|
||||
<port>2281</port>
|
||||
<secure>1</secure>
|
||||
</node>
|
||||
<node index="3">
|
||||
<host>zoo3</host>
|
||||
<port>2281</port>
|
||||
<secure>1</secure>
|
||||
</node>
|
||||
<session_timeout_ms>3000</session_timeout_ms>
|
||||
</zookeeper>
|
||||
</clickhouse>
|
||||
@ -0,0 +1,17 @@
|
||||
<clickhouse>
|
||||
<remote_servers>
|
||||
<test_cluster>
|
||||
<shard>
|
||||
<replica>
|
||||
<host>node1</host>
|
||||
<port>9000</port>
|
||||
</replica>
|
||||
|
||||
<replica>
|
||||
<host>node2</host>
|
||||
<port>9000</port>
|
||||
</replica>
|
||||
</shard>
|
||||
</test_cluster>
|
||||
</remote_servers>
|
||||
</clickhouse>
|
||||
@ -0,0 +1,16 @@
|
||||
<clickhouse>
|
||||
<openSSL>
|
||||
<client>
|
||||
<certificateFile>/etc/clickhouse-server/config.d/first_client.crt</certificateFile>
|
||||
<privateKeyFile>/etc/clickhouse-server/config.d/first_client.key</privateKeyFile>
|
||||
<loadDefaultCAFile>true</loadDefaultCAFile>
|
||||
<cacheSessions>true</cacheSessions>
|
||||
<disableProtocols>sslv2,sslv3</disableProtocols>
|
||||
<preferServerCiphers>true</preferServerCiphers>
|
||||
<verificationMode>none</verificationMode>
|
||||
<invalidCertificateHandler>
|
||||
<name>RejectCertificateHandler</name>
|
||||
</invalidCertificateHandler>
|
||||
</client>
|
||||
</openSSL>
|
||||
</clickhouse>
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user