mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-30 19:42:00 +00:00
Merge branch 'master' into rs/qc-totals-extremes2
This commit is contained in:
Alexey Milovidov
GitHub
commit
4dceee0dc3
35
.github/workflows/pull_request.yml
vendored
35
.github/workflows/pull_request.yml
vendored
@ -1308,6 +1308,40 @@ jobs:
|
||||
docker ps --quiet | xargs --no-run-if-empty docker kill ||:
|
||||
docker ps --all --quiet | xargs --no-run-if-empty docker rm -f ||:
|
||||
sudo rm -fr "$TEMP_PATH"
|
||||
FunctionalStatelessTestReleaseAnalyzer:
|
||||
needs: [BuilderDebRelease]
|
||||
runs-on: [self-hosted, func-tester]
|
||||
steps:
|
||||
- name: Set envs
|
||||
run: |
|
||||
cat >> "$GITHUB_ENV" << 'EOF'
|
||||
TEMP_PATH=${{runner.temp}}/stateless_analyzer
|
||||
REPORTS_PATH=${{runner.temp}}/reports_dir
|
||||
CHECK_NAME=Stateless tests (release, analyzer)
|
||||
REPO_COPY=${{runner.temp}}/stateless_analyzer/ClickHouse
|
||||
KILL_TIMEOUT=10800
|
||||
EOF
|
||||
- name: Download json reports
|
||||
uses: actions/download-artifact@v3
|
||||
with:
|
||||
path: ${{ env.REPORTS_PATH }}
|
||||
- name: Check out repository code
|
||||
uses: ClickHouse/checkout@v1
|
||||
with:
|
||||
clear-repository: true
|
||||
- name: Functional test
|
||||
run: |
|
||||
sudo rm -fr "$TEMP_PATH"
|
||||
mkdir -p "$TEMP_PATH"
|
||||
cp -r "$GITHUB_WORKSPACE" "$TEMP_PATH"
|
||||
cd "$REPO_COPY/tests/ci"
|
||||
python3 functional_test_check.py "$CHECK_NAME" "$KILL_TIMEOUT"
|
||||
- name: Cleanup
|
||||
if: always()
|
||||
run: |
|
||||
docker ps --quiet | xargs --no-run-if-empty docker kill ||:
|
||||
docker ps --all --quiet | xargs --no-run-if-empty docker rm -f ||:
|
||||
sudo rm -fr "$TEMP_PATH"
|
||||
FunctionalStatelessTestReleaseS3_0:
|
||||
needs: [BuilderDebRelease]
|
||||
runs-on: [self-hosted, func-tester]
|
||||
@ -4755,6 +4789,7 @@ jobs:
|
||||
- FunctionalStatelessTestReleaseDatabaseReplicated2
|
||||
- FunctionalStatelessTestReleaseDatabaseReplicated3
|
||||
- FunctionalStatelessTestReleaseWideParts
|
||||
- FunctionalStatelessTestReleaseAnalyzer
|
||||
- FunctionalStatelessTestAarch64
|
||||
- FunctionalStatelessTestAsan0
|
||||
- FunctionalStatelessTestAsan1
|
||||
|
||||
150
CHANGELOG.md
150
CHANGELOG.md
@ -1,4 +1,5 @@
|
||||
### Table of Contents
|
||||
**[ClickHouse release v23.4, 2023-04-26](#234)**<br/>
|
||||
**[ClickHouse release v23.3 LTS, 2023-03-30](#233)**<br/>
|
||||
**[ClickHouse release v23.2, 2023-02-23](#232)**<br/>
|
||||
**[ClickHouse release v23.1, 2023-01-25](#231)**<br/>
|
||||
@ -6,6 +7,155 @@
|
||||
|
||||
# 2023 Changelog
|
||||
|
||||
### <a id="234"></a> ClickHouse release 23.4 LTS, 2023-04-26
|
||||
|
||||
#### Backward Incompatible Change
|
||||
* Formatter '%M' in function formatDateTime() now prints the month name instead of the minutes. This makes the behavior consistent with MySQL. The previous behavior can be restored using setting "formatdatetime_parsedatetime_m_is_month_name = 0". [#47246](https://github.com/ClickHouse/ClickHouse/pull/47246) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* This change makes sense only if you are using the virtual filesystem cache. If `path` in the virtual filesystem cache configuration is not empty and is not an absolute path, then it will be put in `<clickhouse server data directory>/caches/<path_from_cache_config>`. [#48784](https://github.com/ClickHouse/ClickHouse/pull/48784) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Primary/secondary indices and sorting keys with identical expressions are now rejected. This behavior can be disabled using setting `allow_suspicious_indices`. [#48536](https://github.com/ClickHouse/ClickHouse/pull/48536) ([凌涛](https://github.com/lingtaolf)).
|
||||
|
||||
#### New Feature
|
||||
* Support new aggregate function `quantileGK`/`quantilesGK`, like [approx_percentile](https://spark.apache.org/docs/latest/api/sql/index.html#approx_percentile) in spark. Greenwald-Khanna algorithm refer to http://infolab.stanford.edu/~datar/courses/cs361a/papers/quantiles.pdf. [#46428](https://github.com/ClickHouse/ClickHouse/pull/46428) ([李扬](https://github.com/taiyang-li)).
|
||||
* Add a statement `SHOW COLUMNS` which shows distilled information from system.columns. [#48017](https://github.com/ClickHouse/ClickHouse/pull/48017) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Added `LIGHTWEIGHT` and `PULL` modifiers for `SYSTEM SYNC REPLICA` query. `LIGHTWEIGHT` version waits for fetches and drop-ranges only (merges and mutations are ignored). `PULL` version pulls new entries from ZooKeeper and does not wait for them. Fixes [#47794](https://github.com/ClickHouse/ClickHouse/issues/47794). [#48085](https://github.com/ClickHouse/ClickHouse/pull/48085) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* Add `kafkaMurmurHash` function for compatibility with Kafka DefaultPartitioner. Closes [#47834](https://github.com/ClickHouse/ClickHouse/issues/47834). [#48185](https://github.com/ClickHouse/ClickHouse/pull/48185) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Allow to easily create a user with the same grants as the current user by using `GRANT CURRENT GRANTS`. [#48262](https://github.com/ClickHouse/ClickHouse/pull/48262) ([pufit](https://github.com/pufit)).
|
||||
* Add statistical aggregate function `kolmogorovSmirnovTest`. Close [#48228](https://github.com/ClickHouse/ClickHouse/issues/48228). [#48325](https://github.com/ClickHouse/ClickHouse/pull/48325) ([FFFFFFFHHHHHHH](https://github.com/FFFFFFFHHHHHHH)).
|
||||
* Added a `lost_part_count` column to the `system.replicas` table. The column value shows the total number of lost parts in the corresponding table. Value is stored in zookeeper and can be used instead of not persistent `ReplicatedDataLoss` profile event for monitoring. [#48526](https://github.com/ClickHouse/ClickHouse/pull/48526) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Add `soundex` function for compatibility. Closes [#39880](https://github.com/ClickHouse/ClickHouse/issues/39880). [#48567](https://github.com/ClickHouse/ClickHouse/pull/48567) ([FriendLey](https://github.com/FriendLey)).
|
||||
* Support `Map` type for JSONExtract. [#48629](https://github.com/ClickHouse/ClickHouse/pull/48629) ([李扬](https://github.com/taiyang-li)).
|
||||
* Add `PrettyJSONEachRow` format to output pretty JSON with new line delimieters and 4 space indents. [#48898](https://github.com/ClickHouse/ClickHouse/pull/48898) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add `ParquetMetadata` input format to read Parquet file metadata. [#48911](https://github.com/ClickHouse/ClickHouse/pull/48911) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add `extractKeyValuePairs` function to extract key value pairs from strings. Input strings might contain noise (i.e log files / do not need to be 100% formatted in key-value-pair format), the algorithm will look for key value pairs matching the arguments passed to the function. As of now, function accepts the following arguments: `data_column` (mandatory), `key_value_pair_delimiter` (defaults to `:`), `pair_delimiters` (defaults to `\space \, \;`) and `quoting_character` (defaults to double quotes). [#43606](https://github.com/ClickHouse/ClickHouse/pull/43606) ([Arthur Passos](https://github.com/arthurpassos)).
|
||||
* Functions replaceOne(), replaceAll(), replaceRegexpOne() and replaceRegexpAll() can now be called with non-const pattern and replacement arguments. [#46589](https://github.com/ClickHouse/ClickHouse/pull/46589) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Added functions to work with columns of type `Map`: `mapConcat`, `mapSort`, `mapExists`. [#48071](https://github.com/ClickHouse/ClickHouse/pull/48071) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
|
||||
#### Performance Improvement
|
||||
* Reading files in `Parquet` format is now much faster. IO and decoding are parallelized (controlled by `max_threads` setting), and only required data ranges are read. [#47964](https://github.com/ClickHouse/ClickHouse/pull/47964) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* If we run a mutation with IN (subquery) like this: `ALTER TABLE t UPDATE col='new value' WHERE id IN (SELECT id FROM huge_table)` and the table `t` has multiple parts than for each part a set for subquery `SELECT id FROM huge_table` is built in memory. And if there are many parts then this might consume a lot of memory (and lead to an OOM) and CPU. The solution is to introduce a short-lived cache of sets that are currently being built by mutation tasks. If another task of the same mutation is executed concurrently it can lookup the set in the cache, wait for it to be built and reuse it. [#46835](https://github.com/ClickHouse/ClickHouse/pull/46835) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Only check dependencies if necessary when applying `ALTER TABLE` queries. [#48062](https://github.com/ClickHouse/ClickHouse/pull/48062) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Optimize function `mapUpdate`. [#48118](https://github.com/ClickHouse/ClickHouse/pull/48118) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Now an internal query to local replica is sent explicitly and data from it received through loopback interface. Setting `prefer_localhost_replica` is not respected for parallel replicas. This is needed for better scheduling and makes the code cleaner: the initiator is only responsible for coordinating of the reading process and merging results, continiously answering for requests while all the secondary queries read the data. Note: Using loopback interface is not so performant, otherwise some replicas could starve for tasks which could lead to even slower query execution and not utilizing all possible resources. The initialization of the coordinator is now even more lazy. All incoming requests contain the information about the reading algorithm we initialize the coordinator with it when first request comes. If any replica will decide to read with different algorithm - an exception will be thrown and a query will be aborted. [#48246](https://github.com/ClickHouse/ClickHouse/pull/48246) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Do not build set for the right side of `IN` clause with subquery when it is used only for analysis of skip indexes and they are disabled by setting (`use_skip_indexes=0`). Previously it might affect the performance of queries. [#48299](https://github.com/ClickHouse/ClickHouse/pull/48299) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Query processing is parallelized right after reading `FROM file(...)`. Related to [#38755](https://github.com/ClickHouse/ClickHouse/issues/38755). [#48525](https://github.com/ClickHouse/ClickHouse/pull/48525) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Query processing is parallelized right after reading from a data source. Affected data sources are mostly simple or external storages like table functions `url`, `file`. [#48727](https://github.com/ClickHouse/ClickHouse/pull/48727) ([Igor Nikonov](https://github.com/devcrafter)).
|
||||
* Lowered contention of ThreadPool mutex (may increase performance for a huge amount of small jobs). [#48750](https://github.com/ClickHouse/ClickHouse/pull/48750) ([Sergei Trifonov](https://github.com/serxa)).
|
||||
* Reduce memory usage for multiple `ALTER DELETE` mutations. [#48522](https://github.com/ClickHouse/ClickHouse/pull/48522) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).
|
||||
* Remove the excessive connection attempts if the `skip_unavailable_shards` setting is enabled. [#48771](https://github.com/ClickHouse/ClickHouse/pull/48771) ([Azat Khuzhin](https://github.com/azat)).
|
||||
|
||||
#### Experimental Feature
|
||||
* Entries in the query cache are now squashed to max_block_size and compressed. [#45912](https://github.com/ClickHouse/ClickHouse/pull/45912) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* It is now possible to define per-user quotas in the query cache. [#48284](https://github.com/ClickHouse/ClickHouse/pull/48284) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Some fixes for parallel replicas [#48433](https://github.com/ClickHouse/ClickHouse/pull/48433) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Implement zero-copy-replication (an experimental feature) on encrypted disks. [#48741](https://github.com/ClickHouse/ClickHouse/pull/48741) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
|
||||
#### Improvement
|
||||
* Increase default value for `connect_timeout_with_failover_ms` to 1000 ms (because of adding async connections in https://github.com/ClickHouse/ClickHouse/pull/47229) . Closes [#5188](https://github.com/ClickHouse/ClickHouse/issues/5188). [#49009](https://github.com/ClickHouse/ClickHouse/pull/49009) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Several improvements around data lakes: - Make `Iceberg` work with non-partitioned data. - Support `Iceberg` format version v2 (previously only v1 was supported) - Support reading partitioned data for `DeltaLake`/`Hudi` - Faster reading of `DeltaLake` metadata by using Delta's checkpoint files - Fixed incorrect `Hudi` reads: previously it incorrectly chose which data to read and therefore was able to read correctly only small size tables - Made these engines to pickup updates of changed data (previously the state was set on table creation) - Make proper testing for `Iceberg`/`DeltaLake`/`Hudi` using spark. [#47307](https://github.com/ClickHouse/ClickHouse/pull/47307) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Add async connection to socket and async writing to socket. Make creating connections and sending query/external tables async across shards. Refactor code with fibers. Closes [#46931](https://github.com/ClickHouse/ClickHouse/issues/46931). We will be able to increase `connect_timeout_with_failover_ms` by default after this PR (https://github.com/ClickHouse/ClickHouse/issues/5188). [#47229](https://github.com/ClickHouse/ClickHouse/pull/47229) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support config sections `keeper`/`keeper_server` as an alternative to `zookeeper`. Close [#34766](https://github.com/ClickHouse/ClickHouse/issues/34766) , [#34767](https://github.com/ClickHouse/ClickHouse/issues/34767). [#35113](https://github.com/ClickHouse/ClickHouse/pull/35113) ([李扬](https://github.com/taiyang-li)).
|
||||
* It is possible to set _secure_ flag in named_collections for a dictionary with a ClickHouse table source. Addresses [#38450](https://github.com/ClickHouse/ClickHouse/issues/38450) . [#46323](https://github.com/ClickHouse/ClickHouse/pull/46323) ([Ilya Golshtein](https://github.com/ilejn)).
|
||||
* `bitCount` function support `FixedString` and `String` data type. [#49044](https://github.com/ClickHouse/ClickHouse/pull/49044) ([flynn](https://github.com/ucasfl)).
|
||||
* Added configurable retries for all operations with [Zoo]Keeper for Backup queries. [#47224](https://github.com/ClickHouse/ClickHouse/pull/47224) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Enable `use_environment_credentials` for S3 by default, so the entire provider chain is constructed by default. [#47397](https://github.com/ClickHouse/ClickHouse/pull/47397) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Currently, the JSON_VALUE function is similar as spark's get_json_object function, which support to get value from json string by a path like '$.key'. But still has something different - 1. in spark's get_json_object will return null while the path is not exist, but in JSON_VALUE will return empty string; - 2. in spark's get_json_object will return a complext type value, such as a json object/array value, but in JSON_VALUE will return empty string. [#47494](https://github.com/ClickHouse/ClickHouse/pull/47494) ([KevinyhZou](https://github.com/KevinyhZou)).
|
||||
* For `use_structure_from_insertion_table_in_table_functions` more flexible insert table structure propagation to table function. Fixed an issue with name mapping and using virtual columns. No more need for 'auto' setting. [#47962](https://github.com/ClickHouse/ClickHouse/pull/47962) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
|
||||
* Do not continue retrying to connect to ZK if the query is killed or over limits. [#47985](https://github.com/ClickHouse/ClickHouse/pull/47985) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Support Enum output/input in `BSONEachRow`, allow all map key types and avoid extra calculations on output. [#48122](https://github.com/ClickHouse/ClickHouse/pull/48122) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support more ClickHouse types in `ORC`/`Arrow`/`Parquet` formats: Enum(8|16), (U)Int(128|256), Decimal256 (for ORC), allow reading IPv4 from Int32 values (ORC outputs IPv4 as Int32 and we couldn't read it back), fix reading Nullable(IPv6) from binary data for `ORC`. [#48126](https://github.com/ClickHouse/ClickHouse/pull/48126) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Add columns `perform_ttl_move_on_insert`, `load_balancing` for table `system.storage_policies`, modify column `volume_type` type to `Enum8`. [#48167](https://github.com/ClickHouse/ClickHouse/pull/48167) ([lizhuoyu5](https://github.com/lzydmxy)).
|
||||
* Added support for `BACKUP ALL` command which backups all tables and databases, including temporary and system ones. [#48189](https://github.com/ClickHouse/ClickHouse/pull/48189) ([Vitaly Baranov](https://github.com/vitlibar)).
|
||||
* Function mapFromArrays supports `Map` type as an input. [#48207](https://github.com/ClickHouse/ClickHouse/pull/48207) ([李扬](https://github.com/taiyang-li)).
|
||||
* The output of some SHOW PROCESSLIST is now sorted. [#48241](https://github.com/ClickHouse/ClickHouse/pull/48241) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Per-query/per-server throttling for remote IO/local IO/BACKUPs (server settings: `max_remote_read_network_bandwidth_for_server`, `max_remote_write_network_bandwidth_for_server`, `max_local_read_bandwidth_for_server`, `max_local_write_bandwidth_for_server`, `max_backup_bandwidth_for_server`, settings: `max_remote_read_network_bandwidth`, `max_remote_write_network_bandwidth`, `max_local_read_bandwidth`, `max_local_write_bandwidth`, `max_backup_bandwidth`). [#48242](https://github.com/ClickHouse/ClickHouse/pull/48242) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Support more types in `CapnProto` format: Map, (U)Int(128|256), Decimal(128|256). Allow integer conversions during input/output. [#48257](https://github.com/ClickHouse/ClickHouse/pull/48257) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Don't throw CURRENT_WRITE_BUFFER_IS_EXHAUSTED for normal behaviour. [#48288](https://github.com/ClickHouse/ClickHouse/pull/48288) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Add new setting `keeper_map_strict_mode` which enforces extra guarantees on operations made on top of `KeeperMap` tables. [#48293](https://github.com/ClickHouse/ClickHouse/pull/48293) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Check primary key type for simple dictionary is native unsigned integer type Add setting `check_dictionary_primary_key ` for compatibility(set `check_dictionary_primary_key =false` to disable checking). [#48335](https://github.com/ClickHouse/ClickHouse/pull/48335) ([lizhuoyu5](https://github.com/lzydmxy)).
|
||||
* Don't replicate mutations for `KeeperMap` because it's unnecessary. [#48354](https://github.com/ClickHouse/ClickHouse/pull/48354) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Allow write/read unnamed tuple as nested Message in Protobuf format. Tuple elements and Message fields are mathced by position. [#48390](https://github.com/ClickHouse/ClickHouse/pull/48390) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Support `additional_table_filters` and `additional_result_filter` settings in the new planner. Also, add a documentation entry for `additional_result_filter`. [#48405](https://github.com/ClickHouse/ClickHouse/pull/48405) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* `parseDateTime` now understands format string '%f' (fractional seconds). [#48420](https://github.com/ClickHouse/ClickHouse/pull/48420) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Format string "%f" in formatDateTime() now prints "000000" if the formatted value has no fractional seconds, the previous behavior (single zero) can be restored using setting "formatdatetime_f_prints_single_zero = 1". [#48422](https://github.com/ClickHouse/ClickHouse/pull/48422) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Don't replicate DELETE and TRUNCATE for KeeperMap. [#48434](https://github.com/ClickHouse/ClickHouse/pull/48434) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Generate valid Decimals and Bools in generateRandom function. [#48436](https://github.com/ClickHouse/ClickHouse/pull/48436) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Allow trailing commas in expression list of SELECT query, for example `SELECT a, b, c, FROM table`. Closes [#37802](https://github.com/ClickHouse/ClickHouse/issues/37802). [#48438](https://github.com/ClickHouse/ClickHouse/pull/48438) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Override `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` environment variables with `--user` and `--password` client parameters. Closes [#38909](https://github.com/ClickHouse/ClickHouse/issues/38909). [#48440](https://github.com/ClickHouse/ClickHouse/pull/48440) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Added retries to loading of data parts in `MergeTree` tables in case of retryable errors. [#48442](https://github.com/ClickHouse/ClickHouse/pull/48442) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Add support for `Date`, `Date32`, `DateTime`, `DateTime64` data types to `arrayMin`, `arrayMax`, `arrayDifference` functions. Closes [#21645](https://github.com/ClickHouse/ClickHouse/issues/21645). [#48445](https://github.com/ClickHouse/ClickHouse/pull/48445) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Add support for `{server_uuid}` macro. It is useful for identifying replicas in autoscaled clusters when new replicas are constantly added and removed in runtime. This closes [#48554](https://github.com/ClickHouse/ClickHouse/issues/48554). [#48563](https://github.com/ClickHouse/ClickHouse/pull/48563) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* The installation script will create a hard link instead of copying if it is possible. [#48578](https://github.com/ClickHouse/ClickHouse/pull/48578) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Support `SHOW TABLE` syntax meaning the same as `SHOW CREATE TABLE`. Closes [#48580](https://github.com/ClickHouse/ClickHouse/issues/48580). [#48591](https://github.com/ClickHouse/ClickHouse/pull/48591) ([flynn](https://github.com/ucasfl)).
|
||||
* HTTP temporary buffers now support working by evicting data from the virtual filesystem cache. [#48664](https://github.com/ClickHouse/ClickHouse/pull/48664) ([Vladimir C](https://github.com/vdimir)).
|
||||
* Make Schema inference works for `CREATE AS SELECT`. Closes [#47599](https://github.com/ClickHouse/ClickHouse/issues/47599). [#48679](https://github.com/ClickHouse/ClickHouse/pull/48679) ([flynn](https://github.com/ucasfl)).
|
||||
* Added a `replicated_max_mutations_in_one_entry` setting for `ReplicatedMergeTree` that allows limiting the number of mutation commands per one `MUTATE_PART` entry (default is 10000). [#48731](https://github.com/ClickHouse/ClickHouse/pull/48731) ([Alexander Tokmakov](https://github.com/tavplubix)).
|
||||
* In AggregateFunction types, don't count unused arena bytes as `read_bytes`. [#48745](https://github.com/ClickHouse/ClickHouse/pull/48745) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Fix some MySQL-related settings not being handled with the MySQL dictionary source + named collection. Closes [#48402](https://github.com/ClickHouse/ClickHouse/issues/48402). [#48759](https://github.com/ClickHouse/ClickHouse/pull/48759) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* If a user set `max_single_part_upload_size` to a very large value, it can lead to a crash due to a bug in the AWS S3 SDK. This fixes [#47679](https://github.com/ClickHouse/ClickHouse/issues/47679). [#48816](https://github.com/ClickHouse/ClickHouse/pull/48816) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix data race in `RabbitMQ` ([report](https://pastila.nl/?004f7100/de1505289ab5bb355e67ebe6c7cc8707)), refactor the code. [#48845](https://github.com/ClickHouse/ClickHouse/pull/48845) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Add aliases `name` and `part_name` form `system.parts` and `system.part_log`. Closes [#48718](https://github.com/ClickHouse/ClickHouse/issues/48718). [#48850](https://github.com/ClickHouse/ClickHouse/pull/48850) ([sichenzhao](https://github.com/sichenzhao)).
|
||||
* Functions "arrayDifferenceSupport()", "arrayCumSum()" and "arrayCumSumNonNegative()" now support input arrays of wide integer types (U)Int128/256. [#48866](https://github.com/ClickHouse/ClickHouse/pull/48866) ([cluster](https://github.com/infdahai)).
|
||||
* Multi-line history in clickhouse-client is now no longer padded. This makes pasting more natural. [#48870](https://github.com/ClickHouse/ClickHouse/pull/48870) ([Joanna Hulboj](https://github.com/jh0x)).
|
||||
* Implement a slight improvement for the rare case when ClickHouse is run inside LXC and LXCFS is used. The LXCFS has an issue: sometimes it returns an error "Transport endpoint is not connected" on reading from the file inside `/proc`. This error was correctly logged into ClickHouse's server log. We have additionally workaround this issue by reopening a file. This is a minuscule change. [#48922](https://github.com/ClickHouse/ClickHouse/pull/48922) ([Real](https://github.com/RunningXie)).
|
||||
* Improve memory accounting for prefetches. Randomise prefetch settings In CI. [#48973](https://github.com/ClickHouse/ClickHouse/pull/48973) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Correctly set headers for native copy operations on GCS. [#48981](https://github.com/ClickHouse/ClickHouse/pull/48981) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Add support for specifying setting names in the command line with dashes instead of underscores, for example, `--max-threads` instead of `--max_threads`. Additionally, support Unicode dash characters like `—` instead of `--` - this is useful when you communicate with a team in another company, and a manager from that team copy-pasted code from MS Word. [#48985](https://github.com/ClickHouse/ClickHouse/pull/48985) ([alekseygolub](https://github.com/alekseygolub)).
|
||||
* Add fallback to password authentication when authentication with SSL user certificate has failed. Closes [#48974](https://github.com/ClickHouse/ClickHouse/issues/48974). [#48989](https://github.com/ClickHouse/ClickHouse/pull/48989) ([Nikolay Degterinsky](https://github.com/evillique)).
|
||||
* Improve the embedded dashboard. Close [#46671](https://github.com/ClickHouse/ClickHouse/issues/46671). [#49036](https://github.com/ClickHouse/ClickHouse/pull/49036) ([Kevin Zhang](https://github.com/Kinzeng)).
|
||||
* Add profile events for log messages, so you can easily see the count of log messages by severity. [#49042](https://github.com/ClickHouse/ClickHouse/pull/49042) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* In previous versions, the `LineAsString` format worked inconsistently when the parallel parsing was enabled or not, in presence of DOS or MacOS Classic line breaks. This closes [#49039](https://github.com/ClickHouse/ClickHouse/issues/49039). [#49052](https://github.com/ClickHouse/ClickHouse/pull/49052) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* The exception message about the unparsed query parameter will also tell about the name of the parameter. Reimplement [#48878](https://github.com/ClickHouse/ClickHouse/issues/48878). Close [#48772](https://github.com/ClickHouse/ClickHouse/issues/48772). [#49061](https://github.com/ClickHouse/ClickHouse/pull/49061) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
* Update time zones. The following were updated: Africa/Cairo, Africa/Casablanca, Africa/El_Aaiun, America/Bogota, America/Cambridge_Bay, America/Ciudad_Juarez, America/Godthab, America/Inuvik, America/Iqaluit, America/Nuuk, America/Ojinaga, America/Pangnirtung, America/Rankin_Inlet, America/Resolute, America/Whitehorse, America/Yellowknife, Asia/Gaza, Asia/Hebron, Asia/Kuala_Lumpur, Asia/Singapore, Canada/Yukon, Egypt, Europe/Kirov, Europe/Volgograd, Singapore. [#48572](https://github.com/ClickHouse/ClickHouse/pull/48572) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Reduce the number of dependencies in the header files to speed up the build. [#47984](https://github.com/ClickHouse/ClickHouse/pull/47984) ([Dmitry Novik](https://github.com/novikd)).
|
||||
* Randomize compression of marks and indices in tests. [#48286](https://github.com/ClickHouse/ClickHouse/pull/48286) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Bump internal ZSTD from 1.5.4 to 1.5.5. [#46797](https://github.com/ClickHouse/ClickHouse/pull/46797) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Randomize vertical merges from compact to wide parts in tests. [#48287](https://github.com/ClickHouse/ClickHouse/pull/48287) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Support for CRC32 checksum in HDFS. Fix performance issues. [#48614](https://github.com/ClickHouse/ClickHouse/pull/48614) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Remove remainders of GCC support. [#48671](https://github.com/ClickHouse/ClickHouse/pull/48671) ([Robert Schulze](https://github.com/rschu1ze)).
|
||||
* Add CI run with new analyzer infrastructure enabled. [#48719](https://github.com/ClickHouse/ClickHouse/pull/48719) ([Dmitry Novik](https://github.com/novikd)).
|
||||
|
||||
#### Bug Fix (user-visible misbehavior in an official stable release)
|
||||
|
||||
* Fix system.query_views_log for MVs that are pushed from background threads [#46668](https://github.com/ClickHouse/ClickHouse/pull/46668) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix several `RENAME COLUMN` bugs [#46946](https://github.com/ClickHouse/ClickHouse/pull/46946) ([alesapin](https://github.com/alesapin)).
|
||||
* Fix minor hiliting issues in clickhouse-format [#47610](https://github.com/ClickHouse/ClickHouse/pull/47610) ([Natasha Murashkina](https://github.com/murfel)).
|
||||
* Fix a bug in LLVM's libc++ leading to a crash for uploading parts to S3 which size is greater then INT_MAX [#47693](https://github.com/ClickHouse/ClickHouse/pull/47693) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix overflow in the `sparkbar` function [#48121](https://github.com/ClickHouse/ClickHouse/pull/48121) ([Vladimir C](https://github.com/vdimir)).
|
||||
* Fix race in S3 [#48190](https://github.com/ClickHouse/ClickHouse/pull/48190) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Disable JIT for aggregate functions due to inconsistent behavior [#48195](https://github.com/ClickHouse/ClickHouse/pull/48195) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix alter formatting (minor) [#48289](https://github.com/ClickHouse/ClickHouse/pull/48289) ([Natasha Murashkina](https://github.com/murfel)).
|
||||
* Fix cpu usage in RabbitMQ (was worsened in 23.2 after [#44404](https://github.com/ClickHouse/ClickHouse/issues/44404)) [#48311](https://github.com/ClickHouse/ClickHouse/pull/48311) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix crash in EXPLAIN PIPELINE for Merge over Distributed [#48320](https://github.com/ClickHouse/ClickHouse/pull/48320) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix serializing LowCardinality as Arrow dictionary [#48361](https://github.com/ClickHouse/ClickHouse/pull/48361) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Reset downloader for cache file segment in TemporaryFileStream [#48386](https://github.com/ClickHouse/ClickHouse/pull/48386) ([Vladimir C](https://github.com/vdimir)).
|
||||
* Fix possible SYSTEM SYNC REPLICA stuck in case of DROP/REPLACE PARTITION [#48391](https://github.com/ClickHouse/ClickHouse/pull/48391) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix a startup error when loading a distributed table that depends on a dictionary [#48419](https://github.com/ClickHouse/ClickHouse/pull/48419) ([MikhailBurdukov](https://github.com/MikhailBurdukov)).
|
||||
* Don't check dependencies when renaming system tables automatically [#48431](https://github.com/ClickHouse/ClickHouse/pull/48431) ([Raúl Marín](https://github.com/Algunenano)).
|
||||
* Update only affected rows in KeeperMap storage [#48435](https://github.com/ClickHouse/ClickHouse/pull/48435) ([Antonio Andelic](https://github.com/antonio2368)).
|
||||
* Fix possible segfault in the VFS cache [#48469](https://github.com/ClickHouse/ClickHouse/pull/48469) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* `toTimeZone` function throws an error when no constant string is provided [#48471](https://github.com/ClickHouse/ClickHouse/pull/48471) ([Jordi Villar](https://github.com/jrdi)).
|
||||
* Fix logical error with IPv4 in Protobuf, add support for Date32 [#48486](https://github.com/ClickHouse/ClickHouse/pull/48486) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* "changed" flag in system.settings was calculated incorrectly for settings with multiple values [#48516](https://github.com/ClickHouse/ClickHouse/pull/48516) ([MikhailBurdukov](https://github.com/MikhailBurdukov)).
|
||||
* Fix storage `Memory` with enabled compression [#48517](https://github.com/ClickHouse/ClickHouse/pull/48517) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix bracketed-paste mode messing up password input in the event of client reconnection [#48528](https://github.com/ClickHouse/ClickHouse/pull/48528) ([Michael Kolupaev](https://github.com/al13n321)).
|
||||
* Fix nested map for keys of IP and UUID types [#48556](https://github.com/ClickHouse/ClickHouse/pull/48556) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
|
||||
* Fix an uncaught exception in case of parallel loader for hashed dictionaries [#48571](https://github.com/ClickHouse/ClickHouse/pull/48571) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* The `groupArray` aggregate function correctly works for empty result over nullable types [#48593](https://github.com/ClickHouse/ClickHouse/pull/48593) ([lgbo](https://github.com/lgbo-ustc)).
|
||||
* Fix bug in Keeper when a node is not created with scheme `auth` in ACL sometimes. [#48595](https://github.com/ClickHouse/ClickHouse/pull/48595) ([Aleksei Filatov](https://github.com/aalexfvk)).
|
||||
* Allow IPv4 comparison operators with UInt [#48611](https://github.com/ClickHouse/ClickHouse/pull/48611) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
|
||||
* Fix possible error from cache [#48636](https://github.com/ClickHouse/ClickHouse/pull/48636) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Async inserts with empty data will no longer throw exception. [#48663](https://github.com/ClickHouse/ClickHouse/pull/48663) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix table dependencies in case of failed RENAME TABLE [#48683](https://github.com/ClickHouse/ClickHouse/pull/48683) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* If the primary key has duplicate columns (which is only possible for projections), in previous versions it might lead to a bug [#48838](https://github.com/ClickHouse/ClickHouse/pull/48838) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix for a race condition in ZooKeeper when joining send_thread/receive_thread [#48849](https://github.com/ClickHouse/ClickHouse/pull/48849) ([Alexander Gololobov](https://github.com/davenger)).
|
||||
* Fix unexpected part name error when trying to drop a ignored detached part with zero copy replication [#48862](https://github.com/ClickHouse/ClickHouse/pull/48862) ([Michael Lex](https://github.com/mlex)).
|
||||
* Fix reading `Date32` Parquet/Arrow column into not a `Date32` column [#48864](https://github.com/ClickHouse/ClickHouse/pull/48864) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix `UNKNOWN_IDENTIFIER` error while selecting from table with row policy and column with dots [#48976](https://github.com/ClickHouse/ClickHouse/pull/48976) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Fix aggregation by empty nullable strings [#48999](https://github.com/ClickHouse/ClickHouse/pull/48999) ([LiuNeng](https://github.com/liuneng1994)).
|
||||
|
||||
|
||||
### <a id="233"></a> ClickHouse release 23.3 LTS, 2023-03-30
|
||||
|
||||
#### Upgrade Notes
|
||||
|
||||
@ -421,8 +421,11 @@ endif ()
|
||||
|
||||
set (CMAKE_POSTFIX_VARIABLE "CMAKE_${CMAKE_BUILD_TYPE_UC}_POSTFIX")
|
||||
|
||||
set (CMAKE_POSITION_INDEPENDENT_CODE OFF)
|
||||
if (OS_LINUX AND NOT (ARCH_AARCH64 OR ARCH_S390X))
|
||||
if (NOT SANITIZE)
|
||||
set (CMAKE_POSITION_INDEPENDENT_CODE OFF)
|
||||
endif()
|
||||
|
||||
if (OS_LINUX AND NOT (ARCH_AARCH64 OR ARCH_S390X) AND NOT SANITIZE)
|
||||
# Slightly more efficient code can be generated
|
||||
# It's disabled for ARM because otherwise ClickHouse cannot run on Android.
|
||||
set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -fno-pie")
|
||||
|
||||
@ -235,6 +235,17 @@ ssize_t getrandom(void *buf, size_t buflen, unsigned flags)
|
||||
return syscall(SYS_getrandom, buf, buflen, flags);
|
||||
}
|
||||
|
||||
/* Structure for scatter/gather I/O. */
|
||||
struct iovec
|
||||
{
|
||||
void *iov_base; /* Pointer to data. */
|
||||
size_t iov_len; /* Length of data. */
|
||||
};
|
||||
|
||||
ssize_t preadv(int __fd, const struct iovec *__iovec, int __count, __off_t __offset)
|
||||
{
|
||||
return syscall(SYS_preadv, __fd, __iovec, __count, (long)(__offset), (long)(__offset>>32));
|
||||
}
|
||||
|
||||
#include <errno.h>
|
||||
#include <limits.h>
|
||||
|
||||
@ -33,8 +33,7 @@ if (SANITIZE)
|
||||
# RelWithDebInfo, and downgrade optimizations to -O1 but not to -Og, to
|
||||
# keep the binary size down.
|
||||
# TODO: try compiling with -Og and with ld.gold.
|
||||
set (MSAN_FLAGS "-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-track-origins -fno-optimize-sibling-calls -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt")
|
||||
|
||||
set (MSAN_FLAGS "-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-track-origins -fno-optimize-sibling-calls -fPIC -fpie -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt")
|
||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SAN_FLAGS} ${MSAN_FLAGS}")
|
||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SAN_FLAGS} ${MSAN_FLAGS}")
|
||||
|
||||
|
||||
2
contrib/sysroot
vendored
2
contrib/sysroot
vendored
@ -1 +1 @@
|
||||
Subproject commit f0081b2649b94837855f3bc7d05ef326b100bad8
|
||||
Subproject commit e0d1b64da666afbfaa6f1ee0487c33f3fd2cd5cb
|
||||
@ -18,7 +18,7 @@ SUCCESS_FINISH_SIGNS = ["All tests have finished", "No tests were run"]
|
||||
RETRIES_SIGN = "Some tests were restarted"
|
||||
|
||||
|
||||
def process_test_log(log_path):
|
||||
def process_test_log(log_path, broken_tests):
|
||||
total = 0
|
||||
skipped = 0
|
||||
unknown = 0
|
||||
@ -62,8 +62,12 @@ def process_test_log(log_path):
|
||||
failed += 1
|
||||
test_results.append((test_name, "Timeout", test_time, []))
|
||||
elif FAIL_SIGN in line:
|
||||
failed += 1
|
||||
test_results.append((test_name, "FAIL", test_time, []))
|
||||
if test_name in broken_tests:
|
||||
success += 1
|
||||
test_results.append((test_name, "OK", test_time, []))
|
||||
else:
|
||||
failed += 1
|
||||
test_results.append((test_name, "FAIL", test_time, []))
|
||||
elif UNKNOWN_SIGN in line:
|
||||
unknown += 1
|
||||
test_results.append((test_name, "FAIL", test_time, []))
|
||||
@ -71,8 +75,21 @@ def process_test_log(log_path):
|
||||
skipped += 1
|
||||
test_results.append((test_name, "SKIPPED", test_time, []))
|
||||
else:
|
||||
success += int(OK_SIGN in line)
|
||||
test_results.append((test_name, "OK", test_time, []))
|
||||
if OK_SIGN in line and test_name in broken_tests:
|
||||
failed += 1
|
||||
test_results.append(
|
||||
(
|
||||
test_name,
|
||||
"FAIL",
|
||||
test_time,
|
||||
[
|
||||
"Test is expected to fail! Please, update broken_tests.txt!\n"
|
||||
],
|
||||
)
|
||||
)
|
||||
else:
|
||||
success += int(OK_SIGN in line)
|
||||
test_results.append((test_name, "OK", test_time, []))
|
||||
test_end = False
|
||||
elif (
|
||||
len(test_results) > 0 and test_results[-1][1] == "FAIL" and not test_end
|
||||
@ -110,7 +127,7 @@ def process_test_log(log_path):
|
||||
)
|
||||
|
||||
|
||||
def process_result(result_path):
|
||||
def process_result(result_path, broken_tests):
|
||||
test_results = []
|
||||
state = "success"
|
||||
description = ""
|
||||
@ -134,7 +151,7 @@ def process_result(result_path):
|
||||
success_finish,
|
||||
retries,
|
||||
test_results,
|
||||
) = process_test_log(result_path)
|
||||
) = process_test_log(result_path, broken_tests)
|
||||
is_flacky_check = 1 < int(os.environ.get("NUM_TRIES", 1))
|
||||
logging.info("Is flaky check: %s", is_flacky_check)
|
||||
# If no tests were run (success == 0) it indicates an error (e.g. server did not start or crashed immediately)
|
||||
@ -186,9 +203,17 @@ if __name__ == "__main__":
|
||||
parser.add_argument("--in-results-dir", default="/test_output/")
|
||||
parser.add_argument("--out-results-file", default="/test_output/test_results.tsv")
|
||||
parser.add_argument("--out-status-file", default="/test_output/check_status.tsv")
|
||||
parser.add_argument("--broken-tests", default="/broken_tests.txt")
|
||||
args = parser.parse_args()
|
||||
|
||||
state, description, test_results = process_result(args.in_results_dir)
|
||||
broken_tests = list()
|
||||
if os.path.exists(args.broken_tests):
|
||||
logging.info(f"File {args.broken_tests} with broken tests found")
|
||||
with open(args.broken_tests) as f:
|
||||
broken_tests = f.read().splitlines()
|
||||
logging.info(f"Broken tests in the list: {len(broken_tests)}")
|
||||
|
||||
state, description, test_results = process_result(args.in_results_dir, broken_tests)
|
||||
logging.info("Result parsed")
|
||||
status = (state, description)
|
||||
write_results(args.out_results_file, args.out_status_file, test_results, status)
|
||||
|
||||
@ -13,9 +13,11 @@ Supported platforms:
|
||||
- AArch64
|
||||
- Power9 (experimental)
|
||||
|
||||
## Normal Build for Development on Ubuntu
|
||||
## Building on Ubuntu
|
||||
|
||||
The following tutorial is based on the Ubuntu Linux system. With appropriate changes, it should also work on any other Linux distribution.
|
||||
The following tutorial is based on Ubuntu Linux.
|
||||
With appropriate changes, it should also work on any other Linux distribution.
|
||||
The minimum recommended Ubuntu version for development is 22.04 LTS.
|
||||
|
||||
### Install Prerequisites {#install-prerequisites}
|
||||
|
||||
@ -23,13 +25,11 @@ The following tutorial is based on the Ubuntu Linux system. With appropriate cha
|
||||
sudo apt-get install git cmake ccache python3 ninja-build yasm gawk
|
||||
```
|
||||
|
||||
Or cmake3 instead of cmake on older systems.
|
||||
### Install and Use the Clang compiler
|
||||
|
||||
### Install the latest clang (recommended)
|
||||
On Ubuntu/Debian you can use LLVM's automatic installation script, see [here](https://apt.llvm.org/).
|
||||
|
||||
On Ubuntu/Debian you can use the automatic installation script (check [official webpage](https://apt.llvm.org/))
|
||||
|
||||
```bash
|
||||
``` bash
|
||||
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
|
||||
```
|
||||
|
||||
@ -40,19 +40,17 @@ sudo apt-get install software-properties-common
|
||||
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
|
||||
```
|
||||
|
||||
For other Linux distribution - check the availability of the [prebuild packages](https://releases.llvm.org/download.html) or build clang [from sources](https://clang.llvm.org/get_started.html).
|
||||
For other Linux distribution - check the availability of LLVM's [prebuild packages](https://releases.llvm.org/download.html).
|
||||
|
||||
#### Use the latest clang for Builds
|
||||
As of April 2023, any version of Clang >= 15 will work.
|
||||
GCC as a compiler is not supported
|
||||
To build with a specific Clang version:
|
||||
|
||||
``` bash
|
||||
export CC=clang-15
|
||||
export CXX=clang++-15
|
||||
```
|
||||
|
||||
In this example we use version 15 that is the latest as of Sept 2022.
|
||||
|
||||
Gcc cannot be used.
|
||||
|
||||
### Checkout ClickHouse Sources {#checkout-clickhouse-sources}
|
||||
|
||||
``` bash
|
||||
@ -70,79 +68,46 @@ git clone --recursive --shallow-submodules https://github.com/ClickHouse/ClickHo
|
||||
``` bash
|
||||
cd ClickHouse
|
||||
mkdir build
|
||||
cd build
|
||||
cmake ..
|
||||
ninja
|
||||
cmake -S . -B build
|
||||
cmake --build build # or: `cd build; ninja`
|
||||
```
|
||||
|
||||
To create an executable, run `ninja clickhouse`.
|
||||
This will create the `programs/clickhouse` executable, which can be used with `client` or `server` arguments.
|
||||
To create an executable, run `cmake --build --target clickhouse` (or: `cd build; ninja clickhouse`).
|
||||
This will create executable `build/programs/clickhouse` which can be used with `client` or `server` arguments.
|
||||
|
||||
## How to Build ClickHouse on Any Linux {#how-to-build-clickhouse-on-any-linux}
|
||||
## Building on Any Linux {#how-to-build-clickhouse-on-any-linux}
|
||||
|
||||
The build requires the following components:
|
||||
|
||||
- Git (is used only to checkout the sources, it’s not needed for the build)
|
||||
- CMake 3.15 or newer
|
||||
- Git (used to checkout the sources, not needed for the build)
|
||||
- CMake 3.20 or newer
|
||||
- Compiler: Clang 15 or newer
|
||||
- Linker: lld 15 or newer
|
||||
- Ninja
|
||||
- C++ compiler: clang-15 or newer
|
||||
- Linker: lld
|
||||
- Yasm
|
||||
- Gawk
|
||||
|
||||
If all the components are installed, you may build in the same way as the steps above.
|
||||
|
||||
Example for Ubuntu Eoan:
|
||||
``` bash
|
||||
sudo apt update
|
||||
sudo apt install git cmake ninja-build clang++ python yasm gawk
|
||||
git clone --recursive https://github.com/ClickHouse/ClickHouse.git
|
||||
mkdir build && cd build
|
||||
cmake ../ClickHouse
|
||||
ninja
|
||||
```
|
||||
|
||||
Example for OpenSUSE Tumbleweed:
|
||||
|
||||
``` bash
|
||||
sudo zypper install git cmake ninja clang-c++ python lld yasm gawk
|
||||
git clone --recursive https://github.com/ClickHouse/ClickHouse.git
|
||||
mkdir build && cd build
|
||||
cmake ../ClickHouse
|
||||
ninja
|
||||
mkdir build
|
||||
cmake -S . -B build
|
||||
cmake --build build
|
||||
```
|
||||
|
||||
Example for Fedora Rawhide:
|
||||
|
||||
``` bash
|
||||
sudo yum update
|
||||
sudo yum --nogpg install git cmake make clang python3 ccache yasm gawk
|
||||
git clone --recursive https://github.com/ClickHouse/ClickHouse.git
|
||||
mkdir build && cd build
|
||||
cmake ../ClickHouse
|
||||
make -j $(nproc)
|
||||
```
|
||||
|
||||
Here is an example of how to build `clang` and all the llvm infrastructure from sources:
|
||||
|
||||
```
|
||||
git clone git@github.com:llvm/llvm-project.git
|
||||
mkdir llvm-build && cd llvm-build
|
||||
cmake -DCMAKE_BUILD_TYPE:STRING=Release -DLLVM_ENABLE_PROJECTS=all ../llvm-project/llvm/
|
||||
make -j16
|
||||
sudo make install
|
||||
hash clang
|
||||
clang --version
|
||||
```
|
||||
|
||||
You can install the older clang like clang-11 from packages and then use it to build the new clang from sources.
|
||||
|
||||
Here is an example of how to install the new `cmake` from the official website:
|
||||
|
||||
```
|
||||
wget https://github.com/Kitware/CMake/releases/download/v3.22.2/cmake-3.22.2-linux-x86_64.sh
|
||||
chmod +x cmake-3.22.2-linux-x86_64.sh
|
||||
./cmake-3.22.2-linux-x86_64.sh
|
||||
export PATH=/home/milovidov/work/cmake-3.22.2-linux-x86_64/bin/:${PATH}
|
||||
hash cmake
|
||||
mkdir build
|
||||
cmake -S . -B build
|
||||
cmake --build build
|
||||
```
|
||||
|
||||
## You Don’t Have to Build ClickHouse {#you-dont-have-to-build-clickhouse}
|
||||
|
||||
@ -40,7 +40,7 @@ If the user tries to violate the constraints an exception is thrown and the sett
|
||||
There are supported few types of constraints: `min`, `max`, `readonly` (with alias `const`) and `changeable_in_readonly`. The `min` and `max` constraints specify upper and lower boundaries for a numeric setting and can be used in combination. The `readonly` or `const` constraint specifies that the user cannot change the corresponding setting at all. The `changeable_in_readonly` constraint type allows user to change the setting within `min`/`max` range even if `readonly` setting is set to 1, otherwise settings are not allow to be changed in `readonly=1` mode. Note that `changeable_in_readonly` is supported only if `settings_constraints_replace_previous` is enabled:
|
||||
``` xml
|
||||
<access_control_improvements>
|
||||
<settings_constraints_replace_previous>true<settings_constraints_replace_previous>
|
||||
<settings_constraints_replace_previous>true</settings_constraints_replace_previous>
|
||||
</access_control_improvements>
|
||||
```
|
||||
|
||||
|
||||
@ -890,7 +890,7 @@ Write time that processor spent during execution/waiting for data to `system.pro
|
||||
|
||||
See also:
|
||||
|
||||
- [`system.processors_profile_log`](../../operations/system-tables/processors_profile_log.md#system-processors_profile_log)
|
||||
- [`system.processors_profile_log`](../../operations/system-tables/processors_profile_log.md)
|
||||
- [`EXPLAIN PIPELINE`](../../sql-reference/statements/explain.md#explain-pipeline)
|
||||
|
||||
## max_insert_block_size {#settings-max_insert_block_size}
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# system.processors_profile_log {#system-processors_profile_log}
|
||||
# processors_profile_log
|
||||
|
||||
This table contains profiling on processors level (that you can find in [`EXPLAIN PIPELINE`](../../sql-reference/statements/explain.md#explain-pipeline)).
|
||||
|
||||
@ -73,4 +73,4 @@ Here you can see:
|
||||
|
||||
**See Also**
|
||||
|
||||
- [`EXPLAIN PIPELINE`](../../sql-reference/statements/explain.md#explain-pipeline)
|
||||
- [`EXPLAIN PIPELINE`](../../sql-reference/statements/explain.md#explain-pipeline)
|
||||
|
||||
@ -36,6 +36,18 @@ GRANT [ON CLUSTER cluster_name] role [,...] TO {user | another_role | CURRENT_US
|
||||
The `WITH ADMIN OPTION` clause grants [ADMIN OPTION](#admin-option-privilege) privilege to `user` or `role`.
|
||||
The `WITH REPLACE OPTION` clause replace old roles by new role for the `user` or `role`, if is not specified it appends roles.
|
||||
|
||||
## Grant Current Grants Syntax
|
||||
``` sql
|
||||
GRANT CURRENT GRANTS{(privilege[(column_name [,...])] [,...] ON {db.table|db.*|*.*|table|*}) | ON {db.table|db.*|*.*|table|*}} TO {user | role | CURRENT_USER} [,...] [WITH GRANT OPTION] [WITH REPLACE OPTION]
|
||||
```
|
||||
|
||||
- `privilege` — Type of privilege.
|
||||
- `role` — ClickHouse user role.
|
||||
- `user` — ClickHouse user account.
|
||||
|
||||
Using the `CURRENT GRANTS` statement allows you to give all specified privileges to the given user or role.
|
||||
If none of the privileges were specified, then the given user or role will receive all available privileges for `CURRENT_USER`.
|
||||
|
||||
## Usage
|
||||
|
||||
To use `GRANT`, your account must have the `GRANT OPTION` privilege. You can grant privileges only inside the scope of your account privileges.
|
||||
|
||||
@ -37,6 +37,19 @@ GRANT [ON CLUSTER cluster_name] role [,...] TO {user | another_role | CURRENT_US
|
||||
`WITH ADMIN OPTION` присваивает привилегию [ADMIN OPTION](#admin-option-privilege) пользователю или роли.
|
||||
`WITH REPLACE OPTION` заменяет все старые роли новыми ролями для пользователя `user` или `role`, если не указано, добавляет новые новые роли.
|
||||
|
||||
## Синтаксис присвоения текущих привилегий {#grant-current-grants-syntax}
|
||||
|
||||
```sql
|
||||
GRANT CURRENT GRANTS{(privilege[(column_name [,...])] [,...] ON {db.table|db.*|*.*|table|*}) | ON {db.table|db.*|*.*|table|*}} TO {user | role | CURRENT_USER} [,...] [WITH GRANT OPTION] [WITH REPLACE OPTION]
|
||||
```
|
||||
|
||||
- `privilege` — Тип привилегии
|
||||
- `role` — Роль пользователя ClickHouse.
|
||||
- `user` — Пользователь ClickHouse.
|
||||
|
||||
Использование выражения `CURRENT GRANTS` позволяет присвоить все указанные и доступные для присвоения привилегии.
|
||||
Если список привелегий не задан, то указанный пользователь или роль получат все доступные привилегии для `CURRENT_USER`.
|
||||
|
||||
## Использование {#grant-usage}
|
||||
|
||||
Для использования `GRANT` пользователь должен иметь привилегию `GRANT OPTION`. Пользователь может выдавать привилегии только внутри области действий назначенных ему самому привилегий.
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
#include <Common/HashTable/HashSet.h>
|
||||
#include <Common/HashTable/HashMap.h>
|
||||
#include <Common/SipHash.h>
|
||||
#include <IO/ReadHelpersArena.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/ReadHelpersArena.h>
|
||||
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
|
||||
@ -2,7 +2,6 @@
|
||||
|
||||
#include <base/sort.h>
|
||||
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/NaNUtils.h>
|
||||
|
||||
#include <Columns/ColumnVector.h>
|
||||
@ -29,6 +28,7 @@
|
||||
namespace DB
|
||||
{
|
||||
struct Settings;
|
||||
class Arena;
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
|
||||
@ -6,7 +6,6 @@
|
||||
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
|
||||
#include <Common/ArenaAllocator.h>
|
||||
#include <Common/assert_cast.h>
|
||||
#include <base/arithmeticOverflow.h>
|
||||
#include <base/sort.h>
|
||||
|
||||
@ -5,7 +5,6 @@
|
||||
#include <Columns/ColumnTuple.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/assert_cast.h>
|
||||
#include <Common/ArenaAllocator.h>
|
||||
#include <Common/PODArray_fwd.h>
|
||||
#include <base/types.h>
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
|
||||
@ -6,7 +6,6 @@
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnTuple.h>
|
||||
#include <Common/assert_cast.h>
|
||||
#include <Common/ArenaAllocator.h>
|
||||
#include <Common/PODArray_fwd.h>
|
||||
#include <base/types.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
|
||||
@ -14,8 +14,6 @@
|
||||
#include <DataTypes/DataTypeTuple.h>
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
|
||||
#include <Common/ArenaAllocator.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
struct Settings;
|
||||
|
||||
@ -8,7 +8,6 @@
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <Common/ArenaAllocator.h>
|
||||
#include <base/range.h>
|
||||
#include <bitset>
|
||||
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/ReadHelpersArena.h>
|
||||
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
|
||||
@ -6,7 +6,6 @@
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <Common/ArenaAllocator.h>
|
||||
#include <Common/assert_cast.h>
|
||||
|
||||
#include <AggregateFunctions/AggregateFunctionNull.h>
|
||||
|
||||
@ -385,8 +385,7 @@ void ColumnAggregateFunction::updateHashFast(SipHash & hash) const
|
||||
/// threads, so we can't know the size of these data.
|
||||
size_t ColumnAggregateFunction::byteSize() const
|
||||
{
|
||||
return data.size() * sizeof(data[0])
|
||||
+ (my_arena ? my_arena->size() : 0);
|
||||
return data.size() * sizeof(data[0]) + (my_arena ? my_arena->usedBytes() : 0);
|
||||
}
|
||||
|
||||
size_t ColumnAggregateFunction::byteSizeAt(size_t) const
|
||||
@ -395,11 +394,11 @@ size_t ColumnAggregateFunction::byteSizeAt(size_t) const
|
||||
return sizeof(data[0]) + func->sizeOfData();
|
||||
}
|
||||
|
||||
/// Like in byteSize(), the size is underestimated.
|
||||
/// Similar to byteSize() the size is underestimated.
|
||||
/// In this case it's also overestimated at the same time as it counts all the bytes allocated by the arena, used or not
|
||||
size_t ColumnAggregateFunction::allocatedBytes() const
|
||||
{
|
||||
return data.allocated_bytes()
|
||||
+ (my_arena ? my_arena->size() : 0);
|
||||
return data.allocated_bytes() + (my_arena ? my_arena->allocatedBytes() : 0);
|
||||
}
|
||||
|
||||
void ColumnAggregateFunction::protect()
|
||||
|
||||
@ -80,7 +80,8 @@ private:
|
||||
|
||||
/// Last contiguous MemoryChunk of memory.
|
||||
MemoryChunk * head;
|
||||
size_t size_in_bytes;
|
||||
size_t allocated_bytes;
|
||||
size_t used_bytes;
|
||||
size_t page_size;
|
||||
|
||||
static size_t roundUpToPageSize(size_t s, size_t page_size)
|
||||
@ -119,7 +120,7 @@ private:
|
||||
void NO_INLINE addMemoryChunk(size_t min_size)
|
||||
{
|
||||
head = new MemoryChunk(nextSize(min_size + pad_right), head);
|
||||
size_in_bytes += head->size();
|
||||
allocated_bytes += head->size();
|
||||
}
|
||||
|
||||
friend class ArenaAllocator;
|
||||
@ -127,9 +128,12 @@ private:
|
||||
|
||||
public:
|
||||
explicit Arena(size_t initial_size_ = 4096, size_t growth_factor_ = 2, size_t linear_growth_threshold_ = 128 * 1024 * 1024)
|
||||
: growth_factor(growth_factor_), linear_growth_threshold(linear_growth_threshold_),
|
||||
head(new MemoryChunk(initial_size_, nullptr)), size_in_bytes(head->size()),
|
||||
page_size(static_cast<size_t>(::getPageSize()))
|
||||
: growth_factor(growth_factor_)
|
||||
, linear_growth_threshold(linear_growth_threshold_)
|
||||
, head(new MemoryChunk(initial_size_, nullptr))
|
||||
, allocated_bytes(head->size())
|
||||
, used_bytes(0)
|
||||
, page_size(static_cast<size_t>(::getPageSize()))
|
||||
{
|
||||
}
|
||||
|
||||
@ -141,6 +145,7 @@ public:
|
||||
/// Get piece of memory, without alignment.
|
||||
char * alloc(size_t size)
|
||||
{
|
||||
used_bytes += size;
|
||||
if (unlikely(static_cast<std::ptrdiff_t>(size) > head->end - head->pos))
|
||||
addMemoryChunk(size);
|
||||
|
||||
@ -153,6 +158,7 @@ public:
|
||||
/// Get piece of memory with alignment
|
||||
char * alignedAlloc(size_t size, size_t alignment)
|
||||
{
|
||||
used_bytes += size;
|
||||
do
|
||||
{
|
||||
void * head_pos = head->pos;
|
||||
@ -184,6 +190,7 @@ public:
|
||||
*/
|
||||
void * rollback(size_t size)
|
||||
{
|
||||
used_bytes -= size;
|
||||
head->pos -= size;
|
||||
ASAN_POISON_MEMORY_REGION(head->pos, size + pad_right);
|
||||
return head->pos;
|
||||
@ -299,11 +306,11 @@ public:
|
||||
return res;
|
||||
}
|
||||
|
||||
/// Size of MemoryChunks in bytes.
|
||||
size_t size() const
|
||||
{
|
||||
return size_in_bytes;
|
||||
}
|
||||
/// Size of all MemoryChunks in bytes.
|
||||
size_t allocatedBytes() const { return allocated_bytes; }
|
||||
|
||||
/// Total space actually used (not counting padding or space unused by caller allocations) in all MemoryChunks in bytes.
|
||||
size_t usedBytes() const { return used_bytes; }
|
||||
|
||||
/// Bad method, don't use it -- the MemoryChunks are not your business, the entire
|
||||
/// purpose of the arena code is to manage them for you, so if you find
|
||||
|
||||
@ -107,10 +107,7 @@ public:
|
||||
}
|
||||
|
||||
/// Size of the allocated pool in bytes
|
||||
size_t size() const
|
||||
{
|
||||
return pool.size();
|
||||
}
|
||||
size_t allocatedBytes() const { return pool.allocatedBytes(); }
|
||||
};
|
||||
|
||||
class SynchronizedArenaWithFreeLists : private ArenaWithFreeLists
|
||||
@ -135,10 +132,10 @@ public:
|
||||
}
|
||||
|
||||
/// Size of the allocated pool in bytes
|
||||
size_t size() const
|
||||
size_t allocatedBytes() const
|
||||
{
|
||||
std::lock_guard lock{mutex};
|
||||
return ArenaWithFreeLists::size();
|
||||
return ArenaWithFreeLists::allocatedBytes();
|
||||
}
|
||||
private:
|
||||
mutable std::mutex mutex;
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#include <base/defines.h>
|
||||
#include <base/StringRef.h>
|
||||
#include <Common/HashTable/StringHashMap.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Poco/Util/AbstractConfiguration.h>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
@ -11,6 +10,7 @@
|
||||
|
||||
namespace DB
|
||||
{
|
||||
class Arena;
|
||||
|
||||
enum TLDType
|
||||
{
|
||||
|
||||
@ -270,7 +270,7 @@ int main(int argc, char ** argv)
|
||||
|
||||
watch.stop();
|
||||
std::cerr
|
||||
<< "Insert info arena. Bytes: " << arena.size()
|
||||
<< "Insert info arena. Bytes: " << arena.allocatedBytes()
|
||||
<< ", elapsed: " << watch.elapsedSeconds()
|
||||
<< " (" << data.size() / watch.elapsedSeconds() << " elem/sec.,"
|
||||
<< " " << sum_strings_size / 1048576.0 / watch.elapsedSeconds() << " MiB/sec.)"
|
||||
@ -298,7 +298,7 @@ int main(int argc, char ** argv)
|
||||
|
||||
watch.stop();
|
||||
std::cerr
|
||||
<< "Randomly remove and insert elements. Bytes: " << arena.size()
|
||||
<< "Randomly remove and insert elements. Bytes: " << arena.allocatedBytes()
|
||||
<< ", elapsed: " << watch.elapsedSeconds()
|
||||
<< " (" << data.size() / watch.elapsedSeconds() << " elem/sec.,"
|
||||
<< " " << bytes / 1048576.0 / watch.elapsedSeconds() << " MiB/sec.)"
|
||||
@ -331,7 +331,7 @@ int main(int argc, char ** argv)
|
||||
|
||||
watch.stop();
|
||||
std::cerr
|
||||
<< "Filling cache. Bytes: " << arena.size()
|
||||
<< "Filling cache. Bytes: " << arena.allocatedBytes()
|
||||
<< ", elapsed: " << watch.elapsedSeconds()
|
||||
<< " (" << data.size() / watch.elapsedSeconds() << " elem/sec.,"
|
||||
<< " " << bytes / 1048576.0 / watch.elapsedSeconds() << " MiB/sec.)"
|
||||
|
||||
@ -333,10 +333,7 @@ public:
|
||||
}
|
||||
}
|
||||
|
||||
uint64_t keyArenaSize() const
|
||||
{
|
||||
return arena.size();
|
||||
}
|
||||
uint64_t keyArenaSize() const { return arena.allocatedBytes(); }
|
||||
|
||||
iterator begin() { return list.begin(); }
|
||||
const_iterator begin() const { return list.cbegin(); }
|
||||
|
||||
@ -129,6 +129,7 @@ class IColumn;
|
||||
\
|
||||
M(Bool, allow_suspicious_low_cardinality_types, false, "In CREATE TABLE statement allows specifying LowCardinality modifier for types of small fixed size (8 or less). Enabling this may increase merge times and memory consumption.", 0) \
|
||||
M(Bool, allow_suspicious_fixed_string_types, false, "In CREATE TABLE statement allows creating columns of type FixedString(n) with n > 256. FixedString with length >= 256 is suspicious and most likely indicates misusage", 0) \
|
||||
M(Bool, allow_suspicious_indices, false, "Reject primary/secondary indexes and sorting keys with identical expressions", 0) \
|
||||
M(Bool, compile_expressions, true, "Compile some scalar functions and operators to native code.", 0) \
|
||||
M(UInt64, min_count_to_compile_expression, 3, "The number of identical expressions before they are JIT-compiled", 0) \
|
||||
M(Bool, compile_aggregate_expressions, false, "Compile aggregate functions to native code. This feature has a bug and should not be used.", 0) \
|
||||
|
||||
@ -80,6 +80,7 @@ namespace SettingsChangesHistory
|
||||
/// It's used to implement `compatibility` setting (see https://github.com/ClickHouse/ClickHouse/issues/35972)
|
||||
static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> settings_changes_history =
|
||||
{
|
||||
{"23.4", {{"allow_suspicious_indices", true, false, "If true, index can defined with identical expressions"}}},
|
||||
{"23.4", {{"connect_timeout_with_failover_ms", 50, 1000, "Increase default connect timeout because of async connect"},
|
||||
{"connect_timeout_with_failover_secure_ms", 100, 1000, "Increase default secure connect timeout because of async connect"},
|
||||
{"hedged_connection_timeout_ms", 100, 50, "Start new connection in hedged requests after 50 ms instead of 100 to correspond with previous connect timeout"}}},
|
||||
|
||||
@ -17,6 +17,7 @@
|
||||
#include <Databases/PostgreSQL/fetchPostgreSQLTableStructure.h>
|
||||
#include <Common/quoteString.h>
|
||||
#include <Common/filesystemHelpers.h>
|
||||
#include <Common/logger_useful.h>
|
||||
#include <filesystem>
|
||||
|
||||
namespace fs = std::filesystem;
|
||||
@ -51,6 +52,7 @@ DatabasePostgreSQL::DatabasePostgreSQL(
|
||||
, configuration(configuration_)
|
||||
, pool(std::move(pool_))
|
||||
, cache_tables(cache_tables_)
|
||||
, log(&Poco::Logger::get("DatabasePostgreSQL(" + dbname_ + ")"))
|
||||
{
|
||||
cleaner_task = getContext()->getSchedulePool().createTask("PostgreSQLCleanerTask", [this]{ removeOutdatedTables(); });
|
||||
cleaner_task->deactivate();
|
||||
@ -192,7 +194,10 @@ StoragePtr DatabasePostgreSQL::fetchTable(const String & table_name, ContextPtr,
|
||||
ColumnsDescription{columns_info->columns}, ConstraintsDescription{}, String{}, configuration.schema, configuration.on_conflict);
|
||||
|
||||
if (cache_tables)
|
||||
{
|
||||
LOG_TEST(log, "Cached table `{}`", table_name);
|
||||
cached_tables[table_name] = storage;
|
||||
}
|
||||
|
||||
return storage;
|
||||
}
|
||||
|
||||
@ -73,6 +73,7 @@ private:
|
||||
mutable Tables cached_tables;
|
||||
std::unordered_set<std::string> detached_or_dropped;
|
||||
BackgroundSchedulePool::TaskHolder cleaner_task;
|
||||
Poco::Logger * log;
|
||||
|
||||
String getTableNameForLogs(const String & table_name) const;
|
||||
|
||||
|
||||
@ -157,7 +157,7 @@ public:
|
||||
});
|
||||
}
|
||||
|
||||
return arena.size() + sizeof(Cell) * configuration.max_size_in_cells + attributes_size_in_bytes;
|
||||
return arena.allocatedBytes() + sizeof(Cell) * configuration.max_size_in_cells + attributes_size_in_bytes;
|
||||
}
|
||||
|
||||
private:
|
||||
|
||||
@ -1,6 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/HashTable/HashMap.h>
|
||||
#include <Columns/IColumn.h>
|
||||
#include <Columns/ColumnDecimal.h>
|
||||
@ -29,6 +28,8 @@ namespace ErrorCodes
|
||||
extern const int BAD_ARGUMENTS;
|
||||
}
|
||||
|
||||
class Arena;
|
||||
|
||||
/** Simple helper for getting default.
|
||||
* Initialized with default value and default values column.
|
||||
* If default values column is not null default value is taken from column.
|
||||
|
||||
@ -505,7 +505,7 @@ void FlatDictionary::calculateBytesAllocated()
|
||||
bytes_allocated += hierarchical_index_bytes_allocated;
|
||||
}
|
||||
|
||||
bytes_allocated += string_arena.size();
|
||||
bytes_allocated += string_arena.allocatedBytes();
|
||||
}
|
||||
|
||||
FlatDictionary::Attribute FlatDictionary::createAttribute(const DictionaryAttribute & dictionary_attribute)
|
||||
|
||||
@ -797,7 +797,7 @@ void HashedArrayDictionary<dictionary_key_type>::calculateBytesAllocated()
|
||||
bytes_allocated += hierarchical_index_bytes_allocated;
|
||||
}
|
||||
|
||||
bytes_allocated += string_arena.size();
|
||||
bytes_allocated += string_arena.allocatedBytes();
|
||||
}
|
||||
|
||||
template <DictionaryKeyType dictionary_key_type>
|
||||
|
||||
@ -1022,7 +1022,7 @@ void HashedDictionary<dictionary_key_type, sparse, sharded>::calculateBytesAlloc
|
||||
}
|
||||
|
||||
for (const auto & arena : string_arenas)
|
||||
bytes_allocated += arena->size();
|
||||
bytes_allocated += arena->allocatedBytes();
|
||||
}
|
||||
|
||||
template <DictionaryKeyType dictionary_key_type, bool sparse, bool sharded>
|
||||
|
||||
@ -541,7 +541,7 @@ template <>
|
||||
void IPAddressDictionary::addAttributeSize<String>(const Attribute & attribute)
|
||||
{
|
||||
addAttributeSize<StringRef>(attribute);
|
||||
bytes_allocated += sizeof(Arena) + attribute.string_arena->size();
|
||||
bytes_allocated += sizeof(Arena) + attribute.string_arena->allocatedBytes();
|
||||
}
|
||||
|
||||
void IPAddressDictionary::calculateBytesAllocated()
|
||||
|
||||
@ -5,7 +5,6 @@
|
||||
#include <variant>
|
||||
#include <Columns/ColumnDecimal.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/HashTable/HashMap.h>
|
||||
#include <Columns/ColumnFixedString.h>
|
||||
#include <Columns/ColumnVector.h>
|
||||
@ -18,6 +17,8 @@
|
||||
|
||||
namespace DB
|
||||
{
|
||||
class Arena;
|
||||
|
||||
class IPAddressDictionary final : public IDictionary
|
||||
{
|
||||
public:
|
||||
|
||||
@ -726,7 +726,7 @@ void RangeHashedDictionary<dictionary_key_type>::calculateBytesAllocated()
|
||||
if (update_field_loaded_block)
|
||||
bytes_allocated += update_field_loaded_block->allocatedBytes();
|
||||
|
||||
bytes_allocated += string_arena.size();
|
||||
bytes_allocated += string_arena.allocatedBytes();
|
||||
}
|
||||
|
||||
template <DictionaryKeyType dictionary_key_type>
|
||||
|
||||
@ -10,7 +10,6 @@
|
||||
|
||||
#include <Columns/IColumn.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/HashTable/Hash.h>
|
||||
#include <Common/HashTable/HashSet.h>
|
||||
|
||||
@ -1,15 +1,18 @@
|
||||
#pragma once
|
||||
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/DataTypesDecimal.h>
|
||||
#include <DataTypes/DataTypeFixedString.h>
|
||||
#include <DataTypes/DataTypeInterval.h>
|
||||
#include <DataTypes/Native.h>
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnDecimal.h>
|
||||
#include <Columns/ColumnFixedString.h>
|
||||
#include <Functions/IFunction.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
#include <Columns/ColumnVector.h>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <DataTypes/DataTypeFixedString.h>
|
||||
#include <DataTypes/DataTypeInterval.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <DataTypes/DataTypesDecimal.h>
|
||||
#include <DataTypes/DataTypesNumber.h>
|
||||
#include <DataTypes/Native.h>
|
||||
#include <Functions/FunctionHelpers.h>
|
||||
#include <Functions/IFunction.h>
|
||||
#include <Functions/IsOperation.h>
|
||||
#include <Functions/castTypeToEither.h>
|
||||
|
||||
@ -30,7 +33,6 @@ namespace ErrorCodes
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
|
||||
template <typename A, typename Op>
|
||||
struct UnaryOperationImpl
|
||||
{
|
||||
@ -130,6 +132,47 @@ struct FixedStringUnaryOperationImpl
|
||||
}
|
||||
};
|
||||
|
||||
template <typename Op>
|
||||
struct StringUnaryOperationReduceImpl
|
||||
{
|
||||
MULTITARGET_FUNCTION_AVX512BW_AVX512F_AVX2_SSE42(
|
||||
MULTITARGET_FUNCTION_HEADER(static UInt64 NO_INLINE),
|
||||
vectorImpl,
|
||||
MULTITARGET_FUNCTION_BODY((const UInt8 * start, const UInt8 * end) /// NOLINT
|
||||
{
|
||||
UInt64 res = 0;
|
||||
while (start < end)
|
||||
res += Op::apply(*start++);
|
||||
return res;
|
||||
}))
|
||||
|
||||
static UInt64 NO_INLINE vector(const UInt8 * start, const UInt8 * end)
|
||||
{
|
||||
#if USE_MULTITARGET_CODE
|
||||
if (isArchSupported(TargetArch::AVX512BW))
|
||||
{
|
||||
return vectorImplAVX512BW(start, end);
|
||||
}
|
||||
|
||||
if (isArchSupported(TargetArch::AVX512F))
|
||||
{
|
||||
return vectorImplAVX512F(start, end);
|
||||
}
|
||||
|
||||
if (isArchSupported(TargetArch::AVX2))
|
||||
{
|
||||
return vectorImplAVX2(start, end);
|
||||
}
|
||||
|
||||
if (isArchSupported(TargetArch::SSE42))
|
||||
{
|
||||
return vectorImplSSE42(start, end);
|
||||
}
|
||||
#endif
|
||||
|
||||
return vectorImpl(start, end);
|
||||
}
|
||||
};
|
||||
|
||||
template <typename FunctionName>
|
||||
struct FunctionUnaryArithmeticMonotonicity;
|
||||
@ -142,7 +185,8 @@ template <template <typename> class Op, typename Name, bool is_injective>
|
||||
class FunctionUnaryArithmetic : public IFunction
|
||||
{
|
||||
static constexpr bool allow_decimal = IsUnaryOperation<Op>::negate || IsUnaryOperation<Op>::abs || IsUnaryOperation<Op>::sign;
|
||||
static constexpr bool allow_fixed_string = Op<UInt8>::allow_fixed_string;
|
||||
static constexpr bool allow_string_or_fixed_string = Op<UInt8>::allow_string_or_fixed_string;
|
||||
static constexpr bool is_bit_count = IsUnaryOperation<Op>::bit_count;
|
||||
static constexpr bool is_sign_function = IsUnaryOperation<Op>::sign;
|
||||
|
||||
ContextPtr context;
|
||||
@ -170,8 +214,8 @@ class FunctionUnaryArithmetic : public IFunction
|
||||
DataTypeDecimal<Decimal128>,
|
||||
DataTypeDecimal<Decimal256>,
|
||||
DataTypeFixedString,
|
||||
DataTypeInterval
|
||||
>(type, std::forward<F>(f));
|
||||
DataTypeString,
|
||||
DataTypeInterval>(type, std::forward<F>(f));
|
||||
}
|
||||
|
||||
static FunctionOverloadResolverPtr
|
||||
@ -204,7 +248,10 @@ public:
|
||||
|
||||
size_t getNumberOfArguments() const override { return 1; }
|
||||
bool isInjective(const ColumnsWithTypeAndName &) const override { return is_injective; }
|
||||
bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo & /*arguments*/) const override { return false; }
|
||||
bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo & /*arguments*/) const override
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
bool useDefaultImplementationForConstants() const override { return true; }
|
||||
|
||||
@ -232,9 +279,33 @@ public:
|
||||
using DataType = std::decay_t<decltype(type)>;
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType>)
|
||||
{
|
||||
if constexpr (!Op<DataTypeFixedString>::allow_fixed_string)
|
||||
if constexpr (!allow_string_or_fixed_string)
|
||||
return false;
|
||||
result = std::make_shared<DataType>(type.getN());
|
||||
/// For `bitCount`, when argument is FixedString, it's return type
|
||||
/// should be integer instead of FixedString, the return value is
|
||||
/// the sum of `bitCount` apply to each chars.
|
||||
else
|
||||
{
|
||||
/// UInt16 can save bitCount of FixedString less than 8192,
|
||||
/// it's should enough for almost all cases, and the setting
|
||||
/// `allow_suspicious_fixed_string_types` is disabled by default.
|
||||
if constexpr (is_bit_count)
|
||||
result = std::make_shared<DataTypeUInt16>();
|
||||
else
|
||||
result = std::make_shared<DataType>(type.getN());
|
||||
}
|
||||
}
|
||||
else if constexpr (std::is_same_v<DataTypeString, DataType>)
|
||||
{
|
||||
if constexpr (!allow_string_or_fixed_string)
|
||||
return false;
|
||||
else
|
||||
{
|
||||
if constexpr (is_bit_count)
|
||||
result = std::make_shared<DataTypeUInt64>();
|
||||
else
|
||||
result = std::make_shared<DataType>();
|
||||
}
|
||||
}
|
||||
else if constexpr (std::is_same_v<DataTypeInterval, DataType>)
|
||||
{
|
||||
@ -278,16 +349,80 @@ public:
|
||||
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType>)
|
||||
{
|
||||
if constexpr (allow_fixed_string)
|
||||
if constexpr (allow_string_or_fixed_string)
|
||||
{
|
||||
if (const auto * col = checkAndGetColumn<ColumnFixedString>(arguments[0].column.get()))
|
||||
{
|
||||
auto col_res = ColumnFixedString::create(col->getN());
|
||||
auto & vec_res = col_res->getChars();
|
||||
vec_res.resize(col->size() * col->getN());
|
||||
FixedStringUnaryOperationImpl<Op<UInt8>>::vector(col->getChars(), vec_res);
|
||||
result_column = std::move(col_res);
|
||||
return true;
|
||||
if constexpr (is_bit_count)
|
||||
{
|
||||
auto size = col->size();
|
||||
|
||||
auto col_res = ColumnUInt16::create(size);
|
||||
auto & vec_res = col_res->getData();
|
||||
vec_res.resize(col->size());
|
||||
|
||||
const auto & chars = col->getChars();
|
||||
auto n = col->getN();
|
||||
for (size_t i = 0; i < size; ++i)
|
||||
{
|
||||
vec_res[i] = StringUnaryOperationReduceImpl<Op<UInt8>>::vector(
|
||||
chars.data() + n * i, chars.data() + n * (i + 1));
|
||||
}
|
||||
result_column = std::move(col_res);
|
||||
return true;
|
||||
}
|
||||
else
|
||||
{
|
||||
auto col_res = ColumnFixedString::create(col->getN());
|
||||
auto & vec_res = col_res->getChars();
|
||||
vec_res.resize(col->size() * col->getN());
|
||||
FixedStringUnaryOperationImpl<Op<UInt8>>::vector(col->getChars(), vec_res);
|
||||
result_column = std::move(col_res);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
else if constexpr (std::is_same_v<DataTypeString, DataType>)
|
||||
{
|
||||
if constexpr (allow_string_or_fixed_string)
|
||||
{
|
||||
if (const auto * col = checkAndGetColumn<ColumnString>(arguments[0].column.get()))
|
||||
{
|
||||
if constexpr (is_bit_count)
|
||||
{
|
||||
auto size = col->size();
|
||||
|
||||
auto col_res = ColumnUInt64::create(size);
|
||||
auto & vec_res = col_res->getData();
|
||||
|
||||
const auto & chars = col->getChars();
|
||||
const auto & offsets = col->getOffsets();
|
||||
for (size_t i = 0; i < size; ++i)
|
||||
{
|
||||
vec_res[i] = StringUnaryOperationReduceImpl<Op<UInt8>>::vector(
|
||||
chars.data() + offsets[i - 1], chars.data() + offsets[i] - 1);
|
||||
}
|
||||
result_column = std::move(col_res);
|
||||
return true;
|
||||
}
|
||||
else
|
||||
{
|
||||
auto col_res = ColumnString::create();
|
||||
auto & vec_res = col_res->getChars();
|
||||
auto & offset_res = col_res->getOffsets();
|
||||
|
||||
const auto & vec_col = col->getChars();
|
||||
const auto & offset_col = col->getOffsets();
|
||||
|
||||
vec_res.resize(vec_col.size());
|
||||
offset_res.resize(offset_col.size());
|
||||
memcpy(offset_res.data(), offset_col.data(), offset_res.size() * sizeof(UInt64));
|
||||
|
||||
FixedStringUnaryOperationImpl<Op<UInt8>>::vector(vec_col, vec_res);
|
||||
result_column = std::move(col_res);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -350,7 +485,7 @@ public:
|
||||
return castType(arguments[0].get(), [&](const auto & type)
|
||||
{
|
||||
using DataType = std::decay_t<decltype(type)>;
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType>)
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType> || std::is_same_v<DataTypeString, DataType>)
|
||||
return false;
|
||||
else
|
||||
return !IsDataTypeDecimal<DataType> && Op<typename DataType::FieldType>::compilable;
|
||||
@ -365,7 +500,7 @@ public:
|
||||
castType(types[0].get(), [&](const auto & type)
|
||||
{
|
||||
using DataType = std::decay_t<decltype(type)>;
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType>)
|
||||
if constexpr (std::is_same_v<DataTypeFixedString, DataType> || std::is_same_v<DataTypeString, DataType>)
|
||||
return false;
|
||||
else
|
||||
{
|
||||
|
||||
@ -5,7 +5,9 @@ namespace DB
|
||||
|
||||
/// These classes should be present in DB namespace (cannot place them into namelesspace)
|
||||
template <typename> struct AbsImpl;
|
||||
template <typename> struct BitCountImpl;
|
||||
template <typename> struct NegateImpl;
|
||||
template <typename> struct SignImpl;
|
||||
template <typename, typename> struct PlusImpl;
|
||||
template <typename, typename> struct MinusImpl;
|
||||
template <typename, typename> struct MultiplyImpl;
|
||||
@ -22,9 +24,6 @@ template <typename, typename> struct LessOrEqualsOp;
|
||||
template <typename, typename> struct GreaterOrEqualsOp;
|
||||
template <typename, typename> struct BitHammingDistanceImpl;
|
||||
|
||||
template <typename>
|
||||
struct SignImpl;
|
||||
|
||||
template <template <typename, typename> typename Op1, template <typename, typename> typename Op2>
|
||||
struct IsSameOperation
|
||||
{
|
||||
@ -37,6 +36,7 @@ struct IsUnaryOperation
|
||||
static constexpr bool abs = std::is_same_v<Op<Int8>, AbsImpl<Int8>>;
|
||||
static constexpr bool negate = std::is_same_v<Op<Int8>, NegateImpl<Int8>>;
|
||||
static constexpr bool sign = std::is_same_v<Op<Int8>, SignImpl<Int8>>;
|
||||
static constexpr bool bit_count = std::is_same_v<Op<Int8>, BitCountImpl<Int8>>;
|
||||
};
|
||||
|
||||
template <template <typename, typename> typename Op>
|
||||
|
||||
@ -10,8 +10,7 @@ template <typename A>
|
||||
struct AbsImpl
|
||||
{
|
||||
using ResultType = std::conditional_t<is_decimal<A>, A, typename NumberTraits::ResultOfAbs<A>::Type>;
|
||||
static const constexpr bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline NO_SANITIZE_UNDEFINED ResultType apply(A a)
|
||||
{
|
||||
|
||||
@ -16,8 +16,8 @@ template <typename A, typename B>
|
||||
struct BitAndImpl
|
||||
{
|
||||
using ResultType = typename NumberTraits::ResultOfBit<A, B>::Type;
|
||||
static constexpr const bool allow_fixed_string = true;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_fixed_string = true;
|
||||
static constexpr bool allow_string_integer = false;
|
||||
|
||||
template <typename Result = ResultType>
|
||||
static inline Result apply(A a, B b)
|
||||
|
||||
@ -6,15 +6,11 @@
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
template <typename A>
|
||||
struct BitCountImpl
|
||||
{
|
||||
using ResultType = UInt8;
|
||||
static constexpr bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = true;
|
||||
|
||||
static inline ResultType apply(A a)
|
||||
{
|
||||
@ -41,8 +37,6 @@ struct BitCountImpl
|
||||
struct NameBitCount { static constexpr auto name = "bitCount"; };
|
||||
using FunctionBitCount = FunctionUnaryArithmetic<BitCountImpl, NameBitCount, false /* is injective */>;
|
||||
|
||||
}
|
||||
|
||||
/// The function has no ranges of monotonicity.
|
||||
template <> struct FunctionUnaryArithmeticMonotonicity<NameBitCount>
|
||||
{
|
||||
|
||||
@ -8,8 +8,8 @@ template <typename A, typename B>
|
||||
struct BitHammingDistanceImpl
|
||||
{
|
||||
using ResultType = UInt8;
|
||||
static const constexpr bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_fixed_string = false;
|
||||
static constexpr bool allow_string_integer = false;
|
||||
|
||||
template <typename Result = ResultType>
|
||||
static inline NO_SANITIZE_UNDEFINED Result apply(A a, B b)
|
||||
|
||||
@ -17,8 +17,7 @@ template <typename A>
|
||||
struct BitNotImpl
|
||||
{
|
||||
using ResultType = typename NumberTraits::ResultOfBitNot<A>::Type;
|
||||
static const constexpr bool allow_fixed_string = true;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = true;
|
||||
|
||||
static inline ResultType NO_SANITIZE_UNDEFINED apply(A a)
|
||||
{
|
||||
|
||||
@ -15,8 +15,8 @@ template <typename A, typename B>
|
||||
struct BitOrImpl
|
||||
{
|
||||
using ResultType = typename NumberTraits::ResultOfBit<A, B>::Type;
|
||||
static constexpr const bool allow_fixed_string = true;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_fixed_string = true;
|
||||
static constexpr bool allow_string_integer = false;
|
||||
|
||||
template <typename Result = ResultType>
|
||||
static inline Result apply(A a, B b)
|
||||
|
||||
@ -19,8 +19,7 @@ template <typename A>
|
||||
struct BitSwapLastTwoImpl
|
||||
{
|
||||
using ResultType = UInt8;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType NO_SANITIZE_UNDEFINED apply([[maybe_unused]] A a)
|
||||
{
|
||||
|
||||
@ -19,8 +19,7 @@ template <typename A>
|
||||
struct BitWrapperFuncImpl
|
||||
{
|
||||
using ResultType = UInt8;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType NO_SANITIZE_UNDEFINED apply(A a [[maybe_unused]])
|
||||
{
|
||||
|
||||
@ -17,8 +17,7 @@ struct FactorialImpl
|
||||
{
|
||||
using ResultType = UInt64;
|
||||
static const constexpr bool allow_decimal = false;
|
||||
static const constexpr bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static const constexpr bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline NO_SANITIZE_UNDEFINED ResultType apply(A a)
|
||||
{
|
||||
|
||||
@ -17,8 +17,7 @@ template <typename A>
|
||||
struct IntExp10Impl
|
||||
{
|
||||
using ResultType = UInt64;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType apply([[maybe_unused]] A a)
|
||||
{
|
||||
|
||||
@ -18,8 +18,7 @@ template <typename A>
|
||||
struct IntExp2Impl
|
||||
{
|
||||
using ResultType = UInt64;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType apply([[maybe_unused]] A a)
|
||||
{
|
||||
|
||||
@ -9,8 +9,7 @@ template <typename A>
|
||||
struct NegateImpl
|
||||
{
|
||||
using ResultType = std::conditional_t<is_decimal<A>, A, typename NumberTraits::ResultOfNegate<A>::Type>;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline NO_SANITIZE_UNDEFINED ResultType apply(A a)
|
||||
{
|
||||
|
||||
@ -10,8 +10,7 @@ template <typename A>
|
||||
struct RoundAgeImpl
|
||||
{
|
||||
using ResultType = UInt8;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType apply(A x)
|
||||
{
|
||||
|
||||
@ -10,8 +10,7 @@ template <typename A>
|
||||
struct RoundDurationImpl
|
||||
{
|
||||
using ResultType = UInt16;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline ResultType apply(A x)
|
||||
{
|
||||
|
||||
@ -63,8 +63,7 @@ template <typename T>
|
||||
struct RoundToExp2Impl
|
||||
{
|
||||
using ResultType = T;
|
||||
static constexpr const bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr const bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline T apply(T x)
|
||||
{
|
||||
|
||||
@ -9,8 +9,7 @@ template <typename A>
|
||||
struct SignImpl
|
||||
{
|
||||
using ResultType = Int8;
|
||||
static const constexpr bool allow_fixed_string = false;
|
||||
static const constexpr bool allow_string_integer = false;
|
||||
static constexpr bool allow_string_or_fixed_string = false;
|
||||
|
||||
static inline NO_SANITIZE_UNDEFINED ResultType apply(A a)
|
||||
{
|
||||
|
||||
@ -28,7 +28,6 @@
|
||||
#include <Common/Allocator.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/intExp.h>
|
||||
|
||||
#include <Formats/FormatSettings.h>
|
||||
@ -137,22 +136,6 @@ inline void readStringBinary(std::string & s, ReadBuffer & buf, size_t max_strin
|
||||
buf.readStrict(s.data(), size);
|
||||
}
|
||||
|
||||
|
||||
inline StringRef readStringBinaryInto(Arena & arena, ReadBuffer & buf)
|
||||
{
|
||||
size_t size = 0;
|
||||
readVarUInt(size, buf);
|
||||
|
||||
if (unlikely(size > DEFAULT_MAX_STRING_SIZE))
|
||||
throw Exception(ErrorCodes::TOO_LARGE_STRING_SIZE, "Too large string size.");
|
||||
|
||||
char * data = arena.alloc(size);
|
||||
buf.readStrict(data, size);
|
||||
|
||||
return StringRef(data, size);
|
||||
}
|
||||
|
||||
|
||||
template <typename T>
|
||||
void readVectorBinary(std::vector<T> & v, ReadBuffer & buf)
|
||||
{
|
||||
|
||||
33
src/IO/ReadHelpersArena.h
Normal file
33
src/IO/ReadHelpersArena.h
Normal file
@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/VarInt.h>
|
||||
#include <base/StringRef.h>
|
||||
#include <Common/Arena.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int TOO_LARGE_STRING_SIZE;
|
||||
}
|
||||
|
||||
inline StringRef readStringBinaryInto(Arena & arena, ReadBuffer & buf)
|
||||
{
|
||||
size_t size = 0;
|
||||
readVarUInt(size, buf);
|
||||
|
||||
if (unlikely(size > DEFAULT_MAX_STRING_SIZE))
|
||||
throw Exception(ErrorCodes::TOO_LARGE_STRING_SIZE, "Too large string size.");
|
||||
|
||||
char * data = arena.alloc(size);
|
||||
buf.readStrict(data, size);
|
||||
|
||||
return StringRef(data, size);
|
||||
}
|
||||
|
||||
}
|
||||
@ -17,6 +17,7 @@ namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int BAD_ARGUMENTS;
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
@ -330,6 +331,54 @@ namespace
|
||||
updateGrantedAccessRightsAndRolesTemplate(*role, elements_to_grant, elements_to_revoke, roles_to_grant, roles_to_revoke, admin_option);
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
void grantCurrentGrantsTemplate(
|
||||
T & grantee,
|
||||
const AccessRights & rights_to_grant,
|
||||
const AccessRightsElements & elements_to_revoke)
|
||||
{
|
||||
if (!elements_to_revoke.empty())
|
||||
grantee.access.revoke(elements_to_revoke);
|
||||
|
||||
grantee.access.makeUnion(rights_to_grant);
|
||||
}
|
||||
|
||||
/// Grants current user's grants with grant options to specified user.
|
||||
void grantCurrentGrants(
|
||||
IAccessEntity & grantee,
|
||||
const AccessRights & new_rights,
|
||||
const AccessRightsElements & elements_to_revoke)
|
||||
{
|
||||
if (auto * user = typeid_cast<User *>(&grantee))
|
||||
grantCurrentGrantsTemplate(*user, new_rights, elements_to_revoke);
|
||||
else if (auto * role = typeid_cast<Role *>(&grantee))
|
||||
grantCurrentGrantsTemplate(*role, new_rights, elements_to_revoke);
|
||||
}
|
||||
|
||||
/// Calculates all available rights to grant with current user intersection.
|
||||
void calculateCurrentGrantRightsWithIntersection(
|
||||
AccessRights & rights,
|
||||
std::shared_ptr<const ContextAccess> current_user_access,
|
||||
const AccessRightsElements & elements_to_grant)
|
||||
{
|
||||
AccessRightsElements current_user_grantable_elements;
|
||||
auto available_grant_elements = current_user_access->getAccessRights()->getElements();
|
||||
AccessRights current_user_rights;
|
||||
for (auto & element : available_grant_elements)
|
||||
{
|
||||
if (!element.grant_option && !element.is_partial_revoke)
|
||||
continue;
|
||||
|
||||
if (element.is_partial_revoke)
|
||||
current_user_rights.revoke(element);
|
||||
else
|
||||
current_user_rights.grant(element);
|
||||
}
|
||||
|
||||
rights.grant(elements_to_grant);
|
||||
rights.makeIntersection(current_user_rights);
|
||||
}
|
||||
|
||||
/// Updates grants of a specified user or role.
|
||||
void updateFromQuery(IAccessEntity & grantee, const ASTGrantQuery & query)
|
||||
{
|
||||
@ -373,6 +422,9 @@ BlockIO InterpreterGrantQuery::execute()
|
||||
/// Executing on cluster.
|
||||
if (!query.cluster.empty())
|
||||
{
|
||||
if (query.current_grants)
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "GRANT CURRENT GRANTS can't be executed on cluster.");
|
||||
|
||||
auto required_access = getRequiredAccessForExecutingOnCluster(elements_to_grant, elements_to_revoke);
|
||||
checkAdminOptionForExecutingOnCluster(*current_user_access, roles_to_grant, roles_to_revoke);
|
||||
current_user_access->checkGranteesAreAllowed(grantees);
|
||||
@ -386,7 +438,8 @@ BlockIO InterpreterGrantQuery::execute()
|
||||
elements_to_grant.replaceEmptyDatabase(current_database);
|
||||
elements_to_revoke.replaceEmptyDatabase(current_database);
|
||||
bool need_check_grantees_are_allowed = true;
|
||||
checkGrantOption(access_control, *current_user_access, grantees, need_check_grantees_are_allowed, elements_to_grant, elements_to_revoke);
|
||||
if (!query.current_grants)
|
||||
checkGrantOption(access_control, *current_user_access, grantees, need_check_grantees_are_allowed, elements_to_grant, elements_to_revoke);
|
||||
|
||||
/// Check if the current user has corresponding roles granted with admin option.
|
||||
checkAdminOption(access_control, *current_user_access, grantees, need_check_grantees_are_allowed, roles_to_grant, roles_to_revoke, query.admin_option);
|
||||
@ -394,11 +447,18 @@ BlockIO InterpreterGrantQuery::execute()
|
||||
if (need_check_grantees_are_allowed)
|
||||
current_user_access->checkGranteesAreAllowed(grantees);
|
||||
|
||||
AccessRights new_rights;
|
||||
if (query.current_grants)
|
||||
calculateCurrentGrantRightsWithIntersection(new_rights, current_user_access, elements_to_grant);
|
||||
|
||||
/// Update roles and users listed in `grantees`.
|
||||
auto update_func = [&](const AccessEntityPtr & entity) -> AccessEntityPtr
|
||||
{
|
||||
auto clone = entity->clone();
|
||||
updateGrantedAccessRightsAndRoles(*clone, elements_to_grant, elements_to_revoke, roles_to_grant, roles_to_revoke, query.admin_option);
|
||||
if (query.current_grants)
|
||||
grantCurrentGrants(*clone, new_rights, elements_to_revoke);
|
||||
else
|
||||
updateGrantedAccessRightsAndRoles(*clone, elements_to_grant, elements_to_revoke, roles_to_grant, roles_to_revoke, query.admin_option);
|
||||
return clone;
|
||||
};
|
||||
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#include <array>
|
||||
|
||||
#include <Common/SipHash.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/HashTable/Hash.h>
|
||||
#include <Common/memcpySmall.h>
|
||||
#include <Common/assert_cast.h>
|
||||
@ -25,6 +24,8 @@ namespace ErrorCodes
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
class Arena;
|
||||
|
||||
using Sizes = std::vector<size_t>;
|
||||
|
||||
/// When packing the values of nullable columns at a given row, we have to
|
||||
|
||||
@ -7,7 +7,6 @@
|
||||
|
||||
|
||||
#include <base/StringRef.h>
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/HashTable/FixedHashMap.h>
|
||||
#include <Common/HashTable/HashMap.h>
|
||||
#include <Common/HashTable/TwoLevelHashMap.h>
|
||||
@ -47,6 +46,10 @@ namespace ErrorCodes
|
||||
extern const int UNKNOWN_AGGREGATED_DATA_VARIANT;
|
||||
}
|
||||
|
||||
class Arena;
|
||||
using ArenaPtr = std::shared_ptr<Arena>;
|
||||
using Arenas = std::vector<ArenaPtr>;
|
||||
|
||||
/** Different data structures that can be used for aggregation
|
||||
* For efficiency, the aggregation data itself is put into the pool.
|
||||
* Data and pool ownership (states of aggregate functions)
|
||||
|
||||
@ -719,7 +719,7 @@ bool FileCache::tryReserve(const Key & key, size_t offset, size_t size, std::loc
|
||||
{
|
||||
auto queue_iterator = cell_for_reserve->queue_iterator;
|
||||
if (queue_iterator)
|
||||
queue_iterator->incrementSize(size, cache_lock);
|
||||
queue_iterator->updateSize(size, cache_lock);
|
||||
else
|
||||
cell_for_reserve->queue_iterator = main_priority->add(key, offset, size, cache_lock);
|
||||
}
|
||||
@ -836,7 +836,7 @@ bool FileCache::tryReserveForMainList(
|
||||
/// If queue iterator already exists, we need to update the size after each space reservation.
|
||||
auto queue_iterator = cell_for_reserve->queue_iterator;
|
||||
if (queue_iterator)
|
||||
queue_iterator->incrementSize(size, cache_lock);
|
||||
queue_iterator->updateSize(size, cache_lock);
|
||||
else
|
||||
cell_for_reserve->queue_iterator = main_priority->add(key, offset, size, cache_lock);
|
||||
}
|
||||
@ -1152,7 +1152,14 @@ void FileCache::reduceSizeToDownloaded(
|
||||
cell->file_segment = std::make_shared<FileSegment>(
|
||||
offset, downloaded_size, key, this, FileSegment::State::DOWNLOADED, create_settings);
|
||||
|
||||
assert(file_segment->reserved_size == downloaded_size);
|
||||
chassert(cell->queue_iterator);
|
||||
chassert(cell->queue_iterator->size() >= downloaded_size);
|
||||
const ssize_t diff = cell->queue_iterator->size() - downloaded_size;
|
||||
if (diff > 0)
|
||||
cell->queue_iterator->updateSize(-diff, cache_lock);
|
||||
|
||||
chassert(file_segment->reserved_size == downloaded_size);
|
||||
chassert(file_segment->reserved_size == cell->queue_iterator->size());
|
||||
}
|
||||
|
||||
bool FileCache::isLastFileSegmentHolder(
|
||||
@ -1450,7 +1457,7 @@ void FileCache::QueryContext::reserve(const Key & key, size_t offset, size_t siz
|
||||
auto queue_iter = priority->add(key, offset, 0, cache_lock);
|
||||
record = records.insert({{key, offset}, queue_iter}).first;
|
||||
}
|
||||
record->second->incrementSize(size, cache_lock);
|
||||
record->second->updateSize(size, cache_lock);
|
||||
}
|
||||
cache_size += size;
|
||||
}
|
||||
|
||||
@ -11,6 +11,8 @@
|
||||
|
||||
#include <magic_enum.hpp>
|
||||
|
||||
namespace fs = std::filesystem;
|
||||
|
||||
namespace CurrentMetrics
|
||||
{
|
||||
extern const Metric CacheDetachedFileSegments;
|
||||
@ -63,7 +65,7 @@ FileSegment::FileSegment(
|
||||
{
|
||||
reserved_size = downloaded_size = size_;
|
||||
is_downloaded = true;
|
||||
chassert(std::filesystem::file_size(getPathInLocalCache()) == size_);
|
||||
chassert(fs::file_size(getPathInLocalCache()) == size_);
|
||||
break;
|
||||
}
|
||||
case (State::SKIP_CACHE):
|
||||
@ -385,6 +387,28 @@ void FileSegment::write(const char * from, size_t size, size_t offset)
|
||||
|
||||
chassert(std::filesystem::file_size(getPathInLocalCache()) == downloaded_size);
|
||||
}
|
||||
catch (ErrnoException & e)
|
||||
{
|
||||
std::unique_lock segment_lock(mutex);
|
||||
|
||||
wrapWithCacheInfo(e, "while writing into cache", segment_lock);
|
||||
|
||||
int code = e.getErrno();
|
||||
if (code == /* No space left on device */28 || code == /* Quota exceeded */122)
|
||||
{
|
||||
const auto file_size = fs::file_size(getPathInLocalCache());
|
||||
chassert(downloaded_size <= file_size);
|
||||
chassert(reserved_size >= file_size);
|
||||
if (downloaded_size != file_size)
|
||||
downloaded_size = file_size;
|
||||
}
|
||||
|
||||
setDownloadFailedUnlocked(segment_lock);
|
||||
|
||||
cv.notify_all();
|
||||
throw;
|
||||
|
||||
}
|
||||
catch (Exception & e)
|
||||
{
|
||||
std::unique_lock segment_lock(mutex);
|
||||
@ -394,7 +418,6 @@ void FileSegment::write(const char * from, size_t size, size_t offset)
|
||||
setDownloadFailedUnlocked(segment_lock);
|
||||
|
||||
cv.notify_all();
|
||||
|
||||
throw;
|
||||
}
|
||||
|
||||
@ -504,7 +527,7 @@ void FileSegment::setDownloadedUnlocked([[maybe_unused]] std::unique_lock<std::m
|
||||
is_downloaded = true;
|
||||
|
||||
assert(getDownloadedSizeUnlocked(segment_lock) > 0);
|
||||
assert(std::filesystem::file_size(getPathInLocalCache()) > 0);
|
||||
assert(fs::file_size(getPathInLocalCache()) > 0);
|
||||
}
|
||||
|
||||
void FileSegment::setDownloadFailedUnlocked(std::unique_lock<std::mutex> & segment_lock)
|
||||
@ -633,7 +656,7 @@ void FileSegment::completeBasedOnCurrentState(std::lock_guard<std::mutex> & cach
|
||||
case State::DOWNLOADED:
|
||||
{
|
||||
chassert(getDownloadedSizeUnlocked(segment_lock) == range().size());
|
||||
chassert(getDownloadedSizeUnlocked(segment_lock) == std::filesystem::file_size(getPathInLocalCache()));
|
||||
chassert(getDownloadedSizeUnlocked(segment_lock) == fs::file_size(getPathInLocalCache()));
|
||||
chassert(is_downloaded);
|
||||
chassert(!cache_writer);
|
||||
break;
|
||||
@ -746,7 +769,7 @@ void FileSegment::assertCorrectnessUnlocked(std::unique_lock<std::mutex> & segme
|
||||
auto current_downloader = getDownloaderUnlocked(segment_lock);
|
||||
chassert(current_downloader.empty() == (download_state != FileSegment::State::DOWNLOADING));
|
||||
chassert(!current_downloader.empty() == (download_state == FileSegment::State::DOWNLOADING));
|
||||
chassert(download_state != FileSegment::State::DOWNLOADED || std::filesystem::file_size(getPathInLocalCache()) > 0);
|
||||
chassert(download_state != FileSegment::State::DOWNLOADED || fs::file_size(getPathInLocalCache()) > 0);
|
||||
}
|
||||
|
||||
void FileSegment::throwIfDetachedUnlocked(std::unique_lock<std::mutex> & segment_lock) const
|
||||
|
||||
@ -61,7 +61,7 @@ public:
|
||||
/// the iterator should automatically point to the next record.
|
||||
virtual void removeAndGetNext(std::lock_guard<std::mutex> &) = 0;
|
||||
|
||||
virtual void incrementSize(size_t, std::lock_guard<std::mutex> &) = 0;
|
||||
virtual void updateSize(ssize_t, std::lock_guard<std::mutex> &) = 0;
|
||||
};
|
||||
|
||||
public:
|
||||
|
||||
@ -94,11 +94,19 @@ void LRUFileCachePriority::LRUFileCacheIterator::removeAndGetNext(std::lock_guar
|
||||
queue_iter = cache_priority->queue.erase(queue_iter);
|
||||
}
|
||||
|
||||
void LRUFileCachePriority::LRUFileCacheIterator::incrementSize(size_t size_increment, std::lock_guard<std::mutex> &)
|
||||
void LRUFileCachePriority::LRUFileCacheIterator::updateSize(ssize_t size, std::lock_guard<std::mutex> &)
|
||||
{
|
||||
cache_priority->cache_size += size_increment;
|
||||
CurrentMetrics::add(CurrentMetrics::FilesystemCacheSize, size_increment);
|
||||
queue_iter->size += size_increment;
|
||||
cache_priority->cache_size += size;
|
||||
|
||||
if (size > 0)
|
||||
CurrentMetrics::add(CurrentMetrics::FilesystemCacheSize, size);
|
||||
else
|
||||
CurrentMetrics::sub(CurrentMetrics::FilesystemCacheSize, size);
|
||||
|
||||
queue_iter->size += size;
|
||||
|
||||
chassert(queue_iter->size > 0);
|
||||
chassert(cache_priority->cache_size >= 0);
|
||||
}
|
||||
|
||||
void LRUFileCachePriority::LRUFileCacheIterator::use(std::lock_guard<std::mutex> &)
|
||||
|
||||
@ -54,7 +54,7 @@ public:
|
||||
|
||||
void removeAndGetNext(std::lock_guard<std::mutex> &) override;
|
||||
|
||||
void incrementSize(size_t size_increment, std::lock_guard<std::mutex> &) override;
|
||||
void updateSize(ssize_t size, std::lock_guard<std::mutex> &) override;
|
||||
|
||||
void use(std::lock_guard<std::mutex> &) override;
|
||||
|
||||
|
||||
@ -517,7 +517,7 @@ size_t HashJoin::getTotalByteCount() const
|
||||
|
||||
res += data->blocks_allocated_size;
|
||||
res += data->blocks_nullmaps_allocated_size;
|
||||
res += data->pool.size();
|
||||
res += data->pool.allocatedBytes();
|
||||
|
||||
if (data->type != Type::CROSS)
|
||||
{
|
||||
|
||||
@ -156,7 +156,7 @@ void NO_INLINE bench(const std::vector<StringRef> & data, DB::Arena &, const cha
|
||||
}
|
||||
watch.stop();
|
||||
|
||||
std::cerr << "arena-memory " << pool.size() + map.getBufferSizeInBytes() << std::endl;
|
||||
std::cerr << "arena-memory " << pool.allocatedBytes() + map.getBufferSizeInBytes() << std::endl;
|
||||
std::cerr << "single-run " << std::setprecision(3)
|
||||
<< watch.elapsedSeconds() << std::endl;
|
||||
}
|
||||
|
||||
@ -34,7 +34,7 @@ void NO_INLINE bench(const std::vector<StringRef> & data, DB::Arena & pool, cons
|
||||
}
|
||||
watch.stop();
|
||||
|
||||
std::cerr << "arena-memory " << pool.size() + set.getBufferSizeInBytes() << std::endl;
|
||||
std::cerr << "arena-memory " << pool.allocatedBytes() + set.getBufferSizeInBytes() << std::endl;
|
||||
std::cerr << "single-run " << std::setprecision(3)
|
||||
<< watch.elapsedSeconds() << std::endl;
|
||||
}
|
||||
|
||||
@ -147,6 +147,8 @@ void ASTGrantQuery::formatImpl(const FormatSettings & settings, FormatState &, F
|
||||
"ASTGrantQuery can contain either roles or access rights elements "
|
||||
"to grant or revoke, not both of them");
|
||||
}
|
||||
else if (current_grants)
|
||||
settings.ostr << (settings.hilite ? hilite_keyword : "") << " CURRENT GRANTS" << (settings.hilite ? hilite_none : "");
|
||||
else
|
||||
formatElementsWithoutOptions(access_rights_elements, settings);
|
||||
|
||||
|
||||
@ -26,6 +26,8 @@ public:
|
||||
bool admin_option = false;
|
||||
bool replace_access = false;
|
||||
bool replace_granted_roles = false;
|
||||
bool current_grants = false;
|
||||
|
||||
std::shared_ptr<ASTRolesOrUsersSet> grantees;
|
||||
|
||||
String getID(char) const override;
|
||||
|
||||
@ -43,7 +43,6 @@ namespace
|
||||
{
|
||||
if (!str.empty())
|
||||
str += " ";
|
||||
std::string_view word{pos->begin, pos->size()};
|
||||
str += std::string_view(pos->begin, pos->size());
|
||||
++pos;
|
||||
}
|
||||
@ -184,6 +183,37 @@ namespace
|
||||
});
|
||||
}
|
||||
|
||||
bool parseCurrentGrants(IParser::Pos & pos, Expected & expected, AccessRightsElements & elements)
|
||||
{
|
||||
if (ParserToken(TokenType::OpeningRoundBracket).ignore(pos, expected))
|
||||
{
|
||||
if (!parseElementsWithoutOptions(pos, expected, elements))
|
||||
return false;
|
||||
|
||||
if (!ParserToken(TokenType::ClosingRoundBracket).ignore(pos, expected))
|
||||
return false;
|
||||
}
|
||||
else
|
||||
{
|
||||
AccessRightsElement default_element(AccessType::ALL);
|
||||
|
||||
if (!ParserKeyword{"ON"}.ignore(pos, expected))
|
||||
return false;
|
||||
|
||||
String database_name, table_name;
|
||||
bool any_database = false, any_table = false;
|
||||
if (!parseDatabaseAndTableNameOrAsterisks(pos, expected, database_name, any_database, table_name, any_table))
|
||||
return false;
|
||||
|
||||
default_element.any_database = any_database;
|
||||
default_element.database = database_name;
|
||||
default_element.any_table = any_table;
|
||||
default_element.table = table_name;
|
||||
elements.push_back(std::move(default_element));
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
void throwIfNotGrantable(AccessRightsElements & elements)
|
||||
{
|
||||
@ -284,8 +314,19 @@ bool ParserGrantQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
|
||||
|
||||
AccessRightsElements elements;
|
||||
std::shared_ptr<ASTRolesOrUsersSet> roles;
|
||||
if (!parseElementsWithoutOptions(pos, expected, elements) && !parseRoles(pos, expected, is_revoke, attach_mode, roles))
|
||||
return false;
|
||||
|
||||
bool current_grants = false;
|
||||
if (!is_revoke && ParserKeyword{"CURRENT GRANTS"}.ignore(pos, expected))
|
||||
{
|
||||
current_grants = true;
|
||||
if (!parseCurrentGrants(pos, expected, elements))
|
||||
return false;
|
||||
}
|
||||
else
|
||||
{
|
||||
if (!parseElementsWithoutOptions(pos, expected, elements) && !parseRoles(pos, expected, is_revoke, attach_mode, roles))
|
||||
return false;
|
||||
}
|
||||
|
||||
if (cluster.empty())
|
||||
parseOnCluster(pos, expected, cluster);
|
||||
@ -353,6 +394,7 @@ bool ParserGrantQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
|
||||
query->admin_option = admin_option;
|
||||
query->replace_access = replace_access;
|
||||
query->replace_granted_roles = replace_role;
|
||||
query->current_grants = current_grants;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -2,7 +2,6 @@
|
||||
|
||||
#include <Processors/Formats/IOutputFormat.h>
|
||||
|
||||
#include <Common/Arena.h>
|
||||
#include <Common/ThreadPool.h>
|
||||
#include <Common/Stopwatch.h>
|
||||
#include <Common/logger_useful.h>
|
||||
|
||||
@ -168,7 +168,7 @@ AggregatingSortedAlgorithm::AggregatingMergedData::AggregatingMergedData(
|
||||
if (def.allocates_memory_in_arena)
|
||||
{
|
||||
arena = std::make_unique<Arena>();
|
||||
arena_size = arena->size();
|
||||
arena_size = arena->allocatedBytes();
|
||||
}
|
||||
}
|
||||
|
||||
@ -194,10 +194,10 @@ void AggregatingSortedAlgorithm::AggregatingMergedData::startGroup(const ColumnR
|
||||
/// To avoid this, reset arena if and only if:
|
||||
/// - arena is required (i.e. SimpleAggregateFunction(any, String) in PK),
|
||||
/// - arena was used in the previous groups.
|
||||
if (def.allocates_memory_in_arena && arena->size() > arena_size)
|
||||
if (def.allocates_memory_in_arena && arena->allocatedBytes() > arena_size)
|
||||
{
|
||||
arena = std::make_unique<Arena>();
|
||||
arena_size = arena->size();
|
||||
arena_size = arena->allocatedBytes();
|
||||
}
|
||||
|
||||
is_group_started = true;
|
||||
|
||||
@ -508,7 +508,7 @@ SummingSortedAlgorithm::SummingMergedData::SummingMergedData(
|
||||
if (def.allocates_memory_in_arena)
|
||||
{
|
||||
arena = std::make_unique<Arena>();
|
||||
arena_size = arena->size();
|
||||
arena_size = arena->allocatedBytes();
|
||||
}
|
||||
}
|
||||
|
||||
@ -522,10 +522,10 @@ void SummingSortedAlgorithm::SummingMergedData::startGroup(ColumnRawPtrs & raw_c
|
||||
for (auto & desc : def.columns_to_aggregate)
|
||||
desc.createState();
|
||||
|
||||
if (def.allocates_memory_in_arena && arena->size() > arena_size)
|
||||
if (def.allocates_memory_in_arena && arena->allocatedBytes() > arena_size)
|
||||
{
|
||||
arena = std::make_unique<Arena>();
|
||||
arena_size = arena->size();
|
||||
arena_size = arena->allocatedBytes();
|
||||
}
|
||||
|
||||
if (def.maps_to_sum.empty())
|
||||
|
||||
@ -2,14 +2,10 @@
|
||||
|
||||
#include <Processors/ISimpleTransform.h>
|
||||
#include <Processors/Transforms/finalizeChunk.h>
|
||||
#include <Common/Arena.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class Arena;
|
||||
using ArenaPtr = std::shared_ptr<Arena>;
|
||||
|
||||
class ExpressionActions;
|
||||
using ExpressionActionsPtr = std::shared_ptr<ExpressionActions>;
|
||||
|
||||
|
||||
@ -42,9 +42,13 @@
|
||||
#include <Interpreters/PartLog.h>
|
||||
#include <Interpreters/TransactionLog.h>
|
||||
#include <Interpreters/TreeRewriter.h>
|
||||
#include <Interpreters/Context_fwd.h>
|
||||
#include <IO/S3Common.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <Parsers/ASTExpressionList.h>
|
||||
#include <Parsers/ASTIndexDeclaration.h>
|
||||
#include <Parsers/ASTFunction.h>
|
||||
#include <Parsers/ASTLiteral.h>
|
||||
#include <Parsers/ASTNameTypePair.h>
|
||||
@ -54,6 +58,7 @@
|
||||
#include <Parsers/ExpressionListParsers.h>
|
||||
#include <Parsers/parseQuery.h>
|
||||
#include <Parsers/queryToString.h>
|
||||
#include <Parsers/ASTAlterQuery.h>
|
||||
#include <Processors/Formats/IInputFormat.h>
|
||||
#include <Processors/QueryPlan/QueryIdHolder.h>
|
||||
#include <Processors/QueryPlan/ReadFromMergeTree.h>
|
||||
@ -76,6 +81,7 @@
|
||||
|
||||
#include <algorithm>
|
||||
#include <atomic>
|
||||
#include <cmath>
|
||||
#include <chrono>
|
||||
#include <iomanip>
|
||||
#include <limits>
|
||||
@ -173,6 +179,19 @@ namespace ErrorCodes
|
||||
extern const int SOCKET_TIMEOUT;
|
||||
}
|
||||
|
||||
static void checkSuspiciousIndices(const ASTFunction * index_function)
|
||||
{
|
||||
std::unordered_set<UInt64> unique_index_expression_hashes;
|
||||
for (const auto & child : index_function->arguments->children)
|
||||
{
|
||||
IAST::Hash hash = child->getTreeHash();
|
||||
UInt64 first_half_of_hash = hash.first;
|
||||
|
||||
if (!unique_index_expression_hashes.emplace(first_half_of_hash).second)
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS,
|
||||
"Primary key or secondary index contains a duplicate expression. To suppress this exception, rerun the command with setting 'allow_suspicious_indices = 1'");
|
||||
}
|
||||
}
|
||||
|
||||
static void checkSampleExpression(const StorageInMemoryMetadata & metadata, bool allow_sampling_expression_not_in_primary_key, bool check_sample_column_is_correct)
|
||||
{
|
||||
@ -440,7 +459,10 @@ static void checkKeyExpression(const ExpressionActions & expr, const Block & sam
|
||||
}
|
||||
|
||||
void MergeTreeData::checkProperties(
|
||||
const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach) const
|
||||
const StorageInMemoryMetadata & new_metadata,
|
||||
const StorageInMemoryMetadata & old_metadata,
|
||||
bool attach,
|
||||
ContextPtr local_context) const
|
||||
{
|
||||
if (!new_metadata.sorting_key.definition_ast)
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "ORDER BY cannot be empty");
|
||||
@ -454,7 +476,13 @@ void MergeTreeData::checkProperties(
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Primary key must be a prefix of the sorting key, but its length: "
|
||||
"{} is greater than the sorting key length: {}", primary_key_size, sorting_key_size);
|
||||
|
||||
NameSet primary_key_columns_set;
|
||||
bool allow_suspicious_indices = getSettings()->allow_suspicious_indices;
|
||||
if (local_context)
|
||||
allow_suspicious_indices = local_context->getSettingsRef().allow_suspicious_indices;
|
||||
|
||||
if (!allow_suspicious_indices && !attach)
|
||||
if (const auto * index_function = typeid_cast<ASTFunction *>(new_sorting_key.definition_ast.get()))
|
||||
checkSuspiciousIndices(index_function);
|
||||
|
||||
for (size_t i = 0; i < sorting_key_size; ++i)
|
||||
{
|
||||
@ -468,9 +496,6 @@ void MergeTreeData::checkProperties(
|
||||
"Primary key must be a prefix of the sorting key, "
|
||||
"but the column in the position {} is {}", i, sorting_key_column +", not " + pk_column);
|
||||
|
||||
if (!primary_key_columns_set.emplace(pk_column).second)
|
||||
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Primary key contains duplicate columns");
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@ -531,6 +556,13 @@ void MergeTreeData::checkProperties(
|
||||
|
||||
for (const auto & index : new_metadata.secondary_indices)
|
||||
{
|
||||
if (!allow_suspicious_indices && !attach)
|
||||
{
|
||||
const auto * index_ast = typeid_cast<const ASTIndexDeclaration *>(index.definition_ast.get());
|
||||
if (const auto * index_function = typeid_cast<const ASTFunction *>(index_ast->expr))
|
||||
checkSuspiciousIndices(index_function);
|
||||
}
|
||||
|
||||
MergeTreeIndexFactory::instance().validate(index, attach);
|
||||
|
||||
if (indices_names.find(index.name) != indices_names.end())
|
||||
@ -556,9 +588,13 @@ void MergeTreeData::checkProperties(
|
||||
checkKeyExpression(*new_sorting_key.expression, new_sorting_key.sample_block, "Sorting", allow_nullable_key);
|
||||
}
|
||||
|
||||
void MergeTreeData::setProperties(const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach)
|
||||
void MergeTreeData::setProperties(
|
||||
const StorageInMemoryMetadata & new_metadata,
|
||||
const StorageInMemoryMetadata & old_metadata,
|
||||
bool attach,
|
||||
ContextPtr local_context)
|
||||
{
|
||||
checkProperties(new_metadata, old_metadata, attach);
|
||||
checkProperties(new_metadata, old_metadata, attach, local_context);
|
||||
setInMemoryMetadata(new_metadata);
|
||||
}
|
||||
|
||||
@ -990,7 +1026,6 @@ std::optional<UInt64> MergeTreeData::totalRowsByPartitionPredicateImpl(
|
||||
return res;
|
||||
}
|
||||
|
||||
|
||||
String MergeTreeData::MergingParams::getModeName() const
|
||||
{
|
||||
switch (mode)
|
||||
@ -3151,7 +3186,7 @@ void MergeTreeData::checkAlterIsPossible(const AlterCommands & commands, Context
|
||||
}
|
||||
}
|
||||
|
||||
checkProperties(new_metadata, old_metadata);
|
||||
checkProperties(new_metadata, old_metadata, false, local_context);
|
||||
checkTTLExpressions(new_metadata, old_metadata);
|
||||
|
||||
if (!columns_to_check_conversion.empty())
|
||||
|
||||
@ -1218,9 +1218,9 @@ protected:
|
||||
/// The same for clearOldTemporaryDirectories.
|
||||
std::mutex clear_old_temporary_directories_mutex;
|
||||
|
||||
void checkProperties(const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach = false) const;
|
||||
void checkProperties(const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach = false, ContextPtr local_context = nullptr) const;
|
||||
|
||||
void setProperties(const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach = false);
|
||||
void setProperties(const StorageInMemoryMetadata & new_metadata, const StorageInMemoryMetadata & old_metadata, bool attach = false, ContextPtr local_context = nullptr);
|
||||
|
||||
void checkPartitionKeyAndInitMinMax(const KeyDescription & new_partition_key);
|
||||
|
||||
|
||||
@ -130,6 +130,7 @@ struct Settings;
|
||||
M(UInt64, vertical_merge_algorithm_min_columns_to_activate, 11, "Minimal amount of non-PK columns to activate Vertical merge algorithm.", 0) \
|
||||
\
|
||||
/** Compatibility settings */ \
|
||||
M(Bool, allow_suspicious_indices, false, "Reject primary/secondary indexes and sorting keys with identical expressions", 0) \
|
||||
M(Bool, compatibility_allow_sampling_expression_not_in_primary_key, false, "Allow to create a table with sampling expression not in primary key. This is needed only to temporarily allow to run the server with wrong tables for backward compatibility.", 0) \
|
||||
M(Bool, use_minimalistic_checksums_in_zookeeper, true, "Use small format (dozens bytes) for part checksums in ZooKeeper instead of ordinary ones (dozens KB). Before enabling check that all replicas support new format.", 0) \
|
||||
M(Bool, use_minimalistic_part_header_in_zookeeper, true, "Store part header (checksums and columns) in a compact format and a single part znode instead of separate znodes (<part>/columns and <part>/checksums). This can dramatically reduce snapshot size in ZooKeeper. Before enabling check that all replicas support new format.", 0) \
|
||||
|
||||
@ -348,7 +348,7 @@ void StorageMergeTree::alter(
|
||||
changeSettings(new_metadata.settings_changes, table_lock_holder);
|
||||
checkTTLExpressions(new_metadata, old_metadata);
|
||||
/// Reinitialize primary key because primary key column types might have changed.
|
||||
setProperties(new_metadata, old_metadata);
|
||||
setProperties(new_metadata, old_metadata, false, local_context);
|
||||
|
||||
DatabaseCatalog::instance().getDatabase(table_id.database_name)->alterTable(local_context, table_id, new_metadata);
|
||||
|
||||
|
||||
@ -397,7 +397,7 @@ StoragePostgreSQL::Configuration StoragePostgreSQL::processNamedCollectionResult
|
||||
required_arguments.insert("table");
|
||||
|
||||
validateNamedCollection<ValidateKeysMultiset<ExternalDatabaseEqualKeysSet>>(
|
||||
named_collection, required_arguments, {"schema", "on_conflict", "addresses_expr", "host", "hostname", "port"});
|
||||
named_collection, required_arguments, {"schema", "on_conflict", "addresses_expr", "host", "hostname", "port", "use_tables_cache"});
|
||||
|
||||
configuration.addresses_expr = named_collection.getOrDefault<String>("addresses_expr", "");
|
||||
if (configuration.addresses_expr.empty())
|
||||
|
||||
144
tests/broken_tests.txt
Normal file
144
tests/broken_tests.txt
Normal file
@ -0,0 +1,144 @@
|
||||
00223_shard_distributed_aggregation_memory_efficient
|
||||
00562_in_subquery_merge_tree
|
||||
00593_union_all_assert_columns_removed
|
||||
00597_push_down_predicate_long
|
||||
00673_subquery_prepared_set_performance
|
||||
00700_decimal_compare
|
||||
00717_merge_and_distributed
|
||||
00725_memory_tracking
|
||||
00754_distributed_optimize_skip_select_on_unused_shards
|
||||
00754_distributed_optimize_skip_select_on_unused_shards_with_prewhere
|
||||
00838_unique_index
|
||||
00927_asof_joins
|
||||
00940_order_by_read_in_order_query_plan
|
||||
00945_bloom_filter_index
|
||||
00952_input_function
|
||||
00979_set_index_not
|
||||
00981_in_subquery_with_tuple
|
||||
01049_join_low_card_bug_long
|
||||
01062_pm_all_join_with_block_continuation
|
||||
01064_incremental_streaming_from_2_src_with_feedback
|
||||
01071_force_optimize_skip_unused_shards
|
||||
01072_optimize_skip_unused_shards_const_expr_eval
|
||||
01083_expressions_in_engine_arguments
|
||||
01086_odbc_roundtrip
|
||||
01142_join_lc_and_nullable_in_key
|
||||
01142_merge_join_lc_and_nullable_in_key
|
||||
01152_cross_replication
|
||||
01155_rename_move_materialized_view
|
||||
01173_transaction_control_queries
|
||||
01211_optimize_skip_unused_shards_type_mismatch
|
||||
01213_optimize_skip_unused_shards_DISTINCT
|
||||
01214_test_storage_merge_aliases_with_where
|
||||
01231_distributed_aggregation_memory_efficient_mix_levels
|
||||
01232_extremes
|
||||
01244_optimize_distributed_group_by_sharding_key
|
||||
01247_optimize_distributed_group_by_sharding_key_dist_on_dist
|
||||
01268_mv_scalars
|
||||
01268_shard_avgweighted
|
||||
01270_optimize_skip_unused_shards_low_cardinality
|
||||
01319_optimize_skip_unused_shards_nesting
|
||||
01353_low_cardinality_join_types
|
||||
01428_nullable_asof_join
|
||||
01455_shard_leaf_max_rows_bytes_to_read
|
||||
01476_right_full_join_switch
|
||||
01477_lc_in_merge_join_left_key
|
||||
01487_distributed_in_not_default_db
|
||||
01495_subqueries_in_with_statement
|
||||
01504_rocksdb
|
||||
01527_dist_sharding_key_dictGet_reload
|
||||
01528_allow_nondeterministic_optimize_skip_unused_shards
|
||||
01540_verbatim_partition_pruning
|
||||
01560_merge_distributed_join
|
||||
01563_distributed_query_finish
|
||||
01576_alias_column_rewrite
|
||||
01583_const_column_in_set_index
|
||||
01584_distributed_buffer_cannot_find_column
|
||||
01585_use_index_for_global_in
|
||||
01585_use_index_for_global_in_with_null
|
||||
01586_columns_pruning
|
||||
01615_random_one_shard_insertion
|
||||
01624_soft_constraints
|
||||
01651_bugs_from_15889
|
||||
01655_plan_optimizations
|
||||
01655_plan_optimizations_optimize_read_in_window_order
|
||||
01656_test_query_log_factories_info
|
||||
01681_bloom_filter_nullable_column
|
||||
01700_system_zookeeper_path_in
|
||||
01710_projection_additional_filters
|
||||
01721_join_implicit_cast_long
|
||||
01739_index_hint
|
||||
01747_join_view_filter_dictionary
|
||||
01748_partition_id_pruning
|
||||
01756_optimize_skip_unused_shards_rewrite_in
|
||||
01757_optimize_skip_unused_shards_limit
|
||||
01758_optimize_skip_unused_shards_once
|
||||
01759_optimize_skip_unused_shards_zero_shards
|
||||
01761_cast_to_enum_nullable
|
||||
01786_explain_merge_tree
|
||||
01889_key_condition_function_chains
|
||||
01890_materialized_distributed_join
|
||||
01901_in_literal_shard_prune
|
||||
01925_join_materialized_columns
|
||||
01925_test_storage_merge_aliases
|
||||
01930_optimize_skip_unused_shards_rewrite_in
|
||||
01947_mv_subquery
|
||||
01951_distributed_push_down_limit
|
||||
01952_optimize_distributed_group_by_sharding_key
|
||||
02000_join_on_const
|
||||
02001_shard_num_shard_count
|
||||
02024_join_on_or_long
|
||||
02131_used_row_policies_in_query_log
|
||||
02139_MV_with_scalar_subquery
|
||||
02174_cte_scalar_cache_mv
|
||||
02242_join_rocksdb
|
||||
02267_join_dup_columns_issue36199
|
||||
02302_s3_file_pruning
|
||||
02317_distinct_in_order_optimization_explain
|
||||
02341_global_join_cte
|
||||
02343_aggregation_pipeline
|
||||
02345_implicit_transaction
|
||||
02346_additional_filters_distr
|
||||
02352_grouby_shadows_arg
|
||||
02354_annoy
|
||||
02366_union_decimal_conversion
|
||||
02375_rocksdb_with_filters
|
||||
02377_optimize_sorting_by_input_stream_properties_explain
|
||||
02382_join_and_filtering_set
|
||||
02402_merge_engine_with_view
|
||||
02404_memory_bound_merging
|
||||
02421_decimal_in_precision_issue_41125
|
||||
02426_orc_bug
|
||||
02428_decimal_in_floating_point_literal
|
||||
02428_parameterized_view
|
||||
02458_use_structure_from_insertion_table
|
||||
02479_race_condition_between_insert_and_droppin_mv
|
||||
02481_merge_array_join_sample_by
|
||||
02493_inconsistent_hex_and_binary_number
|
||||
02494_optimize_group_by_function_keys_and_alias_columns
|
||||
02511_complex_literals_as_aggregate_function_parameters
|
||||
02521_aggregation_by_partitions
|
||||
02554_fix_grouping_sets_predicate_push_down
|
||||
02575_merge_prewhere_different_default_kind
|
||||
02713_array_low_cardinality_string
|
||||
02707_skip_index_with_in
|
||||
02707_complex_query_fails_analyzer
|
||||
02699_polygons_sym_difference_rollup
|
||||
02680_mysql_ast_logical_err
|
||||
02677_analyzer_bitmap_has_any
|
||||
02661_quantile_approx
|
||||
02540_duplicate_primary_key2
|
||||
02516_join_with_totals_and_subquery_bug
|
||||
02324_map_combinator_bug
|
||||
02241_join_rocksdb_bs
|
||||
02003_WithMergeableStateAfterAggregationAndLimit_LIMIT_BY_LIMIT_OFFSET
|
||||
01626_cnf_fuzz_long
|
||||
01115_join_with_dictionary
|
||||
01009_global_array_join_names
|
||||
00917_multiple_joins_denny_crane
|
||||
00725_join_on_bug_1
|
||||
00636_partition_key_parts_pruning
|
||||
00261_storage_aliases_and_array_join
|
||||
02701_non_parametric_function
|
||||
01825_type_json_multiple_files
|
||||
01281_group_by_limit_memory_tracking
|
||||
@ -64,7 +64,10 @@ def get_scales(runner_type: str) -> Tuple[int, int]:
|
||||
if runner_type == "style-checker":
|
||||
# the style checkers have so many noise, so it scales up too quickly
|
||||
scale_down = 1
|
||||
scale_up = 10
|
||||
# The 5 was too quick, there are complainings regarding too slow with
|
||||
# 10. I am trying 7 now.
|
||||
# UPDATE THE COMMENT ON CHANGES
|
||||
scale_up = 7
|
||||
return scale_down, scale_up
|
||||
|
||||
|
||||
|
||||
@ -259,6 +259,9 @@ CI_CONFIG = {
|
||||
"Stateless tests (release, wide parts enabled)": {
|
||||
"required_build": "package_release",
|
||||
},
|
||||
"Stateless tests (release, analyzer)": {
|
||||
"required_build": "package_release",
|
||||
},
|
||||
"Stateless tests (release, DatabaseOrdinary)": {
|
||||
"required_build": "package_release",
|
||||
},
|
||||
|
||||
@ -53,6 +53,8 @@ def get_additional_envs(check_name, run_by_hash_num, run_by_hash_total):
|
||||
result.append("USE_PARALLEL_REPLICAS=1")
|
||||
if "s3 storage" in check_name:
|
||||
result.append("USE_S3_STORAGE_FOR_MERGE_TREE=1")
|
||||
if "analyzer" in check_name:
|
||||
result.append("USE_NEW_ANALYZER=1")
|
||||
|
||||
if run_by_hash_total != 0:
|
||||
result.append(f"RUN_BY_HASH_NUM={run_by_hash_num}")
|
||||
@ -71,6 +73,7 @@ def get_image_name(check_name):
|
||||
|
||||
|
||||
def get_run_command(
|
||||
check_name,
|
||||
builds_path,
|
||||
repo_tests_path,
|
||||
result_path,
|
||||
@ -103,10 +106,16 @@ def get_run_command(
|
||||
envs += [f"-e {e}" for e in additional_envs]
|
||||
|
||||
env_str = " ".join(envs)
|
||||
volume_with_broken_test = (
|
||||
f"--volume={repo_tests_path}/broken_tests.txt:/broken_tests.txt"
|
||||
if "analyzer" in check_name
|
||||
else ""
|
||||
)
|
||||
|
||||
return (
|
||||
f"docker run --volume={builds_path}:/package_folder "
|
||||
f"--volume={repo_tests_path}:/usr/share/clickhouse-test "
|
||||
f"{volume_with_broken_test} "
|
||||
f"--volume={result_path}:/test_output --volume={server_log_path}:/var/log/clickhouse-server "
|
||||
f"--cap-add=SYS_PTRACE {env_str} {additional_options_str} {image}"
|
||||
)
|
||||
@ -322,6 +331,7 @@ def main():
|
||||
additional_envs.append("GLOBAL_TAGS=no-random-settings")
|
||||
|
||||
run_command = get_run_command(
|
||||
check_name,
|
||||
packages_path,
|
||||
repo_tests_path,
|
||||
result_path,
|
||||
|
||||
@ -142,6 +142,7 @@ if __name__ == "__main__":
|
||||
.replace("(", "_")
|
||||
.replace(")", "_")
|
||||
.replace(",", "_")
|
||||
.replace("/", "_")
|
||||
)
|
||||
|
||||
docker_image = get_image_with_version(reports_path, IMAGE_NAME)
|
||||
|
||||
@ -262,17 +262,20 @@ class ReportColorTheme:
|
||||
ColorTheme = Tuple[str, str, str]
|
||||
|
||||
|
||||
def _format_header(header, branch_name, branch_url=None):
|
||||
result = " ".join([w.capitalize() for w in header.split(" ")])
|
||||
def _format_header(
|
||||
header: str, branch_name: str, branch_url: Optional[str] = None
|
||||
) -> str:
|
||||
# Following line does not lower CI->Ci and SQLancer->Sqlancer. It only
|
||||
# capitalizes the first letter and doesn't touch the rest of the word
|
||||
result = " ".join([w[0].upper() + w[1:] for w in header.split(" ") if w])
|
||||
result = result.replace("Clickhouse", "ClickHouse")
|
||||
result = result.replace("clickhouse", "ClickHouse")
|
||||
if "ClickHouse" not in result:
|
||||
result = "ClickHouse " + result
|
||||
result += " for "
|
||||
result = f"ClickHouse {result}"
|
||||
if branch_url:
|
||||
result += f'<a href="{branch_url}">{branch_name}</a>'
|
||||
result = f'{result} for <a href="{branch_url}">{branch_name}</a>'
|
||||
else:
|
||||
result += branch_name
|
||||
result = f"{result} for {branch_name}"
|
||||
return result
|
||||
|
||||
|
||||
|
||||
@ -59,9 +59,10 @@ def upload_results(
|
||||
additional_files: List[str],
|
||||
check_name: str,
|
||||
) -> str:
|
||||
s3_path_prefix = f"{pr_number}/{commit_sha}/" + check_name.lower().replace(
|
||||
" ", "_"
|
||||
).replace("(", "_").replace(")", "_").replace(",", "_")
|
||||
normalized_check_name = check_name.lower()
|
||||
for r in ((" ", "_"), ("(", "_"), (")", "_"), (",", "_"), ("/", "_")):
|
||||
normalized_check_name = normalized_check_name.replace(*r)
|
||||
s3_path_prefix = f"{pr_number}/{commit_sha}/{normalized_check_name}"
|
||||
additional_urls = process_logs(
|
||||
s3_client, additional_files, s3_path_prefix, test_results
|
||||
)
|
||||
|
||||
@ -284,10 +284,12 @@ def handler(event: dict, _: Any) -> dict:
|
||||
wf_job["runner_group_name"] or "", # nullable
|
||||
repo["full_name"],
|
||||
)
|
||||
logging.info(
|
||||
"Got the next event (private_repo=%s): %s", repo["private"], workflow_job
|
||||
)
|
||||
if repo["private"]:
|
||||
workflow_job.anonimyze()
|
||||
|
||||
logging.info("Got the next event: %s", workflow_job)
|
||||
send_event_workflow_job(workflow_job)
|
||||
|
||||
return {
|
||||
|
||||
@ -579,10 +579,17 @@ class SettingsRandomizer:
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def get_random_settings():
|
||||
def get_random_settings(args):
|
||||
random_settings = []
|
||||
is_debug = BuildFlags.DEBUG in args.build_flags
|
||||
for setting, generator in SettingsRandomizer.settings.items():
|
||||
random_settings.append(f"{setting}={generator()}")
|
||||
if (
|
||||
is_debug
|
||||
and setting == "allow_prefetched_read_pool_for_remote_filesystem"
|
||||
):
|
||||
random_settings.append(f"{setting}=0")
|
||||
else:
|
||||
random_settings.append(f"{setting}={generator()}")
|
||||
return random_settings
|
||||
|
||||
|
||||
@ -817,7 +824,7 @@ class TestCase:
|
||||
)
|
||||
|
||||
if self.randomize_settings:
|
||||
self.random_settings = SettingsRandomizer.get_random_settings()
|
||||
self.random_settings = SettingsRandomizer.get_random_settings(args)
|
||||
|
||||
if self.randomize_merge_tree_settings:
|
||||
self.merge_tree_random_settings = (
|
||||
|
||||
@ -79,6 +79,10 @@ ln -sf $SRC_PATH/users.d/marks.xml $DEST_SERVER_PATH/users.d/
|
||||
ln -sf $SRC_PATH/users.d/insert_keeper_retries.xml $DEST_SERVER_PATH/users.d/
|
||||
ln -sf $SRC_PATH/users.d/prefetch_settings.xml $DEST_SERVER_PATH/users.d/
|
||||
|
||||
if [[ -n "$USE_NEW_ANALYZER" ]] && [[ "$USE_NEW_ANALYZER" -eq 1 ]]; then
|
||||
ln -sf $SRC_PATH/users.d/analyzer.xml $DEST_SERVER_PATH/users.d/
|
||||
fi
|
||||
|
||||
# FIXME DataPartsExchange may hang for http_send_timeout seconds
|
||||
# when nobody is going to read from the other side of socket (due to "Fetching of part was cancelled"),
|
||||
# but socket is owned by HTTPSessionPool, so it's not closed.
|
||||
|
||||
7
tests/config/users.d/analyzer.xml
Normal file
7
tests/config/users.d/analyzer.xml
Normal file
@ -0,0 +1,7 @@
|
||||
<clickhouse>
|
||||
<profiles>
|
||||
<default>
|
||||
<allow_experimental_analyzer>1</allow_experimental_analyzer>
|
||||
</default>
|
||||
</profiles>
|
||||
</clickhouse>
|
||||
@ -20,6 +20,9 @@ def start_cluster():
|
||||
instance.query(
|
||||
"CREATE TABLE test.table(x UInt32, y UInt32) ENGINE = MergeTree ORDER BY tuple()"
|
||||
)
|
||||
instance.query(
|
||||
"CREATE TABLE test.table2(x UInt32, y UInt32) ENGINE = MergeTree ORDER BY tuple()"
|
||||
)
|
||||
instance.query("INSERT INTO test.table VALUES (1,5), (2,10)")
|
||||
|
||||
yield cluster
|
||||
@ -585,3 +588,134 @@ def test_grant_with_replace_option():
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
["GRANT INSERT ON test.table TO B"]
|
||||
)
|
||||
|
||||
|
||||
def test_grant_current_grants():
|
||||
instance.query("CREATE USER A")
|
||||
instance.query(
|
||||
"GRANT SELECT, CREATE TABLE, CREATE VIEW ON test.* TO A WITH GRANT OPTION"
|
||||
)
|
||||
assert instance.query("SHOW GRANTS FOR A") == TSV(
|
||||
["GRANT SELECT, CREATE TABLE, CREATE VIEW ON test.* TO A WITH GRANT OPTION"]
|
||||
)
|
||||
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO B", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
["GRANT SELECT, CREATE TABLE, CREATE VIEW ON test.* TO B"]
|
||||
)
|
||||
|
||||
instance.query("CREATE USER C")
|
||||
instance.query("GRANT CURRENT GRANTS(CREATE ON test.*) TO C", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR C") == TSV(
|
||||
["GRANT CREATE TABLE, CREATE VIEW ON test.* TO C"]
|
||||
)
|
||||
|
||||
instance.query("DROP USER IF EXISTS C")
|
||||
instance.query("CREATE USER C")
|
||||
instance.query("GRANT CURRENT GRANTS(NONE ON *.*) TO C", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR C") == TSV([])
|
||||
|
||||
|
||||
def test_grant_current_grants_with_partial_revoke():
|
||||
instance.query("CREATE USER A")
|
||||
instance.query("GRANT CREATE TABLE ON *.* TO A")
|
||||
instance.query("REVOKE CREATE TABLE ON test.* FROM A")
|
||||
instance.query("GRANT CREATE TABLE ON test.table TO A WITH GRANT OPTION")
|
||||
instance.query("GRANT SELECT ON *.* TO A WITH GRANT OPTION")
|
||||
instance.query("REVOKE SELECT ON test.* FROM A")
|
||||
instance.query("GRANT SELECT ON test.table TO A WITH GRANT OPTION")
|
||||
instance.query("GRANT SELECT ON test.table2 TO A")
|
||||
|
||||
assert instance.query("SHOW GRANTS FOR A") == TSV(
|
||||
[
|
||||
"GRANT CREATE TABLE ON *.* TO A",
|
||||
"GRANT SELECT ON *.* TO A WITH GRANT OPTION",
|
||||
"REVOKE SELECT, CREATE TABLE ON test.* FROM A",
|
||||
"GRANT SELECT, CREATE TABLE ON test.table TO A WITH GRANT OPTION",
|
||||
"GRANT SELECT ON test.table2 TO A",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO B", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO B",
|
||||
"REVOKE SELECT ON test.* FROM B",
|
||||
"GRANT SELECT, CREATE TABLE ON test.table TO B",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("DROP USER IF EXISTS B")
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO B WITH GRANT OPTION", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO B WITH GRANT OPTION",
|
||||
"REVOKE SELECT ON test.* FROM B",
|
||||
"GRANT SELECT, CREATE TABLE ON test.table TO B WITH GRANT OPTION",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("DROP USER IF EXISTS C")
|
||||
instance.query("CREATE USER C")
|
||||
instance.query("GRANT SELECT ON test.* TO B")
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO C", user="B")
|
||||
assert instance.query("SHOW GRANTS FOR C") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO C",
|
||||
"GRANT CREATE TABLE ON test.table TO C",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("DROP USER IF EXISTS B")
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT CURRENT GRANTS ON test.* TO B WITH GRANT OPTION", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
[
|
||||
"GRANT SELECT, CREATE TABLE ON test.table TO B WITH GRANT OPTION",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def test_current_grants_override():
|
||||
instance.query("CREATE USER A")
|
||||
instance.query("GRANT SELECT ON *.* TO A WITH GRANT OPTION")
|
||||
instance.query("REVOKE SELECT ON test.* FROM A")
|
||||
assert instance.query("SHOW GRANTS FOR A") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO A WITH GRANT OPTION",
|
||||
"REVOKE SELECT ON test.* FROM A",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT SELECT ON test.table TO B")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
["GRANT SELECT ON test.table TO B"]
|
||||
)
|
||||
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO B", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO B",
|
||||
"REVOKE SELECT ON test.* FROM B",
|
||||
"GRANT SELECT ON test.table TO B",
|
||||
]
|
||||
)
|
||||
|
||||
instance.query("DROP USER IF EXISTS B")
|
||||
instance.query("CREATE USER B")
|
||||
instance.query("GRANT SELECT ON test.table TO B")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
["GRANT SELECT ON test.table TO B"]
|
||||
)
|
||||
|
||||
instance.query("GRANT CURRENT GRANTS ON *.* TO B WITH REPLACE OPTION", user="A")
|
||||
assert instance.query("SHOW GRANTS FOR B") == TSV(
|
||||
[
|
||||
"GRANT SELECT ON *.* TO B",
|
||||
"REVOKE SELECT ON test.* FROM B",
|
||||
]
|
||||
)
|
||||
|
||||
@ -327,6 +327,17 @@ def test_predefined_connection_configuration(started_cluster):
|
||||
node1.query(f"SELECT count() FROM postgres_database.test_table").rstrip()
|
||||
== "100"
|
||||
)
|
||||
node1.query(
|
||||
"""
|
||||
DROP DATABASE postgres_database;
|
||||
CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres1, use_tables_cache=1);
|
||||
"""
|
||||
)
|
||||
assert (
|
||||
node1.query(f"SELECT count() FROM postgres_database.test_table").rstrip()
|
||||
== "100"
|
||||
)
|
||||
assert node1.contains_in_log("Cached table `test_table`")
|
||||
|
||||
node1.query("DROP DATABASE postgres_database")
|
||||
cursor.execute(f"DROP TABLE test_table ")
|
||||
|
||||
@ -4,10 +4,5 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
# shellcheck source=../shell_config.sh
|
||||
. "$CURDIR"/../shell_config.sh
|
||||
|
||||
# use big-endian version of binary data for s390x
|
||||
if [[ $(uname -a | grep s390x) ]]; then
|
||||
echo -ne '\xdb\x8a\xe9\x59\xf2\x32\x74\x50\x39\xc4\x22\xfb\xa7\x4a\xc6\x37''\x82\x13\x00\x00\x00\x09\x00\x00\x00''\x90SELECT 1\n' | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&decompress=1" --data-binary @-
|
||||
else
|
||||
echo -ne '\x50\x74\x32\xf2\x59\xe9\x8a\xdb\x37\xc6\x4a\xa7\xfb\x22\xc4\x39''\x82\x13\x00\x00\x00\x09\x00\x00\x00''\x90SELECT 1\n' | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&decompress=1" --data-binary @-
|
||||
fi
|
||||
echo -ne 'xxxxxxxxxxxxxxxx''\x82\x13\x00\x00\x00\x09\x00\x00\x00''\x90SELECT 1\n' | ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&decompress=1&http_native_compression_disable_checksumming_on_decompress=1" --data-binary @-
|
||||
|
||||
@ -19,3 +19,7 @@
|
||||
1 10 000000000000F03F
|
||||
-1 11 000000000000F0BF
|
||||
inf 11 000000000000F07F
|
||||
Hello, world!!!! 55
|
||||
67
|
||||
67
|
||||
1
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user