diff --git a/CHANGELOG.md b/CHANGELOG.md index 34d11c6a2cd..c7c054a53a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,114 @@ +### ClickHouse release v21.8, 2021-08-11 + +#### New Features + +* Collect common system metrics (in `system.asynchronous_metrics` and `system.asynchronous_metric_log`) on CPU usage, disk usage, memory usage, IO, network, files, load average, CPU frequencies, thermal sensors, EDAC counters, system uptime; also added metrics about the scheduling jitter and the time spent collecting the metrics. It works similar to `atop` in ClickHouse and allows access to monitoring data even if you have no additional tools installed. Close [#9430](https://github.com/ClickHouse/ClickHouse/issues/9430). [#24416](https://github.com/ClickHouse/ClickHouse/pull/24416) ([Yegor Levankov](https://github.com/elevankoff)). +* Add new functions `leftPad()`, `rightPad()`, `leftPadUTF8()`, `rightPadUTF8()`. [#26075](https://github.com/ClickHouse/ClickHouse/pull/26075) ([Vitaly Baranov](https://github.com/vitlibar)). +* Add the `FIRST` keyword to the `ADD INDEX` command to be able to add the index at the beginning of the indices list. [#25904](https://github.com/ClickHouse/ClickHouse/pull/25904) ([xjewer](https://github.com/xjewer)). +* Introduce `system.data_skipping_indices` table containing information about existing data skipping indices. Close [#7659](https://github.com/ClickHouse/ClickHouse/issues/7659). [#25693](https://github.com/ClickHouse/ClickHouse/pull/25693) ([Dmitry Novik](https://github.com/novikd)). +* Add `bin`/`unbin` functions. [#25609](https://github.com/ClickHouse/ClickHouse/pull/25609) ([zhaoyu](https://github.com/zxc111)). +* Support `Map` and `(U)Int128`, `U(Int256) types in `mapAdd` and `mapSubtract` functions. [#25596](https://github.com/ClickHouse/ClickHouse/pull/25596) ([Ildus Kurbangaliev](https://github.com/ildus)). +* Support `DISTINCT ON (columns)` expression, close [#25404](https://github.com/ClickHouse/ClickHouse/issues/25404). [#25589](https://github.com/ClickHouse/ClickHouse/pull/25589) ([Zijie Lu](https://github.com/TszKitLo40)). +* Add support for a part of SQLJSON standard. [#24148](https://github.com/ClickHouse/ClickHouse/pull/24148) ([l1tsolaiki](https://github.com/l1tsolaiki)). +* Add MaterializedPostgreSQL table engine and database engine. This database engine allows replicating a whole database or any subset of database tables. [#20470](https://github.com/ClickHouse/ClickHouse/pull/20470) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add an ability to reset a custom setting to default and remove it from the table's metadata. It allows rolling back the change without knowing the system/config's default. Closes [#14449](https://github.com/ClickHouse/ClickHouse/issues/14449). [#17769](https://github.com/ClickHouse/ClickHouse/pull/17769) ([xjewer](https://github.com/xjewer)). +* Render pipelines as graphs in Web UI if `EXPLAIN PIPELINE graph = 1` query is submitted. [#26067](https://github.com/ClickHouse/ClickHouse/pull/26067) ([alexey-milovidov](https://github.com/alexey-milovidov)). + +#### Performance Improvements + +* Compile aggregate functions. Use option `compile_aggregate_expressions` to enable it. [#24789](https://github.com/ClickHouse/ClickHouse/pull/24789) ([Maksim Kita](https://github.com/kitaisreal)). +* Improve latency of short queries that require reading from tables with many columns. [#26371](https://github.com/ClickHouse/ClickHouse/pull/26371) ([Anton Popov](https://github.com/CurtizJ)). + +#### Improvements + +* Use `Map` data type for system logs tables (`system.query_log`, `system.query_thread_log`, `system.processes`, `system.opentelemetry_span_log`). These tables will be auto-created with new data types. Virtual columns are created to support old queries. Closes [#18698](https://github.com/ClickHouse/ClickHouse/issues/18698). [#23934](https://github.com/ClickHouse/ClickHouse/pull/23934), [#25773](https://github.com/ClickHouse/ClickHouse/pull/25773) ([hexiaoting](https://github.com/hexiaoting), [sundy-li](https://github.com/sundy-li)). +* For a dictionary with a complex key containing only one attribute, allow not wrapping the key expression in tuple for functions `dictGet`, `dictHas`. [#26130](https://github.com/ClickHouse/ClickHouse/pull/26130) ([Maksim Kita](https://github.com/kitaisreal)). +* Implement function `bin`/`hex` from `AggregateFunction` states. [#26094](https://github.com/ClickHouse/ClickHouse/pull/26094) ([zhaoyu](https://github.com/zxc111)). +* Support arguments of `UUID` type for `empty` and `notEmpty` functions. `UUID` is empty if it is all zeros (nil UUID). Closes [#3446](https://github.com/ClickHouse/ClickHouse/issues/3446). [#25974](https://github.com/ClickHouse/ClickHouse/pull/25974) ([zhaoyu](https://github.com/zxc111)). +* Fix error with query `SET SQL_SELECT_LIMIT` in MySQL protocol. Closes [#17115](https://github.com/ClickHouse/ClickHouse/issues/17115). [#25972](https://github.com/ClickHouse/ClickHouse/pull/25972) ([Kseniia Sumarokova](https://github.com/kssenii)). +* More instrumentation for network interaction: add counters for recv/send bytes; add gauges for recvs/sends. Added missing documentation. Close [#5897](https://github.com/ClickHouse/ClickHouse/issues/5897). [#25962](https://github.com/ClickHouse/ClickHouse/pull/25962) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add setting `optimize_move_to_prewhere_if_final`. If query has `FINAL`, the optimization `move_to_prewhere` will be enabled only if both `optimize_move_to_prewhere` and `optimize_move_to_prewhere_if_final` are enabled. Closes [#8684](https://github.com/ClickHouse/ClickHouse/issues/8684). [#25940](https://github.com/ClickHouse/ClickHouse/pull/25940) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Allow complex quoted identifiers of JOINed tables. Close [#17861](https://github.com/ClickHouse/ClickHouse/issues/17861). [#25924](https://github.com/ClickHouse/ClickHouse/pull/25924) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add support for Unicode (e.g. Chinese, Cyrillic) components in `Nested` data types. Close [#25594](https://github.com/ClickHouse/ClickHouse/issues/25594). [#25923](https://github.com/ClickHouse/ClickHouse/pull/25923) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Allow `quantiles*` functions to work with `aggregate_functions_null_for_empty`. Close [#25892](https://github.com/ClickHouse/ClickHouse/issues/25892). [#25919](https://github.com/ClickHouse/ClickHouse/pull/25919) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Allow parameters for parametric aggregate functions to be arbitrary constant expressions (e.g., `1 + 2`), not just literals. It also allows using the query parameters (in parameterized queries like `{param:UInt8}`) inside parametric aggregate functions. Closes [#11607](https://github.com/ClickHouse/ClickHouse/issues/11607). [#25910](https://github.com/ClickHouse/ClickHouse/pull/25910) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Correctly throw the exception on the attempt to parse an invalid `Date`. Closes [#6481](https://github.com/ClickHouse/ClickHouse/issues/6481). [#25909](https://github.com/ClickHouse/ClickHouse/pull/25909) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Support for multiple includes in configuration. It is possible to include users configuration, remote server configuration from multiple sources. Simply place `` element with `from_zk`, `from_env` or `incl` attribute, and it will be replaced with the substitution. [#24404](https://github.com/ClickHouse/ClickHouse/pull/24404) ([nvartolomei](https://github.com/nvartolomei)). +* Support for queries with a column named `"null"` (it must be specified in back-ticks or double quotes) and `ON CLUSTER`. Closes [#24035](https://github.com/ClickHouse/ClickHouse/issues/24035). [#25907](https://github.com/ClickHouse/ClickHouse/pull/25907) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Support `LowCardinality`, `Decimal`, and `UUID` for `JSONExtract`. Closes [#24606](https://github.com/ClickHouse/ClickHouse/issues/24606). [#25900](https://github.com/ClickHouse/ClickHouse/pull/25900) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Convert history file from `readline` format to `replxx` format. [#25888](https://github.com/ClickHouse/ClickHouse/pull/25888) ([Azat Khuzhin](https://github.com/azat)). +* Fix bug which can lead to intersecting parts after `DROP PART` or background deletion of an empty part. [#25884](https://github.com/ClickHouse/ClickHouse/pull/25884) ([alesapin](https://github.com/alesapin)). +* Better handling of lost parts for `ReplicatedMergeTree` tables. Fixes rare inconsistencies in `ReplicationQueue`. Fixes [#10368](https://github.com/ClickHouse/ClickHouse/issues/10368). [#25820](https://github.com/ClickHouse/ClickHouse/pull/25820) ([alesapin](https://github.com/alesapin)). +* Allow starting clickhouse-client with unreadable working directory. [#25817](https://github.com/ClickHouse/ClickHouse/pull/25817) ([ianton-ru](https://github.com/ianton-ru)). +* Fix "No available columns" error for `Merge` storage. [#25801](https://github.com/ClickHouse/ClickHouse/pull/25801) ([Azat Khuzhin](https://github.com/azat)). +* MySQL Engine now supports the exchange of column comments between MySQL and ClickHouse. [#25795](https://github.com/ClickHouse/ClickHouse/pull/25795) ([Storozhuk Kostiantyn](https://github.com/sand6255)). +* Fix inconsistent behaviour of `GROUP BY` constant on empty set. Closes [#6842](https://github.com/ClickHouse/ClickHouse/issues/6842). [#25786](https://github.com/ClickHouse/ClickHouse/pull/25786) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Cancel already running merges in partition on `DROP PARTITION` and `TRUNCATE` for `ReplicatedMergeTree`. Resolves [#17151](https://github.com/ClickHouse/ClickHouse/issues/17151). [#25684](https://github.com/ClickHouse/ClickHouse/pull/25684) ([tavplubix](https://github.com/tavplubix)). +* Support ENUM` data type for MaterializeMySQL. [#25676](https://github.com/ClickHouse/ClickHouse/pull/25676) ([Storozhuk Kostiantyn](https://github.com/sand6255)). +* Support materialized and aliased columns in JOIN, close [#13274](https://github.com/ClickHouse/ClickHouse/issues/13274). [#25634](https://github.com/ClickHouse/ClickHouse/pull/25634) ([Vladimir C](https://github.com/vdimir)). +* Fix possible logical race condition between `ALTER TABLE ... DETACH` and background merges. [#25605](https://github.com/ClickHouse/ClickHouse/pull/25605) ([Azat Khuzhin](https://github.com/azat)). +* Make `NetworkReceiveElapsedMicroseconds` metric to correctly include the time spent waiting for data from the client to `INSERT`. Close [#9958](https://github.com/ClickHouse/ClickHouse/issues/9958). [#25602](https://github.com/ClickHouse/ClickHouse/pull/25602) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Support `TRUNCATE TABLE` for StorageS3 and StorageHDFS. Close [#25530](https://github.com/ClickHouse/ClickHouse/issues/25530). [#25550](https://github.com/ClickHouse/ClickHouse/pull/25550) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support for dynamic reloading of config to change number of threads in pool for background jobs execution (merges, mutations, fetches). [#25548](https://github.com/ClickHouse/ClickHouse/pull/25548) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Allow extracting of non-string element as string using `JSONExtract`. This is for [#25414](https://github.com/ClickHouse/ClickHouse/issues/25414). [#25452](https://github.com/ClickHouse/ClickHouse/pull/25452) ([Amos Bird](https://github.com/amosbird)). +* Support regular expression in `Database` argument for `StorageMerge`. Close [#776](https://github.com/ClickHouse/ClickHouse/issues/776). [#25064](https://github.com/ClickHouse/ClickHouse/pull/25064) ([flynn](https://github.com/ucasfl)). +* Web UI: if the value looks like a URL, automatically generate a link. [#25965](https://github.com/ClickHouse/ClickHouse/pull/25965) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Make `sudo service clickhouse-server start` to work on systems with `systemd` like Centos 8. Close [#14298](https://github.com/ClickHouse/ClickHouse/issues/14298). Close [#17799](https://github.com/ClickHouse/ClickHouse/issues/17799). [#25921](https://github.com/ClickHouse/ClickHouse/pull/25921) ([alexey-milovidov](https://github.com/alexey-milovidov)). + +#### Bug Fixes + +* Fix incorrect `SET ROLE` in some cases. [#26707](https://github.com/ClickHouse/ClickHouse/pull/26707) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix potential `nullptr` dereference in window functions. Fix [#25276](https://github.com/ClickHouse/ClickHouse/issues/25276). [#26668](https://github.com/ClickHouse/ClickHouse/pull/26668) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Fix incorrect function names of `groupBitmapAnd/Or/Xor`. Fix [#26557](https://github.com/ClickHouse/ClickHouse/pull/26557) ([Amos Bird](https://github.com/amosbird)). +* Fix crash in rabbitmq shutdown in case rabbitmq setup was not started. Closes [#26504](https://github.com/ClickHouse/ClickHouse/issues/26504). [#26529](https://github.com/ClickHouse/ClickHouse/pull/26529) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix issues with `CREATE DICTIONARY` query if dictionary name or database name was quoted. Closes [#26491](https://github.com/ClickHouse/ClickHouse/issues/26491). [#26508](https://github.com/ClickHouse/ClickHouse/pull/26508) ([Maksim Kita](https://github.com/kitaisreal)). +* Fix broken name resolution after rewriting column aliases. Fix [#26432](https://github.com/ClickHouse/ClickHouse/issues/26432). [#26475](https://github.com/ClickHouse/ClickHouse/pull/26475) ([Amos Bird](https://github.com/amosbird)). +* Fix infinite non-joined block stream in `partial_merge_join` close [#26325](https://github.com/ClickHouse/ClickHouse/issues/26325). [#26374](https://github.com/ClickHouse/ClickHouse/pull/26374) ([Vladimir C](https://github.com/vdimir)). +* Fix possible crash when login as dropped user. Fix [#26073](https://github.com/ClickHouse/ClickHouse/issues/26073). [#26363](https://github.com/ClickHouse/ClickHouse/pull/26363) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix `optimize_distributed_group_by_sharding_key` for multiple columns (leads to incorrect result w/ `optimize_skip_unused_shards=1`/`allow_nondeterministic_optimize_skip_unused_shards=1` and multiple columns in sharding key expression). [#26353](https://github.com/ClickHouse/ClickHouse/pull/26353) ([Azat Khuzhin](https://github.com/azat)). + * `CAST` from `Date` to `DateTime` (or `DateTime64`) was not using the timezone of the `DateTime` type. It can also affect the comparison between `Date` and `DateTime`. Inference of the common type for `Date` and `DateTime` also was not using the corresponding timezone. It affected the results of function `if` and array construction. Closes [#24128](https://github.com/ClickHouse/ClickHouse/issues/24128). [#24129](https://github.com/ClickHouse/ClickHouse/pull/24129) ([Maksim Kita](https://github.com/kitaisreal)). +* Fixed rare bug in lost replica recovery that may cause replicas to diverge. [#26321](https://github.com/ClickHouse/ClickHouse/pull/26321) ([tavplubix](https://github.com/tavplubix)). +* Fix zstd decompression in case there are escape sequences at the end of internal buffer. Closes [#26013](https://github.com/ClickHouse/ClickHouse/issues/26013). [#26314](https://github.com/ClickHouse/ClickHouse/pull/26314) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix logical error on join with totals, close [#26017](https://github.com/ClickHouse/ClickHouse/issues/26017). [#26250](https://github.com/ClickHouse/ClickHouse/pull/26250) ([Vladimir C](https://github.com/vdimir)). +* Remove excessive newline in `thread_name` column in `system.stack_trace` table. Fix [#24124](https://github.com/ClickHouse/ClickHouse/issues/24124). [#26210](https://github.com/ClickHouse/ClickHouse/pull/26210) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix `joinGet` with `LowCarinality` columns, close [#25993](https://github.com/ClickHouse/ClickHouse/issues/25993). [#26118](https://github.com/ClickHouse/ClickHouse/pull/26118) ([Vladimir C](https://github.com/vdimir)). +* Fix possible crash in `pointInPolygon` if the setting `validate_polygons` is turned off. [#26113](https://github.com/ClickHouse/ClickHouse/pull/26113) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix throwing exception when iterate over non-existing remote directory. [#26087](https://github.com/ClickHouse/ClickHouse/pull/26087) ([ianton-ru](https://github.com/ianton-ru)). +* Fix rare server crash because of `abort` in ZooKeeper client. Fixes [#25813](https://github.com/ClickHouse/ClickHouse/issues/25813). [#26079](https://github.com/ClickHouse/ClickHouse/pull/26079) ([alesapin](https://github.com/alesapin)). +* Fix wrong thread estimation for right subquery join in some cases. Close [#24075](https://github.com/ClickHouse/ClickHouse/issues/24075). [#26052](https://github.com/ClickHouse/ClickHouse/pull/26052) ([Vladimir C](https://github.com/vdimir)). +* Fixed incorrect `sequence_id` in MySQL protocol packets that ClickHouse sends on exception during query execution. It might cause MySQL client to reset connection to ClickHouse server. Fixes [#21184](https://github.com/ClickHouse/ClickHouse/issues/21184). [#26051](https://github.com/ClickHouse/ClickHouse/pull/26051) ([tavplubix](https://github.com/tavplubix)). +* Fix possible mismatched header when using normal projection with `PREWHERE`. Fix [#26020](https://github.com/ClickHouse/ClickHouse/issues/26020). [#26038](https://github.com/ClickHouse/ClickHouse/pull/26038) ([Amos Bird](https://github.com/amosbird)). +* Fix formatting of type `Map` with integer keys to `JSON`. [#25982](https://github.com/ClickHouse/ClickHouse/pull/25982) ([Anton Popov](https://github.com/CurtizJ)). +* Fix possible deadlock during query profiler stack unwinding. Fix [#25968](https://github.com/ClickHouse/ClickHouse/issues/25968). [#25970](https://github.com/ClickHouse/ClickHouse/pull/25970) ([Maksim Kita](https://github.com/kitaisreal)). +* Fix crash on call `dictGet()` with bad arguments. [#25913](https://github.com/ClickHouse/ClickHouse/pull/25913) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fixed `scram-sha-256` authentication for PostgreSQL engines. Closes [#24516](https://github.com/ClickHouse/ClickHouse/issues/24516). [#25906](https://github.com/ClickHouse/ClickHouse/pull/25906) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix extremely long backoff for background tasks when the background pool is full. Fixes [#25836](https://github.com/ClickHouse/ClickHouse/issues/25836). [#25893](https://github.com/ClickHouse/ClickHouse/pull/25893) ([alesapin](https://github.com/alesapin)). +* Fix ARM exception handling with non default page size. Fixes [#25512](https://github.com/ClickHouse/ClickHouse/issues/25512), [#25044](https://github.com/ClickHouse/ClickHouse/issues/25044), [#24901](https://github.com/ClickHouse/ClickHouse/issues/24901), [#23183](https://github.com/ClickHouse/ClickHouse/issues/23183), [#20221](https://github.com/ClickHouse/ClickHouse/issues/20221), [#19703](https://github.com/ClickHouse/ClickHouse/issues/19703), [#19028](https://github.com/ClickHouse/ClickHouse/issues/19028), [#18391](https://github.com/ClickHouse/ClickHouse/issues/18391), [#18121](https://github.com/ClickHouse/ClickHouse/issues/18121), [#17994](https://github.com/ClickHouse/ClickHouse/issues/17994), [#12483](https://github.com/ClickHouse/ClickHouse/issues/12483). [#25854](https://github.com/ClickHouse/ClickHouse/pull/25854) ([Maksim Kita](https://github.com/kitaisreal)). +* Fix sharding_key from column w/o function for `remote()` (before `select * from remote('127.1', system.one, dummy)` leads to `Unknown column: dummy, there are only columns .` error). [#25824](https://github.com/ClickHouse/ClickHouse/pull/25824) ([Azat Khuzhin](https://github.com/azat)). +* Fixed `Not found column ...` and `Missing column ...` errors when selecting from `MaterializeMySQL`. Fixes [#23708](https://github.com/ClickHouse/ClickHouse/issues/23708), [#24830](https://github.com/ClickHouse/ClickHouse/issues/24830), [#25794](https://github.com/ClickHouse/ClickHouse/issues/25794). [#25822](https://github.com/ClickHouse/ClickHouse/pull/25822) ([tavplubix](https://github.com/tavplubix)). +* Fix `optimize_skip_unused_shards_rewrite_in` for non-UInt64 types (may select incorrect shards eventually or throw `Cannot infer type of an empty tuple` or `Function tuple requires at least one argument`). [#25798](https://github.com/ClickHouse/ClickHouse/pull/25798) ([Azat Khuzhin](https://github.com/azat)). +* Fix rare bug with `DROP PART` query for `ReplicatedMergeTree` tables which can lead to error message `Unexpected merged part intersecting drop range`. [#25783](https://github.com/ClickHouse/ClickHouse/pull/25783) ([alesapin](https://github.com/alesapin)). +* Fix bug in `TTL` with `GROUP BY` expression which refuses to execute `TTL` after first execution in part. [#25743](https://github.com/ClickHouse/ClickHouse/pull/25743) ([alesapin](https://github.com/alesapin)). +* Allow StorageMerge to access tables with aliases. Closes [#6051](https://github.com/ClickHouse/ClickHouse/issues/6051). [#25694](https://github.com/ClickHouse/ClickHouse/pull/25694) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix slow dict join in some cases, close [#24209](https://github.com/ClickHouse/ClickHouse/issues/24209). [#25618](https://github.com/ClickHouse/ClickHouse/pull/25618) ([Vladimir C](https://github.com/vdimir)). +* Fix `ALTER MODIFY COLUMN` of columns, which participates in TTL expressions. [#25554](https://github.com/ClickHouse/ClickHouse/pull/25554) ([Anton Popov](https://github.com/CurtizJ)). +* Fix assertion in `PREWHERE` with non-UInt8 type, close [#19589](https://github.com/ClickHouse/ClickHouse/issues/19589). [#25484](https://github.com/ClickHouse/ClickHouse/pull/25484) ([Vladimir C](https://github.com/vdimir)). +* Fix some fuzzed msan crash. Fixes [#22517](https://github.com/ClickHouse/ClickHouse/issues/22517). [#26428](https://github.com/ClickHouse/ClickHouse/pull/26428) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix empty history file conversion. [#26589](https://github.com/ClickHouse/ClickHouse/pull/26589) ([Azat Khuzhin](https://github.com/azat)). +* Update `chown` cmd check in `clickhouse-server` docker entrypoint. It fixes error 'cluster pod restart failed (or timeout)' on kubernetes. [#26545](https://github.com/ClickHouse/ClickHouse/pull/26545) ([Ky Li](https://github.com/Kylinrix)). + +#### Build/Testing/Packaging Improvements + +* Disabling TestFlows LDAP module due to test fails. [#26065](https://github.com/ClickHouse/ClickHouse/pull/26065) ([vzakaznikov](https://github.com/vzakaznikov)). +* Enabling all TestFlows modules and fixing some tests. [#26011](https://github.com/ClickHouse/ClickHouse/pull/26011) ([vzakaznikov](https://github.com/vzakaznikov)). +* Add new tests for checking access rights for columns used in filters (`WHERE` / `PREWHERE` / row policy) of the `SELECT` statement after changes in [#24405](https://github.com/ClickHouse/ClickHouse/pull/24405). [#25619](https://github.com/ClickHouse/ClickHouse/pull/25619) ([Vitaly Baranov](https://github.com/vitlibar)). + +#### Other + +* Add `clickhouse-keeper-converter` tool which allows converting zookeeper logs and snapshots into `clickhouse-keeper` snapshot format. [#25428](https://github.com/ClickHouse/ClickHouse/pull/25428) ([alesapin](https://github.com/alesapin)). + + + ### ClickHouse release v21.7, 2021-07-09 #### Backward Incompatible Change diff --git a/CMakeLists.txt b/CMakeLists.txt index d3cb5f70c83..1727caea766 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -271,12 +271,6 @@ endif() include(cmake/cpu_features.cmake) -option(ARCH_NATIVE "Add -march=native compiler flag. This makes your binaries non-portable but more performant code may be generated.") - -if (ARCH_NATIVE) - set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native") -endif () - # Asynchronous unwind tables are needed for Query Profiler. # They are already by default on some platforms but possibly not on all platforms. # Enable it explicitly. diff --git a/cmake/cpu_features.cmake b/cmake/cpu_features.cmake index d12eac2e3c4..46e42329958 100644 --- a/cmake/cpu_features.cmake +++ b/cmake/cpu_features.cmake @@ -5,109 +5,128 @@ include (CMakePushCheckState) cmake_push_check_state () -# gcc -dM -E -mno-sse2 - < /dev/null | sort > gcc-dump-nosse2 -# gcc -dM -E -msse2 - < /dev/null | sort > gcc-dump-sse2 -#define __SSE2__ 1 -#define __SSE2_MATH__ 1 +# The variables HAVE_* determine if compiler has support for the flag to use the corresponding instruction set. +# The options ENABLE_* determine if we will tell compiler to actually use the corresponding instruction set if compiler can do it. -# gcc -dM -E -msse4.1 - < /dev/null | sort > gcc-dump-sse41 -#define __SSE4_1__ 1 +# All of them are unrelated to the instruction set at the host machine +# (you can compile for newer instruction set on old machines and vice versa). -set (TEST_FLAG "-msse4.1") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - auto a = _mm_insert_epi8(__m128i(), 0, 0); - (void)a; - return 0; - } -" HAVE_SSE41) -if (HAVE_SSE41) - set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") -endif () +option (ENABLE_SSSE3 "Use SSSE3 instructions on x86_64" 1) +option (ENABLE_SSE41 "Use SSE4.1 instructions on x86_64" 1) +option (ENABLE_SSE42 "Use SSE4.2 instructions on x86_64" 1) +option (ENABLE_PCLMULQDQ "Use pclmulqdq instructions on x86_64" 1) +option (ENABLE_POPCNT "Use popcnt instructions on x86_64" 1) +option (ENABLE_AVX "Use AVX instructions on x86_64" 0) +option (ENABLE_AVX2 "Use AVX2 instructions on x86_64" 0) -if (ARCH_PPC64LE) - set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS") -endif () +option (ARCH_NATIVE "Add -march=native compiler flag. This makes your binaries non-portable but more performant code may be generated. This option overrides ENABLE_* options for specific instruction set. Highly not recommended to use." 0) -# gcc -dM -E -msse4.2 - < /dev/null | sort > gcc-dump-sse42 -#define __SSE4_2__ 1 +if (ARCH_NATIVE) + set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native") -set (TEST_FLAG "-msse4.2") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - auto a = _mm_crc32_u64(0, 0); - (void)a; - return 0; - } -" HAVE_SSE42) -if (HAVE_SSE42) - set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") -endif () +else () + set (TEST_FLAG "-mssse3") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + __m64 a = _mm_abs_pi8(__m64()); + (void)a; + return 0; + } + " HAVE_SSSE3) + if (HAVE_SSSE3 AND ENABLE_SSSE3) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () -set (TEST_FLAG "-mssse3") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - __m64 a = _mm_abs_pi8(__m64()); - (void)a; - return 0; - } -" HAVE_SSSE3) -set (TEST_FLAG "-mavx") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - auto a = _mm256_insert_epi8(__m256i(), 0, 0); - (void)a; - return 0; - } -" HAVE_AVX) + set (TEST_FLAG "-msse4.1") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + auto a = _mm_insert_epi8(__m128i(), 0, 0); + (void)a; + return 0; + } + " HAVE_SSE41) + if (HAVE_SSE41 AND ENABLE_SSE41) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () -set (TEST_FLAG "-mavx2") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - auto a = _mm256_add_epi16(__m256i(), __m256i()); - (void)a; - return 0; - } -" HAVE_AVX2) + if (ARCH_PPC64LE) + set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS") + endif () -set (TEST_FLAG "-mpclmul") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - #include - int main() { - auto a = _mm_clmulepi64_si128(__m128i(), __m128i(), 0); - (void)a; - return 0; - } -" HAVE_PCLMULQDQ) + set (TEST_FLAG "-msse4.2") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + auto a = _mm_crc32_u64(0, 0); + (void)a; + return 0; + } + " HAVE_SSE42) + if (HAVE_SSE42 AND ENABLE_SSE42) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () -# gcc -dM -E -mpopcnt - < /dev/null | sort > gcc-dump-popcnt -#define __POPCNT__ 1 + set (TEST_FLAG "-mpclmul") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + auto a = _mm_clmulepi64_si128(__m128i(), __m128i(), 0); + (void)a; + return 0; + } + " HAVE_PCLMULQDQ) + if (HAVE_PCLMULQDQ AND ENABLE_PCLMULQDQ) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () -set (TEST_FLAG "-mpopcnt") + set (TEST_FLAG "-mpopcnt") -set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") -check_cxx_source_compiles(" - int main() { - auto a = __builtin_popcountll(0); - (void)a; - return 0; - } -" HAVE_POPCNT) + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + int main() { + auto a = __builtin_popcountll(0); + (void)a; + return 0; + } + " HAVE_POPCNT) + if (HAVE_POPCNT AND ENABLE_POPCNT) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () -if (HAVE_POPCNT AND NOT ARCH_AARCH64) - set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + set (TEST_FLAG "-mavx") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + auto a = _mm256_insert_epi8(__m256i(), 0, 0); + (void)a; + return 0; + } + " HAVE_AVX) + if (HAVE_AVX AND ENABLE_AVX) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () + + set (TEST_FLAG "-mavx2") + set (CMAKE_REQUIRED_FLAGS "${TEST_FLAG} -O0") + check_cxx_source_compiles(" + #include + int main() { + auto a = _mm256_add_epi16(__m256i(), __m256i()); + (void)a; + return 0; + } + " HAVE_AVX2) + if (HAVE_AVX2 AND ENABLE_AVX2) + set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") + endif () endif () cmake_pop_check_state () diff --git a/contrib/simdjson-cmake/CMakeLists.txt b/contrib/simdjson-cmake/CMakeLists.txt index d3bcf6c046c..862d8dc50f8 100644 --- a/contrib/simdjson-cmake/CMakeLists.txt +++ b/contrib/simdjson-cmake/CMakeLists.txt @@ -4,3 +4,6 @@ set(SIMDJSON_SRC "${SIMDJSON_SRC_DIR}/simdjson.cpp") add_library(simdjson ${SIMDJSON_SRC}) target_include_directories(simdjson SYSTEM PUBLIC "${SIMDJSON_INCLUDE_DIR}" PRIVATE "${SIMDJSON_SRC_DIR}") + +# simdjson is using its own CPU dispatching and get confused if we enable AVX/AVX2 flags. +target_compile_options(simdjson PRIVATE -mno-avx -mno-avx2) diff --git a/docs/en/engines/database-engines/materialized-mysql.md b/docs/en/engines/database-engines/materialized-mysql.md index 62e58c7876b..ca550776d53 100644 --- a/docs/en/engines/database-engines/materialized-mysql.md +++ b/docs/en/engines/database-engines/materialized-mysql.md @@ -43,10 +43,10 @@ CREATE DATABASE mysql ENGINE = MaterializedMySQL('localhost:3306', 'db', 'user', **Settings on MySQL-server side** -For the correct work of `MaterializeMySQL`, there are few mandatory `MySQL`-side configuration settings that should be set: +For the correct work of `MaterializedMySQL`, there are few mandatory `MySQL`-side configuration settings that should be set: -- `default_authentication_plugin = mysql_native_password` since `MaterializeMySQL` can only authorize with this method. -- `gtid_mode = on` since GTID based logging is a mandatory for providing correct `MaterializeMySQL` replication. Pay attention that while turning this mode `On` you should also specify `enforce_gtid_consistency = on`. +- `default_authentication_plugin = mysql_native_password` since `MaterializedMySQL` can only authorize with this method. +- `gtid_mode = on` since GTID based logging is a mandatory for providing correct `MaterializedMySQL` replication. Pay attention that while turning this mode `On` you should also specify `enforce_gtid_consistency = on`. ## Virtual columns {#virtual-columns} diff --git a/docs/en/sql-reference/functions/bitmap-functions.md b/docs/en/sql-reference/functions/bitmap-functions.md index c695c894784..f8d1fdc69fa 100644 --- a/docs/en/sql-reference/functions/bitmap-functions.md +++ b/docs/en/sql-reference/functions/bitmap-functions.md @@ -125,6 +125,44 @@ Result: └───────────────────────────┘ ``` +## subBitmap {#subBitmap} + +Creates a subset of bitmap limit the results to `cardinality_limit` with offset of `offset`. + +**Syntax** + +``` sql +subBitmap(bitmap, offset, cardinality_limit) +``` + +**Arguments** + +- `bitmap` – [Bitmap object](#bitmap_functions-bitmapbuild). +- `offset` – the number of offsets. Type: [UInt32](../../sql-reference/data-types/int-uint.md). +- `cardinality_limit` – The subset cardinality upper limit. Type: [UInt32](../../sql-reference/data-types/int-uint.md). + +**Returned value** + +The subset. + +Type: `Bitmap object`. + +**Example** + +Query: + +``` sql +SELECT bitmapToArray(subBitmap(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500]), toUInt32(10), toUInt32(10))) AS res; +``` + +Result: + +``` text +┌─res─────────────────────────────┐ +│ [10,11,12,13,14,15,16,17,18,19] │ +└─────────────────────────────────┘ +``` + ## bitmapContains {#bitmap_functions-bitmapcontains} Checks whether the bitmap contains an element. diff --git a/docs/zh/sql-reference/functions/bitmap-functions.md b/docs/zh/sql-reference/functions/bitmap-functions.md index 5a9a88c5be1..c4c1b7293f6 100644 --- a/docs/zh/sql-reference/functions/bitmap-functions.md +++ b/docs/zh/sql-reference/functions/bitmap-functions.md @@ -88,6 +88,30 @@ SELECT bitmapToArray(bitmapSubsetLimit(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12 │ [30,31,32,33,100,200,500] │ └───────────────────────────┘ +## subBitmap {#subBitmap} + +将位图跳过`offset`个元素,限制大小为`limit`个的结果转换为另一个位图。 + + subBitmap(bitmap, offset, limit) + +**参数** + +- `bitmap` – 位图对象. +- `offset` – 跳过多少个元素. +- `limit` – 子位图基数上限. + +**示例** + +``` sql +SELECT bitmapToArray(subBitmap(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500]), toUInt32(10), toUInt32(10))) AS res +``` + +```text +┌─res─────────────────────────────┐ +│ [10,11,12,13,14,15,16,17,18,19] │ +└─────────────────────────────────┘ +``` + ## bitmapContains {#bitmapcontains} 检查位图是否包含指定元素。 diff --git a/programs/client/Client.cpp b/programs/client/Client.cpp index b28ef8f7c7f..14442167042 100644 --- a/programs/client/Client.cpp +++ b/programs/client/Client.cpp @@ -1,3 +1,6 @@ +#include +#include "Common/MemoryTracker.h" +#include "Columns/ColumnsNumber.h" #include "ConnectionParameters.h" #include "QueryFuzzer.h" #include "Suggest.h" @@ -100,6 +103,14 @@ #pragma GCC optimize("-fno-var-tracking-assignments") #endif +namespace CurrentMetrics +{ + extern const Metric Revision; + extern const Metric VersionInteger; + extern const Metric MemoryTracking; + extern const Metric MaxDDLEntryID; +} + namespace fs = std::filesystem; namespace DB @@ -524,6 +535,18 @@ private: { UseSSL use_ssl; + MainThreadStatus::getInstance(); + + /// Limit on total memory usage + size_t max_client_memory_usage = config().getInt64("max_memory_usage_in_client", 0 /*default value*/); + + if (max_client_memory_usage != 0) + { + total_memory_tracker.setHardLimit(max_client_memory_usage); + total_memory_tracker.setDescription("(total)"); + total_memory_tracker.setMetric(CurrentMetrics::MemoryTracking); + } + registerFormats(); registerFunctions(); registerAggregateFunctions(); @@ -2581,6 +2604,7 @@ public: ("opentelemetry-tracestate", po::value(), "OpenTelemetry tracestate header as described by W3C Trace Context recommendation") ("history_file", po::value(), "path to history file") ("no-warnings", "disable warnings when client connects to server") + ("max_memory_usage_in_client", po::value(), "sets memory limit in client") ; Settings cmd_settings; diff --git a/src/Access/IAccessStorage.cpp b/src/Access/IAccessStorage.cpp index 348987899cb..f0fbb95ff4e 100644 --- a/src/Access/IAccessStorage.cpp +++ b/src/Access/IAccessStorage.cpp @@ -455,7 +455,7 @@ UUID IAccessStorage::login( if (!replace_exception_with_cannot_authenticate) throw; - tryLogCurrentException(getLogger(), credentials.getUserName() + ": Authentication failed"); + tryLogCurrentException(getLogger(), "from: " + address.toString() + ", user: " + credentials.getUserName() + ": Authentication failed"); throwCannotAuthenticate(credentials.getUserName()); } } diff --git a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h index 95c7e6075d7..878cbc3219f 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h +++ b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h @@ -579,6 +579,37 @@ public: } } + UInt64 rb_offset_limit(UInt64 offset, UInt64 limit, RoaringBitmapWithSmallSet & r1) const + { + if (limit == 0 || offset >= size()) + return 0; + + if (isSmall()) + { + UInt64 count = 0; + UInt64 offset_count = 0; + auto it = small.begin(); + for (;it != small.end() && offset_count < offset; ++it) + ++offset_count; + + for (;it != small.end() && count < limit; ++it, ++count) + r1.add(it->getValue()); + return count; + } + else + { + UInt64 count = 0; + UInt64 offset_count = 0; + auto it = rb->begin(); + for (;it != rb->end() && offset_count < offset; ++it) + ++offset_count; + + for (;it != rb->end() && count < limit; ++it, ++count) + r1.add(*it); + return count; + } + } + UInt64 rb_min() const { if (isSmall()) diff --git a/src/Columns/ColumnLowCardinality.h b/src/Columns/ColumnLowCardinality.h index 698f65b1281..faf5bb9e712 100644 --- a/src/Columns/ColumnLowCardinality.h +++ b/src/Columns/ColumnLowCardinality.h @@ -187,6 +187,7 @@ public: * So LC(Nullable(T)) would return true, LC(U) -- false. */ bool nestedIsNullable() const { return isColumnNullable(*dictionary.getColumnUnique().getNestedColumn()); } + bool nestedCanBeInsideNullable() const { return dictionary.getColumnUnique().getNestedColumn()->canBeInsideNullable(); } void nestedToNullable() { dictionary.getColumnUnique().nestedToNullable(); } void nestedRemoveNullable() { dictionary.getColumnUnique().nestedRemoveNullable(); } diff --git a/src/Core/Block.cpp b/src/Core/Block.cpp index efd8de43a3c..96667862e41 100644 --- a/src/Core/Block.cpp +++ b/src/Core/Block.cpp @@ -44,6 +44,13 @@ void Block::initializeIndexByName() } +void Block::reserve(size_t count) +{ + index_by_name.reserve(count); + data.reserve(count); +} + + void Block::insert(size_t position, ColumnWithTypeAndName elem) { if (position > data.size()) @@ -287,6 +294,7 @@ std::string Block::dumpIndex() const Block Block::cloneEmpty() const { Block res; + res.reserve(data.size()); for (const auto & elem : data) res.insert(elem.cloneEmpty()); @@ -364,6 +372,8 @@ Block Block::cloneWithColumns(MutableColumns && columns) const Block res; size_t num_columns = data.size(); + res.reserve(num_columns); + for (size_t i = 0; i < num_columns; ++i) res.insert({ std::move(columns[i]), data[i].type, data[i].name }); @@ -381,6 +391,8 @@ Block Block::cloneWithColumns(const Columns & columns) const throw Exception("Cannot clone block with columns because block has " + toString(num_columns) + " columns, " "but " + toString(columns.size()) + " columns given.", ErrorCodes::LOGICAL_ERROR); + res.reserve(num_columns); + for (size_t i = 0; i < num_columns; ++i) res.insert({ columns[i], data[i].type, data[i].name }); @@ -393,6 +405,8 @@ Block Block::cloneWithoutColumns() const Block res; size_t num_columns = data.size(); + res.reserve(num_columns); + for (size_t i = 0; i < num_columns; ++i) res.insert({ nullptr, data[i].type, data[i].name }); diff --git a/src/Core/Block.h b/src/Core/Block.h index a2d91190795..14f82cecd8d 100644 --- a/src/Core/Block.h +++ b/src/Core/Block.h @@ -152,6 +152,7 @@ public: private: void eraseImpl(size_t position); void initializeIndexByName(); + void reserve(size_t count); /// This is needed to allow function execution over data. /// It is safe because functions does not change column names, so index is unaffected. diff --git a/src/Core/ya.make b/src/Core/ya.make index d1e352ee846..6946d7a47bb 100644 --- a/src/Core/ya.make +++ b/src/Core/ya.make @@ -31,6 +31,10 @@ SRCS( MySQL/PacketsProtocolText.cpp MySQL/PacketsReplication.cpp NamesAndTypes.cpp + PostgreSQL/Connection.cpp + PostgreSQL/PoolWithFailover.cpp + PostgreSQL/Utils.cpp + PostgreSQL/insertPostgreSQLValue.cpp PostgreSQLProtocol.cpp QueryProcessingStage.cpp Settings.cpp diff --git a/src/DataStreams/ya.make b/src/DataStreams/ya.make index 2012af76697..b1205828a7e 100644 --- a/src/DataStreams/ya.make +++ b/src/DataStreams/ya.make @@ -49,6 +49,7 @@ SRCS( TTLUpdateInfoAlgorithm.cpp copyData.cpp finalizeBlock.cpp + formatBlock.cpp materializeBlock.cpp narrowBlockInputStreams.cpp diff --git a/src/Functions/FunctionsBitmap.cpp b/src/Functions/FunctionsBitmap.cpp index 72652288872..159d0ff6777 100644 --- a/src/Functions/FunctionsBitmap.cpp +++ b/src/Functions/FunctionsBitmap.cpp @@ -13,6 +13,7 @@ void registerFunctionsBitmap(FunctionFactory & factory) factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); + factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); diff --git a/src/Functions/FunctionsBitmap.h b/src/Functions/FunctionsBitmap.h index bbdc53c3006..8155dbdcf8e 100644 --- a/src/Functions/FunctionsBitmap.h +++ b/src/Functions/FunctionsBitmap.h @@ -460,9 +460,24 @@ public: } }; +struct BitmapSubsetOffsetLimitImpl +{ +public: + static constexpr auto name = "subBitmap"; + template + static void apply( + const AggregateFunctionGroupBitmapData & bitmap_data_0, + UInt64 range_start, + UInt64 range_end, + AggregateFunctionGroupBitmapData & bitmap_data_2) + { + bitmap_data_0.rbs.rb_offset_limit(range_start, range_end, bitmap_data_2.rbs); + } +}; + using FunctionBitmapSubsetInRange = FunctionBitmapSubset; using FunctionBitmapSubsetLimit = FunctionBitmapSubset; - +using FunctionBitmapSubsetOffsetLimit = FunctionBitmapSubset; class FunctionBitmapTransform : public IFunction { diff --git a/src/Interpreters/AsynchronousMetrics.cpp b/src/Interpreters/AsynchronousMetrics.cpp index 1c79dedd978..8efe959a623 100644 --- a/src/Interpreters/AsynchronousMetrics.cpp +++ b/src/Interpreters/AsynchronousMetrics.cpp @@ -1091,7 +1091,14 @@ void AsynchronousMetrics::update(std::chrono::system_clock::time_point update_ti { sensor_file->rewind(); Int64 temperature = 0; - readText(temperature, *sensor_file); + try + { + readText(temperature, *sensor_file); + } + catch (const ErrnoException & e) + { + LOG_DEBUG(&Poco::Logger::get("AsynchronousMetrics"), "Hardware monitor '{}', sensor '{}' exists but could not be read, error {}.", hwmon_name, sensor_name, e.getErrno()); + } if (sensor_name.empty()) new_values[fmt::format("Temperature_{}", hwmon_name)] = temperature * 0.001; diff --git a/src/Interpreters/InterpreterCreateQuery.cpp b/src/Interpreters/InterpreterCreateQuery.cpp index bf2cf6338aa..4c1a3064c3d 100644 --- a/src/Interpreters/InterpreterCreateQuery.cpp +++ b/src/Interpreters/InterpreterCreateQuery.cpp @@ -764,7 +764,7 @@ void InterpreterCreateQuery::assertOrSetUUID(ASTCreateQuery & create, const Data const auto * kind = create.is_dictionary ? "Dictionary" : "Table"; const auto * kind_upper = create.is_dictionary ? "DICTIONARY" : "TABLE"; - if (database->getEngineName() == "Replicated" && getContext()->getClientInfo().query_kind == ClientInfo::QueryKind::SECONDARY_QUERY + if (database->getEngineName() == "Replicated" && getContext()->getClientInfo().is_replicated_database_internal && !internal) { if (create.uuid == UUIDHelpers::Nil) diff --git a/src/Interpreters/join_common.cpp b/src/Interpreters/join_common.cpp index 76bfd7f2899..e9f3e4f3fdd 100644 --- a/src/Interpreters/join_common.cpp +++ b/src/Interpreters/join_common.cpp @@ -2,6 +2,7 @@ #include #include +#include #include @@ -105,25 +106,57 @@ DataTypePtr convertTypeToNullable(const DataTypePtr & type) return type; } +/// Convert column to nullable. If column LowCardinality or Const, convert nested column. +/// Returns nullptr if conversion cannot be performed. +static ColumnPtr tryConvertColumnToNullable(const ColumnPtr & col) +{ + if (isColumnNullable(*col) || col->canBeInsideNullable()) + return makeNullable(col); + + if (col->lowCardinality()) + { + auto mut_col = IColumn::mutate(std::move(col)); + ColumnLowCardinality * col_lc = assert_cast(mut_col.get()); + if (col_lc->nestedIsNullable()) + { + return mut_col; + } + else if (col_lc->nestedCanBeInsideNullable()) + { + col_lc->nestedToNullable(); + return mut_col; + } + } + else if (const ColumnConst * col_const = checkAndGetColumn(*col)) + { + const auto & nested = col_const->getDataColumnPtr(); + if (nested->isNullable() || nested->canBeInsideNullable()) + { + return makeNullable(col); + } + else if (nested->lowCardinality()) + { + ColumnPtr nested_nullable = tryConvertColumnToNullable(nested); + if (nested_nullable) + return ColumnConst::create(nested_nullable, col_const->size()); + } + } + return nullptr; +} + void convertColumnToNullable(ColumnWithTypeAndName & column) { - column.type = convertTypeToNullable(column.type); - if (!column.column) + { + column.type = convertTypeToNullable(column.type); return; - - if (column.column->lowCardinality()) - { - /// Convert nested to nullable, not LowCardinality itself - auto mut_col = IColumn::mutate(std::move(column.column)); - ColumnLowCardinality * col_as_lc = assert_cast(mut_col.get()); - if (!col_as_lc->nestedIsNullable()) - col_as_lc->nestedToNullable(); - column.column = std::move(mut_col); } - else if (column.column->canBeInsideNullable()) + + ColumnPtr nullable_column = tryConvertColumnToNullable(column.column); + if (nullable_column) { - column.column = makeNullable(column.column); + column.type = convertTypeToNullable(column.type); + column.column = std::move(nullable_column); } } diff --git a/src/Parsers/ya.make b/src/Parsers/ya.make index 62e0c2b3225..3b8a9a19bce 100644 --- a/src/Parsers/ya.make +++ b/src/Parsers/ya.make @@ -21,6 +21,7 @@ SRCS( ASTCreateRowPolicyQuery.cpp ASTCreateSettingsProfileQuery.cpp ASTCreateUserQuery.cpp + ASTDatabaseOrNone.cpp ASTDictionary.cpp ASTDictionaryAttributeDeclaration.cpp ASTDropAccessEntityQuery.cpp @@ -95,6 +96,7 @@ SRCS( ParserCreateSettingsProfileQuery.cpp ParserCreateUserQuery.cpp ParserDataType.cpp + ParserDatabaseOrNone.cpp ParserDescribeTableQuery.cpp ParserDictionary.cpp ParserDictionaryAttributeDeclaration.cpp diff --git a/src/Processors/Transforms/WindowTransform.cpp b/src/Processors/Transforms/WindowTransform.cpp index 3ab1a23537b..1b8406682ea 100644 --- a/src/Processors/Transforms/WindowTransform.cpp +++ b/src/Processors/Transforms/WindowTransform.cpp @@ -1166,6 +1166,23 @@ void WindowTransform::appendChunk(Chunk & chunk) // Write out the aggregation results. writeOutCurrentRow(); + if (isCancelled()) + { + // Good time to check if the query is cancelled. Checking once + // per block might not be enough in severe quadratic cases. + // Just leave the work halfway through and return, the 'prepare' + // method will figure out what to do. Note that this doesn't + // handle 'max_execution_time' and other limits, because these + // limits are only updated between blocks. Eventually we should + // start updating them in background and canceling the processor, + // like we do for Ctrl+C handling. + // + // This class is final, so the check should hopefully be + // devirtualized and become a single never-taken branch that is + // basically free. + return; + } + // Move to the next row. The frame will have to be recalculated. // The peer group start is updated at the beginning of the loop, // because current_row might now be past-the-end. @@ -1255,10 +1272,12 @@ IProcessor::Status WindowTransform::prepare() // next_output_block_number, first_not_ready_row, first_block_number, // blocks.size()); - if (output.isFinished()) + if (output.isFinished() || isCancelled()) { // The consumer asked us not to continue (or we decided it ourselves), - // so we abort. + // so we abort. Not sure what the difference between the two conditions + // is, but it seemed that output.isFinished() is not enough to cancel on + // Ctrl+C. Test manually if you change it. input.close(); return Status::Finished; } diff --git a/src/Processors/Transforms/WindowTransform.h b/src/Processors/Transforms/WindowTransform.h index d7211f9edd7..5dc78a34f78 100644 --- a/src/Processors/Transforms/WindowTransform.h +++ b/src/Processors/Transforms/WindowTransform.h @@ -80,8 +80,10 @@ struct RowNumber * the order of input data. This property also trivially holds for the ROWS and * GROUPS frames. For the RANGE frame, the proof requires the additional fact * that the ranges are specified in terms of (the single) ORDER BY column. + * + * `final` is so that the isCancelled() is devirtualized, we call it every row. */ -class WindowTransform : public IProcessor /* public ISimpleTransform */ +class WindowTransform final : public IProcessor { public: WindowTransform( diff --git a/src/Processors/ya.make b/src/Processors/ya.make index 4b95484a828..543a08caca5 100644 --- a/src/Processors/ya.make +++ b/src/Processors/ya.make @@ -7,14 +7,8 @@ PEERDIR( clickhouse/src/Common contrib/libs/msgpack contrib/libs/protobuf - contrib/libs/arrow ) -ADDINCL( - contrib/libs/arrow/src -) - -CFLAGS(-DUSE_ARROW=1) SRCS( Chunk.cpp @@ -31,11 +25,6 @@ SRCS( Formats/IOutputFormat.cpp Formats/IRowInputFormat.cpp Formats/IRowOutputFormat.cpp - Formats/Impl/ArrowBlockInputFormat.cpp - Formats/Impl/ArrowBlockOutputFormat.cpp - Formats/Impl/ArrowBufferedStreams.cpp - Formats/Impl/ArrowColumnToCHColumn.cpp - Formats/Impl/CHColumnToArrowColumn.cpp Formats/Impl/BinaryRowInputFormat.cpp Formats/Impl/BinaryRowOutputFormat.cpp Formats/Impl/CSVRowInputFormat.cpp diff --git a/src/Storages/MergeTree/BackgroundJobsExecutor.cpp b/src/Storages/MergeTree/BackgroundJobsExecutor.cpp index 36803ba5197..f3d957117e8 100644 --- a/src/Storages/MergeTree/BackgroundJobsExecutor.cpp +++ b/src/Storages/MergeTree/BackgroundJobsExecutor.cpp @@ -146,6 +146,9 @@ try catch (...) /// Exception while we looking for a task, reschedule { tryLogCurrentException(__PRETTY_FUNCTION__); + + /// Why do we scheduleTask again? + /// To retry on exception, since it may be some temporary exception. scheduleTask(/* with_backoff = */ true); } @@ -180,10 +183,16 @@ void IBackgroundJobExecutor::triggerTask() } void IBackgroundJobExecutor::backgroundTaskFunction() +try { if (!scheduleJob()) scheduleTask(/* with_backoff = */ true); } +catch (...) /// Catch any exception to avoid thread termination. +{ + tryLogCurrentException(__PRETTY_FUNCTION__); + scheduleTask(/* with_backoff = */ true); +} IBackgroundJobExecutor::~IBackgroundJobExecutor() { diff --git a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp index 6279d2d7d6f..baea7e72b21 100644 --- a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp +++ b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp @@ -1663,7 +1663,12 @@ NameToNameVector MergeTreeDataMergerMutator::collectFilesForRenames( { if (command.type == MutationCommand::Type::DROP_INDEX) { - if (source_part->checksums.has(INDEX_FILE_PREFIX + command.column_name + ".idx")) + if (source_part->checksums.has(INDEX_FILE_PREFIX + command.column_name + ".idx2")) + { + rename_vector.emplace_back(INDEX_FILE_PREFIX + command.column_name + ".idx2", ""); + rename_vector.emplace_back(INDEX_FILE_PREFIX + command.column_name + mrk_extension, ""); + } + else if (source_part->checksums.has(INDEX_FILE_PREFIX + command.column_name + ".idx")) { rename_vector.emplace_back(INDEX_FILE_PREFIX + command.column_name + ".idx", ""); rename_vector.emplace_back(INDEX_FILE_PREFIX + command.column_name + mrk_extension, ""); @@ -1749,6 +1754,7 @@ NameSet MergeTreeDataMergerMutator::collectFilesToSkip( for (const auto & index : indices_to_recalc) { files_to_skip.insert(index->getFileName() + ".idx"); + files_to_skip.insert(index->getFileName() + ".idx2"); files_to_skip.insert(index->getFileName() + mrk_extension); } for (const auto & projection : projections_to_recalc) @@ -1893,8 +1899,11 @@ std::set MergeTreeDataMergerMutator::getIndicesToRecalculate( { const auto & index = indices[i]; + bool has_index = + source_part->checksums.has(INDEX_FILE_PREFIX + index.name + ".idx") || + source_part->checksums.has(INDEX_FILE_PREFIX + index.name + ".idx2"); // If we ask to materialize and it already exists - if (!source_part->checksums.has(INDEX_FILE_PREFIX + index.name + ".idx") && materialized_indices.count(index.name)) + if (!has_index && materialized_indices.count(index.name)) { if (indices_to_recalc.insert(index_factory.get(index)).second) { diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp index 9902add9847..4263640c1e0 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp @@ -9,11 +9,6 @@ namespace ErrorCodes extern const int LOGICAL_ERROR; } -namespace -{ - constexpr auto INDEX_FILE_EXTENSION = ".idx"; -} - void MergeTreeDataPartWriterOnDisk::Stream::finalize() { compressed.next(); @@ -165,7 +160,7 @@ void MergeTreeDataPartWriterOnDisk::initSkipIndices() std::make_unique( stream_name, data_part->volume->getDisk(), - part_path + stream_name, INDEX_FILE_EXTENSION, + part_path + stream_name, index_helper->getSerializedFileExtension(), part_path + stream_name, marks_file_extension, default_codec, settings.max_compress_block_size)); skip_indices_aggregators.push_back(index_helper->createIndexAggregator()); diff --git a/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp b/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp index 0b5351dcf01..f60acca12a7 100644 --- a/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp +++ b/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp @@ -1457,9 +1457,10 @@ MarkRanges MergeTreeDataSelectExecutor::filterMarksUsingIndex( size_t & granules_dropped, Poco::Logger * log) { - if (!part->volume->getDisk()->exists(part->getFullRelativePath() + index_helper->getFileName() + ".idx")) + const std::string & path_prefix = part->getFullRelativePath() + index_helper->getFileName(); + if (!index_helper->getDeserializedFormat(part->volume->getDisk(), path_prefix)) { - LOG_DEBUG(log, "File for index {} does not exist. Skipping it.", backQuote(index_helper->index.name)); + LOG_DEBUG(log, "File for index {} does not exist ({}.*). Skipping it.", backQuote(index_helper->index.name), path_prefix); return ranges; } diff --git a/src/Storages/MergeTree/MergeTreeIndexFullText.cpp b/src/Storages/MergeTree/MergeTreeIndexFullText.cpp index 10136cd1069..1c71d77b334 100644 --- a/src/Storages/MergeTree/MergeTreeIndexFullText.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexFullText.cpp @@ -101,14 +101,17 @@ MergeTreeIndexGranuleFullText::MergeTreeIndexGranuleFullText( void MergeTreeIndexGranuleFullText::serializeBinary(WriteBuffer & ostr) const { if (empty()) - throw Exception("Attempt to write empty fulltext index " + backQuote(index_name), ErrorCodes::LOGICAL_ERROR); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Attempt to write empty fulltext index {}.", backQuote(index_name)); for (const auto & bloom_filter : bloom_filters) ostr.write(reinterpret_cast(bloom_filter.getFilter().data()), params.filter_size); } -void MergeTreeIndexGranuleFullText::deserializeBinary(ReadBuffer & istr) +void MergeTreeIndexGranuleFullText::deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) { + if (version != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unknown index version {}.", version); + for (auto & bloom_filter : bloom_filters) { istr.read(reinterpret_cast( diff --git a/src/Storages/MergeTree/MergeTreeIndexFullText.h b/src/Storages/MergeTree/MergeTreeIndexFullText.h index 1385621f97f..d34cbc61da2 100644 --- a/src/Storages/MergeTree/MergeTreeIndexFullText.h +++ b/src/Storages/MergeTree/MergeTreeIndexFullText.h @@ -45,7 +45,7 @@ struct MergeTreeIndexGranuleFullText final : public IMergeTreeIndexGranule ~MergeTreeIndexGranuleFullText() override = default; void serializeBinary(WriteBuffer & ostr) const override; - void deserializeBinary(ReadBuffer & istr) override; + void deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) override; bool empty() const override { return !has_elems; } diff --git a/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.cpp b/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.cpp index b513437fbe1..6a027b8cb8e 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.cpp @@ -84,10 +84,12 @@ bool MergeTreeIndexGranuleBloomFilter::empty() const return !total_rows; } -void MergeTreeIndexGranuleBloomFilter::deserializeBinary(ReadBuffer & istr) +void MergeTreeIndexGranuleBloomFilter::deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) { if (!empty()) - throw Exception("Cannot read data to a non-empty bloom filter index.", ErrorCodes::LOGICAL_ERROR); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot read data to a non-empty bloom filter index."); + if (version != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unknown index version {}.", version); readVarUInt(total_rows, istr); for (auto & filter : bloom_filters) @@ -102,7 +104,7 @@ void MergeTreeIndexGranuleBloomFilter::deserializeBinary(ReadBuffer & istr) void MergeTreeIndexGranuleBloomFilter::serializeBinary(WriteBuffer & ostr) const { if (empty()) - throw Exception("Attempt to write empty bloom filter index.", ErrorCodes::LOGICAL_ERROR); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Attempt to write empty bloom filter index."); static size_t atom_size = 8; writeVarUInt(total_rows, ostr); diff --git a/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h b/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h index cdd4b92f80c..82bd91138a7 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h +++ b/src/Storages/MergeTree/MergeTreeIndexGranuleBloomFilter.h @@ -16,8 +16,7 @@ public: bool empty() const override; void serializeBinary(WriteBuffer & ostr) const override; - - void deserializeBinary(ReadBuffer & istr) override; + void deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) override; const std::vector & getFilters() const { return bloom_filters; } diff --git a/src/Storages/MergeTree/MergeTreeIndexMinMax.cpp b/src/Storages/MergeTree/MergeTreeIndexMinMax.cpp index ebf553295be..3a83afbd280 100644 --- a/src/Storages/MergeTree/MergeTreeIndexMinMax.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexMinMax.cpp @@ -40,28 +40,12 @@ void MergeTreeIndexGranuleMinMax::serializeBinary(WriteBuffer & ostr) const const DataTypePtr & type = index_sample_block.getByPosition(i).type; auto serialization = type->getDefaultSerialization(); - if (!type->isNullable()) - { - serialization->serializeBinary(hyperrectangle[i].left, ostr); - serialization->serializeBinary(hyperrectangle[i].right, ostr); - } - else - { - /// NOTE: that this serialization differs from - /// IMergeTreeDataPart::MinMaxIndex::store() due to preserve - /// backward compatibility. - bool is_null = hyperrectangle[i].left.isNull() || hyperrectangle[i].right.isNull(); // one is enough - writeBinary(is_null, ostr); - if (!is_null) - { - serialization->serializeBinary(hyperrectangle[i].left, ostr); - serialization->serializeBinary(hyperrectangle[i].right, ostr); - } - } + serialization->serializeBinary(hyperrectangle[i].left, ostr); + serialization->serializeBinary(hyperrectangle[i].right, ostr); } } -void MergeTreeIndexGranuleMinMax::deserializeBinary(ReadBuffer & istr) +void MergeTreeIndexGranuleMinMax::deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) { hyperrectangle.clear(); Field min_val; @@ -72,29 +56,53 @@ void MergeTreeIndexGranuleMinMax::deserializeBinary(ReadBuffer & istr) const DataTypePtr & type = index_sample_block.getByPosition(i).type; auto serialization = type->getDefaultSerialization(); - if (!type->isNullable()) + switch (version) { - serialization->deserializeBinary(min_val, istr); - serialization->deserializeBinary(max_val, istr); - } - else - { - /// NOTE: that this serialization differs from - /// IMergeTreeDataPart::MinMaxIndex::load() due to preserve - /// backward compatibility. - bool is_null; - readBinary(is_null, istr); - if (!is_null) - { + case 1: + if (!type->isNullable()) + { + serialization->deserializeBinary(min_val, istr); + serialization->deserializeBinary(max_val, istr); + } + else + { + /// NOTE: that this serialization differs from + /// IMergeTreeDataPart::MinMaxIndex::load() to preserve + /// backward compatibility. + /// + /// But this is deprecated format, so this is OK. + + bool is_null; + readBinary(is_null, istr); + if (!is_null) + { + serialization->deserializeBinary(min_val, istr); + serialization->deserializeBinary(max_val, istr); + } + else + { + min_val = Null(); + max_val = Null(); + } + } + break; + + /// New format with proper Nullable support for values that includes Null values + case 2: serialization->deserializeBinary(min_val, istr); serialization->deserializeBinary(max_val, istr); - } - else - { - min_val = Null(); - max_val = Null(); - } + + // NULL_LAST + if (min_val.isNull()) + min_val = PositiveInfinity(); + if (max_val.isNull()) + max_val = PositiveInfinity(); + + break; + default: + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unknown index version {}.", version); } + hyperrectangle.emplace_back(min_val, true, max_val, true); } } @@ -203,6 +211,15 @@ bool MergeTreeIndexMinMax::mayBenefitFromIndexForIn(const ASTPtr & node) const return false; } +MergeTreeIndexFormat MergeTreeIndexMinMax::getDeserializedFormat(const DiskPtr disk, const std::string & relative_path_prefix) const +{ + if (disk->exists(relative_path_prefix + ".idx2")) + return {2, ".idx2"}; + else if (disk->exists(relative_path_prefix + ".idx")) + return {1, ".idx"}; + return {0 /* unknown */, ""}; +} + MergeTreeIndexPtr minmaxIndexCreator( const IndexDescription & index) { diff --git a/src/Storages/MergeTree/MergeTreeIndexMinMax.h b/src/Storages/MergeTree/MergeTreeIndexMinMax.h index 97b9b874484..0e05e25fb36 100644 --- a/src/Storages/MergeTree/MergeTreeIndexMinMax.h +++ b/src/Storages/MergeTree/MergeTreeIndexMinMax.h @@ -21,7 +21,7 @@ struct MergeTreeIndexGranuleMinMax final : public IMergeTreeIndexGranule ~MergeTreeIndexGranuleMinMax() override = default; void serializeBinary(WriteBuffer & ostr) const override; - void deserializeBinary(ReadBuffer & istr) override; + void deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) override; bool empty() const override { return hyperrectangle.empty(); } @@ -81,6 +81,9 @@ public: const SelectQueryInfo & query, ContextPtr context) const override; bool mayBenefitFromIndexForIn(const ASTPtr & node) const override; + + const char* getSerializedFileExtension() const override { return ".idx2"; } + MergeTreeIndexFormat getDeserializedFormat(const DiskPtr disk, const std::string & path_prefix) const override; }; } diff --git a/src/Storages/MergeTree/MergeTreeIndexReader.cpp b/src/Storages/MergeTree/MergeTreeIndexReader.cpp index eaba247009b..0a0f2511914 100644 --- a/src/Storages/MergeTree/MergeTreeIndexReader.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexReader.cpp @@ -1,5 +1,29 @@ #include +namespace +{ + +using namespace DB; + +std::unique_ptr makeIndexReader( + const std::string & extension, + MergeTreeIndexPtr index, + MergeTreeData::DataPartPtr part, + size_t marks_count, + const MarkRanges & all_mark_ranges, + MergeTreeReaderSettings settings) +{ + return std::make_unique( + part->volume->getDisk(), + part->getFullRelativePath() + index->getFileName(), extension, marks_count, + all_mark_ranges, + std::move(settings), nullptr, nullptr, + part->getFileSizeOrZero(index->getFileName() + extension), + &part->index_granularity_info, + ReadBufferFromFileBase::ProfileCallback{}, CLOCK_MONOTONIC_COARSE); +} + +} namespace DB { @@ -7,27 +31,28 @@ namespace DB MergeTreeIndexReader::MergeTreeIndexReader( MergeTreeIndexPtr index_, MergeTreeData::DataPartPtr part_, size_t marks_count_, const MarkRanges & all_mark_ranges_, MergeTreeReaderSettings settings) - : index(index_), stream( - part_->volume->getDisk(), - part_->getFullRelativePath() + index->getFileName(), ".idx", marks_count_, - all_mark_ranges_, - std::move(settings), nullptr, nullptr, - part_->getFileSizeOrZero(index->getFileName() + ".idx"), - &part_->index_granularity_info, - ReadBufferFromFileBase::ProfileCallback{}, CLOCK_MONOTONIC_COARSE) + : index(index_) { - stream.seekToStart(); + const std::string & path_prefix = part_->getFullRelativePath() + index->getFileName(); + auto index_format = index->getDeserializedFormat(part_->volume->getDisk(), path_prefix); + + stream = makeIndexReader(index_format.extension, index_, part_, marks_count_, all_mark_ranges_, std::move(settings)); + version = index_format.version; + + stream->seekToStart(); } +MergeTreeIndexReader::~MergeTreeIndexReader() = default; + void MergeTreeIndexReader::seek(size_t mark) { - stream.seekToMark(mark); + stream->seekToMark(mark); } MergeTreeIndexGranulePtr MergeTreeIndexReader::read() { auto granule = index->createIndexGranule(); - granule->deserializeBinary(*stream.data_buffer); + granule->deserializeBinary(*stream->data_buffer, version); return granule; } diff --git a/src/Storages/MergeTree/MergeTreeIndexReader.h b/src/Storages/MergeTree/MergeTreeIndexReader.h index 68d681458be..4facd43c175 100644 --- a/src/Storages/MergeTree/MergeTreeIndexReader.h +++ b/src/Storages/MergeTree/MergeTreeIndexReader.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -16,6 +17,7 @@ public: size_t marks_count_, const MarkRanges & all_mark_ranges_, MergeTreeReaderSettings settings); + ~MergeTreeIndexReader(); void seek(size_t mark); @@ -23,7 +25,8 @@ public: private: MergeTreeIndexPtr index; - MergeTreeReaderStream stream; + std::unique_ptr stream; + uint8_t version = 0; }; } diff --git a/src/Storages/MergeTree/MergeTreeIndexSet.cpp b/src/Storages/MergeTree/MergeTreeIndexSet.cpp index 6cee80983d6..024b87c9a3e 100644 --- a/src/Storages/MergeTree/MergeTreeIndexSet.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexSet.cpp @@ -48,8 +48,7 @@ MergeTreeIndexGranuleSet::MergeTreeIndexGranuleSet( void MergeTreeIndexGranuleSet::serializeBinary(WriteBuffer & ostr) const { if (empty()) - throw Exception( - "Attempt to write empty set index " + backQuote(index_name), ErrorCodes::LOGICAL_ERROR); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Attempt to write empty set index {}.", backQuote(index_name)); const auto & size_type = DataTypePtr(std::make_shared()); auto size_serialization = size_type->getDefaultSerialization(); @@ -80,8 +79,11 @@ void MergeTreeIndexGranuleSet::serializeBinary(WriteBuffer & ostr) const } } -void MergeTreeIndexGranuleSet::deserializeBinary(ReadBuffer & istr) +void MergeTreeIndexGranuleSet::deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) { + if (version != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unknown index version {}.", version); + block.clear(); Field field_rows; diff --git a/src/Storages/MergeTree/MergeTreeIndexSet.h b/src/Storages/MergeTree/MergeTreeIndexSet.h index 28afe4f714d..23b336d274b 100644 --- a/src/Storages/MergeTree/MergeTreeIndexSet.h +++ b/src/Storages/MergeTree/MergeTreeIndexSet.h @@ -28,7 +28,7 @@ struct MergeTreeIndexGranuleSet final : public IMergeTreeIndexGranule MutableColumns && columns_); void serializeBinary(WriteBuffer & ostr) const override; - void deserializeBinary(ReadBuffer & istr) override; + void deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) override; size_t size() const { return block.rows(); } bool empty() const override { return !size(); } diff --git a/src/Storages/MergeTree/MergeTreeIndices.h b/src/Storages/MergeTree/MergeTreeIndices.h index 674daeb480d..557af891b74 100644 --- a/src/Storages/MergeTree/MergeTreeIndices.h +++ b/src/Storages/MergeTree/MergeTreeIndices.h @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -17,13 +18,37 @@ constexpr auto INDEX_FILE_PREFIX = "skp_idx_"; namespace DB { +using MergeTreeIndexVersion = uint8_t; +struct MergeTreeIndexFormat +{ + MergeTreeIndexVersion version; + const char* extension; + + operator bool() const { return version != 0; } +}; + /// Stores some info about a single block of data. struct IMergeTreeIndexGranule { virtual ~IMergeTreeIndexGranule() = default; + /// Serialize always last version. virtual void serializeBinary(WriteBuffer & ostr) const = 0; - virtual void deserializeBinary(ReadBuffer & istr) = 0; + + /// Version of the index to deserialize: + /// + /// - 2 -- minmax index for proper Nullable support, + /// - 1 -- everything else. + /// + /// Implementation is responsible for version check, + /// and throw LOGICAL_ERROR in case of unsupported version. + /// + /// See also: + /// - IMergeTreeIndex::getSerializedFileExtension() + /// - IMergeTreeIndex::getDeserializedFormat() + /// - MergeTreeDataMergerMutator::collectFilesToSkip() + /// - MergeTreeDataMergerMutator::collectFilesForRenames() + virtual void deserializeBinary(ReadBuffer & istr, MergeTreeIndexVersion version) = 0; virtual bool empty() const = 0; }; @@ -73,9 +98,26 @@ struct IMergeTreeIndex virtual ~IMergeTreeIndex() = default; - /// gets filename without extension + /// Returns filename without extension. String getFileName() const { return INDEX_FILE_PREFIX + index.name; } + /// Returns extension for serialization. + /// Reimplement if you want new index format. + /// + /// NOTE: In case getSerializedFileExtension() is reimplemented, + /// getDeserializedFormat() should be reimplemented too, + /// and check all previous extensions too + /// (to avoid breaking backward compatibility). + virtual const char* getSerializedFileExtension() const { return ".idx"; } + + /// Returns extension for deserialization. + /// + /// Return pair. + virtual MergeTreeIndexFormat getDeserializedFormat(const DiskPtr, const std::string & /* relative_path_prefix */) const + { + return {1, ".idx"}; + } + /// Checks whether the column is in data skipping index. virtual bool mayBenefitFromIndexForIn(const ASTPtr & node) const = 0; diff --git a/src/Storages/StorageMergeTree.cpp b/src/Storages/StorageMergeTree.cpp index 0763e2a25c4..32c2c76dd10 100644 --- a/src/Storages/StorageMergeTree.cpp +++ b/src/Storages/StorageMergeTree.cpp @@ -959,9 +959,19 @@ std::shared_ptr StorageMergeTree::se if (!commands_for_size_validation.empty()) { - MutationsInterpreter interpreter( - shared_from_this(), metadata_snapshot, commands_for_size_validation, getContext(), false); - commands_size += interpreter.evaluateCommandsSize(); + try + { + MutationsInterpreter interpreter( + shared_from_this(), metadata_snapshot, commands_for_size_validation, getContext(), false); + commands_size += interpreter.evaluateCommandsSize(); + } + catch (...) + { + MergeTreeMutationEntry & entry = it->second; + entry.latest_fail_time = time(nullptr); + entry.latest_fail_reason = getCurrentExceptionMessage(false); + continue; + } } if (current_ast_elements + commands_size >= max_ast_elements) @@ -971,17 +981,21 @@ std::shared_ptr StorageMergeTree::se commands.insert(commands.end(), it->second.commands.begin(), it->second.commands.end()); } - auto new_part_info = part->info; - new_part_info.mutation = current_mutations_by_version.rbegin()->first; + if (!commands.empty()) + { + auto new_part_info = part->info; + new_part_info.mutation = current_mutations_by_version.rbegin()->first; - future_part.parts.push_back(part); - future_part.part_info = new_part_info; - future_part.name = part->getNewName(new_part_info); - future_part.type = part->getType(); + future_part.parts.push_back(part); + future_part.part_info = new_part_info; + future_part.name = part->getNewName(new_part_info); + future_part.type = part->getType(); - tagger = std::make_unique(future_part, MergeTreeDataMergerMutator::estimateNeededDiskSpace({part}), *this, metadata_snapshot, true); - return std::make_shared(future_part, std::move(tagger), commands); + tagger = std::make_unique(future_part, MergeTreeDataMergerMutator::estimateNeededDiskSpace({part}), *this, metadata_snapshot, true); + return std::make_shared(future_part, std::move(tagger), commands); + } } + return {}; } @@ -1036,6 +1050,7 @@ bool StorageMergeTree::scheduleDataProcessingJob(IBackgroundJobExecutor & execut auto share_lock = lockForShare(RWLockImpl::NO_QUERY, getSettings()->lock_acquire_timeout_for_background_operations); + bool has_mutations; { std::unique_lock lock(currently_processing_in_background_mutex); if (merger_mutator.merges_blocker.isCancelled()) @@ -1044,6 +1059,15 @@ bool StorageMergeTree::scheduleDataProcessingJob(IBackgroundJobExecutor & execut merge_entry = selectPartsToMerge(metadata_snapshot, false, {}, false, nullptr, share_lock, lock); if (!merge_entry) mutate_entry = selectPartsToMutate(metadata_snapshot, nullptr, share_lock); + + has_mutations = !current_mutations_by_version.empty(); + } + + if (!mutate_entry && has_mutations) + { + /// Notify in case of errors + std::lock_guard lock(mutation_wait_mutex); + mutation_wait_event.notify_all(); } if (merge_entry) diff --git a/src/Storages/ya.make b/src/Storages/ya.make index 476449e8e6c..b3494849441 100644 --- a/src/Storages/ya.make +++ b/src/Storages/ya.make @@ -141,6 +141,7 @@ SRCS( StorageMerge.cpp StorageMergeTree.cpp StorageMongoDB.cpp + StorageMongoDBSocketFactory.cpp StorageMySQL.cpp StorageNull.cpp StorageReplicatedMergeTree.cpp diff --git a/tests/clickhouse-test b/tests/clickhouse-test index b734af0bdea..f6833cfbd09 100755 --- a/tests/clickhouse-test +++ b/tests/clickhouse-test @@ -647,7 +647,7 @@ def run_tests_array(all_tests_with_params): failures_chain += 1 status += MSG_FAIL status += print_test_time(total_time) - status += " - having exception:\n{}\n".format( + status += " - having exception in stdout:\n{}\n".format( '\n'.join(stdout.split('\n')[:100])) status += 'Database: ' + testcase_args.testcase_database elif reference_file is None: diff --git a/tests/queries/0_stateless/00829_bitmap_function.reference b/tests/queries/0_stateless/00829_bitmap_function.reference index 02633af568d..f403907bce4 100644 --- a/tests/queries/0_stateless/00829_bitmap_function.reference +++ b/tests/queries/0_stateless/00829_bitmap_function.reference @@ -91,6 +91,14 @@ tag4 [0,1,2,3,4,5,6,7,8,9] [5,999,2] [2,888,20] [0,1,3,4,6,7,8,9,20] [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500] [30,31,32,33,100,200,500] [100,200,500] +[] +[] +[1,5,7,9] +[5,7,9] +[5,7] +[0,1,2,3,4,5,6,7,8,9] +[30,31,32,33,100,200,500] +[100,200,500] 0 0 0 diff --git a/tests/queries/0_stateless/00829_bitmap_function.sql b/tests/queries/0_stateless/00829_bitmap_function.sql index b9e9664a56e..fde0176de5b 100644 --- a/tests/queries/0_stateless/00829_bitmap_function.sql +++ b/tests/queries/0_stateless/00829_bitmap_function.sql @@ -286,6 +286,25 @@ select bitmapToArray(bitmapSubsetLimit(bitmapBuild([ 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33, 100,200,500]), toUInt32(100), toUInt16(200))); +-- subBitmap: +---- Empty +SELECT bitmapToArray(subBitmap(bitmapBuild(emptyArrayUInt32()), toUInt8(0), toUInt32(10))); +SELECT bitmapToArray(subBitmap(bitmapBuild(emptyArrayUInt16()), toUInt32(0), toUInt64(10))); +---- Small +select bitmapToArray(subBitmap(bitmapBuild([1,5,7,9]), toUInt8(0), toUInt32(4))); +select bitmapToArray(subBitmap(bitmapBuild([1,5,7,9]), toUInt32(1), toUInt64(4))); +select bitmapToArray(subBitmap(bitmapBuild([1,5,7,9]), toUInt16(1), toUInt32(2))); +---- Large +select bitmapToArray(subBitmap(bitmapBuild([ + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33, + 100,200,500]), toUInt32(0), toUInt32(10))); +select bitmapToArray(subBitmap(bitmapBuild([ + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33, + 100,200,500]), toUInt32(30), toUInt32(200))); +select bitmapToArray(subBitmap(bitmapBuild([ + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33, + 100,200,500]), toUInt32(34), toUInt16(3))); + -- bitmapMin: ---- Empty SELECT bitmapMin(bitmapBuild(emptyArrayUInt8())); diff --git a/tests/queries/0_stateless/01410_nullable_key_and_index.sql b/tests/queries/0_stateless/01410_nullable_key_and_index.sql index 24ddb226c16..ba473b5c29a 100644 --- a/tests/queries/0_stateless/01410_nullable_key_and_index.sql +++ b/tests/queries/0_stateless/01410_nullable_key_and_index.sql @@ -49,15 +49,11 @@ SET force_primary_key = 0; SELECT * FROM nullable_minmax_index ORDER BY k; SET max_rows_to_read = 6; SELECT * FROM nullable_minmax_index WHERE v IS NULL; --- NOTE: granuals with Null values cannot be filtred in data skipping indexes, --- due to backward compatibility -SET max_rows_to_read = 0; +SET max_rows_to_read = 8; SELECT * FROM nullable_minmax_index WHERE v IS NOT NULL; SET max_rows_to_read = 6; SELECT * FROM nullable_minmax_index WHERE v > 2; --- NOTE: granuals with Null values cannot be filtred in data skipping indexes, --- due to backward compatibility -SET max_rows_to_read = 0; +SET max_rows_to_read = 4; SELECT * FROM nullable_minmax_index WHERE v <= 2; DROP TABLE nullable_key; diff --git a/tests/queries/0_stateless/01572_kill_window_function.reference b/tests/queries/0_stateless/01572_kill_window_function.reference new file mode 100644 index 00000000000..f1218bf5bdf --- /dev/null +++ b/tests/queries/0_stateless/01572_kill_window_function.reference @@ -0,0 +1,3 @@ +Started +Sent kill request +Exit 138 diff --git a/tests/queries/0_stateless/01572_kill_window_function.sh b/tests/queries/0_stateless/01572_kill_window_function.sh new file mode 100755 index 00000000000..7103b7f7210 --- /dev/null +++ b/tests/queries/0_stateless/01572_kill_window_function.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +set -e -o pipefail + +# Run a test query that takes very long to run. +query_id="01572_kill_window_function-$CLICKHOUSE_DATABASE" +$CLICKHOUSE_CLIENT --query_id="$query_id" --query "SELECT count(1048575) OVER (PARTITION BY intDiv(NULL, number) ORDER BY number DESC NULLS FIRST ROWS BETWEEN CURRENT ROW AND 1048575 FOLLOWING) FROM numbers(255, 1048575)" >/dev/null 2>&1 & +client_pid=$! +echo Started + +# Use one query to both kill the test query and verify that it has started, +# because if we try to kill it before it starts, the test will fail. +while [ -z "$($CLICKHOUSE_CLIENT --query "kill query where query_id = '$query_id' and current_database = currentDatabase()")" ] +do + # If we don't yet see the query in the process list, the client should still + # be running. The query is very long. + kill -0 -- $client_pid + sleep 1 +done +echo Sent kill request + +# Wait for the client to terminate. +client_exit_code=0 +wait $client_pid || client_exit_code=$? + +echo "Exit $client_exit_code" + +# We have tested for Ctrl+C. +# The following client flags don't cancel, but should: --max_execution_time, +# --receive_timeout. Probably needs asynchonous calculation of query limits, as +# discussed with Nikolay on TG: https://t.me/c/1214350934/21492 + diff --git a/tests/queries/0_stateless/02003_memory_limit_in_client.expect b/tests/queries/0_stateless/02003_memory_limit_in_client.expect new file mode 100755 index 00000000000..49b81240829 --- /dev/null +++ b/tests/queries/0_stateless/02003_memory_limit_in_client.expect @@ -0,0 +1,40 @@ +#!/usr/bin/expect -f + +# This is a test for system.warnings. Testing in interactive mode is necessary, +# as we want to see certain warnings from client + +log_user 0 +set timeout 60 +match_max 100000 + +# A default timeout action is to do nothing, change it to fail +expect_after { + timeout { + exit 1 + } +} + +set basedir [file dirname $argv0] +spawn bash -c "source $basedir/../shell_config.sh ; \$CLICKHOUSE_CLIENT_BINARY \$CLICKHOUSE_CLIENT_OPT --disable_suggestion --max_memory_usage_in_client=1" +expect ":) " + +send -- "SELECT arrayMap(x -> range(x), range(number)) FROM numbers(1000)\r" +expect "Code: 241" + +expect ":) " + +# Exit. +send -- "\4" +expect eof + +set basedir [file dirname $argv0] +spawn bash -c "source $basedir/../shell_config.sh ; \$CLICKHOUSE_CLIENT_BINARY \$CLICKHOUSE_CLIENT_OPT --disable_suggestion --max_memory_usage_in_client=1" +expect ":) " + +send -- "SELECT * FROM (SELECT * FROM system.numbers LIMIT 600000) as num WHERE num.number=60000\r" +expect "60000" +expect ":) " + +# Exit. +send -- "\4" +expect eof diff --git a/tests/queries/0_stateless/02003_memory_limit_in_client.reference b/tests/queries/0_stateless/02003_memory_limit_in_client.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/02004_invalid_partition_mutation_stuck.reference b/tests/queries/0_stateless/02004_invalid_partition_mutation_stuck.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/02004_invalid_partition_mutation_stuck.sql b/tests/queries/0_stateless/02004_invalid_partition_mutation_stuck.sql new file mode 100644 index 00000000000..481a5565095 --- /dev/null +++ b/tests/queries/0_stateless/02004_invalid_partition_mutation_stuck.sql @@ -0,0 +1,33 @@ +SET mutations_sync=2; + +DROP TABLE IF EXISTS rep_data; +CREATE TABLE rep_data +( + p Int, + t DateTime, + INDEX idx t TYPE minmax GRANULARITY 1 +) +ENGINE = ReplicatedMergeTree('/clickhouse/tables/{database}/rep_data', '1') +PARTITION BY p +ORDER BY t +SETTINGS number_of_free_entries_in_pool_to_execute_mutation=0; +INSERT INTO rep_data VALUES (1, now()); +ALTER TABLE rep_data MATERIALIZE INDEX idx IN PARTITION ID 'NO_SUCH_PART'; -- { serverError 248 } +ALTER TABLE rep_data MATERIALIZE INDEX idx IN PARTITION ID '1'; +ALTER TABLE rep_data MATERIALIZE INDEX idx IN PARTITION ID '2'; + +DROP TABLE IF EXISTS data; +CREATE TABLE data +( + p Int, + t DateTime, + INDEX idx t TYPE minmax GRANULARITY 1 +) +ENGINE = MergeTree +PARTITION BY p +ORDER BY t +SETTINGS number_of_free_entries_in_pool_to_execute_mutation=0; +INSERT INTO data VALUES (1, now()); +ALTER TABLE data MATERIALIZE INDEX idx IN PARTITION ID 'NO_SUCH_PART'; -- { serverError 341 } +ALTER TABLE data MATERIALIZE INDEX idx IN PARTITION ID '1'; +ALTER TABLE data MATERIALIZE INDEX idx IN PARTITION ID '2'; diff --git a/tests/queries/0_stateless/02007_join_use_nulls.reference b/tests/queries/0_stateless/02007_join_use_nulls.reference new file mode 100644 index 00000000000..30ee87bf91d --- /dev/null +++ b/tests/queries/0_stateless/02007_join_use_nulls.reference @@ -0,0 +1,8 @@ +1 2 3 1 3 +1 UInt8 2 UInt8 3 Nullable(UInt8) +1 LowCardinality(UInt8) 2 LowCardinality(UInt8) 3 LowCardinality(Nullable(UInt8)) +1 LowCardinality(UInt8) 2 LowCardinality(UInt8) 1 LowCardinality(Nullable(UInt8)) +1 UInt8 2 UInt8 3 Nullable(UInt8) +1 UInt8 2 UInt8 1 Nullable(UInt8) 3 Nullable(UInt8) +1 LowCardinality(UInt8) 2 LowCardinality(UInt8) 3 LowCardinality(Nullable(UInt8)) +1 LowCardinality(UInt8) 2 LowCardinality(UInt8) 1 LowCardinality(Nullable(UInt8)) 3 LowCardinality(Nullable(UInt8)) diff --git a/tests/queries/0_stateless/02007_join_use_nulls.sql b/tests/queries/0_stateless/02007_join_use_nulls.sql new file mode 100644 index 00000000000..e08fffce3b7 --- /dev/null +++ b/tests/queries/0_stateless/02007_join_use_nulls.sql @@ -0,0 +1,11 @@ +SET join_use_nulls = 1; + +SELECT *, d.* FROM ( SELECT 1 AS id, 2 AS value ) a SEMI LEFT JOIN ( SELECT 1 AS id, 3 AS values ) AS d USING id; + +SELECT id, toTypeName(id), value, toTypeName(value), d.values, toTypeName(d.values) FROM ( SELECT 1 AS id, 2 AS value ) a SEMI LEFT JOIN ( SELECT 1 AS id, 3 AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.values, toTypeName(d.values) FROM ( SELECT toLowCardinality(1) AS id, toLowCardinality(2) AS value ) a SEMI LEFT JOIN ( SELECT toLowCardinality(1) AS id, toLowCardinality(3) AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.id, toTypeName(d.id) FROM ( SELECT toLowCardinality(1) AS id, toLowCardinality(2) AS value ) a SEMI LEFT JOIN ( SELECT toLowCardinality(1) AS id, toLowCardinality(3) AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.values, toTypeName(d.values) FROM ( SELECT 1 AS id, 2 AS value ) a SEMI LEFT JOIN ( SELECT 1 AS id, 3 AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.id, toTypeName(d.id) , d.values, toTypeName(d.values) FROM ( SELECT 1 AS id, 2 AS value ) a SEMI LEFT JOIN ( SELECT 1 AS id, 3 AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.values, toTypeName(d.values) FROM ( SELECT toLowCardinality(1) AS id, toLowCardinality(2) AS value ) a SEMI LEFT JOIN ( SELECT toLowCardinality(1) AS id, toLowCardinality(3) AS values ) AS d USING id; +SELECT id, toTypeName(id), value, toTypeName(value), d.id, toTypeName(d.id) , d.values, toTypeName(d.values) FROM ( SELECT toLowCardinality(1) AS id, toLowCardinality(2) AS value ) a SEMI LEFT JOIN ( SELECT toLowCardinality(1) AS id, toLowCardinality(3) AS values ) AS d USING id; diff --git a/website/README.md b/website/README.md index 9f808c6f658..f02deb0ad91 100644 --- a/website/README.md +++ b/website/README.md @@ -23,7 +23,7 @@ virtualenv build ./build.py --skip-multi-page --skip-single-page --skip-amp --skip-pdf --skip-git-log --skip-docs --livereload 8080 ``` -# How to quickly test the ugly annoying broken links in docs +# How to quickly test the broken links in docs ``` ./build.py --skip-multi-page --skip-amp --skip-pdf --skip-blog --skip-git-log --lang en --livereload 8080 diff --git a/website/templates/index/success.html b/website/templates/index/success.html index 83b5c1427c9..a93efa8bdc5 100644 --- a/website/templates/index/success.html +++ b/website/templates/index/success.html @@ -2,19 +2,9 @@