diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 257040c68b7..8e502c0b36f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,4 +1,2 @@ -dbms/* @ClickHouse/core-assigner -utils/* @ClickHouse/core-assigner docs/* @ClickHouse/docs docs/zh/* @ClickHouse/docs-zh diff --git a/.gitmodules b/.gitmodules index f9afe2eb79c..8333e5544bc 100644 --- a/.gitmodules +++ b/.gitmodules @@ -134,3 +134,6 @@ [submodule "contrib/libc-headers"] path = contrib/libc-headers url = https://github.com/ClickHouse-Extras/libc-headers.git +[submodule "contrib/ryu"] + path = contrib/ryu + url = https://github.com/ClickHouse-Extras/ryu.git diff --git a/CHANGELOG.md b/CHANGELOG.md index a6757c38898..f456a56f1be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,8 @@ -## ClickHouse release v19.17.6.36, 2019-12-27 +## ClickHouse release v19.17 -### Bug Fix +### ClickHouse release v19.17.6.36, 2019-12-27 + +#### Bug Fix * Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that could cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. [#8404](https://github.com/ClickHouse/ClickHouse/pull/8404) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed possible server crash (`std::terminate`) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. [#8384](https://github.com/ClickHouse/ClickHouse/pull/8384) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed dictionaries with source from a clickhouse `VIEW`, now reading such dictionaries doesn't cause the error `There is no query`. [#8351](https://github.com/ClickHouse/ClickHouse/pull/8351) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -32,13 +34,12 @@ next request would interpret this info as the beginning of the next query causin * Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. And now it's possible to use TOP with LIMIT BY. [#7637](https://github.com/ClickHouse/ClickHouse/pull/7637) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Fix dictionary reload if it has `invalidate_query`, which stopped updates and some exception on previous update tries. [#8029](https://github.com/ClickHouse/ClickHouse/pull/8029) ([alesapin](https://github.com/alesapin)) +### ClickHouse release v19.17.4.11, 2019-11-22 -## ClickHouse release v19.17.4.11, 2019-11-22 - -### Backward Incompatible Change +#### Backward Incompatible Change * Using column instead of AST to store scalar subquery results for better performance. Setting `enable_scalar_subquery_optimization` was added in 19.17 and it was enabled by default. It leads to errors like [this](https://github.com/ClickHouse/ClickHouse/issues/7851) during upgrade to 19.17.2 or 19.17.3 from previous versions. This setting was disabled by default in 19.17.4, to make possible upgrading from 19.16 and older versions without errors. [#7392](https://github.com/ClickHouse/ClickHouse/pull/7392) ([Amos Bird](https://github.com/amosbird)) -### New Feature +#### New Feature * Add the ability to create dictionaries with DDL queries. [#7360](https://github.com/ClickHouse/ClickHouse/pull/7360) ([alesapin](https://github.com/alesapin)) * Make `bloom_filter` type of index supporting `LowCardinality` and `Nullable` [#7363](https://github.com/ClickHouse/ClickHouse/issues/7363) [#7561](https://github.com/ClickHouse/ClickHouse/pull/7561) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Add function `isValidJSON` to check that passed string is a valid json. [#5910](https://github.com/ClickHouse/ClickHouse/issues/5910) [#7293](https://github.com/ClickHouse/ClickHouse/pull/7293) ([Vdimir](https://github.com/Vdimir)) @@ -51,10 +52,10 @@ next request would interpret this info as the beginning of the next query causin * Implemented `javaHashUTF16LE()` function [#7651](https://github.com/ClickHouse/ClickHouse/pull/7651) ([achimbab](https://github.com/achimbab)) * Add `_shard_num` virtual column for the Distributed engine [#7624](https://github.com/ClickHouse/ClickHouse/pull/7624) ([Azat Khuzhin](https://github.com/azat)) -### Experimental Feature +#### Experimental Feature * Support for processors (new query execution pipeline) in `MergeTree`. [#7181](https://github.com/ClickHouse/ClickHouse/pull/7181) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -### Bug Fix +#### Bug Fix * Fix incorrect float parsing in `Values` [#7817](https://github.com/ClickHouse/ClickHouse/issues/7817) [#7870](https://github.com/ClickHouse/ClickHouse/pull/7870) ([tavplubix](https://github.com/tavplubix)) * Fix rare deadlock which can happen when trace_log is enabled. [#7838](https://github.com/ClickHouse/ClickHouse/pull/7838) ([filimonov](https://github.com/filimonov)) * Prevent message duplication when producing Kafka table has any MVs selecting from it [#7265](https://github.com/ClickHouse/ClickHouse/pull/7265) ([Ivan](https://github.com/abyss7)) @@ -78,7 +79,7 @@ next request would interpret this info as the beginning of the next query causin * Fixed exception in case of using 1 argument while defining S3, URL and HDFS storages. [#7618](https://github.com/ClickHouse/ClickHouse/pull/7618) ([Vladimir Chebotarev](https://github.com/excitoon)) * Fix scope of the InterpreterSelectQuery for views with query [#7601](https://github.com/ClickHouse/ClickHouse/pull/7601) ([Azat Khuzhin](https://github.com/azat)) -### Improvement +#### Improvement * `Nullable` columns recognized and NULL-values handled correctly by ODBC-bridge [#7402](https://github.com/ClickHouse/ClickHouse/pull/7402) ([Vasily Nemkov](https://github.com/Enmk)) * Write current batch for distributed send atomically [#7600](https://github.com/ClickHouse/ClickHouse/pull/7600) ([Azat Khuzhin](https://github.com/azat)) * Throw an exception if we cannot detect table for column name in query. [#7358](https://github.com/ClickHouse/ClickHouse/pull/7358) ([Artem Zuikov](https://github.com/4ertus2)) @@ -90,14 +91,14 @@ next request would interpret this info as the beginning of the next query causin * Better Null format for tcp handler, so that it's possible to use `select ignore() from table format Null` for perf measure via clickhouse-client [#7606](https://github.com/ClickHouse/ClickHouse/pull/7606) ([Amos Bird](https://github.com/amosbird)) * Queries like `CREATE TABLE ... AS (SELECT (1, 2))` are parsed correctly [#7542](https://github.com/ClickHouse/ClickHouse/pull/7542) ([hcz](https://github.com/hczhcz)) -### Performance Improvement +#### Performance Improvement * The performance of aggregation over short string keys is improved. [#6243](https://github.com/ClickHouse/ClickHouse/pull/6243) ([Alexander Kuzmenkov](https://github.com/akuzm), [Amos Bird](https://github.com/amosbird)) * Run another pass of syntax/expression analysis to get potential optimizations after constant predicates are folded. [#7497](https://github.com/ClickHouse/ClickHouse/pull/7497) ([Amos Bird](https://github.com/amosbird)) * Use storage meta info to evaluate trivial `SELECT count() FROM table;` [#7510](https://github.com/ClickHouse/ClickHouse/pull/7510) ([Amos Bird](https://github.com/amosbird), [alexey-milovidov](https://github.com/alexey-milovidov)) * Vectorize processing `arrayReduce` similar to Aggregator `addBatch`. [#7608](https://github.com/ClickHouse/ClickHouse/pull/7608) ([Amos Bird](https://github.com/amosbird)) * Minor improvements in performance of `Kafka` consumption [#7475](https://github.com/ClickHouse/ClickHouse/pull/7475) ([Ivan](https://github.com/abyss7)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Add support for cross-compiling to the CPU architecture AARCH64. Refactor packager script. [#7370](https://github.com/ClickHouse/ClickHouse/pull/7370) [#7539](https://github.com/ClickHouse/ClickHouse/pull/7539) ([Ivan](https://github.com/abyss7)) * Unpack darwin-x86_64 and linux-aarch64 toolchains into mounted Docker volume when building packages [#7534](https://github.com/ClickHouse/ClickHouse/pull/7534) ([Ivan](https://github.com/abyss7)) * Update Docker Image for Binary Packager [#7474](https://github.com/ClickHouse/ClickHouse/pull/7474) ([Ivan](https://github.com/abyss7)) @@ -108,13 +109,14 @@ next request would interpret this info as the beginning of the next query causin * Remove hardcoded paths in `unwind` target [#7460](https://github.com/ClickHouse/ClickHouse/pull/7460) ([Konstantin Podshumok](https://github.com/podshumok)) * Allow to use mysql format without ssl [#7524](https://github.com/ClickHouse/ClickHouse/pull/7524) ([proller](https://github.com/proller)) -### Other +#### Other * Added ANTLR4 grammar for ClickHouse SQL dialect [#7595](https://github.com/ClickHouse/ClickHouse/issues/7595) [#7596](https://github.com/ClickHouse/ClickHouse/pull/7596) ([alexey-milovidov](https://github.com/alexey-milovidov)) +## ClickHouse release v19.16 -## ClickHouse release v19.16.2.2, 2019-10-30 +### ClickHouse release v19.16.2.2, 2019-10-30 -### Backward Incompatible Change +#### Backward Incompatible Change * Add missing arity validation for count/counIf. [#7095](https://github.com/ClickHouse/ClickHouse/issues/7095) [#7298](https://github.com/ClickHouse/ClickHouse/pull/7298) ([Vdimir](https://github.com/Vdimir)) @@ -125,7 +127,7 @@ Zuikov](https://github.com/4ertus2)) [#7118](https://github.com/ClickHouse/ClickHouse/pull/7118) ([tavplubix](https://github.com/tavplubix)) -### New Feature +#### New Feature * Introduce uniqCombined64() to calculate cardinality greater than UINT_MAX. [#7213](https://github.com/ClickHouse/ClickHouse/pull/7213), [#7222](https://github.com/ClickHouse/ClickHouse/pull/7222) ([Azat @@ -166,7 +168,7 @@ Yu](https://github.com/yuzhichang)) * Support Redis as source of external dictionary. [#4361](https://github.com/ClickHouse/ClickHouse/pull/4361) [#6962](https://github.com/ClickHouse/ClickHouse/pull/6962) ([comunodi](https://github.com/comunodi), [Anton Popov](https://github.com/CurtizJ)) -### Bug Fix +#### Bug Fix * Fix wrong query result if it has `WHERE IN (SELECT ...)` section and `optimize_read_in_order` is used. [#7371](https://github.com/ClickHouse/ClickHouse/pull/7371) ([Anton Popov](https://github.com/CurtizJ)) @@ -207,7 +209,7 @@ Kochetov](https://github.com/KochetovNicolai)) [#7271](https://github.com/ClickHouse/ClickHouse/pull/7271) ([vzakaznikov](https://github.com/vzakaznikov)) -### Improvement +#### Improvement * Add a message in case of queue_wait_max_ms wait takes place. [#7390](https://github.com/ClickHouse/ClickHouse/pull/7390) ([Azat Khuzhin](https://github.com/azat)) @@ -252,7 +254,7 @@ Khuzhin](https://github.com/azat)) memory). Load data back when needed. [#7186](https://github.com/ClickHouse/ClickHouse/pull/7186) ([Artem Zuikov](https://github.com/4ertus2)) -### Performance Improvement +#### Performance Improvement * Speed up joinGet with const arguments by avoiding data duplication. [#7359](https://github.com/ClickHouse/ClickHouse/pull/7359) ([Amos Bird](https://github.com/amosbird)) @@ -262,7 +264,7 @@ Bird](https://github.com/amosbird)) [#6781](https://github.com/ClickHouse/ClickHouse/pull/6781) ([tavplubix](https://github.com/tavplubix)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Disable some contribs for cross-compilation to Mac OS. [#7101](https://github.com/ClickHouse/ClickHouse/pull/7101) ([Ivan](https://github.com/abyss7)) * Add missing linking with PocoXML for clickhouse_common_io. @@ -330,7 +332,7 @@ Bird](https://github.com/amosbird)) [#7063](https://github.com/ClickHouse/ClickHouse/pull/7063) ([proller](https://github.com/proller)) -### Code cleanup +#### Code cleanup * Generalize configuration repository to prepare for DDL for Dictionaries. [#7155](https://github.com/ClickHouse/ClickHouse/pull/7155) ([alesapin](https://github.com/alesapin)) * Parser for dictionaries DDL without any semantic. @@ -364,9 +366,11 @@ fix comments to make obvious that it may throw. [#7350](https://github.com/ClickHouse/ClickHouse/pull/7350) ([tavplubix](https://github.com/tavplubix)) -## ClickHouse release 19.15.4.10, 2019-10-31 +## ClickHouse release 19.15 -### Bug Fix +### ClickHouse release 19.15.4.10, 2019-10-31 + +#### Bug Fix * Added handling of SQL_TINYINT and SQL_BIGINT, and fix handling of SQL_FLOAT data source types in ODBC Bridge. [#7491](https://github.com/ClickHouse/ClickHouse/pull/7491) ([Denis Glazachev](https://github.com/traceon)) * Allowed to have some parts on destination disk or volume in MOVE PARTITION. @@ -391,9 +395,9 @@ fix comments to make obvious that it may throw. [#7158](https://github.com/ClickHouse/ClickHouse/pull/7158) ([Azat Khuzhin](https://github.com/azat)) * Added example config with macros for tests ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.15.3.6, 2019-10-09 +### ClickHouse release 19.15.3.6, 2019-10-09 -### Bug Fix +#### Bug Fix * Fixed bad_variant in hashed dictionary. ([alesapin](https://github.com/alesapin)) * Fixed up bug with segmentation fault in ATTACH PART query. @@ -405,9 +409,9 @@ fix comments to make obvious that it may throw. * Serialize NULL values correctly in min/max indexes of MergeTree parts. [#7234](https://github.com/ClickHouse/ClickHouse/pull/7234) ([Alexander Kuzmenkov](https://github.com/akuzm)) -## ClickHouse release 19.15.2.2, 2019-10-01 +### ClickHouse release 19.15.2.2, 2019-10-01 -### New Feature +#### New Feature * Tiered storage: support to use multiple storage volumes for tables with MergeTree engine. It's possible to store fresh data on SSD and automatically move old data to HDD. ([example](https://clickhouse.github.io/clickhouse-presentations/meetup30/new_features/#12)). [#4918](https://github.com/ClickHouse/ClickHouse/pull/4918) ([Igr](https://github.com/ObjatieGroba)) [#6489](https://github.com/ClickHouse/ClickHouse/pull/6489) ([alesapin](https://github.com/alesapin)) * Add table function `input` for reading incoming data in `INSERT SELECT` query. [#5450](https://github.com/ClickHouse/ClickHouse/pull/5450) ([palasonic1](https://github.com/palasonic1)) [#6832](https://github.com/ClickHouse/ClickHouse/pull/6832) ([Anton Popov](https://github.com/CurtizJ)) * Add a `sparse_hashed` dictionary layout, that is functionally equivalent to the `hashed` layout, but is more memory efficient. It uses about twice as less memory at the cost of slower value retrieval. [#6894](https://github.com/ClickHouse/ClickHouse/pull/6894) ([Azat Khuzhin](https://github.com/azat)) @@ -417,11 +421,11 @@ fix comments to make obvious that it may throw. * Add `bitmapMin` and `bitmapMax` functions. [#6970](https://github.com/ClickHouse/ClickHouse/pull/6970) ([Zhichang Yu](https://github.com/yuzhichang)) * Add function `repeat` related to [issue-6648](https://github.com/yandex/ClickHouse/issues/6648) [#6999](https://github.com/ClickHouse/ClickHouse/pull/6999) ([flynn](https://github.com/ucasFL)) -### Experimental Feature +#### Experimental Feature * Implement (in memory) Merge Join variant that does not change current pipeline. Result is partially sorted by merge key. Set `partial_merge_join = 1` to use this feature. The Merge Join is still in development. [#6940](https://github.com/ClickHouse/ClickHouse/pull/6940) ([Artem Zuikov](https://github.com/4ertus2)) * Add `S3` engine and table function. It is still in development (no authentication support yet). [#5596](https://github.com/ClickHouse/ClickHouse/pull/5596) ([Vladimir Chebotarev](https://github.com/excitoon)) -### Improvement +#### Improvement * Every message read from Kafka is inserted atomically. This resolves almost all known issues with Kafka engine. [#6950](https://github.com/ClickHouse/ClickHouse/pull/6950) ([Ivan](https://github.com/abyss7)) * Improvements for failover of Distributed queries. Shorten recovery time, also it is now configurable and can be seen in `system.clusters`. [#6399](https://github.com/ClickHouse/ClickHouse/pull/6399) ([Vasily Nemkov](https://github.com/Enmk)) * Support numeric values for Enums directly in `IN` section. #6766 [#6941](https://github.com/ClickHouse/ClickHouse/pull/6941) ([dimarub2000](https://github.com/dimarub2000)) @@ -432,7 +436,7 @@ fix comments to make obvious that it may throw. * Add automatically cast type `T` to `LowCardinality(T)` while inserting data in column of type `LowCardinality(T)` in Native format via HTTP. [#6891](https://github.com/ClickHouse/ClickHouse/pull/6891) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Add ability to use function `hex` without using `reinterpretAsString` for `Float32`, `Float64`. [#7024](https://github.com/ClickHouse/ClickHouse/pull/7024) ([Mikhail Korotov](https://github.com/millb)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Add gdb-index to clickhouse binary with debug info. It will speed up startup time of `gdb`. [#6947](https://github.com/ClickHouse/ClickHouse/pull/6947) ([alesapin](https://github.com/alesapin)) * Speed up deb packaging with patched dpkg-deb which uses `pigz`. [#6960](https://github.com/ClickHouse/ClickHouse/pull/6960) ([alesapin](https://github.com/alesapin)) * Set `enable_fuzzing = 1` to enable libfuzzer instrumentation of all the project code. [#7042](https://github.com/ClickHouse/ClickHouse/pull/7042) ([kyprizel](https://github.com/kyprizel)) @@ -440,7 +444,7 @@ fix comments to make obvious that it may throw. * Add build with MemorySanitizer to CI. [#7066](https://github.com/ClickHouse/ClickHouse/pull/7066) ([Alexander Kuzmenkov](https://github.com/akuzm)) * Replace `libsparsehash` with `sparsehash-c11` [#6965](https://github.com/ClickHouse/ClickHouse/pull/6965) ([Azat Khuzhin](https://github.com/azat)) -### Bug Fix +#### Bug Fix * Fixed performance degradation of index analysis on complex keys on large tables. This fixes #6924. [#7075](https://github.com/ClickHouse/ClickHouse/pull/7075) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix logical error causing segfaults when selecting from Kafka empty topic. [#6909](https://github.com/ClickHouse/ClickHouse/pull/6909) ([Ivan](https://github.com/abyss7)) * Fix too early MySQL connection close in `MySQLBlockInputStream.cpp`. [#6882](https://github.com/ClickHouse/ClickHouse/pull/6882) ([Clément Rodriguez](https://github.com/clemrodriguez)) @@ -451,28 +455,29 @@ fix comments to make obvious that it may throw. * Fix `Unknown identifier` error in ORDER BY and GROUP BY with multiple JOINs [#7022](https://github.com/ClickHouse/ClickHouse/pull/7022) ([Artem Zuikov](https://github.com/4ertus2)) * Fixed `MSan` warning while executing function with `LowCardinality` argument. [#7062](https://github.com/ClickHouse/ClickHouse/pull/7062) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -### Backward Incompatible Change +#### Backward Incompatible Change * Changed serialization format of bitmap* aggregate function states to improve performance. Serialized states of bitmap* from previous versions cannot be read. [#6908](https://github.com/ClickHouse/ClickHouse/pull/6908) ([Zhichang Yu](https://github.com/yuzhichang)) -## ClickHouse release 19.14.7.15, 2019-10-02 +## ClickHouse release 19.14 +### ClickHouse release 19.14.7.15, 2019-10-02 -### Bug Fix +#### Bug Fix * This release also contains all bug fixes from 19.11.12.69. * Fixed compatibility for distributed queries between 19.14 and earlier versions. This fixes [#7068](https://github.com/ClickHouse/ClickHouse/issues/7068). [#7069](https://github.com/ClickHouse/ClickHouse/pull/7069) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.14.6.12, 2019-09-19 +### ClickHouse release 19.14.6.12, 2019-09-19 -### Bug Fix +#### Bug Fix * Fix for function `АrrayEnumerateUniqRanked` with empty arrays in params. [#6928](https://github.com/ClickHouse/ClickHouse/pull/6928) ([proller](https://github.com/proller)) * Fixed subquery name in queries with `ARRAY JOIN` and `GLOBAL IN subquery` with alias. Use subquery alias for external table name if it is specified. [#6934](https://github.com/ClickHouse/ClickHouse/pull/6934) ([Ivan](https://github.com/abyss7)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Fix [flapping](https://clickhouse-test-reports.s3.yandex.net/6944/aab95fd5175a513413c7395a73a82044bdafb906/functional_stateless_tests_(debug).html) test `00715_fetch_merged_or_mutated_part_zookeeper` by rewriting it to a shell scripts because it needs to wait for mutations to apply. [#6977](https://github.com/ClickHouse/ClickHouse/pull/6977) ([Alexander Kazakov](https://github.com/Akazz)) * Fixed UBSan and MemSan failure in function `groupUniqArray` with emtpy array argument. It was caused by placing of empty `PaddedPODArray` into hash table zero cell because constructor for zero cell value was not called. [#6937](https://github.com/ClickHouse/ClickHouse/pull/6937) ([Amos Bird](https://github.com/amosbird)) -## ClickHouse release 19.14.3.3, 2019-09-10 +### ClickHouse release 19.14.3.3, 2019-09-10 -### New Feature +#### New Feature * `WITH FILL` modifier for `ORDER BY`. (continuation of [#5069](https://github.com/ClickHouse/ClickHouse/issues/5069)) [#6610](https://github.com/ClickHouse/ClickHouse/pull/6610) ([Anton Popov](https://github.com/CurtizJ)) * `WITH TIES` modifier for `LIMIT`. (continuation of [#5069](https://github.com/ClickHouse/ClickHouse/issues/5069)) [#6610](https://github.com/ClickHouse/ClickHouse/pull/6610) ([Anton Popov](https://github.com/CurtizJ)) * Parse unquoted `NULL` literal as NULL (if setting `format_csv_unquoted_null_literal_as_null=1`). Initialize null fields with default values if data type of this field is not nullable (if setting `input_format_null_as_default=1`). [#5990](https://github.com/ClickHouse/ClickHouse/issues/5990) [#6055](https://github.com/ClickHouse/ClickHouse/pull/6055) ([tavplubix](https://github.com/tavplubix)) @@ -498,11 +503,11 @@ fix comments to make obvious that it may throw. * Added support for `_partition` and `_timestamp` virtual columns to Kafka engine. [#6400](https://github.com/ClickHouse/ClickHouse/pull/6400) ([Ivan](https://github.com/abyss7)) * Possibility to remove sensitive data from `query_log`, server logs, process list with regexp-based rules. [#5710](https://github.com/ClickHouse/ClickHouse/pull/5710) ([filimonov](https://github.com/filimonov)) -### Experimental Feature +#### Experimental Feature * Input and output data format `Template`. It allows to specify custom format string for input and output. [#4354](https://github.com/ClickHouse/ClickHouse/issues/4354) [#6727](https://github.com/ClickHouse/ClickHouse/pull/6727) ([tavplubix](https://github.com/tavplubix)) * Implementation of `LIVE VIEW` tables that were originally proposed in [#2898](https://github.com/ClickHouse/ClickHouse/pull/2898), prepared in [#3925](https://github.com/ClickHouse/ClickHouse/issues/3925), and then updated in [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541). See [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541) for detailed description. [#5541](https://github.com/ClickHouse/ClickHouse/issues/5541) ([vzakaznikov](https://github.com/vzakaznikov)) [#6425](https://github.com/ClickHouse/ClickHouse/pull/6425) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) [#6656](https://github.com/ClickHouse/ClickHouse/pull/6656) ([vzakaznikov](https://github.com/vzakaznikov)) Note that `LIVE VIEW` feature may be removed in next versions. -### Bug Fix +#### Bug Fix * This release also contains all bug fixes from 19.13 and 19.11. * Fix segmentation fault when the table has skip indices and vertical merge happens. [#6723](https://github.com/ClickHouse/ClickHouse/pull/6723) ([alesapin](https://github.com/alesapin)) * Fix per-column TTL with non-trivial column defaults. Previously in case of force TTL merge with `OPTIMIZE ... FINAL` query, expired values was replaced by type defaults instead of user-specified column defaults. [#6796](https://github.com/ClickHouse/ClickHouse/pull/6796) ([Anton Popov](https://github.com/CurtizJ)) @@ -540,7 +545,7 @@ fix comments to make obvious that it may throw. * Fix bug with writing secondary indices marks with adaptive granularity. [#6126](https://github.com/ClickHouse/ClickHouse/pull/6126) ([alesapin](https://github.com/alesapin)) * Fix initialization order while server startup. Since `StorageMergeTree::background_task_handle` is initialized in `startup()` the `MergeTreeBlockOutputStream::write()` may try to use it before initialization. Just check if it is initialized. [#6080](https://github.com/ClickHouse/ClickHouse/pull/6080) ([Ivan](https://github.com/abyss7)) * Clearing the data buffer from the previous read operation that was completed with an error. [#6026](https://github.com/ClickHouse/ClickHouse/pull/6026) ([Nikolay](https://github.com/bopohaa)) -* Fix bug with enabling adaptive granularity when creating a new replica for Replicated*MergeTree table. [#6394](https://github.com/ClickHouse/ClickHouse/issues/6394) [#6452](https://github.com/ClickHouse/ClickHouse/pull/6452) ([alesapin](https://github.com/alesapin)) +* Fix bug with enabling adaptive granularity when creating a new replica for Replicated\*MergeTree table. [#6394](https://github.com/ClickHouse/ClickHouse/issues/6394) [#6452](https://github.com/ClickHouse/ClickHouse/pull/6452) ([alesapin](https://github.com/alesapin)) * Fixed possible crash during server startup in case of exception happened in `libunwind` during exception at access to uninitialized `ThreadStatus` structure. [#6456](https://github.com/ClickHouse/ClickHouse/pull/6456) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Fix crash in `yandexConsistentHash` function. Found by fuzz test. [#6304](https://github.com/ClickHouse/ClickHouse/issues/6304) [#6305](https://github.com/ClickHouse/ClickHouse/pull/6305) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed the possibility of hanging queries when server is overloaded and global thread pool becomes near full. This have higher chance to happen on clusters with large number of shards (hundreds), because distributed queries allocate a thread per connection to each shard. For example, this issue may reproduce if a cluster of 330 shards is processing 30 concurrent distributed queries. This issue affects all versions starting from 19.2. [#6301](https://github.com/ClickHouse/ClickHouse/pull/6301) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -560,11 +565,11 @@ fix comments to make obvious that it may throw. * Typo in the error message ( is -> are ). [#6839](https://github.com/ClickHouse/ClickHouse/pull/6839) ([Denis Zhuravlev](https://github.com/den-crane)) * Fixed error while parsing of columns list from string if type contained a comma (this issue was relevant for `File`, `URL`, `HDFS` storages) [#6217](https://github.com/ClickHouse/ClickHouse/issues/6217). [#6209](https://github.com/ClickHouse/ClickHouse/pull/6209) ([dimarub2000](https://github.com/dimarub2000)) -### Security Fix +#### Security Fix * This release also contains all bug security fixes from 19.13 and 19.11. * Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser. Fixed the possibility of stack overflow in Merge and Distributed tables, materialized views and conditions for row-level security that involve subqueries. [#6433](https://github.com/ClickHouse/ClickHouse/pull/6433) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Improvement +#### Improvement * Correct implementation of ternary logic for `AND/OR`. [#6048](https://github.com/ClickHouse/ClickHouse/pull/6048) ([Alexander Kazakov](https://github.com/Akazz)) * Now values and rows with expired TTL will be removed after `OPTIMIZE ... FINAL` query from old parts without TTL infos or with outdated TTL infos, e.g. after `ALTER ... MODIFY TTL` query. Added queries `SYSTEM STOP/START TTL MERGES` to disallow/allow assign merges with TTL and filter expired values in all merges. [#6274](https://github.com/ClickHouse/ClickHouse/pull/6274) ([Anton Popov](https://github.com/CurtizJ)) * Possibility to change the location of ClickHouse history file for client using `CLICKHOUSE_HISTORY_FILE` env. [#6840](https://github.com/ClickHouse/ClickHouse/pull/6840) ([filimonov](https://github.com/filimonov)) @@ -625,7 +630,7 @@ fix comments to make obvious that it may throw. * `MergeTree` now has an additional option `ttl_only_drop_parts` (disabled by default) to avoid partial pruning of parts, so that they dropped completely when all the rows in a part are expired. [#6191](https://github.com/ClickHouse/ClickHouse/pull/6191) ([Sergi Vladykin](https://github.com/svladykin)) * Type checks for set index functions. Throw exception if function got a wrong type. This fixes fuzz test with UBSan. [#6511](https://github.com/ClickHouse/ClickHouse/pull/6511) ([Nikita Vasilev](https://github.com/nikvas0)) -### Performance Improvement +#### Performance Improvement * Optimize queries with `ORDER BY expressions` clause, where `expressions` have coinciding prefix with sorting key in `MergeTree` tables. This optimization is controlled by `optimize_read_in_order` setting. [#6054](https://github.com/ClickHouse/ClickHouse/pull/6054) [#6629](https://github.com/ClickHouse/ClickHouse/pull/6629) ([Anton Popov](https://github.com/CurtizJ)) * Allow to use multiple threads during parts loading and removal. [#6372](https://github.com/ClickHouse/ClickHouse/issues/6372) [#6074](https://github.com/ClickHouse/ClickHouse/issues/6074) [#6438](https://github.com/ClickHouse/ClickHouse/pull/6438) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Implemented batch variant of updating aggregate function states. It may lead to performance benefits. [#6435](https://github.com/ClickHouse/ClickHouse/pull/6435) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -635,7 +640,7 @@ fix comments to make obvious that it may throw. * Pre-fault pages when allocating memory with `mmap()`. [#6667](https://github.com/ClickHouse/ClickHouse/pull/6667) ([akuzm](https://github.com/akuzm)) * Fix performance bug in `Decimal` comparison. [#6380](https://github.com/ClickHouse/ClickHouse/pull/6380) ([Artem Zuikov](https://github.com/4ertus2)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Remove Compiler (runtime template instantiation) because we've win over it's performance. [#6646](https://github.com/ClickHouse/ClickHouse/pull/6646) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Added performance test to show degradation of performance in gcc-9 in more isolated way. [#6302](https://github.com/ClickHouse/ClickHouse/pull/6302) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Added table function `numbers_mt`, which is multithreaded version of `numbers`. Updated performance tests with hash functions. [#6554](https://github.com/ClickHouse/ClickHouse/pull/6554) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -712,19 +717,19 @@ fix comments to make obvious that it may throw. * Fix "splitted" build. [#6618](https://github.com/ClickHouse/ClickHouse/pull/6618) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Other build fixes: [#6186](https://github.com/ClickHouse/ClickHouse/pull/6186) ([Amos Bird](https://github.com/amosbird)) [#6486](https://github.com/ClickHouse/ClickHouse/pull/6486) [#6348](https://github.com/ClickHouse/ClickHouse/pull/6348) ([vxider](https://github.com/Vxider)) [#6744](https://github.com/ClickHouse/ClickHouse/pull/6744) ([Ivan](https://github.com/abyss7)) [#6016](https://github.com/ClickHouse/ClickHouse/pull/6016) [#6421](https://github.com/ClickHouse/ClickHouse/pull/6421) [#6491](https://github.com/ClickHouse/ClickHouse/pull/6491) ([proller](https://github.com/proller)) -### Backward Incompatible Change +#### Backward Incompatible Change * Removed rarely used table function `catBoostPool` and storage `CatBoostPool`. If you have used this table function, please write email to `clickhouse-feedback@yandex-team.com`. Note that CatBoost integration remains and will be supported. [#6279](https://github.com/ClickHouse/ClickHouse/pull/6279) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Disable `ANY RIGHT JOIN` and `ANY FULL JOIN` by default. Set `any_join_distinct_right_table_keys` setting to enable them. [#5126](https://github.com/ClickHouse/ClickHouse/issues/5126) [#6351](https://github.com/ClickHouse/ClickHouse/pull/6351) ([Artem Zuikov](https://github.com/4ertus2)) -## ClickHouse release 19.13.6.51, 2019-10-02 +## ClickHouse release 19.13 +### ClickHouse release 19.13.6.51, 2019-10-02 -### Bug Fix +#### Bug Fix * This release also contains all bug fixes from 19.11.12.69. +### ClickHouse release 19.13.5.44, 2019-09-20 -## ClickHouse release 19.13.5.44, 2019-09-20 - -### Bug Fix +#### Bug Fix * This release also contains all bug fixes from 19.14.6.12. * Fixed possible inconsistent state of table while executing `DROP` query for replicated table while zookeeper is not accessible. [#6045](https://github.com/ClickHouse/ClickHouse/issues/6045) [#6413](https://github.com/ClickHouse/ClickHouse/pull/6413) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) * Fix for data race in StorageMerge [#6717](https://github.com/ClickHouse/ClickHouse/pull/6717) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -736,9 +741,9 @@ fix comments to make obvious that it may throw. * Fixed parsing of `AggregateFunction` values embedded in query. [#6575](https://github.com/ClickHouse/ClickHouse/issues/6575) [#6773](https://github.com/ClickHouse/ClickHouse/pull/6773) ([Zhichang Yu](https://github.com/yuzhichang)) * Fixed wrong behaviour of `trim` functions family. [#6647](https://github.com/ClickHouse/ClickHouse/pull/6647) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.13.4.32, 2019-09-10 +### ClickHouse release 19.13.4.32, 2019-09-10 -### Bug Fix +#### Bug Fix * This release also contains all bug security fixes from 19.11.9.52 and 19.11.10.54. * Fixed data race in `system.parts` table and `ALTER` query. [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245) [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed mismatched header in streams happened in case of reading from empty distributed table with sample and prewhere. [#6167](https://github.com/ClickHouse/ClickHouse/issues/6167) ([Lixiang Qian](https://github.com/fancyqlx)) [#6823](https://github.com/ClickHouse/ClickHouse/pull/6823) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -749,29 +754,83 @@ fix comments to make obvious that it may throw. * Query transformation for `MySQL`, `ODBC`, `JDBC` table functions now works properly for `SELECT WHERE` queries with multiple `AND` expressions. [#6381](https://github.com/ClickHouse/ClickHouse/issues/6381) [#6676](https://github.com/ClickHouse/ClickHouse/pull/6676) ([dimarub2000](https://github.com/dimarub2000)) * Added previous declaration checks for MySQL 8 integration. [#6569](https://github.com/ClickHouse/ClickHouse/pull/6569) ([Rafael David Tinoco](https://github.com/rafaeldtinoco)) -### Security Fix +#### Security Fix * Fix two vulnerabilities in codecs in decompression phase (malicious user can fabricate compressed data that will lead to buffer overflow in decompression). [#6670](https://github.com/ClickHouse/ClickHouse/pull/6670) ([Artem Zuikov](https://github.com/4ertus2)) -## ClickHouse release 19.11.13.74, 2019-11-01 -### Bug Fix +### ClickHouse release 19.13.3.26, 2019-08-22 + +#### Bug Fix +* Fix `ALTER TABLE ... UPDATE` query for tables with `enable_mixed_granularity_parts=1`. [#6543](https://github.com/ClickHouse/ClickHouse/pull/6543) ([alesapin](https://github.com/alesapin)) +* Fix NPE when using IN clause with a subquery with a tuple. [#6125](https://github.com/ClickHouse/ClickHouse/issues/6125) [#6550](https://github.com/ClickHouse/ClickHouse/pull/6550) ([tavplubix](https://github.com/tavplubix)) +* Fixed an issue that if a stale replica becomes alive, it may still have data parts that were removed by DROP PARTITION. [#6522](https://github.com/ClickHouse/ClickHouse/issues/6522) [#6523](https://github.com/ClickHouse/ClickHouse/pull/6523) ([tavplubix](https://github.com/tavplubix)) +* Fixed issue with parsing CSV [#6426](https://github.com/ClickHouse/ClickHouse/issues/6426) [#6559](https://github.com/ClickHouse/ClickHouse/pull/6559) ([tavplubix](https://github.com/tavplubix)) +* Fixed data race in system.parts table and ALTER query. This fixes [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245). [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address `0x14c0` that may happed due to concurrent `DROP TABLE` and `SELECT` from `system.parts` or `system.parts_columns`. Fixed race condition in preparation of mutation queries. Fixed deadlock caused by `OPTIMIZE` of Replicated tables and concurrent modification operations like ALTERs. [#6514](https://github.com/ClickHouse/ClickHouse/pull/6514) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed possible data loss after `ALTER DELETE` query on table with skipping index. [#6224](https://github.com/ClickHouse/ClickHouse/issues/6224) [#6282](https://github.com/ClickHouse/ClickHouse/pull/6282) ([Nikita Vasilev](https://github.com/nikvas0)) + +#### Security Fix +* If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse run, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. [#6247](https://github.com/ClickHouse/ClickHouse/pull/6247) ([alexey-milovidov](https://github.com/alexey-milovidov)) + +### ClickHouse release 19.13.2.19, 2019-08-14 + +#### New Feature +* Sampling profiler on query level. [Example](https://gist.github.com/alexey-milovidov/92758583dd41c24c360fdb8d6a4da194). [#4247](https://github.com/ClickHouse/ClickHouse/issues/4247) ([laplab](https://github.com/laplab)) [#6124](https://github.com/ClickHouse/ClickHouse/pull/6124) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#6250](https://github.com/ClickHouse/ClickHouse/pull/6250) [#6283](https://github.com/ClickHouse/ClickHouse/pull/6283) [#6386](https://github.com/ClickHouse/ClickHouse/pull/6386) +* Allow to specify a list of columns with `COLUMNS('regexp')` expression that works like a more sophisticated variant of `*` asterisk. [#5951](https://github.com/ClickHouse/ClickHouse/pull/5951) ([mfridental](https://github.com/mfridental)), ([alexey-milovidov](https://github.com/alexey-milovidov)) +* `CREATE TABLE AS table_function()` is now possible [#6057](https://github.com/ClickHouse/ClickHouse/pull/6057) ([dimarub2000](https://github.com/dimarub2000)) +* Adam optimizer for stochastic gradient descent is used by default in `stochasticLinearRegression()` and `stochasticLogisticRegression()` aggregate functions, because it shows good quality without almost any tuning. [#6000](https://github.com/ClickHouse/ClickHouse/pull/6000) ([Quid37](https://github.com/Quid37)) +* Added functions for working with the сustom week number [#5212](https://github.com/ClickHouse/ClickHouse/pull/5212) ([Andy Yang](https://github.com/andyyzh)) +* `RENAME` queries now work with all storages. [#5953](https://github.com/ClickHouse/ClickHouse/pull/5953) ([Ivan](https://github.com/abyss7)) +* Now client receive logs from server with any desired level by setting `send_logs_level` regardless to the log level specified in server settings. [#5964](https://github.com/ClickHouse/ClickHouse/pull/5964) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) + +#### Backward Incompatible Change +* The setting `input_format_defaults_for_omitted_fields` is enabled by default. Inserts in Distributed tables need this setting to be the same on cluster (you need to set it before rolling update). It enables calculation of complex default expressions for omitted fields in `JSONEachRow` and `CSV*` formats. It should be the expected behavior but may lead to negligible performance difference. [#6043](https://github.com/ClickHouse/ClickHouse/pull/6043) ([Artem Zuikov](https://github.com/4ertus2)), [#5625](https://github.com/ClickHouse/ClickHouse/pull/5625) ([akuzm](https://github.com/akuzm)) + +#### Experimental features +* New query processing pipeline. Use `experimental_use_processors=1` option to enable it. Use for your own trouble. [#4914](https://github.com/ClickHouse/ClickHouse/pull/4914) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) + +#### Bug Fix +* Kafka integration has been fixed in this version. +* Fixed `DoubleDelta` encoding of `Int64` for large `DoubleDelta` values, improved `DoubleDelta` encoding for random data for `Int32`. [#5998](https://github.com/ClickHouse/ClickHouse/pull/5998) ([Vasily Nemkov](https://github.com/Enmk)) +* Fixed overestimation of `max_rows_to_read` if the setting `merge_tree_uniform_read_distribution` is set to 0. [#6019](https://github.com/ClickHouse/ClickHouse/pull/6019) ([alexey-milovidov](https://github.com/alexey-milovidov)) + +#### Improvement +* Throws an exception if `config.d` file doesn't have the corresponding root element as the config file [#6123](https://github.com/ClickHouse/ClickHouse/pull/6123) ([dimarub2000](https://github.com/dimarub2000)) + +#### Performance Improvement +* Optimize `count()`. Now it uses the smallest column (if possible). [#6028](https://github.com/ClickHouse/ClickHouse/pull/6028) ([Amos Bird](https://github.com/amosbird)) + +#### Build/Testing/Packaging Improvement +* Report memory usage in performance tests. [#5899](https://github.com/ClickHouse/ClickHouse/pull/5899) ([akuzm](https://github.com/akuzm)) +* Fix build with external `libcxx` [#6010](https://github.com/ClickHouse/ClickHouse/pull/6010) ([Ivan](https://github.com/abyss7)) +* Fix shared build with `rdkafka` library [#6101](https://github.com/ClickHouse/ClickHouse/pull/6101) ([Ivan](https://github.com/abyss7)) + +## ClickHouse release 19.11 + +### ClickHouse release 19.11.13.74, 2019-11-01 + +#### Bug Fix * Fixed rare crash in `ALTER MODIFY COLUMN` and vertical merge when one of merged/altered parts is empty (0 rows). [#6780](https://github.com/ClickHouse/ClickHouse/pull/6780) ([alesapin](https://github.com/alesapin)) * Manual update of `SIMDJSON`. This fixes possible flooding of stderr files with bogus json diagnostic messages. [#7548](https://github.com/ClickHouse/ClickHouse/pull/7548) ([Alexander Kazakov](https://github.com/Akazz)) * Fixed bug with `mrk` file extension for mutations ([alesapin](https://github.com/alesapin)) -## ClickHouse release 19.11.12.69, 2019-10-02 +### ClickHouse release 19.11.12.69, 2019-10-02 -### Bug Fix +#### Bug Fix * Fixed performance degradation of index analysis on complex keys on large tables. This fixes [#6924](https://github.com/ClickHouse/ClickHouse/issues/6924). [#7075](https://github.com/ClickHouse/ClickHouse/pull/7075) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Avoid rare SIGSEGV while sending data in tables with Distributed engine (`Failed to send batch: file with index XXXXX is absent`). [#7032](https://github.com/ClickHouse/ClickHouse/pull/7032) ([Azat Khuzhin](https://github.com/azat)) * Fix `Unknown identifier` with multiple joins. This fixes [#5254](https://github.com/ClickHouse/ClickHouse/issues/5254). [#7022](https://github.com/ClickHouse/ClickHouse/pull/7022) ([Artem Zuikov](https://github.com/4ertus2)) -## ClickHouse release 19.11.10.54, 2019-09-10 +### ClickHouse release 19.11.11.57, 2019-09-13 +* Fix logical error causing segfaults when selecting from Kafka empty topic. [#6902](https://github.com/ClickHouse/ClickHouse/issues/6902) [#6909](https://github.com/ClickHouse/ClickHouse/pull/6909) ([Ivan](https://github.com/abyss7)) +* Fix for function `АrrayEnumerateUniqRanked` with empty arrays in params. [#6928](https://github.com/ClickHouse/ClickHouse/pull/6928) ([proller](https://github.com/proller)) -### Bug Fix +### ClickHouse release 19.11.10.54, 2019-09-10 + +#### Bug Fix * Do store offsets for Kafka messages manually to be able to commit them all at once for all partitions. Fixes potential duplication in "one consumer - many partitions" scenario. [#6872](https://github.com/ClickHouse/ClickHouse/pull/6872) ([Ivan](https://github.com/abyss7)) -## ClickHouse release 19.11.9.52, 2019-09-6 +### ClickHouse release 19.11.9.52, 2019-09-6 * Improve error handling in cache dictionaries. [#6737](https://github.com/ClickHouse/ClickHouse/pull/6737) ([Vitaly Baranov](https://github.com/vitlibar)) * Fixed bug in function `arrayEnumerateUniqRanked`. [#6779](https://github.com/ClickHouse/ClickHouse/pull/6779) ([proller](https://github.com/proller)) * Fix `JSONExtract` function while extracting a `Tuple` from JSON. [#6718](https://github.com/ClickHouse/ClickHouse/pull/6718) ([Vitaly Baranov](https://github.com/vitlibar)) @@ -784,63 +843,12 @@ fix comments to make obvious that it may throw. * Fixed error with processing "timezone" in server configuration file. [#6709](https://github.com/ClickHouse/ClickHouse/pull/6709) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix kafka tests. [#6805](https://github.com/ClickHouse/ClickHouse/pull/6805) ([Ivan](https://github.com/abyss7)) -### Security Fix +#### Security Fix * If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse runs, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. [#6247](https://github.com/ClickHouse/ClickHouse/pull/6247) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.13.3.26, 2019-08-22 +### ClickHouse release 19.11.8.46, 2019-08-22 -### Bug Fix -* Fix `ALTER TABLE ... UPDATE` query for tables with `enable_mixed_granularity_parts=1`. [#6543](https://github.com/ClickHouse/ClickHouse/pull/6543) ([alesapin](https://github.com/alesapin)) -* Fix NPE when using IN clause with a subquery with a tuple. [#6125](https://github.com/ClickHouse/ClickHouse/issues/6125) [#6550](https://github.com/ClickHouse/ClickHouse/pull/6550) ([tavplubix](https://github.com/tavplubix)) -* Fixed an issue that if a stale replica becomes alive, it may still have data parts that were removed by DROP PARTITION. [#6522](https://github.com/ClickHouse/ClickHouse/issues/6522) [#6523](https://github.com/ClickHouse/ClickHouse/pull/6523) ([tavplubix](https://github.com/tavplubix)) -* Fixed issue with parsing CSV [#6426](https://github.com/ClickHouse/ClickHouse/issues/6426) [#6559](https://github.com/ClickHouse/ClickHouse/pull/6559) ([tavplubix](https://github.com/tavplubix)) -* Fixed data race in system.parts table and ALTER query. This fixes [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245). [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) -* Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address `0x14c0` that may happed due to concurrent `DROP TABLE` and `SELECT` from `system.parts` or `system.parts_columns`. Fixed race condition in preparation of mutation queries. Fixed deadlock caused by `OPTIMIZE` of Replicated tables and concurrent modification operations like ALTERs. [#6514](https://github.com/ClickHouse/ClickHouse/pull/6514) ([alexey-milovidov](https://github.com/alexey-milovidov)) -* Fixed possible data loss after `ALTER DELETE` query on table with skipping index. [#6224](https://github.com/ClickHouse/ClickHouse/issues/6224) [#6282](https://github.com/ClickHouse/ClickHouse/pull/6282) ([Nikita Vasilev](https://github.com/nikvas0)) - -### Security Fix -* If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse run, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. [#6247](https://github.com/ClickHouse/ClickHouse/pull/6247) ([alexey-milovidov](https://github.com/alexey-milovidov)) - -## ClickHouse release 19.13.2.19, 2019-08-14 - -### New Feature -* Sampling profiler on query level. [Example](https://gist.github.com/alexey-milovidov/92758583dd41c24c360fdb8d6a4da194). [#4247](https://github.com/ClickHouse/ClickHouse/issues/4247) ([laplab](https://github.com/laplab)) [#6124](https://github.com/ClickHouse/ClickHouse/pull/6124) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#6250](https://github.com/ClickHouse/ClickHouse/pull/6250) [#6283](https://github.com/ClickHouse/ClickHouse/pull/6283) [#6386](https://github.com/ClickHouse/ClickHouse/pull/6386) -* Allow to specify a list of columns with `COLUMNS('regexp')` expression that works like a more sophisticated variant of `*` asterisk. [#5951](https://github.com/ClickHouse/ClickHouse/pull/5951) ([mfridental](https://github.com/mfridental)), ([alexey-milovidov](https://github.com/alexey-milovidov)) -* `CREATE TABLE AS table_function()` is now possible [#6057](https://github.com/ClickHouse/ClickHouse/pull/6057) ([dimarub2000](https://github.com/dimarub2000)) -* Adam optimizer for stochastic gradient descent is used by default in `stochasticLinearRegression()` and `stochasticLogisticRegression()` aggregate functions, because it shows good quality without almost any tuning. [#6000](https://github.com/ClickHouse/ClickHouse/pull/6000) ([Quid37](https://github.com/Quid37)) -* Added functions for working with the сustom week number [#5212](https://github.com/ClickHouse/ClickHouse/pull/5212) ([Andy Yang](https://github.com/andyyzh)) -* `RENAME` queries now work with all storages. [#5953](https://github.com/ClickHouse/ClickHouse/pull/5953) ([Ivan](https://github.com/abyss7)) -* Now client receive logs from server with any desired level by setting `send_logs_level` regardless to the log level specified in server settings. [#5964](https://github.com/ClickHouse/ClickHouse/pull/5964) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) - -### Backward Incompatible Change -* The setting `input_format_defaults_for_omitted_fields` is enabled by default. Inserts in Distributed tables need this setting to be the same on cluster (you need to set it before rolling update). It enables calculation of complex default expressions for omitted fields in `JSONEachRow` and `CSV*` formats. It should be the expected behavior but may lead to negligible performance difference. [#6043](https://github.com/ClickHouse/ClickHouse/pull/6043) ([Artem Zuikov](https://github.com/4ertus2)), [#5625](https://github.com/ClickHouse/ClickHouse/pull/5625) ([akuzm](https://github.com/akuzm)) - -### Experimental features -* New query processing pipeline. Use `experimental_use_processors=1` option to enable it. Use for your own trouble. [#4914](https://github.com/ClickHouse/ClickHouse/pull/4914) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) - -### Bug Fix -* Kafka integration has been fixed in this version. -* Fixed `DoubleDelta` encoding of `Int64` for large `DoubleDelta` values, improved `DoubleDelta` encoding for random data for `Int32`. [#5998](https://github.com/ClickHouse/ClickHouse/pull/5998) ([Vasily Nemkov](https://github.com/Enmk)) -* Fixed overestimation of `max_rows_to_read` if the setting `merge_tree_uniform_read_distribution` is set to 0. [#6019](https://github.com/ClickHouse/ClickHouse/pull/6019) ([alexey-milovidov](https://github.com/alexey-milovidov)) - -### Improvement -* Throws an exception if `config.d` file doesn't have the corresponding root element as the config file [#6123](https://github.com/ClickHouse/ClickHouse/pull/6123) ([dimarub2000](https://github.com/dimarub2000)) - -### Performance Improvement -* Optimize `count()`. Now it uses the smallest column (if possible). [#6028](https://github.com/ClickHouse/ClickHouse/pull/6028) ([Amos Bird](https://github.com/amosbird)) - -### Build/Testing/Packaging Improvement -* Report memory usage in performance tests. [#5899](https://github.com/ClickHouse/ClickHouse/pull/5899) ([akuzm](https://github.com/akuzm)) -* Fix build with external `libcxx` [#6010](https://github.com/ClickHouse/ClickHouse/pull/6010) ([Ivan](https://github.com/abyss7)) -* Fix shared build with `rdkafka` library [#6101](https://github.com/ClickHouse/ClickHouse/pull/6101) ([Ivan](https://github.com/abyss7)) - -## ClickHouse release 19.11.11.57, 2019-09-13 -* Fix logical error causing segfaults when selecting from Kafka empty topic. [#6902](https://github.com/ClickHouse/ClickHouse/issues/6902) [#6909](https://github.com/ClickHouse/ClickHouse/pull/6909) ([Ivan](https://github.com/abyss7)) -* Fix for function `АrrayEnumerateUniqRanked` with empty arrays in params. [#6928](https://github.com/ClickHouse/ClickHouse/pull/6928) ([proller](https://github.com/proller)) - -## ClickHouse release 19.11.8.46, 2019-08-22 - -### Bug Fix +#### Bug Fix * Fix `ALTER TABLE ... UPDATE` query for tables with `enable_mixed_granularity_parts=1`. [#6543](https://github.com/ClickHouse/ClickHouse/pull/6543) ([alesapin](https://github.com/alesapin)) * Fix NPE when using IN clause with a subquery with a tuple. [#6125](https://github.com/ClickHouse/ClickHouse/issues/6125) [#6550](https://github.com/ClickHouse/ClickHouse/pull/6550) ([tavplubix](https://github.com/tavplubix)) * Fixed an issue that if a stale replica becomes alive, it may still have data parts that were removed by DROP PARTITION. [#6522](https://github.com/ClickHouse/ClickHouse/issues/6522) [#6523](https://github.com/ClickHouse/ClickHouse/pull/6523) ([tavplubix](https://github.com/tavplubix)) @@ -848,9 +856,9 @@ fix comments to make obvious that it may throw. * Fixed data race in system.parts table and ALTER query. This fixes [#6245](https://github.com/ClickHouse/ClickHouse/issues/6245). [#6513](https://github.com/ClickHouse/ClickHouse/pull/6513) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address `0x14c0` that may happed due to concurrent `DROP TABLE` and `SELECT` from `system.parts` or `system.parts_columns`. Fixed race condition in preparation of mutation queries. Fixed deadlock caused by `OPTIMIZE` of Replicated tables and concurrent modification operations like ALTERs. [#6514](https://github.com/ClickHouse/ClickHouse/pull/6514) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.11.7.40, 2019-08-14 +### ClickHouse release 19.11.7.40, 2019-08-14 -### Bug fix +#### Bug fix * Kafka integration has been fixed in this version. * Fix segfault when using `arrayReduce` for constant arguments. [#6326](https://github.com/ClickHouse/ClickHouse/pull/6326) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed `toFloat()` monotonicity. [#6374](https://github.com/ClickHouse/ClickHouse/pull/6374) ([dimarub2000](https://github.com/dimarub2000)) @@ -865,12 +873,12 @@ fix comments to make obvious that it may throw. * Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser and possibility of stack overflow in `Merge` and `Distributed` tables [#6433](https://github.com/ClickHouse/ClickHouse/pull/6433) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed Gorilla encoding error on small sequences. [#6444](https://github.com/ClickHouse/ClickHouse/pull/6444) ([Enmk](https://github.com/Enmk)) -### Improvement +#### Improvement * Allow user to override `poll_interval` and `idle_connection_timeout` settings on connection. [#6230](https://github.com/ClickHouse/ClickHouse/pull/6230) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.11.5.28, 2019-08-05 +### ClickHouse release 19.11.5.28, 2019-08-05 -### Bug fix +#### Bug fix * Fixed the possibility of hanging queries when server is overloaded. [#6301](https://github.com/ClickHouse/ClickHouse/pull/6301) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix FPE in yandexConsistentHash function. This fixes [#6304](https://github.com/ClickHouse/ClickHouse/issues/6304). [#6126](https://github.com/ClickHouse/ClickHouse/pull/6126) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed bug in conversion of `LowCardinality` types in `AggregateFunctionFactory`. This fixes [#6257](https://github.com/ClickHouse/ClickHouse/issues/6257). [#6281](https://github.com/ClickHouse/ClickHouse/pull/6281) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -878,12 +886,12 @@ fix comments to make obvious that it may throw. * Fix rare bug with incompatible stream headers in queries to `Distributed` table over `MergeTree` table when part of `WHERE` moves to `PREWHERE`. [#6236](https://github.com/ClickHouse/ClickHouse/pull/6236) ([alesapin](https://github.com/alesapin)) * Fixed overflow in integer division of signed type to unsigned type. This fixes [#6214](https://github.com/ClickHouse/ClickHouse/issues/6214). [#6233](https://github.com/ClickHouse/ClickHouse/pull/6233) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Backward Incompatible Change +#### Backward Incompatible Change * `Kafka` still broken. -## ClickHouse release 19.11.4.24, 2019-08-01 +### ClickHouse release 19.11.4.24, 2019-08-01 -### Bug Fix +#### Bug Fix * Fix bug with writing secondary indices marks with adaptive granularity. [#6126](https://github.com/ClickHouse/ClickHouse/pull/6126) ([alesapin](https://github.com/alesapin)) * Fix `WITH ROLLUP` and `WITH CUBE` modifiers of `GROUP BY` with two-level aggregation. [#6225](https://github.com/ClickHouse/ClickHouse/pull/6225) ([Anton Popov](https://github.com/CurtizJ)) * Fixed hang in `JSONExtractRaw` function. Fixed [#6195](https://github.com/ClickHouse/ClickHouse/issues/6195) [#6198](https://github.com/ClickHouse/ClickHouse/pull/6198) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -898,18 +906,18 @@ fix comments to make obvious that it may throw. * Clearing the Kafka data buffer from the previous read operation that was completed with an error [#6026](https://github.com/ClickHouse/ClickHouse/pull/6026) ([Nikolay](https://github.com/bopohaa)) Note that Kafka is broken in this version. * Since `StorageMergeTree::background_task_handle` is initialized in `startup()` the `MergeTreeBlockOutputStream::write()` may try to use it before initialization. Just check if it is initialized. [#6080](https://github.com/ClickHouse/ClickHouse/pull/6080) ([Ivan](https://github.com/abyss7)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Added official `rpm` packages. [#5740](https://github.com/ClickHouse/ClickHouse/pull/5740) ([proller](https://github.com/proller)) ([alesapin](https://github.com/alesapin)) * Add an ability to build `.rpm` and `.tgz` packages with `packager` script. [#5769](https://github.com/ClickHouse/ClickHouse/pull/5769) ([alesapin](https://github.com/alesapin)) * Fixes for "Arcadia" build system. [#6223](https://github.com/ClickHouse/ClickHouse/pull/6223) ([proller](https://github.com/proller)) -### Backward Incompatible Change +#### Backward Incompatible Change * `Kafka` is broken in this version. -## ClickHouse release 19.11.3.11, 2019-07-18 +### ClickHouse release 19.11.3.11, 2019-07-18 -### New Feature +#### New Feature * Added support for prepared statements. [#5331](https://github.com/ClickHouse/ClickHouse/pull/5331/) ([Alexander](https://github.com/sanych73)) [#5630](https://github.com/ClickHouse/ClickHouse/pull/5630) ([alexey-milovidov](https://github.com/alexey-milovidov)) * `DoubleDelta` and `Gorilla` column codecs [#5600](https://github.com/ClickHouse/ClickHouse/pull/5600) ([Vasily Nemkov](https://github.com/Enmk)) * Added `os_thread_priority` setting that allows to control the "nice" value of query processing threads that is used by OS to adjust dynamic scheduling priority. It requires `CAP_SYS_NICE` capabilities to work. This implements [#5858](https://github.com/ClickHouse/ClickHouse/issues/5858) [#5909](https://github.com/ClickHouse/ClickHouse/pull/5909) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -919,7 +927,7 @@ fix comments to make obvious that it may throw. * Add synonim `arrayFlatten` <-> `flatten` [#5764](https://github.com/ClickHouse/ClickHouse/pull/5764) ([hcz](https://github.com/hczhcz)) * Intergate H3 function `geoToH3` from Uber. [#4724](https://github.com/ClickHouse/ClickHouse/pull/4724) ([Remen Ivan](https://github.com/BHYCHIK)) [#5805](https://github.com/ClickHouse/ClickHouse/pull/5805) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Bug Fix +#### Bug Fix * Implement DNS cache with asynchronous update. Separate thread resolves all hosts and updates DNS cache with period (setting `dns_cache_update_period`). It should help, when ip of hosts changes frequently. [#5857](https://github.com/ClickHouse/ClickHouse/pull/5857) ([Anton Popov](https://github.com/CurtizJ)) * Fix segfault in `Delta` codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. [#5786](https://github.com/ClickHouse/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin)) * Fix segfault in TTL merge with non-physical columns in block. [#5819](https://github.com/ClickHouse/ClickHouse/pull/5819) ([Anton Popov](https://github.com/CurtizJ)) @@ -946,7 +954,7 @@ fix comments to make obvious that it may throw. * Fix shutdown of SystemLogs [#5802](https://github.com/ClickHouse/ClickHouse/pull/5802) ([Anton Popov](https://github.com/CurtizJ)) * Fix hanging when condition in invalidate_query depends on a dictionary. [#6011](https://github.com/ClickHouse/ClickHouse/pull/6011) ([Vitaly Baranov](https://github.com/vitlibar)) -### Improvement +#### Improvement * Allow unresolvable addresses in cluster configuration. They will be considered unavailable and tried to resolve at every connection attempt. This is especially useful for Kubernetes. This fixes [#5714](https://github.com/ClickHouse/ClickHouse/issues/5714) [#5924](https://github.com/ClickHouse/ClickHouse/pull/5924) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Close idle TCP connections (with one hour timeout by default). This is especially important for large clusters with multiple distributed tables on every server, because every server can possibly keep a connection pool to every other server, and after peak query concurrency, connections will stall. This fixes [#5879](https://github.com/ClickHouse/ClickHouse/issues/5879) [#5880](https://github.com/ClickHouse/ClickHouse/pull/5880) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Better quality of `topK` function. Changed the SavingSpace set behavior to remove the last element if the new element have a bigger weight. [#5833](https://github.com/ClickHouse/ClickHouse/issues/5833) [#5850](https://github.com/ClickHouse/ClickHouse/pull/5850) ([Guillaume Tassery](https://github.com/YiuRULE)) @@ -967,10 +975,10 @@ fix comments to make obvious that it may throw. * Update default value of `max_ast_elements parameter` [#5933](https://github.com/ClickHouse/ClickHouse/pull/5933) ([Artem Konovalov](https://github.com/izebit)) * Added a notion of obsolete settings. The obsolete setting `allow_experimental_low_cardinality_type` can be used with no effect. [0f15c01c6802f7ce1a1494c12c846be8c98944cd](https://github.com/ClickHouse/ClickHouse/commit/0f15c01c6802f7ce1a1494c12c846be8c98944cd) [Alexey Milovidov](https://github.com/alexey-milovidov) -### Performance Improvement +#### Performance Improvement * Increase number of streams to SELECT from Merge table for more uniform distribution of threads. Added setting `max_streams_multiplier_for_merge_tables`. This fixes [#5797](https://github.com/ClickHouse/ClickHouse/issues/5797) [#5915](https://github.com/ClickHouse/ClickHouse/pull/5915) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Add a backward compatibility test for client-server interaction with different versions of clickhouse. [#5868](https://github.com/ClickHouse/ClickHouse/pull/5868) ([alesapin](https://github.com/alesapin)) * Test coverage information in every commit and pull request. [#5896](https://github.com/ClickHouse/ClickHouse/pull/5896) ([alesapin](https://github.com/alesapin)) * Cooperate with address sanitizer to support our custom allocators (`Arena` and `ArenaWithFreeLists`) for better debugging of "use-after-free" errors. [#5728](https://github.com/ClickHouse/ClickHouse/pull/5728) ([akuzm](https://github.com/akuzm)) @@ -1007,22 +1015,23 @@ fix comments to make obvious that it may throw. * Performance test concerning the new JIT feature with bigger dataset, as requested here [#5263](https://github.com/ClickHouse/ClickHouse/issues/5263) [#5887](https://github.com/ClickHouse/ClickHouse/pull/5887) ([Guillaume Tassery](https://github.com/YiuRULE)) * Run stateful tests in stress test [12693e568722f11e19859742f56428455501fd2a](https://github.com/ClickHouse/ClickHouse/commit/12693e568722f11e19859742f56428455501fd2a) ([alesapin](https://github.com/alesapin)) -### Backward Incompatible Change +#### Backward Incompatible Change * `Kafka` is broken in this version. * Enable `adaptive_index_granularity` = 10MB by default for new `MergeTree` tables. If you created new MergeTree tables on version 19.11+, downgrade to versions prior to 19.6 will be impossible. [#5628](https://github.com/ClickHouse/ClickHouse/pull/5628) ([alesapin](https://github.com/alesapin)) * Removed obsolete undocumented embedded dictionaries that were used by Yandex.Metrica. The functions `OSIn`, `SEIn`, `OSToRoot`, `SEToRoot`, `OSHierarchy`, `SEHierarchy` are no longer available. If you are using these functions, write email to clickhouse-feedback@yandex-team.com. Note: at the last moment we decided to keep these functions for a while. [#5780](https://github.com/ClickHouse/ClickHouse/pull/5780) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.10.1.5, 2019-07-12 +## ClickHouse release 19.10 +### ClickHouse release 19.10.1.5, 2019-07-12 -### New Feature +#### New Feature * Add new column codec: `T64`. Made for (U)IntX/EnumX/Data(Time)/DecimalX columns. It should be good for columns with constant or small range values. Codec itself allows enlarge or shrink data type without re-compression. [#5557](https://github.com/ClickHouse/ClickHouse/pull/5557) ([Artem Zuikov](https://github.com/4ertus2)) * Add database engine `MySQL` that allow to view all the tables in remote MySQL server [#5599](https://github.com/ClickHouse/ClickHouse/pull/5599) ([Winter Zhang](https://github.com/zhang2014)) * `bitmapContains` implementation. It's 2x faster than `bitmapHasAny` if the second bitmap contains one element. [#5535](https://github.com/ClickHouse/ClickHouse/pull/5535) ([Zhichang Yu](https://github.com/yuzhichang)) * Support for `crc32` function (with behaviour exactly as in MySQL or PHP). Do not use it if you need a hash function. [#5661](https://github.com/ClickHouse/ClickHouse/pull/5661) ([Remen Ivan](https://github.com/BHYCHIK)) * Implemented `SYSTEM START/STOP DISTRIBUTED SENDS` queries to control asynchronous inserts into `Distributed` tables. [#4935](https://github.com/ClickHouse/ClickHouse/pull/4935) ([Winter Zhang](https://github.com/zhang2014)) -### Bug Fix +#### Bug Fix * Ignore query execution limits and max parts size for merge limits while executing mutations. [#5659](https://github.com/ClickHouse/ClickHouse/pull/5659) ([Anton Popov](https://github.com/CurtizJ)) * Fix bug which may lead to deduplication of normal blocks (extremely rare) and insertion of duplicate blocks (more often). [#5549](https://github.com/ClickHouse/ClickHouse/pull/5549) ([alesapin](https://github.com/alesapin)) * Fix of function `arrayEnumerateUniqRanked` for arguments with empty arrays [#5559](https://github.com/ClickHouse/ClickHouse/pull/5559) ([proller](https://github.com/proller)) @@ -1032,7 +1041,7 @@ fix comments to make obvious that it may throw. * Fix Float to Decimal convert overflow [#5607](https://github.com/ClickHouse/ClickHouse/pull/5607) ([coraxster](https://github.com/coraxster)) * Flush buffer when `WriteBufferFromHDFS`'s destructor is called. This fixes writing into `HDFS`. [#5684](https://github.com/ClickHouse/ClickHouse/pull/5684) ([Xindong Peng](https://github.com/eejoin)) -### Improvement +#### Improvement * Treat empty cells in `CSV` as default values when the setting `input_format_defaults_for_omitted_fields` is enabled. [#5625](https://github.com/ClickHouse/ClickHouse/pull/5625) ([akuzm](https://github.com/akuzm)) * Non-blocking loading of external dictionaries. [#5567](https://github.com/ClickHouse/ClickHouse/pull/5567) ([Vitaly Baranov](https://github.com/vitlibar)) * Network timeouts can be dynamically changed for already established connections according to the settings. [#4558](https://github.com/ClickHouse/ClickHouse/pull/4558) ([Konstantin Podshumok](https://github.com/podshumok)) @@ -1043,21 +1052,22 @@ fix comments to make obvious that it may throw. * Support `` section in `clickhouse-local` config file. [#5540](https://github.com/ClickHouse/ClickHouse/pull/5540) ([proller](https://github.com/proller)) * Allow run query with `remote` table function in `clickhouse-local` [#5627](https://github.com/ClickHouse/ClickHouse/pull/5627) ([proller](https://github.com/proller)) -### Performance Improvement +#### Performance Improvement * Add the possibility to write the final mark at the end of MergeTree columns. It allows to avoid useless reads for keys that are out of table data range. It is enabled only if adaptive index granularity is in use. [#5624](https://github.com/ClickHouse/ClickHouse/pull/5624) ([alesapin](https://github.com/alesapin)) * Improved performance of MergeTree tables on very slow filesystems by reducing number of `stat` syscalls. [#5648](https://github.com/ClickHouse/ClickHouse/pull/5648) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed performance degradation in reading from MergeTree tables that was introduced in version 19.6. Fixes #5631. [#5633](https://github.com/ClickHouse/ClickHouse/pull/5633) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Implemented `TestKeeper` as an implementation of ZooKeeper interface used for testing [#5643](https://github.com/ClickHouse/ClickHouse/pull/5643) ([alexey-milovidov](https://github.com/alexey-milovidov)) ([levushkin aleksej](https://github.com/alexey-milovidov)) * From now on `.sql` tests can be run isolated by server, in parallel, with random database. It allows to run them faster, add new tests with custom server configurations, and be sure that different tests doesn't affect each other. [#5554](https://github.com/ClickHouse/ClickHouse/pull/5554) ([Ivan](https://github.com/abyss7)) * Remove `` and `` from performance tests [#5672](https://github.com/ClickHouse/ClickHouse/pull/5672) ([Olga Khvostikova](https://github.com/stavrolia)) * Fixed "select_format" performance test for `Pretty` formats [#5642](https://github.com/ClickHouse/ClickHouse/pull/5642) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.9.3.31, 2019-07-05 +## ClickHouse release 19.9 +### ClickHouse release 19.9.3.31, 2019-07-05 -### Bug Fix +#### Bug Fix * Fix segfault in Delta codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. [#5786](https://github.com/ClickHouse/ClickHouse/pull/5786) ([alesapin](https://github.com/alesapin)) * Fix rare bug in checking of part with LowCardinality column. [#5832](https://github.com/ClickHouse/ClickHouse/pull/5832) ([alesapin](https://github.com/alesapin)) * Fix segfault in TTL merge with non-physical columns in block. [#5819](https://github.com/ClickHouse/ClickHouse/pull/5819) ([Anton Popov](https://github.com/CurtizJ)) @@ -1067,26 +1077,21 @@ fix comments to make obvious that it may throw. * Fix race condition, which cause that some queries may not appear in query_log instantly after SYSTEM FLUSH LOGS query. [#5685](https://github.com/ClickHouse/ClickHouse/pull/5685) ([Anton Popov](https://github.com/CurtizJ)) * Added missing support for constant arguments to `evalMLModel` function. [#5820](https://github.com/ClickHouse/ClickHouse/pull/5820) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.7.5.29, 2019-07-05 +### ClickHouse release 19.9.2.4, 2019-06-24 -### Bug Fix -* Fix performance regression in some queries with JOIN. [#5192](https://github.com/ClickHouse/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014)) - -## ClickHouse release 19.9.2.4, 2019-06-24 - -### New Feature +#### New Feature * Print information about frozen parts in `system.parts` table. [#5471](https://github.com/ClickHouse/ClickHouse/pull/5471) ([proller](https://github.com/proller)) * Ask client password on clickhouse-client start on tty if not set in arguments [#5092](https://github.com/ClickHouse/ClickHouse/pull/5092) ([proller](https://github.com/proller)) * Implement `dictGet` and `dictGetOrDefault` functions for Decimal types. [#5394](https://github.com/ClickHouse/ClickHouse/pull/5394) ([Artem Zuikov](https://github.com/4ertus2)) -### Improvement +#### Improvement * Debian init: Add service stop timeout [#5522](https://github.com/ClickHouse/ClickHouse/pull/5522) ([proller](https://github.com/proller)) * Add setting forbidden by default to create table with suspicious types for LowCardinality [#5448](https://github.com/ClickHouse/ClickHouse/pull/5448) ([Olga Khvostikova](https://github.com/stavrolia)) * Regression functions return model weights when not used as State in function `evalMLMethod`. [#5411](https://github.com/ClickHouse/ClickHouse/pull/5411) ([Quid37](https://github.com/Quid37)) * Rename and improve regression methods. [#5492](https://github.com/ClickHouse/ClickHouse/pull/5492) ([Quid37](https://github.com/Quid37)) * Clearer interfaces of string searchers. [#5586](https://github.com/ClickHouse/ClickHouse/pull/5586) ([Danila Kutenin](https://github.com/danlark1)) -### Bug Fix +#### Bug Fix * Fix potential data loss in Kafka [#5445](https://github.com/ClickHouse/ClickHouse/pull/5445) ([Ivan](https://github.com/abyss7)) * Fix potential infinite loop in `PrettySpace` format when called with zero columns [#5560](https://github.com/ClickHouse/ClickHouse/pull/5560) ([Olga Khvostikova](https://github.com/stavrolia)) * Fixed UInt32 overflow bug in linear models. Allow eval ML model for non-const model argument. [#5516](https://github.com/ClickHouse/ClickHouse/pull/5516) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -1106,7 +1111,7 @@ fix comments to make obvious that it may throw. * Throw an exception on wrong integers in `dictGetT` functions instead of crash. [#5446](https://github.com/ClickHouse/ClickHouse/pull/5446) ([Artem Zuikov](https://github.com/4ertus2)) * Fix wrong element_count and load_factor for hashed dictionary in `system.dictionaries` table. [#5440](https://github.com/ClickHouse/ClickHouse/pull/5440) ([Azat Khuzhin](https://github.com/azat)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Fixed build without `Brotli` HTTP compression support (`ENABLE_BROTLI=OFF` cmake variable). [#5521](https://github.com/ClickHouse/ClickHouse/pull/5521) ([Anton Yuzhaninov](https://github.com/citrin)) * Include roaring.h as roaring/roaring.h [#5523](https://github.com/ClickHouse/ClickHouse/pull/5523) ([Orivej Desh](https://github.com/orivej)) * Fix gcc9 warnings in hyperscan (#line directive is evil!) [#5546](https://github.com/ClickHouse/ClickHouse/pull/5546) ([Danila Kutenin](https://github.com/danlark1)) @@ -1121,9 +1126,10 @@ fix comments to make obvious that it may throw. * Fix build clickhouse as submodule [#5574](https://github.com/ClickHouse/ClickHouse/pull/5574) ([proller](https://github.com/proller)) * Improve JSONExtract performance tests [#5444](https://github.com/ClickHouse/ClickHouse/pull/5444) ([Vitaly Baranov](https://github.com/vitlibar)) -## ClickHouse release 19.8.3.8, 2019-06-11 +## ClickHouse release 19.8 +### ClickHouse release 19.8.3.8, 2019-06-11 -### New Features +#### New Features * Added functions to work with JSON [#4686](https://github.com/ClickHouse/ClickHouse/pull/4686) ([hcz](https://github.com/hczhcz)) [#5124](https://github.com/ClickHouse/ClickHouse/pull/5124). ([Vitaly Baranov](https://github.com/vitlibar)) * Add a function basename, with a similar behaviour to a basename function, which exists in a lot of languages (`os.path.basename` in python, `basename` in PHP, etc...). Work with both an UNIX-like path or a Windows path. [#5136](https://github.com/ClickHouse/ClickHouse/pull/5136) ([Guillaume Tassery](https://github.com/YiuRULE)) * Added `LIMIT n, m BY` or `LIMIT m OFFSET n BY` syntax to set offset of n for LIMIT BY clause. [#5138](https://github.com/ClickHouse/ClickHouse/pull/5138) ([Anton Popov](https://github.com/CurtizJ)) @@ -1144,7 +1150,7 @@ fix comments to make obvious that it may throw. * Added functions `IPv4CIDRtoIPv4Range` and `IPv6CIDRtoIPv6Range` to calculate the lower and higher bounds for an IP in the subnet using a CIDR. [#5095](https://github.com/ClickHouse/ClickHouse/pull/5095) ([Guillaume Tassery](https://github.com/YiuRULE)) * Add a X-ClickHouse-Summary header when we send a query using HTTP with enabled setting `send_progress_in_http_headers`. Return the usual information of X-ClickHouse-Progress, with additional information like how many rows and bytes were inserted in the query. [#5116](https://github.com/ClickHouse/ClickHouse/pull/5116) ([Guillaume Tassery](https://github.com/YiuRULE)) -### Improvements +#### Improvements * Added `max_parts_in_total` setting for MergeTree family of tables (default: 100 000) that prevents unsafe specification of partition key #5166. [#5171](https://github.com/ClickHouse/ClickHouse/pull/5171) ([alexey-milovidov](https://github.com/alexey-milovidov)) * `clickhouse-obfuscator`: derive seed for individual columns by combining initial seed with column name, not column position. This is intended to transform datasets with multiple related tables, so that tables will remain JOINable after transformation. [#5178](https://github.com/ClickHouse/ClickHouse/pull/5178) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Added functions `JSONExtractRaw`, `JSONExtractKeyAndValues`. Renamed functions `jsonExtract` to `JSONExtract`. When something goes wrong these functions return the correspondent values, not `NULL`. Modified function `JSONExtract`, now it gets the return type from its last parameter and doesn't inject nullables. Implemented fallback to RapidJSON in case AVX2 instructions are not available. Simdjson library updated to a new version. [#5235](https://github.com/ClickHouse/ClickHouse/pull/5235) ([Vitaly Baranov](https://github.com/vitlibar)) @@ -1168,7 +1174,7 @@ It allows to set commit mode: after every batch of messages is handled, or after * Respect query settings in asynchronous INSERTs into Distributed tables. [#4936](https://github.com/ClickHouse/ClickHouse/pull/4936) ([TCeason](https://github.com/TCeason)) * Renamed functions `leastSqr` to `simpleLinearRegression`, `LinearRegression` to `linearRegression`, `LogisticRegression` to `logisticRegression`. [#5391](https://github.com/ClickHouse/ClickHouse/pull/5391) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -### Performance Improvements +#### Performance Improvements * Parallelize processing of parts of non-replicated MergeTree tables in ALTER MODIFY query. [#4639](https://github.com/ClickHouse/ClickHouse/pull/4639) ([Ivan Kush](https://github.com/IvanKush)) * Optimizations in regular expressions extraction. [#5193](https://github.com/ClickHouse/ClickHouse/pull/5193) [#5191](https://github.com/ClickHouse/ClickHouse/pull/5191) ([Danila Kutenin](https://github.com/danlark1)) * Do not add right join key column to join result if it's used only in join on section. [#5260](https://github.com/ClickHouse/ClickHouse/pull/5260) ([Artem Zuikov](https://github.com/4ertus2)) @@ -1178,7 +1184,7 @@ It allows to set commit mode: after every batch of messages is handled, or after * Upgrade our LZ4 implementation with reference one to have faster decompression. [#5070](https://github.com/ClickHouse/ClickHouse/pull/5070) ([Danila Kutenin](https://github.com/danlark1)) * Implemented MSD radix sort (based on kxsort), and partial sorting. [#5129](https://github.com/ClickHouse/ClickHouse/pull/5129) ([Evgenii Pravda](https://github.com/kvinty)) -### Bug Fixes +#### Bug Fixes * Fix push require columns with join [#5192](https://github.com/ClickHouse/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014)) * Fixed bug, when ClickHouse is run by systemd, the command `sudo service clickhouse-server forcerestart` was not working as expected. [#5204](https://github.com/ClickHouse/ClickHouse/pull/5204) ([proller](https://github.com/proller)) * Fix http error codes in DataPartsExchange (interserver http server on 9009 port always returned code 200, even on errors). [#5216](https://github.com/ClickHouse/ClickHouse/pull/5216) ([proller](https://github.com/proller)) @@ -1189,7 +1195,7 @@ It allows to set commit mode: after every batch of messages is handled, or after * Fix `retention` function. Now all conditions that satisfy in a row of data are added to the data state. [#5119](https://github.com/ClickHouse/ClickHouse/pull/5119) ([小路](https://github.com/nicelulu)) * Fix result type for `quantileExact` with Decimals. [#5304](https://github.com/ClickHouse/ClickHouse/pull/5304) ([Artem Zuikov](https://github.com/4ertus2)) -### Documentation +#### Documentation * Translate documentation for `CollapsingMergeTree` to chinese. [#5168](https://github.com/ClickHouse/ClickHouse/pull/5168) ([张风啸](https://github.com/AlexZFX)) * Translate some documentation about table engines to chinese. [#5134](https://github.com/ClickHouse/ClickHouse/pull/5134) @@ -1197,7 +1203,7 @@ It allows to set commit mode: after every batch of messages is handled, or after ([never lee](https://github.com/neverlee)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Fix some sanitizer reports that show probable use-after-free.[#5139](https://github.com/ClickHouse/ClickHouse/pull/5139) [#5143](https://github.com/ClickHouse/ClickHouse/pull/5143) [#5393](https://github.com/ClickHouse/ClickHouse/pull/5393) ([Ivan](https://github.com/abyss7)) * Move performance tests out of separate directories for convenience. [#5158](https://github.com/ClickHouse/ClickHouse/pull/5158) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix incorrect performance tests. [#5255](https://github.com/ClickHouse/ClickHouse/pull/5255) ([alesapin](https://github.com/alesapin)) @@ -1206,12 +1212,19 @@ It allows to set commit mode: after every batch of messages is handled, or after * Add small instruction how to write performance tests. [#5408](https://github.com/ClickHouse/ClickHouse/pull/5408) ([alesapin](https://github.com/alesapin)) * Add ability to make substitutions in create, fill and drop query in performance tests [#5367](https://github.com/ClickHouse/ClickHouse/pull/5367) ([Olga Khvostikova](https://github.com/stavrolia)) -## ClickHouse release 19.7.5.27, 2019-06-09 +## ClickHouse release 19.7 -### New features +### ClickHouse release 19.7.5.29, 2019-07-05 + +#### Bug Fix +* Fix performance regression in some queries with JOIN. [#5192](https://github.com/ClickHouse/ClickHouse/pull/5192) ([Winter Zhang](https://github.com/zhang2014)) + +### ClickHouse release 19.7.5.27, 2019-06-09 + +#### New features * Added bitmap related functions `bitmapHasAny` and `bitmapHasAll` analogous to `hasAny` and `hasAll` functions for arrays. [#5279](https://github.com/ClickHouse/ClickHouse/pull/5279) ([Sergi Vladykin](https://github.com/svladykin)) -### Bug Fixes +#### Bug Fixes * Fix segfault on `minmax` INDEX with Null value. [#5246](https://github.com/ClickHouse/ClickHouse/pull/5246) ([Nikita Vasilev](https://github.com/nikvas0)) * Mark all input columns in LIMIT BY as required output. It fixes 'Not found column' error in some distributed queries. [#5407](https://github.com/ClickHouse/ClickHouse/pull/5407) ([Constantin S. Pan](https://github.com/kvap)) * Fix "Column '0' already exists" error in `SELECT .. PREWHERE` on column with DEFAULT [#5397](https://github.com/ClickHouse/ClickHouse/pull/5397) ([proller](https://github.com/proller)) @@ -1232,9 +1245,9 @@ It allows to set commit mode: after every batch of messages is handled, or after did not process it, but already get list of children, will terminate the DDLWorker thread. [#5489](https://github.com/ClickHouse/ClickHouse/pull/5489) ([Azat Khuzhin](https://github.com/azat)) * Fix INSERT into Distributed() table with MATERIALIZED column. [#5429](https://github.com/ClickHouse/ClickHouse/pull/5429) ([Azat Khuzhin](https://github.com/azat)) -## ClickHouse release 19.7.3.9, 2019-05-30 +### ClickHouse release 19.7.3.9, 2019-05-30 -### New Features +#### New Features * Allow to limit the range of a setting that can be specified by user. These constraints can be set up in user settings profile. [#4931](https://github.com/ClickHouse/ClickHouse/pull/4931) ([Vitaly @@ -1250,7 +1263,7 @@ Tassery](https://github.com/YiuRULE)) [#5081](https://github.com/ClickHouse/ClickHouse/pull/5081) ([Alexander](https://github.com/Akazz)) -### Bug Fixes +#### Bug Fixes * Crash with uncompressed_cache + JOIN during merge (#5197) [#5133](https://github.com/ClickHouse/ClickHouse/pull/5133) ([Danila Kutenin](https://github.com/danlark1)) @@ -1262,14 +1275,14 @@ Kutenin](https://github.com/danlark1)) ([Ivan](https://github.com/abyss7)) * Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Performance Improvements +#### Performance Improvements * Use radix sort for sorting by single numeric column in `ORDER BY` without `LIMIT`. [#5106](https://github.com/ClickHouse/ClickHouse/pull/5106), [#4439](https://github.com/ClickHouse/ClickHouse/pull/4439) ([Evgenii Pravda](https://github.com/kvinty), [alexey-milovidov](https://github.com/alexey-milovidov)) -### Documentation +#### Documentation * Translate documentation for some table engines to Chinese. [#5107](https://github.com/ClickHouse/ClickHouse/pull/5107), [#5094](https://github.com/ClickHouse/ClickHouse/pull/5094), @@ -1278,7 +1291,7 @@ Kutenin](https://github.com/danlark1)) [#5068](https://github.com/ClickHouse/ClickHouse/pull/5068) ([never lee](https://github.com/neverlee)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Print UTF-8 characters properly in `clickhouse-test`. [#5084](https://github.com/ClickHouse/ClickHouse/pull/5084) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1294,9 +1307,10 @@ lee](https://github.com/neverlee)) [#5110](https://github.com/ClickHouse/ClickHouse/pull/5110) ([proller](https://github.com/proller)) -## ClickHouse release 19.6.3.18, 2019-06-13 +## ClickHouse release 19.6 +### ClickHouse release 19.6.3.18, 2019-06-13 -### Bug Fixes +#### Bug Fixes * Fixed IN condition pushdown for queries from table functions `mysql` and `odbc` and corresponding table engines. This fixes #3540 and #2384. [#5313](https://github.com/ClickHouse/ClickHouse/pull/5313) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix deadlock in Zookeeper. [#5297](https://github.com/ClickHouse/ClickHouse/pull/5297) ([github1youlc](https://github.com/github1youlc)) * Allow quoted decimals in CSV. [#5284](https://github.com/ClickHouse/ClickHouse/pull/5284) ([Artem Zuikov](https://github.com/4ertus2) @@ -1304,18 +1318,18 @@ lee](https://github.com/neverlee)) * Fix data race in rename query. [#5247](https://github.com/ClickHouse/ClickHouse/pull/5247) ([Winter Zhang](https://github.com/zhang2014)) * Temporarily disable LFAlloc. Usage of LFAlloc might lead to a lot of MAP_FAILED in allocating UncompressedCache and in a result to crashes of queries at high loaded servers. [cfdba93](https://github.com/ClickHouse/ClickHouse/commit/cfdba938ce22f16efeec504f7f90206a515b1280)([Danila Kutenin](https://github.com/danlark1)) -## ClickHouse release 19.6.2.11, 2019-05-13 +### ClickHouse release 19.6.2.11, 2019-05-13 -### New Features +#### New Features * TTL expressions for columns and tables. [#4212](https://github.com/ClickHouse/ClickHouse/pull/4212) ([Anton Popov](https://github.com/CurtizJ)) * Added support for `brotli` compression for HTTP responses (Accept-Encoding: br) [#4388](https://github.com/ClickHouse/ClickHouse/pull/4388) ([Mikhail](https://github.com/fandyushin)) * Added new function `isValidUTF8` for checking whether a set of bytes is correctly utf-8 encoded. [#4934](https://github.com/ClickHouse/ClickHouse/pull/4934) ([Danila Kutenin](https://github.com/danlark1)) * Add new load balancing policy `first_or_random` which sends queries to the first specified host and if it's inaccessible send queries to random hosts of shard. Useful for cross-replication topology setups. [#5012](https://github.com/ClickHouse/ClickHouse/pull/5012) ([nvartolomei](https://github.com/nvartolomei)) -### Experimental Features +#### Experimental Features * Add setting `index_granularity_bytes` (adaptive index granularity) for MergeTree* tables family. [#4826](https://github.com/ClickHouse/ClickHouse/pull/4826) ([alesapin](https://github.com/alesapin)) -### Improvements +#### Improvements * Added support for non-constant and negative size and length arguments for function `substringUTF8`. [#4989](https://github.com/ClickHouse/ClickHouse/pull/4989) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Disable push-down to right table in left join, left table in right join, and both tables in full join. This fixes wrong JOIN results in some cases. [#4846](https://github.com/ClickHouse/ClickHouse/pull/4846) ([Ivan](https://github.com/abyss7)) * `clickhouse-copier`: auto upload task configuration from `--task-file` option [#4876](https://github.com/ClickHouse/ClickHouse/pull/4876) ([proller](https://github.com/proller)) @@ -1323,13 +1337,13 @@ lee](https://github.com/neverlee)) * Support asterisks and qualified asterisks for multiple joins without subqueries [#4898](https://github.com/ClickHouse/ClickHouse/pull/4898) ([Artem Zuikov](https://github.com/4ertus2)) * Make missing column error message more user friendly. [#4915](https://github.com/ClickHouse/ClickHouse/pull/4915) ([Artem Zuikov](https://github.com/4ertus2)) -### Performance Improvements +#### Performance Improvements * Significant speedup of ASOF JOIN [#4924](https://github.com/ClickHouse/ClickHouse/pull/4924) ([Martijn Bakker](https://github.com/Gladdy)) -### Backward Incompatible Changes +#### Backward Incompatible Changes * HTTP header `Query-Id` was renamed to `X-ClickHouse-Query-Id` for consistency. [#4972](https://github.com/ClickHouse/ClickHouse/pull/4972) ([Mikhail](https://github.com/fandyushin)) -### Bug Fixes +#### Bug Fixes * Fixed potential null pointer dereference in `clickhouse-copier`. [#4900](https://github.com/ClickHouse/ClickHouse/pull/4900) ([proller](https://github.com/proller)) * Fixed error on query with JOIN + ARRAY JOIN [#4938](https://github.com/ClickHouse/ClickHouse/pull/4938) ([Artem Zuikov](https://github.com/4ertus2)) * Fixed hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary. [#4962](https://github.com/ClickHouse/ClickHouse/pull/4962) ([Vitaly Baranov](https://github.com/vitlibar)) @@ -1337,7 +1351,7 @@ lee](https://github.com/neverlee)) * Fix potentially wrong result for `SELECT DISTINCT` with `JOIN` [#5001](https://github.com/ClickHouse/ClickHouse/pull/5001) ([Artem Zuikov](https://github.com/4ertus2)) * Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Fixed test failures when running clickhouse-server on different host [#4713](https://github.com/ClickHouse/ClickHouse/pull/4713) ([Vasily Nemkov](https://github.com/Enmk)) * clickhouse-test: Disable color control sequences in non tty environment. [#4937](https://github.com/ClickHouse/ClickHouse/pull/4937) ([alesapin](https://github.com/alesapin)) * clickhouse-test: Allow use any test database (remove `test.` qualification where it possible) [#5008](https://github.com/ClickHouse/ClickHouse/pull/5008) ([proller](https://github.com/proller)) @@ -1346,24 +1360,25 @@ lee](https://github.com/neverlee)) * Python util to help with backports and changelogs. [#4949](https://github.com/ClickHouse/ClickHouse/pull/4949) ([Ivan](https://github.com/abyss7)) -## ClickHouse release 19.5.4.22, 2019-05-13 +## ClickHouse release 19.5 +### ClickHouse release 19.5.4.22, 2019-05-13 -### Bug fixes +#### Bug fixes * Fixed possible crash in bitmap* functions [#5220](https://github.com/ClickHouse/ClickHouse/pull/5220) [#5228](https://github.com/ClickHouse/ClickHouse/pull/5228) ([Andy Yang](https://github.com/andyyzh)) * Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. [#5189](https://github.com/ClickHouse/ClickHouse/pull/5189) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed error `Set for IN is not created yet in case of using single LowCardinality column in the left part of IN`. This error happened if LowCardinality column was the part of primary key. #5031 [#5154](https://github.com/ClickHouse/ClickHouse/pull/5154) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Modification of retention function: If a row satisfies both the first and NTH condition, only the first satisfied condition is added to the data state. Now all conditions that satisfy in a row of data are added to the data state. [#5119](https://github.com/ClickHouse/ClickHouse/pull/5119) ([小路](https://github.com/nicelulu)) -## ClickHouse release 19.5.3.8, 2019-04-18 +### ClickHouse release 19.5.3.8, 2019-04-18 -### Bug fixes +#### Bug fixes * Fixed type of setting `max_partitions_per_insert_block` from boolean to UInt64. [#5028](https://github.com/ClickHouse/ClickHouse/pull/5028) ([Mohammad Hossein Sekhavat](https://github.com/mhsekhavat)) -## ClickHouse release 19.5.2.6, 2019-04-15 +### ClickHouse release 19.5.2.6, 2019-04-15 -### New Features +#### New Features * [Hyperscan](https://github.com/intel/hyperscan) multiple regular expression matching was added (functions `multiMatchAny`, `multiMatchAnyIndex`, `multiFuzzyMatchAny`, `multiFuzzyMatchAnyIndex`). [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780), [#4841](https://github.com/ClickHouse/ClickHouse/pull/4841) ([Danila Kutenin](https://github.com/danlark1)) * `multiSearchFirstPosition` function was added. [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1)) @@ -1372,7 +1387,7 @@ lee](https://github.com/neverlee)) * Added `ASOF JOIN` which allows to run queries that join to the most recent value known. [#4774](https://github.com/ClickHouse/ClickHouse/pull/4774) [#4867](https://github.com/ClickHouse/ClickHouse/pull/4867) [#4863](https://github.com/ClickHouse/ClickHouse/pull/4863) [#4875](https://github.com/ClickHouse/ClickHouse/pull/4875) ([Martijn Bakker](https://github.com/Gladdy), [Artem Zuikov](https://github.com/4ertus2)) * Rewrite multiple `COMMA JOIN` to `CROSS JOIN`. Then rewrite them to `INNER JOIN` if possible. [#4661](https://github.com/ClickHouse/ClickHouse/pull/4661) ([Artem Zuikov](https://github.com/4ertus2)) -### Improvement +#### Improvement * `topK` and `topKWeighted` now supports custom `loadFactor` (fixes issue [#4252](https://github.com/ClickHouse/ClickHouse/issues/4252)). [#4634](https://github.com/ClickHouse/ClickHouse/pull/4634) ([Kirill Danshin](https://github.com/kirillDanshin)) * Allow to use `parallel_replicas_count > 1` even for tables without sampling (the setting is simply ignored for them). In previous versions it was lead to exception. [#4637](https://github.com/ClickHouse/ClickHouse/pull/4637) ([Alexey Elymanov](https://github.com/digitalist)) @@ -1389,7 +1404,7 @@ lee](https://github.com/neverlee)) * Improved data skipping indices calculation. [#4640](https://github.com/ClickHouse/ClickHouse/pull/4640) ([Nikita Vasilev](https://github.com/nikvas0)) * Keep ordinary, `DEFAULT`, `MATERIALIZED` and `ALIAS` columns in a single list (fixes issue [#2867](https://github.com/ClickHouse/ClickHouse/issues/2867)). [#4707](https://github.com/ClickHouse/ClickHouse/pull/4707) ([Alex Zatelepin](https://github.com/ztlpn)) -### Bug Fix +#### Bug Fix * Avoid `std::terminate` in case of memory allocation failure. Now `std::bad_alloc` exception is thrown as expected. [#4665](https://github.com/ClickHouse/ClickHouse/pull/4665) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixes capnproto reading from buffer. Sometimes files wasn't loaded successfully by HTTP. [#4674](https://github.com/ClickHouse/ClickHouse/pull/4674) ([Vladislav](https://github.com/smirnov-vs)) @@ -1429,19 +1444,19 @@ lee](https://github.com/neverlee)) * Fix function `toISOWeek` result for year 1970. [#4988](https://github.com/ClickHouse/ClickHouse/pull/4988) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix `DROP`, `TRUNCATE` and `OPTIMIZE` queries duplication, when executed on `ON CLUSTER` for `ReplicatedMergeTree*` tables family. [#4991](https://github.com/ClickHouse/ClickHouse/pull/4991) ([alesapin](https://github.com/alesapin)) -### Backward Incompatible Change +#### Backward Incompatible Change * Rename setting `insert_sample_with_metadata` to setting `input_format_defaults_for_omitted_fields`. [#4771](https://github.com/ClickHouse/ClickHouse/pull/4771) ([Artem Zuikov](https://github.com/4ertus2)) * Added setting `max_partitions_per_insert_block` (with value 100 by default). If inserted block contains larger number of partitions, an exception is thrown. Set it to 0 if you want to remove the limit (not recommended). [#4845](https://github.com/ClickHouse/ClickHouse/pull/4845) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Multi-search functions were renamed (`multiPosition` to `multiSearchAllPositions`, `multiSearch` to `multiSearchAny`, `firstMatch` to `multiSearchFirstIndex`). [#4780](https://github.com/ClickHouse/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1)) -### Performance Improvement +#### Performance Improvement * Optimize Volnitsky searcher by inlining, giving about 5-10% search improvement for queries with many needles or many similar bigrams. [#4862](https://github.com/ClickHouse/ClickHouse/pull/4862) ([Danila Kutenin](https://github.com/danlark1)) * Fix performance issue when setting `use_uncompressed_cache` is greater than zero, which appeared when all read data contained in cache. [#4913](https://github.com/ClickHouse/ClickHouse/pull/4913) ([alesapin](https://github.com/alesapin)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Hardening debug build: more granular memory mappings and ASLR; add memory protection for mark cache and index. This allows to find more memory stomping bugs in case when ASan and MSan cannot do it. [#4632](https://github.com/ClickHouse/ClickHouse/pull/4632) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add support for cmake variables `ENABLE_PROTOBUF`, `ENABLE_PARQUET` and `ENABLE_BROTLI` which allows to enable/disable the above features (same as we can do for librdkafka, mysql, etc). [#4669](https://github.com/ClickHouse/ClickHouse/pull/4669) ([Silviu Caragea](https://github.com/silviucpp)) @@ -1456,9 +1471,10 @@ lee](https://github.com/neverlee)) * Disable usage of `mremap` when compiled with Thread Sanitizer. Surprisingly enough, TSan does not intercept `mremap` (though it does intercept `mmap`, `munmap`) that leads to false positives. Fixed TSan report in stateful tests. [#4859](https://github.com/ClickHouse/ClickHouse/pull/4859) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add test checking using format schema via HTTP interface. [#4864](https://github.com/ClickHouse/ClickHouse/pull/4864) ([Vitaly Baranov](https://github.com/vitlibar)) -## ClickHouse release 19.4.4.33, 2019-04-17 +## ClickHouse release 19.4 +### ClickHouse release 19.4.4.33, 2019-04-17 -### Bug Fixes +#### Bug Fixes * Avoid `std::terminate` in case of memory allocation failure. Now `std::bad_alloc` exception is thrown as expected. [#4665](https://github.com/ClickHouse/ClickHouse/pull/4665) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixes capnproto reading from buffer. Sometimes files wasn't loaded successfully by HTTP. [#4674](https://github.com/ClickHouse/ClickHouse/pull/4674) ([Vladislav](https://github.com/smirnov-vs)) @@ -1493,34 +1509,34 @@ lee](https://github.com/neverlee)) * Fix function `toISOWeek` result for year 1970. [#4988](https://github.com/ClickHouse/ClickHouse/pull/4988) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix `DROP`, `TRUNCATE` and `OPTIMIZE` queries duplication, when executed on `ON CLUSTER` for `ReplicatedMergeTree*` tables family. [#4991](https://github.com/ClickHouse/ClickHouse/pull/4991) ([alesapin](https://github.com/alesapin)) -### Improvements +#### Improvements * Keep ordinary, `DEFAULT`, `MATERIALIZED` and `ALIAS` columns in a single list (fixes issue [#2867](https://github.com/ClickHouse/ClickHouse/issues/2867)). [#4707](https://github.com/ClickHouse/ClickHouse/pull/4707) ([Alex Zatelepin](https://github.com/ztlpn)) -## ClickHouse release 19.4.3.11, 2019-04-02 +### ClickHouse release 19.4.3.11, 2019-04-02 -### Bug Fixes +#### Bug Fixes * Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/ClickHouse/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2)) * Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/ClickHouse/ClickHouse/pull/4835) ([proller](https://github.com/proller)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Add a way to launch clickhouse-server image from a custom user. [#4753](https://github.com/ClickHouse/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid)) -## ClickHouse release 19.4.2.7, 2019-03-30 +### ClickHouse release 19.4.2.7, 2019-03-30 -### Bug Fixes +#### Bug Fixes * Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/ClickHouse/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -## ClickHouse release 19.4.1.3, 2019-03-19 +### ClickHouse release 19.4.1.3, 2019-03-19 -### Bug Fixes +#### Bug Fixes * Fixed remote queries which contain both `LIMIT BY` and `LIMIT`. Previously, if `LIMIT BY` and `LIMIT` were used for remote query, `LIMIT` could happen before `LIMIT BY`, which led to too filtered result. [#4708](https://github.com/ClickHouse/ClickHouse/pull/4708) ([Constantin S. Pan](https://github.com/kvap)) -## ClickHouse release 19.4.0.49, 2019-03-09 +### ClickHouse release 19.4.0.49, 2019-03-09 -### New Features +#### New Features * Added full support for `Protobuf` format (input and output, nested data structures). [#4174](https://github.com/ClickHouse/ClickHouse/pull/4174) [#4493](https://github.com/ClickHouse/ClickHouse/pull/4493) ([Vitaly Baranov](https://github.com/vitlibar)) * Added bitmap functions with Roaring Bitmaps. [#4207](https://github.com/ClickHouse/ClickHouse/pull/4207) ([Andy Yang](https://github.com/andyyzh)) [#4568](https://github.com/ClickHouse/ClickHouse/pull/4568) ([Vitaly Baranov](https://github.com/vitlibar)) * Parquet format support. [#4448](https://github.com/ClickHouse/ClickHouse/pull/4448) ([proller](https://github.com/proller)) @@ -1531,7 +1547,7 @@ lee](https://github.com/neverlee)) * Added functions `arrayEnumerateDenseRanked` and `arrayEnumerateUniqRanked` (it's like `arrayEnumerateUniq` but allows to fine tune array depth to look inside multidimensional arrays). [#4475](https://github.com/ClickHouse/ClickHouse/pull/4475) ([proller](https://github.com/proller)) [#4601](https://github.com/ClickHouse/ClickHouse/pull/4601) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Multiple JOINS with some restrictions: no asterisks, no complex aliases in ON/WHERE/GROUP BY/... [#4462](https://github.com/ClickHouse/ClickHouse/pull/4462) ([Artem Zuikov](https://github.com/4ertus2)) -### Bug Fixes +#### Bug Fixes * This release also contains all bug fixes from 19.3 and 19.1. * Fixed bug in data skipping indices: order of granules after INSERT was incorrect. [#4407](https://github.com/ClickHouse/ClickHouse/pull/4407) ([Nikita Vasilev](https://github.com/nikvas0)) * Fixed `set` index for `Nullable` and `LowCardinality` columns. Before it, `set` index with `Nullable` or `LowCardinality` column led to error `Data type must be deserialized with multiple streams` while selecting. [#4594](https://github.com/ClickHouse/ClickHouse/pull/4594) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) @@ -1553,19 +1569,19 @@ lee](https://github.com/neverlee)) * Fix lambda function with predicate optimizer. [#4408](https://github.com/ClickHouse/ClickHouse/pull/4408) ([Winter Zhang](https://github.com/zhang2014)) * Multiple JOINs multiple fixes. [#4595](https://github.com/ClickHouse/ClickHouse/pull/4595) ([Artem Zuikov](https://github.com/4ertus2)) -### Improvements +#### Improvements * Support aliases in JOIN ON section for right table columns. [#4412](https://github.com/ClickHouse/ClickHouse/pull/4412) ([Artem Zuikov](https://github.com/4ertus2)) * Result of multiple JOINs need correct result names to be used in subselects. Replace flat aliases with source names in result. [#4474](https://github.com/ClickHouse/ClickHouse/pull/4474) ([Artem Zuikov](https://github.com/4ertus2)) * Improve push-down logic for joined statements. [#4387](https://github.com/ClickHouse/ClickHouse/pull/4387) ([Ivan](https://github.com/abyss7)) -### Performance Improvements +#### Performance Improvements * Improved heuristics of "move to PREWHERE" optimization. [#4405](https://github.com/ClickHouse/ClickHouse/pull/4405) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Use proper lookup tables that uses HashTable's API for 8-bit and 16-bit keys. [#4536](https://github.com/ClickHouse/ClickHouse/pull/4536) ([Amos Bird](https://github.com/amosbird)) * Improved performance of string comparison. [#4564](https://github.com/ClickHouse/ClickHouse/pull/4564) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Cleanup distributed DDL queue in a separate thread so that it doesn't slow down the main loop that processes distributed DDL tasks. [#4502](https://github.com/ClickHouse/ClickHouse/pull/4502) ([Alex Zatelepin](https://github.com/ztlpn)) * When `min_bytes_to_use_direct_io` is set to 1, not every file was opened with O_DIRECT mode because the data size to read was sometimes underestimated by the size of one compressed block. [#4526](https://github.com/ClickHouse/ClickHouse/pull/4526) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Added support for clang-9 [#4604](https://github.com/ClickHouse/ClickHouse/pull/4604) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fix wrong `__asm__` instructions (again) [#4621](https://github.com/ClickHouse/ClickHouse/pull/4621) ([Konstantin Podshumok](https://github.com/podshumok)) * Add ability to specify settings for `clickhouse-performance-test` from command line. [#4437](https://github.com/ClickHouse/ClickHouse/pull/4437) ([alesapin](https://github.com/alesapin)) @@ -1577,29 +1593,30 @@ lee](https://github.com/neverlee)) * Fix compilation on Mac. [#4371](https://github.com/ClickHouse/ClickHouse/pull/4371) ([Vitaly Baranov](https://github.com/vitlibar)) * Build fixes for FreeBSD and various unusual build configurations. [#4444](https://github.com/ClickHouse/ClickHouse/pull/4444) ([proller](https://github.com/proller)) -## ClickHouse release 19.3.9.1, 2019-04-02 +## ClickHouse release 19.3 +### ClickHouse release 19.3.9.1, 2019-04-02 -### Bug Fixes +#### Bug Fixes * Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/ClickHouse/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2)) * Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/ClickHouse/ClickHouse/pull/4835) ([proller](https://github.com/proller)) * Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/ClickHouse/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) -### Build/Testing/Packaging Improvement +#### Build/Testing/Packaging Improvement * Add a way to launch clickhouse-server image from a custom user [#4753](https://github.com/ClickHouse/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid)) -## ClickHouse release 19.3.7, 2019-03-12 +### ClickHouse release 19.3.7, 2019-03-12 -### Bug fixes +#### Bug fixes * Fixed error in #3920. This error manifests itself as random cache corruption (messages `Unknown codec family code`, `Cannot seek through file`) and segfaults. This bug first appeared in version 19.1 and is present in versions up to 19.1.10 and 19.3.6. [#4623](https://github.com/ClickHouse/ClickHouse/pull/4623) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.3.6, 2019-03-02 +### ClickHouse release 19.3.6, 2019-03-02 -### Bug fixes +#### Bug fixes * When there are more than 1000 threads in a thread pool, `std::terminate` may happen on thread exit. [Azat Khuzhin](https://github.com/azat) [#4485](https://github.com/ClickHouse/ClickHouse/pull/4485) [#4505](https://github.com/ClickHouse/ClickHouse/pull/4505) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Now it's possible to create `ReplicatedMergeTree*` tables with comments on columns without defaults and tables with columns codecs without comments and defaults. Also fix comparison of codecs. [#4523](https://github.com/ClickHouse/ClickHouse/pull/4523) ([alesapin](https://github.com/alesapin)) @@ -1608,7 +1625,7 @@ lee](https://github.com/neverlee)) * Fixed hangup on server shutdown if distributed DDLs were used. [#4472](https://github.com/ClickHouse/ClickHouse/pull/4472) ([Alex Zatelepin](https://github.com/ztlpn)) * Incorrect column numbers were printed in error message about text format parsing for columns with number greater than 10. [#4484](https://github.com/ClickHouse/ClickHouse/pull/4484) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Fixed build with AVX enabled. [#4527](https://github.com/ClickHouse/ClickHouse/pull/4527) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Enable extended accounting and IO accounting based on good known version instead of kernel under which it is compiled. [#4541](https://github.com/ClickHouse/ClickHouse/pull/4541) ([nvartolomei](https://github.com/nvartolomei)) @@ -1616,33 +1633,33 @@ lee](https://github.com/neverlee)) * Removed the `inline` tags of `void readBinary(...)` in `Field.cpp`. Also merged redundant `namespace DB` blocks. [#4530](https://github.com/ClickHouse/ClickHouse/pull/4530) ([hcz](https://github.com/hczhcz)) -## ClickHouse release 19.3.5, 2019-02-21 +### ClickHouse release 19.3.5, 2019-02-21 -### Bug fixes +#### Bug fixes * Fixed bug with large http insert queries processing. [#4454](https://github.com/ClickHouse/ClickHouse/pull/4454) ([alesapin](https://github.com/alesapin)) * Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/ClickHouse/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/ClickHouse/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.3.4, 2019-02-16 +### ClickHouse release 19.3.4, 2019-02-16 -### Improvements +#### Improvements * Table index size is not accounted for memory limits when doing `ATTACH TABLE` query. Avoided the possibility that a table cannot be attached after being detached. [#4396](https://github.com/ClickHouse/ClickHouse/pull/4396) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Slightly raised up the limit on max string and array size received from ZooKeeper. It allows to continue to work with increased size of `CLIENT_JVMFLAGS=-Djute.maxbuffer=...` on ZooKeeper. [#4398](https://github.com/ClickHouse/ClickHouse/pull/4398) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Allow to repair abandoned replica even if it already has huge number of nodes in its queue. [#4399](https://github.com/ClickHouse/ClickHouse/pull/4399) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add one required argument to `SET` index (max stored rows number). [#4386](https://github.com/ClickHouse/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0)) -### Bug Fixes +#### Bug Fixes * Fixed `WITH ROLLUP` result for group by single `LowCardinality` key. [#4384](https://github.com/ClickHouse/ClickHouse/pull/4384) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Fixed bug in the set index (dropping a granule if it contains more than `max_rows` rows). [#4386](https://github.com/ClickHouse/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0)) * A lot of FreeBSD build fixes. [#4397](https://github.com/ClickHouse/ClickHouse/pull/4397) ([proller](https://github.com/proller)) * Fixed aliases substitution in queries with subquery containing same alias (issue [#4110](https://github.com/ClickHouse/ClickHouse/issues/4110)). [#4351](https://github.com/ClickHouse/ClickHouse/pull/4351) ([Artem Zuikov](https://github.com/4ertus2)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Add ability to run `clickhouse-server` for stateless tests in docker image. [#4347](https://github.com/ClickHouse/ClickHouse/pull/4347) ([Vasily Nemkov](https://github.com/Enmk)) -## ClickHouse release 19.3.3, 2019-02-13 +### ClickHouse release 19.3.3, 2019-02-13 -### New Features +#### New Features * Added the `KILL MUTATION` statement that allows removing mutations that are for some reasons stuck. Added `latest_failed_part`, `latest_fail_time`, `latest_fail_reason` fields to the `system.mutations` table for easier troubleshooting. [#4287](https://github.com/ClickHouse/ClickHouse/pull/4287) ([Alex Zatelepin](https://github.com/ztlpn)) * Added aggregate function `entropy` which computes Shannon entropy. [#4238](https://github.com/ClickHouse/ClickHouse/pull/4238) ([Quid37](https://github.com/Quid37)) * Added ability to send queries `INSERT INTO tbl VALUES (....` to server without splitting on `query` and `data` parts. [#4301](https://github.com/ClickHouse/ClickHouse/pull/4301) ([alesapin](https://github.com/alesapin)) @@ -1662,11 +1679,11 @@ lee](https://github.com/neverlee)) * Added hints while user make typo in function name or type in command line client. [#4239](https://github.com/ClickHouse/ClickHouse/pull/4239) ([Danila Kutenin](https://github.com/danlark1)) * Added `Query-Id` to Server's HTTP Response header. [#4231](https://github.com/ClickHouse/ClickHouse/pull/4231) ([Mikhail ](https://github.com/fandyushin)) -### Experimental features +#### Experimental features * Added `minmax` and `set` data skipping indices for MergeTree table engines family. [#4143](https://github.com/ClickHouse/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0)) * Added conversion of `CROSS JOIN` to `INNER JOIN` if possible. [#4221](https://github.com/ClickHouse/ClickHouse/pull/4221) [#4266](https://github.com/ClickHouse/ClickHouse/pull/4266) ([Artem Zuikov](https://github.com/4ertus2)) -### Bug Fixes +#### Bug Fixes * Fixed `Not found column` for duplicate columns in `JOIN ON` section. [#4279](https://github.com/ClickHouse/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2)) * Make `START REPLICATED SENDS` command start replicated sends. [#4229](https://github.com/ClickHouse/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei)) * Fixed aggregate functions execution with `Array(LowCardinality)` arguments. [#4055](https://github.com/ClickHouse/ClickHouse/pull/4055) ([KochetovNicolai](https://github.com/KochetovNicolai)) @@ -1696,7 +1713,7 @@ lee](https://github.com/neverlee)) * Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/ClickHouse/ClickHouse/pull/4343) ([proller](https://github.com/proller)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Debian package: correct /etc/clickhouse-server/preprocessed link according to config. [#4205](https://github.com/ClickHouse/ClickHouse/pull/4205) ([proller](https://github.com/proller)) * Various build fixes for FreeBSD. [#4225](https://github.com/ClickHouse/ClickHouse/pull/4225) ([proller](https://github.com/proller)) * Added ability to create, fill and drop tables in perftest. [#4220](https://github.com/ClickHouse/ClickHouse/pull/4220) ([alesapin](https://github.com/alesapin)) @@ -1716,17 +1733,17 @@ lee](https://github.com/neverlee)) * Added checking SSE and AVX instruction at start. [#4234](https://github.com/ClickHouse/ClickHouse/pull/4234) ([Igr](https://github.com/igron99)) * Init script will wait server until start. [#4281](https://github.com/ClickHouse/ClickHouse/pull/4281) ([proller](https://github.com/proller)) -### Backward Incompatible Changes +#### Backward Incompatible Changes * Removed `allow_experimental_low_cardinality_type` setting. `LowCardinality` data types are production ready. [#4323](https://github.com/ClickHouse/ClickHouse/pull/4323) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Reduce mark cache size and uncompressed cache size accordingly to available memory amount. [#4240](https://github.com/ClickHouse/ClickHouse/pull/4240) ([Lopatin Konstantin](https://github.com/k-lopatin) * Added keyword `INDEX` in `CREATE TABLE` query. A column with name `index` must be quoted with backticks or double quotes: `` `index` ``. [#4143](https://github.com/ClickHouse/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0)) * `sumMap` now promote result type instead of overflow. The old `sumMap` behavior can be obtained by using `sumMapWithOverflow` function. [#4151](https://github.com/ClickHouse/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo)) -### Performance Improvements +#### Performance Improvements * `std::sort` replaced by `pdqsort` for queries without `LIMIT`. [#4236](https://github.com/ClickHouse/ClickHouse/pull/4236) ([Evgenii Pravda](https://github.com/kvinty)) * Now server reuse threads from global thread pool. This affects performance in some corner cases. [#4150](https://github.com/ClickHouse/ClickHouse/pull/4150) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Improvements +#### Improvements * Implemented AIO support for FreeBSD. [#4305](https://github.com/ClickHouse/ClickHouse/pull/4305) ([urgordeadbeef](https://github.com/urgordeadbeef)) * `SELECT * FROM a JOIN b USING a, b` now return `a` and `b` columns only from the left table. [#4141](https://github.com/ClickHouse/ClickHouse/pull/4141) ([Artem Zuikov](https://github.com/4ertus2)) * Allow `-C` option of client to work as `-c` option. [#4232](https://github.com/ClickHouse/ClickHouse/pull/4232) ([syominsergey](https://github.com/syominsergey)) @@ -1742,34 +1759,37 @@ lee](https://github.com/neverlee)) * Added info about the replicated_can_become_leader setting to system.replicas and add logging if the replica won't try to become leader. [#4379](https://github.com/ClickHouse/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn)) -## ClickHouse release 19.1.14, 2019-03-14 +## ClickHouse release 19.1 +### ClickHouse release 19.1.14, 2019-03-14 * Fixed error `Column ... queried more than once` that may happen if the setting `asterisk_left_columns_only` is set to 1 in case of using `GLOBAL JOIN` with `SELECT *` (rare case). The issue does not exist in 19.3 and newer. [6bac7d8d](https://github.com/ClickHouse/ClickHouse/pull/4692/commits/6bac7d8d11a9b0d6de0b32b53c47eb2f6f8e7062) ([Artem Zuikov](https://github.com/4ertus2)) -## ClickHouse release 19.1.13, 2019-03-12 +### ClickHouse release 19.1.13, 2019-03-12 This release contains exactly the same set of patches as 19.3.7. -## ClickHouse release 19.1.10, 2019-03-03 +### ClickHouse release 19.1.10, 2019-03-03 This release contains exactly the same set of patches as 19.3.6. -## ClickHouse release 19.1.9, 2019-02-21 +## ClickHouse release 19.1 +### ClickHouse release 19.1.9, 2019-02-21 -### Bug fixes +#### Bug fixes * Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/ClickHouse/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/ClickHouse/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.1.8, 2019-02-16 +### ClickHouse release 19.1.8, 2019-02-16 -### Bug Fixes +#### Bug Fixes * Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/ClickHouse/ClickHouse/pull/4343) ([proller](https://github.com/proller)) -## ClickHouse release 19.1.7, 2019-02-15 +## ClickHouse release 19.1 +### ClickHouse release 19.1.7, 2019-02-15 -### Bug Fixes +#### Bug Fixes * Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/ClickHouse/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird)) * Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/ClickHouse/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/ClickHouse/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1795,9 +1815,9 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed bug with incorrect `Date` and `DateTime` comparison. [#4237](https://github.com/ClickHouse/ClickHouse/pull/4237) ([valexey](https://github.com/valexey)) * Fixed incorrect result when `Date` and `DateTime` arguments are used in branches of conditional operator (function `if`). Added generic case for function `if`. [#4243](https://github.com/ClickHouse/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov)) -## ClickHouse release 19.1.6, 2019-01-24 +### ClickHouse release 19.1.6, 2019-01-24 -### New Features +#### New Features * Custom per column compression codecs for tables. [#3899](https://github.com/ClickHouse/ClickHouse/pull/3899) [#4111](https://github.com/ClickHouse/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin), [Winter Zhang](https://github.com/zhang2014), [Anatoly](https://github.com/Sindbag)) * Added compression codec `Delta`. [#4052](https://github.com/ClickHouse/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin)) @@ -1815,12 +1835,12 @@ This release contains exactly the same set of patches as 19.3.6. * Added table function `remoteSecure`. Function works as `remote`, but uses secure connection. [#4088](https://github.com/ClickHouse/ClickHouse/pull/4088) ([proller](https://github.com/proller)) -### Experimental features +#### Experimental features * Added multiple JOINs emulation (`allow_experimental_multiple_joins_emulation` setting). [#3946](https://github.com/ClickHouse/ClickHouse/pull/3946) ([Artem Zuikov](https://github.com/4ertus2)) -### Bug Fixes +#### Bug Fixes * Make `compiled_expression_cache_size` setting limited by default to lower memory consumption. [#4041](https://github.com/ClickHouse/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin)) * Fix a bug that led to hangups in threads that perform ALTERs of Replicated tables and in the thread that updates configuration from ZooKeeper. [#2947](https://github.com/ClickHouse/ClickHouse/issues/2947) [#3891](https://github.com/ClickHouse/ClickHouse/issues/3891) [#3934](https://github.com/ClickHouse/ClickHouse/pull/3934) ([Alex Zatelepin](https://github.com/ztlpn)) @@ -1849,7 +1869,7 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed column aliases for query with `JOIN ON` syntax and distributed tables. [#3980](https://github.com/ClickHouse/ClickHouse/pull/3980) ([Winter Zhang](https://github.com/zhang2014)) * Fixed error in internal implementation of `quantileTDigest` (found by Artem Vakhrushev). This error never happens in ClickHouse and was relevant only for those who use ClickHouse codebase as a library directly. [#3935](https://github.com/ClickHouse/ClickHouse/pull/3935) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Improvements +#### Improvements * Support for `IF NOT EXISTS` in `ALTER TABLE ADD COLUMN` statements along with `IF EXISTS` in `DROP/MODIFY/CLEAR/COMMENT COLUMN`. [#3900](https://github.com/ClickHouse/ClickHouse/pull/3900) ([Boris Granveaud](https://github.com/bgranvea)) * Function `parseDateTimeBestEffort`: support for formats `DD.MM.YYYY`, `DD.MM.YY`, `DD-MM-YYYY`, `DD-Mon-YYYY`, `DD/Month/YYYY` and similar. [#3922](https://github.com/ClickHouse/ClickHouse/pull/3922) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1864,7 +1884,7 @@ This release contains exactly the same set of patches as 19.3.6. * Add check that `SET send_logs_level = 'value'` query accept appropriate value. [#3873](https://github.com/ClickHouse/ClickHouse/pull/3873) ([Sabyanin Maxim](https://github.com/s-mx)) * Fixed data type check in type conversion functions. [#3896](https://github.com/ClickHouse/ClickHouse/pull/3896) ([Winter Zhang](https://github.com/zhang2014)) -### Performance Improvements +#### Performance Improvements * Add a MergeTree setting `use_minimalistic_part_header_in_zookeeper`. If enabled, Replicated tables will store compact part metadata in a single part znode. This can dramatically reduce ZooKeeper snapshot size (especially if the tables have a lot of columns). Note that after enabling this setting you will not be able to downgrade to a version that doesn't support it. [#3960](https://github.com/ClickHouse/ClickHouse/pull/3960) ([Alex Zatelepin](https://github.com/ztlpn)) * Add an DFA-based implementation for functions `sequenceMatch` and `sequenceCount` in case pattern doesn't contain time. [#4004](https://github.com/ClickHouse/ClickHouse/pull/4004) ([Léo Ercolanelli](https://github.com/ercolanelli-leo)) @@ -1872,13 +1892,13 @@ This release contains exactly the same set of patches as 19.3.6. * Zero left padding PODArray so that -1 element is always valid and zeroed. It's used for branchless calculation of offsets. [#3920](https://github.com/ClickHouse/ClickHouse/pull/3920) ([Amos Bird](https://github.com/amosbird)) * Reverted `jemalloc` version which lead to performance degradation. [#4018](https://github.com/ClickHouse/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov)) -### Backward Incompatible Changes +#### Backward Incompatible Changes * Removed undocumented feature `ALTER MODIFY PRIMARY KEY` because it was superseded by the `ALTER MODIFY ORDER BY` command. [#3887](https://github.com/ClickHouse/ClickHouse/pull/3887) ([Alex Zatelepin](https://github.com/ztlpn)) * Removed function `shardByHash`. [#3833](https://github.com/ClickHouse/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Forbid using scalar subqueries with result of type `AggregateFunction`. [#3865](https://github.com/ClickHouse/ClickHouse/pull/3865) ([Ivan](https://github.com/abyss7)) -### Build/Testing/Packaging Improvements +#### Build/Testing/Packaging Improvements * Added support for PowerPC (`ppc64le`) build. [#4132](https://github.com/ClickHouse/ClickHouse/pull/4132) ([Danila Kutenin](https://github.com/danlark1)) * Stateful functional tests are run on public available dataset. [#3969](https://github.com/ClickHouse/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov)) @@ -1906,24 +1926,25 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed typos in comments. [#4089](https://github.com/ClickHouse/ClickHouse/pull/4089) ([Evgenii Pravda](https://github.com/kvinty)) -## ClickHouse release 18.16.1, 2018-12-21 +## ClickHouse release 18.16 +### ClickHouse release 18.16.1, 2018-12-21 -### Bug fixes: +#### Bug fixes: * Fixed an error that led to problems with updating dictionaries with the ODBC source. [#3825](https://github.com/ClickHouse/ClickHouse/issues/3825), [#3829](https://github.com/ClickHouse/ClickHouse/issues/3829) * JIT compilation of aggregate functions now works with LowCardinality columns. [#3838](https://github.com/ClickHouse/ClickHouse/issues/3838) -### Improvements: +#### Improvements: * Added the `low_cardinality_allow_in_native_format` setting (enabled by default). When disabled, LowCardinality columns will be converted to ordinary columns for SELECT queries and ordinary columns will be expected for INSERT queries. [#3879](https://github.com/ClickHouse/ClickHouse/pull/3879) -### Build improvements: +#### Build improvements: * Fixes for builds on macOS and ARM. -## ClickHouse release 18.16.0, 2018-12-14 +### ClickHouse release 18.16.0, 2018-12-14 -### New features: +#### New features: * `DEFAULT` expressions are evaluated for missing fields when loading data in semi-structured input formats (`JSONEachRow`, `TSKV`). The feature is enabled with the `insert_sample_with_metadata` setting. [#3555](https://github.com/ClickHouse/ClickHouse/pull/3555) * The `ALTER TABLE` query now has the `MODIFY ORDER BY` action for changing the sorting key when adding or removing a table column. This is useful for tables in the `MergeTree` family that perform additional tasks when merging based on this sorting key, such as `SummingMergeTree`, `AggregatingMergeTree`, and so on. [#3581](https://github.com/ClickHouse/ClickHouse/pull/3581) [#3755](https://github.com/ClickHouse/ClickHouse/pull/3755) @@ -1942,7 +1963,7 @@ This release contains exactly the same set of patches as 19.3.6. * Added the `is_in_partition_key`, `is_in_sorting_key`, `is_in_primary_key`, and `is_in_sampling_key` columns to the `system.columns` table. [#3609](https://github.com/ClickHouse/ClickHouse/pull/3609) * Added the `min_time` and `max_time` columns to the `system.parts` table. These columns are populated when the partitioning key is an expression consisting of `DateTime` columns. [Emmanuel Donin de Rosière](https://github.com/ClickHouse/ClickHouse/pull/3800) -### Bug fixes: +#### Bug fixes: * Fixes and performance improvements for the `LowCardinality` data type. `GROUP BY` using `LowCardinality(Nullable(...))`. Getting the values of `extremes`. Processing high-order functions. `LEFT ARRAY JOIN`. Distributed `GROUP BY`. Functions that return `Array`. Execution of `ORDER BY`. Writing to `Distributed` tables (nicelulu). Backward compatibility for `INSERT` queries from old clients that implement the `Native` protocol. Support for `LowCardinality` for `JOIN`. Improved performance when working in a single stream. [#3823](https://github.com/ClickHouse/ClickHouse/pull/3823) [#3803](https://github.com/ClickHouse/ClickHouse/pull/3803) [#3799](https://github.com/ClickHouse/ClickHouse/pull/3799) [#3769](https://github.com/ClickHouse/ClickHouse/pull/3769) [#3744](https://github.com/ClickHouse/ClickHouse/pull/3744) [#3681](https://github.com/ClickHouse/ClickHouse/pull/3681) [#3651](https://github.com/ClickHouse/ClickHouse/pull/3651) [#3649](https://github.com/ClickHouse/ClickHouse/pull/3649) [#3641](https://github.com/ClickHouse/ClickHouse/pull/3641) [#3632](https://github.com/ClickHouse/ClickHouse/pull/3632) [#3568](https://github.com/ClickHouse/ClickHouse/pull/3568) [#3523](https://github.com/ClickHouse/ClickHouse/pull/3523) [#3518](https://github.com/ClickHouse/ClickHouse/pull/3518) * Fixed how the `select_sequential_consistency` option works. Previously, when this setting was enabled, an incomplete result was sometimes returned after beginning to write to a new partition. [#2863](https://github.com/ClickHouse/ClickHouse/pull/2863) @@ -1969,7 +1990,7 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed a race condition when reading from `Buffer` tables and simultaneously performing `ALTER` or `DROP` on the target tables. [#3719](https://github.com/ClickHouse/ClickHouse/pull/3719) * Fixed a segfault if the `max_temporary_non_const_columns` limit was exceeded. [#3788](https://github.com/ClickHouse/ClickHouse/pull/3788) -### Improvements: +#### Improvements: * The server does not write the processed configuration files to the `/etc/clickhouse-server/` directory. Instead, it saves them in the `preprocessed_configs` directory inside `path`. This means that the `/etc/clickhouse-server/` directory doesn't have write access for the `clickhouse` user, which improves security. [#2443](https://github.com/ClickHouse/ClickHouse/pull/2443) * The `min_merge_bytes_to_use_direct_io` option is set to 10 GiB by default. A merge that forms large parts of tables from the MergeTree family will be performed in `O_DIRECT` mode, which prevents excessive page cache eviction. [#3504](https://github.com/ClickHouse/ClickHouse/pull/3504) @@ -2001,7 +2022,7 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed the behavior of stateful functions like `rowNumberInAllBlocks`. They previously output a result that was one number larger due to starting during query analysis. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3729) * If the `force_restore_data` file can't be deleted, an error message is displayed. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3794) -### Build improvements: +#### Build improvements: * Updated the `jemalloc` library, which fixes a potential memory leak. [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/3557) * Profiling with `jemalloc` is enabled by default in order to debug builds. [2cc82f5c](https://github.com/ClickHouse/ClickHouse/commit/2cc82f5cbe266421cd4c1165286c2c47e5ffcb15) @@ -2012,95 +2033,96 @@ This release contains exactly the same set of patches as 19.3.6. * For a Docker image, added support for initializing databases using files in the `/docker-entrypoint-initdb.d` directory. [Konstantin Lebedev](https://github.com/ClickHouse/ClickHouse/pull/3695) * Fixes for builds on ARM. [#3709](https://github.com/ClickHouse/ClickHouse/pull/3709) -### Backward incompatible changes: +#### Backward incompatible changes: * Removed the ability to compare the `Date` type with a number. Instead of `toDate('2018-12-18') = 17883`, you must use explicit type conversion `= toDate(17883)` [#3687](https://github.com/ClickHouse/ClickHouse/pull/3687) -## ClickHouse release 18.14.19, 2018-12-19 +## ClickHouse release 18.14 +### ClickHouse release 18.14.19, 2018-12-19 -### Bug fixes: +#### Bug fixes: * Fixed an error that led to problems with updating dictionaries with the ODBC source. [#3825](https://github.com/ClickHouse/ClickHouse/issues/3825), [#3829](https://github.com/ClickHouse/ClickHouse/issues/3829) * Databases are correctly specified when executing DDL `ON CLUSTER` queries. [#3460](https://github.com/ClickHouse/ClickHouse/pull/3460) * Fixed a segfault if the `max_temporary_non_const_columns` limit was exceeded. [#3788](https://github.com/ClickHouse/ClickHouse/pull/3788) -### Build improvements: +#### Build improvements: * Fixes for builds on ARM. -## ClickHouse release 18.14.18, 2018-12-04 +### ClickHouse release 18.14.18, 2018-12-04 -### Bug fixes: +#### Bug fixes: * Fixed error in `dictGet...` function for dictionaries of type `range`, if one of the arguments is constant and other is not. [#3751](https://github.com/ClickHouse/ClickHouse/pull/3751) * Fixed error that caused messages `netlink: '...': attribute type 1 has an invalid length` to be printed in Linux kernel log, that was happening only on fresh enough versions of Linux kernel. [#3749](https://github.com/ClickHouse/ClickHouse/pull/3749) * Fixed segfault in function `empty` for argument of `FixedString` type. [Daniel, Dao Quang Minh](https://github.com/ClickHouse/ClickHouse/pull/3703) * Fixed excessive memory allocation when using large value of `max_query_size` setting (a memory chunk of `max_query_size` bytes was preallocated at once). [#3720](https://github.com/ClickHouse/ClickHouse/pull/3720) -### Build changes: +#### Build changes: * Fixed build with LLVM/Clang libraries of version 7 from the OS packages (these libraries are used for runtime query compilation). [#3582](https://github.com/ClickHouse/ClickHouse/pull/3582) -## ClickHouse release 18.14.17, 2018-11-30 +### ClickHouse release 18.14.17, 2018-11-30 -### Bug fixes: +#### Bug fixes: * Fixed cases when the ODBC bridge process did not terminate with the main server process. [#3642](https://github.com/ClickHouse/ClickHouse/pull/3642) * Fixed synchronous insertion into the `Distributed` table with a columns list that differs from the column list of the remote table. [#3673](https://github.com/ClickHouse/ClickHouse/pull/3673) * Fixed a rare race condition that can lead to a crash when dropping a MergeTree table. [#3643](https://github.com/ClickHouse/ClickHouse/pull/3643) * Fixed a query deadlock in case when query thread creation fails with the `Resource temporarily unavailable` error. [#3643](https://github.com/ClickHouse/ClickHouse/pull/3643) * Fixed parsing of the `ENGINE` clause when the `CREATE AS table` syntax was used and the `ENGINE` clause was specified before the `AS table` (the error resulted in ignoring the specified engine). [#3692](https://github.com/ClickHouse/ClickHouse/pull/3692) -## ClickHouse release 18.14.15, 2018-11-21 +### ClickHouse release 18.14.15, 2018-11-21 -### Bug fixes: +#### Bug fixes: * The size of memory chunk was overestimated while deserializing the column of type `Array(String)` that leads to "Memory limit exceeded" errors. The issue appeared in version 18.12.13. [#3589](https://github.com/ClickHouse/ClickHouse/issues/3589) -## ClickHouse release 18.14.14, 2018-11-20 +### ClickHouse release 18.14.14, 2018-11-20 -### Bug fixes: +#### Bug fixes: * Fixed `ON CLUSTER` queries when cluster configured as secure (flag ``). [#3599](https://github.com/ClickHouse/ClickHouse/pull/3599) -### Build changes: +#### Build changes: * Fixed problems (llvm-7 from system, macos) [#3582](https://github.com/ClickHouse/ClickHouse/pull/3582) -## ClickHouse release 18.14.13, 2018-11-08 +### ClickHouse release 18.14.13, 2018-11-08 -### Bug fixes: +#### Bug fixes: * Fixed the `Block structure mismatch in MergingSorted stream` error. [#3162](https://github.com/ClickHouse/ClickHouse/issues/3162) * Fixed `ON CLUSTER` queries in case when secure connections were turned on in the cluster config (the `` flag). [#3465](https://github.com/ClickHouse/ClickHouse/pull/3465) * Fixed an error in queries that used `SAMPLE`, `PREWHERE` and alias columns. [#3543](https://github.com/ClickHouse/ClickHouse/pull/3543) * Fixed a rare `unknown compression method` error when the `min_bytes_to_use_direct_io` setting was enabled. [3544](https://github.com/ClickHouse/ClickHouse/pull/3544) -### Performance improvements: +#### Performance improvements: * Fixed performance regression of queries with `GROUP BY` of columns of UInt16 or Date type when executing on AMD EPYC processors. [Igor Lapko](https://github.com/ClickHouse/ClickHouse/pull/3512) * Fixed performance regression of queries that process long strings. [#3530](https://github.com/ClickHouse/ClickHouse/pull/3530) -### Build improvements: +#### Build improvements: * Improvements for simplifying the Arcadia build. [#3475](https://github.com/ClickHouse/ClickHouse/pull/3475), [#3535](https://github.com/ClickHouse/ClickHouse/pull/3535) -## ClickHouse release 18.14.12, 2018-11-02 +### ClickHouse release 18.14.12, 2018-11-02 -### Bug fixes: +#### Bug fixes: * Fixed a crash on joining two unnamed subqueries. [#3505](https://github.com/ClickHouse/ClickHouse/pull/3505) * Fixed generating incorrect queries (with an empty `WHERE` clause) when querying external databases. [hotid](https://github.com/ClickHouse/ClickHouse/pull/3477) * Fixed using an incorrect timeout value in ODBC dictionaries. [Marek Vavruša](https://github.com/ClickHouse/ClickHouse/pull/3511) -## ClickHouse release 18.14.11, 2018-10-29 +### ClickHouse release 18.14.11, 2018-10-29 -### Bug fixes: +#### Bug fixes: * Fixed the error `Block structure mismatch in UNION stream: different number of columns` in LIMIT queries. [#2156](https://github.com/ClickHouse/ClickHouse/issues/2156) * Fixed errors when merging data in tables containing arrays inside Nested structures. [#3397](https://github.com/ClickHouse/ClickHouse/pull/3397) * Fixed incorrect query results if the `merge_tree_uniform_read_distribution` setting is disabled (it is enabled by default). [#3429](https://github.com/ClickHouse/ClickHouse/pull/3429) * Fixed an error on inserts to a Distributed table in Native format. [#3411](https://github.com/ClickHouse/ClickHouse/issues/3411) -## ClickHouse release 18.14.10, 2018-10-23 +### ClickHouse release 18.14.10, 2018-10-23 * The `compile_expressions` setting (JIT compilation of expressions) is disabled by default. [#3410](https://github.com/ClickHouse/ClickHouse/pull/3410) * The `enable_optimize_predicate_expression` setting is disabled by default. -## ClickHouse release 18.14.9, 2018-10-16 +### ClickHouse release 18.14.9, 2018-10-16 -### New features: +#### New features: * The `WITH CUBE` modifier for `GROUP BY` (the alternative syntax `GROUP BY CUBE(...)` is also available). [#3172](https://github.com/ClickHouse/ClickHouse/pull/3172) * Added the `formatDateTime` function. [Alexandr Krasheninnikov](https://github.com/ClickHouse/ClickHouse/pull/2770) @@ -2113,12 +2135,12 @@ This release contains exactly the same set of patches as 19.3.6. * Now you can use pre-defined `database` and `table` macros when declaring `Replicated` tables. [#3251](https://github.com/ClickHouse/ClickHouse/pull/3251) * Added the ability to read `Decimal` type values in engineering notation (indicating powers of ten). [#3153](https://github.com/ClickHouse/ClickHouse/pull/3153) -### Experimental features: +#### Experimental features: * Optimization of the GROUP BY clause for `LowCardinality data types.` [#3138](https://github.com/ClickHouse/ClickHouse/pull/3138) * Optimized calculation of expressions for `LowCardinality data types.` [#3200](https://github.com/ClickHouse/ClickHouse/pull/3200) -### Improvements: +#### Improvements: * Significantly reduced memory consumption for queries with `ORDER BY` and `LIMIT`. See the `max_bytes_before_remerge_sort` setting. [#3205](https://github.com/ClickHouse/ClickHouse/pull/3205) * In the absence of `JOIN` (`LEFT`, `INNER`, ...), `INNER JOIN` is assumed. [#3147](https://github.com/ClickHouse/ClickHouse/pull/3147) @@ -2149,7 +2171,7 @@ This release contains exactly the same set of patches as 19.3.6. * Reduced the number of `open` and `close` system calls when reading from a `MergeTree table`. [#3283](https://github.com/ClickHouse/ClickHouse/pull/3283) * A `TRUNCATE TABLE` query can be executed on any replica (the query is passed to the leader replica). [Kirill Shvakov](https://github.com/ClickHouse/ClickHouse/pull/3375) -### Bug fixes: +#### Bug fixes: * Fixed an issue with `Dictionary` tables for `range_hashed` dictionaries. This error occurred in version 18.12.17. [#1702](https://github.com/ClickHouse/ClickHouse/pull/1702) * Fixed an error when loading `range_hashed` dictionaries (the message `Unsupported type Nullable (...)`). This error occurred in version 18.12.17. [#3362](https://github.com/ClickHouse/ClickHouse/pull/3362) @@ -2185,13 +2207,15 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed segfault that could occur in rare cases after optimization that replaced AND chains from equality evaluations with the corresponding IN expression. [liuyimin-bytedance](https://github.com/ClickHouse/ClickHouse/pull/3339) * Minor corrections to `clickhouse-benchmark`: previously, client information was not sent to the server; now the number of queries executed is calculated more accurately when shutting down and for limiting the number of iterations. [#3351](https://github.com/ClickHouse/ClickHouse/pull/3351) [#3352](https://github.com/ClickHouse/ClickHouse/pull/3352) -### Backward incompatible changes: +#### Backward incompatible changes: * Removed the `allow_experimental_decimal_type` option. The `Decimal` data type is available for default use. [#3329](https://github.com/ClickHouse/ClickHouse/pull/3329) -## ClickHouse release 18.12.17, 2018-09-16 +## ClickHouse release 18.12 -### New features: +### ClickHouse release 18.12.17, 2018-09-16 + +#### New features: * `invalidate_query` (the ability to specify a query to check whether an external dictionary needs to be updated) is implemented for the `clickhouse` source. [#3126](https://github.com/ClickHouse/ClickHouse/pull/3126) * Added the ability to use `UInt*`, `Int*`, and `DateTime` data types (along with the `Date` type) as a `range_hashed` external dictionary key that defines the boundaries of ranges. Now `NULL` can be used to designate an open range. [Vasily Nemkov](https://github.com/ClickHouse/ClickHouse/pull/3123) @@ -2199,32 +2223,32 @@ This release contains exactly the same set of patches as 19.3.6. * The `Decimal` type now supports mathematical functions (`exp`, `sin` and so on.) [#3129](https://github.com/ClickHouse/ClickHouse/pull/3129) * The `system.part_log` table now has the `partition_id` column. [#3089](https://github.com/ClickHouse/ClickHouse/pull/3089) -### Bug fixes: +#### Bug fixes: * `Merge` now works correctly on `Distributed` tables. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3159) * Fixed incompatibility (unnecessary dependency on the `glibc` version) that made it impossible to run ClickHouse on `Ubuntu Precise` and older versions. The incompatibility arose in version 18.12.13. [#3130](https://github.com/ClickHouse/ClickHouse/pull/3130) * Fixed errors in the `enable_optimize_predicate_expression` setting. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3107) * Fixed a minor issue with backwards compatibility that appeared when working with a cluster of replicas on versions earlier than 18.12.13 and simultaneously creating a new replica of a table on a server with a newer version (shown in the message `Can not clone replica, because the ... updated to new ClickHouse version`, which is logical, but shouldn't happen). [#3122](https://github.com/ClickHouse/ClickHouse/pull/3122) -### Backward incompatible changes: +#### Backward incompatible changes: * The `enable_optimize_predicate_expression` option is enabled by default (which is rather optimistic). If query analysis errors occur that are related to searching for the column names, set `enable_optimize_predicate_expression` to 0. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3107) -## ClickHouse release 18.12.14, 2018-09-13 +### ClickHouse release 18.12.14, 2018-09-13 -### New features: +#### New features: * Added support for `ALTER UPDATE` queries. [#3035](https://github.com/ClickHouse/ClickHouse/pull/3035) * Added the `allow_ddl` option, which restricts the user's access to DDL queries. [#3104](https://github.com/ClickHouse/ClickHouse/pull/3104) * Added the `min_merge_bytes_to_use_direct_io` option for `MergeTree` engines, which allows you to set a threshold for the total size of the merge (when above the threshold, data part files will be handled using O_DIRECT). [#3117](https://github.com/ClickHouse/ClickHouse/pull/3117) * The `system.merges` system table now contains the `partition_id` column. [#3099](https://github.com/ClickHouse/ClickHouse/pull/3099) -### Improvements +#### Improvements * If a data part remains unchanged during mutation, it isn't downloaded by replicas. [#3103](https://github.com/ClickHouse/ClickHouse/pull/3103) * Autocomplete is available for names of settings when working with `clickhouse-client`. [#3106](https://github.com/ClickHouse/ClickHouse/pull/3106) -### Bug fixes: +#### Bug fixes: * Added a check for the sizes of arrays that are elements of `Nested` type fields when inserting. [#3118](https://github.com/ClickHouse/ClickHouse/pull/3118) * Fixed an error updating external dictionaries with the `ODBC` source and `hashed` storage. This error occurred in version 18.12.13. @@ -2232,9 +2256,9 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed an error in aggregate functions for arrays that can have `NULL` elements. [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/3097) -## ClickHouse release 18.12.13, 2018-09-10 +### ClickHouse release 18.12.13, 2018-09-10 -### New features: +#### New features: * Added the `DECIMAL(digits, scale)` data type (`Decimal32(scale)`, `Decimal64(scale)`, `Decimal128(scale)`). To enable it, use the setting `allow_experimental_decimal_type`. [#2846](https://github.com/ClickHouse/ClickHouse/pull/2846) [#2970](https://github.com/ClickHouse/ClickHouse/pull/2970) [#3008](https://github.com/ClickHouse/ClickHouse/pull/3008) [#3047](https://github.com/ClickHouse/ClickHouse/pull/3047) * New `WITH ROLLUP` modifier for `GROUP BY` (alternative syntax: `GROUP BY ROLLUP(...)`). [#2948](https://github.com/ClickHouse/ClickHouse/pull/2948) @@ -2258,12 +2282,12 @@ This release contains exactly the same set of patches as 19.3.6. * Now you can add (merge) states of aggregate functions by using the plus operator, and multiply the states of aggregate functions by a nonnegative constant. [#3062](https://github.com/ClickHouse/ClickHouse/pull/3062) [#3034](https://github.com/ClickHouse/ClickHouse/pull/3034) * Tables in the MergeTree family now have the virtual column `_partition_id`. [#3089](https://github.com/ClickHouse/ClickHouse/pull/3089) -### Experimental features: +#### Experimental features: * Added the `LowCardinality(T)` data type. This data type automatically creates a local dictionary of values and allows data processing without unpacking the dictionary. [#2830](https://github.com/ClickHouse/ClickHouse/pull/2830) * Added a cache of JIT-compiled functions and a counter for the number of uses before compiling. To JIT compile expressions, enable the `compile_expressions` setting. [#2990](https://github.com/ClickHouse/ClickHouse/pull/2990) [#3077](https://github.com/ClickHouse/ClickHouse/pull/3077) -### Improvements: +#### Improvements: * Fixed the problem with unlimited accumulation of the replication log when there are abandoned replicas. Added an effective recovery mode for replicas with a long lag. * Improved performance of `GROUP BY` with multiple aggregation fields when one of them is string and the others are fixed length. @@ -2292,7 +2316,7 @@ This release contains exactly the same set of patches as 19.3.6. * Added randomization when running the cleanup thread periodically for `ReplicatedMergeTree` tables in order to avoid periodic load spikes when there are a very large number of `ReplicatedMergeTree` tables. * Support for `ATTACH TABLE ... ON CLUSTER` queries. [#3025](https://github.com/ClickHouse/ClickHouse/pull/3025) -### Bug fixes: +#### Bug fixes: * Fixed an issue with `Dictionary` tables (throws the `Size of offsets doesn't match size of column` or `Unknown compression method` exception). This bug appeared in version 18.10.3. [#2913](https://github.com/ClickHouse/ClickHouse/issues/2913) * Fixed a bug when merging `CollapsingMergeTree` tables if one of the data parts is empty (these parts are formed during merge or `ALTER DELETE` if all data was deleted), and the `vertical` algorithm was used for the merge. [#3049](https://github.com/ClickHouse/ClickHouse/pull/3049) @@ -2316,17 +2340,17 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed incorrect code for adding nested data structures in a `SummingMergeTree`. * When allocating memory for states of aggregate functions, alignment is correctly taken into account, which makes it possible to use operations that require alignment when implementing states of aggregate functions. [chenxing-xc](https://github.com/ClickHouse/ClickHouse/pull/2808) -### Security fix: +#### Security fix: * Safe use of ODBC data sources. Interaction with ODBC drivers uses a separate `clickhouse-odbc-bridge` process. Errors in third-party ODBC drivers no longer cause problems with server stability or vulnerabilities. [#2828](https://github.com/ClickHouse/ClickHouse/pull/2828) [#2879](https://github.com/ClickHouse/ClickHouse/pull/2879) [#2886](https://github.com/ClickHouse/ClickHouse/pull/2886) [#2893](https://github.com/ClickHouse/ClickHouse/pull/2893) [#2921](https://github.com/ClickHouse/ClickHouse/pull/2921) * Fixed incorrect validation of the file path in the `catBoostPool` table function. [#2894](https://github.com/ClickHouse/ClickHouse/pull/2894) * The contents of system tables (`tables`, `databases`, `parts`, `columns`, `parts_columns`, `merges`, `mutations`, `replicas`, and `replication_queue`) are filtered according to the user's configured access to databases (`allow_databases`). [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2856) -### Backward incompatible changes: +#### Backward incompatible changes: * In queries with JOIN, the star character expands to a list of columns in all tables, in compliance with the SQL standard. You can restore the old behavior by setting `asterisk_left_columns_only` to 1 on the user configuration level. -### Build changes: +#### Build changes: * Most integration tests can now be run by commit. * Code style checks can also be run by commit. @@ -2335,16 +2359,18 @@ This release contains exactly the same set of patches as 19.3.6. * Debugging the build uses the `jemalloc` debug option. * The interface of the library for interacting with ZooKeeper is declared abstract. [#2950](https://github.com/ClickHouse/ClickHouse/pull/2950) -## ClickHouse release 18.10.3, 2018-08-13 +## ClickHouse release 18.10 -### New features: +### ClickHouse release 18.10.3, 2018-08-13 + +#### New features: * HTTPS can be used for replication. [#2760](https://github.com/ClickHouse/ClickHouse/pull/2760) * Added the functions `murmurHash2_64`, `murmurHash3_32`, `murmurHash3_64`, and `murmurHash3_128` in addition to the existing `murmurHash2_32`. [#2791](https://github.com/ClickHouse/ClickHouse/pull/2791) * Support for Nullable types in the ClickHouse ODBC driver (`ODBCDriver2` output format). [#2834](https://github.com/ClickHouse/ClickHouse/pull/2834) * Support for `UUID` in the key columns. -### Improvements: +#### Improvements: * Clusters can be removed without restarting the server when they are deleted from the config files. [#2777](https://github.com/ClickHouse/ClickHouse/pull/2777) * External dictionaries can be removed without restarting the server when they are removed from config files. [#2779](https://github.com/ClickHouse/ClickHouse/pull/2779) @@ -2360,7 +2386,7 @@ This release contains exactly the same set of patches as 19.3.6. * Added the `prefer_localhost_replica` setting for disabling the preference for a local replica and going to a local replica without inter-process interaction. [#2832](https://github.com/ClickHouse/ClickHouse/pull/2832) * The `quantileExact` aggregate function returns `nan` in the case of aggregation on an empty `Float32` or `Float64` set. [Sundy Li](https://github.com/ClickHouse/ClickHouse/pull/2855) -### Bug fixes: +#### Bug fixes: * Removed unnecessary escaping of the connection string parameters for ODBC, which made it impossible to establish a connection. This error occurred in version 18.6.0. * Fixed the logic for processing `REPLACE PARTITION` commands in the replication queue. If there are two `REPLACE` commands for the same partition, the incorrect logic could cause one of them to remain in the replication queue and not be executed. [#2814](https://github.com/ClickHouse/ClickHouse/pull/2814) @@ -2371,11 +2397,11 @@ This release contains exactly the same set of patches as 19.3.6. * Fixed incorrect clickhouse-client response code in case of a query error. * Fixed incorrect behavior of materialized views containing DISTINCT. [#2795](https://github.com/ClickHouse/ClickHouse/issues/2795) -### Backward incompatible changes +#### Backward incompatible changes * Removed support for CHECK TABLE queries for Distributed tables. -### Build changes: +#### Build changes: * The allocator has been replaced: `jemalloc` is now used instead of `tcmalloc`. In some scenarios, this increases speed up to 20%. However, there are queries that have slowed by up to 20%. Memory consumption has been reduced by approximately 10% in some scenarios, with improved stability. With highly competitive loads, CPU usage in userspace and in system shows just a slight increase. [#2773](https://github.com/ClickHouse/ClickHouse/pull/2773) * Use of libressl from a submodule. [#1983](https://github.com/ClickHouse/ClickHouse/pull/1983) [#2807](https://github.com/ClickHouse/ClickHouse/pull/2807) @@ -2383,37 +2409,43 @@ This release contains exactly the same set of patches as 19.3.6. * Use of mariadb-connector-c from a submodule. [#2785](https://github.com/ClickHouse/ClickHouse/pull/2785) * Added functional test files to the repository that depend on the availability of test data (for the time being, without the test data itself). -## ClickHouse release 18.6.0, 2018-08-02 +## ClickHouse release 18.6 -### New features: +### ClickHouse release 18.6.0, 2018-08-02 + +#### New features: * Added support for ON expressions for the JOIN ON syntax: `JOIN ON Expr([table.]column ...) = Expr([table.]column, ...) [AND Expr([table.]column, ...) = Expr([table.]column, ...) ...]` The expression must be a chain of equalities joined by the AND operator. Each side of the equality can be an arbitrary expression over the columns of one of the tables. The use of fully qualified column names is supported (`table.name`, `database.table.name`, `table_alias.name`, `subquery_alias.name`) for the right table. [#2742](https://github.com/ClickHouse/ClickHouse/pull/2742) * HTTPS can be enabled for replication. [#2760](https://github.com/ClickHouse/ClickHouse/pull/2760) -### Improvements: +#### Improvements: * The server passes the patch component of its version to the client. Data about the patch version component is in `system.processes` and `query_log`. [#2646](https://github.com/ClickHouse/ClickHouse/pull/2646) -## ClickHouse release 18.5.1, 2018-07-31 +## ClickHouse release 18.5 -### New features: +### ClickHouse release 18.5.1, 2018-07-31 + +#### New features: * Added the hash function `murmurHash2_32` [#2756](https://github.com/ClickHouse/ClickHouse/pull/2756). -### Improvements: +#### Improvements: * Now you can use the `from_env` [#2741](https://github.com/ClickHouse/ClickHouse/pull/2741) attribute to set values in config files from environment variables. * Added case-insensitive versions of the `coalesce`, `ifNull`, and `nullIf functions` [#2752](https://github.com/ClickHouse/ClickHouse/pull/2752). -### Bug fixes: +#### Bug fixes: * Fixed a possible bug when starting a replica [#2759](https://github.com/ClickHouse/ClickHouse/pull/2759). -## ClickHouse release 18.4.0, 2018-07-28 +## ClickHouse release 18.4 -### New features: +### ClickHouse release 18.4.0, 2018-07-28 + +#### New features: * Added system tables: `formats`, `data_type_families`, `aggregate_function_combinators`, `table_functions`, `table_engines`, `collations` [#2721](https://github.com/ClickHouse/ClickHouse/pull/2721). * Added the ability to use a table function instead of a table as an argument of a `remote` or `cluster table function` [#2708](https://github.com/ClickHouse/ClickHouse/pull/2708). @@ -2421,26 +2453,28 @@ The expression must be a chain of equalities joined by the AND operator. Each si * The `has` function now allows searching for a numeric value in an array of `Enum` values [Maxim Khrisanfov](https://github.com/ClickHouse/ClickHouse/pull/2699). * Support for adding arbitrary message separators when reading from `Kafka` [Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2701). -### Improvements: +#### Improvements: * The `ALTER TABLE t DELETE WHERE` query does not rewrite data parts that were not affected by the WHERE condition [#2694](https://github.com/ClickHouse/ClickHouse/pull/2694). * The `use_minimalistic_checksums_in_zookeeper` option for `ReplicatedMergeTree` tables is enabled by default. This setting was added in version 1.1.54378, 2018-04-16. Versions that are older than 1.1.54378 can no longer be installed. * Support for running `KILL` and `OPTIMIZE` queries that specify `ON CLUSTER` [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2689). -### Bug fixes: +#### Bug fixes: * Fixed the error `Column ... is not under an aggregate function and not in GROUP BY` for aggregation with an IN expression. This bug appeared in version 18.1.0. ([bbdd780b](https://github.com/ClickHouse/ClickHouse/commit/bbdd780be0be06a0f336775941cdd536878dd2c2)) * Fixed a bug in the `windowFunnel aggregate function` [Winter Zhang](https://github.com/ClickHouse/ClickHouse/pull/2735). * Fixed a bug in the `anyHeavy` aggregate function ([a2101df2](https://github.com/ClickHouse/ClickHouse/commit/a2101df25a6a0fba99aa71f8793d762af2b801ee)) * Fixed server crash when using the `countArray()` aggregate function. -### Backward incompatible changes: +#### Backward incompatible changes: * Parameters for `Kafka` engine was changed from `Kafka(kafka_broker_list, kafka_topic_list, kafka_group_name, kafka_format[, kafka_schema, kafka_num_consumers])` to `Kafka(kafka_broker_list, kafka_topic_list, kafka_group_name, kafka_format[, kafka_row_delimiter, kafka_schema, kafka_num_consumers])`. If your tables use `kafka_schema` or `kafka_num_consumers` parameters, you have to manually edit the metadata files `path/metadata/database/table.sql` and add `kafka_row_delimiter` parameter with `''` value. -## ClickHouse release 18.1.0, 2018-07-23 +## ClickHouse release 18.1 -### New features: +### ClickHouse release 18.1.0, 2018-07-23 + +#### New features: * Support for the `ALTER TABLE t DELETE WHERE` query for non-replicated MergeTree tables ([#2634](https://github.com/ClickHouse/ClickHouse/pull/2634)). * Support for arbitrary types for the `uniq*` family of aggregate functions ([#2010](https://github.com/ClickHouse/ClickHouse/issues/2010)). @@ -2449,13 +2483,13 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Added the `arrayDistinct` function ([#2670](https://github.com/ClickHouse/ClickHouse/pull/2670)). * The SummingMergeTree engine can now work with AggregateFunction type columns ([Constantin S. Pan](https://github.com/ClickHouse/ClickHouse/pull/2566)). -### Improvements: +#### Improvements: * Changed the numbering scheme for release versions. Now the first part contains the year of release (A.D., Moscow timezone, minus 2000), the second part contains the number for major changes (increases for most releases), and the third part is the patch version. Releases are still backward compatible, unless otherwise stated in the changelog. * Faster conversions of floating-point numbers to a string ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2664)). * If some rows were skipped during an insert due to parsing errors (this is possible with the `input_allow_errors_num` and `input_allow_errors_ratio` settings enabled), the number of skipped rows is now written to the server log ([Leonardo Cecchi](https://github.com/ClickHouse/ClickHouse/pull/2669)). -### Bug fixes: +#### Bug fixes: * Fixed the TRUNCATE command for temporary tables ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2624)). * Fixed a rare deadlock in the ZooKeeper client library that occurred when there was a network error while reading the response ([c315200](https://github.com/ClickHouse/ClickHouse/commit/c315200e64b87e44bdf740707fc857d1fdf7e947)). @@ -2466,18 +2500,20 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Fixed incompatibility between servers with different versions in distributed queries that use a `CAST` function that isn't in uppercase letters ([fe8c4d6](https://github.com/ClickHouse/ClickHouse/commit/fe8c4d64e434cacd4ceef34faa9005129f2190a5)). * Added missing quoting of identifiers for queries to an external DBMS ([#2635](https://github.com/ClickHouse/ClickHouse/issues/2635)). -### Backward incompatible changes: +#### Backward incompatible changes: * Converting a string containing the number zero to DateTime does not work. Example: `SELECT toDateTime('0')`. This is also the reason that `DateTime DEFAULT '0'` does not work in tables, as well as `0` in dictionaries. Solution: replace `0` with `0000-00-00 00:00:00`. -## ClickHouse release 1.1.54394, 2018-07-12 +## ClickHouse release 1.1 -### New features: +### ClickHouse release 1.1.54394, 2018-07-12 + +#### New features: * Added the `histogram` aggregate function ([Mikhail Surin](https://github.com/ClickHouse/ClickHouse/pull/2521)). * Now `OPTIMIZE TABLE ... FINAL` can be used without specifying partitions for `ReplicatedMergeTree` ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2600)). -### Bug fixes: +#### Bug fixes: * Fixed a problem with a very small timeout for sockets (one second) for reading and writing when sending and downloading replicated data, which made it impossible to download larger parts if there is a load on the network or disk (it resulted in cyclical attempts to download parts). This error occurred in version 1.1.54388. * Fixed issues when using chroot in ZooKeeper if you inserted duplicate data blocks in the table. @@ -2486,15 +2522,15 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Fixed how an empty `TinyLog` table works after inserting an empty data block ([#2563](https://github.com/ClickHouse/ClickHouse/issues/2563)). * The `system.zookeeper` table works if the value of the node in ZooKeeper is NULL. -## ClickHouse release 1.1.54390, 2018-07-06 +### ClickHouse release 1.1.54390, 2018-07-06 -### New features: +#### New features: * Queries can be sent in `multipart/form-data` format (in the `query` field), which is useful if external data is also sent for query processing ([Olga Hvostikova](https://github.com/ClickHouse/ClickHouse/pull/2490)). * Added the ability to enable or disable processing single or double quotes when reading data in CSV format. You can configure this in the `format_csv_allow_single_quotes` and `format_csv_allow_double_quotes` settings ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2574)). * Now `OPTIMIZE TABLE ... FINAL` can be used without specifying the partition for non-replicated variants of `MergeTree` ([Amos Bird](https://github.com/ClickHouse/ClickHouse/pull/2599)). -### Improvements: +#### Improvements: * Improved performance, reduced memory consumption, and correct memory consumption tracking with use of the IN operator when a table index could be used ([#2584](https://github.com/ClickHouse/ClickHouse/pull/2584)). * Removed redundant checking of checksums when adding a data part. This is important when there are a large number of replicas, because in these cases the total number of checks was equal to N^2. @@ -2504,7 +2540,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Faster selection of data parts for merging in `ReplicatedMergeTree` tables. Faster recovery of the ZooKeeper session ([#2597](https://github.com/ClickHouse/ClickHouse/pull/2597)). * The `format_version.txt` file for `MergeTree` tables is re-created if it is missing, which makes sense if ClickHouse is launched after copying the directory structure without files ([Ciprian Hacman](https://github.com/ClickHouse/ClickHouse/pull/2593)). -### Bug fixes: +#### Bug fixes: * Fixed a bug when working with ZooKeeper that could make it impossible to recover the session and readonly states of tables before restarting the server. * Fixed a bug when working with ZooKeeper that could result in old nodes not being deleted if the session is interrupted. @@ -2514,13 +2550,13 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Fixed switching to the default database when reconnecting the client ([#2583](https://github.com/ClickHouse/ClickHouse/pull/2583)). * Fixed a bug that occurred when the `use_index_for_in_with_subqueries` setting was disabled. -### Security fix: +#### Security fix: * Sending files is no longer possible when connected to MySQL (`LOAD DATA LOCAL INFILE`). -## ClickHouse release 1.1.54388, 2018-06-28 +### ClickHouse release 1.1.54388, 2018-06-28 -### New features: +#### New features: * Support for the `ALTER TABLE t DELETE WHERE` query for replicated tables. Added the `system.mutations` table to track progress of this type of queries. * Support for the `ALTER TABLE t [REPLACE|ATTACH] PARTITION` query for \*MergeTree tables. @@ -2538,12 +2574,12 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Added the `date_time_input_format` setting. If you switch this setting to `'best_effort'`, DateTime values will be read in a wide range of formats. * Added the `clickhouse-obfuscator` utility for data obfuscation. Usage example: publishing data used in performance tests. -### Experimental features: +#### Experimental features: * Added the ability to calculate `and` arguments only where they are needed ([Anastasia Tsarkova](https://github.com/ClickHouse/ClickHouse/pull/2272)) * JIT compilation to native code is now available for some expressions ([pyos](https://github.com/ClickHouse/ClickHouse/pull/2277)). -### Bug fixes: +#### Bug fixes: * Duplicates no longer appear for a query with `DISTINCT` and `ORDER BY`. * Queries with `ARRAY JOIN` and `arrayFilter` no longer return an incorrect result. @@ -2565,7 +2601,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Fixed SSRF in the remote() table function. * Fixed exit behavior of `clickhouse-client` in multiline mode ([#2510](https://github.com/ClickHouse/ClickHouse/issues/2510)). -### Improvements: +#### Improvements: * Background tasks in replicated tables are now performed in a thread pool instead of in separate threads ([Silviu Caragea](https://github.com/ClickHouse/ClickHouse/pull/1722)). * Improved LZ4 compression performance. @@ -2578,7 +2614,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * When calculating the number of available CPU cores, limits on cgroups are now taken into account ([Atri Sharma](https://github.com/ClickHouse/ClickHouse/pull/2325)). * Added chown for config directories in the systemd config file ([Mikhail Shiryaev](https://github.com/ClickHouse/ClickHouse/pull/2421)). -### Build changes: +#### Build changes: * The gcc8 compiler can be used for builds. * Added the ability to build llvm from submodule. @@ -2589,41 +2625,41 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Added the ability to use the libtinfo library instead of libtermcap ([Georgy Kondratiev](https://github.com/ClickHouse/ClickHouse/pull/2519)). * Fixed a header file conflict in Fedora Rawhide ([#2520](https://github.com/ClickHouse/ClickHouse/issues/2520)). -### Backward incompatible changes: +#### Backward incompatible changes: * Removed escaping in `Vertical` and `Pretty*` formats and deleted the `VerticalRaw` format. * If servers with version 1.1.54388 (or newer) and servers with an older version are used simultaneously in a distributed query and the query has the `cast(x, 'Type')` expression without the `AS` keyword and doesn't have the word `cast` in uppercase, an exception will be thrown with a message like `Not found column cast(0, 'UInt8') in block`. Solution: Update the server on the entire cluster. -## ClickHouse release 1.1.54385, 2018-06-01 +### ClickHouse release 1.1.54385, 2018-06-01 -### Bug fixes: +#### Bug fixes: * Fixed an error that in some cases caused ZooKeeper operations to block. -## ClickHouse release 1.1.54383, 2018-05-22 +### ClickHouse release 1.1.54383, 2018-05-22 -### Bug fixes: +#### Bug fixes: * Fixed a slowdown of replication queue if a table has many replicas. -## ClickHouse release 1.1.54381, 2018-05-14 +### ClickHouse release 1.1.54381, 2018-05-14 -### Bug fixes: +#### Bug fixes: * Fixed a nodes leak in ZooKeeper when ClickHouse loses connection to ZooKeeper server. -## ClickHouse release 1.1.54380, 2018-04-21 +### ClickHouse release 1.1.54380, 2018-04-21 -### New features: +#### New features: * Added the table function `file(path, format, structure)`. An example reading bytes from `/dev/urandom`: `ln -s /dev/urandom /var/lib/clickhouse/user_files/random``clickhouse-client -q "SELECT * FROM file('random', 'RowBinary', 'd UInt8') LIMIT 10"`. -### Improvements: +#### Improvements: * Subqueries can be wrapped in `()` brackets to enhance query readability. For example: `(SELECT 1) UNION ALL (SELECT 1)`. * Simple `SELECT` queries from the `system.processes` table are not included in the `max_concurrent_queries` limit. -### Bug fixes: +#### Bug fixes: * Fixed incorrect behavior of the `IN` operator when select from `MATERIALIZED VIEW`. * Fixed incorrect filtering by partition index in expressions like `partition_key_column IN (...)`. @@ -2632,13 +2668,13 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Fixed freezing of `KILL QUERY`. * Fixed an error in ZooKeeper client library which led to loss of watches, freezing of distributed DDL queue, and slowdowns in the replication queue if a non-empty `chroot` prefix is used in the ZooKeeper configuration. -### Backward incompatible changes: +#### Backward incompatible changes: * Removed support for expressions like `(a, b) IN (SELECT (a, b))` (you can use the equivalent expression `(a, b) IN (SELECT a, b)`). In previous releases, these expressions led to undetermined `WHERE` filtering or caused errors. -## ClickHouse release 1.1.54378, 2018-04-16 +### ClickHouse release 1.1.54378, 2018-04-16 -### New features: +#### New features: * Logging level can be changed without restarting the server. * Added the `SHOW CREATE DATABASE` query. @@ -2652,7 +2688,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Multiple comma-separated `topics` can be specified for the `Kafka` engine (Tobias Adamson) * When a query is stopped by `KILL QUERY` or `replace_running_query`, the client receives the `Query was canceled` exception instead of an incomplete result. -### Improvements: +#### Improvements: * `ALTER TABLE ... DROP/DETACH PARTITION` queries are run at the front of the replication queue. * `SELECT ... FINAL` and `OPTIMIZE ... FINAL` can be used even when the table has a single data part. @@ -2663,7 +2699,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * More robust crash recovery for asynchronous insertion into `Distributed` tables. * The return type of the `countEqual` function changed from `UInt32` to `UInt64` (谢磊). -### Bug fixes: +#### Bug fixes: * Fixed an error with `IN` when the left side of the expression is `Nullable`. * Correct results are now returned when using tuples with `IN` when some of the tuple components are in the table index. @@ -2679,30 +2715,30 @@ The expression must be a chain of equalities joined by the AND operator. Each si * `SummingMergeTree` now works correctly for summation of nested data structures with a composite key. * Fixed the possibility of a race condition when choosing the leader for `ReplicatedMergeTree` tables. -### Build changes: +#### Build changes: * The build supports `ninja` instead of `make` and uses `ninja` by default for building releases. * Renamed packages: `clickhouse-server-base` in `clickhouse-common-static`; `clickhouse-server-common` in `clickhouse-server`; `clickhouse-common-dbg` in `clickhouse-common-static-dbg`. To install, use `clickhouse-server clickhouse-client`. Packages with the old names will still load in the repositories for backward compatibility. -### Backward incompatible changes: +#### Backward incompatible changes: * Removed the special interpretation of an IN expression if an array is specified on the left side. Previously, the expression `arr IN (set)` was interpreted as "at least one `arr` element belongs to the `set`". To get the same behavior in the new version, write `arrayExists(x -> x IN (set), arr)`. * Disabled the incorrect use of the socket option `SO_REUSEPORT`, which was incorrectly enabled by default in the Poco library. Note that on Linux there is no longer any reason to simultaneously specify the addresses `::` and `0.0.0.0` for listen – use just `::`, which allows listening to the connection both over IPv4 and IPv6 (with the default kernel config settings). You can also revert to the behavior from previous versions by specifying `1` in the config. -## ClickHouse release 1.1.54370, 2018-03-16 +### ClickHouse release 1.1.54370, 2018-03-16 -### New features: +#### New features: * Added the `system.macros` table and auto updating of macros when the config file is changed. * Added the `SYSTEM RELOAD CONFIG` query. * Added the `maxIntersections(left_col, right_col)` aggregate function, which returns the maximum number of simultaneously intersecting intervals `[left; right]`. The `maxIntersectionsPosition(left, right)` function returns the beginning of the "maximum" interval. ([Michael Furmur](https://github.com/ClickHouse/ClickHouse/pull/2012)). -### Improvements: +#### Improvements: * When inserting data in a `Replicated` table, fewer requests are made to `ZooKeeper` (and most of the user-level errors have disappeared from the `ZooKeeper` log). * Added the ability to create aliases for data sets. Example: `WITH (1, 2, 3) AS set SELECT number IN set FROM system.numbers LIMIT 10`. -### Bug fixes: +#### Bug fixes: * Fixed the `Illegal PREWHERE` error when reading from Merge tables for `Distributed`tables. * Added fixes that allow you to start clickhouse-server in IPv4-only Docker containers. @@ -2716,9 +2752,9 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Restored the behavior for queries like `SELECT * FROM remote('server2', default.table) WHERE col IN (SELECT col2 FROM default.table)` when the right side of the `IN` should use a remote `default.table` instead of a local one. This behavior was broken in version 1.1.54358. * Removed extraneous error-level logging of `Not found column ... in block`. -## Clickhouse Release 1.1.54362, 2018-03-11 +### Clickhouse Release 1.1.54362, 2018-03-11 -### New features: +#### New features: * Aggregation without `GROUP BY` for an empty set (such as `SELECT count(*) FROM table WHERE 0`) now returns a result with one row with null values for aggregate functions, in compliance with the SQL standard. To restore the old behavior (return an empty result), set `empty_result_for_aggregation_by_empty_set` to 1. * Added type conversion for `UNION ALL`. Different alias names are allowed in `SELECT` positions in `UNION ALL`, in compliance with the SQL standard. @@ -2756,7 +2792,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Added the `odbc_default_field_size` option, which allows you to extend the maximum size of the value loaded from an ODBC source (by default, it is 1024). * The `system.processes` table and `SHOW PROCESSLIST` now have the `is_cancelled` and `peak_memory_usage` columns. -### Improvements: +#### Improvements: * Limits and quotas on the result are no longer applied to intermediate data for `INSERT SELECT` queries or for `SELECT` subqueries. * Fewer false triggers of `force_restore_data` when checking the status of `Replicated` tables when the server starts. @@ -2772,7 +2808,7 @@ The expression must be a chain of equalities joined by the AND operator. Each si * `Enum` values can be used in `min`, `max`, `sum` and some other functions. In these cases, it uses the corresponding numeric values. This feature was previously available but was lost in the release 1.1.54337. * Added `max_expanded_ast_elements` to restrict the size of the AST after recursively expanding aliases. -### Bug fixes: +#### Bug fixes: * Fixed cases when unnecessary columns were removed from subqueries in error, or not removed from subqueries containing `UNION ALL`. * Fixed a bug in merges for `ReplacingMergeTree` tables. @@ -2800,19 +2836,19 @@ The expression must be a chain of equalities joined by the AND operator. Each si * Prohibited the use of queries with `UNION ALL` in a `MATERIALIZED VIEW`. * Fixed an error during initialization of the `part_log` system table when the server starts (by default, `part_log` is disabled). -### Backward incompatible changes: +#### Backward incompatible changes: * Removed the `distributed_ddl_allow_replicated_alter` option. This behavior is enabled by default. * Removed the `strict_insert_defaults` setting. If you were using this functionality, write to `clickhouse-feedback@yandex-team.com`. * Removed the `UnsortedMergeTree` engine. -## Clickhouse Release 1.1.54343, 2018-02-05 +### Clickhouse Release 1.1.54343, 2018-02-05 * Added macros support for defining cluster names in distributed DDL queries and constructors of Distributed tables: `CREATE TABLE distr ON CLUSTER '{cluster}' (...) ENGINE = Distributed('{cluster}', 'db', 'table')`. * Now queries like `SELECT ... FROM table WHERE expr IN (subquery)` are processed using the `table` index. * Improved processing of duplicates when inserting to Replicated tables, so they no longer slow down execution of the replication queue. -## Clickhouse Release 1.1.54342, 2018-01-22 +### Clickhouse Release 1.1.54342, 2018-01-22 This release contains bug fixes for the previous release 1.1.54337: @@ -2824,9 +2860,9 @@ This release contains bug fixes for the previous release 1.1.54337: * Buffer tables now work correctly when MATERIALIZED columns are present in the destination table (by zhang2014). * Fixed a bug in implementation of NULL. -## Clickhouse Release 1.1.54337, 2018-01-18 +### Clickhouse Release 1.1.54337, 2018-01-18 -### New features: +#### New features: * Added support for storage of multi-dimensional arrays and tuples (`Tuple` data type) in tables. * Support for table functions for `DESCRIBE` and `INSERT` queries. Added support for subqueries in `DESCRIBE`. Examples: `DESC TABLE remote('host', default.hits)`; `DESC TABLE (SELECT 1)`; `INSERT INTO TABLE FUNCTION remote('host', default.hits)`. Support for `INSERT INTO TABLE` in addition to `INSERT INTO`. @@ -2857,7 +2893,7 @@ This release contains bug fixes for the previous release 1.1.54337: * Added the `--silent` option for the `clickhouse-local` tool. It suppresses printing query execution info in stderr. * Added support for reading values of type `Date` from text in a format where the month and/or day of the month is specified using a single digit instead of two digits (Amos Bird). -### Performance optimizations: +#### Performance optimizations: * Improved performance of aggregate functions `min`, `max`, `any`, `anyLast`, `anyHeavy`, `argMin`, `argMax` from string arguments. * Improved performance of the functions `isInfinite`, `isFinite`, `isNaN`, `roundToExp2`. @@ -2866,7 +2902,7 @@ This release contains bug fixes for the previous release 1.1.54337: * Lowered memory usage for `JOIN` in the case when the left and right parts have columns with identical names that are not contained in `USING` . * Improved performance of aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr` by reducing computational stability. The old functions are available under the names `varSampStable`, `varPopStable`, `stddevSampStable`, `stddevPopStable`, `covarSampStable`, `covarPopStable`, `corrStable`. -### Bug fixes: +#### Bug fixes: * Fixed data deduplication after running a `DROP` or `DETACH PARTITION` query. In the previous version, dropping a partition and inserting the same data again was not working because inserted blocks were considered duplicates. * Fixed a bug that could lead to incorrect interpretation of the `WHERE` clause for ` CREATE MATERIALIZED VIEW` queries with `POPULATE` . @@ -2905,7 +2941,7 @@ This release contains bug fixes for the previous release 1.1.54337: * Fixed the ` SYSTEM DROP DNS CACHE` query: the cache was flushed but addresses of cluster nodes were not updated. * Fixed the behavior of ` MATERIALIZED VIEW` after executing ` DETACH TABLE` for the table under the view (Marek Vavruša). -### Build improvements: +#### Build improvements: * The `pbuilder` tool is used for builds. The build process is almost completely independent of the build host environment. * A single build is used for different OS versions. Packages and binaries have been made compatible with a wide range of Linux systems. @@ -2919,7 +2955,7 @@ This release contains bug fixes for the previous release 1.1.54337: * Removed usage of GNU extensions from the code. Enabled the `-Wextra` option. When building with `clang` the default is `libc++` instead of `libstdc++`. * Extracted `clickhouse_parsers` and `clickhouse_common_io` libraries to speed up builds of various tools. -### Backward incompatible changes: +#### Backward incompatible changes: * The format for marks in `Log` type tables that contain `Nullable` columns was changed in a backward incompatible way. If you have these tables, you should convert them to the `TinyLog` type before starting up the new server version. To do this, replace `ENGINE = Log` with `ENGINE = TinyLog` in the corresponding `.sql` file in the `metadata` directory. If your table doesn't have `Nullable` columns or if the type of your table is not `Log`, then you don't need to do anything. * Removed the `experimental_allow_extended_storage_definition_syntax` setting. Now this feature is enabled by default. @@ -2930,18 +2966,18 @@ This release contains bug fixes for the previous release 1.1.54337: * In previous server versions there was an undocumented feature: if an aggregate function depends on parameters, you can still specify it without parameters in the AggregateFunction data type. Example: `AggregateFunction(quantiles, UInt64)` instead of `AggregateFunction(quantiles(0.5, 0.9), UInt64)`. This feature was lost. Although it was undocumented, we plan to support it again in future releases. * Enum data types cannot be used in min/max aggregate functions. This ability will be returned in the next release. -### Please note when upgrading: +#### Please note when upgrading: * When doing a rolling update on a cluster, at the point when some of the replicas are running the old version of ClickHouse and some are running the new version, replication is temporarily stopped and the message ` unknown parameter 'shard'` appears in the log. Replication will continue after all replicas of the cluster are updated. * If different versions of ClickHouse are running on the cluster servers, it is possible that distributed queries using the following functions will have incorrect results: `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr`. You should update all cluster nodes. -## ClickHouse release 1.1.54327, 2017-12-21 +### ClickHouse release 1.1.54327, 2017-12-21 This release contains bug fixes for the previous release 1.1.54318: * Fixed bug with possible race condition in replication that could lead to data loss. This issue affects versions 1.1.54310 and 1.1.54318. If you use one of these versions with Replicated tables, the update is strongly recommended. This issue shows in logs in Warning messages like ` Part ... from own log doesn't exist.` The issue is relevant even if you don't see these messages in logs. -## ClickHouse release 1.1.54318, 2017-11-30 +### ClickHouse release 1.1.54318, 2017-11-30 This release contains bug fixes for the previous release 1.1.54310: @@ -2951,9 +2987,9 @@ This release contains bug fixes for the previous release 1.1.54310: * Fixed an issue that was causing the replication queue to stop running * Fixed rotation and archiving of server logs -## ClickHouse release 1.1.54310, 2017-11-01 +### ClickHouse release 1.1.54310, 2017-11-01 -### New features: +#### New features: * Custom partitioning key for the MergeTree family of table engines. * [Kafka](https://clickhouse.yandex/docs/en/operations/table_engines/kafka/) table engine. @@ -2970,13 +3006,13 @@ This release contains bug fixes for the previous release 1.1.54310: * Added support for the Cap'n Proto input format. * You can now customize compression level when using the zstd algorithm. -### Backward incompatible changes: +#### Backward incompatible changes: * Creation of temporary tables with an engine other than Memory is not allowed. * Explicit creation of tables with the View or MaterializedView engine is not allowed. * During table creation, a new check verifies that the sampling key expression is included in the primary key. -### Bug fixes: +#### Bug fixes: * Fixed hangups when synchronously inserting into a Distributed table. * Fixed nonatomic adding and removing of parts in Replicated tables. @@ -2987,17 +3023,17 @@ This release contains bug fixes for the previous release 1.1.54310: * Fixed hangups when the disk volume containing server logs is full. * Fixed an overflow in the toRelativeWeekNum function for the first week of the Unix epoch. -### Build improvements: +#### Build improvements: * Several third-party libraries (notably Poco) were updated and converted to git submodules. -## ClickHouse release 1.1.54304, 2017-10-19 +### ClickHouse release 1.1.54304, 2017-10-19 -### New features: +#### New features: * TLS support in the native protocol (to enable, set `tcp_ssl_port` in `config.xml` ). -### Bug fixes: +#### Bug fixes: * `ALTER` for replicated tables now tries to start running as soon as possible. * Fixed crashing when reading data with the setting `preferred_block_size_bytes=0.` @@ -3011,9 +3047,9 @@ This release contains bug fixes for the previous release 1.1.54310: * Users are updated correctly with invalid `users.xml` * Correct handling when an executable dictionary returns a non-zero response code. -## ClickHouse release 1.1.54292, 2017-09-20 +### ClickHouse release 1.1.54292, 2017-09-20 -### New features: +#### New features: * Added the `pointInPolygon` function for working with coordinates on a coordinate plane. * Added the `sumMap` aggregate function for calculating the sum of arrays, similar to `SummingMergeTree`. @@ -3021,7 +3057,7 @@ This release contains bug fixes for the previous release 1.1.54310: * The ClickHouse executable file is now less dependent on the libc version. The same ClickHouse executable file can run on a wide variety of Linux systems. There is still a dependency when using compiled queries (with the setting ` compile = 1` , which is not used by default). * Reduced the time needed for dynamic compilation of queries. -### Bug fixes: +#### Bug fixes: * Fixed an error that sometimes produced ` part ... intersects previous part` messages and weakened replica consistency. * Fixed an error that caused the server to lock up if ZooKeeper was unavailable during shutdown. @@ -3030,9 +3066,9 @@ This release contains bug fixes for the previous release 1.1.54310: * Fixed an error in the concat function that occurred if the first column in a block has the Array type. * Progress is now displayed correctly in the system.merges table. -## ClickHouse release 1.1.54289, 2017-09-13 +### ClickHouse release 1.1.54289, 2017-09-13 -### New features: +#### New features: * `SYSTEM` queries for server administration: `SYSTEM RELOAD DICTIONARY`, `SYSTEM RELOAD DICTIONARIES`, `SYSTEM DROP DNS CACHE`, `SYSTEM SHUTDOWN`, `SYSTEM KILL`. * Added functions for working with arrays: `concat`, `arraySlice`, `arrayPushBack`, `arrayPushFront`, `arrayPopBack`, `arrayPopFront`. @@ -3048,7 +3084,7 @@ This release contains bug fixes for the previous release 1.1.54310: * Option to set `umask` in the config file. * Improved performance for queries with `DISTINCT` . -### Bug fixes: +#### Bug fixes: * Improved the process for deleting old nodes in ZooKeeper. Previously, old nodes sometimes didn't get deleted if there were very frequent inserts, which caused the server to be slow to shut down, among other things. * Fixed randomization when choosing hosts for the connection to ZooKeeper. @@ -3062,21 +3098,21 @@ This release contains bug fixes for the previous release 1.1.54310: * Resolved the appearance of zombie processes when using a dictionary with an `executable` source. * Fixed segfault for the HEAD query. -### Improved workflow for developing and assembling ClickHouse: +#### Improved workflow for developing and assembling ClickHouse: * You can use `pbuilder` to build ClickHouse. * You can use `libc++` instead of `libstdc++` for builds on Linux. * Added instructions for using static code analysis tools: `Coverage`, `clang-tidy`, `cppcheck`. -### Please note when upgrading: +#### Please note when upgrading: * There is now a higher default value for the MergeTree setting `max_bytes_to_merge_at_max_space_in_pool` (the maximum total size of data parts to merge, in bytes): it has increased from 100 GiB to 150 GiB. This might result in large merges running after the server upgrade, which could cause an increased load on the disk subsystem. If the free space available on the server is less than twice the total amount of the merges that are running, this will cause all other merges to stop running, including merges of small data parts. As a result, INSERT queries will fail with the message "Merges are processing significantly slower than inserts." Use the ` SELECT * FROM system.merges` query to monitor the situation. You can also check the `DiskSpaceReservedForMerge` metric in the `system.metrics` table, or in Graphite. You don't need to do anything to fix this, since the issue will resolve itself once the large merges finish. If you find this unacceptable, you can restore the previous value for the `max_bytes_to_merge_at_max_space_in_pool` setting. To do this, go to the section in config.xml, set ```107374182400` and restart the server. -## ClickHouse release 1.1.54284, 2017-08-29 +### ClickHouse release 1.1.54284, 2017-08-29 * This is a bugfix release for the previous 1.1.54282 release. It fixes leaks in the parts directory in ZooKeeper. -## ClickHouse release 1.1.54282, 2017-08-23 +### ClickHouse release 1.1.54282, 2017-08-23 This release contains bug fixes for the previous release 1.1.54276: @@ -3084,9 +3120,9 @@ This release contains bug fixes for the previous release 1.1.54276: * Fixed parsing when inserting in RowBinary format if input data starts with';'. * Errors during runtime compilation of certain aggregate functions (e.g. `groupArray()`). -## Clickhouse Release 1.1.54276, 2017-08-16 +### Clickhouse Release 1.1.54276, 2017-08-16 -### New features: +#### New features: * Added an optional WITH section for a SELECT query. Example query: `WITH 1+1 AS a SELECT a, a*a` * INSERT can be performed synchronously in a Distributed table: OK is returned only after all the data is saved on all the shards. This is activated by the setting insert_distributed_sync=1. @@ -3097,7 +3133,7 @@ This release contains bug fixes for the previous release 1.1.54276: * Added support for non-constant arguments and negative offsets in the function `substring(str, pos, len).` * Added the max_size parameter for the `groupArray(max_size)(column)` aggregate function, and optimized its performance. -### Main changes: +#### Main changes: * Security improvements: all server files are created with 0640 permissions (can be changed via config parameter). * Improved error messages for queries with invalid syntax. @@ -3105,11 +3141,11 @@ This release contains bug fixes for the previous release 1.1.54276: * Significantly increased the performance of data merges for the ReplacingMergeTree engine. * Improved performance for asynchronous inserts from a Distributed table by combining multiple source inserts. To enable this functionality, use the setting distributed_directory_monitor_batch_inserts=1. -### Backward incompatible changes: +#### Backward incompatible changes: * Changed the binary format of aggregate states of `groupArray(array_column)` functions for arrays. -### Complete list of changes: +#### Complete list of changes: * Added the `output_format_json_quote_denormals` setting, which enables outputting nan and inf values in JSON format. * Optimized stream allocation when reading from a Distributed table. @@ -3128,7 +3164,7 @@ This release contains bug fixes for the previous release 1.1.54276: * It is possible to connect to MySQL through a socket in the file system. * The system.parts table has a new column with information about the size of marks, in bytes. -### Bug fixes: +#### Bug fixes: * Distributed tables using a Merge table now work correctly for a SELECT query with a condition on the `_table` field. * Fixed a rare race condition in ReplicatedMergeTree when checking data parts. @@ -3152,15 +3188,15 @@ This release contains bug fixes for the previous release 1.1.54276: * Fixed the "Cannot mremap" error when using arrays in IN and JOIN clauses with more than 2 billion elements. * Fixed the failover for dictionaries with MySQL as the source. -### Improved workflow for developing and assembling ClickHouse: +#### Improved workflow for developing and assembling ClickHouse: * Builds can be assembled in Arcadia. * You can use gcc 7 to compile ClickHouse. * Parallel builds using ccache+distcc are faster now. -## ClickHouse release 1.1.54245, 2017-07-04 +### ClickHouse release 1.1.54245, 2017-07-04 -### New features: +#### New features: * Distributed DDL (for example, `CREATE TABLE ON CLUSTER`) * The replicated query `ALTER TABLE CLEAR COLUMN IN PARTITION.` @@ -3172,16 +3208,16 @@ This release contains bug fixes for the previous release 1.1.54276: * Sessions in the HTTP interface. * The OPTIMIZE query for a Replicated table can can run not only on the leader. -### Backward incompatible changes: +#### Backward incompatible changes: * Removed SET GLOBAL. -### Minor changes: +#### Minor changes: * Now after an alert is triggered, the log prints the full stack trace. * Relaxed the verification of the number of damaged/extra data parts at startup (there were too many false positives). -### Bug fixes: +#### Bug fixes: * Fixed a bad connection "sticking" when inserting into a Distributed table. * GLOBAL IN now works for a query from a Merge table that looks at a Distributed table. diff --git a/CMakeLists.txt b/CMakeLists.txt index c6ae23c0955..cd32288ec9c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -95,6 +95,8 @@ if (CMAKE_GENERATOR STREQUAL "Ninja") set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fdiagnostics-color=always") endif () +include (cmake/add_warning.cmake) + if (NOT MSVC) set (COMMON_WARNING_FLAGS "${COMMON_WARNING_FLAGS} -Wall") # -Werror is also added inside directories with our own code. endif () @@ -176,7 +178,9 @@ if (ARCH_NATIVE) set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native") endif () -set (CMAKE_CXX_STANDARD 17) +# cmake < 3.12 doesn't supoprt 20. We'll set CMAKE_CXX_FLAGS for now +# set (CMAKE_CXX_STANDARD 20) +set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++2a") set (CMAKE_CXX_EXTENSIONS 0) # https://cmake.org/cmake/help/latest/prop_tgt/CXX_EXTENSIONS.html#prop_tgt:CXX_EXTENSIONS set (CMAKE_CXX_STANDARD_REQUIRED ON) @@ -208,7 +212,7 @@ set (CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -O0 -g3 -ggdb3 if (COMPILER_CLANG) # Exception unwinding doesn't work in clang release build without this option - # TODO investigate if contrib/libcxxabi is out of date + # TODO investigate that set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-omit-frame-pointer") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-omit-frame-pointer") endif () @@ -222,8 +226,8 @@ else () set(NOT_UNBUNDLED 1) endif () -# Using system libs can cause lot of warnings in includes (on macro expansion). -if (UNBUNDLED OR NOT (OS_LINUX OR APPLE) OR ARCH_32) +# Using system libs can cause a lot of warnings in includes (on macro expansion). +if (UNBUNDLED OR NOT (OS_LINUX OR OS_DARWIN) OR ARCH_32) option (NO_WERROR "Disable -Werror compiler option" ON) endif () @@ -248,8 +252,16 @@ endif () string (TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC) set (CMAKE_POSTFIX_VARIABLE "CMAKE_${CMAKE_BUILD_TYPE_UC}_POSTFIX") -if (NOT MAKE_STATIC_LIBRARIES) - set(CMAKE_POSITION_INDEPENDENT_CODE ON) +if (MAKE_STATIC_LIBRARIES) + set (CMAKE_POSITION_INDEPENDENT_CODE OFF) + if (OS_LINUX) + # Slightly more efficient code can be generated + set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -fno-pie") + set (CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} -fno-pie") + set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-no-pie") + endif () +else () + set (CMAKE_POSITION_INDEPENDENT_CODE ON) endif () # Using "include-what-you-use" tool. diff --git a/README.md b/README.md index a545c91886f..21498a22912 100644 --- a/README.md +++ b/README.md @@ -11,3 +11,7 @@ ClickHouse is an open-source column-oriented database management system that all * [Blog](https://clickhouse.yandex/blog/en/) contains various ClickHouse-related articles, as well as announces and reports about events. * [Contacts](https://clickhouse.yandex/#contacts) can help to get your questions answered if there are any. * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person. + +## Upcoming Events + +* [ClickHouse Meetup in San Francisco](https://www.eventbrite.com/e/clickhouse-february-meetup-registration-88496227599) on February 5. diff --git a/cmake/add_warning.cmake b/cmake/add_warning.cmake new file mode 100644 index 00000000000..a7a5f0f035e --- /dev/null +++ b/cmake/add_warning.cmake @@ -0,0 +1,18 @@ +include (CheckCXXSourceCompiles) + +# Try to add -Wflag if compiler supports it +macro (add_warning flag) + string (REPLACE "-" "_" underscored_flag ${flag}) + string (REPLACE "+" "x" underscored_flag ${underscored_flag}) + check_cxx_compiler_flag("-W${flag}" SUPPORTS_FLAG_${underscored_flag}) + if (SUPPORTS_FLAG_${underscored_flag}) + set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -W${flag}") + else () + message (WARNING "Flag -W${flag} is unsupported") + endif () +endmacro () + +# Try to add -Wno flag if compiler supports it +macro (no_warning flag) + add_warning(no-${flag}) +endmacro () diff --git a/cmake/darwin/default_libs.cmake b/cmake/darwin/default_libs.cmake index 679fef8808a..7b57e63f4ee 100644 --- a/cmake/darwin/default_libs.cmake +++ b/cmake/darwin/default_libs.cmake @@ -13,11 +13,12 @@ set(CMAKE_C_STANDARD_LIBRARIES ${DEFAULT_LIBS}) # Minimal supported SDK version -set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mmacosx-version-min=10.14") -set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mmacosx-version-min=10.14") +set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mmacosx-version-min=10.15") +set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mmacosx-version-min=10.15") +set (CMAKE_ASM_FLAGS "${CMAKE_ASM_FLAGS} -mmacosx-version-min=10.15") -set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -mmacosx-version-min=10.14") -set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -mmacosx-version-min=10.14") +set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -mmacosx-version-min=10.15") +set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -mmacosx-version-min=10.15") # Global libraries diff --git a/cmake/darwin/toolchain-x86_64.cmake b/cmake/darwin/toolchain-x86_64.cmake index 9128311e3bb..0be81dfa753 100644 --- a/cmake/darwin/toolchain-x86_64.cmake +++ b/cmake/darwin/toolchain-x86_64.cmake @@ -2,6 +2,7 @@ set (CMAKE_SYSTEM_NAME "Darwin") set (CMAKE_SYSTEM_PROCESSOR "x86_64") set (CMAKE_C_COMPILER_TARGET "x86_64-apple-darwin") set (CMAKE_CXX_COMPILER_TARGET "x86_64-apple-darwin") +set (CMAKE_ASM_COMPILER_TARGET "x86_64-apple-darwin") set (CMAKE_OSX_SYSROOT "${CMAKE_CURRENT_LIST_DIR}/../toolchain/darwin-x86_64") set (CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) # disable linkage check - it doesn't work in CMake diff --git a/cmake/find/icu.cmake b/cmake/find/icu.cmake index 8ebe2f9befd..7beb25626b9 100644 --- a/cmake/find/icu.cmake +++ b/cmake/find/icu.cmake @@ -1,4 +1,8 @@ -option(ENABLE_ICU "Enable ICU" ${ENABLE_LIBRARIES}) +if (OS_LINUX) + option(ENABLE_ICU "Enable ICU" ${ENABLE_LIBRARIES}) +else () + option(ENABLE_ICU "Enable ICU" 0) +endif () if (ENABLE_ICU) diff --git a/cmake/target.cmake b/cmake/target.cmake index 3c6aa225af9..9233af62dcf 100644 --- a/cmake/target.cmake +++ b/cmake/target.cmake @@ -13,7 +13,6 @@ if (CMAKE_CROSSCOMPILING) if (OS_DARWIN) # FIXME: broken dependencies set (USE_SNAPPY OFF CACHE INTERNAL "") - set (ENABLE_SSL OFF CACHE INTERNAL "") set (ENABLE_PROTOBUF OFF CACHE INTERNAL "") set (ENABLE_PARQUET OFF CACHE INTERNAL "") set (ENABLE_READLINE OFF CACHE INTERNAL "") diff --git a/contrib/CMakeLists.txt b/contrib/CMakeLists.txt index 4cc74972d8c..53ad9a0c138 100644 --- a/contrib/CMakeLists.txt +++ b/contrib/CMakeLists.txt @@ -2,10 +2,10 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w") - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w -std=c++1z") + set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w") elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang") set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w") - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w -std=c++1z") + set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w") endif () set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL 1) @@ -32,6 +32,8 @@ if (USE_INTERNAL_DOUBLE_CONVERSION_LIBRARY) add_subdirectory (double-conversion-cmake) endif () +add_subdirectory (ryu-cmake) + if (USE_INTERNAL_CITYHASH_LIBRARY) add_subdirectory (cityhash102) endif () diff --git a/contrib/arrow-cmake/CMakeLists.txt b/contrib/arrow-cmake/CMakeLists.txt index 335106cc7ca..1f09bba8d31 100644 --- a/contrib/arrow-cmake/CMakeLists.txt +++ b/contrib/arrow-cmake/CMakeLists.txt @@ -1,5 +1,7 @@ include(ExternalProject) +set (CMAKE_CXX_STANDARD 17) + # === thrift set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/thrift/lib/cpp) diff --git a/contrib/capnproto-cmake/CMakeLists.txt b/contrib/capnproto-cmake/CMakeLists.txt index c54b4e8eae5..8bdac0beec0 100644 --- a/contrib/capnproto-cmake/CMakeLists.txt +++ b/contrib/capnproto-cmake/CMakeLists.txt @@ -1,5 +1,7 @@ set (CAPNPROTO_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/capnproto/c++/src) +set (CMAKE_CXX_STANDARD 17) + set (KJ_SRCS ${CAPNPROTO_SOURCE_DIR}/kj/array.c++ ${CAPNPROTO_SOURCE_DIR}/kj/common.c++ diff --git a/contrib/googletest b/contrib/googletest index d175c8bf823..703bd9caab5 160000 --- a/contrib/googletest +++ b/contrib/googletest @@ -1 +1 @@ -Subproject commit d175c8bf823e709d570772b038757fadf63bc632 +Subproject commit 703bd9caab50b139428cea1aaff9974ebee5742e diff --git a/contrib/icu-cmake/CMakeLists.txt b/contrib/icu-cmake/CMakeLists.txt index 64e82366076..b4903c141fb 100644 --- a/contrib/icu-cmake/CMakeLists.txt +++ b/contrib/icu-cmake/CMakeLists.txt @@ -1,6 +1,8 @@ set(ICU_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/icu/icu4c/source) set(ICUDATA_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/icudata/) +set (CMAKE_CXX_STANDARD 17) + # These lists of sources were generated from build log of the original ICU build system (configure + make). set(ICUUC_SOURCES diff --git a/contrib/libc-headers b/contrib/libc-headers index cd82fd9d8ee..9676d2645a7 160000 --- a/contrib/libc-headers +++ b/contrib/libc-headers @@ -1 +1 @@ -Subproject commit cd82fd9d8eefe50a47a0adf7c617c3ea7d558d11 +Subproject commit 9676d2645a713e679dc981ffd84dee99fcd68b8e diff --git a/contrib/libcxx b/contrib/libcxx index f7c63235238..a8c45330087 160000 --- a/contrib/libcxx +++ b/contrib/libcxx @@ -1 +1 @@ -Subproject commit f7c63235238a71b7e0563fab8c7c5ec1b54831f6 +Subproject commit a8c453300879d0bf255f9d5959d42e2c8aac1bfb diff --git a/contrib/libcxx-cmake/CMakeLists.txt b/contrib/libcxx-cmake/CMakeLists.txt index ee5fe625079..a56efae9bf2 100644 --- a/contrib/libcxx-cmake/CMakeLists.txt +++ b/contrib/libcxx-cmake/CMakeLists.txt @@ -47,15 +47,19 @@ add_library(cxx ${SRCS}) target_include_directories(cxx SYSTEM BEFORE PUBLIC $) target_compile_definitions(cxx PRIVATE -D_LIBCPP_BUILDING_LIBRARY -DLIBCXX_BUILDING_LIBCXXABI) +# Enable capturing stack traces for all exceptions. +if (USE_UNWIND) + target_compile_definitions(cxx PUBLIC -DSTD_EXCEPTION_HAS_STACK_TRACE=1) +endif () + target_compile_options(cxx PUBLIC $<$:-nostdinc++>) -check_cxx_compiler_flag(-Wreserved-id-macro HAVE_WARNING_RESERVED_ID_MACRO) -if (HAVE_WARNING_RESERVED_ID_MACRO) + +if (SUPPORTS_FLAG_no_reserved_id_macro) target_compile_options(cxx PUBLIC -Wno-reserved-id-macro) endif () -check_cxx_compiler_flag(-Wctad-maybe-unsupported HAVE_WARNING_CTAD_MAYBE_UNSUPPORTED) -if (HAVE_WARNING_CTAD_MAYBE_UNSUPPORTED) +if (SUPPORTS_FLAG_no_ctad_maybe_unsupported) target_compile_options(cxx PUBLIC -Wno-ctad-maybe-unsupported) endif () diff --git a/contrib/libcxxabi-cmake/CMakeLists.txt b/contrib/libcxxabi-cmake/CMakeLists.txt index 68bb5606689..daeb209603b 100644 --- a/contrib/libcxxabi-cmake/CMakeLists.txt +++ b/contrib/libcxxabi-cmake/CMakeLists.txt @@ -32,6 +32,11 @@ target_compile_definitions(cxxabi PRIVATE -D_LIBCPP_BUILDING_LIBRARY) target_compile_options(cxxabi PRIVATE -nostdinc++ -fno-sanitize=undefined -Wno-macro-redefined) # If we don't disable UBSan, infinite recursion happens in dynamic_cast. target_link_libraries(cxxabi PUBLIC ${EXCEPTION_HANDLING_LIBRARY}) +# Enable capturing stack traces for all exceptions. +if (USE_UNWIND) + target_compile_definitions(cxxabi PUBLIC -DSTD_EXCEPTION_HAS_STACK_TRACE=1) +endif () + install( TARGETS cxxabi EXPORT global diff --git a/contrib/libhdfs3-cmake/CMake/Platform.cmake b/contrib/libhdfs3-cmake/CMake/Platform.cmake index d9bc760ee3f..fec1d974519 100644 --- a/contrib/libhdfs3-cmake/CMake/Platform.cmake +++ b/contrib/libhdfs3-cmake/CMake/Platform.cmake @@ -7,10 +7,14 @@ ELSE(CMAKE_SYSTEM_NAME STREQUAL "Linux") ENDIF(CMAKE_SYSTEM_NAME STREQUAL "Linux") IF(CMAKE_COMPILER_IS_GNUCXX) - EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_COMPILER_VERSION) + EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpfullversion OUTPUT_VARIABLE GCC_COMPILER_VERSION) IF (NOT GCC_COMPILER_VERSION) - MESSAGE(FATAL_ERROR "Cannot get gcc version") + EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_COMPILER_VERSION) + + IF (NOT GCC_COMPILER_VERSION) + MESSAGE(FATAL_ERROR "Cannot get gcc version") + ENDIF (NOT GCC_COMPILER_VERSION) ENDIF (NOT GCC_COMPILER_VERSION) STRING(REGEX MATCHALL "[0-9]+" GCC_COMPILER_VERSION ${GCC_COMPILER_VERSION}) diff --git a/contrib/openssl-cmake/CMakeLists.txt b/contrib/openssl-cmake/CMakeLists.txt index c2e74dc0023..8dc0c6ae6c5 100644 --- a/contrib/openssl-cmake/CMakeLists.txt +++ b/contrib/openssl-cmake/CMakeLists.txt @@ -1,16 +1,6 @@ set(OPENSSL_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/openssl) set(OPENSSL_BINARY_DIR ${ClickHouse_BINARY_DIR}/contrib/openssl) -#file(READ ${CMAKE_CURRENT_SOURCE_DIR}/${OPENSSL_SOURCE_DIR}/ssl/VERSION SSL_VERSION) -#string(STRIP ${SSL_VERSION} SSL_VERSION) -#string(REPLACE ":" "." SSL_VERSION ${SSL_VERSION}) -#string(REGEX REPLACE "\\..*" "" SSL_MAJOR_VERSION ${SSL_VERSION}) - -#file(READ ${CMAKE_CURRENT_SOURCE_DIR}/${OPENSSL_SOURCE_DIR}/crypto/VERSION CRYPTO_VERSION) -#string(STRIP ${CRYPTO_VERSION} CRYPTO_VERSION) -#string(REPLACE ":" "." CRYPTO_VERSION ${CRYPTO_VERSION}) -#string(REGEX REPLACE "\\..*" "" CRYPTO_MAJOR_VERSION ${CRYPTO_VERSION}) - set(OPENSSLDIR "/etc/ssl" CACHE PATH "Set the default openssl directory") set(OPENSSL_ENGINESDIR "/usr/lib/engines-3" CACHE PATH "Set the default openssl directory for engines") set(OPENSSL_MODULESDIR "/usr/local/lib/ossl-modules" CACHE PATH "Set the default openssl directory for modules") @@ -27,19 +17,27 @@ elseif(ARCH_AARCH64) endif() enable_language(ASM) + if (COMPILER_CLANG) add_definitions(-Wno-unused-command-line-argument) endif () if (ARCH_AMD64) + if (OS_DARWIN) + set (OPENSSL_SYSTEM "macosx") + endif () + macro(perl_generate_asm FILE_IN FILE_OUT) + get_filename_component(DIRNAME ${FILE_OUT} DIRECTORY) + file(MAKE_DIRECTORY ${DIRNAME}) add_custom_command(OUTPUT ${FILE_OUT} - COMMAND /usr/bin/env perl ${FILE_IN} ${FILE_OUT} + COMMAND /usr/bin/env perl ${FILE_IN} ${OPENSSL_SYSTEM} ${FILE_OUT} # ASM code has broken unwind tables (CFI), strip them. # Otherwise asynchronous unwind (that we use for query profiler) # will lead to segfault while trying to interpret wrong "CFA expression". COMMAND sed -i -e '/^\.cfi_/d' ${FILE_OUT}) endmacro() + perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aes-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aes-x86_64.s) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesni-mb-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesni-mb-x86_64.s) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesni-sha1-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesni-sha1-x86_64.s) @@ -70,12 +68,17 @@ if (ARCH_AMD64) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha256-x86_64.s) # This is not a mistake perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha512-x86_64.s) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/whrlpool/asm/wp-x86_64.pl ${OPENSSL_BINARY_DIR}/crypto/whrlpool/wp-x86_64.s) + elseif (ARCH_AARCH64) + macro(perl_generate_asm FILE_IN FILE_OUT) + get_filename_component(DIRNAME ${FILE_OUT} DIRECTORY) + file(MAKE_DIRECTORY ${DIRNAME}) add_custom_command(OUTPUT ${FILE_OUT} COMMAND /usr/bin/env perl ${FILE_IN} "linux64" ${FILE_OUT}) # Hope that the ASM code for AArch64 doesn't have broken CFI. Otherwise, add the same sed as for x86_64. endmacro() + perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/aesv8-armx.pl ${OPENSSL_BINARY_DIR}/crypto/aes/aesv8-armx.S) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/aes/asm/vpaes-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/aes/vpaes-armv8.S) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/bn/asm/armv8-mont.pl ${OPENSSL_BINARY_DIR}/crypto/bn/armv8-mont.S) @@ -88,6 +91,7 @@ elseif (ARCH_AARCH64) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha1-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha1-armv8.S) perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha256-armv8.S) # This is not a mistake perl_generate_asm(${OPENSSL_SOURCE_DIR}/crypto/sha/asm/sha512-armv8.pl ${OPENSSL_BINARY_DIR}/crypto/sha/sha512-armv8.S) + endif () set(CRYPTO_SRCS diff --git a/contrib/ryu b/contrib/ryu new file mode 160000 index 00000000000..5b4a853534b --- /dev/null +++ b/contrib/ryu @@ -0,0 +1 @@ +Subproject commit 5b4a853534b47438b4d97935370f6b2397137c2b diff --git a/contrib/ryu-cmake/CMakeLists.txt b/contrib/ryu-cmake/CMakeLists.txt new file mode 100644 index 00000000000..bf46fdc61a7 --- /dev/null +++ b/contrib/ryu-cmake/CMakeLists.txt @@ -0,0 +1,10 @@ +SET(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/ryu) + +add_library(ryu +${LIBRARY_DIR}/ryu/d2fixed.c +${LIBRARY_DIR}/ryu/d2s.c +${LIBRARY_DIR}/ryu/f2s.c +${LIBRARY_DIR}/ryu/generic_128.c +) + +target_include_directories(ryu SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}") diff --git a/contrib/zlib-ng b/contrib/zlib-ng index 5673222fbd3..bba56a73be2 160000 --- a/contrib/zlib-ng +++ b/contrib/zlib-ng @@ -1 +1 @@ -Subproject commit 5673222fbd37ea89afb2ea73096f9bf5ec68ea31 +Subproject commit bba56a73be249514acfbc7d49aa2a68994dad8ab diff --git a/dbms/CMakeLists.txt b/dbms/CMakeLists.txt index 466b3daf94f..5194c292f3b 100644 --- a/dbms/CMakeLists.txt +++ b/dbms/CMakeLists.txt @@ -45,36 +45,75 @@ endif () option (WEVERYTHING "Enables -Weverything option with some exceptions. This is intended for exploration of new compiler warnings that may be found to be useful. Only makes sense for clang." ON) -if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang") - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wpedantic -Wno-vla-extension -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types") +if (COMPILER_CLANG) + add_warning(pedantic) + no_warning(gnu-anonymous-struct) + no_warning(nested-anon-types) + no_warning(vla-extension) + no_warning(zero-length-array) - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow -Wshadow-uncaptured-local -Wextra-semi -Wcomma -Winconsistent-missing-destructor-override -Wunused-exception-parameter -Wcovered-switch-default -Wold-style-cast -Wrange-loop-analysis -Wunused-member-function -Wunreachable-code -Wunreachable-code-return -Wnewline-eof -Wembedded-directive -Wgnu-case-range -Wunused-macros -Wconditional-uninitialized -Wdeprecated -Wundef -Wreserved-id-macro -Wredundant-parens -Wzero-as-null-pointer-constant") + add_warning(comma) + add_warning(conditional-uninitialized) + add_warning(covered-switch-default) + add_warning(deprecated) + add_warning(embedded-directive) + add_warning(empty-init-stmt) # linux-only + add_warning(extra-semi-stmt) # linux-only + add_warning(extra-semi) + add_warning(gnu-case-range) + add_warning(inconsistent-missing-destructor-override) + add_warning(newline-eof) + add_warning(old-style-cast) + add_warning(range-loop-analysis) + add_warning(redundant-parens) + add_warning(reserved-id-macro) + add_warning(shadow-field) # clang 8+ + add_warning(shadow-uncaptured-local) + add_warning(shadow) + add_warning(string-plus-int) # clang 8+ + add_warning(undef) + add_warning(unreachable-code-return) + add_warning(unreachable-code) + add_warning(unused-exception-parameter) + add_warning(unused-macros) + add_warning(unused-member-function) + add_warning(zero-as-null-pointer-constant) if (WEVERYTHING) - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-padded -Wno-switch-enum -Wno-deprecated-dynamic-exception-spec -Wno-float-equal -Wno-weak-vtables -Wno-shift-sign-overflow -Wno-sign-conversion -Wno-conversion -Wno-exit-time-destructors -Wno-undefined-func-template -Wno-documentation-unknown-command -Wno-missing-variable-declarations -Wno-unused-template -Wno-global-constructors -Wno-c99-extensions -Wno-missing-prototypes -Wno-weak-template-vtables -Wno-zero-length-array -Wno-gnu-anonymous-struct -Wno-nested-anon-types -Wno-double-promotion -Wno-disabled-macro-expansion -Wno-vla-extension -Wno-vla -Wno-packed") + add_warning(everything) + no_warning(c++98-compat-pedantic) + no_warning(c++98-compat) + no_warning(c99-extensions) + no_warning(conversion) + no_warning(ctad-maybe-unsupported) # clang 9+, linux-only + no_warning(deprecated-dynamic-exception-spec) + no_warning(disabled-macro-expansion) + no_warning(documentation-unknown-command) + no_warning(double-promotion) + no_warning(exit-time-destructors) + no_warning(float-equal) + no_warning(global-constructors) + no_warning(gnu-anonymous-struct) + no_warning(missing-prototypes) + no_warning(missing-variable-declarations) + no_warning(nested-anon-types) + no_warning(packed) + no_warning(padded) + no_warning(return-std-move-in-c++11) # clang 7+ + no_warning(shift-sign-overflow) + no_warning(sign-conversion) + no_warning(switch-enum) + no_warning(undefined-func-template) + no_warning(unused-template) + no_warning(vla-extension) + no_warning(vla) + no_warning(weak-template-vtables) + no_warning(weak-vtables) + no_warning(zero-length-array) # TODO Enable conversion, sign-conversion, double-promotion warnings. endif () - - if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7) - if (WEVERYTHING) - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-return-std-move-in-c++11") - endif () - endif () - - if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8) - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wshadow-field -Wstring-plus-int") - if(NOT APPLE) - set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wempty-init-stmt") - endif() - endif () - - if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9) - if (WEVERYTHING AND NOT APPLE) - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-ctad-maybe-unsupported") - endif () - endif () -elseif (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") +elseif (COMPILER_GCC) # Add compiler options only to c++ compiler function(add_cxx_compile_options option) add_compile_options("$<$,CXX>:${option}>") @@ -156,7 +195,7 @@ if (USE_DEBUG_HELPERS) set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${INCLUDE_DEBUG_HELPERS}") endif () -if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") +if (COMPILER_GCC) # If we leave this optimization enabled, gcc-7 replaces a pair of SSE intrinsics (16 byte load, store) with a call to memcpy. # It leads to slow code. This is compiler bug. It looks like this: # @@ -330,6 +369,7 @@ target_link_libraries (clickhouse_common_io ${LINK_LIBRARIES_ONLY_ON_X86_64} PUBLIC ${DOUBLE_CONVERSION_LIBRARIES} + ryu PUBLIC ${Poco_Net_LIBRARY} ${Poco_Util_LIBRARY} diff --git a/dbms/cmake/version.cmake b/dbms/cmake/version.cmake index 220af3d87dc..6d6700bc649 100644 --- a/dbms/cmake/version.cmake +++ b/dbms/cmake/version.cmake @@ -1,11 +1,11 @@ # This strings autochanged from release_lib.sh: -set(VERSION_REVISION 54430) -set(VERSION_MAJOR 19) -set(VERSION_MINOR 19) +set(VERSION_REVISION 54431) +set(VERSION_MAJOR 20) +set(VERSION_MINOR 1) set(VERSION_PATCH 1) -set(VERSION_GITHASH 8bd9709d1dec3366e35d2efeab213435857f67a9) -set(VERSION_DESCRIBE v19.19.1.1-prestable) -set(VERSION_STRING 19.19.1.1) +set(VERSION_GITHASH 51d4c8a53be94504e3607b2232e12e5ef7a8ec28) +set(VERSION_DESCRIBE v20.1.1.1-prestable) +set(VERSION_STRING 20.1.1.1) # end of autochange set(VERSION_EXTRA "" CACHE STRING "") diff --git a/dbms/programs/client/Client.cpp b/dbms/programs/client/Client.cpp index 4b9cee29ff6..6b3d61d35b1 100644 --- a/dbms/programs/client/Client.cpp +++ b/dbms/programs/client/Client.cpp @@ -300,7 +300,7 @@ private: && std::string::npos == embedded_stack_trace_pos) { std::cerr << "Stack trace:" << std::endl - << e.getStackTrace().toString(); + << e.getStackTraceString(); } /// If exception code isn't zero, we should return non-zero return code anyway. @@ -327,6 +327,78 @@ private: || (now.month() == 1 && now.day() <= 5); } + bool isChineseNewYearMode(const String & local_tz) + { + /// Days of Dec. 20 in Chinese calendar starting from year 2019 to year 2105 + static constexpr UInt16 chineseNewYearIndicators[] + = {18275, 18659, 19014, 19368, 19752, 20107, 20491, 20845, 21199, 21583, 21937, 22292, 22676, 23030, 23414, 23768, 24122, 24506, + 24860, 25215, 25599, 25954, 26308, 26692, 27046, 27430, 27784, 28138, 28522, 28877, 29232, 29616, 29970, 30354, 30708, 31062, + 31446, 31800, 32155, 32539, 32894, 33248, 33632, 33986, 34369, 34724, 35078, 35462, 35817, 36171, 36555, 36909, 37293, 37647, + 38002, 38386, 38740, 39095, 39479, 39833, 40187, 40571, 40925, 41309, 41664, 42018, 42402, 42757, 43111, 43495, 43849, 44233, + 44587, 44942, 45326, 45680, 46035, 46418, 46772, 47126, 47510, 47865, 48249, 48604, 48958, 49342}; + static constexpr size_t N = sizeof(chineseNewYearIndicators) / sizeof(chineseNewYearIndicators[0]); + + /// All time zone names are acquired from https://www.iana.org/time-zones + static constexpr const char * chineseNewYearTimeZoneIndicators[] = { + /// Time zones celebrating Chinese new year. + "Asia/Shanghai", + "Asia/Chongqing", + "Asia/Harbin", + "Asia/Urumqi", + "Asia/Hong_Kong", + "Asia/Chungking", + "Asia/Macao", + "Asia/Macau", + "Asia/Taipei", + "Asia/Singapore", + + /// Time zones celebrating Chinese new year but with different festival names. Let's not print the message for now. + // "Asia/Brunei", + // "Asia/Ho_Chi_Minh", + // "Asia/Hovd", + // "Asia/Jakarta", + // "Asia/Jayapura", + // "Asia/Kashgar", + // "Asia/Kuala_Lumpur", + // "Asia/Kuching", + // "Asia/Makassar", + // "Asia/Pontianak", + // "Asia/Pyongyang", + // "Asia/Saigon", + // "Asia/Seoul", + // "Asia/Ujung_Pandang", + // "Asia/Ulaanbaatar", + // "Asia/Ulan_Bator", + }; + static constexpr size_t M = sizeof(chineseNewYearTimeZoneIndicators) / sizeof(chineseNewYearTimeZoneIndicators[0]); + + time_t current_time = time(nullptr); + + if (chineseNewYearTimeZoneIndicators + M + == std::find_if(chineseNewYearTimeZoneIndicators, chineseNewYearTimeZoneIndicators + M, [&local_tz](const char * tz) + { + return tz == local_tz; + })) + return false; + + /// It's bad to be intrusive. + if (current_time % 3 != 0) + return false; + + auto days = DateLUT::instance().toDayNum(current_time).toUnderType(); + for (auto i = 0ul; i < N; ++i) + { + auto d = chineseNewYearIndicators[i]; + + /// Let's celebrate until Lantern Festival + if (d <= days && d + 25u >= days) + return true; + else if (d > days) + return false; + } + return false; + } + int mainImpl() { UseSSL use_ssl; @@ -374,7 +446,7 @@ private: connect(); /// Initialize DateLUT here to avoid counting time spent here as query execution time. - DateLUT::instance(); + const auto local_tz = DateLUT::instance().getTimeZone(); if (!context.getSettingsRef().use_client_time_zone) { const auto & time_zone = connection->getServerTimezone(connection_parameters.timeouts); @@ -540,7 +612,12 @@ private: loop(); - std::cout << (isNewYearMode() ? "Happy new year." : "Bye.") << std::endl; + if (isNewYearMode()) + std::cout << "Happy new year." << std::endl; + else if (isChineseNewYearMode(local_tz)) + std::cout << "Happy Chinese new year. 春节快乐!" << std::endl; + else + std::cout << "Bye." << std::endl; return 0; } else @@ -714,7 +791,7 @@ private: if (config().getBool("stacktrace", false)) std::cerr << "Stack trace:" << std::endl - << e.getStackTrace().toString() << std::endl; + << e.getStackTraceString() << std::endl; std::cerr << std::endl; diff --git a/dbms/programs/odbc-bridge/MainHandler.cpp b/dbms/programs/odbc-bridge/MainHandler.cpp index 73480bf884f..074aaedd7ce 100644 --- a/dbms/programs/odbc-bridge/MainHandler.cpp +++ b/dbms/programs/odbc-bridge/MainHandler.cpp @@ -115,7 +115,7 @@ void ODBCHandler::handleRequest(Poco::Net::HTTPServerRequest & request, Poco::Ne catch (const Exception & ex) { process_error("Invalid 'columns' parameter in request body '" + ex.message() + "'"); - LOG_WARNING(log, ex.getStackTrace().toString()); + LOG_WARNING(log, ex.getStackTraceString()); return; } diff --git a/dbms/programs/performance-test/PerformanceTest.cpp b/dbms/programs/performance-test/PerformanceTest.cpp index 689f68f8d5e..e1550780b15 100644 --- a/dbms/programs/performance-test/PerformanceTest.cpp +++ b/dbms/programs/performance-test/PerformanceTest.cpp @@ -85,16 +85,6 @@ bool PerformanceTest::checkPreconditions() const for (const std::string & precondition : preconditions) { - if (precondition == "flush_disk_cache") - { - if (system( - "(>&2 echo 'Flushing disk cache...') && (sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches') && (>&2 echo 'Flushed.')")) - { - LOG_WARNING(log, "Failed to flush disk cache"); - return false; - } - } - if (precondition == "ram_size") { size_t ram_size_needed = config->getUInt64("preconditions.ram_size"); @@ -337,7 +327,7 @@ void PerformanceTest::runQueries( { statistics.exception = "Code: " + std::to_string(e.code()) + ", e.displayText() = " + e.displayText(); LOG_WARNING(log, "Code: " << e.code() << ", e.displayText() = " << e.displayText() - << ", Stack trace:\n\n" << e.getStackTrace().toString()); + << ", Stack trace:\n\n" << e.getStackTraceString()); } if (!statistics.got_SIGINT) diff --git a/dbms/programs/performance-test/PerformanceTestInfo.cpp b/dbms/programs/performance-test/PerformanceTestInfo.cpp index 8435b29a67a..b0f877abfc7 100644 --- a/dbms/programs/performance-test/PerformanceTestInfo.cpp +++ b/dbms/programs/performance-test/PerformanceTestInfo.cpp @@ -45,21 +45,11 @@ namespace fs = std::filesystem; PerformanceTestInfo::PerformanceTestInfo( XMLConfigurationPtr config, - const std::string & profiles_file_, const Settings & global_settings_) - : profiles_file(profiles_file_) - , settings(global_settings_) + : settings(global_settings_) { path = config->getString("path"); test_name = fs::path(path).stem().string(); - if (config->has("main_metric")) - { - Strings main_metrics; - config->keys("main_metric", main_metrics); - if (main_metrics.size()) - main_metric = main_metrics[0]; - } - applySettings(config); extractQueries(config); extractAuxiliaryQueries(config); @@ -75,38 +65,8 @@ void PerformanceTestInfo::applySettings(XMLConfigurationPtr config) SettingsChanges settings_to_apply; Strings config_settings; config->keys("settings", config_settings); - - auto settings_contain = [&config_settings] (const std::string & setting) - { - auto position = std::find(config_settings.begin(), config_settings.end(), setting); - return position != config_settings.end(); - - }; - /// Preprocess configuration file - if (settings_contain("profile")) - { - if (!profiles_file.empty()) - { - std::string profile_name = config->getString("settings.profile"); - XMLConfigurationPtr profiles_config(new XMLConfiguration(profiles_file)); - - Strings profile_settings; - profiles_config->keys("profiles." + profile_name, profile_settings); - - extractSettings(profiles_config, "profiles." + profile_name, profile_settings, settings_to_apply); - } - } - extractSettings(config, "settings", config_settings, settings_to_apply); settings.applyChanges(settings_to_apply); - - if (settings_contain("average_rows_speed_precision")) - TestStats::avg_rows_speed_precision = - config->getDouble("settings.average_rows_speed_precision"); - - if (settings_contain("average_bytes_speed_precision")) - TestStats::avg_bytes_speed_precision = - config->getDouble("settings.average_bytes_speed_precision"); } } diff --git a/dbms/programs/performance-test/PerformanceTestInfo.h b/dbms/programs/performance-test/PerformanceTestInfo.h index 4483e56bbfe..8e6b1c5f43a 100644 --- a/dbms/programs/performance-test/PerformanceTestInfo.h +++ b/dbms/programs/performance-test/PerformanceTestInfo.h @@ -26,15 +26,13 @@ using StringToVector = std::map; class PerformanceTestInfo { public: - PerformanceTestInfo(XMLConfigurationPtr config, const std::string & profiles_file_, const Settings & global_settings_); + PerformanceTestInfo(XMLConfigurationPtr config, const Settings & global_settings_); std::string test_name; std::string path; - std::string main_metric; Strings queries; - std::string profiles_file; Settings settings; ExecutionType exec_type; StringToVector substitutions; diff --git a/dbms/programs/performance-test/PerformanceTestSuite.cpp b/dbms/programs/performance-test/PerformanceTestSuite.cpp index 594f04a3906..66ef8eb51c0 100644 --- a/dbms/programs/performance-test/PerformanceTestSuite.cpp +++ b/dbms/programs/performance-test/PerformanceTestSuite.cpp @@ -64,7 +64,6 @@ public: const std::string & password_, const Settings & cmd_settings, const bool lite_output_, - const std::string & profiles_file_, Strings && input_files_, Strings && tests_tags_, Strings && skip_tags_, @@ -86,7 +85,6 @@ public: , skip_names_regexp(std::move(skip_names_regexp_)) , query_indexes(query_indexes_) , lite_output(lite_output_) - , profiles_file(profiles_file_) , input_files(input_files_) , log(&Poco::Logger::get("PerformanceTestSuite")) { @@ -139,7 +137,6 @@ private: using XMLConfigurationPtr = Poco::AutoPtr; bool lite_output; - std::string profiles_file; Strings input_files; std::vector tests_configurations; @@ -197,7 +194,7 @@ private: std::pair runTest(XMLConfigurationPtr & test_config) { - PerformanceTestInfo info(test_config, profiles_file, global_context.getSettingsRef()); + PerformanceTestInfo info(test_config, global_context.getSettingsRef()); LOG_INFO(log, "Config for test '" << info.test_name << "' parsed"); PerformanceTest current(test_config, connection, timeouts, interrupt_listener, info, global_context, query_indexes[info.path]); @@ -332,7 +329,6 @@ try desc.add_options() ("help", "produce help message") ("lite", "use lite version of output") - ("profiles-file", value()->default_value(""), "Specify a file with global profiles") ("host,h", value()->default_value("localhost"), "") ("port", value()->default_value(9000), "") ("secure,s", "Use TLS connection") @@ -401,7 +397,6 @@ try options["password"].as(), cmd_settings, options.count("lite") > 0, - options["profiles-file"].as(), std::move(input_files), std::move(tests_tags), std::move(skip_tags), diff --git a/dbms/programs/performance-test/ReportBuilder.cpp b/dbms/programs/performance-test/ReportBuilder.cpp index cfefc37c470..f95aa025095 100644 --- a/dbms/programs/performance-test/ReportBuilder.cpp +++ b/dbms/programs/performance-test/ReportBuilder.cpp @@ -17,23 +17,25 @@ namespace DB namespace { -const std::regex QUOTE_REGEX{"\""}; std::string getMainMetric(const PerformanceTestInfo & test_info) { - std::string main_metric; - if (test_info.main_metric.empty()) - if (test_info.exec_type == ExecutionType::Loop) - main_metric = "min_time"; - else - main_metric = "rows_per_second"; + if (test_info.exec_type == ExecutionType::Loop) + return "min_time"; else - main_metric = test_info.main_metric; - return main_metric; + return "rows_per_second"; } + bool isASCIIString(const std::string & str) { return std::all_of(str.begin(), str.end(), isASCII); } + +String jsonString(const String & str, FormatSettings & settings) +{ + WriteBufferFromOwnString buffer; + writeJSONString(str, buffer, settings); + return std::move(buffer.str()); +} } ReportBuilder::ReportBuilder(const std::string & server_version_) @@ -55,6 +57,8 @@ std::string ReportBuilder::buildFullReport( std::vector & stats, const std::vector & queries_to_run) const { + FormatSettings settings; + JSONString json_output; json_output.set("hostname", hostname); @@ -65,22 +69,18 @@ std::string ReportBuilder::buildFullReport( json_output.set("time", getCurrentTime()); json_output.set("test_name", test_info.test_name); json_output.set("path", test_info.path); - json_output.set("main_metric", getMainMetric(test_info)); - if (test_info.substitutions.size()) + if (!test_info.substitutions.empty()) { JSONString json_parameters(2); /// here, 2 is the size of \t padding - for (auto it = test_info.substitutions.begin(); it != test_info.substitutions.end(); ++it) + for (auto & [parameter, values] : test_info.substitutions) { - std::string parameter = it->first; - Strings values = it->second; - std::ostringstream array_string; array_string << "["; for (size_t i = 0; i != values.size(); ++i) { - array_string << '"' << std::regex_replace(values[i], QUOTE_REGEX, "\\\"") << '"'; + array_string << jsonString(values[i], settings); if (i != values.size() - 1) { array_string << ", "; @@ -110,13 +110,12 @@ std::string ReportBuilder::buildFullReport( JSONString runJSON; - auto query = std::regex_replace(test_info.queries[query_index], QUOTE_REGEX, "\\\""); - runJSON.set("query", query); + runJSON.set("query", jsonString(test_info.queries[query_index], settings), false); runJSON.set("query_index", query_index); if (!statistics.exception.empty()) { if (isASCIIString(statistics.exception)) - runJSON.set("exception", std::regex_replace(statistics.exception, QUOTE_REGEX, "\\\"")); + runJSON.set("exception", jsonString(statistics.exception, settings), false); else runJSON.set("exception", "Some exception occured with non ASCII message. This may produce invalid JSON. Try reproduce locally."); } @@ -183,7 +182,7 @@ std::string ReportBuilder::buildCompactReport( std::vector & stats, const std::vector & queries_to_run) const { - + FormatSettings settings; std::ostringstream output; for (size_t query_index = 0; query_index < test_info.queries.size(); ++query_index) @@ -194,7 +193,7 @@ std::string ReportBuilder::buildCompactReport( for (size_t number_of_launch = 0; number_of_launch < test_info.times_to_run; ++number_of_launch) { if (test_info.queries.size() > 1) - output << "query \"" << test_info.queries[query_index] << "\", "; + output << "query " << jsonString(test_info.queries[query_index], settings) << ", "; output << "run " << std::to_string(number_of_launch + 1) << ": "; diff --git a/dbms/programs/server/HTTPHandler.cpp b/dbms/programs/server/HTTPHandler.cpp index 29d186def2d..b2b3298693e 100644 --- a/dbms/programs/server/HTTPHandler.cpp +++ b/dbms/programs/server/HTTPHandler.cpp @@ -20,8 +20,6 @@ #include #include #include -#include -#include #include #include #include @@ -300,32 +298,24 @@ void HTTPHandler::processQuery( /// The client can pass a HTTP header indicating supported compression method (gzip or deflate). String http_response_compression_methods = request.get("Accept-Encoding", ""); - bool client_supports_http_compression = false; - CompressionMethod http_response_compression_method {}; + CompressionMethod http_response_compression_method = CompressionMethod::None; if (!http_response_compression_methods.empty()) { + /// If client supports brotli - it's preferred. /// Both gzip and deflate are supported. If the client supports both, gzip is preferred. /// NOTE parsing of the list of methods is slightly incorrect. - if (std::string::npos != http_response_compression_methods.find("gzip")) - { - client_supports_http_compression = true; - http_response_compression_method = CompressionMethod::Gzip; - } - else if (std::string::npos != http_response_compression_methods.find("deflate")) - { - client_supports_http_compression = true; - http_response_compression_method = CompressionMethod::Zlib; - } -#if USE_BROTLI - else if (http_response_compression_methods == "br") - { - client_supports_http_compression = true; + + if (std::string::npos != http_response_compression_methods.find("br")) http_response_compression_method = CompressionMethod::Brotli; - } -#endif + else if (std::string::npos != http_response_compression_methods.find("gzip")) + http_response_compression_method = CompressionMethod::Gzip; + else if (std::string::npos != http_response_compression_methods.find("deflate")) + http_response_compression_method = CompressionMethod::Zlib; } + bool client_supports_http_compression = http_response_compression_method != CompressionMethod::None; + /// Client can pass a 'compress' flag in the query string. In this case the query result is /// compressed using internal algorithm. This is not reflected in HTTP headers. bool internal_compression = params.getParsed("compress", false); @@ -344,8 +334,8 @@ void HTTPHandler::processQuery( unsigned keep_alive_timeout = config.getUInt("keep_alive_timeout", 10); used_output.out = std::make_shared( - request, response, keep_alive_timeout, - client_supports_http_compression, http_response_compression_method, buffer_size_http); + request, response, keep_alive_timeout, client_supports_http_compression, http_response_compression_method); + if (internal_compression) used_output.out_maybe_compressed = std::make_shared(*used_output.out); else @@ -400,32 +390,9 @@ void HTTPHandler::processQuery( std::unique_ptr in_post_raw = std::make_unique(istr); /// Request body can be compressed using algorithm specified in the Content-Encoding header. - std::unique_ptr in_post; String http_request_compression_method_str = request.get("Content-Encoding", ""); - if (!http_request_compression_method_str.empty()) - { - if (http_request_compression_method_str == "gzip") - { - in_post = std::make_unique(std::move(in_post_raw), CompressionMethod::Gzip); - } - else if (http_request_compression_method_str == "deflate") - { - in_post = std::make_unique(std::move(in_post_raw), CompressionMethod::Zlib); - } -#if USE_BROTLI - else if (http_request_compression_method_str == "br") - { - in_post = std::make_unique(std::move(in_post_raw)); - } -#endif - else - { - throw Exception("Unknown Content-Encoding of HTTP request: " + http_request_compression_method_str, - ErrorCodes::UNKNOWN_COMPRESSION_METHOD); - } - } - else - in_post = std::move(in_post_raw); + std::unique_ptr in_post = wrapReadBufferWithCompressionMethod( + std::make_unique(istr), chooseCompressionMethod({}, http_request_compression_method_str)); /// The data can also be compressed using incompatible internal algorithm. This is indicated by /// 'decompress' query parameter. diff --git a/dbms/programs/server/Server.cpp b/dbms/programs/server/Server.cpp index 972401c19d5..bb08abf2161 100644 --- a/dbms/programs/server/Server.cpp +++ b/dbms/programs/server/Server.cpp @@ -947,6 +947,7 @@ int Server::main(const std::vector & /*args*/) }); /// try to load dictionaries immediately, throw on error and die + ext::scope_guard dictionaries_xmls, models_xmls; try { if (!config().getBool("dictionaries_lazy_load", true)) @@ -954,12 +955,10 @@ int Server::main(const std::vector & /*args*/) global_context->tryCreateEmbeddedDictionaries(); global_context->getExternalDictionariesLoader().enableAlwaysLoadEverything(true); } - - auto dictionaries_repository = std::make_unique(config(), "dictionaries_config"); - global_context->getExternalDictionariesLoader().addConfigRepository("", std::move(dictionaries_repository)); - - auto models_repository = std::make_unique(config(), "models_config"); - global_context->getExternalModelsLoader().addConfigRepository("", std::move(models_repository)); + dictionaries_xmls = global_context->getExternalDictionariesLoader().addConfigRepository( + std::make_unique(config(), "dictionaries_config")); + models_xmls = global_context->getExternalModelsLoader().addConfigRepository( + std::make_unique(config(), "models_config")); } catch (...) { diff --git a/dbms/programs/server/TCPHandler.cpp b/dbms/programs/server/TCPHandler.cpp index cb215eb0af8..01c5ae59cea 100644 --- a/dbms/programs/server/TCPHandler.cpp +++ b/dbms/programs/server/TCPHandler.cpp @@ -112,7 +112,7 @@ void TCPHandler::runImpl() { Exception e("Database " + backQuote(default_database) + " doesn't exist", ErrorCodes::UNKNOWN_DATABASE); LOG_ERROR(log, "Code: " << e.code() << ", e.displayText() = " << e.displayText() - << ", Stack trace:\n\n" << e.getStackTrace().toString()); + << ", Stack trace:\n\n" << e.getStackTraceString()); sendException(e, connection_context.getSettingsRef().calculate_text_stack_trace); return; } @@ -158,7 +158,7 @@ void TCPHandler::runImpl() /** An exception during the execution of request (it must be sent over the network to the client). * The client will be able to accept it, if it did not happen while sending another packet and the client has not disconnected yet. */ - std::unique_ptr exception; + std::optional exception; bool network_error = false; bool send_exception_with_stack_trace = connection_context.getSettingsRef().calculate_text_stack_trace; @@ -280,7 +280,7 @@ void TCPHandler::runImpl() catch (const Exception & e) { state.io.onException(); - exception.reset(e.clone()); + exception.emplace(e); if (e.code() == ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT) throw; @@ -298,22 +298,22 @@ void TCPHandler::runImpl() * We will try to send exception to the client in any case - see below. */ state.io.onException(); - exception = std::make_unique(e.displayText(), ErrorCodes::POCO_EXCEPTION); + exception.emplace(Exception::CreateFromPoco, e); } catch (const Poco::Exception & e) { state.io.onException(); - exception = std::make_unique(e.displayText(), ErrorCodes::POCO_EXCEPTION); + exception.emplace(Exception::CreateFromPoco, e); } catch (const std::exception & e) { state.io.onException(); - exception = std::make_unique(e.what(), ErrorCodes::STD_EXCEPTION); + exception.emplace(Exception::CreateFromSTD, e); } catch (...) { state.io.onException(); - exception = std::make_unique("Unknown exception", ErrorCodes::UNKNOWN_EXCEPTION); + exception.emplace("Unknown exception", ErrorCodes::UNKNOWN_EXCEPTION); } try @@ -546,7 +546,7 @@ void TCPHandler::processOrdinaryQueryWithProcessors(size_t num_threads) auto & pipeline = state.io.pipeline; if (pipeline.getMaxThreads()) - num_threads = pipeline.getMaxThreads(); + num_threads = std::min(num_threads, pipeline.getMaxThreads()); /// Send header-block, to allow client to prepare output format for data to send. { diff --git a/dbms/programs/server/config.d/part_log.xml b/dbms/programs/server/config.d/part_log.xml new file mode 100644 index 00000000000..35add3c6cc1 --- /dev/null +++ b/dbms/programs/server/config.d/part_log.xml @@ -0,0 +1,7 @@ + + + system + part_log
+ 7500 + + diff --git a/dbms/programs/server/users.xml b/dbms/programs/server/users.xml index 0058ee51184..87e6c406b0a 100644 --- a/dbms/programs/server/users.xml +++ b/dbms/programs/server/users.xml @@ -83,30 +83,7 @@ default - - - - - - diff --git a/dbms/src/Access/RowPolicyContextFactory.cpp b/dbms/src/Access/RowPolicyContextFactory.cpp index e458f06ca94..77e5056e206 100644 --- a/dbms/src/Access/RowPolicyContextFactory.cpp +++ b/dbms/src/Access/RowPolicyContextFactory.cpp @@ -101,9 +101,6 @@ namespace public: void add(const ASTPtr & condition, bool is_restrictive) { - if (!condition) - return; - if (is_restrictive) restrictions.push_back(condition); else @@ -139,29 +136,32 @@ void RowPolicyContextFactory::PolicyInfo::setPolicy(const RowPolicyPtr & policy_ for (auto index : ext::range_with_static_cast(0, MAX_CONDITION_INDEX)) { + parsed_conditions[index] = nullptr; const String & condition = policy->conditions[index]; + if (condition.empty()) + continue; + auto previous_range = std::pair(std::begin(policy->conditions), std::begin(policy->conditions) + index); auto previous_it = std::find(previous_range.first, previous_range.second, condition); if (previous_it != previous_range.second) { /// The condition is already parsed before. parsed_conditions[index] = parsed_conditions[previous_it - previous_range.first]; + continue; } - else + + /// Try to parse the condition. + try { - /// Try to parse the condition. - try - { - ParserExpression parser; - parsed_conditions[index] = parseQuery(parser, condition, 0); - } - catch (...) - { - tryLogCurrentException( - &Poco::Logger::get("RowPolicy"), - String("Could not parse the condition ") + RowPolicy::conditionIndexToString(index) + " of row policy " - + backQuote(policy->getFullName())); - } + ParserExpression parser; + parsed_conditions[index] = parseQuery(parser, condition, 0); + } + catch (...) + { + tryLogCurrentException( + &Poco::Logger::get("RowPolicy"), + String("Could not parse the condition ") + RowPolicy::conditionIndexToString(index) + " of row policy " + + backQuote(policy->getFullName())); } } } @@ -290,7 +290,8 @@ void RowPolicyContextFactory::mixConditionsForContext(RowPolicyContext & context auto & mixers = map_of_mixers[std::pair{policy.getDatabase(), policy.getTableName()}]; mixers.policy_ids.push_back(policy_id); for (auto index : ext::range(0, MAX_CONDITION_INDEX)) - mixers.mixers[index].add(info.parsed_conditions[index], policy.isRestrictive()); + if (info.parsed_conditions[index]) + mixers.mixers[index].add(info.parsed_conditions[index], policy.isRestrictive()); } } diff --git a/dbms/src/Access/UsersConfigAccessStorage.cpp b/dbms/src/Access/UsersConfigAccessStorage.cpp index c9671afaca1..033e8f557b7 100644 --- a/dbms/src/Access/UsersConfigAccessStorage.cpp +++ b/dbms/src/Access/UsersConfigAccessStorage.cpp @@ -135,13 +135,25 @@ namespace for (const String & database : databases) { const String database_config = databases_config + "." + database; - Poco::Util::AbstractConfiguration::Keys table_names; - config.keys(database_config, table_names); + Poco::Util::AbstractConfiguration::Keys keys_in_database_config; + config.keys(database_config, keys_in_database_config); /// Read table properties - for (const String & table_name : table_names) + for (const String & key_in_database_config : keys_in_database_config) { - const auto filter_config = database_config + "." + table_name + ".filter"; + String table_name = key_in_database_config; + String filter_config = database_config + "." + table_name + ".filter"; + + if (key_in_database_config.starts_with("table[")) + { + const auto table_name_config = database_config + "." + table_name + "[@name]"; + if (config.has(table_name_config)) + { + table_name = config.getString(table_name_config); + filter_config = database_config + ".table[@name='" + table_name + "']"; + } + } + if (config.has(filter_config)) { try diff --git a/dbms/src/Client/Connection.cpp b/dbms/src/Client/Connection.cpp index 75bac5c0cb2..b3f1e341f80 100644 --- a/dbms/src/Client/Connection.cpp +++ b/dbms/src/Client/Connection.cpp @@ -774,9 +774,7 @@ std::unique_ptr Connection::receiveException() { //LOG_TRACE(log_wrapper.get(), "Receiving exception"); - Exception e; - readException(e, *in, "Received from " + getDescription()); - return std::unique_ptr{ e.clone() }; + return std::make_unique(readException(*in, "Received from " + getDescription())); } diff --git a/dbms/src/Common/ErrorCodes.cpp b/dbms/src/Common/ErrorCodes.cpp index 7a30c832759..f20c7920a12 100644 --- a/dbms/src/Common/ErrorCodes.cpp +++ b/dbms/src/Common/ErrorCodes.cpp @@ -138,7 +138,6 @@ namespace ErrorCodes extern const int FUNCTION_IS_SPECIAL = 129; extern const int CANNOT_READ_ARRAY_FROM_TEXT = 130; extern const int TOO_LARGE_STRING_SIZE = 131; - extern const int CANNOT_CREATE_TABLE_FROM_METADATA = 132; extern const int AGGREGATE_FUNCTION_DOESNT_ALLOW_PARAMETERS = 133; extern const int PARAMETERS_TO_AGGREGATE_FUNCTIONS_MUST_BE_LITERALS = 134; extern const int ZERO_ARRAY_OR_TUPLE_INDEX = 135; @@ -474,7 +473,6 @@ namespace ErrorCodes extern const int NOT_ENOUGH_PRIVILEGES = 497; extern const int LIMIT_BY_WITH_TIES_IS_NOT_SUPPORTED = 498; extern const int S3_ERROR = 499; - extern const int CANNOT_CREATE_DICTIONARY_FROM_METADATA = 500; extern const int CANNOT_CREATE_DATABASE = 501; extern const int CANNOT_SIGQUEUE = 502; extern const int AGGREGATE_FUNCTION_THROW = 503; diff --git a/dbms/src/Common/Exception.cpp b/dbms/src/Common/Exception.cpp index 28a86c620d1..25da9674e4d 100644 --- a/dbms/src/Common/Exception.cpp +++ b/dbms/src/Common/Exception.cpp @@ -23,8 +23,60 @@ namespace ErrorCodes extern const int UNKNOWN_EXCEPTION; extern const int CANNOT_TRUNCATE_FILE; extern const int NOT_IMPLEMENTED; + extern const int LOGICAL_ERROR; } + +Exception::Exception() +{ +} + +Exception::Exception(const std::string & msg, int code) + : Poco::Exception(msg, code) +{ + // In debug builds, treat LOGICAL_ERROR as an assertion failure. + assert(code != ErrorCodes::LOGICAL_ERROR); +} + +Exception::Exception(CreateFromPocoTag, const Poco::Exception & exc) + : Poco::Exception(exc.displayText(), ErrorCodes::POCO_EXCEPTION) +{ +#ifdef STD_EXCEPTION_HAS_STACK_TRACE + set_stack_trace(exc.get_stack_trace_frames(), exc.get_stack_trace_size()); +#endif +} + +Exception::Exception(CreateFromSTDTag, const std::exception & exc) + : Poco::Exception(String(typeid(exc).name()) + ": " + String(exc.what()), ErrorCodes::STD_EXCEPTION) +{ +#ifdef STD_EXCEPTION_HAS_STACK_TRACE + set_stack_trace(exc.get_stack_trace_frames(), exc.get_stack_trace_size()); +#endif +} + + +std::string getExceptionStackTraceString(const std::exception & e) +{ +#ifdef STD_EXCEPTION_HAS_STACK_TRACE + return StackTrace::toString(e.get_stack_trace_frames(), 0, e.get_stack_trace_size()); +#else + if (const auto * db_exception = dynamic_cast(&e)) + return db_exception->getStackTraceString(); + return {}; +#endif +} + + +std::string Exception::getStackTraceString() const +{ +#ifdef STD_EXCEPTION_HAS_STACK_TRACE + return StackTrace::toString(get_stack_trace_frames(), 0, get_stack_trace_size()); +#else + return trace.toString(); +#endif +} + + std::string errnoToString(int code, int e) { const size_t buf_size = 128; @@ -141,6 +193,7 @@ std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded { stream << "Poco::Exception. Code: " << ErrorCodes::POCO_EXCEPTION << ", e.code() = " << e.code() << ", e.displayText() = " << e.displayText() + << (with_stacktrace ? getExceptionStackTraceString(e) : "") << (with_extra_info ? getExtraExceptionInfo(e) : "") << " (version " << VERSION_STRING << VERSION_OFFICIAL; } @@ -157,8 +210,9 @@ std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded name += " (demangling status: " + toString(status) + ")"; stream << "std::exception. Code: " << ErrorCodes::STD_EXCEPTION << ", type: " << name << ", e.what() = " << e.what() - << (with_extra_info ? getExtraExceptionInfo(e) : "") - << ", version = " << VERSION_STRING << VERSION_OFFICIAL; + << (with_stacktrace ? getExceptionStackTraceString(e) : "") + << (with_extra_info ? getExtraExceptionInfo(e) : "") + << ", version = " << VERSION_STRING << VERSION_OFFICIAL; } catch (...) {} } @@ -261,7 +315,7 @@ std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool stream << "Code: " << e.code() << ", e.displayText() = " << text; if (with_stacktrace && !has_embedded_stack_trace) - stream << ", Stack trace (when copying this message, always include the lines below):\n\n" << e.getStackTrace().toString(); + stream << ", Stack trace (when copying this message, always include the lines below):\n\n" << e.getStackTraceString(); } catch (...) {} diff --git a/dbms/src/Common/Exception.h b/dbms/src/Common/Exception.h index 5df2879a16d..d14b2eb77a1 100644 --- a/dbms/src/Common/Exception.h +++ b/dbms/src/Common/Exception.h @@ -22,13 +22,14 @@ namespace ErrorCodes class Exception : public Poco::Exception { public: - Exception() {} /// For deferred initialization. - Exception(const std::string & msg, int code) : Poco::Exception(msg, code) {} - Exception(const std::string & msg, const Exception & nested_exception, int code) - : Poco::Exception(msg, nested_exception, code), trace(nested_exception.trace) {} + Exception(); + Exception(const std::string & msg, int code); enum CreateFromPocoTag { CreateFromPoco }; - Exception(CreateFromPocoTag, const Poco::Exception & exc) : Poco::Exception(exc.displayText(), ErrorCodes::POCO_EXCEPTION) {} + enum CreateFromSTDTag { CreateFromSTD }; + + Exception(CreateFromPocoTag, const Poco::Exception & exc); + Exception(CreateFromSTDTag, const std::exception & exc); Exception * clone() const override { return new Exception(*this); } void rethrow() const override { throw *this; } @@ -38,15 +39,20 @@ public: /// Add something to the existing message. void addMessage(const std::string & arg) { extendedMessage(arg); } - const StackTrace & getStackTrace() const { return trace; } + std::string getStackTraceString() const; private: +#ifndef STD_EXCEPTION_HAS_STACK_TRACE StackTrace trace; +#endif const char * className() const throw() override { return "DB::Exception"; } }; +std::string getExceptionStackTraceString(const std::exception & e); + + /// Contains an additional member `saved_errno`. See the throwFromErrno function. class ErrnoException : public Exception { diff --git a/dbms/src/Common/LRUCache.h b/dbms/src/Common/LRUCache.h index 98e88d9c59f..5bcfc8fc2db 100644 --- a/dbms/src/Common/LRUCache.h +++ b/dbms/src/Common/LRUCache.h @@ -23,11 +23,10 @@ struct TrivialWeightFunction }; -/// Thread-safe cache that evicts entries which are not used for a long time or are expired. +/// Thread-safe cache that evicts entries which are not used for a long time. /// WeightFunction is a functor that takes Mapped as a parameter and returns "weight" (approximate size) /// of that value. -/// Cache starts to evict entries when their total weight exceeds max_size and when expiration time of these -/// entries is due. +/// Cache starts to evict entries when their total weight exceeds max_size. /// Value weight should not change after insertion. template , typename WeightFunction = TrivialWeightFunction> class LRUCache @@ -36,15 +35,13 @@ public: using Key = TKey; using Mapped = TMapped; using MappedPtr = std::shared_ptr; - using Delay = std::chrono::seconds; private: using Clock = std::chrono::steady_clock; - using Timestamp = Clock::time_point; public: - LRUCache(size_t max_size_, const Delay & expiration_delay_ = Delay::zero()) - : max_size(std::max(static_cast(1), max_size_)), expiration_delay(expiration_delay_) {} + LRUCache(size_t max_size_) + : max_size(std::max(static_cast(1), max_size_)) {} MappedPtr get(const Key & key) { @@ -167,16 +164,9 @@ protected: struct Cell { - bool expired(const Timestamp & last_timestamp, const Delay & delay) const - { - return (delay == Delay::zero()) || - ((last_timestamp > timestamp) && ((last_timestamp - timestamp) > delay)); - } - MappedPtr value; size_t size; LRUQueueIterator queue_iterator; - Timestamp timestamp; }; using Cells = std::unordered_map; @@ -257,7 +247,6 @@ private: /// Total weight of values. size_t current_size = 0; const size_t max_size; - const Delay expiration_delay; std::atomic hits {0}; std::atomic misses {0}; @@ -273,7 +262,6 @@ private: } Cell & cell = it->second; - updateCellTimestamp(cell); /// Move the key to the end of the queue. The iterator remains valid. queue.splice(queue.end(), queue, cell.queue_iterator); @@ -303,18 +291,11 @@ private: cell.value = mapped; cell.size = cell.value ? weight_function(*cell.value) : 0; current_size += cell.size; - updateCellTimestamp(cell); - removeOverflow(cell.timestamp); + removeOverflow(); } - void updateCellTimestamp(Cell & cell) - { - if (expiration_delay != Delay::zero()) - cell.timestamp = Clock::now(); - } - - void removeOverflow(const Timestamp & last_timestamp) + void removeOverflow() { size_t current_weight_lost = 0; size_t queue_size = cells.size(); @@ -330,8 +311,6 @@ private: } const auto & cell = it->second; - if (!cell.expired(last_timestamp, expiration_delay)) - break; current_size -= cell.size; current_weight_lost += cell.size; diff --git a/dbms/src/Common/ProfileEvents.cpp b/dbms/src/Common/ProfileEvents.cpp index 6cbbc07d8d8..99723cccfb4 100644 --- a/dbms/src/Common/ProfileEvents.cpp +++ b/dbms/src/Common/ProfileEvents.cpp @@ -37,6 +37,8 @@ M(CreatedReadBufferOrdinary, "") \ M(CreatedReadBufferAIO, "") \ M(CreatedReadBufferAIOFailed, "") \ + M(CreatedReadBufferMMap, "") \ + M(CreatedReadBufferMMapFailed, "") \ M(CreatedWriteBufferOrdinary, "") \ M(CreatedWriteBufferAIO, "") \ M(CreatedWriteBufferAIOFailed, "") \ diff --git a/dbms/src/Common/StackTrace.cpp b/dbms/src/Common/StackTrace.cpp index 597ed2028fa..e43bc4c287e 100644 --- a/dbms/src/Common/StackTrace.cpp +++ b/dbms/src/Common/StackTrace.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -226,6 +227,7 @@ void StackTrace::tryCapture() size = 0; #if USE_UNWIND size = unw_backtrace(frames.data(), capacity); + __msan_unpoison(frames.data(), size * sizeof(frames[0])); #endif } @@ -328,3 +330,15 @@ std::string StackTrace::toString() const static SimpleCache func_cached; return func_cached(frames, offset, size); } + +std::string StackTrace::toString(void ** frames_, size_t offset, size_t size) +{ + __msan_unpoison(frames_, size * sizeof(*frames_)); + + StackTrace::Frames frames_copy{}; + for (size_t i = 0; i < size; ++i) + frames_copy[i] = frames_[i]; + + static SimpleCache func_cached; + return func_cached(frames_copy, offset, size); +} diff --git a/dbms/src/Common/StackTrace.h b/dbms/src/Common/StackTrace.h index 13147587a19..401c8344f2d 100644 --- a/dbms/src/Common/StackTrace.h +++ b/dbms/src/Common/StackTrace.h @@ -41,6 +41,8 @@ public: const Frames & getFrames() const; std::string toString() const; + static std::string toString(void ** frames, size_t offset, size_t size); + void toStringEveryLine(std::function callback) const; protected: diff --git a/dbms/src/Common/tests/CMakeLists.txt b/dbms/src/Common/tests/CMakeLists.txt index 36206cf55d6..bce78dbe4a6 100644 --- a/dbms/src/Common/tests/CMakeLists.txt +++ b/dbms/src/Common/tests/CMakeLists.txt @@ -13,9 +13,6 @@ target_link_libraries (sip_hash_perf PRIVATE clickhouse_common_io) add_executable (auto_array auto_array.cpp) target_link_libraries (auto_array PRIVATE clickhouse_common_io) -add_executable (lru_cache lru_cache.cpp) -target_link_libraries (lru_cache PRIVATE clickhouse_common_io) - add_executable (hash_table hash_table.cpp) target_link_libraries (hash_table PRIVATE clickhouse_common_io) diff --git a/dbms/src/Common/tests/lru_cache.cpp b/dbms/src/Common/tests/lru_cache.cpp deleted file mode 100644 index e50d6ad9786..00000000000 --- a/dbms/src/Common/tests/lru_cache.cpp +++ /dev/null @@ -1,317 +0,0 @@ -#include -#include - -#include -#include -#include -#include -#include - - -namespace -{ - -void run(); -void runTest(unsigned int num, const std::function & func); -bool test1(); -bool test2(); -bool test_concurrent(); - -#define ASSERT_CHECK(cond, res) \ -do \ -{ \ - if (!(cond)) \ - { \ - std::cout << __FILE__ << ":" << __LINE__ << ":" \ - << "Assertion " << #cond << " failed.\n"; \ - if ((res)) { (res) = false; } \ - } \ -} \ -while (0) - -void run() -{ - const std::vector> tests = - { - test1, - test2, - test_concurrent - }; - - unsigned int num = 0; - for (const auto & test : tests) - { - ++num; - runTest(num, test); - } -} - -void runTest(unsigned int num, const std::function & func) -{ - bool ok; - - try - { - ok = func(); - } - catch (const DB::Exception & ex) - { - ok = false; - std::cout << "Caught exception " << ex.displayText() << "\n"; - } - catch (const std::exception & ex) - { - ok = false; - std::cout << "Caught exception " << ex.what() << "\n"; - } - catch (...) - { - ok = false; - std::cout << "Caught unhandled exception\n"; - } - - if (ok) - std::cout << "Test " << num << " passed\n"; - else - std::cout << "Test " << num << " failed\n"; -} - -struct Weight -{ - size_t operator()(const std::string & s) const - { - return s.size(); - } -}; - -bool test1() -{ - using Cache = DB::LRUCache, Weight>; - using MappedPtr = Cache::MappedPtr; - - auto ptr = [](const std::string & s) - { - return MappedPtr(new std::string(s)); - }; - - Cache cache(10); - - bool res = true; - - ASSERT_CHECK(!cache.get("asd"), res); - - cache.set("asd", ptr("qwe")); - - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - - cache.set("zxcv", ptr("12345")); - cache.set("01234567891234567", ptr("--")); - - ASSERT_CHECK((*cache.get("zxcv") == "12345"), res); - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res); - ASSERT_CHECK(!cache.get("123x"), res); - - cache.set("321x", ptr("+")); - - ASSERT_CHECK(!cache.get("zxcv"), res); - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res); - ASSERT_CHECK(!cache.get("123x"), res); - ASSERT_CHECK((*cache.get("321x") == "+"), res); - - ASSERT_CHECK((cache.weight() == 6), res); - ASSERT_CHECK((cache.count() == 3), res); - - return res; -} - -bool test2() -{ - using namespace std::literals; - using Cache = DB::LRUCache, Weight>; - using MappedPtr = Cache::MappedPtr; - - auto ptr = [](const std::string & s) - { - return MappedPtr(new std::string(s)); - }; - - Cache cache(10, 3s); - - bool res = true; - - ASSERT_CHECK(!cache.get("asd"), res); - - cache.set("asd", ptr("qwe")); - - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - - cache.set("zxcv", ptr("12345")); - cache.set("01234567891234567", ptr("--")); - - ASSERT_CHECK((*cache.get("zxcv") == "12345"), res); - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res); - ASSERT_CHECK(!cache.get("123x"), res); - - cache.set("321x", ptr("+")); - - ASSERT_CHECK((cache.get("zxcv")), res); - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res); - ASSERT_CHECK(!cache.get("123x"), res); - ASSERT_CHECK((*cache.get("321x") == "+"), res); - - ASSERT_CHECK((cache.weight() == 11), res); - ASSERT_CHECK((cache.count() == 4), res); - - std::this_thread::sleep_for(5s); - - cache.set("123x", ptr("2769")); - - ASSERT_CHECK(!cache.get("zxcv"), res); - ASSERT_CHECK((*cache.get("asd") == "qwe"), res); - ASSERT_CHECK((*cache.get("01234567891234567") == "--"), res); - ASSERT_CHECK((*cache.get("321x") == "+"), res); - - ASSERT_CHECK((cache.weight() == 10), res); - ASSERT_CHECK((cache.count() == 4), res); - - return res; -} - -bool test_concurrent() -{ - using namespace std::literals; - - using Cache = DB::LRUCache, Weight>; - Cache cache(2); - - bool res = true; - - auto load_func = [](const std::string & result, std::chrono::seconds sleep_for, bool throw_exc) - { - std::this_thread::sleep_for(sleep_for); - if (throw_exc) - throw std::runtime_error("Exception!"); - return std::make_shared(result); - }; - - /// Case 1: Both threads are able to load the value. - - std::pair result1; - std::thread thread1([&]() - { - result1 = cache.getOrSet("key", [&]() { return load_func("val1", 1s, false); }); - }); - - std::pair result2; - std::thread thread2([&]() - { - result2 = cache.getOrSet("key", [&]() { return load_func("val2", 1s, false); }); - }); - - thread1.join(); - thread2.join(); - - ASSERT_CHECK((result1.first == result2.first), res); - ASSERT_CHECK((result1.second != result2.second), res); - - /// Case 2: One thread throws an exception during loading. - - cache.reset(); - - bool thrown = false; - thread1 = std::thread([&]() - { - try - { - cache.getOrSet("key", [&]() { return load_func("val1", 2s, true); }); - } - catch (...) - { - thrown = true; - } - }); - - thread2 = std::thread([&]() - { - std::this_thread::sleep_for(1s); - result2 = cache.getOrSet("key", [&]() { return load_func("val2", 1s, false); }); - }); - - thread1.join(); - thread2.join(); - - ASSERT_CHECK((thrown == true), res); - ASSERT_CHECK((result2.second == true), res); - ASSERT_CHECK((result2.first.get() == cache.get("key").get()), res); - ASSERT_CHECK((*result2.first == "val2"), res); - - /// Case 3: All threads throw an exception. - - cache.reset(); - - bool thrown1 = false; - thread1 = std::thread([&]() - { - try - { - cache.getOrSet("key", [&]() { return load_func("val1", 1s, true); }); - } - catch (...) - { - thrown1 = true; - } - }); - - bool thrown2 = false; - thread2 = std::thread([&]() - { - try - { - cache.getOrSet("key", [&]() { return load_func("val1", 1s, true); }); - } - catch (...) - { - thrown2 = true; - } - }); - - thread1.join(); - thread2.join(); - - ASSERT_CHECK((thrown1 == true), res); - ASSERT_CHECK((thrown2 == true), res); - ASSERT_CHECK((cache.get("key") == nullptr), res); - - /// Case 4: Concurrent reset. - - cache.reset(); - - thread1 = std::thread([&]() - { - result1 = cache.getOrSet("key", [&]() { return load_func("val1", 2s, false); }); - }); - - std::this_thread::sleep_for(1s); - cache.reset(); - - thread1.join(); - - ASSERT_CHECK((result1.second == true), res); - ASSERT_CHECK((*result1.first == "val1"), res); - ASSERT_CHECK((cache.get("key") == nullptr), res); - - return res; -} - -} - -int main() -{ - run(); - return 0; -} - diff --git a/dbms/src/Compression/CachedCompressedReadBuffer.cpp b/dbms/src/Compression/CachedCompressedReadBuffer.cpp index b39d04cf03f..a1bcb8a7d66 100644 --- a/dbms/src/Compression/CachedCompressedReadBuffer.cpp +++ b/dbms/src/Compression/CachedCompressedReadBuffer.cpp @@ -19,7 +19,7 @@ void CachedCompressedReadBuffer::initInput() { if (!file_in) { - file_in = createReadBufferFromFileBase(path, estimated_size, aio_threshold, buf_size); + file_in = createReadBufferFromFileBase(path, estimated_size, aio_threshold, mmap_threshold, buf_size); compressed_in = file_in.get(); if (profile_callback) @@ -73,10 +73,11 @@ bool CachedCompressedReadBuffer::nextImpl() CachedCompressedReadBuffer::CachedCompressedReadBuffer( - const std::string & path_, UncompressedCache * cache_, size_t estimated_size_, size_t aio_threshold_, + const std::string & path_, UncompressedCache * cache_, + size_t estimated_size_, size_t aio_threshold_, size_t mmap_threshold_, size_t buf_size_) : ReadBuffer(nullptr, 0), path(path_), cache(cache_), buf_size(buf_size_), estimated_size(estimated_size_), - aio_threshold(aio_threshold_), file_pos(0) + aio_threshold(aio_threshold_), mmap_threshold(mmap_threshold_), file_pos(0) { } diff --git a/dbms/src/Compression/CachedCompressedReadBuffer.h b/dbms/src/Compression/CachedCompressedReadBuffer.h index 174ddb98587..52ef750ff19 100644 --- a/dbms/src/Compression/CachedCompressedReadBuffer.h +++ b/dbms/src/Compression/CachedCompressedReadBuffer.h @@ -26,6 +26,7 @@ private: size_t buf_size; size_t estimated_size; size_t aio_threshold; + size_t mmap_threshold; std::unique_ptr file_in; size_t file_pos; @@ -42,7 +43,8 @@ private: public: CachedCompressedReadBuffer( - const std::string & path_, UncompressedCache * cache_, size_t estimated_size_, size_t aio_threshold_, + const std::string & path_, UncompressedCache * cache_, + size_t estimated_size_, size_t aio_threshold_, size_t mmap_threshold_, size_t buf_size_ = DBMS_DEFAULT_BUFFER_SIZE); diff --git a/dbms/src/Compression/CompressedReadBufferFromFile.cpp b/dbms/src/Compression/CompressedReadBufferFromFile.cpp index e413c5e1086..22eaf9e15e8 100644 --- a/dbms/src/Compression/CompressedReadBufferFromFile.cpp +++ b/dbms/src/Compression/CompressedReadBufferFromFile.cpp @@ -33,9 +33,9 @@ bool CompressedReadBufferFromFile::nextImpl() CompressedReadBufferFromFile::CompressedReadBufferFromFile( - const std::string & path, size_t estimated_size, size_t aio_threshold, size_t buf_size) + const std::string & path, size_t estimated_size, size_t aio_threshold, size_t mmap_threshold, size_t buf_size) : BufferWithOwnMemory(0), - p_file_in(createReadBufferFromFileBase(path, estimated_size, aio_threshold, buf_size)), + p_file_in(createReadBufferFromFileBase(path, estimated_size, aio_threshold, mmap_threshold, buf_size)), file_in(*p_file_in) { compressed_in = &file_in; diff --git a/dbms/src/Compression/CompressedReadBufferFromFile.h b/dbms/src/Compression/CompressedReadBufferFromFile.h index 288a66e321a..641e3d6ed1b 100644 --- a/dbms/src/Compression/CompressedReadBufferFromFile.h +++ b/dbms/src/Compression/CompressedReadBufferFromFile.h @@ -30,7 +30,7 @@ private: public: CompressedReadBufferFromFile( - const std::string & path, size_t estimated_size, size_t aio_threshold, size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE); + const std::string & path, size_t estimated_size, size_t aio_threshold, size_t mmap_threshold, size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE); void seek(size_t offset_in_compressed_file, size_t offset_in_decompressed_block); diff --git a/dbms/src/Compression/CompressionCodecDoubleDelta.cpp b/dbms/src/Compression/CompressionCodecDoubleDelta.cpp index 17eeba9a152..dc4a5084c83 100644 --- a/dbms/src/Compression/CompressionCodecDoubleDelta.cpp +++ b/dbms/src/Compression/CompressionCodecDoubleDelta.cpp @@ -26,7 +26,7 @@ extern const int CANNOT_DECOMPRESS; namespace { -Int64 getMaxValueForByteSize(UInt8 byte_size) +inline Int64 getMaxValueForByteSize(Int8 byte_size) { switch (byte_size) { @@ -51,11 +51,56 @@ struct WriteSpec const UInt8 data_bits; }; -const std::array DELTA_SIZES{7, 9, 12, 32, 64}; +// delta size prefix and data lengths based on few high bits peeked from binary stream +static const WriteSpec WRITE_SPEC_LUT[32] = { + // 0b0 - 1-bit prefix, no data to read + /* 00000 */ {1, 0b0, 0}, + /* 00001 */ {1, 0b0, 0}, + /* 00010 */ {1, 0b0, 0}, + /* 00011 */ {1, 0b0, 0}, + /* 00100 */ {1, 0b0, 0}, + /* 00101 */ {1, 0b0, 0}, + /* 00110 */ {1, 0b0, 0}, + /* 00111 */ {1, 0b0, 0}, + /* 01000 */ {1, 0b0, 0}, + /* 01001 */ {1, 0b0, 0}, + /* 01010 */ {1, 0b0, 0}, + /* 01011 */ {1, 0b0, 0}, + /* 01100 */ {1, 0b0, 0}, + /* 01101 */ {1, 0b0, 0}, + /* 01110 */ {1, 0b0, 0}, + /* 01111 */ {1, 0b0, 0}, + + // 0b10 - 2 bit prefix, 7 bits of data + /* 10000 */ {2, 0b10, 7}, + /* 10001 */ {2, 0b10, 7}, + /* 10010 */ {2, 0b10, 7}, + /* 10011 */ {2, 0b10, 7}, + /* 10100 */ {2, 0b10, 7}, + /* 10101 */ {2, 0b10, 7}, + /* 10110 */ {2, 0b10, 7}, + /* 10111 */ {2, 0b10, 7}, + + // 0b110 - 3 bit prefix, 9 bits of data + /* 11000 */ {3, 0b110, 9}, + /* 11001 */ {3, 0b110, 9}, + /* 11010 */ {3, 0b110, 9}, + /* 11011 */ {3, 0b110, 9}, + + // 0b1110 - 4 bit prefix, 12 bits of data + /* 11100 */ {4, 0b1110, 12}, + /* 11101 */ {4, 0b1110, 12}, + + // 5-bit prefixes + /* 11110 */ {5, 0b11110, 32}, + /* 11111 */ {5, 0b11111, 64}, +}; + template WriteSpec getDeltaWriteSpec(const T & value) { + // TODO: to speed up things a bit by counting number of leading zeroes instead of doing lots of comparisons if (value > -63 && value < 64) { return WriteSpec{2, 0b10, 7}; @@ -107,14 +152,15 @@ UInt32 getCompressedDataSize(UInt8 data_bytes_size, UInt32 uncompressed_size) template UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest) { - // Since only unsinged int has granted 2-compliment overflow handling, we are doing math here on unsigned types. - // To simplify and booletproof code, we operate enforce ValueType to be unsigned too. + // Since only unsinged int has granted 2-complement overflow handling, + // we are doing math here only on unsigned types. + // To simplify and booletproof code, we enforce ValueType to be unsigned too. static_assert(is_unsigned_v, "ValueType must be unsigned."); using UnsignedDeltaType = ValueType; // We use signed delta type to turn huge unsigned values into smaller signed: // ffffffff => -1 - using SignedDeltaType = typename std::make_signed::type; + using SignedDeltaType = typename std::make_signed_t; if (source_size % sizeof(ValueType) != 0) throw Exception("Cannot compress, data size " + toString(source_size) @@ -149,8 +195,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest) prev_value = curr_value; } - WriteBuffer buffer(dest, getCompressedDataSize(sizeof(ValueType), source_size - sizeof(ValueType)*2)); - BitWriter writer(buffer); + BitWriter writer(dest, getCompressedDataSize(sizeof(ValueType), source_size - sizeof(ValueType)*2)); int item = 2; for (; source < source_end; source += sizeof(ValueType), ++item) @@ -170,7 +215,8 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest) else { const SignedDeltaType signed_dd = static_cast(double_delta); - const auto sign = std::signbit(signed_dd); + const auto sign = signed_dd < 0; + // -1 shirnks dd down to fit into number of bits, and there can't be 0, so it is OK. const auto abs_value = static_cast(std::abs(signed_dd) - 1); const auto write_spec = getDeltaWriteSpec(signed_dd); @@ -183,7 +229,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest) writer.flush(); - return sizeof(items_count) + sizeof(prev_value) + sizeof(prev_delta) + buffer.count(); + return sizeof(items_count) + sizeof(prev_value) + sizeof(prev_delta) + writer.count() / 8; } template @@ -220,35 +266,28 @@ void decompressDataForType(const char * source, UInt32 source_size, char * dest) dest += sizeof(prev_value); } - ReadBufferFromMemory buffer(source, source_size - sizeof(prev_value) - sizeof(prev_delta) - sizeof(items_count)); - BitReader reader(buffer); + BitReader reader(source, source_size - sizeof(prev_value) - sizeof(prev_delta) - sizeof(items_count)); // since data is tightly packed, up to 1 bit per value, and last byte is padded with zeroes, // we have to keep track of items to avoid reading more that there is. for (UInt32 items_read = 2; items_read < items_count && !reader.eof(); ++items_read) { UnsignedDeltaType double_delta = 0; - if (reader.readBit() == 1) - { - UInt8 i = 0; - for (; i < sizeof(DELTA_SIZES) - 1; ++i) - { - const auto next_bit = reader.readBit(); - if (next_bit == 0) - { - break; - } - } + static_assert(sizeof(WRITE_SPEC_LUT)/sizeof(WRITE_SPEC_LUT[0]) == 32); // 5-bit prefix lookup table + const auto write_spec = WRITE_SPEC_LUT[reader.peekByte() >> (8 - 5)]; // only 5 high bits of peeked byte value + + reader.skipBufferedBits(write_spec.prefix_bits); // discard the prefix value, since we've already used it + if (write_spec.data_bits != 0) + { const UInt8 sign = reader.readBit(); - SignedDeltaType signed_dd = static_cast(reader.readBits(DELTA_SIZES[i] - 1) + 1); + SignedDeltaType signed_dd = static_cast(reader.readBits(write_spec.data_bits - 1) + 1); if (sign) { signed_dd *= -1; } double_delta = static_cast(signed_dd); } - // else if first bit is zero, no need to read more data. const UnsignedDeltaType delta = double_delta + prev_delta; const ValueType curr_value = prev_value + delta; diff --git a/dbms/src/Compression/CompressionCodecDoubleDelta.h b/dbms/src/Compression/CompressionCodecDoubleDelta.h index 3fdaf5f76d8..edbe2eea01d 100644 --- a/dbms/src/Compression/CompressionCodecDoubleDelta.h +++ b/dbms/src/Compression/CompressionCodecDoubleDelta.h @@ -5,6 +5,92 @@ namespace DB { +/** DoubleDelta column codec implementation. + * + * Based on Gorilla paper: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf, which was extended + * to support 64bit types. The drawback is 1 extra bit for 32-byte wide deltas: 5-bit prefix + * instead of 4-bit prefix. + * + * This codec is best used against monotonic integer sequences with constant (or almost contant) + * stride, like event timestamp for some monitoring application. + * + * Given input sequence a: [a0, a1, ... an]: + * + * First, write number of items (sizeof(int32)*8 bits): n + * Then write first item as is (sizeof(a[0])*8 bits): a[0] + * Second item is written as delta (sizeof(a[0])*8 bits): a[1] - a[0] + * Loop over remaining items and calculate double delta: + * double_delta = a[i] - 2 * a[i - 1] + a[i - 2] + * Write it in compact binary form with `BitWriter` + * if double_delta == 0: + * write 1bit: 0 + * else if -63 < double_delta < 64: + * write 2 bit prefix: 10 + * write sign bit (1 if signed): x + * write 7-1 bits of abs(double_delta - 1): xxxxxx + * else if -255 < double_delta < 256: + * write 3 bit prefix: 110 + * write sign bit (1 if signed): x + * write 9-1 bits of abs(double_delta - 1): xxxxxxxx + * else if -2047 < double_delta < 2048: + * write 4 bit prefix: 1110 + * write sign bit (1 if signed): x + * write 12-1 bits of abs(double_delta - 1): xxxxxxxxxxx + * else if double_delta fits into 32-bit int: + * write 5 bit prefix: 11110 + * write sign bit (1 if signed): x + * write 32-1 bits of abs(double_delta - 1): xxxxxxxxxxx... + * else + * write 5 bit prefix: 11111 + * write sign bit (1 if signed): x + * write 64-1 bits of abs(double_delta - 1): xxxxxxxxxxx... + * + * @example sequence of UInt8 values [1, 2, 3, 4, 5, 6, 7, 8, 9 10] is encoded as (codec header is ommited): + * + * .- 4-byte little-endian sequence length (10 == 0xa) + * | .- 1 byte (sizeof(UInt8) a[0] : 0x01 + * | | .- 1 byte of delta: a[1] - a[0] = 2 - 1 = 1 : 0x01 + * | | | .- 8 zero bits since double delta for remaining 8 elements was 0 : 0x00 + * v_______________v___v___v___ + * \x0a\x00\x00\x00\x01\x01\x00 + * + * @example sequence of Int16 values [-10, 10, -20, 20, -40, 40] is encoded as: + * + * .- 4-byte little endian sequence length = 6 : 0x00000006 + * | .- 2 bytes (sizeof(Int16) a[0] as UInt16 = -10 : 0xfff6 + * | | .- 2 bytes of delta: a[1] - a[0] = 10 - (-10) = 20 : 0x0014 + * | | | .- 4 encoded double deltas (see below) + * v_______________ v______ v______ v______________________ + * \x06\x00\x00\x00\xf6\xff\x14\x00\xb8\xe2\x2e\xb1\xe4\x58 + * + * 4 binary encoded double deltas (\xb8\xe2\x2e\xb1\xe4\x58): + * double_delta (DD) = -20 - 2 * 10 + (-10) = -50 + * .- 2-bit prefix : 0b10 + * | .- sign-bit : 0b1 + * | |.- abs(DD - 1) = 49 : 0b110001 + * | || + * | || DD = 20 - 2 * (-20) + 10 = 70 + * | || .- 3-bit prefix : 0b110 + * | || | .- sign bit : 0b0 + * | || | |.- abs(DD - 1) = 69 : 0b1000101 + * | || | || + * | || | || DD = -40 - 2 * 20 + (-20) = -100 + * | || | || .- 3-bit prefix : 0b110 + * | || | || | .- sign-bit : 0b0 + * | || | || | |.- abs(DD - 1) = 99 : 0b1100011 + * | || | || | || + * | || | || | || DD = 40 - 2 * (-40) + 20 = 140 + * | || | || | || .- 3-bit prefix : 0b110 + * | || | || | || | .- sign bit : 0b0 + * | || | || | || | |.- abs(DD - 1) = 139 : 0b10001011 + * | || | || | || | || + * V_vv______V__vv________V____vv_______V__vv________,- padding bits + * 10111000 11100010 00101110 10110001 11100100 01011000 + * + * Please also see unit tests for: + * * Examples on what output `BitWriter` produces on predefined input. + * * Compatibility tests solidifying encoded binary output on set of predefined sequences. + */ class CompressionCodecDoubleDelta : public ICompressionCodec { public: diff --git a/dbms/src/Compression/CompressionCodecGorilla.cpp b/dbms/src/Compression/CompressionCodecGorilla.cpp index 574e40b06bf..62e7a81aae9 100644 --- a/dbms/src/Compression/CompressionCodecGorilla.cpp +++ b/dbms/src/Compression/CompressionCodecGorilla.cpp @@ -112,8 +112,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest, dest += sizeof(prev_value); } - WriteBuffer buffer(dest, dest_end - dest); - BitWriter writer(buffer); + BitWriter writer(dest, dest_end - dest); while (source < source_end) { @@ -148,7 +147,7 @@ UInt32 compressDataForType(const char * source, UInt32 source_size, char * dest, writer.flush(); - return sizeof(items_count) + sizeof(prev_value) + buffer.count(); + return sizeof(items_count) + sizeof(prev_value) + writer.count() / 8; } template @@ -174,8 +173,7 @@ void decompressDataForType(const char * source, UInt32 source_size, char * dest) dest += sizeof(prev_value); } - ReadBufferFromMemory buffer(source, source_size - sizeof(items_count) - sizeof(prev_value)); - BitReader reader(buffer); + BitReader reader(source, source_size - sizeof(items_count) - sizeof(prev_value)); binary_value_info prev_xored_info{0, 0, 0}; diff --git a/dbms/src/Compression/CompressionCodecGorilla.h b/dbms/src/Compression/CompressionCodecGorilla.h index 0bbd220cb59..94941b7aa06 100644 --- a/dbms/src/Compression/CompressionCodecGorilla.h +++ b/dbms/src/Compression/CompressionCodecGorilla.h @@ -5,6 +5,89 @@ namespace DB { +/** Gorilla column codec implementation. + * + * Based on Gorilla paper: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf + * + * This codec is best used against monotonic floating sequences, like CPU usage percentage + * or any other gauge. + * + * Given input sequence a: [a0, a1, ... an] + * + * First, write number of items (sizeof(int32)*8 bits): n + * Then write first item as is (sizeof(a[0])*8 bits): a[0] + * Loop over remaining items and calculate xor_diff: + * xor_diff = a[i] ^ a[i - 1] (e.g. 00000011'10110100) + * Write it in compact binary form with `BitWriter` + * if xor_diff == 0: + * write 1 bit: 0 + * else: + * calculate leading zero bits (lzb) + * and trailing zero bits (tzb) of xor_diff, + * compare to lzb and tzb of previous xor_diff + * (X = sizeof(a[i]) * 8, e.g. X = 16, lzb = 6, tzb = 2) + * if lzb >= prev_lzb && tzb >= prev_tzb: + * (e.g. prev_lzb=4, prev_tzb=1) + * write 2 bit prefix: 0b10 + * write xor_diff >> prev_tzb (X - prev_lzb - prev_tzb bits):0b00111011010 + * (where X = sizeof(a[i]) * 8, e.g. 16) + * else: + * write 2 bit prefix: 0b11 + * write 5 bits of lzb: 0b00110 + * write 6 bits of (X - lzb - tzb)=(16-6-2)=8: 0b001000 + * write (X - lzb - tzb) non-zero bits of xor_diff: 0b11101101 + * prev_lzb = lzb + * prev_tzb = tzb + * + * @example sequence of Float32 values [0.1, 0.1, 0.11, 0.2, 0.1] is encoded as: + * + * .- 4-byte little endian sequence length: 5 : 0x00000005 + * | .- 4 byte (sizeof(Float32) a[0] as UInt32 : -10 : 0xcdcccc3d + * | | .- 4 encoded xor diffs (see below) + * v_______________ v______________ v__________________________________________________ + * \x05\x00\x00\x00\xcd\xcc\xcc\x3d\x6a\x5a\xd8\xb6\x3c\xcd\x75\xb1\x6c\x77\x00\x00\x00 + * + * 4 binary encoded xor diffs (\x6a\x5a\xd8\xb6\x3c\xcd\x75\xb1\x6c\x77\x00\x00\x00): + * + * ........................................... + * a[i-1] = 00111101110011001100110011001101 + * a[i] = 00111101110011001100110011001101 + * xor_diff = 00000000000000000000000000000000 + * .- 1-bit prefix : 0b0 + * | + * | ........................................... + * | a[i-1] = 00111101110011001100110011001101 + * ! a[i] = 00111101111000010100011110101110 + * | xor_diff = 00000000001011011000101101100011 + * | lzb = 10 + * | tzb = 0 + * |.- 2-bit prefix : 0b11 + * || .- lzb (10) : 0b1010 + * || | .- data length (32-10-0): 22 : 0b010110 + * || | | .- data : 0b1011011000101101100011 + * || | | | + * || | | | ........................................... + * || | | | a[i-1] = 00111101111000010100011110101110 + * || | | | a[i] = 00111110010011001100110011001101 + * || | | | xor_diff = 00000011101011011000101101100011 + * || | | | .- 2-bit prefix : 0b11 + * || | | | | .- lzb = 6 : 0b00110 + * || | | | | | .- data length = (32 - 6) = 26 : 0b011010 + * || | | | | | | .- data : 0b11101011011000101101100011 + * || | | | | | | | + * || | | | | | | | ........................................... + * || | | | | | | | a[i-1] = 00111110010011001100110011001101 + * || | | | | | | | a[i] = 00111101110011001100110011001101 + * || | | | | | | | xor_diff = 00000011100000000000000000000000 + * || | | | | | | | .- 2-bit prefix : 0b10 + * || | | | | | | | | .- data : 0b11100000000000000000000000 + * VV_v____ v_____v________________________V_v_____v______v____________________________V_v_____________________________ + * 01101010 01011010 11011000 10110110 00111100 11001101 01110101 10110001 01101100 01110111 00000000 00000000 00000000 + * + * Please also see unit tests for: + * * Examples on what output `BitWriter` produces on predefined input. + * * Compatibility tests solidifying encoded binary output on set of predefined sequences. + */ class CompressionCodecGorilla : public ICompressionCodec { public: diff --git a/dbms/src/Compression/tests/cached_compressed_read_buffer.cpp b/dbms/src/Compression/tests/cached_compressed_read_buffer.cpp index fb30d691745..01dcd5a9fcd 100644 --- a/dbms/src/Compression/tests/cached_compressed_read_buffer.cpp +++ b/dbms/src/Compression/tests/cached_compressed_read_buffer.cpp @@ -32,7 +32,7 @@ int main(int argc, char ** argv) { Stopwatch watch; - CachedCompressedReadBuffer in(path, &cache, 0, 0); + CachedCompressedReadBuffer in(path, &cache, 0, 0, 0); WriteBufferFromFile out("/dev/null"); copyData(in, out); @@ -44,7 +44,7 @@ int main(int argc, char ** argv) { Stopwatch watch; - CachedCompressedReadBuffer in(path, &cache, 0, 0); + CachedCompressedReadBuffer in(path, &cache, 0, 0, 0); WriteBufferFromFile out("/dev/null"); copyData(in, out); diff --git a/dbms/src/Compression/tests/gtest_compressionCodec.cpp b/dbms/src/Compression/tests/gtest_compressionCodec.cpp index 32fff70d564..8693ce86c2a 100644 --- a/dbms/src/Compression/tests/gtest_compressionCodec.cpp +++ b/dbms/src/Compression/tests/gtest_compressionCodec.cpp @@ -1,6 +1,7 @@ #include #include +#include #include #include #include @@ -62,6 +63,32 @@ std::vector operator+(std::vector && left, std::vector && right) namespace { +template +struct AsHexStringHelper +{ + const T & container; +}; + +template +std::ostream & operator << (std::ostream & ostr, const AsHexStringHelper & helper) +{ + ostr << std::hex; + for (const auto & e : helper.container) + { + ostr << "\\x" << std::setw(2) << std::setfill('0') << (static_cast(e) & 0xFF); + } + + return ostr; +} + +template +AsHexStringHelper AsHexString(const T & container) +{ + static_assert (sizeof(container[0]) == 1 && std::is_pod>::value, "Only works on containers of byte-size PODs."); + + return AsHexStringHelper{container}; +} + template std::string bin(const T & value, size_t bits = sizeof(T)*8) { @@ -113,10 +140,71 @@ DataTypePtr makeDataType() #undef MAKE_DATA_TYPE - assert(false && "unsupported size"); + assert(false && "unknown datatype"); return nullptr; } +template +class BinaryDataAsSequenceOfValuesIterator +{ + const Container & container; + const void * data; + const void * data_end; + + T current_value; + +public: + using Self = BinaryDataAsSequenceOfValuesIterator; + + explicit BinaryDataAsSequenceOfValuesIterator(const Container & container_) + : container(container_), + data(&container[0]), + data_end(reinterpret_cast(data) + container.size()), + current_value(T{}) + { + static_assert(sizeof(container[0]) == 1 && std::is_pod>::value, "Only works on containers of byte-size PODs."); + read(); + } + + const T & operator*() const + { + return current_value; + } + + size_t ItemsLeft() const + { + return reinterpret_cast(data_end) - reinterpret_cast(data); + } + + Self & operator++() + { + read(); + return *this; + } + + operator bool() const + { + return ItemsLeft() > 0; + } + +private: + void read() + { + if (!*this) + { + throw std::runtime_error("No more data to read"); + } + + current_value = unalignedLoad(data); + data = reinterpret_cast(data) + sizeof(T); + } +}; + +template +BinaryDataAsSequenceOfValuesIterator AsSequenceOf(const Container & container) +{ + return BinaryDataAsSequenceOfValuesIterator(container); +} template ::testing::AssertionResult EqualByteContainersAs(const ContainerLeft & left, const ContainerRight & right) @@ -126,9 +214,6 @@ template ::testing::AssertionResult result = ::testing::AssertionSuccess(); - ReadBufferFromMemory left_read_buffer(left.data(), left.size()); - ReadBufferFromMemory right_read_buffer(right.data(), right.size()); - const auto l_size = left.size() / sizeof(T); const auto r_size = right.size() / sizeof(T); const auto size = std::min(l_size, r_size); @@ -137,16 +222,25 @@ template { result = ::testing::AssertionFailure() << "size mismatch" << " expected: " << l_size << " got:" << r_size; } + if (l_size == 0 || r_size == 0) + { + return result; + } + + auto l = AsSequenceOf(left); + auto r = AsSequenceOf(right); const auto MAX_MISMATCHING_ITEMS = 5; int mismatching_items = 0; - for (int i = 0; i < size; ++i) - { - T left_value{}; - left_read_buffer.readStrict(reinterpret_cast(&left_value), sizeof(left_value)); + size_t i = 0; - T right_value{}; - right_read_buffer.readStrict(reinterpret_cast(&right_value), sizeof(right_value)); + while (l && r) + { + const auto left_value = *l; + const auto right_value = *r; + ++l; + ++r; + ++i; if (left_value != right_value) { @@ -157,25 +251,47 @@ template if (++mismatching_items <= MAX_MISMATCHING_ITEMS) { - result << "mismatching " << sizeof(T) << "-byte item #" << i + result << "\nmismatching " << sizeof(T) << "-byte item #" << i << "\nexpected: " << bin(left_value) << " (0x" << std::hex << left_value << ")" - << "\ngot : " << bin(right_value) << " (0x" << std::hex << right_value << ")" - << std::endl; + << "\ngot : " << bin(right_value) << " (0x" << std::hex << right_value << ")"; if (mismatching_items == MAX_MISMATCHING_ITEMS) { - result << "..." << std::endl; + result << "\n..." << std::endl; } } } } if (mismatching_items > 0) { - result << "\ntotal mismatching items:" << mismatching_items << " of " << size; + result << "total mismatching items:" << mismatching_items << " of " << size; } return result; } +template +::testing::AssertionResult EqualByteContainers(UInt8 element_size, const ContainerLeft & left, const ContainerRight & right) +{ + switch (element_size) + { + case 1: + return EqualByteContainersAs(left, right); + break; + case 2: + return EqualByteContainersAs(left, right); + break; + case 4: + return EqualByteContainersAs(left, right); + break; + case 8: + return EqualByteContainersAs(left, right); + break; + default: + assert(false && "Invalid element_size"); + return ::testing::AssertionFailure() << "Invalid element_size: " << element_size; + } +} + struct Codec { std::string codec_statement; @@ -185,10 +301,6 @@ struct Codec : codec_statement(std::move(codec_statement_)), expected_compression_ratio(expected_compression_ratio_) {} - - Codec() - : Codec(std::string()) - {} }; @@ -198,36 +310,28 @@ struct CodecTestSequence std::vector serialized_data; DataTypePtr data_type; - CodecTestSequence() - : name(), - serialized_data(), - data_type() - {} - CodecTestSequence(std::string name_, std::vector serialized_data_, DataTypePtr data_type_) : name(name_), serialized_data(serialized_data_), data_type(data_type_) {} - CodecTestSequence(const CodecTestSequence &) = default; - CodecTestSequence & operator=(const CodecTestSequence &) = default; - CodecTestSequence(CodecTestSequence &&) = default; - CodecTestSequence & operator=(CodecTestSequence &&) = default; + CodecTestSequence & append(const CodecTestSequence & other) + { + assert(data_type->equals(*other.data_type)); + + serialized_data.insert(serialized_data.end(), other.serialized_data.begin(), other.serialized_data.end()); + if (!name.empty()) + name += " + "; + name += other.name; + + return *this; + } }; -CodecTestSequence operator+(CodecTestSequence && left, CodecTestSequence && right) +CodecTestSequence operator+(CodecTestSequence && left, const CodecTestSequence & right) { - assert(left.data_type->equals(*right.data_type)); - - std::vector data(std::move(left.serialized_data)); - data.insert(data.end(), right.serialized_data.begin(), right.serialized_data.end()); - - return CodecTestSequence{ - left.name + " + " + right.name, - std::move(data), - std::move(left.data_type) - }; + return left.append(right); } template @@ -288,17 +392,22 @@ CodecTestSequence makeSeq(Args && ... args) }; } -template -CodecTestSequence generateSeq(Generator gen, const char* gen_name, size_t Begin = 0, size_t End = 10000) +template +CodecTestSequence generateSeq(Generator gen, const char* gen_name, B Begin = 0, E End = 10000) { - assert (End >= Begin); - + const auto direction = std::signbit(End - Begin) ? -1 : 1; std::vector data(sizeof(T) * (End - Begin)); char * write_pos = data.data(); - for (size_t i = Begin; i < End; ++i) + for (auto i = Begin; i < End; i += direction) { const T v = gen(static_cast(i)); + +// if constexpr (debug_log_items) +// { +// std::cerr << "#" << i << " " << type_name() << "(" << sizeof(T) << " bytes) : " << v << std::endl; +// } + unalignedStore(write_pos, v); write_pos += sizeof(v); } @@ -310,6 +419,96 @@ CodecTestSequence generateSeq(Generator gen, const char* gen_name, size_t Begin }; } +struct NoOpTimer +{ + void start() {} + void report(const char*) {} +}; + +struct StopwatchTimer +{ + explicit StopwatchTimer(clockid_t clock_type, size_t estimated_marks = 32) + : stopwatch(clock_type) + { + results.reserve(estimated_marks); + } + + void start() + { + stopwatch.restart(); + } + + void report(const char * mark) + { + results.emplace_back(mark, stopwatch.elapsed()); + } + + void stop() + { + stopwatch.stop(); + } + + const std::vector> & getResults() const + { + return results; + } + +private: + Stopwatch stopwatch; + std::vector> results; +}; + +CompressionCodecPtr makeCodec(const std::string & codec_string, const DataTypePtr data_type) +{ + const std::string codec_statement = "(" + codec_string + ")"; + Tokens tokens(codec_statement.begin().base(), codec_statement.end().base()); + IParser::Pos token_iterator(tokens); + + Expected expected; + ASTPtr codec_ast; + ParserCodec parser; + + parser.parse(token_iterator, codec_ast, expected); + + return CompressionCodecFactory::instance().get(codec_ast, data_type); +} + +template +void testTranscoding(Timer & timer, ICompressionCodec & codec, const CodecTestSequence & test_sequence, std::optional expected_compression_ratio = std::optional{}) +{ + const auto & source_data = test_sequence.serialized_data; + + const UInt32 encoded_max_size = codec.getCompressedReserveSize(source_data.size()); + PODArray encoded(encoded_max_size); + + timer.start(); + + const UInt32 encoded_size = codec.compress(source_data.data(), source_data.size(), encoded.data()); + timer.report("encoding"); + + encoded.resize(encoded_size); + + PODArray decoded(source_data.size()); + + timer.start(); + const UInt32 decoded_size = codec.decompress(encoded.data(), encoded.size(), decoded.data()); + timer.report("decoding"); + + decoded.resize(decoded_size); + + ASSERT_TRUE(EqualByteContainers(test_sequence.data_type->getSizeOfValueInMemory(), source_data, decoded)); + + const auto header_size = codec.getHeaderSize(); + const auto compression_ratio = (encoded_size - header_size) / (source_data.size() * 1.0); + + if (expected_compression_ratio) + { + ASSERT_LE(compression_ratio, *expected_compression_ratio) + << "\n\tdecoded size: " << source_data.size() + << "\n\tencoded size: " << encoded_size + << "(no header: " << encoded_size - header_size << ")"; + } +} class CodecTest : public ::testing::TestWithParam> { @@ -320,67 +519,18 @@ public: CODEC_WITHOUT_DATA_TYPE, }; - CompressionCodecPtr makeCodec(MakeCodecParam with_data_type) const + CompressionCodecPtr makeCodec(MakeCodecParam with_data_type) { const auto & codec_string = std::get<0>(GetParam()).codec_statement; const auto & data_type = with_data_type == CODEC_WITH_DATA_TYPE ? std::get<1>(GetParam()).data_type : nullptr; - const std::string codec_statement = "(" + codec_string + ")"; - Tokens tokens(codec_statement.begin().base(), codec_statement.end().base()); - IParser::Pos token_iterator(tokens); - - Expected expected; - ASTPtr codec_ast; - ParserCodec parser; - - parser.parse(token_iterator, codec_ast, expected); - - return CompressionCodecFactory::instance().get(codec_ast, data_type); + return ::makeCodec(codec_string, data_type); } void testTranscoding(ICompressionCodec & codec) { - const auto & test_sequence = std::get<1>(GetParam()); - const auto & source_data = test_sequence.serialized_data; - - const UInt32 encoded_max_size = codec.getCompressedReserveSize(source_data.size()); - PODArray encoded(encoded_max_size); - - const UInt32 encoded_size = codec.compress(source_data.data(), source_data.size(), encoded.data()); - encoded.resize(encoded_size); - - PODArray decoded(source_data.size()); - const UInt32 decoded_size = codec.decompress(encoded.data(), encoded.size(), decoded.data()); - decoded.resize(decoded_size); - - switch (test_sequence.data_type->getSizeOfValueInMemory()) - { - case 1: - ASSERT_TRUE(EqualByteContainersAs(source_data, decoded)); - break; - case 2: - ASSERT_TRUE(EqualByteContainersAs(source_data, decoded)); - break; - case 4: - ASSERT_TRUE(EqualByteContainersAs(source_data, decoded)); - break; - case 8: - ASSERT_TRUE(EqualByteContainersAs(source_data, decoded)); - break; - default: - FAIL() << "Invalid test sequence data type: " << test_sequence.data_type->getName(); - } - const auto header_size = codec.getHeaderSize(); - const auto compression_ratio = (encoded_size - header_size) / (source_data.size() * 1.0); - - const auto & codec_spec = std::get<0>(GetParam()); - if (codec_spec.expected_compression_ratio) - { - ASSERT_LE(compression_ratio, *codec_spec.expected_compression_ratio) - << "\n\tdecoded size: " << source_data.size() - << "\n\tencoded size: " << encoded_size - << "(no header: " << encoded_size - header_size << ")"; - } + NoOpTimer timer; + ::testTranscoding(timer, codec, std::get<1>(GetParam()), std::get<0>(GetParam()).expected_compression_ratio); } }; @@ -396,10 +546,121 @@ TEST_P(CodecTest, TranscodingWithoutDataType) testTranscoding(*codec); } +// Param is tuple-of-tuple to simplify instantiating with values, since typically group of cases test only one codec. +class CodecTest_Compatibility : public ::testing::TestWithParam>> +{}; + +// Check that iput sequence when encoded matches the encoded string binary. +TEST_P(CodecTest_Compatibility, Encoding) +{ + const auto & codec_spec = std::get<0>(GetParam()); + const auto & [data_sequence, expected] = std::get<1>(GetParam()); + const auto codec = makeCodec(codec_spec.codec_statement, data_sequence.data_type); + + const auto & source_data = data_sequence.serialized_data; + + // Just encode the data with codec + const UInt32 encoded_max_size = codec->getCompressedReserveSize(source_data.size()); + PODArray encoded(encoded_max_size); + + const UInt32 encoded_size = codec->compress(source_data.data(), source_data.size(), encoded.data()); + encoded.resize(encoded_size); + SCOPED_TRACE(::testing::Message("encoded: ") << AsHexString(encoded)); + + ASSERT_TRUE(EqualByteContainersAs(expected, encoded)); +} + +// Check that binary string is exactly decoded into input sequence. +TEST_P(CodecTest_Compatibility, Decoding) +{ + const auto & codec_spec = std::get<0>(GetParam()); + const auto & [expected, encoded_data] = std::get<1>(GetParam()); + const auto codec = makeCodec(codec_spec.codec_statement, expected.data_type); + + PODArray decoded(expected.serialized_data.size()); + const UInt32 decoded_size = codec->decompress(encoded_data.c_str(), encoded_data.size(), decoded.data()); + decoded.resize(decoded_size); + + ASSERT_TRUE(EqualByteContainers(expected.data_type->getSizeOfValueInMemory(), expected.serialized_data, decoded)); +} + +class CodecTest_Performance : public ::testing::TestWithParam> +{}; + +TEST_P(CodecTest_Performance, TranscodingWithDataType) +{ + const auto & [codec_spec, test_seq] = GetParam(); + const auto codec = ::makeCodec(codec_spec.codec_statement, test_seq.data_type); + + const auto runs = 10; + std::map> results; + + for (size_t i = 0; i < runs; ++i) + { + StopwatchTimer timer{CLOCK_THREAD_CPUTIME_ID}; + ::testTranscoding(timer, *codec, test_seq); + timer.stop(); + + for (const auto & [label, value] : timer.getResults()) + { + results[label].push_back(value); + } + } + + auto computeMeanAndStdDev = [](const auto & values) + { + double mean{}; + + if (values.size() < 2) + return std::make_tuple(mean, double{}); + + using ValueType = typename std::decay_t::value_type; + std::vector tmp_v(std::begin(values), std::end(values)); + std::sort(tmp_v.begin(), tmp_v.end()); + + // remove min and max + tmp_v.erase(tmp_v.begin()); + tmp_v.erase(tmp_v.end() - 1); + + for (const auto & v : tmp_v) + { + mean += v; + } + + mean = mean / tmp_v.size(); + double std_dev = 0.0; + for (const auto & v : tmp_v) + { + const auto d = (v - mean); + std_dev += (d * d); + } + std_dev = std::sqrt(std_dev / tmp_v.size()); + + return std::make_tuple(mean, std_dev); + }; + + std::cerr << codec_spec.codec_statement + << " " << test_seq.data_type->getName() + << " (" << test_seq.serialized_data.size() << " bytes, " + << std::hex << CityHash_v1_0_2::CityHash64(test_seq.serialized_data.data(), test_seq.serialized_data.size()) << std::dec + << ", average of " << runs << " runs, μs)"; + + for (const auto & k : {"encoding", "decoding"}) + { + const auto & values = results[k]; + const auto & [mean, std_dev] = computeMeanAndStdDev(values); + // Ensure that Coefficient of variation is reasonably low, otherwise these numbers are meaningless + EXPECT_GT(0.05, std_dev / mean); + std::cerr << "\t" << std::fixed << std::setprecision(1) << mean / 1000.0; + } + + std::cerr << std::endl; +} + /////////////////////////////////////////////////////////////////////////////////////////////////// // Here we use generators to produce test payload for codecs. // Generator is a callable that can produce infinite number of values, -// output value MUST be of the same type input value. +// output value MUST be of the same type as input value. /////////////////////////////////////////////////////////////////////////////////////////////////// auto SameValueGenerator = [](auto value) @@ -543,7 +804,6 @@ std::vector generatePyramidOfSequences(const size_t sequences return sequences; }; - // helper macro to produce human-friendly sequence name from generator #define G(generator) generator, #generator @@ -560,22 +820,22 @@ const auto DefaultCodecsToTest = ::testing::Values( // test cases /////////////////////////////////////////////////////////////////////////////////////////////////// -INSTANTIATE_TEST_CASE_P(Simple, +INSTANTIATE_TEST_SUITE_P(Simple, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( makeSeq(1, 2, 3, 5, 7, 11, 13, 17, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SmallSequences, +INSTANTIATE_TEST_SUITE_P(SmallSequences, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::ValuesIn( - generatePyramidOfSequences(42, G(SequentialGenerator(1))) + generatePyramidOfSequences(42, G(SequentialGenerator(1))) + generatePyramidOfSequences(42, G(SequentialGenerator(1))) + generatePyramidOfSequences(42, G(SequentialGenerator(1))) + generatePyramidOfSequences(42, G(SequentialGenerator(1))) @@ -584,10 +844,10 @@ INSTANTIATE_TEST_CASE_P(SmallSequences, + generatePyramidOfSequences(42, G(SequentialGenerator(1))) + generatePyramidOfSequences(42, G(SequentialGenerator(1))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(Mixed, +INSTANTIATE_TEST_SUITE_P(Mixed, CodecTest, ::testing::Combine( DefaultCodecsToTest, @@ -601,15 +861,15 @@ INSTANTIATE_TEST_CASE_P(Mixed, generateSeq(G(MinMaxGenerator()), 1, 5) + generateSeq(G(SequentialGenerator(1)), 1, 1001), generateSeq(G(MinMaxGenerator()), 1, 5) + generateSeq(G(SequentialGenerator(1)), 1, 1001) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SameValueInt, +INSTANTIATE_TEST_SUITE_P(SameValueInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(SameValueGenerator(1000))), + generateSeq(G(SameValueGenerator(1000))), generateSeq(G(SameValueGenerator(1000))), generateSeq(G(SameValueGenerator(1000))), generateSeq(G(SameValueGenerator(1000))), @@ -618,15 +878,15 @@ INSTANTIATE_TEST_CASE_P(SameValueInt, generateSeq(G(SameValueGenerator(1000))), generateSeq(G(SameValueGenerator(1000))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SameNegativeValueInt, +INSTANTIATE_TEST_SUITE_P(SameNegativeValueInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(SameValueGenerator(-1000))), + generateSeq(G(SameValueGenerator(-1000))), generateSeq(G(SameValueGenerator(-1000))), generateSeq(G(SameValueGenerator(-1000))), generateSeq(G(SameValueGenerator(-1000))), @@ -635,10 +895,10 @@ INSTANTIATE_TEST_CASE_P(SameNegativeValueInt, generateSeq(G(SameValueGenerator(-1000))), generateSeq(G(SameValueGenerator(-1000))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SameValueFloat, +INSTANTIATE_TEST_SUITE_P(SameValueFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -649,10 +909,10 @@ INSTANTIATE_TEST_CASE_P(SameValueFloat, generateSeq(G(SameValueGenerator(M_E))), generateSeq(G(SameValueGenerator(M_E))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SameNegativeValueFloat, +INSTANTIATE_TEST_SUITE_P(SameNegativeValueFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -663,15 +923,15 @@ INSTANTIATE_TEST_CASE_P(SameNegativeValueFloat, generateSeq(G(SameValueGenerator(-1 * M_E))), generateSeq(G(SameValueGenerator(-1 * M_E))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SequentialInt, +INSTANTIATE_TEST_SUITE_P(SequentialInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(SequentialGenerator(1))), + generateSeq(G(SequentialGenerator(1))), generateSeq(G(SequentialGenerator(1))), generateSeq(G(SequentialGenerator(1))), generateSeq(G(SequentialGenerator(1))), @@ -680,17 +940,17 @@ INSTANTIATE_TEST_CASE_P(SequentialInt, generateSeq(G(SequentialGenerator(1))), generateSeq(G(SequentialGenerator(1))) ) - ), + ) ); // -1, -2, -3, ... etc for signed // 0xFF, 0xFE, 0xFD, ... for unsigned -INSTANTIATE_TEST_CASE_P(SequentialReverseInt, +INSTANTIATE_TEST_SUITE_P(SequentialReverseInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(SequentialGenerator(-1))), + generateSeq(G(SequentialGenerator(-1))), generateSeq(G(SequentialGenerator(-1))), generateSeq(G(SequentialGenerator(-1))), generateSeq(G(SequentialGenerator(-1))), @@ -699,10 +959,10 @@ INSTANTIATE_TEST_CASE_P(SequentialReverseInt, generateSeq(G(SequentialGenerator(-1))), generateSeq(G(SequentialGenerator(-1))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SequentialFloat, +INSTANTIATE_TEST_SUITE_P(SequentialFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -713,10 +973,10 @@ INSTANTIATE_TEST_CASE_P(SequentialFloat, generateSeq(G(SequentialGenerator(M_E))), generateSeq(G(SequentialGenerator(M_E))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(SequentialReverseFloat, +INSTANTIATE_TEST_SUITE_P(SequentialReverseFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -727,44 +987,44 @@ INSTANTIATE_TEST_CASE_P(SequentialReverseFloat, generateSeq(G(SequentialGenerator(-1 * M_E))), generateSeq(G(SequentialGenerator(-1 * M_E))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(MonotonicInt, +INSTANTIATE_TEST_SUITE_P(MonotonicInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(MonotonicGenerator(1, 5))), - generateSeq(G(MonotonicGenerator(1, 5))), - generateSeq(G(MonotonicGenerator(1, 5))), - generateSeq(G(MonotonicGenerator(1, 5))), + generateSeq(G(MonotonicGenerator(1, 5))), + generateSeq(G(MonotonicGenerator(1, 5))), + generateSeq(G(MonotonicGenerator(1, 5))), + generateSeq(G(MonotonicGenerator(1, 5))), generateSeq(G(MonotonicGenerator(1, 5))), generateSeq(G(MonotonicGenerator(1, 5))), generateSeq(G(MonotonicGenerator(1, 5))), generateSeq(G(MonotonicGenerator(1, 5))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(MonotonicReverseInt, +INSTANTIATE_TEST_SUITE_P(MonotonicReverseInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, ::testing::Values( - generateSeq(G(MonotonicGenerator(-1, 5))), - generateSeq(G(MonotonicGenerator(-1, 5))), - generateSeq(G(MonotonicGenerator(-1, 5))), - generateSeq(G(MonotonicGenerator(-1, 5))), - generateSeq(G(MonotonicGenerator(-1, 5))), + generateSeq(G(MonotonicGenerator(-1, 5))), + generateSeq(G(MonotonicGenerator(-1, 5))), + generateSeq(G(MonotonicGenerator(-1, 5))), + generateSeq(G(MonotonicGenerator(-1, 5))), + generateSeq(G(MonotonicGenerator(-1, 5))), generateSeq(G(MonotonicGenerator(-1, 5))), generateSeq(G(MonotonicGenerator(-1, 5))), generateSeq(G(MonotonicGenerator(-1, 5))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(MonotonicFloat, +INSTANTIATE_TEST_SUITE_P(MonotonicFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -774,10 +1034,10 @@ INSTANTIATE_TEST_CASE_P(MonotonicFloat, generateSeq(G(MonotonicGenerator(M_E, 5))), generateSeq(G(MonotonicGenerator(M_E, 5))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(MonotonicReverseFloat, +INSTANTIATE_TEST_SUITE_P(MonotonicReverseFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -787,10 +1047,10 @@ INSTANTIATE_TEST_CASE_P(MonotonicReverseFloat, generateSeq(G(MonotonicGenerator(-1 * M_E, 5))), generateSeq(G(MonotonicGenerator(-1 * M_E, 5))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(RandomInt, +INSTANTIATE_TEST_SUITE_P(RandomInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, @@ -800,10 +1060,10 @@ INSTANTIATE_TEST_CASE_P(RandomInt, generateSeq(G(RandomGenerator(0, 0, 1000'000'000))), generateSeq(G(RandomGenerator(0, 0, 1000'000'000))) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(RandomishInt, +INSTANTIATE_TEST_SUITE_P(RandomishInt, CodecTest, ::testing::Combine( DefaultCodecsToTest, @@ -815,10 +1075,10 @@ INSTANTIATE_TEST_CASE_P(RandomishInt, generateSeq(G(RandomishGenerator)), generateSeq(G(RandomishGenerator)) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(RandomishFloat, +INSTANTIATE_TEST_SUITE_P(RandomishFloat, CodecTest, ::testing::Combine( DefaultCodecsToTest, @@ -826,11 +1086,11 @@ INSTANTIATE_TEST_CASE_P(RandomishFloat, generateSeq(G(RandomishGenerator)), generateSeq(G(RandomishGenerator)) ) - ), + ) ); // Double delta overflow case, deltas are out of bounds for target type -INSTANTIATE_TEST_CASE_P(OverflowInt, +INSTANTIATE_TEST_SUITE_P(OverflowInt, CodecTest, ::testing::Combine( ::testing::Values( @@ -843,10 +1103,10 @@ INSTANTIATE_TEST_CASE_P(OverflowInt, generateSeq(G(MinMaxGenerator())), generateSeq(G(MinMaxGenerator())) ) - ), + ) ); -INSTANTIATE_TEST_CASE_P(OverflowFloat, +INSTANTIATE_TEST_SUITE_P(OverflowFloat, CodecTest, ::testing::Combine( ::testing::Values( @@ -859,7 +1119,211 @@ INSTANTIATE_TEST_CASE_P(OverflowFloat, generateSeq(G(FFand0Generator())), generateSeq(G(FFand0Generator())) ) - ), + ) ); +template +auto DDCompatibilityTestSequence() +{ + // Generates sequences with double delta in given range. + auto ddGenerator = [prev_delta = static_cast(0), prev = static_cast(0)](auto dd) mutable + { + const auto curr = dd + prev + prev_delta; + prev = curr; + prev_delta = dd + prev_delta; + return curr; + }; + + auto ret = generateSeq(G(SameValueGenerator(42)), 0, 3); + + // These values are from DoubleDelta paper (and implementation) and represent points at which DD encoded length is changed. + // DD value less that this point is encoded in shorter binary form (bigger - longer binary). + const Int64 dd_corner_points[] = {-63, 64, -255, 256, -2047, 2048, std::numeric_limits::min(), std::numeric_limits::max()}; + for (const auto & p : dd_corner_points) + { + if (std::abs(p) > std::numeric_limits::max()) + { + break; + } + + // - 4 is to allow DD value to settle before transitioning through important point, + // since DD depends on 2 previous values of data, + 2 is arbitrary. + ret.append(generateSeq(G(ddGenerator), p - 4, p + 2)); + } + + return ret; +} + +#define BIN_STR(x) std::string{x, sizeof(x) - 1} + +INSTANTIATE_TEST_SUITE_P(DoubleDelta, + CodecTest_Compatibility, + ::testing::Combine( + ::testing::Values(Codec("DoubleDelta")), + ::testing::ValuesIn(std::initializer_list>{ + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\x21\x00\x00\x00\x0f\x00\x00\x00\x01\x00\x0f\x00\x00\x00\x2a\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xb1\xaa\xf4\xf6\x7d\x87\xf8\x80") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\x27\x00\x00\x00\x15\x00\x00\x00\x01\x00\x15\x00\x00\x00\x2a\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xb1\xaa\xf4\xf6\x7d\x87\xf8\x81\x8e\xd0\xca\x02\x01\x01") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\x70\x00\x00\x00\x4e\x00\x00\x00\x02\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x40\x00\x0f\xf2\x78\x00\x01\x7f\x83\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\x70\x00\x00\x00\x4e\x00\x00\x00\x02\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x40\x00\x0f\xf2\x78\x00\x01\x7f\x83\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\x74\x00\x00\x00\x9c\x00\x00\x00\x04\x00\x27\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\xb5\x00\x00\x00\xcc\x00\x00\x00\x04\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xf3\xff\xf9\x41\xaf\xbf\xff\xd6\x0c\xfc\xff\xff\xff\xfb\xf0\x00\x00\x00\x07\xff\xff\xff\xef\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfa\x69\x74\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf8\x00\x00\x00\x07\xff\xff\xff\xf0") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\xd4\x00\x00\x00\x98\x01\x00\x00\x08\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xfc\x00\x00\x00\x04\x00\x06\xbe\x4f\xbf\xff\xd6\x0c\xff\x00\x00\x00\x01\x00\x00\x00\x03\xf8\x00\x00\x00\x08\x00\x00\x00\x0f\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfb\xe0\x00\x00\x01\xc0\x00\x00\x06\x9f\x80\x00\x00\x0a\x00\x00\x00\x34\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf0\x00\x00\x00\x07\xff\xff\xff\xf0") + }, + { + DDCompatibilityTestSequence(), + BIN_STR("\x94\xd4\x00\x00\x00\x98\x01\x00\x00\x08\x00\x33\x00\x00\x00\x2a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x6b\x65\x5f\x50\x34\xff\x4f\xaf\xbc\xe3\x5d\xa3\xd3\xd9\xf6\x1f\xe2\x07\x7c\x47\x20\x67\x48\x07\x47\xff\x47\xf6\xfe\xf8\x00\x00\x70\x6b\xd0\x00\x02\x83\xd9\xfb\x9f\xdc\x1f\xfc\x20\x1e\x80\x00\x22\xc8\xf0\x00\x00\x66\x67\xa0\x00\x02\x00\x3d\x00\x00\x0f\xff\xe8\x00\x00\x7f\xee\xff\xdf\x00\x00\x70\x0d\x7a\x00\x02\x80\x7b\x9f\xf7\x9f\xfb\xc0\x00\x00\xff\xfe\x00\x00\x08\x00\xfc\x00\x00\x00\x04\x00\x06\xbe\x4f\xbf\xff\xd6\x0c\xff\x00\x00\x00\x01\x00\x00\x00\x03\xf8\x00\x00\x00\x08\x00\x00\x00\x0f\xc0\x00\x00\x00\x3f\xff\xff\xff\xfb\xff\xff\xff\xfb\xe0\x00\x00\x01\xc0\x00\x00\x06\x9f\x80\x00\x00\x0a\x00\x00\x00\x34\xf3\xff\xff\xff\xe7\x9f\xff\xff\xff\x7e\x00\x00\x00\x00\xff\xff\xff\xfd\xf0\x00\x00\x00\x07\xff\xff\xff\xf0") + }, + }) + ) +); + +template +auto DDperformanceTestSequence() +{ + const auto times = 100'000; + return DDCompatibilityTestSequence() * times // average case + + generateSeq(G(MinMaxGenerator()), 0, times) // worst + + generateSeq(G(SameValueGenerator(42)), 0, times); // best +} + +// prime numbers in ascending order with some random repitions hit all the cases of Gorilla. +auto PrimesWithMultiplierGenerator = [](int multiplier = 1) +{ + return [multiplier](auto i) + { + static const int vals[] = { + 2, 3, 5, 7, 11, 11, 13, 17, 19, 23, 29, 29, 31, 37, 41, 43, + 47, 47, 53, 59, 61, 61, 67, 71, 73, 79, 83, 89, 89, 97, 101, 103, + 107, 107, 109, 113, 113, 127, 127, 127 + }; + static const size_t count = sizeof(vals)/sizeof(vals[0]); + + using T = decltype(i); + return static_cast(vals[i % count] * static_cast(multiplier)); + }; +}; + +template +auto GCompatibilityTestSequence() +{ + // Also multiply result by some factor to test large values on types that can hold those. + return generateSeq(G(PrimesWithMultiplierGenerator(intExp10(sizeof(ValueType)))), 0, 42); +} + +INSTANTIATE_TEST_SUITE_P(Gorilla, + CodecTest_Compatibility, + ::testing::Combine( + ::testing::Values(Codec("Gorilla")), + ::testing::ValuesIn(std::initializer_list>{ + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x35\x00\x00\x00\x2a\x00\x00\x00\x01\x00\x2a\x00\x00\x00\x14\xe1\xdd\x25\xe5\x7b\x29\x86\xee\x2a\x16\x5a\xc5\x0b\x23\x75\x1b\x3c\xb1\x97\x8b\x5f\xcb\x43\xd9\xc5\x48\xab\x23\xaf\x62\x93\x71\x4a\x73\x0f\xc6\x0a") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x35\x00\x00\x00\x2a\x00\x00\x00\x01\x00\x2a\x00\x00\x00\x14\xe1\xdd\x25\xe5\x7b\x29\x86\xee\x2a\x16\x5a\xc5\x0b\x23\x75\x1b\x3c\xb1\x97\x8b\x5f\xcb\x43\xd9\xc5\x48\xab\x23\xaf\x62\x93\x71\x4a\x73\x0f\xc6\x0a") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x52\x00\x00\x00\x54\x00\x00\x00\x02\x00\x2a\x00\x00\x00\xc8\x00\xdc\xfe\x66\xdb\x1f\x4e\xa7\xde\xdc\xd5\xec\x6e\xf7\x37\x3a\x23\xe7\x63\xf5\x6a\x8e\x99\x37\x34\xf9\xf8\x2e\x76\x35\x2d\x51\xbb\x3b\xc3\x6d\x13\xbf\x86\x53\x9e\x25\xe4\xaf\xaf\x63\xd5\x6a\x6e\x76\x35\x3a\x27\xd3\x0f\x91\xae\x6b\x33\x57\x6e\x64\xcc\x55\x81\xe4") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x52\x00\x00\x00\x54\x00\x00\x00\x02\x00\x2a\x00\x00\x00\xc8\x00\xdc\xfe\x66\xdb\x1f\x4e\xa7\xde\xdc\xd5\xec\x6e\xf7\x37\x3a\x23\xe7\x63\xf5\x6a\x8e\x99\x37\x34\xf9\xf8\x2e\x76\x35\x2d\x51\xbb\x3b\xc3\x6d\x13\xbf\x86\x53\x9e\x25\xe4\xaf\xaf\x63\xd5\x6a\x6e\x76\x35\x3a\x27\xd3\x0f\x91\xae\x6b\x33\x57\x6e\x64\xcc\x55\x81\xe4") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x65\x00\x00\x00\xa8\x00\x00\x00\x04\x00\x2a\x00\x00\x00\x20\x4e\x00\x00\xe4\x57\x63\xc0\xbb\x67\xbc\xce\x91\x97\x99\x15\x9e\xe3\x36\x3f\x89\x5f\x8e\xf2\xec\x8e\xd3\xbf\x75\x43\x58\xc4\x7e\xcf\x93\x43\x38\xc6\x91\x36\x1f\xe7\xb6\x11\x6f\x02\x73\x46\xef\xe0\xec\x50\xfb\x79\xcb\x9c\x14\xfa\x13\xea\x8d\x66\x43\x48\xa0\xde\x3a\xcf\xff\x26\xe0\x5f\x93\xde\x5e\x7f\x6e\x36\x5e\xe6\xb4\x66\x5d\xb0\x0e\xc4") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x65\x00\x00\x00\xa8\x00\x00\x00\x04\x00\x2a\x00\x00\x00\x20\x4e\x00\x00\xe4\x57\x63\xc0\xbb\x67\xbc\xce\x91\x97\x99\x15\x9e\xe3\x36\x3f\x89\x5f\x8e\xf2\xec\x8e\xd3\xbf\x75\x43\x58\xc4\x7e\xcf\x93\x43\x38\xc6\x91\x36\x1f\xe7\xb6\x11\x6f\x02\x73\x46\xef\xe0\xec\x50\xfb\x79\xcb\x9c\x14\xfa\x13\xea\x8d\x66\x43\x48\xa0\xde\x3a\xcf\xff\x26\xe0\x5f\x93\xde\x5e\x7f\x6e\x36\x5e\xe6\xb4\x66\x5d\xb0\x0e\xc4") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x91\x00\x00\x00\x50\x01\x00\x00\x08\x00\x2a\x00\x00\x00\x00\xc2\xeb\x0b\x00\x00\x00\x00\xe3\x2b\xa0\xa6\x19\x85\x98\xdc\x45\x74\x74\x43\xc2\x57\x41\x4c\x6e\x42\x79\xd9\x8f\x88\xa5\x05\xf3\xf1\x94\xa3\x62\x1e\x02\xdf\x05\x10\xf1\x15\x97\x35\x2a\x50\x71\x0f\x09\x6c\x89\xf7\x65\x1d\x11\xb7\xcc\x7d\x0b\x70\xc1\x86\x88\x48\x47\x87\xb6\x32\x26\xa7\x86\x87\x88\xd3\x93\x3d\xfc\x28\x68\x85\x05\x0b\x13\xc6\x5f\xd4\x70\xe1\x5e\x76\xf1\x9f\xf3\x33\x2a\x14\x14\x5e\x40\xc1\x5c\x28\x3f\xec\x43\x03\x05\x11\x91\xe8\xeb\x8e\x0a\x0e\x27\x21\x55\xcb\x39\xbc\x6a\xff\x11\x5d\x81\xa0\xa6\x10") + }, + { + GCompatibilityTestSequence(), + BIN_STR("\x95\x91\x00\x00\x00\x50\x01\x00\x00\x08\x00\x2a\x00\x00\x00\x00\xc2\xeb\x0b\x00\x00\x00\x00\xe3\x2b\xa0\xa6\x19\x85\x98\xdc\x45\x74\x74\x43\xc2\x57\x41\x4c\x6e\x42\x79\xd9\x8f\x88\xa5\x05\xf3\xf1\x94\xa3\x62\x1e\x02\xdf\x05\x10\xf1\x15\x97\x35\x2a\x50\x71\x0f\x09\x6c\x89\xf7\x65\x1d\x11\xb7\xcc\x7d\x0b\x70\xc1\x86\x88\x48\x47\x87\xb6\x32\x26\xa7\x86\x87\x88\xd3\x93\x3d\xfc\x28\x68\x85\x05\x0b\x13\xc6\x5f\xd4\x70\xe1\x5e\x76\xf1\x9f\xf3\x33\x2a\x14\x14\x5e\x40\xc1\x5c\x28\x3f\xec\x43\x03\x05\x11\x91\xe8\xeb\x8e\x0a\x0e\x27\x21\x55\xcb\x39\xbc\x6a\xff\x11\x5d\x81\xa0\xa6\x10") + }, + }) + ) +); + +// These 'tests' try to measure performance of encoding and decoding and hence only make sence to be run locally, +// also they require pretty big data to run agains and generating this data slows down startup of unit test process. +// So un-comment only at your discretion. + +// Just as if all sequences from generatePyramidOfSequences were appended to one-by-one to the first one. +//template +//CodecTestSequence generatePyramidSequence(const size_t sequences_count, Generator && generator, const char* generator_name) +//{ +// CodecTestSequence sequence; +// sequence.data_type = makeDataType(); +// sequence.serialized_data.reserve(sequences_count * sequences_count * sizeof(T)); +// +// for (size_t i = 1; i < sequences_count; ++i) +// { +// std::string name = generator_name + std::string(" from 0 to ") + std::to_string(i); +// sequence.append(generateSeq(std::forward(generator), name.c_str(), 0, i)); +// } +// +// return sequence; +//}; + +//INSTANTIATE_TEST_SUITE_P(DoubleDelta, +// CodecTest_Performance, +// ::testing::Combine( +// ::testing::Values(Codec("DoubleDelta")), +// ::testing::Values( +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence(), +// DDperformanceTestSequence() +// ) +// ), +//); + +//INSTANTIATE_TEST_SUITE_P(Gorilla, +// CodecTest_Performance, +// ::testing::Combine( +// ::testing::Values(Codec("Gorilla")), +// ::testing::Values( +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000, +// generatePyramidSequence(42, G(PrimesWithMultiplierGenerator())) * 6'000 +// ) +// ), +//); + } diff --git a/dbms/src/Core/Settings.h b/dbms/src/Core/Settings.h index 246fd5fe483..724b31ca642 100644 --- a/dbms/src/Core/Settings.h +++ b/dbms/src/Core/Settings.h @@ -127,12 +127,11 @@ struct Settings : public SettingsCollection M(SettingUInt64, optimize_min_equality_disjunction_chain_length, 3, "The minimum length of the expression `expr = x1 OR ... expr = xN` for optimization ", 0) \ \ M(SettingUInt64, min_bytes_to_use_direct_io, 0, "The minimum number of bytes for reading the data with O_DIRECT option during SELECT queries execution. 0 - disabled.", 0) \ + M(SettingUInt64, min_bytes_to_use_mmap_io, 0, "The minimum number of bytes for reading the data with mmap option during SELECT queries execution. 0 - disabled.", 0) \ \ M(SettingBool, force_index_by_date, 0, "Throw an exception if there is a partition key in a table, and it is not used.", 0) \ M(SettingBool, force_primary_key, 0, "Throw an exception if there is primary key in a table, and it is not used.", 0) \ \ - M(SettingUInt64, mark_cache_min_lifetime, 10000, "If the maximum size of mark_cache is exceeded, delete only records older than mark_cache_min_lifetime seconds.", 0) \ - \ M(SettingFloat, max_streams_to_max_threads_ratio, 1, "Allows you to use more sources than the number of threads - to more evenly distribute work across threads. It is assumed that this is a temporary solution, since it will be possible in the future to make the number of sources equal to the number of threads, but for each source to dynamically select available work for itself.", 0) \ M(SettingFloat, max_streams_multiplier_for_merge_tables, 5, "Ask more streams when reading from Merge table. Streams will be spread across tables that Merge table will use. This allows more even distribution of work across threads and especially helpful when merged tables differ in size.", 0) \ \ @@ -383,6 +382,7 @@ struct Settings : public SettingsCollection M(SettingBool, enable_scalar_subquery_optimization, true, "If it is set to true, prevent scalar subqueries from (de)serializing large scalar values and possibly avoid running the same subquery more than once.", 0) \ M(SettingBool, optimize_trivial_count_query, true, "Process trivial 'SELECT count() FROM table' query from metadata.", 0) \ M(SettingUInt64, mutations_sync, 0, "Wait for synchronous execution of ALTER TABLE UPDATE/DELETE queries (mutations). 0 - execute asynchronously. 1 - wait current server. 2 - wait all replicas if they exist.", 0) \ + M(SettingBool, optimize_if_chain_to_miltiif, false, "Replace if(cond1, then1, if(cond2, ...)) chains to multiIf. Currently it's not beneficial for numeric types.", 0) \ \ /** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \ \ @@ -393,6 +393,7 @@ struct Settings : public SettingsCollection M(SettingBool, allow_experimental_cross_to_join_conversion, true, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \ M(SettingBool, allow_experimental_data_skipping_indices, true, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \ M(SettingBool, merge_tree_uniform_read_distribution, true, "Obsolete setting, does nothing. Will be removed after 2020-05-20", 0) \ + M(SettingUInt64, mark_cache_min_lifetime, 0, "Obsolete setting, does nothing. Will be removed after 2020-05-31", 0) \ DECLARE_SETTINGS_COLLECTION(LIST_OF_SETTINGS) diff --git a/dbms/src/Core/SortCursor.h b/dbms/src/Core/SortCursor.h index 5b4db43024f..84b788c8845 100644 --- a/dbms/src/Core/SortCursor.h +++ b/dbms/src/Core/SortCursor.h @@ -22,8 +22,8 @@ namespace DB */ struct SortCursorImpl { - ColumnRawPtrs all_columns; ColumnRawPtrs sort_columns; + ColumnRawPtrs all_columns; SortDescription desc; size_t sort_columns_size = 0; size_t pos = 0; @@ -110,21 +110,52 @@ using SortCursorImpls = std::vector; /// For easy copying. -struct SortCursor +template +struct SortCursorHelper { SortCursorImpl * impl; - SortCursor(SortCursorImpl * impl_) : impl(impl_) {} + const Derived & derived() const { return static_cast(*this); } + + SortCursorHelper(SortCursorImpl * impl_) : impl(impl_) {} SortCursorImpl * operator-> () { return impl; } const SortCursorImpl * operator-> () const { return impl; } + bool ALWAYS_INLINE greater(const SortCursorHelper & rhs) const + { + return derived().greaterAt(rhs.derived(), impl->pos, rhs.impl->pos); + } + + /// Inverted so that the priority queue elements are removed in ascending order. + bool ALWAYS_INLINE operator< (const SortCursorHelper & rhs) const + { + return derived().greater(rhs.derived()); + } + + /// Checks that all rows in the current block of this cursor are less than or equal to all the rows of the current block of another cursor. + bool ALWAYS_INLINE totallyLessOrEquals(const SortCursorHelper & rhs) const + { + if (impl->rows == 0 || rhs.impl->rows == 0) + return false; + + /// The last row of this cursor is no larger than the first row of the another cursor. + return !derived().greaterAt(rhs.derived(), impl->rows - 1, 0); + } +}; + + +struct SortCursor : SortCursorHelper +{ + using SortCursorHelper::SortCursorHelper; + /// The specified row of this cursor is greater than the specified row of another cursor. - bool greaterAt(const SortCursor & rhs, size_t lhs_pos, size_t rhs_pos) const + bool ALWAYS_INLINE greaterAt(const SortCursor & rhs, size_t lhs_pos, size_t rhs_pos) const { for (size_t i = 0; i < impl->sort_columns_size; ++i) { - int direction = impl->desc[i].direction; - int nulls_direction = impl->desc[i].nulls_direction; + const auto & desc = impl->desc[i]; + int direction = desc.direction; + int nulls_direction = desc.nulls_direction; int res = direction * impl->sort_columns[i]->compareAt(lhs_pos, rhs_pos, *(rhs.impl->sort_columns[i]), nulls_direction); if (res > 0) return true; @@ -133,45 +164,37 @@ struct SortCursor } return impl->order > rhs.impl->order; } +}; - /// Checks that all rows in the current block of this cursor are less than or equal to all the rows of the current block of another cursor. - bool totallyLessOrEquals(const SortCursor & rhs) const + +/// For the case with a single column and when there is no order between different cursors. +struct SimpleSortCursor : SortCursorHelper +{ + using SortCursorHelper::SortCursorHelper; + + bool ALWAYS_INLINE greaterAt(const SimpleSortCursor & rhs, size_t lhs_pos, size_t rhs_pos) const { - if (impl->rows == 0 || rhs.impl->rows == 0) - return false; - - /// The last row of this cursor is no larger than the first row of the another cursor. - return !greaterAt(rhs, impl->rows - 1, 0); - } - - bool greater(const SortCursor & rhs) const - { - return greaterAt(rhs, impl->pos, rhs.impl->pos); - } - - /// Inverted so that the priority queue elements are removed in ascending order. - bool operator< (const SortCursor & rhs) const - { - return greater(rhs); + const auto & desc = impl->desc[0]; + int direction = desc.direction; + int nulls_direction = desc.nulls_direction; + int res = impl->sort_columns[0]->compareAt(lhs_pos, rhs_pos, *(rhs.impl->sort_columns[0]), nulls_direction); + return res != 0 && ((res > 0) == (direction > 0)); } }; /// Separate comparator for locale-sensitive string comparisons -struct SortCursorWithCollation +struct SortCursorWithCollation : SortCursorHelper { - SortCursorImpl * impl; + using SortCursorHelper::SortCursorHelper; - SortCursorWithCollation(SortCursorImpl * impl_) : impl(impl_) {} - SortCursorImpl * operator-> () { return impl; } - const SortCursorImpl * operator-> () const { return impl; } - - bool greaterAt(const SortCursorWithCollation & rhs, size_t lhs_pos, size_t rhs_pos) const + bool ALWAYS_INLINE greaterAt(const SortCursorWithCollation & rhs, size_t lhs_pos, size_t rhs_pos) const { for (size_t i = 0; i < impl->sort_columns_size; ++i) { - int direction = impl->desc[i].direction; - int nulls_direction = impl->desc[i].nulls_direction; + const auto & desc = impl->desc[i]; + int direction = desc.direction; + int nulls_direction = desc.nulls_direction; int res; if (impl->need_collation[i]) { @@ -189,29 +212,11 @@ struct SortCursorWithCollation } return impl->order > rhs.impl->order; } - - bool totallyLessOrEquals(const SortCursorWithCollation & rhs) const - { - if (impl->rows == 0 || rhs.impl->rows == 0) - return false; - - /// The last row of this cursor is no larger than the first row of the another cursor. - return !greaterAt(rhs, impl->rows - 1, 0); - } - - bool greater(const SortCursorWithCollation & rhs) const - { - return greaterAt(rhs, impl->pos, rhs.impl->pos); - } - - bool operator< (const SortCursorWithCollation & rhs) const - { - return greater(rhs); - } }; /** Allows to fetch data from multiple sort cursors in sorted order (merging sorted data streams). + * TODO: Replace with "Loser Tree", see https://en.wikipedia.org/wiki/K-way_merge_algorithm */ template class SortingHeap @@ -225,7 +230,8 @@ public: size_t size = cursors.size(); queue.reserve(size); for (size_t i = 0; i < size; ++i) - queue.emplace_back(&cursors[i]); + if (!cursors[i].empty()) + queue.emplace_back(&cursors[i]); std::make_heap(queue.begin(), queue.end()); } @@ -233,7 +239,11 @@ public: Cursor & current() { return queue.front(); } - void next() + size_t size() { return queue.size(); } + + Cursor & nextChild() { return queue[nextChildIndex()]; } + + void ALWAYS_INLINE next() { assert(isValid()); @@ -246,34 +256,67 @@ public: removeTop(); } + void replaceTop(Cursor new_top) + { + current() = new_top; + updateTop(); + } + + void removeTop() + { + std::pop_heap(queue.begin(), queue.end()); + queue.pop_back(); + next_idx = 0; + } + + void push(SortCursorImpl & cursor) + { + queue.emplace_back(&cursor); + std::push_heap(queue.begin(), queue.end()); + next_idx = 0; + } + private: using Container = std::vector; Container queue; + /// Cache comparison between first and second child if the order in queue has not been changed. + size_t next_idx = 0; + + size_t ALWAYS_INLINE nextChildIndex() + { + if (next_idx == 0) + { + next_idx = 1; + + if (queue.size() > 2 && queue[1] < queue[2]) + ++next_idx; + } + + return next_idx; + } + /// This is adapted version of the function __sift_down from libc++. /// Why cannot simply use std::priority_queue? /// - because it doesn't support updating the top element and requires pop and push instead. - void updateTop() + /// Also look at "Boost.Heap" library. + void ALWAYS_INLINE updateTop() { size_t size = queue.size(); if (size < 2) return; - size_t child_idx = 1; auto begin = queue.begin(); - auto child_it = begin + 1; - /// Right child exists and is greater than left child. - if (size > 2 && *child_it < *(child_it + 1)) - { - ++child_it; - ++child_idx; - } + size_t child_idx = nextChildIndex(); + auto child_it = begin + child_idx; /// Check if we are in order. if (*child_it < *begin) return; + next_idx = 0; + auto curr_it = begin; auto top(std::move(*begin)); do @@ -282,11 +325,12 @@ private: *curr_it = std::move(*child_it); curr_it = child_it; - if ((size - 2) / 2 < child_idx) - break; - // recompute the child based off of the updated parent child_idx = 2 * child_idx + 1; + + if (child_idx >= size) + break; + child_it = begin + child_idx; if ((child_idx + 1) < size && *child_it < *(child_it + 1)) @@ -300,12 +344,6 @@ private: } while (!(*child_it < top)); *curr_it = std::move(top); } - - void removeTop() - { - std::pop_heap(queue.begin(), queue.end()); - queue.pop_back(); - } }; } diff --git a/dbms/src/Core/tests/gtest_DecimalFunctions.cpp b/dbms/src/Core/tests/gtest_DecimalFunctions.cpp index d03be3ff3b8..fc304446057 100644 --- a/dbms/src/Core/tests/gtest_DecimalFunctions.cpp +++ b/dbms/src/Core/tests/gtest_DecimalFunctions.cpp @@ -120,7 +120,7 @@ TEST_P(DecimalUtilsSplitAndCombineTest, getFractionalPart_Decimal128) } // Intentionally small values that fit into 32-bit in order to cover Decimal32, Decimal64 and Decimal128 with single set of data. -INSTANTIATE_TEST_CASE_P(Basic, +INSTANTIATE_TEST_SUITE_P(Basic, DecimalUtilsSplitAndCombineTest, ::testing::ValuesIn(std::initializer_list{ { @@ -168,5 +168,5 @@ INSTANTIATE_TEST_CASE_P(Basic, 89 } } - } -),); + }) +); diff --git a/dbms/src/DataStreams/AddingDefaultsBlockInputStream.cpp b/dbms/src/DataStreams/AddingDefaultsBlockInputStream.cpp index f6b6b290428..112afe61183 100644 --- a/dbms/src/DataStreams/AddingDefaultsBlockInputStream.cpp +++ b/dbms/src/DataStreams/AddingDefaultsBlockInputStream.cpp @@ -62,6 +62,9 @@ Block AddingDefaultsBlockInputStream::readImpl() if (evaluate_block.has(column.first)) evaluate_block.erase(column.first); + if (!evaluate_block.columns()) + evaluate_block.insert({ColumnConst::create(ColumnUInt8::create(1, 0), res.rows()), std::make_shared(), "_dummy"}); + evaluateMissingDefaults(evaluate_block, header.getNamesAndTypesList(), column_defaults, context, false); std::unordered_map mixed_columns; diff --git a/dbms/src/DataStreams/AggregatingSortedBlockInputStream.cpp b/dbms/src/DataStreams/AggregatingSortedBlockInputStream.cpp index 3607d1f917f..d23d93e7e5c 100644 --- a/dbms/src/DataStreams/AggregatingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/AggregatingSortedBlockInputStream.cpp @@ -138,14 +138,14 @@ Block AggregatingSortedBlockInputStream::readImpl() } -void AggregatingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void AggregatingSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { size_t merged_rows = 0; /// We take the rows in the correct order and put them in `merged_block`, while the rows are no more than `max_block_size` - while (!queue.empty()) + while (queue.isValid()) { - SortCursor current = queue.top(); + SortCursor current = queue.current(); setPrimaryKeyRef(next_key, current); @@ -167,8 +167,6 @@ void AggregatingSortedBlockInputStream::merge(MutableColumns & merged_columns, s return; } - queue.pop(); - if (key_differs) { current_key.swap(next_key); @@ -202,8 +200,7 @@ void AggregatingSortedBlockInputStream::merge(MutableColumns & merged_columns, s if (!current->isLast()) { - current->next(); - queue.push(current); + queue.next(); } else { diff --git a/dbms/src/DataStreams/AggregatingSortedBlockInputStream.h b/dbms/src/DataStreams/AggregatingSortedBlockInputStream.h index 0cf4bd64d87..6ef1259e458 100644 --- a/dbms/src/DataStreams/AggregatingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/AggregatingSortedBlockInputStream.h @@ -55,7 +55,7 @@ private: /** We support two different cursors - with Collation and without. * Templates are used instead of polymorphic SortCursor and calls to virtual functions. */ - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /** Extract all states of aggregate functions and merge them with the current group. */ diff --git a/dbms/src/DataStreams/CollapsingSortedBlockInputStream.cpp b/dbms/src/DataStreams/CollapsingSortedBlockInputStream.cpp index 7e4ad04b806..ef82a6d8c5e 100644 --- a/dbms/src/DataStreams/CollapsingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/CollapsingSortedBlockInputStream.cpp @@ -105,15 +105,15 @@ Block CollapsingSortedBlockInputStream::readImpl() } -void CollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void CollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { MergeStopCondition stop_condition(average_block_sizes, max_block_size); size_t current_block_granularity; /// Take rows in correct order and put them into `merged_columns` until the rows no more than `max_block_size` - for (; !queue.empty(); ++current_pos) + for (; queue.isValid(); ++current_pos) { - SortCursor current = queue.top(); + SortCursor current = queue.current(); current_block_granularity = current->rows; if (current_key.empty()) @@ -131,8 +131,6 @@ void CollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, st return; } - queue.pop(); - if (key_differs) { /// We write data for the previous primary key. @@ -185,8 +183,7 @@ void CollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, st if (!current->isLast()) { - current->next(); - queue.push(current); + queue.next(); } else { diff --git a/dbms/src/DataStreams/CollapsingSortedBlockInputStream.h b/dbms/src/DataStreams/CollapsingSortedBlockInputStream.h index 7e114e614f6..2b528d27339 100644 --- a/dbms/src/DataStreams/CollapsingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/CollapsingSortedBlockInputStream.h @@ -73,7 +73,7 @@ private: /** We support two different cursors - with Collation and without. * Templates are used instead of polymorphic SortCursors and calls to virtual functions. */ - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /// Output to result rows for the current primary key. void insertRows(MutableColumns & merged_columns, size_t block_size, MergeStopCondition & condition); diff --git a/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.cpp b/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.cpp index 340e10df12f..64a0d52c1aa 100644 --- a/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.cpp @@ -161,7 +161,7 @@ Block GraphiteRollupSortedBlockInputStream::readImpl() } -void GraphiteRollupSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void GraphiteRollupSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { const DateLUTImpl & date_lut = DateLUT::instance(); @@ -173,9 +173,9 @@ void GraphiteRollupSortedBlockInputStream::merge(MutableColumns & merged_columns /// contribute towards current output row. /// Variables starting with next_* refer to the row at the top of the queue. - while (!queue.empty()) + while (queue.isValid()) { - SortCursor next_cursor = queue.top(); + SortCursor next_cursor = queue.current(); StringRef next_path = next_cursor->all_columns[path_column_num]->getDataAt(next_cursor->pos); bool new_path = is_first || next_path != current_group_path; @@ -253,12 +253,9 @@ void GraphiteRollupSortedBlockInputStream::merge(MutableColumns & merged_columns current_group_path = next_path; } - queue.pop(); - if (!next_cursor->isLast()) { - next_cursor->next(); - queue.push(next_cursor); + queue.next(); } else { diff --git a/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.h b/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.h index 533c267ff02..0dfdf7c300c 100644 --- a/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.h +++ b/dbms/src/DataStreams/GraphiteRollupSortedBlockInputStream.h @@ -225,7 +225,7 @@ private: UInt32 selectPrecision(const Graphite::Retentions & retentions, time_t time) const; - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /// Insert the values into the resulting columns, which will not be changed in the future. template diff --git a/dbms/src/DataStreams/MergeSortingBlockInputStream.cpp b/dbms/src/DataStreams/MergeSortingBlockInputStream.cpp index 1c50316fc3f..52f85f1349c 100644 --- a/dbms/src/DataStreams/MergeSortingBlockInputStream.cpp +++ b/dbms/src/DataStreams/MergeSortingBlockInputStream.cpp @@ -150,10 +150,12 @@ MergeSortingBlocksBlockInputStream::MergeSortingBlocksBlockInputStream( blocks.swap(nonempty_blocks); - if (!has_collation) + if (has_collation) + queue_with_collation = SortingHeap(cursors); + else if (description.size() > 1) queue_without_collation = SortingHeap(cursors); else - queue_with_collation = SortingHeap(cursors); + queue_simple = SortingHeap(cursors); } @@ -169,9 +171,12 @@ Block MergeSortingBlocksBlockInputStream::readImpl() return res; } - return !has_collation - ? mergeImpl(queue_without_collation) - : mergeImpl(queue_with_collation); + if (has_collation) + return mergeImpl(queue_with_collation); + else if (description.size() > 1) + return mergeImpl(queue_without_collation); + else + return mergeImpl(queue_simple); } @@ -179,9 +184,18 @@ template Block MergeSortingBlocksBlockInputStream::mergeImpl(TSortingHeap & queue) { size_t num_columns = header.columns(); - MutableColumns merged_columns = header.cloneEmptyColumns(); - /// TODO: reserve (in each column) + + /// Reserve + if (queue.isValid() && !blocks.empty()) + { + /// The expected size of output block is the same as input block + size_t size_to_reserve = blocks[0].rows(); + for (auto & column : merged_columns) + column->reserve(size_to_reserve); + } + + /// TODO: Optimization when a single block left. /// Take rows from queue in right order and push to 'merged'. size_t merged_rows = 0; @@ -210,6 +224,9 @@ Block MergeSortingBlocksBlockInputStream::mergeImpl(TSortingHeap & queue) break; } + if (!queue.isValid()) + blocks.clear(); + if (merged_rows == 0) return {}; diff --git a/dbms/src/DataStreams/MergeSortingBlockInputStream.h b/dbms/src/DataStreams/MergeSortingBlockInputStream.h index 9492bdb074b..ce82f6bb120 100644 --- a/dbms/src/DataStreams/MergeSortingBlockInputStream.h +++ b/dbms/src/DataStreams/MergeSortingBlockInputStream.h @@ -59,6 +59,7 @@ private: bool has_collation = false; SortingHeap queue_without_collation; + SortingHeap queue_simple; SortingHeap queue_with_collation; /** Two different cursors are supported - with and without Collation. diff --git a/dbms/src/DataStreams/MergingSortedBlockInputStream.cpp b/dbms/src/DataStreams/MergingSortedBlockInputStream.cpp index 8c0707e09b0..3614d9c1d66 100644 --- a/dbms/src/DataStreams/MergingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/MergingSortedBlockInputStream.cpp @@ -59,9 +59,9 @@ void MergingSortedBlockInputStream::init(MutableColumns & merged_columns) } if (has_collation) - initQueue(queue_with_collation); + queue_with_collation = SortingHeap(cursors); else - initQueue(queue_without_collation); + queue_without_collation = SortingHeap(cursors); } /// Let's check that all source blocks have the same structure. @@ -82,15 +82,6 @@ void MergingSortedBlockInputStream::init(MutableColumns & merged_columns) } -template -void MergingSortedBlockInputStream::initQueue(std::priority_queue & queue) -{ - for (size_t i = 0; i < cursors.size(); ++i) - if (!cursors[i].empty()) - queue.push(TSortCursor(&cursors[i])); -} - - Block MergingSortedBlockInputStream::readImpl() { if (finished) @@ -115,7 +106,7 @@ Block MergingSortedBlockInputStream::readImpl() template -void MergingSortedBlockInputStream::fetchNextBlock(const TSortCursor & current, std::priority_queue & queue) +void MergingSortedBlockInputStream::fetchNextBlock(const TSortCursor & current, SortingHeap & queue) { size_t order = current->order; size_t size = cursors.size(); @@ -125,15 +116,19 @@ void MergingSortedBlockInputStream::fetchNextBlock(const TSortCursor & current, while (true) { - source_blocks[order] = new detail::SharedBlock(children[order]->read()); + source_blocks[order] = new detail::SharedBlock(children[order]->read()); /// intrusive ptr if (!*source_blocks[order]) + { + queue.removeTop(); break; + } if (source_blocks[order]->rows()) { cursors[order].reset(*source_blocks[order]); - queue.push(TSortCursor(&cursors[order])); + queue.replaceTop(&cursors[order]); + source_blocks[order]->all_columns = cursors[order].all_columns; source_blocks[order]->sort_columns = cursors[order].sort_columns; break; @@ -154,19 +149,14 @@ bool MergingSortedBlockInputStream::MergeStopCondition::checkStop() const return sum_rows_count >= average; } -template -void MergingSortedBlockInputStream::fetchNextBlock(const SortCursor & current, std::priority_queue & queue); -template -void MergingSortedBlockInputStream::fetchNextBlock(const SortCursorWithCollation & current, std::priority_queue & queue); - - -template -void MergingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +template +void MergingSortedBlockInputStream::merge(MutableColumns & merged_columns, TSortingHeap & queue) { size_t merged_rows = 0; MergeStopCondition stop_condition(average_block_sizes, max_block_size); + /** Increase row counters. * Return true if it's time to finish generating the current data block. */ @@ -186,123 +176,100 @@ void MergingSortedBlockInputStream::merge(MutableColumns & merged_columns, std:: return stop_condition.checkStop(); }; - /// Take rows in required order and put them into `merged_columns`, while the rows are no more than `max_block_size` - while (!queue.empty()) + /// Take rows in required order and put them into `merged_columns`, while the number of rows are no more than `max_block_size` + while (queue.isValid()) { - TSortCursor current = queue.top(); + auto current = queue.current(); size_t current_block_granularity = current->rows; - queue.pop(); - while (true) + /** And what if the block is totally less or equal than the rest for the current cursor? + * Or is there only one data source left in the queue? Then you can take the entire block on current cursor. + */ + if (current->isFirst() + && (queue.size() == 1 + || (queue.size() >= 2 && current.totallyLessOrEquals(queue.nextChild())))) { - /** And what if the block is totally less or equal than the rest for the current cursor? - * Or is there only one data source left in the queue? Then you can take the entire block on current cursor. - */ - if (current->isFirst() && (queue.empty() || current.totallyLessOrEquals(queue.top()))) +// std::cerr << "current block is totally less or equals\n"; + + /// If there are already data in the current block, we first return it. We'll get here again the next time we call the merge function. + if (merged_rows != 0) { - // std::cerr << "current block is totally less or equals\n"; - - /// If there are already data in the current block, we first return it. We'll get here again the next time we call the merge function. - if (merged_rows != 0) - { - //std::cerr << "merged rows is non-zero\n"; - queue.push(current); - return; - } - - /// Actually, current->order stores source number (i.e. cursors[current->order] == current) - size_t source_num = current->order; - - if (source_num >= cursors.size()) - throw Exception("Logical error in MergingSortedBlockInputStream", ErrorCodes::LOGICAL_ERROR); - - for (size_t i = 0; i < num_columns; ++i) - merged_columns[i] = (*std::move(source_blocks[source_num]->getByPosition(i).column)).mutate(); - - // std::cerr << "copied columns\n"; - - merged_rows = merged_columns.at(0)->size(); - - /// Limit output - if (limit && total_merged_rows + merged_rows > limit) - { - merged_rows = limit - total_merged_rows; - for (size_t i = 0; i < num_columns; ++i) - { - auto & column = merged_columns[i]; - column = (*column->cut(0, merged_rows)).mutate(); - } - - cancel(false); - finished = true; - } - - /// Write order of rows for other columns - /// this data will be used in grather stream - if (out_row_sources_buf) - { - RowSourcePart row_source(source_num); - for (size_t i = 0; i < merged_rows; ++i) - out_row_sources_buf->write(row_source.data); - } - - //std::cerr << "fetching next block\n"; - - total_merged_rows += merged_rows; - fetchNextBlock(current, queue); + //std::cerr << "merged rows is non-zero\n"; return; } - // std::cerr << "total_merged_rows: " << total_merged_rows << ", merged_rows: " << merged_rows << "\n"; - // std::cerr << "Inserting row\n"; - for (size_t i = 0; i < num_columns; ++i) - merged_columns[i]->insertFrom(*current->all_columns[i], current->pos); + /// Actually, current->order stores source number (i.e. cursors[current->order] == current) + size_t source_num = current->order; + if (source_num >= cursors.size()) + throw Exception("Logical error in MergingSortedBlockInputStream", ErrorCodes::LOGICAL_ERROR); + + for (size_t i = 0; i < num_columns; ++i) + merged_columns[i] = (*std::move(source_blocks[source_num]->getByPosition(i).column)).mutate(); + +// std::cerr << "copied columns\n"; + + merged_rows = merged_columns.at(0)->size(); + + /// Limit output + if (limit && total_merged_rows + merged_rows > limit) + { + merged_rows = limit - total_merged_rows; + for (size_t i = 0; i < num_columns; ++i) + { + auto & column = merged_columns[i]; + column = (*column->cut(0, merged_rows)).mutate(); + } + + cancel(false); + finished = true; + } + + /// Write order of rows for other columns + /// this data will be used in grather stream if (out_row_sources_buf) { - /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) - RowSourcePart row_source(current->order); - out_row_sources_buf->write(row_source.data); + RowSourcePart row_source(source_num); + for (size_t i = 0; i < merged_rows; ++i) + out_row_sources_buf->write(row_source.data); } - if (!current->isLast()) - { - // std::cerr << "moving to next row\n"; - current->next(); + //std::cerr << "fetching next block\n"; - if (queue.empty() || !(current.greater(queue.top()))) - { - if (count_row_and_check_limit(current_block_granularity)) - { - // std::cerr << "pushing back to queue\n"; - queue.push(current); - return; - } + total_merged_rows += merged_rows; + fetchNextBlock(current, queue); + return; + } - /// Do not put the cursor back in the queue, but continue to work with the current cursor. - // std::cerr << "current is still on top, using current row\n"; - continue; - } - else - { - // std::cerr << "next row is not least, pushing back to queue\n"; - queue.push(current); - } - } - else - { - /// We get the next block from the corresponding source, if there is one. - // std::cerr << "It was last row, fetching next block\n"; - fetchNextBlock(current, queue); - } +// std::cerr << "total_merged_rows: " << total_merged_rows << ", merged_rows: " << merged_rows << "\n"; +// std::cerr << "Inserting row\n"; + for (size_t i = 0; i < num_columns; ++i) + merged_columns[i]->insertFrom(*current->all_columns[i], current->pos); - break; + if (out_row_sources_buf) + { + /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) + RowSourcePart row_source(current->order); + out_row_sources_buf->write(row_source.data); + } + + if (!current->isLast()) + { +// std::cerr << "moving to next row\n"; + queue.next(); + } + else + { + /// We get the next block from the corresponding source, if there is one. +// std::cerr << "It was last row, fetching next block\n"; + fetchNextBlock(current, queue); } if (count_row_and_check_limit(current_block_granularity)) return; } + /// We have read all data. Ask childs to cancel providing more data. cancel(false); finished = true; } diff --git a/dbms/src/DataStreams/MergingSortedBlockInputStream.h b/dbms/src/DataStreams/MergingSortedBlockInputStream.h index beb3c7afc52..e6c2b257013 100644 --- a/dbms/src/DataStreams/MergingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/MergingSortedBlockInputStream.h @@ -1,7 +1,5 @@ #pragma once -#include - #include #include @@ -87,7 +85,7 @@ protected: /// Gets the next block from the source corresponding to the `current`. template - void fetchNextBlock(const TSortCursor & current, std::priority_queue & queue); + void fetchNextBlock(const TSortCursor & current, SortingHeap & queue); Block header; @@ -109,14 +107,10 @@ protected: size_t num_columns = 0; std::vector source_blocks; - using CursorImpls = std::vector; - CursorImpls cursors; + SortCursorImpls cursors; - using Queue = std::priority_queue; - Queue queue_without_collation; - - using QueueWithCollation = std::priority_queue; - QueueWithCollation queue_with_collation; + SortingHeap queue_without_collation; + SortingHeap queue_with_collation; /// Used in Vertical merge algorithm to gather non-PK/non-index columns (on next step) /// If it is not nullptr then it should be populated during execution @@ -177,13 +171,10 @@ protected: private: /** We support two different cursors - with Collation and without. - * Templates are used instead of polymorphic SortCursor and calls to virtual functions. - */ - template - void initQueue(std::priority_queue & queue); - - template - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + * Templates are used instead of polymorphic SortCursor and calls to virtual functions. + */ + template + void merge(MutableColumns & merged_columns, TSortingHeap & queue); Logger * log = &Logger::get("MergingSortedBlockInputStream"); diff --git a/dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp b/dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp index d7fb7bad343..3d5fb426218 100644 --- a/dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp +++ b/dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp @@ -129,7 +129,7 @@ void PushingToViewsBlockOutputStream::write(const Block & block) for (size_t view_num = 0; view_num < views.size(); ++view_num) { auto thread_group = CurrentThread::getGroup(); - pool.scheduleOrThrowOnError([=] + pool.scheduleOrThrowOnError([=, this] { setThreadName("PushingToViews"); if (thread_group) diff --git a/dbms/src/DataStreams/ReplacingSortedBlockInputStream.cpp b/dbms/src/DataStreams/ReplacingSortedBlockInputStream.cpp index e2e99815b93..967b4ebb046 100644 --- a/dbms/src/DataStreams/ReplacingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/ReplacingSortedBlockInputStream.cpp @@ -48,13 +48,14 @@ Block ReplacingSortedBlockInputStream::readImpl() } -void ReplacingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void ReplacingSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { MergeStopCondition stop_condition(average_block_sizes, max_block_size); + /// Take the rows in needed order and put them into `merged_columns` until rows no more than `max_block_size` - while (!queue.empty()) + while (queue.isValid()) { - SortCursor current = queue.top(); + SortCursor current = queue.current(); size_t current_block_granularity = current->rows; if (current_key.empty()) @@ -68,8 +69,6 @@ void ReplacingSortedBlockInputStream::merge(MutableColumns & merged_columns, std if (key_differs && stop_condition.checkStop()) return; - queue.pop(); - if (key_differs) { /// Write the data for the previous primary key. @@ -98,8 +97,7 @@ void ReplacingSortedBlockInputStream::merge(MutableColumns & merged_columns, std if (!current->isLast()) { - current->next(); - queue.push(current); + queue.next(); } else { diff --git a/dbms/src/DataStreams/ReplacingSortedBlockInputStream.h b/dbms/src/DataStreams/ReplacingSortedBlockInputStream.h index 7d85542520d..22920c2eb20 100644 --- a/dbms/src/DataStreams/ReplacingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/ReplacingSortedBlockInputStream.h @@ -52,7 +52,7 @@ private: /// Sources of rows with the current primary key. PODArray current_row_sources; - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /// Output into result the rows for current primary key. void insertRow(MutableColumns & merged_columns); diff --git a/dbms/src/DataStreams/SummingSortedBlockInputStream.cpp b/dbms/src/DataStreams/SummingSortedBlockInputStream.cpp index 9ac7d6a3397..fe29dc55916 100644 --- a/dbms/src/DataStreams/SummingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/SummingSortedBlockInputStream.cpp @@ -314,14 +314,14 @@ Block SummingSortedBlockInputStream::readImpl() } -void SummingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void SummingSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { merged_rows = 0; /// Take the rows in needed order and put them in `merged_columns` until rows no more than `max_block_size` - while (!queue.empty()) + while (queue.isValid()) { - SortCursor current = queue.top(); + SortCursor current = queue.current(); setPrimaryKeyRef(next_key, current); @@ -383,12 +383,9 @@ void SummingSortedBlockInputStream::merge(MutableColumns & merged_columns, std:: current_row_is_zero = false; } - queue.pop(); - if (!current->isLast()) { - current->next(); - queue.push(current); + queue.next(); } else { diff --git a/dbms/src/DataStreams/SummingSortedBlockInputStream.h b/dbms/src/DataStreams/SummingSortedBlockInputStream.h index 4412e5529f8..fc02d36d3fd 100644 --- a/dbms/src/DataStreams/SummingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/SummingSortedBlockInputStream.h @@ -1,5 +1,7 @@ #pragma once +#include + #include #include #include @@ -140,7 +142,7 @@ private: /** We support two different cursors - with Collation and without. * Templates are used instead of polymorphic SortCursor and calls to virtual functions. */ - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /// Insert the summed row for the current group into the result and updates some of per-block flags if the row is not "zero". void insertCurrentRowIfNeeded(MutableColumns & merged_columns); diff --git a/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.cpp b/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.cpp index 4dda97597bd..de6f7027243 100644 --- a/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.cpp +++ b/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.cpp @@ -82,21 +82,18 @@ Block VersionedCollapsingSortedBlockInputStream::readImpl() } -void VersionedCollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, std::priority_queue & queue) +void VersionedCollapsingSortedBlockInputStream::merge(MutableColumns & merged_columns, SortingHeap & queue) { MergeStopCondition stop_condition(average_block_sizes, max_block_size); auto update_queue = [this, & queue](SortCursor & cursor) { - queue.pop(); - if (out_row_sources_buf) current_row_sources.emplace(cursor->order, true); if (!cursor->isLast()) { - cursor->next(); - queue.push(cursor); + queue.next(); } else { @@ -106,9 +103,9 @@ void VersionedCollapsingSortedBlockInputStream::merge(MutableColumns & merged_co }; /// Take rows in correct order and put them into `merged_columns` until the rows no more than `max_block_size` - while (!queue.empty()) + while (queue.isValid()) { - SortCursor current = queue.top(); + SortCursor current = queue.current(); size_t current_block_granularity = current->rows; SharedBlockRowRef next_key; diff --git a/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.h b/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.h index f79b564063d..c64972d9266 100644 --- a/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.h +++ b/dbms/src/DataStreams/VersionedCollapsingSortedBlockInputStream.h @@ -5,7 +5,7 @@ #include #include -#include +#include namespace DB @@ -204,7 +204,7 @@ private: /// Sources of rows for VERTICAL merge algorithm. Size equals to (size + number of gaps) in current_keys. std::queue current_row_sources; - void merge(MutableColumns & merged_columns, std::priority_queue & queue); + void merge(MutableColumns & merged_columns, SortingHeap & queue); /// Output to result row for the current primary key. void insertRow(size_t skip_rows, const SharedBlockRowRef & row, MutableColumns & merged_columns); diff --git a/dbms/src/DataStreams/tests/union_stream2.cpp b/dbms/src/DataStreams/tests/union_stream2.cpp index 3eb1927f80a..ab0b583b8e5 100644 --- a/dbms/src/DataStreams/tests/union_stream2.cpp +++ b/dbms/src/DataStreams/tests/union_stream2.cpp @@ -57,6 +57,6 @@ catch (const Exception & e) std::cerr << e.what() << ", " << e.displayText() << std::endl << std::endl << "Stack trace:" << std::endl - << e.getStackTrace().toString(); + << e.getStackTraceString(); return 1; } diff --git a/dbms/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp b/dbms/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp index 21e5200c1dd..fecf09f08f7 100644 --- a/dbms/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp +++ b/dbms/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp @@ -30,7 +30,7 @@ namespace ErrorCodes extern const int LOGICAL_ERROR; } -static const std::vector supported_functions{"any", "anyLast", "min", "max", "sum"}; +static const std::vector supported_functions{"any", "anyLast", "min", "max", "sum", "groupBitAnd", "groupBitOr", "groupBitXor"}; String DataTypeCustomSimpleAggregateFunction::getName() const diff --git a/dbms/src/DataTypes/tests/gtest_data_type_get_common_type.cpp b/dbms/src/DataTypes/tests/gtest_data_type_get_common_type.cpp index 2ae1c335387..602320f5fca 100644 --- a/dbms/src/DataTypes/tests/gtest_data_type_get_common_type.cpp +++ b/dbms/src/DataTypes/tests/gtest_data_type_get_common_type.cpp @@ -104,7 +104,7 @@ TEST_P(MostSubtypeTest, getLeastSupertype) } } -INSTANTIATE_TEST_CASE_P(data_type, +INSTANTIATE_TEST_SUITE_P(data_type, LeastSuperTypeTest, ::testing::ValuesIn( std::initializer_list{ @@ -159,10 +159,10 @@ INSTANTIATE_TEST_CASE_P(data_type, {"Tuple(Int64,Int8) Tuple(UInt64)", nullptr}, {"Array(Int64) Array(String)", nullptr}, } - ), + ) ); -INSTANTIATE_TEST_CASE_P(data_type, +INSTANTIATE_TEST_SUITE_P(data_type, MostSubtypeTest, ::testing::ValuesIn( std::initializer_list{ @@ -210,5 +210,6 @@ INSTANTIATE_TEST_CASE_P(data_type, {"Int8 String", nullptr}, {"Nothing", nullptr}, {"FixedString(16) FixedString(8) String", nullptr}, - }), + } + ) ); diff --git a/dbms/src/Databases/DatabaseLazy.cpp b/dbms/src/Databases/DatabaseLazy.cpp index e9a90b29341..fc71b3a63a7 100644 --- a/dbms/src/Databases/DatabaseLazy.cpp +++ b/dbms/src/Databases/DatabaseLazy.cpp @@ -23,7 +23,6 @@ namespace ErrorCodes extern const int TABLE_ALREADY_EXISTS; extern const int UNKNOWN_TABLE; extern const int UNSUPPORTED_METHOD; - extern const int CANNOT_CREATE_TABLE_FROM_METADATA; extern const int LOGICAL_ERROR; } @@ -255,10 +254,10 @@ StoragePtr DatabaseLazy::loadTable(const Context & context, const String & table return it->second.table = table; } } - catch (const Exception & e) + catch (Exception & e) { - throw Exception("Cannot create table from metadata file " + table_metadata_path + ". Error: " + DB::getCurrentExceptionMessage(true), - e, DB::ErrorCodes::CANNOT_CREATE_TABLE_FROM_METADATA); + e.addMessage("Cannot create table from metadata file " + table_metadata_path); + throw; } } diff --git a/dbms/src/Databases/DatabaseOrdinary.cpp b/dbms/src/Databases/DatabaseOrdinary.cpp index 02f84671475..f9f6983604c 100644 --- a/dbms/src/Databases/DatabaseOrdinary.cpp +++ b/dbms/src/Databases/DatabaseOrdinary.cpp @@ -10,8 +10,6 @@ #include #include #include -#include -#include #include #include #include @@ -38,7 +36,6 @@ namespace DB namespace ErrorCodes { - extern const int CANNOT_CREATE_TABLE_FROM_METADATA; extern const int CANNOT_CREATE_DICTIONARY_FROM_METADATA; extern const int EMPTY_LIST_OF_COLUMNS_PASSED; extern const int CANNOT_PARSE_TEXT; @@ -68,13 +65,10 @@ namespace = createTableFromAST(query, database_name, database.getTableDataPath(query), context, has_force_restore_data_flag); database.attachTable(table_name, table); } - catch (const Exception & e) + catch (Exception & e) { - throw Exception( - "Cannot attach table '" + query.table + "' from query " + serializeAST(query) - + ". Error: " + DB::getCurrentExceptionMessage(true), - e, - DB::ErrorCodes::CANNOT_CREATE_TABLE_FROM_METADATA); + e.addMessage("Cannot attach table '" + backQuote(query.table) + "' from query " + serializeAST(query)); + throw; } } @@ -89,13 +83,10 @@ namespace { database.attachDictionary(query.table, context); } - catch (const Exception & e) + catch (Exception & e) { - throw Exception( - "Cannot create dictionary '" + query.table + "' from query " + serializeAST(query) - + ". Error: " + DB::getCurrentExceptionMessage(true), - e, - DB::ErrorCodes::CANNOT_CREATE_DICTIONARY_FROM_METADATA); + e.addMessage("Cannot attach table '" + backQuote(query.table) + "' from query " + serializeAST(query)); + throw; } } @@ -144,10 +135,10 @@ void DatabaseOrdinary::loadStoredObjects( total_dictionaries += create_query->is_dictionary; } } - catch (const Exception & e) + catch (Exception & e) { - throw Exception( - "Cannot parse definition from metadata file " + full_path + ". Error: " + DB::getCurrentExceptionMessage(true), e, ErrorCodes::CANNOT_PARSE_TEXT); + e.addMessage("Cannot parse definition from metadata file " + full_path); + throw; } }); @@ -181,12 +172,8 @@ void DatabaseOrdinary::loadStoredObjects( /// After all tables was basically initialized, startup them. startupTables(pool); - /// Add database as repository - auto dictionaries_repository = std::make_unique(shared_from_this(), context); - auto & external_loader = context.getExternalDictionariesLoader(); - external_loader.addConfigRepository(getDatabaseName(), std::move(dictionaries_repository)); - /// Attach dictionaries. + attachToExternalDictionariesLoader(context); for (const auto & name_with_query : file_names) { auto create_query = name_with_query.second->as(); diff --git a/dbms/src/Databases/DatabaseWithDictionaries.cpp b/dbms/src/Databases/DatabaseWithDictionaries.cpp index a10348b31ef..716ed32b676 100644 --- a/dbms/src/Databases/DatabaseWithDictionaries.cpp +++ b/dbms/src/Databases/DatabaseWithDictionaries.cpp @@ -1,6 +1,7 @@ #include #include -#include +#include +#include #include #include #include @@ -74,7 +75,7 @@ void DatabaseWithDictionaries::createDictionary(const Context & context, const S /// A dictionary with the same full name could be defined in *.xml config files. String full_name = getDatabaseName() + "." + dictionary_name; - auto & external_loader = const_cast(context.getExternalDictionariesLoader()); + const auto & external_loader = context.getExternalDictionariesLoader(); if (external_loader.getCurrentStatus(full_name) != ExternalLoader::Status::NOT_EXIST) throw Exception( "Dictionary " + backQuote(getDatabaseName()) + "." + backQuote(dictionary_name) + " already exists.", @@ -106,15 +107,10 @@ void DatabaseWithDictionaries::createDictionary(const Context & context, const S /// Add a temporary repository containing the dictionary. /// We need this temp repository to try loading the dictionary before actually attaching it to the database. - static std::atomic counter = 0; - String temp_repository_name = String(IExternalLoaderConfigRepository::INTERNAL_REPOSITORY_NAME_PREFIX) + " creating " + full_name + " " - + std::to_string(++counter); - external_loader.addConfigRepository( - temp_repository_name, - std::make_unique( - std::vector{std::pair{dictionary_metadata_tmp_path, - getDictionaryConfigurationFromAST(query->as(), getDatabaseName())}})); - SCOPE_EXIT({ external_loader.removeConfigRepository(temp_repository_name); }); + auto temp_repository + = const_cast(external_loader) /// the change of ExternalDictionariesLoader is temporary + .addConfigRepository(std::make_unique( + getDatabaseName(), dictionary_metadata_tmp_path, getDictionaryConfigurationFromAST(query->as()))); bool lazy_load = context.getConfigRef().getBool("dictionaries_lazy_load", true); if (!lazy_load) @@ -253,4 +249,23 @@ ASTPtr DatabaseWithDictionaries::getCreateDictionaryQueryImpl( return ast; } +void DatabaseWithDictionaries::shutdown() +{ + detachFromExternalDictionariesLoader(); + DatabaseOnDisk::shutdown(); +} + +DatabaseWithDictionaries::~DatabaseWithDictionaries() = default; + +void DatabaseWithDictionaries::attachToExternalDictionariesLoader(Context & context) +{ + database_as_config_repo_for_external_loader = context.getExternalDictionariesLoader().addConfigRepository( + std::make_unique(*this, context)); +} + +void DatabaseWithDictionaries::detachFromExternalDictionariesLoader() +{ + database_as_config_repo_for_external_loader = {}; +} + } diff --git a/dbms/src/Databases/DatabaseWithDictionaries.h b/dbms/src/Databases/DatabaseWithDictionaries.h index 27181a10902..c16e11f24c5 100644 --- a/dbms/src/Databases/DatabaseWithDictionaries.h +++ b/dbms/src/Databases/DatabaseWithDictionaries.h @@ -1,4 +1,5 @@ #include +#include namespace DB { @@ -25,15 +26,25 @@ public: bool isDictionaryExist(const Context & context, const String & dictionary_name) const override; + void shutdown() override; + + ~DatabaseWithDictionaries() override; + protected: DatabaseWithDictionaries(const String & name, const String & metadata_path_, const String & logger) : DatabaseOnDisk(name, metadata_path_, logger) {} + void attachToExternalDictionariesLoader(Context & context); + void detachFromExternalDictionariesLoader(); + StoragePtr getDictionaryStorage(const Context & context, const String & table_name) const; ASTPtr getCreateDictionaryQueryImpl(const Context & context, const String & dictionary_name, bool throw_on_error) const override; + +private: + ext::scope_guard database_as_config_repo_for_external_loader; }; } diff --git a/dbms/src/Dictionaries/CacheDictionary.cpp b/dbms/src/Dictionaries/CacheDictionary.cpp index 4dcb87c7b8a..78ab9964e5b 100644 --- a/dbms/src/Dictionaries/CacheDictionary.cpp +++ b/dbms/src/Dictionaries/CacheDictionary.cpp @@ -57,12 +57,15 @@ inline size_t CacheDictionary::getCellIdx(const Key id) const CacheDictionary::CacheDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, const size_t size_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -73,7 +76,7 @@ CacheDictionary::CacheDictionary( , rnd_engine(randomSeed()) { if (!this->source_ptr->supportsSelectiveLoad()) - throw Exception{name + ": source cannot be used with CacheDictionary", ErrorCodes::UNSUPPORTED_METHOD}; + throw Exception{full_name + ": source cannot be used with CacheDictionary", ErrorCodes::UNSUPPORTED_METHOD}; createAttributes(); } @@ -204,7 +207,7 @@ void CacheDictionary::isInConstantVector(const Key child_id, const PaddedPODArra void CacheDictionary::getString(const std::string & attribute_name, const PaddedPODArray & ids, ColumnString * out) const { auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto null_value = StringRef{std::get(attribute.null_values)}; @@ -215,7 +218,7 @@ void CacheDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const ColumnString * const def, ColumnString * const out) const { auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsString(attribute, ids, out, [&](const size_t row) { return def->getDataAt(row); }); } @@ -224,7 +227,7 @@ void CacheDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const String & def, ColumnString * const out) const { auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsString(attribute, ids, out, [&](const size_t) { return StringRef{def}; }); } @@ -352,7 +355,7 @@ void CacheDictionary::createAttributes() hierarchical_attribute = &attributes.back(); if (hierarchical_attribute->type != AttributeUnderlyingType::utUInt64) - throw Exception{name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; + throw Exception{full_name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; } } } @@ -539,7 +542,7 @@ CacheDictionary::Attribute & CacheDictionary::getAttribute(const std::string & a { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -580,7 +583,7 @@ std::exception_ptr CacheDictionary::getLastException() const void registerDictionaryCache(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string & full_name, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -590,22 +593,24 @@ void registerDictionaryCache(DictionaryFactory & factory) throw Exception{"'key' is not supported for dictionary of layout 'cache'", ErrorCodes::UNSUPPORTED_METHOD}; if (dict_struct.range_min || dict_struct.range_max) - throw Exception{name + throw Exception{full_name + ": elements .structure.range_min and .structure.range_max should be defined only " "for a dictionary of layout 'range_hashed'", ErrorCodes::BAD_ARGUMENTS}; const auto & layout_prefix = config_prefix + ".layout"; const auto size = config.getInt(layout_prefix + ".cache.size_in_cells"); if (size == 0) - throw Exception{name + ": dictionary of layout 'cache' cannot have 0 cells", ErrorCodes::TOO_SMALL_BUFFER_SIZE}; + throw Exception{full_name + ": dictionary of layout 'cache' cannot have 0 cells", ErrorCodes::TOO_SMALL_BUFFER_SIZE}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); if (require_nonempty) - throw Exception{name + ": dictionary of layout 'cache' cannot have 'require_nonempty' attribute set", + throw Exception{full_name + ": dictionary of layout 'cache' cannot have 'require_nonempty' attribute set", ErrorCodes::BAD_ARGUMENTS}; + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, size); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, size); }; factory.registerLayout("cache", create_layout, false); } diff --git a/dbms/src/Dictionaries/CacheDictionary.h b/dbms/src/Dictionaries/CacheDictionary.h index b5065a63922..d780b557c03 100644 --- a/dbms/src/Dictionaries/CacheDictionary.h +++ b/dbms/src/Dictionaries/CacheDictionary.h @@ -25,13 +25,16 @@ class CacheDictionary final : public IDictionary { public: CacheDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, const size_t size_); - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "Cache"; } @@ -52,7 +55,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, size); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, size); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -254,7 +257,9 @@ private: template void isInImpl(const PaddedPODArray & child_ids, const AncestorType & ancestor_ids, PaddedPODArray & out) const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; mutable DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/CacheDictionary.inc.h b/dbms/src/Dictionaries/CacheDictionary.inc.h index 87005ac821f..a3a6937e9c5 100644 --- a/dbms/src/Dictionaries/CacheDictionary.inc.h +++ b/dbms/src/Dictionaries/CacheDictionary.inc.h @@ -333,7 +333,7 @@ void CacheDictionary::update( last_exception = std::current_exception(); backoff_end_time = now + std::chrono::seconds(calculateDurationWithBackoff(rnd_engine, error_count)); - tryLogException(last_exception, log, "Could not update cache dictionary '" + getName() + + tryLogException(last_exception, log, "Could not update cache dictionary '" + getFullName() + "', next update is scheduled at " + ext::to_string(backoff_end_time)); } } diff --git a/dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp b/dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp index 8ed917e8f89..e16f389e1ce 100644 --- a/dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp +++ b/dbms/src/Dictionaries/ComplexKeyCacheDictionary.cpp @@ -51,12 +51,15 @@ inline UInt64 ComplexKeyCacheDictionary::getCellIdx(const StringRef key) const ComplexKeyCacheDictionary::ComplexKeyCacheDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, const size_t size_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -65,7 +68,7 @@ ComplexKeyCacheDictionary::ComplexKeyCacheDictionary( , rnd_engine(randomSeed()) { if (!this->source_ptr->supportsSelectiveLoad()) - throw Exception{name + ": source cannot be used with ComplexKeyCacheDictionary", ErrorCodes::UNSUPPORTED_METHOD}; + throw Exception{full_name + ": source cannot be used with ComplexKeyCacheDictionary", ErrorCodes::UNSUPPORTED_METHOD}; createAttributes(); } @@ -77,7 +80,7 @@ void ComplexKeyCacheDictionary::getString( dict_struct.validateKeyTypes(key_types); auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto null_value = StringRef{std::get(attribute.null_values)}; @@ -94,7 +97,7 @@ void ComplexKeyCacheDictionary::getString( dict_struct.validateKeyTypes(key_types); auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsString(attribute, key_columns, out, [&](const size_t row) { return def->getDataAt(row); }); } @@ -109,7 +112,7 @@ void ComplexKeyCacheDictionary::getString( dict_struct.validateKeyTypes(key_types); auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsString(attribute, key_columns, out, [&](const size_t) { return StringRef{def}; }); } @@ -249,7 +252,7 @@ void ComplexKeyCacheDictionary::createAttributes() attributes.push_back(createAttributeWithType(attribute.underlying_type, attribute.null_value)); if (attribute.hierarchical) - throw Exception{name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), + throw Exception{full_name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), ErrorCodes::TYPE_MISMATCH}; } } @@ -258,7 +261,7 @@ ComplexKeyCacheDictionary::Attribute & ComplexKeyCacheDictionary::getAttribute(c { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -394,7 +397,7 @@ BlockInputStreamPtr ComplexKeyCacheDictionary::getBlockInputStream(const Names & void registerDictionaryComplexKeyCache(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string & full_name, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -405,15 +408,17 @@ void registerDictionaryComplexKeyCache(DictionaryFactory & factory) const auto & layout_prefix = config_prefix + ".layout"; const auto size = config.getInt(layout_prefix + ".complex_key_cache.size_in_cells"); if (size == 0) - throw Exception{name + ": dictionary of layout 'cache' cannot have 0 cells", ErrorCodes::TOO_SMALL_BUFFER_SIZE}; + throw Exception{full_name + ": dictionary of layout 'cache' cannot have 0 cells", ErrorCodes::TOO_SMALL_BUFFER_SIZE}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); if (require_nonempty) - throw Exception{name + ": dictionary of layout 'cache' cannot have 'require_nonempty' attribute set", + throw Exception{full_name + ": dictionary of layout 'cache' cannot have 'require_nonempty' attribute set", ErrorCodes::BAD_ARGUMENTS}; + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, size); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, size); }; factory.registerLayout("complex_key_cache", create_layout, true); } diff --git a/dbms/src/Dictionaries/ComplexKeyCacheDictionary.h b/dbms/src/Dictionaries/ComplexKeyCacheDictionary.h index e9269cb165a..4547a305f1d 100644 --- a/dbms/src/Dictionaries/ComplexKeyCacheDictionary.h +++ b/dbms/src/Dictionaries/ComplexKeyCacheDictionary.h @@ -42,6 +42,7 @@ class ComplexKeyCacheDictionary final : public IDictionaryBase { public: ComplexKeyCacheDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -50,7 +51,9 @@ public: std::string getKeyDescription() const { return key_description; } - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "ComplexKeyCache"; } @@ -75,7 +78,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, size); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, size); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -668,7 +671,9 @@ private: bool isEmptyCell(const UInt64 idx) const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp b/dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp index 1dafde39a24..7d8d481e2fa 100644 --- a/dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp +++ b/dbms/src/Dictionaries/ComplexKeyHashedDictionary.cpp @@ -15,13 +15,16 @@ namespace ErrorCodes } ComplexKeyHashedDictionary::ComplexKeyHashedDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, bool require_nonempty_, BlockPtr saved_block_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -40,7 +43,7 @@ ComplexKeyHashedDictionary::ComplexKeyHashedDictionary( dict_struct.validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ const auto null_value = std::get(attribute.null_values); \ \ @@ -72,7 +75,7 @@ void ComplexKeyHashedDictionary::getString( dict_struct.validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto & null_value = StringRef{std::get(attribute.null_values)}; @@ -94,7 +97,7 @@ void ComplexKeyHashedDictionary::getString( dict_struct.validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, \ @@ -128,7 +131,7 @@ void ComplexKeyHashedDictionary::getString( dict_struct.validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -148,7 +151,7 @@ void ComplexKeyHashedDictionary::getString( dict_struct.validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ @@ -179,7 +182,7 @@ void ComplexKeyHashedDictionary::getString( dict_struct.validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -256,7 +259,7 @@ void ComplexKeyHashedDictionary::createAttributes() attributes.push_back(createAttributeWithType(attribute.underlying_type, attribute.null_value)); if (attribute.hierarchical) - throw Exception{name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), + throw Exception{full_name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), ErrorCodes::TYPE_MISMATCH}; } } @@ -397,7 +400,7 @@ void ComplexKeyHashedDictionary::loadData() updateData(); if (require_nonempty && 0 == element_count) - throw Exception{name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; + throw Exception{full_name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; } template @@ -630,7 +633,7 @@ const ComplexKeyHashedDictionary::Attribute & ComplexKeyHashedDictionary::getAtt { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -742,7 +745,7 @@ BlockInputStreamPtr ComplexKeyHashedDictionary::getBlockInputStream(const Names void registerDictionaryComplexKeyHashed(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string &, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -751,12 +754,13 @@ void registerDictionaryComplexKeyHashed(DictionaryFactory & factory) if (!dict_struct.key) throw Exception{"'key' is required for dictionary of layout 'complex_key_hashed'", ErrorCodes::BAD_ARGUMENTS}; + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); }; factory.registerLayout("complex_key_hashed", create_layout, true); } - } diff --git a/dbms/src/Dictionaries/ComplexKeyHashedDictionary.h b/dbms/src/Dictionaries/ComplexKeyHashedDictionary.h index 77941d6c5df..82b2a93b010 100644 --- a/dbms/src/Dictionaries/ComplexKeyHashedDictionary.h +++ b/dbms/src/Dictionaries/ComplexKeyHashedDictionary.h @@ -23,6 +23,7 @@ class ComplexKeyHashedDictionary final : public IDictionaryBase { public: ComplexKeyHashedDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -32,7 +33,9 @@ public: std::string getKeyDescription() const { return key_description; } - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "ComplexKeyHashed"; } @@ -48,7 +51,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, saved_block); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, saved_block); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -233,7 +236,9 @@ private: template std::vector getKeys(const Attribute & attribute) const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/FlatDictionary.cpp b/dbms/src/Dictionaries/FlatDictionary.cpp index 68afdd355b8..a26d566e10c 100644 --- a/dbms/src/Dictionaries/FlatDictionary.cpp +++ b/dbms/src/Dictionaries/FlatDictionary.cpp @@ -21,13 +21,16 @@ static const auto max_array_size = 500000; FlatDictionary::FlatDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, bool require_nonempty_, BlockPtr saved_block_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -107,7 +110,7 @@ void FlatDictionary::isInConstantVector(const Key child_id, const PaddedPODArray void FlatDictionary::get##TYPE(const std::string & attribute_name, const PaddedPODArray & ids, ResultArrayType & out) const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ const auto null_value = std::get(attribute.null_values); \ \ @@ -133,7 +136,7 @@ DECLARE(Decimal128) void FlatDictionary::getString(const std::string & attribute_name, const PaddedPODArray & ids, ColumnString * out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto & null_value = std::get(attribute.null_values); @@ -152,7 +155,7 @@ void FlatDictionary::getString(const std::string & attribute_name, const PaddedP ResultArrayType & out) const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \ @@ -177,7 +180,7 @@ void FlatDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const ColumnString * const def, ColumnString * const out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -191,7 +194,7 @@ void FlatDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const TYPE def, ResultArrayType & out) const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ @@ -216,7 +219,7 @@ void FlatDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const String & def, ColumnString * const out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); FlatDictionary::getItemsImpl( attribute, @@ -297,7 +300,7 @@ void FlatDictionary::createAttributes() hierarchical_attribute = &attributes.back(); if (hierarchical_attribute->type != AttributeUnderlyingType::utUInt64) - throw Exception{name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; + throw Exception{full_name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; } } } @@ -404,7 +407,7 @@ void FlatDictionary::loadData() updateData(); if (require_nonempty && 0 == element_count) - throw Exception{name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; + throw Exception{full_name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; } @@ -578,7 +581,7 @@ template void FlatDictionary::resize(Attribute & attribute, const Key id) { if (id >= max_array_size) - throw Exception{name + ": identifier should be less than " + toString(max_array_size), ErrorCodes::ARGUMENT_OUT_OF_BOUND}; + throw Exception{full_name + ": identifier should be less than " + toString(max_array_size), ErrorCodes::ARGUMENT_OUT_OF_BOUND}; auto & array = std::get>(attribute.arrays); if (id >= array.size()) @@ -666,7 +669,7 @@ const FlatDictionary::Attribute & FlatDictionary::getAttribute(const std::string { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -706,7 +709,7 @@ BlockInputStreamPtr FlatDictionary::getBlockInputStream(const Names & column_nam void registerDictionaryFlat(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string & full_name, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -716,13 +719,16 @@ void registerDictionaryFlat(DictionaryFactory & factory) throw Exception{"'key' is not supported for dictionary of layout 'flat'", ErrorCodes::UNSUPPORTED_METHOD}; if (dict_struct.range_min || dict_struct.range_max) - throw Exception{name + throw Exception{full_name + ": elements .structure.range_min and .structure.range_max should be defined only " "for a dictionary of layout 'range_hashed'", ErrorCodes::BAD_ARGUMENTS}; + + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); }; factory.registerLayout("flat", create_layout, false); } diff --git a/dbms/src/Dictionaries/FlatDictionary.h b/dbms/src/Dictionaries/FlatDictionary.h index 1bb06348aab..636c7b9d092 100644 --- a/dbms/src/Dictionaries/FlatDictionary.h +++ b/dbms/src/Dictionaries/FlatDictionary.h @@ -22,6 +22,7 @@ class FlatDictionary final : public IDictionary { public: FlatDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -29,7 +30,9 @@ public: bool require_nonempty_, BlockPtr saved_block_ = nullptr); - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "Flat"; } @@ -45,7 +48,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, saved_block); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, saved_block); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -222,7 +225,9 @@ private: PaddedPODArray getIds() const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/HashedDictionary.cpp b/dbms/src/Dictionaries/HashedDictionary.cpp index 78c871bebc4..025e2e040b9 100644 --- a/dbms/src/Dictionaries/HashedDictionary.cpp +++ b/dbms/src/Dictionaries/HashedDictionary.cpp @@ -31,6 +31,7 @@ namespace ErrorCodes HashedDictionary::HashedDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -38,7 +39,9 @@ HashedDictionary::HashedDictionary( bool require_nonempty_, bool sparse_, BlockPtr saved_block_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -129,7 +132,7 @@ void HashedDictionary::isInConstantVector(const Key child_id, const PaddedPODArr const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ const auto null_value = std::get(attribute.null_values); \ \ @@ -155,7 +158,7 @@ DECLARE(Decimal128) void HashedDictionary::getString(const std::string & attribute_name, const PaddedPODArray & ids, ColumnString * out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto & null_value = StringRef{std::get(attribute.null_values)}; @@ -174,7 +177,7 @@ void HashedDictionary::getString(const std::string & attribute_name, const Padde ResultArrayType & out) const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t row) { return def[row]; }); \ @@ -199,7 +202,7 @@ void HashedDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const ColumnString * const def, ColumnString * const out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -213,7 +216,7 @@ void HashedDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const TYPE & def, ResultArrayType & out) const \ { \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, ids, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ @@ -238,7 +241,7 @@ void HashedDictionary::getString( const std::string & attribute_name, const PaddedPODArray & ids, const String & def, ColumnString * const out) const { const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -317,7 +320,7 @@ void HashedDictionary::createAttributes() hierarchical_attribute = &attributes.back(); if (hierarchical_attribute->type != AttributeUnderlyingType::utUInt64) - throw Exception{name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; + throw Exception{full_name + ": hierarchical attribute must be UInt64.", ErrorCodes::TYPE_MISMATCH}; } } } @@ -424,7 +427,7 @@ void HashedDictionary::loadData() updateData(); if (require_nonempty && 0 == element_count) - throw Exception{name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; + throw Exception{full_name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; } template @@ -684,7 +687,7 @@ const HashedDictionary::Attribute & HashedDictionary::getAttribute(const std::st { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -768,27 +771,31 @@ BlockInputStreamPtr HashedDictionary::getBlockInputStream(const Names & column_n void registerDictionaryHashed(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string & full_name, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, - DictionarySourcePtr source_ptr) -> DictionaryPtr + DictionarySourcePtr source_ptr, + bool sparse) -> DictionaryPtr { if (dict_struct.key) throw Exception{"'key' is not supported for dictionary of layout 'hashed'", ErrorCodes::UNSUPPORTED_METHOD}; if (dict_struct.range_min || dict_struct.range_max) - throw Exception{name + throw Exception{full_name + ": elements .structure.range_min and .structure.range_max should be defined only " "for a dictionary of layout 'range_hashed'", ErrorCodes::BAD_ARGUMENTS}; + + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); - const bool sparse = name == "sparse_hashed"; - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty, sparse); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty, sparse); }; - factory.registerLayout("hashed", create_layout, false); - factory.registerLayout("sparse_hashed", create_layout, false); + using namespace std::placeholders; + factory.registerLayout("hashed", std::bind(create_layout, _1, _2, _3, _4, _5, /* sparse = */ false), false); + factory.registerLayout("sparse_hashed", std::bind(create_layout, _1, _2, _3, _4, _5, /* sparse = */ true), false); } } diff --git a/dbms/src/Dictionaries/HashedDictionary.h b/dbms/src/Dictionaries/HashedDictionary.h index d4f55dc8e39..3f8eec979bb 100644 --- a/dbms/src/Dictionaries/HashedDictionary.h +++ b/dbms/src/Dictionaries/HashedDictionary.h @@ -26,6 +26,7 @@ class HashedDictionary final : public IDictionary { public: HashedDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -34,7 +35,9 @@ public: bool sparse_, BlockPtr saved_block_ = nullptr); - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return sparse ? "SparseHashed" : "Hashed"; } @@ -50,7 +53,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, sparse, saved_block); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty, sparse, saved_block); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -262,7 +265,9 @@ private: template void isInImpl(const ChildType & child_ids, const AncestorType & ancestor_ids, PaddedPODArray & out) const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/IDictionary.h b/dbms/src/Dictionaries/IDictionary.h index 9c74c98e88a..e2b6f078a8e 100644 --- a/dbms/src/Dictionaries/IDictionary.h +++ b/dbms/src/Dictionaries/IDictionary.h @@ -25,6 +25,11 @@ struct IDictionaryBase : public IExternalLoadable { using Key = UInt64; + virtual const std::string & getDatabase() const = 0; + virtual const std::string & getName() const = 0; + virtual const std::string & getFullName() const = 0; + const std::string & getLoadableName() const override { return getFullName(); } + virtual std::string getTypeName() const = 0; virtual size_t getBytesAllocated() const = 0; diff --git a/dbms/src/Dictionaries/RangeHashedDictionary.cpp b/dbms/src/Dictionaries/RangeHashedDictionary.cpp index 1d80ea8c497..8fc16da2e32 100644 --- a/dbms/src/Dictionaries/RangeHashedDictionary.cpp +++ b/dbms/src/Dictionaries/RangeHashedDictionary.cpp @@ -68,12 +68,15 @@ static bool operator<(const RangeHashedDictionary::Range & left, const RangeHash RangeHashedDictionary::RangeHashedDictionary( - const std::string & dictionary_name_, + const std::string & database_, + const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, bool require_nonempty_) - : dictionary_name{dictionary_name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -156,7 +159,7 @@ void RangeHashedDictionary::createAttributes() attributes.push_back(createAttributeWithType(attribute.underlying_type, attribute.null_value)); if (attribute.hierarchical) - throw Exception{dictionary_name + ": hierarchical attributes not supported by " + getName() + " dictionary.", + throw Exception{full_name + ": hierarchical attributes not supported by " + getName() + " dictionary.", ErrorCodes::BAD_ARGUMENTS}; } } @@ -207,7 +210,7 @@ void RangeHashedDictionary::loadData() stream->readSuffix(); if (require_nonempty && 0 == element_count) - throw Exception{dictionary_name + ": dictionary source is empty and 'require_nonempty' property is set.", + throw Exception{full_name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; } @@ -520,7 +523,7 @@ const RangeHashedDictionary::Attribute & RangeHashedDictionary::getAttribute(con { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{dictionary_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -674,7 +677,7 @@ BlockInputStreamPtr RangeHashedDictionary::getBlockInputStream(const Names & col void registerDictionaryRangeHashed(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string & full_name, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -684,12 +687,14 @@ void registerDictionaryRangeHashed(DictionaryFactory & factory) throw Exception{"'key' is not supported for dictionary of layout 'range_hashed'", ErrorCodes::UNSUPPORTED_METHOD}; if (!dict_struct.range_min || !dict_struct.range_max) - throw Exception{name + ": dictionary of layout 'range_hashed' requires .structure.range_min and .structure.range_max", + throw Exception{full_name + ": dictionary of layout 'range_hashed' requires .structure.range_min and .structure.range_max", ErrorCodes::BAD_ARGUMENTS}; + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); }; factory.registerLayout("range_hashed", create_layout, false); } diff --git a/dbms/src/Dictionaries/RangeHashedDictionary.h b/dbms/src/Dictionaries/RangeHashedDictionary.h index 829553c68b3..eba10bbbdbb 100644 --- a/dbms/src/Dictionaries/RangeHashedDictionary.h +++ b/dbms/src/Dictionaries/RangeHashedDictionary.h @@ -18,13 +18,16 @@ class RangeHashedDictionary final : public IDictionaryBase { public: RangeHashedDictionary( - const std::string & dictionary_name_, + const std::string & database_, + const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, bool require_nonempty_); - std::string getName() const override { return dictionary_name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "RangeHashed"; } @@ -40,7 +43,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(dictionary_name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -208,7 +211,9 @@ private: friend struct RangeHashedDIctionaryCallGetBlockInputStreamImpl; - const std::string dictionary_name; + const std::string database; + const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/TrieDictionary.cpp b/dbms/src/Dictionaries/TrieDictionary.cpp index 4432a1ec548..a16d30e32e9 100644 --- a/dbms/src/Dictionaries/TrieDictionary.cpp +++ b/dbms/src/Dictionaries/TrieDictionary.cpp @@ -35,12 +35,15 @@ namespace ErrorCodes } TrieDictionary::TrieDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, const DictionaryLifetime dict_lifetime_, bool require_nonempty_) - : name{name_} + : database(database_) + , name(name_) + , full_name{database_.empty() ? name_ : (database_ + "." + name_)} , dict_struct(dict_struct_) , source_ptr{std::move(source_ptr_)} , dict_lifetime(dict_lifetime_) @@ -75,7 +78,7 @@ TrieDictionary::~TrieDictionary() validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ const auto null_value = std::get(attribute.null_values); \ \ @@ -107,7 +110,7 @@ void TrieDictionary::getString( validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); const auto & null_value = StringRef{std::get(attribute.null_values)}; @@ -129,7 +132,7 @@ void TrieDictionary::getString( validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, \ @@ -163,7 +166,7 @@ void TrieDictionary::getString( validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -183,7 +186,7 @@ void TrieDictionary::getString( validateKeyTypes(key_types); \ \ const auto & attribute = getAttribute(attribute_name); \ - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::ut##TYPE); \ \ getItemsImpl( \ attribute, key_columns, [&](const size_t row, const auto value) { out[row] = value; }, [&](const size_t) { return def; }); \ @@ -214,7 +217,7 @@ void TrieDictionary::getString( validateKeyTypes(key_types); const auto & attribute = getAttribute(attribute_name); - checkAttributeType(name, attribute_name, attribute.type, AttributeUnderlyingType::utString); + checkAttributeType(full_name, attribute_name, attribute.type, AttributeUnderlyingType::utString); getItemsImpl( attribute, @@ -291,7 +294,7 @@ void TrieDictionary::createAttributes() attributes.push_back(createAttributeWithType(attribute.underlying_type, attribute.null_value)); if (attribute.hierarchical) - throw Exception{name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), + throw Exception{full_name + ": hierarchical attributes not supported for dictionary of type " + getTypeName(), ErrorCodes::TYPE_MISMATCH}; } } @@ -337,7 +340,7 @@ void TrieDictionary::loadData() stream->readSuffix(); if (require_nonempty && 0 == element_count) - throw Exception{name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; + throw Exception{full_name + ": dictionary source is empty and 'require_nonempty' property is set.", ErrorCodes::DICTIONARY_IS_EMPTY}; } template @@ -627,7 +630,7 @@ const TrieDictionary::Attribute & TrieDictionary::getAttribute(const std::string { const auto it = attribute_index_by_name.find(attribute_name); if (it == std::end(attribute_index_by_name)) - throw Exception{name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; + throw Exception{full_name + ": no such attribute '" + attribute_name + "'", ErrorCodes::BAD_ARGUMENTS}; return attributes[it->second]; } @@ -767,7 +770,7 @@ BlockInputStreamPtr TrieDictionary::getBlockInputStream(const Names & column_nam void registerDictionaryTrie(DictionaryFactory & factory) { - auto create_layout = [=](const std::string & name, + auto create_layout = [=](const std::string &, const DictionaryStructure & dict_struct, const Poco::Util::AbstractConfiguration & config, const std::string & config_prefix, @@ -776,10 +779,12 @@ void registerDictionaryTrie(DictionaryFactory & factory) if (!dict_struct.key) throw Exception{"'key' is required for dictionary of layout 'ip_trie'", ErrorCodes::BAD_ARGUMENTS}; + const String database = config.getString(config_prefix + ".database", ""); + const String name = config.getString(config_prefix + ".name"); const DictionaryLifetime dict_lifetime{config, config_prefix + ".lifetime"}; const bool require_nonempty = config.getBool(config_prefix + ".require_nonempty", false); // This is specialised trie for storing IPv4 and IPv6 prefixes. - return std::make_unique(name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); + return std::make_unique(database, name, dict_struct, std::move(source_ptr), dict_lifetime, require_nonempty); }; factory.registerLayout("ip_trie", create_layout, true); } diff --git a/dbms/src/Dictionaries/TrieDictionary.h b/dbms/src/Dictionaries/TrieDictionary.h index 7e41942b873..5168eec5a74 100644 --- a/dbms/src/Dictionaries/TrieDictionary.h +++ b/dbms/src/Dictionaries/TrieDictionary.h @@ -23,6 +23,7 @@ class TrieDictionary final : public IDictionaryBase { public: TrieDictionary( + const std::string & database_, const std::string & name_, const DictionaryStructure & dict_struct_, DictionarySourcePtr source_ptr_, @@ -33,7 +34,9 @@ public: std::string getKeyDescription() const { return key_description; } - std::string getName() const override { return name; } + const std::string & getDatabase() const override { return database; } + const std::string & getName() const override { return name; } + const std::string & getFullName() const override { return full_name; } std::string getTypeName() const override { return "Trie"; } @@ -49,7 +52,7 @@ public: std::shared_ptr clone() const override { - return std::make_shared(name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty); + return std::make_shared(database, name, dict_struct, source_ptr->clone(), dict_lifetime, require_nonempty); } const IDictionarySource * getSource() const override { return source_ptr.get(); } @@ -232,7 +235,9 @@ private: Columns getKeyColumns() const; + const std::string database; const std::string name; + const std::string full_name; const DictionaryStructure dict_struct; const DictionarySourcePtr source_ptr; const DictionaryLifetime dict_lifetime; diff --git a/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.cpp b/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.cpp index f0e49bcc4ac..dbbcc0e41a8 100644 --- a/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.cpp +++ b/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.cpp @@ -421,7 +421,7 @@ void checkPrimaryKey(const NamesToTypeNames & all_attrs, const Names & key_attrs } -DictionaryConfigurationPtr getDictionaryConfigurationFromAST(const ASTCreateQuery & query, const String & database_name) +DictionaryConfigurationPtr getDictionaryConfigurationFromAST(const ASTCreateQuery & query) { checkAST(query); @@ -434,10 +434,14 @@ DictionaryConfigurationPtr getDictionaryConfigurationFromAST(const ASTCreateQuer AutoPtr name_element(xml_document->createElement("name")); current_dictionary->appendChild(name_element); - String full_name = (!database_name.empty() ? database_name : query.database) + "." + query.table; - AutoPtr name(xml_document->createTextNode(full_name)); + AutoPtr name(xml_document->createTextNode(query.table)); name_element->appendChild(name); + AutoPtr database_element(xml_document->createElement("database")); + current_dictionary->appendChild(database_element); + AutoPtr database(xml_document->createTextNode(query.database)); + database_element->appendChild(database); + AutoPtr structure_element(xml_document->createElement("structure")); current_dictionary->appendChild(structure_element); Names pk_attrs = getPrimaryKeyColumns(query.dictionary->primary_key); diff --git a/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.h b/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.h index adfcd7f2768..bb48765c492 100644 --- a/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.h +++ b/dbms/src/Dictionaries/getDictionaryConfigurationFromAST.h @@ -10,6 +10,6 @@ using DictionaryConfigurationPtr = Poco::AutoPtrgetString("dictionary.name"), "test.dict1"); + EXPECT_EQ(config->getString("dictionary.database"), "test"); + EXPECT_EQ(config->getString("dictionary.name"), "dict1"); /// lifetime EXPECT_EQ(config->getInt("dictionary.lifetime.min"), 1); diff --git a/dbms/src/Functions/FunctionsExternalDictionaries.h b/dbms/src/Functions/FunctionsExternalDictionaries.h index 33cb05e2e7b..94bc861de13 100644 --- a/dbms/src/Functions/FunctionsExternalDictionaries.h +++ b/dbms/src/Functions/FunctionsExternalDictionaries.h @@ -127,10 +127,10 @@ private: auto dict = dictionaries_loader.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); - if (!context.hasDictionaryAccessRights(dict_ptr->getName())) + if (!context.hasDictionaryAccessRights(dict_ptr->getFullName())) { throw Exception{"For function " + getName() + ", cannot access dictionary " - + dict->getName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; + + dict->getFullName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; } if (!executeDispatchSimple(block, arguments, result, dict_ptr) && @@ -302,10 +302,10 @@ private: auto dict = dictionaries_loader.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); - if (!context.hasDictionaryAccessRights(dict_ptr->getName())) + if (!context.hasDictionaryAccessRights(dict_ptr->getFullName())) { throw Exception{"For function " + getName() + ", cannot access dictionary " - + dict->getName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; + + dict->getFullName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; } if (!executeDispatch(block, arguments, result, dict_ptr) && @@ -488,10 +488,10 @@ private: auto dict = dictionaries_loader.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); - if (!context.hasDictionaryAccessRights(dict_ptr->getName())) + if (!context.hasDictionaryAccessRights(dict_ptr->getFullName())) { throw Exception{"For function " + getName() + ", cannot access dictionary " - + dict->getName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; + + dict->getFullName() + " on database " + context.getCurrentDatabase(), ErrorCodes::DICTIONARY_ACCESS_DENIED}; } if (!executeDispatch(block, arguments, result, dict_ptr) && diff --git a/dbms/src/Functions/FunctionsHashing.h b/dbms/src/Functions/FunctionsHashing.h index bf36e1999e2..fd52bbeb316 100644 --- a/dbms/src/Functions/FunctionsHashing.h +++ b/dbms/src/Functions/FunctionsHashing.h @@ -371,7 +371,7 @@ struct JavaHashUTF16LEImpl } if (size % 2 != 0) - throw Exception("Arguments for javaHashUTF16LE must be in the form of UTF-16", ErrorCodes::LOGICAL_ERROR); + throw Exception("Arguments for javaHashUTF16LE must be in the form of UTF-16", ErrorCodes::BAD_ARGUMENTS); UInt32 h = 0; for (size_t i = 0; i < size; i += 2) diff --git a/dbms/src/Functions/GatherUtils/Sources.h b/dbms/src/Functions/GatherUtils/Sources.h index c21a6fc523c..7c881bba0c5 100644 --- a/dbms/src/Functions/GatherUtils/Sources.h +++ b/dbms/src/Functions/GatherUtils/Sources.h @@ -238,7 +238,7 @@ struct StringSource size_t getElementSize() const { - return offsets[row_num] - prev_offset; + return offsets[row_num] - prev_offset - 1; } Slice getWhole() const diff --git a/dbms/src/Functions/Regexps.h b/dbms/src/Functions/Regexps.h index 9a8366fb543..5d93e823419 100644 --- a/dbms/src/Functions/Regexps.h +++ b/dbms/src/Functions/Regexps.h @@ -36,6 +36,7 @@ namespace ErrorCodes { extern const int CANNOT_ALLOCATE_MEMORY; extern const int LOGICAL_ERROR; + extern const int BAD_ARGUMENTS; } namespace Regexps @@ -205,7 +206,7 @@ namespace MultiRegexps else throw Exception( "Pattern '" + str_patterns[error->expression] + "' failed with error '" + String(error->message), - ErrorCodes::LOGICAL_ERROR); + ErrorCodes::BAD_ARGUMENTS); } ProfileEvents::increment(ProfileEvents::RegexpCreated); diff --git a/dbms/src/Functions/formatString.h b/dbms/src/Functions/formatString.h index c1f9b6d3783..3b08d313c4d 100644 --- a/dbms/src/Functions/formatString.h +++ b/dbms/src/Functions/formatString.h @@ -18,6 +18,7 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; + extern const int BAD_ARGUMENTS; } struct FormatImpl @@ -45,11 +46,11 @@ struct FormatImpl for (UInt64 pos = l; pos < r; pos++) { if (!isNumericASCII(description[pos])) - throw Exception("Not a number in curly braces at position " + std::to_string(pos), ErrorCodes::LOGICAL_ERROR); + throw Exception("Not a number in curly braces at position " + std::to_string(pos), ErrorCodes::BAD_ARGUMENTS); res = res * 10 + description[pos] - '0'; if (res >= argument_threshold) throw Exception( - "Too big number for arguments, must be at most " + std::to_string(argument_threshold), ErrorCodes::LOGICAL_ERROR); + "Too big number for arguments, must be at most " + std::to_string(argument_threshold), ErrorCodes::BAD_ARGUMENTS); } } @@ -114,7 +115,7 @@ struct FormatImpl } if (is_open_curly) - throw Exception("Two open curly braces without close one at position " + std::to_string(i), ErrorCodes::LOGICAL_ERROR); + throw Exception("Two open curly braces without close one at position " + std::to_string(i), ErrorCodes::BAD_ARGUMENTS); String to_add = String(pattern.data() + start_pos, i - start_pos); double_brace_removal(to_add); @@ -137,7 +138,7 @@ struct FormatImpl } if (!is_open_curly) - throw Exception("Closed curly brace without open one at position " + std::to_string(i), ErrorCodes::LOGICAL_ERROR); + throw Exception("Closed curly brace without open one at position " + std::to_string(i), ErrorCodes::BAD_ARGUMENTS); is_open_curly = false; @@ -145,17 +146,17 @@ struct FormatImpl { if (is_plain_numbering && !*is_plain_numbering) throw Exception( - "Cannot switch from automatic field numbering to manual field specification", ErrorCodes::LOGICAL_ERROR); + "Cannot switch from automatic field numbering to manual field specification", ErrorCodes::BAD_ARGUMENTS); is_plain_numbering = true; if (index_if_plain >= argument_number) - throw Exception("Argument is too big for formatting", ErrorCodes::LOGICAL_ERROR); + throw Exception("Argument is too big for formatting", ErrorCodes::BAD_ARGUMENTS); *index_positions_ptr = index_if_plain++; } else { if (is_plain_numbering && *is_plain_numbering) throw Exception( - "Cannot switch from automatic field numbering to manual field specification", ErrorCodes::LOGICAL_ERROR); + "Cannot switch from automatic field numbering to manual field specification", ErrorCodes::BAD_ARGUMENTS); is_plain_numbering = false; UInt64 arg; @@ -163,7 +164,7 @@ struct FormatImpl if (arg >= argument_number) throw Exception( - "Argument is too big for formatting. Note that indexing starts from zero", ErrorCodes::LOGICAL_ERROR); + "Argument is too big for formatting. Note that indexing starts from zero", ErrorCodes::BAD_ARGUMENTS); *index_positions_ptr = arg; } @@ -183,7 +184,7 @@ struct FormatImpl } if (is_open_curly) - throw Exception("Last open curly brace is not closed", ErrorCodes::LOGICAL_ERROR); + throw Exception("Last open curly brace is not closed", ErrorCodes::BAD_ARGUMENTS); String to_add = String(pattern.data() + start_pos, pattern.size() - start_pos); double_brace_removal(to_add); diff --git a/dbms/src/Functions/neighbor.cpp b/dbms/src/Functions/neighbor.cpp index 0253aed65d3..c37a3313a80 100644 --- a/dbms/src/Functions/neighbor.cpp +++ b/dbms/src/Functions/neighbor.cpp @@ -40,6 +40,8 @@ public: bool isVariadic() const override { return true; } + bool isStateful() const override { return true; } + bool isDeterministic() const override { return false; } bool isDeterministicInScopeOfQuery() const override { return false; } diff --git a/dbms/src/Functions/randomPrintableASCII.cpp b/dbms/src/Functions/randomPrintableASCII.cpp index 7af72d68074..a6f0fd65e42 100644 --- a/dbms/src/Functions/randomPrintableASCII.cpp +++ b/dbms/src/Functions/randomPrintableASCII.cpp @@ -74,6 +74,7 @@ public: data_to.resize(next_offset); offsets_to[row_num] = next_offset; + auto * data_to_ptr = data_to.data(); /// avoid assert on array indexing after end for (size_t pos = offset, end = offset + length; pos < end; pos += 4) /// We have padding in column buffers that we can overwrite. { UInt64 rand = thread_local_rng(); @@ -86,10 +87,10 @@ public: /// Printable characters are from range [32; 126]. /// https://lemire.me/blog/2016/06/27/a-fast-alternative-to-the-modulo-reduction/ - data_to[pos + 0] = 32 + ((rand1 * 95) >> 16); - data_to[pos + 1] = 32 + ((rand2 * 95) >> 16); - data_to[pos + 2] = 32 + ((rand3 * 95) >> 16); - data_to[pos + 3] = 32 + ((rand4 * 95) >> 16); + data_to_ptr[pos + 0] = 32 + ((rand1 * 95) >> 16); + data_to_ptr[pos + 1] = 32 + ((rand2 * 95) >> 16); + data_to_ptr[pos + 2] = 32 + ((rand3 * 95) >> 16); + data_to_ptr[pos + 3] = 32 + ((rand4 * 95) >> 16); /// NOTE gcc failed to vectorize this code (aliasing of char?) /// TODO Implement SIMD optimizations from Danila Kutenin. diff --git a/dbms/src/Functions/trap.cpp b/dbms/src/Functions/trap.cpp index 0589fd40ebe..a7f8d81576d 100644 --- a/dbms/src/Functions/trap.cpp +++ b/dbms/src/Functions/trap.cpp @@ -124,6 +124,10 @@ public: t1.join(); t2.join(); } + else if (mode == "throw exception") + { + std::vector().at(0); + } else if (mode == "access context") { (void)context.getCurrentQueryId(); diff --git a/dbms/src/IO/BitHelpers.h b/dbms/src/IO/BitHelpers.h index 1947d9d99ba..321fb4d254e 100644 --- a/dbms/src/IO/BitHelpers.h +++ b/dbms/src/IO/BitHelpers.h @@ -1,9 +1,10 @@ #pragma once -#include -#include #include #include +#include + +#include #if defined(__OpenBSD__) || defined(__FreeBSD__) # include @@ -14,9 +15,16 @@ # define be64toh(x) OSSwapBigToHostInt64(x) #endif + namespace DB { +namespace ErrorCodes +{ +extern const int CANNOT_WRITE_AFTER_END_OF_BUFFER; +extern const int ATTEMPT_TO_READ_AFTER_EOF; +} + /** Reads data from underlying ReadBuffer bit by bit, max 64 bits at once. * * reads MSB bits first, imagine that you have a data: @@ -34,15 +42,20 @@ namespace DB class BitReader { - ReadBuffer & buf; + using BufferType = unsigned __int128; - UInt64 bits_buffer; + const char * source_begin; + const char * source_current; + const char * source_end; + + BufferType bits_buffer; UInt8 bits_count; - static constexpr UInt8 BIT_BUFFER_SIZE = sizeof(bits_buffer) * 8; public: - BitReader(ReadBuffer & buf_) - : buf(buf_), + BitReader(const char * begin, size_t size) + : source_begin(begin), + source_current(begin), + source_end(begin + size), bits_buffer(0), bits_count(0) {} @@ -50,44 +63,21 @@ public: ~BitReader() {} - inline UInt64 readBits(UInt8 bits) + // reads bits_to_read high-bits from bits_buffer + inline UInt64 readBits(UInt8 bits_to_read) { - UInt64 result = 0; - bits = std::min(static_cast(sizeof(result) * 8), bits); + if (bits_to_read > bits_count) + fillBitBuffer(); - while (bits != 0) - { - if (bits_count == 0) - { - fillBuffer(); - if (bits_count == 0) - { - // EOF. - break; - } - } - - const auto to_read = std::min(bits, bits_count); - - const UInt64 v = bits_buffer >> (bits_count - to_read); - const UInt64 mask = maskLowBits(to_read); - const UInt64 value = v & mask; - result |= value; - - // unset bits that were read - bits_buffer &= ~(mask << (bits_count - to_read)); - bits_count -= to_read; - bits -= to_read; - - result <<= std::min(bits, BIT_BUFFER_SIZE); - } - - return result; + return getBitsFromBitBuffer(bits_to_read); } - inline UInt64 peekBits(UInt8 /*bits*/) + inline UInt8 peekByte() { - return 0; + if (bits_count < 8) + fillBitBuffer(); + + return getBitsFromBitBuffer(8); } inline UInt8 readBit() @@ -95,34 +85,95 @@ public: return static_cast(readBits(1)); } + // skip bits from bits_buffer + inline void skipBufferedBits(UInt8 bits) + { + bits_buffer <<= bits; + bits_count -= bits; + } + + inline bool eof() const { - return bits_count == 0 && buf.eof(); + return bits_count == 0 && source_current >= source_end; + } + + // number of bits that was already read by clients with readBits() + inline UInt64 count() const + { + return (source_current - source_begin) * 8 - bits_count; + } + + inline UInt64 remaining() const + { + return (source_end - source_current) * 8 + bits_count; } private: - void fillBuffer() + enum GetBitsMode {CONSUME, PEEK}; + // read data from internal buffer, if it has not enough bits, result is undefined. + template + inline UInt64 getBitsFromBitBuffer(UInt8 bits_to_read) { - auto read = buf.read(reinterpret_cast(&bits_buffer), BIT_BUFFER_SIZE / 8); - bits_buffer = be64toh(bits_buffer); - bits_buffer >>= BIT_BUFFER_SIZE - read * 8; + // push down the high-bits + const UInt64 result = static_cast(bits_buffer >> (sizeof(bits_buffer) * 8 - bits_to_read)); - bits_count = static_cast(read) * 8; + if constexpr (mode == CONSUME) + { + // 'erase' high-bits that were have read + skipBufferedBits(bits_to_read); + } + + return result; + } + + + // Fills internal bits_buffer with data from source, reads at most 64 bits + size_t fillBitBuffer() + { + const size_t available = source_end - source_current; + const auto bytes_to_read = std::min(64 / 8, available); + if (available == 0) + { + if (bytes_to_read == 0) + return 0; + + throw Exception("Buffer is empty, but requested to read " + + std::to_string(bytes_to_read) + " more bytes.", + ErrorCodes::ATTEMPT_TO_READ_AFTER_EOF); + } + + UInt64 tmp_buffer = 0; + memcpy(&tmp_buffer, source_current, bytes_to_read); + source_current += bytes_to_read; + + tmp_buffer = be64toh(tmp_buffer); + + bits_buffer |= BufferType(tmp_buffer) << ((sizeof(BufferType) - sizeof(tmp_buffer)) * 8 - bits_count); + bits_count += static_cast(bytes_to_read) * 8; + + return bytes_to_read; } }; class BitWriter { - WriteBuffer & buf; + using BufferType = unsigned __int128; - UInt64 bits_buffer; + char * dest_begin; + char * dest_current; + char * dest_end; + + BufferType bits_buffer; UInt8 bits_count; static constexpr UInt8 BIT_BUFFER_SIZE = sizeof(bits_buffer) * 8; public: - BitWriter(WriteBuffer & buf_) - : buf(buf_), + BitWriter(char * begin, size_t size) + : dest_begin(begin), + dest_current(begin), + dest_end(begin + size), bits_buffer(0), bits_count(0) {} @@ -132,54 +183,59 @@ public: flush(); } - inline void writeBits(UInt8 bits, UInt64 value) + // write `bits_to_write` low-bits of `value` to the buffer + inline void writeBits(UInt8 bits_to_write, UInt64 value) { - bits = std::min(static_cast(sizeof(value) * 8), bits); - - while (bits > 0) + UInt32 capacity = BIT_BUFFER_SIZE - bits_count; + if (capacity < bits_to_write) { - auto v = value; - auto to_write = bits; - - const UInt8 capacity = BIT_BUFFER_SIZE - bits_count; - if (capacity < bits) - { - v >>= bits - capacity; - to_write = capacity; - } - - const UInt64 mask = maskLowBits(to_write); - v &= mask; - - bits_buffer <<= to_write; - bits_buffer |= v; - bits_count += to_write; - - if (bits_count < BIT_BUFFER_SIZE) - break; - doFlush(); - bits -= to_write; + capacity = BIT_BUFFER_SIZE - bits_count; } + +// write low bits of value as high bits of bits_buffer + const UInt64 mask = maskLowBits(bits_to_write); + BufferType v = value & mask; + v <<= capacity - bits_to_write; + + bits_buffer |= v; + bits_count += bits_to_write; } + // flush contents of bits_buffer to the dest_current, partial bytes are completed with zeroes. inline void flush() { - if (bits_count != 0) - { - bits_buffer <<= (BIT_BUFFER_SIZE - bits_count); + bits_count = (bits_count + 8 - 1) & ~(8 - 1); // align UP to 8-bytes, so doFlush will write ALL data from bits_buffer + while (bits_count != 0) doFlush(); - } + } + + inline UInt64 count() const + { + return (dest_current - dest_begin) * 8 + bits_count; } private: void doFlush() { - bits_buffer = htobe64(bits_buffer); - buf.write(reinterpret_cast(&bits_buffer), (bits_count + 7) / 8); + // write whole bytes to the dest_current, leaving partial bits in bits_buffer + const size_t available = dest_end - dest_current; + const size_t to_write = std::min(sizeof(UInt64), bits_count / 8); // align to 8-bit boundary - bits_count = 0; - bits_buffer = 0; + if (available < to_write) + { + throw Exception("Can not write past end of buffer. Space available " + + std::to_string(available) + " bytes, required to write: " + + std::to_string(to_write) + ".", + ErrorCodes::CANNOT_WRITE_AFTER_END_OF_BUFFER); + } + + const auto tmp_buffer = htobe64(static_cast(bits_buffer >> (sizeof(bits_buffer) - sizeof(UInt64)) * 8)); + memcpy(dest_current, &tmp_buffer, to_write); + dest_current += to_write; + + bits_buffer <<= to_write * 8; + bits_count -= to_write * 8; } }; diff --git a/dbms/src/IO/BrotliWriteBuffer.cpp b/dbms/src/IO/BrotliWriteBuffer.cpp index 0a0eeb52956..ac1e2b3c188 100644 --- a/dbms/src/IO/BrotliWriteBuffer.cpp +++ b/dbms/src/IO/BrotliWriteBuffer.cpp @@ -30,14 +30,14 @@ public: BrotliEncoderState * state; }; -BrotliWriteBuffer::BrotliWriteBuffer(WriteBuffer & out_, int compression_level, size_t buf_size, char * existing_memory, size_t alignment) - : BufferWithOwnMemory(buf_size, existing_memory, alignment) - , brotli(std::make_unique()) - , in_available(0) - , in_data(nullptr) - , out_capacity(0) - , out_data(nullptr) - , out(out_) +BrotliWriteBuffer::BrotliWriteBuffer(std::unique_ptr out_, int compression_level, size_t buf_size, char * existing_memory, size_t alignment) + : BufferWithOwnMemory(buf_size, existing_memory, alignment) + , brotli(std::make_unique()) + , in_available(0) + , in_data(nullptr) + , out_capacity(0) + , out_data(nullptr) + , out(std::move(out_)) { BrotliEncoderSetParameter(brotli->state, BROTLI_PARAM_QUALITY, static_cast(compression_level)); // Set LZ77 window size. According to brotli sources default value is 24 (c/tools/brotli.c:81) @@ -68,9 +68,9 @@ void BrotliWriteBuffer::nextImpl() do { - out.nextIfAtEnd(); - out_data = reinterpret_cast(out.position()); - out_capacity = out.buffer().end() - out.position(); + out->nextIfAtEnd(); + out_data = reinterpret_cast(out->position()); + out_capacity = out->buffer().end() - out->position(); int result = BrotliEncoderCompressStream( brotli->state, @@ -81,7 +81,7 @@ void BrotliWriteBuffer::nextImpl() &out_data, nullptr); - out.position() = out.buffer().end() - out_capacity; + out->position() = out->buffer().end() - out_capacity; if (result == 0) { @@ -100,9 +100,9 @@ void BrotliWriteBuffer::finish() while (true) { - out.nextIfAtEnd(); - out_data = reinterpret_cast(out.position()); - out_capacity = out.buffer().end() - out.position(); + out->nextIfAtEnd(); + out_data = reinterpret_cast(out->position()); + out_capacity = out->buffer().end() - out->position(); int result = BrotliEncoderCompressStream( brotli->state, @@ -113,7 +113,7 @@ void BrotliWriteBuffer::finish() &out_data, nullptr); - out.position() = out.buffer().end() - out_capacity; + out->position() = out->buffer().end() - out_capacity; if (BrotliEncoderIsFinished(brotli->state)) { diff --git a/dbms/src/IO/BrotliWriteBuffer.h b/dbms/src/IO/BrotliWriteBuffer.h index 6cc2a4ec4b7..5a294354f49 100644 --- a/dbms/src/IO/BrotliWriteBuffer.h +++ b/dbms/src/IO/BrotliWriteBuffer.h @@ -10,11 +10,11 @@ class BrotliWriteBuffer : public BufferWithOwnMemory { public: BrotliWriteBuffer( - WriteBuffer & out_, - int compression_level, - size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, - char * existing_memory = nullptr, - size_t alignment = 0); + std::unique_ptr out_, + int compression_level, + size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, + char * existing_memory = nullptr, + size_t alignment = 0); ~BrotliWriteBuffer() override; @@ -30,9 +30,9 @@ private: const uint8_t * in_data; size_t out_capacity; - uint8_t * out_data; + uint8_t * out_data; - WriteBuffer & out; + std::unique_ptr out; bool finished = false; }; diff --git a/dbms/src/IO/CompressionMethod.cpp b/dbms/src/IO/CompressionMethod.cpp new file mode 100644 index 00000000000..20f1ea44301 --- /dev/null +++ b/dbms/src/IO/CompressionMethod.cpp @@ -0,0 +1,104 @@ +#include + +#include +#include +#include +#include +#include +#include + +#include + + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + + +std::string toContentEncodingName(CompressionMethod method) +{ + switch (method) + { + case CompressionMethod::Gzip: return "gzip"; + case CompressionMethod::Zlib: return "deflate"; + case CompressionMethod::Brotli: return "br"; + case CompressionMethod::None: return ""; + } + __builtin_unreachable(); +} + + +CompressionMethod chooseCompressionMethod(const std::string & path, const std::string & hint) +{ + std::string file_extension; + if (hint.empty() || hint == "auto") + { + auto pos = path.find_last_of('.'); + if (pos != std::string::npos) + file_extension = path.substr(pos + 1, std::string::npos); + } + + const std::string * method_str = file_extension.empty() ? &hint : &file_extension; + + if (*method_str == "gzip" || *method_str == "gz") + return CompressionMethod::Gzip; + if (*method_str == "deflate") + return CompressionMethod::Zlib; + if (*method_str == "brotli" || *method_str == "br") + return CompressionMethod::Brotli; + if (hint.empty() || hint == "auto" || hint == "none") + return CompressionMethod::None; + + throw Exception("Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br' are supported as compression methods", + ErrorCodes::NOT_IMPLEMENTED); +} + + +std::unique_ptr wrapReadBufferWithCompressionMethod( + std::unique_ptr nested, + CompressionMethod method, + size_t buf_size, + char * existing_memory, + size_t alignment) +{ + if (method == CompressionMethod::Gzip || method == CompressionMethod::Zlib) + return std::make_unique(std::move(nested), method, buf_size, existing_memory, alignment); +#if USE_BROTLI + if (method == CompressionMethod::Brotli) + return std::make_unique(std::move(nested), buf_size, existing_memory, alignment); +#endif + + if (method == CompressionMethod::None) + return nested; + + throw Exception("Unsupported compression method", ErrorCodes::NOT_IMPLEMENTED); +} + + +std::unique_ptr wrapWriteBufferWithCompressionMethod( + std::unique_ptr nested, + CompressionMethod method, + int level, + size_t buf_size, + char * existing_memory, + size_t alignment) +{ + if (method == DB::CompressionMethod::Gzip || method == CompressionMethod::Zlib) + return std::make_unique(std::move(nested), method, level, buf_size, existing_memory, alignment); + +#if USE_BROTLI + if (method == DB::CompressionMethod::Brotli) + return std::make_unique(std::move(nested), level, buf_size, existing_memory, alignment); +#endif + + if (method == CompressionMethod::None) + return nested; + + throw Exception("Unsupported compression method", ErrorCodes::NOT_IMPLEMENTED); +} + +} diff --git a/dbms/src/IO/CompressionMethod.h b/dbms/src/IO/CompressionMethod.h index c54d2b581fd..64c2ba3341f 100644 --- a/dbms/src/IO/CompressionMethod.h +++ b/dbms/src/IO/CompressionMethod.h @@ -1,18 +1,57 @@ #pragma once +#include +#include + +#include + + namespace DB { +class ReadBuffer; +class WriteBuffer; + +/** These are "generally recognizable" compression methods for data import/export. + * Do not mess with more efficient compression methods used by ClickHouse internally + * (they use non-standard framing, indexes, checksums...) + */ + enum class CompressionMethod { + None, /// DEFLATE compression with gzip header and CRC32 checksum. /// This option corresponds to files produced by gzip(1) or HTTP Content-Encoding: gzip. Gzip, /// DEFLATE compression with zlib header and Adler32 checksum. /// This option corresponds to HTTP Content-Encoding: deflate. Zlib, - Brotli, - None + Brotli }; +/// How the compression method is named in HTTP. +std::string toContentEncodingName(CompressionMethod method); + +/** Choose compression method from path and hint. + * if hint is "auto" or empty string, then path is analyzed, + * otherwise path parameter is ignored and hint is used as compression method name. + * path is arbitrary string that will be analyzed for file extension (gz, br...) that determines compression. + */ +CompressionMethod chooseCompressionMethod(const std::string & path, const std::string & hint); + +std::unique_ptr wrapReadBufferWithCompressionMethod( + std::unique_ptr nested, + CompressionMethod method, + size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, + char * existing_memory = nullptr, + size_t alignment = 0); + +std::unique_ptr wrapWriteBufferWithCompressionMethod( + std::unique_ptr nested, + CompressionMethod method, + int level, + size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, + char * existing_memory = nullptr, + size_t alignment = 0); + } diff --git a/dbms/src/IO/MMapReadBufferFromFile.cpp b/dbms/src/IO/MMapReadBufferFromFile.cpp index 45558b540e5..32271103724 100644 --- a/dbms/src/IO/MMapReadBufferFromFile.cpp +++ b/dbms/src/IO/MMapReadBufferFromFile.cpp @@ -22,7 +22,7 @@ namespace ErrorCodes } -void MMapReadBufferFromFile::open(const std::string & file_name) +void MMapReadBufferFromFile::open() { ProfileEvents::increment(ProfileEvents::FileOpen); @@ -34,16 +34,24 @@ void MMapReadBufferFromFile::open(const std::string & file_name) } -MMapReadBufferFromFile::MMapReadBufferFromFile(const std::string & file_name, size_t offset, size_t length_) +std::string MMapReadBufferFromFile::getFileName() const { - open(file_name); + return file_name; +} + + +MMapReadBufferFromFile::MMapReadBufferFromFile(const std::string & file_name_, size_t offset, size_t length_) + : file_name(file_name_) +{ + open(); init(fd, offset, length_); } -MMapReadBufferFromFile::MMapReadBufferFromFile(const std::string & file_name, size_t offset) +MMapReadBufferFromFile::MMapReadBufferFromFile(const std::string & file_name_, size_t offset) + : file_name(file_name_) { - open(file_name); + open(); init(fd, offset); } diff --git a/dbms/src/IO/MMapReadBufferFromFile.h b/dbms/src/IO/MMapReadBufferFromFile.h index 6790f817b93..bc566a0489c 100644 --- a/dbms/src/IO/MMapReadBufferFromFile.h +++ b/dbms/src/IO/MMapReadBufferFromFile.h @@ -16,21 +16,24 @@ namespace DB class MMapReadBufferFromFile : public MMapReadBufferFromFileDescriptor { public: - MMapReadBufferFromFile(const std::string & file_name, size_t offset, size_t length_); + MMapReadBufferFromFile(const std::string & file_name_, size_t offset, size_t length_); /// Map till end of file. - MMapReadBufferFromFile(const std::string & file_name, size_t offset); + MMapReadBufferFromFile(const std::string & file_name_, size_t offset); ~MMapReadBufferFromFile() override; void close(); + std::string getFileName() const override; + private: int fd = -1; + std::string file_name; CurrentMetrics::Increment metric_increment{CurrentMetrics::OpenFileForRead}; - void open(const std::string & file_name); + void open(); }; } diff --git a/dbms/src/IO/MMapReadBufferFromFileDescriptor.cpp b/dbms/src/IO/MMapReadBufferFromFileDescriptor.cpp index 4852f9e57e9..034c8524f83 100644 --- a/dbms/src/IO/MMapReadBufferFromFileDescriptor.cpp +++ b/dbms/src/IO/MMapReadBufferFromFileDescriptor.cpp @@ -5,6 +5,8 @@ #include #include +#include +#include #include @@ -18,6 +20,8 @@ namespace ErrorCodes extern const int CANNOT_STAT; extern const int BAD_ARGUMENTS; extern const int LOGICAL_ERROR; + extern const int ARGUMENT_OUT_OF_BOUND; + extern const int CANNOT_SEEK_THROUGH_FILE; } @@ -34,6 +38,7 @@ void MMapReadBufferFromFileDescriptor::init(int fd_, size_t offset, size_t lengt ErrorCodes::CANNOT_ALLOCATE_MEMORY); BufferBase::set(static_cast(buf), length, 0); + ReadBuffer::padded = (length % 4096) > 0 && (length % 4096) <= (4096 - 15); /// TODO determine page size } } @@ -58,14 +63,12 @@ void MMapReadBufferFromFileDescriptor::init(int fd_, size_t offset) MMapReadBufferFromFileDescriptor::MMapReadBufferFromFileDescriptor(int fd_, size_t offset_, size_t length_) - : MMapReadBufferFromFileDescriptor() { init(fd_, offset_, length_); } MMapReadBufferFromFileDescriptor::MMapReadBufferFromFileDescriptor(int fd_, size_t offset_) - : MMapReadBufferFromFileDescriptor() { init(fd_, offset_); } @@ -87,4 +90,39 @@ void MMapReadBufferFromFileDescriptor::finish() length = 0; } +std::string MMapReadBufferFromFileDescriptor::getFileName() const +{ + return "(fd = " + toString(fd) + ")"; +} + +int MMapReadBufferFromFileDescriptor::getFD() const +{ + return fd; +} + +off_t MMapReadBufferFromFileDescriptor::getPositionInFile() +{ + return count(); +} + +off_t MMapReadBufferFromFileDescriptor::doSeek(off_t offset, int whence) +{ + off_t new_pos; + if (whence == SEEK_SET) + new_pos = offset; + else if (whence == SEEK_CUR) + new_pos = count() + offset; + else + throw Exception("MMapReadBufferFromFileDescriptor::seek expects SEEK_SET or SEEK_CUR as whence", ErrorCodes::ARGUMENT_OUT_OF_BOUND); + + working_buffer = internal_buffer; + if (new_pos < 0 || new_pos > off_t(working_buffer.size())) + throw Exception("Cannot seek through file " + getFileName() + + " because seek position (" + toString(new_pos) + ") is out of bounds [0, " + toString(working_buffer.size()) + "]", + ErrorCodes::CANNOT_SEEK_THROUGH_FILE); + + position() = working_buffer.begin() + new_pos; + return new_pos; +} + } diff --git a/dbms/src/IO/MMapReadBufferFromFileDescriptor.h b/dbms/src/IO/MMapReadBufferFromFileDescriptor.h index aaef8c3212a..3cf5e89de7a 100644 --- a/dbms/src/IO/MMapReadBufferFromFileDescriptor.h +++ b/dbms/src/IO/MMapReadBufferFromFileDescriptor.h @@ -1,6 +1,6 @@ #pragma once -#include +#include namespace DB @@ -11,14 +11,16 @@ namespace DB * Also you cannot control whether and how long actual IO take place, * so this method is not manageable and not recommended for anything except benchmarks. */ -class MMapReadBufferFromFileDescriptor : public ReadBuffer +class MMapReadBufferFromFileDescriptor : public ReadBufferFromFileBase { protected: - MMapReadBufferFromFileDescriptor() : ReadBuffer(nullptr, 0) {} + MMapReadBufferFromFileDescriptor() {} void init(int fd_, size_t offset, size_t length_); void init(int fd_, size_t offset); + off_t doSeek(off_t off, int whence) override; + public: MMapReadBufferFromFileDescriptor(int fd_, size_t offset_, size_t length_); @@ -30,6 +32,10 @@ public: /// unmap memory before call to destructor void finish(); + off_t getPositionInFile() override; + std::string getFileName() const override; + int getFD() const override; + private: size_t length = 0; int fd = -1; diff --git a/dbms/src/IO/ReadBufferFromFileBase.cpp b/dbms/src/IO/ReadBufferFromFileBase.cpp index 320053bad12..e1f9694c85d 100644 --- a/dbms/src/IO/ReadBufferFromFileBase.cpp +++ b/dbms/src/IO/ReadBufferFromFileBase.cpp @@ -3,6 +3,11 @@ namespace DB { +ReadBufferFromFileBase::ReadBufferFromFileBase() + : BufferWithOwnMemory(0) +{ +} + ReadBufferFromFileBase::ReadBufferFromFileBase(size_t buf_size, char * existing_memory, size_t alignment) : BufferWithOwnMemory(buf_size, existing_memory, alignment) { diff --git a/dbms/src/IO/ReadBufferFromFileBase.h b/dbms/src/IO/ReadBufferFromFileBase.h index 755f0269664..cf1dff9a574 100644 --- a/dbms/src/IO/ReadBufferFromFileBase.h +++ b/dbms/src/IO/ReadBufferFromFileBase.h @@ -14,6 +14,7 @@ namespace DB class ReadBufferFromFileBase : public BufferWithOwnMemory { public: + ReadBufferFromFileBase(); ReadBufferFromFileBase(size_t buf_size, char * existing_memory, size_t alignment); ReadBufferFromFileBase(ReadBufferFromFileBase &&) = default; ~ReadBufferFromFileBase() override; diff --git a/dbms/src/IO/ReadBufferFromFileDescriptor.cpp b/dbms/src/IO/ReadBufferFromFileDescriptor.cpp index db79d078c65..776d5fc828d 100644 --- a/dbms/src/IO/ReadBufferFromFileDescriptor.cpp +++ b/dbms/src/IO/ReadBufferFromFileDescriptor.cpp @@ -101,10 +101,12 @@ bool ReadBufferFromFileDescriptor::nextImpl() /// If 'offset' is small enough to stay in buffer after seek, then true seek in file does not happen. off_t ReadBufferFromFileDescriptor::doSeek(off_t offset, int whence) { - off_t new_pos = offset; - if (whence == SEEK_CUR) + off_t new_pos; + if (whence == SEEK_SET) + new_pos = offset; + else if (whence == SEEK_CUR) new_pos = pos_in_file - (working_buffer.end() - pos) + offset; - else if (whence != SEEK_SET) + else throw Exception("ReadBufferFromFileDescriptor::seek expects SEEK_SET or SEEK_CUR as whence", ErrorCodes::ARGUMENT_OUT_OF_BOUND); /// Position is unchanged. diff --git a/dbms/src/IO/ReadHelpers.cpp b/dbms/src/IO/ReadHelpers.cpp index ea54d37b1b1..eba724f2193 100644 --- a/dbms/src/IO/ReadHelpers.cpp +++ b/dbms/src/IO/ReadHelpers.cpp @@ -959,13 +959,13 @@ void skipJSONField(ReadBuffer & buf, const StringRef & name_of_field) } -void readException(Exception & e, ReadBuffer & buf, const String & additional_message) +Exception readException(ReadBuffer & buf, const String & additional_message) { int code = 0; String name; String message; String stack_trace; - bool has_nested = false; + bool has_nested = false; /// Obsolete readBinary(code, buf); readBinary(name, buf); @@ -986,21 +986,12 @@ void readException(Exception & e, ReadBuffer & buf, const String & additional_me if (!stack_trace.empty()) out << " Stack trace:\n\n" << stack_trace; - if (has_nested) - { - Exception nested; - readException(nested, buf); - e = Exception(out.str(), nested, code); - } - else - e = Exception(out.str(), code); + return Exception(out.str(), code); } void readAndThrowException(ReadBuffer & buf, const String & additional_message) { - Exception e; - readException(e, buf, additional_message); - e.rethrow(); + readException(buf, additional_message).rethrow(); } diff --git a/dbms/src/IO/ReadHelpers.h b/dbms/src/IO/ReadHelpers.h index 47206039435..fc8e444330c 100644 --- a/dbms/src/IO/ReadHelpers.h +++ b/dbms/src/IO/ReadHelpers.h @@ -29,22 +29,13 @@ #include #include #include +#include #include -#include #include -#ifdef __clang__ -#pragma clang diagnostic push -#pragma clang diagnostic ignored "-Wdouble-promotion" -#endif - #include -#ifdef __clang__ -#pragma clang diagnostic pop -#endif - /// 1 GiB #define DEFAULT_MAX_STRING_SIZE (1ULL << 30) @@ -939,7 +930,7 @@ void skipJSONField(ReadBuffer & buf, const StringRef & name_of_field); * (type is cut to base class, 'message' replaced by 'displayText', and stack trace is appended to 'message') * Some additional message could be appended to exception (example: you could add information about from where it was received). */ -void readException(Exception & e, ReadBuffer & buf, const String & additional_message = ""); +Exception readException(ReadBuffer & buf, const String & additional_message = ""); void readAndThrowException(ReadBuffer & buf, const String & additional_message = ""); @@ -1024,21 +1015,11 @@ void skipToNextLineOrEOF(ReadBuffer & buf); /// Skip to next character after next unescaped \n. If no \n in stream, skip to end. Does not throw on invalid escape sequences. void skipToUnescapedNextLineOrEOF(ReadBuffer & buf); -template -std::unique_ptr getReadBuffer(const DB::CompressionMethod method, Types&&... args) -{ - if (method == DB::CompressionMethod::Gzip) - { - auto read_buf = std::make_unique(std::forward(args)...); - return std::make_unique(std::move(read_buf), method); - } - return std::make_unique(args...); -} /** This function just copies the data from buffer's internal position (in.position()) * to current position (from arguments) into memory. */ -void saveUpToPosition(ReadBuffer & in, DB::Memory<> & memory, char * current); +void saveUpToPosition(ReadBuffer & in, Memory<> & memory, char * current); /** This function is negative to eof(). * In fact it returns whether the data was loaded to internal ReadBuffers's buffer or not. @@ -1047,6 +1028,6 @@ void saveUpToPosition(ReadBuffer & in, DB::Memory<> & memory, char * current); * of our buffer and the current cursor in the end of the buffer. When we call eof() it calls next(). * And this function can fill the buffer with new data, so we will lose the data from previous buffer state. */ -bool loadAtPosition(ReadBuffer & in, DB::Memory<> & memory, char * & current); +bool loadAtPosition(ReadBuffer & in, Memory<> & memory, char * & current); } diff --git a/dbms/src/IO/S3Common.cpp b/dbms/src/IO/S3Common.cpp index a9015ca5982..b981c34c2d2 100644 --- a/dbms/src/IO/S3Common.cpp +++ b/dbms/src/IO/S3Common.cpp @@ -29,7 +29,7 @@ const std::pair & convertLogLevel(Aws::Utils::Logg return mapping.at(log_level); } -class AWSLogger : public Aws::Utils::Logging::LogSystemInterface +class AWSLogger final : public Aws::Utils::Logging::LogSystemInterface { public: ~AWSLogger() final = default; diff --git a/dbms/src/IO/WriteBufferFromHTTPServerResponse.cpp b/dbms/src/IO/WriteBufferFromHTTPServerResponse.cpp index f8bd166a4dd..7fbdeab7ab5 100644 --- a/dbms/src/IO/WriteBufferFromHTTPServerResponse.cpp +++ b/dbms/src/IO/WriteBufferFromHTTPServerResponse.cpp @@ -105,67 +105,41 @@ void WriteBufferFromHTTPServerResponse::nextImpl() { if (compress) { - if (compression_method == CompressionMethod::Gzip) - { -#if defined(POCO_CLICKHOUSE_PATCH) - *response_header_ostr << "Content-Encoding: gzip\r\n"; -#else - response.set("Content-Encoding", "gzip"); - response_body_ostr = &(response.send()); -#endif - out_raw = std::make_unique(*response_body_ostr); - deflating_buf.emplace(std::move(out_raw), compression_method, compression_level, working_buffer.size(), working_buffer.begin()); - out = &*deflating_buf; - } - else if (compression_method == CompressionMethod::Zlib) - { -#if defined(POCO_CLICKHOUSE_PATCH) - *response_header_ostr << "Content-Encoding: deflate\r\n"; -#else - response.set("Content-Encoding", "deflate"); - response_body_ostr = &(response.send()); -#endif - out_raw = std::make_unique(*response_body_ostr); - deflating_buf.emplace(std::move(out_raw), compression_method, compression_level, working_buffer.size(), working_buffer.begin()); - out = &*deflating_buf; - } -#if USE_BROTLI - else if (compression_method == CompressionMethod::Brotli) - { -#if defined(POCO_CLICKHOUSE_PATCH) - *response_header_ostr << "Content-Encoding: br\r\n"; -#else - response.set("Content-Encoding", "br"); - response_body_ostr = &(response.send()); -#endif - out_raw = std::make_unique(*response_body_ostr); - brotli_buf.emplace(*out_raw, compression_level, working_buffer.size(), working_buffer.begin()); - out = &*brotli_buf; - } -#endif + auto content_encoding_name = toContentEncodingName(compression_method); - else - throw Exception("Logical error: unknown compression method passed to WriteBufferFromHTTPServerResponse", - ErrorCodes::LOGICAL_ERROR); - /// Use memory allocated for the outer buffer in the buffer pointed to by out. This avoids extra allocation and copy. +#if defined(POCO_CLICKHOUSE_PATCH) + *response_header_ostr << "Content-Encoding: " << content_encoding_name << "\r\n"; +#else + response.set("Content-Encoding", content_encoding_name); +#endif } - else - { + #if !defined(POCO_CLICKHOUSE_PATCH) - response_body_ostr = &(response.send()); + response_body_ostr = &(response.send()); #endif - out_raw = std::make_unique(*response_body_ostr, working_buffer.size(), working_buffer.begin()); - out = &*out_raw; - } + /// We reuse our buffer in "out" to avoid extra allocations and copies. + + if (compress) + out = wrapWriteBufferWithCompressionMethod( + std::make_unique(*response_body_ostr), + compress ? compression_method : CompressionMethod::None, + compression_level, + working_buffer.size(), + working_buffer.begin()); + else + out = std::make_unique( + *response_body_ostr, + working_buffer.size(), + working_buffer.begin()); } finishSendHeaders(); - } if (out) { + out->buffer() = buffer(); out->position() = position(); out->next(); } @@ -177,9 +151,8 @@ WriteBufferFromHTTPServerResponse::WriteBufferFromHTTPServerResponse( Poco::Net::HTTPServerResponse & response_, unsigned keep_alive_timeout_, bool compress_, - CompressionMethod compression_method_, - size_t size) - : BufferWithOwnMemory(size) + CompressionMethod compression_method_) + : BufferWithOwnMemory(DBMS_DEFAULT_BUFFER_SIZE) , request(request_) , response(response_) , keep_alive_timeout(keep_alive_timeout_) @@ -215,6 +188,9 @@ void WriteBufferFromHTTPServerResponse::finalize() if (offset()) { next(); + + if (out) + out.reset(); } else { diff --git a/dbms/src/IO/WriteBufferFromHTTPServerResponse.h b/dbms/src/IO/WriteBufferFromHTTPServerResponse.h index 642e59e4921..f0b614c7406 100644 --- a/dbms/src/IO/WriteBufferFromHTTPServerResponse.h +++ b/dbms/src/IO/WriteBufferFromHTTPServerResponse.h @@ -8,8 +8,6 @@ #include #include #include -#include -#include #include #include #include @@ -52,7 +50,7 @@ private: unsigned keep_alive_timeout = 0; bool compress = false; CompressionMethod compression_method; - int compression_level = Z_DEFAULT_COMPRESSION; + int compression_level = 1; std::ostream * response_body_ostr = nullptr; @@ -60,13 +58,7 @@ private: std::ostream * response_header_ostr = nullptr; #endif - std::unique_ptr out_raw; - std::optional deflating_buf; -#if USE_BROTLI - std::optional brotli_buf; -#endif - - WriteBuffer * out = nullptr; /// Uncompressed HTTP body is written to this buffer. Points to out_raw or possibly to deflating_buf. + std::unique_ptr out; bool headers_started_sending = false; bool headers_finished_sending = false; /// If true, you could not add any headers. @@ -99,8 +91,7 @@ public: Poco::Net::HTTPServerResponse & response_, unsigned keep_alive_timeout_, bool compress_ = false, /// If true - set Content-Encoding header and compress the result. - CompressionMethod compression_method_ = CompressionMethod::Gzip, - size_t size = DBMS_DEFAULT_BUFFER_SIZE); + CompressionMethod compression_method_ = CompressionMethod::None); /// Writes progess in repeating HTTP headers. void onProgress(const Progress & progress); diff --git a/dbms/src/IO/WriteHelpers.cpp b/dbms/src/IO/WriteHelpers.cpp index fe64983c18a..d2605dce9fe 100644 --- a/dbms/src/IO/WriteHelpers.cpp +++ b/dbms/src/IO/WriteHelpers.cpp @@ -48,7 +48,6 @@ void formatUUID(std::reverse_iterator src16, UInt8 * dst36) } - void writeException(const Exception & e, WriteBuffer & buf, bool with_stack_trace) { writeBinary(e.code(), buf); @@ -56,14 +55,11 @@ void writeException(const Exception & e, WriteBuffer & buf, bool with_stack_trac writeBinary(e.displayText(), buf); if (with_stack_trace) - writeBinary(e.getStackTrace().toString(), buf); + writeBinary(e.getStackTraceString(), buf); else writeBinary(String(), buf); - bool has_nested = e.nested() != nullptr; + bool has_nested = false; writeBinary(has_nested, buf); - - if (has_nested) - writeException(Exception(Exception::CreateFromPoco, *e.nested()), buf, with_stack_trace); } } diff --git a/dbms/src/IO/WriteHelpers.h b/dbms/src/IO/WriteHelpers.h index 082bf63e6b7..3e6d579cb16 100644 --- a/dbms/src/IO/WriteHelpers.h +++ b/dbms/src/IO/WriteHelpers.h @@ -26,10 +26,12 @@ #include #include #include -#include + +#include #include + namespace DB { @@ -115,21 +117,108 @@ inline void writeBoolText(bool x, WriteBuffer & buf) writeChar(x ? '1' : '0', buf); } -template -inline size_t writeFloatTextFastPath(T x, char * buffer, int len) + +struct DecomposedFloat64 { - using Converter = DoubleConverter; - double_conversion::StringBuilder builder{buffer, len}; + DecomposedFloat64(double x) + { + memcpy(&x_uint, &x, sizeof(x)); + } + + uint64_t x_uint; + + bool sign() const + { + return x_uint >> 63; + } + + uint16_t exponent() const + { + return (x_uint >> 52) & 0x7FF; + } + + int16_t normalized_exponent() const + { + return int16_t(exponent()) - 1023; + } + + uint64_t mantissa() const + { + return x_uint & 0x5affffffffffffful; + } + + /// NOTE Probably floating point instructions can be better. + bool is_inside_int64() const + { + return x_uint == 0 + || (normalized_exponent() >= 0 && normalized_exponent() <= 52 + && ((mantissa() & ((1ULL << (52 - normalized_exponent())) - 1)) == 0)); + } +}; + +struct DecomposedFloat32 +{ + DecomposedFloat32(float x) + { + memcpy(&x_uint, &x, sizeof(x)); + } + + uint32_t x_uint; + + bool sign() const + { + return x_uint >> 31; + } + + uint16_t exponent() const + { + return (x_uint >> 23) & 0xFF; + } + + int16_t normalized_exponent() const + { + return int16_t(exponent()) - 127; + } + + uint32_t mantissa() const + { + return x_uint & 0x7fffff; + } + + bool is_inside_int32() const + { + return x_uint == 0 + || (normalized_exponent() >= 0 && normalized_exponent() <= 23 + && ((mantissa() & ((1ULL << (23 - normalized_exponent())) - 1)) == 0)); + } +}; + +template +inline size_t writeFloatTextFastPath(T x, char * buffer) +{ + int result = 0; - bool result = false; if constexpr (std::is_same_v) - result = Converter::instance().ToShortest(x, &builder); - else - result = Converter::instance().ToShortestSingle(x, &builder); + { + /// The library Ryu has low performance on integers. + /// This workaround improves performance 6..10 times. - if (!result) + if (DecomposedFloat64(x).is_inside_int64()) + result = itoa(Int64(x), buffer) - buffer; + else + result = d2s_buffered_n(x, buffer); + } + else + { + if (DecomposedFloat32(x).is_inside_int32()) + result = itoa(Int32(x), buffer) - buffer; + else + result = f2s_buffered_n(x, buffer); + } + + if (result <= 0) throw Exception("Cannot print floating point number", ErrorCodes::CANNOT_PRINT_FLOAT_OR_DOUBLE_NUMBER); - return builder.position(); + return result; } template @@ -140,23 +229,13 @@ inline void writeFloatText(T x, WriteBuffer & buf) using Converter = DoubleConverter; if (likely(buf.available() >= Converter::MAX_REPRESENTATION_LENGTH)) { - buf.position() += writeFloatTextFastPath(x, buf.position(), Converter::MAX_REPRESENTATION_LENGTH); + buf.position() += writeFloatTextFastPath(x, buf.position()); return; } Converter::BufferType buffer; - double_conversion::StringBuilder builder{buffer, sizeof(buffer)}; - - bool result = false; - if constexpr (std::is_same_v) - result = Converter::instance().ToShortest(x, &builder); - else - result = Converter::instance().ToShortestSingle(x, &builder); - - if (!result) - throw Exception("Cannot print floating point number", ErrorCodes::CANNOT_PRINT_FLOAT_OR_DOUBLE_NUMBER); - - buf.write(buffer, builder.position()); + size_t result = writeFloatTextFastPath(x, buffer); + buf.write(buffer, result); } @@ -955,15 +1034,4 @@ inline String toString(const T & x) return buf.str(); } -template -std::unique_ptr getWriteBuffer(const DB::CompressionMethod method, Types&&... args) -{ - if (method == DB::CompressionMethod::Gzip) - { - auto write_buf = std::make_unique(std::forward(args)...); - return std::make_unique(std::move(write_buf), method, 1 /* compression level */); - } - return std::make_unique(args...); -} - } diff --git a/dbms/src/IO/ZlibDeflatingWriteBuffer.cpp b/dbms/src/IO/ZlibDeflatingWriteBuffer.cpp index c4d7fac56a6..8efe96877e4 100644 --- a/dbms/src/IO/ZlibDeflatingWriteBuffer.cpp +++ b/dbms/src/IO/ZlibDeflatingWriteBuffer.cpp @@ -5,6 +5,12 @@ namespace DB { +namespace ErrorCodes +{ + extern const int ZLIB_DEFLATE_FAILED; +} + + ZlibDeflatingWriteBuffer::ZlibDeflatingWriteBuffer( std::unique_ptr out_, CompressionMethod compression_method, @@ -84,6 +90,21 @@ void ZlibDeflatingWriteBuffer::finish() next(); + /// https://github.com/zlib-ng/zlib-ng/issues/494 + do + { + out->nextIfAtEnd(); + zstr.next_out = reinterpret_cast(out->position()); + zstr.avail_out = out->buffer().end() - out->position(); + + int rc = deflate(&zstr, Z_FULL_FLUSH); + out->position() = out->buffer().end() - zstr.avail_out; + + if (rc != Z_OK) + throw Exception(std::string("deflate failed: ") + zError(rc), ErrorCodes::ZLIB_DEFLATE_FAILED); + } + while (zstr.avail_out == 0); + while (true) { out->nextIfAtEnd(); diff --git a/dbms/src/IO/ZlibDeflatingWriteBuffer.h b/dbms/src/IO/ZlibDeflatingWriteBuffer.h index 86eee1cffe5..f9df8f8157b 100644 --- a/dbms/src/IO/ZlibDeflatingWriteBuffer.h +++ b/dbms/src/IO/ZlibDeflatingWriteBuffer.h @@ -10,11 +10,6 @@ namespace DB { -namespace ErrorCodes -{ - extern const int ZLIB_DEFLATE_FAILED; -} - /// Performs compression using zlib library and writes compressed data to out_ WriteBuffer. class ZlibDeflatingWriteBuffer : public BufferWithOwnMemory { diff --git a/dbms/src/IO/createReadBufferFromFileBase.cpp b/dbms/src/IO/createReadBufferFromFileBase.cpp index 8fc5923e6ff..9fa560620dd 100644 --- a/dbms/src/IO/createReadBufferFromFileBase.cpp +++ b/dbms/src/IO/createReadBufferFromFileBase.cpp @@ -3,6 +3,7 @@ #if defined(__linux__) || defined(__FreeBSD__) #include #endif +#include #include @@ -11,13 +12,17 @@ namespace ProfileEvents extern const Event CreatedReadBufferOrdinary; extern const Event CreatedReadBufferAIO; extern const Event CreatedReadBufferAIOFailed; + extern const Event CreatedReadBufferMMap; + extern const Event CreatedReadBufferMMapFailed; } namespace DB { -std::unique_ptr createReadBufferFromFileBase(const std::string & filename_, size_t estimated_size, - size_t aio_threshold, size_t buffer_size_, int flags_, char * existing_memory_, size_t alignment) +std::unique_ptr createReadBufferFromFileBase( + const std::string & filename_, + size_t estimated_size, size_t aio_threshold, size_t mmap_threshold, + size_t buffer_size_, int flags_, char * existing_memory_, size_t alignment) { #if defined(__linux__) || defined(__FreeBSD__) if (aio_threshold && estimated_size >= aio_threshold) @@ -40,6 +45,21 @@ std::unique_ptr createReadBufferFromFileBase(const std:: (void)estimated_size; #endif + if (!existing_memory_ && mmap_threshold && estimated_size >= mmap_threshold) + { + try + { + auto res = std::make_unique(filename_, 0); + ProfileEvents::increment(ProfileEvents::CreatedReadBufferMMap); + return res; + } + catch (const ErrnoException &) + { + /// Fallback if mmap is not supported (example: pipe). + ProfileEvents::increment(ProfileEvents::CreatedReadBufferMMapFailed); + } + } + ProfileEvents::increment(ProfileEvents::CreatedReadBufferOrdinary); return std::make_unique(filename_, buffer_size_, flags_, existing_memory_, alignment); } diff --git a/dbms/src/IO/createReadBufferFromFileBase.h b/dbms/src/IO/createReadBufferFromFileBase.h index fa98e536a46..61dfde6229f 100644 --- a/dbms/src/IO/createReadBufferFromFileBase.h +++ b/dbms/src/IO/createReadBufferFromFileBase.h @@ -19,6 +19,7 @@ std::unique_ptr createReadBufferFromFileBase( const std::string & filename_, size_t estimated_size, size_t aio_threshold, + size_t mmap_threshold, size_t buffer_size_ = DBMS_DEFAULT_BUFFER_SIZE, int flags_ = -1, char * existing_memory_ = nullptr, diff --git a/dbms/src/IO/tests/CMakeLists.txt b/dbms/src/IO/tests/CMakeLists.txt index 38802718dd1..e168e704814 100644 --- a/dbms/src/IO/tests/CMakeLists.txt +++ b/dbms/src/IO/tests/CMakeLists.txt @@ -78,7 +78,7 @@ add_executable (parse_date_time_best_effort parse_date_time_best_effort.cpp) target_link_libraries (parse_date_time_best_effort PRIVATE clickhouse_common_io) add_executable (zlib_ng_bug zlib_ng_bug.cpp) -target_link_libraries (zlib_ng_bug PRIVATE ${Poco_Foundation_LIBRARY}) -if(NOT USE_INTERNAL_POCO_LIBRARY) - target_include_directories(zlib_ng_bug SYSTEM BEFORE PRIVATE ${Poco_INCLUDE_DIRS}) -endif() +target_link_libraries (zlib_ng_bug PRIVATE ${Poco_Foundation_LIBRARY} ${ZLIB_LIBRARY}) + +add_executable (ryu_test ryu_test.cpp) +target_link_libraries (ryu_test PRIVATE ryu) diff --git a/dbms/src/IO/tests/gtest_DateTime64_parsing_and_writing.cpp b/dbms/src/IO/tests/gtest_DateTime64_parsing_and_writing.cpp index 08ca5dc88ee..04fdb6f4a34 100644 --- a/dbms/src/IO/tests/gtest_DateTime64_parsing_and_writing.cpp +++ b/dbms/src/IO/tests/gtest_DateTime64_parsing_and_writing.cpp @@ -79,7 +79,7 @@ TEST_P(DateTime64StringParseBestEffortTest, parse) // YYYY-MM-DD HH:MM:SS.NNNNNNNNN -INSTANTIATE_TEST_CASE_P(Basic, +INSTANTIATE_TEST_SUITE_P(Basic, DateTime64StringParseTest, ::testing::ValuesIn(std::initializer_list{ { @@ -130,10 +130,10 @@ INSTANTIATE_TEST_CASE_P(Basic, 1568650817'1ULL, 1 } - }), + }) ); -INSTANTIATE_TEST_CASE_P(BestEffort, +INSTANTIATE_TEST_SUITE_P(BestEffort, DateTime64StringParseBestEffortTest, ::testing::ValuesIn(std::initializer_list{ { @@ -142,13 +142,13 @@ INSTANTIATE_TEST_CASE_P(BestEffort, 1568650817'123456ULL, 6 } - }), + }) ); // TODO: add negative test cases for invalid strings, verifying that error is reported properly -INSTANTIATE_TEST_CASE_P(Basic, +INSTANTIATE_TEST_SUITE_P(Basic, DateTime64StringWriteTest, ::testing::ValuesIn(std::initializer_list{ { @@ -181,6 +181,6 @@ INSTANTIATE_TEST_CASE_P(Basic, 1568650817'001ULL, 3 } - }), + }) ); diff --git a/dbms/src/IO/tests/gtest_bit_io.cpp b/dbms/src/IO/tests/gtest_bit_io.cpp index 3def664231f..57fe45ca1a6 100644 --- a/dbms/src/IO/tests/gtest_bit_io.cpp +++ b/dbms/src/IO/tests/gtest_bit_io.cpp @@ -36,11 +36,11 @@ std::string bin(const T & value, size_t bits = sizeof(T)*8) .to_string().substr(MAX_BITS - bits, bits); } +// gets N low bits of value template T getBits(UInt8 bits, const T & value) { - const T mask = ((static_cast(1) << static_cast(bits)) - 1); - return value & mask; + return value & maskLowBits(bits); } template @@ -83,12 +83,36 @@ std::string dumpContents(const T& container, return sstr.str(); } +template +::testing::AssertionResult BinaryEqual(const ValueLeft & left, const ValueRight & right) +{ +// ::testing::AssertionResult result = ::testing::AssertionSuccess(); + if (sizeof(left) != sizeof(right)) + return ::testing::AssertionFailure() + << "Sizes do not match, expected: " << sizeof(left) << " actual: " << sizeof(right); + + const auto size = std::min(sizeof(left), sizeof(right)); + if (memcmp(&left, &right, size) != 0) + { + const auto l_bits = left ? static_cast(std::log2(left)) : 0; + const auto r_bits = right ? static_cast(std::log2(right)) : 0; + const size_t bits = std::max(l_bits, r_bits) + 1; + + return ::testing::AssertionFailure() + << "Values are binary different,\n" + << "\texpected: 0b" << bin(left, bits) << " (" << std::hex << left << "),\n" + << "\tactual : 0b" << bin(right, bits) << " (" <> bits_and_vals; std::string expected_buffer_binary; - explicit TestCaseParameter(std::vector> vals, std::string binary = std::string{}) + TestCaseParameter(std::vector> vals, std::string binary = std::string{}) : bits_and_vals(std::move(vals)), expected_buffer_binary(binary) {} @@ -114,8 +138,7 @@ TEST_P(BitIO, WriteAndRead) PODArray data(max_buffer_size); { - WriteBuffer write_buffer(data.data(), data.size()); - BitWriter writer(write_buffer); + BitWriter writer(data.data(), data.size()); for (const auto & bv : bits_and_vals) { writer.writeBits(bv.first, bv.second); @@ -133,38 +156,73 @@ TEST_P(BitIO, WriteAndRead) ASSERT_EQ(expected_buffer_binary, actual_buffer_binary); } - BitReader reader(read_buffer); + BitReader reader(data.data(), data.size()); + int bitpos = 0; int item = 0; for (const auto & bv : bits_and_vals) { SCOPED_TRACE(::testing::Message() - << "item #" << item << ", width: " << static_cast(bv.first) - << ", value: " << bin(bv.second) - << ".\n\n\nBuffer memory:\n" << dumpContents(data)); + << "item #" << item << " of " << bits_and_vals.size() << ", width: " << static_cast(bv.first) + << ", value: " << bv.second << "(" << bin(bv.second) << ")" + << ", at bit position: " << std::dec << reader.count() + << ".\nBuffer memory:\n" << dumpContents(data)); - //EXPECT_EQ(getBits(bv.first, bv.second), reader.peekBits(bv.first)); - EXPECT_EQ(getBits(bv.first, bv.second), reader.readBits(bv.first)); +// const UInt8 next_byte = getBits(bv.first, bv.second) & + ASSERT_TRUE(BinaryEqual(getBits(bv.first, bv.second), reader.readBits(bv.first))); ++item; + bitpos += bv.first; } } } -INSTANTIATE_TEST_CASE_P(Simple, - BitIO, - ::testing::Values( - TestCaseParameter( - {{9, 0xFFFFFFFF}, {9, 0x00}, {9, 0xFFFFFFFF}, {9, 0x00}, {9, 0xFFFFFFFF}}, - "11111111 10000000 00111111 11100000 00001111 11111000 "), - TestCaseParameter( - {{7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {3, 0xFFFF}}, - "01111110 11111101 11111011 11110111 11101111 11011111 10111111 01111111 11000000 "), - TestCaseParameter({{33, 0xFF110d0b07050300}, {33, 0xAAEE29251f1d1713}}), - TestCaseParameter({{33, BIT_PATTERN}, {33, BIT_PATTERN}}), - TestCaseParameter({{24, 0xFFFFFFFF}}, - "11111111 11111111 11111111 ") -),); +INSTANTIATE_TEST_SUITE_P(Simple, + BitIO, + ::testing::ValuesIn(std::initializer_list{ + { + {{9, 0xFFFFFFFF}, {9, 0x00}, {9, 0xFFFFFFFF}, {9, 0x00}, {9, 0xFFFFFFFF}}, + "11111111 10000000 00111111 11100000 00001111 11111000 " + }, + { + {{7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {7, 0x3f}, {3, 0xFFFF}}, + "01111110 11111101 11111011 11110111 11101111 11011111 10111111 01111111 11000000 " + }, + { + {{33, 0xFF110d0b07050300}, {33, 0xAAEE29251f1d1713}} + }, + { + {{33, BIT_PATTERN}, {33, BIT_PATTERN}} + }, + { + {{24, 0xFFFFFFFF}}, + "11111111 11111111 11111111 " + }, + { + // Note that we take only N lower bits of the number: {3, 0b01011} => 011 + {{5, 0b01010}, {3, 0b111}, {7, 0b11001100}, {6, 0}, {5, 0b11111111}, {4, 0}, {3, 0b101}, {2, 0}, {1, 0b11111111}}, + "01010111 10011000 00000111 11000010 10010000 " + }, + { + {{64, BIT_PATTERN}, {56, BIT_PATTERN} , {4, 0b1111}, {4, 0}, // 128 + {8, 0b11111111}, {64, BIT_PATTERN}, {48, BIT_PATTERN}, {8, 0}}, // 256 + "11101011 11101111 10111010 11101111 10101111 10111010 11101011 10101001 " // 64 + "11101111 10111010 11101111 10101111 10111010 11101011 10101001 11110000 " // 128 + "11111111 11101011 11101111 10111010 11101111 10101111 10111010 11101011 " // 192 + "10101001 10111010 11101111 10101111 10111010 11101011 10101001 00000000 " // 256 + }, + { + {{64, BIT_PATTERN}, {56, BIT_PATTERN} , {5, 0b11111}, {3, 0}, // 128 + {8, 0b11111111}, {64, BIT_PATTERN}, {48, BIT_PATTERN}, {8, 0}, //256 + {32, BIT_PATTERN}, {12, 0xff}, {8, 0}, {12, 0xAEff}}, + "11101011 11101111 10111010 11101111 10101111 10111010 11101011 10101001 " // 64 + "11101111 10111010 11101111 10101111 10111010 11101011 10101001 11111000 " // 128 + "11111111 11101011 11101111 10111010 11101111 10101111 10111010 11101011 " // 192 + "10101001 10111010 11101111 10101111 10111010 11101011 10101001 00000000 " // 256 + "10101111 10111010 11101011 10101001 00001111 11110000 00001110 11111111 " // 320 + } + }) +); TestCaseParameter primes_case(UInt8 repeat_times, UInt64 pattern) { @@ -183,12 +241,13 @@ TestCaseParameter primes_case(UInt8 repeat_times, UInt64 pattern) return TestCaseParameter(test_data); } -INSTANTIATE_TEST_CASE_P(Primes, - BitIO, - ::testing::Values( - primes_case(11, 0xFFFFFFFFFFFFFFFFULL), - primes_case(11, BIT_PATTERN) -),); +INSTANTIATE_TEST_SUITE_P(Primes, + BitIO, + ::testing::Values( + primes_case(11, 0xFFFFFFFFFFFFFFFFULL), + primes_case(11, BIT_PATTERN) + ) +); TEST(BitHelpers, maskLowBits) { diff --git a/dbms/src/IO/tests/ryu_test.cpp b/dbms/src/IO/tests/ryu_test.cpp new file mode 100644 index 00000000000..0866f6afd3d --- /dev/null +++ b/dbms/src/IO/tests/ryu_test.cpp @@ -0,0 +1,92 @@ +#include +#include +#include + + +struct DecomposedFloat64 +{ + DecomposedFloat64(double x) + { + memcpy(&x_uint, &x, sizeof(x)); + } + + uint64_t x_uint; + + bool sign() const + { + return x_uint >> 63; + } + + uint16_t exponent() const + { + return (x_uint >> 52) & 0x7FF; + } + + int16_t normalized_exponent() const + { + return int16_t(exponent()) - 1023; + } + + uint64_t mantissa() const + { + return x_uint & 0x5affffffffffffful; + } + + bool is_inside_int64() const + { + return x_uint == 0 + || (normalized_exponent() >= 0 && normalized_exponent() <= 52 + && ((mantissa() & ((1ULL << (52 - normalized_exponent())) - 1)) == 0)); + } +}; + +struct DecomposedFloat32 +{ + DecomposedFloat32(float x) + { + memcpy(&x_uint, &x, sizeof(x)); + } + + uint32_t x_uint; + + bool sign() const + { + return x_uint >> 31; + } + + uint16_t exponent() const + { + return (x_uint >> 23) & 0xFF; + } + + int16_t normalized_exponent() const + { + return int16_t(exponent()) - 127; + } + + uint32_t mantissa() const + { + return x_uint & 0x7fffff; + } + + bool is_inside_int32() const + { + return x_uint == 0 + || (normalized_exponent() >= 0 && normalized_exponent() <= 23 + && ((mantissa() & ((1ULL << (23 - normalized_exponent())) - 1)) == 0)); + } +}; + + +int main(int argc, char ** argv) +{ + double x = argc > 1 ? std::stod(argv[1]) : 0; + char buf[32]; + + d2s_buffered(x, buf); + std::cout << buf << "\n"; + + std::cout << DecomposedFloat64(x).is_inside_int64() << "\n"; + + return 0; +} diff --git a/dbms/src/IO/tests/zlib_ng_bug.cpp b/dbms/src/IO/tests/zlib_ng_bug.cpp index 8b94b4e49d2..e9b3c448b88 100644 --- a/dbms/src/IO/tests/zlib_ng_bug.cpp +++ b/dbms/src/IO/tests/zlib_ng_bug.cpp @@ -1,32 +1,50 @@ -#include -#include -#include -#include +#include +#include +#include +#include -/** This script reproduces the bug in zlib-ng library. - * Put the following content to "data.bin" file: -abcdefghijklmn!@Aab#AAabcdefghijklmn$% -xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx - * There are two lines. First line make sense. Second line contains padding to make file size large enough. - * Compile with - * cmake -D SANITIZE=address - * and run: +#pragma GCC diagnostic ignored "-Wold-style-cast" -./zlib_ng_bug data2.bin -================================================================= -==204952==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6310000147ff at pc 0x000000596d7a bp 0x7ffd139edd50 sp 0x7ffd139edd48 -READ of size 1 at 0x6310000147ff thread T0 - */ -int main(int argc, char ** argv) +/// https://github.com/zlib-ng/zlib-ng/issues/494 +int main(int, char **) { - using namespace Poco; + std::vector in(1048576); + std::vector out(1048576); - std::string filename(argc >= 2 ? argv[1] : "data.bin"); - FileInputStream istr(filename); - NullOutputStream ostr; - DeflatingOutputStream deflater(ostr, DeflatingStreamBuf::STREAM_GZIP); - StreamCopier::copyStream(istr, deflater); + ssize_t in_size = read(STDIN_FILENO, in.data(), 1048576); + if (in_size < 0) + throw std::runtime_error("Cannot read"); + in.resize(in_size); + + z_stream zstr{}; + if (Z_OK != deflateInit2(&zstr, 1, Z_DEFLATED, 15 + 16, 8, Z_DEFAULT_STRATEGY)) + throw std::runtime_error("Cannot deflateInit2"); + + zstr.next_in = in.data(); + zstr.avail_in = in.size(); + zstr.next_out = out.data(); + zstr.avail_out = out.size(); + + while (zstr.avail_in > 0) + if (Z_OK != deflate(&zstr, Z_NO_FLUSH)) + throw std::runtime_error("Cannot deflate"); + + while (true) + { + int rc = deflate(&zstr, Z_FINISH); + + if (rc == Z_STREAM_END) + break; + + if (rc != Z_OK) + throw std::runtime_error("Cannot finish deflate"); + } + + deflateEnd(&zstr); + + if (ssize_t(zstr.total_out) != write(STDOUT_FILENO, out.data(), zstr.total_out)) + throw std::runtime_error("Cannot write"); return 0; } diff --git a/dbms/src/Interpreters/BloomFilter.cpp b/dbms/src/Interpreters/BloomFilter.cpp index 709dd7fbddf..a6a5bedb8a8 100644 --- a/dbms/src/Interpreters/BloomFilter.cpp +++ b/dbms/src/Interpreters/BloomFilter.cpp @@ -96,7 +96,7 @@ DataTypePtr BloomFilter::getPrimitiveType(const DataTypePtr & data_type) if (!typeid_cast(array_type->getNestedType().get())) return getPrimitiveType(array_type->getNestedType()); else - throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::LOGICAL_ERROR); + throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::BAD_ARGUMENTS); } if (const auto * nullable_type = typeid_cast(data_type.get())) diff --git a/dbms/src/Interpreters/BloomFilterHash.h b/dbms/src/Interpreters/BloomFilterHash.h index 77bd5cc7ffd..38bedb213cd 100644 --- a/dbms/src/Interpreters/BloomFilterHash.h +++ b/dbms/src/Interpreters/BloomFilterHash.h @@ -23,6 +23,7 @@ namespace DB namespace ErrorCodes { extern const int ILLEGAL_COLUMN; + extern const int BAD_ARGUMENTS; } struct BloomFilterHash @@ -69,7 +70,7 @@ struct BloomFilterHash unexpected_type = true; if (unexpected_type) - throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::LOGICAL_ERROR); + throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::BAD_ARGUMENTS); return ColumnConst::create(ColumnUInt64::create(1, hash), 1); } @@ -82,7 +83,7 @@ struct BloomFilterHash const auto * array_col = typeid_cast(column.get()); if (checkAndGetColumn(array_col->getData())) - throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::LOGICAL_ERROR); + throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::BAD_ARGUMENTS); const auto & offsets = array_col->getOffsets(); limit = offsets[pos + limit - 1] - offsets[pos - 1]; /// PaddedPODArray allows access on index -1. @@ -127,7 +128,7 @@ struct BloomFilterHash else if (which.isFloat64()) getNumberTypeHash(column, vec, pos); else if (which.isString()) getStringTypeHash(column, vec, pos); else if (which.isFixedString()) getStringTypeHash(column, vec, pos); - else throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::LOGICAL_ERROR); + else throw Exception("Unexpected type " + data_type->getName() + " of bloom filter index.", ErrorCodes::BAD_ARGUMENTS); } template diff --git a/dbms/src/Interpreters/CatBoostModel.h b/dbms/src/Interpreters/CatBoostModel.h index 541dd111c82..820c26a1fb4 100644 --- a/dbms/src/Interpreters/CatBoostModel.h +++ b/dbms/src/Interpreters/CatBoostModel.h @@ -60,7 +60,7 @@ public: const ExternalLoadableLifetime & getLifetime() const override; - std::string getName() const override { return name; } + const std::string & getLoadableName() const override { return name; } bool supportUpdates() const override { return true; } @@ -69,7 +69,7 @@ public: std::shared_ptr clone() const override; private: - std::string name; + const std::string name; std::string model_path; std::string lib_path; ExternalLoadableLifetime lifetime; diff --git a/dbms/src/Interpreters/Cluster.cpp b/dbms/src/Interpreters/Cluster.cpp index 2c75bd821fe..71bd89b2b6f 100644 --- a/dbms/src/Interpreters/Cluster.cpp +++ b/dbms/src/Interpreters/Cluster.cpp @@ -10,6 +10,7 @@ #include #include #include +#include namespace DB { @@ -449,18 +450,64 @@ void Cluster::initMisc() } } +std::unique_ptr Cluster::getClusterWithReplicasAsShards(const Settings & settings) const +{ + return std::unique_ptr{ new Cluster(ReplicasAsShardsTag{}, *this, settings)}; +} std::unique_ptr Cluster::getClusterWithSingleShard(size_t index) const { - return std::unique_ptr{ new Cluster(*this, {index}) }; + return std::unique_ptr{ new Cluster(SubclusterTag{}, *this, {index}) }; } std::unique_ptr Cluster::getClusterWithMultipleShards(const std::vector & indices) const { - return std::unique_ptr{ new Cluster(*this, indices) }; + return std::unique_ptr{ new Cluster(SubclusterTag{}, *this, indices) }; } -Cluster::Cluster(const Cluster & from, const std::vector & indices) +Cluster::Cluster(Cluster::ReplicasAsShardsTag, const Cluster & from, const Settings & settings) + : shards_info{}, addresses_with_failover{} +{ + if (from.addresses_with_failover.empty()) + throw Exception("Cluster is empty", ErrorCodes::LOGICAL_ERROR); + + std::set> unique_hosts; + for (size_t shard_index : ext::range(0, from.shards_info.size())) + { + const auto & replicas = from.addresses_with_failover[shard_index]; + for (const auto & address : replicas) + { + if (!unique_hosts.emplace(address.host_name, address.port).second) + continue; /// Duplicate host, skip. + + ShardInfo info; + if (address.is_local) + info.local_addresses.push_back(address); + + ConnectionPoolPtr pool = std::make_shared( + settings.distributed_connections_pool_size, + address.host_name, + address.port, + address.default_database, + address.user, + address.password, + "server", + address.compression, + address.secure); + + info.pool = std::make_shared(ConnectionPoolPtrs{pool}, settings.load_balancing); + info.per_replica_pools = {std::move(pool)}; + + addresses_with_failover.emplace_back(Addresses{address}); + shards_info.emplace_back(std::move(info)); + } + } + + initMisc(); +} + + +Cluster::Cluster(Cluster::SubclusterTag, const Cluster & from, const std::vector & indices) : shards_info{} { for (size_t index : indices) diff --git a/dbms/src/Interpreters/Cluster.h b/dbms/src/Interpreters/Cluster.h index e778c9bcf6f..ef12f9fe78f 100644 --- a/dbms/src/Interpreters/Cluster.h +++ b/dbms/src/Interpreters/Cluster.h @@ -26,7 +26,7 @@ public: const String & username, const String & password, UInt16 clickhouse_port, bool treat_local_as_remote, bool secure = false); - Cluster(const Cluster &) = delete; + Cluster(const Cluster &)= delete; Cluster & operator=(const Cluster &) = delete; /// is used to set a limit on the size of the timeout @@ -148,6 +148,9 @@ public: /// Get a subcluster consisting of one or multiple shards - indexes by count (from 0) of the shard of this cluster. std::unique_ptr getClusterWithMultipleShards(const std::vector & indices) const; + /// Get a new Cluster that contains all servers (all shards with all replicas) from existing cluster as independent shards. + std::unique_ptr getClusterWithReplicasAsShards(const Settings & settings) const; + private: using SlotToShard = std::vector; SlotToShard slot_to_shard; @@ -159,7 +162,12 @@ private: void initMisc(); /// For getClusterWithMultipleShards implementation. - Cluster(const Cluster & from, const std::vector & indices); + struct SubclusterTag {}; + Cluster(SubclusterTag, const Cluster & from, const std::vector & indices); + + /// For getClusterWithReplicasAsShards implementation + struct ReplicasAsShardsTag {}; + Cluster(ReplicasAsShardsTag, const Cluster & from, const Settings & settings); String hash_of_addresses; /// Description of the cluster shards. diff --git a/dbms/src/Interpreters/Context.cpp b/dbms/src/Interpreters/Context.cpp index f7f68f7c951..874a6e5f486 100644 --- a/dbms/src/Interpreters/Context.cpp +++ b/dbms/src/Interpreters/Context.cpp @@ -33,8 +33,6 @@ #include #include #include -#include -#include #include #include #include @@ -924,21 +922,21 @@ StoragePtr Context::tryGetExternalTable(const String & table_name) const StoragePtr Context::getTable(const String & database_name, const String & table_name) const { - Exception exc; + std::optional exc; auto res = getTableImpl(database_name, table_name, &exc); if (!res) - throw exc; + throw *exc; return res; } StoragePtr Context::tryGetTable(const String & database_name, const String & table_name) const { - return getTableImpl(database_name, table_name, nullptr); + return getTableImpl(database_name, table_name, {}); } -StoragePtr Context::getTableImpl(const String & database_name, const String & table_name, Exception * exception) const +StoragePtr Context::getTableImpl(const String & database_name, const String & table_name, std::optional * exception) const { String db; DatabasePtr database; @@ -960,7 +958,7 @@ StoragePtr Context::getTableImpl(const String & database_name, const String & ta if (shared->databases.end() == it) { if (exception) - *exception = Exception("Database " + backQuoteIfNeed(db) + " doesn't exist", ErrorCodes::UNKNOWN_DATABASE); + exception->emplace("Database " + backQuoteIfNeed(db) + " doesn't exist", ErrorCodes::UNKNOWN_DATABASE); return {}; } @@ -971,7 +969,7 @@ StoragePtr Context::getTableImpl(const String & database_name, const String & ta if (!table) { if (exception) - *exception = Exception("Table " + backQuoteIfNeed(db) + "." + backQuoteIfNeed(table_name) + " doesn't exist.", ErrorCodes::UNKNOWN_TABLE); + exception->emplace("Table " + backQuoteIfNeed(db) + "." + backQuoteIfNeed(table_name) + " doesn't exist.", ErrorCodes::UNKNOWN_TABLE); return {}; } @@ -1088,7 +1086,6 @@ DatabasePtr Context::detachDatabase(const String & database_name) { auto lock = getLock(); auto res = getDatabase(database_name); - getExternalDictionariesLoader().removeConfigRepository(database_name); shared->databases.erase(database_name); return res; @@ -1436,7 +1433,7 @@ void Context::setMarkCache(size_t cache_size_in_bytes) if (shared->mark_cache) throw Exception("Mark cache has been already created.", ErrorCodes::LOGICAL_ERROR); - shared->mark_cache = std::make_shared(cache_size_in_bytes, std::chrono::seconds(settings.mark_cache_min_lifetime)); + shared->mark_cache = std::make_shared(cache_size_in_bytes); } diff --git a/dbms/src/Interpreters/Context.h b/dbms/src/Interpreters/Context.h index b0a4b5bb580..e1466713088 100644 --- a/dbms/src/Interpreters/Context.h +++ b/dbms/src/Interpreters/Context.h @@ -589,7 +589,7 @@ private: EmbeddedDictionaries & getEmbeddedDictionariesImpl(bool throw_on_error) const; - StoragePtr getTableImpl(const String & database_name, const String & table_name, Exception * exception) const; + StoragePtr getTableImpl(const String & database_name, const String & table_name, std::optional * exception) const; SessionKey getSessionKey(const String & session_id) const; diff --git a/dbms/src/Interpreters/DatabaseAndTableWithAlias.h b/dbms/src/Interpreters/DatabaseAndTableWithAlias.h index 3567a351b14..ad1f747b6fd 100644 --- a/dbms/src/Interpreters/DatabaseAndTableWithAlias.h +++ b/dbms/src/Interpreters/DatabaseAndTableWithAlias.h @@ -72,4 +72,6 @@ private: std::vector getDatabaseAndTables(const ASTSelectQuery & select_query, const String & current_database); std::optional getDatabaseAndTable(const ASTSelectQuery & select, size_t table_number); +using TablesWithColumnNames = std::vector; + } diff --git a/dbms/src/Interpreters/EmbeddedDictionaries.cpp b/dbms/src/Interpreters/EmbeddedDictionaries.cpp index c73850073cd..9ab3cf2dcbe 100644 --- a/dbms/src/Interpreters/EmbeddedDictionaries.cpp +++ b/dbms/src/Interpreters/EmbeddedDictionaries.cpp @@ -72,7 +72,7 @@ bool EmbeddedDictionaries::reloadImpl(const bool throw_on_error, const bool forc bool was_exception = false; - DictionaryReloader reload_regions_hierarchies = [=] (const Poco::Util::AbstractConfiguration & config) + DictionaryReloader reload_regions_hierarchies = [=, this] (const Poco::Util::AbstractConfiguration & config) { return geo_dictionaries_loader->reloadRegionsHierarchies(config); }; @@ -80,7 +80,7 @@ bool EmbeddedDictionaries::reloadImpl(const bool throw_on_error, const bool forc if (!reloadDictionary(regions_hierarchies, std::move(reload_regions_hierarchies), throw_on_error, force_reload)) was_exception = true; - DictionaryReloader reload_regions_names = [=] (const Poco::Util::AbstractConfiguration & config) + DictionaryReloader reload_regions_names = [=, this] (const Poco::Util::AbstractConfiguration & config) { return geo_dictionaries_loader->reloadRegionsNames(config); }; diff --git a/dbms/src/Interpreters/ExpressionAnalyzer.cpp b/dbms/src/Interpreters/ExpressionAnalyzer.cpp index 59dff858cf0..fa199f94efc 100644 --- a/dbms/src/Interpreters/ExpressionAnalyzer.cpp +++ b/dbms/src/Interpreters/ExpressionAnalyzer.cpp @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include diff --git a/dbms/src/Interpreters/ExternalDictionariesLoader.cpp b/dbms/src/Interpreters/ExternalDictionariesLoader.cpp index 53dc70fe5d4..d5f995a8db3 100644 --- a/dbms/src/Interpreters/ExternalDictionariesLoader.cpp +++ b/dbms/src/Interpreters/ExternalDictionariesLoader.cpp @@ -9,6 +9,7 @@ ExternalDictionariesLoader::ExternalDictionariesLoader(Context & context_) : ExternalLoader("external dictionary", &Logger::get("ExternalDictionariesLoader")) , context(context_) { + setConfigSettings({"dictionary", "name", "database"}); enableAsyncLoading(true); enablePeriodicUpdates(true); } @@ -23,11 +24,4 @@ ExternalLoader::LoadablePtr ExternalDictionariesLoader::create( bool dictionary_from_database = !repository_name.empty(); return DictionaryFactory::instance().create(name, config, key_in_config, context, dictionary_from_database); } - -void ExternalDictionariesLoader::addConfigRepository( - const std::string & repository_name, std::unique_ptr config_repository) -{ - ExternalLoader::addConfigRepository(repository_name, std::move(config_repository), {"dictionary", "name"}); -} - } diff --git a/dbms/src/Interpreters/ExternalDictionariesLoader.h b/dbms/src/Interpreters/ExternalDictionariesLoader.h index 15293ac09c0..90c49876ca4 100644 --- a/dbms/src/Interpreters/ExternalDictionariesLoader.h +++ b/dbms/src/Interpreters/ExternalDictionariesLoader.h @@ -29,10 +29,6 @@ public: return std::static_pointer_cast(tryLoad(name)); } - void addConfigRepository( - const std::string & repository_name, - std::unique_ptr config_repository); - protected: LoadablePtr create(const std::string & name, const Poco::Util::AbstractConfiguration & config, const std::string & key_in_config, const std::string & repository_name) const override; diff --git a/dbms/src/Interpreters/ExternalLoader.cpp b/dbms/src/Interpreters/ExternalLoader.cpp index 215263d8b3c..4b907b521e9 100644 --- a/dbms/src/Interpreters/ExternalLoader.cpp +++ b/dbms/src/Interpreters/ExternalLoader.cpp @@ -92,6 +92,7 @@ struct ExternalLoader::ObjectConfig Poco::AutoPtr config; String key_in_config; String repository_name; + bool from_temp_repository = false; String path; }; @@ -107,26 +108,30 @@ public: } ~LoadablesConfigReader() = default; - using RepositoryPtr = std::unique_ptr; + using Repository = IExternalLoaderConfigRepository; - void addConfigRepository(const String & repository_name, RepositoryPtr repository, const ExternalLoaderConfigSettings & settings) + void addConfigRepository(std::unique_ptr repository) { std::lock_guard lock{mutex}; - RepositoryInfo repository_info{std::move(repository), settings, {}}; - repositories.emplace(repository_name, std::move(repository_info)); + auto * ptr = repository.get(); + repositories.emplace(ptr, RepositoryInfo{std::move(repository), {}}); need_collect_object_configs = true; } - RepositoryPtr removeConfigRepository(const String & repository_name) + void removeConfigRepository(Repository * repository) { std::lock_guard lock{mutex}; - auto it = repositories.find(repository_name); + auto it = repositories.find(repository); if (it == repositories.end()) - return nullptr; - auto repository = std::move(it->second.repository); + return; repositories.erase(it); need_collect_object_configs = true; - return repository; + } + + void setConfigSettings(const ExternalLoaderConfigSettings & settings_) + { + std::lock_guard lock{mutex}; + settings = settings_; } using ObjectConfigsPtr = std::shared_ptr>; @@ -170,8 +175,7 @@ private: struct RepositoryInfo { - RepositoryPtr repository; - ExternalLoaderConfigSettings settings; + std::unique_ptr repository; std::unordered_map files; }; @@ -179,18 +183,10 @@ private: /// Checks last modification times of files and read those files which are new or changed. void readRepositories(const std::optional & only_repository_name = {}, const std::optional & only_path = {}) { - Strings repository_names; - if (only_repository_name) + for (auto & [repository, repository_info] : repositories) { - if (repositories.count(*only_repository_name)) - repository_names.push_back(*only_repository_name); - } - else - boost::copy(repositories | boost::adaptors::map_keys, std::back_inserter(repository_names)); - - for (const auto & repository_name : repository_names) - { - auto & repository_info = repositories[repository_name]; + if (only_repository_name && (repository->getName() != *only_repository_name)) + continue; for (auto & file_info : repository_info.files | boost::adaptors::map_values) file_info.in_use = false; @@ -198,11 +194,11 @@ private: Strings existing_paths; if (only_path) { - if (repository_info.repository->exists(*only_path)) + if (repository->exists(*only_path)) existing_paths.push_back(*only_path); } else - boost::copy(repository_info.repository->getAllLoadablesDefinitionNames(), std::back_inserter(existing_paths)); + boost::copy(repository->getAllLoadablesDefinitionNames(), std::back_inserter(existing_paths)); for (const auto & path : existing_paths) { @@ -210,13 +206,13 @@ private: if (it != repository_info.files.end()) { FileInfo & file_info = it->second; - if (readFileInfo(file_info, *repository_info.repository, path, repository_info.settings)) + if (readFileInfo(file_info, *repository, path)) need_collect_object_configs = true; } else { FileInfo file_info; - if (readFileInfo(file_info, *repository_info.repository, path, repository_info.settings)) + if (readFileInfo(file_info, *repository, path)) { repository_info.files.emplace(path, std::move(file_info)); need_collect_object_configs = true; @@ -249,8 +245,7 @@ private: bool readFileInfo( FileInfo & file_info, IExternalLoaderConfigRepository & repository, - const String & path, - const ExternalLoaderConfigSettings & settings) const + const String & path) const { try { @@ -293,7 +288,13 @@ private: continue; } - object_configs_from_file.emplace_back(object_name, ObjectConfig{file_contents, key, {}, {}}); + String database; + if (!settings.external_database.empty()) + database = file_contents->getString(key + "." + settings.external_database, ""); + if (!database.empty()) + object_name = database + "." + object_name; + + object_configs_from_file.emplace_back(object_name, ObjectConfig{file_contents, key, {}, {}, {}}); } file_info.objects = std::move(object_configs_from_file); @@ -318,7 +319,7 @@ private: // Generate new result. auto new_configs = std::make_shared>(); - for (const auto & [repository_name, repository_info] : repositories) + for (const auto & [repository, repository_info] : repositories) { for (const auto & [path, file_info] : repository_info.files) { @@ -328,19 +329,19 @@ private: if (already_added_it == new_configs->end()) { auto & new_config = new_configs->emplace(object_name, object_config).first->second; - new_config.repository_name = repository_name; + new_config.from_temp_repository = repository->isTemporary(); + new_config.repository_name = repository->getName(); new_config.path = path; } else { const auto & already_added = already_added_it->second; - if (!startsWith(repository_name, IExternalLoaderConfigRepository::INTERNAL_REPOSITORY_NAME_PREFIX) && - !startsWith(already_added.repository_name, IExternalLoaderConfigRepository::INTERNAL_REPOSITORY_NAME_PREFIX)) + if (!already_added.from_temp_repository && !repository->isTemporary()) { LOG_WARNING( log, type_name << " '" << object_name << "' is found " - << (((path == already_added.path) && repository_name == already_added.repository_name) + << (((path == already_added.path) && (repository->getName() == already_added.repository_name)) ? ("twice in the same file '" + path + "'") : ("both in file '" + already_added.path + "' and '" + path + "'"))); } @@ -356,7 +357,8 @@ private: Logger * log; std::mutex mutex; - std::unordered_map repositories; + ExternalLoaderConfigSettings settings; + std::unordered_map repositories; ObjectConfigsPtr object_configs; bool need_collect_object_configs = false; }; @@ -613,7 +615,7 @@ public: } catch (...) { - tryLogCurrentException(log, "Could not check if " + type_name + " '" + object->getName() + "' was modified"); + tryLogCurrentException(log, "Could not check if " + type_name + " '" + object->getLoadableName() + "' was modified"); /// Cannot check isModified, so update should_update_flag = true; } @@ -1151,20 +1153,23 @@ ExternalLoader::ExternalLoader(const String & type_name_, Logger * log_) ExternalLoader::~ExternalLoader() = default; -void ExternalLoader::addConfigRepository( - const std::string & repository_name, - std::unique_ptr config_repository, - const ExternalLoaderConfigSettings & config_settings) +ext::scope_guard ExternalLoader::addConfigRepository(std::unique_ptr repository) { - config_files_reader->addConfigRepository(repository_name, std::move(config_repository), config_settings); - reloadConfig(repository_name); + auto * ptr = repository.get(); + String name = ptr->getName(); + config_files_reader->addConfigRepository(std::move(repository)); + reloadConfig(name); + + return [this, ptr, name]() + { + config_files_reader->removeConfigRepository(ptr); + reloadConfig(name); + }; } -std::unique_ptr ExternalLoader::removeConfigRepository(const std::string & repository_name) +void ExternalLoader::setConfigSettings(const ExternalLoaderConfigSettings & settings) { - auto repository = config_files_reader->removeConfigRepository(repository_name); - reloadConfig(repository_name); - return repository; + config_files_reader->setConfigSettings(settings); } void ExternalLoader::enableAlwaysLoadEverything(bool enable) diff --git a/dbms/src/Interpreters/ExternalLoader.h b/dbms/src/Interpreters/ExternalLoader.h index 9ccdc4bf39c..1f65cb80f94 100644 --- a/dbms/src/Interpreters/ExternalLoader.h +++ b/dbms/src/Interpreters/ExternalLoader.h @@ -7,6 +7,7 @@ #include #include #include +#include namespace DB @@ -24,9 +25,9 @@ struct ExternalLoaderConfigSettings { std::string external_config; std::string external_name; + std::string external_database; }; - /** Interface for manage user-defined objects. * Monitors configuration file and automatically reloads objects in separate threads. * The monitoring thread wakes up every 'check_period_sec' seconds and checks @@ -87,13 +88,9 @@ public: virtual ~ExternalLoader(); /// Adds a repository which will be used to read configurations from. - void addConfigRepository( - const std::string & repository_name, - std::unique_ptr config_repository, - const ExternalLoaderConfigSettings & config_settings); + ext::scope_guard addConfigRepository(std::unique_ptr config_repository); - /// Removes a repository which were used to read configurations. - std::unique_ptr removeConfigRepository(const std::string & repository_name); + void setConfigSettings(const ExternalLoaderConfigSettings & settings_); /// Sets whether all the objects from the configuration should be always loaded (even those which are never used). void enableAlwaysLoadEverything(bool enable); diff --git a/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.cpp b/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.cpp index 3f5e271bd57..33469d95e08 100644 --- a/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.cpp +++ b/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.cpp @@ -1,4 +1,5 @@ #include +#include #include namespace DB @@ -11,40 +12,46 @@ namespace ErrorCodes namespace { -String trimDatabaseName(const std::string & loadable_definition_name, const DatabasePtr database) +String trimDatabaseName(const std::string & loadable_definition_name, const IDatabase & database) { - const auto & dbname = database->getDatabaseName(); + const auto & dbname = database.getDatabaseName(); if (!startsWith(loadable_definition_name, dbname)) throw Exception( - "Loadable '" + loadable_definition_name + "' is not from database '" + database->getDatabaseName(), ErrorCodes::UNKNOWN_DICTIONARY); + "Loadable '" + loadable_definition_name + "' is not from database '" + database.getDatabaseName(), ErrorCodes::UNKNOWN_DICTIONARY); /// dbname.loadable_name ///--> remove <--- return loadable_definition_name.substr(dbname.length() + 1); } } -LoadablesConfigurationPtr ExternalLoaderDatabaseConfigRepository::load(const std::string & loadable_definition_name) const +ExternalLoaderDatabaseConfigRepository::ExternalLoaderDatabaseConfigRepository(IDatabase & database_, const Context & context_) + : name(database_.getDatabaseName()) + , database(database_) + , context(context_) { - String dictname = trimDatabaseName(loadable_definition_name, database); - return getDictionaryConfigurationFromAST(database->getCreateDictionaryQuery(context, dictname)->as()); } -bool ExternalLoaderDatabaseConfigRepository::exists(const std::string & loadable_definition_name) const +LoadablesConfigurationPtr ExternalLoaderDatabaseConfigRepository::load(const std::string & loadable_definition_name) { - return database->isDictionaryExist( - context, trimDatabaseName(loadable_definition_name, database)); + String dictname = trimDatabaseName(loadable_definition_name, database); + return getDictionaryConfigurationFromAST(database.getCreateDictionaryQuery(context, dictname)->as()); +} + +bool ExternalLoaderDatabaseConfigRepository::exists(const std::string & loadable_definition_name) +{ + return database.isDictionaryExist(context, trimDatabaseName(loadable_definition_name, database)); } Poco::Timestamp ExternalLoaderDatabaseConfigRepository::getUpdateTime(const std::string & loadable_definition_name) { - return database->getObjectMetadataModificationTime(trimDatabaseName(loadable_definition_name, database)); + return database.getObjectMetadataModificationTime(trimDatabaseName(loadable_definition_name, database)); } -std::set ExternalLoaderDatabaseConfigRepository::getAllLoadablesDefinitionNames() const +std::set ExternalLoaderDatabaseConfigRepository::getAllLoadablesDefinitionNames() { std::set result; - const auto & dbname = database->getDatabaseName(); - auto itr = database->getDictionariesIterator(context); + const auto & dbname = database.getDatabaseName(); + auto itr = database.getDictionariesIterator(context); while (itr && itr->isValid()) { result.insert(dbname + "." + itr->name()); diff --git a/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.h b/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.h index 343ed8cf038..2afff035d9d 100644 --- a/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.h +++ b/dbms/src/Interpreters/ExternalLoaderDatabaseConfigRepository.h @@ -12,22 +12,21 @@ namespace DB class ExternalLoaderDatabaseConfigRepository : public IExternalLoaderConfigRepository { public: - ExternalLoaderDatabaseConfigRepository(const DatabasePtr & database_, const Context & context_) - : database(database_) - , context(context_) - { - } + ExternalLoaderDatabaseConfigRepository(IDatabase & database_, const Context & context_); - std::set getAllLoadablesDefinitionNames() const override; + const std::string & getName() const override { return name; } - bool exists(const std::string & loadable_definition_name) const override; + std::set getAllLoadablesDefinitionNames() override; + + bool exists(const std::string & loadable_definition_name) override; Poco::Timestamp getUpdateTime(const std::string & loadable_definition_name) override; - LoadablesConfigurationPtr load(const std::string & loadable_definition_name) const override; + LoadablesConfigurationPtr load(const std::string & loadable_definition_name) override; private: - DatabasePtr database; + const String name; + IDatabase & database; Context context; }; diff --git a/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.cpp b/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.cpp deleted file mode 100644 index 16c8a3aa59c..00000000000 --- a/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.cpp +++ /dev/null @@ -1,49 +0,0 @@ -#include -#include -#include -#include - - -namespace DB -{ -namespace ErrorCodes -{ - extern const int BAD_ARGUMENTS; -} - -ExternalLoaderPresetConfigRepository::ExternalLoaderPresetConfigRepository(const std::vector> & preset_) -{ - boost::range::copy(preset_, std::inserter(preset, preset.end())); -} - -ExternalLoaderPresetConfigRepository::~ExternalLoaderPresetConfigRepository() = default; - -std::set ExternalLoaderPresetConfigRepository::getAllLoadablesDefinitionNames() const -{ - std::set paths; - boost::range::copy(preset | boost::adaptors::map_keys, std::inserter(paths, paths.end())); - return paths; -} - -bool ExternalLoaderPresetConfigRepository::exists(const String& path) const -{ - return preset.count(path); -} - -Poco::Timestamp ExternalLoaderPresetConfigRepository::getUpdateTime(const String & path) -{ - if (!exists(path)) - throw Exception("Loadable " + path + " not found", ErrorCodes::BAD_ARGUMENTS); - return creation_time; -} - -/// May contain definition about several entities (several dictionaries in one .xml file) -LoadablesConfigurationPtr ExternalLoaderPresetConfigRepository::load(const String & path) const -{ - auto it = preset.find(path); - if (it == preset.end()) - throw Exception("Loadable " + path + " not found", ErrorCodes::BAD_ARGUMENTS); - return it->second; -} - -} diff --git a/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.h b/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.h deleted file mode 100644 index b35209a7fb9..00000000000 --- a/dbms/src/Interpreters/ExternalLoaderPresetConfigRepository.h +++ /dev/null @@ -1,28 +0,0 @@ -#pragma once - -#include -#include -#include -#include - - -namespace DB -{ -/// A config repository filled with preset loadables used by ExternalLoader. -class ExternalLoaderPresetConfigRepository : public IExternalLoaderConfigRepository -{ -public: - ExternalLoaderPresetConfigRepository(const std::vector> & preset_); - ~ExternalLoaderPresetConfigRepository() override; - - std::set getAllLoadablesDefinitionNames() const override; - bool exists(const String & path) const override; - Poco::Timestamp getUpdateTime(const String & path) override; - LoadablesConfigurationPtr load(const String & path) const override; - -private: - std::unordered_map preset; - Poco::Timestamp creation_time; -}; - -} diff --git a/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.cpp b/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.cpp new file mode 100644 index 00000000000..c4210875867 --- /dev/null +++ b/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.cpp @@ -0,0 +1,46 @@ +#include +#include + + +namespace DB +{ +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + + +ExternalLoaderTempConfigRepository::ExternalLoaderTempConfigRepository(const String & repository_name_, const String & path_, const LoadablesConfigurationPtr & config_) + : name(repository_name_), path(path_), config(config_) {} + + +std::set ExternalLoaderTempConfigRepository::getAllLoadablesDefinitionNames() +{ + std::set paths; + paths.insert(path); + return paths; +} + + +bool ExternalLoaderTempConfigRepository::exists(const String & path_) +{ + return path == path_; +} + + +Poco::Timestamp ExternalLoaderTempConfigRepository::getUpdateTime(const String & path_) +{ + if (!exists(path_)) + throw Exception("Loadable " + path_ + " not found", ErrorCodes::BAD_ARGUMENTS); + return creation_time; +} + + +LoadablesConfigurationPtr ExternalLoaderTempConfigRepository::load(const String & path_) +{ + if (!exists(path_)) + throw Exception("Loadable " + path_ + " not found", ErrorCodes::BAD_ARGUMENTS); + return config; +} + +} diff --git a/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.h b/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.h new file mode 100644 index 00000000000..6ee717631cc --- /dev/null +++ b/dbms/src/Interpreters/ExternalLoaderTempConfigRepository.h @@ -0,0 +1,31 @@ +#pragma once + +#include +#include +#include + + +namespace DB +{ +/// A config repository filled with preset loadables used by ExternalLoader. +class ExternalLoaderTempConfigRepository : public IExternalLoaderConfigRepository +{ +public: + ExternalLoaderTempConfigRepository(const String & repository_name_, const String & path_, const LoadablesConfigurationPtr & config_); + + const String & getName() const override { return name; } + bool isTemporary() const override { return true; } + + std::set getAllLoadablesDefinitionNames() override; + bool exists(const String & path) override; + Poco::Timestamp getUpdateTime(const String & path) override; + LoadablesConfigurationPtr load(const String & path) override; + +private: + String name; + String path; + LoadablesConfigurationPtr config; + Poco::Timestamp creation_time; +}; + +} diff --git a/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.cpp b/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.cpp index 9a5e32697df..63755ee1839 100644 --- a/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.cpp +++ b/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.cpp @@ -11,13 +11,18 @@ namespace DB { +ExternalLoaderXMLConfigRepository::ExternalLoaderXMLConfigRepository( + const Poco::Util::AbstractConfiguration & main_config_, const std::string & config_key_) + : main_config(main_config_), config_key(config_key_) +{ +} Poco::Timestamp ExternalLoaderXMLConfigRepository::getUpdateTime(const std::string & definition_entity_name) { return Poco::File(definition_entity_name).getLastModified(); } -std::set ExternalLoaderXMLConfigRepository::getAllLoadablesDefinitionNames() const +std::set ExternalLoaderXMLConfigRepository::getAllLoadablesDefinitionNames() { std::set files; @@ -52,13 +57,13 @@ std::set ExternalLoaderXMLConfigRepository::getAllLoadablesDefiniti return files; } -bool ExternalLoaderXMLConfigRepository::exists(const std::string & definition_entity_name) const +bool ExternalLoaderXMLConfigRepository::exists(const std::string & definition_entity_name) { return Poco::File(definition_entity_name).exists(); } Poco::AutoPtr ExternalLoaderXMLConfigRepository::load( - const std::string & config_file) const + const std::string & config_file) { ConfigProcessor config_processor{config_file}; ConfigProcessor::LoadedConfig preprocessed = config_processor.loadConfig(); diff --git a/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.h b/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.h index b8676209c14..808fa66fdbf 100644 --- a/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.h +++ b/dbms/src/Interpreters/ExternalLoaderXMLConfigRepository.h @@ -13,26 +13,25 @@ namespace DB class ExternalLoaderXMLConfigRepository : public IExternalLoaderConfigRepository { public: + ExternalLoaderXMLConfigRepository(const Poco::Util::AbstractConfiguration & main_config_, const std::string & config_key_); - ExternalLoaderXMLConfigRepository(const Poco::Util::AbstractConfiguration & main_config_, const std::string & config_key_) - : main_config(main_config_) - , config_key(config_key_) - { - } + const String & getName() const override { return name; } /// Return set of .xml files from path in main_config (config_key) - std::set getAllLoadablesDefinitionNames() const override; + std::set getAllLoadablesDefinitionNames() override; /// Checks that file with name exists on filesystem - bool exists(const std::string & definition_entity_name) const override; + bool exists(const std::string & definition_entity_name) override; /// Return xml-file modification time via stat call Poco::Timestamp getUpdateTime(const std::string & definition_entity_name) override; /// May contain definition about several entities (several dictionaries in one .xml file) - LoadablesConfigurationPtr load(const std::string & definition_entity_name) const override; + LoadablesConfigurationPtr load(const std::string & definition_entity_name) override; private: + const String name; + /// Main server config (config.xml). const Poco::Util::AbstractConfiguration & main_config; diff --git a/dbms/src/Interpreters/ExternalModelsLoader.cpp b/dbms/src/Interpreters/ExternalModelsLoader.cpp index 2a83b8324a4..7334cfbaa3a 100644 --- a/dbms/src/Interpreters/ExternalModelsLoader.cpp +++ b/dbms/src/Interpreters/ExternalModelsLoader.cpp @@ -14,6 +14,7 @@ ExternalModelsLoader::ExternalModelsLoader(Context & context_) : ExternalLoader("external model", &Logger::get("ExternalModelsLoader")) , context(context_) { + setConfigSettings({"models", "name", {}}); enablePeriodicUpdates(true); } @@ -38,9 +39,4 @@ std::shared_ptr ExternalModelsLoader::create( throw Exception("Unknown model type: " + type, ErrorCodes::INVALID_CONFIG_PARAMETER); } } - -void ExternalModelsLoader::addConfigRepository(const String & name, std::unique_ptr config_repository) -{ - ExternalLoader::addConfigRepository(name, std::move(config_repository), {"models", "name"}); -} } diff --git a/dbms/src/Interpreters/ExternalModelsLoader.h b/dbms/src/Interpreters/ExternalModelsLoader.h index 753bad20ca0..4d09cebc307 100644 --- a/dbms/src/Interpreters/ExternalModelsLoader.h +++ b/dbms/src/Interpreters/ExternalModelsLoader.h @@ -25,10 +25,6 @@ public: return std::static_pointer_cast(load(name)); } - void addConfigRepository(const String & name, - std::unique_ptr config_repository); - - protected: LoadablePtr create(const std::string & name, const Poco::Util::AbstractConfiguration & config, const std::string & key_in_config, const std::string & repository_name) const override; diff --git a/dbms/src/Interpreters/ExtractExpressionInfoVisitor.cpp b/dbms/src/Interpreters/ExtractExpressionInfoVisitor.cpp new file mode 100644 index 00000000000..1240b6a09d6 --- /dev/null +++ b/dbms/src/Interpreters/ExtractExpressionInfoVisitor.cpp @@ -0,0 +1,79 @@ +#include +#include +#include +#include +#include + + +namespace DB +{ + +void ExpressionInfoMatcher::visit(const ASTPtr & ast, Data & data) +{ + if (const auto * function = ast->as()) + visit(*function, ast, data); + else if (const auto * identifier = ast->as()) + visit(*identifier, ast, data); +} + +void ExpressionInfoMatcher::visit(const ASTFunction & ast_function, const ASTPtr &, Data & data) +{ + if (ast_function.name == "arrayJoin") + data.is_array_join = true; + else if (AggregateFunctionFactory::instance().isAggregateFunctionName(ast_function.name)) + data.is_aggregate_function = true; + else + { + const auto & function = FunctionFactory::instance().tryGet(ast_function.name, data.context); + + /// Skip lambda, tuple and other special functions + if (function && function->isStateful()) + data.is_stateful_function = true; + } +} + +void ExpressionInfoMatcher::visit(const ASTIdentifier & identifier, const ASTPtr &, Data & data) +{ + if (!identifier.compound()) + { + for (size_t index = 0; index < data.tables.size(); ++index) + { + const auto & columns = data.tables[index].columns; + + // TODO: make sure no collision ever happens + if (std::find(columns.begin(), columns.end(), identifier.name) != columns.end()) + { + data.unique_reference_tables_pos.emplace(index); + break; + } + } + } + else + { + size_t best_table_pos = 0; + if (IdentifierSemantic::chooseTable(identifier, data.tables, best_table_pos)) + data.unique_reference_tables_pos.emplace(best_table_pos); + } +} + +bool ExpressionInfoMatcher::needChildVisit(const ASTPtr & node, const ASTPtr &) +{ + return !node->as(); +} + +bool hasStatefulFunction(const ASTPtr & node, const Context & context) +{ + for (const auto & select_expression : node->children) + { + ExpressionInfoVisitor::Data expression_info{.context = context, .tables = {}}; + ExpressionInfoVisitor(expression_info).visit(select_expression); + + if (expression_info.is_stateful_function) + return true; + } + + return false; +} + +} + diff --git a/dbms/src/Interpreters/ExtractExpressionInfoVisitor.h b/dbms/src/Interpreters/ExtractExpressionInfoVisitor.h new file mode 100644 index 00000000000..65d23057e52 --- /dev/null +++ b/dbms/src/Interpreters/ExtractExpressionInfoVisitor.h @@ -0,0 +1,40 @@ +#pragma once + +#include +#include +#include +#include +#include + +namespace DB +{ + +class Context; + +struct ExpressionInfoMatcher +{ + struct Data + { + const Context & context; + const std::vector & tables; + + bool is_array_join = false; + bool is_stateful_function = false; + bool is_aggregate_function = false; + std::unordered_set unique_reference_tables_pos = {}; + }; + + static void visit(const ASTPtr & ast, Data & data); + + static bool needChildVisit(const ASTPtr & node, const ASTPtr &); + + static void visit(const ASTFunction & ast_function, const ASTPtr &, Data & data); + + static void visit(const ASTIdentifier & identifier, const ASTPtr &, Data & data); +}; + +using ExpressionInfoVisitor = ConstInDepthNodeVisitor; + +bool hasStatefulFunction(const ASTPtr & node, const Context & context); + +} diff --git a/dbms/src/Interpreters/ExtractFunctionDataVisitor.cpp b/dbms/src/Interpreters/ExtractFunctionDataVisitor.cpp deleted file mode 100644 index d7a0d9001d5..00000000000 --- a/dbms/src/Interpreters/ExtractFunctionDataVisitor.cpp +++ /dev/null @@ -1,16 +0,0 @@ -#include -#include - - -namespace DB -{ - -void ExtractFunctionData::visit(ASTFunction & function, ASTPtr &) -{ - if (AggregateFunctionFactory::instance().isAggregateFunctionName(function.name)) - aggregate_functions.emplace_back(&function); - else - functions.emplace_back(&function); -} - -} diff --git a/dbms/src/Interpreters/ExtractFunctionDataVisitor.h b/dbms/src/Interpreters/ExtractFunctionDataVisitor.h deleted file mode 100644 index ed3dbb868c4..00000000000 --- a/dbms/src/Interpreters/ExtractFunctionDataVisitor.h +++ /dev/null @@ -1,25 +0,0 @@ -#pragma once - -#include -#include -#include -#include -#include - -namespace DB -{ - -struct ExtractFunctionData -{ - using TypeToVisit = ASTFunction; - - std::vector functions; - std::vector aggregate_functions; - - void visit(ASTFunction & identifier, ASTPtr &); -}; - -using ExtractFunctionMatcher = OneTypeMatcher; -using ExtractFunctionVisitor = InDepthNodeVisitor; - -} diff --git a/dbms/src/Interpreters/FindIdentifierBestTableVisitor.cpp b/dbms/src/Interpreters/FindIdentifierBestTableVisitor.cpp deleted file mode 100644 index 56897ec15c7..00000000000 --- a/dbms/src/Interpreters/FindIdentifierBestTableVisitor.cpp +++ /dev/null @@ -1,40 +0,0 @@ -#include -#include - - -namespace DB -{ - -FindIdentifierBestTableData::FindIdentifierBestTableData(const std::vector & tables_) - : tables(tables_) -{ -} - -void FindIdentifierBestTableData::visit(ASTIdentifier & identifier, ASTPtr &) -{ - const DatabaseAndTableWithAlias * best_table = nullptr; - - if (!identifier.compound()) - { - for (const auto & table_names : tables) - { - auto & columns = table_names.columns; - if (std::find(columns.begin(), columns.end(), identifier.name) != columns.end()) - { - // TODO: make sure no collision ever happens - if (!best_table) - best_table = &table_names.table; - } - } - } - else - { - size_t best_table_pos = 0; - if (IdentifierSemantic::chooseTable(identifier, tables, best_table_pos)) - best_table = &tables[best_table_pos].table; - } - - identifier_table.emplace_back(&identifier, best_table); -} - -} diff --git a/dbms/src/Interpreters/FindIdentifierBestTableVisitor.h b/dbms/src/Interpreters/FindIdentifierBestTableVisitor.h deleted file mode 100644 index 498ee60ab0b..00000000000 --- a/dbms/src/Interpreters/FindIdentifierBestTableVisitor.h +++ /dev/null @@ -1,27 +0,0 @@ -#pragma once - -#include -#include -#include -#include - -namespace DB -{ - -struct FindIdentifierBestTableData -{ - using TypeToVisit = ASTIdentifier; - using IdentifierWithTable = std::pair; - - const std::vector & tables; - std::vector identifier_table; - - FindIdentifierBestTableData(const std::vector & tables_); - - void visit(ASTIdentifier & identifier, ASTPtr &); -}; - -using FindIdentifierBestTableMatcher = OneTypeMatcher; -using FindIdentifierBestTableVisitor = InDepthNodeVisitor; - -} diff --git a/dbms/src/Interpreters/IExternalLoadable.h b/dbms/src/Interpreters/IExternalLoadable.h index d4b93c56d2a..f9f24a9bbac 100644 --- a/dbms/src/Interpreters/IExternalLoadable.h +++ b/dbms/src/Interpreters/IExternalLoadable.h @@ -36,7 +36,7 @@ public: virtual const ExternalLoadableLifetime & getLifetime() const = 0; - virtual std::string getName() const = 0; + virtual const std::string & getLoadableName() const = 0; /// True if object can be updated when lifetime exceeded. virtual bool supportUpdates() const = 0; /// If lifetime exceeded and isModified(), ExternalLoader replace current object with the result of clone(). diff --git a/dbms/src/Interpreters/IExternalLoaderConfigRepository.cpp b/dbms/src/Interpreters/IExternalLoaderConfigRepository.cpp deleted file mode 100644 index 968a9bca9de..00000000000 --- a/dbms/src/Interpreters/IExternalLoaderConfigRepository.cpp +++ /dev/null @@ -1,7 +0,0 @@ -#include - - -namespace DB -{ -const char * IExternalLoaderConfigRepository::INTERNAL_REPOSITORY_NAME_PREFIX = "\xFF internal repo "; -} diff --git a/dbms/src/Interpreters/IExternalLoaderConfigRepository.h b/dbms/src/Interpreters/IExternalLoaderConfigRepository.h index bcac36d9807..866aa0b877f 100644 --- a/dbms/src/Interpreters/IExternalLoaderConfigRepository.h +++ b/dbms/src/Interpreters/IExternalLoaderConfigRepository.h @@ -4,13 +4,13 @@ #include #include +#include #include #include #include namespace DB { - using LoadablesConfigurationPtr = Poco::AutoPtr; /// Base interface for configurations source for Loadble objects, which can be @@ -22,24 +22,27 @@ using LoadablesConfigurationPtr = Poco::AutoPtr getAllLoadablesDefinitionNames() const = 0; + virtual std::set getAllLoadablesDefinitionNames() = 0; /// Checks that source of loadables configuration exist. - virtual bool exists(const std::string & loadable_definition_name) const = 0; + virtual bool exists(const std::string & path) = 0; /// Returns entity last update time - virtual Poco::Timestamp getUpdateTime(const std::string & loadable_definition_name) = 0; + virtual Poco::Timestamp getUpdateTime(const std::string & path) = 0; /// Load configuration from some concrete source to AbstractConfiguration - virtual LoadablesConfigurationPtr load(const std::string & loadable_definition_name) const = 0; + virtual LoadablesConfigurationPtr load(const std::string & path) = 0; - virtual ~IExternalLoaderConfigRepository() = default; - - static const char * INTERNAL_REPOSITORY_NAME_PREFIX; + virtual ~IExternalLoaderConfigRepository() {} }; -using ExternalLoaderConfigRepositoryPtr = std::unique_ptr; - } diff --git a/dbms/src/Interpreters/InDepthNodeVisitor.h b/dbms/src/Interpreters/InDepthNodeVisitor.h index 18b84b11b24..7bb4f5e4d54 100644 --- a/dbms/src/Interpreters/InDepthNodeVisitor.h +++ b/dbms/src/Interpreters/InDepthNodeVisitor.h @@ -59,7 +59,13 @@ public: using Data = Data_; using TypeToVisit = typename Data::TypeToVisit; - static bool needChildVisit(const ASTPtr &, const ASTPtr &) { return visit_children; } + static bool needChildVisit(const ASTPtr & node, const ASTPtr &) + { + if (node && node->as()) + return visit_children; + + return true; + } static void visit(T & ast, Data & data) { diff --git a/dbms/src/Interpreters/InterpreterCreateQuery.cpp b/dbms/src/Interpreters/InterpreterCreateQuery.cpp index 2a2160ca058..db48c3bed92 100644 --- a/dbms/src/Interpreters/InterpreterCreateQuery.cpp +++ b/dbms/src/Interpreters/InterpreterCreateQuery.cpp @@ -514,14 +514,21 @@ void InterpreterCreateQuery::setEngine(ASTCreateQuery & create) const ASTPtr as_create_ptr = context.getDatabase(as_database_name)->getCreateTableQuery(context, as_table_name); const auto & as_create = as_create_ptr->as(); + const String qualified_name = backQuoteIfNeed(as_database_name) + "." + backQuoteIfNeed(as_table_name); + if (as_create.is_view) throw Exception( - "Cannot CREATE a table AS " + as_database_name + "." + as_table_name + ", it is a View", + "Cannot CREATE a table AS " + qualified_name + ", it is a View", ErrorCodes::INCORRECT_QUERY); if (as_create.is_live_view) throw Exception( - "Cannot CREATE a table AS " + as_database_name + "." + as_table_name + ", it is a Live View", + "Cannot CREATE a table AS " + qualified_name + ", it is a Live View", + ErrorCodes::INCORRECT_QUERY); + + if (as_create.is_dictionary) + throw Exception( + "Cannot CREATE a table AS " + qualified_name + ", it is a Dictionary", ErrorCodes::INCORRECT_QUERY); create.set(create.storage, as_create.storage->ptr()); @@ -691,7 +698,9 @@ BlockIO InterpreterCreateQuery::createDictionary(ASTCreateQuery & create) String dictionary_name = create.table; - String database_name = !create.database.empty() ? create.database : context.getCurrentDatabase(); + if (create.database.empty()) + create.database = context.getCurrentDatabase(); + const String & database_name = create.database; auto guard = context.getDDLGuard(database_name, dictionary_name); DatabasePtr database = context.getDatabase(database_name); diff --git a/dbms/src/Interpreters/InterpreterSelectQuery.cpp b/dbms/src/Interpreters/InterpreterSelectQuery.cpp index 45ace212c68..b147c5f4887 100644 --- a/dbms/src/Interpreters/InterpreterSelectQuery.cpp +++ b/dbms/src/Interpreters/InterpreterSelectQuery.cpp @@ -405,9 +405,19 @@ InterpreterSelectQuery::InterpreterSelectQuery( query.setExpression(ASTSelectQuery::Expression::WHERE, std::make_shared(0u)); need_analyze_again = true; } + if (query.prewhere() && query.where()) + { + /// Filter block in WHERE instead to get better performance + query.setExpression(ASTSelectQuery::Expression::WHERE, makeASTFunction("and", query.prewhere()->clone(), query.where()->clone())); + need_analyze_again = true; + } if (need_analyze_again) analyze(); + /// If there is no WHERE, filter blocks as usual + if (query.prewhere() && !query.where()) + analysis_result.prewhere_info->need_filter = true; + /// Blocks used in expression analysis contains size 1 const columns for constant folding and /// null non-const columns to avoid useless memory allocations. However, a valid block sample /// requires all columns to be of size 0, thus we need to sanitize the block here. @@ -478,8 +488,8 @@ BlockInputStreams InterpreterSelectQuery::executeWithMultipleStreams(QueryPipeli QueryPipeline InterpreterSelectQuery::executeWithProcessors() { QueryPipeline query_pipeline; - query_pipeline.setMaxThreads(context->getSettingsRef().max_threads); executeImpl(query_pipeline, input, query_pipeline); + query_pipeline.setMaxThreads(max_streams); query_pipeline.addInterpreterContext(context); query_pipeline.addStorageHolder(storage); return query_pipeline; @@ -493,28 +503,31 @@ Block InterpreterSelectQuery::getSampleBlockImpl() /// Do all AST changes here, because actions from analysis_result will be used later in readImpl. - /// PREWHERE optimization. - /// Turn off, if the table filter (row-level security) is applied. - if (storage && !context->getRowPolicy()->getCondition(storage->getDatabaseName(), storage->getTableName(), RowPolicy::SELECT_FILTER)) + if (storage) { query_analyzer->makeSetsForIndex(query.where()); query_analyzer->makeSetsForIndex(query.prewhere()); - auto optimize_prewhere = [&](auto & merge_tree) + /// PREWHERE optimization. + /// Turn off, if the table filter (row-level security) is applied. + if (!context->getRowPolicy()->getCondition(storage->getDatabaseName(), storage->getTableName(), RowPolicy::SELECT_FILTER)) { - SelectQueryInfo current_info; - current_info.query = query_ptr; - current_info.syntax_analyzer_result = syntax_analyzer_result; - current_info.sets = query_analyzer->getPreparedSets(); + auto optimize_prewhere = [&](auto & merge_tree) + { + SelectQueryInfo current_info; + current_info.query = query_ptr; + current_info.syntax_analyzer_result = syntax_analyzer_result; + current_info.sets = query_analyzer->getPreparedSets(); - /// Try transferring some condition from WHERE to PREWHERE if enabled and viable - if (settings.optimize_move_to_prewhere && query.where() && !query.prewhere() && !query.final()) - MergeTreeWhereOptimizer{current_info, *context, merge_tree, - syntax_analyzer_result->requiredSourceColumns(), log}; - }; + /// Try transferring some condition from WHERE to PREWHERE if enabled and viable + if (settings.optimize_move_to_prewhere && query.where() && !query.prewhere() && !query.final()) + MergeTreeWhereOptimizer{current_info, *context, merge_tree, + syntax_analyzer_result->requiredSourceColumns(), log}; + }; - if (const auto * merge_tree_data = dynamic_cast(storage.get())) - optimize_prewhere(*merge_tree_data); + if (const auto * merge_tree_data = dynamic_cast(storage.get())) + optimize_prewhere(*merge_tree_data); + } } if (storage && !options.only_analyze) @@ -1170,7 +1183,6 @@ void InterpreterSelectQuery::executeImpl(TPipeline & pipeline, const BlockInputS if (expressions.second_stage) { bool need_second_distinct_pass = false; - bool need_merge_streams = false; if (expressions.need_aggregate) { @@ -1206,7 +1218,7 @@ void InterpreterSelectQuery::executeImpl(TPipeline & pipeline, const BlockInputS } else if (query.group_by_with_totals || query.group_by_with_rollup || query.group_by_with_cube) - throw Exception("WITH TOTALS, ROLLUP or CUBE are not supported without aggregation", ErrorCodes::LOGICAL_ERROR); + throw Exception("WITH TOTALS, ROLLUP or CUBE are not supported without aggregation", ErrorCodes::NOT_IMPLEMENTED); need_second_distinct_pass = query.distinct && pipeline.hasMixedStreams(); @@ -1231,13 +1243,11 @@ void InterpreterSelectQuery::executeImpl(TPipeline & pipeline, const BlockInputS executePreLimit(pipeline); } - if (need_second_distinct_pass - || query.limitLength() - || query.limitBy() - || pipeline.hasDelayedStream()) - { - need_merge_streams = true; - } + bool need_merge_streams = need_second_distinct_pass || query.limitLength() || query.limitBy(); + + if constexpr (!pipeline_with_processors) + if (pipeline.hasDelayedStream()) + need_merge_streams = true; if (need_merge_streams) { @@ -1783,6 +1793,9 @@ void InterpreterSelectQuery::executeFetchColumns( // pipes[i].pinSources(i); // } + for (auto & pipe : pipes) + pipe.enableQuota(); + pipeline.init(std::move(pipes)); } else @@ -1920,7 +1933,7 @@ void InterpreterSelectQuery::executeAggregation(QueryPipeline & pipeline, const * 1. Parallel aggregation is done, and the results should be merged in parallel. * 2. An aggregation is done with store of temporary data on the disk, and they need to be merged in a memory efficient way. */ - bool allow_to_use_two_level_group_by = pipeline.getNumMainStreams() > 1 || settings.max_bytes_before_external_group_by != 0; + bool allow_to_use_two_level_group_by = pipeline.getNumStreams() > 1 || settings.max_bytes_before_external_group_by != 0; Aggregator::Params params(header_before_aggregation, keys, aggregates, overflow_row, settings.max_rows_to_group_by, settings.group_by_overflow_mode, @@ -1934,12 +1947,12 @@ void InterpreterSelectQuery::executeAggregation(QueryPipeline & pipeline, const pipeline.dropTotalsIfHas(); /// If there are several sources, then we perform parallel aggregation - if (pipeline.getNumMainStreams() > 1) + if (pipeline.getNumStreams() > 1) { /// Add resize transform to uniformly distribute data between aggregating streams. - pipeline.resize(pipeline.getNumMainStreams(), true); + pipeline.resize(pipeline.getNumStreams(), true); - auto many_data = std::make_shared(pipeline.getNumMainStreams()); + auto many_data = std::make_shared(pipeline.getNumStreams()); auto merge_threads = settings.aggregation_memory_efficient_merge_threads ? static_cast(settings.aggregation_memory_efficient_merge_threads) : static_cast(settings.max_threads); @@ -1961,6 +1974,8 @@ void InterpreterSelectQuery::executeAggregation(QueryPipeline & pipeline, const return std::make_shared(header, transform_params); }); } + + pipeline.enableQuotaForCurrentStreams(); } @@ -2074,6 +2089,8 @@ void InterpreterSelectQuery::executeMergeAggregated(QueryPipeline & pipeline, bo pipeline.addPipe(std::move(pipe)); } + + pipeline.enableQuotaForCurrentStreams(); } @@ -2257,17 +2274,17 @@ void InterpreterSelectQuery::executeOrder(Pipeline & pipeline, InputSortingInfoP limits.size_limits = SizeLimits(settings.max_rows_to_sort, settings.max_bytes_to_sort, settings.sort_overflow_mode); sorting_stream->setLimits(limits); - stream = sorting_stream; + auto merging_stream = std::make_shared( + sorting_stream, output_order_descr, settings.max_block_size, limit, + settings.max_bytes_before_remerge_sort, + settings.max_bytes_before_external_sort / pipeline.streams.size(), + context->getTemporaryPath(), settings.min_free_disk_space_for_temporary_data); + + stream = merging_stream; }); /// If there are several streams, we merge them into one - executeUnion(pipeline, {}); - - /// Merge the sorted blocks. - pipeline.firstStream() = std::make_shared( - pipeline.firstStream(), output_order_descr, settings.max_block_size, limit, - settings.max_bytes_before_remerge_sort, - settings.max_bytes_before_external_sort, context->getTemporaryPath(), settings.min_free_disk_space_for_temporary_data); + executeMergeSorted(pipeline, output_order_descr, limit); } } @@ -2307,6 +2324,8 @@ void InterpreterSelectQuery::executeOrder(QueryPipeline & pipeline, InputSorting pipeline.addPipe({ std::move(transform) }); } + pipeline.enableQuotaForCurrentStreams(); + if (need_finish_sorting) { pipeline.addSimpleTransform([&](const Block & header, QueryPipeline::StreamType stream_type) @@ -2332,9 +2351,6 @@ void InterpreterSelectQuery::executeOrder(QueryPipeline & pipeline, InputSorting return std::make_shared(header, output_order_descr, limit, do_count_rows); }); - /// If there are several streams, we merge them into one - pipeline.resize(1); - /// Merge the sorted blocks. pipeline.addSimpleTransform([&](const Block & header, QueryPipeline::StreamType stream_type) -> ProcessorPtr { @@ -2343,9 +2359,12 @@ void InterpreterSelectQuery::executeOrder(QueryPipeline & pipeline, InputSorting return std::make_shared( header, output_order_descr, settings.max_block_size, limit, - settings.max_bytes_before_remerge_sort, + settings.max_bytes_before_remerge_sort / pipeline.getNumStreams(), settings.max_bytes_before_external_sort, context->getTemporaryPath(), settings.min_free_disk_space_for_temporary_data); }); + + /// If there are several streams, we merge them into one + executeMergeSorted(pipeline, output_order_descr, limit); } @@ -2407,6 +2426,8 @@ void InterpreterSelectQuery::executeMergeSorted(QueryPipeline & pipeline, const settings.max_block_size, limit); pipeline.addPipe({ std::move(transform) }); + + pipeline.enableQuotaForCurrentStreams(); } } @@ -2784,11 +2805,7 @@ void InterpreterSelectQuery::executeSubqueriesInSetsAndJoins(Pipeline & pipeline void InterpreterSelectQuery::executeSubqueriesInSetsAndJoins(QueryPipeline & pipeline, SubqueriesForSets & subqueries_for_sets) { if (query_info.input_sorting_info) - { - if (pipeline.hasDelayedStream()) - throw Exception("Using read in order optimization, but has delayed stream in pipeline", ErrorCodes::LOGICAL_ERROR); executeMergeSorted(pipeline, query_info.input_sorting_info->order_key_prefix_descr, 0); - } const Settings & settings = context->getSettingsRef(); @@ -2805,7 +2822,7 @@ void InterpreterSelectQuery::unifyStreams(Pipeline & pipeline, Block header) { /// Unify streams in case they have different headers. - /// TODO: remove previos addition of _dummy column. + /// TODO: remove previous addition of _dummy column. if (header.columns() > 1 && header.has("_dummy")) header.erase("_dummy"); diff --git a/dbms/src/Interpreters/OptimizeIfChains.cpp b/dbms/src/Interpreters/OptimizeIfChains.cpp new file mode 100644 index 00000000000..d440b204d54 --- /dev/null +++ b/dbms/src/Interpreters/OptimizeIfChains.cpp @@ -0,0 +1,92 @@ +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; + extern const int UNEXPECTED_AST_STRUCTURE; +} + +void OptimizeIfChainsVisitor::visit(ASTPtr & current_ast) +{ + if (!current_ast) + return; + + for (ASTPtr & child : current_ast->children) + { + /// Fallthrough cases + + const auto * function_node = child->as(); + if (!function_node || function_node->name != "if" || !function_node->arguments) + { + visit(child); + continue; + } + + const auto * function_args = function_node->arguments->as(); + if (!function_args || function_args->children.size() != 3 || !function_args->children[2]) + { + visit(child); + continue; + } + + const auto * else_arg = function_args->children[2]->as(); + if (!else_arg || else_arg->name != "if") + { + visit(child); + continue; + } + + /// The case of: + /// if(cond, a, if(...)) + + auto chain = ifChain(child); + std::reverse(chain.begin(), chain.end()); + child->as()->name = "multiIf"; + child->as()->arguments->children = std::move(chain); + } +} + +ASTs OptimizeIfChainsVisitor::ifChain(const ASTPtr & child) +{ + const auto * function_node = child->as(); + if (!function_node || !function_node->arguments) + throw Exception("Unexpected AST for function 'if'", ErrorCodes::UNEXPECTED_AST_STRUCTURE); + + const auto * function_args = function_node->arguments->as(); + + if (!function_args || function_args->children.size() != 3) + throw Exception("Wrong number of arguments for function 'if' (" + toString(function_args->children.size()) + " instead of 3)", + ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); + + const auto * else_arg = function_args->children[2]->as(); + + /// Recursively collect arguments from the innermost if ("head-resursion"). + /// Arguments will be returned in reverse order. + + if (else_arg && else_arg->name == "if") + { + auto cur = ifChain(function_node->arguments->children[2]); + cur.push_back(function_node->arguments->children[1]); + cur.push_back(function_node->arguments->children[0]); + return cur; + } + else + { + ASTs end; + end.reserve(3); + end.push_back(function_node->arguments->children[2]); + end.push_back(function_node->arguments->children[1]); + end.push_back(function_node->arguments->children[0]); + return end; + } +} + +} diff --git a/dbms/src/Interpreters/OptimizeIfChains.h b/dbms/src/Interpreters/OptimizeIfChains.h new file mode 100644 index 00000000000..5dbdb9bee50 --- /dev/null +++ b/dbms/src/Interpreters/OptimizeIfChains.h @@ -0,0 +1,19 @@ +#pragma once + +#include + +namespace DB +{ + +/// It converts if-chain to multiIf. +class OptimizeIfChainsVisitor +{ +public: + OptimizeIfChainsVisitor() = default; + void visit(ASTPtr & ast); + +private: + ASTs ifChain(const ASTPtr & child); +}; + +} diff --git a/dbms/src/Interpreters/PredicateExpressionsOptimizer.cpp b/dbms/src/Interpreters/PredicateExpressionsOptimizer.cpp index 050ee637d18..9927091874c 100644 --- a/dbms/src/Interpreters/PredicateExpressionsOptimizer.cpp +++ b/dbms/src/Interpreters/PredicateExpressionsOptimizer.cpp @@ -1,32 +1,13 @@ -#include - -#include -#include #include -#include -#include -#include + #include #include -#include #include -#include -#include #include -#include -#include -#include -#include #include -#include -#include -#include -#include -#include -#include -#include #include -#include +#include +#include namespace DB @@ -38,155 +19,105 @@ namespace ErrorCodes extern const int UNKNOWN_ELEMENT_IN_AST; } -namespace -{ - -constexpr auto and_function_name = "and"; - -String qualifiedName(ASTIdentifier * identifier, const String & prefix) -{ - if (identifier->isShort()) - return prefix + identifier->getAliasOrColumnName(); - return identifier->getAliasOrColumnName(); -} - -} - PredicateExpressionsOptimizer::PredicateExpressionsOptimizer( - ASTSelectQuery * ast_select_, ExtractedSettings && settings_, const Context & context_) - : ast_select(ast_select_), settings(settings_), context(context_) + const Context & context_, const TablesWithColumnNames & tables_with_columns_, const Settings & settings_) + : context(context_), tables_with_columns(tables_with_columns_), settings(settings_) { } -bool PredicateExpressionsOptimizer::optimize() +bool PredicateExpressionsOptimizer::optimize(ASTSelectQuery & select_query) { - if (!settings.enable_optimize_predicate_expression || !ast_select || !ast_select->tables() || ast_select->tables()->children.empty()) + if (!settings.enable_optimize_predicate_expression) return false; - if (!ast_select->where() && !ast_select->prewhere()) + if (select_query.having() && (!select_query.group_by_with_cube && !select_query.group_by_with_rollup && !select_query.group_by_with_totals)) + tryMovePredicatesFromHavingToWhere(select_query); + + if (!select_query.tables() || select_query.tables()->children.empty()) return false; - if (ast_select->array_join_expression_list()) + if ((!select_query.where() && !select_query.prewhere()) || select_query.array_join_expression_list()) return false; - SubqueriesProjectionColumns all_subquery_projection_columns = getAllSubqueryProjectionColumns(); + const auto & tables_predicates = extractTablesPredicates(select_query.where(), select_query.prewhere()); - bool is_rewrite_subqueries = false; - if (!all_subquery_projection_columns.empty()) - { - is_rewrite_subqueries |= optimizeImpl(ast_select->where(), all_subquery_projection_columns, OptimizeKind::PUSH_TO_WHERE); - is_rewrite_subqueries |= optimizeImpl(ast_select->prewhere(), all_subquery_projection_columns, OptimizeKind::PUSH_TO_PREWHERE); - } + if (!tables_predicates.empty()) + return tryRewritePredicatesToTables(select_query.refTables()->children, tables_predicates); - return is_rewrite_subqueries; + return false; } -bool PredicateExpressionsOptimizer::optimizeImpl( - const ASTPtr & outer_expression, const SubqueriesProjectionColumns & subqueries_projection_columns, OptimizeKind expression_kind) +static ASTs splitConjunctionPredicate(const std::initializer_list & predicates) { - /// split predicate with `and` - std::vector outer_predicate_expressions = splitConjunctionPredicate(outer_expression); + std::vector res; - std::vector table_expressions = getTableExpressions(*ast_select); - std::vector tables_with_columns = getDatabaseAndTablesWithColumnNames(table_expressions, context); - - bool is_rewrite_subquery = false; - for (auto & outer_predicate : outer_predicate_expressions) + auto remove_expression_at_index = [&res] (const size_t index) { - if (isArrayJoinFunction(outer_predicate)) + if (index < res.size() - 1) + std::swap(res[index], res.back()); + res.pop_back(); + }; + + for (const auto & predicate : predicates) + { + if (!predicate) continue; - auto outer_predicate_dependencies = getDependenciesAndQualifiers(outer_predicate, tables_with_columns); + res.emplace_back(predicate); - /// TODO: remove origin expression - for (const auto & [subquery, projection_columns] : subqueries_projection_columns) + for (size_t idx = 0; idx < res.size();) { - OptimizeKind optimize_kind = OptimizeKind::NONE; - if (allowPushDown(subquery, outer_predicate, projection_columns, outer_predicate_dependencies, optimize_kind)) + const auto & expression = res.at(idx); + + if (const auto * function = expression->as(); function && function->name == "and") { - if (optimize_kind == OptimizeKind::NONE) - optimize_kind = expression_kind; + for (auto & child : function->arguments->children) + res.emplace_back(child); - ASTPtr inner_predicate = outer_predicate->clone(); - cleanExpressionAlias(inner_predicate); /// clears the alias name contained in the outer predicate + remove_expression_at_index(idx); + continue; + } + ++idx; + } + } - std::vector inner_predicate_dependencies = - getDependenciesAndQualifiers(inner_predicate, tables_with_columns); + return res; +} - setNewAliasesForInnerPredicate(projection_columns, inner_predicate_dependencies); +std::vector PredicateExpressionsOptimizer::extractTablesPredicates(const ASTPtr & where, const ASTPtr & prewhere) +{ + std::vector tables_predicates(tables_with_columns.size()); - switch (optimize_kind) - { - case OptimizeKind::NONE: continue; - case OptimizeKind::PUSH_TO_WHERE: - is_rewrite_subquery |= optimizeExpression(inner_predicate, subquery, ASTSelectQuery::Expression::WHERE); - continue; - case OptimizeKind::PUSH_TO_HAVING: - is_rewrite_subquery |= optimizeExpression(inner_predicate, subquery, ASTSelectQuery::Expression::HAVING); - continue; - case OptimizeKind::PUSH_TO_PREWHERE: - is_rewrite_subquery |= optimizeExpression(inner_predicate, subquery, ASTSelectQuery::Expression::PREWHERE); - continue; - } + for (const auto & predicate_expression : splitConjunctionPredicate({where, prewhere})) + { + ExpressionInfoVisitor::Data expression_info{.context = context, .tables = tables_with_columns}; + ExpressionInfoVisitor(expression_info).visit(predicate_expression); + + if (expression_info.is_stateful_function) + return {}; /// give up the optimization when the predicate contains stateful function + + if (!expression_info.is_array_join) + { + if (expression_info.unique_reference_tables_pos.size() == 1) + tables_predicates[*expression_info.unique_reference_tables_pos.begin()].emplace_back(predicate_expression); + else if (expression_info.unique_reference_tables_pos.size() == 0) + { + for (size_t index = 0; index < tables_predicates.size(); ++index) + tables_predicates[index].emplace_back(predicate_expression); } } } - return is_rewrite_subquery; + + return tables_predicates; /// everything is OK, it can be optimized } -bool PredicateExpressionsOptimizer::allowPushDown( - const ASTSelectQuery * subquery, - const ASTPtr &, - const std::vector & projection_columns, - const std::vector & dependencies, - OptimizeKind & optimize_kind) +bool PredicateExpressionsOptimizer::tryRewritePredicatesToTables(ASTs & tables_element, const std::vector & tables_predicates) { - if (!subquery - || (!settings.enable_optimize_predicate_expression_to_final_subquery && subquery->final()) - || subquery->limitBy() || subquery->limitLength() - || subquery->with() || subquery->withFill()) - return false; - else + bool is_rewrite_tables = false; + + for (size_t index = tables_element.size(); index > 0; --index) { - ASTPtr expr_list = ast_select->select(); - ExtractFunctionVisitor::Data extract_data; - ExtractFunctionVisitor(extract_data).visit(expr_list); - - for (const auto & subquery_function : extract_data.functions) - { - const auto & function = FunctionFactory::instance().tryGet(subquery_function->name, context); - - /// Skip lambda, tuple and other special functions - if (function && function->isStateful()) - return false; - } - } - - const auto * ast_join = ast_select->join(); - const ASTTableExpression * left_table_expr = nullptr; - const ASTTableExpression * right_table_expr = nullptr; - const ASTSelectQuery * left_subquery = nullptr; - const ASTSelectQuery * right_subquery = nullptr; - - if (ast_join) - { - left_table_expr = ast_select - ->tables()->as() - ->children[0]->as() - ->table_expression->as(); - right_table_expr = ast_select - ->tables()->as() - ->children[1]->as() - ->table_expression->as(); - - if (left_table_expr && left_table_expr->subquery) - left_subquery = left_table_expr->subquery - ->children[0]->as() - ->list_of_selects->children[0]->as(); - if (right_table_expr && right_table_expr->subquery) - right_subquery = right_table_expr->subquery - ->children[0]->as() - ->list_of_selects->children[0]->as(); + size_t table_pos = index - 1; /// NOTE: the syntactic way of pushdown has limitations and should be partially disabled in case of JOINs. /// Let's take a look at the query: @@ -201,326 +132,84 @@ bool PredicateExpressionsOptimizer::allowPushDown( /// It happens because the not-matching columns are replaced with a global default values on JOIN. /// Same is true for RIGHT JOIN and FULL JOIN. - /// Check right side for LEFT'o'FULL JOIN - if (isLeftOrFull(ast_join->table_join->as()->kind) && right_subquery == subquery) - return false; - - /// Check left side for RIGHT'o'FULL JOIN - if (isRightOrFull(ast_join->table_join->as()->kind) && left_subquery == subquery) - return false; - } - - return checkDependencies(projection_columns, dependencies, optimize_kind); -} - -bool PredicateExpressionsOptimizer::checkDependencies( - const std::vector & projection_columns, - const std::vector & dependencies, - OptimizeKind & optimize_kind) -{ - for (const auto & [identifier, prefix] : dependencies) - { - bool is_found = false; - String qualified_name = qualifiedName(identifier, prefix); - - for (const auto & [ast, alias] : projection_columns) + if (const auto & table_element = tables_element[table_pos]->as()) { - if (alias == qualified_name) - { - is_found = true; - ASTPtr projection_column = ast; - ExtractFunctionVisitor::Data extract_data; - ExtractFunctionVisitor(extract_data).visit(projection_column); + if (table_element->table_join && isLeft(table_element->table_join->as()->kind)) + continue; /// Skip right table optimization - if (!extract_data.aggregate_functions.empty()) - optimize_kind = OptimizeKind::PUSH_TO_HAVING; - } - } + if (table_element->table_join && isFull(table_element->table_join->as()->kind)) + break; /// Skip left and right table optimization - if (!is_found) - return false; - } + is_rewrite_tables |= tryRewritePredicatesToTable(tables_element[table_pos], tables_predicates[table_pos], + tables_with_columns[table_pos].columns); - return true; -} - -std::vector PredicateExpressionsOptimizer::splitConjunctionPredicate(const ASTPtr & predicate_expression) -{ - std::vector predicate_expressions; - - if (predicate_expression) - { - predicate_expressions.emplace_back(predicate_expression); - - auto remove_expression_at_index = [&predicate_expressions] (const size_t index) - { - if (index < predicate_expressions.size() - 1) - std::swap(predicate_expressions[index], predicate_expressions.back()); - predicate_expressions.pop_back(); - }; - - for (size_t idx = 0; idx < predicate_expressions.size();) - { - const auto expression = predicate_expressions.at(idx); - - if (const auto * function = expression->as()) - { - if (function->name == and_function_name) - { - for (auto & child : function->arguments->children) - predicate_expressions.emplace_back(child); - - remove_expression_at_index(idx); - continue; - } - } - ++idx; + if (table_element->table_join && isRight(table_element->table_join->as()->kind)) + break; /// Skip left table optimization } } - return predicate_expressions; + + return is_rewrite_tables; } -std::vector -PredicateExpressionsOptimizer::getDependenciesAndQualifiers(ASTPtr & expression, std::vector & tables) +bool PredicateExpressionsOptimizer::tryRewritePredicatesToTable(ASTPtr & table_element, const ASTs & table_predicates, const Names & table_column) const { - FindIdentifierBestTableVisitor::Data find_data(tables); - FindIdentifierBestTableVisitor(find_data).visit(expression); - - std::vector dependencies; - - for (const auto & [identifier, table] : find_data.identifier_table) + if (!table_predicates.empty()) { - String table_alias; - if (table) - table_alias = table->getQualifiedNamePrefix(); + auto optimize_final = settings.enable_optimize_predicate_expression_to_final_subquery; + PredicateRewriteVisitor::Data data(context, table_predicates, table_column, optimize_final); - dependencies.emplace_back(identifier, table_alias); + PredicateRewriteVisitor(data).visit(table_element); + return data.is_rewrite; } - return dependencies; -} - -void PredicateExpressionsOptimizer::setNewAliasesForInnerPredicate( - const std::vector & projection_columns, - const std::vector & dependencies) -{ - for (auto & [identifier, prefix] : dependencies) - { - String qualified_name = qualifiedName(identifier, prefix); - - for (auto & [ast, alias] : projection_columns) - { - if (alias == qualified_name) - { - String name; - if (auto * id = ast->as()) - { - name = id->tryGetAlias(); - if (name.empty()) - name = id->shortName(); - } - else - { - if (ast->tryGetAlias().empty()) - ast->setAlias(ast->getColumnName()); - name = ast->getAliasOrColumnName(); - } - - identifier->setShortName(name); - } - } - } -} - -bool PredicateExpressionsOptimizer::isArrayJoinFunction(const ASTPtr & node) -{ - if (const auto * function = node->as()) - { - if (function->name == "arrayJoin") - return true; - } - - for (auto & child : node->children) - if (isArrayJoinFunction(child)) - return true; - return false; } -bool PredicateExpressionsOptimizer::optimizeExpression(const ASTPtr & outer_expression, ASTSelectQuery * subquery, - ASTSelectQuery::Expression expr) +bool PredicateExpressionsOptimizer::tryMovePredicatesFromHavingToWhere(ASTSelectQuery & select_query) { - ASTPtr subquery_expression = subquery->getExpression(expr, false); - subquery_expression = subquery_expression ? makeASTFunction(and_function_name, outer_expression, subquery_expression) : outer_expression; + ASTs where_predicates; + ASTs having_predicates; + + const auto & reduce_predicates = [&](const ASTs & predicates) + { + ASTPtr res = predicates[0]; + for (size_t index = 1; index < predicates.size(); ++index) + res = makeASTFunction("and", res, predicates[index]); + + return res; + }; + + for (const auto & moving_predicate: splitConjunctionPredicate({select_query.having()})) + { + ExpressionInfoVisitor::Data expression_info{.context = context, .tables = {}}; + ExpressionInfoVisitor(expression_info).visit(moving_predicate); + + /// TODO: If there is no group by, where, and prewhere expression, we can push down the stateful function + if (expression_info.is_stateful_function) + return false; + + if (expression_info.is_aggregate_function) + having_predicates.emplace_back(moving_predicate); + else + where_predicates.emplace_back(moving_predicate); + } + + if (having_predicates.empty()) + select_query.setExpression(ASTSelectQuery::Expression::HAVING, {}); + else + { + auto having_predicate = reduce_predicates(having_predicates); + select_query.setExpression(ASTSelectQuery::Expression::HAVING, std::move(having_predicate)); + } + + if (!where_predicates.empty()) + { + auto moved_predicate = reduce_predicates(where_predicates); + moved_predicate = select_query.where() ? makeASTFunction("and", select_query.where(), moved_predicate) : moved_predicate; + select_query.setExpression(ASTSelectQuery::Expression::WHERE, std::move(moved_predicate)); + } - subquery->setExpression(expr, std::move(subquery_expression)); return true; } -PredicateExpressionsOptimizer::SubqueriesProjectionColumns PredicateExpressionsOptimizer::getAllSubqueryProjectionColumns() -{ - SubqueriesProjectionColumns projection_columns; - - for (const auto & table_expression : getTableExpressions(*ast_select)) - if (table_expression->subquery) - getSubqueryProjectionColumns(table_expression->subquery, projection_columns); - - return projection_columns; -} - -void PredicateExpressionsOptimizer::getSubqueryProjectionColumns(const ASTPtr & subquery, SubqueriesProjectionColumns & projection_columns) -{ - String qualified_name_prefix = subquery->tryGetAlias(); - if (!qualified_name_prefix.empty()) - qualified_name_prefix += '.'; - - const ASTPtr & subselect = subquery->children[0]; - - ASTs select_with_union_projections; - const auto * select_with_union_query = subselect->as(); - - for (auto & select : select_with_union_query->list_of_selects->children) - { - std::vector subquery_projections; - auto select_projection_columns = getSelectQueryProjectionColumns(select); - - if (!select_projection_columns.empty()) - { - if (select_with_union_projections.empty()) - select_with_union_projections = select_projection_columns; - - for (size_t i = 0; i < select_projection_columns.size(); i++) - subquery_projections.emplace_back(std::pair(select_projection_columns[i], - qualified_name_prefix + select_with_union_projections[i]->getAliasOrColumnName())); - - projection_columns.insert(std::pair(select->as(), subquery_projections)); - } - } -} - -ASTs PredicateExpressionsOptimizer::getSelectQueryProjectionColumns(ASTPtr & ast) -{ - ASTs projection_columns; - auto * select_query = ast->as(); - - /// first should normalize query tree. - std::unordered_map aliases; - std::vector tables = getDatabaseAndTables(*select_query, context.getCurrentDatabase()); - - /// TODO: get tables from evaluateAsterisk instead of tablesOnly() to extract asterisks in general way - std::vector tables_with_columns = TranslateQualifiedNamesVisitor::Data::tablesOnly(tables); - TranslateQualifiedNamesVisitor::Data qn_visitor_data({}, std::move(tables_with_columns), false); - TranslateQualifiedNamesVisitor(qn_visitor_data).visit(ast); - - QueryAliasesVisitor::Data query_aliases_data{aliases}; - QueryAliasesVisitor(query_aliases_data).visit(ast); - - MarkTableIdentifiersVisitor::Data mark_tables_data{aliases}; - MarkTableIdentifiersVisitor(mark_tables_data).visit(ast); - - QueryNormalizer::Data normalizer_data(aliases, settings); - QueryNormalizer(normalizer_data).visit(ast); - - for (const auto & projection_column : select_query->select()->children) - { - if (projection_column->as() || projection_column->as() || projection_column->as()) - { - ASTs evaluated_columns = evaluateAsterisk(select_query, projection_column); - - for (const auto & column : evaluated_columns) - projection_columns.emplace_back(column); - - continue; - } - - projection_columns.emplace_back(projection_column); - } - return projection_columns; -} - -ASTs PredicateExpressionsOptimizer::evaluateAsterisk(ASTSelectQuery * select_query, const ASTPtr & asterisk) -{ - /// SELECT *, SELECT dummy, SELECT 1 AS id - if (!select_query->tables() || select_query->tables()->children.empty()) - return {}; - - std::vector tables_expression = getTableExpressions(*select_query); - - if (const auto * qualified_asterisk = asterisk->as()) - { - if (qualified_asterisk->children.size() != 1) - throw Exception("Logical error: qualified asterisk must have exactly one child", ErrorCodes::LOGICAL_ERROR); - - DatabaseAndTableWithAlias ident_db_and_name(qualified_asterisk->children[0]); - - for (auto it = tables_expression.begin(); it != tables_expression.end();) - { - const ASTTableExpression * table_expression = *it; - DatabaseAndTableWithAlias database_and_table_with_alias(*table_expression, context.getCurrentDatabase()); - - if (ident_db_and_name.satisfies(database_and_table_with_alias, true)) - ++it; - else - it = tables_expression.erase(it); /// It's not a required table - } - } - - ASTs projection_columns; - for (auto & table_expression : tables_expression) - { - if (table_expression->subquery) - { - const auto * subquery = table_expression->subquery->as(); - const auto * select_with_union_query = subquery->children[0]->as(); - const auto subquery_projections = getSelectQueryProjectionColumns(select_with_union_query->list_of_selects->children[0]); - projection_columns.insert(projection_columns.end(), subquery_projections.begin(), subquery_projections.end()); - } - else - { - StoragePtr storage; - - if (table_expression->table_function) - { - auto query_context = const_cast(&context.getQueryContext()); - storage = query_context->executeTableFunction(table_expression->table_function); - } - else if (table_expression->database_and_table_name) - { - const auto * database_and_table_ast = table_expression->database_and_table_name->as(); - DatabaseAndTableWithAlias database_and_table_name(*database_and_table_ast); - storage = context.getTable(database_and_table_name.database, database_and_table_name.table); - } - else - throw Exception("Logical error: unexpected table expression", ErrorCodes::LOGICAL_ERROR); - - const auto block = storage->getSampleBlock(); - if (const auto * asterisk_pattern = asterisk->as()) - { - for (size_t idx = 0; idx < block.columns(); ++idx) - { - auto & col = block.getByPosition(idx); - if (asterisk_pattern->isColumnMatching(col.name)) - projection_columns.emplace_back(std::make_shared(col.name)); - } - } - else - { - for (size_t idx = 0; idx < block.columns(); ++idx) - projection_columns.emplace_back(std::make_shared(block.getByPosition(idx).name)); - } - } - } - return projection_columns; -} - -void PredicateExpressionsOptimizer::cleanExpressionAlias(ASTPtr & expression) -{ - const auto my_alias = expression->tryGetAlias(); - if (!my_alias.empty()) - expression->setAlias(""); - - for (auto & child : expression->children) - cleanExpressionAlias(child); -} - } diff --git a/dbms/src/Interpreters/PredicateExpressionsOptimizer.h b/dbms/src/Interpreters/PredicateExpressionsOptimizer.h index ca2c8b8766d..da6b98987a6 100644 --- a/dbms/src/Interpreters/PredicateExpressionsOptimizer.h +++ b/dbms/src/Interpreters/PredicateExpressionsOptimizer.h @@ -1,110 +1,53 @@ #pragma once -#include "DatabaseAndTableWithAlias.h" #include -#include +#include namespace DB { -class ASTIdentifier; -class ASTSubquery; class Context; +struct Settings; -/** This class provides functions for Push-Down predicate expressions - * - * The Example: - * - Query before optimization : - * SELECT id_1, name_1 FROM (SELECT id_1, name_1 FROM table_a UNION ALL SELECT id_2, name_2 FROM table_b) - * WHERE id_1 = 1 - * - Query after optimization : - * SELECT id_1, name_1 FROM (SELECT id_1, name_1 FROM table_a WHERE id_1 = 1 UNION ALL SELECT id_2, name_2 FROM table_b WHERE id_2 = 1) - * WHERE id_1 = 1 +/** Predicate optimization based on rewriting ast rules * For more details : https://github.com/ClickHouse/ClickHouse/pull/2015#issuecomment-374283452 + * The optimizer does two different optimizations + * - Move predicates from having to where + * - Push the predicate down from the current query to the having of the subquery */ class PredicateExpressionsOptimizer { - using ProjectionWithAlias = std::pair; - using SubqueriesProjectionColumns = std::map>; - using IdentifierWithQualifier = std::pair; +public: + PredicateExpressionsOptimizer(const Context & context_, const TablesWithColumnNames & tables_with_columns_, const Settings & settings_); + bool optimize(ASTSelectQuery & select_query); + +private: /// Extracts settings, mostly to show which are used and which are not. struct ExtractedSettings { - /// QueryNormalizer settings - const UInt64 max_ast_depth; - const UInt64 max_expanded_ast_elements; - const String count_distinct_implementation; - - /// for PredicateExpressionsOptimizer const bool enable_optimize_predicate_expression; const bool enable_optimize_predicate_expression_to_final_subquery; - const bool join_use_nulls; template ExtractedSettings(const T & settings_) - : max_ast_depth(settings_.max_ast_depth), - max_expanded_ast_elements(settings_.max_expanded_ast_elements), - count_distinct_implementation(settings_.count_distinct_implementation), - enable_optimize_predicate_expression(settings_.enable_optimize_predicate_expression), - enable_optimize_predicate_expression_to_final_subquery(settings_.enable_optimize_predicate_expression_to_final_subquery), - join_use_nulls(settings_.join_use_nulls) + : enable_optimize_predicate_expression(settings_.enable_optimize_predicate_expression), + enable_optimize_predicate_expression_to_final_subquery(settings_.enable_optimize_predicate_expression_to_final_subquery) {} }; -public: - PredicateExpressionsOptimizer(ASTSelectQuery * ast_select_, ExtractedSettings && settings_, const Context & context_); - - bool optimize(); - -private: - ASTSelectQuery * ast_select; - const ExtractedSettings settings; const Context & context; + const std::vector & tables_with_columns; - enum OptimizeKind - { - NONE, - PUSH_TO_PREWHERE, - PUSH_TO_WHERE, - PUSH_TO_HAVING, - }; + const ExtractedSettings settings; - bool isArrayJoinFunction(const ASTPtr & node); + std::vector extractTablesPredicates(const ASTPtr & where, const ASTPtr & prewhere); - std::vector splitConjunctionPredicate(const ASTPtr & predicate_expression); + bool tryRewritePredicatesToTables(ASTs & tables_element, const std::vector & tables_predicates); - std::vector getDependenciesAndQualifiers(ASTPtr & expression, - std::vector & tables_with_aliases); + bool tryRewritePredicatesToTable(ASTPtr & table_element, const ASTs & table_predicates, const Names & table_column) const; - bool optimizeExpression(const ASTPtr & outer_expression, ASTSelectQuery * subquery, ASTSelectQuery::Expression expr); - - bool optimizeImpl(const ASTPtr & outer_expression, const SubqueriesProjectionColumns & subqueries_projection_columns, OptimizeKind optimize_kind); - - bool allowPushDown( - const ASTSelectQuery * subquery, - const ASTPtr & outer_predicate, - const std::vector & subquery_projection_columns, - const std::vector & outer_predicate_dependencies, - OptimizeKind & optimize_kind); - - bool checkDependencies( - const std::vector & projection_columns, - const std::vector & dependencies, - OptimizeKind & optimize_kind); - - void setNewAliasesForInnerPredicate(const std::vector & projection_columns, - const std::vector & inner_predicate_dependencies); - - SubqueriesProjectionColumns getAllSubqueryProjectionColumns(); - - void getSubqueryProjectionColumns(const ASTPtr & subquery, SubqueriesProjectionColumns & all_subquery_projection_columns); - - ASTs getSelectQueryProjectionColumns(ASTPtr & ast); - - ASTs evaluateAsterisk(ASTSelectQuery * select_query, const ASTPtr & asterisk); - - void cleanExpressionAlias(ASTPtr & expression); + bool tryMovePredicatesFromHavingToWhere(ASTSelectQuery & select_query); }; } diff --git a/dbms/src/Interpreters/PredicateRewriteVisitor.cpp b/dbms/src/Interpreters/PredicateRewriteVisitor.cpp new file mode 100644 index 00000000000..6bd16ddc066 --- /dev/null +++ b/dbms/src/Interpreters/PredicateRewriteVisitor.cpp @@ -0,0 +1,119 @@ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +namespace DB +{ + +PredicateRewriteVisitorData::PredicateRewriteVisitorData( + const Context & context_, const ASTs & predicates_, const Names & column_names_, bool optimize_final_) + : context(context_), predicates(predicates_), column_names(column_names_), optimize_final(optimize_final_) +{ +} + +void PredicateRewriteVisitorData::visit(ASTSelectWithUnionQuery & union_select_query, ASTPtr &) +{ + auto & internal_select_list = union_select_query.list_of_selects->children; + + if (internal_select_list.size() > 0) + visitFirstInternalSelect(*internal_select_list[0]->as(), internal_select_list[0]); + + for (size_t index = 1; index < internal_select_list.size(); ++index) + visitOtherInternalSelect(*internal_select_list[index]->as(), internal_select_list[index]); +} + +void PredicateRewriteVisitorData::visitFirstInternalSelect(ASTSelectQuery & select_query, ASTPtr &) +{ + is_rewrite |= rewriteSubquery(select_query, column_names, column_names); +} + +void PredicateRewriteVisitorData::visitOtherInternalSelect(ASTSelectQuery & select_query, ASTPtr &) +{ + /// For non first select, its alias has no more significance, so we can set a temporary alias for them + ASTPtr temp_internal_select = select_query.clone(); + ASTSelectQuery * temp_select_query = temp_internal_select->as(); + + size_t alias_index = 0; + for (auto & ref_select : temp_select_query->refSelect()->children) + { + if (!ref_select->as() && !ref_select->as() && !ref_select->as() && + !ref_select->as()) + { + if (const auto & alias = ref_select->tryGetAlias(); alias.empty()) + ref_select->setAlias("--predicate_optimizer_" + toString(alias_index++)); + } + } + + const Names & internal_columns = InterpreterSelectQuery( + temp_internal_select, context, SelectQueryOptions().analyze()).getSampleBlock().getNames(); + + if (rewriteSubquery(*temp_select_query, column_names, internal_columns)) + { + is_rewrite |= true; + select_query.setExpression(ASTSelectQuery::Expression::SELECT, std::move(temp_select_query->refSelect())); + select_query.setExpression(ASTSelectQuery::Expression::HAVING, std::move(temp_select_query->refHaving())); + } +} + +static void cleanAliasAndCollectIdentifiers(ASTPtr & predicate, std::vector & identifiers) +{ + /// Skip WHERE x in (SELECT ...) + if (!predicate->as()) + { + for (auto & children : predicate->children) + cleanAliasAndCollectIdentifiers(children, identifiers); + } + + if (const auto alias = predicate->tryGetAlias(); !alias.empty()) + predicate->setAlias(""); + + if (ASTIdentifier * identifier = predicate->as()) + identifiers.emplace_back(identifier); +} + +bool PredicateRewriteVisitorData::rewriteSubquery(ASTSelectQuery & subquery, const Names & outer_columns, const Names & inner_columns) +{ + if ((!optimize_final && subquery.final()) + || subquery.with() || subquery.withFill() + || subquery.limitBy() || subquery.limitLength() + || hasStatefulFunction(subquery.select(), context)) + return false; + + for (const auto & predicate : predicates) + { + std::vector identifiers; + ASTPtr optimize_predicate = predicate->clone(); + cleanAliasAndCollectIdentifiers(optimize_predicate, identifiers); + + for (size_t index = 0; index < identifiers.size(); ++index) + { + const auto & column_name = identifiers[index]->shortName(); + const auto & outer_column_iterator = std::find(outer_columns.begin(), outer_columns.end(), column_name); + + /// For lambda functions, we can't always find them in the list of columns + /// For example: SELECT * FROM system.one WHERE arrayMap(x -> x, [dummy]) = [0] + if (outer_column_iterator != outer_columns.end()) + identifiers[index]->setShortName(inner_columns[outer_column_iterator - outer_columns.begin()]); + } + + /// We only need to push all the predicates to subquery having + /// The subquery optimizer will move the appropriate predicates from having to where + subquery.setExpression(ASTSelectQuery::Expression::HAVING, + subquery.having() ? makeASTFunction("and", optimize_predicate, subquery.having()) : optimize_predicate); + } + + return true; +} + +} diff --git a/dbms/src/Interpreters/PredicateRewriteVisitor.h b/dbms/src/Interpreters/PredicateRewriteVisitor.h new file mode 100644 index 00000000000..e07df922c15 --- /dev/null +++ b/dbms/src/Interpreters/PredicateRewriteVisitor.h @@ -0,0 +1,36 @@ +#pragma once + +#include +#include +#include +#include + +namespace DB +{ + +class PredicateRewriteVisitorData +{ +public: + bool is_rewrite = false; + using TypeToVisit = ASTSelectWithUnionQuery; + + void visit(ASTSelectWithUnionQuery & union_select_query, ASTPtr &); + + PredicateRewriteVisitorData(const Context & context_, const ASTs & predicates_, const Names & column_names_, bool optimize_final_); + +private: + const Context & context; + const ASTs & predicates; + const Names & column_names; + bool optimize_final; + + void visitFirstInternalSelect(ASTSelectQuery & select_query, ASTPtr &); + + void visitOtherInternalSelect(ASTSelectQuery & select_query, ASTPtr &); + + bool rewriteSubquery(ASTSelectQuery & subquery, const Names & outer_columns, const Names & inner_columns); +}; + +using PredicateRewriteMatcher = OneTypeMatcher; +using PredicateRewriteVisitor = InDepthNodeVisitor; +} diff --git a/dbms/src/Interpreters/ReplaceQueryParameterVisitor.cpp b/dbms/src/Interpreters/ReplaceQueryParameterVisitor.cpp index 1cbcb758bf3..5c29c722f88 100644 --- a/dbms/src/Interpreters/ReplaceQueryParameterVisitor.cpp +++ b/dbms/src/Interpreters/ReplaceQueryParameterVisitor.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -54,10 +55,12 @@ void ReplaceQueryParameterVisitor::visitQueryParameter(ASTPtr & ast) IColumn & temp_column = *temp_column_ptr; ReadBufferFromString read_buffer{value}; FormatSettings format_settings; - data_type->deserializeAsWholeText(temp_column, read_buffer, format_settings); + data_type->deserializeAsTextEscaped(temp_column, read_buffer, format_settings); if (!read_buffer.eof()) - throw Exception("Value " + value + " cannot be parsed as " + type_name + " for query parameter '" + ast_param.name + "'", ErrorCodes::BAD_QUERY_PARAMETER); + throw Exception("Value " + value + " cannot be parsed as " + type_name + " for query parameter '" + ast_param.name + "'" + " because it isn't parsed completely: only " + toString(read_buffer.count()) + " of " + toString(value.size()) + " bytes was parsed: " + + value.substr(0, read_buffer.count()), ErrorCodes::BAD_QUERY_PARAMETER); ast = addTypeConversionToAST(std::make_shared(temp_column[0]), type_name); } diff --git a/dbms/src/Interpreters/SyntaxAnalyzer.cpp b/dbms/src/Interpreters/SyntaxAnalyzer.cpp index a26d8b8253a..3680947c8f1 100644 --- a/dbms/src/Interpreters/SyntaxAnalyzer.cpp +++ b/dbms/src/Interpreters/SyntaxAnalyzer.cpp @@ -21,6 +21,7 @@ #include #include /// getSmallestColumn() #include +#include #include #include @@ -914,8 +915,14 @@ SyntaxAnalyzerResultPtr SyntaxAnalyzer::analyze( /// Optimize if with constant condition after constants was substituted instead of scalar subqueries. OptimizeIfWithConstantConditionVisitor(result.aliases).visit(query); + if (settings.optimize_if_chain_to_miltiif) + OptimizeIfChainsVisitor().visit(query); + if (select_query) { + /// Push the predicate expression down to the subqueries. + result.rewrite_subqueries = PredicateExpressionsOptimizer(context, tables_with_columns, settings).optimize(*select_query); + /// GROUP BY injective function elimination. optimizeGroupBy(select_query, source_columns_set, context); @@ -931,9 +938,6 @@ SyntaxAnalyzerResultPtr SyntaxAnalyzer::analyze( /// array_join_alias_to_name, array_join_result_to_source. getArrayJoinedColumns(query, result, select_query, result.source_columns, source_columns_set); - /// Push the predicate expression down to the subqueries. - result.rewrite_subqueries = PredicateExpressionsOptimizer(select_query, settings, context).optimize(); - setJoinStrictness(*select_query, settings.join_default_strictness, settings.any_join_distinct_right_table_keys, result.analyzed_join->table_join); collectJoinedColumns(*result.analyzed_join, *select_query, tables_with_columns, result.aliases); diff --git a/dbms/src/Interpreters/executeQuery.cpp b/dbms/src/Interpreters/executeQuery.cpp index 2c6bf087f8d..77b0256d121 100644 --- a/dbms/src/Interpreters/executeQuery.cpp +++ b/dbms/src/Interpreters/executeQuery.cpp @@ -129,9 +129,9 @@ static void setExceptionStackTrace(QueryLogElement & elem) { throw; } - catch (const Exception & e) + catch (const std::exception & e) { - elem.stack_trace = e.getStackTrace().toString(); + elem.stack_trace = getExceptionStackTraceString(e); } catch (...) {} } diff --git a/dbms/src/Interpreters/tests/create_query.cpp b/dbms/src/Interpreters/tests/create_query.cpp index fc487f4b7bb..e159afced58 100644 --- a/dbms/src/Interpreters/tests/create_query.cpp +++ b/dbms/src/Interpreters/tests/create_query.cpp @@ -97,6 +97,6 @@ catch (const Exception & e) std::cerr << e.what() << ", " << e.displayText() << std::endl << std::endl << "Stack trace:" << std::endl - << e.getStackTrace().toString(); + << e.getStackTraceString(); return 1; } diff --git a/dbms/src/Interpreters/tests/select_query.cpp b/dbms/src/Interpreters/tests/select_query.cpp index 54613fffd8e..197d1c55faf 100644 --- a/dbms/src/Interpreters/tests/select_query.cpp +++ b/dbms/src/Interpreters/tests/select_query.cpp @@ -55,6 +55,6 @@ catch (const Exception & e) std::cerr << e.what() << ", " << e.displayText() << std::endl << std::endl << "Stack trace:" << std::endl - << e.getStackTrace().toString(); + << e.getStackTraceString(); return 1; } diff --git a/dbms/src/Processors/DelayedPortsProcessor.cpp b/dbms/src/Processors/DelayedPortsProcessor.cpp new file mode 100644 index 00000000000..672f2645c16 --- /dev/null +++ b/dbms/src/Processors/DelayedPortsProcessor.cpp @@ -0,0 +1,95 @@ +#include + +namespace DB +{ + +DelayedPortsProcessor::DelayedPortsProcessor(const Block & header, size_t num_ports, const PortNumbers & delayed_ports) + : IProcessor(InputPorts(num_ports, header), OutputPorts(num_ports, header)) + , num_delayed(delayed_ports.size()) +{ + port_pairs.resize(num_ports); + + auto input_it = inputs.begin(); + auto output_it = outputs.begin(); + for (size_t i = 0; i < num_ports; ++i) + { + port_pairs[i].input_port = &*input_it; + port_pairs[i].output_port = &*output_it; + ++input_it; + ++output_it; + } + + for (auto & delayed : delayed_ports) + port_pairs[delayed].is_delayed = true; +} + +bool DelayedPortsProcessor::processPair(PortsPair & pair) +{ + auto finish = [&]() + { + if (!pair.is_finished) + { + pair.is_finished = true; + ++num_finished; + } + }; + + if (pair.output_port->isFinished()) + { + pair.input_port->close(); + finish(); + return false; + } + + if (pair.input_port->isFinished()) + { + pair.output_port->finish(); + finish(); + return false; + } + + if (!pair.output_port->canPush()) + return false; + + pair.input_port->setNeeded(); + if (pair.input_port->hasData()) + pair.output_port->pushData(pair.input_port->pullData()); + + return true; +} + +IProcessor::Status DelayedPortsProcessor::prepare(const PortNumbers & updated_inputs, const PortNumbers & updated_outputs) +{ + bool skip_delayed = (num_finished + num_delayed) < port_pairs.size(); + bool need_data = false; + + for (auto & output_number : updated_outputs) + { + if (!skip_delayed || !port_pairs[output_number].is_delayed) + need_data = processPair(port_pairs[output_number]) || need_data; + } + + for (auto & input_number : updated_inputs) + { + if (!skip_delayed || !port_pairs[input_number].is_delayed) + need_data = processPair(port_pairs[input_number]) || need_data; + } + + /// In case if main streams are finished at current iteration, start processing delayed streams. + if (skip_delayed && (num_finished + num_delayed) >= port_pairs.size()) + { + for (auto & pair : port_pairs) + if (pair.is_delayed) + need_data = processPair(pair) || need_data; + } + + if (num_finished == port_pairs.size()) + return Status::Finished; + + if (need_data) + return Status::NeedData; + + return Status::PortFull; +} + +} diff --git a/dbms/src/Processors/DelayedPortsProcessor.h b/dbms/src/Processors/DelayedPortsProcessor.h new file mode 100644 index 00000000000..44dd632f8a8 --- /dev/null +++ b/dbms/src/Processors/DelayedPortsProcessor.h @@ -0,0 +1,37 @@ +#pragma once +#include + +namespace DB +{ + +/// Processor with N inputs and N outputs. Only moves data from i-th input to i-th output as is. +/// Some ports are delayed. Delayed ports are processed after other outputs are all finished. +/// Data between ports is not mixed. It is important because this processor can be used before MergingSortedTransform. +/// Delayed ports are appeared after joins, when some non-matched data need to be processed at the end. +class DelayedPortsProcessor : public IProcessor +{ +public: + DelayedPortsProcessor(const Block & header, size_t num_ports, const PortNumbers & delayed_ports); + + String getName() const override { return "DelayedPorts"; } + + Status prepare(const PortNumbers &, const PortNumbers &) override; + +private: + + struct PortsPair + { + InputPort * input_port = nullptr; + OutputPort * output_port = nullptr; + bool is_delayed = false; + bool is_finished = false; + }; + + std::vector port_pairs; + size_t num_delayed; + size_t num_finished = 0; + + bool processPair(PortsPair & pair); +}; + +} diff --git a/dbms/src/Processors/Executors/PipelineExecutor.cpp b/dbms/src/Processors/Executors/PipelineExecutor.cpp index 6addec11975..70cd2e2405f 100644 --- a/dbms/src/Processors/Executors/PipelineExecutor.cpp +++ b/dbms/src/Processors/Executors/PipelineExecutor.cpp @@ -64,13 +64,6 @@ bool PipelineExecutor::addEdges(UInt64 node) throwUnknownProcessor(to_proc, cur, true); UInt64 proc_num = it->second; - - for (auto & edge : edges) - { - if (edge.to == proc_num) - throw Exception("Multiple edges are not allowed for the same processors.", ErrorCodes::LOGICAL_ERROR); - } - auto & edge = edges.emplace_back(proc_num, is_backward, input_port_number, output_port_number, update_list); from_port.setUpdateInfo(&edge.update_info); @@ -177,10 +170,20 @@ void PipelineExecutor::addJob(ExecutionState * execution_state) execution_state->job = std::move(job); } -void PipelineExecutor::expandPipeline(Stack & stack, UInt64 pid) +bool PipelineExecutor::expandPipeline(Stack & stack, UInt64 pid) { auto & cur_node = graph[pid]; - auto new_processors = cur_node.processor->expandPipeline(); + Processors new_processors; + + try + { + new_processors = cur_node.processor->expandPipeline(); + } + catch (...) + { + cur_node.execution_state->exception = std::current_exception(); + return false; + } for (const auto & processor : new_processors) { @@ -220,20 +223,22 @@ void PipelineExecutor::expandPipeline(Stack & stack, UInt64 pid) } } } + + return true; } -bool PipelineExecutor::tryAddProcessorToStackIfUpdated(Edge & edge, Stack & stack) +bool PipelineExecutor::tryAddProcessorToStackIfUpdated(Edge & edge, Queue & queue, size_t thread_number) { /// In this method we have ownership on edge, but node can be concurrently accessed. auto & node = graph[edge.to]; - std::lock_guard guard(node.status_mutex); + std::unique_lock lock(node.status_mutex); ExecStatus status = node.status; if (status == ExecStatus::Finished) - return false; + return true; if (edge.backward) node.updated_output_ports.push_back(edge.output_port_number); @@ -243,14 +248,13 @@ bool PipelineExecutor::tryAddProcessorToStackIfUpdated(Edge & edge, Stack & stac if (status == ExecStatus::Idle) { node.status = ExecStatus::Preparing; - stack.push(edge.to); - return true; + return prepareProcessor(edge.to, thread_number, queue, std::move(lock)); } - return false; + return true; } -bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & parents, size_t thread_number, bool async) +bool PipelineExecutor::prepareProcessor(UInt64 pid, size_t thread_number, Queue & queue, std::unique_lock node_lock) { /// In this method we have ownership on node. auto & node = graph[pid]; @@ -264,14 +268,22 @@ bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & pa { /// Stopwatch watch; - std::lock_guard guard(node.status_mutex); + std::unique_lock lock(std::move(node_lock)); - auto status = node.processor->prepare(node.updated_input_ports, node.updated_output_ports); - node.updated_input_ports.clear(); - node.updated_output_ports.clear(); + try + { + node.last_processor_status = node.processor->prepare(node.updated_input_ports, node.updated_output_ports); + } + catch (...) + { + node.execution_state->exception = std::current_exception(); + return false; + } /// node.execution_state->preparation_time_ns += watch.elapsed(); - node.last_processor_status = status; + + node.updated_input_ports.clear(); + node.updated_output_ports.clear(); switch (node.last_processor_status) { @@ -291,7 +303,8 @@ bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & pa case IProcessor::Status::Ready: { node.status = ExecStatus::Executing; - return true; + queue.push(node.execution_state.get()); + break; } case IProcessor::Status::Async: { @@ -303,9 +316,7 @@ bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & pa } case IProcessor::Status::Wait: { - if (!async) - throw Exception("Processor returned status Wait before Async.", ErrorCodes::LOGICAL_ERROR); - break; + throw Exception("Wait is temporary not supported.", ErrorCodes::LOGICAL_ERROR); } case IProcessor::Status::ExpandPipeline: { @@ -337,18 +348,26 @@ bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & pa if (need_traverse) { - for (auto & edge : updated_back_edges) - tryAddProcessorToStackIfUpdated(*edge, parents); - for (auto & edge : updated_direct_edges) - tryAddProcessorToStackIfUpdated(*edge, children); + { + if (!tryAddProcessorToStackIfUpdated(*edge, queue, thread_number)) + return false; + } + + for (auto & edge : updated_back_edges) + { + if (!tryAddProcessorToStackIfUpdated(*edge, queue, thread_number)) + return false; + } } if (need_expand_pipeline) { + Stack stack; + executor_contexts[thread_number]->task_list.emplace_back( node.execution_state.get(), - &parents + &stack ); ExpandPipelineTask * desired = &executor_contexts[thread_number]->task_list.back(); @@ -356,20 +375,32 @@ bool PipelineExecutor::prepareProcessor(UInt64 pid, Stack & children, Stack & pa while (!expand_pipeline_task.compare_exchange_strong(expected, desired)) { - doExpandPipeline(expected, true); + if (!doExpandPipeline(expected, true)) + return false; + expected = nullptr; } - doExpandPipeline(desired, true); + if (!doExpandPipeline(desired, true)) + return false; /// Add itself back to be prepared again. - children.push(pid); + stack.push(pid); + + while (!stack.empty()) + { + auto item = stack.top(); + if (!prepareProcessor(item, thread_number, queue, std::unique_lock(graph[item].status_mutex))) + return false; + + stack.pop(); + } } - return false; + return true; } -void PipelineExecutor::doExpandPipeline(ExpandPipelineTask * task, bool processing) +bool PipelineExecutor::doExpandPipeline(ExpandPipelineTask * task, bool processing) { std::unique_lock lock(task->mutex); @@ -381,16 +412,20 @@ void PipelineExecutor::doExpandPipeline(ExpandPipelineTask * task, bool processi return task->num_waiting_processing_threads >= num_processing_executors || expand_pipeline_task != task; }); + bool result = true; + /// After condvar.wait() task may point to trash. Can change it only if it is still in expand_pipeline_task. if (expand_pipeline_task == task) { - expandPipeline(*task->stack, task->node_to_expand->processors_id); + result = expandPipeline(*task->stack, task->node_to_expand->processors_id); expand_pipeline_task = nullptr; lock.unlock(); task->condvar.notify_all(); } + + return result; } void PipelineExecutor::cancel() @@ -459,49 +494,31 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads #if !defined(__APPLE__) && !defined(__FreeBSD__) /// Specify CPU core for thread if can. /// It may reduce the number of context swithches. - cpu_set_t cpu_set; - CPU_ZERO(&cpu_set); - CPU_SET(thread_num, &cpu_set); - if (sched_setaffinity(0, sizeof(cpu_set_t), &cpu_set) == -1) - LOG_TRACE(log, "Cannot set affinity for thread " << num_threads); + /* + if (num_threads > 1) + { + cpu_set_t cpu_set; + CPU_ZERO(&cpu_set); + CPU_SET(thread_num, &cpu_set); + if (sched_setaffinity(0, sizeof(cpu_set_t), &cpu_set) == -1) + LOG_TRACE(log, "Cannot set affinity for thread " << num_threads); + } + */ #endif - UInt64 total_time_ns = 0; - UInt64 execution_time_ns = 0; - UInt64 processing_time_ns = 0; - UInt64 wait_time_ns = 0; +// UInt64 total_time_ns = 0; +// UInt64 execution_time_ns = 0; +// UInt64 processing_time_ns = 0; +// UInt64 wait_time_ns = 0; - Stopwatch total_time_watch; +// Stopwatch total_time_watch; ExecutionState * state = nullptr; - auto prepare_processor = [&](UInt64 pid, Stack & children, Stack & parents) + auto prepare_processor = [&](UInt64 pid, Queue & queue) { - try - { - return prepareProcessor(pid, children, parents, thread_num, false); - } - catch (...) - { - graph[pid].execution_state->exception = std::current_exception(); + if (!prepareProcessor(pid, thread_num, queue, std::unique_lock(graph[pid].status_mutex))) finish(); - } - - return false; - }; - - using Queue = std::queue; - - auto prepare_all_processors = [&](Queue & queue, Stack & stack, Stack & children, Stack & parents) - { - while (!stack.empty() && !finished) - { - auto current_processor = stack.top(); - stack.pop(); - - if (prepare_processor(current_processor, children, parents)) - queue.push(graph[current_processor].execution_state.get()); - } }; auto wake_up_executor = [&](size_t executor) @@ -511,63 +528,6 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads executor_contexts[executor]->condvar.notify_one(); }; - auto process_pinned_tasks = [&](Queue & queue) - { - Queue tmp_queue; - - struct PinnedTask - { - ExecutionState * task; - size_t thread_num; - }; - - std::stack pinned_tasks; - - while (!queue.empty()) - { - auto task = queue.front(); - queue.pop(); - - auto stream = task->processor->getStream(); - if (stream != IProcessor::NO_STREAM) - pinned_tasks.push({.task = task, .thread_num = stream % num_threads}); - else - tmp_queue.push(task); - } - - if (!pinned_tasks.empty()) - { - std::stack threads_to_wake; - - { - std::lock_guard lock(task_queue_mutex); - - while (!pinned_tasks.empty()) - { - auto & pinned_task = pinned_tasks.top(); - auto thread = pinned_task.thread_num; - - executor_contexts[thread]->pinned_tasks.push(pinned_task.task); - pinned_tasks.pop(); - - if (threads_queue.has(thread)) - { - threads_queue.pop(thread); - threads_to_wake.push(thread); - } - } - } - - while (!threads_to_wake.empty()) - { - wake_up_executor(threads_to_wake.top()); - threads_to_wake.pop(); - } - } - - queue.swap(tmp_queue); - }; - while (!finished) { /// First, find any processor to execute. @@ -577,20 +537,11 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads { std::unique_lock lock(task_queue_mutex); - if (!executor_contexts[thread_num]->pinned_tasks.empty()) - { - state = executor_contexts[thread_num]->pinned_tasks.front(); - executor_contexts[thread_num]->pinned_tasks.pop(); - - break; - } - if (!task_queue.empty()) { - state = task_queue.front(); - task_queue.pop(); + state = task_queue.pop(thread_num); - if (!task_queue.empty() && !threads_queue.empty()) + if (!task_queue.empty() && !threads_queue.empty() /*&& task_queue.quota() > threads_queue.size()*/) { auto thread_to_wake = threads_queue.pop_any(); lock.unlock(); @@ -648,8 +599,6 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads /// Try to execute neighbour processor. { - Stack children; - Stack parents; Queue queue; ++num_processing_executors; @@ -657,36 +606,16 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads doExpandPipeline(task, true); /// Execute again if can. - if (!prepare_processor(state->processors_id, children, parents)) - state = nullptr; - - /// Process all neighbours. Children will be on the top of stack, then parents. - prepare_all_processors(queue, children, children, parents); - process_pinned_tasks(queue); + prepare_processor(state->processors_id, queue); + state = nullptr; /// Take local task from queue if has one. - if (!state && !queue.empty()) + if (!queue.empty()) { state = queue.front(); queue.pop(); } - prepare_all_processors(queue, parents, parents, parents); - process_pinned_tasks(queue); - - /// Take pinned task if has one. - { - std::lock_guard guard(task_queue_mutex); - if (!executor_contexts[thread_num]->pinned_tasks.empty()) - { - if (state) - queue.push(state); - - state = executor_contexts[thread_num]->pinned_tasks.front(); - executor_contexts[thread_num]->pinned_tasks.pop(); - } - } - /// Push other tasks to global queue. if (!queue.empty()) { @@ -694,14 +623,15 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads while (!queue.empty() && !finished) { - task_queue.push(queue.front()); + task_queue.push(queue.front(), thread_num); queue.pop(); } - if (!threads_queue.empty()) + if (!threads_queue.empty() /* && task_queue.quota() > threads_queue.size()*/) { auto thread_to_wake = threads_queue.pop_any(); lock.unlock(); + wake_up_executor(thread_to_wake); } } @@ -715,14 +645,15 @@ void PipelineExecutor::executeSingleThread(size_t thread_num, size_t num_threads } } - total_time_ns = total_time_watch.elapsed(); - wait_time_ns = total_time_ns - execution_time_ns - processing_time_ns; - +// total_time_ns = total_time_watch.elapsed(); +// wait_time_ns = total_time_ns - execution_time_ns - processing_time_ns; +/* LOG_TRACE(log, "Thread finished." << " Total time: " << (total_time_ns / 1e9) << " sec." << " Execution time: " << (execution_time_ns / 1e9) << " sec." << " Processing time: " << (processing_time_ns / 1e9) << " sec." << " Wait time: " << (wait_time_ns / 1e9) << "sec."); +*/ } void PipelineExecutor::executeImpl(size_t num_threads) @@ -730,6 +661,7 @@ void PipelineExecutor::executeImpl(size_t num_threads) Stack stack; threads_queue.init(num_threads); + task_queue.init(num_threads); { std::lock_guard guard(executor_contexts_mutex); @@ -763,42 +695,57 @@ void PipelineExecutor::executeImpl(size_t num_threads) { std::lock_guard lock(task_queue_mutex); + Queue queue; + size_t next_thread = 0; + while (!stack.empty()) { UInt64 proc = stack.top(); stack.pop(); - if (prepareProcessor(proc, stack, stack, 0, false)) + prepareProcessor(proc, 0, queue, std::unique_lock(graph[proc].status_mutex)); + + while (!queue.empty()) { - auto cur_state = graph[proc].execution_state.get(); - task_queue.push(cur_state); + task_queue.push(queue.front(), next_thread); + queue.pop(); + + ++next_thread; + if (next_thread >= num_threads) + next_thread = 0; } } } - for (size_t i = 0; i < num_threads; ++i) + if (num_threads > 1) { - threads.emplace_back([this, thread_group, thread_num = i, num_threads] + + for (size_t i = 0; i < num_threads; ++i) { - /// ThreadStatus thread_status; + threads.emplace_back([this, thread_group, thread_num = i, num_threads] + { + /// ThreadStatus thread_status; - setThreadName("QueryPipelineEx"); + setThreadName("QueryPipelineEx"); - if (thread_group) - CurrentThread::attachTo(thread_group); + if (thread_group) + CurrentThread::attachTo(thread_group); - SCOPE_EXIT( - if (thread_group) - CurrentThread::detachQueryIfNotDetached(); - ); + SCOPE_EXIT( + if (thread_group) + CurrentThread::detachQueryIfNotDetached(); + ); - executeSingleThread(thread_num, num_threads); - }); + executeSingleThread(thread_num, num_threads); + }); + } + + for (auto & thread : threads) + if (thread.joinable()) + thread.join(); } - - for (auto & thread : threads) - if (thread.joinable()) - thread.join(); + else + executeSingleThread(0, num_threads); finished_flag = true; } diff --git a/dbms/src/Processors/Executors/PipelineExecutor.h b/dbms/src/Processors/Executors/PipelineExecutor.h index aded3de3008..2231c19284b 100644 --- a/dbms/src/Processors/Executors/PipelineExecutor.h +++ b/dbms/src/Processors/Executors/PipelineExecutor.h @@ -84,6 +84,7 @@ private: IProcessor * processor = nullptr; UInt64 processors_id = 0; + bool has_quota = false; /// Counters for profiling. size_t num_executed_jobs = 0; @@ -117,6 +118,7 @@ private: execution_state = std::make_unique(); execution_state->processor = processor; execution_state->processors_id = processor_id; + execution_state->has_quota = processor->hasQuota(); } Node(Node && other) noexcept @@ -132,7 +134,59 @@ private: using Stack = std::stack; - using TaskQueue = std::queue; + class TaskQueue + { + public: + void init(size_t num_threads) { queues.resize(num_threads); } + + void push(ExecutionState * state, size_t thread_num) + { + queues[thread_num].push(state); + + ++size_; + + if (state->has_quota) + ++quota_; + } + + ExecutionState * pop(size_t thread_num) + { + if (size_ == 0) + throw Exception("TaskQueue is not empty.", ErrorCodes::LOGICAL_ERROR); + + for (size_t i = 0; i < queues.size(); ++i) + { + if (!queues[thread_num].empty()) + { + ExecutionState * state = queues[thread_num].front(); + queues[thread_num].pop(); + + --size_; + + if (state->has_quota) + ++quota_; + + return state; + } + + ++thread_num; + if (thread_num >= queues.size()) + thread_num = 0; + } + + throw Exception("TaskQueue is not empty.", ErrorCodes::LOGICAL_ERROR); + } + + size_t size() const { return size_; } + bool empty() const { return size_ == 0; } + size_t quota() const { return quota_; } + + private: + using Queue = std::queue; + std::vector queues; + size_t size_ = 0; + size_t quota_ = 0; + }; /// Queue with pointers to tasks. Each thread will concurrently read from it until finished flag is set. /// Stores processors need to be prepared. Preparing status is already set for them. @@ -173,7 +227,7 @@ private: std::mutex mutex; bool wake_flag = false; - std::queue pinned_tasks; + /// std::queue pinned_tasks; }; std::vector> executor_contexts; @@ -186,19 +240,21 @@ private: /// Graph related methods. bool addEdges(UInt64 node); void buildGraph(); - void expandPipeline(Stack & stack, UInt64 pid); + bool expandPipeline(Stack & stack, UInt64 pid); + + using Queue = std::queue; /// Pipeline execution related methods. void addChildlessProcessorsToStack(Stack & stack); - bool tryAddProcessorToStackIfUpdated(Edge & edge, Stack & stack); + bool tryAddProcessorToStackIfUpdated(Edge & edge, Queue & queue, size_t thread_number); static void addJob(ExecutionState * execution_state); // TODO: void addAsyncJob(UInt64 pid); /// Prepare processor with pid number. /// Check parents and children of current processor and push them to stacks if they also need to be prepared. /// If processor wants to be expanded, ExpandPipelineTask from thread_number's execution context will be used. - bool prepareProcessor(UInt64 pid, Stack & children, Stack & parents, size_t thread_number, bool async); - void doExpandPipeline(ExpandPipelineTask * task, bool processing); + bool prepareProcessor(UInt64 pid, size_t thread_number, Queue & queue, std::unique_lock node_lock); + bool doExpandPipeline(ExpandPipelineTask * task, bool processing); void executeImpl(size_t num_threads); void executeSingleThread(size_t thread_num, size_t num_threads); diff --git a/dbms/src/Processors/IProcessor.h b/dbms/src/Processors/IProcessor.h index 852bde2d467..5296f36de87 100644 --- a/dbms/src/Processors/IProcessor.h +++ b/dbms/src/Processors/IProcessor.h @@ -272,12 +272,17 @@ public: size_t getStream() const { return stream_number; } constexpr static size_t NO_STREAM = std::numeric_limits::max(); + void enableQuota() { has_quota = true; } + bool hasQuota() const { return has_quota; } + private: std::atomic is_cancelled{false}; std::string processor_description; size_t stream_number = NO_STREAM; + + bool has_quota = false; }; diff --git a/dbms/src/Processors/Pipe.cpp b/dbms/src/Processors/Pipe.cpp index 17b44a48ea1..27daadac8a0 100644 --- a/dbms/src/Processors/Pipe.cpp +++ b/dbms/src/Processors/Pipe.cpp @@ -115,4 +115,13 @@ void Pipe::pinSources(size_t executor_number) } } +void Pipe::enableQuota() +{ + for (auto & processor : processors) + { + if (auto * source = dynamic_cast(processor.get())) + source->enableQuota(); + } +} + } diff --git a/dbms/src/Processors/Pipe.h b/dbms/src/Processors/Pipe.h index d734c89f485..3d121d3b2e3 100644 --- a/dbms/src/Processors/Pipe.h +++ b/dbms/src/Processors/Pipe.h @@ -42,6 +42,8 @@ public: /// Set information about preferred executor number for sources. void pinSources(size_t executor_number); + void enableQuota(); + void setTotalsPort(OutputPort * totals_) { totals = totals_; } OutputPort * getTotalsPort() const { return totals; } diff --git a/dbms/src/Processors/QueryPipeline.cpp b/dbms/src/Processors/QueryPipeline.cpp index fd75d7f57cf..25abeb6c6d3 100644 --- a/dbms/src/Processors/QueryPipeline.cpp +++ b/dbms/src/Processors/QueryPipeline.cpp @@ -18,6 +18,7 @@ #include #include #include +#include namespace DB { @@ -165,7 +166,6 @@ void QueryPipeline::addSimpleTransformImpl(const TProcessorGetter & getter) for (size_t stream_num = 0; stream_num < streams.size(); ++stream_num) add_transform(streams[stream_num], StreamType::Main, stream_num); - add_transform(delayed_stream_port, StreamType::Main); add_transform(totals_having_port, StreamType::Totals); add_transform(extremes_port, StreamType::Extremes); @@ -185,7 +185,6 @@ void QueryPipeline::addSimpleTransform(const ProcessorGetterWithStreamKind & get void QueryPipeline::addPipe(Processors pipe) { checkInitialized(); - concatDelayedStream(); if (pipe.empty()) throw Exception("Can't add empty processors list to QueryPipeline.", ErrorCodes::LOGICAL_ERROR); @@ -224,41 +223,20 @@ void QueryPipeline::addDelayedStream(ProcessorPtr source) { checkInitialized(); - if (delayed_stream_port) - throw Exception("QueryPipeline already has stream with non joined data.", ErrorCodes::LOGICAL_ERROR); - checkSource(source, false); assertBlocksHaveEqualStructure(current_header, source->getOutputs().front().getHeader(), "QueryPipeline"); - delayed_stream_port = &source->getOutputs().front(); + IProcessor::PortNumbers delayed_streams = { streams.size() }; + streams.emplace_back(&source->getOutputs().front()); processors.emplace_back(std::move(source)); -} -void QueryPipeline::concatDelayedStream() -{ - if (!delayed_stream_port) - return; - - auto resize = std::make_shared(current_header, getNumMainStreams(), 1); - auto stream = streams.begin(); - for (auto & input : resize->getInputs()) - connect(**(stream++), input); - - auto concat = std::make_shared(current_header, 2); - connect(resize->getOutputs().front(), concat->getInputs().front()); - connect(*delayed_stream_port, concat->getInputs().back()); - - streams = { &concat->getOutputs().front() }; - processors.emplace_back(std::move(resize)); - processors.emplace_back(std::move(concat)); - - delayed_stream_port = nullptr; + auto processor = std::make_shared(current_header, streams.size(), delayed_streams); + addPipe({ std::move(processor) }); } void QueryPipeline::resize(size_t num_streams, bool force) { checkInitialized(); - concatDelayedStream(); if (!force && num_streams == getNumStreams()) return; @@ -278,6 +256,12 @@ void QueryPipeline::resize(size_t num_streams, bool force) processors.emplace_back(std::move(resize)); } +void QueryPipeline::enableQuotaForCurrentStreams() +{ + for (auto & stream : streams) + stream->getProcessor().enableQuota(); +} + void QueryPipeline::addTotalsHavingTransform(ProcessorPtr transform) { checkInitialized(); @@ -437,7 +421,6 @@ void QueryPipeline::unitePipelines( std::vector && pipelines, const Block & common_header, const Context & context) { checkInitialized(); - concatDelayedStream(); addSimpleTransform([&](const Block & header) { @@ -450,7 +433,6 @@ void QueryPipeline::unitePipelines( for (auto & pipeline : pipelines) { pipeline.checkInitialized(); - pipeline.concatDelayedStream(); pipeline.addSimpleTransform([&](const Block & header) { @@ -490,6 +472,8 @@ void QueryPipeline::unitePipelines( table_locks.insert(table_locks.end(), std::make_move_iterator(pipeline.table_locks.begin()), std::make_move_iterator(pipeline.table_locks.end())); interpreter_context.insert(interpreter_context.end(), pipeline.interpreter_context.begin(), pipeline.interpreter_context.end()); storage_holder.insert(storage_holder.end(), pipeline.storage_holder.begin(), pipeline.storage_holder.end()); + + max_threads = std::max(max_threads, pipeline.max_threads); } if (!extremes.empty()) diff --git a/dbms/src/Processors/QueryPipeline.h b/dbms/src/Processors/QueryPipeline.h index e32ed6a0abe..29ebaf22955 100644 --- a/dbms/src/Processors/QueryPipeline.h +++ b/dbms/src/Processors/QueryPipeline.h @@ -57,18 +57,19 @@ public: /// Will read from this stream after all data was read from other streams. void addDelayedStream(ProcessorPtr source); - bool hasDelayedStream() const { return delayed_stream_port; } + /// Check if resize transform was used. (In that case another distinct transform will be added). bool hasMixedStreams() const { return has_resize || hasMoreThanOneStream(); } void resize(size_t num_streams, bool force = false); + void enableQuotaForCurrentStreams(); + void unitePipelines(std::vector && pipelines, const Block & common_header, const Context & context); PipelineExecutorPtr execute(); - size_t getNumStreams() const { return streams.size() + (hasDelayedStream() ? 1 : 0); } - size_t getNumMainStreams() const { return streams.size(); } + size_t getNumStreams() const { return streams.size(); } bool hasMoreThanOneStream() const { return getNumStreams() > 1; } bool hasTotals() const { return totals_having_port != nullptr; } @@ -101,9 +102,6 @@ private: OutputPort * totals_having_port = nullptr; OutputPort * extremes_port = nullptr; - /// Special port for delayed stream. - OutputPort * delayed_stream_port = nullptr; - /// If resize processor was added to pipeline. bool has_resize = false; @@ -124,7 +122,6 @@ private: void checkInitialized(); void checkSource(const ProcessorPtr & source, bool can_have_totals); - void concatDelayedStream(); template void addSimpleTransformImpl(const TProcessorGetter & getter); diff --git a/dbms/src/Processors/Transforms/AggregatingTransform.cpp b/dbms/src/Processors/Transforms/AggregatingTransform.cpp index 72a5ff3bb7c..7a5bf77da68 100644 --- a/dbms/src/Processors/Transforms/AggregatingTransform.cpp +++ b/dbms/src/Processors/Transforms/AggregatingTransform.cpp @@ -72,15 +72,33 @@ namespace class ConvertingAggregatedToChunksSource : public ISource { public: + static constexpr UInt32 NUM_BUCKETS = 256; + + struct SharedData + { + std::atomic next_bucket_to_merge = 0; + std::array, NUM_BUCKETS> source_for_bucket; + + SharedData() + { + for (auto & source : source_for_bucket) + source = -1; + } + }; + + using SharedDataPtr = std::shared_ptr; + ConvertingAggregatedToChunksSource( AggregatingTransformParamsPtr params_, ManyAggregatedDataVariantsPtr data_, - Arena * arena_, - std::shared_ptr> next_bucket_to_merge_) + SharedDataPtr shared_data_, + Int32 source_number_, + Arena * arena_) : ISource(params_->getHeader()) , params(std::move(params_)) , data(std::move(data_)) - , next_bucket_to_merge(std::move(next_bucket_to_merge_)) + , shared_data(std::move(shared_data_)) + , source_number(source_number_) , arena(arena_) {} @@ -89,23 +107,25 @@ public: protected: Chunk generate() override { - UInt32 bucket_num = next_bucket_to_merge->fetch_add(1); + UInt32 bucket_num = shared_data->next_bucket_to_merge.fetch_add(1); if (bucket_num >= NUM_BUCKETS) return {}; Block block = params->aggregator.mergeAndConvertOneBucketToBlock(*data, arena, params->final, bucket_num); + Chunk chunk = convertToChunk(block); - return convertToChunk(block); + shared_data->source_for_bucket[bucket_num] = source_number; + + return chunk; } private: AggregatingTransformParamsPtr params; ManyAggregatedDataVariantsPtr data; - std::shared_ptr> next_bucket_to_merge; + SharedDataPtr shared_data; + Int32 source_number; Arena * arena; - - static constexpr UInt32 NUM_BUCKETS = 256; }; /// Generates chunks with aggregated data. @@ -159,6 +179,7 @@ public: auto & out = source->getOutputs().front(); inputs.emplace_back(out.getHeader(), this); connect(out, inputs.back()); + inputs.back().setNeeded(); } return std::move(processors); @@ -200,7 +221,7 @@ public: return Status::Ready; /// Two-level case. - return preparePullFromInputs(); + return prepareTwoLevel(); } private: @@ -220,38 +241,37 @@ private: } /// Read all sources and try to push current bucket. - IProcessor::Status preparePullFromInputs() + IProcessor::Status prepareTwoLevel() { - bool all_inputs_are_finished = true; + auto & output = outputs.front(); - for (auto & input : inputs) + Int32 next_input_num = shared_data->source_for_bucket[current_bucket_num]; + if (next_input_num < 0) + return Status::NeedData; + + auto next_input = std::next(inputs.begin(), next_input_num); + /// next_input can't be finished till data was not pulled. + if (!next_input->hasData()) + return Status::NeedData; + + output.push(next_input->pull()); + + ++current_bucket_num; + if (current_bucket_num == NUM_BUCKETS) { - if (input.isFinished()) - continue; - - all_inputs_are_finished = false; - - input.setNeeded(); - - if (input.hasData()) - ready_chunks.emplace_back(input.pull()); + output.finish(); + /// Do not close inputs, they must be finished. + return Status::Finished; } - moveReadyChunksToMap(); - - if (trySetCurrentChunkFromCurrentBucket()) - return preparePushToOutput(); - - if (all_inputs_are_finished) - throw Exception("All sources have finished before getting enough data in " - "ConvertingAggregatedToChunksTransform.", ErrorCodes::LOGICAL_ERROR); - - return Status::NeedData; + return Status::PortFull; } private: AggregatingTransformParamsPtr params; ManyAggregatedDataVariantsPtr data; + ConvertingAggregatedToChunksSource::SharedDataPtr shared_data; + size_t num_threads; bool is_initialized = false; @@ -259,49 +279,12 @@ private: bool finished = false; Chunk current_chunk; - Chunks ready_chunks; UInt32 current_bucket_num = 0; static constexpr Int32 NUM_BUCKETS = 256; - std::map bucket_to_chunk; Processors processors; - static Int32 getBucketFromChunk(const Chunk & chunk) - { - auto & info = chunk.getChunkInfo(); - if (!info) - throw Exception("Chunk info was not set for chunk in " - "ConvertingAggregatedToChunksTransform.", ErrorCodes::LOGICAL_ERROR); - - auto * agg_info = typeid_cast(info.get()); - if (!agg_info) - throw Exception("Chunk should have AggregatedChunkInfo in " - "ConvertingAggregatedToChunksTransform.", ErrorCodes::LOGICAL_ERROR); - - return agg_info->bucket_num; - } - - void moveReadyChunksToMap() - { - for (auto & chunk : ready_chunks) - { - auto bucket = getBucketFromChunk(chunk); - - if (bucket < 0 || bucket >= NUM_BUCKETS) - throw Exception("Invalid bucket number " + toString(bucket) + " in " - "ConvertingAggregatedToChunksTransform.", ErrorCodes::LOGICAL_ERROR); - - if (bucket_to_chunk.count(bucket)) - throw Exception("Found several chunks with the same bucket number in " - "ConvertingAggregatedToChunksTransform.", ErrorCodes::LOGICAL_ERROR); - - bucket_to_chunk[bucket] = std::move(chunk); - } - - ready_chunks.clear(); - } - void setCurrentChunk(Chunk chunk) { if (has_input) @@ -366,34 +349,17 @@ private: void createSources() { AggregatedDataVariantsPtr & first = data->at(0); - auto next_bucket_to_merge = std::make_shared>(0); + shared_data = std::make_shared(); for (size_t thread = 0; thread < num_threads; ++thread) { Arena * arena = first->aggregates_pools.at(thread).get(); auto source = std::make_shared( - params, data, arena, next_bucket_to_merge); + params, data, shared_data, thread, arena); processors.emplace_back(std::move(source)); } } - - bool trySetCurrentChunkFromCurrentBucket() - { - auto it = bucket_to_chunk.find(current_bucket_num); - if (it != bucket_to_chunk.end()) - { - setCurrentChunk(std::move(it->second)); - ++current_bucket_num; - - if (current_bucket_num == NUM_BUCKETS) - finished = true; - - return true; - } - - return false; - } }; AggregatingTransform::AggregatingTransform(Block header, AggregatingTransformParamsPtr params_) diff --git a/dbms/src/Processors/Transforms/MergeSortingTransform.cpp b/dbms/src/Processors/Transforms/MergeSortingTransform.cpp index 83d80d42e05..39da24ba149 100644 --- a/dbms/src/Processors/Transforms/MergeSortingTransform.cpp +++ b/dbms/src/Processors/Transforms/MergeSortingTransform.cpp @@ -1,11 +1,10 @@ -#include #include #include #include -#include #include -#include #include +#include +#include #include #include #include @@ -21,6 +20,13 @@ namespace ProfileEvents namespace DB { +namespace ErrorCodes +{ + extern const int NOT_ENOUGH_SPACE; +} +class MergeSorter; + + class BufferingToFileTransform : public IAccumulatingTransform { public: diff --git a/dbms/src/Processors/Transforms/MergeSortingTransform.h b/dbms/src/Processors/Transforms/MergeSortingTransform.h index ee51f29565a..ecfaeb4f272 100644 --- a/dbms/src/Processors/Transforms/MergeSortingTransform.h +++ b/dbms/src/Processors/Transforms/MergeSortingTransform.h @@ -1,25 +1,14 @@ #pragma once + #include #include #include -#include -#include -#include -#include - #include -#include namespace DB { -namespace ErrorCodes -{ - extern const int NOT_ENOUGH_SPACE; -} -class MergeSorter; - class MergeSortingTransform : public SortingTransform { public: diff --git a/dbms/src/Processors/Transforms/MergingSortedTransform.cpp b/dbms/src/Processors/Transforms/MergingSortedTransform.cpp index 705116ca081..ddbd91b38d1 100644 --- a/dbms/src/Processors/Transforms/MergingSortedTransform.cpp +++ b/dbms/src/Processors/Transforms/MergingSortedTransform.cpp @@ -148,9 +148,9 @@ IProcessor::Status MergingSortedTransform::prepare() return Status::NeedData; if (has_collation) - initQueue(queue_with_collation); + queue_with_collation = SortingHeap(cursors); else - initQueue(queue_without_collation); + queue_without_collation = SortingHeap(cursors); is_initialized = true; return Status::Ready; @@ -169,7 +169,6 @@ IProcessor::Status MergingSortedTransform::prepare() if (need_data) { - auto & input = *std::next(inputs.begin(), next_input_to_read); if (!input.isFinished()) { @@ -183,7 +182,11 @@ IProcessor::Status MergingSortedTransform::prepare() return Status::NeedData; updateCursor(std::move(chunk), next_input_to_read); - pushToQueue(next_input_to_read); + + if (has_collation) + queue_with_collation.push(cursors[next_input_to_read]); + else + queue_without_collation.push(cursors[next_input_to_read]); } need_data = false; @@ -201,8 +204,8 @@ void MergingSortedTransform::work() merge(queue_without_collation); } -template -void MergingSortedTransform::merge(std::priority_queue & queue) +template +void MergingSortedTransform::merge(TSortingHeap & queue) { /// Returns MergeStatus which we should return if we are going to finish now. auto can_read_another_row = [&, this]() @@ -224,77 +227,66 @@ void MergingSortedTransform::merge(std::priority_queue & queue) }; /// Take rows in required order and put them into `merged_data`, while the rows are no more than `max_block_size` - while (!queue.empty()) + while (queue.isValid()) { /// Shouldn't happen at first iteration, but check just in case. if (!can_read_another_row()) return; - TSortCursor current = queue.top(); - queue.pop(); - bool first_iteration = true; + auto current = queue.current(); - while (true) + /** And what if the block is totally less or equal than the rest for the current cursor? + * Or is there only one data source left in the queue? Then you can take the entire block on current cursor. + */ + if (current.impl->isFirst() + && (queue.size() == 1 + || (queue.size() >= 2 && current.totallyLessOrEquals(queue.nextChild())))) { - if (!first_iteration && !can_read_another_row()) + //std::cerr << "current block is totally less or equals\n"; + + /// If there are already data in the current block, we first return it. We'll get here again the next time we call the merge function. + if (merged_data.mergedRows() != 0) { - queue.push(current); - return; - } - first_iteration = false; - - /** And what if the block is totally less or equal than the rest for the current cursor? - * Or is there only one data source left in the queue? Then you can take the entire block on current cursor. - */ - if (current.impl->isFirst() && (queue.empty() || current.totallyLessOrEquals(queue.top()))) - { - //std::cerr << "current block is totally less or equals\n"; - - /// If there are already data in the current block, we first return it. We'll get here again the next time we call the merge function. - if (merged_data.mergedRows() != 0) - { - //std::cerr << "merged rows is non-zero\n"; - queue.push(current); - return; - } - - /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) - size_t source_num = current.impl->order; - insertFromChunk(source_num); + //std::cerr << "merged rows is non-zero\n"; return; } - //std::cerr << "total_merged_rows: " << total_merged_rows << ", merged_rows: " << merged_rows << "\n"; - //std::cerr << "Inserting row\n"; - merged_data.insertRow(current->all_columns, current->pos); + /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) + size_t source_num = current.impl->order; + insertFromChunk(source_num); + queue.removeTop(); + return; + } - if (out_row_sources_buf) - { - /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) - RowSourcePart row_source(current.impl->order); - out_row_sources_buf->write(row_source.data); - } + //std::cerr << "total_merged_rows: " << total_merged_rows << ", merged_rows: " << merged_rows << "\n"; + //std::cerr << "Inserting row\n"; + merged_data.insertRow(current->all_columns, current->pos); - if (current->isLast()) - { - need_data = true; - next_input_to_read = current.impl->order; + if (out_row_sources_buf) + { + /// Actually, current.impl->order stores source number (i.e. cursors[current.impl->order] == current.impl) + RowSourcePart row_source(current.impl->order); + out_row_sources_buf->write(row_source.data); + } - if (limit && merged_data.totalMergedRows() >= limit) - is_finished = true; + if (!current->isLast()) + { +// std::cerr << "moving to next row\n"; + queue.next(); + } + else + { + /// We will get the next block from the corresponding source, if there is one. + queue.removeTop(); - return; - } +// std::cerr << "It was last row, fetching next block\n"; + need_data = true; + next_input_to_read = current.impl->order; - //std::cerr << "moving to next row\n"; - current->next(); + if (limit && merged_data.totalMergedRows() >= limit) + is_finished = true; - if (!queue.empty() && current.greater(queue.top())) - { - //std::cerr << "next row is not least, pushing back to queue\n"; - queue.push(current); - break; - } + return; } } is_finished = true; diff --git a/dbms/src/Processors/Transforms/MergingSortedTransform.h b/dbms/src/Processors/Transforms/MergingSortedTransform.h index b32dd076c5f..aa88fb09623 100644 --- a/dbms/src/Processors/Transforms/MergingSortedTransform.h +++ b/dbms/src/Processors/Transforms/MergingSortedTransform.h @@ -1,10 +1,10 @@ #pragma once + #include #include #include #include -#include namespace DB { @@ -111,14 +111,10 @@ protected: /// Chunks currently being merged. std::vector source_chunks; - using CursorImpls = std::vector; - CursorImpls cursors; + SortCursorImpls cursors; - using Queue = std::priority_queue; - Queue queue_without_collation; - - using QueueWithCollation = std::priority_queue; - QueueWithCollation queue_with_collation; + SortingHeap queue_without_collation; + SortingHeap queue_with_collation; private: @@ -128,8 +124,8 @@ private: bool need_data = false; size_t next_input_to_read = 0; - template - void merge(std::priority_queue & queue); + template + void merge(TSortingHeap & queue); void insertFromChunk(size_t source_num); @@ -159,22 +155,6 @@ private: shared_chunk_ptr->all_columns = cursors[source_num].all_columns; shared_chunk_ptr->sort_columns = cursors[source_num].sort_columns; } - - void pushToQueue(size_t source_num) - { - if (has_collation) - queue_with_collation.push(SortCursorWithCollation(&cursors[source_num])); - else - queue_without_collation.push(SortCursor(&cursors[source_num])); - } - - template - void initQueue(std::priority_queue & queue) - { - for (auto & cursor : cursors) - if (!cursor.empty()) - queue.push(TSortCursor(&cursor)); - } }; } diff --git a/dbms/src/Processors/Transforms/SortingTransform.cpp b/dbms/src/Processors/Transforms/SortingTransform.cpp index ab87591c0d6..30f53742ec0 100644 --- a/dbms/src/Processors/Transforms/SortingTransform.cpp +++ b/dbms/src/Processors/Transforms/SortingTransform.cpp @@ -40,16 +40,12 @@ MergeSorter::MergeSorter(Chunks chunks_, SortDescription & description_, size_t chunks.swap(nonempty_chunks); - if (!has_collation) - { - for (auto & cursor : cursors) - queue_without_collation.push(SortCursor(&cursor)); - } + if (has_collation) + queue_with_collation = SortingHeap(cursors); + else if (description.size() > 1) + queue_without_collation = SortingHeap(cursors); else - { - for (auto & cursor : cursors) - queue_with_collation.push(SortCursorWithCollation(&cursor)); - } + queue_simple = SortingHeap(cursors); } @@ -65,50 +61,61 @@ Chunk MergeSorter::read() return res; } - return !has_collation - ? mergeImpl(queue_without_collation) - : mergeImpl(queue_with_collation); + if (has_collation) + return mergeImpl(queue_with_collation); + else if (description.size() > 1) + return mergeImpl(queue_without_collation); + else + return mergeImpl(queue_simple); } -template -Chunk MergeSorter::mergeImpl(std::priority_queue & queue) +template +Chunk MergeSorter::mergeImpl(TSortingHeap & queue) { size_t num_columns = chunks[0].getNumColumns(); - MutableColumns merged_columns = chunks[0].cloneEmptyColumns(); - /// TODO: reserve (in each column) + + /// Reserve + if (queue.isValid()) + { + /// The expected size of output block is the same as input block + size_t size_to_reserve = chunks[0].getNumRows(); + for (auto & column : merged_columns) + column->reserve(size_to_reserve); + } + + /// TODO: Optimization when a single block left. /// Take rows from queue in right order and push to 'merged'. size_t merged_rows = 0; - while (!queue.empty()) + while (queue.isValid()) { - TSortCursor current = queue.top(); - queue.pop(); + auto current = queue.current(); + /// Append a row from queue. for (size_t i = 0; i < num_columns; ++i) merged_columns[i]->insertFrom(*current->all_columns[i], current->pos); ++total_merged_rows; ++merged_rows; - if (!current->isLast()) - { - current->next(); - queue.push(current); - } - + /// We don't need more rows because of limit has reached. if (limit && total_merged_rows == limit) { chunks.clear(); - return Chunk(std::move(merged_columns), merged_rows); + break; } + queue.next(); + + /// It's enough for current output block but we will continue. if (merged_rows == max_merged_block_size) - return Chunk(std::move(merged_columns), merged_rows); + break; } - chunks.clear(); + if (!queue.isValid()) + chunks.clear(); if (merged_rows == 0) return {}; diff --git a/dbms/src/Processors/Transforms/SortingTransform.h b/dbms/src/Processors/Transforms/SortingTransform.h index 2703501c81a..49bdf303c7f 100644 --- a/dbms/src/Processors/Transforms/SortingTransform.h +++ b/dbms/src/Processors/Transforms/SortingTransform.h @@ -1,10 +1,10 @@ #pragma once + #include #include #include #include #include -#include namespace DB @@ -27,19 +27,19 @@ private: UInt64 limit; size_t total_merged_rows = 0; - using CursorImpls = std::vector; - CursorImpls cursors; + SortCursorImpls cursors; bool has_collation = false; - std::priority_queue queue_without_collation; - std::priority_queue queue_with_collation; + SortingHeap queue_without_collation; + SortingHeap queue_simple; + SortingHeap queue_with_collation; /** Two different cursors are supported - with and without Collation. * Templates are used (instead of virtual functions in SortCursor) for zero-overhead. */ - template - Chunk mergeImpl(std::priority_queue & queue); + template + Chunk mergeImpl(TSortingHeap & queue); }; diff --git a/dbms/src/Storages/AlterCommands.cpp b/dbms/src/Storages/AlterCommands.cpp index c586ea54c98..fc7bf608b17 100644 --- a/dbms/src/Storages/AlterCommands.cpp +++ b/dbms/src/Storages/AlterCommands.cpp @@ -249,6 +249,9 @@ void AlterCommand::apply(StorageInMemoryMetadata & metadata) const /// let's use info about old type if (data_type == nullptr) codec->useInfoAboutType(column.type); + else /// use info about new DataType + codec->useInfoAboutType(data_type); + column.codec = codec; } @@ -316,7 +319,7 @@ void AlterCommand::apply(StorageInMemoryMetadata & metadata) const if (insert_it == metadata.indices.indices.end()) throw Exception("Wrong index name. Cannot find index " + backQuote(after_index_name) + " to insert after.", - ErrorCodes::LOGICAL_ERROR); + ErrorCodes::BAD_ARGUMENTS); ++insert_it; } @@ -338,7 +341,7 @@ void AlterCommand::apply(StorageInMemoryMetadata & metadata) const if (if_exists) return; throw Exception("Wrong index name. Cannot find index " + backQuote(index_name) + " to drop.", - ErrorCodes::LOGICAL_ERROR); + ErrorCodes::BAD_ARGUMENTS); } metadata.indices.indices.erase(erase_it); @@ -378,7 +381,7 @@ void AlterCommand::apply(StorageInMemoryMetadata & metadata) const if (if_exists) return; throw Exception("Wrong constraint name. Cannot find constraint `" + constraint_name + "` to drop.", - ErrorCodes::LOGICAL_ERROR); + ErrorCodes::BAD_ARGUMENTS); } metadata.constraints.constraints.erase(erase_it); } diff --git a/dbms/src/Storages/IStorage.cpp b/dbms/src/Storages/IStorage.cpp index 9dabfe0b604..e48e9896597 100644 --- a/dbms/src/Storages/IStorage.cpp +++ b/dbms/src/Storages/IStorage.cpp @@ -425,21 +425,4 @@ BlockInputStreams IStorage::read( return res; } -DB::CompressionMethod IStorage::chooseCompressionMethod(const String & uri, const String & compression_method) -{ - if (compression_method == "auto" || compression_method == "") - { - if (endsWith(uri, ".gz")) - return DB::CompressionMethod::Gzip; - else - return DB::CompressionMethod::None; - } - else if (compression_method == "gzip") - return DB::CompressionMethod::Gzip; - else if (compression_method == "none") - return DB::CompressionMethod::None; - else - throw Exception("Only auto, none, gzip supported as compression method", ErrorCodes::NOT_IMPLEMENTED); -} - } diff --git a/dbms/src/Storages/IStorage.h b/dbms/src/Storages/IStorage.h index 8f8a363aec1..69bbca86879 100644 --- a/dbms/src/Storages/IStorage.h +++ b/dbms/src/Storages/IStorage.h @@ -5,7 +5,6 @@ #include #include #include -#include #include #include #include @@ -440,8 +439,6 @@ public: return {}; } - static DB::CompressionMethod chooseCompressionMethod(const String & uri, const String & compression_method); - private: /// You always need to take the next three locks in this order. diff --git a/dbms/src/Storages/LiveView/LiveViewCommands.h b/dbms/src/Storages/LiveView/LiveViewCommands.h index 54048c28a5f..6757acdadab 100644 --- a/dbms/src/Storages/LiveView/LiveViewCommands.h +++ b/dbms/src/Storages/LiveView/LiveViewCommands.h @@ -9,13 +9,13 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ - #pragma once #include #include #include + namespace DB { diff --git a/dbms/src/Storages/LiveView/LiveViewEventsBlockInputStream.h b/dbms/src/Storages/LiveView/LiveViewEventsBlockInputStream.h index 14cfade338d..bf971d75d01 100644 --- a/dbms/src/Storages/LiveView/LiveViewEventsBlockInputStream.h +++ b/dbms/src/Storages/LiveView/LiveViewEventsBlockInputStream.h @@ -9,7 +9,6 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ - #pragma once #include diff --git a/dbms/src/Storages/LiveView/ProxyStorage.h b/dbms/src/Storages/LiveView/ProxyStorage.h deleted file mode 100644 index 60faa907209..00000000000 --- a/dbms/src/Storages/LiveView/ProxyStorage.h +++ /dev/null @@ -1,66 +0,0 @@ -#pragma once - -#include - -namespace DB -{ - -class ProxyStorage : public IStorage -{ -public: - ProxyStorage(StoragePtr storage_, BlockInputStreams streams_, QueryProcessingStage::Enum to_stage_) - : storage(std::move(storage_)), streams(std::move(streams_)), to_stage(to_stage_) {} - -public: - std::string getName() const override { return "ProxyStorage(" + storage->getName() + ")"; } - std::string getTableName() const override { return storage->getTableName(); } - - bool isRemote() const override { return storage->isRemote(); } - bool supportsSampling() const override { return storage->supportsSampling(); } - bool supportsFinal() const override { return storage->supportsFinal(); } - bool supportsPrewhere() const override { return storage->supportsPrewhere(); } - bool supportsReplication() const override { return storage->supportsReplication(); } - bool supportsDeduplication() const override { return storage->supportsDeduplication(); } - - QueryProcessingStage::Enum getQueryProcessingStage(const Context & /*context*/) const override { return to_stage; } - - BlockInputStreams read( - const Names & /*column_names*/, - const SelectQueryInfo & /*query_info*/, - const Context & /*context*/, - QueryProcessingStage::Enum /*processed_stage*/, - size_t /*max_block_size*/, - unsigned /*num_streams*/) override - { - return streams; - } - - bool supportsIndexForIn() const override { return storage->supportsIndexForIn(); } - bool mayBenefitFromIndexForIn(const ASTPtr & left_in_operand, const Context & query_context) const override { return storage->mayBenefitFromIndexForIn(left_in_operand, query_context); } - ASTPtr getPartitionKeyAST() const override { return storage->getPartitionKeyAST(); } - ASTPtr getSortingKeyAST() const override { return storage->getSortingKeyAST(); } - ASTPtr getPrimaryKeyAST() const override { return storage->getPrimaryKeyAST(); } - ASTPtr getSamplingKeyAST() const override { return storage->getSamplingKeyAST(); } - Names getColumnsRequiredForPartitionKey() const override { return storage->getColumnsRequiredForPartitionKey(); } - Names getColumnsRequiredForSortingKey() const override { return storage->getColumnsRequiredForSortingKey(); } - Names getColumnsRequiredForPrimaryKey() const override { return storage->getColumnsRequiredForPrimaryKey(); } - Names getColumnsRequiredForSampling() const override { return storage->getColumnsRequiredForSampling(); } - Names getColumnsRequiredForFinal() const override { return storage->getColumnsRequiredForFinal(); } - - const ColumnsDescription & getColumns() const override { return storage->getColumns(); } - void setColumns(ColumnsDescription columns_) override { return storage->setColumns(columns_); } - NameAndTypePair getColumn(const String & column_name) const override { return storage->getColumn(column_name); } - bool hasColumn(const String & column_name) const override { return storage->hasColumn(column_name); } - static StoragePtr createProxyStorage(StoragePtr storage, BlockInputStreams streams, QueryProcessingStage::Enum to_stage) - { - return std::make_shared(std::move(storage), std::move(streams), to_stage); - } -private: - StoragePtr storage; - BlockInputStreams streams; - QueryProcessingStage::Enum to_stage; -}; - - - -} diff --git a/dbms/src/Storages/LiveView/StorageBlocks.h b/dbms/src/Storages/LiveView/StorageBlocks.h new file mode 100644 index 00000000000..bd60e6f0b97 --- /dev/null +++ b/dbms/src/Storages/LiveView/StorageBlocks.h @@ -0,0 +1,51 @@ +#pragma once + +#include + + +namespace DB +{ + +class StorageBlocks : public IStorage +{ +/* Storage based on the prepared streams that already contain data blocks. + * Used by Live Views to complete stored query based on the mergeable blocks. + */ +public: + StorageBlocks(const std::string & database_name_, const std::string & table_name_, + const ColumnsDescription & columns_, BlockInputStreams streams_, + QueryProcessingStage::Enum to_stage_) + : database_name(database_name_), table_name(table_name_), streams(streams_), to_stage(to_stage_) + { + setColumns(columns_); + } + static StoragePtr createStorage(const std::string & database_name, const std::string & table_name, + const ColumnsDescription & columns, BlockInputStreams streams, QueryProcessingStage::Enum to_stage) + { + return std::make_shared(database_name, table_name, columns, streams, to_stage); + } + std::string getName() const override { return "Blocks"; } + std::string getTableName() const override { return table_name; } + std::string getDatabaseName() const override { return database_name; } + QueryProcessingStage::Enum getQueryProcessingStage(const Context & /*context*/) const override { return to_stage; } + + BlockInputStreams read( + const Names & /*column_names*/, + const SelectQueryInfo & /*query_info*/, + const Context & /*context*/, + QueryProcessingStage::Enum /*processed_stage*/, + size_t /*max_block_size*/, + unsigned /*num_streams*/) override + { + return streams; + } + +private: + std::string database_name; + std::string table_name; + Block res_block; + BlockInputStreams streams; + QueryProcessingStage::Enum to_stage; +}; + +} diff --git a/dbms/src/Storages/LiveView/StorageLiveView.cpp b/dbms/src/Storages/LiveView/StorageLiveView.cpp index 6118ef26bba..eae8eaa1d3c 100644 --- a/dbms/src/Storages/LiveView/StorageLiveView.cpp +++ b/dbms/src/Storages/LiveView/StorageLiveView.cpp @@ -9,7 +9,6 @@ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ - #include #include #include @@ -32,14 +31,17 @@ limitations under the License. */ #include #include #include -#include +#include #include #include #include +#include +#include #include #include + namespace DB { @@ -52,13 +54,16 @@ namespace ErrorCodes extern const int SUPPORT_IS_DISABLED; } -static void extractDependentTable(ASTSelectQuery & query, String & select_database_name, String & select_table_name) +static void extractDependentTable(ASTPtr & query, String & select_database_name, String & select_table_name, const String & table_name, ASTPtr & inner_subquery) { - auto db_and_table = getDatabaseAndTable(query, 0); - ASTPtr subquery = extractTableExpression(query, 0); + ASTSelectQuery & select_query = typeid_cast(*query); + auto db_and_table = getDatabaseAndTable(select_query, 0); + ASTPtr subquery = extractTableExpression(select_query, 0); if (!db_and_table && !subquery) + { return; + } if (db_and_table) { @@ -68,19 +73,21 @@ static void extractDependentTable(ASTSelectQuery & query, String & select_databa { db_and_table->database = select_database_name; AddDefaultDatabaseVisitor visitor(select_database_name); - visitor.visit(query); + visitor.visit(select_query); } else select_database_name = db_and_table->database; + + select_query.replaceDatabaseAndTable("", table_name + "_blocks"); } else if (auto * ast_select = subquery->as()) { if (ast_select->list_of_selects->children.size() != 1) throw Exception("UNION is not supported for LIVE VIEW", ErrorCodes::QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW); - auto & inner_query = ast_select->list_of_selects->children.at(0); + inner_subquery = ast_select->list_of_selects->children.at(0)->clone(); - extractDependentTable(inner_query->as(), select_database_name, select_table_name); + extractDependentTable(ast_select->list_of_selects->children.at(0), select_database_name, select_table_name, table_name, inner_subquery); } else throw Exception("Logical error while creating StorageLiveView." @@ -88,6 +95,66 @@ static void extractDependentTable(ASTSelectQuery & query, String & select_databa DB::ErrorCodes::LOGICAL_ERROR); } +MergeableBlocksPtr StorageLiveView::collectMergeableBlocks(const Context & context) +{ + ASTPtr mergeable_query = inner_query; + + if (inner_subquery) + mergeable_query = inner_subquery; + + MergeableBlocksPtr new_mergeable_blocks = std::make_shared(); + BlocksPtrs new_blocks = std::make_shared>(); + BlocksPtr base_blocks = std::make_shared(); + + InterpreterSelectQuery interpreter(mergeable_query->clone(), context, SelectQueryOptions(QueryProcessingStage::WithMergeableState), Names()); + + auto view_mergeable_stream = std::make_shared(interpreter.execute().in); + + while (Block this_block = view_mergeable_stream->read()) + base_blocks->push_back(this_block); + + new_blocks->push_back(base_blocks); + + new_mergeable_blocks->blocks = new_blocks; + new_mergeable_blocks->sample_block = view_mergeable_stream->getHeader(); + + return new_mergeable_blocks; +} + +BlockInputStreams StorageLiveView::blocksToInputStreams(BlocksPtrs blocks, Block & sample_block) +{ + BlockInputStreams streams; + for (auto & blocks_ : *blocks) + { + BlockInputStreamPtr stream = std::make_shared(std::make_shared(blocks_), sample_block); + streams.push_back(std::move(stream)); + } + return streams; +} + +/// Complete query using input streams from mergeable blocks +BlockInputStreamPtr StorageLiveView::completeQuery(BlockInputStreams from) +{ + auto block_context = std::make_unique(global_context); + block_context->makeQueryContext(); + + auto blocks_storage = StorageBlocks::createStorage(database_name, table_name, parent_storage->getColumns(), + std::move(from), QueryProcessingStage::WithMergeableState); + + block_context->addExternalTable(table_name + "_blocks", blocks_storage); + + InterpreterSelectQuery select(inner_blocks_query->clone(), *block_context, StoragePtr(), SelectQueryOptions(QueryProcessingStage::Complete)); + BlockInputStreamPtr data = std::make_shared(select.execute().in); + + /// Squashing is needed here because the view query can generate a lot of blocks + /// even when only one block is inserted into the parent table (e.g. if the query is a GROUP BY + /// and two-level aggregation is triggered). + data = std::make_shared( + data, global_context.getSettingsRef().min_insert_block_size_rows, + global_context.getSettingsRef().min_insert_block_size_bytes); + + return data; +} void StorageLiveView::writeIntoLiveView( StorageLiveView & live_view, @@ -110,49 +177,40 @@ void StorageLiveView::writeIntoLiveView( bool is_block_processed = false; BlockInputStreams from; - BlocksPtrs mergeable_blocks; + MergeableBlocksPtr mergeable_blocks; BlocksPtr new_mergeable_blocks = std::make_shared(); { std::lock_guard lock(live_view.mutex); mergeable_blocks = live_view.getMergeableBlocks(); - if (!mergeable_blocks || mergeable_blocks->size() >= context.getGlobalContext().getSettingsRef().max_live_view_insert_blocks_before_refresh) + if (!mergeable_blocks || mergeable_blocks->blocks->size() >= context.getGlobalContext().getSettingsRef().max_live_view_insert_blocks_before_refresh) { - mergeable_blocks = std::make_shared>(); - BlocksPtr base_mergeable_blocks = std::make_shared(); - InterpreterSelectQuery interpreter(live_view.getInnerQuery(), context, SelectQueryOptions(QueryProcessingStage::WithMergeableState), Names()); - auto view_mergeable_stream = std::make_shared( - interpreter.execute().in); - while (Block this_block = view_mergeable_stream->read()) - base_mergeable_blocks->push_back(this_block); - mergeable_blocks->push_back(base_mergeable_blocks); + mergeable_blocks = live_view.collectMergeableBlocks(context); live_view.setMergeableBlocks(mergeable_blocks); - - /// Create from streams - for (auto & blocks_ : *mergeable_blocks) - { - if (blocks_->empty()) - continue; - auto sample_block = blocks_->front().cloneEmpty(); - BlockInputStreamPtr stream = std::make_shared(std::make_shared(blocks_), sample_block); - from.push_back(std::move(stream)); - } - + from = live_view.blocksToInputStreams(mergeable_blocks->blocks, mergeable_blocks->sample_block); is_block_processed = true; } } if (!is_block_processed) { - auto parent_storage = context.getTable(live_view.getSelectDatabaseName(), live_view.getSelectTableName()); + ASTPtr mergeable_query = live_view.getInnerQuery(); + + if (live_view.getInnerSubQuery()) + mergeable_query = live_view.getInnerSubQuery(); + BlockInputStreams streams = {std::make_shared(block)}; - auto proxy_storage = std::make_shared(parent_storage, std::move(streams), QueryProcessingStage::FetchColumns); - InterpreterSelectQuery select_block(live_view.getInnerQuery(), - context, proxy_storage, + + auto blocks_storage = StorageBlocks::createStorage(live_view.database_name, live_view.table_name, + live_view.getParentStorage()->getColumns(), std::move(streams), QueryProcessingStage::FetchColumns); + + InterpreterSelectQuery select_block(mergeable_query, context, blocks_storage, QueryProcessingStage::WithMergeableState); + auto data_mergeable_stream = std::make_shared( select_block.execute().in); + while (Block this_block = data_mergeable_stream->read()) new_mergeable_blocks->push_back(this_block); @@ -163,31 +221,12 @@ void StorageLiveView::writeIntoLiveView( std::lock_guard lock(live_view.mutex); mergeable_blocks = live_view.getMergeableBlocks(); - mergeable_blocks->push_back(new_mergeable_blocks); - - /// Create from streams - for (auto & blocks_ : *mergeable_blocks) - { - if (blocks_->empty()) - continue; - auto sample_block = blocks_->front().cloneEmpty(); - BlockInputStreamPtr stream = std::make_shared(std::make_shared(blocks_), sample_block); - from.push_back(std::move(stream)); - } + mergeable_blocks->blocks->push_back(new_mergeable_blocks); + from = live_view.blocksToInputStreams(mergeable_blocks->blocks, mergeable_blocks->sample_block); } } - auto parent_storage = context.getTable(live_view.getSelectDatabaseName(), live_view.getSelectTableName()); - auto proxy_storage = std::make_shared(parent_storage, std::move(from), QueryProcessingStage::WithMergeableState); - InterpreterSelectQuery select(live_view.getInnerQuery(), context, proxy_storage, QueryProcessingStage::Complete); - BlockInputStreamPtr data = std::make_shared(select.execute().in); - - /// Squashing is needed here because the view query can generate a lot of blocks - /// even when only one block is inserted into the parent table (e.g. if the query is a GROUP BY - /// and two-level aggregation is triggered). - data = std::make_shared( - data, context.getGlobalContext().getSettingsRef().min_insert_block_size_rows, context.getGlobalContext().getSettingsRef().min_insert_block_size_bytes); - + BlockInputStreamPtr data = live_view.completeQuery(from); copyData(*data, *output); } @@ -201,6 +240,9 @@ StorageLiveView::StorageLiveView( : table_name(table_name_), database_name(database_name_), global_context(local_context.getGlobalContext()) { + live_view_context = std::make_unique(global_context); + live_view_context->makeQueryContext(); + setColumns(columns_); if (!query.select) @@ -212,9 +254,11 @@ StorageLiveView::StorageLiveView( throw Exception("UNION is not supported for LIVE VIEW", ErrorCodes::QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW); inner_query = query.select->list_of_selects->children.at(0); + inner_blocks_query = inner_query->clone(); - ASTSelectQuery & select_query = typeid_cast(*inner_query); - extractDependentTable(select_query, select_database_name, select_table_name); + InterpreterSelectQuery(inner_blocks_query, *live_view_context, SelectQueryOptions().modify().analyze()); + + extractDependentTable(inner_blocks_query, select_database_name, select_table_name, table_name, inner_subquery); /// If the table is not specified - use the table `system.one` if (select_table_name.empty()) @@ -227,6 +271,8 @@ StorageLiveView::StorageLiveView( DatabaseAndTableName(select_database_name, select_table_name), DatabaseAndTableName(database_name, table_name)); + parent_storage = local_context.getTable(select_database_name, select_table_name); + is_temporary = query.temporary; temporary_live_view_timeout = local_context.getSettingsRef().temporary_live_view_timeout.totalSeconds(); @@ -257,9 +303,7 @@ Block StorageLiveView::getHeader() const if (!sample_block) { - auto storage = global_context.getTable(select_database_name, select_table_name); - sample_block = InterpreterSelectQuery(inner_query, global_context, storage, - SelectQueryOptions(QueryProcessingStage::Complete)).getSampleBlock(); + sample_block = InterpreterSelectQuery(inner_query->clone(), *live_view_context, SelectQueryOptions(QueryProcessingStage::Complete)).getSampleBlock(); sample_block.insert({DataTypeUInt64().createColumnConst( sample_block.rows(), 0)->convertToFullColumnIfConst(), std::make_shared(), @@ -271,7 +315,6 @@ Block StorageLiveView::getHeader() const sample_block.safeGetByPosition(i).column = sample_block.safeGetByPosition(i).column->convertToFullColumnIfConst(); } } - return sample_block; } @@ -281,26 +324,10 @@ bool StorageLiveView::getNewBlocks() UInt128 key; BlocksPtr new_blocks = std::make_shared(); BlocksMetadataPtr new_blocks_metadata = std::make_shared(); - BlocksPtr new_mergeable_blocks = std::make_shared(); - InterpreterSelectQuery interpreter(inner_query->clone(), global_context, SelectQueryOptions(QueryProcessingStage::WithMergeableState), Names()); - auto mergeable_stream = std::make_shared(interpreter.execute().in); - - while (Block block = mergeable_stream->read()) - new_mergeable_blocks->push_back(block); - - mergeable_blocks = std::make_shared>(); - mergeable_blocks->push_back(new_mergeable_blocks); - BlockInputStreamPtr from = std::make_shared(std::make_shared(new_mergeable_blocks), mergeable_stream->getHeader()); - auto proxy_storage = ProxyStorage::createProxyStorage(global_context.getTable(select_database_name, select_table_name), {from}, QueryProcessingStage::WithMergeableState); - InterpreterSelectQuery select(inner_query->clone(), global_context, proxy_storage, SelectQueryOptions(QueryProcessingStage::Complete)); - BlockInputStreamPtr data = std::make_shared(select.execute().in); - - /// Squashing is needed here because the view query can generate a lot of blocks - /// even when only one block is inserted into the parent table (e.g. if the query is a GROUP BY - /// and two-level aggregation is triggered). - data = std::make_shared( - data, global_context.getSettingsRef().min_insert_block_size_rows, global_context.getSettingsRef().min_insert_block_size_bytes); + mergeable_blocks = collectMergeableBlocks(*live_view_context); + BlockInputStreams from = blocksToInputStreams(mergeable_blocks->blocks, mergeable_blocks->sample_block); + BlockInputStreamPtr data = completeQuery({from}); while (Block block = data->read()) { diff --git a/dbms/src/Storages/LiveView/StorageLiveView.h b/dbms/src/Storages/LiveView/StorageLiveView.h index 3f1dffb898c..916406a1dbd 100644 --- a/dbms/src/Storages/LiveView/StorageLiveView.h +++ b/dbms/src/Storages/LiveView/StorageLiveView.h @@ -27,9 +27,16 @@ struct BlocksMetadata UInt64 version; }; +struct MergeableBlocks +{ + BlocksPtrs blocks; + Block sample_block; +}; + class IAST; using ASTPtr = std::shared_ptr; using BlocksMetadataPtr = std::shared_ptr; +using MergeableBlocksPtr = std::shared_ptr; class StorageLiveView : public ext::shared_ptr_helper, public IStorage { @@ -45,12 +52,24 @@ public: String getDatabaseName() const override { return database_name; } String getSelectDatabaseName() const { return select_database_name; } String getSelectTableName() const { return select_table_name; } + StoragePtr getParentStorage() const { return parent_storage; } NameAndTypePair getColumn(const String & column_name) const override; bool hasColumn(const String & column_name) const override; - // const NamesAndTypesList & getColumnsListImpl() const override { return *columns; } ASTPtr getInnerQuery() const { return inner_query->clone(); } + ASTPtr getInnerSubQuery() const + { + if (inner_subquery) + return inner_subquery->clone(); + return nullptr; + } + ASTPtr getInnerBlocksQuery() const + { + if (inner_blocks_query) + return inner_blocks_query->clone(); + return nullptr; + } /// It is passed inside the query and solved at its level. bool supportsSampling() const override { return true; } @@ -127,8 +146,14 @@ public: unsigned num_streams) override; std::shared_ptr getBlocksPtr() { return blocks_ptr; } - BlocksPtrs getMergeableBlocks() { return mergeable_blocks; } - void setMergeableBlocks(BlocksPtrs blocks) { mergeable_blocks = blocks; } + MergeableBlocksPtr getMergeableBlocks() { return mergeable_blocks; } + + /// Collect mergeable blocks and their sample. Must be called holding mutex + MergeableBlocksPtr collectMergeableBlocks(const Context & context); + /// Complete query using input streams from mergeable blocks + BlockInputStreamPtr completeQuery(BlockInputStreams from); + + void setMergeableBlocks(MergeableBlocksPtr blocks) { mergeable_blocks = blocks; } std::shared_ptr getActivePtr() { return active_ptr; } /// Read new data blocks that store query result @@ -136,6 +161,9 @@ public: Block getHeader() const; + /// convert blocks to input streams + static BlockInputStreams blocksToInputStreams(BlocksPtrs blocks, Block & sample_block); + static void writeIntoLiveView( StorageLiveView & live_view, const Block & block, @@ -146,8 +174,13 @@ private: String select_table_name; String table_name; String database_name; - ASTPtr inner_query; + ASTPtr inner_query; /// stored query : SELECT * FROM ( SELECT a FROM A) + ASTPtr inner_subquery; /// stored query's innermost subquery if any + ASTPtr inner_blocks_query; /// query over the mergeable blocks to produce final result Context & global_context; + std::unique_ptr live_view_context; + StoragePtr parent_storage; + bool is_temporary = false; /// Mutex to protect access to sample block mutable std::mutex sample_block_lock; @@ -165,7 +198,7 @@ private: std::shared_ptr blocks_ptr; /// Current data blocks metadata std::shared_ptr blocks_metadata_ptr; - BlocksPtrs mergeable_blocks; + MergeableBlocksPtr mergeable_blocks; /// Background thread for temporary tables /// which drops this table if there are no users diff --git a/dbms/src/Storages/MarkCache.h b/dbms/src/Storages/MarkCache.h index 9ce04c01e43..6e36a941fff 100644 --- a/dbms/src/Storages/MarkCache.h +++ b/dbms/src/Storages/MarkCache.h @@ -38,8 +38,8 @@ private: using Base = LRUCache; public: - MarkCache(size_t max_size_in_bytes, const Delay & expiration_delay_) - : Base(max_size_in_bytes, expiration_delay_) {} + MarkCache(size_t max_size_in_bytes) + : Base(max_size_in_bytes) {} /// Calculate key from path to file and offset. static UInt128 hash(const String & path_to_file) diff --git a/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp b/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp index af43a0b8a6f..a519e2a4b71 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp @@ -25,6 +25,7 @@ MergeTreeBaseSelectProcessor::MergeTreeBaseSelectProcessor( UInt64 preferred_block_size_bytes_, UInt64 preferred_max_column_in_block_size_bytes_, UInt64 min_bytes_to_use_direct_io_, + UInt64 min_bytes_to_use_mmap_io_, UInt64 max_read_buffer_size_, bool use_uncompressed_cache_, bool save_marks_in_cache_, @@ -37,6 +38,7 @@ MergeTreeBaseSelectProcessor::MergeTreeBaseSelectProcessor( preferred_block_size_bytes(preferred_block_size_bytes_), preferred_max_column_in_block_size_bytes(preferred_max_column_in_block_size_bytes_), min_bytes_to_use_direct_io(min_bytes_to_use_direct_io_), + min_bytes_to_use_mmap_io(min_bytes_to_use_mmap_io_), max_read_buffer_size(max_read_buffer_size_), use_uncompressed_cache(use_uncompressed_cache_), save_marks_in_cache(save_marks_in_cache_), @@ -76,35 +78,23 @@ void MergeTreeBaseSelectProcessor::initializeRangeReaders(MergeTreeReadTask & cu { if (reader->getColumns().empty()) { - current_task.range_reader = MergeTreeRangeReader( - pre_reader.get(), nullptr, - prewhere_info->alias_actions, prewhere_info->prewhere_actions, - &prewhere_info->prewhere_column_name, - current_task.remove_prewhere_column, true); + current_task.range_reader = MergeTreeRangeReader(pre_reader.get(), nullptr, prewhere_info, true); } else { MergeTreeRangeReader * pre_reader_ptr = nullptr; if (pre_reader != nullptr) { - current_task.pre_range_reader = MergeTreeRangeReader( - pre_reader.get(), nullptr, - prewhere_info->alias_actions, prewhere_info->prewhere_actions, - &prewhere_info->prewhere_column_name, - current_task.remove_prewhere_column, false); + current_task.pre_range_reader = MergeTreeRangeReader(pre_reader.get(), nullptr, prewhere_info, false); pre_reader_ptr = ¤t_task.pre_range_reader; } - current_task.range_reader = MergeTreeRangeReader( - reader.get(), pre_reader_ptr, nullptr, nullptr, - nullptr, false, true); + current_task.range_reader = MergeTreeRangeReader(reader.get(), pre_reader_ptr, nullptr, true); } } else { - current_task.range_reader = MergeTreeRangeReader( - reader.get(), nullptr, nullptr, nullptr, - nullptr, false, true); + current_task.range_reader = MergeTreeRangeReader(reader.get(), nullptr, nullptr, true); } } @@ -333,6 +323,12 @@ void MergeTreeBaseSelectProcessor::executePrewhereActions(Block & block, const P prewhere_info->prewhere_actions->execute(block); if (prewhere_info->remove_prewhere_column) block.erase(prewhere_info->prewhere_column_name); + else + { + auto & ctn = block.getByName(prewhere_info->prewhere_column_name); + ctn.type = std::make_shared(); + ctn.column = ctn.type->createColumnConst(block.rows(), 1u)->convertToFullColumnIfConst(); + } if (!block) block.insert({nullptr, std::make_shared(), "_nothing"}); diff --git a/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h b/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h index 7f3367b74c8..ba6404bfb4e 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h +++ b/dbms/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h @@ -27,6 +27,7 @@ public: UInt64 preferred_block_size_bytes_, UInt64 preferred_max_column_in_block_size_bytes_, UInt64 min_bytes_to_use_direct_io_, + UInt64 min_bytes_to_use_mmap_io_, UInt64 max_read_buffer_size_, bool use_uncompressed_cache_, bool save_marks_in_cache_ = true, @@ -64,6 +65,7 @@ protected: UInt64 preferred_max_column_in_block_size_bytes; UInt64 min_bytes_to_use_direct_io; + UInt64 min_bytes_to_use_mmap_io; UInt64 max_read_buffer_size; bool use_uncompressed_cache; diff --git a/dbms/src/Storages/MergeTree/MergeTreeData.cpp b/dbms/src/Storages/MergeTree/MergeTreeData.cpp index d4451af3273..c10bd78afec 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeData.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeData.cpp @@ -3216,18 +3216,20 @@ ReservationPtr MergeTreeData::tryReserveSpace(UInt64 expected_size, SpacePtr spa ReservationPtr MergeTreeData::reserveSpacePreferringTTLRules(UInt64 expected_size, const MergeTreeDataPart::TTLInfos & ttl_infos, - time_t time_of_move) const + time_t time_of_move, + size_t min_volume_index) const { expected_size = std::max(RESERVATION_MIN_ESTIMATION_SIZE, expected_size); - ReservationPtr reservation = tryReserveSpacePreferringTTLRules(expected_size, ttl_infos, time_of_move); + ReservationPtr reservation = tryReserveSpacePreferringTTLRules(expected_size, ttl_infos, time_of_move, min_volume_index); return checkAndReturnReservation(expected_size, std::move(reservation)); } ReservationPtr MergeTreeData::tryReserveSpacePreferringTTLRules(UInt64 expected_size, const MergeTreeDataPart::TTLInfos & ttl_infos, - time_t time_of_move) const + time_t time_of_move, + size_t min_volume_index) const { expected_size = std::max(RESERVATION_MIN_ESTIMATION_SIZE, expected_size); @@ -3253,10 +3255,19 @@ ReservationPtr MergeTreeData::tryReserveSpacePreferringTTLRules(UInt64 expected_ reservation = destination_ptr->reserve(expected_size); if (reservation) return reservation; + else + if (ttl_entry->destination_type == PartDestinationType::VOLUME) + LOG_WARNING(log, "Would like to reserve space on volume '" + << ttl_entry->destination_name << "' by TTL rule of table '" + << log_name << "' but there is not enough space"); + else if (ttl_entry->destination_type == PartDestinationType::DISK) + LOG_WARNING(log, "Would like to reserve space on disk '" + << ttl_entry->destination_name << "' by TTL rule of table '" + << log_name << "' but there is not enough space"); } } - reservation = storage_policy->reserve(expected_size); + reservation = storage_policy->reserve(expected_size, min_volume_index); return reservation; } diff --git a/dbms/src/Storages/MergeTree/MergeTreeData.h b/dbms/src/Storages/MergeTree/MergeTreeData.h index 4fb09277b1e..f458fb3e7d3 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeData.h +++ b/dbms/src/Storages/MergeTree/MergeTreeData.h @@ -675,10 +675,12 @@ public: /// Reserves space at least 1MB preferring best destination according to `ttl_infos`. ReservationPtr reserveSpacePreferringTTLRules(UInt64 expected_size, const MergeTreeDataPart::TTLInfos & ttl_infos, - time_t time_of_move) const; + time_t time_of_move, + size_t min_volume_index = 0) const; ReservationPtr tryReserveSpacePreferringTTLRules(UInt64 expected_size, const MergeTreeDataPart::TTLInfos & ttl_infos, - time_t time_of_move) const; + time_t time_of_move, + size_t min_volume_index = 0) const; /// Choose disk with max available free space /// Reserves 0 bytes ReservationPtr makeEmptyReservationOnLargestDisk() { return storage_policy->makeEmptyReservationOnLargestDisk(); } diff --git a/dbms/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp b/dbms/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp index 5919e5a2670..05db73ce215 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp @@ -169,9 +169,12 @@ UInt64 MergeTreeDataMergerMutator::getMaxSourcePartSizeForMutation() const auto data_settings = data.getSettings(); size_t busy_threads_in_pool = CurrentMetrics::values[CurrentMetrics::BackgroundPoolTask].load(std::memory_order_relaxed); + /// DataPart can be store only at one disk. Get Max of free space at all disks + UInt64 disk_space = data.storage_policy->getMaxUnreservedFreeSpace(); + /// Allow mutations only if there are enough threads, leave free threads for merges else if (background_pool_size - busy_threads_in_pool >= data_settings->number_of_free_entries_in_pool_to_execute_mutation) - return static_cast(data.storage_policy->getMaxUnreservedFreeSpace() / DISK_USAGE_COEFFICIENT_TO_RESERVE); + return static_cast(disk_space / DISK_USAGE_COEFFICIENT_TO_RESERVE); return 0; } @@ -816,7 +819,7 @@ MergeTreeData::MutableDataPartPtr MergeTreeDataMergerMutator::mergePartsToTempor + ") differs from number of bytes written to rows_sources file (" + toString(rows_sources_count) + "). It is a bug.", ErrorCodes::LOGICAL_ERROR); - CompressedReadBufferFromFile rows_sources_read_buf(rows_sources_file_path, 0, 0); + CompressedReadBufferFromFile rows_sources_read_buf(rows_sources_file_path, 0, 0, 0); IMergedBlockOutputStream::WrittenOffsetColumns written_offset_columns; for (size_t column_num = 0, gathering_column_names_size = gathering_column_names.size(); diff --git a/dbms/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp b/dbms/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp index 36969ed18fe..09c4fe835d6 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp @@ -793,7 +793,8 @@ Pipes MergeTreeDataSelectExecutor::spreadMarkRangesAmongStreams( auto source = std::make_shared( data, part.data_part, max_block_size, settings.preferred_block_size_bytes, settings.preferred_max_column_in_block_size_bytes, column_names, part.ranges, use_uncompressed_cache, - query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.max_read_buffer_size, true, + query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.min_bytes_to_use_mmap_io, + settings.max_read_buffer_size, true, virt_columns, part.part_index_in_query); res.emplace_back(std::move(source)); @@ -973,7 +974,7 @@ Pipes MergeTreeDataSelectExecutor::spreadMarkRangesAmongStreamsWithOrder( pipes.emplace_back(std::make_shared( data, part.data_part, max_block_size, settings.preferred_block_size_bytes, settings.preferred_max_column_in_block_size_bytes, column_names, ranges_to_get_from_part, - use_uncompressed_cache, query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, + use_uncompressed_cache, query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.min_bytes_to_use_mmap_io, settings.max_read_buffer_size, true, virt_columns, part.part_index_in_query)); } else @@ -981,7 +982,7 @@ Pipes MergeTreeDataSelectExecutor::spreadMarkRangesAmongStreamsWithOrder( pipes.emplace_back(std::make_shared( data, part.data_part, max_block_size, settings.preferred_block_size_bytes, settings.preferred_max_column_in_block_size_bytes, column_names, ranges_to_get_from_part, - use_uncompressed_cache, query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, + use_uncompressed_cache, query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.min_bytes_to_use_mmap_io, settings.max_read_buffer_size, true, virt_columns, part.part_index_in_query)); pipes.back().addSimpleTransform(std::make_shared(pipes.back().getHeader())); @@ -1054,7 +1055,8 @@ Pipes MergeTreeDataSelectExecutor::spreadMarkRangesAmongStreamsFinal( auto source_processor = std::make_shared( data, part.data_part, max_block_size, settings.preferred_block_size_bytes, settings.preferred_max_column_in_block_size_bytes, column_names, part.ranges, use_uncompressed_cache, - query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.max_read_buffer_size, true, + query_info.prewhere_info, true, settings.min_bytes_to_use_direct_io, settings.min_bytes_to_use_mmap_io, + settings.max_read_buffer_size, true, virt_columns, part.part_index_in_query); Pipe pipe(std::move(source_processor)); diff --git a/dbms/src/Storages/MergeTree/MergeTreeIndexFullText.cpp b/dbms/src/Storages/MergeTree/MergeTreeIndexFullText.cpp index da3f1df8130..8041ad4dbe7 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeIndexFullText.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeIndexFullText.cpp @@ -636,15 +636,16 @@ bool SplitTokenExtractor::next(const char * data, size_t len, size_t * pos, size { if (isASCII(data[*pos]) && !isAlphaNumericASCII(data[*pos])) { + /// Finish current token if any if (*token_len > 0) return true; *token_start = ++*pos; } else { - const size_t sz = UTF8::seqLength(static_cast(data[*pos])); - *pos += sz; - *token_len += sz; + /// Note that UTF-8 sequence is completely consisted of non-ASCII bytes. + ++*pos; + ++*token_len; } } return *token_len > 0; diff --git a/dbms/src/Storages/MergeTree/MergeTreeIndexReader.cpp b/dbms/src/Storages/MergeTree/MergeTreeIndexReader.cpp index 05f09041fed..8b567b39304 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeIndexReader.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeIndexReader.cpp @@ -9,7 +9,7 @@ MergeTreeIndexReader::MergeTreeIndexReader( : index(index_), stream( part_->getFullPath() + index->getFileName(), ".idx", marks_count_, all_mark_ranges_, nullptr, false, nullptr, - part_->getFileSizeOrZero(index->getFileName() + ".idx"), 0, DBMS_DEFAULT_BUFFER_SIZE, + part_->getFileSizeOrZero(index->getFileName() + ".idx"), 0, 0, DBMS_DEFAULT_BUFFER_SIZE, &part_->index_granularity_info, ReadBufferFromFileBase::ProfileCallback{}, CLOCK_MONOTONIC_COARSE) { diff --git a/dbms/src/Storages/MergeTree/MergeTreeRangeReader.cpp b/dbms/src/Storages/MergeTree/MergeTreeRangeReader.cpp index d03160d7ec2..a09bd548b64 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeRangeReader.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeRangeReader.cpp @@ -255,6 +255,36 @@ void MergeTreeRangeReader::ReadResult::clear() filter = nullptr; } +void MergeTreeRangeReader::ReadResult::shrink(Columns & old_columns) +{ + for (size_t i = 0; i < old_columns.size(); ++i) + { + if (!old_columns[i]) + continue; + auto new_column = old_columns[i]->cloneEmpty(); + new_column->reserve(total_rows_per_granule); + for (size_t j = 0, pos = 0; j < rows_per_granule_original.size(); pos += rows_per_granule_original[i], ++j) + { + if (rows_per_granule[j]) + new_column->insertRangeFrom(*old_columns[i], pos, rows_per_granule[j]); + } + old_columns[i] = std::move(new_column); + } +} + +void MergeTreeRangeReader::ReadResult::setFilterConstTrue() +{ + clearFilter(); + filter_holder = DataTypeUInt8().createColumnConst(num_rows, 1u); +} + +void MergeTreeRangeReader::ReadResult::setFilterConstFalse() +{ + clearFilter(); + columns.clear(); + num_rows = 0; +} + void MergeTreeRangeReader::ReadResult::optimize() { if (total_rows_per_granule == 0 || filter == nullptr) @@ -268,30 +298,47 @@ void MergeTreeRangeReader::ReadResult::optimize() clear(); return; } - else if (total_zero_rows_in_tails == 0 && countBytesInFilter(filter->getData()) == filter->size()) + else if (total_zero_rows_in_tails == 0 && countBytesInResultFilter(filter->getData()) == filter->size()) { - filter_holder = nullptr; - filter = nullptr; + setFilterConstTrue(); return; } - /// Just a guess. If only a few rows may be skipped, it's better not to skip at all. - if (2 * total_zero_rows_in_tails > filter->size()) + else if (2 * total_zero_rows_in_tails > filter->size()) { + for (auto i : ext::range(0, rows_per_granule.size())) + { + rows_per_granule_original.push_back(rows_per_granule[i]); + rows_per_granule[i] -= zero_tails[i]; + } + num_rows_to_skip_in_last_granule += rows_per_granule_original.back() - rows_per_granule.back(); - auto new_filter = ColumnUInt8::create(filter->size() - total_zero_rows_in_tails); - IColumn::Filter & new_data = new_filter->getData(); + /// Check if const 1 after shrink + if (countBytesInResultFilter(filter->getData()) + total_zero_rows_in_tails == total_rows_per_granule) + { + total_rows_per_granule = total_rows_per_granule - total_zero_rows_in_tails; + num_rows = total_rows_per_granule; + setFilterConstTrue(); + shrink(columns); /// shrink acts as filtering in such case + } + else + { + auto new_filter = ColumnUInt8::create(filter->size() - total_zero_rows_in_tails); + IColumn::Filter & new_data = new_filter->getData(); - size_t rows_in_last_granule = rows_per_granule.back(); - - collapseZeroTails(filter->getData(), new_data, zero_tails); - - total_rows_per_granule = new_filter->size(); - num_rows_to_skip_in_last_granule += rows_in_last_granule - rows_per_granule.back(); - - filter = new_filter.get(); - filter_holder = std::move(new_filter); + collapseZeroTails(filter->getData(), new_data); + total_rows_per_granule = new_filter->size(); + num_rows = total_rows_per_granule; + filter_original = filter; + filter_holder_original = std::move(filter_holder); + filter = new_filter.get(); + filter_holder = std::move(new_filter); + } + need_filter = true; } + /// Another guess, if it's worth filtering at PREWHERE + else if (countBytesInResultFilter(filter->getData()) < 0.6 * filter->size()) + need_filter = true; } size_t MergeTreeRangeReader::ReadResult::countZeroTails(const IColumn::Filter & filter_vec, NumRows & zero_tails) const @@ -314,24 +361,16 @@ size_t MergeTreeRangeReader::ReadResult::countZeroTails(const IColumn::Filter & return total_zero_rows_in_tails; } -void MergeTreeRangeReader::ReadResult::collapseZeroTails(const IColumn::Filter & filter_vec, IColumn::Filter & new_filter_vec, - const NumRows & zero_tails) +void MergeTreeRangeReader::ReadResult::collapseZeroTails(const IColumn::Filter & filter_vec, IColumn::Filter & new_filter_vec) { auto filter_data = filter_vec.data(); auto new_filter_data = new_filter_vec.data(); for (auto i : ext::range(0, rows_per_granule.size())) { - auto & rows_to_read = rows_per_granule[i]; - auto filtered_rows_num_at_granule_end = zero_tails[i]; - - rows_to_read -= filtered_rows_num_at_granule_end; - - memcpySmallAllowReadWriteOverflow15(new_filter_data, filter_data, rows_to_read); - filter_data += rows_to_read; - new_filter_data += rows_to_read; - - filter_data += filtered_rows_num_at_granule_end; + memcpySmallAllowReadWriteOverflow15(new_filter_data, filter_data, rows_per_granule[i]); + filter_data += rows_per_granule_original[i]; + new_filter_data += rows_per_granule[i]; } new_filter_vec.resize(new_filter_data - new_filter_vec.data()); @@ -405,15 +444,27 @@ void MergeTreeRangeReader::ReadResult::setFilter(const ColumnPtr & new_filter) } +size_t MergeTreeRangeReader::ReadResult::countBytesInResultFilter(const IColumn::Filter & filter_) +{ + auto it = filter_bytes_map.find(&filter_); + if (it == filter_bytes_map.end()) + { + auto bytes = countBytesInFilter(filter_); + filter_bytes_map[&filter_] = bytes; + return bytes; + } + else + return it->second; +} + MergeTreeRangeReader::MergeTreeRangeReader( - MergeTreeReader * merge_tree_reader_, MergeTreeRangeReader * prev_reader_, - ExpressionActionsPtr alias_actions_, ExpressionActionsPtr prewhere_actions_, - const String * prewhere_column_name_, bool remove_prewhere_column_, bool last_reader_in_chain_) - : merge_tree_reader(merge_tree_reader_), index_granularity(&(merge_tree_reader->data_part->index_granularity)) - , prev_reader(prev_reader_), prewhere_column_name(prewhere_column_name_) - , alias_actions(std::move(alias_actions_)), prewhere_actions(std::move(prewhere_actions_)) - , remove_prewhere_column(remove_prewhere_column_) - , last_reader_in_chain(last_reader_in_chain_), is_initialized(true) + MergeTreeReader * merge_tree_reader_, + MergeTreeRangeReader * prev_reader_, + const PrewhereInfoPtr & prewhere_, + bool last_reader_in_chain_) + : merge_tree_reader(merge_tree_reader_) + , index_granularity(&(merge_tree_reader->data_part->index_granularity)), prev_reader(prev_reader_) + , prewhere(prewhere_), last_reader_in_chain(last_reader_in_chain_), is_initialized(true) { if (prev_reader) sample_block = prev_reader->getSampleBlock(); @@ -421,14 +472,18 @@ MergeTreeRangeReader::MergeTreeRangeReader( for (auto & name_and_type : merge_tree_reader->getColumns()) sample_block.insert({name_and_type.type->createColumn(), name_and_type.type, name_and_type.name}); - if (alias_actions) - alias_actions->execute(sample_block, true); + if (prewhere) + { + if (prewhere->alias_actions) + prewhere->alias_actions->execute(sample_block, true); - if (prewhere_actions) - prewhere_actions->execute(sample_block, true); + sample_block_before_prewhere = sample_block; + if (prewhere->prewhere_actions) + prewhere->prewhere_actions->execute(sample_block, true); - if (remove_prewhere_column) - sample_block.erase(*prewhere_column_name); + if (prewhere->remove_prewhere_column) + sample_block.erase(prewhere->prewhere_column_name); + } } bool MergeTreeRangeReader::isReadingFinished() const @@ -488,12 +543,10 @@ MergeTreeRangeReader::ReadResult MergeTreeRangeReader::read(size_t max_rows, Mar throw Exception("Expected at least 1 row to read, got 0.", ErrorCodes::LOGICAL_ERROR); ReadResult read_result; - size_t prev_bytes = 0; if (prev_reader) { read_result = prev_reader->read(max_rows, ranges); - prev_bytes = read_result.numBytesRead(); size_t num_read_rows; Columns columns = continueReadingChain(read_result, num_read_rows); @@ -509,6 +562,15 @@ MergeTreeRangeReader::ReadResult MergeTreeRangeReader::read(size_t max_rows, Mar has_columns = true; } + size_t total_bytes = 0; + for (auto & column : columns) + { + if (column) + total_bytes += column->byteSize(); + } + + read_result.addNumBytesRead(total_bytes); + bool should_evaluate_missing_defaults = false; if (has_columns) @@ -533,8 +595,30 @@ MergeTreeRangeReader::ReadResult MergeTreeRangeReader::read(size_t max_rows, Mar } if (!columns.empty() && should_evaluate_missing_defaults) - merge_tree_reader->evaluateMissingDefaults( - prev_reader->getSampleBlock().cloneWithColumns(read_result.columns), columns); + { + auto block = prev_reader->sample_block.cloneWithColumns(read_result.columns); + auto block_before_prewhere = read_result.block_before_prewhere; + for (auto & ctn : block) + { + if (block_before_prewhere.has(ctn.name)) + block_before_prewhere.erase(ctn.name); + } + + if (block_before_prewhere) + { + if (read_result.need_filter) + { + auto old_columns = block_before_prewhere.getColumns(); + filterColumns(old_columns, read_result.getFilter()->getData()); + block_before_prewhere.setColumns(std::move(old_columns)); + } + + for (auto && ctn : block_before_prewhere) + block.insert(std::move(ctn)); + } + + merge_tree_reader->evaluateMissingDefaults(block, columns); + } read_result.columns.reserve(read_result.columns.size() + columns.size()); for (auto & column : columns) @@ -556,17 +640,17 @@ MergeTreeRangeReader::ReadResult MergeTreeRangeReader::read(size_t max_rows, Mar } else read_result.columns.clear(); + + size_t total_bytes = 0; + for (auto & column : read_result.columns) + total_bytes += column->byteSize(); + + read_result.addNumBytesRead(total_bytes); } if (read_result.num_rows == 0) return read_result; - size_t total_bytes = 0; - for (auto & column : read_result.columns) - total_bytes += column->byteSize(); - - read_result.addNumBytesRead(total_bytes - prev_bytes); - executePrewhereActionsAndFilterColumns(read_result); return read_result; @@ -674,7 +758,7 @@ Columns MergeTreeRangeReader::continueReadingChain(ReadResult & result, size_t & void MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(ReadResult & result) { - if (!prewhere_actions) + if (!prewhere) return; auto & header = merge_tree_reader->getColumns(); @@ -705,12 +789,14 @@ void MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(ReadResult & r for (auto name_and_type = header.begin(); pos < num_columns; ++pos, ++name_and_type) block.insert({result.columns[pos], name_and_type->type, name_and_type->name}); - if (alias_actions) - alias_actions->execute(block); + if (prewhere && prewhere->alias_actions) + prewhere->alias_actions->execute(block); - prewhere_actions->execute(block); + /// Columns might be projected out. We need to store them here so that default columns can be evaluated later. + result.block_before_prewhere = block; + prewhere->prewhere_actions->execute(block); - prewhere_column_pos = block.getPositionByName(*prewhere_column_name); + prewhere_column_pos = block.getPositionByName(prewhere->prewhere_column_name); result.columns.clear(); result.columns.reserve(block.columns()); @@ -729,51 +815,38 @@ void MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(ReadResult & r } result.setFilter(filter); + + /// If there is a WHERE, we filter in there, and only optimize IO and shrink columns here if (!last_reader_in_chain) result.optimize(); - bool filter_always_true = !result.getFilter() && result.totalRowsPerGranule() == filter->size(); - + /// If we read nothing or filter gets optimized to nothing if (result.totalRowsPerGranule() == 0) + result.setFilterConstFalse(); + /// If we need to filter in PREWHERE + else if (prewhere->need_filter || result.need_filter) { - result.columns.clear(); - result.num_rows = 0; - } - else if (!filter_always_true) - { - FilterDescription filter_description(*filter); - - size_t num_bytes_in_filter = 0; - bool calculated_num_bytes_in_filter = false; - - auto getNumBytesInFilter = [&]() + /// If there is a filter and without optimized + if (result.getFilter() && last_reader_in_chain) { - if (!calculated_num_bytes_in_filter) - num_bytes_in_filter = countBytesInFilter(*filter_description.data); - - calculated_num_bytes_in_filter = true; - return num_bytes_in_filter; - }; - - if (last_reader_in_chain) - { - size_t bytes_in_filter = getNumBytesInFilter(); + auto result_filter = result.getFilter(); + /// optimize is not called, need to check const 1 and const 0 + size_t bytes_in_filter = result.countBytesInResultFilter(result_filter->getData()); if (bytes_in_filter == 0) - { - result.columns.clear(); - result.num_rows = 0; - } - else if (bytes_in_filter == filter->size()) - filter_always_true = true; + result.setFilterConstFalse(); + else if (bytes_in_filter == result.num_rows) + result.setFilterConstTrue(); } - if (!filter_always_true) + /// If there is still a filter, do the filtering now + if (result.getFilter()) { - filterColumns(result.columns, *filter_description.data); + /// filter might be shrinked while columns not + auto result_filter = result.getFilterOriginal() ? result.getFilterOriginal() : result.getFilter(); + filterColumns(result.columns, result_filter->getData()); + result.need_filter = true; - /// Get num rows after filtration. bool has_column = false; - for (auto & column : result.columns) { if (column) @@ -784,19 +857,26 @@ void MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(ReadResult & r } } + /// There is only one filter column. Record the actual number if (!has_column) - result.num_rows = getNumBytesInFilter(); + result.num_rows = result.countBytesInResultFilter(result_filter->getData()); + } + + /// Check if the PREWHERE column is needed + if (result.columns.size()) + { + if (prewhere->remove_prewhere_column) + result.columns.erase(result.columns.begin() + prewhere_column_pos); + else + result.columns[prewhere_column_pos] = DataTypeUInt8().createColumnConst(result.num_rows, 1u)->convertToFullColumnIfConst(); } } - - if (result.num_rows == 0) - return; - - if (remove_prewhere_column) - result.columns.erase(result.columns.begin() + prewhere_column_pos); + /// Filter in WHERE instead else - result.columns[prewhere_column_pos] = - DataTypeUInt8().createColumnConst(result.num_rows, 1u)->convertToFullColumnIfConst(); + { + result.columns[prewhere_column_pos] = result.getFilterHolder()->convertToFullColumnIfConst(); + result.clearFilter(); // Acting as a flag to not filter in PREWHERE + } } } diff --git a/dbms/src/Storages/MergeTree/MergeTreeRangeReader.h b/dbms/src/Storages/MergeTree/MergeTreeRangeReader.h index 67d5cbc3908..345f537d2aa 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeRangeReader.h +++ b/dbms/src/Storages/MergeTree/MergeTreeRangeReader.h @@ -13,6 +13,8 @@ using ColumnUInt8 = ColumnVector; class MergeTreeReader; class MergeTreeIndexGranularity; +struct PrewhereInfo; +using PrewhereInfoPtr = std::shared_ptr; /// MergeTreeReader iterator which allows sequential reading for arbitrary number of rows between pairs of marks in the same part. /// Stores reading state, which can be inside granule. Can skip rows in current granule and start reading from next mark. @@ -20,9 +22,11 @@ class MergeTreeIndexGranularity; class MergeTreeRangeReader { public: - MergeTreeRangeReader(MergeTreeReader * merge_tree_reader_, MergeTreeRangeReader * prev_reader_, - ExpressionActionsPtr alias_actions_, ExpressionActionsPtr prewhere_actions_, - const String * prewhere_column_name_, bool remove_prewhere_column_, bool last_reader_in_chain_); + MergeTreeRangeReader( + MergeTreeReader * merge_tree_reader_, + MergeTreeRangeReader * prev_reader_, + const PrewhereInfoPtr & prewhere_, + bool last_reader_in_chain_); MergeTreeRangeReader() = default; @@ -140,7 +144,9 @@ public: /// The number of bytes read from disk. size_t numBytesRead() const { return num_bytes_read; } /// Filter you need to apply to newly-read columns in order to add them to block. + const ColumnUInt8 * getFilterOriginal() const { return filter_original; } const ColumnUInt8 * getFilter() const { return filter; } + ColumnPtr & getFilterHolder() { return filter_holder; } void addGranule(size_t num_rows_); void adjustLastGranule(); @@ -154,10 +160,21 @@ public: /// Remove all rows from granules. void clear(); + void clearFilter() { filter = nullptr; } + void setFilterConstTrue(); + void setFilterConstFalse(); + void addNumBytesRead(size_t count) { num_bytes_read += count; } + void shrink(Columns & old_columns); + + size_t countBytesInResultFilter(const IColumn::Filter & filter); + Columns columns; size_t num_rows = 0; + bool need_filter = false; + + Block block_before_prewhere; private: RangesInfo started_ranges; @@ -165,6 +182,7 @@ public: /// Granule here is not number of rows between two marks /// It's amount of rows per single reading act NumRows rows_per_granule; + NumRows rows_per_granule_original; /// Sum(rows_per_granule) size_t total_rows_per_granule = 0; /// The number of rows was read at first step. May be zero if no read columns present in part. @@ -175,11 +193,15 @@ public: size_t num_bytes_read = 0; /// nullptr if prev reader hasn't prewhere_actions. Otherwise filter.size() >= total_rows_per_granule. ColumnPtr filter_holder; + ColumnPtr filter_holder_original; const ColumnUInt8 * filter = nullptr; + const ColumnUInt8 * filter_original = nullptr; - void collapseZeroTails(const IColumn::Filter & filter, IColumn::Filter & new_filter, const NumRows & zero_tails); + void collapseZeroTails(const IColumn::Filter & filter, IColumn::Filter & new_filter); size_t countZeroTails(const IColumn::Filter & filter, NumRows & zero_tails) const; static size_t numZerosInTail(const UInt8 * begin, const UInt8 * end); + + std::map filter_bytes_map; }; ReadResult read(size_t max_rows, MarkRanges & ranges); @@ -196,16 +218,13 @@ private: MergeTreeReader * merge_tree_reader = nullptr; const MergeTreeIndexGranularity * index_granularity = nullptr; MergeTreeRangeReader * prev_reader = nullptr; /// If not nullptr, read from prev_reader firstly. - - const String * prewhere_column_name = nullptr; - ExpressionActionsPtr alias_actions = nullptr; /// If not nullptr, calculate aliases. - ExpressionActionsPtr prewhere_actions = nullptr; /// If not nullptr, calculate filter. + PrewhereInfoPtr prewhere; Stream stream; Block sample_block; + Block sample_block_before_prewhere; - bool remove_prewhere_column = false; bool last_reader_in_chain = false; bool is_initialized = false; }; diff --git a/dbms/src/Storages/MergeTree/MergeTreeReader.cpp b/dbms/src/Storages/MergeTree/MergeTreeReader.cpp index 29d1dac7587..72c31a9dcb1 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReader.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeReader.cpp @@ -1,7 +1,6 @@ #include #include #include -#include #include #include #include @@ -40,6 +39,7 @@ MergeTreeReader::MergeTreeReader( const MergeTreeData & storage_, MarkRanges all_mark_ranges_, size_t aio_threshold_, + size_t mmap_threshold_, size_t max_read_buffer_size_, ValueSizeMap avg_value_size_hints_, const ReadBufferFromFileBase::ProfileCallback & profile_callback_, @@ -53,6 +53,7 @@ MergeTreeReader::MergeTreeReader( , storage(storage_) , all_mark_ranges(std::move(all_mark_ranges_)) , aio_threshold(aio_threshold_) + , mmap_threshold(mmap_threshold_) , max_read_buffer_size(max_read_buffer_size_) { try @@ -198,7 +199,7 @@ void MergeTreeReader::addStreams(const String & name, const IDataType & type, path + stream_name, DATA_FILE_EXTENSION, data_part->getMarksCount(), all_mark_ranges, mark_cache, save_marks_in_cache, uncompressed_cache, data_part->getFileSizeOrZero(stream_name + DATA_FILE_EXTENSION), - aio_threshold, max_read_buffer_size, + aio_threshold, mmap_threshold, max_read_buffer_size, &data_part->index_granularity_info, profile_callback, clock_type)); }; diff --git a/dbms/src/Storages/MergeTree/MergeTreeReader.h b/dbms/src/Storages/MergeTree/MergeTreeReader.h index 140fbcb51b0..b0642c06108 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReader.h +++ b/dbms/src/Storages/MergeTree/MergeTreeReader.h @@ -28,6 +28,7 @@ public: const MergeTreeData & storage_, MarkRanges all_mark_ranges_, size_t aio_threshold_, + size_t mmap_threshold_, size_t max_read_buffer_size_, ValueSizeMap avg_value_size_hints_ = ValueSizeMap{}, const ReadBufferFromFileBase::ProfileCallback & profile_callback_ = ReadBufferFromFileBase::ProfileCallback{}, @@ -81,6 +82,7 @@ private: const MergeTreeData & storage; MarkRanges all_mark_ranges; size_t aio_threshold; + size_t mmap_threshold; size_t max_read_buffer_size; void addStreams(const String & name, const IDataType & type, diff --git a/dbms/src/Storages/MergeTree/MergeTreeReaderStream.cpp b/dbms/src/Storages/MergeTree/MergeTreeReaderStream.cpp index 3dbfc61a00b..1691f01d794 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReaderStream.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeReaderStream.cpp @@ -1,4 +1,5 @@ #include +#include #include @@ -19,7 +20,7 @@ MergeTreeReaderStream::MergeTreeReaderStream( const MarkRanges & all_mark_ranges, MarkCache * mark_cache_, bool save_marks_in_cache_, UncompressedCache * uncompressed_cache, - size_t file_size, size_t aio_threshold, size_t max_read_buffer_size, + size_t file_size, size_t aio_threshold, size_t mmap_threshold, size_t max_read_buffer_size, const MergeTreeIndexGranularityInfo * index_granularity_info_, const ReadBufferFromFileBase::ProfileCallback & profile_callback, clockid_t clock_type) : path_prefix(path_prefix_), data_file_extension(data_file_extension_), marks_count(marks_count_) @@ -79,7 +80,7 @@ MergeTreeReaderStream::MergeTreeReaderStream( if (uncompressed_cache) { auto buffer = std::make_unique( - path_prefix + data_file_extension, uncompressed_cache, sum_mark_range_bytes, aio_threshold, buffer_size); + path_prefix + data_file_extension, uncompressed_cache, sum_mark_range_bytes, aio_threshold, mmap_threshold, buffer_size); if (profile_callback) buffer->setProfileCallback(profile_callback, clock_type); @@ -90,7 +91,7 @@ MergeTreeReaderStream::MergeTreeReaderStream( else { auto buffer = std::make_unique( - path_prefix + data_file_extension, sum_mark_range_bytes, aio_threshold, buffer_size); + path_prefix + data_file_extension, sum_mark_range_bytes, aio_threshold, mmap_threshold, buffer_size); if (profile_callback) buffer->setProfileCallback(profile_callback, clock_type); diff --git a/dbms/src/Storages/MergeTree/MergeTreeReaderStream.h b/dbms/src/Storages/MergeTree/MergeTreeReaderStream.h index d60689f1e91..1b0a973797b 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReaderStream.h +++ b/dbms/src/Storages/MergeTree/MergeTreeReaderStream.h @@ -15,13 +15,13 @@ class MergeTreeReaderStream { public: MergeTreeReaderStream( - const String & path_prefix_, const String & data_file_extension_, size_t marks_count_, - const MarkRanges & all_mark_ranges, - MarkCache * mark_cache, bool save_marks_in_cache, - UncompressedCache * uncompressed_cache, - size_t file_size, size_t aio_threshold, size_t max_read_buffer_size, - const MergeTreeIndexGranularityInfo * index_granularity_info_, - const ReadBufferFromFileBase::ProfileCallback & profile_callback, clockid_t clock_type); + const String & path_prefix_, const String & data_file_extension_, size_t marks_count_, + const MarkRanges & all_mark_ranges, + MarkCache * mark_cache, bool save_marks_in_cache, + UncompressedCache * uncompressed_cache, + size_t file_size, size_t aio_threshold, size_t mmap_threshold, size_t max_read_buffer_size, + const MergeTreeIndexGranularityInfo * index_granularity_info_, + const ReadBufferFromFileBase::ProfileCallback & profile_callback, clockid_t clock_type); void seekToMark(size_t index); diff --git a/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp b/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp index af8c02318d7..25425b62aa9 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp @@ -43,6 +43,7 @@ MergeTreeReverseSelectProcessor::MergeTreeReverseSelectProcessor( const PrewhereInfoPtr & prewhere_info_, bool check_columns, size_t min_bytes_to_use_direct_io_, + size_t min_bytes_to_use_mmap_io_, size_t max_read_buffer_size_, bool save_marks_in_cache_, const Names & virt_column_names_, @@ -52,7 +53,7 @@ MergeTreeReverseSelectProcessor::MergeTreeReverseSelectProcessor( MergeTreeBaseSelectProcessor{ replaceTypes(storage_.getSampleBlockForColumns(required_columns_), owned_data_part_), storage_, prewhere_info_, max_block_size_rows_, - preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, min_bytes_to_use_direct_io_, + preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, min_bytes_to_use_direct_io_, min_bytes_to_use_mmap_io_, max_read_buffer_size_, use_uncompressed_cache_, save_marks_in_cache_, virt_column_names_}, required_columns{std::move(required_columns_)}, data_part{owned_data_part_}, @@ -93,13 +94,13 @@ MergeTreeReverseSelectProcessor::MergeTreeReverseSelectProcessor( reader = std::make_unique( path, data_part, task_columns.columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, storage, - all_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size); + all_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size); if (prewhere_info) pre_reader = std::make_unique( path, data_part, task_columns.pre_columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, storage, - all_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size); + all_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size); } bool MergeTreeReverseSelectProcessor::getNewTask() @@ -111,7 +112,7 @@ try return false; } - /// We have some blocks to return in buffer. + /// We have some blocks to return in buffer. /// Return true to continue reading, but actually don't create a task. if (all_mark_ranges.empty()) return true; diff --git a/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h b/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h index 58202988e4c..9c37e60ab10 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h +++ b/dbms/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h @@ -28,6 +28,7 @@ public: const PrewhereInfoPtr & prewhere_info, bool check_columns, size_t min_bytes_to_use_direct_io, + size_t min_bytes_to_use_mmap_io, size_t max_read_buffer_size, bool save_marks_in_cache, const Names & virt_column_names = {}, diff --git a/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp b/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp index 51ed337367d..dac42859eef 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp @@ -43,6 +43,7 @@ MergeTreeSelectProcessor::MergeTreeSelectProcessor( const PrewhereInfoPtr & prewhere_info_, bool check_columns_, size_t min_bytes_to_use_direct_io_, + size_t min_bytes_to_use_mmap_io_, size_t max_read_buffer_size_, bool save_marks_in_cache_, const Names & virt_column_names_, @@ -52,7 +53,7 @@ MergeTreeSelectProcessor::MergeTreeSelectProcessor( MergeTreeBaseSelectProcessor{ replaceTypes(storage_.getSampleBlockForColumns(required_columns_), owned_data_part_), storage_, prewhere_info_, max_block_size_rows_, - preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, min_bytes_to_use_direct_io_, + preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, min_bytes_to_use_direct_io_, min_bytes_to_use_mmap_io_, max_read_buffer_size_, use_uncompressed_cache_, save_marks_in_cache_, virt_column_names_}, required_columns{std::move(required_columns_)}, data_part{owned_data_part_}, @@ -122,13 +123,13 @@ try reader = std::make_unique( path, data_part, task_columns.columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, storage, - all_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size); + all_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size); if (prewhere_info) pre_reader = std::make_unique( path, data_part, task_columns.pre_columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, storage, - all_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size); + all_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size); } return true; diff --git a/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.h b/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.h index c0d93842a81..9b0aac9bab1 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.h +++ b/dbms/src/Storages/MergeTree/MergeTreeSelectProcessor.h @@ -28,6 +28,7 @@ public: const PrewhereInfoPtr & prewhere_info, bool check_columns, size_t min_bytes_to_use_direct_io, + size_t min_bytes_to_use_mmap_io, size_t max_read_buffer_size, bool save_marks_in_cache, const Names & virt_column_names = {}, diff --git a/dbms/src/Storages/MergeTree/MergeTreeSequentialBlockInputStream.cpp b/dbms/src/Storages/MergeTree/MergeTreeSequentialBlockInputStream.cpp index 081ad289d28..372c29a3ac3 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeSequentialBlockInputStream.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeSequentialBlockInputStream.cpp @@ -57,7 +57,7 @@ MergeTreeSequentialBlockInputStream::MergeTreeSequentialBlockInputStream( MarkRanges{MarkRange(0, data_part->getMarksCount())}, /* bytes to use AIO (this is hack) */ read_with_direct_io ? 1UL : std::numeric_limits::max(), - DBMS_DEFAULT_BUFFER_SIZE); + 0, DBMS_DEFAULT_BUFFER_SIZE); } diff --git a/dbms/src/Storages/MergeTree/MergeTreeThreadSelectBlockInputProcessor.cpp b/dbms/src/Storages/MergeTree/MergeTreeThreadSelectBlockInputProcessor.cpp index cc090833f1e..ff2c058ede8 100644 --- a/dbms/src/Storages/MergeTree/MergeTreeThreadSelectBlockInputProcessor.cpp +++ b/dbms/src/Storages/MergeTree/MergeTreeThreadSelectBlockInputProcessor.cpp @@ -20,10 +20,11 @@ MergeTreeThreadSelectBlockInputProcessor::MergeTreeThreadSelectBlockInputProcess const Settings & settings, const Names & virt_column_names_) : - MergeTreeBaseSelectProcessor{pool_->getHeader(), storage_, prewhere_info_, max_block_size_rows_, - preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, - settings.min_bytes_to_use_direct_io, settings.max_read_buffer_size, - use_uncompressed_cache_, true, virt_column_names_}, + MergeTreeBaseSelectProcessor{ + pool_->getHeader(), storage_, prewhere_info_, max_block_size_rows_, + preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, + settings.min_bytes_to_use_direct_io, settings.min_bytes_to_use_mmap_io, settings.max_read_buffer_size, + use_uncompressed_cache_, true, virt_column_names_}, thread{thread_}, pool{pool_} { @@ -73,12 +74,12 @@ bool MergeTreeThreadSelectBlockInputProcessor::getNewTask() reader = std::make_unique( path, task->data_part, task->columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, - storage, rest_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size, MergeTreeReader::ValueSizeMap{}, profile_callback); + storage, rest_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size, MergeTreeReader::ValueSizeMap{}, profile_callback); if (prewhere_info) pre_reader = std::make_unique( path, task->data_part, task->pre_columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, - storage, rest_mark_ranges, min_bytes_to_use_direct_io, + storage, rest_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size, MergeTreeReader::ValueSizeMap{}, profile_callback); } else @@ -90,13 +91,13 @@ bool MergeTreeThreadSelectBlockInputProcessor::getNewTask() /// retain avg_value_size_hints reader = std::make_unique( path, task->data_part, task->columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, - storage, rest_mark_ranges, min_bytes_to_use_direct_io, max_read_buffer_size, + storage, rest_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size, reader->getAvgValueSizeHints(), profile_callback); if (prewhere_info) pre_reader = std::make_unique( path, task->data_part, task->pre_columns, owned_uncompressed_cache.get(), owned_mark_cache.get(), save_marks_in_cache, - storage, rest_mark_ranges, min_bytes_to_use_direct_io, + storage, rest_mark_ranges, min_bytes_to_use_direct_io, min_bytes_to_use_mmap_io, max_read_buffer_size, pre_reader->getAvgValueSizeHints(), profile_callback); } } diff --git a/dbms/src/Storages/SelectQueryInfo.h b/dbms/src/Storages/SelectQueryInfo.h index 11907151575..84cf3a32aa1 100644 --- a/dbms/src/Storages/SelectQueryInfo.h +++ b/dbms/src/Storages/SelectQueryInfo.h @@ -20,6 +20,7 @@ struct PrewhereInfo ExpressionActionsPtr remove_columns_actions; String prewhere_column_name; bool remove_prewhere_column = false; + bool need_filter = false; PrewhereInfo() = default; explicit PrewhereInfo(ExpressionActionsPtr prewhere_actions_, String prewhere_column_name_) diff --git a/dbms/src/Storages/StorageFile.cpp b/dbms/src/Storages/StorageFile.cpp index 64a603717e2..a15d7ba414c 100644 --- a/dbms/src/Storages/StorageFile.cpp +++ b/dbms/src/Storages/StorageFile.cpp @@ -23,6 +23,8 @@ #include #include +#include +#include #include #include @@ -39,6 +41,7 @@ namespace ErrorCodes { extern const int CANNOT_WRITE_TO_FILE_DESCRIPTOR; extern const int CANNOT_SEEK_THROUGH_FILE; + extern const int CANNOT_TRUNCATE_FILE; extern const int DATABASE_ACCESS_DENIED; extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; extern const int UNKNOWN_IDENTIFIER; @@ -61,11 +64,12 @@ static std::vector listFilesWithRegexpMatching(const std::string & const std::string suffix_with_globs = for_match.substr(end_of_path_without_globs); /// begin with '/' const size_t next_slash = suffix_with_globs.find('/', 1); - re2::RE2 matcher(makeRegexpPatternFromGlobs(suffix_with_globs.substr(0, next_slash))); + auto regexp = makeRegexpPatternFromGlobs(suffix_with_globs.substr(0, next_slash)); + re2::RE2 matcher(regexp); std::vector result; const std::string prefix_without_globs = path_for_ls + for_match.substr(1, end_of_path_without_globs); - if (!fs::exists(fs::path(prefix_without_globs.data()))) + if (!fs::exists(fs::path(prefix_without_globs))) { return result; } @@ -110,11 +114,11 @@ static void checkCreationIsAllowed(const Context & context_global, const std::st /// "/dev/null" is allowed for perf testing if (!startsWith(table_path, db_dir_path) && table_path != "/dev/null") - throw Exception("Part path " + table_path + " is not inside " + db_dir_path, ErrorCodes::DATABASE_ACCESS_DENIED); + throw Exception("File is not inside " + db_dir_path, ErrorCodes::DATABASE_ACCESS_DENIED); Poco::File table_path_poco_file = Poco::File(table_path); if (table_path_poco_file.exists() && table_path_poco_file.isDirectory()) - throw Exception("File " + table_path + " must not be a directory", ErrorCodes::INCORRECT_FILE_NAME); + throw Exception("File must not be a directory", ErrorCodes::INCORRECT_FILE_NAME); } } @@ -145,11 +149,10 @@ StorageFile::StorageFile(const std::string & table_path_, const std::string & us const std::string path = poco_path.absolute().toString(); if (path.find_first_of("*?{") == std::string::npos) - { paths.push_back(path); - } else paths = listFilesWithRegexpMatching("/", path); + for (const auto & cur_path : paths) checkCreationIsAllowed(args.context, user_files_absolute_path, cur_path); } @@ -200,12 +203,12 @@ public: } storage->table_fd_was_used = true; - read_buf = getReadBuffer(compression_method, storage->table_fd); + read_buf = wrapReadBufferWithCompressionMethod(std::make_unique(storage->table_fd), compression_method); } else { shared_lock = std::shared_lock(storage->rwlock); - read_buf = getReadBuffer(compression_method, file_path); + read_buf = wrapReadBufferWithCompressionMethod(std::make_unique(file_path), compression_method); } reader = FormatFactory::instance().getInput(storage->format_name, *read_buf, storage->getSampleBlock(), context, max_block_size); @@ -266,7 +269,7 @@ BlockInputStreams StorageFile::read( for (const auto & file_path : paths) { BlockInputStreamPtr cur_block = std::make_shared( - std::static_pointer_cast(shared_from_this()), context, max_block_size, file_path, IStorage::chooseCompressionMethod(file_path, compression_method)); + std::static_pointer_cast(shared_from_this()), context, max_block_size, file_path, chooseCompressionMethod(file_path, compression_method)); blocks_input.push_back(column_defaults.empty() ? cur_block : std::make_shared(cur_block, column_defaults, context)); } return narrowBlockInputStreams(blocks_input, num_streams); @@ -288,13 +291,15 @@ public: * INSERT data; SELECT *; last SELECT returns only insert_data */ storage.table_fd_was_used = true; - write_buf = getWriteBuffer(compression_method, storage.table_fd); + write_buf = wrapWriteBufferWithCompressionMethod(std::make_unique(storage.table_fd), compression_method, 3); } else { if (storage.paths.size() != 1) throw Exception("Table '" + storage.table_name + "' is in readonly mode because of globs in filepath", ErrorCodes::DATABASE_ACCESS_DENIED); - write_buf = getWriteBuffer(compression_method, storage.paths[0], DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_APPEND | O_CREAT); + write_buf = wrapWriteBufferWithCompressionMethod( + std::make_unique(storage.paths[0], DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_APPEND | O_CREAT), + compression_method, 3); } writer = FormatFactory::instance().getOutput(storage.format_name, *write_buf, storage.getSampleBlock(), context); @@ -333,8 +338,7 @@ BlockOutputStreamPtr StorageFile::write( const ASTPtr & /*query*/, const Context & context) { - return std::make_shared(*this, - IStorage::chooseCompressionMethod(paths[0], compression_method), context); + return std::make_shared(*this, chooseCompressionMethod(paths[0], compression_method), context); } Strings StorageFile::getDataPaths() const @@ -363,6 +367,28 @@ void StorageFile::rename(const String & new_path_to_table_data, const String & n database_name = new_database_name; } +void StorageFile::truncate(const ASTPtr & /*query*/, const Context & /* context */, TableStructureWriteLockHolder &) +{ + if (paths.size() != 1) + throw Exception("Can't truncate table '" + table_name + "' in readonly mode", ErrorCodes::DATABASE_ACCESS_DENIED); + + std::unique_lock lock(rwlock); + + if (use_table_fd) + { + if (0 != ::ftruncate(table_fd, 0)) + throwFromErrno("Cannot truncate file at fd " + toString(table_fd), ErrorCodes::CANNOT_TRUNCATE_FILE); + } + else + { + if (!Poco::File(paths[0]).exists()) + return; + + if (0 != ::truncate(paths[0].c_str(), 0)) + throwFromErrnoWithPath("Cannot truncate file " + paths[0], paths[0], ErrorCodes::CANNOT_TRUNCATE_FILE); + } +} + void registerStorageFile(StorageFactory & factory) { diff --git a/dbms/src/Storages/StorageFile.h b/dbms/src/Storages/StorageFile.h index e3871166f03..23a6d6e7ff5 100644 --- a/dbms/src/Storages/StorageFile.h +++ b/dbms/src/Storages/StorageFile.h @@ -38,6 +38,8 @@ public: const ASTPtr & query, const Context & context) override; + void truncate(const ASTPtr & /*query*/, const Context & /* context */, TableStructureWriteLockHolder &) override; + void rename(const String & new_path_to_table_data, const String & new_database_name, const String & new_table_name, TableStructureWriteLockHolder &) override; Strings getDataPaths() const override; diff --git a/dbms/src/Storages/StorageHDFS.cpp b/dbms/src/Storages/StorageHDFS.cpp index 3f1386cca5e..8e5db910092 100644 --- a/dbms/src/Storages/StorageHDFS.cpp +++ b/dbms/src/Storages/StorageHDFS.cpp @@ -67,7 +67,7 @@ public: UInt64 max_block_size, const CompressionMethod compression_method) { - auto read_buf = getReadBuffer(compression_method, uri); + auto read_buf = wrapReadBufferWithCompressionMethod(std::make_unique(uri), compression_method); auto input_stream = FormatFactory::instance().getInput(format, *read_buf, sample_block, context, max_block_size); reader = std::make_shared>(input_stream, std::move(read_buf)); @@ -112,7 +112,7 @@ public: const CompressionMethod compression_method) : sample_block(sample_block_) { - write_buf = getWriteBuffer(compression_method, uri); + write_buf = wrapWriteBufferWithCompressionMethod(std::make_unique(uri), compression_method, 3); writer = FormatFactory::instance().getOutput(format, *write_buf, sample_block, context); } @@ -213,7 +213,7 @@ BlockInputStreams StorageHDFS::read( for (const auto & res_path : res_paths) { result.push_back(std::make_shared(uri_without_path + res_path, format_name, getSampleBlock(), context_, - max_block_size, IStorage::chooseCompressionMethod(res_path, compression_method))); + max_block_size, chooseCompressionMethod(res_path, compression_method))); } return narrowBlockInputStreams(result, num_streams); @@ -231,7 +231,7 @@ BlockOutputStreamPtr StorageHDFS::write(const ASTPtr & /*query*/, const Context format_name, getSampleBlock(), context, - IStorage::chooseCompressionMethod(uri, compression_method)); + chooseCompressionMethod(uri, compression_method)); } void registerStorageHDFS(StorageFactory & factory) diff --git a/dbms/src/Storages/StorageMerge.cpp b/dbms/src/Storages/StorageMerge.cpp index 7a5e5999fc9..5be6353514e 100644 --- a/dbms/src/Storages/StorageMerge.cpp +++ b/dbms/src/Storages/StorageMerge.cpp @@ -252,7 +252,7 @@ BlockInputStreams StorageMerge::read( else { source_streams.emplace_back(std::make_shared( - header, [=]() mutable -> BlockInputStreamPtr + header, [=, this]() mutable -> BlockInputStreamPtr { BlockInputStreams streams = createSourceStreams(query_info, processed_stage, max_block_size, header, storage, struct_lock, real_column_names, diff --git a/dbms/src/Storages/StorageMergeTree.cpp b/dbms/src/Storages/StorageMergeTree.cpp index 9a0583a464a..f857602cdde 100644 --- a/dbms/src/Storages/StorageMergeTree.cpp +++ b/dbms/src/Storages/StorageMergeTree.cpp @@ -328,10 +328,14 @@ public: else { MergeTreeDataPart::TTLInfos ttl_infos; + size_t max_volume_index = 0; for (auto & part_ptr : future_part_.parts) + { ttl_infos.update(part_ptr->ttl_infos); + max_volume_index = std::max(max_volume_index, storage.getStoragePolicy()->getVolumeIndexByDisk(part_ptr->disk)); + } - reserved_space = storage.tryReserveSpacePreferringTTLRules(total_size, ttl_infos, time(nullptr)); + reserved_space = storage.tryReserveSpacePreferringTTLRules(total_size, ttl_infos, time(nullptr), max_volume_index); } if (!reserved_space) { @@ -612,7 +616,13 @@ bool StorageMergeTree::merge( if (!selected) { if (out_disable_reason) - *out_disable_reason = "Cannot select parts for optimization"; + { + if (!out_disable_reason->empty()) + { + *out_disable_reason += ". "; + } + *out_disable_reason += "Cannot select parts for optimization"; + } return false; } @@ -697,9 +707,6 @@ bool StorageMergeTree::tryMutatePart() /// You must call destructor with unlocked `currently_processing_in_background_mutex`. std::optional tagger; { - /// DataPart can be store only at one disk. Get Max of free space at all disks - UInt64 disk_space = storage_policy->getMaxUnreservedFreeSpace(); - std::lock_guard lock(currently_processing_in_background_mutex); if (current_mutations_by_version.empty()) @@ -715,7 +722,7 @@ bool StorageMergeTree::tryMutatePart() if (mutations_begin_it == mutations_end_it) continue; - if (merger_mutator.getMaxSourcePartSizeForMutation() > disk_space) + if (merger_mutator.getMaxSourcePartSizeForMutation() < part->bytes_on_disk) continue; size_t current_ast_elements = 0; @@ -1145,7 +1152,7 @@ void StorageMergeTree::replacePartitionFrom(const StoragePtr & source_table, con if (!canReplacePartition(src_part)) throw Exception( "Cannot replace partition '" + partition_id + "' because part '" + src_part->name + "' has inconsistent granularity with table", - ErrorCodes::LOGICAL_ERROR); + ErrorCodes::BAD_ARGUMENTS); /// This will generate unique name in scope of current server process. Int64 temp_index = insert_increment.get(); diff --git a/dbms/src/Storages/StorageReplicatedMergeTree.cpp b/dbms/src/Storages/StorageReplicatedMergeTree.cpp index 61cd7b550ca..67b9cbd5ca4 100644 --- a/dbms/src/Storages/StorageReplicatedMergeTree.cpp +++ b/dbms/src/Storages/StorageReplicatedMergeTree.cpp @@ -524,7 +524,7 @@ void StorageReplicatedMergeTree::setTableStructure(ColumnsDescription new_column if (metadata_diff.ttl_table_changed) { - ParserExpression parser; + ParserTTLExpressionList parser; metadata.ttl_for_table_ast = parseQuery(parser, metadata_diff.new_ttl_table, 0); } } @@ -1053,12 +1053,14 @@ bool StorageReplicatedMergeTree::tryExecuteMerge(const LogEntry & entry) /// Can throw an exception while reserving space. MergeTreeDataPart::TTLInfos ttl_infos; + size_t max_volume_index = 0; for (auto & part_ptr : parts) { ttl_infos.update(part_ptr->ttl_infos); + max_volume_index = std::max(max_volume_index, getStoragePolicy()->getVolumeIndexByDisk(part_ptr->disk)); } ReservationPtr reserved_space = reserveSpacePreferringTTLRules(estimated_space_for_merge, - ttl_infos, time(nullptr)); + ttl_infos, time(nullptr), max_volume_index); auto table_lock = lockStructureForShare(false, RWLockImpl::NO_QUERY); @@ -3181,7 +3183,7 @@ bool StorageReplicatedMergeTree::optimize(const ASTPtr & query, const ASTPtr & p { /// NOTE Table lock must not be held while waiting. Some combination of R-W-R locks from different threads will yield to deadlock. for (auto & merge_entry : merge_entries) - waitForAllReplicasToProcessLogEntry(merge_entry); + waitForAllReplicasToProcessLogEntry(merge_entry, false); } return true; @@ -3889,13 +3891,19 @@ StorageReplicatedMergeTree::allocateBlockNumber( } -void StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry(const ReplicatedMergeTreeLogEntryData & entry) +void StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry(const ReplicatedMergeTreeLogEntryData & entry, bool wait_for_non_active) { LOG_DEBUG(log, "Waiting for all replicas to process " << entry.znode_name); - Strings replicas = getZooKeeper()->getChildren(zookeeper_path + "/replicas"); + auto zookeeper = getZooKeeper(); + Strings replicas = zookeeper->getChildren(zookeeper_path + "/replicas"); for (const String & replica : replicas) - waitForReplicaToProcessLogEntry(replica, entry); + { + if (wait_for_non_active || zookeeper->exists(zookeeper_path + "/replicas/" + replica + "/is_active")) + { + waitForReplicaToProcessLogEntry(replica, entry); + } + } LOG_DEBUG(log, "Finished waiting for all replicas to process " << entry.znode_name); } diff --git a/dbms/src/Storages/StorageReplicatedMergeTree.h b/dbms/src/Storages/StorageReplicatedMergeTree.h index 9c97abdff40..60c2ea0b870 100644 --- a/dbms/src/Storages/StorageReplicatedMergeTree.h +++ b/dbms/src/Storages/StorageReplicatedMergeTree.h @@ -486,7 +486,7 @@ private: * Because it effectively waits for other thread that usually has to also acquire a lock to proceed and this yields deadlock. * TODO: There are wrong usages of this method that are not fixed yet. */ - void waitForAllReplicasToProcessLogEntry(const ReplicatedMergeTreeLogEntryData & entry); + void waitForAllReplicasToProcessLogEntry(const ReplicatedMergeTreeLogEntryData & entry, bool wait_for_non_active = true); /** Wait until the specified replica executes the specified action from the log. * NOTE: See comment about locks above. diff --git a/dbms/src/Storages/StorageS3.cpp b/dbms/src/Storages/StorageS3.cpp index cf0b3df44fd..14732a291b1 100644 --- a/dbms/src/Storages/StorageS3.cpp +++ b/dbms/src/Storages/StorageS3.cpp @@ -49,7 +49,7 @@ namespace const String & key) : name(name_) { - read_buf = getReadBuffer(compression_method, client, bucket, key); + read_buf = wrapReadBufferWithCompressionMethod(std::make_unique(client, bucket, key), compression_method); reader = FormatFactory::instance().getInput(format, *read_buf, sample_block, context, max_block_size); } @@ -98,7 +98,8 @@ namespace const String & key) : sample_block(sample_block_) { - write_buf = getWriteBuffer(compression_method, client, bucket, key, min_upload_part_size); + write_buf = wrapWriteBufferWithCompressionMethod( + std::make_unique(client, bucket, key, min_upload_part_size), compression_method, 3); writer = FormatFactory::instance().getOutput(format, *write_buf, sample_block, context); } @@ -173,7 +174,7 @@ BlockInputStreams StorageS3::read( getHeaderBlock(column_names), context, max_block_size, - IStorage::chooseCompressionMethod(uri.endpoint, compression_method), + chooseCompressionMethod(uri.endpoint, compression_method), client, uri.bucket, uri.key); @@ -194,7 +195,7 @@ BlockOutputStreamPtr StorageS3::write(const ASTPtr & /*query*/, const Context & { return std::make_shared( format_name, min_upload_part_size, getSampleBlock(), context_global, - IStorage::chooseCompressionMethod(uri.endpoint, compression_method), + chooseCompressionMethod(uri.endpoint, compression_method), client, uri.bucket, uri.key); } diff --git a/dbms/src/Storages/StorageStripeLog.cpp b/dbms/src/Storages/StorageStripeLog.cpp index 1be6adb8037..90dac4a53d7 100644 --- a/dbms/src/Storages/StorageStripeLog.cpp +++ b/dbms/src/Storages/StorageStripeLog.cpp @@ -250,7 +250,7 @@ BlockInputStreams StorageStripeLog::read( if (!Poco::File(fullPath() + "index.mrk").exists()) return { std::make_shared(getSampleBlockForColumns(column_names)) }; - CompressedReadBufferFromFile index_in(fullPath() + "index.mrk", 0, 0, INDEX_BUFFER_SIZE); + CompressedReadBufferFromFile index_in(fullPath() + "index.mrk", 0, 0, 0, INDEX_BUFFER_SIZE); std::shared_ptr index{std::make_shared(index_in, column_names_set)}; BlockInputStreams res; diff --git a/dbms/src/Storages/StorageURL.cpp b/dbms/src/Storages/StorageURL.cpp index 907e18b21cf..efe15dc1928 100644 --- a/dbms/src/Storages/StorageURL.cpp +++ b/dbms/src/Storages/StorageURL.cpp @@ -60,17 +60,18 @@ namespace const CompressionMethod compression_method) : name(name_) { - read_buf = getReadBuffer( - compression_method, - uri, - method, - callback, - timeouts, - context.getSettingsRef().max_http_get_redirects, - Poco::Net::HTTPBasicCredentials{}, - DBMS_DEFAULT_BUFFER_SIZE, - ReadWriteBufferFromHTTP::HTTPHeaderEntries{}, - context.getRemoteHostFilter()); + read_buf = wrapReadBufferWithCompressionMethod( + std::make_unique( + uri, + method, + callback, + timeouts, + context.getSettingsRef().max_http_get_redirects, + Poco::Net::HTTPBasicCredentials{}, + DBMS_DEFAULT_BUFFER_SIZE, + ReadWriteBufferFromHTTP::HTTPHeaderEntries{}, + context.getRemoteHostFilter()), + compression_method); reader = FormatFactory::instance().getInput(format, *read_buf, sample_block, context, max_block_size); } @@ -117,7 +118,9 @@ namespace const CompressionMethod compression_method) : sample_block(sample_block_) { - write_buf = getWriteBuffer(compression_method, uri, Poco::Net::HTTPRequest::HTTP_POST, timeouts); + write_buf = wrapWriteBufferWithCompressionMethod( + std::make_unique(uri, Poco::Net::HTTPRequest::HTTP_POST, timeouts), + compression_method, 3); writer = FormatFactory::instance().getOutput(format, *write_buf, sample_block, context); } @@ -196,7 +199,7 @@ BlockInputStreams IStorageURLBase::read(const Names & column_names, context, max_block_size, ConnectionTimeouts::getHTTPTimeouts(context), - IStorage::chooseCompressionMethod(request_uri.getPath(), compression_method)); + chooseCompressionMethod(request_uri.getPath(), compression_method)); auto column_defaults = getColumns().getDefaults(); if (column_defaults.empty()) @@ -215,7 +218,7 @@ BlockOutputStreamPtr IStorageURLBase::write(const ASTPtr & /*query*/, const Cont return std::make_shared( uri, format_name, getSampleBlock(), context_global, ConnectionTimeouts::getHTTPTimeouts(context_global), - IStorage::chooseCompressionMethod(uri.toString(), compression_method)); + chooseCompressionMethod(uri.toString(), compression_method)); } void registerStorageURL(StorageFactory & factory) diff --git a/dbms/src/Storages/StorageView.cpp b/dbms/src/Storages/StorageView.cpp index 824856dfc4e..5c8543bbb33 100644 --- a/dbms/src/Storages/StorageView.cpp +++ b/dbms/src/Storages/StorageView.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -23,6 +24,7 @@ namespace ErrorCodes { extern const int INCORRECT_QUERY; extern const int LOGICAL_ERROR; + extern const int ALIAS_REQUIRED; } @@ -62,8 +64,23 @@ BlockInputStreams StorageView::read( replaceTableNameWithSubquery(new_outer_select, new_inner_query); - if (PredicateExpressionsOptimizer(new_outer_select, context.getSettings(), context).optimize()) - current_inner_query = new_inner_query; + /// TODO: remove getTableExpressions and getTablesWithColumns + { + const auto & table_expressions = getTableExpressions(*new_outer_select); + const auto & tables_with_columns = getDatabaseAndTablesWithColumnNames(table_expressions, context); + + auto & settings = context.getSettingsRef(); + if (settings.joined_subquery_requires_alias && tables_with_columns.size() > 1) + { + for (auto & pr : tables_with_columns) + if (pr.table.table.empty() && pr.table.alias.empty()) + throw Exception("Not unique subquery in FROM requires an alias (or joined_subquery_requires_alias=0 to disable restriction).", + ErrorCodes::ALIAS_REQUIRED); + } + + if (PredicateExpressionsOptimizer(context, tables_with_columns, context.getSettings()).optimize(*new_outer_select)) + current_inner_query = new_inner_query; + } } QueryPipeline pipeline; diff --git a/dbms/src/Storages/StorageXDBC.cpp b/dbms/src/Storages/StorageXDBC.cpp index 222eebd6377..0dcbf372b28 100644 --- a/dbms/src/Storages/StorageXDBC.cpp +++ b/dbms/src/Storages/StorageXDBC.cpp @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include diff --git a/dbms/src/Storages/System/StorageSystemDictionaries.cpp b/dbms/src/Storages/System/StorageSystemDictionaries.cpp index c8e19fed086..60ae65427a1 100644 --- a/dbms/src/Storages/System/StorageSystemDictionaries.cpp +++ b/dbms/src/Storages/System/StorageSystemDictionaries.cpp @@ -50,19 +50,25 @@ void StorageSystemDictionaries::fillData(MutableColumns & res_columns, const Con const auto & external_dictionaries = context.getExternalDictionariesLoader(); for (const auto & load_result : external_dictionaries.getCurrentLoadResults()) { - if (startsWith(load_result.repository_name, IExternalLoaderConfigRepository::INTERNAL_REPOSITORY_NAME_PREFIX)) - continue; + const auto dict_ptr = std::dynamic_pointer_cast(load_result.object); - size_t i = 0; - String database; - String short_name = load_result.name; - - if (!load_result.repository_name.empty() && startsWith(load_result.name, load_result.repository_name + ".")) + String database, short_name; + if (dict_ptr) { - database = load_result.repository_name; - short_name = load_result.name.substr(load_result.repository_name.length() + 1); + database = dict_ptr->getDatabase(); + short_name = dict_ptr->getName(); + } + else + { + short_name = load_result.name; + if (!load_result.repository_name.empty() && startsWith(short_name, load_result.repository_name + ".")) + { + database = load_result.repository_name; + short_name = short_name.substr(database.length() + 1); + } } + size_t i = 0; res_columns[i++]->insert(database); res_columns[i++]->insert(short_name); res_columns[i++]->insert(static_cast(load_result.status)); @@ -70,7 +76,6 @@ void StorageSystemDictionaries::fillData(MutableColumns & res_columns, const Con std::exception_ptr last_exception = load_result.exception; - const auto dict_ptr = std::dynamic_pointer_cast(load_result.object); if (dict_ptr) { res_columns[i++]->insert(dict_ptr->getTypeName()); diff --git a/dbms/src/Storages/tests/get_abandonable_lock_in_all_partitions.cpp b/dbms/src/Storages/tests/get_abandonable_lock_in_all_partitions.cpp index f0c5a3d158e..533db3236fd 100644 --- a/dbms/src/Storages/tests/get_abandonable_lock_in_all_partitions.cpp +++ b/dbms/src/Storages/tests/get_abandonable_lock_in_all_partitions.cpp @@ -54,7 +54,7 @@ try catch (const Exception & e) { std::cerr << e.what() << ", " << e.displayText() << ": " << std::endl - << e.getStackTrace().toString() << std::endl; + << e.getStackTraceString() << std::endl; throw; } catch (Poco::Exception & e) diff --git a/dbms/src/Storages/tests/get_current_inserts_in_replicated.cpp b/dbms/src/Storages/tests/get_current_inserts_in_replicated.cpp index 9012ccb6eb8..aba69684045 100644 --- a/dbms/src/Storages/tests/get_current_inserts_in_replicated.cpp +++ b/dbms/src/Storages/tests/get_current_inserts_in_replicated.cpp @@ -112,7 +112,7 @@ try catch (const Exception & e) { std::cerr << e.what() << ", " << e.displayText() << ": " << std::endl - << e.getStackTrace().toString() << std::endl; + << e.getStackTraceString() << std::endl; throw; } catch (Poco::Exception & e) diff --git a/dbms/src/TableFunctions/ITableFunctionFileLike.cpp b/dbms/src/TableFunctions/ITableFunctionFileLike.cpp index 3e0ddafaa90..7b1d342a64a 100644 --- a/dbms/src/TableFunctions/ITableFunctionFileLike.cpp +++ b/dbms/src/TableFunctions/ITableFunctionFileLike.cpp @@ -42,12 +42,10 @@ StoragePtr ITableFunctionFileLike::executeImpl(const ASTPtr & ast_function, cons std::string filename = args[0]->as().value.safeGet(); std::string format = args[1]->as().value.safeGet(); std::string structure = args[2]->as().value.safeGet(); - std::string compression_method; + std::string compression_method = "auto"; if (args.size() == 4) - { compression_method = args[3]->as().value.safeGet(); - } else compression_method = "auto"; ColumnsDescription columns = parseColumnsListFromString(structure, context); diff --git a/dbms/src/TableFunctions/TableFunctionRemote.cpp b/dbms/src/TableFunctions/TableFunctionRemote.cpp index 87c8989cbe2..033839009bb 100644 --- a/dbms/src/TableFunctions/TableFunctionRemote.cpp +++ b/dbms/src/TableFunctions/TableFunctionRemote.cpp @@ -14,6 +14,7 @@ #include #include #include +#include #include "registerTableFunctions.h" @@ -140,7 +141,10 @@ StoragePtr TableFunctionRemote::executeImpl(const ASTPtr & ast_function, const C if (!cluster_name.empty()) { /// Use an existing cluster from the main config - cluster = context.getCluster(cluster_name); + if (name != "clusterAllReplicas") + cluster = context.getCluster(cluster_name); + else + cluster = context.getCluster(cluster_name)->getClusterWithReplicasAsShards(context.getSettings()); } else { @@ -164,13 +168,22 @@ StoragePtr TableFunctionRemote::executeImpl(const ASTPtr & ast_function, const C { size_t colon = host.find(':'); if (colon == String::npos) - context.getRemoteHostFilter().checkHostAndPort(host, toString((secure ? (maybe_secure_port ? *maybe_secure_port : DBMS_DEFAULT_SECURE_PORT) : context.getTCPPort()))); + context.getRemoteHostFilter().checkHostAndPort( + host, + toString((secure ? (maybe_secure_port ? *maybe_secure_port : DBMS_DEFAULT_SECURE_PORT) : context.getTCPPort()))); else context.getRemoteHostFilter().checkHostAndPort(host.substr(0, colon), host.substr(colon + 1)); } } - cluster = std::make_shared(context.getSettings(), names, username, password, (secure ? (maybe_secure_port ? *maybe_secure_port : DBMS_DEFAULT_SECURE_PORT) : context.getTCPPort()), false, secure); + cluster = std::make_shared( + context.getSettings(), + names, + username, + password, + (secure ? (maybe_secure_port ? *maybe_secure_port : DBMS_DEFAULT_SECURE_PORT) : context.getTCPPort()), + false, + secure); } auto structure_remote_table = getStructureOfRemoteTable(*cluster, remote_database, remote_table, context, remote_table_function_ptr); @@ -198,7 +211,7 @@ StoragePtr TableFunctionRemote::executeImpl(const ASTPtr & ast_function, const C TableFunctionRemote::TableFunctionRemote(const std::string & name_, bool secure_) : name{name_}, secure{secure_} { - is_cluster_function = name == "cluster"; + is_cluster_function = (name == "cluster" || name == "clusterAllReplicas"); std::stringstream ss; ss << "Table function '" << name + "' requires from 2 to " << (is_cluster_function ? 3 : 5) << " parameters" @@ -213,6 +226,7 @@ void registerTableFunctionRemote(TableFunctionFactory & factory) factory.registerFunction("remote", [] () -> TableFunctionPtr { return std::make_shared("remote"); }); factory.registerFunction("remoteSecure", [] () -> TableFunctionPtr { return std::make_shared("remote", /* secure = */ true); }); factory.registerFunction("cluster", [] () -> TableFunctionPtr { return std::make_shared("cluster"); }); + factory.registerFunction("clusterAllReplicas", [] () -> TableFunctionPtr { return std::make_shared("clusterAllReplicas"); }); } } diff --git a/dbms/tests/integration/test_cluster_all_replicas/__init__.py b/dbms/tests/integration/test_cluster_all_replicas/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dbms/tests/integration/test_cluster_all_replicas/configs/remote_servers.xml b/dbms/tests/integration/test_cluster_all_replicas/configs/remote_servers.xml new file mode 100644 index 00000000000..68dcfcc1460 --- /dev/null +++ b/dbms/tests/integration/test_cluster_all_replicas/configs/remote_servers.xml @@ -0,0 +1,16 @@ + + + + + + node1 + 9000 + + + node2 + 9000 + + + + + diff --git a/dbms/tests/integration/test_cluster_all_replicas/test.py b/dbms/tests/integration/test_cluster_all_replicas/test.py new file mode 100644 index 00000000000..0af5693fc75 --- /dev/null +++ b/dbms/tests/integration/test_cluster_all_replicas/test.py @@ -0,0 +1,21 @@ +import pytest + +from helpers.cluster import ClickHouseCluster + +cluster = ClickHouseCluster(__file__) + +node1 = cluster.add_instance('node1', main_configs=['configs/remote_servers.xml'], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=['configs/remote_servers.xml'], with_zookeeper=True) + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def test_remote(start_cluster): + assert node1.query('''SELECT hostName() FROM clusterAllReplicas("two_shards", system.one)''') == 'node1\nnode2\n' + assert node1.query('''SELECT hostName() FROM cluster("two_shards", system.one)''') == 'node1\n' diff --git a/dbms/tests/integration/test_row_policy/multiple_tags_with_table_names.xml b/dbms/tests/integration/test_row_policy/multiple_tags_with_table_names.xml new file mode 100644 index 00000000000..87b22047e7e --- /dev/null +++ b/dbms/tests/integration/test_row_policy/multiple_tags_with_table_names.xml @@ -0,0 +1,26 @@ + + + + + + + + + + a = 1 +
+ + + + a + b < 1 or c - d > 5 +
+ + + + c = 1 +
+ + + + + diff --git a/dbms/tests/integration/test_row_policy/tag_with_table_name.xml b/dbms/tests/integration/test_row_policy/tag_with_table_name.xml new file mode 100644 index 00000000000..4affd2d9038 --- /dev/null +++ b/dbms/tests/integration/test_row_policy/tag_with_table_name.xml @@ -0,0 +1,26 @@ + + + + + + + + + + a = 1 +
+ + + + a + b < 1 or c - d > 5 +
+ + + + c = 1 + + + + + + diff --git a/dbms/tests/integration/test_row_policy/test.py b/dbms/tests/integration/test_row_policy/test.py index 421a4b0510c..ae64d1e5a3a 100644 --- a/dbms/tests/integration/test_row_policy/test.py +++ b/dbms/tests/integration/test_row_policy/test.py @@ -28,13 +28,19 @@ def started_cluster(): CREATE TABLE mydb.filtered_table1 (a UInt8, b UInt8) ENGINE MergeTree ORDER BY a; INSERT INTO mydb.filtered_table1 values (0, 0), (0, 1), (1, 0), (1, 1); + CREATE TABLE mydb.table (a UInt8, b UInt8) ENGINE MergeTree ORDER BY a; + INSERT INTO mydb.table values (0, 0), (0, 1), (1, 0), (1, 1); + CREATE TABLE mydb.filtered_table2 (a UInt8, b UInt8, c UInt8, d UInt8) ENGINE MergeTree ORDER BY a; INSERT INTO mydb.filtered_table2 values (0, 0, 0, 0), (1, 2, 3, 4), (4, 3, 2, 1), (0, 0, 6, 0); CREATE TABLE mydb.filtered_table3 (a UInt8, b UInt8, c UInt16 ALIAS a + b) ENGINE MergeTree ORDER BY a; INSERT INTO mydb.filtered_table3 values (0, 0), (0, 1), (1, 0), (1, 1); + + CREATE TABLE mydb.`.filtered_table4` (a UInt8, b UInt8, c UInt16 ALIAS a + b) ENGINE MergeTree ORDER BY a; + INSERT INTO mydb.`.filtered_table4` values (0, 0), (0, 1), (1, 0), (1, 1); ''') - + yield cluster finally: @@ -58,6 +64,7 @@ def test_smoke(): assert instance.query("SELECT a FROM mydb.filtered_table1") == "1\n1\n" assert instance.query("SELECT b FROM mydb.filtered_table1") == "0\n1\n" assert instance.query("SELECT a FROM mydb.filtered_table1 WHERE a = 1") == "1\n1\n" + assert instance.query("SELECT a FROM mydb.filtered_table1 WHERE a IN (1)") == "1\n1\n" assert instance.query("SELECT a = 1 FROM mydb.filtered_table1") == "1\n1\n" assert instance.query("SELECT a FROM mydb.filtered_table3") == "0\n1\n" @@ -88,6 +95,46 @@ def test_prewhere_not_supported(): assert expected_error in instance.query_and_get_error("SELECT * FROM mydb.filtered_table3 PREWHERE 1") +def test_single_table_name(): + copy_policy_xml('tag_with_table_name.xml') + assert instance.query("SELECT * FROM mydb.table") == "1\t0\n1\t1\n" + assert instance.query("SELECT * FROM mydb.filtered_table2") == "0\t0\t0\t0\n0\t0\t6\t0\n" + assert instance.query("SELECT * FROM mydb.filtered_table3") == "0\t1\n1\t0\n" + + assert instance.query("SELECT a FROM mydb.table") == "1\n1\n" + assert instance.query("SELECT b FROM mydb.table") == "0\n1\n" + assert instance.query("SELECT a FROM mydb.table WHERE a = 1") == "1\n1\n" + assert instance.query("SELECT a = 1 FROM mydb.table") == "1\n1\n" + + assert instance.query("SELECT a FROM mydb.filtered_table3") == "0\n1\n" + assert instance.query("SELECT b FROM mydb.filtered_table3") == "1\n0\n" + assert instance.query("SELECT c FROM mydb.filtered_table3") == "1\n1\n" + assert instance.query("SELECT a + b FROM mydb.filtered_table3") == "1\n1\n" + assert instance.query("SELECT a FROM mydb.filtered_table3 WHERE c = 1") == "0\n1\n" + assert instance.query("SELECT c = 1 FROM mydb.filtered_table3") == "1\n1\n" + assert instance.query("SELECT a + b = 1 FROM mydb.filtered_table3") == "1\n1\n" + + +def test_custom_table_name(): + copy_policy_xml('multiple_tags_with_table_names.xml') + assert instance.query("SELECT * FROM mydb.table") == "1\t0\n1\t1\n" + assert instance.query("SELECT * FROM mydb.filtered_table2") == "0\t0\t0\t0\n0\t0\t6\t0\n" + assert instance.query("SELECT * FROM mydb.`.filtered_table4`") == "0\t1\n1\t0\n" + + assert instance.query("SELECT a FROM mydb.table") == "1\n1\n" + assert instance.query("SELECT b FROM mydb.table") == "0\n1\n" + assert instance.query("SELECT a FROM mydb.table WHERE a = 1") == "1\n1\n" + assert instance.query("SELECT a = 1 FROM mydb.table") == "1\n1\n" + + assert instance.query("SELECT a FROM mydb.`.filtered_table4`") == "0\n1\n" + assert instance.query("SELECT b FROM mydb.`.filtered_table4`") == "1\n0\n" + assert instance.query("SELECT c FROM mydb.`.filtered_table4`") == "1\n1\n" + assert instance.query("SELECT a + b FROM mydb.`.filtered_table4`") == "1\n1\n" + assert instance.query("SELECT a FROM mydb.`.filtered_table4` WHERE c = 1") == "0\n1\n" + assert instance.query("SELECT c = 1 FROM mydb.`.filtered_table4`") == "1\n1\n" + assert instance.query("SELECT a + b = 1 FROM mydb.`.filtered_table4`") == "1\n1\n" + + def test_change_of_users_xml_changes_row_policies(): copy_policy_xml('normal_filters.xml') assert instance.query("SELECT * FROM mydb.filtered_table1") == "1\t0\n1\t1\n" diff --git a/dbms/tests/performance/README.md b/dbms/tests/performance/README.md index ecda08a80b1..d436eb7bce3 100644 --- a/dbms/tests/performance/README.md +++ b/dbms/tests/performance/README.md @@ -16,7 +16,7 @@ After you have choosen type, you have to specify `preconditions`. It contains ta The most important part of test is `stop_conditions`. For `loop` test you should always use `min_time_not_changing_for_ms` stop condition. For `once` test you can choose between `average_speed_not_changing_for_ms` and `max_speed_not_changing_for_ms`, but first is preferable. Also you should always specify `total_time_ms` metric. Endless tests will be ignored by CI. -`metrics` and `main_metric` settings are not important and can be ommited, because `loop` tests are always compared by `min_time` metric and `once` tests compared by `max_rows_per_second`. +`loop` tests are always compared by `min_time` metric and `once` tests compared by `max_rows_per_second`. You can use `substitions`, `create`, `fill` and `drop` queries to prepare test. You can find examples in this folder. diff --git a/dbms/tests/performance/agg_functions_min_max_any.xml b/dbms/tests/performance/agg_functions_min_max_any.xml index 85799a9a94a..8a132bb79a9 100644 --- a/dbms/tests/performance/agg_functions_min_max_any.xml +++ b/dbms/tests/performance/agg_functions_min_max_any.xml @@ -12,9 +12,6 @@ - - - default.hits_1000m_single diff --git a/dbms/tests/performance/and_function.xml b/dbms/tests/performance/and_function.xml index 08fd07ea7e5..fb1bcb9fcf8 100644 --- a/dbms/tests/performance/and_function.xml +++ b/dbms/tests/performance/and_function.xml @@ -12,9 +12,6 @@ - - - select count() from numbers(10000000) where number != 96594 AND number != 18511 AND number != 98085 AND number != 84177 AND number != 70314 AND number != 28083 AND number != 54202 AND number != 66522 AND number != 66939 AND number != 99469 AND number != 65776 AND number != 22876 AND number != 42151 AND number != 19924 AND number != 66681 AND number != 63022 AND number != 17487 AND number != 83914 AND number != 59754 AND number != 968 AND number != 73334 AND number != 68569 AND number != 49853 AND number != 33155 AND number != 31777 AND number != 99698 AND number != 26708 AND number != 76409 AND number != 42191 AND number != 55397 AND number != 25724 AND number != 39170 AND number != 22728 AND number != 98238 AND number != 86052 AND number != 12756 AND number != 13948 AND number != 57774 AND number != 82511 AND number != 11337 AND number != 23506 AND number != 11875 AND number != 58536 AND number != 56919 AND number != 25986 AND number != 80710 AND number != 61797 AND number != 99244 AND number != 11665 AND number != 15758 AND number != 82899 AND number != 63150 AND number != 7198 AND number != 40071 AND number != 46310 AND number != 78488 AND number != 9273 AND number != 91878 AND number != 57904 AND number != 53941 AND number != 75675 AND number != 12093 AND number != 50090 AND number != 59675 AND number != 41632 AND number != 81448 AND number != 46821 AND number != 51919 AND number != 49028 AND number != 71059 AND number != 15673 AND number != 6132 AND number != 15473 AND number != 32527 AND number != 63842 AND number != 33121 AND number != 53271 AND number != 86033 AND number != 96807 AND number != 4791 AND number != 80089 AND number != 51616 AND number != 46311 AND number != 82844 AND number != 59353 AND number != 63538 AND number != 64857 AND number != 58471 AND number != 29870 AND number != 80209 AND number != 61000 AND number != 75991 AND number != 44506 AND number != 11283 AND number != 6335 AND number != 73502 AND number != 22354 AND number != 72816 AND number != 66399 AND number != 61703 diff --git a/dbms/tests/performance/array_element.xml b/dbms/tests/performance/array_element.xml index 92dcf0bb5e1..672683fe146 100644 --- a/dbms/tests/performance/array_element.xml +++ b/dbms/tests/performance/array_element.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore([[1], [2]][number % 2 + 2]) SELECT count() FROM system.numbers WHERE NOT ignore([[], [2]][number % 2 + 2]) diff --git a/dbms/tests/performance/array_fill.xml b/dbms/tests/performance/array_fill.xml index c4c0955dfc6..25ed745158b 100644 --- a/dbms/tests/performance/array_fill.xml +++ b/dbms/tests/performance/array_fill.xml @@ -7,9 +7,6 @@ - - - SELECT arraySlice(arrayFill(x -> ((x % 2) >= 0), range(100000000)), 1, 10) SELECT arraySlice(arrayFill(x -> (((x.1) % 2) >= 0), arrayMap(x -> (x, toString(x)), range(100000000))), 1, 10) diff --git a/dbms/tests/performance/array_join.xml b/dbms/tests/performance/array_join.xml index 7220f35d881..d2eb213ce03 100644 --- a/dbms/tests/performance/array_join.xml +++ b/dbms/tests/performance/array_join.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM (SELECT [number] a, [number * 2] b FROM system.numbers) AS t ARRAY JOIN a, b WHERE NOT ignore(a + b) SELECT count() FROM (SELECT [number] a, [number * 2] b FROM system.numbers) AS t LEFT ARRAY JOIN a, b WHERE NOT ignore(a + b) diff --git a/dbms/tests/performance/base64.xml b/dbms/tests/performance/base64.xml index a479b10d48a..651412c2752 100644 --- a/dbms/tests/performance/base64.xml +++ b/dbms/tests/performance/base64.xml @@ -11,9 +11,6 @@ - - - diff --git a/dbms/tests/performance/base64_hits.xml b/dbms/tests/performance/base64_hits.xml index be693a16bee..7b07f3badb7 100644 --- a/dbms/tests/performance/base64_hits.xml +++ b/dbms/tests/performance/base64_hits.xml @@ -15,9 +15,6 @@ - - - diff --git a/dbms/tests/performance/basename.xml b/dbms/tests/performance/basename.xml index e204f050bfe..6af67bc94c4 100644 --- a/dbms/tests/performance/basename.xml +++ b/dbms/tests/performance/basename.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/cidr.xml b/dbms/tests/performance/cidr.xml index 257fcf6fb0d..1ca7f691881 100644 --- a/dbms/tests/performance/cidr.xml +++ b/dbms/tests/performance/cidr.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/codecs_float_insert.xml b/dbms/tests/performance/codecs_float_insert.xml index 2a39dfc48d6..1bbbf6b2d92 100644 --- a/dbms/tests/performance/codecs_float_insert.xml +++ b/dbms/tests/performance/codecs_float_insert.xml @@ -37,14 +37,20 @@ rnd + + num_rows + + 1000000 + + CREATE TABLE IF NOT EXISTS codec_{seq_type}_{type}_{codec} (n {type} CODEC({codec})) ENGINE = MergeTree PARTITION BY tuple() ORDER BY tuple(); - INSERT INTO codec_seq_Float64_{codec} (n) SELECT number/pi() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_mon_Float64_{codec} (n) SELECT number+sin(number) FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_rnd_Float64_{codec} (n) SELECT (rand() - 4294967295)/pi() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 + INSERT INTO codec_seq_{type}_{codec} (n) SELECT number/pi() FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_mon_{type}_{codec} (n) SELECT number+sin(number) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_rnd_{type}_{codec} (n) SELECT (intHash64(number) - 4294967295)/pi() FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 DROP TABLE IF EXISTS codec_{seq_type}_{type}_{codec} diff --git a/dbms/tests/performance/codecs_float_select.xml b/dbms/tests/performance/codecs_float_select.xml index f23b363b914..1d3957c8da9 100644 --- a/dbms/tests/performance/codecs_float_select.xml +++ b/dbms/tests/performance/codecs_float_select.xml @@ -37,18 +37,21 @@ rnd + + num_rows + + 1000000 + + CREATE TABLE IF NOT EXISTS codec_{seq_type}_{type}_{codec} (n {type} CODEC({codec})) ENGINE = MergeTree PARTITION BY tuple() ORDER BY tuple(); - - INSERT INTO codec_seq_Float64_{codec} (n) SELECT number/pi() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_mon_Float64_{codec} (n) SELECT number+sin(number) FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_rnd_Float64_{codec} (n) SELECT (rand() - 4294967295)/pi() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 + INSERT INTO codec_seq_{type}_{codec} (n) SELECT number/pi() FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_mon_{type}_{codec} (n) SELECT number+sin(number) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_rnd_{type}_{codec} (n) SELECT (intHash64(number) - 4294967295)/pi() FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 - - SELECT count(n) FROM codec_{seq_type}_{type}_{codec} WHERE ignore(n) LIMIT 100000 SETTINGS max_threads=1 + SELECT count(n) FROM codec_{seq_type}_{type}_{codec} WHERE ignore(n) == 0 LIMIT {num_rows} SETTINGS max_threads=1 DROP TABLE IF EXISTS codec_{seq_type}_{type}_{codec} diff --git a/dbms/tests/performance/codecs_int_insert.xml b/dbms/tests/performance/codecs_int_insert.xml index 742693d49fe..3ee293f9d16 100644 --- a/dbms/tests/performance/codecs_int_insert.xml +++ b/dbms/tests/performance/codecs_int_insert.xml @@ -39,14 +39,20 @@ rnd + + num_rows + + 1000000 + + CREATE TABLE IF NOT EXISTS codec_{seq_type}_{type}_{codec} (n {type} CODEC({codec})) ENGINE = MergeTree PARTITION BY tuple() ORDER BY tuple(); - INSERT INTO codec_seq_UInt64_{codec} (n) SELECT number FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_mon_UInt64_{codec} (n) SELECT number*512+(rand()%512) FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_rnd_UInt64_{codec} (n) SELECT rand() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 + INSERT INTO codec_seq_{type}_{codec} (n) SELECT number FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_mon_{type}_{codec} (n) SELECT number*512+(intHash64(number)%512) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_rnd_{type}_{codec} (n) SELECT intHash64(number) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 DROP TABLE IF EXISTS codec_{seq_type}_{type}_{codec} diff --git a/dbms/tests/performance/codecs_int_select.xml b/dbms/tests/performance/codecs_int_select.xml index 9c007863cd8..40ebfd4d000 100644 --- a/dbms/tests/performance/codecs_int_select.xml +++ b/dbms/tests/performance/codecs_int_select.xml @@ -39,18 +39,21 @@ rnd + + num_rows + + 1000000 + + CREATE TABLE IF NOT EXISTS codec_{seq_type}_{type}_{codec} (n {type} CODEC({codec})) ENGINE = MergeTree PARTITION BY tuple() ORDER BY tuple(); - - INSERT INTO codec_seq_UInt64_{codec} (n) SELECT number FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_mon_UInt64_{codec} (n) SELECT number*512+(rand()%512) FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 - INSERT INTO codec_rnd_UInt64_{codec} (n) SELECT rand() FROM system.numbers LIMIT 100000 SETTINGS max_threads=1 + INSERT INTO codec_seq_{type}_{codec} (n) SELECT number FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_mon_{type}_{codec} (n) SELECT number*512+(intHash64(number)%512) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 + INSERT INTO codec_rnd_{type}_{codec} (n) SELECT intHash64(number) FROM system.numbers LIMIT {num_rows} SETTINGS max_threads=1 - - SELECT count(n) FROM codec_{seq_type}_{type}_{codec} WHERE ignore(n) LIMIT 100000 SETTINGS max_threads=1 + SELECT count(n) FROM codec_{seq_type}_{type}_{codec} WHERE ignore(n) == 0 LIMIT {num_rows} SETTINGS max_threads=1 DROP TABLE IF EXISTS codec_{seq_type}_{type}_{codec} diff --git a/dbms/tests/performance/collations.xml b/dbms/tests/performance/collations.xml index 9bc48d76bce..1bec38dd103 100644 --- a/dbms/tests/performance/collations.xml +++ b/dbms/tests/performance/collations.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/column_column_comparison.xml b/dbms/tests/performance/column_column_comparison.xml index f80a673ce22..9d4446d7c2d 100644 --- a/dbms/tests/performance/column_column_comparison.xml +++ b/dbms/tests/performance/column_column_comparison.xml @@ -40,7 +40,4 @@ - - - diff --git a/dbms/tests/performance/columns_hashing.xml b/dbms/tests/performance/columns_hashing.xml index fe9777b8a38..138855dae89 100644 --- a/dbms/tests/performance/columns_hashing.xml +++ b/dbms/tests/performance/columns_hashing.xml @@ -40,7 +40,4 @@ - - - diff --git a/dbms/tests/performance/complex_array_creation.xml b/dbms/tests/performance/complex_array_creation.xml index b572041e8ec..a5ff824d6de 100644 --- a/dbms/tests/performance/complex_array_creation.xml +++ b/dbms/tests/performance/complex_array_creation.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore([[number], [number]]) SELECT count() FROM system.numbers WHERE NOT ignore([[], [number]]) diff --git a/dbms/tests/performance/concat_hits.xml b/dbms/tests/performance/concat_hits.xml index 0c4c52afa02..e2c6fc23c08 100644 --- a/dbms/tests/performance/concat_hits.xml +++ b/dbms/tests/performance/concat_hits.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/consistent_hashes.xml b/dbms/tests/performance/consistent_hashes.xml index 98a621e4248..7219aa00c1a 100644 --- a/dbms/tests/performance/consistent_hashes.xml +++ b/dbms/tests/performance/consistent_hashes.xml @@ -8,10 +8,6 @@ - - - - diff --git a/dbms/tests/performance/constant_column_comparison.xml b/dbms/tests/performance/constant_column_comparison.xml index 14fa7fa623e..f32ed444a0c 100644 --- a/dbms/tests/performance/constant_column_comparison.xml +++ b/dbms/tests/performance/constant_column_comparison.xml @@ -42,7 +42,4 @@ - - - diff --git a/dbms/tests/performance/constant_column_search.xml b/dbms/tests/performance/constant_column_search.xml index 76a4d2b7e74..9953c2797a2 100644 --- a/dbms/tests/performance/constant_column_search.xml +++ b/dbms/tests/performance/constant_column_search.xml @@ -61,7 +61,4 @@ - - - diff --git a/dbms/tests/performance/count.xml b/dbms/tests/performance/count.xml index da972c69059..0244adf4b38 100644 --- a/dbms/tests/performance/count.xml +++ b/dbms/tests/performance/count.xml @@ -12,9 +12,6 @@ - - - CREATE TABLE data(k UInt64, v UInt64) ENGINE = MergeTree ORDER BY k diff --git a/dbms/tests/performance/cpu_synthetic.xml b/dbms/tests/performance/cpu_synthetic.xml index cb6e41b34ac..16bef3fd42e 100644 --- a/dbms/tests/performance/cpu_synthetic.xml +++ b/dbms/tests/performance/cpu_synthetic.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/cryptographic_hashes.xml b/dbms/tests/performance/cryptographic_hashes.xml index 2f5a0a1d779..7840a7b382a 100644 --- a/dbms/tests/performance/cryptographic_hashes.xml +++ b/dbms/tests/performance/cryptographic_hashes.xml @@ -11,9 +11,6 @@ - - - diff --git a/dbms/tests/performance/date_parsing.xml b/dbms/tests/performance/date_parsing.xml index 10a2812b067..8ecf3681804 100644 --- a/dbms/tests/performance/date_parsing.xml +++ b/dbms/tests/performance/date_parsing.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/date_time_64.xml b/dbms/tests/performance/date_time_64.xml index 60c77ca22f8..b345550b335 100644 --- a/dbms/tests/performance/date_time_64.xml +++ b/dbms/tests/performance/date_time_64.xml @@ -21,9 +21,6 @@ - - - SELECT count() FROM dt where not ignore(x) diff --git a/dbms/tests/performance/decimal_aggregates.xml b/dbms/tests/performance/decimal_aggregates.xml index f22cb89de36..86830fedce6 100644 --- a/dbms/tests/performance/decimal_aggregates.xml +++ b/dbms/tests/performance/decimal_aggregates.xml @@ -11,9 +11,6 @@ - - - SELECT min(d32), max(d32), argMin(x, d32), argMax(x, d32) FROM t SELECT min(d64), max(d64), argMin(x, d64), argMax(x, d64) FROM t diff --git a/dbms/tests/performance/early_constant_folding.xml b/dbms/tests/performance/early_constant_folding.xml index 04fb4057d17..ad2d1619eb9 100644 --- a/dbms/tests/performance/early_constant_folding.xml +++ b/dbms/tests/performance/early_constant_folding.xml @@ -11,9 +11,6 @@ - - - default.hits_100m_single diff --git a/dbms/tests/performance/entropy.xml b/dbms/tests/performance/entropy.xml index a7b8f76fcaf..dcede345792 100644 --- a/dbms/tests/performance/entropy.xml +++ b/dbms/tests/performance/entropy.xml @@ -15,9 +15,6 @@ - - - diff --git a/dbms/tests/performance/first_significant_subdomain.xml b/dbms/tests/performance/first_significant_subdomain.xml index 8d38b871c62..705e70b86f9 100644 --- a/dbms/tests/performance/first_significant_subdomain.xml +++ b/dbms/tests/performance/first_significant_subdomain.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/fixed_string16.xml b/dbms/tests/performance/fixed_string16.xml index 34fa6a94707..398f09aba3d 100644 --- a/dbms/tests/performance/fixed_string16.xml +++ b/dbms/tests/performance/fixed_string16.xml @@ -22,7 +22,4 @@ - - - diff --git a/dbms/tests/performance/float_formatting.xml b/dbms/tests/performance/float_formatting.xml new file mode 100644 index 00000000000..0216e524735 --- /dev/null +++ b/dbms/tests/performance/float_formatting.xml @@ -0,0 +1,58 @@ + + once + + long + + + + + 10000 + + + 5000 + 20000 + + + + + + + expr + + 1 / rand() + rand() / 0xFFFFFFFF + 0xFFFFFFFF / rand() + toFloat64(number) + toFloat64(number % 2) + toFloat64(number % 10) + toFloat64(number % 100) + toFloat64(number % 1000) + toFloat64(number % 10000) + toFloat64(number % 100 + 0.5) + toFloat64(number % 100 + 0.123) + toFloat64(number % 1000 + 0.123456) + number / 2 + number / 3 + number / 7 + number / 16 + toFloat64(rand()) + toFloat64(rand64()) + toFloat32(number) + toFloat32(number % 2) + toFloat32(number % 10) + toFloat32(number % 100) + toFloat32(number % 1000) + toFloat32(number % 10000) + toFloat32(number % 100 + 0.5) + toFloat32(number % 100 + 0.123) + toFloat32(number % 1000 + 0.123456) + toFloat32(rand()) + toFloat32(rand64()) + reinterpretAsFloat32(reinterpretAsString(rand())) + reinterpretAsFloat64(reinterpretAsString(rand64())) + + + + + SELECT count() FROM system.numbers WHERE NOT ignore(toString({expr})) + diff --git a/dbms/tests/performance/float_parsing.xml b/dbms/tests/performance/float_parsing.xml index 1cded361871..81f30540dd1 100644 --- a/dbms/tests/performance/float_parsing.xml +++ b/dbms/tests/performance/float_parsing.xml @@ -14,9 +14,6 @@ - - - diff --git a/dbms/tests/performance/general_purpose_hashes.xml b/dbms/tests/performance/general_purpose_hashes.xml index b7a1b915ff0..122e69f374c 100644 --- a/dbms/tests/performance/general_purpose_hashes.xml +++ b/dbms/tests/performance/general_purpose_hashes.xml @@ -12,9 +12,6 @@ - - - diff --git a/dbms/tests/performance/general_purpose_hashes_on_UUID.xml b/dbms/tests/performance/general_purpose_hashes_on_UUID.xml index 23e00909bbe..c7fb0a3676b 100644 --- a/dbms/tests/performance/general_purpose_hashes_on_UUID.xml +++ b/dbms/tests/performance/general_purpose_hashes_on_UUID.xml @@ -12,9 +12,6 @@ - - - diff --git a/dbms/tests/performance/group_array_moving_sum.xml b/dbms/tests/performance/group_array_moving_sum.xml index ee2686c9af8..504a8b133a1 100644 --- a/dbms/tests/performance/group_array_moving_sum.xml +++ b/dbms/tests/performance/group_array_moving_sum.xml @@ -12,9 +12,6 @@ - - - CREATE TABLE moving_sum_1m(k UInt64, v UInt64) ENGINE = MergeTree ORDER BY k CREATE TABLE moving_sum_10m(k UInt64, v UInt64) ENGINE = MergeTree ORDER BY k diff --git a/dbms/tests/performance/if_array_num.xml b/dbms/tests/performance/if_array_num.xml index 3d1359bb55a..417b82a9d0c 100644 --- a/dbms/tests/performance/if_array_num.xml +++ b/dbms/tests/performance/if_array_num.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? [1, 2, 3] : [4, 5]) SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? [1, 2, 3] : materialize([4, 5])) diff --git a/dbms/tests/performance/if_array_string.xml b/dbms/tests/performance/if_array_string.xml index c135cf9c8ce..e1d8485adc2 100644 --- a/dbms/tests/performance/if_array_string.xml +++ b/dbms/tests/performance/if_array_string.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? ['Hello', 'World'] : ['a', 'b', 'c']) SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? materialize(['Hello', 'World']) : ['a', 'b', 'c']) diff --git a/dbms/tests/performance/if_string_const.xml b/dbms/tests/performance/if_string_const.xml index be2c4629519..15a281685ae 100644 --- a/dbms/tests/performance/if_string_const.xml +++ b/dbms/tests/performance/if_string_const.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? 'hello' : 'world') SELECT count() FROM system.numbers WHERE NOT ignore(rand() % 2 ? 'hello' : '') diff --git a/dbms/tests/performance/if_string_hits.xml b/dbms/tests/performance/if_string_hits.xml index e0ee7109f0c..267c8b039e5 100644 --- a/dbms/tests/performance/if_string_hits.xml +++ b/dbms/tests/performance/if_string_hits.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/if_to_multiif.xml b/dbms/tests/performance/if_to_multiif.xml new file mode 100644 index 00000000000..54d4b8ba842 --- /dev/null +++ b/dbms/tests/performance/if_to_multiif.xml @@ -0,0 +1,19 @@ + + once + + + + 1000 + 10000 + + + + + + + + + + diff --git a/dbms/tests/performance/information_value.xml b/dbms/tests/performance/information_value.xml index 63d61f6a432..ed054eda40d 100644 --- a/dbms/tests/performance/information_value.xml +++ b/dbms/tests/performance/information_value.xml @@ -15,9 +15,6 @@ - - - SELECT categoricalInformationValue(Age < 15, IsMobile) SELECT categoricalInformationValue(Age < 15, Age >= 15 and Age < 30, Age >= 30 and Age < 45, Age >= 45 and Age < 60, Age >= 60, IsMobile) diff --git a/dbms/tests/performance/int_parsing.xml b/dbms/tests/performance/int_parsing.xml index 625bdaadc86..51f740523ba 100644 --- a/dbms/tests/performance/int_parsing.xml +++ b/dbms/tests/performance/int_parsing.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/jit_small_requests.xml b/dbms/tests/performance/jit_small_requests.xml index 9c481994b97..d65e14cb97e 100644 --- a/dbms/tests/performance/jit_small_requests.xml +++ b/dbms/tests/performance/jit_small_requests.xml @@ -12,9 +12,6 @@ - - - @@ -25,10 +22,11 @@ bitXor(x2, bitShiftRight(x2, 33)) AS x3, x3 * 0xc4ceb9fe1a85ec53 AS x4, bitXor(x4, bitShiftRight(x4, 33)) AS x5 - SELECT x5, intHash64(number) FROM system.numbers LIMIT 10 + SELECT count() FROM numbers(10000000) WHERE NOT ignore(x5) SETTINGS compile_expressions = 0 + WITH bitXor(number, 0x4CF2D2BAAE6DA887) AS x0, @@ -37,8 +35,13 @@ bitXor(x2, bitShiftRight(x2, 33)) AS x3, x3 * 0xc4ceb9fe1a85ec53 AS x4, bitXor(x4, bitShiftRight(x4, 33)) AS x5 - SELECT x5, intHash64(number) FROM system.numbers LIMIT 10 + SELECT count() FROM numbers(10000000) WHERE NOT ignore(x5) SETTINGS compile_expressions = 1 + + + + SELECT count() FROM numbers(10000000) WHERE NOT ignore(intHash64(number)) + diff --git a/dbms/tests/performance/joins_in_memory.xml b/dbms/tests/performance/joins_in_memory.xml index 1da400c48f4..f624030d7d4 100644 --- a/dbms/tests/performance/joins_in_memory.xml +++ b/dbms/tests/performance/joins_in_memory.xml @@ -7,9 +7,6 @@ - - - CREATE TABLE ints (i64 Int64, i32 Int32, i16 Int16, i8 Int8) ENGINE = Memory diff --git a/dbms/tests/performance/joins_in_memory_pmj.xml b/dbms/tests/performance/joins_in_memory_pmj.xml index 3908df8c978..0352268c846 100644 --- a/dbms/tests/performance/joins_in_memory_pmj.xml +++ b/dbms/tests/performance/joins_in_memory_pmj.xml @@ -7,9 +7,6 @@ - - - CREATE TABLE ints (i64 Int64, i32 Int32, i16 Int16, i8 Int8) ENGINE = Memory SET partial_merge_join = 1 diff --git a/dbms/tests/performance/json_extract_rapidjson.xml b/dbms/tests/performance/json_extract_rapidjson.xml index 74f04c9736d..8a2718d4a56 100644 --- a/dbms/tests/performance/json_extract_rapidjson.xml +++ b/dbms/tests/performance/json_extract_rapidjson.xml @@ -12,9 +12,6 @@ - - - diff --git a/dbms/tests/performance/json_extract_simdjson.xml b/dbms/tests/performance/json_extract_simdjson.xml index 6ba1746d5e3..f3a38912b0f 100644 --- a/dbms/tests/performance/json_extract_simdjson.xml +++ b/dbms/tests/performance/json_extract_simdjson.xml @@ -12,9 +12,6 @@ - - - diff --git a/dbms/tests/performance/leftpad.xml b/dbms/tests/performance/leftpad.xml index a38668c745d..a0717adbbd8 100644 --- a/dbms/tests/performance/leftpad.xml +++ b/dbms/tests/performance/leftpad.xml @@ -21,9 +21,6 @@ - - - diff --git a/dbms/tests/performance/linear_regression.xml b/dbms/tests/performance/linear_regression.xml index a04683c2b60..50634b6a60a 100644 --- a/dbms/tests/performance/linear_regression.xml +++ b/dbms/tests/performance/linear_regression.xml @@ -12,9 +12,6 @@ test.hits - - - DROP TABLE IF EXISTS test_model CREATE TABLE test_model engine = Memory as select stochasticLinearRegressionState(0.0001)(Age, Income, ParamPrice, Robotness, RefererHash) as state from test.hits diff --git a/dbms/tests/performance/math.xml b/dbms/tests/performance/math.xml index ea6d79b696d..5f4f302a0e8 100644 --- a/dbms/tests/performance/math.xml +++ b/dbms/tests/performance/math.xml @@ -8,9 +8,6 @@ - - - diff --git a/dbms/tests/performance/merge_table_streams.xml b/dbms/tests/performance/merge_table_streams.xml index 3f19c21109e..f1816e85097 100644 --- a/dbms/tests/performance/merge_table_streams.xml +++ b/dbms/tests/performance/merge_table_streams.xml @@ -15,9 +15,6 @@ - - - 5 diff --git a/dbms/tests/performance/merge_tree_many_partitions.xml b/dbms/tests/performance/merge_tree_many_partitions.xml index 8450a34dc59..6eb110bfab9 100644 --- a/dbms/tests/performance/merge_tree_many_partitions.xml +++ b/dbms/tests/performance/merge_tree_many_partitions.xml @@ -15,9 +15,6 @@ - - - 0 diff --git a/dbms/tests/performance/merge_tree_many_partitions_2.xml b/dbms/tests/performance/merge_tree_many_partitions_2.xml index 1ad06fcbf2f..038bff93057 100644 --- a/dbms/tests/performance/merge_tree_many_partitions_2.xml +++ b/dbms/tests/performance/merge_tree_many_partitions_2.xml @@ -15,9 +15,6 @@ - - - 0 diff --git a/dbms/tests/performance/modulo.xml b/dbms/tests/performance/modulo.xml index 931b160ea00..8e6674d0980 100644 --- a/dbms/tests/performance/modulo.xml +++ b/dbms/tests/performance/modulo.xml @@ -7,9 +7,6 @@ - - - SELECT number % 128 FROM numbers(300000000) FORMAT Null SELECT number % 255 FROM numbers(300000000) FORMAT Null diff --git a/dbms/tests/performance/ngram_distance.xml b/dbms/tests/performance/ngram_distance.xml index 8204a9e9d0b..9a8e4ac72a2 100644 --- a/dbms/tests/performance/ngram_distance.xml +++ b/dbms/tests/performance/ngram_distance.xml @@ -43,7 +43,4 @@ SELECT DISTINCT URL, ngramDistanceCaseInsensitiveUTF8(URL, 'как дЕлА') AS distance FROM hits_100m_single ORDER BY distance ASC LIMIT 50 SELECT DISTINCT URL, ngramDistanceCaseInsensitiveUTF8(URL, 'Чем зАнимаешЬся') AS distance FROM hits_100m_single ORDER BY distance ASC LIMIT 50 - - - diff --git a/dbms/tests/performance/number_formatting_formats.xml b/dbms/tests/performance/number_formatting_formats.xml index df83c5cbf11..aa9929464fb 100644 --- a/dbms/tests/performance/number_formatting_formats.xml +++ b/dbms/tests/performance/number_formatting_formats.xml @@ -14,9 +14,6 @@ - - - diff --git a/dbms/tests/performance/nyc_taxi.xml b/dbms/tests/performance/nyc_taxi.xml index ac97a8e1475..7648e377433 100644 --- a/dbms/tests/performance/nyc_taxi.xml +++ b/dbms/tests/performance/nyc_taxi.xml @@ -12,9 +12,6 @@ - - - default.trips_mergetree diff --git a/dbms/tests/performance/order_by_decimals.xml b/dbms/tests/performance/order_by_decimals.xml index ad6937cd1d6..faf2841e993 100644 --- a/dbms/tests/performance/order_by_decimals.xml +++ b/dbms/tests/performance/order_by_decimals.xml @@ -24,7 +24,4 @@ SELECT toDecimal64(number, 8) AS n FROM numbers(1000000) ORDER BY n DESC SELECT toDecimal128(number, 10) AS n FROM numbers(1000000) ORDER BY n DESC - - - diff --git a/dbms/tests/performance/order_by_read_in_order.xml b/dbms/tests/performance/order_by_read_in_order.xml index d0c5350b3c6..a99dd89846e 100644 --- a/dbms/tests/performance/order_by_read_in_order.xml +++ b/dbms/tests/performance/order_by_read_in_order.xml @@ -12,17 +12,7 @@ - - - - - - - - - - default.hits_100m_single diff --git a/dbms/tests/performance/order_by_single_column.xml b/dbms/tests/performance/order_by_single_column.xml index 98f2bdac17e..ed247641ca8 100644 --- a/dbms/tests/performance/order_by_single_column.xml +++ b/dbms/tests/performance/order_by_single_column.xml @@ -29,7 +29,4 @@ SELECT PageCharset as col FROM hits_100m_single ORDER BY col LIMIT 10000,1 SELECT Title as col FROM hits_100m_single ORDER BY col LIMIT 1000,1 - - - diff --git a/dbms/tests/performance/parse_engine_file.xml b/dbms/tests/performance/parse_engine_file.xml index 8308d8f049f..080acbd53f2 100644 --- a/dbms/tests/performance/parse_engine_file.xml +++ b/dbms/tests/performance/parse_engine_file.xml @@ -16,9 +16,6 @@ - - - diff --git a/dbms/tests/performance/prewhere.xml b/dbms/tests/performance/prewhere.xml index 2ba028562e5..e9a7c4c5a7f 100644 --- a/dbms/tests/performance/prewhere.xml +++ b/dbms/tests/performance/prewhere.xml @@ -12,9 +12,6 @@ - - - default.hits_10m_single diff --git a/dbms/tests/performance/random_printable_ascii.xml b/dbms/tests/performance/random_printable_ascii.xml index a92488b4423..b37469c0aee 100644 --- a/dbms/tests/performance/random_printable_ascii.xml +++ b/dbms/tests/performance/random_printable_ascii.xml @@ -8,12 +8,6 @@ - - - - - - SELECT count() FROM system.numbers WHERE NOT ignore(randomPrintableASCII(10)) SELECT count() FROM system.numbers WHERE NOT ignore(randomPrintableASCII(100)) diff --git a/dbms/tests/performance/range.xml b/dbms/tests/performance/range.xml index b075bad5e43..48463b535ef 100644 --- a/dbms/tests/performance/range.xml +++ b/dbms/tests/performance/range.xml @@ -8,9 +8,6 @@ - - - SELECT count() FROM (SELECT range(number % 100) FROM system.numbers limit 10000000) SELECT count() FROM (SELECT range(0, number % 100, 1) FROM system.numbers limit 10000000) diff --git a/dbms/tests/performance/right.xml b/dbms/tests/performance/right.xml index 8d1304a4604..06d4bdaa93f 100644 --- a/dbms/tests/performance/right.xml +++ b/dbms/tests/performance/right.xml @@ -15,9 +15,6 @@ - - - diff --git a/dbms/tests/performance/round_down.xml b/dbms/tests/performance/round_down.xml index 34c030672b2..5275d69ad84 100644 --- a/dbms/tests/performance/round_down.xml +++ b/dbms/tests/performance/round_down.xml @@ -11,9 +11,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore(roundDuration(rand() % 65536)) SELECT count() FROM system.numbers WHERE NOT ignore(roundDown(rand() % 65536, [0, 1, 10, 30, 60, 120, 180, 240, 300, 600, 1200, 1800, 3600, 7200, 18000, 36000])) diff --git a/dbms/tests/performance/round_methods.xml b/dbms/tests/performance/round_methods.xml index d999feaedd4..b80a8977c33 100644 --- a/dbms/tests/performance/round_methods.xml +++ b/dbms/tests/performance/round_methods.xml @@ -11,9 +11,6 @@ - - - SELECT count() FROM system.numbers WHERE NOT ignore(round(toInt64(number), -2)) SELECT count() FROM system.numbers WHERE NOT ignore(roundBankers(toInt64(number), -2)) diff --git a/dbms/tests/performance/scalar.xml b/dbms/tests/performance/scalar.xml index bb8044685d3..d1bc661c58f 100644 --- a/dbms/tests/performance/scalar.xml +++ b/dbms/tests/performance/scalar.xml @@ -11,9 +11,6 @@ - - - CREATE TABLE cdp_tags (tag_id String, mid_seqs AggregateFunction(groupBitmap, UInt32)) engine=MergeTree() ORDER BY (tag_id) SETTINGS index_granularity=1 CREATE TABLE cdp_orders(order_id UInt64, order_complete_time DateTime, order_total_sales Float32, mid_seq UInt32) engine=MergeTree() PARTITION BY toYYYYMMDD(order_complete_time) ORDER BY (order_complete_time, order_id) diff --git a/dbms/tests/performance/select_format.xml b/dbms/tests/performance/select_format.xml index 723aca5fb97..621247fee1e 100644 --- a/dbms/tests/performance/select_format.xml +++ b/dbms/tests/performance/select_format.xml @@ -14,9 +14,6 @@ - - - 1000000 diff --git a/dbms/tests/performance/set.xml b/dbms/tests/performance/set.xml index 1e62840d8d1..7f3ee4fd4c1 100644 --- a/dbms/tests/performance/set.xml +++ b/dbms/tests/performance/set.xml @@ -14,9 +14,6 @@ - - - diff --git a/dbms/tests/performance/set_hits.xml b/dbms/tests/performance/set_hits.xml index b4afc111ad1..f124de84e64 100644 --- a/dbms/tests/performance/set_hits.xml +++ b/dbms/tests/performance/set_hits.xml @@ -15,9 +15,6 @@ - - - SELECT count() FROM hits_100m_single WHERE UserID IN (SELECT UserID FROM hits_100m_single WHERE AdvEngineID != 0) SELECT count() FROM hits_100m_single WHERE UserID IN (SELECT UserID FROM hits_100m_single) diff --git a/dbms/tests/performance/simple_join_query.xml b/dbms/tests/performance/simple_join_query.xml index 7d8981db2ff..1f6d6ba74d6 100644 --- a/dbms/tests/performance/simple_join_query.xml +++ b/dbms/tests/performance/simple_join_query.xml @@ -11,9 +11,6 @@ - - - CREATE TABLE join_table(A Int64, S0 String, S1 String, S2 String, S3 String)ENGINE = MergeTree ORDER BY A diff --git a/dbms/tests/performance/slices_hits.xml b/dbms/tests/performance/slices_hits.xml index 84aee81c8a1..ad01a607b8a 100644 --- a/dbms/tests/performance/slices_hits.xml +++ b/dbms/tests/performance/slices_hits.xml @@ -12,9 +12,6 @@ - - - test.hits diff --git a/dbms/tests/performance/string_join.xml b/dbms/tests/performance/string_join.xml index 6c0ad83d5b4..228fe3182b8 100644 --- a/dbms/tests/performance/string_join.xml +++ b/dbms/tests/performance/string_join.xml @@ -7,9 +7,6 @@ - - - default.hits_10m_single diff --git a/dbms/tests/performance/string_set.xml b/dbms/tests/performance/string_set.xml index 08a74e3e8f9..cf6261d6d60 100644 --- a/dbms/tests/performance/string_set.xml +++ b/dbms/tests/performance/string_set.xml @@ -7,9 +7,6 @@ - - - hits_10m_single diff --git a/dbms/tests/performance/string_sort.xml b/dbms/tests/performance/string_sort.xml index 7b33bb20d52..c4c9463a9aa 100644 --- a/dbms/tests/performance/string_sort.xml +++ b/dbms/tests/performance/string_sort.xml @@ -47,7 +47,4 @@ - - - diff --git a/dbms/tests/performance/trim_numbers.xml b/dbms/tests/performance/trim_numbers.xml index 48fdb710cd9..997272e95f6 100644 --- a/dbms/tests/performance/trim_numbers.xml +++ b/dbms/tests/performance/trim_numbers.xml @@ -11,9 +11,6 @@ - - - diff --git a/dbms/tests/performance/trim_urls.xml b/dbms/tests/performance/trim_urls.xml index 2b672d9d97d..23dd3f77f6e 100644 --- a/dbms/tests/performance/trim_urls.xml +++ b/dbms/tests/performance/trim_urls.xml @@ -15,9 +15,6 @@ - - - diff --git a/dbms/tests/performance/trim_whitespace.xml b/dbms/tests/performance/trim_whitespace.xml index 26d89859432..e886359a368 100644 --- a/dbms/tests/performance/trim_whitespace.xml +++ b/dbms/tests/performance/trim_whitespace.xml @@ -10,9 +10,6 @@ - - - diff --git a/dbms/tests/performance/uniq.xml b/dbms/tests/performance/uniq.xml index c44a4e2ca58..307ad6f88ef 100644 --- a/dbms/tests/performance/uniq.xml +++ b/dbms/tests/performance/uniq.xml @@ -13,9 +13,6 @@ - - - 30000000000 diff --git a/dbms/tests/performance/url_hits.xml b/dbms/tests/performance/url_hits.xml index a251a2706b8..d4e504cd1b8 100644 --- a/dbms/tests/performance/url_hits.xml +++ b/dbms/tests/performance/url_hits.xml @@ -15,9 +15,6 @@ - - - diff --git a/dbms/tests/performance/vectorize_aggregation_combinators.xml b/dbms/tests/performance/vectorize_aggregation_combinators.xml index a1afb2e6cc8..73024f454f9 100644 --- a/dbms/tests/performance/vectorize_aggregation_combinators.xml +++ b/dbms/tests/performance/vectorize_aggregation_combinators.xml @@ -12,9 +12,6 @@ - - - 1 diff --git a/dbms/tests/performance/visit_param_extract_raw.xml b/dbms/tests/performance/visit_param_extract_raw.xml index 02b8224c361..0faa43088e7 100644 --- a/dbms/tests/performance/visit_param_extract_raw.xml +++ b/dbms/tests/performance/visit_param_extract_raw.xml @@ -8,12 +8,6 @@ - - - - - - diff --git a/dbms/tests/performance/website.xml b/dbms/tests/performance/website.xml index 4cb350a60a1..83a1c3607c7 100644 --- a/dbms/tests/performance/website.xml +++ b/dbms/tests/performance/website.xml @@ -16,9 +16,6 @@ - - - 20000000000 diff --git a/dbms/tests/queries/0_stateless/00597_push_down_predicate.reference b/dbms/tests/queries/0_stateless/00597_push_down_predicate.reference index f64243e9be7..9fde80689f1 100644 --- a/dbms/tests/queries/0_stateless/00597_push_down_predicate.reference +++ b/dbms/tests/queries/0_stateless/00597_push_down_predicate.reference @@ -13,7 +13,7 @@ SELECT \n a, \n b\nFROM \n(\n SELECT \n 1 AS a, \n 1 AS b -------Need push down------- SELECT toString(value) AS value\nFROM \n(\n SELECT 1 AS value\n) 1 -SELECT id\nFROM \n(\n SELECT 1 AS id\n UNION ALL\n SELECT 2 AS `2`\n WHERE 0\n)\nWHERE id = 1 +SELECT id\nFROM \n(\n SELECT 1 AS id\n UNION ALL\n SELECT 2 AS `--predicate_optimizer_0`\n WHERE 0\n)\nWHERE id = 1 1 SELECT id\nFROM \n(\n SELECT arrayJoin([1, 2, 3]) AS id\n WHERE id = 1\n)\nWHERE id = 1 1 diff --git a/dbms/tests/queries/0_stateless/00909_kill_not_initialized_query.sh b/dbms/tests/queries/0_stateless/00909_kill_not_initialized_query.sh index 677709dd2c0..b5982fd0dc8 100755 --- a/dbms/tests/queries/0_stateless/00909_kill_not_initialized_query.sh +++ b/dbms/tests/queries/0_stateless/00909_kill_not_initialized_query.sh @@ -34,7 +34,7 @@ $CLICKHOUSE_CLIENT -q "KILL QUERY WHERE query='$query_to_kill' ASYNC" &>/dev/nul sleep 1 # Kill $query_for_pending SYNC. This query is not blocker, so it should be killed fast. -timeout 10 $CLICKHOUSE_CLIENT -q "KILL QUERY WHERE query='$query_for_pending' SYNC" &>/dev/null +timeout 20 $CLICKHOUSE_CLIENT -q "KILL QUERY WHERE query='$query_for_pending' SYNC" &>/dev/null # Both queries have to be killed, doesn't matter with SYNC or ASYNC kill for run in {1..15} diff --git a/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.reference b/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.reference index f2965419d0a..f60d942d406 100644 --- a/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.reference +++ b/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.reference @@ -39,6 +39,6 @@ SimpleAggregateFunction(sum, Double) 7 14 8 16 9 18 -1 1 2 2.2.2.2 -10 2222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222 20 20.20.20.20 -SimpleAggregateFunction(anyLast, Nullable(String)) SimpleAggregateFunction(anyLast, LowCardinality(Nullable(String))) SimpleAggregateFunction(anyLast, IPv4) +1 1 2 2.2.2.2 3 +10 2222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222 20 20.20.20.20 5 +SimpleAggregateFunction(anyLast, Nullable(String)) SimpleAggregateFunction(anyLast, LowCardinality(Nullable(String))) SimpleAggregateFunction(anyLast, IPv4) SimpleAggregateFunction(groupBitOr, UInt32) diff --git a/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.sql b/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.sql index c010aad7e5a..037032a84cc 100644 --- a/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.sql +++ b/dbms/tests/queries/0_stateless/00915_simple_aggregate_function.sql @@ -19,14 +19,20 @@ select * from simple; -- complex types drop table if exists simple; -create table simple (id UInt64,nullable_str SimpleAggregateFunction(anyLast,Nullable(String)),low_str SimpleAggregateFunction(anyLast,LowCardinality(Nullable(String))),ip SimpleAggregateFunction(anyLast,IPv4)) engine=AggregatingMergeTree order by id; -insert into simple values(1,'1','1','1.1.1.1'); -insert into simple values(1,null,'2','2.2.2.2'); +create table simple ( + id UInt64, + nullable_str SimpleAggregateFunction(anyLast,Nullable(String)), + low_str SimpleAggregateFunction(anyLast,LowCardinality(Nullable(String))), + ip SimpleAggregateFunction(anyLast,IPv4), + status SimpleAggregateFunction(groupBitOr, UInt32) +) engine=AggregatingMergeTree order by id; +insert into simple values(1,'1','1','1.1.1.1', 1); +insert into simple values(1,null,'2','2.2.2.2', 2); -- String longer then MAX_SMALL_STRING_SIZE (actual string length is 100) -insert into simple values(10,'10','10','10.10.10.10'); -insert into simple values(10,'2222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222','20','20.20.20.20'); +insert into simple values(10,'10','10','10.10.10.10', 4); +insert into simple values(10,'2222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222','20','20.20.20.20', 1); select * from simple final; -select toTypeName(nullable_str),toTypeName(low_str),toTypeName(ip) from simple limit 1; +select toTypeName(nullable_str),toTypeName(low_str),toTypeName(ip),toTypeName(status) from simple limit 1; drop table simple; diff --git a/dbms/tests/queries/0_stateless/00929_multi_match_edit_distance.sql b/dbms/tests/queries/0_stateless/00929_multi_match_edit_distance.sql index 48b31070204..c0f39a4f201 100644 --- a/dbms/tests/queries/0_stateless/00929_multi_match_edit_distance.sql +++ b/dbms/tests/queries/0_stateless/00929_multi_match_edit_distance.sql @@ -3,8 +3,8 @@ SET send_logs_level = 'none'; select 0 = multiFuzzyMatchAny('abc', 0, ['a1c']) from system.numbers limit 5; select 1 = multiFuzzyMatchAny('abc', 1, ['a1c']) from system.numbers limit 5; select 1 = multiFuzzyMatchAny('abc', 2, ['a1c']) from system.numbers limit 5; -select 1 = multiFuzzyMatchAny('abc', 3, ['a1c']) from system.numbers limit 5; -- { serverError 49 } -select 1 = multiFuzzyMatchAny('abc', 4, ['a1c']) from system.numbers limit 5; -- { serverError 49 } +select 1 = multiFuzzyMatchAny('abc', 3, ['a1c']) from system.numbers limit 5; -- { serverError 36 } +select 1 = multiFuzzyMatchAny('abc', 4, ['a1c']) from system.numbers limit 5; -- { serverError 36 } select 1 = multiFuzzyMatchAny('leftabcright', 1, ['a1c']) from system.numbers limit 5; @@ -14,7 +14,7 @@ select 0 = multiFuzzyMatchAny('halo some wrld', 2, ['^hello.*world$']); select 1 = multiFuzzyMatchAny('halo some wrld', 2, ['^hello.*world$', '^halo.*world$']); select 1 = multiFuzzyMatchAny('halo some wrld', 2, ['^halo.*world$', '^hello.*world$']); select 1 = multiFuzzyMatchAny('halo some wrld', 3, ['^hello.*world$']); -select 1 = multiFuzzyMatchAny('hello some world', 10, ['^hello.*world$']); -- { serverError 49 } +select 1 = multiFuzzyMatchAny('hello some world', 10, ['^hello.*world$']); -- { serverError 36 } select 1 = multiFuzzyMatchAny('hello some world', -1, ['^hello.*world$']); -- { serverError 43 } select 1 = multiFuzzyMatchAny('hello some world', 10000000000, ['^hello.*world$']); -- { serverError 44 } select 1 = multiFuzzyMatchAny('http://hyperscan_is_nice.ru/st', 2, ['http://hyperscan_is_nice.ru/(st\\d\\d$|st\\d\\d\\.|st1[0-4]\\d|st150|st\\d$|gl|rz|ch)']); diff --git a/dbms/tests/queries/0_stateless/00949_format.sql b/dbms/tests/queries/0_stateless/00949_format.sql index 1786a2a3e1e..433ababde9d 100644 --- a/dbms/tests/queries/0_stateless/00949_format.sql +++ b/dbms/tests/queries/0_stateless/00949_format.sql @@ -16,25 +16,25 @@ select 100 = length(format(concat((select arrayStringConcat(arrayMap(x ->'}', ra select format('', 'first'); select concat('third', 'first', 'second')=format('{2}{0}{1}', 'first', 'second', 'third'); -select format('{', ''); -- { serverError 49 } -select format('{{}', ''); -- { serverError 49 } -select format('{ {}', ''); -- { serverError 49 } -select format('}', ''); -- { serverError 49 } +select format('{', ''); -- { serverError 36 } +select format('{{}', ''); -- { serverError 36 } +select format('{ {}', ''); -- { serverError 36 } +select format('}', ''); -- { serverError 36 } select format('{{', ''); -select format('{}}', ''); -- { serverError 49 } +select format('{}}', ''); -- { serverError 36 } select format('}}', ''); -select format('{2 }', ''); -- { serverError 49 } -select format('{}{}{}{}{}{} }{}', '', '', '', '', '', '', ''); -- { serverError 49 } -select format('{sometext}', ''); -- { serverError 49 } -select format('{\0sometext}', ''); -- { serverError 49 } -select format('{1023}', ''); -- { serverError 49 } -select format('{10000000000000000000000000000000000000000000000000}', ''); -- { serverError 49 } -select format('{} {0}', '', ''); -- { serverError 49 } -select format('{0} {}', '', ''); -- { serverError 49 } -select format('Hello {} World {} {}{}', 'first', 'second', 'third') from system.numbers limit 2; -- { serverError 49 } -select format('Hello {0} World {1} {2}{3}', 'first', 'second', 'third') from system.numbers limit 2; -- { serverError 49 } +select format('{2 }', ''); -- { serverError 36 } +select format('{}{}{}{}{}{} }{}', '', '', '', '', '', '', ''); -- { serverError 36 } +select format('{sometext}', ''); -- { serverError 36 } +select format('{\0sometext}', ''); -- { serverError 36 } +select format('{1023}', ''); -- { serverError 36 } +select format('{10000000000000000000000000000000000000000000000000}', ''); -- { serverError 36 } +select format('{} {0}', '', ''); -- { serverError 36 } +select format('{0} {}', '', ''); -- { serverError 36 } +select format('Hello {} World {} {}{}', 'first', 'second', 'third') from system.numbers limit 2; -- { serverError 36 } +select format('Hello {0} World {1} {2}{3}', 'first', 'second', 'third') from system.numbers limit 2; -- { serverError 36 } -select 50 = length(format((select arrayStringConcat(arrayMap(x ->'{', range(101)))), '')); -- { serverError 49 } +select 50 = length(format((select arrayStringConcat(arrayMap(x ->'{', range(101)))), '')); -- { serverError 36 } select format('{}{}{}', materialize(toFixedString('a', 1)), materialize(toFixedString('b', 1)), materialize(toFixedString('c', 1))) == 'abc'; select format('{}{}{}', materialize(toFixedString('a', 1)), materialize('b'), materialize(toFixedString('c', 1))) == 'abc'; diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.reference new file mode 100644 index 00000000000..ebf18a51290 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.reference @@ -0,0 +1,18 @@ +1 1 +2 1 +3 1 +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.sql new file mode 100644 index 00000000000..2e256615925 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT a FROM (SELECT a FROM test.mt); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.reference new file mode 100644 index 00000000000..7a596e87ed6 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.reference @@ -0,0 +1,6 @@ +1 hello 2 +1 hello 2 +1 hello 3 +2 hello 3 +1 hello 3 +2 hello 3 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.sql new file mode 100644 index 00000000000..f89455803f3 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join.sql @@ -0,0 +1,29 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.A; +DROP TABLE IF EXISTS test.B; + +CREATE TABLE test.A (id Int32) Engine=Memory; +CREATE TABLE test.B (id Int32, name String) Engine=Memory; + +CREATE LIVE VIEW test.lv AS SELECT id, name FROM ( SELECT A.id, B.name FROM test.A as A, test.B as B WHERE A.id = B.id ); + +SELECT * FROM test.lv; + +INSERT INTO test.A VALUES (1); +INSERT INTO test.B VALUES (1, 'hello'); + +SELECT *,_version FROM test.lv ORDER BY id; +SELECT *,_version FROM test.lv ORDER BY id; + +INSERT INTO test.A VALUES (2) +INSERT INTO test.B VALUES (2, 'hello') + +SELECT *,_version FROM test.lv ORDER BY id; +SELECT *,_version FROM test.lv ORDER BY id; + +DROP TABLE test.lv; +DROP TABLE test.A; +DROP TABLE test.B; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.reference new file mode 100644 index 00000000000..7a596e87ed6 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.reference @@ -0,0 +1,6 @@ +1 hello 2 +1 hello 2 +1 hello 3 +2 hello 3 +1 hello 3 +2 hello 3 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.sql new file mode 100644 index 00000000000..b8eea8c71e5 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_join_no_alias.sql @@ -0,0 +1,29 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.A; +DROP TABLE IF EXISTS test.B; + +CREATE TABLE test.A (id Int32) Engine=Memory; +CREATE TABLE test.B (id Int32, name String) Engine=Memory; + +CREATE LIVE VIEW test.lv AS SELECT id, name FROM ( SELECT test.A.id, test.B.name FROM test.A, test.B WHERE test.A.id = test.B.id); + +SELECT * FROM test.lv; + +INSERT INTO test.A VALUES (1); +INSERT INTO test.B VALUES (1, 'hello'); + +SELECT *,_version FROM test.lv ORDER BY id; +SELECT *,_version FROM test.lv ORDER BY id; + +INSERT INTO test.A VALUES (2) +INSERT INTO test.B VALUES (2, 'hello') + +SELECT *,_version FROM test.lv ORDER BY id; +SELECT *,_version FROM test.lv ORDER BY id; + +DROP TABLE test.lv; +DROP TABLE test.A; +DROP TABLE test.B; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.reference new file mode 100644 index 00000000000..ebf18a51290 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.reference @@ -0,0 +1,18 @@ +1 1 +2 1 +3 1 +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.sql new file mode 100644 index 00000000000..f2decda148b --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT a FROM ( SELECT * FROM ( SELECT a FROM (SELECT a FROM test.mt) ) ); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.reference new file mode 100644 index 00000000000..75236c0daf7 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.reference @@ -0,0 +1,4 @@ +6 1 +6 1 +12 2 +12 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.sql new file mode 100644 index 00000000000..4dc7b02fc51 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT * FROM ( SELECT sum(a) FROM ( SELECT a FROM (SELECT a FROM test.mt) ) ); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.reference new file mode 100644 index 00000000000..75236c0daf7 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.reference @@ -0,0 +1,4 @@ +6 1 +6 1 +12 2 +12 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.sql new file mode 100644 index 00000000000..2e10eefda49 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_nested_with_aggregation_table_alias.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT * FROM ( SELECT sum(boo.x) FROM ( SELECT foo.x FROM (SELECT a AS x FROM test.mt) AS foo) AS boo ); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.reference new file mode 100644 index 00000000000..ebf18a51290 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.reference @@ -0,0 +1,18 @@ +1 1 +2 1 +3 1 +1 1 +2 1 +3 1 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 +1 2 +2 2 +3 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.sql new file mode 100644 index 00000000000..d5da0854899 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_table_alias.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT foo.x FROM (SELECT a AS x FROM test.mt) AS foo; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.reference new file mode 100644 index 00000000000..75236c0daf7 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.reference @@ -0,0 +1,4 @@ +6 1 +6 1 +12 2 +12 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.sql new file mode 100644 index 00000000000..bc15e8a7356 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT sum(a) FROM (SELECT a FROM test.mt); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.reference b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.reference new file mode 100644 index 00000000000..75236c0daf7 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.reference @@ -0,0 +1,4 @@ +6 1 +6 1 +12 2 +12 2 diff --git a/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.sql b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.sql new file mode 100644 index 00000000000..4dd7a12b190 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00973_live_view_with_subquery_select_with_aggregation_in_subquery.sql @@ -0,0 +1,21 @@ +SET allow_experimental_live_view = 1; + +DROP TABLE IF EXISTS test.lv; +DROP TABLE IF EXISTS test.mt; + +CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); +CREATE LIVE VIEW test.lv AS SELECT * FROM (SELECT sum(a) FROM test.mt); + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +INSERT INTO test.mt VALUES (1),(2),(3); + +SELECT *,_version FROM test.lv; +SELECT *,_version FROM test.lv; + +DROP TABLE test.lv; +DROP TABLE test.mt; + diff --git a/dbms/tests/queries/0_stateless/00979_live_view_watch_live_moving_avg.py b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_moving_avg.py new file mode 100755 index 00000000000..30d5e6d67b3 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_moving_avg.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python +import os +import sys +import signal + +CURDIR = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(CURDIR, 'helpers')) + +from client import client, prompt, end_of_block + +log = None +# uncomment the line below for debugging +#log=sys.stdout + +with client(name='client1>', log=log) as client1, client(name='client2>', log=log) as client2: + client1.expect(prompt) + client2.expect(prompt) + + client1.send('SET allow_experimental_live_view = 1') + client1.expect(prompt) + client2.send('SET allow_experimental_live_view = 1') + client2.expect(prompt) + + client1.send('DROP TABLE IF EXISTS test.lv') + client1.expect(prompt) + client1.send(' DROP TABLE IF EXISTS test.mt') + client1.expect(prompt) + client1.send('CREATE TABLE test.mt (a Int32, id Int32) Engine=Memory') + client1.expect(prompt) + client1.send('CREATE LIVE VIEW test.lv AS SELECT sum(a)/2 FROM (SELECT a, id FROM ( SELECT a, id FROM test.mt ORDER BY id DESC LIMIT 2 ) ORDER BY id DESC LIMIT 2)') + client1.expect(prompt) + client1.send('WATCH test.lv') + client1.expect(r'0.*1' + end_of_block) + client2.send('INSERT INTO test.mt VALUES (1, 1),(2, 2),(3, 3)') + client1.expect(r'2\.5.*2' + end_of_block) + client2.expect(prompt) + client2.send('INSERT INTO test.mt VALUES (4, 4),(5, 5),(6, 6)') + client1.expect(r'5\.5.*3' + end_of_block) + client2.expect(prompt) + for v, i in enumerate(range(7,129)): + client2.send('INSERT INTO test.mt VALUES (%d, %d)' % (i, i)) + client1.expect(r'%.1f.*%d' % (i-0.5, 4+v) + end_of_block) + client2.expect(prompt) + # send Ctrl-C + client1.send('\x03', eol='') + match = client1.expect('(%s)|([#\$] )' % prompt) + if match.groups()[1]: + client1.send(client1.command) + client1.expect(prompt) + client1.send('DROP TABLE test.lv') + client1.expect(prompt) + client1.send('DROP TABLE test.mt') + client1.expect(prompt) diff --git a/dbms/tests/queries/0_stateless/00979_live_view_watch_live_moving_avg.reference b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_moving_avg.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dbms/tests/queries/0_stateless/00979_live_view_watch_live_with_subquery.py b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_with_subquery.py new file mode 100755 index 00000000000..44c923d75d8 --- /dev/null +++ b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_with_subquery.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python +import os +import sys +import signal + +CURDIR = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(CURDIR, 'helpers')) + +from client import client, prompt, end_of_block + +log = None +# uncomment the line below for debugging +#log=sys.stdout + +with client(name='client1>', log=log) as client1, client(name='client2>', log=log) as client2: + client1.expect(prompt) + client2.expect(prompt) + + client1.send('SET allow_experimental_live_view = 1') + client1.expect(prompt) + client2.send('SET allow_experimental_live_view = 1') + client2.expect(prompt) + + client1.send('DROP TABLE IF EXISTS test.lv') + client1.expect(prompt) + client1.send(' DROP TABLE IF EXISTS test.mt') + client1.expect(prompt) + client1.send('CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple()') + client1.expect(prompt) + client1.send('CREATE LIVE VIEW test.lv AS SELECT * FROM ( SELECT sum(A.a) FROM (SELECT * FROM test.mt) AS A )') + client1.expect(prompt) + client1.send('WATCH test.lv') + client1.expect(r'0.*1' + end_of_block) + client2.send('INSERT INTO test.mt VALUES (1),(2),(3)') + client1.expect(r'6.*2' + end_of_block) + client2.expect(prompt) + client2.send('INSERT INTO test.mt VALUES (4),(5),(6)') + client1.expect(r'21.*3' + end_of_block) + client2.expect(prompt) + for i in range(1,129): + client2.send('INSERT INTO test.mt VALUES (1)') + client1.expect(r'%d.*%d' % (21+i, 3+i) + end_of_block) + client2.expect(prompt) + # send Ctrl-C + client1.send('\x03', eol='') + match = client1.expect('(%s)|([#\$] )' % prompt) + if match.groups()[1]: + client1.send(client1.command) + client1.expect(prompt) + client1.send('DROP TABLE test.lv') + client1.expect(prompt) + client1.send('DROP TABLE test.mt') + client1.expect(prompt) diff --git a/dbms/tests/queries/0_stateless/00979_live_view_watch_live_with_subquery.reference b/dbms/tests/queries/0_stateless/00979_live_view_watch_live_with_subquery.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.reference b/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.reference index 7f9fcbb2e9c..49d86fc2fbf 100644 --- a/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.reference +++ b/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.reference @@ -1,3 +1,4 @@ temporary_live_view_timeout 5 live_view_heartbeat_interval 15 +lv 0 diff --git a/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.sql b/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.sql index 037c2a9e587..0c3bb7f815d 100644 --- a/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.sql +++ b/dbms/tests/queries/0_stateless/00980_create_temporary_live_view.sql @@ -10,8 +10,8 @@ SET temporary_live_view_timeout=1; CREATE TABLE test.mt (a Int32) Engine=MergeTree order by tuple(); CREATE TEMPORARY LIVE VIEW test.lv AS SELECT sum(a) FROM test.mt; -SHOW TABLES LIKE 'lv'; +SHOW TABLES FROM test LIKE 'lv'; SELECT sleep(2); -SHOW TABLES LIKE 'lv'; +SHOW TABLES FROM test LIKE 'lv'; DROP TABLE test.mt; diff --git a/dbms/tests/queries/0_stateless/01013_totals_without_aggregation.sql b/dbms/tests/queries/0_stateless/01013_totals_without_aggregation.sql index bed393b63d3..584a8994767 100755 --- a/dbms/tests/queries/0_stateless/01013_totals_without_aggregation.sql +++ b/dbms/tests/queries/0_stateless/01013_totals_without_aggregation.sql @@ -1,6 +1,6 @@ SELECT 11 AS n GROUP BY n WITH TOTALS; SELECT 12 AS n GROUP BY n WITH ROLLUP; SELECT 13 AS n GROUP BY n WITH CUBE; -SELECT 1 AS n WITH TOTALS; -- { serverError 49 } -SELECT 1 AS n WITH ROLLUP; -- { serverError 49 } -SELECT 1 AS n WITH CUBE; -- { serverError 49 } +SELECT 1 AS n WITH TOTALS; -- { serverError 48 } +SELECT 1 AS n WITH ROLLUP; -- { serverError 48 } +SELECT 1 AS n WITH CUBE; -- { serverError 48 } diff --git a/dbms/tests/queries/0_stateless/01033_substr_negative_size_arg.reference b/dbms/tests/queries/0_stateless/01033_substr_negative_size_arg.reference index 98c07557034..db3a106ac7f 100644 --- a/dbms/tests/queries/0_stateless/01033_substr_negative_size_arg.reference +++ b/dbms/tests/queries/0_stateless/01033_substr_negative_size_arg.reference @@ -1,8 +1,8 @@ -lickhous -lickhous -lickhous -lickhous -lickhous -lickhous -lickhous -lickhous +lickhou +lickhou +lickhou +lickhou +lickhou +lickhou +lickhou +lickhou diff --git a/dbms/tests/queries/0_stateless/01052_array_reduce_exception.sql b/dbms/tests/queries/0_stateless/01052_array_reduce_exception.sql index 71c030a055c..2bdfc2136a2 100644 --- a/dbms/tests/queries/0_stateless/01052_array_reduce_exception.sql +++ b/dbms/tests/queries/0_stateless/01052_array_reduce_exception.sql @@ -1 +1 @@ -SELECT arrayReduce('aggThrow(0.0001)', range(number % 10)) FROM system.numbers; -- { serverError 503 } +SELECT arrayReduce('aggThrow(0.0001)', range(number % 10)) FROM system.numbers FORMAT Null; -- { serverError 503 } diff --git a/dbms/tests/queries/0_stateless/01053_if_chain_check.reference b/dbms/tests/queries/0_stateless/01053_if_chain_check.reference new file mode 100644 index 00000000000..4211be303d5 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01053_if_chain_check.reference @@ -0,0 +1,2002 @@ +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +1 +2 +4 +5 +7 +8 +10 +11 +14 +17 +19 +20 +23 +25 +29 +31 +35 +38 +40 +41 +43 +47 +49 +50 +53 +55 +59 +61 +62 +67 +70 +71 +73 +77 +79 +82 +83 +85 +86 +89 +94 +95 +97 +98 +100 +101 +103 +106 +107 +109 +113 +115 +118 +119 +121 +122 +124 +125 +127 +131 +133 +134 +137 +139 +142 +145 +146 +149 +151 +155 +157 +158 +161 +163 +164 +166 +167 +172 +173 +175 +178 +179 +181 +187 +188 +190 +191 +193 +194 +197 +199 +200 +202 +203 +205 +206 +209 +211 +212 +214 +215 +217 +218 +223 +226 +227 +229 +233 +235 +236 +239 +241 +244 +245 +248 +250 +251 +253 +254 +257 +262 +263 +265 +266 +268 +269 +271 +274 +275 +277 +278 +281 +283 +284 +287 +289 +292 +293 +295 +298 +301 +302 +305 +307 +310 +311 +313 +314 +316 +317 +319 +323 +326 +328 +329 +331 +332 +334 +335 +337 +341 +343 +344 +346 +347 +349 +350 +353 +355 +356 +358 +359 +361 +362 +365 +367 +371 +373 +376 +379 +382 +383 +385 +386 +388 +389 +391 +394 +395 +397 +398 +401 +404 +409 +410 +412 +413 +415 +419 +421 +422 +424 +425 +427 +428 +430 +431 +433 +434 +436 +437 +439 +443 +445 +446 +449 +451 +452 +454 +457 +458 +461 +463 +466 +467 +469 +470 +472 +473 +475 +478 +479 +482 +485 +487 +488 +490 +491 +493 +497 +499 +500 +502 +503 +505 +508 +509 +511 +514 +515 +517 +521 +523 +524 +526 +527 +529 +530 +535 +536 +538 +539 +541 +542 +545 +547 +548 +551 +553 +554 +556 +557 +562 +563 +565 +566 +568 +569 +571 +574 +575 +577 +581 +583 +584 +586 +587 +589 +590 +593 +595 +596 +599 +601 +602 +604 +605 +607 +610 +613 +614 +617 +619 +620 +622 +623 +625 +626 +628 +631 +632 +634 +635 +641 +643 +647 +649 +652 +653 +655 +658 +659 +661 +662 +664 +665 +667 +668 +670 +671 +673 +674 +677 +679 +683 +685 +686 +691 +692 +694 +695 +697 +698 +701 +706 +707 +709 +710 +712 +713 +716 +718 +719 +721 +722 +724 +725 +727 +730 +731 +733 +734 +737 +739 +742 +743 +745 +746 +749 +751 +755 +757 +758 +761 +763 +764 +766 +769 +772 +773 +775 +776 +778 +779 +781 +785 +787 +788 +790 +791 +794 +796 +797 +799 +802 +803 +805 +808 +809 +811 +815 +817 +818 +820 +821 +823 +824 +826 +827 +829 +830 +833 +835 +838 +839 +841 +842 +844 +847 +853 +854 +856 +857 +859 +860 +862 +863 +865 +866 +869 +872 +875 +877 +878 +881 +883 +886 +887 +889 +890 +892 +893 +895 +898 +899 +901 +904 +905 +907 +908 +911 +913 +914 +916 +917 +919 +922 +926 +929 +931 +932 +934 +935 +937 +938 +940 +941 +943 +947 +950 +953 +955 +956 +958 +959 +961 +964 +965 +967 +970 +971 +973 +974 +977 +979 +982 +983 +985 +989 +991 +994 +995 +997 +998 +1000 +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +\N +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 +42 diff --git a/dbms/tests/queries/0_stateless/01053_if_chain_check.sql b/dbms/tests/queries/0_stateless/01053_if_chain_check.sql new file mode 100644 index 00000000000..3a98b85c473 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01053_if_chain_check.sql @@ -0,0 +1,3 @@ +SELECT x FROM (SELECT number % 16 = 0 ? nan : (number % 24 = 0 ? NULL : (number % 37 = 0 ? nan : (number % 34 = 0 ? nan : (number % 3 = 0 ? NULL : (number % 68 = 0 ? 42 : (number % 28 = 0 ? nan : (number % 46 = 0 ? nan : (number % 13 = 0 ? nan : (number % 27 = 0 ? NULL : (number % 39 = 0 ? NULL : (number % 27 = 0 ? NULL : (number % 30 = 0 ? NULL : (number % 72 = 0 ? NULL : (number % 36 = 0 ? NULL : (number % 51 = 0 ? NULL : (number % 58 = 0 ? nan : (number % 26 = 0 ? 42 : (number % 13 = 0 ? nan : (number % 12 = 0 ? NULL : (number % 22 = 0 ? nan : (number % 36 = 0 ? NULL : (number % 63 = 0 ? NULL : (number % 27 = 0 ? NULL : (number % 18 = 0 ? NULL : (number % 69 = 0 ? NULL : (number % 76 = 0 ? nan : (number % 42 = 0 ? NULL : (number % 9 = 0 ? NULL : (toFloat64(number)))))))))))))))))))))))))))))) AS x FROM system.numbers LIMIT 1001) ORDER BY x ASC NULLS FIRST; + +SELECT x FROM (SELECT number % 22 = 0 ? nan : (number % 56 = 0 ? 42 : (number % 45 = 0 ? NULL : (number % 47 = 0 ? 42 : (number % 39 = 0 ? NULL : (number % 1 = 0 ? nan : (number % 43 = 0 ? nan : (number % 40 = 0 ? nan : (number % 42 = 0 ? NULL : (number % 26 = 0 ? 42 : (number % 41 = 0 ? 42 : (number % 6 = 0 ? NULL : (number % 39 = 0 ? NULL : (number % 34 = 0 ? nan : (number % 74 = 0 ? 42 : (number % 40 = 0 ? nan : (number % 37 = 0 ? nan : (number % 51 = 0 ? NULL : (number % 46 = 0 ? nan : (toFloat64(number)))))))))))))))))))) AS x FROM system.numbers LIMIT 1001) ORDER BY x ASC NULLS FIRST; \ No newline at end of file diff --git a/dbms/tests/queries/0_stateless/01055_prewhere_bugs.reference b/dbms/tests/queries/0_stateless/01055_prewhere_bugs.reference new file mode 100644 index 00000000000..cd0e6a397a1 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01055_prewhere_bugs.reference @@ -0,0 +1,2 @@ +43 + 1 diff --git a/dbms/tests/queries/0_stateless/01055_prewhere_bugs.sql b/dbms/tests/queries/0_stateless/01055_prewhere_bugs.sql new file mode 100644 index 00000000000..d9a0256ce52 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01055_prewhere_bugs.sql @@ -0,0 +1,17 @@ +DROP TABLE IF EXISTS test_prewhere_default_column; +DROP TABLE IF EXISTS test_prewhere_column_type; + +CREATE TABLE test_prewhere_default_column (APIKey UInt8, SessionType UInt8) ENGINE = MergeTree() PARTITION BY APIKey ORDER BY tuple(); +INSERT INTO test_prewhere_default_column VALUES( 42, 42 ); +ALTER TABLE test_prewhere_default_column ADD COLUMN OperatingSystem UInt64 DEFAULT SessionType+1; + +SELECT OperatingSystem FROM test_prewhere_default_column PREWHERE SessionType = 42; + + +CREATE TABLE test_prewhere_column_type (`a` LowCardinality(String), `x` Nullable(Int32)) ENGINE = MergeTree ORDER BY tuple(); +INSERT INTO test_prewhere_column_type VALUES ('', 2); + +SELECT a, y FROM test_prewhere_column_type prewhere (x = 2) AS y; + +DROP TABLE test_prewhere_default_column; +DROP TABLE test_prewhere_column_type; diff --git a/dbms/tests/queries/0_stateless/01056_create_table_as.reference b/dbms/tests/queries/0_stateless/01056_create_table_as.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dbms/tests/queries/0_stateless/01056_create_table_as.sql b/dbms/tests/queries/0_stateless/01056_create_table_as.sql new file mode 100644 index 00000000000..868e1f082dd --- /dev/null +++ b/dbms/tests/queries/0_stateless/01056_create_table_as.sql @@ -0,0 +1,32 @@ +DROP TABLE IF EXISTS t1; +CREATE TABLE t1 (key Int) Engine=Memory(); +CREATE TABLE t2 AS t1; +DROP TABLE t2; + +-- live view +SET allow_experimental_live_view=1; +CREATE LIVE VIEW lv AS SELECT * FROM t1; +CREATE TABLE t3 AS lv; -- { serverError 80; } +DROP TABLE lv; + +-- view +CREATE VIEW v AS SELECT * FROM t1; +CREATE TABLE t3 AS v; -- { serverError 80; } +DROP TABLE v; + +-- dictionary +DROP DATABASE if exists test_01056_dict_data; +CREATE DATABASE test_01056_dict_data; +CREATE TABLE test_01056_dict_data.dict_data (key Int, value UInt16) Engine=Memory(); +CREATE DICTIONARY dict +( + `key` UInt64, + `value` UInt16 +) +PRIMARY KEY key +SOURCE(CLICKHOUSE( + HOST '127.0.0.1' PORT 9000 + TABLE 'dict_data' DB 'test_01056_dict_data' USER 'default' PASSWORD '')) +LIFETIME(MIN 0 MAX 0) +LAYOUT(SPARSE_HASHED()); +CREATE TABLE t3 AS dict; -- { serverError 80; } diff --git a/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.reference b/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.reference new file mode 100644 index 00000000000..019e95cb359 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.reference @@ -0,0 +1,28 @@ +SELECT \n k, \n v, \n d, \n i\nFROM \n(\n SELECT \n t.1 AS k, \n t.2 AS v, \n runningDifference(v) AS d, \n runningDifference(cityHash64(t.1)) AS i\n FROM \n (\n SELECT arrayJoin([(\'a\', 1), (\'a\', 2), (\'a\', 3), (\'b\', 11), (\'b\', 13), (\'b\', 15)]) AS t\n )\n)\nWHERE i = 0 +a 1 0 0 +a 2 1 0 +a 3 1 0 +b 13 2 0 +b 15 2 0 +SELECT \n co, \n co2, \n co3, \n num\nFROM \n(\n SELECT \n co, \n co2, \n co3, \n count() AS num\n FROM \n (\n SELECT \n 1 AS co, \n 2 AS co2, \n 3 AS co3\n )\n GROUP BY \n co, \n co2, \n co3\n WITH CUBE\n HAVING (co != 0) AND (co2 != 2)\n)\nWHERE (co != 0) AND (co2 != 2) +1 0 3 1 +1 0 0 1 +SELECT alias AS name\nFROM \n(\n SELECT name AS alias\n FROM system.settings\n WHERE alias = \'enable_optimize_predicate_expression\'\n)\nANY INNER JOIN \n(\n SELECT name\n FROM system.settings\n) USING (name)\nWHERE name = \'enable_optimize_predicate_expression\' +enable_optimize_predicate_expression +1 val11 val21 val31 +SELECT ccc\nFROM \n(\n SELECT 1 AS ccc\n WHERE 0\n UNION ALL\n SELECT ccc\n FROM \n (\n SELECT 2 AS ccc\n )\n ANY INNER JOIN \n (\n SELECT 2 AS ccc\n ) USING (ccc)\n WHERE ccc > 1\n)\nWHERE ccc > 1 +2 +SELECT \n ts, \n id, \n id_b, \n b.ts, \n b.id, \n id_c\nFROM \n(\n SELECT \n ts, \n id, \n id_b\n FROM A\n WHERE ts <= toDateTime(\'1970-01-01 03:00:00\')\n) AS a\nALL LEFT JOIN B AS b ON b.id = id_b\nWHERE ts <= toDateTime(\'1970-01-01 03:00:00\') +SELECT \n ts AS `--a.ts`, \n id AS `--a.id`, \n id_b AS `--a.id_b`, \n b.ts AS `--b.ts`, \n b.id AS `--b.id`, \n id_c AS `--b.id_c`\nFROM \n(\n SELECT \n ts, \n id, \n id_b\n FROM A\n WHERE ts <= toDateTime(\'1970-01-01 03:00:00\')\n) AS a\nALL LEFT JOIN B AS b ON `--b.id` = `--a.id_b`\nWHERE `--a.ts` <= toDateTime(\'1970-01-01 03:00:00\') +2 3 +3 4 +4 5 +5 0 +2 4 +4 0 +2 3 +4 5 +SELECT dummy\nFROM \n(\n SELECT dummy\n FROM system.one\n WHERE arrayMap(x -> (x + 1), [dummy]) = [1]\n)\nWHERE arrayMap(x -> (x + 1), [dummy]) = [1] +0 +SELECT \n id, \n value, \n value_1\nFROM \n(\n SELECT \n 1 AS id, \n 2 AS value\n)\nALL INNER JOIN \n(\n SELECT \n 1 AS id, \n 3 AS value_1\n) USING (id)\nWHERE arrayMap(x -> ((x + value) + value_1), [1]) = [6] +1 2 3 diff --git a/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.sql b/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.sql new file mode 100644 index 00000000000..e1e185be076 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01056_predicate_optimizer_bugs.sql @@ -0,0 +1,75 @@ +SET enable_debug_queries = 1; +SET enable_optimize_predicate_expression = 1; + +-- https://github.com/ClickHouse/ClickHouse/issues/3885 +-- https://github.com/ClickHouse/ClickHouse/issues/5485 +ANALYZE SELECT k, v, d, i FROM (SELECT t.1 AS k, t.2 AS v, runningDifference(v) AS d, runningDifference(cityHash64(t.1)) AS i FROM ( SELECT arrayJoin([('a', 1), ('a', 2), ('a', 3), ('b', 11), ('b', 13), ('b', 15)]) AS t)) WHERE i = 0; +SELECT k, v, d, i FROM (SELECT t.1 AS k, t.2 AS v, runningDifference(v) AS d, runningDifference(cityHash64(t.1)) AS i FROM ( SELECT arrayJoin([('a', 1), ('a', 2), ('a', 3), ('b', 11), ('b', 13), ('b', 15)]) AS t)) WHERE i = 0; + +-- https://github.com/ClickHouse/ClickHouse/issues/5682 +ANALYZE SELECT co,co2,co3,num FROM ( SELECT co,co2,co3,count() AS num FROM ( SELECT 1 AS co,2 AS co2 ,3 AS co3 ) GROUP BY cube (co,co2,co3) ) WHERE co!=0 AND co2 !=2; +SELECT co,co2,co3,num FROM ( SELECT co,co2,co3,count() AS num FROM ( SELECT 1 AS co,2 AS co2 ,3 AS co3 ) GROUP BY cube (co,co2,co3) ) WHERE co!=0 AND co2 !=2; + +-- https://github.com/ClickHouse/ClickHouse/issues/6734 +ANALYZE SELECT alias AS name FROM ( SELECT name AS alias FROM system.settings ) ANY INNER JOIN ( SELECT name FROM system.settings ) USING (name) WHERE name = 'enable_optimize_predicate_expression'; +SELECT alias AS name FROM ( SELECT name AS alias FROM system.settings ) ANY INNER JOIN ( SELECT name FROM system.settings ) USING (name) WHERE name = 'enable_optimize_predicate_expression'; + +-- https://github.com/ClickHouse/ClickHouse/issues/6767 +DROP TABLE IF EXISTS t1; +DROP TABLE IF EXISTS t2; +DROP TABLE IF EXISTS t3; +DROP TABLE IF EXISTS view1; + +CREATE TABLE t1 (id UInt32, value1 String ) ENGINE ReplacingMergeTree() ORDER BY id; +CREATE TABLE t2 (id UInt32, value2 String ) ENGINE ReplacingMergeTree() ORDER BY id; +CREATE TABLE t3 (id UInt32, value3 String ) ENGINE ReplacingMergeTree() ORDER BY id; + +INSERT INTO t1 (id, value1) VALUES (1, 'val11'); +INSERT INTO t2 (id, value2) VALUES (1, 'val21'); +INSERT INTO t3 (id, value3) VALUES (1, 'val31'); + +CREATE VIEW IF NOT EXISTS view1 AS SELECT t1.id AS id, t1.value1 AS value1, t2.value2 AS value2, t3.value3 AS value3 FROM t1 LEFT JOIN t2 ON t1.id = t2.id LEFT JOIN t3 ON t1.id = t3.id WHERE t1.id > 0; +SELECT * FROM view1 WHERE id = 1; + +DROP TABLE IF EXISTS t1; +DROP TABLE IF EXISTS t2; +DROP TABLE IF EXISTS t3; +DROP TABLE IF EXISTS view1; + +-- https://github.com/ClickHouse/ClickHouse/issues/7136 +ANALYZE SELECT ccc FROM ( SELECT 1 AS ccc UNION ALL SELECT * FROM ( SELECT 2 AS ccc ) ANY INNER JOIN ( SELECT 2 AS ccc ) USING (ccc) ) WHERE ccc > 1; +SELECT ccc FROM ( SELECT 1 AS ccc UNION ALL SELECT * FROM ( SELECT 2 AS ccc ) ANY INNER JOIN ( SELECT 2 AS ccc ) USING (ccc) ) WHERE ccc > 1; + +-- https://github.com/ClickHouse/ClickHouse/issues/5674 +-- https://github.com/ClickHouse/ClickHouse/issues/4731 +-- https://github.com/ClickHouse/ClickHouse/issues/4904 +DROP TABLE IF EXISTS A; +DROP TABLE IF EXISTS B; + +CREATE TABLE A (ts DateTime, id String, id_b String) ENGINE = MergeTree PARTITION BY toStartOfHour(ts) ORDER BY (ts,id); +CREATE TABLE B (ts DateTime, id String, id_c String) ENGINE = MergeTree PARTITION BY toStartOfHour(ts) ORDER BY (ts,id); + +ANALYZE SELECT ts, id, id_b, b.ts, b.id, id_c FROM (SELECT ts, id, id_b FROM A) AS a ALL LEFT JOIN B AS b ON b.id = a.id_b WHERE a.ts <= toDateTime('1970-01-01 03:00:00'); +ANALYZE SELECT ts AS `--a.ts`, id AS `--a.id`, id_b AS `--a.id_b`, b.ts AS `--b.ts`, b.id AS `--b.id`, id_c AS `--b.id_c` FROM (SELECT ts, id, id_b FROM A) AS a ALL LEFT JOIN B AS b ON `--b.id` = `--a.id_b` WHERE `--a.ts` <= toDateTime('1970-01-01 03:00:00'); + +DROP TABLE IF EXISTS A; +DROP TABLE IF EXISTS B; + +-- https://github.com/ClickHouse/ClickHouse/issues/7802 +DROP TABLE IF EXISTS test; + +CREATE TABLE test ( A Int32, B Int32 ) ENGINE = Memory(); + +INSERT INTO test VALUES(1, 2)(0, 3)(1, 4)(0, 5); + +SELECT B, neighbor(B, 1) AS next_B FROM (SELECT * FROM test ORDER BY B); +SELECT B, neighbor(B, 1) AS next_B FROM (SELECT * FROM test ORDER BY B) WHERE A == 1; +SELECT B, next_B FROM (SELECT A, B, neighbor(B, 1) AS next_B FROM (SELECT * FROM test ORDER BY B)) WHERE A == 1; + +DROP TABLE IF EXISTS test; + +ANALYZE SELECT * FROM (SELECT * FROM system.one) WHERE arrayMap(x -> x + 1, [dummy]) = [1]; +SELECT * FROM (SELECT * FROM system.one) WHERE arrayMap(x -> x + 1, [dummy]) = [1]; + +ANALYZE SELECT * FROM (SELECT 1 AS id, 2 AS value) INNER JOIN (SELECT 1 AS id, 3 AS value_1) USING id WHERE arrayMap(x -> x + value + value_1, [1]) = [6]; +SELECT * FROM (SELECT 1 AS id, 2 AS value) INNER JOIN (SELECT 1 AS id, 3 AS value_1) USING id WHERE arrayMap(x -> x + value + value_1, [1]) = [6]; diff --git a/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.reference b/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.reference new file mode 100644 index 00000000000..b1ec6c8a4ed --- /dev/null +++ b/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.reference @@ -0,0 +1,8 @@ +Hello, World +Hello,\tWorld +Hello,\nWorld +Hello,\tWorld +Hello,\nWorld +\N +457 +457 diff --git a/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.sh b/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.sh new file mode 100755 index 00000000000..4840a8f672d --- /dev/null +++ b/dbms/tests/queries/0_stateless/01056_prepared_statements_null_and_escaping.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. $CURDIR/../shell_config.sh + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%20World" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%5CtWorld" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%5CnWorld" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%5C%09World" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%5C%0AWorld" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=%5CN" \ + -d "SELECT {x:Nullable(String)}"; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%09World" \ + -d "SELECT {x:Nullable(String)}" 2>&1 | grep -oF '457'; + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}¶m_x=Hello,%0AWorld" \ + -d "SELECT {x:Nullable(String)}" 2>&1 | grep -oF '457'; diff --git a/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.reference b/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.reference new file mode 100644 index 00000000000..5dd396a38c9 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.reference @@ -0,0 +1,11 @@ +1 +1 +1 +1 +1 +999997 +999998 +999999 +999997 +999998 +999999 diff --git a/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.sh b/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.sh new file mode 100755 index 00000000000..419f774e502 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01057_http_compression_prefer_brotli.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. $CURDIR/../shell_config.sh + +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: br' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT 1' | brotli -d +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: br,gzip' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT 1' | brotli -d +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: gzip,br' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT 1' | brotli -d +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: gzip,deflate,br' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT 1' | brotli -d +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: gzip,deflate' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT 1' | gzip -d +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: gzip' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT number FROM numbers(1000000)' | gzip -d | tail -n3 +${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: br' "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT number FROM numbers(1000000)' | brotli -d | tail -n3 diff --git a/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.reference b/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.reference new file mode 100644 index 00000000000..1036eecb9b0 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.reference @@ -0,0 +1,101 @@ +34529 +34530 +34531 +34532 +34533 +34534 +34535 +34536 +34537 +34538 +34539 +34540 +34541 +34542 +34543 +34544 +34545 +34546 +34547 +34548 +34549 +34550 +34551 +34552 +34553 +34554 +34555 +34556 +34557 +34558 +34559 +34560 +34561 +34562 +34563 +34564 +34565 +34566 +34567 +34568 +34569 +34570 +34571 +34572 +34573 +34574 +34575 +34576 +34577 +34578 +34579 +34580 +34581 +34582 +34583 +34584 +34585 +34586 +34587 +34588 +34589 +34590 +34591 +34592 +34593 +34594 +34595 +34596 +34597 +34598 +34599 +34600 +34601 +34602 +34603 +34604 +34605 +34606 +34607 +34608 +34609 +34610 +34611 +34612 +34613 +34614 +34615 +34616 +34617 +34618 +34619 +34620 +34621 +34622 +34623 +34624 +34625 +34626 +34627 +34628 +34629 diff --git a/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.sh b/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.sh new file mode 100755 index 00000000000..f554aec4fca --- /dev/null +++ b/dbms/tests/queries/0_stateless/01058_zlib_ng_level1_bug.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. $CURDIR/../shell_config.sh + +for i in $(seq 34530 1 34630); do ${CLICKHOUSE_CURL} -sS -H 'Accept-Encoding: gzip' "${CLICKHOUSE_URL}&enable_http_compression=1&http_zlib_compression_level=1" -d "SELECT * FROM numbers($i)" | gzip -d | tail -n1; done diff --git a/dbms/tests/queries/0_stateless/01059_storage_file_brotli.reference b/dbms/tests/queries/0_stateless/01059_storage_file_brotli.reference new file mode 100644 index 00000000000..6c545e9faec --- /dev/null +++ b/dbms/tests/queries/0_stateless/01059_storage_file_brotli.reference @@ -0,0 +1,5 @@ +1000000 999999 +1000000 999999 +2000000 999999 +1 255 +1 255 diff --git a/dbms/tests/queries/0_stateless/01059_storage_file_brotli.sql b/dbms/tests/queries/0_stateless/01059_storage_file_brotli.sql new file mode 100644 index 00000000000..e7d5a87b2af --- /dev/null +++ b/dbms/tests/queries/0_stateless/01059_storage_file_brotli.sql @@ -0,0 +1,22 @@ +DROP TABLE IF EXISTS file; +CREATE TABLE file (x UInt64) ENGINE = File(TSV, 'data1.tsv.br'); +TRUNCATE TABLE file; + +INSERT INTO file SELECT * FROM numbers(1000000); +SELECT count(), max(x) FROM file; + +DROP TABLE file; + +CREATE TABLE file (x UInt64) ENGINE = File(TSV, 'data2.tsv.gz'); +TRUNCATE TABLE file; + +INSERT INTO file SELECT * FROM numbers(1000000); +SELECT count(), max(x) FROM file; + +DROP TABLE file; + +SELECT count(), max(x) FROM file('data{1,2}.tsv.{gz,br}', TSV, 'x UInt64'); + +-- check that they are compressed +SELECT count() < 1000000, max(x) FROM file('data1.tsv.br', RowBinary, 'x UInt8', 'none'); +SELECT count() < 3000000, max(x) FROM file('data2.tsv.gz', RowBinary, 'x UInt8', 'none'); diff --git a/dbms/tests/queries/0_stateless/01060_defaults_all_columns.reference b/dbms/tests/queries/0_stateless/01060_defaults_all_columns.reference new file mode 100644 index 00000000000..68b4657ca60 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01060_defaults_all_columns.reference @@ -0,0 +1,4 @@ +1 hello +2 test42 +42 test42 +42 world diff --git a/dbms/tests/queries/0_stateless/01060_defaults_all_columns.sql b/dbms/tests/queries/0_stateless/01060_defaults_all_columns.sql new file mode 100644 index 00000000000..afbb01b8cb2 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01060_defaults_all_columns.sql @@ -0,0 +1,10 @@ +DROP TABLE IF EXISTS defaults_all_columns; + +CREATE TABLE defaults_all_columns (n UInt8 DEFAULT 42, s String DEFAULT concat('test', CAST(n, 'String'))) ENGINE = Memory; + +INSERT INTO defaults_all_columns FORMAT JSONEachRow {"n": 1, "s": "hello"} {}; +INSERT INTO defaults_all_columns FORMAT JSONEachRow {"n": 2}, {"s": "world"}; + +SELECT * FROM defaults_all_columns ORDER BY n, s; + +DROP TABLE defaults_all_columns; diff --git a/dbms/tests/queries/0_stateless/01060_substring_negative_size.reference b/dbms/tests/queries/0_stateless/01060_substring_negative_size.reference new file mode 100644 index 00000000000..b25696dc7d6 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01060_substring_negative_size.reference @@ -0,0 +1,27 @@ +bcdef +bcdef +bcdef +bcdef +- +bcdef +bcdef +bcdef +bcdef +- +bcdef +23456 +bcdef +3456 +bcdef +2345 +bcdef +345 +- +bcdef +23456 +bcdef +3456 +bcdef +2345 +bcdef +345 diff --git a/dbms/tests/queries/0_stateless/01060_substring_negative_size.sql b/dbms/tests/queries/0_stateless/01060_substring_negative_size.sql new file mode 100644 index 00000000000..23cab14a6e0 --- /dev/null +++ b/dbms/tests/queries/0_stateless/01060_substring_negative_size.sql @@ -0,0 +1,36 @@ +select substring('abcdefgh', 2, -2); +select substring('abcdefgh', materialize(2), -2); +select substring('abcdefgh', 2, materialize(-2)); +select substring('abcdefgh', materialize(2), materialize(-2)); + +select '-'; + +select substring(cast('abcdefgh' as FixedString(8)), 2, -2); +select substring(cast('abcdefgh' as FixedString(8)), materialize(2), -2); +select substring(cast('abcdefgh' as FixedString(8)), 2, materialize(-2)); +select substring(cast('abcdefgh' as FixedString(8)), materialize(2), materialize(-2)); + +select '-'; + +drop table if exists t; +create table t (s String, l Int8, r Int8) engine = Memory; +insert into t values ('abcdefgh', 2, -2), ('12345678', 3, -3); + +select substring(s, 2, -2) from t; +select substring(s, l, -2) from t; +select substring(s, 2, r) from t; +select substring(s, l, r) from t; + +select '-'; + +drop table if exists t; +create table t (s FixedString(8), l Int8, r Int8) engine = Memory; +insert into t values ('abcdefgh', 2, -2), ('12345678', 3, -3); + +select substring(s, 2, -2) from t; +select substring(s, l, -2) from t; +select substring(s, 2, r) from t; +select substring(s, l, r) from t; + +drop table if exists t; + diff --git a/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.reference b/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.reference new file mode 100644 index 00000000000..836a5f20d7a --- /dev/null +++ b/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.reference @@ -0,0 +1,5 @@ +epoch UInt64 CODEC(Delta(8), LZ4) +_time_dec Float64 +epoch UInt64 toUInt64(_time_dec) CODEC(Delta(8), LZ4) +_time_dec Float64 +1577351080 1577351080 diff --git a/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.sql b/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.sql new file mode 100644 index 00000000000..7f662c7463d --- /dev/null +++ b/dbms/tests/queries/0_stateless/01061_alter_codec_with_type.sql @@ -0,0 +1,19 @@ +DROP TABLE IF EXISTS alter_bug; + +create table alter_bug ( + epoch UInt64 CODEC(Delta,LZ4), + _time_dec Float64 +) Engine = MergeTree ORDER BY (epoch); + + +SELECT name, type, compression_codec FROM system.columns WHERE table='alter_bug' AND database=currentDatabase(); + +ALTER TABLE alter_bug MODIFY COLUMN epoch DEFAULT toUInt64(_time_dec) CODEC(Delta,LZ4); + +SELECT name, type, default_expression, compression_codec FROM system.columns WHERE table='alter_bug' AND database=currentDatabase(); + +INSERT INTO alter_bug(_time_dec) VALUES(1577351080); + +SELECT * FROM alter_bug; + +DROP TABLE IF EXISTS alter_bug; diff --git a/dbms/tests/queries/1_stateful/00091_prewhere_two_conditions.sql b/dbms/tests/queries/1_stateful/00091_prewhere_two_conditions.sql index cc660ed3f24..201ff788006 100644 --- a/dbms/tests/queries/1_stateful/00091_prewhere_two_conditions.sql +++ b/dbms/tests/queries/1_stateful/00091_prewhere_two_conditions.sql @@ -1,4 +1,4 @@ -SET max_bytes_to_read = 200000000; +SET max_bytes_to_read = 600000000; SET optimize_move_to_prewhere = 1; diff --git a/dbms/tests/queries/1_stateful/00151_replace_partition_with_different_granularity.sql b/dbms/tests/queries/1_stateful/00151_replace_partition_with_different_granularity.sql index c907f353768..bea90dade3c 100644 --- a/dbms/tests/queries/1_stateful/00151_replace_partition_with_different_granularity.sql +++ b/dbms/tests/queries/1_stateful/00151_replace_partition_with_different_granularity.sql @@ -24,7 +24,7 @@ CREATE TABLE non_mixed_granularity_adaptive_table AS test.hits; INSERT INTO non_mixed_granularity_adaptive_table SELECT * FROM test.hits LIMIT 10; -ALTER TABLE non_mixed_granularity_adaptive_table REPLACE PARTITION 201403 FROM test.hits; -- { serverError 49 } +ALTER TABLE non_mixed_granularity_adaptive_table REPLACE PARTITION 201403 FROM test.hits; -- { serverError 36 } DROP TABLE IF EXISTS non_mixed_granularity_adaptive_table; @@ -35,7 +35,7 @@ CREATE TABLE non_mixed_granularity_non_adaptive_table (`WatchID` UInt64, `JavaEn INSERT INTO non_mixed_granularity_non_adaptive_table SELECT * FROM test.hits LIMIT 10; -- after optimize mixed_granularity_table will have .mrk2 parts -ALTER TABLE non_mixed_granularity_non_adaptive_table REPLACE PARTITION 201403 FROM mixed_granularity_table; -- { serverError 49 } +ALTER TABLE non_mixed_granularity_non_adaptive_table REPLACE PARTITION 201403 FROM mixed_granularity_table; -- { serverError 36 } DROP TABLE IF EXISTS non_mixed_granularity_non_adaptive_table; @@ -46,7 +46,7 @@ CREATE TABLE mixed_granularity_strictly_non_adaptive_table (`WatchID` UInt64, `J INSERT INTO mixed_granularity_strictly_non_adaptive_table SELECT * FROM test.hits LIMIT 10; -ALTER TABLE mixed_granularity_strictly_non_adaptive_table REPLACE PARTITION 201403 FROM mixed_granularity_table; -- { serverError 49 } +ALTER TABLE mixed_granularity_strictly_non_adaptive_table REPLACE PARTITION 201403 FROM mixed_granularity_table; -- { serverError 36 } DROP TABLE IF EXISTS mixed_granularity_table; diff --git a/dbms/tests/queries/bugs/01060_defaults_all_columns.reference b/dbms/tests/queries/bugs/01060_defaults_all_columns.reference new file mode 100644 index 00000000000..7b1fdfb6817 --- /dev/null +++ b/dbms/tests/queries/bugs/01060_defaults_all_columns.reference @@ -0,0 +1,4 @@ +1 hello +2 test2 +42 test42 +42 world diff --git a/debian/clickhouse-server.init b/debian/clickhouse-server.init index 32282756719..8a50298ecd2 100755 --- a/debian/clickhouse-server.init +++ b/debian/clickhouse-server.init @@ -3,8 +3,8 @@ # Provides: clickhouse-server # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 -# Required-Start: -# Required-Stop: +# Required-Start: $network +# Required-Stop: $network # Short-Description: Yandex clickhouse-server daemon ### END INIT INFO @@ -20,7 +20,16 @@ CLICKHOUSE_LOGDIR=/var/log/clickhouse-server CLICKHOUSE_LOGDIR_USER=root CLICKHOUSE_DATADIR_OLD=/opt/clickhouse CLICKHOUSE_DATADIR=/var/lib/clickhouse -LOCALSTATEDIR=/var/lock +if [ -d "/var/lock" ]; then + LOCALSTATEDIR=/var/lock +else + LOCALSTATEDIR=/run/lock +fi + +if [ ! -d "$LOCALSTATEDIR" ]; then + mkdir -p "$LOCALSTATEDIR" +fi + CLICKHOUSE_BINDIR=/usr/bin CLICKHOUSE_CRONFILE=/etc/cron.d/clickhouse-server CLICKHOUSE_CONFIG=$CLICKHOUSE_CONFDIR/config.xml diff --git a/debian/clickhouse-server.postinst b/debian/clickhouse-server.postinst index 4a1f4d9d387..81a9582c063 100644 --- a/debian/clickhouse-server.postinst +++ b/debian/clickhouse-server.postinst @@ -13,19 +13,14 @@ CLICKHOUSE_GENERIC_PROGRAM=${CLICKHOUSE_GENERIC_PROGRAM:=clickhouse} EXTRACT_FROM_CONFIG=${CLICKHOUSE_GENERIC_PROGRAM}-extract-from-config CLICKHOUSE_CONFIG=$CLICKHOUSE_CONFDIR/config.xml -OS=${OS=`lsb_release -is 2>/dev/null ||:`} -[ -z "$OS" ] && [ -f /etc/os-release ] && . /etc/os-release && OS=$ID -[ -z "$OS" ] && [ -f /etc/centos-release ] && OS=centos -[ -z "$OS" ] && OS=`uname -s ||:` - [ -f /usr/share/debconf/confmodule ] && . /usr/share/debconf/confmodule [ -f /etc/default/clickhouse ] && . /etc/default/clickhouse -if [ "$OS" = "rhel" ] || [ "$OS" = "centos" ] || [ "$OS" = "fedora" ] || [ "$OS" = "CentOS" ] || [ "$OS" = "Fedora" ] || [ "$OS" = "ol" ]; then - is_rh=1 +if [ ! -f "/etc/debian_version" ]; then + not_deb_os=1 fi -if [ "$1" = configure ] || [ -n "$is_rh" ]; then +if [ "$1" = configure ] || [ -n "$not_deb_os" ]; then if [ -x "/bin/systemctl" ] && [ -f /etc/systemd/system/clickhouse-server.service ] && [ -d /run/systemd/system ]; then # if old rc.d service present - remove it if [ -x "/etc/init.d/clickhouse-server" ] && [ -x "/usr/sbin/update-rc.d" ]; then @@ -48,9 +43,8 @@ if [ "$1" = configure ] || [ -n "$is_rh" ]; then # Make sure the administrative user exists if ! getent passwd ${CLICKHOUSE_USER} > /dev/null; then - if [ -n "$is_rh" ]; then - adduser --system --no-create-home --home /nonexistent \ - --shell /bin/false ${CLICKHOUSE_USER} > /dev/null + if [ -n "$not_deb_os" ]; then + useradd -r -s /bin/false --home-dir /nonexistent ${CLICKHOUSE_USER} > /dev/null else adduser --system --disabled-login --no-create-home --home /nonexistent \ --shell /bin/false --group --gecos "ClickHouse server" ${CLICKHOUSE_USER} > /dev/null @@ -59,12 +53,12 @@ if [ "$1" = configure ] || [ -n "$is_rh" ]; then # if the user was created manually, make sure the group is there as well if ! getent group ${CLICKHOUSE_GROUP} > /dev/null; then - addgroup --system ${CLICKHOUSE_GROUP} > /dev/null + groupadd -r ${CLICKHOUSE_GROUP} > /dev/null fi # make sure user is in the correct group if ! id -Gn ${CLICKHOUSE_USER} | grep -qw ${CLICKHOUSE_USER}; then - adduser ${CLICKHOUSE_USER} ${CLICKHOUSE_GROUP} > /dev/null + usermod -a -G ${CLICKHOUSE_GROUP} ${CLICKHOUSE_USER} > /dev/null fi # check validity of user and group @@ -81,6 +75,9 @@ Please fix this and reinstall this package." >&2 fi if [ -x "$CLICKHOUSE_BINDIR/$EXTRACT_FROM_CONFIG" ] && [ -f "$CLICKHOUSE_CONFIG" ]; then + if [ -z "$SHELL" ]; then + SHELL="/bin/sh" + fi CLICKHOUSE_DATADIR_FROM_CONFIG=$(su -s $SHELL ${CLICKHOUSE_USER} -c "$CLICKHOUSE_BINDIR/$EXTRACT_FROM_CONFIG --config-file=\"$CLICKHOUSE_CONFIG\" --key=path") ||: echo "Path to data directory in ${CLICKHOUSE_CONFIG}: ${CLICKHOUSE_DATADIR_FROM_CONFIG}" fi diff --git a/debian/clickhouse-server.service b/debian/clickhouse-server.service index 4543b304197..b9681f9279e 100644 --- a/debian/clickhouse-server.service +++ b/debian/clickhouse-server.service @@ -1,5 +1,7 @@ [Unit] Description=ClickHouse Server (analytic DBMS for big data) +Requires=network-online.target +After=network-online.target [Service] Type=simple diff --git a/docker/packager/packager b/docker/packager/packager index 8e385786c5f..5e8ffbf1cb9 100755 --- a/docker/packager/packager +++ b/docker/packager/packager @@ -124,6 +124,7 @@ def parse_env_variables(build_type, compiler, sanitizer, package_type, image_typ if is_cross_darwin: cc = compiler[:-len(DARWIN_SUFFIX)] cmake_flags.append("-DCMAKE_AR:FILEPATH=/cctools/bin/x86_64-apple-darwin-ar") + cmake_flags.append("-DCMAKE_INSTALL_NAME_TOOL=/cctools/bin/x86_64-apple-darwin-install_name_tool") cmake_flags.append("-DCMAKE_RANLIB:FILEPATH=/cctools/bin/x86_64-apple-darwin-ranlib") cmake_flags.append("-DLINKER_NAME=/cctools/bin/x86_64-apple-darwin-ld") cmake_flags.append("-DCMAKE_TOOLCHAIN_FILE=/build/cmake/darwin/toolchain-x86_64.cmake") diff --git a/docker/test/performance-comparison/Dockerfile b/docker/test/performance-comparison/Dockerfile index 1e08ec0f521..6c67e724477 100644 --- a/docker/test/performance-comparison/Dockerfile +++ b/docker/test/performance-comparison/Dockerfile @@ -3,15 +3,16 @@ FROM ubuntu:18.04 RUN apt-get update \ && apt-get install --yes --no-install-recommends \ - p7zip-full bash ncdu wget python3 python3-pip python3-dev g++ \ + p7zip-full bash git ncdu wget psmisc python3 python3-pip python3-dev g++ \ && pip3 --no-cache-dir install clickhouse_driver \ && apt-get purge --yes python3-dev g++ \ && apt-get autoremove --yes \ - && apt-get clean + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* COPY * / CMD /entrypoint.sh -# docker run --network=host --volume :/workspace --volume=:/output -e LEFT_PR=<> -e LEFT_SHA=<> -e RIGHT_PR=<> -e RIGHT_SHA=<> yandex/clickhouse-performance-comparison +# docker run --network=host --volume :/workspace --volume=:/output -e PR_TO_TEST=<> -e SHA_TO_TEST=<> yandex/clickhouse-performance-comparison diff --git a/docker/test/performance-comparison/entrypoint.sh b/docker/test/performance-comparison/entrypoint.sh index 7ef5a9553a0..823832f2881 100755 --- a/docker/test/performance-comparison/entrypoint.sh +++ b/docker/test/performance-comparison/entrypoint.sh @@ -1,8 +1,19 @@ #!/bin/bash +set -ex cd /workspace -../compare.sh $LEFT_PR $LEFT_SHA $RIGHT_PR $RIGHT_SHA > compare.log 2>&1 +# We will compare to the most recent testing tag in master branch, let's find it. +rm -rf ch ||: +git clone --branch master --single-branch --depth 50 --bare https://github.com/ClickHouse/ClickHouse ch +ref_tag=$(cd ch && git describe --match='v*-testing' --abbrev=0 --first-parent master) +echo Reference tag is $ref_tag +# We use annotated tags which have their own shas, so we have to further +# dereference the tag to get the commit it points to, hence the '~0' thing. +ref_sha=$(cd ch && git rev-parse $ref_tag~0) +echo Reference SHA is $ref_sha + +../compare.sh 0 $ref_sha $PR_TO_TEST $SHA_TO_TEST > compare.log 2>&1 7z a /output/output.7z *.log *.tsv cp compare.log /output diff --git a/docker/test/performance-comparison/perf.py b/docker/test/performance-comparison/perf.py index 5517a71cc44..8d2fe1bc476 100755 --- a/docker/test/performance-comparison/perf.py +++ b/docker/test/performance-comparison/perf.py @@ -7,7 +7,8 @@ import argparse import pprint parser = argparse.ArgumentParser(description='Run performance test.') -parser.add_argument('file', metavar='FILE', type=argparse.FileType('r'), nargs=1, help='test description file') +# Explicitly decode files as UTF-8 because sometimes we have Russian characters in queries, and LANG=C is set. +parser.add_argument('file', metavar='FILE', type=argparse.FileType('r', encoding='utf-8'), nargs=1, help='test description file') args = parser.parse_args() tree = et.parse(args.file[0]) diff --git a/docs/en/getting_started/example_datasets/star_schema.md b/docs/en/getting_started/example_datasets/star_schema.md index 2e66ced7149..43c1b1df6fa 100644 --- a/docs/en/getting_started/example_datasets/star_schema.md +++ b/docs/en/getting_started/example_datasets/star_schema.md @@ -10,6 +10,10 @@ $ make Generating data: +!!! warning "Attention" + -s 100 -- dbgen generates 600 million rows (67 GB) + -s 1000 -- dbgen generates 6 billion rows (takes a lot of time) + ```bash $ ./dbgen -s 1000 -T c $ ./dbgen -s 1000 -T l @@ -101,68 +105,236 @@ CREATE TABLE lineorder_flat ENGINE = MergeTree PARTITION BY toYear(LO_ORDERDATE) ORDER BY (LO_ORDERDATE, LO_ORDERKEY) AS -SELECT l.*, c.*, s.*, p.* -FROM lineorder l - ANY INNER JOIN customer c ON (c.C_CUSTKEY = l.LO_CUSTKEY) - ANY INNER JOIN supplier s ON (s.S_SUPPKEY = l.LO_SUPPKEY) - ANY INNER JOIN part p ON (p.P_PARTKEY = l.LO_PARTKEY); +SELECT + l.LO_ORDERKEY AS LO_ORDERKEY, + l.LO_LINENUMBER AS LO_LINENUMBER, + l.LO_CUSTKEY AS LO_CUSTKEY, + l.LO_PARTKEY AS LO_PARTKEY, + l.LO_SUPPKEY AS LO_SUPPKEY, + l.LO_ORDERDATE AS LO_ORDERDATE, + l.LO_ORDERPRIORITY AS LO_ORDERPRIORITY, + l.LO_SHIPPRIORITY AS LO_SHIPPRIORITY, + l.LO_QUANTITY AS LO_QUANTITY, + l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE, + l.LO_ORDTOTALPRICE AS LO_ORDTOTALPRICE, + l.LO_DISCOUNT AS LO_DISCOUNT, + l.LO_REVENUE AS LO_REVENUE, + l.LO_SUPPLYCOST AS LO_SUPPLYCOST, + l.LO_TAX AS LO_TAX, + l.LO_COMMITDATE AS LO_COMMITDATE, + l.LO_SHIPMODE AS LO_SHIPMODE, + c.C_NAME AS C_NAME, + c.C_ADDRESS AS C_ADDRESS, + c.C_CITY AS C_CITY, + c.C_NATION AS C_NATION, + c.C_REGION AS C_REGION, + c.C_PHONE AS C_PHONE, + c.C_MKTSEGMENT AS C_MKTSEGMENT, + s.S_NAME AS S_NAME, + s.S_ADDRESS AS S_ADDRESS, + s.S_CITY AS S_CITY, + s.S_NATION AS S_NATION, + s.S_REGION AS S_REGION, + s.S_PHONE AS S_PHONE, + p.P_NAME AS P_NAME, + p.P_MFGR AS P_MFGR, + p.P_CATEGORY AS P_CATEGORY, + p.P_BRAND AS P_BRAND, + p.P_COLOR AS P_COLOR, + p.P_TYPE AS P_TYPE, + p.P_SIZE AS P_SIZE, + p.P_CONTAINER AS P_CONTAINER +FROM lineorder AS l +INNER JOIN customer AS c ON c.C_CUSTKEY = l.LO_CUSTKEY +INNER JOIN supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY +INNER JOIN part AS p ON p.P_PARTKEY = l.LO_PARTKEY; -ALTER TABLE lineorder_flat DROP COLUMN C_CUSTKEY, DROP COLUMN S_SUPPKEY, DROP COLUMN P_PARTKEY; ``` Running the queries: Q1.1 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toYear(LO_ORDERDATE) = 1993 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toYear(LO_ORDERDATE) = 1993 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25; ``` Q1.2 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toYYYYMM(LO_ORDERDATE) = 199401 AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toYYYYMM(LO_ORDERDATE) = 199401 AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35; ``` Q1.3 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toISOWeek(LO_ORDERDATE) = 6 AND toYear(LO_ORDERDATE) = 1994 AND LO_DISCOUNT BETWEEN 5 AND 7 AND LO_QUANTITY BETWEEN 26 AND 35; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toISOWeek(LO_ORDERDATE) = 6 AND toYear(LO_ORDERDATE) = 1994 + AND LO_DISCOUNT BETWEEN 5 AND 7 AND LO_QUANTITY BETWEEN 26 AND 35; ``` Q2.1 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q2.2 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_BRAND BETWEEN 'MFGR#2221' AND 'MFGR#2228' AND S_REGION = 'ASIA' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_BRAND >= 'MFGR#2221' AND P_BRAND <= 'MFGR#2228' AND S_REGION = 'ASIA' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q2.3 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q3.1 ```sql -SELECT C_NATION, S_NATION, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' AND year >= 1992 AND year <= 1997 GROUP BY C_NATION, S_NATION, year ORDER BY year asc, revenue desc; +SELECT + C_NATION, + S_NATION, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' AND year >= 1992 AND year <= 1997 +GROUP BY + C_NATION, + S_NATION, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.2 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' AND year >= 1992 AND year <= 1997 GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' AND year >= 1992 AND year <= 1997 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.3 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND year >= 1992 AND year <= 1997 GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND year >= 1992 AND year <= 1997 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.4 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND toYYYYMM(LO_ORDERDATE) = '199712' GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND toYYYYMM(LO_ORDERDATE) = 199712 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q4.1 ```sql -SELECT toYear(LO_ORDERDATE) AS year, C_NATION, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year, C_NATION ORDER BY year, C_NATION; +SELECT + toYear(LO_ORDERDATE) AS year, + C_NATION, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') +GROUP BY + year, + C_NATION +ORDER BY + year ASC, + C_NATION ASC; ``` Q4.2 ```sql -SELECT toYear(LO_ORDERDATE) AS year, S_NATION, P_CATEGORY, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (year = 1997 OR year = 1998) AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year, S_NATION, P_CATEGORY ORDER BY year, S_NATION, P_CATEGORY; +SELECT + toYear(LO_ORDERDATE) AS year, + S_NATION, + P_CATEGORY, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (year = 1997 OR year = 1998) AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') +GROUP BY + year, + S_NATION, + P_CATEGORY +ORDER BY + year ASC, + S_NATION ASC, + P_CATEGORY ASC; ``` Q4.3 ```sql -SELECT toYear(LO_ORDERDATE) AS year, S_CITY, P_BRAND, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE S_NATION = 'UNITED STATES' AND (year = 1997 OR year = 1998) AND P_CATEGORY = 'MFGR#14' GROUP BY year, S_CITY, P_BRAND ORDER BY year, S_CITY, P_BRAND; +SELECT + toYear(LO_ORDERDATE) AS year, + S_CITY, + P_BRAND, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE S_NATION = 'UNITED STATES' AND (year = 1997 OR year = 1998) AND P_CATEGORY = 'MFGR#14' +GROUP BY + year, + S_CITY, + P_BRAND +ORDER BY + year ASC, + S_CITY ASC, + P_BRAND ASC; ``` [Original article](https://clickhouse.yandex/docs/en/getting_started/example_datasets/star_schema/) diff --git a/docs/en/getting_started/install.md b/docs/en/getting_started/install.md index e47500fa22f..f1ec980fa70 100644 --- a/docs/en/getting_started/install.md +++ b/docs/en/getting_started/install.md @@ -59,6 +59,35 @@ sudo yum install clickhouse-server clickhouse-client You can also download and install packages manually from here: . +### From tgz archives {#from-tgz-archives} + +It is recommended to use official pre-compiled `tgz` archives for all Linux distributions, where installation of `deb` or `rpm` packages is not possible. + +Required version can be dowloaded with `curl` or `wget` from repository . +After that downloaded archives should be unpacked and installed with installation scripts. Example for the latest version: +```bash +export LATEST_VERSION=`curl https://api.github.com/repos/ClickHouse/ClickHouse/tags 2>/dev/null | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -n 1` +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-common-static-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-common-static-dbg-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-server-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-client-$LATEST_VERSION.tgz + +tar -xzvf clickhouse-common-static-$LATEST_VERSION.tgz +sudo clickhouse-common-static-$LATEST_VERSION/install/doinst.sh + +tar -xzvf clickhouse-common-static-dbg-$LATEST_VERSION.tgz +sudo clickhouse-common-static-dbg-$LATEST_VERSION/install/doinst.sh + +tar -xzvf clickhouse-server-$LATEST_VERSION.tgz +sudo clickhouse-server-$LATEST_VERSION/install/doinst.sh +sudo /etc/init.d/clickhouse-server start + +tar -xzvf clickhouse-client-$LATEST_VERSION.tgz +sudo clickhouse-client-$LATEST_VERSION/install/doinst.sh +``` + +For production environments it's recommended to use latest `stable`-version. You can find it's number on github page https://github.com/ClickHouse/ClickHouse/tags with postfix `-stable`. + ### From Docker Image To run ClickHouse inside Docker follow the guide on [Docker Hub](https://hub.docker.com/r/yandex/clickhouse-server/). Those images use official `deb` packages inside. diff --git a/docs/en/getting_started/tutorial.md b/docs/en/getting_started/tutorial.md index acdd9074beb..bffee808122 100644 --- a/docs/en/getting_started/tutorial.md +++ b/docs/en/getting_started/tutorial.md @@ -444,7 +444,7 @@ SAMPLE BY intHash32(UserID) SETTINGS index_granularity = 8192 ``` -You can execute those queries using interactive mode of `clickhouse-client` (just launch it in terminal without specifying a query in advance) or try some [alternative interface](../interfaces/index.md) if you ant. +You can execute those queries using interactive mode of `clickhouse-client` (just launch it in terminal without specifying a query in advance) or try some [alternative interface](../interfaces/index.md) if you want. As we can see, `hits_v1` uses the [basic MergeTree engine](../operations/table_engines/mergetree.md), while the `visits_v1` uses the [Collapsing](../operations/table_engines/collapsingmergetree.md) variant. diff --git a/docs/en/guides/apply_catboost_model.md b/docs/en/guides/apply_catboost_model.md index 4665809bfa0..06863bb48f9 100644 --- a/docs/en/guides/apply_catboost_model.md +++ b/docs/en/guides/apply_catboost_model.md @@ -74,7 +74,7 @@ $ clickhouse client ROLE_FAMILY UInt32, ROLE_CODE UInt32 ) -ENGINE = MergeTree() +ENGINE = MergeTree ORDER BY date ``` **3.** Exit from ClickHouse console client: @@ -227,4 +227,4 @@ FROM ``` !!! note "Note" - More info about [avg()](../query_language/agg_functions/reference.md#agg_function-avg) and [log()](../query_language/functions/math_functions.md) functions. \ No newline at end of file + More info about [avg()](../query_language/agg_functions/reference.md#agg_function-avg) and [log()](../query_language/functions/math_functions.md) functions. diff --git a/docs/en/interfaces/cli.md b/docs/en/interfaces/cli.md index 198e5f5c094..86c7a104670 100644 --- a/docs/en/interfaces/cli.md +++ b/docs/en/interfaces/cli.md @@ -117,7 +117,7 @@ You can pass parameters to `clickhouse-client` (all parameters have a default va - `--query, -q` – The query to process when using non-interactive mode. - `--database, -d` – Select the current default database. Default value: the current database from the server settings ('default' by default). - `--multiline, -m` – If specified, allow multiline queries (do not send the query on Enter). -- `--multiquery, -n` – If specified, allow processing multiple queries separated by commas. Only works in non-interactive mode. +- `--multiquery, -n` – If specified, allow processing multiple queries separated by semicolons. - `--format, -f` – Use the specified default format to output the result. - `--vertical, -E` – If specified, use the Vertical format by default to output the result. This is the same as '--format=Vertical'. In this format, each value is printed on a separate line, which is helpful when displaying wide tables. - `--time, -t` – If specified, print the query execution time to 'stderr' in non-interactive mode. diff --git a/docs/en/interfaces/third-party/integrations.md b/docs/en/interfaces/third-party/integrations.md index f96507320a5..692dfae9776 100644 --- a/docs/en/interfaces/third-party/integrations.md +++ b/docs/en/interfaces/third-party/integrations.md @@ -33,7 +33,9 @@ - Monitoring - [Graphite](https://graphiteapp.org) - [graphouse](https://github.com/yandex/graphouse) - - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + + - [graphite-clickhouse](https://github.com/lomik/graphite-clickhouse) + - [graphite-ch-optimizer](https://github.com/innogames/graphite-ch-optimizer) - optimizes staled partitions in [\*GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#graphitemergetree) if rules from [rollup configuration](../../operations/table_engines/graphitemergetree.md#rollup-configuration) could be applied - [Grafana](https://grafana.com/) - [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) - [Prometheus](https://prometheus.io/) diff --git a/docs/en/operations/index.md b/docs/en/operations/index.md index 547fc4de260..3ce48c8e57e 100644 --- a/docs/en/operations/index.md +++ b/docs/en/operations/index.md @@ -13,6 +13,7 @@ ClickHouse operations manual consists of the following major sections: - [Quotas](quotas.md) - [System Tables](system_tables.md) - [Server Configuration Parameters](server_settings/index.md) + - [How To Test Your Hardware With ClickHouse](performance_test.md) - [Settings](settings/index.md) - [Utilities](utils/index.md) diff --git a/docs/en/operations/performance_test.md b/docs/en/operations/performance_test.md new file mode 100644 index 00000000000..a56490ac8ba --- /dev/null +++ b/docs/en/operations/performance_test.md @@ -0,0 +1,72 @@ +# How To Test Your Hardware With ClickHouse + +Draft. + +With this instruction you can run basic ClickHouse performance test on any server without installation of ClickHouse packages. + +1. Go to "commits" page: https://github.com/ClickHouse/ClickHouse/commits/master + +2. Click on the first green check mark or red cross with green "ClickHouse Build Check" and click on the "Details" link near "ClickHouse Build Check". + +3. Copy the link to "clickhouse" binary for amd64 or aarch64. + +4. ssh to the server and download it with wget: +``` +# For amd64: +wget https://clickhouse-builds.s3.yandex.net/0/00ba767f5d2a929394ea3be193b1f79074a1c4bc/1578163263_binary/clickhouse +# For aarch64: +wget https://clickhouse-builds.s3.yandex.net/0/00ba767f5d2a929394ea3be193b1f79074a1c4bc/1578161264_binary/clickhouse +# Then do: +chmod a+x clickhouse +``` + +5. Download configs: +``` +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/programs/server/config.xml +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/programs/server/users.xml +mkdir config.d +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/programs/server/config.d/path.xml -O config.d/path.xml +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/programs/server/config.d/log_to_console.xml -O config.d/log_to_console.xml +``` + +6. Download benchmark files: +``` +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/benchmark/clickhouse/benchmark-new.sh +chmod a+x benchmark-new.sh +wget https://raw.githubusercontent.com/ClickHouse/ClickHouse/master/dbms/benchmark/clickhouse/queries.sql +``` + +7. Download test data: +According to the instruction: +https://clickhouse.yandex/docs/en/getting_started/example_datasets/metrica/ +("hits" table containing 100 million rows) + +``` +wget https://clickhouse-datasets.s3.yandex.net/hits/partitions/hits_100m_obfuscated_v1.tar.xz +tar xvf hits_100m_obfuscated_v1.tar.xz -C . +mv hits_100m_obfuscated_v1/* . +``` + +8. Run the server: +``` +./clickhouse server +``` + +9. Check the data: +ssh to the server in another terminal +``` +./clickhouse client --query "SELECT count() FROM hits_100m_obfuscated" +100000000 +``` + +10. Edit the benchmark-new.sh, change "clickhouse-client" to "./clickhouse client" and add "--max_memory_usage 100000000000" parameter. +``` +mcedit benchmark-new.sh +``` + +11. Run the benchmark: +``` +./benchmark-new.sh hits_100m_obfuscated +``` + +12. Send the numbers and the info about your hardware configuration to clickhouse-feedback@yandex-team.com diff --git a/docs/en/operations/server_settings/settings.md b/docs/en/operations/server_settings/settings.md index c76637cc927..89bb7ef33ae 100644 --- a/docs/en/operations/server_settings/settings.md +++ b/docs/en/operations/server_settings/settings.md @@ -372,9 +372,6 @@ Approximate size (in bytes) of the cache of marks used by table engines of the [ The cache is shared for the server and memory is allocated as needed. The cache size must be at least 5368709120. -!!! warning "Warning" - This parameter could be exceeded by the [mark_cache_min_lifetime](../settings/settings.md#settings-mark_cache_min_lifetime) setting. - **Example** ```xml diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index ba7370fef03..21d2f46e71b 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -80,7 +80,7 @@ This parameter is useful when you are using formats that require a schema defini Enables or disables [fsync](http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html) when writing `.sql` files. Enabled by default. -It makes sense to disable it if the server has millions of tiny table chunks that are constantly being created and destroyed. +It makes sense to disable it if the server has millions of tiny tables that are constantly being created and destroyed. ## enable_http_compression {#settings-enable_http_compression} @@ -218,11 +218,11 @@ Ok. Enables or disables template deduction for an SQL expressions in [Values](../../interfaces/formats.md#data-format-values) format. It allows to parse and interpret expressions in `Values` much faster if expressions in consecutive rows have the same structure. ClickHouse will try to deduce template of an expression, parse the following rows using this template and evaluate the expression on batch of successfully parsed rows. For the following query: ```sql INSERT INTO test VALUES (lower('Hello')), (lower('world')), (lower('INSERT')), (upper('Values')), ... -``` +``` - if `input_format_values_interpret_expressions=1` and `format_values_deduce_templates_of_expressions=0` expressions will be interpreted separately for each row (this is very slow for large number of rows) - if `input_format_values_interpret_expressions=0` and `format_values_deduce_templates_of_expressions=1` expressions in the first, second and third rows will be parsed using template `lower(String)` and interpreted together, expression is the forth row will be parsed with another template (`upper(String)`) - if `input_format_values_interpret_expressions=1` and `format_values_deduce_templates_of_expressions=1` - the same as in previous case, but also allows fallback to interpreting expressions separately if it's not possible to deduce template. - + Enabled by default. ## input_format_values_accurate_types_of_literals {#settings-input_format_values_accurate_types_of_literals} @@ -232,7 +232,7 @@ This setting is used only when `input_format_values_deduce_templates_of_expressi (..., abs(3.141592654), ...), -- Float64 literal (..., abs(-1), ...), -- Int64 literal ``` -When this setting is enabled, ClickHouse will check actual type of literal and will use expression template of the corresponding type. In some cases it may significantly slow down expression evaluation in `Values`. +When this setting is enabled, ClickHouse will check actual type of literal and will use expression template of the corresponding type. In some cases it may significantly slow down expression evaluation in `Values`. When disabled, ClickHouse may use more general type for some literals (e.g. `Float64` or `Int64` instead of `UInt64` for `42`), but it may cause overflow and precision issues. Enabled by default. @@ -477,7 +477,7 @@ Default value: 8. ## merge_tree_max_rows_to_use_cache {#setting-merge_tree_max_rows_to_use_cache} -If ClickHouse should read more than `merge_tree_max_rows_to_use_cache` rows in one query, it doesn't use the cache of uncompressed blocks. +If ClickHouse should read more than `merge_tree_max_rows_to_use_cache` rows in one query, it doesn't use the cache of uncompressed blocks. The cache of uncompressed blocks stores data extracted for queries. ClickHouse uses this cache to speed up responses to repeated small queries. This setting protects the cache from trashing by queries that read a large amount of data. The [uncompressed_cache_size](../server_settings/settings.md#server-settings-uncompressed_cache_size) server setting defines the size of the cache of uncompressed blocks. @@ -591,12 +591,6 @@ We are writing a URL column with the String type (average size of 60 bytes per v There usually isn't any reason to change this setting. -## mark_cache_min_lifetime {#settings-mark_cache_min_lifetime} - -If the value of [mark_cache_size](../server_settings/settings.md#server-mark-cache-size) setting is exceeded, delete only records older than mark_cache_min_lifetime seconds. If your hosts have low amount of RAM, it makes sense to lower this parameter. - -Default value: 10000 seconds. - ## max_query_size {#settings-max_query_size} The maximum part of a query that can be taken to RAM for parsing with the SQL parser. @@ -960,7 +954,7 @@ Possible values: - 1 — skipping enabled. - If a shard is unavailable, ClickHouse returns a result based on partial data and doesn't report node availability issues. + If a shard is unavailable, ClickHouse returns a result based on partial data and doesn't report node availability issues. - 0 — skipping disabled. @@ -987,7 +981,7 @@ Default value: 0. - Type: seconds - Default value: 60 seconds -Controls how fast errors of distributed tables are zeroed. Given that currently a replica was unavailabe for some time and accumulated 5 errors and distributed_replica_error_half_life is set to 1 second, then said replica is considered back to normal in 3 seconds since last error. +Controls how fast errors in distributed tables are zeroed. If a replica is unavailabe for some time, accumulates 5 errors, and distributed_replica_error_half_life is set to 1 second, then the replica is considered normal 3 seconds after last error. ** See also ** @@ -1000,7 +994,7 @@ Controls how fast errors of distributed tables are zeroed. Given that currently - Type: unsigned int - Default value: 1000 -Error count of each replica is capped at this value, preventing a single replica from accumulating to many errors. +Error count of each replica is capped at this value, preventing a single replica from accumulating too many errors. ** See also ** @@ -1010,7 +1004,7 @@ Error count of each replica is capped at this value, preventing a single replica ## distributed_directory_monitor_sleep_time_ms {#distributed_directory_monitor_sleep_time_ms} -Base interval of data sending by the [Distributed](../table_engines/distributed.md) table engine. Actual interval grows exponentially in case of any errors. +Base interval for the [Distributed](../table_engines/distributed.md) table engine to send data. The actual interval grows exponentially in the event of errors. Possible values: @@ -1021,7 +1015,7 @@ Default value: 100 milliseconds. ## distributed_directory_monitor_max_sleep_time_ms {#distributed_directory_monitor_max_sleep_time_ms} -Maximum interval of data sending by the [Distributed](../table_engines/distributed.md) table engine. Limits exponential growth of the interval set in the [distributed_directory_monitor_sleep_time_ms](#distributed_directory_monitor_sleep_time_ms) setting. +Maximum interval for the [Distributed](../table_engines/distributed.md) table engine to send data. Limits exponential growth of the interval set in the [distributed_directory_monitor_sleep_time_ms](#distributed_directory_monitor_sleep_time_ms) setting. Possible values: @@ -1033,14 +1027,14 @@ Default value: 30000 milliseconds (30 seconds). Enables/disables sending of inserted data in batches. -When batch sending is enabled, [Distributed](../table_engines/distributed.md) table engine tries to send multiple files of inserted data in one operation instead of sending them separately. Batch sending improves cluster performance by better server and network resources utilization. +When batch sending is enabled, the [Distributed](../table_engines/distributed.md) table engine tries to send multiple files of inserted data in one operation instead of sending them separately. Batch sending improves cluster performance by better utilizing server and network resources. Possible values: - 1 — Enabled. - 0 — Disabled. -Defaule value: 0. +Default value: 0. ## os_thread_priority {#setting-os_thread_priority} @@ -1067,7 +1061,7 @@ Possible values: - Positive integer number, in nanoseconds. Recommended values: - + - 10000000 (100 times a second) nanoseconds and less for single queries. - 1000000000 (once a second) for cluster-wide profiling. @@ -1090,7 +1084,7 @@ Possible values: - Positive integer number of nanoseconds. Recommended values: - + - 10000000 (100 times a second) nanosecods and more for for single queries. - 1000000000 (once a second) for cluster-wide profiling. diff --git a/docs/en/operations/table_engines/distributed.md b/docs/en/operations/table_engines/distributed.md index a22fd43b34f..24a61998b39 100644 --- a/docs/en/operations/table_engines/distributed.md +++ b/docs/en/operations/table_engines/distributed.md @@ -87,9 +87,9 @@ The Distributed engine requires writing clusters to the config file. Clusters fr There are two methods for writing data to a cluster: -First, you can define which servers to write which data to, and perform the write directly on each shard. In other words, perform INSERT in the tables that the distributed table "looks at". This is the most flexible solution – you can use any sharding scheme, which could be non-trivial due to the requirements of the subject area. This is also the most optimal solution, since data can be written to different shards completely independently. +First, you can define which servers to write which data to and perform the write directly on each shard. In other words, perform INSERT in the tables that the distributed table "looks at". This is the most flexible solution as you can use any sharding scheme, which could be non-trivial due to the requirements of the subject area. This is also the most optimal solution, since data can be written to different shards completely independently. -Second, you can perform INSERT in a Distributed table. In this case, the table will distribute the inserted data across servers itself. In order to write to a Distributed table, it must have a sharding key set (the last parameter). In addition, if there is only one shard, the write operation works without specifying the sharding key, since it doesn't have any meaning in this case. +Second, you can perform INSERT in a Distributed table. In this case, the table will distribute the inserted data across servers itself. In order to write to a Distributed table, it must have a sharding key set (the last parameter). In addition, if there is only one shard, the write operation works without specifying the sharding key, since it doesn't mean anything in this case. Each shard can have a weight defined in the config file. By default, the weight is equal to one. Data is distributed across shards in the amount proportional to the shard weight. For example, if there are two shards and the first has a weight of 9 while the second has a weight of 10, the first will be sent 9 / 19 parts of the rows, and the second will be sent 10 / 19. @@ -112,7 +112,7 @@ You should be concerned about the sharding scheme in the following cases: - Queries are used that require joining data (IN or JOIN) by a specific key. If data is sharded by this key, you can use local IN or JOIN instead of GLOBAL IN or GLOBAL JOIN, which is much more efficient. - A large number of servers is used (hundreds or more) with a large number of small queries (queries of individual clients - websites, advertisers, or partners). In order for the small queries to not affect the entire cluster, it makes sense to locate data for a single client on a single shard. Alternatively, as we've done in Yandex.Metrica, you can set up bi-level sharding: divide the entire cluster into "layers", where a layer may consist of multiple shards. Data for a single client is located on a single layer, but shards can be added to a layer as necessary, and data is randomly distributed within them. Distributed tables are created for each layer, and a single shared distributed table is created for global queries. -Data is written asynchronously. When inserted to the table, the data block is just written to the local file system. The data is sent to the remote servers in the background as soon as possible. The period of data sending is managed by the [distributed_directory_monitor_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_sleep_time_ms) and [distributed_directory_monitor_max_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_max_sleep_time_ms) settings. The `Distributed` engine sends each file with inserted data separately, but you can enable batch sending of files with the [distributed_directory_monitor_batch_inserts](../settings/settings.md#distributed_directory_monitor_batch_inserts) setting. This setting improves cluster performance by better local server and network resources utilization. You should check whether data is sent successfully by checking the list of files (data waiting to be sent) in the table directory: `/var/lib/clickhouse/data/database/table/`. +Data is written asynchronously. When inserted in the table, the data block is just written to the local file system. The data is sent to the remote servers in the background as soon as possible. The period for sending data is managed by the [distributed_directory_monitor_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_sleep_time_ms) and [distributed_directory_monitor_max_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_max_sleep_time_ms) settings. The `Distributed` engine sends each file with inserted data separately, but you can enable batch sending of files with the [distributed_directory_monitor_batch_inserts](../settings/settings.md#distributed_directory_monitor_batch_inserts) setting. This setting improves cluster performance by better utilizing local server and network resources. You should check whether data is sent successfully by checking the list of files (data waiting to be sent) in the table directory: `/var/lib/clickhouse/data/database/table/`. If the server ceased to exist or had a rough restart (for example, after a device failure) after an INSERT to a Distributed table, the inserted data might be lost. If a damaged data part is detected in the table directory, it is transferred to the 'broken' subdirectory and no longer used. diff --git a/docs/en/operations/table_engines/graphitemergetree.md b/docs/en/operations/table_engines/graphitemergetree.md index a8ed8aaaddf..7a47eabfe22 100644 --- a/docs/en/operations/table_engines/graphitemergetree.md +++ b/docs/en/operations/table_engines/graphitemergetree.md @@ -1,5 +1,4 @@ - -# GraphiteMergeTree +# GraphiteMergeTree {#graphitemergetree} This engine is designed for thinning and aggregating/averaging (rollup) [Graphite](http://graphite.readthedocs.io/en/latest/index.html) data. It may be helpful to developers who want to use ClickHouse as a data store for Graphite. @@ -7,7 +6,7 @@ You can use any ClickHouse table engine to store the Graphite data if you don't The engine inherits properties from [MergeTree](mergetree.md). -## Creating a Table +## Creating a Table {#creating-table} ```sql CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] @@ -67,7 +66,7 @@ All of the parameters excepting `config_section` have the same meaning as in `Me - `config_section` — Name of the section in the configuration file, where are the rules of rollup set. -## Rollup configuration +## Rollup configuration {#rollup-configuration} The settings for rollup are defined by the [graphite_rollup](../server_settings/settings.md#server_settings-graphite_rollup) parameter in the server configuration. The name of the parameter could be any. You can create several configurations and use them for different tables. @@ -78,14 +77,14 @@ required-columns patterns ``` -### Required Columns +### Required Columns {#required-columns} - `path_column_name` — The name of the column storing the metric name (Graphite sensor). Default value: `Path`. - `time_column_name` — The name of the column storing the time of measuring the metric. Default value: `Time`. - `value_column_name` — The name of the column storing the value of the metric at the time set in `time_column_name`. Default value: `Value`. - `version_column_name` — The name of the column storing the version of the metric. Default value: `Timestamp`. -### Patterns +### Patterns {#patterns} Structure of the `patterns` section: @@ -127,7 +126,7 @@ Fields for `pattern` and `default` sections: - `function` – The name of the aggregating function to apply to data whose age falls within the range `[age, age + precision]`. -### Configuration Example +### Configuration Example {#configuration-example} ```xml diff --git a/docs/en/operations/table_engines/mergetree.md b/docs/en/operations/table_engines/mergetree.md index f1c888e4480..e9c9ec26c53 100644 --- a/docs/en/operations/table_engines/mergetree.md +++ b/docs/en/operations/table_engines/mergetree.md @@ -504,21 +504,25 @@ Disks, volumes and storage policies should be declared inside the ` - - /mnt/fast_ssd/clickhouse - - - /mnt/hdd1/clickhouse - 10485760_ - - - /mnt/hdd2/clickhouse - 10485760_ - + + + + /mnt/fast_ssd/clickhouse + + + /mnt/hdd1/clickhouse + 10485760 + + + /mnt/hdd2/clickhouse + 10485760 + + ... + + ... - + ``` Tags: @@ -532,26 +536,30 @@ The order of the disk definition is not important. Storage policies configuration markup: ```xml - - - - - disk_name_from_disks_configuration - 1073741824 - - - - - - - 0.2 - - - - + + ... + + + + + disk_name_from_disks_configuration + 1073741824 + + + + + + + 0.2 + + + + - - + + + ... + ``` Tags: @@ -565,29 +573,33 @@ Tags: Cofiguration examples: ```xml - - - - - disk1 - disk2 - - - + + ... + + + + + disk1 + disk2 + + + - - - - fast_ssd - 1073741824 - - - disk1 - - - 0.2 - - + + + + fast_ssd + 1073741824 + + + disk1 + + + 0.2 + + + ... + ``` In given example, the `hdd_in_order` policy implements the [round-robin](https://en.wikipedia.org/wiki/Round-robin_scheduling) approach. Thus this policy defines only one volume (`single`), the data parts are stored on all its disks in circular order. Such policy can be quite useful if there are several similar disks are mounted to the system, but RAID is not configured. Keep in mind that each individual disk drive is not reliable and you might want to compensate it with replication factor of 3 or more. diff --git a/docs/en/query_language/functions/conditional_functions.md b/docs/en/query_language/functions/conditional_functions.md index 074df25303f..8b46566af31 100644 --- a/docs/en/query_language/functions/conditional_functions.md +++ b/docs/en/query_language/functions/conditional_functions.md @@ -1,19 +1,66 @@ # Conditional functions -## if(cond, then, else), cond ? operator then : else +## `if` function -Returns `then` if `cond != 0`, or `else` if `cond = 0`. -`cond` must be of type `UInt8`, and `then` and `else` must have the lowest common type. +Syntax: `if(cond, then, else)` -`then` and `else` can be `NULL` +Returns `then` if the `cond` is truthy(greater than zero), otherwise returns `else`. + +* `cond` must be of type of `UInt8`, and `then` and `else` must have the lowest common type. + +* `then` and `else` can be `NULL` + +**Example:** + +Take this `LEFT_RIGHT` table: + +```sql +SELECT * +FROM LEFT_RIGHT + +┌─left─┬─right─┐ +│ ᴺᵁᴸᴸ │ 4 │ +│ 1 │ 3 │ +│ 2 │ 2 │ +│ 3 │ 1 │ +│ 4 │ ᴺᵁᴸᴸ │ +└──────┴───────┘ +``` +The following query compares `left` and `right` values: + +```sql +SELECT + left, + right, + if(left < right, 'left is smaller than right', 'right is greater or equal than left') AS is_smaller +FROM LEFT_RIGHT +WHERE isNotNull(left) AND isNotNull(right) + +┌─left─┬─right─┬─is_smaller──────────────────────────┐ +│ 1 │ 3 │ left is smaller than right │ +│ 2 │ 2 │ right is greater or equal than left │ +│ 3 │ 1 │ right is greater or equal than left │ +└──────┴───────┴─────────────────────────────────────┘ +``` +Note: `NULL` values are not used in this example, check [NULL values in conditionals](#null-values-in-conditionals) section. + +## Ternary operator + +It works same as `if` function. + +Syntax: `cond ? then : else` + +Returns `then` if the `cond` is truthy(greater than zero), otherwise returns `else`. + +* `cond` must be of type of `UInt8`, and `then` and `else` must have the lowest common type. + +* `then` and `else` can be `NULL` ## multiIf Allows you to write the [CASE](../operators.md#operator_case) operator more compactly in the query. -```sql -multiIf(cond_1, then_1, cond_2, then_2...else) -``` +Syntax: `multiIf(cond_1, then_1, cond_2, then_2, ..., else)` **Parameters:** @@ -29,22 +76,76 @@ The function returns one of the values `then_N` or `else`, depending on the cond **Example** -Take the table +Again using `LEFT_RIGHT` table. -```text -┌─x─┬────y─┐ -│ 1 │ ᴺᵁᴸᴸ │ -│ 2 │ 3 │ -└───┴──────┘ +```sql +SELECT + left, + right, + multiIf(left < right, 'left is smaller', left > right, 'left is greater', left = right, 'Both equal', 'Null value') AS result +FROM LEFT_RIGHT + +┌─left─┬─right─┬─result──────────┐ +│ ᴺᵁᴸᴸ │ 4 │ Null value │ +│ 1 │ 3 │ left is smaller │ +│ 2 │ 2 │ Both equal │ +│ 3 │ 1 │ left is greater │ +│ 4 │ ᴺᵁᴸᴸ │ Null value │ +└──────┴───────┴─────────────────┘ +``` +## Using conditional results directly + +Conditionals always result to `0`, `1` or `NULL`. So you can use conditional results directly like this: + +```sql +SELECT left < right AS is_small +FROM LEFT_RIGHT + +┌─is_small─┐ +│ ᴺᵁᴸᴸ │ +│ 1 │ +│ 0 │ +│ 0 │ +│ ᴺᵁᴸᴸ │ +└──────────┘ ``` -Run the query `SELECT multiIf(isNull(y) x, y < 3, y, NULL) FROM t_null`. Result: -```text -┌─multiIf(isNull(y), x, less(y, 3), y, NULL)─┐ -│ 1 │ -│ ᴺᵁᴸᴸ │ -└────────────────────────────────────────────┘ +## NULL values in conditionals + +When `NULL` values are involved in conditionals, the result will also be `NULL`. + +```sql +SELECT + NULL < 1, + 2 < NULL, + NULL < NULL, + NULL = NULL + +┌─less(NULL, 1)─┬─less(2, NULL)─┬─less(NULL, NULL)─┬─equals(NULL, NULL)─┐ +│ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ +└───────────────┴───────────────┴──────────────────┴────────────────────┘ ``` +So you should construct your queries carefully if the types are `Nullable`. + +The following example demonstrates this by failing to add equals condition to `multiIf`. + +```sql +SELECT + left, + right, + multiIf(left < right, 'left is smaller', left > right, 'right is smaller', 'Both equal') AS faulty_result +FROM LEFT_RIGHT + +┌─left─┬─right─┬─faulty_result────┐ +│ ᴺᵁᴸᴸ │ 4 │ Both equal │ +│ 1 │ 3 │ left is smaller │ +│ 2 │ 2 │ Both equal │ +│ 3 │ 1 │ right is smaller │ +│ 4 │ ᴺᵁᴸᴸ │ Both equal │ +└──────┴───────┴──────────────────┘ +``` + + [Original article](https://clickhouse.yandex/docs/en/query_language/functions/conditional_functions/) diff --git a/docs/en/query_language/functions/string_functions.md b/docs/en/query_language/functions/string_functions.md index 33e5700f355..8b73f1b3c19 100644 --- a/docs/en/query_language/functions/string_functions.md +++ b/docs/en/query_language/functions/string_functions.md @@ -217,6 +217,44 @@ Result: └───────────────────────────────────┘ ``` +## trim {#trim} + +Removes all specified characters from the start or end of a string. +By default removes all consecutive occurrences of common whitespace (ASCII character 32) from both ends of a string. + +**Syntax** + +```sql +trim([[LEADING|TRAILING|BOTH] trim_character FROM] input_string) +``` + +**Parameters** + +- `trim_character` — specified characters for trim. [String](../../data_types/string.md). +- `input_string` — string for trim. [String](../../data_types/string.md). + +**Returned value** + +A string without leading and (or) trailing specified characters. + +Type: `String`. + +**Example** + +Query: + +```sql +SELECT trim(BOTH ' ()' FROM '( Hello, world! )') +``` + +Result: + +```text +┌─trim(BOTH ' ()' FROM '( Hello, world! )')─┐ +│ Hello, world! │ +└───────────────────────────────────────────────┘ +``` + ## trimLeft {#trimleft} Removes all consecutive occurrences of common whitespace (ASCII character 32) from the beginning of a string. It doesn't remove other kinds of whitespace characters (tab, no-break space, etc.). @@ -224,14 +262,14 @@ Removes all consecutive occurrences of common whitespace (ASCII character 32) fr **Syntax** ```sql -trimLeft() +trimLeft(input_string) ``` -Alias: `ltrim`. +Alias: `ltrim(input_string)`. **Parameters** -- `string` — string to trim. [String](../../data_types/string.md). +- `input_string` — string to trim. [String](../../data_types/string.md). **Returned value** @@ -262,14 +300,14 @@ Removes all consecutive occurrences of common whitespace (ASCII character 32) fr **Syntax** ```sql -trimRight() +trimRight(input_string) ``` -Alias: `rtrim`. +Alias: `rtrim(input_string)`. **Parameters** -- `string` — string to trim. [String](../../data_types/string.md). +- `input_string` — string to trim. [String](../../data_types/string.md). **Returned value** @@ -300,14 +338,14 @@ Removes all consecutive occurrences of common whitespace (ASCII character 32) fr **Syntax** ```sql -trimBoth() +trimBoth(input_string) ``` -Alias: `trim`. +Alias: `trim(input_string)`. **Parameters** -- `string` — string to trim. [String](../../data_types/string.md). +- `input_string` — string to trim. [String](../../data_types/string.md). **Returned value** diff --git a/docs/en/security_changelog.md b/docs/en/security_changelog.md index 0847300cc19..92b35868f94 100644 --- a/docs/en/security_changelog.md +++ b/docs/en/security_changelog.md @@ -1,6 +1,27 @@ +## Fixed in ClickHouse Release 19.14.3.3, 2019-09-10 + +### CVE-2019-15024 + +Аn attacker having write access to ZooKeeper and who is able to run a custom server available from the network where ClickHouse runs, can create a custom-built malicious server that will act as a ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from the malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. + +Credits: Eldar Zaitov of Yandex Information Security Team + +### CVE-2019-16535 + +Аn OOB read, OOB write and integer underflow in decompression algorithms can be used to achieve RCE or DoS via native protocol. + +Credits: Eldar Zaitov of Yandex Information Security Team + +### CVE-2019-16536 + +Stack overflow leading to DoS can be triggered by malicious authenticated client. + +Credits: Eldar Zaitov of Yandex Information Security Team + ## Fixed in ClickHouse Release 19.13.6.1, 2019-09-20 ### CVE-2019-18657 + Table function `url` had the vulnerability allowed the attacker to inject arbitrary HTTP headers in the request. Credits: [Nikita Tikhomirov](https://github.com/NSTikhomirov) @@ -24,6 +45,7 @@ Credits: Andrey Krasichkov and Evgeny Sidorov of Yandex Information Security Tea ## Fixed in ClickHouse Release 1.1.54388, 2018-06-28 ### CVE-2018-14668 + "remote" table function allowed arbitrary symbols in "user", "password" and "default_database" fields which led to Cross Protocol Request Forgery Attacks. Credits: Andrey Krasichkov of Yandex Information Security Team @@ -31,6 +53,7 @@ Credits: Andrey Krasichkov of Yandex Information Security Team ## Fixed in ClickHouse Release 1.1.54390, 2018-07-06 ### CVE-2018-14669 + ClickHouse MySQL client had "LOAD DATA LOCAL INFILE" functionality enabled that allowed a malicious MySQL database read arbitrary files from the connected ClickHouse server. Credits: Andrey Krasichkov and Evgeny Sidorov of Yandex Information Security Team diff --git a/docs/fa/interfaces/cli.md b/docs/fa/interfaces/cli.md index 7680348aef6..e5f869e1c0d 100644 --- a/docs/fa/interfaces/cli.md +++ b/docs/fa/interfaces/cli.md @@ -91,7 +91,7 @@ command line برا پایه 'readline' (و 'history' یا 'libedit'، یه بد - `--query, -q` – مشخص کردن query برای پردازش در هنگام استفاده از حالت non-interactive. - `--database, -d` – انتخاب دیتابیس در بدو ورود به کلاینت. مقدار پیش فرض: دیتابیس مشخص شده در تنظیمات سرور (پیش فرض 'default') - `--multiline, -m` – اگر مشخص شود، یعنی اجازه ی نوشتن query های چند خطی را بده. (بعد از Enter، query را ارسال نکن). -- `--multiquery, -n` – اگر مشخص شود، اجازه ی اجرای چندین query که از طریق کاما جدا شده اند را می دهد. فقط در حالت non-interactive کار می کند. +- `--multiquery, -n` – اگر مشخص شود، اجازه ی اجرای چندین query که از طریق جمع و حلقه ها جدا شده اند را می دهد. فقط در حالت non-interactive کار می کند. - `--format, -f` مشخص کردن نوع فرمت خروجی - `--vertical, -E` اگر مشخص شود، از فرمت Vertical برای نمایش خروجی استفاده می شود. این گزینه مشابه '--format=Vertical' می باشد. در این فرمت، هر مقدار در یک خط جدید چاپ می شود، که در هنگام نمایش جداول عریض مفید است. - `--time, -t` اگر مشخص شود، در حالت non-interactive زمان اجرای query در 'stderr' جاپ می شود. diff --git a/docs/fa/interfaces/third-party/integrations.md b/docs/fa/interfaces/third-party/integrations.md index e7def6bca58..7aed10d3762 100644 --- a/docs/fa/interfaces/third-party/integrations.md +++ b/docs/fa/interfaces/third-party/integrations.md @@ -34,7 +34,9 @@ - نظارت بر - [Graphite](https://graphiteapp.org) - [graphouse](https://github.com/yandex/graphouse) - - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + + - [graphite-clickhouse](https://github.com/lomik/graphite-clickhouse) + - [graphite-ch-optimizer](https://github.com/innogames/graphite-ch-optimizer) - optimizes staled partitions in [\*GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#graphitemergetree) if rules from [rollup configuration](../../operations/table_engines/graphitemergetree.md#rollup-configuration) could be applied - [Grafana](https://grafana.com/) - [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) - [Prometheus](https://prometheus.io/) diff --git a/docs/fa/operations/performance_test.md b/docs/fa/operations/performance_test.md new file mode 120000 index 00000000000..a74c126c63f --- /dev/null +++ b/docs/fa/operations/performance_test.md @@ -0,0 +1 @@ +../../en/operations/performance_test.md \ No newline at end of file diff --git a/docs/ja/operations/performance_test.md b/docs/ja/operations/performance_test.md new file mode 120000 index 00000000000..a74c126c63f --- /dev/null +++ b/docs/ja/operations/performance_test.md @@ -0,0 +1 @@ +../../en/operations/performance_test.md \ No newline at end of file diff --git a/docs/ru/extended_roadmap.md b/docs/ru/extended_roadmap.md index 206a87abb00..d5cbcf5f8c8 100644 --- a/docs/ru/extended_roadmap.md +++ b/docs/ru/extended_roadmap.md @@ -1413,7 +1413,7 @@ ClickHouse поддерживает LZ4 и ZSTD для сжатия данных ### 24.4. Шифрование в ClickHouse на уровне кусков данных. -Yuchen Dong, ICS. +Yuchen Dong, ICT. Данные в ClickHouse хранятся без шифрования. При наличии доступа к дискам, злоумышленник может прочитать данные. Предлагается реализовать два подхода к шифрованию: @@ -1422,7 +1422,7 @@ Yuchen Dong, ICS. ### 24.5. Поддержка функций шифрования для отдельных значений. -Yuchen Dong, ICS. +Yuchen Dong, ICT. Смотрите также 24.5. @@ -1574,7 +1574,7 @@ https://github.com/yandex/ClickHouse/issues/6874 ### 24.27. Реализация алгоритмов min-hash, sim-hash для нечёткого поиска полудубликатов. -ucasFL, ICS. +ucasFL, ICT. Алгоритмы min-hash и sim-hash позволяют вычислить для текста несколько хэш-значений таких, что при небольшом изменении текста, по крайней мере один из хэшей не меняется. Вычисления можно реализовать на n-грамах и словарных шинглах. Предлагается добавить поддержку этих алгоритмов в виде функций в ClickHouse и изучить их применимость для задачи нечёткого поиска полудубликатов. @@ -1681,7 +1681,7 @@ Amos Bird, но его решение слишком громоздкое и п Требуется проработать вопрос безопасности и изоляции инстансов (поднятие в контейнерах с ограничениями по сети), подключение тестовых датасетов с помощью copy-on-write файловой системы; органичения ресурсов. -### 25.17. Взаимодействие с ВУЗами: ВШЭ, УрФУ, ICS Beijing. +### 25.17. Взаимодействие с ВУЗами: ВШЭ, УрФУ, ICT Beijing. Алексей Миловидов и вся группа разработки diff --git a/docs/ru/getting_started/example_datasets/star_schema.md b/docs/ru/getting_started/example_datasets/star_schema.md index 2e66ced7149..28ab1e0fd2b 100644 --- a/docs/ru/getting_started/example_datasets/star_schema.md +++ b/docs/ru/getting_started/example_datasets/star_schema.md @@ -1,6 +1,6 @@ # Star Schema Benchmark -Compiling dbgen: +Компиляция dbgen: ```bash $ git clone git@github.com:vadimtk/ssb-dbgen.git @@ -8,7 +8,12 @@ $ cd ssb-dbgen $ make ``` -Generating data: +Генерация данных: + +!!! warning "Внимание" + -s 100 -- dbgen генерирует 600 миллионов строк (67 ГБ) + -s 1000 -- dbgen генерирует 6 миллиардов строк (занимает много времени) + ```bash $ ./dbgen -s 1000 -T c @@ -18,7 +23,7 @@ $ ./dbgen -s 1000 -T s $ ./dbgen -s 1000 -T d ``` -Creating tables in ClickHouse: +Создание таблиц в Кликхауз: ```sql CREATE TABLE customer @@ -83,7 +88,7 @@ CREATE TABLE supplier ENGINE = MergeTree ORDER BY S_SUPPKEY; ``` -Inserting data: +Вставка данных: ```bash $ clickhouse-client --query "INSERT INTO customer FORMAT CSV" < customer.tbl @@ -92,77 +97,244 @@ $ clickhouse-client --query "INSERT INTO supplier FORMAT CSV" < supplier.tbl $ clickhouse-client --query "INSERT INTO lineorder FORMAT CSV" < lineorder.tbl ``` -Converting "star schema" to denormalized "flat schema": +Конвертация схемы-звезда в денормализованную плоскую схему: ```sql SET max_memory_usage = 20000000000, allow_experimental_multiple_joins_emulation = 1; - CREATE TABLE lineorder_flat ENGINE = MergeTree PARTITION BY toYear(LO_ORDERDATE) ORDER BY (LO_ORDERDATE, LO_ORDERKEY) AS -SELECT l.*, c.*, s.*, p.* -FROM lineorder l - ANY INNER JOIN customer c ON (c.C_CUSTKEY = l.LO_CUSTKEY) - ANY INNER JOIN supplier s ON (s.S_SUPPKEY = l.LO_SUPPKEY) - ANY INNER JOIN part p ON (p.P_PARTKEY = l.LO_PARTKEY); +SELECT + l.LO_ORDERKEY AS LO_ORDERKEY, + l.LO_LINENUMBER AS LO_LINENUMBER, + l.LO_CUSTKEY AS LO_CUSTKEY, + l.LO_PARTKEY AS LO_PARTKEY, + l.LO_SUPPKEY AS LO_SUPPKEY, + l.LO_ORDERDATE AS LO_ORDERDATE, + l.LO_ORDERPRIORITY AS LO_ORDERPRIORITY, + l.LO_SHIPPRIORITY AS LO_SHIPPRIORITY, + l.LO_QUANTITY AS LO_QUANTITY, + l.LO_EXTENDEDPRICE AS LO_EXTENDEDPRICE, + l.LO_ORDTOTALPRICE AS LO_ORDTOTALPRICE, + l.LO_DISCOUNT AS LO_DISCOUNT, + l.LO_REVENUE AS LO_REVENUE, + l.LO_SUPPLYCOST AS LO_SUPPLYCOST, + l.LO_TAX AS LO_TAX, + l.LO_COMMITDATE AS LO_COMMITDATE, + l.LO_SHIPMODE AS LO_SHIPMODE, + c.C_NAME AS C_NAME, + c.C_ADDRESS AS C_ADDRESS, + c.C_CITY AS C_CITY, + c.C_NATION AS C_NATION, + c.C_REGION AS C_REGION, + c.C_PHONE AS C_PHONE, + c.C_MKTSEGMENT AS C_MKTSEGMENT, + s.S_NAME AS S_NAME, + s.S_ADDRESS AS S_ADDRESS, + s.S_CITY AS S_CITY, + s.S_NATION AS S_NATION, + s.S_REGION AS S_REGION, + s.S_PHONE AS S_PHONE, + p.P_NAME AS P_NAME, + p.P_MFGR AS P_MFGR, + p.P_CATEGORY AS P_CATEGORY, + p.P_BRAND AS P_BRAND, + p.P_COLOR AS P_COLOR, + p.P_TYPE AS P_TYPE, + p.P_SIZE AS P_SIZE, + p.P_CONTAINER AS P_CONTAINER +FROM lineorder AS l +INNER JOIN customer AS c ON c.C_CUSTKEY = l.LO_CUSTKEY +INNER JOIN supplier AS s ON s.S_SUPPKEY = l.LO_SUPPKEY +INNER JOIN part AS p ON p.P_PARTKEY = l.LO_PARTKEY; -ALTER TABLE lineorder_flat DROP COLUMN C_CUSTKEY, DROP COLUMN S_SUPPKEY, DROP COLUMN P_PARTKEY; ``` Running the queries: Q1.1 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toYear(LO_ORDERDATE) = 1993 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toYear(LO_ORDERDATE) = 1993 AND LO_DISCOUNT BETWEEN 1 AND 3 AND LO_QUANTITY < 25; ``` Q1.2 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toYYYYMM(LO_ORDERDATE) = 199401 AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toYYYYMM(LO_ORDERDATE) = 199401 AND LO_DISCOUNT BETWEEN 4 AND 6 AND LO_QUANTITY BETWEEN 26 AND 35; ``` Q1.3 ```sql -SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue FROM lineorder_flat WHERE toISOWeek(LO_ORDERDATE) = 6 AND toYear(LO_ORDERDATE) = 1994 AND LO_DISCOUNT BETWEEN 5 AND 7 AND LO_QUANTITY BETWEEN 26 AND 35; +SELECT sum(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue +FROM lineorder_flat +WHERE toISOWeek(LO_ORDERDATE) = 6 AND toYear(LO_ORDERDATE) = 1994 + AND LO_DISCOUNT BETWEEN 5 AND 7 AND LO_QUANTITY BETWEEN 26 AND 35; ``` Q2.1 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q2.2 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_BRAND BETWEEN 'MFGR#2221' AND 'MFGR#2228' AND S_REGION = 'ASIA' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_BRAND >= 'MFGR#2221' AND P_BRAND <= 'MFGR#2228' AND S_REGION = 'ASIA' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q2.3 ```sql -SELECT sum(LO_REVENUE), toYear(LO_ORDERDATE) AS year, P_BRAND FROM lineorder_flat WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE' GROUP BY year, P_BRAND ORDER BY year, P_BRAND; +SELECT + sum(LO_REVENUE), + toYear(LO_ORDERDATE) AS year, + P_BRAND +FROM lineorder_flat +WHERE P_BRAND = 'MFGR#2239' AND S_REGION = 'EUROPE' +GROUP BY + year, + P_BRAND +ORDER BY + year, + P_BRAND; ``` Q3.1 ```sql -SELECT C_NATION, S_NATION, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' AND year >= 1992 AND year <= 1997 GROUP BY C_NATION, S_NATION, year ORDER BY year asc, revenue desc; +SELECT + C_NATION, + S_NATION, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE C_REGION = 'ASIA' AND S_REGION = 'ASIA' AND year >= 1992 AND year <= 1997 +GROUP BY + C_NATION, + S_NATION, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.2 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' AND year >= 1992 AND year <= 1997 GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE C_NATION = 'UNITED STATES' AND S_NATION = 'UNITED STATES' AND year >= 1992 AND year <= 1997 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.3 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND year >= 1992 AND year <= 1997 GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND year >= 1992 AND year <= 1997 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q3.4 ```sql -SELECT C_CITY, S_CITY, toYear(LO_ORDERDATE) AS year, sum(LO_REVENUE) AS revenue FROM lineorder_flat WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND toYYYYMM(LO_ORDERDATE) = '199712' GROUP BY C_CITY, S_CITY, year ORDER BY year asc, revenue desc; +SELECT + C_CITY, + S_CITY, + toYear(LO_ORDERDATE) AS year, + sum(LO_REVENUE) AS revenue +FROM lineorder_flat +WHERE (C_CITY = 'UNITED KI1' OR C_CITY = 'UNITED KI5') AND (S_CITY = 'UNITED KI1' OR S_CITY = 'UNITED KI5') AND toYYYYMM(LO_ORDERDATE) = 199712 +GROUP BY + C_CITY, + S_CITY, + year +ORDER BY + year ASC, + revenue DESC; ``` Q4.1 ```sql -SELECT toYear(LO_ORDERDATE) AS year, C_NATION, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year, C_NATION ORDER BY year, C_NATION; +SELECT + toYear(LO_ORDERDATE) AS year, + C_NATION, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') +GROUP BY + year, + C_NATION +ORDER BY + year ASC, + C_NATION ASC; ``` Q4.2 ```sql -SELECT toYear(LO_ORDERDATE) AS year, S_NATION, P_CATEGORY, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (year = 1997 OR year = 1998) AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') GROUP BY year, S_NATION, P_CATEGORY ORDER BY year, S_NATION, P_CATEGORY; +SELECT + toYear(LO_ORDERDATE) AS year, + S_NATION, + P_CATEGORY, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE C_REGION = 'AMERICA' AND S_REGION = 'AMERICA' AND (year = 1997 OR year = 1998) AND (P_MFGR = 'MFGR#1' OR P_MFGR = 'MFGR#2') +GROUP BY + year, + S_NATION, + P_CATEGORY +ORDER BY + year ASC, + S_NATION ASC, + P_CATEGORY ASC; ``` Q4.3 ```sql -SELECT toYear(LO_ORDERDATE) AS year, S_CITY, P_BRAND, sum(LO_REVENUE - LO_SUPPLYCOST) AS profit FROM lineorder_flat WHERE S_NATION = 'UNITED STATES' AND (year = 1997 OR year = 1998) AND P_CATEGORY = 'MFGR#14' GROUP BY year, S_CITY, P_BRAND ORDER BY year, S_CITY, P_BRAND; +SELECT + toYear(LO_ORDERDATE) AS year, + S_CITY, + P_BRAND, + sum(LO_REVENUE - LO_SUPPLYCOST) AS profit +FROM lineorder_flat +WHERE S_NATION = 'UNITED STATES' AND (year = 1997 OR year = 1998) AND P_CATEGORY = 'MFGR#14' +GROUP BY + year, + S_CITY, + P_BRAND +ORDER BY + year ASC, + S_CITY ASC, + P_BRAND ASC; ``` [Original article](https://clickhouse.yandex/docs/en/getting_started/example_datasets/star_schema/) diff --git a/docs/ru/getting_started/install.md b/docs/ru/getting_started/install.md index 29ccd2b14f4..cd1a04b6192 100644 --- a/docs/ru/getting_started/install.md +++ b/docs/ru/getting_started/install.md @@ -50,7 +50,6 @@ sudo yum-config-manager --add-repo https://repo.yandex.ru/clickhouse/rpm/stable/ Для использования наиболее свежих версий нужно заменить `stable` на `testing` (рекомендуется для тестовых окружений). -Then run these commands to actually install packages: Для, собственно, установки пакетов необходимо выполнить следующие команды: ```bash @@ -59,6 +58,35 @@ sudo yum install clickhouse-server clickhouse-client Также есть возможность установить пакеты вручную, скачав отсюда: . +### Из tgz архивов {#from-tgz-archives} + +Команда ClickHouse в Яндексе рекомендует использовать предкомпилированные бинарники из `tgz` архивов для всех дистрибутивов, где невозможна установка `deb` и `rpm` пакетов. + +Интересующую версию архивов можно скачать вручную с помощью `curl` или `wget` из репозитория . +После этого архивы нужно распаковать и воспользоваться скриптами установки. Пример установки самой свежей версии: +```bash +export LATEST_VERSION=`curl https://api.github.com/repos/ClickHouse/ClickHouse/tags 2>/dev/null | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | head -n 1` +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-common-static-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-common-static-dbg-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-server-$LATEST_VERSION.tgz +curl -O https://repo.yandex.ru/clickhouse/tgz/clickhouse-client-$LATEST_VERSION.tgz + +tar -xzvf clickhouse-common-static-$LATEST_VERSION.tgz +sudo clickhouse-common-static-$LATEST_VERSION/install/doinst.sh + +tar -xzvf clickhouse-common-static-dbg-$LATEST_VERSION.tgz +sudo clickhouse-common-static-dbg-$LATEST_VERSION/install/doinst.sh + +tar -xzvf clickhouse-server-$LATEST_VERSION.tgz +sudo clickhouse-server-$LATEST_VERSION/install/doinst.sh +sudo /etc/init.d/clickhouse-server start + +tar -xzvf clickhouse-client-$LATEST_VERSION.tgz +sudo clickhouse-client-$LATEST_VERSION/install/doinst.sh +``` + +Для production окружений рекомендуется использовать последнюю `stable`-версию. Её номер также можно найти на github с на вкладке https://github.com/ClickHouse/ClickHouse/tags c постфиксом `-stable`. + ### Из Docker образа {#from-docker-image} Для запуска ClickHouse в Docker нужно следовать инструкции на [Docker Hub](https://hub.docker.com/r/yandex/clickhouse-server/). Внутри образов используются официальные `deb` пакеты. diff --git a/docs/ru/guides/apply_catboost_model.md b/docs/ru/guides/apply_catboost_model.md index 9f93aacbd22..69aa0faccb2 100644 --- a/docs/ru/guides/apply_catboost_model.md +++ b/docs/ru/guides/apply_catboost_model.md @@ -74,7 +74,7 @@ $ clickhouse client ROLE_FAMILY UInt32, ROLE_CODE UInt32 ) -ENGINE = MergeTree() +ENGINE = MergeTree ORDER BY date ``` **3.** Выйдите из клиента ClickHouse: @@ -227,4 +227,4 @@ FROM ``` !!! note "Примечание" - Подробнее про функции [avg()](../query_language/agg_functions/reference.md#agg_function-avg), [log()](../query_language/functions/math_functions.md). \ No newline at end of file + Подробнее про функции [avg()](../query_language/agg_functions/reference.md#agg_function-avg), [log()](../query_language/functions/math_functions.md). diff --git a/docs/ru/interfaces/cli.md b/docs/ru/interfaces/cli.md index a67ae87f6ab..71742d02740 100644 --- a/docs/ru/interfaces/cli.md +++ b/docs/ru/interfaces/cli.md @@ -119,7 +119,7 @@ $ clickhouse-client --param_tuple_in_tuple="(10, ('dt', 10))" -q "SELECT * FROM - `--query, -q` — запрос для выполнения, при использовании в неинтерактивном режиме. - `--database, -d` — выбрать текущую БД, по умолчанию — текущая БД из настроек сервера (по умолчанию — БД default). - `--multiline, -m` — если указано — разрешить многострочные запросы, не отправлять запрос по нажатию Enter. -- `--multiquery, -n` — если указано — разрешить выполнять несколько запросов, разделённых точкой с запятой. Работает только в неинтерактивном режиме. +- `--multiquery, -n` — если указано — разрешить выполнять несколько запросов, разделённых точкой с запятой. - `--format, -f` — использовать указанный формат по умолчанию для вывода результата. - `--vertical, -E` — если указано, использовать формат Vertical по умолчанию для вывода результата. То же самое, что --format=Vertical. В этом формате каждое значение выводится на отдельной строке, что удобно для отображения широких таблиц. - `--time, -t` — если указано, в неинтерактивном режиме вывести время выполнения запроса в stderr. diff --git a/docs/ru/interfaces/third-party/integrations.md b/docs/ru/interfaces/third-party/integrations.md index 5ab706da67b..470d02bea7d 100644 --- a/docs/ru/interfaces/third-party/integrations.md +++ b/docs/ru/interfaces/third-party/integrations.md @@ -32,7 +32,9 @@ - Мониторинг - [Graphite](https://graphiteapp.org) - [graphouse](https://github.com/yandex/graphouse) - - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + + - [graphite-clickhouse](https://github.com/lomik/graphite-clickhouse) + - [graphite-ch-optimizer](https://github.com/innogames/graphite-ch-optimizer) - оптимизирует партиции таблиц [\*GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#graphitemergetree) согласно правилам в [конфигурации rollup](../../operations/table_engines/graphitemergetree.md#rollup-configuration) - [Grafana](https://grafana.com/) - [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) - [Prometheus](https://prometheus.io/) diff --git a/docs/ru/operations/index.md b/docs/ru/operations/index.md index a10f7c377b1..371afaf2af0 100644 --- a/docs/ru/operations/index.md +++ b/docs/ru/operations/index.md @@ -13,6 +13,7 @@ - [Квоты](quotas.md) - [Системные таблицы](system_tables.md) - [Конфигурационные параметры сервера](server_settings/index.md) + - [Тестирование севреров с помощью ClickHouse](performance_test.md) - [Настройки](settings/index.md) - [Утилиты](utils/index.md) diff --git a/docs/ru/operations/performance_test.md b/docs/ru/operations/performance_test.md new file mode 120000 index 00000000000..a74c126c63f --- /dev/null +++ b/docs/ru/operations/performance_test.md @@ -0,0 +1 @@ +../../en/operations/performance_test.md \ No newline at end of file diff --git a/docs/ru/operations/server_settings/settings.md b/docs/ru/operations/server_settings/settings.md index ca1c255bee3..9a0fa781f63 100644 --- a/docs/ru/operations/server_settings/settings.md +++ b/docs/ru/operations/server_settings/settings.md @@ -370,10 +370,8 @@ ClickHouse проверит условия `min_part_size` и `min_part_size_rat Приблизительный размер (в байтах) кэша засечек, используемых движками таблиц семейства [MergeTree](../../operations/table_engines/mergetree.md). -Кэш общий для сервера, память выделяется по мере необходимости. Кэш не может быть меньше, чем 5368709120. +Кэш общий для сервера, память выделяется по мере необходимости. -!!! warning "Внимание" - Этот параметр может быть превышен при большом значении настройки [mark_cache_min_lifetime](../settings/settings.md#settings-mark_cache_min_lifetime). **Пример** diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 2d5a11bec86..71ff9b494a4 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -566,12 +566,6 @@ ClickHouse использует этот параметр при чтении д Как правило, не имеет смысла менять эту настройку. -## mark_cache_min_lifetime {#settings-mark_cache_min_lifetime} - -Если превышено значение параметра [mark_cache_size](../server_settings/settings.md#server-mark-cache-size), то будут удалены только записи старше чем значение этого параметра. Имеет смысл понижать данный параметр при малом количестве RAM на хост-системах. - -Default value: 10000 seconds. - ## max_query_size {#settings-max_query_size} Максимальный кусок запроса, который будет считан в оперативку для разбора парсером языка SQL. @@ -957,6 +951,39 @@ load_balancing = first_or_random Значение по умолчанию — 0. +## distributed_directory_monitor_sleep_time_ms {#distributed_directory_monitor_sleep_time_ms} + +Основной интервал отправки данных движком таблиц [Distributed](../table_engines/distributed.md). Фактический интервал растёт экспоненциально при возникновении ошибок. + +Возможные значения: + +- Положительное целое количество миллисекунд. + +Значение по умолчанию: 100 миллисекунд. + + +## distributed_directory_monitor_max_sleep_time_ms {#distributed_directory_monitor_max_sleep_time_ms} + +Максимальный интервал отправки данных движком таблиц [Distributed](../table_engines/distributed.md). Ограничивает экпоненциальный рост интервала, установленого настройкой [distributed_directory_monitor_sleep_time_ms](#distributed_directory_monitor_sleep_time_ms). + +Возможные значения: + +- Положительное целое количество миллисекунд. + +Значение по умолчанию: 30000 миллисекунд (30 секунд). + +## distributed_directory_monitor_batch_inserts {#distributed_directory_monitor_batch_inserts} + +Включает/выключает пакетную отправку вставленных данных. + +Если пакетная отправка включена, то движок таблиц [Distributed](../table_engines/distributed.md) вместо того, чтобы отправлять каждый файл со вставленными данными по отдельности, старается отправить их все за одну операцию. Пакетная отправка улучшает производительность кластера за счет более оптимального использования ресурсов сервера и сети. + +Возможные значения: + +- 1 — включено. +- 0 — выключено. + +Значение по умолчанию: 0. ## os_thread_priority {#setting-os_thread_priority} diff --git a/docs/ru/operations/table_engines/distributed.md b/docs/ru/operations/table_engines/distributed.md index ceea785d84e..53391fe8125 100644 --- a/docs/ru/operations/table_engines/distributed.md +++ b/docs/ru/operations/table_engines/distributed.md @@ -87,12 +87,10 @@ logs - имя кластера в конфигурационном файле с Есть два способа записывать данные на кластер: -Во первых, вы можете самостоятельно определять, на какие серверы какие данные записывать, и выполнять запись непосредственно на каждый шард. То есть, делать INSERT в те таблицы, на которые "смотрит" распределённая таблица. -Это наиболее гибкое решение - вы можете использовать любую схему шардирования, которая может быть нетривиальной из-за требований предметной области. +Во-первых, вы можете самостоятельно определять, на какие серверы какие данные записывать, и выполнять запись непосредственно на каждый шард. То есть, делать INSERT в те таблицы, на которые "смотрит" распределённая таблица. Это наиболее гибкое решение поскольку вы можете использовать любую схему шардирования, которая может быть нетривиальной из-за требований предметной области. Также это является наиболее оптимальным решением, так как данные могут записываться на разные шарды полностью независимо. -Во вторых, вы можете делать INSERT в Distributed таблицу. В этом случае, таблица будет сама распределять вставляемые данные по серверам. -Для того, чтобы писать в Distributed таблицу, у неё должен быть задан ключ шардирования (последний параметр). Также, если шард всего-лишь один, то запись работает и без указания ключа шардирования (так как в этом случае он не имеет смысла). +Во-вторых, вы можете делать INSERT в Distributed таблицу. В этом случае, таблица будет сама распределять вставляемые данные по серверам. Для того, чтобы писать в Distributed таблицу, у неё должен быть задан ключ шардирования (последний параметр). Также, если шард всего-лишь один, то запись работает и без указания ключа шардирования (так как в этом случае он не имеет смысла). У каждого шарда в конфигурационном файле может быть задан "вес" (weight). По умолчанию, вес равен единице. Данные будут распределяться по шардам в количестве, пропорциональном весу шарда. Например, если есть два шарда, и у первого выставлен вес 9, а у второго 10, то на первый будет отправляться 9 / 19 доля строк, а на второй - 10 / 19. @@ -114,7 +112,7 @@ logs - имя кластера в конфигурационном файле с - используются запросы, требующие соединение данных (IN, JOIN) по определённому ключу - тогда если данные шардированы по этому ключу, то можно использовать локальные IN, JOIN вместо GLOBAL IN, GLOBAL JOIN, что кардинально более эффективно. - используется большое количество серверов (сотни и больше) и большое количество маленьких запросов (запросы отдельных клиентов - сайтов, рекламодателей, партнёров) - тогда, для того, чтобы маленькие запросы не затрагивали весь кластер, имеет смысл располагать данные одного клиента на одном шарде, или (вариант, который используется в Яндекс.Метрике) сделать двухуровневое шардирование: разбить весь кластер на "слои", где слой может состоять из нескольких шардов; данные для одного клиента располагаются на одном слое, но в один слой можно по мере необходимости добавлять шарды, в рамках которых данные распределены произвольным образом; создаются распределённые таблицы на каждый слой и одна общая распределённая таблица для глобальных запросов. -Запись данных осуществляется полностью асинхронно. При INSERT-е в Distributed таблицу, блок данных всего лишь записывается в локальную файловую систему. Данные отправляются на удалённые серверы в фоне, при первой возможности. Вы должны проверять, успешно ли отправляются данные, проверяя список файлов (данные, ожидающие отправки) в директории таблицы: /var/lib/clickhouse/data/database/table/. +Запись данных осуществляется полностью асинхронно. При вставке в таблицу, блок данных сначала записывается в файловую систему. Затем, в фоновом режиме отправляются на удалённые серверы при первой возможности. Период отправки регулируется настройками [distributed_directory_monitor_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_sleep_time_ms) и [distributed_directory_monitor_max_sleep_time_ms](../settings/settings.md#distributed_directory_monitor_max_sleep_time_ms). Движок таблиц `Distributed` отправляет каждый файл со вставленными данными отдельно, но можно включить пакетную отправку данных настройкой [distributed_directory_monitor_batch_inserts](../settings/settings.md#distributed_directory_monitor_batch_inserts). Эта настройка улучшает производительность кластера за счет более оптимального использования ресурсов сервера-отправителя и сети. Необходимо проверять, что данные отправлены успешно, для этого проверьте список файлов (данных, ожидающих отправки) в каталоге таблицы `/var/lib/clickhouse/data/database/table/`. Если после INSERT-а в Distributed таблицу, сервер перестал существовать или был грубо перезапущен (например, в следствие аппаратного сбоя), то записанные данные могут быть потеряны. Если в директории таблицы обнаружен повреждённый кусок данных, то он переносится в поддиректорию broken и больше не используется. diff --git a/docs/ru/operations/table_engines/graphitemergetree.md b/docs/ru/operations/table_engines/graphitemergetree.md index 40948512a2c..cbb4cc746df 100644 --- a/docs/ru/operations/table_engines/graphitemergetree.md +++ b/docs/ru/operations/table_engines/graphitemergetree.md @@ -1,4 +1,4 @@ -# GraphiteMergeTree +# GraphiteMergeTree {#graphitemergetree} Движок предназначен для прореживания и агрегирования/усреднения (rollup) данных [Graphite](http://graphite.readthedocs.io/en/latest/index.html). Он может быть интересен разработчикам, которые хотят использовать ClickHouse как хранилище данных для Graphite. @@ -6,7 +6,7 @@ Движок наследует свойства от [MergeTree](mergetree.md). -## Создание таблицы +## Создание таблицы {#creating-table} ```sql CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] @@ -70,7 +70,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] -## Конфигурация rollup +## Конфигурация rollup {#rollup-configuration} Настройки прореживания данных задаются параметром [graphite_rollup](../server_settings/settings.md#server_settings-graphite_rollup) в конфигурации сервера . Имя параметра может быть любым. Можно создать несколько конфигураций и использовать их для разных таблиц. @@ -81,14 +81,14 @@ required-columns patterns ``` -### Требуемые столбцы (required-columns) +### Требуемые столбцы (required-columns) {#required-columns} - `path_column_name` — столбец, в котором хранится название метрики (сенсор Graphite). Значение по умолчанию: `Path`. - `time_column_name` — столбец, в котором хранится время измерения метрики. Значение по умолчанию: `Time`. - `value_column_name` — столбец со значением метрики в момент времени, установленный в `time_column_name`. Значение по умолчанию: `Value`. - `version_column_name` — столбец, в котором хранится версия метрики. Значение по умолчанию: `Timestamp`. -### Правила (patterns) +### Правила (patterns) {#patterns} Структура раздела `patterns`: @@ -129,7 +129,7 @@ default - `precision` – точность определения возраста данных в секундах. Должен быть делителем для 86400 (количество секунд в сутках). - `function` – имя агрегирующей функции, которую следует применить к данным, чей возраст оказался в интервале `[age, age + precision]`. -### Пример конфигурации +### Пример конфигурации {#configuration-example} ```xml diff --git a/docs/ru/operations/table_engines/mergetree.md b/docs/ru/operations/table_engines/mergetree.md index f3eba70f0e2..058e46eed99 100644 --- a/docs/ru/operations/table_engines/mergetree.md +++ b/docs/ru/operations/table_engines/mergetree.md @@ -491,21 +491,25 @@ ALTER TABLE example_table Структура конфигурации: ```xml - - - /mnt/fast_ssd/clickhouse - - - /mnt/hdd1/clickhouse - 10485760_ - - - /mnt/hdd2/clickhouse - 10485760_ - + + + + /mnt/fast_ssd/clickhouse + + + /mnt/hdd1/clickhouse + 10485760 + + + /mnt/hdd2/clickhouse + 10485760 + + ... + + ... - + ``` Теги: @@ -519,26 +523,30 @@ ALTER TABLE example_table Общий вид конфигурации политик хранения: ```xml - - - - - disk_name_from_disks_configuration - 1073741824 - - - - - - - 0.2 - - - - + + ... + + + + + disk_name_from_disks_configuration + 1073741824 + + + + + + + 0.2 + + + + - - + + + ... + ``` Тэги: @@ -552,29 +560,33 @@ ALTER TABLE example_table Примеры конфигураций: ```xml - - - - - disk1 - disk2 - - - + + ... + + + + + disk1 + disk2 + + + - - - - fast_ssd - 1073741824 - - - disk1 - - - 0.2 - - + + + + fast_ssd + 1073741824 + + + disk1 + + + 0.2 + + + ... + ``` В приведенном примере, политика `hdd_in_order` реализует прицип [round-robin](https://ru.wikipedia.org/wiki/Round-robin_(%D0%B0%D0%BB%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%BC)). Так как в политике есть всего один том (`single`), то все записи производятся на его диски по круговому циклу. Такая политика может быть полезна при наличии в системе нескольких похожих дисков, но при этом не сконфигурирован RAID. Учтите, что каждый отдельный диск ненадёжен и чтобы не потерять важные данные это необходимо скомпенсировать за счет хранения данных в трёх копиях. diff --git a/docs/ru/query_language/functions/string_functions.md b/docs/ru/query_language/functions/string_functions.md index 2169cb794e0..56886e83a3d 100644 --- a/docs/ru/query_language/functions/string_functions.md +++ b/docs/ru/query_language/functions/string_functions.md @@ -189,6 +189,44 @@ SELECT startsWith('Hello, world!', 'He'); └───────────────────────────────────┘ ``` +## trim {#trim} + +Удаляет все указанные символы с начала или окончания строки. +По умолчанию удаляет все последовательные вхождения обычных пробелов (32 символ ASCII) с обоих концов строки. + +**Синтаксис** + +```sql +trim([[LEADING|TRAILING|BOTH] trim_character FROM] input_string) +``` + +**Параметры** + +- `trim_character` — один или несколько символов, подлежащие удалению. [String](../../data_types/string.md). +- `input_string` — строка для обрезки. [String](../../data_types/string.md). + +**Возвращаемое значение** + +Исходную строку после обрезки с левого и (или) правого концов строки. + +Тип: `String`. + +**Пример** + +Запрос: + +```sql +SELECT trim(BOTH ' ()' FROM '( Hello, world! )') +``` + +Ответ: + +```text +┌─trim(BOTH ' ()' FROM '( Hello, world! )')─┐ +│ Hello, world! │ +└───────────────────────────────────────────────┘ +``` + ## trimLeft {#trimleft} Удаляет все последовательные вхождения обычных пробелов (32 символ ASCII) с левого конца строки. Не удаляет другие виды пробелов (табуляция, пробел без разрыва и т. д.). @@ -196,14 +234,14 @@ SELECT startsWith('Hello, world!', 'He'); **Синтаксис** ```sql -trimLeft() +trimLeft(input_string) ``` -Алиас: `ltrim`. +Алиас: `ltrim(input_string)`. **Параметры** -- `string` — строка для обрезки. [String](../../data_types/string.md). +- `input_string` — строка для обрезки. [String](../../data_types/string.md). **Возвращаемое значение** @@ -234,14 +272,14 @@ SELECT trimLeft(' Hello, world! ') **Синтаксис** ```sql -trimRight() +trimRight(input_string) ``` -Алиас: `rtrim`. +Алиас: `rtrim(input_string)`. **Параметры** -- `string` — строка для обрезки. [String](../../data_types/string.md). +- `input_string` — строка для обрезки. [String](../../data_types/string.md). **Возвращаемое значение** @@ -272,14 +310,14 @@ SELECT trimRight(' Hello, world! ') **Синтаксис** ```sql -trimBoth() +trimBoth(input_string) ``` -Алиас: `trim`. +Алиас: `trim(input_string)`. **Параметры** -- `string` — строка для обрезки. [String](../../data_types/string.md). +- `input_string` — строка для обрезки. [String](../../data_types/string.md). **Возвращаемое значение** diff --git a/docs/ru/security_changelog.md b/docs/ru/security_changelog.md index 17ae1eba19d..db742b5f990 100644 --- a/docs/ru/security_changelog.md +++ b/docs/ru/security_changelog.md @@ -1,3 +1,23 @@ +## Исправлено в релизе 19.14.3.3, 2019-09-10 + +### CVE-2019-15024 + +Злоумышленник с доступом на запись к ZooKeeper и возможностью запустить собственный сервер в сети доступной ClickHouse может создать вредоносный сервер, который будет вести себя как реплика ClickHouse и зарегистрируется в ZooKeeper. В процессе репликации вредоносный сервер может указать любой путь на файловой системе в который будут записаны данные. + +Обнаружено благодаря: Эльдару Заитову из Службы Информационной Безопасности Яндекса + +### CVE-2019-16535 + +Интерфейс декомпрессии позволял совершать OOB чтения и записи данных в памяти, а также переполнение целочисленных переменных, что могло приводить к отказу в обслуживании. Также потенциально могло использоваьтся для удаленного выполнения кода. + +Обнаружено благодаря: Эльдару Заитову из Службы Информационной Безопасности Яндекса + +### CVE-2019-16536 + +Аутентифицированный клиент злоумышленника имел возможность вызвать переполнение стека, что могло привести к отказу в обслуживании. + +Обнаружено благодаря: Эльдару Заитову из Службы Информационной Безопасности Яндекса + ## Исправлено в релизе 19.13.6.1 от 20 сентября 2019 ### CVE-2019-18657 @@ -19,7 +39,7 @@ unixODBC позволял указать путь для подключения Обнаружено благодаря: Андрею Красичкову и Евгению Сидорову из Службы Информационной Безопасности Яндекса -## Исправлено в релизе 1.1.54388 от 28 июня 2018 +## Исправлено в релизе 1.1.54388 от 28 июня 2018 ### CVE-2018-14668 Табличная функция "remote" допускала произвольные символы в полях "user", "password" и "default_database", что позволяло производить атаки класса Cross Protocol Request Forgery. diff --git a/docs/toc_en.yml b/docs/toc_en.yml index a11c40e4907..604aca5d18e 100644 --- a/docs/toc_en.yml +++ b/docs/toc_en.yml @@ -197,6 +197,7 @@ nav: - 'Configuration Files': 'operations/configuration_files.md' - 'Quotas': 'operations/quotas.md' - 'System Tables': 'operations/system_tables.md' + - 'Testing Hardware': 'operations/performance_test.md' - 'Server Configuration Parameters': - 'Introduction': 'operations/server_settings/index.md' - 'Server Settings': 'operations/server_settings/settings.md' diff --git a/docs/toc_fa.yml b/docs/toc_fa.yml index 710a2ee20f8..b13c8092889 100644 --- a/docs/toc_fa.yml +++ b/docs/toc_fa.yml @@ -193,6 +193,7 @@ nav: - 'Configuration Files': 'operations/configuration_files.md' - 'Quotas': 'operations/quotas.md' - 'System Tables': 'operations/system_tables.md' + - 'Testing Hardware': 'operations/performance_test.md' - 'Server Configuration Parameters': - 'Introduction': 'operations/server_settings/index.md' - 'Server Settings': 'operations/server_settings/settings.md' diff --git a/docs/toc_ja.yml b/docs/toc_ja.yml index 945042f0fef..31b384f97b5 100644 --- a/docs/toc_ja.yml +++ b/docs/toc_ja.yml @@ -197,6 +197,7 @@ nav: - 'Configuration Files': 'operations/configuration_files.md' - 'Quotas': 'operations/quotas.md' - 'System Tables': 'operations/system_tables.md' + - 'Testing Hardware': 'operations/performance_test.md' - 'Server Configuration Parameters': - 'Introduction': 'operations/server_settings/index.md' - 'Server Settings': 'operations/server_settings/settings.md' diff --git a/docs/toc_ru.yml b/docs/toc_ru.yml index 469590b6bc8..dc6a0d9227c 100644 --- a/docs/toc_ru.yml +++ b/docs/toc_ru.yml @@ -196,6 +196,7 @@ nav: - 'Конфигурационные файлы': 'operations/configuration_files.md' - 'Квоты': 'operations/quotas.md' - 'Системные таблицы': 'operations/system_tables.md' + - 'Тестирование оборудования': 'operations/performance_test.md' - 'Конфигурационные параметры сервера': - 'Введение': 'operations/server_settings/index.md' - 'Серверные настройки': 'operations/server_settings/settings.md' diff --git a/docs/toc_zh.yml b/docs/toc_zh.yml index 09f9875069b..c07c3bb2711 100644 --- a/docs/toc_zh.yml +++ b/docs/toc_zh.yml @@ -192,6 +192,7 @@ nav: - '配置文件': 'operations/configuration_files.md' - '配额': 'operations/quotas.md' - '系统表': 'operations/system_tables.md' + - 'Testing Hardware': 'operations/performance_test.md' - 'Server参数配置': - '介绍': 'operations/server_settings/index.md' - 'Server参数说明': 'operations/server_settings/settings.md' diff --git a/docs/zh/interfaces/third-party/integrations.md b/docs/zh/interfaces/third-party/integrations.md index 3e9fcfcd410..253939827df 100644 --- a/docs/zh/interfaces/third-party/integrations.md +++ b/docs/zh/interfaces/third-party/integrations.md @@ -31,7 +31,9 @@ - 监控 - [Graphite](https://graphiteapp.org) - [graphouse](https://github.com/yandex/graphouse) - - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + - [carbon-clickhouse](https://github.com/lomik/carbon-clickhouse) + + - [graphite-clickhouse](https://github.com/lomik/graphite-clickhouse) + - [graphite-ch-optimizer](https://github.com/innogames/graphite-ch-optimizer) - optimizes staled partitions in [\*GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#graphitemergetree) if rules from [rollup configuration](../../operations/table_engines/graphitemergetree.md#rollup-configuration) could be applied - [Grafana](https://grafana.com/) - [clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) - [Prometheus](https://prometheus.io/) diff --git a/docs/zh/operations/performance_test.md b/docs/zh/operations/performance_test.md new file mode 120000 index 00000000000..a74c126c63f --- /dev/null +++ b/docs/zh/operations/performance_test.md @@ -0,0 +1 @@ +../../en/operations/performance_test.md \ No newline at end of file diff --git a/libs/libcommon/include/ext/scope_guard.h b/libs/libcommon/include/ext/scope_guard.h index c2c7e5ec630..c12d17d0398 100644 --- a/libs/libcommon/include/ext/scope_guard.h +++ b/libs/libcommon/include/ext/scope_guard.h @@ -1,22 +1,56 @@ #pragma once #include +#include + namespace ext { - -template class scope_guard { - const F function; - +template +class [[nodiscard]] basic_scope_guard +{ public: - constexpr scope_guard(const F & function_) : function{function_} {} - constexpr scope_guard(F && function_) : function{std::move(function_)} {} - ~scope_guard() { function(); } + constexpr basic_scope_guard() = default; + constexpr basic_scope_guard(basic_scope_guard && src) : function{std::exchange(src.function, F{})} {} + + constexpr basic_scope_guard & operator=(basic_scope_guard && src) + { + if (this != &src) + { + invoke(); + function = std::exchange(src.function, F{}); + } + return *this; + } + + template , void>> + constexpr basic_scope_guard(const G & function_) : function{function_} {} + + template , void>> + constexpr basic_scope_guard(G && function_) : function{std::move(function_)} {} + + ~basic_scope_guard() { invoke(); } + +private: + void invoke() + { + if constexpr (std::is_constructible_v) + { + if (!function) + return; + } + + function(); + } + + F function = F{}; }; -template -inline scope_guard make_scope_guard(F && function_) { return std::forward(function_); } +using scope_guard = basic_scope_guard>; + +template +inline basic_scope_guard make_scope_guard(F && function_) { return std::forward(function_); } } #define SCOPE_EXIT_CONCAT(n, ...) \ diff --git a/libs/libdaemon/include/daemon/BaseDaemon.h b/libs/libdaemon/include/daemon/BaseDaemon.h index 462cbb95418..56f7dc5f06b 100644 --- a/libs/libdaemon/include/daemon/BaseDaemon.h +++ b/libs/libdaemon/include/daemon/BaseDaemon.h @@ -168,22 +168,11 @@ protected: { std::string file; - /// Создать объект, не создавая PID файл - PID() {} - - /// Создать объект, создать PID файл - PID(const std::string & file_) { seed(file_); } - - /// Создать PID файл - void seed(const std::string & file_); - - /// Удалить PID файл - void clear(); - - ~PID() { clear(); } + PID(const std::string & file_); + ~PID(); }; - PID pid; + std::optional pid; std::atomic_bool is_cancelled{false}; diff --git a/libs/libdaemon/src/BaseDaemon.cpp b/libs/libdaemon/src/BaseDaemon.cpp index 50bacb895ef..9d9ea6a0a15 100644 --- a/libs/libdaemon/src/BaseDaemon.cpp +++ b/libs/libdaemon/src/BaseDaemon.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -209,7 +210,7 @@ public: /// This allows to receive more signals if failure happens inside onFault function. /// Example: segfault while symbolizing stack trace. - std::thread([=] { onFault(sig, info, context, stack_trace, thread_num, query_id); }).detach(); + std::thread([=, this] { onFault(sig, info, context, stack_trace, thread_num, query_id); }).detach(); } } } @@ -498,7 +499,7 @@ void BaseDaemon::terminate() void BaseDaemon::kill() { dumpCoverageReportIfPossible(); - pid.clear(); + pid.reset(); if (::raise(SIGKILL) != 0) throw Poco::SystemException("cannot kill process"); } @@ -668,7 +669,7 @@ void BaseDaemon::initialize(Application & self) /// Create pid file. if (config().has("pid")) - pid.seed(config().getString("pid")); + pid.emplace(config().getString("pid")); /// Change path for logging. if (!log_path.empty()) @@ -835,7 +836,7 @@ bool isPidRunning(pid_t pid) return 0; } -void BaseDaemon::PID::seed(const std::string & file_) +BaseDaemon::PID::PID(const std::string & file_) { file = Poco::Path(file_).absolute().toString(); Poco::File poco_file(file); @@ -862,34 +863,28 @@ void BaseDaemon::PID::seed(const std::string & file_) if (-1 == fd) { - file.clear(); if (EEXIST == errno) throw Poco::Exception("Pid file exists, should not start daemon."); throw Poco::CreateFileException("Cannot create pid file."); } + SCOPE_EXIT({ close(fd); }); + + std::stringstream s; + s << getpid(); + if (static_cast(s.str().size()) != write(fd, s.str().c_str(), s.str().size())) + throw Poco::Exception("Cannot write to pid file."); +} + +BaseDaemon::PID::~PID() +{ try { - std::stringstream s; - s << getpid(); - if (static_cast(s.str().size()) != write(fd, s.str().c_str(), s.str().size())) - throw Poco::Exception("Cannot write to pid file."); + Poco::File(file).remove(); } catch (...) { - close(fd); - throw; - } - - close(fd); -} - -void BaseDaemon::PID::clear() -{ - if (!file.empty()) - { - Poco::File(file).remove(); - file.clear(); + DB::tryLogCurrentException(__PRETTY_FUNCTION__); } } diff --git a/utils/check-marks/main.cpp b/utils/check-marks/main.cpp index 0e075f6be4e..a0a204a0185 100644 --- a/utils/check-marks/main.cpp +++ b/utils/check-marks/main.cpp @@ -76,7 +76,7 @@ void checkCompressedHeaders(const std::string & mrk_path, const std::string & bi void checkByCompressedReadBuffer(const std::string & mrk_path, const std::string & bin_path) { DB::ReadBufferFromFile mrk_in(mrk_path); - DB::CompressedReadBufferFromFile bin_in(bin_path, 0, 0); + DB::CompressedReadBufferFromFile bin_in(bin_path, 0, 0, 0); DB::WriteBufferFromFileDescriptor out(STDOUT_FILENO); bool mrk2_format = boost::algorithm::ends_with(mrk_path, ".mrk2"); @@ -133,7 +133,7 @@ int main(int argc, char ** argv) std::cerr << e.what() << ", " << e.message() << std::endl << std::endl << "Stack trace:" << std::endl - << e.getStackTrace().toString() + << e.getStackTraceString() << std::endl; throw; } diff --git a/utils/compressor/CMakeLists.txt b/utils/compressor/CMakeLists.txt index c032054187b..3498640acd1 100644 --- a/utils/compressor/CMakeLists.txt +++ b/utils/compressor/CMakeLists.txt @@ -1,17 +1,5 @@ -find_package (Threads) - -add_executable (zstd_test zstd_test.cpp) -if(ZSTD_LIBRARY) - target_link_libraries(zstd_test PRIVATE ${ZSTD_LIBRARY}) -endif() -target_link_libraries (zstd_test PRIVATE common) - add_executable (mutator mutator.cpp) target_link_libraries(mutator PRIVATE clickhouse_common_io) add_executable (decompress_perf decompress_perf.cpp) target_link_libraries(decompress_perf PRIVATE dbms ${LZ4_LIBRARY}) - -if (NOT USE_INTERNAL_ZSTD_LIBRARY AND ZSTD_INCLUDE_DIR) - target_include_directories (zstd_test BEFORE PRIVATE ${ZSTD_INCLUDE_DIR}) -endif () diff --git a/utils/compressor/zstd_test.cpp b/utils/compressor/zstd_test.cpp deleted file mode 100644 index 01016830e7d..00000000000 --- a/utils/compressor/zstd_test.cpp +++ /dev/null @@ -1,68 +0,0 @@ -#include -#include -#include -#include -#include - - -int main(int argc, char ** argv) -{ - bool compress = argc == 1; - - const size_t size = 1048576; - std::vector src_buf(size); - std::vector dst_buf; - - size_t pos = 0; - while (true) - { - ssize_t read_res = read(STDIN_FILENO, &src_buf[pos], size - pos); - if (read_res < 0) - throw std::runtime_error("Cannot read from stdin"); - if (read_res == 0) - break; - pos += read_res; - } - - src_buf.resize(pos); - - size_t zstd_res; - - if (compress) - { - dst_buf.resize(ZSTD_compressBound(src_buf.size())); - - zstd_res = ZSTD_compress( - &dst_buf[0], - dst_buf.size(), - &src_buf[0], - src_buf.size(), - 1); - } - else - { - dst_buf.resize(size); - - zstd_res = ZSTD_decompress( - &dst_buf[0], - dst_buf.size(), - &src_buf[0], - src_buf.size()); - } - - if (ZSTD_isError(zstd_res)) - throw std::runtime_error(ZSTD_getErrorName(zstd_res)); - - dst_buf.resize(zstd_res); - - pos = 0; - while (pos < dst_buf.size()) - { - ssize_t write_res = write(STDOUT_FILENO, &dst_buf[pos], dst_buf.size()); - if (write_res <= 0) - throw std::runtime_error("Cannot write to stdout"); - pos += write_res; - } - - return 0; -} diff --git a/utils/fill-factor/main.cpp b/utils/fill-factor/main.cpp index fa7ec6252ce..b492be1be85 100644 --- a/utils/fill-factor/main.cpp +++ b/utils/fill-factor/main.cpp @@ -53,7 +53,7 @@ int main(int argc, char ** argv) std::cerr << e.what() << ", " << e.message() << std::endl << std::endl << "Stack trace:" << std::endl - << e.getStackTrace().toString() + << e.getStackTraceString() << std::endl; throw; } diff --git a/utils/make_changelog.py b/utils/make_changelog.py index 50959eb09b7..808fafa6d92 100755 --- a/utils/make_changelog.py +++ b/utils/make_changelog.py @@ -207,7 +207,7 @@ def get_users_info(pull_requests, commits_info, token, max_retries, retry_timeou # List of unknown commits -> text description. def process_unknown_commits(commits, commits_info, users): - pattern = 'Commit: [{}]({})\nAuthor: {}\nMessage: {}' + pattern = u'Commit: [{}]({})\nAuthor: {}\nMessage: {}' texts = [] @@ -455,7 +455,7 @@ def make_changelog(new_tag, prev_tag, pull_requests_nums, repo, repo_folder, sta # Remove double whitespaces and trailing whitespaces changelog = re.sub(r' {2,}| +$', r''.format(repo), changelog) - print(changelog) + print(changelog.encode('utf-8')) if __name__ == '__main__': diff --git a/utils/package/arch/PKGBUILD.in b/utils/package/arch/PKGBUILD.in index b3482b04907..20de555f8a7 100644 --- a/utils/package/arch/PKGBUILD.in +++ b/utils/package/arch/PKGBUILD.in @@ -7,6 +7,12 @@ url='https://clickhouse.yandex/' license=('Apache') package() { + install -dm 755 $pkgdir/usr/lib/tmpfiles.d + install -dm 755 $pkgdir/usr/lib/sysusers.d + install -Dm 644 ${CMAKE_CURRENT_SOURCE_DIR}/clickhouse.tmpfiles $pkgdir/usr/lib/tmpfiles.d/clickhouse.conf + install -Dm 644 ${CMAKE_CURRENT_SOURCE_DIR}/clickhouse.sysusers $pkgdir/usr/lib/sysusers.d/clickhouse.conf + install -dm 755 $pkgdir/etc/clickhouse-server/config.d + install -Dm 644 ${CMAKE_CURRENT_SOURCE_DIR}/logging.xml $pkgdir/etc/clickhouse-server/config.d/logging.xml # This code was requisited from kmeaw@ https://aur.archlinux.org/packages/clickhouse/ . SRC=${ClickHouse_SOURCE_DIR} BIN=${ClickHouse_BINARY_DIR} diff --git a/utils/package/arch/README.md b/utils/package/arch/README.md index 10bdae7367a..0db5aac8080 100644 --- a/utils/package/arch/README.md +++ b/utils/package/arch/README.md @@ -1,9 +1,17 @@ -### Build Arch linux package +### Build Arch Linux package From binary directory: ``` make -cd arch +cd utils/package/arch makepkg ``` + +### Install and start ClickHouse server + +``` +pacman -U clickhouse-*.pkg.tar.xz +systemctl enable clickhouse-server +systemctl start clickhouse-server +``` diff --git a/utils/package/arch/clickhouse.sysusers b/utils/package/arch/clickhouse.sysusers new file mode 100644 index 00000000000..4381c52c4f2 --- /dev/null +++ b/utils/package/arch/clickhouse.sysusers @@ -0,0 +1,3 @@ +u clickhouse - "ClickHouse user" /nonexistent /bin/false +g clickhouse - "ClickHouse group" +m clickhouse clickhouse diff --git a/utils/package/arch/clickhouse.tmpfiles b/utils/package/arch/clickhouse.tmpfiles new file mode 100644 index 00000000000..631aa895f2f --- /dev/null +++ b/utils/package/arch/clickhouse.tmpfiles @@ -0,0 +1 @@ +d /var/lib/clickhouse 0700 clickhouse clickhouse diff --git a/utils/package/arch/logging.xml b/utils/package/arch/logging.xml new file mode 100644 index 00000000000..0f9c51dff80 --- /dev/null +++ b/utils/package/arch/logging.xml @@ -0,0 +1,6 @@ + + + + + + diff --git a/utils/release/release_lib.sh b/utils/release/release_lib.sh index 5aa48a60926..ab395c9ad37 100644 --- a/utils/release/release_lib.sh +++ b/utils/release/release_lib.sh @@ -275,8 +275,39 @@ function make_tgz { PACKAGE_DIR=${PACKAGE_DIR=../} for PACKAGE in clickhouse-server clickhouse-client clickhouse-test clickhouse-common-static clickhouse-common-static-dbg; do - alien --verbose --to-tgz ${PACKAGE_DIR}${PACKAGE}_${VERSION_FULL}_*.deb + alien --verbose --scripts --generate --to-tgz ${PACKAGE_DIR}${PACKAGE}_${VERSION_FULL}_*.deb + PKGDIR="./${PACKAGE}-${VERSION_FULL}" + if [ ! -d "$PKGDIR/install" ]; then + mkdir "$PKGDIR/install" + fi + + if [ ! -f "$PKGDIR/install/doinst.sh" ]; then + echo '#!/bin/sh' > "$PKGDIR/install/doinst.sh" + echo 'set -e' >> "$PKGDIR/install/doinst.sh" + fi + + SCRIPT_TEXT=' +SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )" +for filepath in `find $SCRIPTPATH/.. -type f -or -type l | grep -v "\.\./install/"`; do + destpath=${filepath##$SCRIPTPATH/..} + mkdir -p $(dirname "$destpath") + cp -r "$filepath" "$destpath" +done +' + + echo "$SCRIPT_TEXT" | sed -i "2r /dev/stdin" "$PKGDIR/install/doinst.sh" + + chmod +x "$PKGDIR/install/doinst.sh" + + if [ -f "/usr/bin/pigz" ]; then + tar --use-compress-program=pigz -cf "${PACKAGE}-${VERSION_FULL}.tgz" "$PKGDIR" + else + tar -czf "${PACKAGE}-${VERSION_FULL}.tgz" "$PKGDIR" + fi + + rm -r $PKGDIR done + mv clickhouse-*-${VERSION_FULL}.tgz ${PACKAGE_DIR} } diff --git a/website/index.html b/website/index.html index afe8d2abcf5..ec686bceefb 100644 --- a/website/index.html +++ b/website/index.html @@ -398,7 +398,7 @@

System requirements: Linux, x86_64 with SSE 4.2.

-

Install packages for Ubuntu/Debian or CentOS/RedHat:

+

Install packages for Ubuntu/Debian or CentOS/RedHat or Other Linux:

@@ -422,6 +422,28 @@ sudo yum install clickhouse-server clickhouse-client
 sudo /etc/init.d/clickhouse-server start
 clickhouse-client
+ + +

For other operating systems the easiest way to get started is using @@ -549,7 +571,7 @@ clickhouse-client window.location.host = hostParts[0] + '.' + hostParts[1]; } - var available_distributives = ['deb', 'rpm']; + var available_distributives = ['deb', 'rpm', 'tgz']; var selected_distributive = 'deb'; function refresh_distributives() {