diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e1c1d59b99..950bdc7e374 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,196 @@ +## ClickHouse release 20.7 + +### ClickHouse release v20.7.2.30-stable, 2020-08-31 + +#### Backward Incompatible Change + +* Function `modulo` (operator `%`) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added alias `mod`. This closes [#7323](https://github.com/ClickHouse/ClickHouse/issues/7323). [#12585](https://github.com/ClickHouse/ClickHouse/pull/12585) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Deprecate special printing of zero Date/DateTime values as `0000-00-00` and `0000-00-00 00:00:00`. [#12442](https://github.com/ClickHouse/ClickHouse/pull/12442) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* The function `groupArrayMoving*` was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The function `groupArrayMovingAvg` was returning integer number that was inconsistent with the `avg` function. This fixes [#12568](https://github.com/ClickHouse/ClickHouse/issues/12568). [#12622](https://github.com/ClickHouse/ClickHouse/pull/12622) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. [#13153](https://github.com/ClickHouse/ClickHouse/pull/13153) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Protect from the cases when user may set `background_pool_size` to value lower than `number_of_free_entries_in_pool_to_execute_mutation` or `number_of_free_entries_in_pool_to_lower_max_size_of_merge`. In these cases ALTERs won't work or the maximum size of merge will be too limited. It will throw exception explaining what to do. This closes [#10897](https://github.com/ClickHouse/ClickHouse/issues/10897). [#12728](https://github.com/ClickHouse/ClickHouse/pull/12728) ([alexey-milovidov](https://github.com/alexey-milovidov)). + +#### New Feature + +* Polygon dictionary type that provides efficient "reverse geocoding" lookups - to find the region by coordinates in a dictionary of many polygons (world map). It is using carefully optimized algorithm with recursive grids to maintain low CPU and memory usage. [#9278](https://github.com/ClickHouse/ClickHouse/pull/9278) ([achulkov2](https://github.com/achulkov2)). +* Added support of LDAP authentication for preconfigured users ("Simple Bind" method). [#11234](https://github.com/ClickHouse/ClickHouse/pull/11234) ([Denis Glazachev](https://github.com/traceon)). +* Introduce setting `alter_partition_verbose_result` which outputs information about touched parts for some types of `ALTER TABLE ... PARTITION ...` queries (currently `ATTACH` and `FREEZE`). Closes [#8076](https://github.com/ClickHouse/ClickHouse/issues/8076). [#13017](https://github.com/ClickHouse/ClickHouse/pull/13017) ([alesapin](https://github.com/alesapin)). +* Add `bayesAB` function for bayesian-ab-testing. [#12327](https://github.com/ClickHouse/ClickHouse/pull/12327) ([achimbab](https://github.com/achimbab)). +* Added `system.crash_log` table into which stack traces for fatal errors are collected. This table should be empty. [#12316](https://github.com/ClickHouse/ClickHouse/pull/12316) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Added http headers `X-ClickHouse-Database` and `X-ClickHouse-Format` which may be used to set default database and output format. [#12981](https://github.com/ClickHouse/ClickHouse/pull/12981) ([hcz](https://github.com/hczhcz)). +* Add `minMap` and `maxMap` functions support to `SimpleAggregateFunction`. [#12662](https://github.com/ClickHouse/ClickHouse/pull/12662) ([Ildus Kurbangaliev](https://github.com/ildus)). +* Add setting `allow_non_metadata_alters` which restricts to execute `ALTER` queries which modify data on disk. Disabled be default. Closes [#11547](https://github.com/ClickHouse/ClickHouse/issues/11547). [#12635](https://github.com/ClickHouse/ClickHouse/pull/12635) ([alesapin](https://github.com/alesapin)). +* A function `formatRow` is added to support turning arbitrary expressions into a string via given format. It's useful for manipulating SQL outputs and is quite versatile combined with the `columns` function. [#12574](https://github.com/ClickHouse/ClickHouse/pull/12574) ([Amos Bird](https://github.com/amosbird)). +* Add `FROM_UNIXTIME` function for compatibility with MySQL, related to [12149](https://github.com/ClickHouse/ClickHouse/issues/12149). [#12484](https://github.com/ClickHouse/ClickHouse/pull/12484) ([flynn](https://github.com/ucasFL)). +* Allow Nullable types as keys in MergeTree tables if `allow_nullable_key` table setting is enabled. https://github.com/ClickHouse/ClickHouse/issues/5319. [#12433](https://github.com/ClickHouse/ClickHouse/pull/12433) ([Amos Bird](https://github.com/amosbird)). +* Integration with [COS](https://intl.cloud.tencent.com/product/cos). [#12386](https://github.com/ClickHouse/ClickHouse/pull/12386) ([fastio](https://github.com/fastio)). +* Add mapAdd and mapSubtract functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). + +#### Bug Fix + +* Fix premature `ON CLUSTER` timeouts for queries that must be executed on a single replica. Fixes [#6704](https://github.com/ClickHouse/ClickHouse/issues/6704), [#7228](https://github.com/ClickHouse/ClickHouse/issues/7228), [#13361](https://github.com/ClickHouse/ClickHouse/issues/13361), [#11884](https://github.com/ClickHouse/ClickHouse/issues/11884). [#13450](https://github.com/ClickHouse/ClickHouse/pull/13450) ([alesapin](https://github.com/alesapin)). +* Fix crash in mark inclusion search introduced in https://github.com/ClickHouse/ClickHouse/pull/12277. [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). +* Fix race condition in external dictionaries with cache layout which can lead server crash. [#12566](https://github.com/ClickHouse/ClickHouse/pull/12566) ([alesapin](https://github.com/alesapin)). +* Fix visible data clobbering by progress bar in client in interactive mode. This fixes [#12562](https://github.com/ClickHouse/ClickHouse/issues/12562) and [#13369](https://github.com/ClickHouse/ClickHouse/issues/13369) and [#13584](https://github.com/ClickHouse/ClickHouse/issues/13584) and fixes [#12964](https://github.com/ClickHouse/ClickHouse/issues/12964). [#13691](https://github.com/ClickHouse/ClickHouse/pull/13691) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed incorrect sorting order for `LowCardinality` columns when ORDER BY multiple columns is used. This fixes [#13958](https://github.com/ClickHouse/ClickHouse/issues/13958). [#14223](https://github.com/ClickHouse/ClickHouse/pull/14223) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Removed hardcoded timeout, which wrongly overruled `query_wait_timeout_milliseconds` setting for cache-dictionary. [#14105](https://github.com/ClickHouse/ClickHouse/pull/14105) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fixed wrong mount point in extra info for `Poco::Exception: no space left on device`. [#14050](https://github.com/ClickHouse/ClickHouse/pull/14050) ([tavplubix](https://github.com/tavplubix)). +* Fix wrong query optimization of select queries with `DISTINCT` keyword when subqueries also have `DISTINCT` in case `optimize_duplicate_order_by_and_distinct` setting is enabled. [#13925](https://github.com/ClickHouse/ClickHouse/pull/13925) ([Artem Zuikov](https://github.com/4ertus2)). +* Fixed potential deadlock when renaming `Distributed` table. [#13922](https://github.com/ClickHouse/ClickHouse/pull/13922) ([tavplubix](https://github.com/tavplubix)). +* Fix incorrect sorting for `FixedString` columns when ORDER BY multiple columns is used. Fixes [#13182](https://github.com/ClickHouse/ClickHouse/issues/13182). [#13887](https://github.com/ClickHouse/ClickHouse/pull/13887) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix potentially lower precision of `topK`/`topKWeighted` aggregations (with non-default parameters). [#13817](https://github.com/ClickHouse/ClickHouse/pull/13817) ([Azat Khuzhin](https://github.com/azat)). +* Fix reading from MergeTree table with INDEX of type SET fails when compared against NULL. This fixes [#13686](https://github.com/ClickHouse/ClickHouse/issues/13686). [#13793](https://github.com/ClickHouse/ClickHouse/pull/13793) ([Amos Bird](https://github.com/amosbird)). +* Fix step overflow in function `range()`. [#13790](https://github.com/ClickHouse/ClickHouse/pull/13790) ([Azat Khuzhin](https://github.com/azat)). +* Fixed `Directory not empty` error when concurrently executing `DROP DATABASE` and `CREATE TABLE`. [#13756](https://github.com/ClickHouse/ClickHouse/pull/13756) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add range check for `h3KRing` function. This fixes [#13633](https://github.com/ClickHouse/ClickHouse/issues/13633). [#13752](https://github.com/ClickHouse/ClickHouse/pull/13752) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of [#8602](https://github.com/ClickHouse/ClickHouse/issues/8602) that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. [#13746](https://github.com/ClickHouse/ClickHouse/pull/13746) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix logging Settings.Names/Values when `log_queries_min_type` greater than `QUERY_START`. [#13737](https://github.com/ClickHouse/ClickHouse/pull/13737) ([Azat Khuzhin](https://github.com/azat)). +* Fix incorrect message in `clickhouse-server.init` while checking user and group. [#13711](https://github.com/ClickHouse/ClickHouse/pull/13711) ([ylchou](https://github.com/ylchou)). +* Do not optimize `any(arrayJoin())` to `arrayJoin()` under `optimize_move_functions_out_of_any`. [#13681](https://github.com/ClickHouse/ClickHouse/pull/13681) ([Azat Khuzhin](https://github.com/azat)). +* Fixed possible deadlock in concurrent `ALTER ... REPLACE/MOVE PARTITION ...` queries. [#13626](https://github.com/ClickHouse/ClickHouse/pull/13626) ([tavplubix](https://github.com/tavplubix)). +* Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. [#13624](https://github.com/ClickHouse/ClickHouse/pull/13624) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix secondary indices corruption in compact parts (compact parts is an experimental feature). [#13538](https://github.com/ClickHouse/ClickHouse/pull/13538) ([Anton Popov](https://github.com/CurtizJ)). +* Fix wrong code in function `netloc`. This fixes [#13335](https://github.com/ClickHouse/ClickHouse/issues/13335). [#13446](https://github.com/ClickHouse/ClickHouse/pull/13446) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix error in `parseDateTimeBestEffort` function when unix timestamp was passed as an argument. This fixes [#13362](https://github.com/ClickHouse/ClickHouse/issues/13362). [#13441](https://github.com/ClickHouse/ClickHouse/pull/13441) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix invalid return type for comparison of tuples with `NULL` elements. Fixes [#12461](https://github.com/ClickHouse/ClickHouse/issues/12461). [#13420](https://github.com/ClickHouse/ClickHouse/pull/13420) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix wrong optimization caused `aggregate function any(x) is found inside another aggregate function in query` error with `SET optimize_move_functions_out_of_any = 1` and aliases inside `any()`. [#13419](https://github.com/ClickHouse/ClickHouse/pull/13419) ([Artem Zuikov](https://github.com/4ertus2)). +* Fix possible race in `StorageMemory`. [#13416](https://github.com/ClickHouse/ClickHouse/pull/13416) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix empty output for `Arrow` and `Parquet` formats in case if query return zero rows. It was done because empty output is not valid for this formats. [#13399](https://github.com/ClickHouse/ClickHouse/pull/13399) ([hcz](https://github.com/hczhcz)). +* Fix select queries with constant columns and prefix of primary key in `ORDER BY` clause. [#13396](https://github.com/ClickHouse/ClickHouse/pull/13396) ([Anton Popov](https://github.com/CurtizJ)). +* Fix `PrettyCompactMonoBlock` for clickhouse-local. Fix extremes/totals with `PrettyCompactMonoBlock`. Fixes [#7746](https://github.com/ClickHouse/ClickHouse/issues/7746). [#13394](https://github.com/ClickHouse/ClickHouse/pull/13394) ([Azat Khuzhin](https://github.com/azat)). +* Fixed deadlock in system.text_log. [#12452](https://github.com/ClickHouse/ClickHouse/pull/12452) ([alexey-milovidov](https://github.com/alexey-milovidov)). It is a part of [#12339](https://github.com/ClickHouse/ClickHouse/issues/12339). This fixes [#12325](https://github.com/ClickHouse/ClickHouse/issues/12325). [#13386](https://github.com/ClickHouse/ClickHouse/pull/13386) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fixed `File(TSVWithNames*)` (header was written multiple times), fixed `clickhouse-local --format CSVWithNames*` (lacks header, broken after [#12197](https://github.com/ClickHouse/ClickHouse/issues/12197)), fixed `clickhouse-local --format CSVWithNames*` with zero rows (lacks header). [#13343](https://github.com/ClickHouse/ClickHouse/pull/13343) ([Azat Khuzhin](https://github.com/azat)). +* Fix segfault when function `groupArrayMovingSum` deserializes empty state. Fixes [#13339](https://github.com/ClickHouse/ClickHouse/issues/13339). [#13341](https://github.com/ClickHouse/ClickHouse/pull/13341) ([alesapin](https://github.com/alesapin)). +* Throw error on `arrayJoin()` function in `JOIN ON` section. [#13330](https://github.com/ClickHouse/ClickHouse/pull/13330) ([Artem Zuikov](https://github.com/4ertus2)). +* Fix crash in `LEFT ASOF JOIN` with `join_use_nulls=1`. [#13291](https://github.com/ClickHouse/ClickHouse/pull/13291) ([Artem Zuikov](https://github.com/4ertus2)). +* Fix possible error `Totals having transform was already added to pipeline` in case of a query from delayed replica. [#13290](https://github.com/ClickHouse/ClickHouse/pull/13290) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* The server may crash if user passed specifically crafted arguments to the function `h3ToChildren`. This fixes [#13275](https://github.com/ClickHouse/ClickHouse/issues/13275). [#13277](https://github.com/ClickHouse/ClickHouse/pull/13277) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix potentially low performance and slightly incorrect result for `uniqExact`, `topK`, `sumDistinct` and similar aggregate functions called on Float types with `NaN` values. It also triggered assert in debug build. This fixes [#12491](https://github.com/ClickHouse/ClickHouse/issues/12491). [#13254](https://github.com/ClickHouse/ClickHouse/pull/13254) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes [#12465](https://github.com/ClickHouse/ClickHouse/issues/12465). [#13251](https://github.com/ClickHouse/ClickHouse/pull/13251) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Return passed number for numbers with MSB set in function roundUpToPowerOfTwoOrZero(). It prevents potential errors in case of overflow of array sizes. [#13234](https://github.com/ClickHouse/ClickHouse/pull/13234) ([Azat Khuzhin](https://github.com/azat)). +* Fix function if with nullable constexpr as cond that is not literal NULL. Fixes [#12463](https://github.com/ClickHouse/ClickHouse/issues/12463). [#13226](https://github.com/ClickHouse/ClickHouse/pull/13226) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix assert in `arrayElement` function in case of array elements are Nullable and array subscript is also Nullable. This fixes [#12172](https://github.com/ClickHouse/ClickHouse/issues/12172). [#13224](https://github.com/ClickHouse/ClickHouse/pull/13224) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix DateTime64 conversion functions with constant argument. [#13205](https://github.com/ClickHouse/ClickHouse/pull/13205) ([Azat Khuzhin](https://github.com/azat)). +* Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes https://github.com/ClickHouse/ClickHouse/issues/5779, https://github.com/ClickHouse/ClickHouse/issues/12527. [#13199](https://github.com/ClickHouse/ClickHouse/pull/13199) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix access to `redis` dictionary after connection was dropped once. It may happen with `cache` and `direct` dictionary layouts. [#13082](https://github.com/ClickHouse/ClickHouse/pull/13082) ([Anton Popov](https://github.com/CurtizJ)). +* Fix wrong index analysis with functions. It could lead to some data parts being skipped when reading from `MergeTree` tables. Fixes [#13060](https://github.com/ClickHouse/ClickHouse/issues/13060). Fixes [#12406](https://github.com/ClickHouse/ClickHouse/issues/12406). [#13081](https://github.com/ClickHouse/ClickHouse/pull/13081) ([Anton Popov](https://github.com/CurtizJ)). +* Fix error `Cannot convert column because it is constant but values of constants are different in source and result` for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like `now()`, `now64()`, `randConstant()`. Fixes [#11327](https://github.com/ClickHouse/ClickHouse/issues/11327). [#13075](https://github.com/ClickHouse/ClickHouse/pull/13075) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix crash which was possible for queries with `ORDER BY` tuple and small `LIMIT`. Fixes [#12623](https://github.com/ClickHouse/ClickHouse/issues/12623). [#13009](https://github.com/ClickHouse/ClickHouse/pull/13009) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix `Block structure mismatch` error for queries with `UNION` and `JOIN`. Fixes [#12602](https://github.com/ClickHouse/ClickHouse/issues/12602). [#12989](https://github.com/ClickHouse/ClickHouse/pull/12989) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Corrected `merge_with_ttl_timeout` logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). [#12982](https://github.com/ClickHouse/ClickHouse/pull/12982) ([Alexander Kazakov](https://github.com/Akazz)). +* Fix columns duplication for range hashed dictionary created from DDL query. This fixes [#10605](https://github.com/ClickHouse/ClickHouse/issues/10605). [#12857](https://github.com/ClickHouse/ClickHouse/pull/12857) ([alesapin](https://github.com/alesapin)). +* Fix unnecessary limiting for the number of threads for selects from local replica. [#12840](https://github.com/ClickHouse/ClickHouse/pull/12840) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix rare bug when `ALTER DELETE` and `ALTER MODIFY COLUMN` queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in `count.txt` and as a consequence incorrect data in part. Also, fix a small bug with simultaneous `ALTER RENAME COLUMN` and `ALTER ADD COLUMN`. [#12760](https://github.com/ClickHouse/ClickHouse/pull/12760) ([alesapin](https://github.com/alesapin)). +* Wrong credentials being used when using `clickhouse` dictionary source to query remote tables. [#12756](https://github.com/ClickHouse/ClickHouse/pull/12756) ([sundyli](https://github.com/sundy-li)). +* Fix `CAST(Nullable(String), Enum())`. [#12745](https://github.com/ClickHouse/ClickHouse/pull/12745) ([Azat Khuzhin](https://github.com/azat)). +* Fix performance with large tuples, which are interpreted as functions in `IN` section. The case when user writes `WHERE x IN tuple(1, 2, ...)` instead of `WHERE x IN (1, 2, ...)` for some obscure reason. [#12700](https://github.com/ClickHouse/ClickHouse/pull/12700) ([Anton Popov](https://github.com/CurtizJ)). +* Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). [#12672](https://github.com/ClickHouse/ClickHouse/pull/12672) ([Azat Khuzhin](https://github.com/azat)). +* Fix wrong optimization `optimize_move_functions_out_of_any=1` in case of `any(func())`. [#12664](https://github.com/ClickHouse/ClickHouse/pull/12664) ([Artem Zuikov](https://github.com/4ertus2)). +* Fixed [#10572](https://github.com/ClickHouse/ClickHouse/issues/10572) fix bloom filter index with const expression. [#12659](https://github.com/ClickHouse/ClickHouse/pull/12659) ([Winter Zhang](https://github.com/zhang2014)). +* Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). [#12658](https://github.com/ClickHouse/ClickHouse/pull/12658) ([Azat Khuzhin](https://github.com/azat)). +* Add support for function `if` with `Array(UUID)` arguments. This fixes [#11066](https://github.com/ClickHouse/ClickHouse/issues/11066). [#12648](https://github.com/ClickHouse/ClickHouse/pull/12648) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. This fixes https://github.com/ClickHouse/ClickHouse/issues/12507. [#12646](https://github.com/ClickHouse/ClickHouse/pull/12646) ([Vitaly Baranov](https://github.com/vitlibar)). +* Exception `There is no supertype...` can be thrown during `ALTER ... UPDATE` in unexpected cases (e.g. when subtracting from UInt64 column). This fixes [#7306](https://github.com/ClickHouse/ClickHouse/issues/7306). This fixes [#4165](https://github.com/ClickHouse/ClickHouse/issues/4165). [#12633](https://github.com/ClickHouse/ClickHouse/pull/12633) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix possible `Pipeline stuck` error for queries with external sorting. Fixes [#12617](https://github.com/ClickHouse/ClickHouse/issues/12617). [#12618](https://github.com/ClickHouse/ClickHouse/pull/12618) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix error `Output of TreeExecutor is not sorted` for `OPTIMIZE DEDUPLICATE`. Fixes [#11572](https://github.com/ClickHouse/ClickHouse/issues/11572). [#12613](https://github.com/ClickHouse/ClickHouse/pull/12613) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix the issue when alias on result of function `any` can be lost during query optimization. [#12593](https://github.com/ClickHouse/ClickHouse/pull/12593) ([Anton Popov](https://github.com/CurtizJ)). +* Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. [#12556](https://github.com/ClickHouse/ClickHouse/pull/12556) ([Azat Khuzhin](https://github.com/azat)). +* Now ClickHouse will recalculate checksums for parts when file `checksums.txt` is absent. Broken since [#9827](https://github.com/ClickHouse/ClickHouse/issues/9827). [#12545](https://github.com/ClickHouse/ClickHouse/pull/12545) ([alesapin](https://github.com/alesapin)). +* Fix bug which lead to broken old parts after `ALTER DELETE` query when `enable_mixed_granularity_parts=1`. Fixes [#12536](https://github.com/ClickHouse/ClickHouse/issues/12536). [#12543](https://github.com/ClickHouse/ClickHouse/pull/12543) ([alesapin](https://github.com/alesapin)). +* Fixing race condition in live view tables which could cause data duplication. LIVE VIEW is an experimental feature. [#12519](https://github.com/ClickHouse/ClickHouse/pull/12519) ([vzakaznikov](https://github.com/vzakaznikov)). +* Fix backwards compatibility in binary format of `AggregateFunction(avg, ...)` values. This fixes [#12342](https://github.com/ClickHouse/ClickHouse/issues/12342). [#12486](https://github.com/ClickHouse/ClickHouse/pull/12486) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix crash in JOIN with dictionary when we are joining over expression of dictionary key: `t JOIN dict ON expr(dict.id) = t.id`. Disable dictionary join optimisation for this case. [#12458](https://github.com/ClickHouse/ClickHouse/pull/12458) ([Artem Zuikov](https://github.com/4ertus2)). +* Fix overflow when very large LIMIT or OFFSET is specified. This fixes [#10470](https://github.com/ClickHouse/ClickHouse/issues/10470). This fixes [#11372](https://github.com/ClickHouse/ClickHouse/issues/11372). [#12427](https://github.com/ClickHouse/ClickHouse/pull/12427) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* kafka: fix SIGSEGV if there is a message with error in the middle of the batch. [#12302](https://github.com/ClickHouse/ClickHouse/pull/12302) ([Azat Khuzhin](https://github.com/azat)). + +#### Improvement + +* Keep smaller amount of logs in ZooKeeper. Avoid excessive growing of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. [#13100](https://github.com/ClickHouse/ClickHouse/pull/13100) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes [#11329](https://github.com/ClickHouse/ClickHouse/issues/11329). [#12666](https://github.com/ClickHouse/ClickHouse/pull/12666) ([alesapin](https://github.com/alesapin)). +* Add `QueryTimeMicroseconds`, `SelectQueryTimeMicroseconds` and `InsertQueryTimeMicroseconds` to `system.events`, along with system.metrics, processes, query_log, etc. [#13028](https://github.com/ClickHouse/ClickHouse/pull/13028) ([ianton-ru](https://github.com/ianton-ru)). +* Added `SelectedRows` and `SelectedBytes` to `system.events`, along with system.metrics, processes, query_log, etc. [#12638](https://github.com/ClickHouse/ClickHouse/pull/12638) ([ianton-ru](https://github.com/ianton-ru)). +* Added `current_database` information to `system.query_log`. [#12652](https://github.com/ClickHouse/ClickHouse/pull/12652) ([Amos Bird](https://github.com/amosbird)). +* Allow `TabSeparatedRaw` as input format. [#12009](https://github.com/ClickHouse/ClickHouse/pull/12009) ([hcz](https://github.com/hczhcz)). +* Now `joinGet` supports multi-key lookup. [#12418](https://github.com/ClickHouse/ClickHouse/pull/12418) ([Amos Bird](https://github.com/amosbird)). +* Allow `*Map` aggregate functions to work on Arrays with NULLs. Fixes [#13157](https://github.com/ClickHouse/ClickHouse/issues/13157). [#13225](https://github.com/ClickHouse/ClickHouse/pull/13225) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example, `1970-01-01 00:00:00` in Moscow). Saturate to zero instead. This fixes [#3470](https://github.com/ClickHouse/ClickHouse/issues/3470). This fixes [#4172](https://github.com/ClickHouse/ClickHouse/issues/4172). [#12443](https://github.com/ClickHouse/ClickHouse/pull/12443) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* AvroConfluent: Skip Kafka tombstone records - Support skipping broken records [#13203](https://github.com/ClickHouse/ClickHouse/pull/13203) ([Andrew Onyshchuk](https://github.com/oandrew)). +* Fix wrong error for long queries. It was possible to get syntax error other than `Max query size exceeded` for correct query. [#13928](https://github.com/ClickHouse/ClickHouse/pull/13928) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix data race in `lgamma` function. This race was caught only in `tsan`, no side effects really happened. [#13842](https://github.com/ClickHouse/ClickHouse/pull/13842) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix a 'Week'-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. [#13417](https://github.com/ClickHouse/ClickHouse/pull/13417) ([vladimir-golovchenko](https://github.com/vladimir-golovchenko)). +* Now broken parts are also reported when encountered in compact part processing. Compact parts is an experimental feature. [#13282](https://github.com/ClickHouse/ClickHouse/pull/13282) ([Amos Bird](https://github.com/amosbird)). +* Fix assert in `geohashesInBox`. This fixes [#12554](https://github.com/ClickHouse/ClickHouse/issues/12554). [#13229](https://github.com/ClickHouse/ClickHouse/pull/13229) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix assert in `parseDateTimeBestEffort`. This fixes [#12649](https://github.com/ClickHouse/ClickHouse/issues/12649). [#13227](https://github.com/ClickHouse/ClickHouse/pull/13227) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Minor optimization in Processors/PipelineExecutor: breaking out of a loop because it makes sense to do so. [#13058](https://github.com/ClickHouse/ClickHouse/pull/13058) ([Mark Papadakis](https://github.com/markpapadakis)). +* Support TRUNCATE table without TABLE keyword. [#12653](https://github.com/ClickHouse/ClickHouse/pull/12653) ([Winter Zhang](https://github.com/zhang2014)). +* Fix explain query format overwrite by default, issue https://github.com/ClickHouse/ClickHouse/issues/12432. [#12541](https://github.com/ClickHouse/ClickHouse/pull/12541) ([BohuTANG](https://github.com/BohuTANG)). +* Allow to set JOIN kind and type in more standad way: `LEFT SEMI JOIN` instead of `SEMI LEFT JOIN`. For now both are correct. [#12520](https://github.com/ClickHouse/ClickHouse/pull/12520) ([Artem Zuikov](https://github.com/4ertus2)). +* Changes default value for `multiple_joins_rewriter_version` to 2. It enables new multiple joins rewriter that knows about column names. [#12469](https://github.com/ClickHouse/ClickHouse/pull/12469) ([Artem Zuikov](https://github.com/4ertus2)). +* Add several metrics for requests to S3 storages. [#12464](https://github.com/ClickHouse/ClickHouse/pull/12464) ([ianton-ru](https://github.com/ianton-ru)). +* Use correct default secure port for clickhouse-benchmark with `--secure` argument. This fixes [#11044](https://github.com/ClickHouse/ClickHouse/issues/11044). [#12440](https://github.com/ClickHouse/ClickHouse/pull/12440) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Rollback insertion errors in `Log`, `TinyLog`, `StripeLog` engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes [#12402](https://github.com/ClickHouse/ClickHouse/issues/12402). [#12426](https://github.com/ClickHouse/ClickHouse/pull/12426) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Implement `RENAME DATABASE` and `RENAME DICTIONARY` for `Atomic` database engine - Add implicit `{uuid}` macro, which can be used in ZooKeeper path for `ReplicatedMergeTree`. It works with `CREATE ... ON CLUSTER ...` queries. Set `show_table_uuid_in_table_create_query_if_not_nil` to `true` to use it. - Make `ReplicatedMergeTree` engine arguments optional, `/clickhouse/tables/{uuid}/{shard}/` and `{replica}` are used by default. Closes [#12135](https://github.com/ClickHouse/ClickHouse/issues/12135). - Minor fixes. - These changes break backward compatibility of `Atomic` database engine. Previously created `Atomic` databases must be manually converted to new format. Atomic database is an experimental feature. [#12343](https://github.com/ClickHouse/ClickHouse/pull/12343) ([tavplubix](https://github.com/tavplubix)). +* Separated `AWSAuthV4Signer` into different logger, removed excessive `AWSClient: AWSClient` from log messages. [#12320](https://github.com/ClickHouse/ClickHouse/pull/12320) ([Vladimir Chebotarev](https://github.com/excitoon)). +* Better exception message in disk access storage. [#12625](https://github.com/ClickHouse/ClickHouse/pull/12625) ([alesapin](https://github.com/alesapin)). +* Better exception for function `in` with invalid number of arguments. [#12529](https://github.com/ClickHouse/ClickHouse/pull/12529) ([Anton Popov](https://github.com/CurtizJ)). +* Fix error message about adaptive granularity. [#12624](https://github.com/ClickHouse/ClickHouse/pull/12624) ([alesapin](https://github.com/alesapin)). +* Fix SETTINGS parse after FORMAT. [#12480](https://github.com/ClickHouse/ClickHouse/pull/12480) ([Azat Khuzhin](https://github.com/azat)). +* If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed [#7941](https://github.com/ClickHouse/ClickHouse/issues/7941). [#12382](https://github.com/ClickHouse/ClickHouse/pull/12382) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). [#13086](https://github.com/ClickHouse/ClickHouse/pull/13086) ([Azat Khuzhin](https://github.com/azat)). + +#### Performance Improvement + +* Lower memory usage for some operations up to 2 times. [#12424](https://github.com/ClickHouse/ClickHouse/pull/12424) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Optimize PK lookup for queries that match exact PK range. [#12277](https://github.com/ClickHouse/ClickHouse/pull/12277) ([Ivan Babrou](https://github.com/bobrik)). +* Slightly optimize very short queries with `LowCardinality`. [#14129](https://github.com/ClickHouse/ClickHouse/pull/14129) ([Anton Popov](https://github.com/CurtizJ)). +* Slightly improve performance of aggregation by UInt8/UInt16 keys. [#13091](https://github.com/ClickHouse/ClickHouse/pull/13091) and [#13055](https://github.com/ClickHouse/ClickHouse/pull/13055) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Push down `LIMIT` step for query plan (inside subqueries). [#13016](https://github.com/ClickHouse/ClickHouse/pull/13016) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Parallel primary key lookup and skipping index stages on parts, as described in [#11564](https://github.com/ClickHouse/ClickHouse/issues/11564). [#12589](https://github.com/ClickHouse/ClickHouse/pull/12589) ([Ivan Babrou](https://github.com/bobrik)). +* Converting String-type arguments of function "if" and "transform" into enum if `set optimize_if_transform_strings_to_enum = 1`. [#12515](https://github.com/ClickHouse/ClickHouse/pull/12515) ([Artem Zuikov](https://github.com/4ertus2)). +* Replaces monotonic functions with its argument in `ORDER BY` if `set optimize_monotonous_functions_in_order_by=1`. [#12467](https://github.com/ClickHouse/ClickHouse/pull/12467) ([Artem Zuikov](https://github.com/4ertus2)). +* Add order by optimization that rewrites `ORDER BY x, f(x)` with `ORDER by x` if `set optimize_redundant_functions_in_order_by = 1`. [#12404](https://github.com/ClickHouse/ClickHouse/pull/12404) ([Artem Zuikov](https://github.com/4ertus2)). +* Allow pushdown predicate when subquery contains `WITH` clause. This fixes [#12293](https://github.com/ClickHouse/ClickHouse/issues/12293) [#12663](https://github.com/ClickHouse/ClickHouse/pull/12663) ([Winter Zhang](https://github.com/zhang2014)). +* Improve performance of reading from compact parts. Compact parts is an experimental feature. [#12492](https://github.com/ClickHouse/ClickHouse/pull/12492) ([Anton Popov](https://github.com/CurtizJ)). +* Attempt to implement streaming optimization in `DiskS3`. DiskS3 is an experimental feature. [#12434](https://github.com/ClickHouse/ClickHouse/pull/12434) ([Vladimir Chebotarev](https://github.com/excitoon)). + +#### Build/Testing/Packaging Improvement + +* Use `shellcheck` for sh tests linting. [#13200](https://github.com/ClickHouse/ClickHouse/pull/13200) [#13207](https://github.com/ClickHouse/ClickHouse/pull/13207) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add script which set labels for pull requests in GitHub hook. [#13183](https://github.com/ClickHouse/ClickHouse/pull/13183) ([alesapin](https://github.com/alesapin)). +* Remove some of recursive submodules. See [#13378](https://github.com/ClickHouse/ClickHouse/issues/13378). [#13379](https://github.com/ClickHouse/ClickHouse/pull/13379) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Ensure that all the submodules are from proper URLs. Continuation of [#13379](https://github.com/ClickHouse/ClickHouse/issues/13379). This fixes [#13378](https://github.com/ClickHouse/ClickHouse/issues/13378). [#13397](https://github.com/ClickHouse/ClickHouse/pull/13397) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Added support for user-declared settings, which can be accessed from inside queries. This is needed when ClickHouse engine is used as a component of another system. [#13013](https://github.com/ClickHouse/ClickHouse/pull/13013) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added testing for RBAC functionality of INSERT privilege in TestFlows. Expanded tables on which SELECT is being tested. Added Requirements to match new table engine tests. [#13340](https://github.com/ClickHouse/ClickHouse/pull/13340) ([MyroTk](https://github.com/MyroTk)). +* Fix timeout error during server restart in the stress test. [#13321](https://github.com/ClickHouse/ClickHouse/pull/13321) ([alesapin](https://github.com/alesapin)). +* Now fast test will wait server with retries. [#13284](https://github.com/ClickHouse/ClickHouse/pull/13284) ([alesapin](https://github.com/alesapin)). +* Function `materialize()` (the function for ClickHouse testing) will work for NULL as expected - by transforming it to non-constant column. [#13212](https://github.com/ClickHouse/ClickHouse/pull/13212) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix libunwind build in AArch64. This fixes [#13204](https://github.com/ClickHouse/ClickHouse/issues/13204). [#13208](https://github.com/ClickHouse/ClickHouse/pull/13208) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Even more retries in zkutil gtest to prevent test flakiness. [#13165](https://github.com/ClickHouse/ClickHouse/pull/13165) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Small fixes to the RBAC TestFlows. [#13152](https://github.com/ClickHouse/ClickHouse/pull/13152) ([vzakaznikov](https://github.com/vzakaznikov)). +* Fixing `00960_live_view_watch_events_live.py` test. [#13108](https://github.com/ClickHouse/ClickHouse/pull/13108) ([vzakaznikov](https://github.com/vzakaznikov)). +* Improve cache purge in documentation deploy script. [#13107](https://github.com/ClickHouse/ClickHouse/pull/13107) ([alesapin](https://github.com/alesapin)). +* Rewrote some orphan tests to gtest. Removed useless includes from tests. [#13073](https://github.com/ClickHouse/ClickHouse/pull/13073) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Added tests for RBAC functionality of `SELECT` privilege in TestFlows. [#13061](https://github.com/ClickHouse/ClickHouse/pull/13061) ([Ritaank Tiwari](https://github.com/ritaank)). +* Rerun some tests in fast test check. [#12992](https://github.com/ClickHouse/ClickHouse/pull/12992) ([alesapin](https://github.com/alesapin)). +* Fix MSan error in "rdkafka" library. This closes [#12990](https://github.com/ClickHouse/ClickHouse/issues/12990). Updated `rdkafka` to version 1.5 (master). [#12991](https://github.com/ClickHouse/ClickHouse/pull/12991) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes [#12318](https://github.com/ClickHouse/ClickHouse/issues/12318). Author: @qoega. [#12441](https://github.com/ClickHouse/ClickHouse/pull/12441) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix UBSan report in HDFS library. This closes [#12330](https://github.com/ClickHouse/ClickHouse/issues/12330). [#12453](https://github.com/ClickHouse/ClickHouse/pull/12453) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Check an ability that we able to restore the backup from an old version to the new version. This closes [#8979](https://github.com/ClickHouse/ClickHouse/issues/8979). [#12959](https://github.com/ClickHouse/ClickHouse/pull/12959) ([alesapin](https://github.com/alesapin)). +* Do not build helper_container image inside integrational tests. Build docker container in CI and use pre-built helper_container in integration tests. [#12953](https://github.com/ClickHouse/ClickHouse/pull/12953) ([Ilya Yatsishin](https://github.com/qoega)). +* Add a test for `ALTER TABLE CLEAR COLUMN` query for primary key columns. [#12951](https://github.com/ClickHouse/ClickHouse/pull/12951) ([alesapin](https://github.com/alesapin)). +* Increased timeouts in testflows tests. [#12949](https://github.com/ClickHouse/ClickHouse/pull/12949) ([vzakaznikov](https://github.com/vzakaznikov)). +* Fix build of test under Mac OS X. This closes [#12767](https://github.com/ClickHouse/ClickHouse/issues/12767). [#12772](https://github.com/ClickHouse/ClickHouse/pull/12772) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Connector-ODBC updated to mysql-connector-odbc-8.0.21. [#12739](https://github.com/ClickHouse/ClickHouse/pull/12739) ([Ilya Yatsishin](https://github.com/qoega)). +* Adding RBAC syntax tests in TestFlows. [#12642](https://github.com/ClickHouse/ClickHouse/pull/12642) ([vzakaznikov](https://github.com/vzakaznikov)). +* Improve performance of TestKeeper. This will speedup tests with heavy usage of Replicated tables. [#12505](https://github.com/ClickHouse/ClickHouse/pull/12505) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Now we check that server is able to start after stress tests run. This fixes [#12473](https://github.com/ClickHouse/ClickHouse/issues/12473). [#12496](https://github.com/ClickHouse/ClickHouse/pull/12496) ([alesapin](https://github.com/alesapin)). +* Update fmtlib to master (7.0.1). [#12446](https://github.com/ClickHouse/ClickHouse/pull/12446) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add docker image for fast tests. [#12294](https://github.com/ClickHouse/ClickHouse/pull/12294) ([alesapin](https://github.com/alesapin)). +* Rework configuration paths for integration tests. [#12285](https://github.com/ClickHouse/ClickHouse/pull/12285) ([Ilya Yatsishin](https://github.com/qoega)). +* Add compiler option to control that stack frames are not too large. This will help to run the code in fibers with small stack size. [#11524](https://github.com/ClickHouse/ClickHouse/pull/11524) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Update gitignore-files. [#13447](https://github.com/ClickHouse/ClickHouse/pull/13447) ([vladimir-golovchenko](https://github.com/vladimir-golovchenko)). + + ## ClickHouse release 20.6 ### ClickHouse release v20.6.3.28-stable diff --git a/README.md b/README.md index 5daf152109f..300ef4555a2 100644 --- a/README.md +++ b/README.md @@ -17,5 +17,4 @@ ClickHouse is an open-source column-oriented database management system that all ## Upcoming Events -* [ClickHouse at ByteDance (in Chinese)](https://mp.weixin.qq.com/s/Em-HjPylO8D7WPui4RREAQ) on August 28, 2020. * [ClickHouse Data Integration Virtual Meetup](https://www.eventbrite.com/e/clickhouse-september-virtual-meetup-data-integration-tickets-117421895049) on September 10, 2020. diff --git a/base/common/arithmeticOverflow.h b/base/common/arithmeticOverflow.h index 3dfbdbc1346..e228af287e2 100644 --- a/base/common/arithmeticOverflow.h +++ b/base/common/arithmeticOverflow.h @@ -38,18 +38,18 @@ namespace common } template <> - inline bool addOverflow(bInt256 x, bInt256 y, bInt256 & res) + inline bool addOverflow(wInt256 x, wInt256 y, wInt256 & res) { res = x + y; - return (y > 0 && x > std::numeric_limits::max() - y) || - (y < 0 && x < std::numeric_limits::min() - y); + return (y > 0 && x > std::numeric_limits::max() - y) || + (y < 0 && x < std::numeric_limits::min() - y); } template <> - inline bool addOverflow(bUInt256 x, bUInt256 y, bUInt256 & res) + inline bool addOverflow(wUInt256 x, wUInt256 y, wUInt256 & res) { res = x + y; - return x > std::numeric_limits::max() - y; + return x > std::numeric_limits::max() - y; } template @@ -86,15 +86,15 @@ namespace common } template <> - inline bool subOverflow(bInt256 x, bInt256 y, bInt256 & res) + inline bool subOverflow(wInt256 x, wInt256 y, wInt256 & res) { res = x - y; - return (y < 0 && x > std::numeric_limits::max() + y) || - (y > 0 && x < std::numeric_limits::min() + y); + return (y < 0 && x > std::numeric_limits::max() + y) || + (y > 0 && x < std::numeric_limits::min() + y); } template <> - inline bool subOverflow(bUInt256 x, bUInt256 y, bUInt256 & res) + inline bool subOverflow(wUInt256 x, wUInt256 y, wUInt256 & res) { res = x - y; return x < y; @@ -137,19 +137,19 @@ namespace common } template <> - inline bool mulOverflow(bInt256 x, bInt256 y, bInt256 & res) + inline bool mulOverflow(wInt256 x, wInt256 y, wInt256 & res) { res = x * y; if (!x || !y) return false; - bInt256 a = (x > 0) ? x : -x; - bInt256 b = (y > 0) ? y : -y; + wInt256 a = (x > 0) ? x : -x; + wInt256 b = (y > 0) ? y : -y; return (a * b) / b != a; } template <> - inline bool mulOverflow(bUInt256 x, bUInt256 y, bUInt256 & res) + inline bool mulOverflow(wUInt256 x, wUInt256 y, wUInt256 & res) { res = x * y; if (!x || !y) diff --git a/base/common/types.h b/base/common/types.h index 0a394de9f5c..682fe94366c 100644 --- a/base/common/types.h +++ b/base/common/types.h @@ -6,7 +6,7 @@ #include #include -#include +#include using Int8 = int8_t; using Int16 = int16_t; @@ -25,12 +25,11 @@ using UInt64 = uint64_t; using Int128 = __int128; -/// We have to use 127 and 255 bit integers to safe a bit for a sign serialization -//using bInt256 = boost::multiprecision::int256_t; -using bInt256 = boost::multiprecision::number >; -using bUInt256 = boost::multiprecision::uint256_t; +using wInt256 = std::wide_integer<256, signed>; +using wUInt256 = std::wide_integer<256, unsigned>; +static_assert(sizeof(wInt256) == 32); +static_assert(sizeof(wUInt256) == 32); using String = std::string; @@ -44,7 +43,7 @@ struct is_signed }; template <> struct is_signed { static constexpr bool value = true; }; -template <> struct is_signed { static constexpr bool value = true; }; +template <> struct is_signed { static constexpr bool value = true; }; template inline constexpr bool is_signed_v = is_signed::value; @@ -55,7 +54,7 @@ struct is_unsigned static constexpr bool value = std::is_unsigned_v; }; -template <> struct is_unsigned { static constexpr bool value = true; }; +template <> struct is_unsigned { static constexpr bool value = true; }; template inline constexpr bool is_unsigned_v = is_unsigned::value; @@ -69,8 +68,8 @@ struct is_integer }; template <> struct is_integer { static constexpr bool value = true; }; -template <> struct is_integer { static constexpr bool value = true; }; -template <> struct is_integer { static constexpr bool value = true; }; +template <> struct is_integer { static constexpr bool value = true; }; +template <> struct is_integer { static constexpr bool value = true; }; template inline constexpr bool is_integer_v = is_integer::value; @@ -93,9 +92,9 @@ struct make_unsigned typedef std::make_unsigned_t type; }; -template <> struct make_unsigned<__int128> { using type = unsigned __int128; }; -template <> struct make_unsigned { using type = bUInt256; }; -template <> struct make_unsigned { using type = bUInt256; }; +template <> struct make_unsigned { using type = unsigned __int128; }; +template <> struct make_unsigned { using type = wUInt256; }; +template <> struct make_unsigned { using type = wUInt256; }; template using make_unsigned_t = typename make_unsigned::type; @@ -105,8 +104,8 @@ struct make_signed typedef std::make_signed_t type; }; -template <> struct make_signed { typedef bInt256 type; }; -template <> struct make_signed { typedef bInt256 type; }; +template <> struct make_signed { using type = wInt256; }; +template <> struct make_signed { using type = wInt256; }; template using make_signed_t = typename make_signed::type; @@ -116,8 +115,8 @@ struct is_big_int static constexpr bool value = false; }; -template <> struct is_big_int { static constexpr bool value = true; }; -template <> struct is_big_int { static constexpr bool value = true; }; +template <> struct is_big_int { static constexpr bool value = true; }; +template <> struct is_big_int { static constexpr bool value = true; }; template inline constexpr bool is_big_int_v = is_big_int::value; @@ -125,14 +124,11 @@ inline constexpr bool is_big_int_v = is_big_int::value; template inline std::string bigintToString(const T & x) { - return x.str(); + return to_string(x); } template inline To bigint_cast(const From & x [[maybe_unused]]) { - if constexpr ((is_big_int_v && std::is_same_v) || (is_big_int_v && std::is_same_v)) - return static_cast(x); - else - return static_cast(x); + return static_cast(x); } diff --git a/base/common/wide_integer.h b/base/common/wide_integer.h new file mode 100644 index 00000000000..67d0b3f04da --- /dev/null +++ b/base/common/wide_integer.h @@ -0,0 +1,249 @@ +#pragma once + +/////////////////////////////////////////////////////////////// +// Distributed under the Boost Software License, Version 1.0. +// (See at http://www.boost.org/LICENSE_1_0.txt) +/////////////////////////////////////////////////////////////// + +/* Divide and multiply + * + * + * Copyright (c) 2008 + * Evan Teran + * + * Permission to use, copy, modify, and distribute this software and its + * documentation for any purpose and without fee is hereby granted, provided + * that the above copyright notice appears in all copies and that both the + * copyright notice and this permission notice appear in supporting + * documentation, and that the same name not be used in advertising or + * publicity pertaining to distribution of the software without specific, + * written prior permission. We make no representations about the + * suitability this software for any purpose. It is provided "as is" + * without express or implied warranty. + */ + +#include // CHAR_BIT +#include +#include +#include +#include + +namespace std +{ +template +class wide_integer; + +template +struct common_type, wide_integer>; + +template +struct common_type, Arithmetic>; + +template +struct common_type>; + +template +class wide_integer +{ +public: + using base_type = uint8_t; + using signed_base_type = int8_t; + + // ctors + wide_integer() = default; + + template + constexpr wide_integer(T rhs) noexcept; + template + constexpr wide_integer(std::initializer_list il) noexcept; + + // assignment + template + constexpr wide_integer & operator=(const wide_integer & rhs) noexcept; + + template + constexpr wide_integer & operator=(Arithmetic rhs) noexcept; + + template + constexpr wide_integer & operator*=(const Arithmetic & rhs); + + template + constexpr wide_integer & operator/=(const Arithmetic & rhs); + + template + constexpr wide_integer & operator+=(const Arithmetic & rhs) noexcept(is_same::value); + + template + constexpr wide_integer & operator-=(const Arithmetic & rhs) noexcept(is_same::value); + + template + constexpr wide_integer & operator%=(const Integral & rhs); + + template + constexpr wide_integer & operator&=(const Integral & rhs) noexcept; + + template + constexpr wide_integer & operator|=(const Integral & rhs) noexcept; + + template + constexpr wide_integer & operator^=(const Integral & rhs) noexcept; + + constexpr wide_integer & operator<<=(int n); + constexpr wide_integer & operator>>=(int n) noexcept; + + constexpr wide_integer & operator++() noexcept(is_same::value); + constexpr wide_integer operator++(int) noexcept(is_same::value); + constexpr wide_integer & operator--() noexcept(is_same::value); + constexpr wide_integer operator--(int) noexcept(is_same::value); + + // observers + + constexpr explicit operator bool() const noexcept; + + template + using __integral_not_wide_integer_class = typename std::enable_if::value, T>::type; + + template > + constexpr operator T() const noexcept; + + constexpr operator long double() const noexcept; + constexpr operator double() const noexcept; + constexpr operator float() const noexcept; + + struct _impl; + +private: + template + friend class wide_integer; + + friend class numeric_limits>; + friend class numeric_limits>; + + base_type m_arr[_impl::arr_size]; +}; + +template +static constexpr bool ArithmeticConcept() noexcept; +template +using __only_arithmetic = typename std::enable_if() && ArithmeticConcept()>::type; + +template +static constexpr bool IntegralConcept() noexcept; +template +using __only_integer = typename std::enable_if() && IntegralConcept()>::type; + +// Unary operators +template +constexpr wide_integer operator~(const wide_integer & lhs) noexcept; + +template +constexpr wide_integer operator-(const wide_integer & lhs) noexcept(is_same::value); + +template +constexpr wide_integer operator+(const wide_integer & lhs) noexcept(is_same::value); + +// Binary operators +template +std::common_type_t, wide_integer> constexpr +operator*(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator*(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator/(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator/(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator+(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator+(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator-(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator-(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator%(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator%(const Integral & rhs, const Integral2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator&(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator&(const Integral & rhs, const Integral2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator|(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator|(const Integral & rhs, const Integral2 & lhs); + +template +std::common_type_t, wide_integer> constexpr +operator^(const wide_integer & lhs, const wide_integer & rhs); +template > +std::common_type_t constexpr operator^(const Integral & rhs, const Integral2 & lhs); + +// TODO: Integral +template +constexpr wide_integer operator<<(const wide_integer & lhs, int n) noexcept; +template +constexpr wide_integer operator>>(const wide_integer & lhs, int n) noexcept; + +template >> +constexpr wide_integer operator<<(const wide_integer & lhs, Int n) noexcept +{ + return lhs << int(n); +} +template >> +constexpr wide_integer operator>>(const wide_integer & lhs, Int n) noexcept +{ + return lhs >> int(n); +} + +template +constexpr bool operator<(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator<(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +constexpr bool operator>(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator>(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +constexpr bool operator<=(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator<=(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +constexpr bool operator>=(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator>=(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +constexpr bool operator==(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator==(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +constexpr bool operator!=(const wide_integer & lhs, const wide_integer & rhs); +template > +constexpr bool operator!=(const Arithmetic & rhs, const Arithmetic2 & lhs); + +template +std::string to_string(const wide_integer & n); + +template +struct hash>; + +} + +#include "wide_integer_impl.h" diff --git a/base/common/wide_integer_impl.h b/base/common/wide_integer_impl.h new file mode 100644 index 00000000000..c77a9120a55 --- /dev/null +++ b/base/common/wide_integer_impl.h @@ -0,0 +1,1301 @@ +/// Original is here https://github.com/cerevra/int +#pragma once + +#include "wide_integer.h" + +#include +#include + +namespace std +{ +#define CT(x) \ + std::common_type_t, std::decay_t> { x } + +// numeric limits +template +class numeric_limits> +{ +public: + static constexpr bool is_specialized = true; + static constexpr bool is_signed = is_same::value; + static constexpr bool is_integer = true; + static constexpr bool is_exact = true; + static constexpr bool has_infinity = false; + static constexpr bool has_quiet_NaN = false; + static constexpr bool has_signaling_NaN = true; + static constexpr std::float_denorm_style has_denorm = std::denorm_absent; + static constexpr bool has_denorm_loss = false; + static constexpr std::float_round_style round_style = std::round_toward_zero; + static constexpr bool is_iec559 = false; + static constexpr bool is_bounded = true; + static constexpr bool is_modulo = true; + static constexpr int digits = Bits - (is_same::value ? 1 : 0); + static constexpr int digits10 = digits * 0.30103 /*std::log10(2)*/; + static constexpr int max_digits10 = 0; + static constexpr int radix = 2; + static constexpr int min_exponent = 0; + static constexpr int min_exponent10 = 0; + static constexpr int max_exponent = 0; + static constexpr int max_exponent10 = 0; + static constexpr bool traps = true; + static constexpr bool tinyness_before = false; + + static constexpr wide_integer min() noexcept + { + if (is_same::value) + { + using T = wide_integer; + T res{}; + res.m_arr[T::_impl::big(0)] = std::numeric_limits::signed_base_type>::min(); + return res; + } + return 0; + } + + static constexpr wide_integer max() noexcept + { + using T = wide_integer; + T res{}; + res.m_arr[T::_impl::big(0)] = is_same::value + ? std::numeric_limits::signed_base_type>::max() + : std::numeric_limits::base_type>::max(); + for (int i = 1; i < wide_integer::_impl::arr_size; ++i) + { + res.m_arr[T::_impl::big(i)] = std::numeric_limits::base_type>::max(); + } + return res; + } + + static constexpr wide_integer lowest() noexcept { return min(); } + static constexpr wide_integer epsilon() noexcept { return 0; } + static constexpr wide_integer round_error() noexcept { return 0; } + static constexpr wide_integer infinity() noexcept { return 0; } + static constexpr wide_integer quiet_NaN() noexcept { return 0; } + static constexpr wide_integer signaling_NaN() noexcept { return 0; } + static constexpr wide_integer denorm_min() noexcept { return 0; } +}; + +template +struct IsWideInteger +{ + static const constexpr bool value = false; +}; + +template +struct IsWideInteger> +{ + static const constexpr bool value = true; +}; + +template +static constexpr bool ArithmeticConcept() noexcept +{ + return std::is_arithmetic_v || IsWideInteger::value; +} + +template +static constexpr bool IntegralConcept() noexcept +{ + return std::is_integral_v || IsWideInteger::value; +} + +// type traits +template +struct common_type, wide_integer> +{ + using type = std::conditional_t < Bits == Bits2, + wide_integer< + Bits, + std::conditional_t<(std::is_same::value && std::is_same::value), signed, unsigned>>, + std::conditional_t, wide_integer>>; +}; + +template +struct common_type, Arithmetic> +{ + static_assert(ArithmeticConcept(), ""); + + using type = std::conditional_t< + std::is_floating_point::value, + Arithmetic, + std::conditional_t< + sizeof(Arithmetic) < Bits * sizeof(long), + wide_integer, + std::conditional_t< + Bits * sizeof(long) < sizeof(Arithmetic), + Arithmetic, + std::conditional_t< + Bits * sizeof(long) == sizeof(Arithmetic) && (is_same::value || std::is_signed::value), + Arithmetic, + wide_integer>>>>; +}; + +template +struct common_type> : std::common_type, Arithmetic> +{ +}; + +template +struct wide_integer::_impl +{ + static_assert(Bits % CHAR_BIT == 0, "=)"); + + // utils + static const int base_bits = sizeof(base_type) * CHAR_BIT; + static const int arr_size = Bits / base_bits; + static constexpr size_t _Bits = Bits; + static constexpr bool _is_wide_integer = true; + + // The original implementation is big-endian. We need little one. + static constexpr unsigned little(unsigned idx) { return idx; } + static constexpr unsigned big(unsigned idx) { return arr_size - 1 - idx; } + static constexpr unsigned any(unsigned idx) { return idx; } + + template + constexpr static bool is_negative(const wide_integer & n) noexcept + { + if constexpr (std::is_same_v) + return static_cast(n.m_arr[big(0)]) < 0; + else + return false; + } + + template + constexpr static wide_integer make_positive(const wide_integer & n) noexcept + { + return is_negative(n) ? operator_unary_minus(n) : n; + } + + template + constexpr static auto to_Integral(T f) noexcept + { + if constexpr (std::is_same_v) + return f; + else if constexpr (std::is_signed_v) + return static_cast(f); + else + return static_cast(f); + } + + template + constexpr static void wide_integer_from_bultin(wide_integer & self, Integral rhs) noexcept + { + auto r = _impl::to_Integral(rhs); + + int r_idx = 0; + for (; static_cast(r_idx) < sizeof(Integral) && r_idx < arr_size; ++r_idx) + { + base_type & curr = self.m_arr[little(r_idx)]; + base_type curr_rhs = (r >> (r_idx * CHAR_BIT)) & std::numeric_limits::max(); + curr = curr_rhs; + } + + for (; r_idx < arr_size; ++r_idx) + { + base_type & curr = self.m_arr[little(r_idx)]; + curr = r < 0 ? std::numeric_limits::max() : 0; + } + } + + constexpr static void wide_integer_from_bultin(wide_integer & self, double rhs) noexcept + { + if ((rhs > 0 && rhs < std::numeric_limits::max()) || (rhs < 0 && rhs > std::numeric_limits::min())) + { + self = to_Integral(rhs); + return; + } + + long double r = rhs; + if (r < 0) + r = -r; + + size_t count = r / std::numeric_limits::max(); + self = count; + self *= std::numeric_limits::max(); + long double to_diff = count; + to_diff *= std::numeric_limits::max(); + + self += to_Integral(r - to_diff); + + if (rhs < 0) + self = -self; + } + + template + constexpr static void + wide_integer_from_wide_integer(wide_integer & self, const wide_integer & rhs) noexcept + { + // int Bits_to_copy = std::min(arr_size, rhs.arr_size); + auto rhs_arr_size = wide_integer::_impl::arr_size; + int base_elems_to_copy = _impl::arr_size < rhs_arr_size ? _impl::arr_size : rhs_arr_size; + for (int i = 0; i < base_elems_to_copy; ++i) + { + self.m_arr[little(i)] = rhs.m_arr[little(i)]; + } + for (int i = 0; i < arr_size - base_elems_to_copy; ++i) + { + self.m_arr[big(i)] = is_negative(rhs) ? std::numeric_limits::max() : 0; + } + } + + template + constexpr static bool should_keep_size() + { + return sizeof(T) * CHAR_BIT <= Bits; + } + + constexpr static wide_integer shift_left(const wide_integer & rhs, int n) + { + if (static_cast(n) >= base_bits * arr_size) + return 0; + if (n <= 0) + return rhs; + + wide_integer lhs = rhs; + int bit_shift = n % base_bits; + unsigned n_bytes = n / base_bits; + if (bit_shift) + { + lhs.m_arr[big(0)] <<= bit_shift; + for (int i = 1; i < arr_size; ++i) + { + lhs.m_arr[big(i - 1)] |= lhs.m_arr[big(i)] >> (base_bits - bit_shift); + lhs.m_arr[big(i)] <<= bit_shift; + } + } + if (n_bytes) + { + for (unsigned i = 0; i < arr_size - n_bytes; ++i) + { + lhs.m_arr[big(i)] = lhs.m_arr[big(i + n_bytes)]; + } + for (unsigned i = arr_size - n_bytes; i < arr_size; ++i) + lhs.m_arr[big(i)] = 0; + } + return lhs; + } + + constexpr static wide_integer shift_left(const wide_integer & rhs, int n) + { + // static_assert(is_negative(rhs), "shift left for negative lhsbers is underfined!"); + if (is_negative(rhs)) + throw std::runtime_error("shift left for negative lhsbers is underfined!"); + + return wide_integer(shift_left(wide_integer(rhs), n)); + } + + constexpr static wide_integer shift_right(const wide_integer & rhs, int n) noexcept + { + if (static_cast(n) >= base_bits * arr_size) + return 0; + if (n <= 0) + return rhs; + + wide_integer lhs = rhs; + int bit_shift = n % base_bits; + unsigned n_bytes = n / base_bits; + if (bit_shift) + { + lhs.m_arr[little(0)] >>= bit_shift; + for (int i = 1; i < arr_size; ++i) + { + lhs.m_arr[little(i - 1)] |= lhs.m_arr[little(i)] << (base_bits - bit_shift); + lhs.m_arr[little(i)] >>= bit_shift; + } + } + if (n_bytes) + { + for (unsigned i = 0; i < arr_size - n_bytes; ++i) + { + lhs.m_arr[little(i)] = lhs.m_arr[little(i + n_bytes)]; + } + for (unsigned i = arr_size - n_bytes; i < arr_size; ++i) + lhs.m_arr[little(i)] = 0; + } + return lhs; + } + + constexpr static wide_integer shift_right(const wide_integer & rhs, int n) noexcept + { + if (static_cast(n) >= base_bits * arr_size) + return 0; + if (n <= 0) + return rhs; + + bool is_neg = is_negative(rhs); + if (!is_neg) + return shift_right(wide_integer(rhs), n); + + wide_integer lhs = rhs; + int bit_shift = n % base_bits; + unsigned n_bytes = n / base_bits; + if (bit_shift) + { + lhs = shift_right(wide_integer(lhs), bit_shift); + lhs.m_arr[big(0)] |= std::numeric_limits::max() << (base_bits - bit_shift); + } + if (n_bytes) + { + for (unsigned i = 0; i < arr_size - n_bytes; ++i) + { + lhs.m_arr[little(i)] = lhs.m_arr[little(i + n_bytes)]; + } + for (unsigned i = arr_size - n_bytes; i < arr_size; ++i) + { + lhs.m_arr[little(i)] = std::numeric_limits::max(); + } + } + return lhs; + } + + template + constexpr static wide_integer + operator_plus_T(const wide_integer & lhs, T rhs) noexcept(is_same::value) + { + if (rhs < 0) + return _operator_minus_T(lhs, -rhs); + else + return _operator_plus_T(lhs, rhs); + } + +private: + template + constexpr static wide_integer + _operator_minus_T(const wide_integer & lhs, T rhs) noexcept(is_same::value) + { + wide_integer res = lhs; + + bool is_underflow = false; + int r_idx = 0; + for (; static_cast(r_idx) < sizeof(T) && r_idx < arr_size; ++r_idx) + { + base_type & res_i = res.m_arr[little(r_idx)]; + base_type curr_rhs = (rhs >> (r_idx * CHAR_BIT)) & std::numeric_limits::max(); + + if (is_underflow) + { + --res_i; + is_underflow = res_i == std::numeric_limits::max(); + } + + if (res_i < curr_rhs) + is_underflow = true; + res_i -= curr_rhs; + } + + if (is_underflow && r_idx < arr_size) + { + --res.m_arr[little(r_idx)]; + for (int i = arr_size - 1 - r_idx - 1; i >= 0; --i) + { + if (res.m_arr[big(i + 1)] == std::numeric_limits::max()) + --res.m_arr[big(i)]; + else + break; + } + } + + return res; + } + + template + constexpr static wide_integer + _operator_plus_T(const wide_integer & lhs, T rhs) noexcept(is_same::value) + { + wide_integer res = lhs; + + bool is_overflow = false; + int r_idx = 0; + for (; static_cast(r_idx) < sizeof(T) && r_idx < arr_size; ++r_idx) + { + base_type & res_i = res.m_arr[little(r_idx)]; + base_type curr_rhs = (rhs >> (r_idx * CHAR_BIT)) & std::numeric_limits::max(); + + if (is_overflow) + { + ++res_i; + is_overflow = res_i == 0; + } + + res_i += curr_rhs; + if (res_i < curr_rhs) + is_overflow = true; + } + + if (is_overflow && r_idx < arr_size) + { + ++res.m_arr[little(r_idx)]; + for (int i = arr_size - 1 - r_idx - 1; i >= 0; --i) + { + if (res.m_arr[big(i + 1)] == 0) + ++res.m_arr[big(i)]; + else + break; + } + } + + return res; + } + +public: + constexpr static wide_integer operator_unary_tilda(const wide_integer & lhs) noexcept + { + wide_integer res{}; + + for (int i = 0; i < arr_size; ++i) + res.m_arr[any(i)] = ~lhs.m_arr[any(i)]; + return res; + } + + constexpr static wide_integer + operator_unary_minus(const wide_integer & lhs) noexcept(is_same::value) + { + return operator_plus_T(operator_unary_tilda(lhs), 1); + } + + template + constexpr static auto operator_plus(const wide_integer & lhs, const T & rhs) noexcept(is_same::value) + { + if constexpr (should_keep_size()) + { + wide_integer t = rhs; + if (is_negative(t)) + return _operator_minus_wide_integer(lhs, operator_unary_minus(t)); + else + return _operator_plus_wide_integer(lhs, t); + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, wide_integer>::_impl::operator_plus( + wide_integer(lhs), rhs); + } + } + + template + constexpr static auto operator_minus(const wide_integer & lhs, const T & rhs) noexcept(is_same::value) + { + if constexpr (should_keep_size()) + { + wide_integer t = rhs; + if (is_negative(t)) + return _operator_plus_wide_integer(lhs, operator_unary_minus(t)); + else + return _operator_minus_wide_integer(lhs, t); + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, wide_integer>::_impl::operator_minus( + wide_integer(lhs), rhs); + } + } + +private: + constexpr static wide_integer _operator_minus_wide_integer( + const wide_integer & lhs, const wide_integer & rhs) noexcept(is_same::value) + { + wide_integer res = lhs; + + bool is_underflow = false; + for (int idx = 0; idx < arr_size; ++idx) + { + base_type & res_i = res.m_arr[little(idx)]; + const base_type rhs_i = rhs.m_arr[little(idx)]; + + if (is_underflow) + { + --res_i; + is_underflow = res_i == std::numeric_limits::max(); + } + + if (res_i < rhs_i) + is_underflow = true; + + res_i -= rhs_i; + } + + return res; + } + + constexpr static wide_integer _operator_plus_wide_integer( + const wide_integer & lhs, const wide_integer & rhs) noexcept(is_same::value) + { + wide_integer res = lhs; + + bool is_overflow = false; + for (int idx = 0; idx < arr_size; ++idx) + { + base_type & res_i = res.m_arr[little(idx)]; + const base_type rhs_i = rhs.m_arr[little(idx)]; + + if (is_overflow) + { + ++res_i; + is_overflow = res_i == 0; + } + + res_i += rhs_i; + + if (res_i < rhs_i) + is_overflow = true; + } + + return res; + } + +public: + template + constexpr static auto operator_star(const wide_integer & lhs, const T & rhs) + { + if constexpr (should_keep_size()) + { + const wide_integer a = make_positive(lhs); + wide_integer t = make_positive(wide_integer(rhs)); + + wide_integer res = 0; + + for (size_t i = 0; i < arr_size * base_bits; ++i) + { + if (t.m_arr[little(0)] & 1) + res = operator_plus(res, shift_left(a, i)); + + t = shift_right(t, 1); + } + + if (is_same::value && is_negative(wide_integer(rhs)) != is_negative(lhs)) + res = operator_unary_minus(res); + + return res; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_star(T(lhs), rhs); + } + } + + template + constexpr static bool operator_more(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + // static_assert(Signed == std::is_signed::value, + // "warning: operator_more: comparison of integers of different signs"); + + wide_integer t = rhs; + + if (std::numeric_limits::is_signed && (is_negative(lhs) != is_negative(t))) + return is_negative(t); + + for (int i = 0; i < arr_size; ++i) + { + if (lhs.m_arr[big(i)] != t.m_arr[big(i)]) + return lhs.m_arr[big(i)] > t.m_arr[big(i)]; + } + + return false; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_more(T(lhs), rhs); + } + } + + template + constexpr static bool operator_less(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + // static_assert(Signed == std::is_signed::value, + // "warning: operator_less: comparison of integers of different signs"); + + wide_integer t = rhs; + + if (std::numeric_limits::is_signed && (is_negative(lhs) != is_negative(t))) + return is_negative(lhs); + + for (int i = 0; i < arr_size; ++i) + if (lhs.m_arr[big(i)] != t.m_arr[big(i)]) + return lhs.m_arr[big(i)] < t.m_arr[big(i)]; + + return false; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_less(T(lhs), rhs); + } + } + + template + constexpr static bool operator_eq(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + wide_integer t = rhs; + + for (int i = 0; i < arr_size; ++i) + if (lhs.m_arr[any(i)] != t.m_arr[any(i)]) + return false; + + return true; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_eq(T(lhs), rhs); + } + } + + template + constexpr static auto operator_pipe(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + wide_integer t = rhs; + wide_integer res = lhs; + + for (int i = 0; i < arr_size; ++i) + res.m_arr[any(i)] |= t.m_arr[any(i)]; + return res; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_pipe(T(lhs), rhs); + } + } + + template + constexpr static auto operator_amp(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + wide_integer t = rhs; + wide_integer res = lhs; + + for (int i = 0; i < arr_size; ++i) + res.m_arr[any(i)] &= t.m_arr[any(i)]; + return res; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, T>::_impl::operator_amp(T(lhs), rhs); + } + } + +private: + template + constexpr static void divide(const T & lhserator, const T & denominator, T & quotient, T & remainder) + { + bool is_zero = true; + for (auto c : denominator.m_arr) + { + if (c != 0) + { + is_zero = false; + break; + } + } + + if (is_zero) + throw std::domain_error("divide by zero"); + + T n = lhserator; + T d = denominator; + T x = 1; + T answer = 0; + + while (!operator_more(d, n) && operator_eq(operator_amp(shift_right(d, base_bits * arr_size - 1), 1), 0)) + { + x = shift_left(x, 1); + d = shift_left(d, 1); + } + + while (!operator_eq(x, 0)) + { + if (!operator_more(d, n)) + { + n = operator_minus(n, d); + answer = operator_pipe(answer, x); + } + + x = shift_right(x, 1); + d = shift_right(d, 1); + } + + quotient = answer; + remainder = n; + } + +public: + template + constexpr static auto operator_slash(const wide_integer & lhs, const T & rhs) + { + if constexpr (should_keep_size()) + { + wide_integer o = rhs; + wide_integer quotient{}, remainder{}; + divide(make_positive(lhs), make_positive(o), quotient, remainder); + + if (is_same::value && is_negative(o) != is_negative(lhs)) + quotient = operator_unary_minus(quotient); + + return quotient; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, wide_integer>::operator_slash(T(lhs), rhs); + } + } + + template + constexpr static auto operator_percent(const wide_integer & lhs, const T & rhs) + { + if constexpr (should_keep_size()) + { + wide_integer o = rhs; + wide_integer quotient{}, remainder{}; + divide(make_positive(lhs), make_positive(o), quotient, remainder); + + if (is_same::value && is_negative(lhs)) + remainder = operator_unary_minus(remainder); + + return remainder; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return std::common_type_t, wide_integer>::operator_percent(T(lhs), rhs); + } + } + + // ^ + template + constexpr static auto operator_circumflex(const wide_integer & lhs, const T & rhs) noexcept + { + if constexpr (should_keep_size()) + { + wide_integer t(rhs); + wide_integer res = lhs; + + for (int i = 0; i < arr_size; ++i) + res.m_arr[any(i)] ^= t.m_arr[any(i)]; + return res; + } + else + { + static_assert(T::_impl::_is_wide_integer, ""); + return T::operator_circumflex(T(lhs), rhs); + } + } + + constexpr static wide_integer from_str(const char * c) + { + wide_integer res = 0; + + bool is_neg = is_same::value && *c == '-'; + if (is_neg) + ++c; + + if (*c == '0' && (*(c + 1) == 'x' || *(c + 1) == 'X')) + { // hex + ++c; + ++c; + while (*c) + { + if (*c >= '0' && *c <= '9') + { + res = operator_star(res, 16U); + res = operator_plus_T(res, *c - '0'); + ++c; + } + else if (*c >= 'a' && *c <= 'f') + { + res = operator_star(res, 16U); + res = operator_plus_T(res, *c - 'a' + 10U); + ++c; + } + else if (*c >= 'A' && *c <= 'F') + { // tolower must be used, but it is not constexpr + res = operator_star(res, 16U); + res = operator_plus_T(res, *c - 'A' + 10U); + ++c; + } + else + throw std::runtime_error("invalid char from"); + } + } + else + { // dec + while (*c) + { + if (*c < '0' || *c > '9') + throw std::runtime_error("invalid char from"); + + res = operator_star(res, 10U); + res = operator_plus_T(res, *c - '0'); + ++c; + } + } + + if (is_neg) + res = operator_unary_minus(res); + + return res; + } +}; + +// Members + +template +template +constexpr wide_integer::wide_integer(T rhs) noexcept + : m_arr{} +{ + if constexpr (IsWideInteger::value) + _impl::wide_integer_from_wide_integer(*this, rhs); + else + _impl::wide_integer_from_bultin(*this, rhs); +} + +template +template +constexpr wide_integer::wide_integer(std::initializer_list il) noexcept + : m_arr{} +{ + if (il.size() == 1) + { + if constexpr (IsWideInteger::value) + _impl::wide_integer_from_wide_integer(*this, *il.begin()); + else + _impl::wide_integer_from_bultin(*this, *il.begin()); + } + else + _impl::wide_integer_from_bultin(*this, 0); +} + +template +template +constexpr wide_integer & wide_integer::operator=(const wide_integer & rhs) noexcept +{ + _impl::wide_integer_from_wide_integer(*this, rhs); + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator=(T rhs) noexcept +{ + _impl::wide_integer_from_bultin(*this, rhs); + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator*=(const T & rhs) +{ + *this = *this * rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator/=(const T & rhs) +{ + *this = *this / rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator+=(const T & rhs) noexcept(is_same::value) +{ + *this = *this + rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator-=(const T & rhs) noexcept(is_same::value) +{ + *this = *this - rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator%=(const T & rhs) +{ + *this = *this % rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator&=(const T & rhs) noexcept +{ + *this = *this & rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator|=(const T & rhs) noexcept +{ + *this = *this | rhs; + return *this; +} + +template +template +constexpr wide_integer & wide_integer::operator^=(const T & rhs) noexcept +{ + *this = *this ^ rhs; + return *this; +} + +template +constexpr wide_integer & wide_integer::operator<<=(int n) +{ + *this = _impl::shift_left(*this, n); + return *this; +} + +template +constexpr wide_integer & wide_integer::operator>>=(int n) noexcept +{ + *this = _impl::shift_right(*this, n); + return *this; +} + +template +constexpr wide_integer & wide_integer::operator++() noexcept(is_same::value) +{ + *this = _impl::operator_plus(*this, 1); + return *this; +} + +template +constexpr wide_integer wide_integer::operator++(int) noexcept(is_same::value) +{ + auto tmp = *this; + *this = _impl::operator_plus(*this, 1); + return tmp; +} + +template +constexpr wide_integer & wide_integer::operator--() noexcept(is_same::value) +{ + *this = _impl::operator_minus(*this, 1); + return *this; +} + +template +constexpr wide_integer wide_integer::operator--(int) noexcept(is_same::value) +{ + auto tmp = *this; + *this = _impl::operator_minus(*this, 1); + return tmp; +} + +template +constexpr wide_integer::operator bool() const noexcept +{ + return !_impl::operator_eq(*this, 0); +} + +template +template +constexpr wide_integer::operator T() const noexcept +{ + static_assert(std::numeric_limits::is_integer, ""); + T res = 0; + for (size_t r_idx = 0; r_idx < _impl::arr_size && r_idx < sizeof(T); ++r_idx) + { + res |= (T(m_arr[_impl::little(r_idx)]) << (_impl::base_bits * r_idx)); + } + return res; +} + +template +constexpr wide_integer::operator long double() const noexcept +{ + if (_impl::operator_eq(*this, 0)) + return 0; + + wide_integer tmp = *this; + if (_impl::is_negative(*this)) + tmp = -tmp; + + long double res = 0; + for (size_t idx = 0; idx < _impl::arr_size; ++idx) + { + long double t = res; + res *= std::numeric_limits::max(); + res += t; + res += tmp.m_arr[_impl::big(idx)]; + } + + if (_impl::is_negative(*this)) + res = -res; + + return res; +} + +template +constexpr wide_integer::operator double() const noexcept +{ + return static_cast(*this); +} + +template +constexpr wide_integer::operator float() const noexcept +{ + return static_cast(*this); +} + +// Unary operators +template +constexpr wide_integer operator~(const wide_integer & lhs) noexcept +{ + return wide_integer::_impl::operator_unary_tilda(lhs); +} + +template +constexpr wide_integer operator-(const wide_integer & lhs) noexcept(is_same::value) +{ + return wide_integer::_impl::operator_unary_minus(lhs); +} + +template +constexpr wide_integer operator+(const wide_integer & lhs) noexcept(is_same::value) +{ + return lhs; +} + +// Binary operators +template +std::common_type_t, wide_integer> constexpr +operator*(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_star(lhs, rhs); +} + +template +std::common_type_t constexpr operator*(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) * CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator/(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_slash(lhs, rhs); +} +template +std::common_type_t constexpr operator/(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) / CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator+(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_plus(lhs, rhs); +} +template +std::common_type_t constexpr operator+(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) + CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator-(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_minus(lhs, rhs); +} +template +std::common_type_t constexpr operator-(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) - CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator%(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_percent(lhs, rhs); +} +template +std::common_type_t constexpr operator%(const Integral & lhs, const Integral2 & rhs) +{ + return CT(lhs) % CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator&(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_amp(lhs, rhs); +} +template +std::common_type_t constexpr operator&(const Integral & lhs, const Integral2 & rhs) +{ + return CT(lhs) & CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator|(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_pipe(lhs, rhs); +} +template +std::common_type_t constexpr operator|(const Integral & lhs, const Integral2 & rhs) +{ + return CT(lhs) | CT(rhs); +} + +template +std::common_type_t, wide_integer> constexpr +operator^(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_circumflex(lhs, rhs); +} +template +std::common_type_t constexpr operator^(const Integral & lhs, const Integral2 & rhs) +{ + return CT(lhs) ^ CT(rhs); +} + +template +constexpr wide_integer operator<<(const wide_integer & lhs, int n) noexcept +{ + return wide_integer::_impl::shift_left(lhs, n); +} +template +constexpr wide_integer operator>>(const wide_integer & lhs, int n) noexcept +{ + return wide_integer::_impl::shift_right(lhs, n); +} + +template +constexpr bool operator<(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_less(lhs, rhs); +} +template +constexpr bool operator<(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) < CT(rhs); +} + +template +constexpr bool operator>(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_more(lhs, rhs); +} +template +constexpr bool operator>(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) > CT(rhs); +} + +template +constexpr bool operator<=(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_less(lhs, rhs) + || std::common_type_t, wide_integer>::_impl::operator_eq(lhs, rhs); +} +template +constexpr bool operator<=(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) <= CT(rhs); +} + +template +constexpr bool operator>=(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_more(lhs, rhs) + || std::common_type_t, wide_integer>::_impl::operator_eq(lhs, rhs); +} +template +constexpr bool operator>=(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) >= CT(rhs); +} + +template +constexpr bool operator==(const wide_integer & lhs, const wide_integer & rhs) +{ + return std::common_type_t, wide_integer>::_impl::operator_eq(lhs, rhs); +} +template +constexpr bool operator==(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) == CT(rhs); +} + +template +constexpr bool operator!=(const wide_integer & lhs, const wide_integer & rhs) +{ + return !std::common_type_t, wide_integer>::_impl::operator_eq(lhs, rhs); +} +template +constexpr bool operator!=(const Arithmetic & lhs, const Arithmetic2 & rhs) +{ + return CT(lhs) != CT(rhs); +} + +template +inline std::string to_string(const wide_integer & n) +{ + std::string res; + if (wide_integer::_impl::operator_eq(n, 0U)) + return "0"; + + wide_integer t; + bool is_neg = wide_integer::_impl::is_negative(n); + if (is_neg) + t = wide_integer::_impl::operator_unary_minus(n); + else + t = n; + + while (!wide_integer::_impl::operator_eq(t, 0U)) + { + res.insert(res.begin(), '0' + char(wide_integer::_impl::operator_percent(t, 10U))); + t = wide_integer::_impl::operator_slash(t, 10U); + } + + if (is_neg) + res.insert(res.begin(), '-'); + return res; +} + +template +struct hash> +{ + std::size_t operator()(const wide_integer & lhs) const + { + static_assert(Bits % (sizeof(size_t) * 8) == 0); + + const auto * ptr = reinterpret_cast(lhs.m_arr); + unsigned count = Bits / (sizeof(size_t) * 8); + + size_t res = 0; + for (unsigned i = 0; i < count; ++i) + res ^= ptr[i]; + return hash()(res); + } +}; + +#undef CT +} diff --git a/contrib/capnproto-cmake/CMakeLists.txt b/contrib/capnproto-cmake/CMakeLists.txt index 8bdac0beec0..b655ad3e5d9 100644 --- a/contrib/capnproto-cmake/CMakeLists.txt +++ b/contrib/capnproto-cmake/CMakeLists.txt @@ -78,8 +78,9 @@ if (COMPILER_GCC) -Wno-deprecated-declarations -Wno-class-memaccess) elseif (COMPILER_CLANG) set (SUPPRESS_WARNINGS -Wno-non-virtual-dtor -Wno-sign-compare -Wno-strict-aliasing -Wno-deprecated-declarations) + set (CAPNP_PRIVATE_CXX_FLAGS -fno-char8_t) endif () -target_compile_options(kj PRIVATE ${SUPPRESS_WARNINGS}) -target_compile_options(capnp PRIVATE ${SUPPRESS_WARNINGS}) -target_compile_options(capnpc PRIVATE ${SUPPRESS_WARNINGS}) +target_compile_options(kj PRIVATE ${SUPPRESS_WARNINGS} ${CAPNP_PRIVATE_CXX_FLAGS}) +target_compile_options(capnp PRIVATE ${SUPPRESS_WARNINGS} ${CAPNP_PRIVATE_CXX_FLAGS}) +target_compile_options(capnpc PRIVATE ${SUPPRESS_WARNINGS} ${CAPNP_PRIVATE_CXX_FLAGS}) diff --git a/debian/clickhouse-server.init b/debian/clickhouse-server.init index b82c70bd6e0..f56164759bf 100755 --- a/debian/clickhouse-server.init +++ b/debian/clickhouse-server.init @@ -67,13 +67,6 @@ if uname -mpi | grep -q 'x86_64'; then fi -SUPPORTED_COMMANDS="{start|stop|status|restart|forcestop|forcerestart|reload|condstart|condstop|condrestart|condreload|initdb}" -is_supported_command() -{ - echo "$SUPPORTED_COMMANDS" | grep -E "(\{|\|)$1(\||})" &> /dev/null -} - - is_running() { pgrep --pidfile "$CLICKHOUSE_PIDFILE" $(echo "${PROGRAM}" | cut -c1-15) 1> /dev/null 2> /dev/null @@ -283,13 +276,12 @@ use_cron() fi return 0 } - +# returns false if cron disabled (with systemd) enable_cron() { use_cron && sed -i 's/^#*//' "$CLICKHOUSE_CRONFILE" } - - +# returns false if cron disabled (with systemd) disable_cron() { use_cron && sed -i 's/^#*/#/' "$CLICKHOUSE_CRONFILE" @@ -312,15 +304,14 @@ main() EXIT_STATUS=0 case "$1" in start) - start && enable_cron + service_or_func start && enable_cron ;; stop) - # disable_cron returns false if cron disabled (with systemd) - not checking return status disable_cron - stop + service_or_func stop ;; restart) - restart && enable_cron + service_or_func restart && enable_cron ;; forcestop) disable_cron @@ -330,7 +321,7 @@ main() forcerestart && enable_cron ;; reload) - restart + service_or_func restart ;; condstart) is_running || service_or_func start @@ -354,7 +345,7 @@ main() disable_cron ;; *) - echo "Usage: $0 $SUPPORTED_COMMANDS" + echo "Usage: $0 {start|stop|status|restart|forcestop|forcerestart|reload|condstart|condstop|condrestart|condreload|initdb}" exit 2 ;; esac diff --git a/debian/rules b/debian/rules index 5b271a8691f..ffe1f9e1228 100755 --- a/debian/rules +++ b/debian/rules @@ -18,7 +18,7 @@ ifeq ($(CCACHE_PREFIX),distcc) THREADS_COUNT=$(shell distcc -j) endif ifeq ($(THREADS_COUNT),) - THREADS_COUNT=$(shell nproc || grep -c ^processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 4) + THREADS_COUNT=$(shell echo $$(( $$(nproc || grep -c ^processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 8) / 2 )) ) endif DEB_BUILD_OPTIONS+=parallel=$(THREADS_COUNT) diff --git a/docker/builder/Dockerfile b/docker/builder/Dockerfile index b7dadc3ec6d..d4a121d13eb 100644 --- a/docker/builder/Dockerfile +++ b/docker/builder/Dockerfile @@ -6,7 +6,7 @@ RUN apt-get update \ && apt-get install ca-certificates lsb-release wget gnupg apt-transport-https \ --yes --no-install-recommends --verbose-versions \ && export LLVM_PUBKEY_HASH="bda960a8da687a275a2078d43c111d66b1c6a893a3275271beedf266c1ff4a0cdecb429c7a5cccf9f486ea7aa43fd27f" \ - && wget -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ + && wget -nv -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ && echo "${LLVM_PUBKEY_HASH} /tmp/llvm-snapshot.gpg.key" | sha384sum -c \ && apt-key add /tmp/llvm-snapshot.gpg.key \ && export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ diff --git a/docker/packager/binary/Dockerfile b/docker/packager/binary/Dockerfile index b8650b945e1..e1133f337a9 100644 --- a/docker/packager/binary/Dockerfile +++ b/docker/packager/binary/Dockerfile @@ -7,7 +7,7 @@ RUN apt-get update \ && apt-get install ca-certificates lsb-release wget gnupg apt-transport-https \ --yes --no-install-recommends --verbose-versions \ && export LLVM_PUBKEY_HASH="bda960a8da687a275a2078d43c111d66b1c6a893a3275271beedf266c1ff4a0cdecb429c7a5cccf9f486ea7aa43fd27f" \ - && wget -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ + && wget -nv -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ && echo "${LLVM_PUBKEY_HASH} /tmp/llvm-snapshot.gpg.key" | sha384sum -c \ && apt-key add /tmp/llvm-snapshot.gpg.key \ && export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ @@ -55,7 +55,6 @@ RUN apt-get update \ cmake \ gdb \ rename \ - wget \ build-essential \ --yes --no-install-recommends @@ -83,14 +82,14 @@ RUN git clone https://github.com/tpoechtrager/cctools-port.git \ && rm -rf cctools-port # Download toolchain for Darwin -RUN wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.14-beta4/MacOSX10.14.sdk.tar.xz +RUN wget -nv https://github.com/phracker/MacOSX-SDKs/releases/download/10.14-beta4/MacOSX10.14.sdk.tar.xz # Download toolchain for ARM # It contains all required headers and libraries. Note that it's named as "gcc" but actually we are using clang for cross compiling. -RUN wget "https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz?revision=2e88a73f-d233-4f96-b1f4-d8b36e9bb0b9&la=en" -O gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz +RUN wget -nv "https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz?revision=2e88a73f-d233-4f96-b1f4-d8b36e9bb0b9&la=en" -O gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz # Download toolchain for FreeBSD 11.3 -RUN wget https://clickhouse-datasets.s3.yandex.net/toolchains/toolchains/freebsd-11.3-toolchain.tar.xz +RUN wget -nv https://clickhouse-datasets.s3.yandex.net/toolchains/toolchains/freebsd-11.3-toolchain.tar.xz COPY build.sh / CMD ["/bin/bash", "/build.sh"] diff --git a/docker/packager/deb/Dockerfile b/docker/packager/deb/Dockerfile index 6d0fdca2310..87f4582f8e2 100644 --- a/docker/packager/deb/Dockerfile +++ b/docker/packager/deb/Dockerfile @@ -7,7 +7,7 @@ RUN apt-get update \ && apt-get install ca-certificates lsb-release wget gnupg apt-transport-https \ --yes --no-install-recommends --verbose-versions \ && export LLVM_PUBKEY_HASH="bda960a8da687a275a2078d43c111d66b1c6a893a3275271beedf266c1ff4a0cdecb429c7a5cccf9f486ea7aa43fd27f" \ - && wget -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ + && wget -nv -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ && echo "${LLVM_PUBKEY_HASH} /tmp/llvm-snapshot.gpg.key" | sha384sum -c \ && apt-key add /tmp/llvm-snapshot.gpg.key \ && export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ @@ -34,7 +34,7 @@ RUN curl -O https://clickhouse-builds.s3.yandex.net/utils/1/dpkg-deb \ ENV APACHE_PUBKEY_HASH="bba6987b63c63f710fd4ed476121c588bc3812e99659d27a855f8c4d312783ee66ad6adfce238765691b04d62fa3688f" RUN export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ - && wget -O /tmp/arrow-keyring.deb "https://apache.bintray.com/arrow/ubuntu/apache-arrow-archive-keyring-latest-${CODENAME}.deb" \ + && wget -nv -O /tmp/arrow-keyring.deb "https://apache.bintray.com/arrow/ubuntu/apache-arrow-archive-keyring-latest-${CODENAME}.deb" \ && echo "${APACHE_PUBKEY_HASH} /tmp/arrow-keyring.deb" | sha384sum -c \ && dpkg -i /tmp/arrow-keyring.deb diff --git a/docker/test/base/Dockerfile b/docker/test/base/Dockerfile index c9b0700ecfc..8117d2907bc 100644 --- a/docker/test/base/Dockerfile +++ b/docker/test/base/Dockerfile @@ -7,7 +7,7 @@ RUN apt-get update \ && apt-get install ca-certificates lsb-release wget gnupg apt-transport-https \ --yes --no-install-recommends --verbose-versions \ && export LLVM_PUBKEY_HASH="bda960a8da687a275a2078d43c111d66b1c6a893a3275271beedf266c1ff4a0cdecb429c7a5cccf9f486ea7aa43fd27f" \ - && wget -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ + && wget -nv -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ && echo "${LLVM_PUBKEY_HASH} /tmp/llvm-snapshot.gpg.key" | sha384sum -c \ && apt-key add /tmp/llvm-snapshot.gpg.key \ && export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ diff --git a/docker/test/codebrowser/Dockerfile b/docker/test/codebrowser/Dockerfile index f9d239ef8ef..cb3462cad0e 100644 --- a/docker/test/codebrowser/Dockerfile +++ b/docker/test/codebrowser/Dockerfile @@ -15,7 +15,7 @@ RUN apt-get --allow-unauthenticated update -y \ gpg-agent \ git -RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add - +RUN wget -nv -O - https://apt.kitware.com/keys/kitware-archive-latest.asc | sudo apt-key add - RUN sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' RUN sudo echo "deb [trusted=yes] http://apt.llvm.org/bionic/ llvm-toolchain-bionic-8 main" >> /etc/apt/sources.list diff --git a/docker/test/fasttest/Dockerfile b/docker/test/fasttest/Dockerfile index 49845d72f1d..9b4bb574f8f 100644 --- a/docker/test/fasttest/Dockerfile +++ b/docker/test/fasttest/Dockerfile @@ -7,7 +7,7 @@ RUN apt-get update \ && apt-get install ca-certificates lsb-release wget gnupg apt-transport-https \ --yes --no-install-recommends --verbose-versions \ && export LLVM_PUBKEY_HASH="bda960a8da687a275a2078d43c111d66b1c6a893a3275271beedf266c1ff4a0cdecb429c7a5cccf9f486ea7aa43fd27f" \ - && wget -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ + && wget -nv -O /tmp/llvm-snapshot.gpg.key https://apt.llvm.org/llvm-snapshot.gpg.key \ && echo "${LLVM_PUBKEY_HASH} /tmp/llvm-snapshot.gpg.key" | sha384sum -c \ && apt-key add /tmp/llvm-snapshot.gpg.key \ && export CODENAME="$(lsb_release --codename --short | tr 'A-Z' 'a-z')" \ @@ -61,7 +61,6 @@ RUN apt-get update \ software-properties-common \ tzdata \ unixodbc \ - wget \ --yes --no-install-recommends # This symlink required by gcc to find lld compiler @@ -70,7 +69,7 @@ RUN ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/ld.lld ARG odbc_driver_url="https://github.com/ClickHouse/clickhouse-odbc/releases/download/v1.1.4.20200302/clickhouse-odbc-1.1.4-Linux.tar.gz" RUN mkdir -p /tmp/clickhouse-odbc-tmp \ - && wget --quiet -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ + && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ && cp /tmp/clickhouse-odbc-tmp/lib64/*.so /usr/local/lib/ \ && odbcinst -i -d -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbcinst.ini.sample \ && odbcinst -i -s -l -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbc.ini.sample \ diff --git a/docker/test/fuzzer/query-fuzzer-tweaks-users.xml b/docker/test/fuzzer/query-fuzzer-tweaks-users.xml index 8d430aa5c54..356d3212932 100644 --- a/docker/test/fuzzer/query-fuzzer-tweaks-users.xml +++ b/docker/test/fuzzer/query-fuzzer-tweaks-users.xml @@ -2,6 +2,15 @@ 10 + + + + 10 + + diff --git a/docker/test/fuzzer/run-fuzzer.sh b/docker/test/fuzzer/run-fuzzer.sh index 5c1f2fdb586..3d70faca5e0 100755 --- a/docker/test/fuzzer/run-fuzzer.sh +++ b/docker/test/fuzzer/run-fuzzer.sh @@ -227,5 +227,4 @@ EOF ;& esac -exit $task_exit_code - +exit $task_exit_code \ No newline at end of file diff --git a/docker/test/integration/runner/Dockerfile b/docker/test/integration/runner/Dockerfile index 95ab516cdaa..bfbe8da816f 100644 --- a/docker/test/integration/runner/Dockerfile +++ b/docker/test/integration/runner/Dockerfile @@ -46,7 +46,7 @@ RUN set -eux; \ \ # this "case" statement is generated via "update.sh" \ - if ! wget -O docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz"; then \ + if ! wget -nv -O docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz"; then \ echo >&2 "error: failed to download 'docker-${DOCKER_VERSION}' from '${DOCKER_CHANNEL}' for '${x86_64}'"; \ exit 1; \ fi; \ diff --git a/docker/test/integration/runner/compose/docker_compose_mysql.yml b/docker/test/integration/runner/compose/docker_compose_mysql.yml index 2e3afce117d..2f09c2c01e3 100644 --- a/docker/test/integration/runner/compose/docker_compose_mysql.yml +++ b/docker/test/integration/runner/compose/docker_compose_mysql.yml @@ -7,3 +7,4 @@ services: MYSQL_ROOT_PASSWORD: clickhouse ports: - 3308:3306 + command: --server_id=100 --log-bin='mysql-bin-1.log' --default-time-zone='+3:00' --gtid-mode="ON" --enforce-gtid-consistency \ No newline at end of file diff --git a/docker/test/integration/runner/compose/docker_compose_mysql_5_7.yml b/docker/test/integration/runner/compose/docker_compose_mysql_5_7.yml deleted file mode 100644 index f42d2c6dd79..00000000000 --- a/docker/test/integration/runner/compose/docker_compose_mysql_5_7.yml +++ /dev/null @@ -1,10 +0,0 @@ -version: '2.3' -services: - mysql5_7: - image: mysql:5.7 - restart: always - environment: - MYSQL_ROOT_PASSWORD: clickhouse - ports: - - 33307:3306 - command: --server_id=100 --log-bin='mysql-bin-1.log' --default-time-zone='+3:00' --gtid-mode="ON" --enforce-gtid-consistency diff --git a/docker/test/integration/runner/compose/docker_compose_redis.yml b/docker/test/integration/runner/compose/docker_compose_redis.yml index 2dc79ed5910..2c9ace96d0c 100644 --- a/docker/test/integration/runner/compose/docker_compose_redis.yml +++ b/docker/test/integration/runner/compose/docker_compose_redis.yml @@ -5,3 +5,4 @@ services: restart: always ports: - 6380:6379 + command: redis-server --requirepass "clickhouse" diff --git a/docker/test/performance-comparison/compare.sh b/docker/test/performance-comparison/compare.sh index d3b9fc2214e..f54d0a022f6 100755 --- a/docker/test/performance-comparison/compare.sh +++ b/docker/test/performance-comparison/compare.sh @@ -536,7 +536,9 @@ create table queries engine File(TSVWithNamesAndTypes, 'report/queries.tsv') left join query_display_names on query_metric_stats.test = query_display_names.test and query_metric_stats.query_index = query_display_names.query_index - where metric_name = 'server_time' + -- 'server_time' is rounded down to ms, which might be bad for very short queries. + -- Use 'client_time' instead. + where metric_name = 'client_time' order by test, query_index, metric_name ; @@ -563,40 +565,54 @@ create table unstable_queries_report engine File(TSV, 'report/unstable-queries.t toDecimal64(stat_threshold, 3), unstable_fail, test, query_index, query_display_name from queries where unstable_show order by stat_threshold desc; -create table test_time_changes engine File(TSV, 'report/test-time-changes.tsv') as - select test, queries, average_time_change from ( - select test, count(*) queries, - sum(left) as left, sum(right) as right, - (right - left) / right average_time_change - from queries - group by test - order by abs(average_time_change) desc - ) - ; -create table unstable_tests engine File(TSV, 'report/unstable-tests.tsv') as - select test, sum(unstable_show) total_unstable, sum(changed_show) total_changed +create view test_speedup as + select + test, + exp2(avg(log2(left / right))) times_speedup, + count(*) queries, + unstable + changed bad, + sum(changed_show) changed, + sum(unstable_show) unstable from queries group by test - order by total_unstable + total_changed desc + order by times_speedup desc + ; + +create view total_speedup as + select + 'Total' test, + exp2(avg(log2(times_speedup))) times_speedup, + sum(queries) queries, + unstable + changed bad, + sum(changed) changed, + sum(unstable) unstable + from test_speedup ; create table test_perf_changes_report engine File(TSV, 'report/test-perf-changes.tsv') as - select test, - queries, - coalesce(total_unstable, 0) total_unstable, - coalesce(total_changed, 0) total_changed, - total_unstable + total_changed total_bad, - coalesce(toString(toDecimal64(average_time_change, 3)), '??') average_time_change_str - from test_time_changes - full join unstable_tests - using test - where (abs(average_time_change) > 0.05 and queries > 5) - or (total_bad > 0) - order by total_bad desc, average_time_change desc - settings join_use_nulls = 1 + with + (times_speedup >= 1 + ? '-' || toString(toDecimal64(times_speedup, 3)) || 'x' + : '+' || toString(toDecimal64(1 / times_speedup, 3)) || 'x') + as times_speedup_str + select test, times_speedup_str, queries, bad, changed, unstable + -- Not sure what's the precedence of UNION ALL vs WHERE & ORDER BY, hence all + -- the braces. + from ( + ( + select * from total_speedup + ) union all ( + select * from test_speedup + where + (times_speedup >= 1 ? times_speedup : (1 / times_speedup)) >= 1.005 + or bad + ) + ) + order by test = 'Total' desc ; + create view total_client_time_per_query as select * from file('analyze/client-times.tsv', TSV, 'test text, query_index int, client float, server float'); @@ -888,7 +904,10 @@ for log in *-err.log do test=$(basename "$log" "-err.log") { - grep -H -m2 -i '\(Exception\|Error\):[^:]' "$log" \ + # The second grep is a heuristic for error messages like + # "socket.timeout: timed out". + grep -h -m2 -i '\(Exception\|Error\):[^:]' "$log" \ + || grep -h -m2 -i '^[^ ]\+: ' "$log" \ || head -2 "$log" } | sed "s/^/$test\t/" >> run-errors.tsv ||: done diff --git a/docker/test/performance-comparison/report.py b/docker/test/performance-comparison/report.py index ecf9c7a45e5..1003a6d0e1a 100755 --- a/docker/test/performance-comparison/report.py +++ b/docker/test/performance-comparison/report.py @@ -168,12 +168,6 @@ def nextRowAnchor(): global table_anchor return f'{table_anchor}.{row_anchor + 1}' -def setRowAnchor(anchor_row_part): - global row_anchor - global table_anchor - row_anchor = anchor_row_part - return currentRowAnchor() - def advanceRowAnchor(): global row_anchor global table_anchor @@ -376,7 +370,7 @@ if args.report == 'main': columns = [ 'Old, s', # 0 'New, s', # 1 - 'Times speedup / slowdown', # 2 + 'Ratio of speedup (-) or slowdown (+)', # 2 'Relative difference (new − old) / old', # 3 'p < 0.001 threshold', # 4 # Failed # 5 @@ -453,7 +447,7 @@ if args.report == 'main': addSimpleTable('Skipped tests', ['Test', 'Reason'], skipped_tests_rows) addSimpleTable('Test performance changes', - ['Test', 'Queries', 'Unstable', 'Changed perf', 'Total not OK', 'Avg relative time diff'], + ['Test', 'Ratio of speedup (-) or slowdown (+)', 'Queries', 'Total not OK', 'Changed perf', 'Unstable'], tsvRows('report/test-perf-changes.tsv')) def add_test_times(): @@ -480,11 +474,12 @@ if args.report == 'main': total_runs = (nominal_runs + 1) * 2 # one prewarm run, two servers attrs = ['' for c in columns] for r in rows: + anchor = f'{currentTableAnchor()}.{r[0]}' if float(r[6]) > 1.5 * total_runs: # FIXME should be 15s max -- investigate parallel_insert slow_average_tests += 1 attrs[6] = f'style="background: {color_bad}"' - errors_explained.append([f'The test \'{r[0]}\' is too slow to run as a whole. Investigate whether the create and fill queries can be sped up']) + errors_explained.append([f'The test \'{r[0]}\' is too slow to run as a whole. Investigate whether the create and fill queries can be sped up']) else: attrs[6] = '' @@ -495,7 +490,7 @@ if args.report == 'main': else: attrs[5] = '' - text += tableRow(r, attrs) + text += tableRow(r, attrs, anchor) text += tableEnd() tables.append(text) @@ -652,7 +647,7 @@ elif args.report == 'all-queries': # Unstable #1 'Old, s', #2 'New, s', #3 - 'Times speedup / slowdown', #4 + 'Ratio of speedup (-) or slowdown (+)', #4 'Relative difference (new − old) / old', #5 'p < 0.001 threshold', #6 'Test', #7 diff --git a/docker/test/pvs/Dockerfile b/docker/test/pvs/Dockerfile index ebd9c105705..0aedb67e572 100644 --- a/docker/test/pvs/Dockerfile +++ b/docker/test/pvs/Dockerfile @@ -12,8 +12,8 @@ RUN apt-get update --yes \ strace \ --yes --no-install-recommends -#RUN wget -q -O - http://files.viva64.com/etc/pubkey.txt | sudo apt-key add - -#RUN sudo wget -O /etc/apt/sources.list.d/viva64.list http://files.viva64.com/etc/viva64.list +#RUN wget -nv -O - http://files.viva64.com/etc/pubkey.txt | sudo apt-key add - +#RUN sudo wget -nv -O /etc/apt/sources.list.d/viva64.list http://files.viva64.com/etc/viva64.list # #RUN apt-get --allow-unauthenticated update -y \ # && env DEBIAN_FRONTEND=noninteractive \ @@ -24,10 +24,10 @@ ENV PKG_VERSION="pvs-studio-latest" RUN set -x \ && export PUBKEY_HASHSUM="486a0694c7f92e96190bbfac01c3b5ac2cb7823981db510a28f744c99eabbbf17a7bcee53ca42dc6d84d4323c2742761" \ - && wget https://files.viva64.com/etc/pubkey.txt -O /tmp/pubkey.txt \ + && wget -nv https://files.viva64.com/etc/pubkey.txt -O /tmp/pubkey.txt \ && echo "${PUBKEY_HASHSUM} /tmp/pubkey.txt" | sha384sum -c \ && apt-key add /tmp/pubkey.txt \ - && wget "https://files.viva64.com/${PKG_VERSION}.deb" \ + && wget -nv "https://files.viva64.com/${PKG_VERSION}.deb" \ && { debsig-verify ${PKG_VERSION}.deb \ || echo "WARNING: Some file was just downloaded from the internet without any validation and we are installing it into the system"; } \ && dpkg -i "${PKG_VERSION}.deb" diff --git a/docker/test/stateful/run.sh b/docker/test/stateful/run.sh index 5be14970914..c3576acc0e4 100755 --- a/docker/test/stateful/run.sh +++ b/docker/test/stateful/run.sh @@ -29,17 +29,26 @@ if [[ -n "$USE_DATABASE_ATOMIC" ]] && [[ "$USE_DATABASE_ATOMIC" -eq 1 ]]; then ln -s /usr/share/clickhouse-test/config/database_atomic_usersd.xml /etc/clickhouse-server/users.d/ fi -echo "TSAN_OPTIONS='verbosity=1000 halt_on_error=1 history_size=7'" >> /etc/environment -echo "TSAN_SYMBOLIZER_PATH=/usr/lib/llvm-10/bin/llvm-symbolizer" >> /etc/environment -echo "UBSAN_OPTIONS='print_stacktrace=1'" >> /etc/environment -echo "ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-10/bin/llvm-symbolizer" >> /etc/environment -echo "UBSAN_SYMBOLIZER_PATH=/usr/lib/llvm-10/bin/llvm-symbolizer" >> /etc/environment -echo "LLVM_SYMBOLIZER_PATH=/usr/lib/llvm-10/bin/llvm-symbolizer" >> /etc/environment +function start() +{ + counter=0 + until clickhouse-client --query "SELECT 1" + do + if [ "$counter" -gt 120 ] + then + echo "Cannot start clickhouse-server" + cat /var/log/clickhouse-server/stdout.log + tail -n1000 /var/log/clickhouse-server/stderr.log + tail -n1000 /var/log/clickhouse-server/clickhouse-server.log + break + fi + timeout 120 service clickhouse-server start + sleep 0.5 + counter=$(($counter + 1)) + done +} -service zookeeper start -sleep 5 -service clickhouse-server start -sleep 5 +start /s3downloader --dataset-names $DATASETS chmod 777 -R /var/lib/clickhouse clickhouse-client --query "SHOW DATABASES" diff --git a/docker/test/stateful_with_coverage/run.sh b/docker/test/stateful_with_coverage/run.sh index 8928fc28f80..c2434b319b9 100755 --- a/docker/test/stateful_with_coverage/run.sh +++ b/docker/test/stateful_with_coverage/run.sh @@ -71,14 +71,26 @@ ln -s /usr/share/clickhouse-test/config/macros.xml /etc/clickhouse-server/config ln -s --backup=simple --suffix=_original.xml \ /usr/share/clickhouse-test/config/query_masking_rules.xml /etc/clickhouse-server/config.d/ +function start() +{ + counter=0 + until clickhouse-client --query "SELECT 1" + do + if [ "$counter" -gt 120 ] + then + echo "Cannot start clickhouse-server" + cat /var/log/clickhouse-server/stdout.log + tail -n1000 /var/log/clickhouse-server/stderr.log + tail -n1000 /var/log/clickhouse-server/clickhouse-server.log + break + fi + timeout 120 service clickhouse-server start + sleep 0.5 + counter=$(($counter + 1)) + done +} -service zookeeper start - -sleep 5 - -start_clickhouse - -sleep 5 +start if ! /s3downloader --dataset-names $DATASETS; then echo "Cannot download datatsets" diff --git a/docker/test/stateless/Dockerfile b/docker/test/stateless/Dockerfile index d3bc03a8f92..409a1b07bef 100644 --- a/docker/test/stateless/Dockerfile +++ b/docker/test/stateless/Dockerfile @@ -26,7 +26,7 @@ RUN apt-get update -y \ zookeeperd RUN mkdir -p /tmp/clickhouse-odbc-tmp \ - && wget --quiet -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ + && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ && cp /tmp/clickhouse-odbc-tmp/lib64/*.so /usr/local/lib/ \ && odbcinst -i -d -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbcinst.ini.sample \ && odbcinst -i -s -l -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbc.ini.sample \ diff --git a/docker/test/stateless_unbundled/Dockerfile b/docker/test/stateless_unbundled/Dockerfile index 7de29fede72..b05e46406da 100644 --- a/docker/test/stateless_unbundled/Dockerfile +++ b/docker/test/stateless_unbundled/Dockerfile @@ -71,7 +71,7 @@ RUN apt-get --allow-unauthenticated update -y \ zookeeperd RUN mkdir -p /tmp/clickhouse-odbc-tmp \ - && wget --quiet -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ + && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ && cp /tmp/clickhouse-odbc-tmp/lib64/*.so /usr/local/lib/ \ && odbcinst -i -d -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbcinst.ini.sample \ && odbcinst -i -s -l -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbc.ini.sample \ diff --git a/docker/test/stateless_with_coverage/Dockerfile b/docker/test/stateless_with_coverage/Dockerfile index f3539804852..77357d5142f 100644 --- a/docker/test/stateless_with_coverage/Dockerfile +++ b/docker/test/stateless_with_coverage/Dockerfile @@ -33,7 +33,7 @@ RUN apt-get update -y \ qemu-user-static RUN mkdir -p /tmp/clickhouse-odbc-tmp \ - && wget --quiet -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ + && wget -nv -O - ${odbc_driver_url} | tar --strip-components=1 -xz -C /tmp/clickhouse-odbc-tmp \ && cp /tmp/clickhouse-odbc-tmp/lib64/*.so /usr/local/lib/ \ && odbcinst -i -d -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbcinst.ini.sample \ && odbcinst -i -s -l -f /tmp/clickhouse-odbc-tmp/share/doc/clickhouse-odbc/config/odbc.ini.sample \ diff --git a/docker/test/testflows/runner/Dockerfile b/docker/test/testflows/runner/Dockerfile index 6b4ec12b80c..898552ade56 100644 --- a/docker/test/testflows/runner/Dockerfile +++ b/docker/test/testflows/runner/Dockerfile @@ -44,7 +44,7 @@ RUN set -eux; \ \ # this "case" statement is generated via "update.sh" \ - if ! wget -O docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz"; then \ + if ! wget -nv -O docker.tgz "https://download.docker.com/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz"; then \ echo >&2 "error: failed to download 'docker-${DOCKER_VERSION}' from '${DOCKER_CHANNEL}' for '${x86_64}'"; \ exit 1; \ fi; \ diff --git a/docs/en/engines/table-engines/integrations/rabbitmq.md b/docs/en/engines/table-engines/integrations/rabbitmq.md index 7d09c6f72a5..1bf1c1d3754 100644 --- a/docs/en/engines/table-engines/integrations/rabbitmq.md +++ b/docs/en/engines/table-engines/integrations/rabbitmq.md @@ -7,7 +7,7 @@ toc_title: RabbitMQ This engine allows integrating ClickHouse with [RabbitMQ](https://www.rabbitmq.com). -RabbitMQ lets you: +`RabbitMQ` lets you: - Publish or subscribe to data flows. - Process streams as they become available. @@ -44,8 +44,8 @@ Optional parameters: - `rabbitmq_routing_key_list` – A comma-separated list of routing keys. - `rabbitmq_row_delimiter` – Delimiter character, which ends the message. - `rabbitmq_num_consumers` – The number of consumers per table. Default: `1`. Specify more consumers if the throughput of one consumer is insufficient. -- `rabbitmq_num_queues` – The number of queues per consumer. Default: `1`. Specify more queues if the capacity of one queue per consumer is insufficient. Single queue can contain up to 50K messages at the same time. -- `rabbitmq_transactional_channel` – Wrap insert queries in transactions. Default: `0`. +- `rabbitmq_num_queues` – The number of queues per consumer. Default: `1`. Specify more queues if the capacity of one queue per consumer is insufficient. A single queue can contain up to 50K messages at the same time. +- `rabbitmq_transactional_channel` – Wrap `INSERT` queries in transactions. Default: `0`. Required configuration: @@ -72,7 +72,7 @@ Example: ## Description {#description} -`SELECT` is not particularly useful for reading messages (except for debugging), because each message can be read only once. It is more practical to create real-time threads using materialized views. To do this: +`SELECT` is not particularly useful for reading messages (except for debugging), because each message can be read only once. It is more practical to create real-time threads using [materialized views](../../../sql-reference/statements/create/view.md). To do this: 1. Use the engine to create a RabbitMQ consumer and consider it a data stream. 2. Create a table with the desired structure. @@ -86,13 +86,13 @@ There can be no more than one exchange per table. One exchange can be shared bet Exchange type options: -- `direct` - Routing is based on exact matching of keys. Example table key list: `key1,key2,key3,key4,key5`, message key can eqaul any of them. +- `direct` - Routing is based on the exact matching of keys. Example table key list: `key1,key2,key3,key4,key5`, message key can equal any of them. - `fanout` - Routing to all tables (where exchange name is the same) regardless of the keys. - `topic` - Routing is based on patterns with dot-separated keys. Examples: `*.logs`, `records.*.*.2020`, `*.2018,*.2019,*.2020`. - `headers` - Routing is based on `key=value` matches with a setting `x-match=all` or `x-match=any`. Example table key list: `x-match=all,format=logs,type=report,year=2020`. -- `consistent-hash` - Data is evenly distributed between all bound tables (where exchange name is the same). Note that this exchange type must be enabled with RabbitMQ plugin: `rabbitmq-plugins enable rabbitmq_consistent_hash_exchange`. +- `consistent-hash` - Data is evenly distributed between all bound tables (where the exchange name is the same). Note that this exchange type must be enabled with RabbitMQ plugin: `rabbitmq-plugins enable rabbitmq_consistent_hash_exchange`. -If exchange type is not specified, then default is `fanout` and routing keys for data publishing must be randomized in range `[1, num_consumers]` for every message/batch (or in range `[1, num_consumers * num_queues]` if `rabbitmq_num_queues` is set). This table configuration works quicker then any other, especially when `rabbitmq_num_consumers` and/or `rabbitmq_num_queues` parameters are set. +If exchange type is not specified, then default is `fanout` and routing keys for data publishing must be randomized in range `[1, num_consumers]` for every message/batch (or in range `[1, num_consumers * num_queues]` if `rabbitmq_num_queues` is set). This table configuration works quicker than any other, especially when `rabbitmq_num_consumers` and/or `rabbitmq_num_queues` parameters are set. If `rabbitmq_num_consumers` and/or `rabbitmq_num_queues` parameters are specified along with `rabbitmq_exchange_type`, then: diff --git a/docs/en/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/en/engines/table-engines/mergetree-family/replacingmergetree.md index 109ae6c4601..684e7e28112 100644 --- a/docs/en/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/en/engines/table-engines/mergetree-family/replacingmergetree.md @@ -31,7 +31,7 @@ For a description of request parameters, see [statement description](../../../sq **ReplacingMergeTree Parameters** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` or `DateTime64`. Optional parameter. +- `ver` — column with version. Type `UInt*`, `Date` or `DateTime`. Optional parameter. When merging, `ReplacingMergeTree` from all the rows with the same sorting key leaves only one: diff --git a/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md b/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md index a010a395c64..b23139b402b 100644 --- a/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md +++ b/docs/en/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md @@ -121,7 +121,7 @@ To find out why we need two rows for each change, see [Algorithm](#table_engines **Notes on Usage** -1. The program that writes the data should remember the state of an object in order to cancel it. The “cancel” string should be a copy of the “state” string with the opposite `Sign`. This increases the initial size of storage but allows to write the data quickly. +1. The program that writes the data should remember the state of an object to be able to cancel it. “Cancel” string should contain copies of the primary key fields and the version of the “state” string and the opposite `Sign`. It increases the initial size of storage but allows to write the data quickly. 2. Long growing arrays in columns reduce the efficiency of the engine due to the load for writing. The more straightforward the data, the better the efficiency. 3. `SELECT` results depend strongly on the consistency of the history of object changes. Be accurate when preparing data for inserting. You can get unpredictable results with inconsistent data, such as negative values for non-negative metrics like session depth. diff --git a/docs/en/interfaces/http.md b/docs/en/interfaces/http.md index a5e7ef22558..35c79b5ee02 100644 --- a/docs/en/interfaces/http.md +++ b/docs/en/interfaces/http.md @@ -36,7 +36,7 @@ Examples: $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 4995c04f712..76fcfa2a616 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -1290,6 +1290,47 @@ Possible values: Default value: 0. +## distributed\_group\_by\_no\_merge {#distributed-group-by-no-merge} + +Do not merge aggregation states from different servers for distributed query processing, you can use this in case it is for certain that there are different keys on different shards + +Possible values: + +- 0 — Disabled (final query processing is done on the initiator node). +- 1 - Do not merge aggregation states from different servers for distributed query processing (query completelly processed on the shard, initiator only proxy the data). +- 2 - Same as 1 but apply `ORDER BY` and `LIMIT` on the initiator (can be used for queries with `ORDER BY` and/or `LIMIT`). + +**Example** + +```sql +SELECT * +FROM remote('127.0.0.{2,3}', system.one) +GROUP BY dummy +LIMIT 1 +SETTINGS distributed_group_by_no_merge = 1 +FORMAT PrettyCompactMonoBlock + +┌─dummy─┐ +│ 0 │ +│ 0 │ +└───────┘ +``` + +```sql +SELECT * +FROM remote('127.0.0.{2,3}', system.one) +GROUP BY dummy +LIMIT 1 +SETTINGS distributed_group_by_no_merge = 2 +FORMAT PrettyCompactMonoBlock + +┌─dummy─┐ +│ 0 │ +└───────┘ +``` + +Default value: 0 + ## optimize\_skip\_unused\_shards {#optimize-skip-unused-shards} Enables or disables skipping of unused shards for [SELECT](../../sql-reference/statements/select/index.md) queries that have sharding key condition in `WHERE/PREWHERE` (assuming that the data is distributed by sharding key, otherwise does nothing). @@ -1337,6 +1378,40 @@ Possible values: Default value: 0 +## optimize\_distributed\_group\_by\_sharding\_key {#optimize-distributed-group-by-sharding-key} + +Optimize `GROUP BY sharding_key` queries, by avoiding costly aggregation on the initiator server (which will reduce memory usage for the query on the initiator server). + +The following types of queries are supported (and all combinations of them): + +- `SELECT DISTINCT [..., ]sharding_key[, ...] FROM dist` +- `SELECT ... FROM dist GROUP BY sharding_key[, ...]` +- `SELECT ... FROM dist GROUP BY sharding_key[, ...] ORDER BY x` +- `SELECT ... FROM dist GROUP BY sharding_key[, ...] LIMIT 1` +- `SELECT ... FROM dist GROUP BY sharding_key[, ...] LIMIT 1 BY x` + +The following types of queries are not supported (support for some of them may be added later): + +- `SELECT ... GROUP BY sharding_key[, ...] WITH TOTALS` +- `SELECT ... GROUP BY sharding_key[, ...] WITH ROLLUP` +- `SELECT ... GROUP BY sharding_key[, ...] WITH CUBE` +- `SELECT ... GROUP BY sharding_key[, ...] SETTINGS extremes=1` + +Possible values: + +- 0 — Disabled. +- 1 — Enabled. + +Default value: 0 + +See also: + +- [distributed\_group\_by\_no\_merge](#distributed-group-by-no-merge) +- [optimize\_skip\_unused\_shards](#optimize-skip-unused-shards) + +!!! note "Note" + Right now it requires `optimize_skip_unused_shards` (the reason behind this is that one day it may be enabled by default, and it will work correctly only if data was inserted via Distributed table, i.e. data is distributed according to sharding_key). + ## optimize\_throw\_if\_noop {#setting-optimize_throw_if_noop} Enables or disables throwing an exception if an [OPTIMIZE](../../sql-reference/statements/misc.md#misc_operations-optimize) query didn’t perform a merge. @@ -1894,9 +1969,9 @@ Locking timeout is used to protect from deadlocks while executing read/write ope Possible values: -- Positive integer. +- Positive integer (in seconds). - 0 — No locking timeout. -Default value: `120`. +Default value: `120` seconds. [Original article](https://clickhouse.tech/docs/en/operations/settings/settings/) diff --git a/docs/es/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/es/engines/table-engines/mergetree-family/replacingmergetree.md index cb3c6aea34b..a1e95c5b5f4 100644 --- a/docs/es/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/es/engines/table-engines/mergetree-family/replacingmergetree.md @@ -33,7 +33,7 @@ Para obtener una descripción de los parámetros de solicitud, consulte [descrip **ReplacingMergeTree Parámetros** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` o `DateTime64`. Parámetro opcional. +- `ver` — column with version. Type `UInt*`, `Date` o `DateTime`. Parámetro opcional. Al fusionar, `ReplacingMergeTree` de todas las filas con la misma clave primaria deja solo una: diff --git a/docs/es/interfaces/http.md b/docs/es/interfaces/http.md index abc5cf63188..ebce0ec7a51 100644 --- a/docs/es/interfaces/http.md +++ b/docs/es/interfaces/http.md @@ -38,7 +38,7 @@ Ejemplos: $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/fa/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/fa/engines/table-engines/mergetree-family/replacingmergetree.md index 4ece20461cb..0ace0e05afc 100644 --- a/docs/fa/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/fa/engines/table-engines/mergetree-family/replacingmergetree.md @@ -33,7 +33,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **پارامترهای جایگزین** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` یا `DateTime64`. پارامتر اختیاری. +- `ver` — column with version. Type `UInt*`, `Date` یا `DateTime`. پارامتر اختیاری. هنگام ادغام, `ReplacingMergeTree` از تمام ردیف ها با همان کلید اصلی تنها یک برگ دارد: diff --git a/docs/fa/interfaces/http.md b/docs/fa/interfaces/http.md index 774980cf8fb..9ce40c17e6f 100644 --- a/docs/fa/interfaces/http.md +++ b/docs/fa/interfaces/http.md @@ -38,7 +38,7 @@ Ok. $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/fr/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/fr/engines/table-engines/mergetree-family/replacingmergetree.md index 755249c1a38..ac3c0f3b021 100644 --- a/docs/fr/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/fr/engines/table-engines/mergetree-family/replacingmergetree.md @@ -33,7 +33,7 @@ Pour une description des paramètres de requête, voir [demande de description]( **ReplacingMergeTree Paramètres** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` ou `DateTime64`. Paramètre facultatif. +- `ver` — column with version. Type `UInt*`, `Date` ou `DateTime`. Paramètre facultatif. Lors de la fusion, `ReplacingMergeTree` de toutes les lignes avec la même clé primaire ne laisse qu'un: diff --git a/docs/fr/interfaces/http.md b/docs/fr/interfaces/http.md index 2de32747d4a..a414bba2c2f 100644 --- a/docs/fr/interfaces/http.md +++ b/docs/fr/interfaces/http.md @@ -38,7 +38,7 @@ Exemple: $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/ja/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/ja/engines/table-engines/mergetree-family/replacingmergetree.md index e2cce893e3a..c3df9559415 100644 --- a/docs/ja/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/ja/engines/table-engines/mergetree-family/replacingmergetree.md @@ -33,7 +33,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **ReplacingMergeTreeパラメータ** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` または `DateTime64`. 任意パラメータ。 +- `ver` — column with version. Type `UInt*`, `Date` または `DateTime`. 任意パラメータ。 マージ時, `ReplacingMergeTree` 同じ主キーを持つすべての行から、一つだけを残します: diff --git a/docs/ja/interfaces/http.md b/docs/ja/interfaces/http.md index c76b1ba0827..31f2b54af6d 100644 --- a/docs/ja/interfaces/http.md +++ b/docs/ja/interfaces/http.md @@ -38,7 +38,7 @@ GETメソッドを使用する場合, ‘readonly’ 設定されています。 $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/ja/sql-reference/data-types/date.md b/docs/ja/sql-reference/data-types/date.md index ff6e028e885..bcdc8f7224d 100644 --- a/docs/ja/sql-reference/data-types/date.md +++ b/docs/ja/sql-reference/data-types/date.md @@ -1,14 +1,11 @@ --- -machine_translated: true -machine_translated_rev: 72537a2d527c63c07aa5d2361a8829f3895cf2bd toc_priority: 47 toc_title: "\u65E5\u4ED8" --- # 日付 {#date} -デートだ 1970-01-01(符号なし)以降の日数として二バイト単位で格納されます。 Unixエポックの開始直後から、コンパイル段階で定数によって定義される上限しきい値までの値を格納できます（現在は2106年までですが、完全にサポート -最小値は1970-01-01として出力されます。 +日付型です。 1970-01-01 からの日数が2バイトの符号なし整数として格納されます。 UNIX時間の開始直後から、変換段階で定数として定義される上限しきい値までの値を格納できます（現在は2106年までですが、一年分を完全にサポートしているのは2105年までです）。 日付値は、タイムゾーンなしで格納されます。 diff --git a/docs/ru/engines/table-engines/integrations/rabbitmq.md b/docs/ru/engines/table-engines/integrations/rabbitmq.md new file mode 100644 index 00000000000..b6b239f0eee --- /dev/null +++ b/docs/ru/engines/table-engines/integrations/rabbitmq.md @@ -0,0 +1,122 @@ +--- +toc_priority: 6 +toc_title: RabbitMQ +--- + +# RabbitMQ {#rabbitmq-engine} + +Движок работает с [RabbitMQ](https://www.rabbitmq.com). + +`RabbitMQ` позволяет: + +- Публиковать/подписываться на потоки данных. +- Обрабатывать потоки по мере их появления. + +## Создание таблицы {#table_engine-rabbitmq-creating-a-table} + +``` sql +CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] +( + name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1], + name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2], + ... +) ENGINE = RabbitMQ SETTINGS + rabbitmq_host_port = 'host:port', + rabbitmq_exchange_name = 'exchange_name', + rabbitmq_format = 'data_format'[,] + [rabbitmq_exchange_type = 'exchange_type',] + [rabbitmq_routing_key_list = 'key1,key2,...',] + [rabbitmq_row_delimiter = 'delimiter_symbol',] + [rabbitmq_num_consumers = N,] + [rabbitmq_num_queues = N,] + [rabbitmq_transactional_channel = 0] +``` + +Обязательные параметры: + +- `rabbitmq_host_port` – адрес сервера (`хост:порт`). Например: `localhost:5672`. +- `rabbitmq_exchange_name` – имя точки обмена в RabbitMQ. +- `rabbitmq_format` – формат сообщения. Используется такое же обозначение, как и в функции `FORMAT` в SQL, например, `JSONEachRow`. Подробнее см. в разделе [Форматы входных и выходных данных](../../../interfaces/formats.md). + +Дополнительные параметры: + +- `rabbitmq_exchange_type` – тип точки обмена в RabbitMQ: `direct`, `fanout`, `topic`, `headers`, `consistent-hash`. По умолчанию: `fanout`. +- `rabbitmq_routing_key_list` – список ключей маршрутизации, через запятую. +- `rabbitmq_row_delimiter` – символ-разделитель, который завершает сообщение. +- `rabbitmq_num_consumers` – количество потребителей на таблицу. По умолчанию: `1`. Укажите больше потребителей, если пропускная способность одного потребителя недостаточна. +- `rabbitmq_num_queues` – количество очередей на потребителя. По умолчанию: `1`. Укажите больше потребителей, если пропускная способность одной очереди на потребителя недостаточна. Одна очередь поддерживает до 50 тысяч сообщений одновременно. +- `rabbitmq_transactional_channel` – обернутые запросы `INSERT` в транзакциях. По умолчанию: `0`. + +Требуемая конфигурация: + +Конфигурация сервера RabbitMQ добавляется с помощью конфигурационного файла ClickHouse. + +``` xml + + root + clickhouse + +``` + +Example: + +``` sql + CREATE TABLE queue ( + key UInt64, + value UInt64 + ) ENGINE = RabbitMQ SETTINGS rabbitmq_host_port = 'localhost:5672', + rabbitmq_exchange_name = 'exchange1', + rabbitmq_format = 'JSONEachRow', + rabbitmq_num_consumers = 5; +``` + +## Описание {#description} + +Запрос `SELECT` не очень полезен для чтения сообщений (за исключением отладки), поскольку каждое сообщение может быть прочитано только один раз. Практичнее создавать потоки реального времени с помощью [материализованных преставлений](../../../sql-reference/statements/create/view.md). Для этого: + +1. Создайте потребителя RabbitMQ с помощью движка и рассматривайте его как поток данных. +2. Создайте таблицу с необходимой структурой. +3. Создайте материализованное представление, которое преобразует данные от движка и помещает их в ранее созданную таблицу. + +Когда к движку присоединяется материализованное представление, оно начинает в фоновом режиме собирать данные. Это позволяет непрерывно получать сообщения от RabbitMQ и преобразовывать их в необходимый формат с помощью `SELECT`. +У одной таблицы RabbitMQ может быть неограниченное количество материализованных представлений. + +Данные передаются с помощью параметров `rabbitmq_exchange_type` и `rabbitmq_routing_key_list`. +Может быть не более одной точки обмена на таблицу. Одна точка обмена может использоваться несколькими таблицами: это позволяет выполнять маршрутизацию по нескольким таблицам одновременно. + +Параметры точек обмена: + +- `direct` - маршрутизация основана на точном совпадении ключей. Пример списка ключей: `key1,key2,key3,key4,key5`. Ключ сообщения может совпадать с одним из них. +- `fanout` - маршрутизация по всем таблицам, где имя точки обмена совпадает, независимо от ключей. +- `topic` - маршрутизация основана на правилах с ключами, разделенными точками. Например: `*.logs`, `records.*.*.2020`, `*.2018,*.2019,*.2020`. +- `headers` - маршрутизация основана на совпадении `key=value` с настройкой `x-match=all` или `x-match=any`. Пример списка ключей таблицы: `x-match=all,format=logs,type=report,year=2020`. +- `consistent-hash` - данные равномерно распределяются между всеми связанными таблицами, где имя точки обмена совпадает. Обратите внимание, что этот тип обмена должен быть включен с помощью плагина RabbitMQ: `rabbitmq-plugins enable rabbitmq_consistent_hash_exchange`. + +Если тип точки обмена не задан, по умолчанию используется `fanout`. В таком случае ключи маршрутизации для публикации данных должны быть рандомизированы в диапазоне `[1, num_consumers]` за каждое сообщение/пакет (или в диапазоне `[1, num_consumers * num_queues]`, если `rabbitmq_num_queues` задано). Эта конфигурация таблицы работает быстрее, чем любая другая, особенно когда заданы параметры `rabbitmq_num_consumers` и/или `rabbitmq_num_queues`. + +Если параметры`rabbitmq_num_consumers` и/или `rabbitmq_num_queues` заданы вместе с параметром `rabbitmq_exchange_type`: + +- плагин `rabbitmq-consistent-hash-exchange` должен быть включен. +- свойство `message_id` должно быть определено (уникальное для каждого сообщения/пакета). + +Пример: + +``` sql + CREATE TABLE queue ( + key UInt64, + value UInt64 + ) ENGINE = RabbitMQ SETTINGS rabbitmq_host_port = 'localhost:5672', + rabbitmq_exchange_name = 'exchange1', + rabbitmq_exchange_type = 'headers', + rabbitmq_routing_key_list = 'format=logs,type=report,year=2020', + rabbitmq_format = 'JSONEachRow', + rabbitmq_num_consumers = 5; + + CREATE TABLE daily (key UInt64, value UInt64) + ENGINE = MergeTree(); + + CREATE MATERIALIZED VIEW consumer TO daily + AS SELECT key, value FROM queue; + + SELECT key, value FROM daily ORDER BY key; +``` diff --git a/docs/ru/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/ru/engines/table-engines/mergetree-family/replacingmergetree.md index fefc3c65b38..4aa1eb556f3 100644 --- a/docs/ru/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/ru/engines/table-engines/mergetree-family/replacingmergetree.md @@ -25,7 +25,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **Параметры ReplacingMergeTree** -- `ver` — столбец с версией, тип `UInt*`, `Date`, `DateTime` или `DateTime64`. Необязательный параметр. +- `ver` — столбец с версией, тип `UInt*`, `Date` или `DateTime`. Необязательный параметр. При слиянии, из всех строк с одинаковым значением ключа сортировки `ReplacingMergeTree` оставляет только одну: diff --git a/docs/ru/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md b/docs/ru/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md index 5dc9589bef5..bf280eb52bc 100644 --- a/docs/ru/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md +++ b/docs/ru/engines/table-engines/mergetree-family/versionedcollapsingmergetree.md @@ -116,7 +116,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **Примечания по использованию** -1. Программа, которая записывает данные, должна помнить состояние объекта, чтобы иметь возможность отменить его. Строка отмены состояния должна быть копией предыдущей строки состояния с противоположным значением `Sign`. Это увеличивает начальный размер хранилища, но позволяет быстро записывать данные. +1. Программа, которая записывает данные, должна помнить состояние объекта, чтобы иметь возможность отменить его. Строка отмены состояния должна содержать копии полей первичного ключа и копию версии строки состояния и противоположное значение `Sign`. Это увеличивает начальный размер хранилища, но позволяет быстро записывать данные. 2. Длинные растущие массивы в столбцах снижают эффективность работы движка за счёт нагрузки на запись. Чем проще данные, тем выше эффективность. 3. `SELECT` результаты сильно зависят от согласованности истории изменений объекта. Будьте точны при подготовке данных для вставки. Вы можете получить непредсказуемые результаты с несогласованными данными, такими как отрицательные значения для неотрицательных метрик, таких как глубина сеанса. diff --git a/docs/ru/interfaces/http.md b/docs/ru/interfaces/http.md index afd4d083365..b1cc4c79b25 100644 --- a/docs/ru/interfaces/http.md +++ b/docs/ru/interfaces/http.md @@ -31,7 +31,7 @@ Ok. $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 9a487b6c166..2c6e0f05fb5 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -401,13 +401,91 @@ INSERT INTO test VALUES (lower('Hello')), (lower('world')), (lower('INSERT')), ( Устанавливает тип поведения [JOIN](../../sql-reference/statements/select/join.md). При объединении таблиц могут появиться пустые ячейки. ClickHouse заполняет их по-разному в зависимости от настроек. -Возможные значения +Возможные значения: - 0 — пустые ячейки заполняются значением по умолчанию соответствующего типа поля. - 1 — `JOIN` ведёт себя как в стандартном SQL. Тип соответствующего поля преобразуется в [Nullable](../../sql-reference/data-types/nullable.md#data_type-nullable), а пустые ячейки заполняются значениями [NULL](../../sql-reference/syntax.md). +## partial_merge_join_optimizations {#partial_merge_join_optimizations} + +Отключает все оптимизации для запросов [JOIN](../../sql-reference/statements/select/join.md) с частичным MergeJoin алгоритмом. + +По умолчанию оптимизации включены, что может привести к неправильным результатам. Если вы видите подозрительные результаты в своих запросах, отключите оптимизацию с помощью этого параметра. В различных версиях сервера ClickHouse, оптимизация может отличаться. + +Возможные значения: + +- 0 — Оптимизация отключена. +- 1 — Оптимизация включена. + +Значение по умолчанию: 1. + +## partial_merge_join_rows_in_right_blocks {#partial_merge_join_rows_in_right_blocks} + +Устанавливает предельные размеры блоков данных «правого» соединения, для запросов [JOIN](../../sql-reference/statements/select/join.md) с частичным MergeJoin алгоритмом. + +Сервер ClickHouse: + +1. Разделяет данные правого соединения на блоки с заданным числом строк. +2. Индексирует для каждого блока минимальное и максимальное значение. +3. Выгружает подготовленные блоки на диск, если это возможно. + +Возможные значения: + +- Положительное целое число. Рекомендуемый диапазон значений [1000, 100000]. + +Значение по умолчанию: 65536. + +## join_on_disk_max_files_to_merge {#join_on_disk_max_files_to_merge} + +Устанавливет количество файлов, разрешенных для параллельной сортировки, при выполнении операций MergeJoin на диске. + +Чем больше значение параметра, тем больше оперативной памяти используется и тем меньше используется диск (I/O). + +Возможные значения: + +- Положительное целое число, больше 2. + +Значение по умолчанию: 64. + +## temporary_files_codec {#temporary_files_codec} + +Устанавливает метод сжатия для временных файлов на диске, используемых при сортировки и объединения. + +Возможные значения: + +- LZ4 — применять сжатие, используя алгоритм [LZ4](https://ru.wikipedia.org/wiki/LZ4) +- NONE — не применять сжатие. + +Значение по умолчанию: LZ4. + +## any_join_distinct_right_table_keys {#any_join_distinct_right_table_keys} + +Включает устаревшее поведение сервера ClickHouse при выполнении операций `ANY INNER|LEFT JOIN`. + +!!! note "Внимание" + Используйте этот параметр только в целях обратной совместимости, если ваши варианты использования требуют устаревшего поведения `JOIN`. + +Когда включено устаревшее поведение: + +- Результаты операций "t1 ANY LEFT JOIN t2" и "t2 ANY RIGHT JOIN t1" не равны, поскольку ClickHouse использует логику с сопоставлением ключей таблицы "многие к одному слева направо". +- Результаты операций `ANY INNER JOIN` содержат все строки из левой таблицы, аналогично операции `SEMI LEFT JOIN`. + +Когда устаревшее поведение отключено: + +- Результаты операций `t1 ANY LEFT JOIN t2` и `t2 ANY RIGHT JOIN t1` равно, потому что ClickHouse использует логику сопоставления ключей один-ко-многим в операциях `ANY RIGHT JOIN`. +- Результаты операций `ANY INNER JOIN` содержат по одной строке на ключ из левой и правой таблиц. + +Возможные значения: + +- 0 — Устаревшее поведение отключено. +- 1 — Устаревшее поведение включено. + Значение по умолчанию: 0. +См. также: + +- [JOIN strictness](../../sql-reference/statements/select/join.md#select-join-strictness) + ## max\_block\_size {#setting-max_block_size} Данные в ClickHouse обрабатываются по блокам (наборам кусочков столбцов). Внутренние циклы обработки для одного блока достаточно эффективны, но есть заметные издержки на каждый блок. Настройка `max_block_size` — это рекомендация, какой размер блока (в количестве строк) загружать из таблиц. Размер блока не должен быть слишком маленьким, чтобы затраты на каждый блок были заметны, но не слишком велики, чтобы запрос с LIMIT, который завершается после первого блока, обрабатывался быстро. Цель состоит в том, чтобы не использовалось слишком много оперативки при вынимании большого количества столбцов в несколько потоков; чтобы оставалась хоть какая-нибудь кэш-локальность. @@ -1678,4 +1756,17 @@ SELECT idx, i FROM null_in WHERE i IN (1, NULL) SETTINGS transform_null_in = 1; - [Секции и настройки запроса CREATE TABLE](../../engines/table-engines/mergetree-family/mergetree.md#mergetree-query-clauses) (настройка `merge_with_ttl_timeout`) - [Table TTL](../../engines/table-engines/mergetree-family/mergetree.md#mergetree-table-ttl) +## lock_acquire_timeout {#lock_acquire_timeout} + +Устанавливает, сколько секунд сервер ожидает возможности выполнить блокировку таблицы. + +Таймаут устанавливается для защиты от взаимоблокировки при выполнении операций чтения или записи. Если время ожидания истекло, а блокировку выполнить не удалось, сервер возвращает исключение с кодом `DEADLOCK_AVOIDED` и сообщением "Locking attempt timed out! Possible deadlock avoided. Client should retry." ("Время ожидания блокировки истекло! Возможная взаимоблокировка предотвращена. Повторите запрос."). + +Возможные значения: + +- Положительное целое число (в секундах). +- 0 — таймаут не устанавливается. + +Значение по умолчанию: `120` секунд. + [Оригинальная статья](https://clickhouse.tech/docs/ru/operations/settings/settings/) diff --git a/docs/ru/operations/utilities/clickhouse-copier.md b/docs/ru/operations/utilities/clickhouse-copier.md index b05db93b28b..b43f5ccaf7a 100644 --- a/docs/ru/operations/utilities/clickhouse-copier.md +++ b/docs/ru/operations/utilities/clickhouse-copier.md @@ -24,7 +24,7 @@ Утилиту следует запускать вручную следующим образом: ``` bash -$ clickhouse-copier copier --daemon --config zookeeper.xml --task-path /task/path --base-dir /path/to/dir +$ clickhouse-copier --daemon --config zookeeper.xml --task-path /task/path --base-dir /path/to/dir ``` Параметры запуска: diff --git a/docs/ru/sql-reference/statements/select/join.md b/docs/ru/sql-reference/statements/select/join.md index 2a5bcff0cbb..800f07a7c66 100644 --- a/docs/ru/sql-reference/statements/select/join.md +++ b/docs/ru/sql-reference/statements/select/join.md @@ -36,7 +36,9 @@ FROM !!! note "Примечание" Значение строгости по умолчанию может быть переопределено с помощью настройки [join\_default\_strictness](../../../operations/settings/settings.md#settings-join_default_strictness). - + +Поведение сервера ClickHouse для операций `ANY JOIN` зависит от параметра [any_join_distinct_right_table_keys](../../../operations/settings/settings.md#any_join_distinct_right_table_keys). + ### Использование ASOF JOIN {#asof-join-usage} `ASOF JOIN` применим в том случае, когда необходимо объединять записи, которые не имеют точного совпадения. diff --git a/docs/tools/build.py b/docs/tools/build.py index ac675897fca..120af33c8fb 100755 --- a/docs/tools/build.py +++ b/docs/tools/build.py @@ -180,12 +180,13 @@ def build(args): if not args.skip_website: website.build_website(args) - test.test_templates(args.website_dir) + if not args.skip_test_templates: + test.test_templates(args.website_dir) - build_docs(args) - - from github import build_releases - build_releases(args, build_docs) + if not args.skip_docs: + build_docs(args) + from github import build_releases + build_releases(args, build_docs) if not args.skip_blog: blog.build_blog(args) @@ -220,6 +221,8 @@ if __name__ == '__main__': arg_parser.add_argument('--skip-website', action='store_true') arg_parser.add_argument('--skip-blog', action='store_true') arg_parser.add_argument('--skip-git-log', action='store_true') + arg_parser.add_argument('--skip-docs', action='store_true') + arg_parser.add_argument('--skip-test-templates', action='store_true') arg_parser.add_argument('--test-only', action='store_true') arg_parser.add_argument('--minify', action='store_true') arg_parser.add_argument('--htmlproofer', action='store_true') diff --git a/docs/tr/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/tr/engines/table-engines/mergetree-family/replacingmergetree.md index f586b97cb2f..a24c84e9a16 100644 --- a/docs/tr/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/tr/engines/table-engines/mergetree-family/replacingmergetree.md @@ -33,7 +33,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **ReplacingMergeTree Parametreleri** -- `ver` — column with version. Type `UInt*`, `Date`, `DateTime` veya `DateTime64`. İsteğe bağlı parametre. +- `ver` — column with version. Type `UInt*`, `Date` veya `DateTime`. İsteğe bağlı parametre. Birleş whenirken, `ReplacingMergeTree` aynı birincil anahtara sahip tüm satırlardan sadece bir tane bırakır: diff --git a/docs/tr/interfaces/http.md b/docs/tr/interfaces/http.md index 2b92dd0ed9b..49d20ef6655 100644 --- a/docs/tr/interfaces/http.md +++ b/docs/tr/interfaces/http.md @@ -38,7 +38,7 @@ GET yöntemini kullanırken, ‘readonly’ ayar .lanmıştır. Başka bir deyi $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ echo -ne 'GET /?query=SELECT%201 HTTP/1.0\r\n\r\n' | nc localhost 8123 diff --git a/docs/zh/engines/table-engines/mergetree-family/mergetree.md b/docs/zh/engines/table-engines/mergetree-family/mergetree.md index e92621c12df..0b886547229 100644 --- a/docs/zh/engines/table-engines/mergetree-family/mergetree.md +++ b/docs/zh/engines/table-engines/mergetree-family/mergetree.md @@ -2,44 +2,47 @@ Clickhouse 中最强大的表引擎当属 `MergeTree` （合并树）引擎及该系列（`*MergeTree`）中的其他引擎。 -`MergeTree` 引擎系列的基本理念如下。当你有巨量数据要插入到表中，你要高效地一批批写入数据片段，并希望这些数据片段在后台按照一定规则合并。相比在插入时不断修改（重写）数据进存储，这种策略会高效很多。 +`MergeTree` 系列的引擎被设计用于插入极大量的数据到一张表当中。数据可以以数据片段的形式一个接着一个的快速写入，数据片段在后台按照一定的规则进行合并。相比在插入时不断修改（重写）已存储的数据，这种策略会高效很多。 主要特点: - 存储的数据按主键排序。 - 这让你可以创建一个用于快速检索数据的小稀疏索引。 + 这使得你能够创建一个小型的稀疏索引来加快数据检索。 -- 允许使用分区，如果指定了 [分区键](custom-partitioning-key.md) 的话。 +- 支持数据分区，如果指定了 [分区键](custom-partitioning-key.md) 的话。 在相同数据集和相同结果集的情况下 ClickHouse 中某些带分区的操作会比普通操作更快。查询中指定了分区键时 ClickHouse 会自动截取分区数据。这也有效增加了查询性能。 - 支持数据副本。 - `ReplicatedMergeTree` 系列的表便是用于此。更多信息，请参阅 [数据副本](replication.md) 一节。 + `ReplicatedMergeTree` 系列的表提供了数据副本功能。更多信息，请参阅 [数据副本](replication.md) 一节。 - 支持数据采样。 需要的话，你可以给表设置一个采样方法。 -!!! 注意 "注意" +!!! note "注意" [合并](../special/merge.md#merge) 引擎并不属于 `*MergeTree` 系列。 ## 建表 {#table_engine-mergetree-creating-a-table} - CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] - ( - name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1], - name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2], - ... - INDEX index_name1 expr1 TYPE type1(...) GRANULARITY value1, - INDEX index_name2 expr2 TYPE type2(...) GRANULARITY value2 - ) ENGINE = MergeTree() - [PARTITION BY expr] - [ORDER BY expr] - [PRIMARY KEY expr] - [SAMPLE BY expr] - [SETTINGS name=value, ...] +``` sql +CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] +( + name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1] [TTL expr1], + name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2] [TTL expr2], + ... + INDEX index_name1 expr1 TYPE type1(...) GRANULARITY value1, + INDEX index_name2 expr2 TYPE type2(...) GRANULARITY value2 +) ENGINE = MergeTree() +ORDER BY expr +[PARTITION BY expr] +[PRIMARY KEY expr] +[SAMPLE BY expr] +[TTL expr [DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'], ...] +[SETTINGS name=value, ...] +``` 对于以上参数的描述，可参考 [CREATE 语句 的描述](../../../engines/table-engines/mergetree-family/mergetree.md) 。 @@ -62,7 +65,7 @@ Clickhouse 中最强大的表引擎当属 `MergeTree` （合并树）引擎及 要按月分区，可以使用表达式 `toYYYYMM(date_column)` ，这里的 `date_column` 是一个 [Date](../../../engines/table-engines/mergetree-family/mergetree.md) 类型的列。分区名的格式会是 `"YYYYMM"` 。 -- `PRIMARY KEY` - 主键，如果要设成 [跟排序键不相同](#xuan-ze-gen-pai-xu-jian-bu-yi-yang-zhu-jian)，可选。 +- `PRIMARY KEY` - 主键，如果要 [选择与排序键不同的主键](#choosing-a-primary-key-that-differs-from-the-sorting-key)，可选。 默认情况下主键跟排序键（由 `ORDER BY` 子句指定）相同。 因此，大部分情况下不需要再专门指定一个 `PRIMARY KEY` 子句。 @@ -72,17 +75,19 @@ Clickhouse 中最强大的表引擎当属 `MergeTree` （合并树）引擎及 如果要用抽样表达式，主键中必须包含这个表达式。例如： `SAMPLE BY intHash32(UserID) ORDER BY (CounterID, EventDate, intHash32(UserID))` 。 -- TTL 指定行存储的持续时间并定义 PART 在硬盘和卷上的移动逻辑的规则列表，可选。 +- TTL 指定行存储的持续时间并定义数据片段在硬盘和卷上的移动逻辑的规则列表，可选。 表达式中必须存在至少一个 `Date` 或 `DateTime` 类型的列，比如： `TTL date + INTERVAl 1 DAY` - 规则的类型 `DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'`指定了当满足条件（到达当前时间）时所要执行的动作：移除过期的行，还是将 PART （如果PART中的所有行都满足表达式的话）移动到指定的磁盘（`TO DISK 'xxx'`) 或 卷（`TO VOLUME 'xxx'`）。默认的规则是移除（`DELETE`）。可以在列表中指定多个规则，但最多只能有一个`DELETE`的规则。 + 规则的类型 `DELETE|TO DISK 'xxx'|TO VOLUME 'xxx'`指定了当满足条件（到达指定时间）时所要执行的动作：移除过期的行，还是将数据片段（如果数据片段中的所有行都满足表达式的话）移动到指定的磁盘（`TO DISK 'xxx'`) 或 卷（`TO VOLUME 'xxx'`）。默认的规则是移除（`DELETE`）。可以在列表中指定多个规则，但最多只能有一个`DELETE`的规则。 + + 更多细节，请查看 [表和列的 TTL](#table_engine-mergetree-ttl) -- `SETTINGS` — 影响 `MergeTree` 性能的额外参数： +- `SETTINGS` — 控制 `MergeTree` 行为的额外参数： - - `index_granularity` — 索引粒度。索引中相邻的『标记』间的数据行数。默认值，8192 。参考[Data Storage](#mergetree-data-storage)。 + - `index_granularity` — 索引粒度。索引中相邻的『标记』间的数据行数。默认值，8192 。参考[数据存储](#mergetree-data-storage)。 - `index_granularity_bytes` — 索引粒度，以字节为单位，默认值: 10Mb。如果想要仅按数据行数限制索引粒度, 请设置为0(不建议)。 - `enable_mixed_granularity_parts` — 是否启用通过 `index_granularity_bytes` 控制索引粒度的大小。在19.11版本之前, 只有 `index_granularity` 配置能够用于限制索引粒度的大小。当从具有很大的行（几十上百兆字节）的表中查询数据时候，`index_granularity_bytes` 配置能够提升ClickHouse的性能。如果你的表里有很大的行，可以开启这项配置来提升`SELECT` 查询的性能。 - `use_minimalistic_part_header_in_zookeeper` — 是否在 ZooKeeper 中启用最小的数据片段头 。如果设置了 `use_minimalistic_part_header_in_zookeeper=1` ，ZooKeeper 会存储更少的数据。更多信息参考『服务配置参数』这章中的 [设置描述](../../../operations/server-configuration-parameters/settings.md#server-settings-use_minimalistic_part_header_in_zookeeper) 。 @@ -90,18 +95,21 @@ Clickhouse 中最强大的表引擎当属 `MergeTree` （合并树）引擎及 - `merge_with_ttl_timeout` — TTL合并频率的最小间隔时间，单位：秒。默认值: 86400 (1 天)。 - `write_final_mark` — 是否启用在数据片段尾部写入最终索引标记。默认值: 1（不建议更改）。 - - `storage_policy` — 存储策略。 参见 [使用多个区块装置进行数据存储](#table_engine-mergetree-multiple-volumes). - - `min_bytes_for_wide_part`,`min_rows_for_wide_part` 在数据分段中可以使用`Wide`格式进行存储的最小字节数/行数。你可以不设置、只设置一个，或全都设置。参考：[Data Storage](#mergetree-data-storage) + - `merge_max_block_size` — 在块中进行合并操作时的最大行数限制。默认值：8192 + - `storage_policy` — 存储策略。 参见 [使用具有多个块的设备进行数据存储](#table_engine-mergetree-multiple-volumes). + - `min_bytes_for_wide_part`,`min_rows_for_wide_part` 在数据片段中可以使用`Wide`格式进行存储的最小字节数/行数。你可以不设置、只设置一个，或全都设置。参考：[数据存储](#mergetree-data-storage) **示例配置** - ENGINE MergeTree() PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID)) SAMPLE BY intHash32(UserID) SETTINGS index_granularity=8192 +``` sql +ENGINE MergeTree() PARTITION BY toYYYYMM(EventDate) ORDER BY (CounterID, EventDate, intHash32(UserID)) SAMPLE BY intHash32(UserID) SETTINGS index_granularity=8192 +``` -示例中，我们设为按月分区。 +在这个例子中，我们设置了按月进行分区。 -同时我们设置了一个按用户ID哈希的抽样表达式。这让你可以有该表中每个 `CounterID` 和 `EventDate` 下面的数据的伪随机分布。如果你在查询时指定了 [SAMPLE](../../../engines/table-engines/mergetree-family/mergetree.md#select-sample-clause) 子句。 ClickHouse会返回对于用户子集的一个均匀的伪随机数据采样。 +同时我们设置了一个按用户 ID 哈希的抽样表达式。这使得你可以对该表中每个 `CounterID` 和 `EventDate` 的数据伪随机分布。如果你在查询时指定了 [SAMPLE](../../../engines/table-engines/mergetree-family/mergetree.md#select-sample-clause) 子句。 ClickHouse会返回对于用户子集的一个均匀的伪随机数据采样。 -`index_granularity` 可省略，默认值为 8192 。 +`index_granularity` 可省略因为 8192 是默认设置 。
@@ -133,15 +141,20 @@ Clickhouse 中最强大的表引擎当属 `MergeTree` （合并树）引擎及 ## 数据存储 {#mergetree-data-storage} -表由按主键排序的数据 *片段* 组成。 +表由按主键排序的数据片段（DATA PART）组成。 -当数据被插入到表中时，会分成数据片段并按主键的字典序排序。例如，主键是 `(CounterID, Date)` 时，片段中数据按 `CounterID` 排序，具有相同 `CounterID` 的部分按 `Date` 排序。 +当数据被插入到表中时，会创建多个数据片段并按主键的字典序排序。例如，主键是 `(CounterID, Date)` 时，片段中数据首先按 `CounterID` 排序，具有相同 `CounterID` 的部分按 `Date` 排序。 -不同分区的数据会被分成不同的片段，ClickHouse 在后台合并数据片段以便更高效存储。不会合并来自不同分区的数据片段。这个合并机制并不保证相同主键的所有行都会合并到同一个数据片段中。 +不同分区的数据会被分成不同的片段，ClickHouse 在后台合并数据片段以便更高效存储。不同分区的数据片段不会进行合并。合并机制并不保证具有相同主键的行全都合并到同一个数据片段中。 -ClickHouse 会为每个数据片段创建一个索引文件，索引文件包含每个索引行（『标记』）的主键值。索引行号定义为 `n * index_granularity` 。最大的 `n` 等于总行数除以 `index_granularity` 的值的整数部分。对于每列，跟主键相同的索引行处也会写入『标记』。这些『标记』让你可以直接找到数据所在的列。 +数据片段可以以 `Wide` 或 `Compact` 格式存储。在 `Wide` 格式下，每一列都会在文件系统中存储为单独的文件，在 `Compact` 格式下所有列都存储在一个文件中。`Compact` 格式可以提高插入量少插入频率频繁时的性能。 -你可以只用一单一大表并不断地一块块往里面加入数据 – `MergeTree` 引擎的就是为了这样的场景。 +数据存储格式由 `min_bytes_for_wide_part` 和 `min_rows_for_wide_part` 表引擎参数控制。如果数据片段中的字节数或行数少于相应的设置值，数据片段会以 `Compact` 格式存储，否则会以 `Wide` 格式存储。 + +每个数据片段被逻辑的分割成颗粒（granules）。颗粒是 ClickHouse 中进行数据查询时的最小不可分割数据集。ClickHouse 不会对行或值进行拆分，所以每个颗粒总是包含整数个行。每个颗粒的第一行通过该行的主键值进行标记， +ClickHouse 会为每个数据片段创建一个索引文件来存储这些标记。对于每列，无论它是否包含在主键当中，ClickHouse 都会存储类似标记。这些标记让你可以在列文件中直接找到数据。 + +颗粒的大小通过表引擎参数 `index_granularity` 和 `index_granularity_bytes` 控制。取决于行的大小，颗粒的行数的在 `[1, index_granularity]` 范围中。如果单行的大小超过了 `index_granularity_bytes` 设置的值，那么一个颗粒的大小会超过 `index_granularity_bytes`。在这种情况下，颗粒的大小等于该行的大小。 ## 主键和索引在查询中的表现 {#primary-keys-and-indexes-in-queries} @@ -162,56 +175,53 @@ ClickHouse 会为每个数据片段创建一个索引文件，索引文件包含 上面例子可以看出使用索引通常会比全表描述要高效。 -稀疏索引会引起额外的数据读取。当读取主键单个区间范围的数据时，每个数据块中最多会多读 `index_granularity * 2` 行额外的数据。大部分情况下，当 `index_granularity = 8192` 时，ClickHouse的性能并不会降级。 +稀疏索引会引起额外的数据读取。当读取主键单个区间范围的数据时，每个数据块中最多会多读 `index_granularity * 2` 行额外的数据。 -稀疏索引让你能操作有巨量行的表。因为这些索引是常驻内存（RAM）的。 +稀疏索引使得你可以处理极大量的行，因为大多数情况下，这些索引常驻与内存（RAM）中。 -ClickHouse 不要求主键惟一。所以，你可以插入多条具有相同主键的行。 +ClickHouse 不要求主键惟一，所以你可以插入多条具有相同主键的行。 ### 主键的选择 {#zhu-jian-de-xuan-ze} -主键中列的数量并没有明确的限制。依据数据结构，你应该让主键包含多些或少些列。这样可以： +主键中列的数量并没有明确的限制。依据数据结构，你可以在主键包含多些或少些列。这样可以： - 改善索引的性能。 - 如果当前主键是 `(a, b)` ，然后加入另一个 `c` 列，满足下面条件时，则可以改善性能： - - 有带有 `c` 列条件的查询。 - - 很长的数据范围（ `index_granularity` 的数倍）里 `(a, b)` 都是相同的值，并且这种的情况很普遍。换言之，就是加入另一列后，可以让你的查询略过很长的数据范围。 + 如果当前主键是 `(a, b)` ，在下列情况下添加另一个 `c` 列会提升性能： + + - 查询会使用 `c` 列作为条件 + - 很长的数据范围（ `index_granularity` 的数倍）里 `(a, b)` 都是相同的值，并且这样的情况很普遍。换言之，就是加入另一列后，可以让你的查询略过很长的数据范围。 - 改善数据压缩。 - ClickHouse 以主键排序片段数据，所以，数据的一致性越高，压缩越好。 + ClickHouse 以主键排序片段数据，所以，数据的一致性越高，压缩越好。 -- [折叠树](collapsingmergetree.md#table_engine-collapsingmergetree) 和 [SummingMergeTree](summingmergetree.md) 引擎里，数据合并时，会有额外的处理逻辑。 +- 在[CollapsingMergeTree](collapsingmergetree.md#table_engine-collapsingmergetree) 和 [SummingMergeTree](summingmergetree.md) 引擎里进行数据合并时会提供额外的处理逻辑。 - 在这种情况下，指定一个跟主键不同的 *排序键* 也是有意义的。 + 在这种情况下，指定与主键不同的 *排序键* 也是有意义的。 长的主键会对插入性能和内存消耗有负面影响，但主键中额外的列并不影响 `SELECT` 查询的性能。 -### 选择跟排序键不一样主键 {#xuan-ze-gen-pai-xu-jian-bu-yi-yang-zhu-jian} +可以使用 `ORDER BY tuple()` 语法创建没有主键的表。在这种情况下 ClickHouse 根据数据插入的顺序存储。如果在使用 `INSERT ... SELECT` 时希望保持数据的排序，请设置 [max\_insert\_threads = 1](../../../operations/settings/settings.md#settings-max-insert-threads)。 -指定一个跟排序键（用于排序数据片段中行的表达式） -不一样的主键（用于计算写到索引文件的每个标记值的表达式）是可以的。 -这种情况下，主键表达式元组必须是排序键表达式元组的一个前缀。 +想要根据初始顺序进行数据查询，使用 [单线程查询](../../../operations/settings/settings.md#settings-max_threads) -当使用 [SummingMergeTree](summingmergetree.md) 和 -[AggregatingMergeTree](aggregatingmergetree.md) 引擎时，这个特性非常有用。 -通常，使用这类引擎时，表里列分两种：*维度* 和 *度量* 。 -典型的查询是在 `GROUP BY` 并过虑维度的情况下统计度量列的值。 -像 SummingMergeTree 和 AggregatingMergeTree ，用相同的排序键值统计行时， -通常会加上所有的维度。结果就是，这键的表达式会是一长串的列组成， -并且这组列还会因为新加维度必须频繁更新。 +### 选择与排序键不同主键 {#choosing-a-primary-key-that-differs-from-the-sorting-key} -这种情况下，主键中仅预留少量列保证高效范围扫描， -剩下的维度列放到排序键元组里。这样是合理的。 +指定一个跟排序键不一样的主键是可以的，此时排序键用于在数据片段中进行排序，主键用于在索引文件中进行标记的写入。这种情况下，主键表达式元组必须是排序键表达式元组的前缀。 -[排序键的修改](../../../engines/table-engines/mergetree-family/mergetree.md) 是轻量级的操作，因为一个新列同时被加入到表里和排序键后时，已存在的数据片段并不需要修改。由于旧的排序键是新排序键的前缀，并且刚刚添加的列中没有数据，因此在表修改时的数据对于新旧的排序键来说都是有序的。 +当使用 [SummingMergeTree](summingmergetree.md) 和 [AggregatingMergeTree](aggregatingmergetree.md) 引擎时，这个特性非常有用。通常在使用这类引擎时，表里的列分两种：*维度* 和 *度量* 。典型的查询会通过任意的 `GROUP BY` 对度量列进行聚合并通过维度列进行过滤。由于 SummingMergeTree 和 AggregatingMergeTree 会对排序键相同的行进行聚合，所以把所有的维度放进排序键是很自然的做法。但这将导致排序键中包含大量的列，并且排序键会伴随着新添加的维度不断的更新。 -### 索引和分区在查询中的应用 {#suo-yin-he-fen-qu-zai-cha-xun-zhong-de-ying-yong} +在这种情况下合理的做法是，只保留少量的列在主键当中用于提升扫描效率，将维度列添加到排序键中。 -对于 `SELECT` 查询，ClickHouse 分析是否可以使用索引。如果 `WHERE/PREWHERE` 子句具有下面这些表达式（作为谓词链接一子项或整个）则可以使用索引：基于主键或分区键的列或表达式的部分的等式或比较运算表达式；基于主键或分区键的列或表达式的固定前缀的 `IN` 或 `LIKE` 表达式；基于主键或分区键的列的某些函数；基于主键或分区键的表达式的逻辑表达式。 +对排序键进行 [ALTER](../../../sql-reference/statements/alter.md) 是轻量级的操作，因为当一个新列同时被加入到表里和排序键里时，已存在的数据片段并不需要修改。由于旧的排序键是新排序键的前缀，并且新添加的列中没有数据，因此在表修改时的数据对于新旧的排序键来说都是有序的。 -因此，在索引键的一个或多个区间上快速地跑查询都是可能的。下面例子中，指定标签；指定标签和日期范围；指定标签和日期；指定多个标签和日期范围等运行查询，都会非常快。 +### 索引和分区在查询中的应用 {#use-of-indexes-and-partitions-in-queries} + +对于 `SELECT` 查询，ClickHouse 分析是否可以使用索引。如果 `WHERE/PREWHERE` 子句具有下面这些表达式（作为谓词链接一子项或整个）则可以使用索引：包含一个表示与主键/分区键中的部分字段或全部字段相等/不等的比较表达式；基于主键/分区键的字段上的 `IN` 或 固定前缀的`LIKE` 表达式；基于主键/分区键的字段上的某些函数；基于主键/分区键的表达式的逻辑表达式。 + + +因此，在索引键的一个或多个区间上快速地执行查询都是可能的。下面例子中，指定标签；指定标签和日期范围；指定标签和日期；指定多个标签和日期范围等执行查询，都会非常快。 当引擎配置如下时： @@ -237,11 +247,18 @@ SELECT count() FROM table WHERE CounterID = 34 OR URL LIKE '%upyachka%' 要检查 ClickHouse 执行一个查询时能否使用索引，可设置 [force\_index\_by\_date](../../../operations/settings/settings.md#settings-force_index_by_date) 和 [force\_primary\_key](../../../operations/settings/settings.md) 。 -按月分区的分区键是只能读取包含适当范围日期的数据块。这种情况下，数据块会包含很多天（最多整月）的数据。在块中，数据按主键排序，主键第一列可能不包含日期。因此，仅使用日期而没有带主键前缀条件的查询将会导致读取超过这个日期范围。 +按月分区的分区键是只能读取包含适当范围日期的数据块。这种情况下，数据块会包含很多天（最多整月）的数据。在块中，数据按主键排序，主键第一列可能不包含日期。因此，仅使用日期而没有带主键前几个字段作为条件的查询将会导致需要读取超过这个指定日期以外的数据。 -### 跳数索引（分段汇总索引，实验性的） {#tiao-shu-suo-yin-fen-duan-hui-zong-suo-yin-shi-yan-xing-de} +### 部分单调主键的使用 -需要设置 `allow_experimental_data_skipping_indices` 为 1 才能使用此索引。（执行 `SET allow_experimental_data_skipping_indices = 1`）。 +考虑这样的场景，比如一个月中的几天。它们在一个月的范围内形成一个[单调序列](https://zh.wikipedia.org/wiki/单调函数) ，但如果扩展到更大的时间范围它们就不再单调了。这就是一个部分单调序列。如果用户使用部分单调的主键创建表，ClickHouse同样会创建一个稀疏索引。当用户从这类表中查询数据时，ClickHouse 会对查询条件进行分析。如果用户希望获取两个索引标记之间的数据并且这两个标记在一个月以内，ClickHouse 可以在这种特殊情况下使用到索引，因为它可以计算出查询参数与索引标记之间的距离。 + +如果查询参数范围内的主键不是单调序列，那么 ClickHouse 无法使用索引。在这种情况下，ClickHouse 会进行全表扫描。 + +ClickHouse 在任何主键代表一个部分单调序列的情况下都会使用这个逻辑。 + + +### 跳数索引 {#tiao-shu-suo-yin-fen-duan-hui-zong-suo-yin-shi-yan-xing-de} 此索引在 `CREATE` 语句的列部分里定义。 @@ -249,12 +266,14 @@ SELECT count() FROM table WHERE CounterID = 34 OR URL LIKE '%upyachka%' INDEX index_name expr TYPE type(...) GRANULARITY granularity_value ``` -`*MergeTree` 系列的表都能指定跳数索引。 +`*MergeTree` 系列的表可以指定跳数索引。 这些索引是由数据块按粒度分割后的每部分在指定表达式上汇总信息 `granularity_value` 组成（粒度大小用表引擎里 `index_granularity` 的指定）。 这些汇总信息有助于用 `where` 语句跳过大片不满足的数据，从而减少 `SELECT` 查询从磁盘读取的数据量， -示例 +这些索引会在数据块上聚合指定表达式的信息，这些信息以 granularity_value 指定的粒度组成 （粒度的大小通过在表引擎中定义 index_granularity 定义）。这些汇总信息有助于跳过大片不满足 `where` 条件的数据，从而减少 `SELECT` 查询从磁盘读取的数据量。 + +**示例** ``` sql CREATE TABLE table_name @@ -282,19 +301,27 @@ SELECT count() FROM table WHERE u64 * i32 == 10 AND u64 * length(s) >= 1234 存储指定表达式的极值（如果表达式是 `tuple` ，则存储 `tuple` 中每个元素的极值），这些信息用于跳过数据块，类似主键。 - `set(max_rows)` - 存储指定表达式的惟一值（不超过 `max_rows` 个，`max_rows=0` 则表示『无限制』）。这些信息可用于检查 `WHERE` 表达式是否满足某个数据块。 + 存储指定表达式的不重复值（不超过 `max_rows` 个，`max_rows=0` 则表示『无限制』）。这些信息可用于检查 数据块是否满足 `WHERE` 条件。 - `ngrambf_v1(n, size_of_bloom_filter_in_bytes, number_of_hash_functions, random_seed)` - 存储包含数据块中所有 n 元短语的 [布隆过滤器](https://en.wikipedia.org/wiki/Bloom_filter) 。只可用在字符串上。 + 存储一个包含数据块中所有 n元短语（ngram） 的 [布隆过滤器](https://en.wikipedia.org/wiki/Bloom_filter) 。只可用在字符串上。 可用于优化 `equals` ， `like` 和 `in` 表达式的性能。 `n` – 短语长度。 - `size_of_bloom_filter_in_bytes` – 布隆过滤器大小，单位字节。（因为压缩得好，可以指定比较大的值，如256或512）。 - `number_of_hash_functions` – 布隆过滤器中使用的 hash 函数的个数。 - `random_seed` – hash 函数的随机种子。 + `size_of_bloom_filter_in_bytes` – 布隆过滤器大小，单位字节。（因为压缩得好，可以指定比较大的值，如 256 或 512）。 + `number_of_hash_functions` – 布隆过滤器中使用的哈希函数的个数。 + `random_seed` – 哈希函数的随机种子。 - `tokenbf_v1(size_of_bloom_filter_in_bytes, number_of_hash_functions, random_seed)` - 跟 `ngrambf_v1` 类似，不同于 ngrams 存储字符串指定长度的所有片段。它只存储被非字母数据字符分割的片段。 + 跟 `ngrambf_v1` 类似，不同于 ngrams 存储字符串指定长度的所有片段。它只存储被非字母数字字符分割的片段。 +- `bloom_filter(bloom_filter([false_positive])` – 为指定的列存储布隆过滤器 + + 可选的参数 false_positive 用来指定从布隆过滤器收到错误响应的几率。取值范围是 (0,1)，默认值：0.025 + + 支持的数据类型：`Int*`, `UInt*`, `Float*`, `Enum`, `Date`, `DateTime`, `String`, `FixedString`, `Array`, `LowCardinality`, `Nullable`。 + + 以下函数会用到这个索引： [equals](../../../sql-reference/functions/comparison-functions.md), [notEquals](../../../sql-reference/functions/comparison-functions.md), [in](../../../sql-reference/functions/in-functions.md), [notIn](../../../sql-reference/functions/in-functions.md), [has](../../../sql-reference/functions/array-functions.md) + ``` sql @@ -303,17 +330,62 @@ INDEX sample_index2 (u64 * length(str), i32 + f64 * 100, date, str) TYPE set(100 INDEX sample_index3 (lower(str), str) TYPE ngrambf_v1(3, 256, 2, 0) GRANULARITY 4 ``` -## 并发数据访问 {#bing-fa-shu-ju-fang-wen} +#### 函数支持 {#functions-support} + +WHERE 子句中的条件包含对列的函数调用，如果列是索引的一部分，ClickHouse 会在执行函数时尝试使用索引。不同的函数对索引的支持是不同的。 + +`set` 索引会对所有函数生效，其他索引对函数的生效情况见下表 + +| 函数 (操作符) / 索引 | primary key | minmax | ngrambf\_v1 | tokenbf\_v1 | bloom\_filter | +|------------------------------------------------------------------------------------------------------------|-------------|--------|-------------|-------------|---------------| +| [equals (=, ==)](../../../sql-reference/functions/comparison-functions.md#function-equals) | ✔ | ✔ | ✔ | ✔ | ✔ | +| [notEquals(!=, \<\>)](../../../sql-reference/functions/comparison-functions.md#function-notequals) | ✔ | ✔ | ✔ | ✔ | ✔ | +| [like](../../../sql-reference/functions/string-search-functions.md#function-like) | ✔ | ✔ | ✔ | ✔ | ✔ | +| [notLike](../../../sql-reference/functions/string-search-functions.md#function-notlike) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [startsWith](../../../sql-reference/functions/string-functions.md#startswith) | ✔ | ✔ | ✔ | ✔ | ✗ | +| [endsWith](../../../sql-reference/functions/string-functions.md#endswith) | ✗ | ✗ | ✔ | ✔ | ✗ | +| [multiSearchAny](../../../sql-reference/functions/string-search-functions.md#function-multisearchany) | ✗ | ✗ | ✔ | ✗ | ✗ | +| [in](../../../sql-reference/functions/in-functions.md#in-functions) | ✔ | ✔ | ✔ | ✔ | ✔ | +| [notIn](../../../sql-reference/functions/in-functions.md#in-functions) | ✔ | ✔ | ✔ | ✔ | ✔ | +| [less (\<)](../../../sql-reference/functions/comparison-functions.md#function-less) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [greater (\>)](../../../sql-reference/functions/comparison-functions.md#function-greater) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [lessOrEquals (\<=)](../../../sql-reference/functions/comparison-functions.md#function-lessorequals) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [greaterOrEquals (\>=)](../../../sql-reference/functions/comparison-functions.md#function-greaterorequals) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [empty](../../../sql-reference/functions/array-functions.md#function-empty) | ✔ | ✔ | ✗ | ✗ | ✗ | +| [notEmpty](../../../sql-reference/functions/array-functions.md#function-notempty) | ✔ | ✔ | ✗ | ✗ | ✗ | +| hasToken | ✗ | ✗ | ✗ | ✔ | ✗ | + +常量参数小于 ngram 大小的函数不能使用 `ngrambf_v1` 进行查询优化。 + +!!! note "注意" +布隆过滤器可能会包含不符合条件的匹配，所以 `ngrambf_v1`, `tokenbf_v1` 和 `bloom_filter` 索引不能用于负向的函数，例如： + +- 可以用来优化的场景 + - `s LIKE '%test%'` + - `NOT s NOT LIKE '%test%'` + - `s = 1` + - `NOT s != 1` + - `startsWith(s, 'test')` +- 不能用来优化的场景 + - `NOT s LIKE '%test%'` + - `s NOT LIKE '%test%'` + - `NOT s = 1` + - `s != 1` + - `NOT startsWith(s, 'test')` + +## 并发数据访问 {#concurrent-data-access} 应对表的并发访问，我们使用多版本机制。换言之，当同时读和更新表时，数据从当前查询到的一组片段中读取。没有冗长的的锁。插入不会阻碍读取。 对表的读操作是自动并行的。 -## 列和表的TTL {#table_engine-mergetree-ttl} +## 列和表的 TTL {#table_engine-mergetree-ttl} -TTL可以设置值的生命周期，它既可以为整张表设置，也可以为每个列字段单独设置。如果`TTL`同时作用于表和字段，ClickHouse会使用先到期的那个。 +TTL 可以设置值的生命周期，它既可以为整张表设置，也可以为每个列字段单独设置。表级别的 TTL 还会指定数据在磁盘和卷上自动转移的逻辑。 -被设置TTL的表，必须拥有[日期](../../../engines/table-engines/mergetree-family/mergetree.md) 或 [日期时间](../../../engines/table-engines/mergetree-family/mergetree.md) 类型的字段。要定义数据的生命周期，需要在这个日期字段上使用操作符，例如: +TTL 表达式的计算结果必须是 [日期](../../../engines/table-engines/mergetree-family/mergetree.md) 或 [日期时间](../../../engines/table-engines/mergetree-family/mergetree.md) 类型的字段。 + +示例： ``` sql TTL time_column @@ -327,15 +399,15 @@ TTL date_time + INTERVAL 1 MONTH TTL date_time + INTERVAL 15 HOUR ``` -### 列字段 TTL {#mergetree-column-ttl} +### 列 TTL {#mergetree-column-ttl} -当列字段中的值过期时, ClickHouse会将它们替换成数据类型的默认值。如果分区内，某一列的所有值均已过期，则ClickHouse会从文件系统中删除这个分区目录下的列文件。 +当列中的值过期时, ClickHouse会将它们替换成该列数据类型的默认值。如果数据片段中列的所有值均已过期，则ClickHouse 会从文件系统中的数据片段中此列。 `TTL`子句不能被用于主键字段。 -示例说明: +示例: -创建一张包含 `TTL` 的表 +创建表时指定 `TTL` ``` sql CREATE TABLE example_table @@ -368,11 +440,21 @@ ALTER TABLE example_table ### 表 TTL {#mergetree-table-ttl} -当表内的数据过期时, ClickHouse会删除所有对应的行。 +表可以设置一个用于移除过期行的表达式，以及多个用于在磁盘或卷上自动转移数据片段的表达式。当表中的行过期时，ClickHouse 会删除所有对应的行。对于数据片段的转移特性，必须所有的行都满足转移条件。 -举例说明: +``` sql +TTL expr [DELETE|TO DISK 'aaa'|TO VOLUME 'bbb'], ... +``` -创建一张包含 `TTL` 的表 +TTL 规则的类型紧跟在每个 TTL 表达式后面，它会影响满足表达式时（到达指定时间时）应当执行的操作： + +- `DELETE` - 删除过期的行（默认操作）; +- `TO DISK 'aaa'` - 将数据片段移动到磁盘 `aaa`; +- `TO VOLUME 'bbb'` - 将数据片段移动到卷 `bbb`. + +示例: + +创建时指定 TTL ``` sql CREATE TABLE example_table @@ -383,7 +465,9 @@ CREATE TABLE example_table ENGINE = MergeTree PARTITION BY toYYYYMM(d) ORDER BY d -TTL d + INTERVAL 1 MONTH; +TTL d + INTERVAL 1 MONTH [DELETE], + d + INTERVAL 1 WEEK TO VOLUME 'aaa', + d + INTERVAL 2 WEEK TO DISK 'bbb'; ``` 修改表的 `TTL` @@ -395,14 +479,179 @@ ALTER TABLE example_table **删除数据** -当ClickHouse合并数据分区时, 会删除TTL过期的数据。 +ClickHouse 在数据片段合并时会删除掉过期的数据。 -当ClickHouse发现数据过期时, 它将会执行一个计划外的合并。要控制这类合并的频率, 你可以设置 `merge_with_ttl_timeout`。如果该值被设置的太低, 它将导致执行许多的计划外合并，这可能会消耗大量资源。 +当ClickHouse发现数据过期时, 它将会执行一个计划外的合并。要控制这类合并的频率, 你可以设置 `merge_with_ttl_timeout`。如果该值被设置的太低, 它将引发大量计划外的合并，这可能会消耗大量资源。 -如果在合并的时候执行`SELECT` 查询, 则可能会得到过期的数据。为了避免这种情况，可以在`SELECT`之前使用 [OPTIMIZE](../../../engines/table-engines/mergetree-family/mergetree.md#misc_operations-optimize) 查询。 +如果在合并的过程中执行 `SELECT` 查询, 则可能会得到过期的数据。为了避免这种情况，可以在 `SELECT` 之前使用 [OPTIMIZE](../../../engines/table-engines/mergetree-family/mergetree.md#misc_operations-optimize) 查询。 -## 使用多个块设备进行数据存储 {#table_engine-mergetree-multiple-volumes} +## 使用具有多个块的设备进行数据存储 {#table_engine-mergetree-multiple-volumes} + +### 介绍 {#introduction} + +MergeTree 系列表引擎可以将数据存储在多块设备上。这对某些可以潜在被划分为“冷”“热”的表来说是很有用的。近期数据被定期的查询但只需要很小的空间。相反，详尽的历史数据很少被用到。如果有多块磁盘可用，那么“热”的数据可以放置在快速的磁盘上（比如 NVMe 固态硬盘或内存），“冷”的数据可以放在相对较慢的磁盘上（比如机械硬盘）。 + +数据片段是 `MergeTree` 引擎表的最小可移动单元。属于同一个数据片段的数据被存储在同一块磁盘上。数据片段会在后台自动的在磁盘间移动，也可以通过 [ALTER](../../../sql-reference/statements/alter.md#alter_move-partition) 查询来移动。 + +### 术语 {#terms} + +- 磁盘 — 挂载到文件系统的块设备 +- 默认磁盘 — 在服务器设置中通过 [path](../../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-path) 参数指定的数据存储 +- 卷 — 磁盘的等效有序集合 （类似于 [JBOD](https://en.wikipedia.org/wiki/Non-RAID_drive_architectures)） +- 存储策略 — 卷的集合及他们之间的数据移动规则 ### 配置 {#table_engine-mergetree-multiple-volumes_configure} -[来源文章](https://clickhouse.tech/docs/en/operations/table_engines/mergetree/) +磁盘、卷和存储策略应当在主文件 `config.xml` 或 `config.d` 目录中的独立文件中的 `` 标签内定义。 + +配置结构： + +``` xml + + + + /mnt/fast_ssd/clickhouse/ + + + /mnt/hdd1/clickhouse/ + 10485760 + + + /mnt/hdd2/clickhouse/ + 10485760 + + + ... + + + ... + +``` + +标签： + +- `` — 磁盘名，名称必须与其他磁盘不同. +- `path` — 服务器将用来存储数据 (`data` 和 `shadow` 目录) 的路径, 应当以 ‘/’ 结尾. +- `keep_free_space_bytes` — 需要保留的剩余磁盘空间. + +磁盘定义的顺序无关紧要。 + +存储策略配置： + +``` xml + + ... + + + + + disk_name_from_disks_configuration + 1073741824 + + + + + + + 0.2 + + + + + + + + ... + +``` + +标签： + +- `policy_name_N` — 策略名称，不能重复。 +- `volume_name_N` — 卷名称，不能重复。 +- `disk` — 卷中的磁盘。 +- `max_data_part_size_bytes` — 任意卷上的磁盘可以存储的数据片段的最大大小。 +- `move_factor` — 当可用空间少于这个因子时，数据将自动的向下一个卷（如果有的话）移动 (默认值为 0.1)。 + +配置示例： + +``` xml + + ... + + + + + disk1 + disk2 + + + + + + + + fast_ssd + 1073741824 + + + disk1 + + + 0.2 + + + ... + +``` + +在给出的例子中， `hdd_in_order` 策略实现了 [循环制](https://zh.wikipedia.org/wiki/循环制) 方法。因此这个策略只定义了一个卷（`single`），数据片段会以循环的顺序全部存储到它的磁盘上。当有多个类似的磁盘挂载到系统上，但没有配置 RAID 时，这种策略非常有用。请注意一个每个独立的磁盘驱动都并不可靠，你可能需要用 3 或更大的复制因此来补偿它。 + +如果在系统中有不同类型的磁盘可用，可以使用 `moving_from_ssd_to_hdd`。`hot` 卷由 SSD 磁盘（`fast_ssd`）组成，这个卷上可以存储的数据片段的最大大小为 1GB。所有大于 1GB 的数据片段都会被直接存储到 `cold` 卷上，`cold` 卷包含一个名为 `disk1` 的 HDD 磁盘。 +同样，一旦 `fast_ssd` 被填充超过 80%，数据会通过后台进程向 `disk1` 进行转移。 + +存储策略中卷的枚举顺序是很重要的。因为当一个卷被充满时，数据会向下一个卷转移。磁盘的枚举顺序同样重要，因为数据是依次存储在磁盘上的。 + +在创建表时，可以将一个配置好的策略应用到表： + +``` sql +CREATE TABLE table_with_non_default_policy ( + EventDate Date, + OrderID UInt64, + BannerID UInt64, + SearchPhrase String +) ENGINE = MergeTree +ORDER BY (OrderID, BannerID) +PARTITION BY toYYYYMM(EventDate) +SETTINGS storage_policy = 'moving_from_ssd_to_hdd' +``` + +`default` 存储策略意味着只使用一个卷，这个卷只包含一个在 `` 中定义的磁盘。表创建后，它的存储策略就不能改变了。 + +可以通过 [background\_move\_pool\_size](../../../operations/settings/settings.md#background_move_pool_size) 设置调整执行后台任务的线程数。 + +### 详细说明 {#details} + +对于 `MergeTree` 表，数据通过以下不同的方式写入到磁盘当中： + +- 作为插入（`INSERT`查询）的结果 +- 在后台合并和[数据变异](../../../sql-reference/statements/alter.md#alter-mutations)期间 +- 当从另一个副本下载时 +- 作为 [ALTER TABLE … FREEZE PARTITION](../../../sql-reference/statements/alter.md#alter_freeze-partition) 冻结分区的结果 + +除了数据变异和冻结分区以外的情况下，数据按照以下逻辑存储到卷或磁盘上： + +1. 首个卷（按定义顺序）拥有足够的磁盘空间存储数据片段（`unreserved_space > current_part_size`）并且允许存储给定数据片段的大小（`max_data_part_size_bytes > current_part_size`） +2. 在这个数据卷内，紧挨着先前存储数据的那块磁盘之后的磁盘，拥有比数据片段大的剩余空间。（`unreserved_space - keep_free_space_bytes > current_part_size`） + +更进一步，数据变异和分区冻结使用的是 [硬链接](https://en.wikipedia.org/wiki/Hard_link)。不同磁盘之间的硬链接是不支持的，所以在这种情况下数据片段都会被存储到初始化的那一块磁盘上。 + +在后台，数据片段基于剩余空间（`move_factor`参数）根据卷在配置文件中定义的顺序进行转移。数据永远不会从最后一个移出也不会从第一个移入。可以通过系统表 [system.part\_log](../../../operations/system-tables/part_log.md#system_tables-part-log) (字段 `type = MOVE_PART`) 和 [system.parts](../../../operations/system-tables/parts.md#system_tables-parts) (字段 `path` 和 `disk`) 来监控后台的移动情况。同时，具体细节可以通过服务器日志查看。 + +用户可以通过 [ALTER TABLE … MOVE PART\|PARTITION … TO VOLUME\|DISK …](../../../sql-reference/statements/alter.md#alter_move-partition) 强制移动一个数据片段或分区到另外一个卷，所有后台移动的限制都会被考虑在内。这个查询会自行启动，无需等待后台操作完成。如果没有足够的可用空间或任何必须条件没有被满足，用户会收到报错信息。 + +数据移动不会妨碍到数据复制。也就是说，同一张表的不同副本可以指定不同的存储策略。 + +在后台合并和数据变异之后，就的数据片段会在一定时间后被移除 (`old_parts_lifetime`)。在这期间，他们不能被移动到其他的卷或磁盘。也就是说，直到数据片段被完全移除，它们仍然会被磁盘占用空间计算在内。 + +[原始文章](https://clickhouse.tech/docs/en/operations/table_engines/mergetree/) diff --git a/docs/zh/engines/table-engines/mergetree-family/replacingmergetree.md b/docs/zh/engines/table-engines/mergetree-family/replacingmergetree.md index 03b47172400..626597eeaf0 100644 --- a/docs/zh/engines/table-engines/mergetree-family/replacingmergetree.md +++ b/docs/zh/engines/table-engines/mergetree-family/replacingmergetree.md @@ -25,7 +25,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster] **参数** -- `ver` — 版本列。类型为 `UInt*`, `Date`, `DateTime` 或 `DateTime64`。可选参数。 +- `ver` — 版本列。类型为 `UInt*`, `Date` 或 `DateTime`。可选参数。 合并的时候，`ReplacingMergeTree` 从所有具有相同主键的行中选择一行留下： - 如果 `ver` 列未指定，选择最后一条。 diff --git a/docs/zh/interfaces/http.md b/docs/zh/interfaces/http.md index 0fecb1873db..9feb8c5d69d 100644 --- a/docs/zh/interfaces/http.md +++ b/docs/zh/interfaces/http.md @@ -23,7 +23,7 @@ Ok. $ curl 'http://localhost:8123/?query=SELECT%201' 1 -$ wget -O- -q 'http://localhost:8123/?query=SELECT 1' +$ wget -nv -O- 'http://localhost:8123/?query=SELECT 1' 1 $ GET 'http://localhost:8123/?query=SELECT 1' diff --git a/programs/benchmark/Benchmark.cpp b/programs/benchmark/Benchmark.cpp index 3ae3980c273..c8fdde3d3a6 100644 --- a/programs/benchmark/Benchmark.cpp +++ b/programs/benchmark/Benchmark.cpp @@ -104,6 +104,8 @@ public: query_processing_stage = QueryProcessingStage::FetchColumns; else if (stage == "with_mergeable_state") query_processing_stage = QueryProcessingStage::WithMergeableState; + else if (stage == "with_mergeable_state_after_aggregation") + query_processing_stage = QueryProcessingStage::WithMergeableStateAfterAggregation; else throw Exception("Unknown query processing stage: " + stage, ErrorCodes::BAD_ARGUMENTS); @@ -564,8 +566,8 @@ int mainEntryClickHouseBenchmark(int argc, char ** argv) desc.add_options() ("help", "produce help message") ("concurrency,c", value()->default_value(1), "number of parallel queries") - ("delay,d", value()->default_value(1), "delay between intermediate reports in seconds (set 0 to disable reports)") - ("stage", value()->default_value("complete"), "request query processing up to specified stage: complete,fetch_columns,with_mergeable_state") + ("delay,d", value()->default_value(1), "delay between intermediate reports in seconds (set 0 to disable reports)") + ("stage", value()->default_value("complete"), "request query processing up to specified stage: complete,fetch_columns,with_mergeable_state,with_mergeable_state_after_aggregation") ("iterations,i", value()->default_value(0), "amount of queries to be executed") ("timelimit,t", value()->default_value(0.), "stop launch of queries after specified time limit") ("randomize,r", value()->default_value(false), "randomize order of execution") diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 3a975325851..f24ba444203 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -716,6 +716,7 @@ int Server::main(const std::vector & /*args*/) { /// Disable DNS caching at all DNSResolver::instance().setDisableCacheFlag(); + LOG_DEBUG(log, "DNS caching disabled"); } else { diff --git a/src/Access/AccessControlManager.cpp b/src/Access/AccessControlManager.cpp index 6158be1b603..1fa26c85354 100644 --- a/src/Access/AccessControlManager.cpp +++ b/src/Access/AccessControlManager.cpp @@ -281,41 +281,33 @@ void AccessControlManager::addStoragesFromMainConfig( String config_dir = std::filesystem::path{config_path}.remove_filename().string(); String dbms_dir = config.getString("path", DBMS_DEFAULT_PATH); String include_from_path = config.getString("include_from", "/etc/metrika.xml"); + bool has_user_directories = config.has("user_directories"); - if (config.has("user_directories")) + /// If path to users' config isn't absolute, try guess its root (current) dir. + /// At first, try to find it in dir of main config, after will use current dir. + String users_config_path = config.getString("users_config", ""); + if (users_config_path.empty()) { - if (config.has("users_config")) - LOG_WARNING(getLogger(), " is specified, the path from won't be used: " + config.getString("users_config")); - if (config.has("access_control_path")) - LOG_WARNING(getLogger(), " is specified, the path from won't be used: " + config.getString("access_control_path")); - - addStoragesFromUserDirectoriesConfig( - config, - "user_directories", - config_dir, - dbms_dir, - include_from_path, - get_zookeeper_function); - } - else - { - /// If path to users' config isn't absolute, try guess its root (current) dir. - /// At first, try to find it in dir of main config, after will use current dir. - String users_config_path = config.getString("users_config", ""); - if (users_config_path.empty()) + if (!has_user_directories) users_config_path = config_path; - else if (std::filesystem::path{users_config_path}.is_relative() && std::filesystem::exists(config_dir + users_config_path)) - users_config_path = config_dir + users_config_path; + } + else if (std::filesystem::path{users_config_path}.is_relative() && std::filesystem::exists(config_dir + users_config_path)) + users_config_path = config_dir + users_config_path; + if (!users_config_path.empty()) + { if (users_config_path != config_path) checkForUsersNotInMainConfig(config, config_path, users_config_path, getLogger()); addUsersConfigStorage(users_config_path, include_from_path, dbms_dir, get_zookeeper_function); - - String disk_storage_dir = config.getString("access_control_path", ""); - if (!disk_storage_dir.empty()) - addDiskStorage(disk_storage_dir); } + + String disk_storage_dir = config.getString("access_control_path", ""); + if (!disk_storage_dir.empty()) + addDiskStorage(disk_storage_dir); + + if (has_user_directories) + addStoragesFromUserDirectoriesConfig(config, "user_directories", config_dir, dbms_dir, include_from_path, get_zookeeper_function); } diff --git a/src/Access/AccessControlManager.h b/src/Access/AccessControlManager.h index ad9fb48d263..d7cf59cfb28 100644 --- a/src/Access/AccessControlManager.h +++ b/src/Access/AccessControlManager.h @@ -47,7 +47,7 @@ class AccessControlManager : public MultipleAccessStorage { public: AccessControlManager(); - ~AccessControlManager(); + ~AccessControlManager() override; /// Parses access entities from a configuration loaded from users.xml. /// This function add UsersConfigAccessStorage if it wasn't added before. diff --git a/src/Access/AccessFlags.h b/src/Access/AccessFlags.h index 9b801fd88a3..3cb92b6b855 100644 --- a/src/Access/AccessFlags.h +++ b/src/Access/AccessFlags.h @@ -96,6 +96,22 @@ public: /// Returns all the flags related to a dictionary. static AccessFlags allDictionaryFlags(); + /// Returns all the flags which could be granted on the global level. + /// The same as allFlags(). + static AccessFlags allFlagsGrantableOnGlobalLevel(); + + /// Returns all the flags which could be granted on the database level. + /// Returns allDatabaseFlags() | allTableFlags() | allDictionaryFlags() | allColumnFlags(). + static AccessFlags allFlagsGrantableOnDatabaseLevel(); + + /// Returns all the flags which could be granted on the table level. + /// Returns allTableFlags() | allDictionaryFlags() | allColumnFlags(). + static AccessFlags allFlagsGrantableOnTableLevel(); + + /// Returns all the flags which could be granted on the global level. + /// The same as allColumnFlags(). + static AccessFlags allFlagsGrantableOnColumnLevel(); + private: static constexpr size_t NUM_FLAGS = 128; using Flags = std::bitset; @@ -193,6 +209,10 @@ public: const Flags & getTableFlags() const { return all_flags_for_target[TABLE]; } const Flags & getColumnFlags() const { return all_flags_for_target[COLUMN]; } const Flags & getDictionaryFlags() const { return all_flags_for_target[DICTIONARY]; } + const Flags & getAllFlagsGrantableOnGlobalLevel() const { return getAllFlags(); } + const Flags & getAllFlagsGrantableOnDatabaseLevel() const { return all_flags_grantable_on_database_level; } + const Flags & getAllFlagsGrantableOnTableLevel() const { return all_flags_grantable_on_table_level; } + const Flags & getAllFlagsGrantableOnColumnLevel() const { return getColumnFlags(); } private: enum NodeType @@ -381,6 +401,9 @@ private: } for (const auto & child : start_node->children) collectAllFlags(child.get()); + + all_flags_grantable_on_table_level = all_flags_for_target[TABLE] | all_flags_for_target[DICTIONARY] | all_flags_for_target[COLUMN]; + all_flags_grantable_on_database_level = all_flags_for_target[DATABASE] | all_flags_grantable_on_table_level; } Impl() @@ -431,6 +454,8 @@ private: std::vector access_type_to_flags_mapping; Flags all_flags; Flags all_flags_for_target[static_cast(DICTIONARY) + 1]; + Flags all_flags_grantable_on_database_level; + Flags all_flags_grantable_on_table_level; }; @@ -447,6 +472,10 @@ inline AccessFlags AccessFlags::allDatabaseFlags() { return Impl<>::instance().g inline AccessFlags AccessFlags::allTableFlags() { return Impl<>::instance().getTableFlags(); } inline AccessFlags AccessFlags::allColumnFlags() { return Impl<>::instance().getColumnFlags(); } inline AccessFlags AccessFlags::allDictionaryFlags() { return Impl<>::instance().getDictionaryFlags(); } +inline AccessFlags AccessFlags::allFlagsGrantableOnGlobalLevel() { return Impl<>::instance().getAllFlagsGrantableOnGlobalLevel(); } +inline AccessFlags AccessFlags::allFlagsGrantableOnDatabaseLevel() { return Impl<>::instance().getAllFlagsGrantableOnDatabaseLevel(); } +inline AccessFlags AccessFlags::allFlagsGrantableOnTableLevel() { return Impl<>::instance().getAllFlagsGrantableOnTableLevel(); } +inline AccessFlags AccessFlags::allFlagsGrantableOnColumnLevel() { return Impl<>::instance().getAllFlagsGrantableOnColumnLevel(); } inline AccessFlags operator |(AccessType left, AccessType right) { return AccessFlags(left) | right; } inline AccessFlags operator &(AccessType left, AccessType right) { return AccessFlags(left) & right; } diff --git a/src/Access/AccessRights.cpp b/src/Access/AccessRights.cpp index 65c78f39e86..8ce71dd8da8 100644 --- a/src/Access/AccessRights.cpp +++ b/src/Access/AccessRights.cpp @@ -1,5 +1,4 @@ #include -#include #include #include #include @@ -8,12 +7,6 @@ namespace DB { -namespace ErrorCodes -{ - extern const int INVALID_GRANT; -} - - namespace { using Kind = AccessRightsElementWithOptions::Kind; @@ -214,30 +207,14 @@ namespace COLUMN_LEVEL, }; - AccessFlags getAcceptableFlags(Level level) + AccessFlags getAllGrantableFlags(Level level) { switch (level) { - case GLOBAL_LEVEL: - { - static const AccessFlags res = AccessFlags::allFlags(); - return res; - } - case DATABASE_LEVEL: - { - static const AccessFlags res = AccessFlags::allDatabaseFlags() | AccessFlags::allTableFlags() | AccessFlags::allDictionaryFlags() | AccessFlags::allColumnFlags(); - return res; - } - case TABLE_LEVEL: - { - static const AccessFlags res = AccessFlags::allTableFlags() | AccessFlags::allDictionaryFlags() | AccessFlags::allColumnFlags(); - return res; - } - case COLUMN_LEVEL: - { - static const AccessFlags res = AccessFlags::allColumnFlags(); - return res; - } + case GLOBAL_LEVEL: return AccessFlags::allFlagsGrantableOnGlobalLevel(); + case DATABASE_LEVEL: return AccessFlags::allFlagsGrantableOnDatabaseLevel(); + case TABLE_LEVEL: return AccessFlags::allFlagsGrantableOnTableLevel(); + case COLUMN_LEVEL: return AccessFlags::allFlagsGrantableOnColumnLevel(); } __builtin_unreachable(); } @@ -276,21 +253,7 @@ public: void grant(const AccessFlags & flags_) { - if (!flags_) - return; - - AccessFlags flags_to_add = flags_ & getAcceptableFlags(); - - if (!flags_to_add) - { - if (level == DATABASE_LEVEL) - throw Exception(flags_.toString() + " cannot be granted on the database level", ErrorCodes::INVALID_GRANT); - else if (level == TABLE_LEVEL) - throw Exception(flags_.toString() + " cannot be granted on the table level", ErrorCodes::INVALID_GRANT); - else if (level == COLUMN_LEVEL) - throw Exception(flags_.toString() + " cannot be granted on the column level", ErrorCodes::INVALID_GRANT); - } - + AccessFlags flags_to_add = flags_ & getAllGrantableFlags(); addGrantsRec(flags_to_add); optimizeTree(); } @@ -456,8 +419,8 @@ public: } private: - AccessFlags getAcceptableFlags() const { return ::DB::getAcceptableFlags(level); } - AccessFlags getChildAcceptableFlags() const { return ::DB::getAcceptableFlags(static_cast(level + 1)); } + AccessFlags getAllGrantableFlags() const { return ::DB::getAllGrantableFlags(level); } + AccessFlags getChildAllGrantableFlags() const { return ::DB::getAllGrantableFlags(static_cast(level + 1)); } Node * tryGetChild(const std::string_view & name) const { @@ -480,7 +443,7 @@ private: Node & new_child = (*children)[*new_child_name]; new_child.node_name = std::move(new_child_name); new_child.level = static_cast(level + 1); - new_child.flags = flags & new_child.getAcceptableFlags(); + new_child.flags = flags & new_child.getAllGrantableFlags(); return new_child; } @@ -496,12 +459,12 @@ private: bool canEraseChild(const Node & child) const { - return ((flags & child.getAcceptableFlags()) == child.flags) && !child.children; + return ((flags & child.getAllGrantableFlags()) == child.flags) && !child.children; } void addGrantsRec(const AccessFlags & flags_) { - if (auto flags_to_add = flags_ & getAcceptableFlags()) + if (auto flags_to_add = flags_ & getAllGrantableFlags()) { flags |= flags_to_add; if (children) @@ -547,7 +510,7 @@ private: const AccessFlags & parent_flags) { auto flags = node.flags; - auto parent_fl = parent_flags & node.getAcceptableFlags(); + auto parent_fl = parent_flags & node.getAllGrantableFlags(); auto revokes = parent_fl - flags; auto grants = flags - parent_fl; @@ -576,9 +539,9 @@ private: const Node * node_go, const AccessFlags & parent_flags_go) { - auto acceptable_flags = ::DB::getAcceptableFlags(static_cast(full_name.size())); - auto parent_fl = parent_flags & acceptable_flags; - auto parent_fl_go = parent_flags_go & acceptable_flags; + auto grantable_flags = ::DB::getAllGrantableFlags(static_cast(full_name.size())); + auto parent_fl = parent_flags & grantable_flags; + auto parent_fl_go = parent_flags_go & grantable_flags; auto flags = node ? node->flags : parent_fl; auto flags_go = node_go ? node_go->flags : parent_fl_go; auto revokes = parent_fl - flags; @@ -672,8 +635,8 @@ private: } max_flags_with_children |= max_among_children; - AccessFlags add_acceptable_flags = getAcceptableFlags() - getChildAcceptableFlags(); - min_flags_with_children &= min_among_children | add_acceptable_flags; + AccessFlags add_flags = getAllGrantableFlags() - getChildAllGrantableFlags(); + min_flags_with_children &= min_among_children | add_flags; } void makeUnionRec(const Node & rhs) @@ -689,7 +652,7 @@ private: for (auto & [lhs_childname, lhs_child] : *children) { if (!rhs.tryGetChild(lhs_childname)) - lhs_child.flags |= rhs.flags & lhs_child.getAcceptableFlags(); + lhs_child.flags |= rhs.flags & lhs_child.getAllGrantableFlags(); } } } @@ -738,7 +701,7 @@ private: if (new_flags != flags) { - new_flags &= getAcceptableFlags(); + new_flags &= getAllGrantableFlags(); flags_added |= static_cast(new_flags - flags); flags_removed |= static_cast(flags - new_flags); flags = new_flags; diff --git a/src/Access/AccessRightsElement.h b/src/Access/AccessRightsElement.h index f9f7c433308..36cb64e6eba 100644 --- a/src/Access/AccessRightsElement.h +++ b/src/Access/AccessRightsElement.h @@ -71,6 +71,8 @@ struct AccessRightsElement { } + bool empty() const { return !access_flags || (!any_column && columns.empty()); } + auto toTuple() const { return std::tie(access_flags, any_database, database, any_table, table, any_column, columns); } friend bool operator==(const AccessRightsElement & left, const AccessRightsElement & right) { return left.toTuple() == right.toTuple(); } friend bool operator!=(const AccessRightsElement & left, const AccessRightsElement & right) { return !(left == right); } @@ -86,6 +88,9 @@ struct AccessRightsElement /// If the database is empty, replaces it with `new_database`. Otherwise does nothing. void replaceEmptyDatabase(const String & new_database); + /// Resets flags which cannot be granted. + void removeNonGrantableFlags(); + /// Returns a human-readable representation like "SELECT, UPDATE(x, y) ON db.table". String toString() const; }; @@ -111,6 +116,9 @@ struct AccessRightsElementWithOptions : public AccessRightsElement friend bool operator==(const AccessRightsElementWithOptions & left, const AccessRightsElementWithOptions & right) { return left.toTuple() == right.toTuple(); } friend bool operator!=(const AccessRightsElementWithOptions & left, const AccessRightsElementWithOptions & right) { return !(left == right); } + /// Resets flags which cannot be granted. + void removeNonGrantableFlags(); + /// Returns a human-readable representation like "GRANT SELECT, UPDATE(x, y) ON db.table". String toString() const; }; @@ -120,9 +128,14 @@ struct AccessRightsElementWithOptions : public AccessRightsElement class AccessRightsElements : public std::vector { public: + bool empty() const { return std::all_of(begin(), end(), [](const AccessRightsElement & e) { return e.empty(); }); } + /// Replaces the empty database with `new_database`. void replaceEmptyDatabase(const String & new_database); + /// Resets flags which cannot be granted. + void removeNonGrantableFlags(); + /// Returns a human-readable representation like "GRANT SELECT, UPDATE(x, y) ON db.table". String toString() const; }; @@ -134,6 +147,9 @@ public: /// Replaces the empty database with `new_database`. void replaceEmptyDatabase(const String & new_database); + /// Resets flags which cannot be granted. + void removeNonGrantableFlags(); + /// Returns a human-readable representation like "GRANT SELECT, UPDATE(x, y) ON db.table". String toString() const; }; @@ -157,4 +173,34 @@ inline void AccessRightsElementsWithOptions::replaceEmptyDatabase(const String & element.replaceEmptyDatabase(new_database); } +inline void AccessRightsElement::removeNonGrantableFlags() +{ + if (!any_column) + access_flags &= AccessFlags::allFlagsGrantableOnColumnLevel(); + else if (!any_table) + access_flags &= AccessFlags::allFlagsGrantableOnTableLevel(); + else if (!any_database) + access_flags &= AccessFlags::allFlagsGrantableOnDatabaseLevel(); + else + access_flags &= AccessFlags::allFlagsGrantableOnGlobalLevel(); +} + +inline void AccessRightsElementWithOptions::removeNonGrantableFlags() +{ + if (kind == Kind::GRANT) + AccessRightsElement::removeNonGrantableFlags(); +} + +inline void AccessRightsElements::removeNonGrantableFlags() +{ + for (auto & element : *this) + element.removeNonGrantableFlags(); +} + +inline void AccessRightsElementsWithOptions::removeNonGrantableFlags() +{ + for (auto & element : *this) + element.removeNonGrantableFlags(); +} + } diff --git a/src/AggregateFunctions/AggregateFunctionTopK.cpp b/src/AggregateFunctions/AggregateFunctionTopK.cpp index a8cea5eb59b..e32da02f442 100644 --- a/src/AggregateFunctions/AggregateFunctionTopK.cpp +++ b/src/AggregateFunctions/AggregateFunctionTopK.cpp @@ -85,12 +85,12 @@ AggregateFunctionPtr createAggregateFunctionTopK(const std::string & name, const load_factor = applyVisitor(FieldVisitorConvertToNumber(), params[1]); if (load_factor < 1) - throw Exception("Too small parameter for aggregate function " + name + ". Minimum: 1", + throw Exception("Too small parameter 'load_factor' for aggregate function " + name + ". Minimum: 1", ErrorCodes::ARGUMENT_OUT_OF_BOUND); } - if (k > TOP_K_MAX_SIZE) - throw Exception("Too large parameter for aggregate function " + name + ". Maximum: " + toString(TOP_K_MAX_SIZE), + if (k > TOP_K_MAX_SIZE || load_factor > TOP_K_MAX_SIZE || k * load_factor > TOP_K_MAX_SIZE) + throw Exception("Too large parameter(s) for aggregate function " + name + ". Maximum: " + toString(TOP_K_MAX_SIZE), ErrorCodes::ARGUMENT_OUT_OF_BOUND); if (k == 0) diff --git a/src/Columns/ColumnDecimal.h b/src/Columns/ColumnDecimal.h index 11ab294c1a3..c33ab34b541 100644 --- a/src/Columns/ColumnDecimal.h +++ b/src/Columns/ColumnDecimal.h @@ -126,7 +126,7 @@ public: bool isNumeric() const override { return false; } bool canBeInsideNullable() const override { return true; } - bool isFixedAndContiguous() const override { return is_POD; } + bool isFixedAndContiguous() const override { return true; } size_t sizeOfValueIfFixed() const override { return sizeof(T); } size_t size() const override { return data.size(); } diff --git a/src/Columns/ColumnVector.h b/src/Columns/ColumnVector.h index d2c4846193c..1090de556a0 100644 --- a/src/Columns/ColumnVector.h +++ b/src/Columns/ColumnVector.h @@ -12,11 +12,6 @@ namespace DB { -namespace ErrorCodes -{ - extern const int NOT_IMPLEMENTED; -} - /** Stuff for comparing numbers. * Integer values are compared as usual. * Floating-point numbers are compared this way that NaNs always end up at the end @@ -298,23 +293,17 @@ public: void gather(ColumnGathererStream & gatherer_stream) override; bool canBeInsideNullable() const override { return true; } - bool isFixedAndContiguous() const override { return is_POD; } + bool isFixedAndContiguous() const override { return true; } size_t sizeOfValueIfFixed() const override { return sizeof(T); } StringRef getRawData() const override { - if constexpr (is_POD) - return StringRef(reinterpret_cast(data.data()), byteSize()); - else - throw Exception("getRawData() is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); + return StringRef(reinterpret_cast(data.data()), byteSize()); } StringRef getDataAt(size_t n) const override { - if constexpr (is_POD) - return StringRef(reinterpret_cast(&data[n]), sizeof(data[n])); - else - throw Exception("getDataAt() is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); + return StringRef(reinterpret_cast(&data[n]), sizeof(data[n])); } bool structureEquals(const IColumn & rhs) const override diff --git a/src/Common/HashTable/Hash.h b/src/Common/HashTable/Hash.h index 1d850ab2d32..c561933ab80 100644 --- a/src/Common/HashTable/Hash.h +++ b/src/Common/HashTable/Hash.h @@ -1,6 +1,7 @@ #pragma once #include +#include #include #include @@ -89,8 +90,7 @@ template inline typename std::enable_if, DB::UInt64>::type intHashCRC32(const T & x, DB::UInt64 updated_value) { - std::vector parts; - export_bits(x, std::back_inserter(parts), sizeof(UInt64), false); + std::vector parts = DB::BigInt::toIntArray(x); for (const auto & part : parts) updated_value = intHashCRC32(part, updated_value); @@ -199,7 +199,7 @@ inline size_t DefaultHash64(std::enable_if_t<(sizeof(T) > sizeof(UInt64)), T> ke { return intHash64(key.low ^ key.high); } - else if constexpr (std::is_same_v || std::is_same_v) + else if constexpr (is_big_int_v && sizeof(T) == 32) { return intHash64(static_cast(key) ^ static_cast(key >> 64) ^ @@ -256,7 +256,7 @@ inline size_t hashCRC32(std::enable_if_t<(sizeof(T) > sizeof(UInt64)), T> key) { return intHashCRC32(key.low ^ key.high); } - else if constexpr (std::is_same_v || std::is_same_v) + else if constexpr (is_big_int_v && sizeof(T) == 32) { return intHashCRC32(static_cast(key) ^ static_cast(key >> 64) ^ @@ -358,7 +358,7 @@ struct IntHash32 { return intHash32(key.low ^ key.high); } - else if constexpr (std::is_same_v || std::is_same_v) + else if constexpr (is_big_int_v && sizeof(T) == 32) { return intHash32(static_cast(key) ^ static_cast(key >> 64) ^ diff --git a/src/Common/SipHash.h b/src/Common/SipHash.h index df4c09da6d9..f4f86951652 100644 --- a/src/Common/SipHash.h +++ b/src/Common/SipHash.h @@ -148,7 +148,7 @@ public: } template - std::enable_if_t, void> update(const T & x) + std::enable_if_t && !std::has_unique_object_representations_v, void> update(const T & x) { update(DB::BigInt::serialize(x)); } @@ -213,7 +213,7 @@ std::enable_if_t, UInt64> sipHash64( } template -std::enable_if_t<(std::is_floating_point_v || is_big_int_v), UInt64> sipHash64(const T & x) +std::enable_if_t<(std::is_floating_point_v || (is_big_int_v && !std::has_unique_object_representations_v)), UInt64> sipHash64(const T & x) { SipHash hash; hash.update(x); diff --git a/src/Common/SpaceSaving.h b/src/Common/SpaceSaving.h index 56063340240..cb6fee1ad91 100644 --- a/src/Common/SpaceSaving.h +++ b/src/Common/SpaceSaving.h @@ -147,16 +147,17 @@ public: { // Increase weight of a key that already exists auto hash = counter_map.hash(key); - auto counter = findCounter(key, hash); - if (counter) + + if (auto counter = findCounter(key, hash); counter) { counter->count += increment; counter->error += error; percolate(counter); return; } + // Key doesn't exist, but can fit in the top K - else if (unlikely(size() < capacity())) + if (unlikely(size() < capacity())) { auto c = new Counter(arena.emplace(key), increment, error, hash); push(c); diff --git a/src/Common/StringSearcher.h b/src/Common/StringSearcher.h index 13ed67eac93..a4da499837b 100644 --- a/src/Common/StringSearcher.h +++ b/src/Common/StringSearcher.h @@ -254,7 +254,7 @@ public: const auto offset = __builtin_ctz(mask); haystack += offset; - if (haystack < haystack_end && haystack + n <= haystack_end && pageSafe(haystack)) + if (haystack + n <= haystack_end && pageSafe(haystack)) { const auto v_haystack_offset = _mm_loadu_si128(reinterpret_cast(haystack)); const auto v_against_l_offset = _mm_cmpeq_epi8(v_haystack_offset, cachel); @@ -463,7 +463,7 @@ public: const auto offset = __builtin_ctz(mask); haystack += offset; - if (haystack < haystack_end && haystack + n <= haystack_end && pageSafe(haystack)) + if (haystack + n <= haystack_end && pageSafe(haystack)) { const auto v_haystack_offset = _mm_loadu_si128(reinterpret_cast(haystack)); const auto v_against_l_offset = _mm_cmpeq_epi8(v_haystack_offset, cachel); @@ -652,7 +652,7 @@ public: const auto offset = __builtin_ctz(mask); haystack += offset; - if (haystack < haystack_end && haystack + n <= haystack_end && pageSafe(haystack)) + if (haystack + n <= haystack_end && pageSafe(haystack)) { /// check for first 16 octets const auto v_haystack_offset = _mm_loadu_si128(reinterpret_cast(haystack)); diff --git a/src/Common/UInt128.h b/src/Common/UInt128.h index 2b1177e970c..3944d8073c2 100644 --- a/src/Common/UInt128.h +++ b/src/Common/UInt128.h @@ -67,6 +67,11 @@ struct UInt128 bool inline operator <= (const Int128 rhs) const { return *this <= UInt128(rhs, rhs >> 64) && rhs >= 0; } bool inline operator < (const Int128 rhs) const { return *this < UInt128(rhs, rhs >> 64) && rhs >= 0; } + bool inline operator > (const Int256 rhs) const { return (rhs < 0) || ((Int256(high) << 64) + low) > rhs; } + bool inline operator > (const UInt256 rhs) const { return ((UInt256(high) << 64) + low) > rhs; } + bool inline operator < (const Int256 rhs) const { return (rhs >= 0) && ((Int256(high) << 64) + low) < rhs; } + bool inline operator < (const UInt256 rhs) const { return ((UInt256(high) << 64) + low) < rhs; } + template bool inline operator== (const T rhs) const { return *this == UInt128(rhs); } template bool inline operator!= (const T rhs) const { return *this != UInt128(rhs); } template bool inline operator>= (const T rhs) const { return *this >= UInt128(rhs); } diff --git a/src/Common/intExp.h b/src/Common/intExp.h index d585eaa9c67..8a52015c54a 100644 --- a/src/Common/intExp.h +++ b/src/Common/intExp.h @@ -138,9 +138,9 @@ constexpr inline Int128 exp10_i128(int x) } -inline bInt256 exp10_i256(int x) +inline wInt256 exp10_i256(int x) { - using Int256 = bInt256; + using Int256 = wInt256; static constexpr Int256 i10e18{1000000000000000000ll}; static const Int256 values[] = { static_cast(1ll), diff --git a/src/Core/BigInt.h b/src/Core/BigInt.h index f1da96d330a..5abd7110062 100644 --- a/src/Core/BigInt.h +++ b/src/Core/BigInt.h @@ -7,46 +7,15 @@ namespace DB { template -struct BigIntPayload +struct BigInt { - static_assert(!is_big_int_v); - static constexpr size_t size = 0; -}; - -template <> struct BigIntPayload { static constexpr size_t size = 32; }; - -template <> struct BigIntPayload -{ - using UnsingedType = bUInt256; + static_assert(sizeof(T) == 32); static constexpr size_t size = 32; -}; - -template -struct BigInt : BigIntPayload -{ - using BigIntPayload::size; - - static constexpr size_t lastBit() - { - return size * 8 - 1; - } static StringRef serialize(const T & x, char * pos) { - if constexpr (is_signed_v) - { - using UnsignedT = typename BigIntPayload::UnsingedType; - - if (x < 0) - { - UnsignedT unsigned_x = UnsignedT{0} - static_cast(-x); - export_bits(unsigned_x, pos, 8, false); - } - else - export_bits(x, pos, 8, false); - } - else - export_bits(x, pos, 8, false); + //unalignedStore(pos, x); + memcpy(pos, &x, size); return StringRef(pos, size); } @@ -59,24 +28,20 @@ struct BigInt : BigIntPayload static T deserialize(const char * pos) { - if constexpr (is_signed_v) - { - using UnsignedT = typename BigIntPayload::UnsingedType; + //return unalignedLoad(pos); + T res; + memcpy(&res, pos, size); + return res; + } - UnsignedT unsigned_x; - import_bits(unsigned_x, pos, pos + size, false); - - bool is_negative = bit_test(unsigned_x, lastBit()); - if (is_negative) - unsigned_x = UnsignedT{0} - unsigned_x; - return static_cast(unsigned_x); - } - else - { - T x; - import_bits(x, pos, pos + size, false); - return x; - } + static std::vector toIntArray(const T & x) + { + std::vector parts(4, 0); + parts[0] = UInt64(x); + parts[1] = UInt64(x >> 64); + parts[2] = UInt64(x >> 128); + parts[4] = UInt64(x >> 192); + return parts; } }; diff --git a/src/Core/DecimalComparison.h b/src/Core/DecimalComparison.h index a43ce8d803c..93992029634 100644 --- a/src/Core/DecimalComparison.h +++ b/src/Core/DecimalComparison.h @@ -226,25 +226,25 @@ private: static NO_INLINE UInt8 apply(A a, B b, CompareInt scale [[maybe_unused]]) { CompareInt x; - if constexpr (is_big_int_v && IsDecimalNumber) + if constexpr (IsDecimalNumber) x = a.value; else - x = bigint_cast(a); + x = a; CompareInt y; - if constexpr (is_big_int_v && IsDecimalNumber) + if constexpr (IsDecimalNumber) y = b.value; else - y = bigint_cast(b); + y = b; if constexpr (_check_overflow) { bool overflow = false; if constexpr (sizeof(A) > sizeof(CompareInt)) - overflow |= (A(x) != a); + overflow |= (bigint_cast(x) != a); if constexpr (sizeof(B) > sizeof(CompareInt)) - overflow |= (B(y) != b); + overflow |= (bigint_cast(y) != b); if constexpr (is_unsigned_v) overflow |= (x < 0); if constexpr (is_unsigned_v) diff --git a/src/Core/MySQL/MySQLReplication.cpp b/src/Core/MySQL/MySQLReplication.cpp index f26436440b8..e3848c52aec 100644 --- a/src/Core/MySQL/MySQLReplication.cpp +++ b/src/Core/MySQL/MySQLReplication.cpp @@ -13,6 +13,7 @@ namespace DB namespace ErrorCodes { extern const int UNKNOWN_EXCEPTION; + extern const int LOGICAL_ERROR; } namespace MySQLReplication @@ -110,12 +111,12 @@ namespace MySQLReplication else if (query.starts_with("XA")) { if (query.starts_with("XA ROLLBACK")) - throw ReplicationError("ParseQueryEvent: Unsupported query event:" + query, ErrorCodes::UNKNOWN_EXCEPTION); + throw ReplicationError("ParseQueryEvent: Unsupported query event:" + query, ErrorCodes::LOGICAL_ERROR); typ = QUERY_EVENT_XA; } else if (query.starts_with("SAVEPOINT")) { - throw ReplicationError("ParseQueryEvent: Unsupported query event:" + query, ErrorCodes::UNKNOWN_EXCEPTION); + throw ReplicationError("ParseQueryEvent: Unsupported query event:" + query, ErrorCodes::LOGICAL_ERROR); } } @@ -741,7 +742,9 @@ namespace MySQLReplication void GTIDEvent::dump(std::ostream & out) const { - auto gtid_next = gtid.uuid.toUnderType().toHexString() + ":" + std::to_string(gtid.seq_no); + WriteBufferFromOwnString ws; + writeUUIDText(gtid.uuid, ws); + auto gtid_next = ws.str() + ":" + std::to_string(gtid.seq_no); header.dump(out); out << "GTID Next: " << gtid_next << std::endl; diff --git a/src/Core/QueryProcessingStage.h b/src/Core/QueryProcessingStage.h index 6de65d32f93..658b504fc2c 100644 --- a/src/Core/QueryProcessingStage.h +++ b/src/Core/QueryProcessingStage.h @@ -10,17 +10,36 @@ namespace DB namespace QueryProcessingStage { /// Numbers matter - the later stage has a larger number. + /// + /// It is part of Protocol ABI, add values only to the end. + /// Also keep in mind that the code may depends on the order of fields, so be double aware when you will add new values. enum Enum { - FetchColumns = 0, /// Only read/have been read the columns specified in the query. - WithMergeableState = 1, /// Until the stage where the results of processing on different servers can be combined. - Complete = 2, /// Completely. + /// Only read/have been read the columns specified in the query. + FetchColumns = 0, + /// Until the stage where the results of processing on different servers can be combined. + WithMergeableState = 1, + /// Completely. + Complete = 2, + /// Until the stage where the aggregate functions were calculated and finalized. + /// + /// It is used for auto distributed_group_by_no_merge optimization for distributed engine. + /// (See comments in StorageDistributed). + WithMergeableStateAfterAggregation = 3, + + MAX = 4, }; inline const char * toString(UInt64 stage) { - static const char * data[] = { "FetchColumns", "WithMergeableState", "Complete" }; - return stage < 3 + static const char * data[] = + { + "FetchColumns", + "WithMergeableState", + "Complete", + "WithMergeableStateAfterAggregation", + }; + return stage < MAX ? data[stage] : "Unknown stage"; } diff --git a/src/Core/Settings.h b/src/Core/Settings.h index 8c4f6b8eb6f..00df95c2b69 100644 --- a/src/Core/Settings.h +++ b/src/Core/Settings.h @@ -107,8 +107,8 @@ class IColumn; \ M(Bool, skip_unavailable_shards, false, "If 1, ClickHouse silently skips unavailable shards and nodes unresolvable through DNS. Shard is marked as unavailable when none of the replicas can be reached.", 0) \ \ - M(Bool, distributed_group_by_no_merge, false, "Do not merge aggregation states from different servers for distributed query processing - in case it is for certain that there are different keys on different shards.", 0) \ M(UInt64, parallel_distributed_insert_select, 0, "Process distributed INSERT SELECT query in the same cluster on local tables on every shard, if 1 SELECT is executed on each shard, if 2 SELECT and INSERT is executed on each shard", 0) \ + M(UInt64, distributed_group_by_no_merge, 0, "If 1, Do not merge aggregation states from different servers for distributed query processing - in case it is for certain that there are different keys on different shards. If 2 - same as 1 but also apply ORDER BY and LIMIT stages", 0) \ M(Bool, optimize_distributed_group_by_sharding_key, false, "Optimize GROUP BY sharding_key queries (by avodiing costly aggregation on the initiator server).", 0) \ M(Bool, optimize_skip_unused_shards, false, "Assumes that data is distributed by sharding_key. Optimization to skip unused shards if SELECT query filters by sharding_key.", 0) \ M(UInt64, force_optimize_skip_unused_shards, 0, "Throw an exception if unused shards cannot be skipped (1 - throw only if the table has the sharding key, 2 - always throw.", 0) \ diff --git a/src/Core/Types.h b/src/Core/Types.h index 39c152ce48b..c23ac4a1379 100644 --- a/src/Core/Types.h +++ b/src/Core/Types.h @@ -58,14 +58,14 @@ using UInt8 = ::UInt8; using UInt16 = ::UInt16; using UInt32 = ::UInt32; using UInt64 = ::UInt64; -using UInt256 = ::bUInt256; +using UInt256 = ::wUInt256; using Int8 = ::Int8; using Int16 = ::Int16; using Int32 = ::Int32; using Int64 = ::Int64; using Int128 = ::Int128; -using Int256 = ::bInt256; +using Int256 = ::wInt256; using Float32 = float; using Float64 = double; diff --git a/src/DataTypes/DataTypeDateTime.cpp b/src/DataTypes/DataTypeDateTime.cpp index c860766406e..9ea698d4fbb 100644 --- a/src/DataTypes/DataTypeDateTime.cpp +++ b/src/DataTypes/DataTypeDateTime.cpp @@ -185,31 +185,4 @@ bool DataTypeDateTime::equals(const IDataType & rhs) const return typeid(rhs) == typeid(*this); } -namespace ErrorCodes -{ - extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; - extern const int ILLEGAL_TYPE_OF_ARGUMENT; -} - -static DataTypePtr create(const ASTPtr & arguments) -{ - if (!arguments) - return std::make_shared(); - - if (arguments->children.size() != 1) - throw Exception("DateTime data type can optionally have only one argument - time zone name", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); - - const auto * arg = arguments->children[0]->as(); - if (!arg || arg->value.getType() != Field::Types::String) - throw Exception("Parameter for DateTime data type must be string literal", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - - return std::make_shared(arg->value.get()); -} - -void registerDataTypeDateTime(DataTypeFactory & factory) -{ - factory.registerDataType("DateTime", create, DataTypeFactory::CaseInsensitive); - factory.registerAlias("TIMESTAMP", "DateTime", DataTypeFactory::CaseInsensitive); -} - } diff --git a/src/DataTypes/DataTypeDateTime64.cpp b/src/DataTypes/DataTypeDateTime64.cpp index 97dd28439d7..ee4139c2b7a 100644 --- a/src/DataTypes/DataTypeDateTime64.cpp +++ b/src/DataTypes/DataTypeDateTime64.cpp @@ -201,65 +201,4 @@ bool DataTypeDateTime64::equals(const IDataType & rhs) const return false; } -namespace ErrorCodes -{ - extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; - extern const int ILLEGAL_TYPE_OF_ARGUMENT; -} - -enum class ArgumentKind -{ - Optional, - Mandatory -}; - -template -std::conditional_t, T> -getArgument(const ASTPtr & arguments, size_t argument_index, const char * argument_name, const std::string context_data_type_name) -{ - using NearestResultType = NearestFieldType; - const auto field_type = Field::TypeToEnum::value; - const ASTLiteral * argument = nullptr; - - auto exception_message = [=](const String & message) - { - return std::string("Parameter #") + std::to_string(argument_index) + " '" - + argument_name + "' for " + context_data_type_name - + message - + ", expected: " + Field::Types::toString(field_type) + " literal."; - }; - - if (!arguments || arguments->children.size() <= argument_index - || !(argument = arguments->children[argument_index]->as())) - { - if constexpr (Kind == ArgumentKind::Optional) - return {}; - else - throw Exception(exception_message(" is missing"), - ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); - } - - if (argument->value.getType() != field_type) - throw Exception(exception_message(String(" has wrong type: ") + argument->value.getTypeName()), - ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - - return argument->value.get(); -} - -static DataTypePtr create64(const ASTPtr & arguments) -{ - if (!arguments || arguments->size() == 0) - return std::make_shared(DataTypeDateTime64::default_scale); - - const auto scale = getArgument(arguments, 0, "scale", "DateType64"); - const auto timezone = getArgument(arguments, !!scale, "timezone", "DateType64"); - - return std::make_shared(scale.value_or(DataTypeDateTime64::default_scale), timezone.value_or(String{})); -} - -void registerDataTypeDateTime64(DataTypeFactory & factory) -{ - factory.registerDataType("DateTime64", create64, DataTypeFactory::CaseInsensitive); -} - } diff --git a/src/DataTypes/DataTypeFactory.cpp b/src/DataTypes/DataTypeFactory.cpp index 664927389b5..9386f4b39f1 100644 --- a/src/DataTypes/DataTypeFactory.cpp +++ b/src/DataTypes/DataTypeFactory.cpp @@ -165,7 +165,6 @@ DataTypeFactory::DataTypeFactory() registerDataTypeDecimal(*this); registerDataTypeDate(*this); registerDataTypeDateTime(*this); - registerDataTypeDateTime64(*this); registerDataTypeString(*this); registerDataTypeFixedString(*this); registerDataTypeEnum(*this); diff --git a/src/DataTypes/DataTypeFactory.h b/src/DataTypes/DataTypeFactory.h index 67b72945acc..ea77c50170c 100644 --- a/src/DataTypes/DataTypeFactory.h +++ b/src/DataTypes/DataTypeFactory.h @@ -82,7 +82,6 @@ void registerDataTypeInterval(DataTypeFactory & factory); void registerDataTypeLowCardinality(DataTypeFactory & factory); void registerDataTypeDomainIPv4AndIPv6(DataTypeFactory & factory); void registerDataTypeDomainSimpleAggregateFunction(DataTypeFactory & factory); -void registerDataTypeDateTime64(DataTypeFactory & factory); void registerDataTypeDomainGeo(DataTypeFactory & factory); } diff --git a/src/DataTypes/NumberTraits.h b/src/DataTypes/NumberTraits.h index 8c70306ba5d..603449150db 100644 --- a/src/DataTypes/NumberTraits.h +++ b/src/DataTypes/NumberTraits.h @@ -28,21 +28,13 @@ constexpr size_t min(size_t x, size_t y) return x < y ? x : y; } +/// @note There's no auto scale to larger big integer, only for integral ones. +/// It's cause of (U)Int64 backward compatibilty and very big performance penalties. constexpr size_t nextSize(size_t size) { - return min(size * 2, 8); -} - -template -constexpr size_t nextSize2(size_t size) -{ - // old way for built-in integers - if (size <= 8) return nextSize(size); - - if constexpr (is_signed) - return size <= 32 ? 32 : 48; - else - return size <= 32 ? 16 : 48; + if (size < 8) + return size * 2; + return size; } template @@ -55,9 +47,8 @@ template <> struct Construct { using Type = UInt8; }; template <> struct Construct { using Type = UInt16; }; template <> struct Construct { using Type = UInt32; }; template <> struct Construct { using Type = UInt64; }; -template <> struct Construct { using Type = UInt256; }; +template <> struct Construct { using Type = UInt256; }; /// TODO: we cannot use our UInt128 here template <> struct Construct { using Type = UInt256; }; -template <> struct Construct { using Type = UInt256; }; template <> struct Construct { using Type = Float32; }; template <> struct Construct { using Type = Float32; }; template <> struct Construct { using Type = Float32; }; @@ -67,8 +58,7 @@ template <> struct Construct { using Type = Int16; }; template <> struct Construct { using Type = Int32; }; template <> struct Construct { using Type = Int64; }; template <> struct Construct { using Type = Int128; }; -template <> struct Construct { using Type = Int128; }; -template <> struct Construct { using Type = Int256; }; +template <> struct Construct { using Type = Int256; }; template <> struct Construct { using Type = Float32; }; template <> struct Construct { using Type = Float32; }; template <> struct Construct { using Type = Float32; }; @@ -86,7 +76,7 @@ template struct ResultOfAdditionMultiplication using Type = typename Construct< is_signed_v || is_signed_v, std::is_floating_point_v || std::is_floating_point_v, - nextSize2< is_signed_v || is_signed_v >(max(sizeof(A), sizeof(B)))>::Type; + nextSize(max(sizeof(A), sizeof(B)))>::Type; }; template struct ResultOfSubtraction @@ -94,7 +84,7 @@ template struct ResultOfSubtraction using Type = typename Construct< true, std::is_floating_point_v || std::is_floating_point_v, - nextSize2< is_signed_v || is_signed_v >(max(sizeof(A), sizeof(B)))>::Type; + nextSize(max(sizeof(A), sizeof(B)))>::Type; }; /** When dividing, you always get a floating-point number. @@ -127,7 +117,7 @@ template struct ResultOfNegate using Type = typename Construct< true, std::is_floating_point_v, - is_signed_v ? sizeof(A) : nextSize2(sizeof(A))>::Type; + is_signed_v ? sizeof(A) : nextSize(sizeof(A))>::Type; }; template struct ResultOfAbs diff --git a/src/DataTypes/registerDataTypeDateTime.cpp b/src/DataTypes/registerDataTypeDateTime.cpp new file mode 100644 index 00000000000..815948c6531 --- /dev/null +++ b/src/DataTypes/registerDataTypeDateTime.cpp @@ -0,0 +1,118 @@ + +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; +} + +enum class ArgumentKind +{ + Optional, + Mandatory +}; + +String getExceptionMessage( + const String & message, size_t argument_index, const char * argument_name, + const std::string & context_data_type_name, Field::Types::Which field_type) +{ + return std::string("Parameter #") + std::to_string(argument_index) + " '" + + argument_name + "' for " + context_data_type_name + + message + + ", expected: " + Field::Types::toString(field_type) + " literal."; +} + +template +std::conditional_t, T> +getArgument(const ASTPtr & arguments, size_t argument_index, const char * argument_name [[maybe_unused]], const std::string context_data_type_name) +{ + using NearestResultType = NearestFieldType; + const auto field_type = Field::TypeToEnum::value; + const ASTLiteral * argument = nullptr; + + if (!arguments || arguments->children.size() <= argument_index + || !(argument = arguments->children[argument_index]->as()) + || argument->value.getType() != field_type) + { + if constexpr (Kind == ArgumentKind::Optional) + return {}; + else + { + if (argument && argument->value.getType() != field_type) + throw Exception(getExceptionMessage(String(" has wrong type: ") + argument->value.getTypeName(), + argument_index, argument_name, context_data_type_name, field_type), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + else + throw Exception(getExceptionMessage(" is missing", argument_index, argument_name, context_data_type_name, field_type), + ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); + } + } + + return argument->value.get(); +} + +static DataTypePtr create(const ASTPtr & arguments) +{ + if (!arguments || arguments->children.empty()) + return std::make_shared(); + + const auto scale = getArgument(arguments, 0, "scale", "DateTime"); + const auto timezone = getArgument(arguments, !!scale, "timezone", "DateTime"); + + if (!scale && !timezone) + throw Exception(getExceptionMessage(" has wrong type: ", 0, "scale", "DateTime", Field::Types::Which::UInt64), + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + + /// If scale is defined, the data type is DateTime when scale = 0 otherwise the data type is DateTime64 + if (scale && scale.value() != 0) + return std::make_shared(scale.value(), timezone.value_or(String{})); + + return std::make_shared(timezone.value_or(String{})); +} + +static DataTypePtr create32(const ASTPtr & arguments) +{ + if (!arguments || arguments->children.empty()) + return std::make_shared(); + + if (arguments->children.size() != 1) + throw Exception("DateTime32 data type can optionally have only one argument - time zone name", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); + + const auto timezone = getArgument(arguments, 0, "timezone", "DateTime32"); + + return std::make_shared(timezone); +} + +static DataTypePtr create64(const ASTPtr & arguments) +{ + if (!arguments || arguments->children.empty()) + return std::make_shared(DataTypeDateTime64::default_scale); + + if (arguments->children.size() > 2) + throw Exception("DateTime64 data type can optionally have two argument - scale and time zone name", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); + + const auto scale = getArgument(arguments, 0, "scale", "DateTime64"); + const auto timezone = getArgument(arguments, 1, "timezone", "DateTime64"); + + return std::make_shared(scale, timezone.value_or(String{})); +} + +void registerDataTypeDateTime(DataTypeFactory & factory) +{ + factory.registerDataType("DateTime", create, DataTypeFactory::CaseInsensitive); + factory.registerDataType("DateTime32", create32, DataTypeFactory::CaseInsensitive); + factory.registerDataType("DateTime64", create64, DataTypeFactory::CaseInsensitive); + + factory.registerAlias("TIMESTAMP", "DateTime", DataTypeFactory::CaseInsensitive); +} + +} diff --git a/src/DataTypes/ya.make b/src/DataTypes/ya.make index 82e9baf76f2..4237ca920ae 100644 --- a/src/DataTypes/ya.make +++ b/src/DataTypes/ya.make @@ -38,6 +38,7 @@ SRCS( getMostSubtype.cpp IDataType.cpp NestedUtils.cpp + registerDataTypeDateTime.cpp ) diff --git a/src/Databases/DatabaseOnDisk.cpp b/src/Databases/DatabaseOnDisk.cpp index b69454e3b83..a2a2dd992ed 100644 --- a/src/Databases/DatabaseOnDisk.cpp +++ b/src/Databases/DatabaseOnDisk.cpp @@ -221,6 +221,11 @@ void DatabaseOnDisk::dropTable(const Context & context, const String & table_nam throw Exception(ErrorCodes::LOGICAL_ERROR, "Path is empty"); StoragePtr table = detachTable(table_name); + + /// This is possible for Lazy database. + if (!table) + return; + bool renamed = false; try { diff --git a/src/Dictionaries/ExternalQueryBuilder.cpp b/src/Dictionaries/ExternalQueryBuilder.cpp index b682aaeb557..e8d71b1fd85 100644 --- a/src/Dictionaries/ExternalQueryBuilder.cpp +++ b/src/Dictionaries/ExternalQueryBuilder.cpp @@ -13,6 +13,7 @@ namespace DB namespace ErrorCodes { extern const int UNSUPPORTED_METHOD; + extern const int LOGICAL_ERROR; } @@ -239,12 +240,15 @@ std::string ExternalQueryBuilder::composeLoadIdsQuery(const std::vector } -std::string -ExternalQueryBuilder::composeLoadKeysQuery(const Columns & key_columns, const std::vector & requested_rows, LoadKeysMethod method, size_t partition_key_prefix) +std::string ExternalQueryBuilder::composeLoadKeysQuery( + const Columns & key_columns, const std::vector & requested_rows, LoadKeysMethod method, size_t partition_key_prefix) { if (!dict_struct.key) throw Exception{"Composite key required for method", ErrorCodes::UNSUPPORTED_METHOD}; + if (key_columns.size() != dict_struct.key->size()) + throw Exception{"The size of key_columns does not equal to the size of dictionary key", ErrorCodes::LOGICAL_ERROR}; + WriteBufferFromOwnString out; writeString("SELECT ", out); diff --git a/src/Dictionaries/RedisDictionarySource.cpp b/src/Dictionaries/RedisDictionarySource.cpp index 27acd723938..89ed19cd8a8 100644 --- a/src/Dictionaries/RedisDictionarySource.cpp +++ b/src/Dictionaries/RedisDictionarySource.cpp @@ -52,12 +52,14 @@ namespace DB const String & host_, UInt16 port_, UInt8 db_index_, + const String & password_, RedisStorageType storage_type_, const Block & sample_block_) : dict_struct{dict_struct_} , host{host_} , port{port_} , db_index{db_index_} + , password{password_} , storage_type{storage_type_} , sample_block{sample_block_} , client{std::make_shared(host, port)} @@ -78,16 +80,22 @@ namespace DB ErrorCodes::INVALID_CONFIG_PARAMETER}; // suppose key[0] is primary key, key[1] is secondary key } + if (!password.empty()) + { + RedisCommand command("AUTH"); + command << password; + String reply = client->execute(command); + if (reply != "OK") + throw Exception{"Authentication failed with reason " + + reply, ErrorCodes::INTERNAL_REDIS_ERROR}; + } if (db_index != 0) { RedisCommand command("SELECT"); - // Use poco's Int64, because it is defined as long long, and on - // MacOS, for the purposes of template instantiation, this type is - // distinct from int64_t, which is our Int64. - command << static_cast(db_index); + command << std::to_string(db_index); String reply = client->execute(command); - if (reply != "+OK\r\n") + if (reply != "OK") throw Exception{"Selecting database with index " + DB::toString(db_index) + " failed with reason " + reply, ErrorCodes::INTERNAL_REDIS_ERROR}; } @@ -104,6 +112,7 @@ namespace DB config_.getString(config_prefix_ + ".host"), config_.getUInt(config_prefix_ + ".port"), config_.getUInt(config_prefix_ + ".db_index", 0), + config_.getString(config_prefix_ + ".password",""), parseStorageType(config_.getString(config_prefix_ + ".storage_type", "")), sample_block_) { @@ -115,6 +124,7 @@ namespace DB other.host, other.port, other.db_index, + other.password, other.storage_type, other.sample_block} { diff --git a/src/Dictionaries/RedisDictionarySource.h b/src/Dictionaries/RedisDictionarySource.h index b30c428cb2d..75dcc2fb081 100644 --- a/src/Dictionaries/RedisDictionarySource.h +++ b/src/Dictionaries/RedisDictionarySource.h @@ -41,6 +41,7 @@ namespace ErrorCodes const std::string & host, UInt16 port, UInt8 db_index, + const std::string & password, RedisStorageType storage_type, const Block & sample_block); @@ -91,6 +92,7 @@ namespace ErrorCodes const std::string host; const UInt16 port; const UInt8 db_index; + const std::string password; const RedisStorageType storage_type; Block sample_block; diff --git a/src/Dictionaries/SSDComplexKeyCacheDictionary.cpp b/src/Dictionaries/SSDComplexKeyCacheDictionary.cpp index 826a61f7312..972d10da24d 100644 --- a/src/Dictionaries/SSDComplexKeyCacheDictionary.cpp +++ b/src/Dictionaries/SSDComplexKeyCacheDictionary.cpp @@ -1120,6 +1120,8 @@ void SSDComplexKeyCacheStorage::update( AbsentIdHandler && on_key_not_found, const DictionaryLifetime lifetime) { + assert(key_columns.size() == key_types.size()); + auto append_block = [&key_types, this]( const Columns & new_keys, const SSDComplexKeyCachePartition::Attributes & new_attributes, @@ -1447,6 +1449,12 @@ void SSDComplexKeyCacheDictionary::getItemsNumberImpl( const Columns & key_columns, const DataTypes & key_types, ResultArrayType & out, DefaultGetter && get_default) const { + assert(dict_struct.key); + assert(key_columns.size() == key_types.size()); + assert(key_columns.size() == dict_struct.key->size()); + + dict_struct.validateKeyTypes(key_types); + const auto now = std::chrono::system_clock::now(); TemporalComplexKeysPool not_found_pool; @@ -1527,6 +1535,8 @@ void SSDComplexKeyCacheDictionary::getItemsStringImpl( ColumnString * out, DefaultGetter && get_default) const { + dict_struct.validateKeyTypes(key_types); + const auto now = std::chrono::system_clock::now(); TemporalComplexKeysPool not_found_pool; diff --git a/src/Dictionaries/SSDComplexKeyCacheDictionary.h b/src/Dictionaries/SSDComplexKeyCacheDictionary.h index 89e88982eee..af9a0c0a7ee 100644 --- a/src/Dictionaries/SSDComplexKeyCacheDictionary.h +++ b/src/Dictionaries/SSDComplexKeyCacheDictionary.h @@ -427,9 +427,8 @@ private: using SSDComplexKeyCachePartitionPtr = std::shared_ptr; -/* - Class for managing SSDCachePartition and getting data from source. -*/ +/** Class for managing SSDCachePartition and getting data from source. + */ class SSDComplexKeyCacheStorage { public: @@ -515,9 +514,8 @@ private: }; -/* - Dictionary interface -*/ +/** Dictionary interface + */ class SSDComplexKeyCacheDictionary final : public IDictionaryBase { public: diff --git a/src/Functions/DivisionUtils.h b/src/Functions/DivisionUtils.h index 99fd7795bb8..7a816df70e5 100644 --- a/src/Functions/DivisionUtils.h +++ b/src/Functions/DivisionUtils.h @@ -81,8 +81,10 @@ struct DivideIntegralImpl /// NOTE: overflow is still possible when dividing large signed number to large unsigned number or vice-versa. But it's less harmful. if constexpr (is_integer_v && is_integer_v && (is_signed_v || is_signed_v)) { - return checkedDivision(make_signed_t(a), - sizeof(A) > sizeof(B) ? make_signed_t(CastB(b)) : make_signed_t(b)); + using SignedCastA = make_signed_t; + using SignedCastB = std::conditional_t, SignedCastA>; + + return bigint_cast(checkedDivision(bigint_cast(a), bigint_cast(b))); } else return bigint_cast(checkedDivision(CastA(a), CastB(b))); @@ -108,7 +110,7 @@ struct ModuloImpl if constexpr (std::is_floating_point_v) { /// This computation is similar to `fmod` but the latter is not inlined and has 40 times worse performance. - return ResultType(a) - trunc(ResultType(a) / ResultType(b)) * ResultType(b); + return bigint_cast(a) - trunc(bigint_cast(a) / bigint_cast(b)) * bigint_cast(b); } else { @@ -125,7 +127,7 @@ struct ModuloImpl if constexpr (is_big_int_v && sizeof(IntegerAType) <= sizeof(IntegerBType)) return bigint_cast(bigint_cast(int_a) % int_b); else - return bigint_cast(int_a % int_b); + return bigint_cast(int_a % bigint_cast(int_b)); } else return IntegerAType(a) % IntegerBType(b); diff --git a/src/Functions/FunctionBinaryArithmetic.h b/src/Functions/FunctionBinaryArithmetic.h index 241f7b2fae0..2a467451684 100644 --- a/src/Functions/FunctionBinaryArithmetic.h +++ b/src/Functions/FunctionBinaryArithmetic.h @@ -361,12 +361,8 @@ private: return apply(a.value, b); else if constexpr (IsDecimalNumber) return apply(a, b.value); - else if constexpr (std::is_same_v) - return apply(UInt16(a), b); - else if constexpr (std::is_same_v) - return apply(a, UInt16(b)); else - return applyNative(static_cast(a), static_cast(b)); + return applyNative(bigint_cast(a), bigint_cast(b)); } else return applyNative(a, b); @@ -381,12 +377,8 @@ private: return applyScaled(a.value, b, scale); else if constexpr (IsDecimalNumber) return applyScaled(a, b.value, scale); - else if constexpr (std::is_same_v) - return applyScaled(UInt16(a), b, scale); - else if constexpr (std::is_same_v) - return applyScaled(a, UInt16(b), scale); else - return applyNativeScaled(static_cast(a), static_cast(b), scale); + return applyNativeScaled(bigint_cast(a), bigint_cast(b), scale); } else return applyNativeScaled(a, b, scale); @@ -401,12 +393,8 @@ private: return applyScaledDiv(a.value, b, scale); else if constexpr (IsDecimalNumber) return applyScaledDiv(a, b.value, scale); - else if constexpr (std::is_same_v) - return applyScaledDiv(UInt16(a), b, scale); - else if constexpr (std::is_same_v) - return applyScaledDiv(a, UInt16(b), scale); else - return applyNativeScaledDiv(static_cast(a), static_cast(b), scale); + return applyNativeScaledDiv(bigint_cast(a), bigint_cast(b), scale); } else return applyNativeScaledDiv(a, b, scale); diff --git a/src/Functions/FunctionsConversion.cpp b/src/Functions/FunctionsConversion.cpp index 535dbb328f1..3f38614f584 100644 --- a/src/Functions/FunctionsConversion.cpp +++ b/src/Functions/FunctionsConversion.cpp @@ -36,6 +36,7 @@ void registerFunctionsConversion(FunctionFactory & factory) factory.registerFunction(); factory.registerFunction(); + factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); @@ -93,6 +94,9 @@ void registerFunctionsConversion(FunctionFactory & factory) factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); + factory.registerFunction(); + factory.registerFunction(); + factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); factory.registerFunction(); diff --git a/src/Functions/FunctionsConversion.h b/src/Functions/FunctionsConversion.h index 125aec860fe..a18139fd4c8 100644 --- a/src/Functions/FunctionsConversion.h +++ b/src/Functions/FunctionsConversion.h @@ -977,6 +977,7 @@ struct ConvertImpl /// Declared early because used below. struct NameToDate { static constexpr auto name = "toDate"; }; struct NameToDateTime { static constexpr auto name = "toDateTime"; }; +struct NameToDateTime32 { static constexpr auto name = "toDateTime32"; }; struct NameToDateTime64 { static constexpr auto name = "toDateTime64"; }; struct NameToString { static constexpr auto name = "toString"; }; struct NameToDecimal32 { static constexpr auto name = "toDecimal32"; }; @@ -1003,6 +1004,26 @@ DEFINE_NAME_TO_INTERVAL(Year) #undef DEFINE_NAME_TO_INTERVAL +struct NameParseDateTimeBestEffort; +struct NameParseDateTimeBestEffortOrZero; +struct NameParseDateTimeBestEffortOrNull; + +template +static inline bool isDateTime64(const ColumnsWithTypeAndName & arguments, const ColumnNumbers & arguments_index = {}) +{ + if constexpr (std::is_same_v) + return true; + else if constexpr (std::is_same_v || std::is_same_v + || std::is_same_v || std::is_same_v) + { + if (arguments_index.empty()) + return (arguments.size() == 2 && isUnsignedInteger(arguments[1].type)) || arguments.size() == 3; + else + return (arguments_index.size() == 2 && isUnsignedInteger(arguments[arguments_index[1]].type)) || arguments_index.size() == 3; + } + + return false; +} template class FunctionConvert : public IFunction @@ -1034,10 +1055,16 @@ public: FunctionArgumentDescriptors mandatory_args = {{"Value", nullptr, nullptr, nullptr}}; FunctionArgumentDescriptors optional_args; - if constexpr (to_decimal || to_datetime64) + if constexpr (to_decimal) { mandatory_args.push_back({"scale", &isNativeInteger, &isColumnConst, "const Integer"}); } + + if (!to_decimal && isDateTime64(arguments)) + { + mandatory_args.push_back({"scale", &isNativeInteger, &isColumnConst, "const Integer"}); + } + // toString(DateTime or DateTime64, [timezone: String]) if ((std::is_same_v && arguments.size() > 0 && (isDateTime64(arguments[0].type) || isDateTime(arguments[0].type))) // toUnixTimestamp(value[, timezone : String]) @@ -1080,16 +1107,22 @@ public: UInt32 scale [[maybe_unused]] = DataTypeDateTime64::default_scale; // DateTime64 requires more arguments: scale and timezone. Since timezone is optional, scale should be first. - if constexpr (to_datetime64) + if (isDateTime64(arguments)) { timezone_arg_position += 1; scale = static_cast(arguments[1].column->get64(0)); + + if (to_datetime64 || scale != 0) /// toDateTime('xxxx-xx-xx xx:xx:xx', 0) return DateTime + return std::make_shared(scale, + extractTimeZoneNameFromFunctionArguments(arguments, timezone_arg_position, 0)); + + return std::make_shared(extractTimeZoneNameFromFunctionArguments(arguments, timezone_arg_position, 0)); } if constexpr (std::is_same_v) return std::make_shared(extractTimeZoneNameFromFunctionArguments(arguments, timezone_arg_position, 0)); - else if constexpr (to_datetime64) - return std::make_shared(scale, extractTimeZoneNameFromFunctionArguments(arguments, timezone_arg_position, 0)); + else if constexpr (std::is_same_v) + throw Exception("LOGICAL ERROR: It is a bug.", ErrorCodes::LOGICAL_ERROR); else return std::make_shared(); } @@ -1208,6 +1241,22 @@ private: return true; }; + if (isDateTime64(block.getColumnsWithTypeAndName(), arguments)) + { + /// For toDateTime('xxxx-xx-xx xx:xx:xx.00', 2[, 'timezone']) we need to it convert to DateTime64 + const ColumnWithTypeAndName & scale_column = block.getByPosition(arguments[1]); + UInt32 scale = extractToDecimalScale(scale_column); + + if (to_datetime64 || scale != 0) /// When scale = 0, the data type is DateTime otherwise the data type is DateTime64 + { + if (!callOnIndexAndDataType(from_type->getTypeId(), call)) + throw Exception("Illegal type " + block.getByPosition(arguments[0]).type->getName() + " of argument of function " + getName(), + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + + return; + } + } + bool done = callOnIndexAndDataType(from_type->getTypeId(), call); if (!done) { @@ -1262,7 +1311,8 @@ public: DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override { DataTypePtr res; - if constexpr (to_datetime64) + + if (isDateTime64(arguments)) { validateFunctionArgumentTypes(*this, arguments, FunctionArgumentDescriptors{{"string", isStringOrFixedString, nullptr, "String or FixedString"}}, @@ -1272,11 +1322,12 @@ public: {"timezone", isStringOrFixedString, isColumnConst, "const String or FixedString"}, }); - UInt64 scale = DataTypeDateTime64::default_scale; + UInt64 scale = to_datetime64 ? DataTypeDateTime64::default_scale : 0; if (arguments.size() > 1) scale = extractToDecimalScale(arguments[1]); const auto timezone = extractTimeZoneNameFromFunctionArguments(arguments, 2, 0); - res = std::make_shared(scale, timezone); + + res = scale == 0 ? res = std::make_shared(timezone) : std::make_shared(scale, timezone); } else { @@ -1323,6 +1374,8 @@ public: if constexpr (std::is_same_v) res = std::make_shared(extractTimeZoneNameFromFunctionArguments(arguments, 1, 0)); + else if constexpr (std::is_same_v) + throw Exception("LOGICAL ERROR: It is a bug.", ErrorCodes::LOGICAL_ERROR); else if constexpr (to_decimal) { UInt64 scale = extractToDecimalScale(arguments[1]); @@ -1340,42 +1393,53 @@ public: return res; } - void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) const override + template + bool executeInternal(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count, UInt32 scale = 0) const { const IDataType * from_type = block.getByPosition(arguments[0]).type.get(); - bool ok = true; - if constexpr (to_decimal || to_datetime64) + if (checkAndGetDataType(from_type)) { - const UInt32 scale = assert_cast(*removeNullable(block.getByPosition(result).type)).getScale(); - - if (checkAndGetDataType(from_type)) - { - ConvertThroughParsing::execute( - block, arguments, result, input_rows_count, scale); - } - else if (checkAndGetDataType(from_type)) - { - ConvertThroughParsing::execute( - block, arguments, result, input_rows_count, scale); - } - else - ok = false; + ConvertThroughParsing::execute( + block, arguments, result, input_rows_count, scale); + return true; } + else if (checkAndGetDataType(from_type)) + { + ConvertThroughParsing::execute( + block, arguments, result, input_rows_count, scale); + return true; + } + + return false; + } + + void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) const override + { + bool ok = true; + + if constexpr (to_decimal) + ok = executeInternal(block, arguments, result, input_rows_count, + assert_cast(*removeNullable(block.getByPosition(result).type)).getScale()); else { - if (checkAndGetDataType(from_type)) + if (isDateTime64(block.getColumnsWithTypeAndName(), arguments)) { - ConvertThroughParsing::execute( - block, arguments, result, input_rows_count); - } - else if (checkAndGetDataType(from_type)) - { - ConvertThroughParsing::execute( - block, arguments, result, input_rows_count); + UInt64 scale = to_datetime64 ? DataTypeDateTime64::default_scale : 0; + if (arguments.size() > 1) + scale = extractToDecimalScale(block.getColumnsWithTypeAndName()[arguments[1]]); + + if (scale == 0) + ok = executeInternal(block, arguments, result, input_rows_count); + else + { + ok = executeInternal(block, arguments, result, input_rows_count, static_cast(scale)); + } } else - ok = false; + { + ok = executeInternal(block, arguments, result, input_rows_count); + } } if (!ok) @@ -1638,6 +1702,7 @@ using FunctionToFloat32 = FunctionConvert>; using FunctionToDate = FunctionConvert; using FunctionToDateTime = FunctionConvert; +using FunctionToDateTime32 = FunctionConvert; using FunctionToDateTime64 = FunctionConvert; using FunctionToUUID = FunctionConvert>; using FunctionToString = FunctionConvert; @@ -1767,6 +1832,9 @@ struct NameParseDateTimeBestEffort { static constexpr auto name = "parseDateTime struct NameParseDateTimeBestEffortUS { static constexpr auto name = "parseDateTimeBestEffortUS"; }; struct NameParseDateTimeBestEffortOrZero { static constexpr auto name = "parseDateTimeBestEffortOrZero"; }; struct NameParseDateTimeBestEffortOrNull { static constexpr auto name = "parseDateTimeBestEffortOrNull"; }; +struct NameParseDateTime32BestEffort { static constexpr auto name = "parseDateTime32BestEffort"; }; +struct NameParseDateTime32BestEffortOrZero { static constexpr auto name = "parseDateTime32BestEffortOrZero"; }; +struct NameParseDateTime32BestEffortOrNull { static constexpr auto name = "parseDateTime32BestEffortOrNull"; }; struct NameParseDateTime64BestEffort { static constexpr auto name = "parseDateTime64BestEffort"; }; struct NameParseDateTime64BestEffortOrZero { static constexpr auto name = "parseDateTime64BestEffortOrZero"; }; struct NameParseDateTime64BestEffortOrNull { static constexpr auto name = "parseDateTime64BestEffortOrNull"; }; @@ -1781,6 +1849,13 @@ using FunctionParseDateTimeBestEffortOrZero = FunctionConvertFromString< using FunctionParseDateTimeBestEffortOrNull = FunctionConvertFromString< DataTypeDateTime, NameParseDateTimeBestEffortOrNull, ConvertFromStringExceptionMode::Null, ConvertFromStringParsingMode::BestEffort>; +using FunctionParseDateTime32BestEffort = FunctionConvertFromString< + DataTypeDateTime, NameParseDateTime32BestEffort, ConvertFromStringExceptionMode::Throw, ConvertFromStringParsingMode::BestEffort>; +using FunctionParseDateTime32BestEffortOrZero = FunctionConvertFromString< + DataTypeDateTime, NameParseDateTime32BestEffortOrZero, ConvertFromStringExceptionMode::Zero, ConvertFromStringParsingMode::BestEffort>; +using FunctionParseDateTime32BestEffortOrNull = FunctionConvertFromString< + DataTypeDateTime, NameParseDateTime32BestEffortOrNull, ConvertFromStringExceptionMode::Null, ConvertFromStringParsingMode::BestEffort>; + using FunctionParseDateTime64BestEffort = FunctionConvertFromString< DataTypeDateTime64, NameParseDateTime64BestEffort, ConvertFromStringExceptionMode::Throw, ConvertFromStringParsingMode::BestEffort>; using FunctionParseDateTime64BestEffortOrZero = FunctionConvertFromString< diff --git a/src/Functions/GatherUtils/Algorithms.h b/src/Functions/GatherUtils/Algorithms.h index bf13de0c594..e54538c76b3 100644 --- a/src/Functions/GatherUtils/Algorithms.h +++ b/src/Functions/GatherUtils/Algorithms.h @@ -558,7 +558,7 @@ bool sliceEqualElements(const NumericArraySlice & first [[maybe_unused]], { /// TODO: Decimal scale if constexpr (IsDecimalNumber && IsDecimalNumber) - return accurate::equalsOp(typename T::NativeType(first.data[first_ind]), typename U::NativeType(second.data[second_ind])); + return accurate::equalsOp(first.data[first_ind].value, second.data[second_ind].value); else if constexpr (IsDecimalNumber || IsDecimalNumber) return false; else @@ -588,7 +588,7 @@ bool insliceEqualElements(const NumericArraySlice & first [[maybe_unused]], size_t second_ind [[maybe_unused]]) { if constexpr (IsDecimalNumber) - return accurate::equalsOp(typename T::NativeType(first.data[first_ind]), typename T::NativeType(first.data[second_ind])); + return accurate::equalsOp(first.data[first_ind].value, first.data[second_ind].value); else return accurate::equalsOp(first.data[first_ind], first.data[second_ind]); } diff --git a/src/Functions/URL/tldLookup.sh b/src/Functions/URL/tldLookup.sh index a61f2b09660..a7893c3a168 100755 --- a/src/Functions/URL/tldLookup.sh +++ b/src/Functions/URL/tldLookup.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -[ ! -f public_suffix_list.dat ] && wget -O public_suffix_list.dat https://publicsuffix.org/list/public_suffix_list.dat +[ ! -f public_suffix_list.dat ] && wget -nv -O public_suffix_list.dat https://publicsuffix.org/list/public_suffix_list.dat echo '%language=C++ %define lookup-function-name is_valid diff --git a/src/Functions/abs.cpp b/src/Functions/abs.cpp index a7d31f4e030..f0c530e0e8f 100644 --- a/src/Functions/abs.cpp +++ b/src/Functions/abs.cpp @@ -16,11 +16,10 @@ struct AbsImpl { if constexpr (IsDecimalNumber) return a < A(0) ? A(-a) : a; - else if constexpr (is_big_int_v) - // from boost/multiprecision/number.hpp - return static_cast(abs(a)); + else if constexpr (is_big_int_v && is_signed_v) + return (a < 0) ? -a : a; else if constexpr (is_integer_v && is_signed_v) - return a < 0 ? static_cast(~a) + 1 : a; + return a < 0 ? static_cast(~a) + 1 : static_cast(a); else if constexpr (is_integer_v && is_unsigned_v) return static_cast(a); else if constexpr (std::is_floating_point_v) diff --git a/src/Functions/bitRotateLeft.cpp b/src/Functions/bitRotateLeft.cpp index 3bef0bb5ff3..a6975468c1e 100644 --- a/src/Functions/bitRotateLeft.cpp +++ b/src/Functions/bitRotateLeft.cpp @@ -18,7 +18,7 @@ struct BitRotateLeftImpl template static inline NO_SANITIZE_UNDEFINED Result apply(A a [[maybe_unused]], B b [[maybe_unused]]) { - if constexpr (is_big_int_v) + if constexpr (is_big_int_v || is_big_int_v) throw Exception("Bit rotate is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); else return (static_cast(a) << static_cast(b)) diff --git a/src/Functions/bitRotateRight.cpp b/src/Functions/bitRotateRight.cpp index e94e13a8e60..71d7385bbdf 100644 --- a/src/Functions/bitRotateRight.cpp +++ b/src/Functions/bitRotateRight.cpp @@ -18,7 +18,7 @@ struct BitRotateRightImpl template static inline NO_SANITIZE_UNDEFINED Result apply(A a [[maybe_unused]], B b [[maybe_unused]]) { - if constexpr (is_big_int_v) + if constexpr (is_big_int_v || is_big_int_v) throw Exception("Bit rotate is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); else return (static_cast(a) >> static_cast(b)) diff --git a/src/Functions/bitShiftLeft.cpp b/src/Functions/bitShiftLeft.cpp index 1ea69af73e9..d42082d7778 100644 --- a/src/Functions/bitShiftLeft.cpp +++ b/src/Functions/bitShiftLeft.cpp @@ -19,9 +19,9 @@ struct BitShiftLeftImpl static inline NO_SANITIZE_UNDEFINED Result apply(A a [[maybe_unused]], B b [[maybe_unused]]) { if constexpr (is_big_int_v) - throw Exception("BitShiftLeftImpl is not implemented for big integers as second argument", ErrorCodes::NOT_IMPLEMENTED); + throw Exception("BitShiftLeft is not implemented for big integers as second argument", ErrorCodes::NOT_IMPLEMENTED); else if constexpr (is_big_int_v) - return static_cast(a) << bigint_cast(b); + return bigint_cast(a) << bigint_cast(b); else return static_cast(a) << static_cast(b); } diff --git a/src/Functions/bitShiftRight.cpp b/src/Functions/bitShiftRight.cpp index 25eb86bf353..249a86d6961 100644 --- a/src/Functions/bitShiftRight.cpp +++ b/src/Functions/bitShiftRight.cpp @@ -19,9 +19,9 @@ struct BitShiftRightImpl static inline NO_SANITIZE_UNDEFINED Result apply(A a [[maybe_unused]], B b [[maybe_unused]]) { if constexpr (is_big_int_v) - throw Exception("BitRotate is not implemented for big integers as second argument", ErrorCodes::NOT_IMPLEMENTED); + throw Exception("BitShiftRight is not implemented for big integers as second argument", ErrorCodes::NOT_IMPLEMENTED); else if constexpr (is_big_int_v) - return static_cast(a) >> bigint_cast(b); + return bigint_cast(a) >> bigint_cast(b); else return static_cast(a) >> static_cast(b); } diff --git a/src/Functions/bitTest.cpp b/src/Functions/bitTest.cpp index 19afa1da84b..f34e300d675 100644 --- a/src/Functions/bitTest.cpp +++ b/src/Functions/bitTest.cpp @@ -19,10 +19,8 @@ struct BitTestImpl template NO_SANITIZE_UNDEFINED static inline Result apply(A a [[maybe_unused]], B b [[maybe_unused]]) { - if constexpr (is_big_int_v) + if constexpr (is_big_int_v || is_big_int_v) throw Exception("bitTest is not implemented for big integers as second argument", ErrorCodes::NOT_IMPLEMENTED); - else if constexpr (is_big_int_v) - return bit_test(a, static_cast(b)); else return (typename NumberTraits::ToInteger::Type(a) >> typename NumberTraits::ToInteger::Type(b)) & 1; } diff --git a/src/Functions/gcd.cpp b/src/Functions/gcd.cpp index 4ee39f3f2fd..b5d1ed6e92c 100644 --- a/src/Functions/gcd.cpp +++ b/src/Functions/gcd.cpp @@ -20,7 +20,7 @@ struct GCDImpl template static inline Result apply([[maybe_unused]] A a, [[maybe_unused]] B b) { - if constexpr (is_big_int_v || is_big_int_v) + if constexpr (is_big_int_v || is_big_int_v || is_big_int_v) throw Exception("GCD is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); else { diff --git a/src/Functions/lcm.cpp b/src/Functions/lcm.cpp index c61337c75ad..ceca495ddce 100644 --- a/src/Functions/lcm.cpp +++ b/src/Functions/lcm.cpp @@ -40,14 +40,14 @@ struct LCMImpl static const constexpr bool allow_fixed_string = false; template - static inline std::enable_if_t || is_big_int_v, Result> + static inline std::enable_if_t || is_big_int_v || is_big_int_v, Result> apply([[maybe_unused]] A a, [[maybe_unused]] B b) { throw Exception("LCM is not implemented for big integers", ErrorCodes::NOT_IMPLEMENTED); } template - static inline std::enable_if_t && !is_big_int_v, Result> + static inline std::enable_if_t && !is_big_int_v && !is_big_int_v, Result> apply([[maybe_unused]] A a, [[maybe_unused]] B b) { throwIfDivisionLeadsToFPE(typename NumberTraits::ToInteger::Type(a), typename NumberTraits::ToInteger::Type(b)); diff --git a/src/Functions/roundToExp2.cpp b/src/Functions/roundToExp2.cpp index 12856a7930e..c6b6f672c66 100644 --- a/src/Functions/roundToExp2.cpp +++ b/src/Functions/roundToExp2.cpp @@ -6,6 +6,11 @@ namespace DB { +namespace ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + template inline std::enable_if_t && (sizeof(T) <= sizeof(UInt32)), T> roundDownToPowerOfTwo(T x) @@ -48,10 +53,9 @@ roundDownToPowerOfTwo(T x) template inline std::enable_if_t, T> -roundDownToPowerOfTwo(T x) +roundDownToPowerOfTwo(T) { - // extention from boost/multiprecision/number.hpp - return T(1) << msb(x); + throw Exception("roundToExp2() for big integers is not implemented", ErrorCodes::NOT_IMPLEMENTED); } /** For integer data types: diff --git a/src/IO/WriteHelpers.h b/src/IO/WriteHelpers.h index fb7c4e0cd80..3b9eced09bd 100644 --- a/src/IO/WriteHelpers.h +++ b/src/IO/WriteHelpers.h @@ -831,6 +831,7 @@ template <> inline void writeText(const bool & x, WriteBuffer & buf) { wri inline void writeText(const char * x, WriteBuffer & buf) { writeEscapedString(x, strlen(x), buf); } inline void writeText(const char * x, size_t size, WriteBuffer & buf) { writeEscapedString(x, size, buf); } +inline void writeText(const DayNum & x, WriteBuffer & buf) { writeDateText(LocalDate(x), buf); } inline void writeText(const LocalDate & x, WriteBuffer & buf) { writeDateText(x, buf); } inline void writeText(const LocalDateTime & x, WriteBuffer & buf) { writeDateTimeText(x, buf); } inline void writeText(const UUID & x, WriteBuffer & buf) { writeUUIDText(x, buf); } diff --git a/src/IO/readDecimalText.h b/src/IO/readDecimalText.h index c69a56dd83e..727dd67c389 100644 --- a/src/IO/readDecimalText.h +++ b/src/IO/readDecimalText.h @@ -160,7 +160,7 @@ inline void readDecimalText(ReadBuffer & buf, T & x, uint32_t precision, uint32_ " Expected to read decimal with scale {} and precision {}"; if constexpr (is_big_int_v) - throw Exception(fmt::format(pattern, digits, x.value.str(), exponent, scale, precision), ErrorCodes::ARGUMENT_OUT_OF_BOUND); + throw Exception(fmt::format(pattern, digits, bigintToString(x.value), exponent, scale, precision), ErrorCodes::ARGUMENT_OUT_OF_BOUND); else throw Exception(fmt::format(pattern, digits, x, exponent, scale, precision), ErrorCodes::ARGUMENT_OUT_OF_BOUND); } @@ -180,7 +180,7 @@ inline void readDecimalText(ReadBuffer & buf, T & x, uint32_t precision, uint32_ { /// Too many digits after point. Just cut off excessive digits. auto divisor = intExp10OfSize(divisor_exp); - assert(divisor > T(0)); /// This is for Clang Static Analyzer. It is not smart enough to infer it automatically. + assert(divisor > 0); /// This is for Clang Static Analyzer. It is not smart enough to infer it automatically. x.value /= divisor; scale = 0; return; diff --git a/src/Interpreters/ActionsVisitor.cpp b/src/Interpreters/ActionsVisitor.cpp index f2a1d570773..0df83f11c1f 100644 --- a/src/Interpreters/ActionsVisitor.cpp +++ b/src/Interpreters/ActionsVisitor.cpp @@ -447,6 +447,19 @@ void ScopeStack::addAction(const ExpressionAction & action) } } +void ScopeStack::addActionNoInput(const ExpressionAction & action) +{ + size_t level = 0; + Names required = action.getNeededColumns(); + for (const auto & elem : required) + level = std::max(level, getColumnLevel(elem)); + + Names added; + stack[level].actions->add(action, added); + + stack[level].new_columns.insert(added.begin(), added.end()); +} + ExpressionActionsPtr ScopeStack::popLevel() { ExpressionActionsPtr res = stack.back().actions; @@ -549,7 +562,7 @@ void ActionsMatcher::visit(const ASTFunction & node, const ASTPtr & ast, Data & /// It could have been possible to implement arrayJoin which keeps source column, /// but in this case it will always be replicated (as many arrays), which is expensive. String tmp_name = data.getUniqueName("_array_join_" + arg->getColumnName()); - data.addAction(ExpressionAction::copyColumn(arg->getColumnName(), tmp_name)); + data.addActionNoInput(ExpressionAction::copyColumn(arg->getColumnName(), tmp_name)); data.addAction(ExpressionAction::arrayJoin(tmp_name, result_name)); } diff --git a/src/Interpreters/ActionsVisitor.h b/src/Interpreters/ActionsVisitor.h index dbcc54c01d6..d8d85f1c0bf 100644 --- a/src/Interpreters/ActionsVisitor.h +++ b/src/Interpreters/ActionsVisitor.h @@ -12,6 +12,7 @@ namespace DB class Context; class ASTFunction; +struct ExpressionAction; class ExpressionActions; using ExpressionActionsPtr = std::shared_ptr; @@ -49,6 +50,8 @@ struct ScopeStack size_t getColumnLevel(const std::string & name); void addAction(const ExpressionAction & action); + /// For arrayJoin() to avoid double columns in the input. + void addActionNoInput(const ExpressionAction & action); ExpressionActionsPtr popLevel(); @@ -115,6 +118,10 @@ public: { actions_stack.addAction(action); } + void addActionNoInput(const ExpressionAction & action) + { + actions_stack.addActionNoInput(action); + } const Block & getSampleBlock() const { diff --git a/src/Interpreters/Aggregator.cpp b/src/Interpreters/Aggregator.cpp index c011aac1349..86a33dccb53 100644 --- a/src/Interpreters/Aggregator.cpp +++ b/src/Interpreters/Aggregator.cpp @@ -362,7 +362,9 @@ AggregatedDataVariants::Type Aggregator::chooseAggregationMethod() return AggregatedDataVariants::Type::key64; if (size_of_field == 16) return AggregatedDataVariants::Type::keys128; - throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16.", ErrorCodes::LOGICAL_ERROR); + if (size_of_field == 32) + return AggregatedDataVariants::Type::keys256; + throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16, 32.", ErrorCodes::LOGICAL_ERROR); } /// If all keys fits in N bits, will use hash table with all keys packed (placed contiguously) to single N-bit key. diff --git a/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp b/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp index 986de85d712..ed7bd2cf71f 100644 --- a/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp +++ b/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp @@ -13,6 +13,7 @@ #include #include #include +#include namespace ProfileEvents { @@ -68,14 +69,19 @@ SelectStreamFactory::SelectStreamFactory( namespace { -QueryPipeline createLocalStream( +auto createLocalPipe( const ASTPtr & query_ast, const Block & header, const Context & context, QueryProcessingStage::Enum processed_stage) { checkStackSize(); - InterpreterSelectQuery interpreter{query_ast, context, SelectQueryOptions(processed_stage)}; + InterpreterSelectQuery interpreter(query_ast, context, SelectQueryOptions(processed_stage)); + auto query_plan = std::make_unique(); - auto pipeline = interpreter.execute().pipeline; + interpreter.buildQueryPlan(*query_plan); + auto pipeline = std::move(*query_plan->buildQueryPipeline()); + + /// Avoid going it out-of-scope for EXPLAIN + pipeline.addQueryPlan(std::move(query_plan)); pipeline.addSimpleTransform([&](const Block & source_header) { @@ -94,7 +100,7 @@ QueryPipeline createLocalStream( /// return std::make_shared(stream); pipeline.setMaxThreads(1); - return pipeline; + return QueryPipeline::getPipe(std::move(pipeline)); } String formattedAST(const ASTPtr & ast) @@ -130,7 +136,7 @@ void SelectStreamFactory::createForShard( auto emplace_local_stream = [&]() { - pipes.emplace_back(QueryPipeline::getPipe(createLocalStream(modified_query_ast, header, context, processed_stage))); + pipes.emplace_back(createLocalPipe(modified_query_ast, header, context, processed_stage)); }; String modified_query = formattedAST(modified_query_ast); @@ -270,7 +276,7 @@ void SelectStreamFactory::createForShard( } if (try_results.empty() || local_delay < max_remote_delay) - return QueryPipeline::getPipe(createLocalStream(modified_query_ast, header, context, stage)); + return createLocalPipe(modified_query_ast, header, context, stage); else { std::vector connections; diff --git a/src/Interpreters/DNSCacheUpdater.cpp b/src/Interpreters/DNSCacheUpdater.cpp index e5a97dc76d9..248c0ffa4dd 100644 --- a/src/Interpreters/DNSCacheUpdater.cpp +++ b/src/Interpreters/DNSCacheUpdater.cpp @@ -42,6 +42,7 @@ void DNSCacheUpdater::run() void DNSCacheUpdater::start() { + LOG_INFO(&Poco::Logger::get("DNSCacheUpdater"), "Update period {} seconds", update_period_seconds); task_handle->activateAndSchedule(); } diff --git a/src/Interpreters/HashJoin.cpp b/src/Interpreters/HashJoin.cpp index e3f265af004..9818f89d13c 100644 --- a/src/Interpreters/HashJoin.cpp +++ b/src/Interpreters/HashJoin.cpp @@ -221,7 +221,9 @@ HashJoin::Type HashJoin::chooseMethod(const ColumnRawPtrs & key_columns, Sizes & return Type::key64; if (size_of_field == 16) return Type::keys128; - throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16.", ErrorCodes::LOGICAL_ERROR); + if (size_of_field == 32) + return Type::keys256; + throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16, 32.", ErrorCodes::LOGICAL_ERROR); } /// If the keys fit in N bits, we will use a hash table for N-bit-packed keys diff --git a/src/Interpreters/InterpreterAlterQuery.cpp b/src/Interpreters/InterpreterAlterQuery.cpp index e0313803e9a..8cf581eb463 100644 --- a/src/Interpreters/InterpreterAlterQuery.cpp +++ b/src/Interpreters/InterpreterAlterQuery.cpp @@ -101,7 +101,7 @@ BlockIO InterpreterAlterQuery::execute() switch (command.type) { case LiveViewCommand::REFRESH: - live_view->refresh(context); + live_view->refresh(); break; } } diff --git a/src/Interpreters/InterpreterExplainQuery.cpp b/src/Interpreters/InterpreterExplainQuery.cpp index 9960509a5d7..c936556ce39 100644 --- a/src/Interpreters/InterpreterExplainQuery.cpp +++ b/src/Interpreters/InterpreterExplainQuery.cpp @@ -269,7 +269,9 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() if (settings.graph) { - auto processors = Pipe::detachProcessors(QueryPipeline::getPipe(std::move(*pipeline))); + /// Pipe holds QueryPlan, should not go out-of-scope + auto pipe = QueryPipeline::getPipe(std::move(*pipeline)); + const auto & processors = pipe.getProcessors(); if (settings.compact) printPipelineCompact(processors, buffer, settings.query_pipeline_options.header); diff --git a/src/Interpreters/InterpreterGrantQuery.cpp b/src/Interpreters/InterpreterGrantQuery.cpp index 2f468507eb6..57cb701036e 100644 --- a/src/Interpreters/InterpreterGrantQuery.cpp +++ b/src/Interpreters/InterpreterGrantQuery.cpp @@ -29,7 +29,6 @@ namespace current_access.grant(access_to_grant); } - AccessRightsElements getFilteredAccessRightsElementsToRevoke( const AccessRights & current_access, const AccessRightsElements & access_to_revoke, bool grant_option) { @@ -214,6 +213,7 @@ BlockIO InterpreterGrantQuery::execute() auto access = context.getAccess(); auto & access_control = context.getAccessControlManager(); query.replaceEmptyDatabaseWithCurrent(context.getCurrentDatabase()); + query.removeNonGrantableFlags(); RolesOrUsersSet roles_from_query; if (query.roles) diff --git a/src/Interpreters/InterpreterSelectQuery.cpp b/src/Interpreters/InterpreterSelectQuery.cpp index 604bf55649a..c10716aac32 100644 --- a/src/Interpreters/InterpreterSelectQuery.cpp +++ b/src/Interpreters/InterpreterSelectQuery.cpp @@ -553,6 +553,11 @@ Block InterpreterSelectQuery::getSampleBlockImpl() return res; } + if (options.to_stage == QueryProcessingStage::Enum::WithMergeableStateAfterAggregation) + { + return analysis_result.before_order_and_select->getSampleBlock(); + } + return analysis_result.final_projection->getSampleBlock(); } @@ -740,6 +745,8 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu auto & expressions = analysis_result; const auto & subqueries_for_sets = query_analyzer->getSubqueriesForSets(); bool intermediate_stage = false; + bool to_aggregation_stage = false; + bool from_aggregation_stage = false; if (options.only_analyze) { @@ -788,6 +795,14 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu options.to_stage == QueryProcessingStage::WithMergeableState) intermediate_stage = true; + /// Support optimize_distributed_group_by_sharding_key + /// Is running on the initiating server during distributed processing? + if (from_stage == QueryProcessingStage::WithMergeableStateAfterAggregation) + from_aggregation_stage = true; + /// Is running on remote servers during distributed processing? + if (options.to_stage == QueryProcessingStage::WithMergeableStateAfterAggregation) + to_aggregation_stage = true; + if (storage && expressions.filter_info && expressions.prewhere_info) throw Exception("PREWHERE is not supported if the table is filtered by row-level security expression", ErrorCodes::ILLEGAL_PREWHERE); @@ -848,6 +863,12 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu if (expressions.need_aggregate) executeMergeAggregated(query_plan, aggregate_overflow_row, aggregate_final); } + if (from_aggregation_stage) + { + if (intermediate_stage || expressions.first_stage || expressions.second_stage) + throw Exception("Query with after aggregation stage cannot have any other stages", ErrorCodes::LOGICAL_ERROR); + } + if (expressions.first_stage) { @@ -939,9 +960,13 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu executeSubqueriesInSetsAndJoins(query_plan, subqueries_for_sets); } - if (expressions.second_stage) + if (expressions.second_stage || from_aggregation_stage) { - if (expressions.need_aggregate) + if (from_aggregation_stage) + { + /// No need to aggregate anything, since this was done on remote shards. + } + else if (expressions.need_aggregate) { /// If you need to combine aggregated results from multiple servers if (!expressions.first_stage) @@ -994,7 +1019,8 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu * limiting the number of rows in each up to `offset + limit`. */ bool has_prelimit = false; - if (query.limitLength() && !query.limit_with_ties && !hasWithTotalsInAnySubqueryInFromClause(query) && + if (!to_aggregation_stage && + query.limitLength() && !query.limit_with_ties && !hasWithTotalsInAnySubqueryInFromClause(query) && !query.arrayJoinExpressionList() && !query.distinct && !expressions.hasLimitBy() && !settings.extremes) { executePreLimit(query_plan, false); @@ -1023,18 +1049,23 @@ void InterpreterSelectQuery::executeImpl(QueryPlan & query_plan, const BlockInpu has_prelimit = true; } - /** We must do projection after DISTINCT because projection may remove some columns. - */ - executeProjection(query_plan, expressions.final_projection); + /// Projection not be done on the shards, since then initiator will not find column in blocks. + /// (significant only for WithMergeableStateAfterAggregation). + if (!to_aggregation_stage) + { + /// We must do projection after DISTINCT because projection may remove some columns. + executeProjection(query_plan, expressions.final_projection); + } - /** Extremes are calculated before LIMIT, but after LIMIT BY. This is Ok. - */ + /// Extremes are calculated before LIMIT, but after LIMIT BY. This is Ok. executeExtremes(query_plan); - if (!has_prelimit) /// Limit is no longer needed if there is prelimit. + /// Limit is no longer needed if there is prelimit. + if (!to_aggregation_stage && !has_prelimit) executeLimit(query_plan); - executeOffset(query_plan); + if (!to_aggregation_stage) + executeOffset(query_plan); } } diff --git a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp index 461dd997cd1..70916fe386d 100644 --- a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp +++ b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -124,8 +125,37 @@ static NamesAndTypesList getNames(const ASTFunction & expr, const Context & cont return expression->getRequiredColumnsWithTypes(); } +static NamesAndTypesList modifyPrimaryKeysToNonNullable(const NamesAndTypesList & primary_keys, NamesAndTypesList & columns) +{ + /// https://dev.mysql.com/doc/refman/5.7/en/create-table.html#create-table-indexes-keys + /// PRIMARY KEY: + /// A unique index where all key columns must be defined as NOT NULL. + /// If they are not explicitly declared as NOT NULL, MySQL declares them so implicitly (and silently). + /// A table can have only one PRIMARY KEY. The name of a PRIMARY KEY is always PRIMARY, + /// which thus cannot be used as the name for any other kind of index. + NamesAndTypesList non_nullable_primary_keys; + for (const auto & primary_key : primary_keys) + { + if (!primary_key.type->isNullable()) + non_nullable_primary_keys.emplace_back(primary_key); + else + { + non_nullable_primary_keys.emplace_back( + NameAndTypePair(primary_key.name, assert_cast(primary_key.type.get())->getNestedType())); + + for (auto & column : columns) + { + if (column.name == primary_key.name) + column.type = assert_cast(column.type.get())->getNestedType(); + } + } + } + + return non_nullable_primary_keys; +} + static inline std::tuple getKeys( - ASTExpressionList * columns_define, ASTExpressionList * indices_define, const Context & context, const NamesAndTypesList & columns) + ASTExpressionList * columns_define, ASTExpressionList * indices_define, const Context & context, NamesAndTypesList & columns) { NameSet increment_columns; auto keys = makeASTFunction("tuple"); @@ -171,8 +201,9 @@ static inline std::tuple ASTPtr { - ASTPtr column = std::make_shared(column_name); + if (type_max_size <= 1000) + return std::make_shared(column_name); - if (is_nullable) - column = makeASTFunction("assumeNotNull", column); - - return makeASTFunction("intDiv", column, std::make_shared(UInt64(type_max_size / 1000))); + return makeASTFunction("intDiv", std::make_shared(column_name), + std::make_shared(UInt64(type_max_size / 1000))); }; ASTPtr best_partition; @@ -219,16 +249,12 @@ static ASTPtr getPartitionPolicy(const NamesAndTypesList & primary_keys) WhichDataType which(type); if (which.isNullable()) - { - type = (static_cast(*type)).getNestedType(); - which = WhichDataType(type); - } + throw Exception("LOGICAL ERROR: MySQL primary key must be not null, it is a bug.", ErrorCodes::LOGICAL_ERROR); if (which.isDateOrDateTime()) { /// In any case, date or datetime is always the best partitioning key - ASTPtr res = std::make_shared(primary_key.name); - return makeASTFunction("toYYYYMM", primary_key.type->isNullable() ? makeASTFunction("assumeNotNull", res) : res); + return makeASTFunction("toYYYYMM", std::make_shared(primary_key.name)); } if (type->haveMaximumSizeOfValue() && (!best_size || type->getSizeOfValueInMemory() < best_size)) @@ -236,25 +262,22 @@ static ASTPtr getPartitionPolicy(const NamesAndTypesList & primary_keys) if (which.isInt8() || which.isUInt8()) { best_size = type->getSizeOfValueInMemory(); - best_partition = std::make_shared(primary_key.name); - - if (primary_key.type->isNullable()) - best_partition = makeASTFunction("assumeNotNull", best_partition); + best_partition = numbers_partition(primary_key.name, std::numeric_limits::max()); } else if (which.isInt16() || which.isUInt16()) { best_size = type->getSizeOfValueInMemory(); - best_partition = numbers_partition(primary_key.name, primary_key.type->isNullable(), std::numeric_limits::max()); + best_partition = numbers_partition(primary_key.name, std::numeric_limits::max()); } else if (which.isInt32() || which.isUInt32()) { best_size = type->getSizeOfValueInMemory(); - best_partition = numbers_partition(primary_key.name, primary_key.type->isNullable(), std::numeric_limits::max()); + best_partition = numbers_partition(primary_key.name, std::numeric_limits::max()); } else if (which.isInt64() || which.isUInt64()) { best_size = type->getSizeOfValueInMemory(); - best_partition = numbers_partition(primary_key.name, primary_key.type->isNullable(), std::numeric_limits::max()); + best_partition = numbers_partition(primary_key.name, std::numeric_limits::max()); } } } @@ -266,12 +289,12 @@ static ASTPtr getOrderByPolicy( const NamesAndTypesList & primary_keys, const NamesAndTypesList & unique_keys, const NamesAndTypesList & keys, const NameSet & increment_columns) { NameSet order_by_columns_set; - std::deque> order_by_columns_list; + std::deque order_by_columns_list; const auto & add_order_by_expression = [&](const NamesAndTypesList & names_and_types) { - std::vector increment_keys; - std::vector non_increment_keys; + NamesAndTypesList increment_keys; + NamesAndTypesList non_increment_keys; for (const auto & [name, type] : names_and_types) { @@ -280,13 +303,13 @@ static ASTPtr getOrderByPolicy( if (increment_columns.count(name)) { - increment_keys.emplace_back(name); order_by_columns_set.emplace(name); + increment_keys.emplace_back(NameAndTypePair(name, type)); } else { order_by_columns_set.emplace(name); - non_increment_keys.emplace_back(name); + non_increment_keys.emplace_back(NameAndTypePair(name, type)); } } @@ -305,8 +328,13 @@ static ASTPtr getOrderByPolicy( for (const auto & order_by_columns : order_by_columns_list) { - for (const auto & order_by_column : order_by_columns) - order_by_expression->arguments->children.emplace_back(std::make_shared(order_by_column)); + for (const auto & [name, type] : order_by_columns) + { + order_by_expression->arguments->children.emplace_back(std::make_shared(name)); + + if (type->isNullable()) + order_by_expression->arguments->children.back() = makeASTFunction("assumeNotNull", order_by_expression->arguments->children.back()); + } } return order_by_expression; diff --git a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp index b9bfe28ea1b..b940e4e0c95 100644 --- a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp +++ b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp @@ -103,21 +103,12 @@ TEST(MySQLCreateRewritten, PartitionPolicy) {"TIMESTAMP", "DateTime", " PARTITION BY toYYYYMM(key)"}, {"BOOLEAN", "Int8", " PARTITION BY key"} }; - const auto & replace_string = [](const String & str, const String & old_str, const String & new_str) - { - String new_string = str; - size_t pos = new_string.find(old_str); - if (pos != std::string::npos) - new_string = new_string.replace(pos, old_str.size(), new_str); - return new_string; - }; - for (const auto & [test_type, mapped_type, partition_policy] : test_types) { EXPECT_EQ(queryToString(tryRewrittenCreateQuery( "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + " PRIMARY KEY)", context_holder.context)), - "CREATE TABLE test_database.test_table_1 (`key` Nullable(" + mapped_type + "), `_sign` Int8() MATERIALIZED 1, " - "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + replace_string(partition_policy, "key", "assumeNotNull(key)") + " ORDER BY tuple(key)"); + "CREATE TABLE test_database.test_table_1 (`key` " + mapped_type + ", `_sign` Int8() MATERIALIZED 1, " + "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + partition_policy + " ORDER BY tuple(key)"); EXPECT_EQ(queryToString(tryRewrittenCreateQuery( "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + " NOT NULL PRIMARY KEY)", context_holder.context)), @@ -126,6 +117,45 @@ TEST(MySQLCreateRewritten, PartitionPolicy) } } +TEST(MySQLCreateRewritten, OrderbyPolicy) +{ + tryRegisterFunctions(); + const auto & context_holder = getContext(); + + std::vector> test_types + { + {"TINYINT", "Int8", " PARTITION BY key"}, {"SMALLINT", "Int16", " PARTITION BY intDiv(key, 65)"}, + {"MEDIUMINT", "Int32", " PARTITION BY intDiv(key, 4294967)"}, {"INT", "Int32", " PARTITION BY intDiv(key, 4294967)"}, + {"INTEGER", "Int32", " PARTITION BY intDiv(key, 4294967)"}, {"BIGINT", "Int64", " PARTITION BY intDiv(key, 18446744073709551)"}, + {"FLOAT", "Float32", ""}, {"DOUBLE", "Float64", ""}, {"VARCHAR(10)", "String", ""}, {"CHAR(10)", "String", ""}, + {"Date", "Date", " PARTITION BY toYYYYMM(key)"}, {"DateTime", "DateTime", " PARTITION BY toYYYYMM(key)"}, + {"TIMESTAMP", "DateTime", " PARTITION BY toYYYYMM(key)"}, {"BOOLEAN", "Int8", " PARTITION BY key"} + }; + + for (const auto & [test_type, mapped_type, partition_policy] : test_types) + { + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + " PRIMARY KEY, `key2` " + test_type + " UNIQUE KEY)", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` " + mapped_type + ", `key2` Nullable(" + mapped_type + "), `_sign` Int8() MATERIALIZED 1, " + "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + partition_policy + " ORDER BY (key, assumeNotNull(key2))"); + + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + " NOT NULL PRIMARY KEY, `key2` " + test_type + " NOT NULL UNIQUE KEY)", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` " + mapped_type + ", `key2` " + mapped_type + ", `_sign` Int8() MATERIALIZED 1, " + "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + partition_policy + " ORDER BY (key, key2)"); + + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + " KEY UNIQUE KEY)", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` " + mapped_type + ", `_sign` Int8() MATERIALIZED 1, " + "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + partition_policy + " ORDER BY tuple(key)"); + + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` " + test_type + ", `key2` " + test_type + " UNIQUE KEY, PRIMARY KEY(`key`, `key2`))", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` " + mapped_type + ", `key2` " + mapped_type + ", `_sign` Int8() MATERIALIZED 1, " + "`_version` UInt64() MATERIALIZED 1) ENGINE = ReplacingMergeTree(_version)" + partition_policy + " ORDER BY (key, key2)"); + } +} + TEST(MySQLCreateRewritten, RewrittenQueryWithPrimaryKey) { tryRegisterFunctions(); diff --git a/src/Interpreters/QueryNormalizer.cpp b/src/Interpreters/QueryNormalizer.cpp index 324c401eb8a..59233218a50 100644 --- a/src/Interpreters/QueryNormalizer.cpp +++ b/src/Interpreters/QueryNormalizer.cpp @@ -20,6 +20,7 @@ namespace ErrorCodes extern const int TOO_DEEP_AST; extern const int CYCLIC_ALIASES; extern const int UNKNOWN_QUERY_PARAMETER; + extern const int BAD_ARGUMENTS; } @@ -151,6 +152,13 @@ void QueryNormalizer::visitChildren(const ASTPtr & node, Data & data) { if (const auto * func_node = node->as()) { + if (func_node->query) + { + if (func_node->name != "view") + throw Exception("Query argument can only be used in the `view` TableFunction", ErrorCodes::BAD_ARGUMENTS); + /// Don't go into query argument. + return; + } /// We skip the first argument. We also assume that the lambda function can not have parameters. size_t first_pos = 0; if (func_node->name == "lambda") diff --git a/src/Interpreters/SetVariants.cpp b/src/Interpreters/SetVariants.cpp index b026ce56705..eb21333e3ec 100644 --- a/src/Interpreters/SetVariants.cpp +++ b/src/Interpreters/SetVariants.cpp @@ -110,9 +110,8 @@ typename SetVariantsTemplate::Type SetVariantsTemplate::choose size_t size_of_field = nested_key_columns[0]->sizeOfValueIfFixed(); if ((size_of_field == 1) || (size_of_field == 2) || (size_of_field == 4) || (size_of_field == 8)) return Type::nullable_keys128; - else - throw Exception{"Logical error: numeric column has sizeOfField not in 1, 2, 4, 8.", - ErrorCodes::LOGICAL_ERROR}; + + /// Pass to more generic method } if (all_fixed) @@ -145,7 +144,9 @@ typename SetVariantsTemplate::Type SetVariantsTemplate::choose return Type::key64; if (size_of_field == 16) return Type::keys128; - throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16.", ErrorCodes::LOGICAL_ERROR); + if (size_of_field == 32) + return Type::keys256; + throw Exception("Logical error: numeric column has sizeOfField not in 1, 2, 4, 8, 16, 32.", ErrorCodes::LOGICAL_ERROR); } /// If the keys fit in N bits, we will use a hash table for N-bit-packed keys diff --git a/src/Interpreters/TranslateQualifiedNamesVisitor.cpp b/src/Interpreters/TranslateQualifiedNamesVisitor.cpp index fcc4948d88a..e28997f0ad6 100644 --- a/src/Interpreters/TranslateQualifiedNamesVisitor.cpp +++ b/src/Interpreters/TranslateQualifiedNamesVisitor.cpp @@ -18,6 +18,7 @@ #include #include #include +#include namespace DB @@ -135,8 +136,8 @@ void TranslateQualifiedNamesMatcher::visit(ASTFunction & node, const ASTPtr &, D void TranslateQualifiedNamesMatcher::visit(const ASTQualifiedAsterisk &, const ASTPtr & ast, Data & data) { - if (ast->children.size() != 1) - throw Exception("Logical error: qualified asterisk must have exactly one child", ErrorCodes::LOGICAL_ERROR); + if (ast->children.empty()) + throw Exception("Logical error: qualified asterisk must have children", ErrorCodes::LOGICAL_ERROR); auto & ident = ast->children[0]; @@ -242,6 +243,10 @@ void TranslateQualifiedNamesMatcher::visit(ASTExpressionList & node, const ASTPt first_table = false; } + for (const auto & transformer : asterisk->children) + { + IASTColumnsTransformer::transform(transformer, node.children); + } } else if (const auto * asterisk_pattern = child->as()) { @@ -258,6 +263,11 @@ void TranslateQualifiedNamesMatcher::visit(ASTExpressionList & node, const ASTPt first_table = false; } + // ColumnsMatcher's transformers start to appear at child 1 + for (auto it = asterisk_pattern->children.begin() + 1; it != asterisk_pattern->children.end(); ++it) + { + IASTColumnsTransformer::transform(*it, node.children); + } } else if (const auto * qualified_asterisk = child->as()) { @@ -274,6 +284,11 @@ void TranslateQualifiedNamesMatcher::visit(ASTExpressionList & node, const ASTPt break; } } + // QualifiedAsterisk's transformers start to appear at child 1 + for (auto it = qualified_asterisk->children.begin() + 1; it != qualified_asterisk->children.end(); ++it) + { + IASTColumnsTransformer::transform(*it, node.children); + } } else node.children.emplace_back(child); diff --git a/src/Parsers/ASTAsterisk.cpp b/src/Parsers/ASTAsterisk.cpp index 9f38b955d00..95a63586685 100644 --- a/src/Parsers/ASTAsterisk.cpp +++ b/src/Parsers/ASTAsterisk.cpp @@ -13,9 +13,14 @@ ASTPtr ASTAsterisk::clone() const void ASTAsterisk::appendColumnName(WriteBuffer & ostr) const { ostr.write('*'); } -void ASTAsterisk::formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const +void ASTAsterisk::formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const { settings.ostr << "*"; + for (const auto & child : children) + { + settings.ostr << ' '; + child->formatImpl(settings, state, frame); + } } } diff --git a/src/Parsers/ASTAsterisk.h b/src/Parsers/ASTAsterisk.h index 620394ec65a..9c4c9a2df6d 100644 --- a/src/Parsers/ASTAsterisk.h +++ b/src/Parsers/ASTAsterisk.h @@ -9,6 +9,9 @@ namespace DB struct AsteriskSemantic; struct AsteriskSemanticImpl; +/** SELECT * is expanded to all visible columns of the source table. + * Optional transformers can be attached to further manipulate these expanded columns. + */ class ASTAsterisk : public IAST { public: diff --git a/src/Parsers/ASTColumnsMatcher.cpp b/src/Parsers/ASTColumnsMatcher.cpp index b6eb4889a09..191ca52c0e8 100644 --- a/src/Parsers/ASTColumnsMatcher.cpp +++ b/src/Parsers/ASTColumnsMatcher.cpp @@ -28,10 +28,15 @@ void ASTColumnsMatcher::updateTreeHashImpl(SipHash & hash_state) const IAST::updateTreeHashImpl(hash_state); } -void ASTColumnsMatcher::formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const +void ASTColumnsMatcher::formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const { settings.ostr << (settings.hilite ? hilite_keyword : "") << "COLUMNS" << (settings.hilite ? hilite_none : "") << "(" << quoteString(original_pattern) << ")"; + for (ASTs::const_iterator it = children.begin() + 1; it != children.end(); ++it) + { + settings.ostr << ' '; + (*it)->formatImpl(settings, state, frame); + } } void ASTColumnsMatcher::setPattern(String pattern) diff --git a/src/Parsers/ASTColumnsMatcher.h b/src/Parsers/ASTColumnsMatcher.h index 3fa85769712..47a9b86a519 100644 --- a/src/Parsers/ASTColumnsMatcher.h +++ b/src/Parsers/ASTColumnsMatcher.h @@ -23,6 +23,7 @@ struct AsteriskSemanticImpl; /** SELECT COLUMNS('regexp') is expanded to multiple columns like * (asterisk). + * Optional transformers can be attached to further manipulate these expanded columns. */ class ASTColumnsMatcher : public IAST { diff --git a/src/Parsers/ASTColumnsTransformers.cpp b/src/Parsers/ASTColumnsTransformers.cpp new file mode 100644 index 00000000000..2625a03830b --- /dev/null +++ b/src/Parsers/ASTColumnsTransformers.cpp @@ -0,0 +1,159 @@ +#include +#include "ASTColumnsTransformers.h" +#include +#include +#include +#include +#include + + +namespace DB +{ +namespace ErrorCodes +{ + extern const int ILLEGAL_TYPE_OF_ARGUMENT; +} + +void IASTColumnsTransformer::transform(const ASTPtr & transformer, ASTs & nodes) +{ + if (const auto * apply = transformer->as()) + { + apply->transform(nodes); + } + else if (const auto * except = transformer->as()) + { + except->transform(nodes); + } + else if (const auto * replace = transformer->as()) + { + replace->transform(nodes); + } +} + +void ASTColumnsApplyTransformer::formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const +{ + settings.ostr << (settings.hilite ? hilite_keyword : "") << "APPLY" << (settings.hilite ? hilite_none : "") << "(" << func_name << ")"; +} + +void ASTColumnsApplyTransformer::transform(ASTs & nodes) const +{ + for (auto & column : nodes) + { + column = makeASTFunction(func_name, column); + } +} + +void ASTColumnsExceptTransformer::formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const +{ + settings.ostr << (settings.hilite ? hilite_keyword : "") << "EXCEPT" << (settings.hilite ? hilite_none : "") << "("; + + for (ASTs::const_iterator it = children.begin(); it != children.end(); ++it) + { + if (it != children.begin()) + { + settings.ostr << ", "; + } + (*it)->formatImpl(settings, state, frame); + } + + settings.ostr << ")"; +} + +void ASTColumnsExceptTransformer::transform(ASTs & nodes) const +{ + nodes.erase( + std::remove_if( + nodes.begin(), + nodes.end(), + [this](const ASTPtr & node_child) + { + if (const auto * id = node_child->as()) + { + for (const auto & except_child : children) + { + if (except_child->as().name == id->shortName()) + return true; + } + } + return false; + }), + nodes.end()); +} + +void ASTColumnsReplaceTransformer::Replacement::formatImpl( + const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const +{ + expr->formatImpl(settings, state, frame); + settings.ostr << (settings.hilite ? hilite_keyword : "") << " AS " << (settings.hilite ? hilite_none : "") << name; +} + +void ASTColumnsReplaceTransformer::formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const +{ + settings.ostr << (settings.hilite ? hilite_keyword : "") << "REPLACE" << (settings.hilite ? hilite_none : "") << "("; + + for (ASTs::const_iterator it = children.begin(); it != children.end(); ++it) + { + if (it != children.begin()) + { + settings.ostr << ", "; + } + (*it)->formatImpl(settings, state, frame); + } + + settings.ostr << ")"; +} + +void ASTColumnsReplaceTransformer::replaceChildren(ASTPtr & node, const ASTPtr & replacement, const String & name) +{ + for (auto & child : node->children) + { + if (const auto * id = child->as()) + { + if (id->shortName() == name) + child = replacement; + } + else + replaceChildren(child, replacement, name); + } +} + +void ASTColumnsReplaceTransformer::transform(ASTs & nodes) const +{ + std::map replace_map; + for (const auto & replace_child : children) + { + auto & replacement = replace_child->as(); + if (replace_map.find(replacement.name) != replace_map.end()) + throw Exception( + "Expressions in columns transformer REPLACE should not contain the same replacement more than once", + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + replace_map.emplace(replacement.name, replacement.expr); + } + + for (auto & column : nodes) + { + if (const auto * id = column->as()) + { + auto replace_it = replace_map.find(id->shortName()); + if (replace_it != replace_map.end()) + { + column = replace_it->second; + column->setAlias(replace_it->first); + } + } + else if (auto * ast_with_alias = dynamic_cast(column.get())) + { + auto replace_it = replace_map.find(ast_with_alias->alias); + if (replace_it != replace_map.end()) + { + auto new_ast = replace_it->second->clone(); + ast_with_alias->alias = ""; // remove the old alias as it's useless after replace transformation + replaceChildren(new_ast, column, replace_it->first); + column = new_ast; + column->setAlias(replace_it->first); + } + } + } +} + +} diff --git a/src/Parsers/ASTColumnsTransformers.h b/src/Parsers/ASTColumnsTransformers.h new file mode 100644 index 00000000000..ddf0d70dc35 --- /dev/null +++ b/src/Parsers/ASTColumnsTransformers.h @@ -0,0 +1,85 @@ +#pragma once + +#include + +namespace DB +{ +class IASTColumnsTransformer : public IAST +{ +public: + virtual void transform(ASTs & nodes) const = 0; + static void transform(const ASTPtr & transformer, ASTs & nodes); +}; + +class ASTColumnsApplyTransformer : public IASTColumnsTransformer +{ +public: + String getID(char) const override { return "ColumnsApplyTransformer"; } + ASTPtr clone() const override + { + auto res = std::make_shared(*this); + return res; + } + void transform(ASTs & nodes) const override; + String func_name; + +protected: + void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; +}; + +class ASTColumnsExceptTransformer : public IASTColumnsTransformer +{ +public: + String getID(char) const override { return "ColumnsExceptTransformer"; } + ASTPtr clone() const override + { + auto clone = std::make_shared(*this); + clone->cloneChildren(); + return clone; + } + void transform(ASTs & nodes) const override; + +protected: + void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; +}; + +class ASTColumnsReplaceTransformer : public IASTColumnsTransformer +{ +public: + class Replacement : public IAST + { + public: + String getID(char) const override { return "ColumnsReplaceTransformer::Replacement"; } + ASTPtr clone() const override + { + auto replacement = std::make_shared(*this); + replacement->name = name; + replacement->expr = expr->clone(); + replacement->children.push_back(replacement->expr); + return replacement; + } + + String name; + ASTPtr expr; + + protected: + void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; + }; + + String getID(char) const override { return "ColumnsReplaceTransformer"; } + ASTPtr clone() const override + { + auto clone = std::make_shared(*this); + clone->cloneChildren(); + return clone; + } + void transform(ASTs & nodes) const override; + +protected: + void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; + +private: + static void replaceChildren(ASTPtr & node, const ASTPtr & replacement, const String & name); +}; + +} diff --git a/src/Parsers/ASTFunction.cpp b/src/Parsers/ASTFunction.cpp index e8e6efc7fd9..07429c8104f 100644 --- a/src/Parsers/ASTFunction.cpp +++ b/src/Parsers/ASTFunction.cpp @@ -48,6 +48,7 @@ ASTPtr ASTFunction::clone() const auto res = std::make_shared(*this); res->children.clear(); + if (query) { res->query = query->clone(); res->children.push_back(res->query); } if (arguments) { res->arguments = arguments->clone(); res->children.push_back(res->arguments); } if (parameters) { res->parameters = parameters->clone(); res->children.push_back(res->parameters); } @@ -118,6 +119,18 @@ void ASTFunction::formatImplWithoutAlias(const FormatSettings & settings, Format nested_need_parens.need_parens = true; nested_dont_need_parens.need_parens = false; + if (query) + { + std::string nl_or_nothing = settings.one_line ? "" : "\n"; + std::string indent_str = settings.one_line ? "" : std::string(4u * frame.indent, ' '); + settings.ostr << (settings.hilite ? hilite_function : "") << name << "(" << nl_or_nothing; + FormatStateStacked frame_nested = frame; + frame_nested.need_parens = false; + ++frame_nested.indent; + query->formatImpl(settings, state, frame_nested); + settings.ostr << nl_or_nothing << indent_str << ")"; + return; + } /// Should this function to be written as operator? bool written = false; if (arguments && !parameters) diff --git a/src/Parsers/ASTFunction.h b/src/Parsers/ASTFunction.h index 127f50ee586..b94614426d8 100644 --- a/src/Parsers/ASTFunction.h +++ b/src/Parsers/ASTFunction.h @@ -13,6 +13,7 @@ class ASTFunction : public ASTWithAlias { public: String name; + ASTPtr query; // It's possible for a function to accept a query as its only argument. ASTPtr arguments; /// parameters - for parametric aggregate function. Example: quantile(0.9)(x) - what in first parens are 'parameters'. ASTPtr parameters; diff --git a/src/Parsers/ASTGrantQuery.cpp b/src/Parsers/ASTGrantQuery.cpp index ae9649cdddc..63489e0417f 100644 --- a/src/Parsers/ASTGrantQuery.cpp +++ b/src/Parsers/ASTGrantQuery.cpp @@ -144,4 +144,12 @@ void ASTGrantQuery::replaceCurrentUserTagWithName(const String & current_user_na if (to_roles) to_roles->replaceCurrentUserTagWithName(current_user_name); } + + +void ASTGrantQuery::removeNonGrantableFlags() +{ + if (kind == Kind::GRANT) + access_rights_elements.removeNonGrantableFlags(); +} + } diff --git a/src/Parsers/ASTGrantQuery.h b/src/Parsers/ASTGrantQuery.h index c36e42689a5..5f172fe3298 100644 --- a/src/Parsers/ASTGrantQuery.h +++ b/src/Parsers/ASTGrantQuery.h @@ -33,6 +33,7 @@ public: void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; void replaceEmptyDatabaseWithCurrent(const String & current_database); void replaceCurrentUserTagWithName(const String & current_user_name) const; + void removeNonGrantableFlags(); ASTPtr getRewrittenASTWithoutOnCluster(const std::string &) const override { return removeOnCluster(clone()); } }; } diff --git a/src/Parsers/ASTQualifiedAsterisk.cpp b/src/Parsers/ASTQualifiedAsterisk.cpp index cbde6d4f15d..0cda01cecac 100644 --- a/src/Parsers/ASTQualifiedAsterisk.cpp +++ b/src/Parsers/ASTQualifiedAsterisk.cpp @@ -16,6 +16,11 @@ void ASTQualifiedAsterisk::formatImpl(const FormatSettings & settings, FormatSta const auto & qualifier = children.at(0); qualifier->formatImpl(settings, state, frame); settings.ostr << ".*"; + for (ASTs::const_iterator it = children.begin() + 1; it != children.end(); ++it) + { + settings.ostr << ' '; + (*it)->formatImpl(settings, state, frame); + } } } diff --git a/src/Parsers/ASTQualifiedAsterisk.h b/src/Parsers/ASTQualifiedAsterisk.h index 2cead406647..2c3689d0ace 100644 --- a/src/Parsers/ASTQualifiedAsterisk.h +++ b/src/Parsers/ASTQualifiedAsterisk.h @@ -11,6 +11,7 @@ struct AsteriskSemanticImpl; /** Something like t.* * It will have qualifier as its child ASTIdentifier. + * Optional transformers can be attached to further manipulate these expanded columns. */ class ASTQualifiedAsterisk : public IAST { diff --git a/src/Parsers/ExpressionElementParsers.cpp b/src/Parsers/ExpressionElementParsers.cpp index e24bb9c4129..985507071be 100644 --- a/src/Parsers/ExpressionElementParsers.cpp +++ b/src/Parsers/ExpressionElementParsers.cpp @@ -18,9 +18,13 @@ #include #include #include +#include +#include #include #include +#include +#include #include #include #include @@ -217,10 +221,12 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) ParserIdentifier id_parser; ParserKeyword distinct("DISTINCT"); ParserExpressionList contents(false); + ParserSelectWithUnionQuery select; bool has_distinct_modifier = false; ASTPtr identifier; + ASTPtr query; ASTPtr expr_list_args; ASTPtr expr_list_params; @@ -231,8 +237,36 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) return false; ++pos; + if (distinct.ignore(pos, expected)) has_distinct_modifier = true; + else + { + auto old_pos = pos; + auto maybe_an_subquery = pos->type == TokenType::OpeningRoundBracket; + + if (select.parse(pos, query, expected)) + { + auto & select_ast = query->as(); + if (select_ast.list_of_selects->children.size() == 1 && maybe_an_subquery) + { + // It's an subquery. Bail out. + pos = old_pos; + } + else + { + if (pos->type != TokenType::ClosingRoundBracket) + return false; + ++pos; + auto function_node = std::make_shared(); + tryGetIdentifierNameInto(identifier, function_node->name); + function_node->query = query; + function_node->children.push_back(function_node->query); + node = function_node; + return true; + } + } + } const char * contents_begin = pos->begin; if (!contents.parse(pos, expr_list_args, expected)) @@ -1172,17 +1206,131 @@ bool ParserColumnsMatcher::parseImpl(Pos & pos, ASTPtr & node, Expected & expect auto res = std::make_shared(); res->setPattern(regex_node->as().value.get()); res->children.push_back(regex_node); + ParserColumnsTransformers transformers_p; + ASTPtr transformer; + while (transformers_p.parse(pos, transformer, expected)) + { + res->children.push_back(transformer); + } node = std::move(res); return true; } -bool ParserAsterisk::parseImpl(Pos & pos, ASTPtr & node, Expected &) +bool ParserColumnsTransformers::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) +{ + ParserKeyword apply("APPLY"); + ParserKeyword except("EXCEPT"); + ParserKeyword replace("REPLACE"); + ParserKeyword as("AS"); + + if (apply.ignore(pos, expected)) + { + if (pos->type != TokenType::OpeningRoundBracket) + return false; + ++pos; + + String func_name; + if (!parseIdentifierOrStringLiteral(pos, expected, func_name)) + return false; + + if (pos->type != TokenType::ClosingRoundBracket) + return false; + ++pos; + + auto res = std::make_shared(); + res->func_name = func_name; + node = std::move(res); + return true; + } + else if (except.ignore(pos, expected)) + { + if (pos->type != TokenType::OpeningRoundBracket) + return false; + ++pos; + + ASTs identifiers; + auto parse_id = [&identifiers, &pos, &expected] + { + ASTPtr identifier; + if (!ParserIdentifier().parse(pos, identifier, expected)) + return false; + + identifiers.emplace_back(std::move(identifier)); + return true; + }; + + if (!ParserList::parseUtil(pos, expected, parse_id, false)) + return false; + + if (pos->type != TokenType::ClosingRoundBracket) + return false; + ++pos; + + auto res = std::make_shared(); + res->children = std::move(identifiers); + node = std::move(res); + return true; + } + else if (replace.ignore(pos, expected)) + { + if (pos->type != TokenType::OpeningRoundBracket) + return false; + ++pos; + + ASTs replacements; + ParserExpression element_p; + ParserIdentifier ident_p; + auto parse_id = [&] + { + ASTPtr expr; + + if (!element_p.parse(pos, expr, expected)) + return false; + if (!as.ignore(pos, expected)) + return false; + + ASTPtr ident; + if (!ident_p.parse(pos, ident, expected)) + return false; + + auto replacement = std::make_shared(); + replacement->name = getIdentifierName(ident); + replacement->expr = std::move(expr); + replacements.emplace_back(std::move(replacement)); + return true; + }; + + if (!ParserList::parseUtil(pos, expected, parse_id, false)) + return false; + + if (pos->type != TokenType::ClosingRoundBracket) + return false; + ++pos; + + auto res = std::make_shared(); + res->children = std::move(replacements); + node = std::move(res); + return true; + } + + return false; +} + + +bool ParserAsterisk::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) { if (pos->type == TokenType::Asterisk) { ++pos; - node = std::make_shared(); + auto asterisk = std::make_shared(); + ParserColumnsTransformers transformers_p; + ASTPtr transformer; + while (transformers_p.parse(pos, transformer, expected)) + { + asterisk->children.push_back(transformer); + } + node = asterisk; return true; } return false; @@ -1204,6 +1352,12 @@ bool ParserQualifiedAsterisk::parseImpl(Pos & pos, ASTPtr & node, Expected & exp auto res = std::make_shared(); res->children.push_back(node); + ParserColumnsTransformers transformers_p; + ASTPtr transformer; + while (transformers_p.parse(pos, transformer, expected)) + { + res->children.push_back(transformer); + } node = std::move(res); return true; } diff --git a/src/Parsers/ExpressionElementParsers.h b/src/Parsers/ExpressionElementParsers.h index 13e3febcebe..702d757761a 100644 --- a/src/Parsers/ExpressionElementParsers.h +++ b/src/Parsers/ExpressionElementParsers.h @@ -88,6 +88,15 @@ protected: bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; }; +/** *, t.*, db.table.*, COLUMNS('') APPLY(...) or EXCEPT(...) or REPLACE(...) + */ +class ParserColumnsTransformers : public IParserBase +{ +protected: + const char * getName() const override { return "COLUMNS transformers"; } + bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; +}; + /** A function, for example, f(x, y + 1, g(z)). * Or an aggregate function: sum(x + f(y)), corr(x, y). The syntax is the same as the usual function. * Or a parametric aggregate function: quantile(0.9)(x + y). diff --git a/src/Parsers/MySQL/ASTDeclareColumn.cpp b/src/Parsers/MySQL/ASTDeclareColumn.cpp index 56a92291f06..6d21f934858 100644 --- a/src/Parsers/MySQL/ASTDeclareColumn.cpp +++ b/src/Parsers/MySQL/ASTDeclareColumn.cpp @@ -46,10 +46,10 @@ static inline bool parseColumnDeclareOptions(IParser::Pos & pos, ASTPtr & node, OptionDescribe("DEFAULT", "default", std::make_unique()), OptionDescribe("ON UPDATE", "on_update", std::make_unique()), OptionDescribe("AUTO_INCREMENT", "auto_increment", std::make_unique()), - OptionDescribe("UNIQUE", "unique_key", std::make_unique()), OptionDescribe("UNIQUE KEY", "unique_key", std::make_unique()), - OptionDescribe("KEY", "primary_key", std::make_unique()), OptionDescribe("PRIMARY KEY", "primary_key", std::make_unique()), + OptionDescribe("UNIQUE", "unique_key", std::make_unique()), + OptionDescribe("KEY", "primary_key", std::make_unique()), OptionDescribe("COMMENT", "comment", std::make_unique()), OptionDescribe("CHARACTER SET", "charset_name", std::make_unique()), OptionDescribe("COLLATE", "collate", std::make_unique()), diff --git a/src/Parsers/ParserGrantQuery.cpp b/src/Parsers/ParserGrantQuery.cpp index 6e42b165b21..7dd721c9af2 100644 --- a/src/Parsers/ParserGrantQuery.cpp +++ b/src/Parsers/ParserGrantQuery.cpp @@ -14,6 +14,7 @@ namespace DB { namespace ErrorCodes { + extern const int INVALID_GRANT; extern const int SYNTAX_ERROR; } @@ -156,6 +157,29 @@ namespace } + void removeNonGrantableFlags(AccessRightsElements & elements) + { + for (auto & element : elements) + { + if (element.empty()) + continue; + auto old_flags = element.access_flags; + element.removeNonGrantableFlags(); + if (!element.empty()) + continue; + + if (!element.any_column) + throw Exception(old_flags.toString() + " cannot be granted on the column level", ErrorCodes::INVALID_GRANT); + else if (!element.any_table) + throw Exception(old_flags.toString() + " cannot be granted on the table level", ErrorCodes::INVALID_GRANT); + else if (!element.any_database) + throw Exception(old_flags.toString() + " cannot be granted on the database level", ErrorCodes::INVALID_GRANT); + else + throw Exception(old_flags.toString() + " cannot be granted", ErrorCodes::INVALID_GRANT); + } + } + + bool parseRoles(IParser::Pos & pos, Expected & expected, Kind kind, bool id_mode, std::shared_ptr & roles) { return IParserBase::wrapParseImpl(pos, [&] @@ -274,6 +298,9 @@ bool ParserGrantQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) if (admin_option && !elements.empty()) throw Exception("ADMIN OPTION should be specified for roles", ErrorCodes::SYNTAX_ERROR); + if (kind == Kind::GRANT) + removeNonGrantableFlags(elements); + auto query = std::make_shared(); node = query; diff --git a/src/Parsers/ya.make b/src/Parsers/ya.make index 1b03bae100b..b6ef322e426 100644 --- a/src/Parsers/ya.make +++ b/src/Parsers/ya.make @@ -10,6 +10,7 @@ SRCS( ASTAsterisk.cpp ASTColumnDeclaration.cpp ASTColumnsMatcher.cpp + ASTColumnsTransformers.cpp ASTConstraintDeclaration.cpp ASTCreateQuery.cpp ASTCreateQuotaQuery.cpp diff --git a/src/Processors/Executors/PipelineExecutor.cpp b/src/Processors/Executors/PipelineExecutor.cpp index cacd8fced8d..858f53cb047 100644 --- a/src/Processors/Executors/PipelineExecutor.cpp +++ b/src/Processors/Executors/PipelineExecutor.cpp @@ -402,6 +402,11 @@ void PipelineExecutor::execute(size_t num_threads) for (auto & node : graph->nodes) if (node->exception) std::rethrow_exception(node->exception); + + /// Exception which happened in executing thread, but not at processor. + for (auto & executor_context : executor_contexts) + if (executor_context->exception) + std::rethrow_exception(executor_context->exception); } catch (...) { @@ -726,7 +731,16 @@ void PipelineExecutor::executeImpl(size_t num_threads) CurrentThread::detachQueryIfNotDetached(); ); - executeSingleThread(thread_num, num_threads); + try + { + executeSingleThread(thread_num, num_threads); + } + catch (...) + { + /// In case of exception from executor itself, stop other threads. + finish(); + executor_contexts[thread_num]->exception = std::current_exception(); + } }); } diff --git a/src/Processors/Executors/PipelineExecutor.h b/src/Processors/Executors/PipelineExecutor.h index 927b9d891e4..b457cca34b1 100644 --- a/src/Processors/Executors/PipelineExecutor.h +++ b/src/Processors/Executors/PipelineExecutor.h @@ -97,6 +97,9 @@ private: /// Currently processing node. ExecutingGraph::Node * node = nullptr; + /// Exception from executing thread itself. + std::exception_ptr exception; + #ifndef NDEBUG /// Time for different processing stages. UInt64 total_time_ns = 0; diff --git a/src/Processors/Pipe.cpp b/src/Processors/Pipe.cpp index 93dcd561c00..d28e54dae58 100644 --- a/src/Processors/Pipe.cpp +++ b/src/Processors/Pipe.cpp @@ -102,6 +102,8 @@ Pipe::Holder & Pipe::Holder::operator=(Holder && rhs) storage_holders.insert(storage_holders.end(), rhs.storage_holders.begin(), rhs.storage_holders.end()); interpreter_context.insert(interpreter_context.end(), rhs.interpreter_context.begin(), rhs.interpreter_context.end()); + for (auto & plan : rhs.query_plans) + query_plans.emplace_back(std::move(plan)); return *this; } diff --git a/src/Processors/Pipe.h b/src/Processors/Pipe.h index 28b64937aeb..f5f8b117db9 100644 --- a/src/Processors/Pipe.h +++ b/src/Processors/Pipe.h @@ -1,6 +1,7 @@ #pragma once #include #include +#include namespace DB { @@ -8,6 +9,8 @@ namespace DB class Pipe; using Pipes = std::vector; +class QueryPipeline; + class IStorage; using StoragePtr = std::shared_ptr; @@ -86,6 +89,8 @@ public: /// Get processors from Pipe. Use it with cautious, it is easy to loss totals and extremes ports. static Processors detachProcessors(Pipe pipe) { return std::move(pipe.processors); } + /// Get processors from Pipe w/o destroying pipe (used for EXPLAIN to keep QueryPlan). + const Processors & getProcessors() const { return processors; } /// Specify quotas and limits for every ISourceWithProgress. void setLimits(const SourceWithProgress::LocalLimits & limits); @@ -96,6 +101,8 @@ public: /// This methods are from QueryPipeline. Needed to make conversion from pipeline to pipe possible. void addInterpreterContext(std::shared_ptr context) { holder.interpreter_context.emplace_back(std::move(context)); } void addStorageHolder(StoragePtr storage) { holder.storage_holders.emplace_back(std::move(storage)); } + /// For queries with nested interpreters (i.e. StorageDistributed) + void addQueryPlan(std::unique_ptr plan) { holder.query_plans.emplace_back(std::move(plan)); } private: /// Destruction order: processors, header, locks, temporary storages, local contexts @@ -113,6 +120,7 @@ private: std::vector> interpreter_context; std::vector storage_holders; std::vector table_locks; + std::vector> query_plans; }; Holder holder; diff --git a/src/Processors/Port.h b/src/Processors/Port.h index acce2371dea..b9fe8be218a 100644 --- a/src/Processors/Port.h +++ b/src/Processors/Port.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include diff --git a/src/Processors/QueryPipeline.h b/src/Processors/QueryPipeline.h index 385cf77198e..94de753bebc 100644 --- a/src/Processors/QueryPipeline.h +++ b/src/Processors/QueryPipeline.h @@ -21,6 +21,8 @@ class QueryPipelineProcessorsCollector; struct AggregatingTransformParams; using AggregatingTransformParamsPtr = std::shared_ptr; +class QueryPlan; + class QueryPipeline { public: @@ -93,6 +95,7 @@ public: void addTableLock(const TableLockHolder & lock) { pipe.addTableLock(lock); } void addInterpreterContext(std::shared_ptr context) { pipe.addInterpreterContext(std::move(context)); } void addStorageHolder(StoragePtr storage) { pipe.addStorageHolder(std::move(storage)); } + void addQueryPlan(std::unique_ptr plan) { pipe.addQueryPlan(std::move(plan)); } /// For compatibility with IBlockInputStream. void setProgressCallback(const ProgressCallback & callback); diff --git a/src/Server/MySQLHandler.cpp b/src/Server/MySQLHandler.cpp index 2cf6ad546eb..ca18752f60b 100644 --- a/src/Server/MySQLHandler.cpp +++ b/src/Server/MySQLHandler.cpp @@ -394,6 +394,7 @@ static bool isFederatedServerSetupSetCommand(const String & query) "|(^(SET FOREIGN_KEY_CHECKS(.*)))" "|(^(SET AUTOCOMMIT(.*)))" "|(^(SET sql_mode(.*)))" + "|(^(SET @@(.*)))" "|(^(SET SESSION TRANSACTION ISOLATION LEVEL(.*)))" , std::regex::icase}; return 1 == std::regex_match(query, expr); diff --git a/src/Storages/LiveView/StorageLiveView.cpp b/src/Storages/LiveView/StorageLiveView.cpp index 54ac5bcc791..4da02365232 100644 --- a/src/Storages/LiveView/StorageLiveView.cpp +++ b/src/Storages/LiveView/StorageLiveView.cpp @@ -518,9 +518,10 @@ void StorageLiveView::drop() condition.notify_all(); } -void StorageLiveView::refresh(const Context & context) +void StorageLiveView::refresh() { - auto table_lock = lockExclusively(context.getCurrentQueryId(), context.getSettingsRef().lock_acquire_timeout); + // Lock is already acquired exclusively from InterperterAlterQuery.cpp InterpreterAlterQuery::execute() method. + // So, reacquiring lock is not needed and will result in an exception. { std::lock_guard lock(mutex); if (getNewBlocks()) diff --git a/src/Storages/LiveView/StorageLiveView.h b/src/Storages/LiveView/StorageLiveView.h index 43afd169a92..0c099d01a29 100644 --- a/src/Storages/LiveView/StorageLiveView.h +++ b/src/Storages/LiveView/StorageLiveView.h @@ -122,7 +122,7 @@ public: void startup() override; void shutdown() override; - void refresh(const Context & context); + void refresh(); Pipe read( const Names & column_names, diff --git a/src/Storages/MergeTree/AllMergeSelector.cpp b/src/Storages/MergeTree/AllMergeSelector.cpp index 053f91395c6..79080df1570 100644 --- a/src/Storages/MergeTree/AllMergeSelector.cpp +++ b/src/Storages/MergeTree/AllMergeSelector.cpp @@ -6,14 +6,14 @@ namespace DB { -AllMergeSelector::PartsInPartition AllMergeSelector::select( - const Partitions & partitions, +AllMergeSelector::PartsRange AllMergeSelector::select( + const PartsRanges & parts_ranges, const size_t /*max_total_size_to_merge*/) { size_t min_partition_size = 0; - Partitions::const_iterator best_partition; + PartsRanges::const_iterator best_partition; - for (auto it = partitions.begin(); it != partitions.end(); ++it) + for (auto it = parts_ranges.begin(); it != parts_ranges.end(); ++it) { if (it->size() <= 1) continue; diff --git a/src/Storages/MergeTree/AllMergeSelector.h b/src/Storages/MergeTree/AllMergeSelector.h index eade7954144..d3b399b2fc5 100644 --- a/src/Storages/MergeTree/AllMergeSelector.h +++ b/src/Storages/MergeTree/AllMergeSelector.h @@ -11,8 +11,8 @@ class AllMergeSelector : public IMergeSelector { public: /// Parameter max_total_size_to_merge is ignored. - PartsInPartition select( - const Partitions & partitions, + PartsRange select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) override; }; diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.cpp b/src/Storages/MergeTree/IMergeTreeDataPart.cpp index 0c4ab1c8678..486e444763d 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPart.cpp @@ -1047,6 +1047,37 @@ void IMergeTreeDataPart::accumulateColumnSizes(ColumnToSize & column_to_size) co column_to_size[column_name] = size.data_compressed; } + +bool IMergeTreeDataPart::checkAllTTLCalculated(const StorageMetadataPtr & metadata_snapshot) const +{ + if (!metadata_snapshot->hasAnyTTL()) + return false; + + if (metadata_snapshot->hasRowsTTL()) + { + if (isEmpty()) /// All rows were finally deleted and we don't store TTL + return true; + else if (ttl_infos.table_ttl.min == 0) + return false; + } + + for (const auto & [column, desc] : metadata_snapshot->getColumnTTLs()) + { + /// Part has this column, but we don't calculated TTL for it + if (!ttl_infos.columns_ttl.count(column) && getColumns().contains(column)) + return false; + } + + for (const auto & move_desc : metadata_snapshot->getMoveTTLs()) + { + /// Move TTL is not calculated + if (!ttl_infos.moves_ttl.count(move_desc.result_column)) + return false; + } + + return true; +} + bool isCompactPart(const MergeTreeDataPartPtr & data_part) { return (data_part && data_part->getType() == MergeTreeDataPartType::COMPACT); diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.h b/src/Storages/MergeTree/IMergeTreeDataPart.h index 35e82e0e94a..7df0468dc13 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.h +++ b/src/Storages/MergeTree/IMergeTreeDataPart.h @@ -344,6 +344,11 @@ public: static inline constexpr auto DELETE_ON_DESTROY_MARKER_FILE_NAME = "delete-on-destroy.txt"; + /// Checks that all TTLs (table min/max, column ttls, so on) for part + /// calculated. Part without calculated TTL may exist if TTL was added after + /// part creation (using alter query with materialize_ttl setting). + bool checkAllTTLCalculated(const StorageMetadataPtr & metadata_snapshot) const; + protected: /// Total size of all columns, calculated once in calcuateColumnSizesOnDisk diff --git a/src/Storages/MergeTree/LevelMergeSelector.cpp b/src/Storages/MergeTree/LevelMergeSelector.cpp index ba7ca0257eb..7bcfbf6160a 100644 --- a/src/Storages/MergeTree/LevelMergeSelector.cpp +++ b/src/Storages/MergeTree/LevelMergeSelector.cpp @@ -14,7 +14,7 @@ namespace */ struct Estimator { - using Iterator = LevelMergeSelector::PartsInPartition::const_iterator; + using Iterator = LevelMergeSelector::PartsRange::const_iterator; void consider(Iterator begin, Iterator end, size_t sum_size) { @@ -28,9 +28,9 @@ struct Estimator } } - LevelMergeSelector::PartsInPartition getBest() const + LevelMergeSelector::PartsRange getBest() const { - return LevelMergeSelector::PartsInPartition(best_begin, best_end); + return LevelMergeSelector::PartsRange(best_begin, best_end); } double min_score = 0; @@ -40,7 +40,7 @@ struct Estimator void selectWithinPartition( - const LevelMergeSelector::PartsInPartition & parts, + const LevelMergeSelector::PartsRange & parts, const size_t max_total_size_to_merge, Estimator & estimator, const LevelMergeSelector::Settings & settings) @@ -103,14 +103,14 @@ void selectWithinPartition( } -LevelMergeSelector::PartsInPartition LevelMergeSelector::select( - const Partitions & partitions, +LevelMergeSelector::PartsRange LevelMergeSelector::select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) { Estimator estimator; - for (const auto & partition : partitions) - selectWithinPartition(partition, max_total_size_to_merge, estimator, settings); + for (const auto & parts_range: parts_ranges) + selectWithinPartition(parts_range, max_total_size_to_merge, estimator, settings); return estimator.getBest(); } diff --git a/src/Storages/MergeTree/LevelMergeSelector.h b/src/Storages/MergeTree/LevelMergeSelector.h index 4ce6624bea1..5849b34e320 100644 --- a/src/Storages/MergeTree/LevelMergeSelector.h +++ b/src/Storages/MergeTree/LevelMergeSelector.h @@ -19,8 +19,8 @@ public: explicit LevelMergeSelector(const Settings & settings_) : settings(settings_) {} - PartsInPartition select( - const Partitions & partitions, + PartsRange select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) override; private: diff --git a/src/Storages/MergeTree/MergeSelector.h b/src/Storages/MergeTree/MergeSelector.h index 24612b367d5..f9854d0de6a 100644 --- a/src/Storages/MergeTree/MergeSelector.h +++ b/src/Storages/MergeTree/MergeSelector.h @@ -48,16 +48,16 @@ public: }; /// Parts are belong to partitions. Only parts within same partition could be merged. - using PartsInPartition = std::vector; + using PartsRange = std::vector; /// Parts are in some specific order. Parts could be merged only in contiguous ranges. - using Partitions = std::vector; + using PartsRanges = std::vector; /** Function could be called at any frequency and it must decide, should you do any merge at all. * If better not to do any merge, it returns empty result. */ - virtual PartsInPartition select( - const Partitions & partitions, + virtual PartsRange select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) = 0; virtual ~IMergeSelector() = default; diff --git a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp index 17dc2dc83c3..8296e31eccd 100644 --- a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp +++ b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp @@ -228,13 +228,25 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( time_t current_time = std::time(nullptr); - IMergeSelector::Partitions partitions; + IMergeSelector::PartsRanges parts_ranges; const String * prev_partition_id = nullptr; /// Previous part only in boundaries of partition frame const MergeTreeData::DataPartPtr * prev_part = nullptr; + for (const MergeTreeData::DataPartPtr & part : data_parts) { + const String & partition_id = part->info.partition_id; + + if (!prev_partition_id || partition_id != *prev_partition_id) + { + if (parts_ranges.empty() || !parts_ranges.back().empty()) + parts_ranges.emplace_back(); + /// New partition frame. + prev_partition_id = &partition_id; + prev_part = nullptr; + } + /// Check predicate only for first part in each partition. if (!prev_part) { @@ -245,15 +257,19 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( if (!can_merge_callback(nullptr, part, nullptr)) continue; } - - const String & partition_id = part->info.partition_id; - if (!prev_partition_id || partition_id != *prev_partition_id || (prev_part && !can_merge_callback(*prev_part, part, nullptr))) + else { - if (partitions.empty() || !partitions.back().empty()) - partitions.emplace_back(); - /// New partition frame. - prev_partition_id = &partition_id; - prev_part = nullptr; + /// If we cannot merge with previous part we had to start new parts + /// interval (in the same partition) + if (!can_merge_callback(*prev_part, part, nullptr)) + { + /// Starting new interval in the same partition + if (!parts_ranges.back().empty()) + parts_ranges.emplace_back(); + + /// Now we have no previous part, but it affects only logging + prev_part = nullptr; + } } IMergeSelector::Part part_info; @@ -264,7 +280,7 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( part_info.min_ttl = part->ttl_infos.part_min_ttl; part_info.max_ttl = part->ttl_infos.part_max_ttl; - partitions.back().emplace_back(part_info); + parts_ranges.back().emplace_back(part_info); /// Check for consistency of data parts. If assertion is failed, it requires immediate investigation. if (prev_part && part->info.partition_id == (*prev_part)->info.partition_id @@ -276,7 +292,7 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( prev_part = ∂ } - IMergeSelector::PartsInPartition parts_to_merge; + IMergeSelector::PartsRange parts_to_merge; if (!ttl_merges_blocker.isCancelled()) { @@ -285,7 +301,7 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( current_time, data_settings->merge_with_ttl_timeout, data_settings->ttl_only_drop_parts); - parts_to_merge = merge_selector.select(partitions, max_total_size_to_merge); + parts_to_merge = merge_selector.select(parts_ranges, max_total_size_to_merge); } if (parts_to_merge.empty()) @@ -295,7 +311,7 @@ bool MergeTreeDataMergerMutator::selectPartsToMerge( merge_settings.base = 1; parts_to_merge = SimpleMergeSelector(merge_settings) - .select(partitions, max_total_size_to_merge); + .select(parts_ranges, max_total_size_to_merge); /// Do not allow to "merge" part with itself for regular merges, unless it is a TTL-merge where it is ok to remove some values with expired ttl if (parts_to_merge.size() == 1) @@ -643,8 +659,17 @@ MergeTreeData::MutableDataPartPtr MergeTreeDataMergerMutator::mergePartsToTempor new_data_part->is_temp = true; bool need_remove_expired_values = false; + bool force_ttl = false; for (const auto & part : parts) + { new_data_part->ttl_infos.update(part->ttl_infos); + if (metadata_snapshot->hasAnyTTL() && !part->checkAllTTLCalculated(metadata_snapshot)) + { + LOG_INFO(log, "Some TTL values were not calculated for part {}. Will calculate them forcefully during merge.", part->name); + need_remove_expired_values = true; + force_ttl = true; + } + } const auto & part_min_ttl = new_data_part->ttl_infos.part_min_ttl; if (part_min_ttl && part_min_ttl <= time_of_merge) @@ -822,7 +847,7 @@ MergeTreeData::MutableDataPartPtr MergeTreeDataMergerMutator::mergePartsToTempor merged_stream = std::make_shared(merged_stream, sort_description, SizeLimits(), 0 /*limit_hint*/, Names()); if (need_remove_expired_values) - merged_stream = std::make_shared(merged_stream, data, metadata_snapshot, new_data_part, time_of_merge, false); + merged_stream = std::make_shared(merged_stream, data, metadata_snapshot, new_data_part, time_of_merge, force_ttl); if (metadata_snapshot->hasSecondaryIndices()) { diff --git a/src/Storages/MergeTree/MergeTreeDataPartTTLInfo.h b/src/Storages/MergeTree/MergeTreeDataPartTTLInfo.h index 209d7181b66..d7a6add8171 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartTTLInfo.h +++ b/src/Storages/MergeTree/MergeTreeDataPartTTLInfo.h @@ -38,7 +38,7 @@ struct MergeTreeDataPartTTLInfos MergeTreeDataPartTTLInfo table_ttl; /// `part_min_ttl` and `part_max_ttl` are TTLs which are used for selecting parts - /// to merge in order to remove expired rows. + /// to merge in order to remove expired rows. time_t part_min_ttl = 0; time_t part_max_ttl = 0; @@ -58,7 +58,7 @@ struct MergeTreeDataPartTTLInfos part_max_ttl = time_max; } - bool empty() + bool empty() const { return !part_min_ttl && moves_ttl.empty(); } diff --git a/src/Storages/MergeTree/SimpleMergeSelector.cpp b/src/Storages/MergeTree/SimpleMergeSelector.cpp index ccb56188431..cbb24d1494e 100644 --- a/src/Storages/MergeTree/SimpleMergeSelector.cpp +++ b/src/Storages/MergeTree/SimpleMergeSelector.cpp @@ -15,7 +15,7 @@ namespace */ struct Estimator { - using Iterator = SimpleMergeSelector::PartsInPartition::const_iterator; + using Iterator = SimpleMergeSelector::PartsRange::const_iterator; void consider(Iterator begin, Iterator end, size_t sum_size, size_t size_prev_at_left, const SimpleMergeSelector::Settings & settings) { @@ -42,9 +42,9 @@ struct Estimator } } - SimpleMergeSelector::PartsInPartition getBest() const + SimpleMergeSelector::PartsRange getBest() const { - return SimpleMergeSelector::PartsInPartition(best_begin, best_end); + return SimpleMergeSelector::PartsRange(best_begin, best_end); } static double score(double count, double sum_size, double sum_size_fixed_cost) @@ -137,7 +137,7 @@ bool allow( void selectWithinPartition( - const SimpleMergeSelector::PartsInPartition & parts, + const SimpleMergeSelector::PartsRange & parts, const size_t max_total_size_to_merge, Estimator & estimator, const SimpleMergeSelector::Settings & settings) @@ -185,14 +185,14 @@ void selectWithinPartition( } -SimpleMergeSelector::PartsInPartition SimpleMergeSelector::select( - const Partitions & partitions, +SimpleMergeSelector::PartsRange SimpleMergeSelector::select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) { Estimator estimator; - for (const auto & partition : partitions) - selectWithinPartition(partition, max_total_size_to_merge, estimator, settings); + for (const auto & part_range : parts_ranges) + selectWithinPartition(part_range, max_total_size_to_merge, estimator, settings); return estimator.getBest(); } diff --git a/src/Storages/MergeTree/SimpleMergeSelector.h b/src/Storages/MergeTree/SimpleMergeSelector.h index 729eaa966e9..9aeb73a40a8 100644 --- a/src/Storages/MergeTree/SimpleMergeSelector.h +++ b/src/Storages/MergeTree/SimpleMergeSelector.h @@ -73,8 +73,8 @@ public: explicit SimpleMergeSelector(const Settings & settings_) : settings(settings_) {} - PartsInPartition select( - const Partitions & partitions, + PartsRange select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) override; private: diff --git a/src/Storages/MergeTree/TTLMergeSelector.cpp b/src/Storages/MergeTree/TTLMergeSelector.cpp index 1966f2a4f0a..048aa026014 100644 --- a/src/Storages/MergeTree/TTLMergeSelector.cpp +++ b/src/Storages/MergeTree/TTLMergeSelector.cpp @@ -15,18 +15,18 @@ const String & getPartitionIdForPart(const TTLMergeSelector::Part & part_info) } -IMergeSelector::PartsInPartition TTLMergeSelector::select( - const Partitions & partitions, +IMergeSelector::PartsRange TTLMergeSelector::select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) { - using Iterator = IMergeSelector::PartsInPartition::const_iterator; + using Iterator = IMergeSelector::PartsRange::const_iterator; Iterator best_begin; ssize_t partition_to_merge_index = -1; time_t partition_to_merge_min_ttl = 0; - for (size_t i = 0; i < partitions.size(); ++i) + for (size_t i = 0; i < parts_ranges.size(); ++i) { - const auto & mergeable_parts_in_partition = partitions[i]; + const auto & mergeable_parts_in_partition = parts_ranges[i]; if (mergeable_parts_in_partition.empty()) continue; @@ -51,7 +51,7 @@ IMergeSelector::PartsInPartition TTLMergeSelector::select( if (partition_to_merge_index == -1 || partition_to_merge_min_ttl > current_time) return {}; - const auto & best_partition = partitions[partition_to_merge_index]; + const auto & best_partition = parts_ranges[partition_to_merge_index]; Iterator best_end = best_begin + 1; size_t total_size = 0; @@ -88,7 +88,7 @@ IMergeSelector::PartsInPartition TTLMergeSelector::select( const auto & best_partition_id = getPartitionIdForPart(best_partition.front()); merge_due_times[best_partition_id] = current_time + merge_cooldown_time; - return PartsInPartition(best_begin, best_end); + return PartsRange(best_begin, best_end); } } diff --git a/src/Storages/MergeTree/TTLMergeSelector.h b/src/Storages/MergeTree/TTLMergeSelector.h index 5b7361d2d2b..0559fcc14d3 100644 --- a/src/Storages/MergeTree/TTLMergeSelector.h +++ b/src/Storages/MergeTree/TTLMergeSelector.h @@ -10,7 +10,7 @@ namespace DB { /** Merge selector, which is used to remove values with expired ttl. - * It selects parts to merge by greedy algorithm: + * It selects parts to merge by greedy algorithm: * 1. Finds part with the most earliest expired ttl and includes it to result. * 2. Tries to find the longest range of parts with expired ttl, that includes part from step 1. * Finally, merge selector updates TTL merge timer for the selected partition @@ -26,8 +26,8 @@ public: merge_cooldown_time(merge_cooldown_time_), only_drop_parts(only_drop_parts_) {} - PartsInPartition select( - const Partitions & partitions, + PartsRange select( + const PartsRanges & parts_ranges, const size_t max_total_size_to_merge) override; private: diff --git a/src/Storages/StorageDistributed.cpp b/src/Storages/StorageDistributed.cpp index f536c6ee763..703bb6dcb96 100644 --- a/src/Storages/StorageDistributed.cpp +++ b/src/Storages/StorageDistributed.cpp @@ -56,12 +56,15 @@ #include #include +#include namespace { const UInt64 FORCE_OPTIMIZE_SKIP_UNUSED_SHARDS_HAS_SHARDING_KEY = 1; const UInt64 FORCE_OPTIMIZE_SKIP_UNUSED_SHARDS_ALWAYS = 2; + +const UInt64 DISTRIBUTED_GROUP_BY_NO_MERGE_AFTER_AGGREGATION = 2; } namespace DB @@ -242,22 +245,82 @@ void replaceConstantExpressions( visitor.visit(node); } -QueryProcessingStage::Enum getQueryProcessingStageImpl(const Context & context, QueryProcessingStage::Enum to_stage, const ClusterPtr & cluster) +/// Returns one of the following: +/// - QueryProcessingStage::Complete +/// - QueryProcessingStage::WithMergeableStateAfterAggregation +/// - none (in this case regular WithMergeableState should be used) +std::optional getOptimizedQueryProcessingStage(const ASTPtr & query_ptr, bool extremes, const Block & sharding_key_block) { - const Settings & settings = context.getSettingsRef(); + const auto & select = query_ptr->as(); + auto sharding_block_has = [&](const auto & exprs, size_t limit = SIZE_MAX) -> bool + { + size_t i = 0; + for (auto & expr : exprs) + { + ++i; + if (i > limit) + break; + + auto id = expr->template as(); + if (!id) + return false; + /// TODO: if GROUP BY contains multiIf()/if() it should contain only columns from sharding_key + if (!sharding_key_block.has(id->name)) + return false; + } + return true; + }; + + // GROUP BY qualifiers + // - TODO: WITH TOTALS can be implemented + // - TODO: WITH ROLLUP can be implemented (I guess) + if (select.group_by_with_totals || select.group_by_with_rollup || select.group_by_with_cube) + return {}; + + // TODO: extremes support can be implemented + if (extremes) + return {}; + + // DISTINCT + if (select.distinct) + { + if (!sharding_block_has(select.select()->children)) + return {}; + } + + // GROUP BY + const ASTPtr group_by = select.groupBy(); + if (!group_by) + { + if (!select.distinct) + return {}; + } + else + { + if (!sharding_block_has(group_by->children, 1)) + return {}; + } + + // ORDER BY + const ASTPtr order_by = select.orderBy(); + if (order_by) + return QueryProcessingStage::WithMergeableStateAfterAggregation; + + // LIMIT BY + // LIMIT + if (select.limitBy() || select.limitLength()) + return QueryProcessingStage::WithMergeableStateAfterAggregation; + + // Only simple SELECT FROM GROUP BY sharding_key can use Complete state. + return QueryProcessingStage::Complete; +} + +size_t getClusterQueriedNodes(const Settings & settings, const ClusterPtr & cluster) +{ size_t num_local_shards = cluster->getLocalShardCount(); size_t num_remote_shards = cluster->getRemoteShardCount(); - size_t result_size = (num_remote_shards * settings.max_parallel_replicas) + num_local_shards; - - if (settings.distributed_group_by_no_merge) - return QueryProcessingStage::Complete; - /// Nested distributed query cannot return Complete stage, - /// since the parent query need to aggregate the results after. - if (to_stage == QueryProcessingStage::WithMergeableState) - return QueryProcessingStage::WithMergeableState; - return result_size == 1 ? QueryProcessingStage::Complete - : QueryProcessingStage::WithMergeableState; + return (num_remote_shards * settings.max_parallel_replicas) + num_local_shards; } } @@ -374,87 +437,23 @@ StoragePtr StorageDistributed::createWithOwnCluster( return res; } - -bool StorageDistributed::canForceGroupByNoMerge(const Context &context, QueryProcessingStage::Enum to_stage, const ASTPtr & query_ptr) const -{ - const auto & settings = context.getSettingsRef(); - std::string reason; - - if (settings.distributed_group_by_no_merge) - return true; - if (!settings.optimize_distributed_group_by_sharding_key) - return false; - - /// Distributed-over-Distributed (see getQueryProcessingStageImpl()) - if (to_stage == QueryProcessingStage::WithMergeableState) - return false; - if (!settings.optimize_skip_unused_shards) - return false; - if (!has_sharding_key) - return false; - - const auto & select = query_ptr->as(); - - if (select.group_by_with_totals || select.group_by_with_rollup || select.group_by_with_cube) - return false; - - // TODO: The following can be optimized too (but with some caveats, will be addressed later): - // - ORDER BY - // - LIMIT BY - // - LIMIT - if (select.orderBy()) - return false; - if (select.limitBy() || select.limitLength()) - return false; - - if (select.distinct) - { - for (auto & expr : select.select()->children) - { - const auto * id = expr->as(); - if (!id) - return false; - if (!sharding_key_expr->getSampleBlock().has(id->name)) - return false; - } - - reason = "DISTINCT " + backQuote(serializeAST(*select.select(), true)); - } - - const ASTPtr group_by = select.groupBy(); - if (!group_by) - { - if (!select.distinct) - return false; - } - else - { - // injective functions are optimized out in optimizeGroupBy() - // hence all we need to check is that column in GROUP BY matches sharding expression - auto & group_exprs = group_by->children; - if (group_exprs.empty()) - throw Exception("No ASTExpressionList in GROUP BY", ErrorCodes::LOGICAL_ERROR); - - const auto * id = group_exprs[0]->as(); - if (!id) - return false; - if (!sharding_key_expr->getSampleBlock().has(id->name)) - return false; - - reason = "GROUP BY " + backQuote(serializeAST(*group_by, true)); - } - - LOG_DEBUG(log, "Force distributed_group_by_no_merge for {} (injective)", reason); - return true; -} - QueryProcessingStage::Enum StorageDistributed::getQueryProcessingStage(const Context &context, QueryProcessingStage::Enum to_stage, const ASTPtr & query_ptr) const { const auto & settings = context.getSettingsRef(); auto metadata_snapshot = getInMemoryMetadataPtr(); - if (canForceGroupByNoMerge(context, to_stage, query_ptr)) - return QueryProcessingStage::Complete; + if (settings.distributed_group_by_no_merge) + { + if (settings.distributed_group_by_no_merge == DISTRIBUTED_GROUP_BY_NO_MERGE_AFTER_AGGREGATION) + return QueryProcessingStage::WithMergeableStateAfterAggregation; + else + return QueryProcessingStage::Complete; + } + + /// Nested distributed query cannot return Complete stage, + /// since the parent query need to aggregate the results after. + if (to_stage == QueryProcessingStage::WithMergeableState) + return QueryProcessingStage::WithMergeableState; ClusterPtr cluster = getCluster(); if (settings.optimize_skip_unused_shards) @@ -464,7 +463,26 @@ QueryProcessingStage::Enum StorageDistributed::getQueryProcessingStage(const Con cluster = optimized_cluster; } - return getQueryProcessingStageImpl(context, to_stage, cluster); + /// If there is only one node, the query can be fully processed by the + /// shard, initiator will work as a proxy only. + if (getClusterQueriedNodes(settings, cluster) == 1) + return QueryProcessingStage::Complete; + + if (settings.optimize_skip_unused_shards && + settings.optimize_distributed_group_by_sharding_key && + has_sharding_key && + sharding_key_is_deterministic) + { + Block sharding_key_block = sharding_key_expr->getSampleBlock(); + auto stage = getOptimizedQueryProcessingStage(query_ptr, settings.extremes, sharding_key_block); + if (stage) + { + LOG_DEBUG(log, "Force processing stage to {}", QueryProcessingStage::toString(*stage)); + return *stage; + } + } + + return QueryProcessingStage::WithMergeableState; } Pipe StorageDistributed::read( diff --git a/src/Storages/StorageDistributed.h b/src/Storages/StorageDistributed.h index aafd5b754f9..a29a147d4bc 100644 --- a/src/Storages/StorageDistributed.h +++ b/src/Storages/StorageDistributed.h @@ -66,8 +66,6 @@ public: bool isRemote() const override { return true; } - /// Return true if distributed_group_by_no_merge may be applied. - bool canForceGroupByNoMerge(const Context &, QueryProcessingStage::Enum to_stage, const ASTPtr &) const; QueryProcessingStage::Enum getQueryProcessingStage(const Context &, QueryProcessingStage::Enum to_stage, const ASTPtr &) const override; Pipe read( diff --git a/src/Storages/StorageMerge.cpp b/src/Storages/StorageMerge.cpp index 2d96a59392b..42ac9cb6a5b 100644 --- a/src/Storages/StorageMerge.cpp +++ b/src/Storages/StorageMerge.cpp @@ -452,6 +452,8 @@ Block StorageMerge::getQueryHeader( } case QueryProcessingStage::WithMergeableState: case QueryProcessingStage::Complete: + case QueryProcessingStage::WithMergeableStateAfterAggregation: + case QueryProcessingStage::MAX: { auto query = query_info.query->clone(); removeJoin(*query->as()); diff --git a/src/Storages/StorageView.cpp b/src/Storages/StorageView.cpp index 1a95b7ea21f..4b7733c1cd2 100644 --- a/src/Storages/StorageView.cpp +++ b/src/Storages/StorageView.cpp @@ -104,7 +104,13 @@ void StorageView::replaceWithSubquery(ASTSelectQuery & outer_query, ASTPtr view_ ASTTableExpression * table_expression = getFirstTableExpression(outer_query); if (!table_expression->database_and_table_name) - throw Exception("Logical error: incorrect table expression", ErrorCodes::LOGICAL_ERROR); + { + // If it's a view table function, add a fake db.table name. + if (table_expression->table_function && table_expression->table_function->as()->name == "view") + table_expression->database_and_table_name = std::make_shared("__view"); + else + throw Exception("Logical error: incorrect table expression", ErrorCodes::LOGICAL_ERROR); + } DatabaseAndTableWithAlias db_table(table_expression->database_and_table_name); String alias = db_table.alias.empty() ? db_table.table : db_table.alias; diff --git a/src/Storages/tests/merge_selector.cpp b/src/Storages/tests/merge_selector.cpp index b4b668f1b02..9433e38c648 100644 --- a/src/Storages/tests/merge_selector.cpp +++ b/src/Storages/tests/merge_selector.cpp @@ -14,8 +14,8 @@ int main(int, char **) { using namespace DB; - IMergeSelector::Partitions partitions(1); - IMergeSelector::PartsInPartition & parts = partitions.back(); + IMergeSelector::PartsRanges partitions(1); + IMergeSelector::PartsRange & parts = partitions.back(); SimpleMergeSelector::Settings settings; // settings.base = 2; @@ -53,7 +53,7 @@ int main(int, char **) while (parts.size() > 1) { - IMergeSelector::PartsInPartition selected_parts = selector.select(partitions, 0); + IMergeSelector::PartsRange selected_parts = selector.select(partitions, 0); if (selected_parts.empty()) { diff --git a/src/Storages/tests/merge_selector2.cpp b/src/Storages/tests/merge_selector2.cpp index a2c30733326..fd05b2f3bb1 100644 --- a/src/Storages/tests/merge_selector2.cpp +++ b/src/Storages/tests/merge_selector2.cpp @@ -19,8 +19,8 @@ int main(int, char **) { using namespace DB; - IMergeSelector::Partitions partitions(1); - IMergeSelector::PartsInPartition & parts = partitions.back(); + IMergeSelector::PartsRanges partitions(1); + IMergeSelector::PartsRange & parts = partitions.back(); /* SimpleMergeSelector::Settings settings; SimpleMergeSelector selector(settings);*/ @@ -52,7 +52,7 @@ int main(int, char **) while (parts.size() > 1) { - IMergeSelector::PartsInPartition selected_parts = selector.select(partitions, 100ULL * 1024 * 1024 * 1024); + IMergeSelector::PartsRange selected_parts = selector.select(partitions, 100ULL * 1024 * 1024 * 1024); if (selected_parts.empty()) { diff --git a/src/TableFunctions/TableFunctionView.cpp b/src/TableFunctions/TableFunctionView.cpp new file mode 100644 index 00000000000..6166fa56f47 --- /dev/null +++ b/src/TableFunctions/TableFunctionView.cpp @@ -0,0 +1,45 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include "registerTableFunctions.h" + + +namespace DB +{ +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +StoragePtr TableFunctionView::executeImpl(const ASTPtr & ast_function, const Context & context, const std::string & table_name) const +{ + if (const auto * function = ast_function->as()) + { + if (function->query) + { + if (auto * select = function->query->as()) + { + auto sample = InterpreterSelectWithUnionQuery::getSampleBlock(function->query, context); + auto columns = ColumnsDescription(sample.getNamesAndTypesList()); + ASTCreateQuery create; + create.select = select; + auto res = StorageView::create(StorageID(getDatabaseName(), table_name), create, columns); + res->startup(); + return res; + } + } + } + throw Exception("Table function '" + getName() + "' requires a query argument.", ErrorCodes::BAD_ARGUMENTS); +} + +void registerTableFunctionView(TableFunctionFactory & factory) +{ + factory.registerFunction(); +} + +} diff --git a/src/TableFunctions/TableFunctionView.h b/src/TableFunctions/TableFunctionView.h new file mode 100644 index 00000000000..49f51823735 --- /dev/null +++ b/src/TableFunctions/TableFunctionView.h @@ -0,0 +1,27 @@ +#pragma once + +#include +#include + + +namespace DB +{ + +/* view(query) + * Turning subquery into a table. + * Useful for passing subquery around. + */ +class TableFunctionView : public ITableFunction +{ +public: + static constexpr auto name = "view"; + std::string getName() const override { return name; } +private: + StoragePtr executeImpl(const ASTPtr & ast_function, const Context & context, const std::string & table_name) const override; + const char * getStorageTypeName() const override { return "View"; } + + UInt64 evaluateArgument(const Context & context, ASTPtr & argument) const; +}; + + +} diff --git a/src/TableFunctions/registerTableFunctions.cpp b/src/TableFunctions/registerTableFunctions.cpp index d312fa2085d..25a495a9185 100644 --- a/src/TableFunctions/registerTableFunctions.cpp +++ b/src/TableFunctions/registerTableFunctions.cpp @@ -30,6 +30,8 @@ void registerTableFunctions() registerTableFunctionODBC(factory); registerTableFunctionJDBC(factory); + registerTableFunctionView(factory); + #if USE_MYSQL registerTableFunctionMySQL(factory); #endif diff --git a/src/TableFunctions/registerTableFunctions.h b/src/TableFunctions/registerTableFunctions.h index a695c1926a0..8ff64a22fea 100644 --- a/src/TableFunctions/registerTableFunctions.h +++ b/src/TableFunctions/registerTableFunctions.h @@ -30,6 +30,8 @@ void registerTableFunctionHDFS(TableFunctionFactory & factory); void registerTableFunctionODBC(TableFunctionFactory & factory); void registerTableFunctionJDBC(TableFunctionFactory & factory); +void registerTableFunctionView(TableFunctionFactory & factory); + #if USE_MYSQL void registerTableFunctionMySQL(TableFunctionFactory & factory); #endif diff --git a/src/TableFunctions/ya.make b/src/TableFunctions/ya.make index 3f73df7e3e2..e87c96073bd 100644 --- a/src/TableFunctions/ya.make +++ b/src/TableFunctions/ya.make @@ -21,6 +21,7 @@ SRCS( TableFunctionRemote.cpp TableFunctionURL.cpp TableFunctionValues.cpp + TableFunctionView.cpp TableFunctionZeros.cpp ) diff --git a/tests/integration/CMakeLists.txt b/tests/integration/CMakeLists.txt index 8280464051f..f57ade79471 100644 --- a/tests/integration/CMakeLists.txt +++ b/tests/integration/CMakeLists.txt @@ -18,7 +18,7 @@ if(MAKE_STATIC_LIBRARIES AND DOCKER_CMD) if(NOT INTEGRATION_USE_RUNNER AND DOCKER_COMPOSE_CMD AND PYTEST_CMD) # To run one test with debug: # cmake . -DPYTEST_OPT="-ss;test_cluster_copier" - add_test(NAME integration-pytest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} COMMAND env ${TEST_USE_BINARIES} "CLICKHOUSE_TESTS_BASE_CONFIG_DIR=${ClickHouse_SOURCE_DIR}/programs/server/" ${PYTEST_STARTER} ${PYTEST_CMD} ${PYTEST_OPT}) + add_test(NAME integration-pytest WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} COMMAND env ${TEST_USE_BINARIES} "CLICKHOUSE_TESTS_BASE_CONFIG_DIR=${ClickHouse_SOURCE_DIR}/programs/server/" "CLICKHOUSE_TESTS_CONFIG_DIR=${ClickHouse_SOURCE_DIR}/tests/config/" ${PYTEST_STARTER} ${PYTEST_CMD} ${PYTEST_OPT}) message(STATUS "Using tests in docker DOCKER=${DOCKER_CMD}; DOCKER_COMPOSE=${DOCKER_COMPOSE_CMD}; PYTEST=${PYTEST_STARTER} ${PYTEST_CMD} ${PYTEST_OPT}") endif() endif() diff --git a/tests/integration/helpers/0_common_enable_dictionaries.xml b/tests/integration/helpers/0_common_enable_dictionaries.xml new file mode 100644 index 00000000000..b6e52983db2 --- /dev/null +++ b/tests/integration/helpers/0_common_enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/dictionaries/*.xml + diff --git a/tests/integration/helpers/client.py b/tests/integration/helpers/client.py index 0ca6a977868..d88a21fbe46 100644 --- a/tests/integration/helpers/client.py +++ b/tests/integration/helpers/client.py @@ -71,7 +71,7 @@ class CommandRequest: self.stderr_file = tempfile.TemporaryFile() self.ignore_error = ignore_error - #print " ".join(command) + print " ".join(command) # we suppress stderror on client becase sometimes thread sanitizer # can print some debug information there diff --git a/tests/integration/helpers/cluster.py b/tests/integration/helpers/cluster.py index f421f979947..8f87518944f 100644 --- a/tests/integration/helpers/cluster.py +++ b/tests/integration/helpers/cluster.py @@ -1,25 +1,25 @@ import base64 +import cassandra.cluster import distutils.dir_util +import docker import errno +import httplib +import logging import os import os.path as p +import pprint +import psycopg2 import pwd +import pymongo +import pymysql import re +import requests import shutil import socket import subprocess import time import urllib -import httplib -import requests import xml.dom.minidom -import logging -import docker -import pprint -import psycopg2 -import pymongo -import pymysql -import cassandra.cluster from dicttoxml import dicttoxml from kazoo.client import KazooClient from kazoo.exceptions import KazooException @@ -88,12 +88,14 @@ class ClickHouseCluster: these directories will contain logs, database files, docker-compose config, ClickHouse configs etc. """ - def __init__(self, base_path, name=None, base_configs_dir=None, server_bin_path=None, client_bin_path=None, + def __init__(self, base_path, name=None, base_config_dir=None, server_bin_path=None, client_bin_path=None, odbc_bridge_bin_path=None, zookeeper_config_path=None, custom_dockerd_host=None): + for param in os.environ.keys(): + print "ENV %40s %s" % (param,os.environ[param]) self.base_dir = p.dirname(base_path) self.name = name if name is not None else '' - self.base_configs_dir = base_configs_dir or os.environ.get('CLICKHOUSE_TESTS_BASE_CONFIG_DIR', + self.base_config_dir = base_config_dir or os.environ.get('CLICKHOUSE_TESTS_BASE_CONFIG_DIR', '/etc/clickhouse-server/') self.server_bin_path = p.realpath( server_bin_path or os.environ.get('CLICKHOUSE_TESTS_SERVER_BIN_PATH', '/usr/bin/clickhouse')) @@ -111,6 +113,7 @@ class ClickHouseCluster: custom_dockerd_host = custom_dockerd_host or os.environ.get('CLICKHOUSE_TESTS_DOCKERD_HOST') self.docker_api_version = os.environ.get("DOCKER_API_VERSION") + self.docker_base_tag = os.environ.get("DOCKER_BASE_TAG", "latest") self.base_cmd = ['docker-compose'] if custom_dockerd_host: @@ -154,6 +157,7 @@ class ClickHouseCluster: self.docker_client = None self.is_up = False + print "CLUSTER INIT base_config_dir:{}".format(self.base_config_dir) def get_client_cmd(self): cmd = self.client_bin_path @@ -161,17 +165,17 @@ class ClickHouseCluster: cmd += " client" return cmd - def add_instance(self, name, config_dir=None, main_configs=None, user_configs=None, macros=None, + def add_instance(self, name, base_config_dir=None, main_configs=None, user_configs=None, dictionaries = None, macros=None, with_zookeeper=False, with_mysql=False, with_kafka=False, with_rabbitmq=False, clickhouse_path_dir=None, with_odbc_drivers=False, with_postgres=False, with_hdfs=False, with_mongo=False, with_redis=False, with_minio=False, with_cassandra=False, - hostname=None, env_variables=None, image="yandex/clickhouse-integration-test", + hostname=None, env_variables=None, image="yandex/clickhouse-integration-test", tag=None, stay_alive=False, ipv4_address=None, ipv6_address=None, with_installed_binary=False, tmpfs=None, zookeeper_docker_compose_path=None, zookeeper_use_tmpfs=True, minio_certs_dir=None): """Add an instance to the cluster. name - the name of the instance directory and the value of the 'instance' macro in ClickHouse. - config_dir - a directory with config files which content will be copied to /etc/clickhouse-server/ directory + base_config_dir - a directory with config.xml and users.xml files which will be copied to /etc/clickhouse-server/ directory main_configs - a list of config files that will be added to config.d/ directory user_configs - a list of config files that will be added to users.d/ directory with_zookeeper - if True, add ZooKeeper configuration to configs and ZooKeeper instances to the cluster. @@ -183,15 +187,40 @@ class ClickHouseCluster: if name in self.instances: raise Exception("Can\'t add instance `%s': there is already an instance with the same name!" % name) + if tag is None: + tag = self.docker_base_tag + instance = ClickHouseInstance( - self, self.base_dir, name, config_dir, main_configs or [], user_configs or [], macros or {}, - with_zookeeper, - self.zookeeper_config_path, with_mysql, with_kafka, with_rabbitmq, with_mongo, with_redis, with_minio, with_cassandra, - self.base_configs_dir, self.server_bin_path, - self.odbc_bridge_bin_path, clickhouse_path_dir, with_odbc_drivers, hostname=hostname, - env_variables=env_variables or {}, image=image, stay_alive=stay_alive, ipv4_address=ipv4_address, + cluster=self, + base_path=self.base_dir, + name=name, + base_config_dir=base_config_dir if base_config_dir else self.base_config_dir, + custom_main_configs=main_configs or [], + custom_user_configs=user_configs or [], + custom_dictionaries=dictionaries or [], + macros=macros or {}, + with_zookeeper=with_zookeeper, + zookeeper_config_path=self.zookeeper_config_path, + with_mysql=with_mysql, + with_kafka=with_kafka, + with_rabbitmq=with_rabbitmq, + with_mongo=with_mongo, + with_redis=with_redis, + with_minio=with_minio, + with_cassandra=with_cassandra, + server_bin_path=self.server_bin_path, + odbc_bridge_bin_path=self.odbc_bridge_bin_path, + clickhouse_path_dir=clickhouse_path_dir, + with_odbc_drivers=with_odbc_drivers, + hostname=hostname, + env_variables=env_variables or {}, + image=image, + tag=tag, + stay_alive=stay_alive, + ipv4_address=ipv4_address, ipv6_address=ipv6_address, - with_installed_binary=with_installed_binary, tmpfs=tmpfs or []) + with_installed_binary=with_installed_binary, + tmpfs=tmpfs or []) docker_compose_yml_dir = get_docker_compose_path() @@ -458,19 +487,19 @@ class ClickHouseCluster: try: minio_client.list_buckets() - logging.info("Connected to Minio.") + print("Connected to Minio.") if minio_client.bucket_exists(self.minio_bucket): minio_client.remove_bucket(self.minio_bucket) minio_client.make_bucket(self.minio_bucket) - logging.info("S3 bucket '%s' created", self.minio_bucket) + print("S3 bucket '%s' created", self.minio_bucket) self.minio_client = minio_client return except Exception as ex: - logging.warning("Can't connect to Minio: %s", str(ex)) + print("Can't connect to Minio: %s", str(ex)) time.sleep(1) raise Exception("Can't wait Minio to start") @@ -482,10 +511,10 @@ class ClickHouseCluster: try: sr_client._send_request(sr_client.url) self.schema_registry_client = sr_client - logging.info("Connected to SchemaRegistry") + print("Connected to SchemaRegistry") return except Exception as ex: - logging.warning("Can't connect to SchemaRegistry: %s", str(ex)) + print("Can't connect to SchemaRegistry: %s", str(ex)) time.sleep(1) def wait_cassandra_to_start(self, timeout=30): @@ -501,25 +530,27 @@ class ClickHouseCluster: time.sleep(1) def start(self, destroy_dirs=True): + print "Cluster start called. is_up={}, destroy_dirs={}".format(self.is_up, destroy_dirs) if self.is_up: return # Just in case kill unstopped containers from previous launch try: - logging.info("Trying to kill unstopped containers...") + print("Trying to kill unstopped containers...") if not subprocess_call(['docker-compose', 'kill']): subprocess_call(['docker-compose', 'down', '--volumes']) - logging.info("Unstopped containers killed") + print("Unstopped containers killed") except: pass try: if destroy_dirs and p.exists(self.instances_dir): - logging.info("Removing instances dir %s", self.instances_dir) + print("Removing instances dir %s", self.instances_dir) shutil.rmtree(self.instances_dir) for instance in self.instances.values(): + print('Setup directory for instance: {} destroy_dirs: {}'.format(instance.name, destroy_dirs)) instance.create_dir(destroy_dir=destroy_dirs) self.docker_client = docker.from_env(version=self.docker_api_version) @@ -527,6 +558,7 @@ class ClickHouseCluster: common_opts = ['up', '-d', '--force-recreate'] if self.with_zookeeper and self.base_zookeeper_cmd: + print('Setup ZooKeeper') env = os.environ.copy() if not self.zookeeper_use_tmpfs: env['ZK_FS'] = 'bind' @@ -545,14 +577,17 @@ class ClickHouseCluster: self.wait_zookeeper_to_start(120) if self.with_mysql and self.base_mysql_cmd: + print('Setup MySQL') subprocess_check_call(self.base_mysql_cmd + common_opts) self.wait_mysql_to_start(120) if self.with_postgres and self.base_postgres_cmd: + print('Setup Postgres') subprocess_check_call(self.base_postgres_cmd + common_opts) self.wait_postgres_to_start(120) if self.with_kafka and self.base_kafka_cmd: + print('Setup Kafka') subprocess_check_call(self.base_kafka_cmd + common_opts + ['--renew-anon-volumes']) self.kafka_docker_id = self.get_instance_docker_id('kafka1') self.wait_schema_registry_to_start(120) @@ -562,14 +597,17 @@ class ClickHouseCluster: self.rabbitmq_docker_id = self.get_instance_docker_id('rabbitmq1') if self.with_hdfs and self.base_hdfs_cmd: + print('Setup HDFS') subprocess_check_call(self.base_hdfs_cmd + common_opts) self.wait_hdfs_to_start(120) if self.with_mongo and self.base_mongo_cmd: + print('Setup Mongo') subprocess_check_call(self.base_mongo_cmd + common_opts) self.wait_mongo_to_start(30) if self.with_redis and self.base_redis_cmd: + print('Setup Redis') subprocess_check_call(self.base_redis_cmd + ['up', '-d', '--force-recreate']) time.sleep(10) @@ -608,18 +646,19 @@ class ClickHouseCluster: self.wait_cassandra_to_start() clickhouse_start_cmd = self.base_cmd + ['up', '-d', '--no-recreate'] - logging.info("Trying to create ClickHouse instance by command %s", ' '.join(map(str, clickhouse_start_cmd))) + print("Trying to create ClickHouse instance by command %s", ' '.join(map(str, clickhouse_start_cmd))) subprocess_check_call(clickhouse_start_cmd) - logging.info("ClickHouse instance created") + print("ClickHouse instance created") + start_deadline = time.time() + 20.0 # seconds for instance in self.instances.itervalues(): instance.docker_client = self.docker_client instance.ip_address = self.get_instance_ip(instance.name) - logging.info("Waiting for ClickHouse start...") + print("Waiting for ClickHouse start...") instance.wait_for_start(start_deadline) - logging.info("ClickHouse started") + print("ClickHouse started") instance.client = Client(instance.ip_address, command=self.client_bin_path) @@ -633,7 +672,10 @@ class ClickHouseCluster: def shutdown(self, kill=True): sanitizer_assert_instance = None with open(self.docker_logs_path, "w+") as f: - subprocess.check_call(self.base_cmd + ['logs'], stdout=f) + try: + subprocess.check_call(self.base_cmd + ['logs'], stdout=f) + except Exception as e: + print "Unable to get logs from docker." f.seek(0) for line in f: if SANITIZER_SIGN in line: @@ -641,8 +683,15 @@ class ClickHouseCluster: break if kill: - subprocess_check_call(self.base_cmd + ['kill']) - subprocess_check_call(self.base_cmd + ['down', '--volumes', '--remove-orphans']) + try: + subprocess_check_call(self.base_cmd + ['kill']) + except Exception as e: + print "Kill command failed durung shutdown. {}".format(repr(e)) + + try: + subprocess_check_call(self.base_cmd + ['down', '--volumes', '--remove-orphans']) + except Exception as e: + print "Down + remove orphans failed durung shutdown. {}".format(repr(e)) self.is_up = False @@ -704,10 +753,10 @@ DOCKER_COMPOSE_TEMPLATE = ''' version: '2.3' services: {name}: - image: {image} + image: {image}:{tag} hostname: {hostname} volumes: - - {configs_dir}:/etc/clickhouse-server/ + - {instance_config_dir}:/etc/clickhouse-server/ - {db_dir}:/var/lib/clickhouse/ - {logs_dir}:/var/log/clickhouse-server/ {binary_volume} @@ -723,6 +772,11 @@ services: - {env_file} security_opt: - label:disable + dns_opt: + - attempts:2 + - timeout:1 + - inet6 + - rotate {networks} {app_net} {ipv4_address} @@ -735,11 +789,10 @@ services: class ClickHouseInstance: def __init__( - self, cluster, base_path, name, custom_config_dir, custom_main_configs, custom_user_configs, macros, - with_zookeeper, zookeeper_config_path, with_mysql, with_kafka, with_rabbitmq, with_mongo, with_redis, with_minio, with_cassandra, - base_configs_dir, server_bin_path, odbc_bridge_bin_path, - clickhouse_path_dir, with_odbc_drivers, hostname=None, env_variables=None, - image="yandex/clickhouse-integration-test", + self, cluster, base_path, name, base_config_dir, custom_main_configs, custom_user_configs, custom_dictionaries, + macros, with_zookeeper, zookeeper_config_path, with_mysql, with_kafka, with_rabbitmq, with_mongo, with_redis, with_minio, + with_cassandra, server_bin_path, odbc_bridge_bin_path, clickhouse_path_dir, with_odbc_drivers, hostname=None, env_variables=None, + image="yandex/clickhouse-integration-test", tag="latest", stay_alive=False, ipv4_address=None, ipv6_address=None, with_installed_binary=False, tmpfs=None): self.name = name @@ -749,15 +802,15 @@ class ClickHouseInstance: self.hostname = hostname if hostname is not None else self.name self.tmpfs = tmpfs or [] - self.custom_config_dir = p.abspath(p.join(base_path, custom_config_dir)) if custom_config_dir else None + self.base_config_dir = p.abspath(p.join(base_path, base_config_dir)) if base_config_dir else None self.custom_main_config_paths = [p.abspath(p.join(base_path, c)) for c in custom_main_configs] self.custom_user_config_paths = [p.abspath(p.join(base_path, c)) for c in custom_user_configs] + self.custom_dictionaries_paths = [p.abspath(p.join(base_path, c)) for c in custom_dictionaries] self.clickhouse_path_dir = p.abspath(p.join(base_path, clickhouse_path_dir)) if clickhouse_path_dir else None self.macros = macros if macros is not None else {} self.with_zookeeper = with_zookeeper self.zookeeper_config_path = zookeeper_config_path - self.base_configs_dir = base_configs_dir self.server_bin_path = server_bin_path self.odbc_bridge_bin_path = odbc_bridge_bin_path @@ -773,7 +826,7 @@ class ClickHouseInstance: self.docker_compose_path = p.join(self.path, 'docker_compose.yml') self.env_variables = env_variables or {} if with_odbc_drivers: - self.odbc_ini_path = os.path.dirname(self.docker_compose_path) + "/odbc.ini:/etc/odbc.ini" + self.odbc_ini_path = self.path + "/odbc.ini:/etc/odbc.ini" self.with_mysql = True else: self.odbc_ini_path = "" @@ -783,6 +836,7 @@ class ClickHouseInstance: self.client = None self.default_timeout = 20.0 # 20 sec self.image = image + self.tag = tag self.stay_alive = stay_alive self.ipv4_address = ipv4_address self.ipv6_address = ipv6_address @@ -975,7 +1029,7 @@ class ClickHouseInstance: time_left = deadline - current_time if deadline is not None and current_time >= deadline: raise Exception("Timed out while waiting for instance `{}' with ip address {} to start. " - "Container status: {}".format(self.name, self.ip_address, status)) + "Container status: {}, logs: {}".format(self.name, self.ip_address, status, handle.logs())) # Repeatedly poll the instance address until there is something that listens there. # Usually it means that ClickHouse is ready to accept queries. @@ -1057,40 +1111,46 @@ class ClickHouseInstance: os.makedirs(self.path) - configs_dir = p.abspath(p.join(self.path, 'configs')) - os.mkdir(configs_dir) + instance_config_dir = p.abspath(p.join(self.path, 'configs')) + os.makedirs(instance_config_dir) - shutil.copy(p.join(self.base_configs_dir, 'config.xml'), configs_dir) - shutil.copy(p.join(self.base_configs_dir, 'users.xml'), configs_dir) + print "Copy common default production configuration from {}".format(self.base_config_dir) + shutil.copyfile(p.join(self.base_config_dir, 'config.xml'), p.join(instance_config_dir, 'config.xml')) + shutil.copyfile(p.join(self.base_config_dir, 'users.xml'), p.join(instance_config_dir, 'users.xml')) + print "Create directory for configuration generated in this helper" # used by all utils with any config - conf_d_dir = p.abspath(p.join(configs_dir, 'conf.d')) - # used by server with main config.xml - self.config_d_dir = p.abspath(p.join(configs_dir, 'config.d')) - users_d_dir = p.abspath(p.join(configs_dir, 'users.d')) + conf_d_dir = p.abspath(p.join(instance_config_dir, 'conf.d')) os.mkdir(conf_d_dir) - os.mkdir(self.config_d_dir) - os.mkdir(users_d_dir) + print "Create directory for common tests configuration" + # used by server with main config.xml + self.config_d_dir = p.abspath(p.join(instance_config_dir, 'config.d')) + os.mkdir(self.config_d_dir) + users_d_dir = p.abspath(p.join(instance_config_dir, 'users.d')) + os.mkdir(users_d_dir) + dictionaries_dir = p.abspath(p.join(instance_config_dir, 'dictionaries')) + os.mkdir(dictionaries_dir) + + print "Copy common configuration from helpers" # The file is named with 0_ prefix to be processed before other configuration overloads. shutil.copy(p.join(HELPERS_DIR, '0_common_instance_config.xml'), self.config_d_dir) shutil.copy(p.join(HELPERS_DIR, '0_common_instance_users.xml'), users_d_dir) + if len(self.custom_dictionaries_paths): + shutil.copy(p.join(HELPERS_DIR, '0_common_enable_dictionaries.xml'), self.config_d_dir) - # Generate and write macros file + print "Generate and write macros file" macros = self.macros.copy() macros['instance'] = self.name - with open(p.join(self.config_d_dir, 'macros.xml'), 'w') as macros_config: + with open(p.join(conf_d_dir, 'macros.xml'), 'w') as macros_config: macros_config.write(self.dict_to_xml({"macros": macros})) # Put ZooKeeper config if self.with_zookeeper: shutil.copy(self.zookeeper_config_path, conf_d_dir) - # Copy config dir - if self.custom_config_dir: - distutils.dir_util.copy_tree(self.custom_config_dir, configs_dir) - # Copy config.d configs + print "Copy custom test config files {} to {}".format(self.custom_main_config_paths, self.config_d_dir) for path in self.custom_main_config_paths: shutil.copy(path, self.config_d_dir) @@ -1098,12 +1158,19 @@ class ClickHouseInstance: for path in self.custom_user_config_paths: shutil.copy(path, users_d_dir) + # Copy dictionaries configs to configs/dictionaries + for path in self.custom_dictionaries_paths: + shutil.copy(path, dictionaries_dir) + db_dir = p.abspath(p.join(self.path, 'database')) + print "Setup database dir {}".format(db_dir) os.mkdir(db_dir) if self.clickhouse_path_dir is not None: + print "Database files taken from {}".format(self.clickhouse_path_dir) distutils.dir_util.copy_tree(self.clickhouse_path_dir, db_dir) logs_dir = p.abspath(p.join(self.path, 'logs')) + print "Setup logs dir {}".format(logs_dir) os.mkdir(logs_dir) depends_on = [] @@ -1128,6 +1195,8 @@ class ClickHouseInstance: env_file = _create_env_file(os.path.dirname(self.docker_compose_path), self.env_variables) + print "Env {} stored in {}".format(self.env_variables, env_file) + odbc_ini_path = "" if self.odbc_ini_path: self._create_odbc_config_file() @@ -1138,6 +1207,8 @@ class ClickHouseInstance: if self.stay_alive: entrypoint_cmd = CLICKHOUSE_STAY_ALIVE_COMMAND + print "Entrypoint cmd: {}".format(entrypoint_cmd) + networks = app_net = ipv4_address = ipv6_address = net_aliases = net_alias1 = "" if self.ipv4_address is not None or self.ipv6_address is not None or self.hostname != self.name: networks = "networks:" @@ -1157,14 +1228,16 @@ class ClickHouseInstance: binary_volume = "- " + self.server_bin_path + ":/usr/share/clickhouse_fresh" odbc_bridge_volume = "- " + self.odbc_bridge_bin_path + ":/usr/share/clickhouse-odbc-bridge_fresh" + with open(self.docker_compose_path, 'w') as docker_compose: docker_compose.write(DOCKER_COMPOSE_TEMPLATE.format( image=self.image, + tag=self.tag, name=self.name, hostname=self.hostname, binary_volume=binary_volume, odbc_bridge_volume=odbc_bridge_volume, - configs_dir=configs_dir, + instance_config_dir=instance_config_dir, config_d_dir=self.config_d_dir, db_dir=db_dir, tmpfs=str(self.tmpfs), diff --git a/tests/integration/helpers/dictonaries/decimals_dictionary.xml b/tests/integration/helpers/dictonaries/decimals_dictionary.xml new file mode 100644 index 00000000000..f728fa774a7 --- /dev/null +++ b/tests/integration/helpers/dictonaries/decimals_dictionary.xml @@ -0,0 +1,197 @@ + + + flat_decimals + + + localhost + 9000 + default + + system + decimals
+ + + 0 + + + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + hashed_decimals + + + localhost + 9000 + default + + system + decimals
+ + + 0 + + + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + cache_decimals + + + localhost + 9000 + default + + system + decimals
+ + + 0 + + 1000 + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + complex_hashed_decimals + + + localhost + 9000 + default + + system + decimals
+ + + 0 + + + + + + + key + UInt64 + + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + complex_cache_decimals + + + localhost + 9000 + default + + system + decimals
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + diff --git a/tests/integration/helpers/dictonaries/ints_dictionary.xml b/tests/integration/helpers/dictonaries/ints_dictionary.xml new file mode 100644 index 00000000000..a22dab8933c --- /dev/null +++ b/tests/integration/helpers/dictonaries/ints_dictionary.xml @@ -0,0 +1,514 @@ + + + flat_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + hashed_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + hashed_sparse_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + cache_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + 1000 + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + complex_hashed_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + + + + + + key + UInt64 + + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + complex_cache_ints + + + localhost + 9000 + default + + system + ints
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + +one_cell_cache_ints + + + localhost + 9000 + default + + test_01054 + ints
+ + +0 + + 1 + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + + one_cell_cache_ints_overflow + + + localhost + 9000 + default + + test_01054_overflow + ints
+ + + 0 + + 1 + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + \ No newline at end of file diff --git a/tests/integration/helpers/dictonaries/strings_dictionary.xml b/tests/integration/helpers/dictonaries/strings_dictionary.xml new file mode 100644 index 00000000000..c5643eecb68 --- /dev/null +++ b/tests/integration/helpers/dictonaries/strings_dictionary.xml @@ -0,0 +1,209 @@ + + + flat_strings + + + localhost + 9000 + default + + system + strings
+ + + 0 + + + + + + key + + + str + String + + + + + + + hashed_strings + + + localhost + 9000 + default + + system + strings
+ + + 0 + + + + + + key + + + str + String + + + + + + + cache_strings + + + localhost + 9000 + default + + system + strings
+ + + 0 + + 1000 + + + + key + + + str + String + + + + + + + complex_hashed_strings + + + localhost + 9000 + default + + system + strings
+ + + 0 + + + + + + + key + UInt64 + + + + str + String + + + + + + + complex_cache_strings + + + localhost + 9000 + default + + system + strings
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + str + String + + + + + + + complex_hashed_strings_key + + + localhost + 9000 + default + + system + strings
+ + + 0 + + + + + + + str + String + + + + key + UInt64 + 0 + + + + + + complex_cache_strings_key + + + localhost + 9000 + default + + system + strings
+ + + 0 + + 1000 + + + + + str + String + + + + key + UInt64 + 0 + + + + diff --git a/tests/integration/helpers/test_tools.py b/tests/integration/helpers/test_tools.py index 93265d280df..67ca025c58a 100644 --- a/tests/integration/helpers/test_tools.py +++ b/tests/integration/helpers/test_tools.py @@ -11,6 +11,9 @@ class TSV: raw_lines = contents.splitlines(True) elif isinstance(contents, list): raw_lines = ['\t'.join(map(str, l)) if isinstance(l, list) else str(l) for l in contents] + elif isinstance(contents, TSV): + self.lines = contents.lines + return else: raise TypeError("contents must be either file or string or list, actual type: " + type(contents).__name__) self.lines = [l.strip() for l in raw_lines if l.strip()] diff --git a/tests/integration/runner b/tests/integration/runner index 9fbb87de253..f097a42e52a 100755 --- a/tests/integration/runner +++ b/tests/integration/runner @@ -53,7 +53,7 @@ def check_args_and_update_paths(args): logging.info("base_configs_dir: {}, binary: {}, cases_dir: {} ".format(args.base_configs_dir, args.binary, args.cases_dir)) - for path in [args.binary, args.base_configs_dir, args.cases_dir, CLICKHOUSE_ROOT]: + for path in [args.binary, args.bridge_binary, args.base_configs_dir, args.cases_dir, CLICKHOUSE_ROOT]: if not os.path.exists(path): raise Exception("Path {} doesn't exist".format(path)) @@ -154,6 +154,8 @@ if __name__ == "__main__": env_tags += "-e {}={} ".format("DOCKER_MYSQL_PHP_CLIENT_TAG", tag) elif image == "yandex/clickhouse-postgresql-java-client": env_tags += "-e {}={} ".format("DOCKER_POSTGRESQL_JAVA_CLIENT_TAG", tag) + elif image == "yandex/clickhouse-integration-test": + env_tags += "-e {}={}".format("DOCKER_BASE_TAG", tag) else: logging.info("Unknown image {}".format(image)) diff --git a/tests/integration/test_access_control_on_cluster/test.py b/tests/integration/test_access_control_on_cluster/test.py index 07c72e94be0..9f053afb607 100644 --- a/tests/integration/test_access_control_on_cluster/test.py +++ b/tests/integration/test_access_control_on_cluster/test.py @@ -4,9 +4,9 @@ from helpers.cluster import ClickHouseCluster from helpers.client import QueryRuntimeException cluster = ClickHouseCluster(__file__) -ch1 = cluster.add_instance('ch1', config_dir="configs", with_zookeeper=True) -ch2 = cluster.add_instance('ch2', config_dir="configs", with_zookeeper=True) -ch3 = cluster.add_instance('ch3', config_dir="configs", with_zookeeper=True) +ch1 = cluster.add_instance('ch1', main_configs=["configs/config.d/clusters.xml"], with_zookeeper=True) +ch2 = cluster.add_instance('ch2', main_configs=["configs/config.d/clusters.xml"], with_zookeeper=True) +ch3 = cluster.add_instance('ch3', main_configs=["configs/config.d/clusters.xml"], with_zookeeper=True) @pytest.fixture(scope="module", autouse=True) def started_cluster(): diff --git a/tests/integration/test_adaptive_granularity/test.py b/tests/integration/test_adaptive_granularity/test.py index f60a932b7e8..247a0c8919d 100644 --- a/tests/integration/test_adaptive_granularity/test.py +++ b/tests/integration/test_adaptive_granularity/test.py @@ -9,23 +9,23 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node1 = cluster.add_instance('node1', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node3 = cluster.add_instance('node3', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.6.3.18', with_installed_binary=True) -node4 = cluster.add_instance('node4', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node3 = cluster.add_instance('node3', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.6.3.18', with_installed_binary=True) +node4 = cluster.add_instance('node4', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node5 = cluster.add_instance('node5', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.1.15', with_installed_binary=True) -node6 = cluster.add_instance('node6', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node5 = cluster.add_instance('node5', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.15', with_installed_binary=True) +node6 = cluster.add_instance('node6', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node7 = cluster.add_instance('node7', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.6.3.18', stay_alive=True, with_installed_binary=True) -node8 = cluster.add_instance('node8', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.1.15', stay_alive=True, with_installed_binary=True) +node7 = cluster.add_instance('node7', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.6.3.18', stay_alive=True, with_installed_binary=True) +node8 = cluster.add_instance('node8', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.15', stay_alive=True, with_installed_binary=True) -node9 = cluster.add_instance('node9', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml', 'configs/merge_tree_settings.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.1.15', stay_alive=True, with_installed_binary=True) -node10 = cluster.add_instance('node10', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml', 'configs/merge_tree_settings.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.6.3.18', stay_alive=True, with_installed_binary=True) +node9 = cluster.add_instance('node9', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml', 'configs/merge_tree_settings.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.15', stay_alive=True, with_installed_binary=True) +node10 = cluster.add_instance('node10', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml', 'configs/merge_tree_settings.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.6.3.18', stay_alive=True, with_installed_binary=True) -node11 = cluster.add_instance('node11', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.1.15', stay_alive=True, with_installed_binary=True) -node12 = cluster.add_instance('node12', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server:19.1.15', stay_alive=True, with_installed_binary=True) +node11 = cluster.add_instance('node11', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.15', stay_alive=True, with_installed_binary=True) +node12 = cluster.add_instance('node12', main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.15', stay_alive=True, with_installed_binary=True) def prepare_single_pair_with_setting(first_node, second_node, group): diff --git a/tests/integration/test_adaptive_granularity_different_settings/test.py b/tests/integration/test_adaptive_granularity_different_settings/test.py index 23c95ef1701..d84b438f77f 100644 --- a/tests/integration/test_adaptive_granularity_different_settings/test.py +++ b/tests/integration/test_adaptive_granularity_different_settings/test.py @@ -7,7 +7,7 @@ node1 = cluster.add_instance('node1', with_zookeeper=True) node2 = cluster.add_instance('node2', with_zookeeper=True) # no adaptive granularity by default -node3 = cluster.add_instance('node3', image='yandex/clickhouse-server:19.9.5.36', with_installed_binary=True, stay_alive=True) +node3 = cluster.add_instance('node3', image='yandex/clickhouse-server', tag='19.9.5.36', with_installed_binary=True, stay_alive=True) @pytest.fixture(scope="module") def start_cluster(): diff --git a/tests/integration/test_adaptive_granularity_replicated/test.py b/tests/integration/test_adaptive_granularity_replicated/test.py index 52978ade685..87956c82661 100644 --- a/tests/integration/test_adaptive_granularity_replicated/test.py +++ b/tests/integration/test_adaptive_granularity_replicated/test.py @@ -11,7 +11,7 @@ cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', with_zookeeper=True) node2 = cluster.add_instance('node2', with_zookeeper=True) -node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server:19.1.14', with_installed_binary=True) +node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.14', with_installed_binary=True) @pytest.fixture(scope="module") def start_cluster(): diff --git a/tests/integration/test_allowed_client_hosts/test.py b/tests/integration/test_allowed_client_hosts/test.py index 23f7f0a4abd..f187b6d889c 100644 --- a/tests/integration/test_allowed_client_hosts/test.py +++ b/tests/integration/test_allowed_client_hosts/test.py @@ -4,7 +4,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -server = cluster.add_instance('server', config_dir="configs") +server = cluster.add_instance('server', user_configs=["configs/users.d/network.xml"]) clientA1 = cluster.add_instance('clientA1', hostname = 'clientA1.com') clientA2 = cluster.add_instance('clientA2', hostname = 'clientA2.com') @@ -20,7 +20,12 @@ clientD2 = cluster.add_instance('clientD2', hostname = 'xxx.clientD0002.ru') clientD3 = cluster.add_instance('clientD3', hostname = 'clientD0003.ru') +def check_clickhouse_is_ok(client_node, server_node): + assert client_node.exec_in_container(["bash", "-c", "/usr/bin/curl -s {}:8123 ".format(server_node.hostname)]) == "Ok.\n" + + def query_from_one_node_to_another(client_node, server_node, query): + check_clickhouse_is_ok(client_node, server_node) return client_node.exec_in_container(["bash", "-c", "/usr/bin/clickhouse client --host {} --query {!r}".format(server_node.hostname, query)]) @@ -56,5 +61,6 @@ def test_allowed_host(): for client_node in expected_to_fail: with pytest.raises(Exception) as e: - query_from_one_node_to_another(client_node, server, "SELECT * FROM test_table") + result = query_from_one_node_to_another(client_node, server, "SELECT * FROM test_table") + print("Client node: {} Server node: {} Result: {}".format(client_node, server_node, result)) assert "default: Authentication failed" in str(e) diff --git a/tests/integration/test_allowed_url_from_config/test.py b/tests/integration/test_allowed_url_from_config/test.py index 688f94cb058..2a666e4e2ec 100644 --- a/tests/integration/test_allowed_url_from_config/test.py +++ b/tests/integration/test_allowed_url_from_config/test.py @@ -40,7 +40,7 @@ def test_config_with_only_regexp_hosts(start_cluster): assert node3.query("CREATE TABLE table_test_3_1 (word String) Engine=URL('https://host:80', HDFS)") == "" assert node3.query("CREATE TABLE table_test_3_2 (word String) Engine=URL('https://yandex.ru', CSV)") == "" assert "not allowed" in node3.query_and_get_error("CREATE TABLE table_test_3_3 (word String) Engine=URL('https://host', CSV)") - assert "not allowed" in node3.query_and_get_error("CREATE TABLE table_test_3_4 (word String) Engine=URL('https://yandex2.ru', S3)") + assert "not allowed" in node3.query_and_get_error("CREATE TABLE table_test_3_4 (word String) Engine=URL('https://yandex2.ru', S3)") def test_config_without_allowed_hosts(start_cluster): assert node4.query("CREATE TABLE table_test_4_1 (word String) Engine=URL('https://host:80', CSV)") == "" @@ -49,18 +49,18 @@ def test_config_without_allowed_hosts(start_cluster): assert node4.query("CREATE TABLE table_test_4_4 (word String) Engine=URL('ftp://something.com', S3)") == "" def test_table_function_remote(start_cluster): + assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-{1|2}', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-1,example01-02-1', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remote('example01-0{1,2}-1', system, events", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remote('example01-0{1,2}-{1|2}', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-{01..02}-{1|2}', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed" in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-1,example01-03-1', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed" in node6.query_and_get_error("SELECT * FROM remote('example01-01-{1|3}', system, events)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) + assert "not allowed" in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-0{1,3}-1', system, metrics)", settings={"connections_with_failover_max_tries":1, "connect_timeout_with_failover_ms": 1000, "connect_timeout_with_failover_secure_ms": 1000, "connect_timeout": 1, "send_timeout":1}) assert node6.query("SELECT * FROM remote('localhost', system, events)") != "" assert node6.query("SELECT * FROM remoteSecure('localhost', system, metrics)") != "" assert "URL \"localhost:800\" is not allowed in config.xml" in node6.query_and_get_error("SELECT * FROM remoteSecure('localhost:800', system, events)") assert "URL \"localhost:800\" is not allowed in config.xml" in node6.query_and_get_error("SELECT * FROM remote('localhost:800', system, metrics)") - assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-1,example01-02-1', system, events)") - assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remote('example01-0{1,2}-1', system, events") - assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-{1|2}', system, events)") - assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remote('example01-0{1,2}-{1|2}', system, events)") - assert "not allowed in config.xml" not in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-{01..02}-{1|2}', system, events)") - assert "not allowed" in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-01-1,example01-03-1', system, events)") - assert "not allowed" in node6.query_and_get_error("SELECT * FROM remote('example01-01-{1|3}', system, events)") - assert "not allowed" in node6.query_and_get_error("SELECT * FROM remoteSecure('example01-0{1,3}-1', system, metrics)") def test_redirect(start_cluster): hdfs_api = HDFSApi("root") diff --git a/tests/integration/test_alter_codec/test.py b/tests/integration/test_alter_codec/test.py index 7e038081110..4d251f60b16 100644 --- a/tests/integration/test_alter_codec/test.py +++ b/tests/integration/test_alter_codec/test.py @@ -6,11 +6,9 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - config_dir='configs', main_configs=['configs/logs_config.xml']) node2 = cluster.add_instance('node2', - config_dir='configs', main_configs=['configs/logs_config.xml']) diff --git a/tests/integration/test_atomic_drop_table/test.py b/tests/integration/test_atomic_drop_table/test.py index 279d13ac4da..ee79a3ff080 100644 --- a/tests/integration/test_atomic_drop_table/test.py +++ b/tests/integration/test_atomic_drop_table/test.py @@ -6,7 +6,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs", with_zookeeper=True) +node1 = cluster.add_instance('node1', main_configs=["configs/config.d/zookeeper_session_timeout.xml", "configs/remote_servers.xml"], with_zookeeper=True) @pytest.fixture(scope="module") diff --git a/tests/integration/test_backup_with_other_granularity/test.py b/tests/integration/test_backup_with_other_granularity/test.py index d4ca9bd1bac..c27cd732a05 100644 --- a/tests/integration/test_backup_with_other_granularity/test.py +++ b/tests/integration/test_backup_with_other_granularity/test.py @@ -5,9 +5,9 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server:19.4.5.35', stay_alive=True, with_installed_binary=True) -node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server:19.4.5.35', stay_alive=True, with_installed_binary=True) -node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server:19.4.5.35', stay_alive=True, with_installed_binary=True) +node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.4.5.35', stay_alive=True, with_installed_binary=True) +node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.4.5.35', stay_alive=True, with_installed_binary=True) +node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.4.5.35', stay_alive=True, with_installed_binary=True) node4 = cluster.add_instance('node4') diff --git a/tests/integration/test_backward_compatibility/test.py b/tests/integration/test_backward_compatibility/test.py index 914153342f8..5b51823d361 100644 --- a/tests/integration/test_backward_compatibility/test.py +++ b/tests/integration/test_backward_compatibility/test.py @@ -4,7 +4,7 @@ import helpers.client as client from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server:19.17.8.54', stay_alive=True, with_installed_binary=True) +node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.17.8.54', stay_alive=True, with_installed_binary=True) node2 = cluster.add_instance('node2', with_zookeeper=True) @pytest.fixture(scope="module") diff --git a/tests/integration/test_backward_compatibility/test_aggregate_function_state_avg.py b/tests/integration/test_backward_compatibility/test_aggregate_function_state_avg.py index c9f3acc2e2e..b1b9fecf54e 100644 --- a/tests/integration/test_backward_compatibility/test_aggregate_function_state_avg.py +++ b/tests/integration/test_backward_compatibility/test_aggregate_function_state_avg.py @@ -5,9 +5,9 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - with_zookeeper=False, image='yandex/clickhouse-server:19.16.9.37', stay_alive=True, with_installed_binary=True) + with_zookeeper=False, image='yandex/clickhouse-server', tag='19.16.9.37', stay_alive=True, with_installed_binary=True) node2 = cluster.add_instance('node2', - with_zookeeper=False, image='yandex/clickhouse-server:19.16.9.37', stay_alive=True, with_installed_binary=True) + with_zookeeper=False, image='yandex/clickhouse-server', tag='19.16.9.37', stay_alive=True, with_installed_binary=True) node3 = cluster.add_instance('node3', with_zookeeper=False) node4 = cluster.add_instance('node4', with_zookeeper=False) diff --git a/tests/integration/test_backward_compatibility/test_short_strings_aggregation.py b/tests/integration/test_backward_compatibility/test_short_strings_aggregation.py index 1c264d1e636..5cf78b481b9 100644 --- a/tests/integration/test_backward_compatibility/test_short_strings_aggregation.py +++ b/tests/integration/test_backward_compatibility/test_short_strings_aggregation.py @@ -4,8 +4,8 @@ import helpers.client as client from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', with_zookeeper=False, image='yandex/clickhouse-server:19.16.9.37', stay_alive=True, with_installed_binary=True) -node2 = cluster.add_instance('node2', with_zookeeper=False, image='yandex/clickhouse-server:19.16.9.37', stay_alive=True, with_installed_binary=True) +node1 = cluster.add_instance('node1', with_zookeeper=False, image='yandex/clickhouse-server', tag='19.16.9.37', stay_alive=True, with_installed_binary=True) +node2 = cluster.add_instance('node2', with_zookeeper=False, image='yandex/clickhouse-server', tag='19.16.9.37', stay_alive=True, with_installed_binary=True) node3 = cluster.add_instance('node3', with_zookeeper=False) @pytest.fixture(scope="module") diff --git a/tests/integration/test_cluster_copier/configs/conf.d/clusters.xml b/tests/integration/test_cluster_copier/configs/conf.d/clusters.xml index 54a8822fa98..632ab84d6a2 100644 --- a/tests/integration/test_cluster_copier/configs/conf.d/clusters.xml +++ b/tests/integration/test_cluster_copier/configs/conf.d/clusters.xml @@ -1,80 +1,74 @@ + - - - - - true - - s0_0_0 - 9000 - - - s0_0_1 - 9000 - - - - true - - s0_1_0 - 9000 - - - - - - - true - - s1_0_0 - 9000 - - - s1_0_1 - 9000 - - - - true - - s1_1_0 - 9000 - - - - - - - true - - s0_0_0 - 9000 - - - s0_0_1 - 9000 - - - - - - - - s0_0_0 - 9000 - - - - - - - - - s1_0_0 - 9000 - - - - - + + + + true + + s0_0_0 + 9000 + + + s0_0_1 + 9000 + + + + true + + s0_1_0 + 9000 + + + + + + true + + s1_0_0 + 9000 + + + s1_0_1 + 9000 + + + + true + + s1_1_0 + 9000 + + + + + + true + + s0_0_0 + 9000 + + + s0_0_1 + 9000 + + + + + + + s0_0_0 + 9000 + + + + + + + s1_0_0 + 9000 + + + + diff --git a/tests/integration/test_cluster_copier/test.py b/tests/integration/test_cluster_copier/test.py index 983cac596dc..3f9ca8a053c 100644 --- a/tests/integration/test_cluster_copier/test.py +++ b/tests/integration/test_cluster_copier/test.py @@ -54,7 +54,8 @@ def started_cluster(): for replica_name in replicas: name = "s{}_{}_{}".format(cluster_name, shard_name, replica_name) cluster.add_instance(name, - config_dir="configs", + main_configs=["configs/conf.d/query_log.xml", "configs/conf.d/ddl.xml", "configs/conf.d/clusters.xml"], + user_configs=["configs/users.xml"], macros={"cluster": cluster_name, "shard": shard_name, "replica": replica_name}, with_zookeeper=True) @@ -226,6 +227,7 @@ def execute_task(task, cmd_options): zk.ensure_path(zk_task_path) zk.create(zk_task_path + "/description", task.copier_task_config) + # Run cluster-copier processes on each node docker_api = docker.from_env().api copiers_exec_ids = [] @@ -241,9 +243,11 @@ def execute_task(task, cmd_options): for instance_name in copiers: instance = cluster.instances[instance_name] container = instance.get_docker_handle() + instance.copy_file_to_container(os.path.join(CURRENT_TEST_DIR, "configs/config-copier.xml"), "/etc/clickhouse-server/config-copier.xml") + print "Copied copier config to {}".format(instance.name) exec_id = docker_api.exec_create(container.id, cmd, stderr=True) - docker_api.exec_start(exec_id, detach=True) - + output = docker_api.exec_start(exec_id).decode('utf8') + print(output) copiers_exec_ids.append(exec_id) print "Copier for {} ({}) has started".format(instance.name, instance.ip_address) diff --git a/tests/integration/test_cluster_copier/trivial_test.py b/tests/integration/test_cluster_copier/trivial_test.py index 70c66653cb2..1697f8bbdfa 100644 --- a/tests/integration/test_cluster_copier/trivial_test.py +++ b/tests/integration/test_cluster_copier/trivial_test.py @@ -34,7 +34,7 @@ def started_cluster(): for replica_name in replicas: name = "s{}_{}_{}".format(cluster_name, shard_name, replica_name) cluster.add_instance(name, - config_dir="configs", + main_configs=[], user_configs=[], macros={"cluster": cluster_name, "shard": shard_name, "replica": replica_name}, with_zookeeper=True) diff --git a/tests/integration/test_concurrent_ttl_merges/__init__.py b/tests/integration/test_concurrent_ttl_merges/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/integration/test_concurrent_ttl_merges/configs/fast_background_pool.xml b/tests/integration/test_concurrent_ttl_merges/configs/fast_background_pool.xml new file mode 100644 index 00000000000..e62a0105907 --- /dev/null +++ b/tests/integration/test_concurrent_ttl_merges/configs/fast_background_pool.xml @@ -0,0 +1,9 @@ + + 1 + 0 + 0.0 + 0 + 1 + 1 + 0 + diff --git a/tests/integration/test_concurrent_ttl_merges/test.py b/tests/integration/test_concurrent_ttl_merges/test.py new file mode 100644 index 00000000000..a82da2a1a8b --- /dev/null +++ b/tests/integration/test_concurrent_ttl_merges/test.py @@ -0,0 +1,86 @@ +import time +import pytest + +import helpers.client as client +from helpers.cluster import ClickHouseCluster +from helpers.test_tools import TSV +from helpers.test_tools import assert_eq_with_retry + + +cluster = ClickHouseCluster(__file__) +node1 = cluster.add_instance('node1', main_configs=['configs/fast_background_pool.xml'], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=['configs/fast_background_pool.xml'], with_zookeeper=True) + + +@pytest.fixture(scope="module") +def started_cluster(): + try: + cluster.start() + + yield cluster + + finally: + cluster.shutdown() + + +def count_ttl_merges_in_queue(node, table): + result = node.query("SELECT count() FROM system.replication_queue WHERE merge_type = 'TTL_DELETE' and table = '{}'".format(table)) + if not result: + return 0 + return int(result.strip()) + + +def count_regular_merges_in_queue(node, table): + result = node.query("SELECT count() FROM system.replication_queue WHERE merge_type = 'REGULAR' and table = '{}'".format(table)) + if not result: + return 0 + return int(result.strip()) + + +def count_ttl_merges_in_background_pool(node, table): + result = node.query("SELECT count() FROM system.merges WHERE merge_type = 'TTL_DELETE' and table = '{}'".format(table)) + if not result: + return 0 + return int(result.strip()) + + +def count_regular_merges_in_background_pool(node, table): + result = node.query("SELECT count() FROM system.merges WHERE merge_type = 'REGULAR' and table = '{}'".format(table)) + if not result: + return 0 + return int(result.strip()) + + +def count_running_mutations(node, table): + result = node.query("SELECT count() FROM system.merges WHERE table = '{}' and is_mutation=1".format(table)) + if not result: + return 0 + return int(result.strip()) + + +# This test was introduced to check concurrency for TTLs merges and mutations +# but it revealed a bug when we assign different merges to the same part +# on the borders of partitions. +def test_no_ttl_merges_in_busy_pool(started_cluster): + node1.query("CREATE TABLE test_ttl (d DateTime, key UInt64, data UInt64) ENGINE = MergeTree() ORDER BY tuple() PARTITION BY key TTL d + INTERVAL 1 MONTH SETTINGS merge_with_ttl_timeout = 0, number_of_free_entries_in_pool_to_execute_mutation = 0") + + node1.query("SYSTEM STOP TTL MERGES") + + for i in range(1, 7): + node1.query("INSERT INTO test_ttl SELECT now() - INTERVAL 1 MONTH + number - 1, {}, number FROM numbers(5)".format(i)) + + node1.query("ALTER TABLE test_ttl UPDATE data = data + 1 WHERE sleepEachRow(1) = 0") + + while count_running_mutations(node1, "test_ttl") < 6: + print "Mutations count", count_running_mutations(node1, "test_ttl") + assert count_ttl_merges_in_background_pool(node1, "test_ttl") == 0 + time.sleep(0.5) + + node1.query("SYSTEM START TTL MERGES") + + while count_running_mutations(node1, "test_ttl") == 6: + print "Mutations count after start TTL", count_running_mutations(node1, "test_ttl") + assert node1.query("SELECT count() FROM test_ttl") == "30\n" + time.sleep(0.5) + + assert_eq_with_retry(node1, "SELECT COUNT() FROM test_ttl", "0") diff --git a/tests/integration/test_config_corresponding_root/test.py b/tests/integration/test_config_corresponding_root/test.py index fd5d3eae3ff..1c714654820 100644 --- a/tests/integration/test_config_corresponding_root/test.py +++ b/tests/integration/test_config_corresponding_root/test.py @@ -4,10 +4,9 @@ import pytest from helpers.cluster import ClickHouseCluster SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -config_dir = os.path.join(SCRIPT_DIR, './configs') cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir = config_dir) +node = cluster.add_instance('node', main_configs=["configs/config.d/bad.xml"]) caught_exception = "" @pytest.fixture(scope="module") @@ -19,4 +18,5 @@ def start_cluster(): caught_exception = str(e) def test_work(start_cluster): + print(caught_exception) assert caught_exception.find("Root element doesn't have the corresponding root element as the config file.") != -1 diff --git a/tests/integration/test_custom_settings/test.py b/tests/integration/test_custom_settings/test.py index 444a4d21881..62c765a6ba0 100644 --- a/tests/integration/test_custom_settings/test.py +++ b/tests/integration/test_custom_settings/test.py @@ -2,7 +2,7 @@ import pytest from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir='configs') +node = cluster.add_instance('node', main_configs=["configs/config.d/text_log.xml"], user_configs=["configs/users.d/custom_settings.xml"]) @pytest.fixture(scope="module", autouse=True) diff --git a/tests/integration/test_default_compression_codec/test.py b/tests/integration/test_default_compression_codec/test.py index 2474394abe2..d312a93ba01 100644 --- a/tests/integration/test_default_compression_codec/test.py +++ b/tests/integration/test_default_compression_codec/test.py @@ -8,7 +8,7 @@ cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', main_configs=['configs/default_compression.xml'], with_zookeeper=True) node2 = cluster.add_instance('node2', main_configs=['configs/default_compression.xml'], with_zookeeper=True) -node3 = cluster.add_instance('node3', main_configs=['configs/default_compression.xml'], image='yandex/clickhouse-server:20.3.16', stay_alive=True, with_installed_binary=True) +node3 = cluster.add_instance('node3', main_configs=['configs/default_compression.xml'], image='yandex/clickhouse-server', tag='20.3.16', stay_alive=True, with_installed_binary=True) @pytest.fixture(scope="module") def start_cluster(): diff --git a/tests/integration/test_dictionaries_all_layouts_and_sources/configs/disable_ssl_verification.xml b/tests/integration/test_dictionaries_all_layouts_and_sources/configs/disable_ssl_verification.xml new file mode 100644 index 00000000000..dc9958934d2 --- /dev/null +++ b/tests/integration/test_dictionaries_all_layouts_and_sources/configs/disable_ssl_verification.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + \ No newline at end of file diff --git a/tests/integration/test_dictionaries_all_layouts_and_sources/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_all_layouts_and_sources/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..8a3d6704670 --- /dev/null +++ b/tests/integration/test_dictionaries_all_layouts_and_sources/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*.xml + diff --git a/tests/integration/test_dictionaries_all_layouts_and_sources/external_sources.py b/tests/integration/test_dictionaries_all_layouts_and_sources/external_sources.py index f6985e7de54..fac7dcdea1e 100644 --- a/tests/integration/test_dictionaries_all_layouts_and_sources/external_sources.py +++ b/tests/integration/test_dictionaries_all_layouts_and_sources/external_sources.py @@ -483,23 +483,27 @@ class SourceRedis(ExternalSource): name, internal_hostname, internal_port, docker_hostname, docker_port, user, password ) self.storage_type = storage_type + self.db_index = 1 def get_source_str(self, table_name): return ''' {host} {port} - 0 + {password} + {db_index} {storage_type} '''.format( host=self.docker_hostname, port=self.docker_port, + password=self.password, storage_type=self.storage_type, # simple or hash_map + db_index=self.db_index, ) def prepare(self, structure, table_name, cluster): - self.client = redis.StrictRedis(host=self.internal_hostname, port=self.internal_port) + self.client = redis.StrictRedis(host=self.internal_hostname, port=self.internal_port, db=self.db_index, password=self.password or None) self.prepared = True self.ordered_names = structure.get_ordered_names() @@ -525,7 +529,6 @@ class SourceRedis(ExternalSource): return True return False - class SourceAerospike(ExternalSource): def __init__(self, name, internal_hostname, internal_port, docker_hostname, docker_port, user, password): diff --git a/tests/integration/test_dictionaries_all_layouts_and_sources/test.py b/tests/integration/test_dictionaries_all_layouts_and_sources/test.py index f4b0ba9c1e4..3e11a544229 100644 --- a/tests/integration/test_dictionaries_all_layouts_and_sources/test.py +++ b/tests/integration/test_dictionaries_all_layouts_and_sources/test.py @@ -134,8 +134,8 @@ DICTIONARIES = [] # Key-value dictionaries with only one possible field for key SOURCES_KV = [ - SourceRedis("RedisSimple", "localhost", "6380", "redis1", "6379", "", "", storage_type="simple"), - SourceRedis("RedisHash", "localhost", "6380", "redis1", "6379", "", "", storage_type="hash_map"), + SourceRedis("RedisSimple", "localhost", "6380", "redis1", "6379", "", "clickhouse", storage_type="simple"), + SourceRedis("RedisHash", "localhost", "6380", "redis1", "6379", "", "clickhouse", storage_type="hash_map"), ] DICTIONARIES_KV = [] @@ -181,12 +181,18 @@ def setup_module(module): if not (field.is_key or field.is_range or field.is_range_key): DICTIONARIES_KV.append(get_dict(source, layout, field_keys + [field], field.name)) + cluster = ClickHouseCluster(__file__) + main_configs = [] + main_configs.append(os.path.join('configs', 'disable_ssl_verification.xml')) + + cluster.add_instance('clickhouse1', main_configs=main_configs) + + dictionaries = [] for fname in os.listdir(dict_configs_path): - main_configs.append(os.path.join(dict_configs_path, fname)) - cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) - node = cluster.add_instance('node', main_configs=main_configs, with_mysql=True, with_mongo=True, with_redis=True, with_cassandra=True) - cluster.add_instance('clickhouse1') + dictionaries.append(os.path.join(dict_configs_path, fname)) + + node = cluster.add_instance('node', main_configs=main_configs, dictionaries=dictionaries, with_mysql=True, with_mongo=True, with_redis=True, with_cassandra=True) @pytest.fixture(scope="module") @@ -238,8 +244,8 @@ def remove_mysql_dicts(): TODO remove this when open ssl will be fixed or thread sanitizer will be suppressed """ - global DICTIONARIES - DICTIONARIES = [d for d in DICTIONARIES if not d.name.startswith("MySQL")] + #global DICTIONARIES + #DICTIONARIES = [d for d in DICTIONARIES if not d.name.startswith("MySQL")] @pytest.mark.parametrize("fold", list(range(10))) diff --git a/tests/integration/test_dictionaries_complex_key_cache_string/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_complex_key_cache_string/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..46d148ad9b9 --- /dev/null +++ b/tests/integration/test_dictionaries_complex_key_cache_string/configs/enable_dictionaries.xml @@ -0,0 +1,5 @@ + + + /etc/clickhouse-server/config.d/complex_key_cache_string.xml + /etc/clickhouse-server/config.d/ssd_complex_key_cache_string.xml + diff --git a/tests/integration/test_dictionaries_complex_key_cache_string/test.py b/tests/integration/test_dictionaries_complex_key_cache_string/test.py index 2a62d66a5f8..8c676841f16 100644 --- a/tests/integration/test_dictionaries_complex_key_cache_string/test.py +++ b/tests/integration/test_dictionaries_complex_key_cache_string/test.py @@ -5,13 +5,12 @@ from helpers.cluster import ClickHouseCluster @pytest.fixture(scope="function") def cluster(request): SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) - cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) - + cluster = ClickHouseCluster(__file__) try: if request.param == "memory": - node = cluster.add_instance('node', main_configs=['configs/dictionaries/complex_key_cache_string.xml']) + node = cluster.add_instance('node', main_configs=['configs/enable_dictionaries.xml', 'configs/dictionaries/complex_key_cache_string.xml']) if request.param == "ssd": - node = cluster.add_instance('node', main_configs=['configs/dictionaries/ssd_complex_key_cache_string.xml']) + node = cluster.add_instance('node', main_configs=['configs/enable_dictionaries.xml', 'configs/dictionaries/ssd_complex_key_cache_string.xml']) cluster.start() node.query("create table radars_table (radar_id String, radar_ip String, client_id String) engine=MergeTree() order by radar_id") diff --git a/tests/integration/test_dictionaries_ddl/configs/allow_remote_node.xml b/tests/integration/test_dictionaries_ddl/configs/allow_remote_node.xml new file mode 100644 index 00000000000..5e616865fef --- /dev/null +++ b/tests/integration/test_dictionaries_ddl/configs/allow_remote_node.xml @@ -0,0 +1,5 @@ + + + node1 + + diff --git a/tests/integration/test_dictionaries_ddl/configs/config.xml b/tests/integration/test_dictionaries_ddl/configs/config.xml deleted file mode 100644 index 6ecc7c089ca..00000000000 --- a/tests/integration/test_dictionaries_ddl/configs/config.xml +++ /dev/null @@ -1,23 +0,0 @@ - - - trace - /var/log/clickhouse-server/clickhouse-server.log - /var/log/clickhouse-server/clickhouse-server.err.log - 1000M - 10 - /var/log/clickhouse-server/stderr.log - /var/log/clickhouse-server/stdout.log - - - 9000 - 127.0.0.1 - 500 - 5368709120 - ./clickhouse/ - users.xml - /etc/clickhouse-server/config.d/*.xml - - - node1 - - diff --git a/tests/integration/test_dictionaries_ddl/configs/config_password.xml b/tests/integration/test_dictionaries_ddl/configs/config_password.xml index 3e9ef48f2db..19a56748612 100644 --- a/tests/integration/test_dictionaries_ddl/configs/config_password.xml +++ b/tests/integration/test_dictionaries_ddl/configs/config_password.xml @@ -1,23 +1,7 @@ - - - - - default - default - default - - - - - ::/0 - - default - default - diff --git a/tests/integration/test_dictionaries_ddl/configs/dictionaries/conflict_name_dictionary.xml b/tests/integration/test_dictionaries_ddl/configs/dictionaries/conflict_name_dictionary.xml new file mode 100644 index 00000000000..75e6f8953eb --- /dev/null +++ b/tests/integration/test_dictionaries_ddl/configs/dictionaries/conflict_name_dictionary.xml @@ -0,0 +1,41 @@ + + + test.conflicting_dictionary + + + localhost + 9000 + default + + test + xml_dictionary_table
+ + + + + 0 + 0 + + + + 128 + + + + + id + + + SomeValue1 + UInt8 + 1 + + + + SomeValue2 + String + '' + + + + diff --git a/tests/integration/test_dictionaries_ddl/configs/dictionaries/lazy_load_dictionary.xml b/tests/integration/test_dictionaries_ddl/configs/dictionaries/lazy_load_dictionary.xml new file mode 100644 index 00000000000..d01f7a0155b --- /dev/null +++ b/tests/integration/test_dictionaries_ddl/configs/dictionaries/lazy_load_dictionary.xml @@ -0,0 +1,4 @@ + + false + + diff --git a/tests/integration/test_dictionaries_ddl/configs/user_admin.xml b/tests/integration/test_dictionaries_ddl/configs/user_admin.xml new file mode 100644 index 00000000000..cb41216f71c --- /dev/null +++ b/tests/integration/test_dictionaries_ddl/configs/user_admin.xml @@ -0,0 +1,13 @@ + + + + + + + ::/0 + + default + default + + + diff --git a/tests/integration/test_dictionaries_ddl/configs/user_default.xml b/tests/integration/test_dictionaries_ddl/configs/user_default.xml new file mode 100644 index 00000000000..9b9770acc2d --- /dev/null +++ b/tests/integration/test_dictionaries_ddl/configs/user_default.xml @@ -0,0 +1,11 @@ + + + + + + default + test + + + + diff --git a/tests/integration/test_dictionaries_ddl/configs/users.xml b/tests/integration/test_dictionaries_ddl/configs/users.xml deleted file mode 100644 index 3e53e05aee1..00000000000 --- a/tests/integration/test_dictionaries_ddl/configs/users.xml +++ /dev/null @@ -1,36 +0,0 @@ - - - - - - - - - - - - ::/0 - - default - default - - default - test - - - - - - - ::/0 - - default - default - - - - - - - - diff --git a/tests/integration/test_dictionaries_ddl/test.py b/tests/integration/test_dictionaries_ddl/test.py index 0e613f74e9f..cc7536d4b36 100644 --- a/tests/integration/test_dictionaries_ddl/test.py +++ b/tests/integration/test_dictionaries_ddl/test.py @@ -7,11 +7,11 @@ import warnings SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) -node1 = cluster.add_instance('node1', with_mysql=True, main_configs=['configs/dictionaries/simple_dictionary.xml']) -node2 = cluster.add_instance('node2', with_mysql=True, main_configs=['configs/dictionaries/simple_dictionary.xml', 'configs/dictionaries/lazy_load.xml']) -node3 = cluster.add_instance('node3', main_configs=['configs/dictionaries/dictionary_with_conflict_name.xml']) -node4 = cluster.add_instance('node4', user_configs=['configs/config_password.xml']) # hardcoded value 33333 +cluster = ClickHouseCluster(__file__) +node1 = cluster.add_instance('node1', with_mysql=True, dictionaries=['configs/dictionaries/simple_dictionary.xml'], user_configs=['configs/user_admin.xml', 'configs/user_default.xml']) +node2 = cluster.add_instance('node2', with_mysql=True, dictionaries=['configs/dictionaries/simple_dictionary.xml'], main_configs=['configs/dictionaries/lazy_load.xml', 'configs/allow_remote_node.xml'], user_configs=['configs/user_admin.xml', 'configs/user_default.xml']) +node3 = cluster.add_instance('node3', main_configs=['configs/allow_remote_node.xml'], dictionaries=['configs/dictionaries/dictionary_with_conflict_name.xml', 'configs/dictionaries/conflict_name_dictionary.xml'], user_configs=['configs/user_admin.xml']) +node4 = cluster.add_instance('node4', user_configs=['configs/user_admin.xml', 'configs/config_password.xml']) def create_mysql_conn(user, password, hostname, port): @@ -50,7 +50,7 @@ def started_cluster(): (node2, 'complex_node2_hashed', 'LAYOUT(COMPLEX_KEY_HASHED())'), (node2, 'complex_node2_cache', 'LAYOUT(COMPLEX_KEY_CACHE(SIZE_IN_CELLS 10))'), ]) -def test_crete_and_select_mysql(started_cluster, clickhouse, name, layout): +def test_create_and_select_mysql(started_cluster, clickhouse, name, layout): mysql_conn = create_mysql_conn("root", "clickhouse", "localhost", 3308) execute_mysql_query(mysql_conn, "CREATE DATABASE IF NOT EXISTS clickhouse") execute_mysql_query(mysql_conn, "CREATE TABLE clickhouse.{} (key_field1 int, key_field2 bigint, value1 text, value2 float, PRIMARY KEY (key_field1, key_field2))".format(name)) @@ -94,8 +94,8 @@ def test_crete_and_select_mysql(started_cluster, clickhouse, name, layout): for i in range(172, 200): assert clickhouse.query("SELECT dictGetString('default.{}', 'value1', tuple(toInt32({}), toInt64({})))".format(name, i, i * i)) == str(i) * 3 + '\n' - stroka = clickhouse.query("SELECT dictGetFloat32('default.{}', 'value2', tuple(toInt32({}), toInt64({})))".format(name, i, i * i)).strip() - value = float(stroka) + string = clickhouse.query("SELECT dictGetFloat32('default.{}', 'value2', tuple(toInt32({}), toInt64({})))".format(name, i, i * i)).strip() + value = float(string) assert int(value) == int(i * 2.718) clickhouse.query("select dictGetUInt8('xml_dictionary', 'SomeValue1', toUInt64(17))") == "17\n" @@ -272,4 +272,4 @@ def test_clickhouse_remote(started_cluster): """) node3.query("select dictGetUInt8('test.clickhouse_remote', 'SomeValue1', toUInt64(17))") == '17\n' - + diff --git a/tests/integration/test_dictionaries_dependency_xml/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_dependency_xml/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..89a4c99ef7a --- /dev/null +++ b/tests/integration/test_dictionaries_dependency_xml/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/dep_*.xml + diff --git a/tests/integration/test_dictionaries_dependency_xml/test.py b/tests/integration/test_dictionaries_dependency_xml/test.py index c0ce0af0313..da1146cd54c 100644 --- a/tests/integration/test_dictionaries_dependency_xml/test.py +++ b/tests/integration/test_dictionaries_dependency_xml/test.py @@ -3,11 +3,11 @@ import os from helpers.cluster import ClickHouseCluster from helpers.test_tools import assert_eq_with_retry -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) +ENABLE_DICT_CONFIG = ['configs/enable_dictionaries.xml'] DICTIONARY_FILES = ['configs/dictionaries/dep_x.xml', 'configs/dictionaries/dep_y.xml', 'configs/dictionaries/dep_z.xml'] -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) -instance = cluster.add_instance('instance', main_configs=DICTIONARY_FILES) +cluster = ClickHouseCluster(__file__) +instance = cluster.add_instance('instance', main_configs=ENABLE_DICT_CONFIG+DICTIONARY_FILES,) @pytest.fixture(scope="module") diff --git a/tests/integration/test_dictionaries_mysql/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_mysql/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..76ed6af89ba --- /dev/null +++ b/tests/integration/test_dictionaries_mysql/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/mysql_dict*.xml + diff --git a/tests/integration/test_dictionaries_mysql/test.py b/tests/integration/test_dictionaries_mysql/test.py index 647e36c71b3..4d2a063e91d 100644 --- a/tests/integration/test_dictionaries_mysql/test.py +++ b/tests/integration/test_dictionaries_mysql/test.py @@ -8,10 +8,9 @@ import pymysql.cursors from helpers.cluster import ClickHouseCluster from helpers.test_tools import assert_eq_with_retry -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) CONFIG_FILES = ['configs/dictionaries/mysql_dict1.xml', 'configs/dictionaries/mysql_dict2.xml', 'configs/remote_servers.xml'] - -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) +CONFIG_FILES += ['configs/enable_dictionaries.xml'] +cluster = ClickHouseCluster(__file__) instance = cluster.add_instance('instance', main_configs=CONFIG_FILES, with_mysql = True) create_table_mysql_template = """ diff --git a/tests/integration/test_dictionaries_null_value/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_null_value/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..8a3d6704670 --- /dev/null +++ b/tests/integration/test_dictionaries_null_value/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*.xml + diff --git a/tests/integration/test_dictionaries_null_value/test.py b/tests/integration/test_dictionaries_null_value/test.py index bb840d8f8f7..c4ad3782498 100644 --- a/tests/integration/test_dictionaries_null_value/test.py +++ b/tests/integration/test_dictionaries_null_value/test.py @@ -3,11 +3,11 @@ import os from helpers.cluster import ClickHouseCluster from helpers.test_tools import TSV, assert_eq_with_retry -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) +ENABLE_DICT_CONFIG = ['configs/enable_dictionaries.xml'] DICTIONARY_FILES = ['configs/dictionaries/cache.xml'] -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) -instance = cluster.add_instance('instance', main_configs=DICTIONARY_FILES) +cluster = ClickHouseCluster(__file__) +instance = cluster.add_instance('instance', main_configs=ENABLE_DICT_CONFIG+DICTIONARY_FILES) @pytest.fixture(scope="module") diff --git a/tests/integration/test_dictionaries_select_all/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_select_all/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..fa26ed7ec3d --- /dev/null +++ b/tests/integration/test_dictionaries_select_all/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/dictionary*.xml + diff --git a/tests/integration/test_dictionaries_select_all/test.py b/tests/integration/test_dictionaries_select_all/test.py index 8bad8a9b214..7dc93b2df44 100644 --- a/tests/integration/test_dictionaries_select_all/test.py +++ b/tests/integration/test_dictionaries_select_all/test.py @@ -19,12 +19,12 @@ def setup_module(module): structure = generate_structure() dictionary_files = generate_dictionaries(os.path.join(SCRIPT_DIR, 'configs/dictionaries'), structure) - cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) - instance = cluster.add_instance('instance', main_configs=dictionary_files) + cluster = ClickHouseCluster(__file__) + instance = cluster.add_instance('instance', main_configs=dictionary_files+['configs/enable_dictionaries.xml']) test_table = DictionaryTestTable(os.path.join(SCRIPT_DIR, 'configs/dictionaries/source.tsv')) -@pytest.fixture(scope="module") +@pytest.fixture(scope="module", autouse=True) def started_cluster(): try: cluster.start() diff --git a/tests/integration/test_dictionaries_update_and_reload/configs/enable_dictionaries.xml b/tests/integration/test_dictionaries_update_and_reload/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..8a3d6704670 --- /dev/null +++ b/tests/integration/test_dictionaries_update_and_reload/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*.xml + diff --git a/tests/integration/test_dictionaries_update_and_reload/test.py b/tests/integration/test_dictionaries_update_and_reload/test.py index 5e5c6514dd2..762fd3adc28 100644 --- a/tests/integration/test_dictionaries_update_and_reload/test.py +++ b/tests/integration/test_dictionaries_update_and_reload/test.py @@ -6,10 +6,11 @@ from helpers.client import QueryTimeoutExceedException from helpers.test_tools import assert_eq_with_retry SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) +ENABLE_DICT_CONFIG = ['configs/enable_dictionaries.xml'] DICTIONARY_FILES = ['configs/dictionaries/cache_xypairs.xml', 'configs/dictionaries/executable.xml', 'configs/dictionaries/file.xml', 'configs/dictionaries/file.txt', 'configs/dictionaries/slow.xml'] -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) -instance = cluster.add_instance('instance', main_configs=DICTIONARY_FILES) +cluster = ClickHouseCluster(__file__) +instance = cluster.add_instance('instance', main_configs=ENABLE_DICT_CONFIG+DICTIONARY_FILES) @pytest.fixture(scope="module") diff --git a/tests/integration/test_dictionary_allow_read_expired_keys/configs/enable_dictionaries.xml b/tests/integration/test_dictionary_allow_read_expired_keys/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..8a3d6704670 --- /dev/null +++ b/tests/integration/test_dictionary_allow_read_expired_keys/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*.xml + diff --git a/tests/integration/test_dictionary_allow_read_expired_keys/test_default_reading.py b/tests/integration/test_dictionary_allow_read_expired_keys/test_default_reading.py index 8da882679bd..b6b742c1de8 100644 --- a/tests/integration/test_dictionary_allow_read_expired_keys/test_default_reading.py +++ b/tests/integration/test_dictionary_allow_read_expired_keys/test_default_reading.py @@ -8,11 +8,11 @@ from helpers.cluster import ClickHouseCluster from helpers.cluster import ClickHouseKiller from helpers.network import PartitionManager -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) +cluster = ClickHouseCluster(__file__) dictionary_node = cluster.add_instance('dictionary_node', stay_alive=True) -main_node = cluster.add_instance('main_node', main_configs=['configs/dictionaries/cache_ints_dictionary.xml']) +main_node = cluster.add_instance('main_node', main_configs=['configs/enable_dictionaries.xml', + 'configs/dictionaries/cache_ints_dictionary.xml']) @pytest.fixture(scope="module") def started_cluster(): diff --git a/tests/integration/test_dictionary_allow_read_expired_keys/test_default_string.py b/tests/integration/test_dictionary_allow_read_expired_keys/test_default_string.py index 7d762db2a6d..d6517379086 100644 --- a/tests/integration/test_dictionary_allow_read_expired_keys/test_default_string.py +++ b/tests/integration/test_dictionary_allow_read_expired_keys/test_default_string.py @@ -9,10 +9,10 @@ from helpers.cluster import ClickHouseCluster from helpers.test_tools import TSV SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) +cluster = ClickHouseCluster(__file__) dictionary_node = cluster.add_instance('dictionary_node', stay_alive=True) -main_node = cluster.add_instance('main_node', main_configs=['configs/dictionaries/cache_strings_default_settings.xml']) +main_node = cluster.add_instance('main_node', main_configs=['configs/enable_dictionaries.xml','configs/dictionaries/cache_ints_dictionary.xml','configs/dictionaries/cache_strings_default_settings.xml']) def get_random_string(string_length=8): @@ -26,7 +26,7 @@ def started_cluster(): dictionary_node.query("CREATE DATABASE IF NOT EXISTS test;") dictionary_node.query("DROP TABLE IF EXISTS test.strings;") dictionary_node.query(""" - CREATE TABLE test.strings + CREATE TABLE test.strings (key UInt64, value String) ENGINE = Memory; """) diff --git a/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get.py b/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get.py index 6b0e1936259..44698b380e3 100644 --- a/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get.py +++ b/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get.py @@ -9,11 +9,10 @@ from helpers.cluster import ClickHouseKiller from helpers.network import PartitionManager from helpers.network import PartitionManagerDisabler -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) +cluster = ClickHouseCluster(__file__) dictionary_node = cluster.add_instance('dictionary_node', stay_alive=True) -main_node = cluster.add_instance('main_node', main_configs=['configs/dictionaries/cache_ints_dictionary.xml']) +main_node = cluster.add_instance('main_node', main_configs=['configs/enable_dictionaries.xml', 'configs/dictionaries/cache_ints_dictionary.xml']) @pytest.fixture(scope="module") def started_cluster(): diff --git a/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get_or_default.py b/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get_or_default.py index 3fce7b7398d..e0b546aae24 100644 --- a/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get_or_default.py +++ b/tests/integration/test_dictionary_allow_read_expired_keys/test_dict_get_or_default.py @@ -8,11 +8,10 @@ from helpers.cluster import ClickHouseCluster from helpers.cluster import ClickHouseKiller from helpers.network import PartitionManager -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) +cluster = ClickHouseCluster(__file__) dictionary_node = cluster.add_instance('dictionary_node', stay_alive=True) -main_node = cluster.add_instance('main_node', main_configs=['configs/dictionaries/cache_ints_dictionary.xml']) +main_node = cluster.add_instance('main_node', main_configs=['configs/enable_dictionaries.xml','configs/dictionaries/cache_ints_dictionary.xml']) @pytest.fixture(scope="module") def started_cluster(): diff --git a/tests/integration/test_dictionary_custom_settings/configs/enable_dictionaries.xml b/tests/integration/test_dictionary_custom_settings/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..8a3d6704670 --- /dev/null +++ b/tests/integration/test_dictionary_custom_settings/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*.xml + diff --git a/tests/integration/test_dictionary_custom_settings/test.py b/tests/integration/test_dictionary_custom_settings/test.py index 97874879525..e58b40df527 100644 --- a/tests/integration/test_dictionary_custom_settings/test.py +++ b/tests/integration/test_dictionary_custom_settings/test.py @@ -3,8 +3,7 @@ import pytest from helpers.cluster import ClickHouseCluster -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -config_dir = os.path.join(SCRIPT_DIR, './configs') +ENABLE_DICT_CONFIG = ['configs/enable_dictionaries.xml'] DICTIONARY_FILES = [ 'configs/dictionaries/FileSourceConfig.xml', 'configs/dictionaries/ExecutableSourceConfig.xml', @@ -13,8 +12,8 @@ DICTIONARY_FILES = [ 'configs/dictionaries/ClickHouseSourceConfig.xml' ] -cluster = ClickHouseCluster(__file__, base_configs_dir=config_dir) -instance = cluster.add_instance('node', main_configs=DICTIONARY_FILES, config_dir=config_dir) +cluster = ClickHouseCluster(__file__) +instance = cluster.add_instance('node', main_configs=ENABLE_DICT_CONFIG+DICTIONARY_FILES) def prepare(): node = instance diff --git a/tests/integration/test_dictionary_ddl_on_cluster/test.py b/tests/integration/test_dictionary_ddl_on_cluster/test.py index 909d2e06377..6239fda1752 100644 --- a/tests/integration/test_dictionary_ddl_on_cluster/test.py +++ b/tests/integration/test_dictionary_ddl_on_cluster/test.py @@ -4,10 +4,10 @@ from helpers.cluster import ClickHouseCluster from helpers.client import QueryRuntimeException cluster = ClickHouseCluster(__file__) -ch1 = cluster.add_instance('ch1', config_dir="configs", with_zookeeper=True) -ch2 = cluster.add_instance('ch2', config_dir="configs", with_zookeeper=True) -ch3 = cluster.add_instance('ch3', config_dir="configs", with_zookeeper=True) -ch4 = cluster.add_instance('ch4', config_dir="configs", with_zookeeper=True) +ch1 = cluster.add_instance('ch1', main_configs=["configs/config.d/clusters.xml", "configs/config.d/ddl.xml"], with_zookeeper=True) +ch2 = cluster.add_instance('ch2', main_configs=["configs/config.d/clusters.xml", "configs/config.d/ddl.xml"], with_zookeeper=True) +ch3 = cluster.add_instance('ch3', main_configs=["configs/config.d/clusters.xml", "configs/config.d/ddl.xml"], with_zookeeper=True) +ch4 = cluster.add_instance('ch4', main_configs=["configs/config.d/clusters.xml", "configs/config.d/ddl.xml"], with_zookeeper=True) @pytest.fixture(scope="module") def started_cluster(): diff --git a/tests/integration/test_disk_types/configs/storage.xml b/tests/integration/test_disk_types/configs/storage.xml new file mode 100644 index 00000000000..2bf9a2e363a --- /dev/null +++ b/tests/integration/test_disk_types/configs/storage.xml @@ -0,0 +1,16 @@ + + + + + + s3 + http://minio1:9001/root/data/ + minio + minio123 + + + memory + + + + diff --git a/tests/integration/test_disk_types/test.py b/tests/integration/test_disk_types/test.py index 04346388b47..3c65315a7e3 100644 --- a/tests/integration/test_disk_types/test.py +++ b/tests/integration/test_disk_types/test.py @@ -1,3 +1,4 @@ + import pytest from helpers.cluster import ClickHouseCluster @@ -12,7 +13,7 @@ disk_types = { def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/storage.xml"], with_minio=True) cluster.start() yield cluster finally: diff --git a/tests/integration/test_distributed_backward_compatability/test.py b/tests/integration/test_distributed_backward_compatability/test.py index a0362cea49e..7ce7edb2860 100644 --- a/tests/integration/test_distributed_backward_compatability/test.py +++ b/tests/integration/test_distributed_backward_compatability/test.py @@ -8,7 +8,7 @@ from helpers.test_tools import TSV cluster = ClickHouseCluster(__file__) -node_old = cluster.add_instance('node1', main_configs=['configs/remote_servers.xml'], image='yandex/clickhouse-server:19.17.8.54', stay_alive=True, with_installed_binary=True) +node_old = cluster.add_instance('node1', main_configs=['configs/remote_servers.xml'], image='yandex/clickhouse-server', tag='19.17.8.54', stay_alive=True, with_installed_binary=True) node_new = cluster.add_instance('node2', main_configs=['configs/remote_servers.xml']) @pytest.fixture(scope="module") diff --git a/tests/integration/test_distributed_ddl/cluster.py b/tests/integration/test_distributed_ddl/cluster.py index 082a76cd88d..258478de990 100644 --- a/tests/integration/test_distributed_ddl/cluster.py +++ b/tests/integration/test_distributed_ddl/cluster.py @@ -17,10 +17,17 @@ class ClickHouseClusterWithDDLHelpers(ClickHouseCluster): def prepare(self, replace_hostnames_with_ips=True): try: + main_configs_files = ["clusters.xml", "zookeeper_session_timeout.xml", "macro.xml", "query_log.xml","ddl.xml"] + main_configs = [os.path.join(self.test_config_dir, "config.d", f) for f in main_configs_files] + user_configs = [os.path.join(self.test_config_dir, "users.d", f) for f in ["restricted_user.xml", "query_log.xml"]] + if self.test_config_dir == "configs_secure": + main_configs += [os.path.join(self.test_config_dir, f) for f in ["server.crt", "server.key", "dhparam.pem", "config.d/ssl_conf.xml"]] + for i in xrange(4): self.add_instance( 'ch{}'.format(i+1), - config_dir=self.test_config_dir, + main_configs=main_configs, + user_configs=user_configs, macros={"layer": 0, "shard": i/2 + 1, "replica": i%2 + 1}, with_zookeeper=True) diff --git a/tests/integration/test_distributed_ddl/configs_secure/config.d/ssl_conf.xml b/tests/integration/test_distributed_ddl/configs_secure/config.d/ssl_conf.xml index 696695ddc69..fe39e3712b8 100644 --- a/tests/integration/test_distributed_ddl/configs_secure/config.d/ssl_conf.xml +++ b/tests/integration/test_distributed_ddl/configs_secure/config.d/ssl_conf.xml @@ -1,8 +1,9 @@ - /etc/clickhouse-server/server.crt - /etc/clickhouse-server/server.key + /etc/clickhouse-server/config.d/server.crt + /etc/clickhouse-server/config.d/server.key + /etc/clickhouse-server/config.d/dhparam.pem none true diff --git a/tests/integration/test_distributed_ddl_password/test.py b/tests/integration/test_distributed_ddl_password/test.py index f957f001df1..961b60857dd 100644 --- a/tests/integration/test_distributed_ddl_password/test.py +++ b/tests/integration/test_distributed_ddl_password/test.py @@ -6,12 +6,12 @@ from helpers.test_tools import assert_eq_with_retry from helpers.client import QueryRuntimeException cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs", with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", with_zookeeper=True) -node3 = cluster.add_instance('node3', config_dir="configs", with_zookeeper=True) -node4 = cluster.add_instance('node4', config_dir="configs", with_zookeeper=True) -node5 = cluster.add_instance('node5', config_dir="configs", with_zookeeper=True) -node6 = cluster.add_instance('node6', config_dir="configs", with_zookeeper=True) +node1 = cluster.add_instance('node1', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) +node3 = cluster.add_instance('node3', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) +node4 = cluster.add_instance('node4', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) +node5 = cluster.add_instance('node5', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) +node6 = cluster.add_instance('node6', main_configs=["configs/config.d/clusters.xml"], user_configs=["configs/users.d/default_with_password.xml"], with_zookeeper=True) @pytest.fixture(scope="module") diff --git a/tests/integration/test_distributed_format/test.py b/tests/integration/test_distributed_format/test.py index 291db89ae4c..251ec766b74 100644 --- a/tests/integration/test_distributed_format/test.py +++ b/tests/integration/test_distributed_format/test.py @@ -9,7 +9,7 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir="configs", main_configs=['configs/remote_servers.xml']) +node = cluster.add_instance('node', main_configs=['configs/remote_servers.xml']) cluster_param = pytest.mark.parametrize("cluster", [ ('test_cluster'), diff --git a/tests/integration/test_distributed_respect_user_timeouts/configs_secure/config.d/ssl_conf.xml b/tests/integration/test_distributed_respect_user_timeouts/configs_secure/config.d/ssl_conf.xml index 696695ddc69..fe39e3712b8 100644 --- a/tests/integration/test_distributed_respect_user_timeouts/configs_secure/config.d/ssl_conf.xml +++ b/tests/integration/test_distributed_respect_user_timeouts/configs_secure/config.d/ssl_conf.xml @@ -1,8 +1,9 @@ - /etc/clickhouse-server/server.crt - /etc/clickhouse-server/server.key + /etc/clickhouse-server/config.d/server.crt + /etc/clickhouse-server/config.d/server.key + /etc/clickhouse-server/config.d/dhparam.pem none true diff --git a/tests/integration/test_distributed_respect_user_timeouts/test.py b/tests/integration/test_distributed_respect_user_timeouts/test.py index ba760e90412..dc5168bfdad 100644 --- a/tests/integration/test_distributed_respect_user_timeouts/test.py +++ b/tests/integration/test_distributed_respect_user_timeouts/test.py @@ -1,6 +1,6 @@ import itertools import timeit - +import os.path import pytest from helpers.cluster import ClickHouseCluster @@ -91,8 +91,16 @@ def started_cluster(request): cluster = ClickHouseCluster(__file__) cluster.__with_ssl_config = request.param == "configs_secure" + main_configs = [] + main_configs += [os.path.join(request.param, "config.d/remote_servers.xml")] + if cluster.__with_ssl_config: + main_configs += [os.path.join(request.param, "server.crt")] + main_configs += [os.path.join(request.param, "server.key")] + main_configs += [os.path.join(request.param, "dhparam.pem")] + main_configs += [os.path.join(request.param, "config.d/ssl_conf.xml")] + user_configs = [os.path.join(request.param, "users.d/set_distributed_defaults.xml")] for name in NODES: - NODES[name] = cluster.add_instance(name, config_dir=request.param) + NODES[name] = cluster.add_instance(name, main_configs=main_configs, user_configs=user_configs) try: cluster.start() diff --git a/tests/integration/test_distributed_storage_configuration/test.py b/tests/integration/test_distributed_storage_configuration/test.py index 8dfaab659cb..716dd3e3075 100644 --- a/tests/integration/test_distributed_storage_configuration/test.py +++ b/tests/integration/test_distributed_storage_configuration/test.py @@ -9,7 +9,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node = cluster.add_instance('node', - config_dir='configs', + main_configs=["configs/config.d/storage_configuration.xml"], tmpfs=['/disk1:size=100M', '/disk2:size=100M']) @pytest.fixture(scope='module') diff --git a/tests/integration/test_enabling_access_management/test.py b/tests/integration/test_enabling_access_management/test.py index abb8cd6c07a..4a6ad59f0bb 100644 --- a/tests/integration/test_enabling_access_management/test.py +++ b/tests/integration/test_enabling_access_management/test.py @@ -2,7 +2,7 @@ import pytest from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -instance = cluster.add_instance('instance', config_dir="configs") +instance = cluster.add_instance('instance', user_configs=["configs/users.d/extra_users.xml"]) @pytest.fixture(scope="module", autouse=True) def started_cluster(): diff --git a/tests/integration/test_extreme_deduplication/test.py b/tests/integration/test_extreme_deduplication/test.py index 5c1ae389857..a7e6f10c1f6 100644 --- a/tests/integration/test_extreme_deduplication/test.py +++ b/tests/integration/test_extreme_deduplication/test.py @@ -12,8 +12,8 @@ from helpers.client import QueryTimeoutExceedException cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir='configs', with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 1}) -node2 = cluster.add_instance('node2', config_dir='configs', with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 2}) +node1 = cluster.add_instance('node1', main_configs=["configs/conf.d/merge_tree.xml", "configs/conf.d/remote_servers.xml"], with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 1}) +node2 = cluster.add_instance('node2', main_configs=["configs/conf.d/merge_tree.xml", "configs/conf.d/remote_servers.xml"], with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 2}) nodes = [node1, node2] @pytest.fixture(scope="module") diff --git a/tests/integration/test_grant_and_revoke/test.py b/tests/integration/test_grant_and_revoke/test.py index 92ffb78a1cb..1557e81bce8 100644 --- a/tests/integration/test_grant_and_revoke/test.py +++ b/tests/integration/test_grant_and_revoke/test.py @@ -107,6 +107,15 @@ def test_revoke_requires_grant_option(): assert instance.query("SHOW GRANTS FOR B") == "" +def test_grant_all_on_table(): + instance.query("CREATE USER A, B") + instance.query("GRANT ALL ON test.table TO A WITH GRANT OPTION") + instance.query("GRANT ALL ON test.table TO B", user='A') + assert instance.query("SHOW GRANTS FOR B") == "GRANT SHOW TABLES, SHOW COLUMNS, SHOW DICTIONARIES, SELECT, INSERT, ALTER, CREATE TABLE, CREATE VIEW, CREATE DICTIONARY, DROP TABLE, DROP VIEW, DROP DICTIONARY, TRUNCATE, OPTIMIZE, SYSTEM MERGES, SYSTEM TTL MERGES, SYSTEM FETCHES, SYSTEM MOVES, SYSTEM SENDS, SYSTEM REPLICATION QUEUES, SYSTEM DROP REPLICA, SYSTEM SYNC REPLICA, SYSTEM RESTART REPLICA, SYSTEM FLUSH DISTRIBUTED, dictGet ON test.table TO B\n" + instance.query("REVOKE ALL ON test.table FROM B", user='A') + assert instance.query("SHOW GRANTS FOR B") == "" + + def test_implicit_show_grants(): instance.query("CREATE USER A") assert instance.query("select count() FROM system.databases WHERE name='test'", user="A") == "0\n" diff --git a/tests/integration/test_host_ip_change/configs/dns_update_short.xml b/tests/integration/test_host_ip_change/configs/dns_update_short.xml index 2bfafe2ef21..3317f709b4a 100644 --- a/tests/integration/test_host_ip_change/configs/dns_update_short.xml +++ b/tests/integration/test_host_ip_change/configs/dns_update_short.xml @@ -1,3 +1,3 @@ - 2 + 1 diff --git a/tests/integration/test_host_ip_change/test.py b/tests/integration/test_host_ip_change/test.py index ac35478277c..a2a38158dc4 100644 --- a/tests/integration/test_host_ip_change/test.py +++ b/tests/integration/test_host_ip_change/test.py @@ -106,11 +106,20 @@ def test_ip_change_update_dns_cache(cluster_with_dns_cache_update): # Put some data to source node3 node3.query("INSERT INTO test_table_update VALUES ('2018-10-01', 5), ('2018-10-02', 6), ('2018-10-03', 7)") + + # Check that data is placed on node3 assert node3.query("SELECT count(*) from test_table_update") == "6\n" + curl_result = node4.exec_in_container(["bash", "-c", "curl -s 'node3:8123'"]) + assert curl_result == 'Ok.\n' + cat_resolv = node4.exec_in_container(["bash", "-c", "cat /etc/resolv.conf"]) + print("RESOLV {}".format(cat_resolv)) + + assert_eq_with_retry(node4, "SELECT * FROM remote('node3', 'system', 'one')", "0", sleep_time=0.5) + # Because of DNS cache update, ip of node3 would be updated - assert_eq_with_retry(node4, "SELECT count(*) from test_table_update", "6") + assert_eq_with_retry(node4, "SELECT count(*) from test_table_update", "6", sleep_time=3) # Just to be sure check one more time node3.query("INSERT INTO test_table_update VALUES ('2018-10-01', 8)") diff --git a/tests/integration/test_https_replication/configs/dhparam.pem b/tests/integration/test_https_replication/configs/dhparam.pem new file mode 100644 index 00000000000..2e6cee0798d --- /dev/null +++ b/tests/integration/test_https_replication/configs/dhparam.pem @@ -0,0 +1,8 @@ +-----BEGIN DH PARAMETERS----- +MIIBCAKCAQEAua92DDli13gJ+//ZXyGaggjIuidqB0crXfhUlsrBk9BV1hH3i7fR +XGP9rUdk2ubnB3k2ejBStL5oBrkHm9SzUFSQHqfDjLZjKoUpOEmuDc4cHvX1XTR5 +Pr1vf5cd0yEncJWG5W4zyUB8k++SUdL2qaeslSs+f491HBLDYn/h8zCgRbBvxhxb +9qeho1xcbnWeqkN6Kc9bgGozA16P9NLuuLttNnOblkH+lMBf42BSne/TWt3AlGZf +slKmmZcySUhF8aKfJnLKbkBCFqOtFRh8zBA9a7g+BT/lSANATCDPaAk1YVih2EKb +dpc3briTDbRsiqg2JKMI7+VdULY9bh3EawIBAg== +-----END DH PARAMETERS----- diff --git a/tests/integration/test_https_replication/configs/ssl_conf.xml b/tests/integration/test_https_replication/configs/ssl_conf.xml index 237bbc6af1c..ad7b874ebd3 100644 --- a/tests/integration/test_https_replication/configs/ssl_conf.xml +++ b/tests/integration/test_https_replication/configs/ssl_conf.xml @@ -1,8 +1,9 @@ - /etc/clickhouse-server/server.crt - /etc/clickhouse-server/server.key + /etc/clickhouse-server/config.d/server.crt + /etc/clickhouse-server/config.d/server.key + /etc/clickhouse-server/config.d/dhparam.pem none true @@ -15,4 +16,5 @@ 9010 + diff --git a/tests/integration/test_https_replication/test.py b/tests/integration/test_https_replication/test.py index a34c5faeccc..4974da850b4 100644 --- a/tests/integration/test_https_replication/test.py +++ b/tests/integration/test_https_replication/test.py @@ -23,8 +23,8 @@ def _fill_nodes(nodes, shard): '''.format(shard=shard, replica=node.name)) cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml'], with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml'], with_zookeeper=True) +node1 = cluster.add_instance('node1', main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml', "configs/server.crt", "configs/server.key", "configs/dhparam.pem"], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml', "configs/server.crt", "configs/server.key", "configs/dhparam.pem"], with_zookeeper=True) @pytest.fixture(scope="module") def both_https_cluster(): @@ -78,8 +78,8 @@ def test_replication_after_partition(both_https_cluster): -node3 = cluster.add_instance('node3', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) -node4 = cluster.add_instance('node4', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) +node3 = cluster.add_instance('node3', main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) +node4 = cluster.add_instance('node4', main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) @pytest.fixture(scope="module") def both_http_cluster(): @@ -104,8 +104,8 @@ def test_both_http(both_http_cluster): assert_eq_with_retry(node3, "SELECT id FROM test_table order by id", '111\n222') assert_eq_with_retry(node4, "SELECT id FROM test_table order by id", '111\n222') -node5 = cluster.add_instance('node5', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml'], with_zookeeper=True) -node6 = cluster.add_instance('node6', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) +node5 = cluster.add_instance('node5', main_configs=['configs/remote_servers.xml', 'configs/ssl_conf.xml', "configs/server.crt", "configs/server.key", "configs/dhparam.pem"], with_zookeeper=True) +node6 = cluster.add_instance('node6', main_configs=['configs/remote_servers.xml', 'configs/no_ssl_conf.xml'], with_zookeeper=True) @pytest.fixture(scope="module") def mixed_protocol_cluster(): diff --git a/tests/integration/test_log_family_s3/configs/minio.xml b/tests/integration/test_log_family_s3/configs/minio.xml new file mode 100644 index 00000000000..6c9329a2bbc --- /dev/null +++ b/tests/integration/test_log_family_s3/configs/minio.xml @@ -0,0 +1,13 @@ + + + + + + s3 + http://minio1:9001/root/data/ + minio + minio123 + + + + diff --git a/tests/integration/test_log_family_s3/configs/ssl.xml b/tests/integration/test_log_family_s3/configs/ssl.xml new file mode 100644 index 00000000000..95cdc918bd0 --- /dev/null +++ b/tests/integration/test_log_family_s3/configs/ssl.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + diff --git a/tests/integration/test_log_family_s3/test.py b/tests/integration/test_log_family_s3/test.py index 50e5b2ad19e..3b0d847967b 100644 --- a/tests/integration/test_log_family_s3/test.py +++ b/tests/integration/test_log_family_s3/test.py @@ -11,7 +11,7 @@ logging.getLogger().addHandler(logging.StreamHandler()) def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/minio.xml", "configs/ssl.xml", "configs/config.d/log_conf.xml"], with_minio=True) logging.info("Starting cluster...") cluster.start() logging.info("Cluster started") diff --git a/tests/integration/test_materialize_mysql_database/materialize_with_ddl.py b/tests/integration/test_materialize_mysql_database/materialize_with_ddl.py index 0dff05df3a1..37e204aae48 100644 --- a/tests/integration/test_materialize_mysql_database/materialize_with_ddl.py +++ b/tests/integration/test_materialize_mysql_database/materialize_with_ddl.py @@ -178,7 +178,7 @@ def alter_add_column_with_materialize_mysql_database(clickhouse_node, mysql_node mysql_node.query("ALTER TABLE test_database.test_table_1 ADD COLUMN add_column_1 INT NOT NULL") mysql_node.query("ALTER TABLE test_database.test_table_1 ADD COLUMN add_column_2 INT NOT NULL FIRST") mysql_node.query("ALTER TABLE test_database.test_table_1 ADD COLUMN add_column_3 INT NOT NULL AFTER add_column_1") - mysql_node.query("ALTER TABLE test_database.test_table_1 ADD COLUMN add_column_4 INT NOT NULL DEFAULT " + ("0" if service_name == "mysql5_7" else "(id)")) + mysql_node.query("ALTER TABLE test_database.test_table_1 ADD COLUMN add_column_4 INT NOT NULL DEFAULT " + ("0" if service_name == "mysql1" else "(id)")) # create mapping clickhouse_node.query( @@ -194,9 +194,9 @@ def alter_add_column_with_materialize_mysql_database(clickhouse_node, mysql_node mysql_node.query("ALTER TABLE test_database.test_table_2 ADD COLUMN add_column_1 INT NOT NULL, ADD COLUMN add_column_2 INT NOT NULL FIRST") mysql_node.query( "ALTER TABLE test_database.test_table_2 ADD COLUMN add_column_3 INT NOT NULL AFTER add_column_1, ADD COLUMN add_column_4 INT NOT NULL DEFAULT " + ( - "0" if service_name == "mysql5_7" else "(id)")) + "0" if service_name == "mysql1" else "(id)")) - default_expression = "DEFAULT\t0" if service_name == "mysql5_7" else "DEFAULT\tid" + default_expression = "DEFAULT\t0" if service_name == "mysql1" else "DEFAULT\tid" check_query(clickhouse_node, "DESC test_database.test_table_2 FORMAT TSV", "add_column_2\tInt32\t\t\t\t\t\nid\tInt32\t\t\t\t\t\nadd_column_1\tInt32\t\t\t\t\t\nadd_column_3\tInt32\t\t\t\t\t\nadd_column_4\tInt32\t" + default_expression + "\t\t\t\n_sign\tInt8\tMATERIALIZED\t1\t\t\t\n_version\tUInt64\tMATERIALIZED\t1\t\t\t\n") @@ -323,7 +323,6 @@ def alter_rename_table_with_materialize_mysql_database(clickhouse_node, mysql_no clickhouse_node.query("DROP DATABASE test_database") mysql_node.query("DROP DATABASE test_database") - def query_event_with_empty_transaction(clickhouse_node, mysql_node, service_name): mysql_node.query("CREATE DATABASE test_database") @@ -336,12 +335,10 @@ def query_event_with_empty_transaction(clickhouse_node, mysql_node, service_name service_name)) # Reject one empty GTID QUERY event with 'BEGIN' and 'COMMIT' - mysql_cursor = mysql_node.cursor(pymysql.cursors.DictCursor) + mysql_cursor = mysql_node.alloc_connection().cursor(pymysql.cursors.DictCursor) mysql_cursor.execute("SHOW MASTER STATUS") (uuid, seqs) = mysql_cursor.fetchall()[0]["Executed_Gtid_Set"].split(":") (seq_begin, seq_end) = seqs.split("-") - assert int(seq_begin) == 1 - assert int(seq_end) == 3 next_gtid = uuid + ":" + str(int(seq_end) + 1) mysql_node.query("SET gtid_next='" + next_gtid + "'") mysql_node.query("BEGIN") diff --git a/tests/integration/test_materialize_mysql_database/test.py b/tests/integration/test_materialize_mysql_database/test.py index d45a5e3ceaf..c00b310436d 100644 --- a/tests/integration/test_materialize_mysql_database/test.py +++ b/tests/integration/test_materialize_mysql_database/test.py @@ -11,7 +11,7 @@ from helpers.cluster import ClickHouseCluster, get_docker_compose_path DOCKER_COMPOSE_PATH = get_docker_compose_path() cluster = ClickHouseCluster(__file__) -clickhouse_node = cluster.add_instance('node1', config_dir="configs", with_mysql=False) +clickhouse_node = cluster.add_instance('node1', user_configs=["configs/users.xml"], with_mysql=False) @pytest.fixture(scope="module") @@ -61,8 +61,8 @@ class MySQLNodeInstance: @pytest.fixture(scope="module") def started_mysql_5_7(): - mysql_node = MySQLNodeInstance('root', 'clickhouse', '127.0.0.1', 33307) - docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_5_7.yml') + mysql_node = MySQLNodeInstance('root', 'clickhouse', '127.0.0.1', 3308) + docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql.yml') try: subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d']) @@ -88,7 +88,7 @@ def started_mysql_8_0(): def test_materialize_database_dml_with_mysql_5_7(started_cluster, started_mysql_5_7): - materialize_with_ddl.dml_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") + materialize_with_ddl.dml_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") def test_materialize_database_dml_with_mysql_8_0(started_cluster, started_mysql_8_0): @@ -96,15 +96,15 @@ def test_materialize_database_dml_with_mysql_8_0(started_cluster, started_mysql_ def test_materialize_database_ddl_with_mysql_5_7(started_cluster, started_mysql_5_7): try: - materialize_with_ddl.drop_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.create_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.rename_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.alter_add_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.alter_drop_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") + materialize_with_ddl.drop_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.create_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.rename_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.alter_add_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.alter_drop_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") # mysql 5.7 cannot support alter rename column - # materialize_with_ddl.alter_rename_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.alter_rename_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") - materialize_with_ddl.alter_modify_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql5_7") + # materialize_with_ddl.alter_rename_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.alter_rename_table_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") + materialize_with_ddl.alter_modify_column_with_materialize_mysql_database(clickhouse_node, started_mysql_5_7, "mysql1") except: print(clickhouse_node.query("select '\n', thread_id, query_id, arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n ', addressToLine(x)), trace), '\n') AS sym from system.stack_trace format TSVRaw")) raise @@ -121,7 +121,7 @@ def test_materialize_database_ddl_with_mysql_8_0(started_cluster, started_mysql_ materialize_with_ddl.alter_modify_column_with_materialize_mysql_database(clickhouse_node, started_mysql_8_0, "mysql8_0") def test_materialize_database_ddl_with_empty_transaction_5_7(started_cluster, started_mysql_5_7): - materialize_with_ddl.query_event_with_empty_transaction(clickhouse_node, started_mysql_5_7.alloc_connection(), "mysql5_7") + materialize_with_ddl.query_event_with_empty_transaction(clickhouse_node, started_mysql_5_7, "mysql1") def test_materialize_database_ddl_with_empty_transaction_8_0(started_cluster, started_mysql_8_0): - materialize_with_ddl.query_event_with_empty_transaction(clickhouse_node, started_mysql_8_0.alloc_connection(), "mysql8_0") + materialize_with_ddl.query_event_with_empty_transaction(clickhouse_node, started_mysql_8_0, "mysql8_0") diff --git a/tests/integration/test_max_http_connections_for_replication/test.py b/tests/integration/test_max_http_connections_for_replication/test.py index c421d36c315..0317aa19cc3 100644 --- a/tests/integration/test_max_http_connections_for_replication/test.py +++ b/tests/integration/test_max_http_connections_for_replication/test.py @@ -22,8 +22,8 @@ def _fill_nodes(nodes, shard, connections_count): '''.format(shard=shard, replica=node.name, connections=connections_count)) cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node1 = cluster.add_instance('node1', user_configs=[], main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node2 = cluster.add_instance('node2', user_configs=[], main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) @pytest.fixture(scope="module") def start_small_cluster(): @@ -68,9 +68,9 @@ def test_keepalive_timeout(start_small_cluster): assert not node2.contains_in_log("No message received"), "Found 'No message received' in clickhouse-server.log" -node3 = cluster.add_instance('node3', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node4 = cluster.add_instance('node4', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) -node5 = cluster.add_instance('node5', config_dir="configs", main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node3 = cluster.add_instance('node3', user_configs=[], main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node4 = cluster.add_instance('node4', user_configs=[], main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) +node5 = cluster.add_instance('node5', user_configs=[], main_configs=['configs/remote_servers.xml', 'configs/log_conf.xml'], with_zookeeper=True) @pytest.fixture(scope="module") def start_big_cluster(): diff --git a/tests/integration/test_merge_tree_s3/test.py b/tests/integration/test_merge_tree_s3/test.py index 146383b2e48..c4e7bbefd87 100644 --- a/tests/integration/test_merge_tree_s3/test.py +++ b/tests/integration/test_merge_tree_s3/test.py @@ -14,7 +14,7 @@ logging.getLogger().addHandler(logging.StreamHandler()) def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/config.d/storage_conf.xml", "configs/config.d/bg_processing_pool_conf.xml", "configs/config.d/log_conf.xml"], user_configs=[], with_minio=True) logging.info("Starting cluster...") cluster.start() logging.info("Cluster started") diff --git a/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/query_log.xml b/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/query_log.xml new file mode 100644 index 00000000000..afcc8ba5c67 --- /dev/null +++ b/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/query_log.xml @@ -0,0 +1,9 @@ + + + + system + query_log
+ toYYYYMM(event_date) + 1000 + + diff --git a/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/ssl_conf.xml b/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/ssl_conf.xml new file mode 100644 index 00000000000..95cdc918bd0 --- /dev/null +++ b/tests/integration/test_merge_tree_s3_with_cache/configs/config.d/ssl_conf.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + diff --git a/tests/integration/test_merge_tree_s3_with_cache/test.py b/tests/integration/test_merge_tree_s3_with_cache/test.py index 72c7d97cfed..25c08777ae5 100644 --- a/tests/integration/test_merge_tree_s3_with_cache/test.py +++ b/tests/integration/test_merge_tree_s3_with_cache/test.py @@ -11,7 +11,9 @@ logging.getLogger().addHandler(logging.StreamHandler()) def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/config.d/log_conf.xml", "configs/config.d/storage_conf.xml", + "configs/config.d/ssl_conf.xml", "configs/config.d/query_log.xml"], + user_configs=["configs/config.d/users.xml"], with_minio=True) logging.info("Starting cluster...") cluster.start() logging.info("Cluster started") diff --git a/tests/integration/test_multiple_disks/test.py b/tests/integration/test_multiple_disks/test.py index bb54920593f..8a4c0a9c30f 100644 --- a/tests/integration/test_multiple_disks/test.py +++ b/tests/integration/test_multiple_disks/test.py @@ -13,16 +13,14 @@ from helpers.test_tools import TSV cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - config_dir='configs', - main_configs=['configs/logs_config.xml'], + main_configs=['configs/logs_config.xml', 'configs/config.d/storage_configuration.xml', 'configs/config.d/cluster.xml'], with_zookeeper=True, stay_alive=True, tmpfs=['/jbod1:size=40M', '/jbod2:size=40M', '/external:size=200M'], macros={"shard": 0, "replica": 1} ) node2 = cluster.add_instance('node2', - config_dir='configs', - main_configs=['configs/logs_config.xml'], + main_configs=['configs/logs_config.xml', 'configs/config.d/storage_configuration.xml', 'configs/config.d/cluster.xml'], with_zookeeper=True, stay_alive=True, tmpfs=['/jbod1:size=40M', '/jbod2:size=40M', '/external:size=200M'], diff --git a/tests/integration/test_mysql_database_engine/test.py b/tests/integration/test_mysql_database_engine/test.py index 2791cc7b382..efbbe6d4104 100644 --- a/tests/integration/test_mysql_database_engine/test.py +++ b/tests/integration/test_mysql_database_engine/test.py @@ -127,7 +127,6 @@ def test_bad_arguments_for_mysql_database_engine(started_cluster): with contextlib.closing(MySQLNodeInstance('root', 'clickhouse', '127.0.0.1', port=3308)) as mysql_node: with pytest.raises(QueryRuntimeException) as exception: mysql_node.query("CREATE DATABASE IF NOT EXISTS test_bad_arguments DEFAULT CHARACTER SET 'utf8'") - clickhouse_node.query("CREATE DATABASE test_database ENGINE = MySQL('mysql1:3306', test_bad_arguments, root, 'clickhouse')") - + clickhouse_node.query("CREATE DATABASE test_database_bad_arguments ENGINE = MySQL('mysql1:3306', test_bad_arguments, root, 'clickhouse')") assert 'Database engine MySQL requested literal argument.' in str(exception.value) mysql_node.query("DROP DATABASE test_bad_arguments") diff --git a/tests/integration/test_mysql_protocol/configs/log_conf.xml b/tests/integration/test_mysql_protocol/configs/log_conf.xml new file mode 100644 index 00000000000..0346e43c81d --- /dev/null +++ b/tests/integration/test_mysql_protocol/configs/log_conf.xml @@ -0,0 +1,10 @@ + + + + trace + /var/log/clickhouse-server/clickhouse-server.log + /var/log/clickhouse-server/clickhouse-server.err.log + 1000M + 10 + + diff --git a/tests/integration/test_mysql_protocol/configs/mysql.xml b/tests/integration/test_mysql_protocol/configs/mysql.xml new file mode 100644 index 00000000000..a3ebc6e8576 --- /dev/null +++ b/tests/integration/test_mysql_protocol/configs/mysql.xml @@ -0,0 +1,4 @@ + + + 9001 + diff --git a/tests/integration/test_mysql_protocol/configs/ssl_conf.xml b/tests/integration/test_mysql_protocol/configs/ssl_conf.xml new file mode 100644 index 00000000000..5938b80ccb8 --- /dev/null +++ b/tests/integration/test_mysql_protocol/configs/ssl_conf.xml @@ -0,0 +1,18 @@ + + + + + + + /etc/clickhouse-server/config.d/server.crt + /etc/clickhouse-server/config.d/server.key + + /etc/clickhouse-server/config.d/dhparam.pem + none + true + true + sslv2,sslv3 + true + + + diff --git a/tests/integration/test_mysql_protocol/test.py b/tests/integration/test_mysql_protocol/test.py index a31961dbd16..510c2821687 100644 --- a/tests/integration/test_mysql_protocol/test.py +++ b/tests/integration/test_mysql_protocol/test.py @@ -17,9 +17,10 @@ from helpers.cluster import ClickHouseCluster, get_docker_compose_path SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) DOCKER_COMPOSE_PATH = get_docker_compose_path() -config_dir = os.path.join(SCRIPT_DIR, './configs') cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir=config_dir, env_variables={'UBSAN_OPTIONS': 'print_stacktrace=1'}) +node = cluster.add_instance('node', main_configs=["configs/log_conf.xml", "configs/ssl_conf.xml", "configs/mysql.xml", + "configs/dhparam.pem", "configs/server.crt", "configs/server.key"], + user_configs=["configs/users.xml"], env_variables={'UBSAN_OPTIONS': 'print_stacktrace=1'}) server_port = 9001 @@ -36,7 +37,7 @@ def server_address(): @pytest.fixture(scope='module') def mysql_client(): docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_client.yml') - subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--build']) + subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--no-build']) yield docker.from_env().containers.get(cluster.project_name + '_mysql1_1') @@ -62,28 +63,28 @@ def mysql_server(mysql_client): @pytest.fixture(scope='module') def golang_container(): docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_golang_client.yml') - subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--build']) + subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--no-build']) yield docker.from_env().containers.get(cluster.project_name + '_golang1_1') @pytest.fixture(scope='module') def php_container(): docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_php_client.yml') - subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--build']) + subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--no-build']) yield docker.from_env().containers.get(cluster.project_name + '_php1_1') @pytest.fixture(scope='module') def nodejs_container(): docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_js_client.yml') - subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--build']) + subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--no-build']) yield docker.from_env().containers.get(cluster.project_name + '_mysqljs1_1') @pytest.fixture(scope='module') def java_container(): docker_compose = os.path.join(DOCKER_COMPOSE_PATH, 'docker_compose_mysql_java_client.yml') - subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--build']) + subprocess.check_call(['docker-compose', '-p', cluster.project_name, '-f', docker_compose, 'up', '--no-recreate', '-d', '--no-build']) yield docker.from_env().containers.get(cluster.project_name + '_java1_1') @@ -282,6 +283,23 @@ def test_mysql_federated(mysql_server, server_address): assert stdout == '\n'.join(['col', '0', '0', '1', '1', '5', '5', '']) +def test_mysql_set_variables(mysql_client, server_address): + code, (stdout, stderr) = mysql_client.exec_run(''' + mysql --protocol tcp -h {host} -P {port} default -u default --password=123 + -e + " + SET NAMES=default; + SET character_set_results=default; + SET FOREIGN_KEY_CHECKS=false; + SET AUTOCOMMIT=1; + SET sql_mode='strict'; + SET @@wait_timeout = 2147483; + SET SESSION TRANSACTION ISOLATION LEVEL READ; + " + '''.format(host=server_address, port=server_port), demux=True) + assert code == 0 + + def test_python_client(server_address): client = pymysql.connections.Connection(host=server_address, user='user_with_double_sha1', password='abacaba', database='default', port=server_port) @@ -329,7 +347,7 @@ def test_python_client(server_address): def test_golang_client(server_address, golang_container): # type: (str, Container) -> None - with open(os.path.join(SCRIPT_DIR,'golang.reference')) as fp: + with open(os.path.join(SCRIPT_DIR, 'golang.reference')) as fp: reference = fp.read() code, (stdout, stderr) = golang_container.exec_run('./main --host {host} --port {port} --user default --password 123 --database ' diff --git a/tests/integration/test_odbc_interaction/configs/enable_dictionaries.xml b/tests/integration/test_odbc_interaction/configs/enable_dictionaries.xml new file mode 100644 index 00000000000..93780125e8e --- /dev/null +++ b/tests/integration/test_odbc_interaction/configs/enable_dictionaries.xml @@ -0,0 +1,4 @@ + + + /etc/clickhouse-server/config.d/*dictionary.xml + diff --git a/tests/integration/test_odbc_interaction/configs/odbc_logging.xml b/tests/integration/test_odbc_interaction/configs/odbc_logging.xml new file mode 100644 index 00000000000..029275eb09c --- /dev/null +++ b/tests/integration/test_odbc_interaction/configs/odbc_logging.xml @@ -0,0 +1,8 @@ + + + + /var/log/clickhouse-server/clickhouse-odbc-bridge.log + /var/log/clickhouse-server/clickhouse-odbc-bridge.err.log + trace + + diff --git a/tests/integration/test_odbc_interaction/configs/openssl.xml b/tests/integration/test_odbc_interaction/configs/openssl.xml new file mode 100644 index 00000000000..95cdc918bd0 --- /dev/null +++ b/tests/integration/test_odbc_interaction/configs/openssl.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + diff --git a/tests/integration/test_odbc_interaction/test.py b/tests/integration/test_odbc_interaction/test.py index 46845802083..0577917ded8 100644 --- a/tests/integration/test_odbc_interaction/test.py +++ b/tests/integration/test_odbc_interaction/test.py @@ -7,10 +7,9 @@ import psycopg2 from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT from helpers.cluster import ClickHouseCluster -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -cluster = ClickHouseCluster(__file__, base_configs_dir=os.path.join(SCRIPT_DIR, 'configs')) -node1 = cluster.add_instance('node1', with_odbc_drivers=True, with_mysql=True, image='yandex/clickhouse-integration-test', main_configs=['configs/dictionaries/sqlite3_odbc_hashed_dictionary.xml', 'configs/dictionaries/sqlite3_odbc_cached_dictionary.xml', 'configs/dictionaries/postgres_odbc_hashed_dictionary.xml'], stay_alive=True) +cluster = ClickHouseCluster(__file__) +node1 = cluster.add_instance('node1', with_odbc_drivers=True, with_mysql=True, main_configs=['configs/openssl.xml','configs/odbc_logging.xml','configs/enable_dictionaries.xml','configs/dictionaries/sqlite3_odbc_hashed_dictionary.xml','configs/dictionaries/sqlite3_odbc_cached_dictionary.xml','configs/dictionaries/postgres_odbc_hashed_dictionary.xml'], stay_alive=True) create_table_sql_template = """ CREATE TABLE `clickhouse`.`{}` ( diff --git a/tests/integration/test_old_versions/test.py b/tests/integration/test_old_versions/test.py index d77b4af016a..0336a1196c4 100644 --- a/tests/integration/test_old_versions/test.py +++ b/tests/integration/test_old_versions/test.py @@ -1,3 +1,4 @@ + import time import os import pytest @@ -9,13 +10,13 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node18_14 = cluster.add_instance('node18_14', image='yandex/clickhouse-server:18.14.19', with_installed_binary=True, config_dir="configs") -node19_1 = cluster.add_instance('node19_1', image='yandex/clickhouse-server:19.1.16', with_installed_binary=True, config_dir="configs") -node19_4 = cluster.add_instance('node19_4', image='yandex/clickhouse-server:19.4.5.35', with_installed_binary=True, config_dir="configs") -node19_8 = cluster.add_instance('node19_8', image='yandex/clickhouse-server:19.8.3.8', with_installed_binary=True, config_dir="configs") -node19_11 = cluster.add_instance('node19_11', image='yandex/clickhouse-server:19.11.13.74', with_installed_binary=True, config_dir="configs") -node19_13 = cluster.add_instance('node19_13', image='yandex/clickhouse-server:19.13.7.57', with_installed_binary=True, config_dir="configs") -node19_16 = cluster.add_instance('node19_16', image='yandex/clickhouse-server:19.16.2.2', with_installed_binary=True, config_dir="configs") +node18_14 = cluster.add_instance('node18_14', image='yandex/clickhouse-server', tag='18.14.19', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_1 = cluster.add_instance('node19_1', image='yandex/clickhouse-server', tag='19.1.16', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_4 = cluster.add_instance('node19_4', image='yandex/clickhouse-server', tag='19.4.5.35', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_8 = cluster.add_instance('node19_8', image='yandex/clickhouse-server', tag='19.8.3.8', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_11 = cluster.add_instance('node19_11', image='yandex/clickhouse-server', tag='19.11.13.74', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_13 = cluster.add_instance('node19_13', image='yandex/clickhouse-server', tag='19.13.7.57', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) +node19_16 = cluster.add_instance('node19_16', image='yandex/clickhouse-server', tag='19.16.2.2', with_installed_binary=True, main_configs=["configs/config.d/test_cluster.xml"]) old_nodes = [node18_14, node19_1, node19_4, node19_8, node19_11, node19_13, node19_16] new_node = cluster.add_instance('node_new') diff --git a/tests/integration/test_polymorphic_parts/test.py b/tests/integration/test_polymorphic_parts/test.py index d3ebbd8c7a8..ed89f768d4c 100644 --- a/tests/integration/test_polymorphic_parts/test.py +++ b/tests/integration/test_polymorphic_parts/test.py @@ -53,29 +53,29 @@ def create_tables_old_format(name, nodes, shard): ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/{shard}/{name}', '{repl}', date, id, 64) '''.format(name=name, shard=shard, repl=i)) -node1 = cluster.add_instance('node1', config_dir="configs", with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", with_zookeeper=True) +node1 = cluster.add_instance('node1', main_configs=[], user_configs=["configs/users.d/not_optimize_count.xml"], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=[], user_configs=["configs/users.d/not_optimize_count.xml"], with_zookeeper=True) settings_default = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 512, 'min_rows_for_compact_part' : 0} settings_compact_only = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 1000000, 'min_rows_for_compact_part' : 0} settings_not_adaptive = {'index_granularity_bytes' : 0, 'min_rows_for_wide_part' : 512, 'min_rows_for_compact_part' : 0} -node3 = cluster.add_instance('node3', config_dir="configs", with_zookeeper=True) -node4 = cluster.add_instance('node4', config_dir="configs", main_configs=['configs/no_leader.xml'], with_zookeeper=True) +node3 = cluster.add_instance('node3', main_configs=[], user_configs=["configs/users.d/not_optimize_count.xml"], with_zookeeper=True) +node4 = cluster.add_instance('node4', user_configs=["configs/users.d/not_optimize_count.xml"], main_configs=['configs/no_leader.xml'], with_zookeeper=True) settings_compact = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 512, 'min_rows_for_compact_part' : 0} settings_wide = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 0, 'min_rows_for_compact_part' : 0} -node5 = cluster.add_instance('node5', config_dir='configs', main_configs=['configs/compact_parts.xml'], with_zookeeper=True) -node6 = cluster.add_instance('node6', config_dir='configs', main_configs=['configs/compact_parts.xml'], with_zookeeper=True) +node5 = cluster.add_instance('node5', main_configs=['configs/compact_parts.xml'], with_zookeeper=True) +node6 = cluster.add_instance('node6', main_configs=['configs/compact_parts.xml'], with_zookeeper=True) settings_in_memory = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 512, 'min_rows_for_compact_part' : 256} -node9 = cluster.add_instance('node9', config_dir="configs", with_zookeeper=True, stay_alive=True) -node10 = cluster.add_instance('node10', config_dir="configs", with_zookeeper=True) +node9 = cluster.add_instance('node9', with_zookeeper=True, stay_alive=True) +node10 = cluster.add_instance('node10', with_zookeeper=True) -node11 = cluster.add_instance('node11', config_dir="configs", main_configs=['configs/do_not_merge.xml'], with_zookeeper=True, stay_alive=True) -node12 = cluster.add_instance('node12', config_dir="configs", main_configs=['configs/do_not_merge.xml'], with_zookeeper=True, stay_alive=True) +node11 = cluster.add_instance('node11', main_configs=['configs/do_not_merge.xml'], with_zookeeper=True, stay_alive=True) +node12 = cluster.add_instance('node12', main_configs=['configs/do_not_merge.xml'], with_zookeeper=True, stay_alive=True) @pytest.fixture(scope="module") def start_cluster(): @@ -213,8 +213,8 @@ def test_different_part_types_on_replicas(start_cluster, table, part_type): "WHERE table = '{}' AND active GROUP BY part_type ORDER BY part_type".format(table))) == TSV(expected) -node7 = cluster.add_instance('node7', config_dir="configs_old", with_zookeeper=True, image='yandex/clickhouse-server:19.17.8.54', stay_alive=True, with_installed_binary=True) -node8 = cluster.add_instance('node8', config_dir="configs", with_zookeeper=True) +node7 = cluster.add_instance('node7', user_configs=["configs_old/users.d/not_optimize_count.xml"], with_zookeeper=True, image='yandex/clickhouse-server', tag='19.17.8.54', stay_alive=True, with_installed_binary=True) +node8 = cluster.add_instance('node8', user_configs=["configs/users.d/not_optimize_count.xml"], with_zookeeper=True) settings7 = {'index_granularity_bytes' : 10485760} settings8 = {'index_granularity_bytes' : 10485760, 'min_rows_for_wide_part' : 512, 'min_rows_for_compact_part' : 0} diff --git a/tests/integration/test_postgresql_protocol/configs/default_passwd.xml b/tests/integration/test_postgresql_protocol/configs/default_passwd.xml new file mode 100644 index 00000000000..86f5b6657c2 --- /dev/null +++ b/tests/integration/test_postgresql_protocol/configs/default_passwd.xml @@ -0,0 +1,13 @@ + + + + + + + + + + 123 + + + diff --git a/tests/integration/test_postgresql_protocol/configs/log.xml b/tests/integration/test_postgresql_protocol/configs/log.xml new file mode 100644 index 00000000000..7f6380b0393 --- /dev/null +++ b/tests/integration/test_postgresql_protocol/configs/log.xml @@ -0,0 +1,10 @@ + + + + trace + /var/log/clickhouse-server/clickhouse-server.log + /var/log/clickhouse-server/clickhouse-server.err.log + 1000M + 10 + + diff --git a/tests/integration/test_postgresql_protocol/configs/postresql.xml b/tests/integration/test_postgresql_protocol/configs/postresql.xml new file mode 100644 index 00000000000..aedfb59bedb --- /dev/null +++ b/tests/integration/test_postgresql_protocol/configs/postresql.xml @@ -0,0 +1,4 @@ + + + 5433 + diff --git a/tests/integration/test_postgresql_protocol/configs/ssl_conf.xml b/tests/integration/test_postgresql_protocol/configs/ssl_conf.xml new file mode 100644 index 00000000000..271cb987218 --- /dev/null +++ b/tests/integration/test_postgresql_protocol/configs/ssl_conf.xml @@ -0,0 +1,18 @@ + + + + + + + /etc/clickhouse-server/config.d/server.crt + /etc/clickhouse-server/config.d/server.key + + /etc/clickhouse-server/config.d/dhparam.pem + none + true + true + sslv2,sslv3 + true + + + diff --git a/tests/integration/test_postgresql_protocol/test.py b/tests/integration/test_postgresql_protocol/test.py index 527c652229e..939e8231931 100644 --- a/tests/integration/test_postgresql_protocol/test.py +++ b/tests/integration/test_postgresql_protocol/test.py @@ -19,11 +19,12 @@ from helpers.cluster import ClickHouseCluster, get_docker_compose_path psycopg2.extras.register_uuid() SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -config_dir = os.path.join(SCRIPT_DIR, './configs') DOCKER_COMPOSE_PATH = get_docker_compose_path() cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir=config_dir, env_variables={'UBSAN_OPTIONS': 'print_stacktrace=1'}) +node = cluster.add_instance('node', main_configs=["configs/postresql.xml", "configs/log.xml", "configs/ssl_conf.xml", + "configs/dhparam.pem", "configs/server.crt", "configs/server.key"], + user_configs=["configs/default_passwd.xml"], env_variables={'UBSAN_OPTIONS': 'print_stacktrace=1'}) server_port = 5433 diff --git a/tests/integration/test_profile_events_s3/configs/log.xml b/tests/integration/test_profile_events_s3/configs/log.xml new file mode 100644 index 00000000000..0346e43c81d --- /dev/null +++ b/tests/integration/test_profile_events_s3/configs/log.xml @@ -0,0 +1,10 @@ + + + + trace + /var/log/clickhouse-server/clickhouse-server.log + /var/log/clickhouse-server/clickhouse-server.err.log + 1000M + 10 + + diff --git a/tests/integration/test_profile_events_s3/configs/query_log.xml b/tests/integration/test_profile_events_s3/configs/query_log.xml new file mode 100644 index 00000000000..afcc8ba5c67 --- /dev/null +++ b/tests/integration/test_profile_events_s3/configs/query_log.xml @@ -0,0 +1,9 @@ + + + + system + query_log
+ toYYYYMM(event_date) + 1000 + + diff --git a/tests/integration/test_profile_events_s3/configs/ssl_conf.xml b/tests/integration/test_profile_events_s3/configs/ssl_conf.xml new file mode 100644 index 00000000000..95cdc918bd0 --- /dev/null +++ b/tests/integration/test_profile_events_s3/configs/ssl_conf.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + diff --git a/tests/integration/test_profile_events_s3/test.py b/tests/integration/test_profile_events_s3/test.py index f98505757bf..e2cb10499e7 100644 --- a/tests/integration/test_profile_events_s3/test.py +++ b/tests/integration/test_profile_events_s3/test.py @@ -17,7 +17,7 @@ def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/config.d/storage_conf.xml", "configs/log.xml", "configs/query_log.xml", "configs/ssl_conf.xml"], with_minio=True) logging.info("Starting cluster...") cluster.start() diff --git a/tests/integration/test_quorum_inserts/test.py b/tests/integration/test_quorum_inserts/test.py index f490c13ca27..e89611c0d99 100644 --- a/tests/integration/test_quorum_inserts/test.py +++ b/tests/integration/test_quorum_inserts/test.py @@ -7,18 +7,15 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -zero = cluster.add_instance("zero", - config_dir="configs", +zero = cluster.add_instance("zero", user_configs=["configs/users.d/settings.xml"], macros={"cluster": "anime", "shard": "0", "replica": "zero"}, with_zookeeper=True) -first = cluster.add_instance("first", - config_dir="configs", +first = cluster.add_instance("first", user_configs=["configs/users.d/settings.xml"], macros={"cluster": "anime", "shard": "0", "replica": "first"}, with_zookeeper=True) -second = cluster.add_instance("second", - config_dir="configs", +second = cluster.add_instance("second", user_configs=["configs/users.d/settings.xml"], macros={"cluster": "anime", "shard": "0", "replica": "second"}, with_zookeeper=True) diff --git a/tests/integration/test_quota/test.py b/tests/integration/test_quota/test.py index 27aa353b9b1..4c97d127ad0 100644 --- a/tests/integration/test_quota/test.py +++ b/tests/integration/test_quota/test.py @@ -6,28 +6,38 @@ import re import time cluster = ClickHouseCluster(__file__) -instance = cluster.add_instance('instance', - config_dir="configs") +instance = cluster.add_instance('instance', user_configs=["configs/users.d/assign_myquota.xml", "configs/users.d/drop_default_quota.xml", "configs/users.d/quota.xml"]) -def system_quotas(): - return TSV(instance.query("SELECT * FROM system.quotas ORDER BY name")) +def check_system_quotas(canonical): + canonical_tsv = TSV(canonical) + r = TSV(instance.query("SELECT * FROM system.quotas ORDER BY name")) + print("system_quotas: {},\ncanonical: {}".format(r, TSV(canonical_tsv))) + assert r == canonical_tsv -def system_quota_limits(): - return TSV(instance.query("SELECT * FROM system.quota_limits ORDER BY quota_name, duration")) +def system_quota_limits(canonical): + canonical_tsv = TSV(canonical) + r = TSV(instance.query("SELECT * FROM system.quota_limits ORDER BY quota_name, duration")) + print("system_quota_limits: {},\ncanonical: {}".format(r, TSV(canonical_tsv))) + assert r == canonical_tsv -def system_quota_usage(): +def system_quota_usage(canonical): + canonical_tsv = TSV(canonical) query = "SELECT quota_name, quota_key, duration, queries, max_queries, errors, max_errors, result_rows, max_result_rows,"\ "result_bytes, max_result_bytes, read_rows, max_read_rows, read_bytes, max_read_bytes, max_execution_time "\ "FROM system.quota_usage ORDER BY duration" - return TSV(instance.query(query)) + r = TSV(instance.query(query)) + print("system_quota_usage: {},\ncanonical: {}".format(r, TSV(canonical_tsv))) + assert r == canonical_tsv -def system_quotas_usage(): +def system_quotas_usage(canonical): + canonical_tsv = TSV(canonical) query = "SELECT quota_name, quota_key, is_current, duration, queries, max_queries, errors, max_errors, result_rows, max_result_rows, "\ "result_bytes, max_result_bytes, read_rows, max_read_rows, read_bytes, max_read_bytes, max_execution_time "\ "FROM system.quotas_usage ORDER BY quota_name, quota_key, duration" - return TSV(instance.query(query)) - + r = TSV(instance.query(query)) + print("system_quotas_usage: {},\ncanonical: {}".format(r, TSV(canonical_tsv))) + assert r == canonical_tsv def copy_quota_xml(local_file_name, reload_immediately = True): script_dir = os.path.dirname(os.path.realpath(__file__)) @@ -40,7 +50,7 @@ def copy_quota_xml(local_file_name, reload_immediately = True): def started_cluster(): try: cluster.start() - + instance.query("CREATE TABLE test_table(x UInt32) ENGINE = MergeTree ORDER BY tuple()") instance.query("INSERT INTO test_table SELECT number FROM numbers(50)") @@ -61,141 +71,141 @@ def reset_quotas_and_usage_info(): def test_quota_from_users_xml(): - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] - assert system_quotas_usage() == [["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) + system_quotas_usage([["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"]]) instance.query("SELECT COUNT() from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 2, 1000, 0, "\N", 51, "\N", 208, "\N", 50, 1000, 200, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 2, 1000, 0, "\N", 51, "\N", 208, "\N", 50, 1000, 200, "\N", "\N"]]) def test_simpliest_quota(): # Simpliest quota doesn't even track usage. copy_quota_xml('simpliest.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[]", 0, "['default']", "[]"]] - assert system_quota_limits() == "" - assert system_quota_usage() == [["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[]", 0, "['default']", "[]"]]) + system_quota_limits("") + system_quota_usage([["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]]) instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]] + system_quota_usage([["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]]) def test_tracking_quota(): # Now we're tracking usage. copy_quota_xml('tracking.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, "\N", "\N", "\N", "\N", "\N", "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, "\N", 0, "\N", 0, "\N", 0, "\N", 0, "\N", 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, "\N", "\N", "\N", "\N", "\N", "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 0, "\N", 0, "\N", 0, "\N", 0, "\N", 0, "\N", 0, "\N", "\N"]]) instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, "\N", 0, "\N", 50, "\N", 200, "\N", 50, "\N", 200, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 1, "\N", 0, "\N", 50, "\N", 200, "\N", 50, "\N", 200, "\N", "\N"]]) instance.query("SELECT COUNT() from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 2, "\N", 0, "\N", 51, "\N", 208, "\N", 50, "\N", 200, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 2, "\N", 0, "\N", 51, "\N", 208, "\N", 50, "\N", 200, "\N", "\N"]]) def test_exceed_quota(): # Change quota, now the limits are tiny so we will exceed the quota. copy_quota_xml('tiny_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1, 1, 1, "\N", 1, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, 1, 0, 1, 0, 1, 0, "\N", 0, 1, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1, 1, 1, "\N", 1, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 0, 1, 0, 1, 0, 1, 0, "\N", 0, 1, 0, "\N", "\N"]]) assert re.search("Quota.*has\ been\ exceeded", instance.query_and_get_error("SELECT * from test_table")) - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, 1, 1, 1, 0, 1, 0, "\N", 50, 1, 0, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 1, 1, 1, 1, 0, 1, 0, "\N", 50, 1, 0, "\N", "\N"]]) # Change quota, now the limits are enough to execute queries. copy_quota_xml('normal_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, 1000, 1, "\N", 0, "\N", 0, "\N", 50, 1000, 0, "\N", "\N"]] - + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 1, 1000, 1, "\N", 0, "\N", 0, "\N", 50, 1000, 0, "\N", "\N"]]) + instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 2, 1000, 1, "\N", 50, "\N", 200, "\N", 100, 1000, 200, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 2, 1000, 1, "\N", 50, "\N", 200, "\N", 100, 1000, 200, "\N", "\N"]]) def test_add_remove_interval(): - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) # Add interval. copy_quota_xml('two_intervals.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952,63113904]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"], - ["myQuota", 63113904, 1, "\N", "\N", "\N", 30000, "\N", 20000, 120]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"], - ["myQuota", "default", 63113904, 0, "\N", 0, "\N", 0, "\N", 0, 30000, 0, "\N", 0, 20000, 120]] - + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952,63113904]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"], + ["myQuota", 63113904, 1, "\N", "\N", "\N", 30000, "\N", 20000, 120]]) + system_quota_usage([["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"], + ["myQuota", "default", 63113904, 0, "\N", 0, "\N", 0, "\N", 0, 30000, 0, "\N", 0, 20000, 120]]) + instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"], - ["myQuota", "default", 63113904, 1, "\N", 0, "\N", 50, "\N", 200, 30000, 50, "\N", 200, 20000, 120]] + system_quota_usage([["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"], + ["myQuota", "default", 63113904, 1, "\N", 0, "\N", 50, "\N", 200, 30000, 50, "\N", 200, 20000, 120]]) # Remove interval. copy_quota_xml('normal_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"]] - + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 1, 1000, 0, "\N", 50, "\N", 200, "\N", 50, 1000, 200, "\N", "\N"]]) + instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", 31556952, 2, 1000, 0, "\N", 100, "\N", 400, "\N", 100, 1000, 400, "\N", "\N"]] + system_quota_usage([["myQuota", "default", 31556952, 2, 1000, 0, "\N", 100, "\N", 400, "\N", 100, 1000, 400, "\N", "\N"]]) # Remove all intervals. copy_quota_xml('simpliest.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[]", 0, "['default']", "[]"]] - assert system_quota_limits() == "" - assert system_quota_usage() == [["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]] - + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[]", 0, "['default']", "[]"]]) + system_quota_limits("") + system_quota_usage([["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]]) + instance.query("SELECT * from test_table") - assert system_quota_usage() == [["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]] + system_quota_usage([["myQuota", "default", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N", "\N"]]) # Add one interval back. copy_quota_xml('normal_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quota_usage() == [["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quota_usage([["myQuota", "default", 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) def test_add_remove_quota(): - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quotas_usage() == [["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", [31556952], 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quotas_usage([["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) # Add quota. copy_quota_xml('two_quotas.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"], - ["myQuota2", "4590510c-4d13-bf21-ec8a-c2187b092e73", "users.xml", "['client_key','user_name']", "[3600,2629746]", 0, "[]", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"], + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"], + ["myQuota2", "4590510c-4d13-bf21-ec8a-c2187b092e73", "users.xml", "['client_key','user_name']", "[3600,2629746]", 0, "[]", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"], ["myQuota2", 3600, 1, "\N", "\N", 4000, 400000, 4000, 400000, 60], - ["myQuota2", 2629746, 0, "\N", "\N", "\N", "\N", "\N", "\N", 1800]] - assert system_quotas_usage() == [["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + ["myQuota2", 2629746, 0, "\N", "\N", "\N", "\N", "\N", "\N", 1800]]) + system_quotas_usage([["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) # Drop quota. copy_quota_xml('normal_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quotas_usage() == [["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quotas_usage([["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) # Drop all quotas. copy_quota_xml('no_quotas.xml') - assert system_quotas() == "" - assert system_quota_limits() == "" - assert system_quotas_usage() == "" + check_system_quotas("") + system_quota_limits("") + system_quotas_usage("") # Add one quota back. copy_quota_xml('normal_limits.xml') - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] - assert system_quotas_usage() == [["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) + system_quotas_usage([["myQuota", "default", 1, 31556952, 0, 1000, 0, "\N", 0, "\N", 0, "\N", 0, 1000, 0, "\N", "\N"]]) def test_reload_users_xml_by_timer(): - assert system_quotas() == [["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]] - assert system_quota_limits() == [["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]] + check_system_quotas([["myQuota", "e651da9c-a748-8703-061a-7e5e5096dae7", "users.xml", "['user_name']", "[31556952]", 0, "['default']", "[]"]]) + system_quota_limits([["myQuota", 31556952, 0, 1000, "\N", "\N", "\N", 1000, "\N", "\N"]]) time.sleep(1) # The modification time of the 'quota.xml' file should be different, # because config files are reload by timer only when the modification time is changed. @@ -246,7 +256,7 @@ def test_dcl_introspection(): def test_dcl_management(): copy_quota_xml('no_quotas.xml') assert instance.query("SHOW QUOTA") == "" - + instance.query("CREATE QUOTA qA FOR INTERVAL 15 MONTH MAX QUERIES 123 TO CURRENT_USER") assert instance.query("SHOW CREATE QUOTA qA") == "CREATE QUOTA qA FOR INTERVAL 5 quarter MAX queries = 123 TO default\n" assert re.match("qA\\t\\t.*\\t39446190\\t0\\t123\\t0\\t\\\\N\\t0\\t\\\\N\\t0\\t\\\\N\\t0\\t\\\\N\\t0\\t\\\\N\\t.*\\t\\\\N\n", diff --git a/tests/integration/test_random_inserts/test.py b/tests/integration/test_random_inserts/test.py index eb644a7a19c..4e3d8db7e53 100644 --- a/tests/integration/test_random_inserts/test.py +++ b/tests/integration/test_random_inserts/test.py @@ -14,8 +14,8 @@ from helpers.client import CommandRequest cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir='configs', with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 1}) -node2 = cluster.add_instance('node2', config_dir='configs', with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 2}) +node1 = cluster.add_instance('node1', main_configs=["configs/conf.d/merge_tree.xml", "configs/conf.d/remote_servers.xml" ], with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 1}) +node2 = cluster.add_instance('node2', main_configs=["configs/conf.d/merge_tree.xml", "configs/conf.d/remote_servers.xml" ], with_zookeeper=True, macros={"layer": 0, "shard": 0, "replica": 2}) nodes = [node1, node2] @pytest.fixture(scope="module") diff --git a/tests/integration/test_reload_max_table_size_to_drop/configs/max_table_size_to_drop.xml b/tests/integration/test_reload_max_table_size_to_drop/configs/max_table_size_to_drop.xml new file mode 100644 index 00000000000..03d5e33646f --- /dev/null +++ b/tests/integration/test_reload_max_table_size_to_drop/configs/max_table_size_to_drop.xml @@ -0,0 +1,5 @@ + + + 1 + 1 + diff --git a/tests/integration/test_reload_max_table_size_to_drop/test.py b/tests/integration/test_reload_max_table_size_to_drop/test.py index 3959b383fc5..9d0bc244521 100644 --- a/tests/integration/test_reload_max_table_size_to_drop/test.py +++ b/tests/integration/test_reload_max_table_size_to_drop/test.py @@ -1,3 +1,4 @@ + import time import pytest import os @@ -6,10 +7,10 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir="configs") +node = cluster.add_instance('node', main_configs=["configs/max_table_size_to_drop.xml"]) SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -CONFIG_PATH = os.path.join(SCRIPT_DIR, './_instances/node/configs/config.xml') +CONFIG_PATH = os.path.join(SCRIPT_DIR, './_instances/node/configs/config.d/max_table_size_to_drop.xml') @pytest.fixture(scope="module") diff --git a/tests/integration/test_reloading_storage_configuration/test.py b/tests/integration/test_reloading_storage_configuration/test.py index c9effcdd67a..a30d4029d7c 100644 --- a/tests/integration/test_reloading_storage_configuration/test.py +++ b/tests/integration/test_reloading_storage_configuration/test.py @@ -14,7 +14,6 @@ import helpers.cluster cluster = helpers.cluster.ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - config_dir='configs', main_configs=['configs/logs_config.xml'], with_zookeeper=True, stay_alive=True, @@ -22,7 +21,6 @@ node1 = cluster.add_instance('node1', macros={"shard": 0, "replica": 1} ) node2 = cluster.add_instance('node2', - config_dir='configs', main_configs=['configs/logs_config.xml'], with_zookeeper=True, stay_alive=True, diff --git a/tests/integration/test_rename_column/test.py b/tests/integration/test_rename_column/test.py index 029d140d0ed..9a108583347 100644 --- a/tests/integration/test_rename_column/test.py +++ b/tests/integration/test_rename_column/test.py @@ -12,8 +12,9 @@ from helpers.test_tools import TSV node_options = dict( with_zookeeper=True, - main_configs=['configs/remote_servers.xml'], - config_dir='configs', + main_configs=["configs/remote_servers.xml", "configs/config.d/instant_moves.xml", + "configs/config.d/part_log.xml", "configs/config.d/zookeeper_session_timeout.xml", + "configs/config.d/storage_configuration.xml"], tmpfs=['/external:size=200M', '/internal:size=1M']) cluster = ClickHouseCluster(__file__) diff --git a/tests/integration/test_replicated_merge_tree_s3/test.py b/tests/integration/test_replicated_merge_tree_s3/test.py index b5c35c8b330..a77a69b842b 100644 --- a/tests/integration/test_replicated_merge_tree_s3/test.py +++ b/tests/integration/test_replicated_merge_tree_s3/test.py @@ -15,9 +15,9 @@ def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node1", config_dir="configs", macros={'cluster': 'test1'}, with_minio=True, with_zookeeper=True) - cluster.add_instance("node2", config_dir="configs", macros={'cluster': 'test1'}, with_zookeeper=True) - cluster.add_instance("node3", config_dir="configs", macros={'cluster': 'test1'}, with_zookeeper=True) + cluster.add_instance("node1", main_configs=["configs/config.d/storage_conf.xml"], macros={'cluster': 'test1'}, with_minio=True, with_zookeeper=True) + cluster.add_instance("node2", main_configs=["configs/config.d/storage_conf.xml"], macros={'cluster': 'test1'}, with_zookeeper=True) + cluster.add_instance("node3", main_configs=["configs/config.d/storage_conf.xml"], macros={'cluster': 'test1'}, with_zookeeper=True) logging.info("Starting cluster...") cluster.start() diff --git a/tests/integration/test_replicating_constants/test.py b/tests/integration/test_replicating_constants/test.py index f340817b584..b72b9089f65 100644 --- a/tests/integration/test_replicating_constants/test.py +++ b/tests/integration/test_replicating_constants/test.py @@ -5,7 +5,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', with_zookeeper=True) -node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server:19.1.14', with_installed_binary=True) +node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server', tag='19.1.14', with_installed_binary=True) @pytest.fixture(scope="module") def start_cluster(): diff --git a/tests/integration/test_row_policy/configs/users.d/another_user.xml b/tests/integration/test_row_policy/configs/users.d/another_user.xml new file mode 100644 index 00000000000..fb9608e5313 --- /dev/null +++ b/tests/integration/test_row_policy/configs/users.d/another_user.xml @@ -0,0 +1,13 @@ + + + + + + + ::/0 + + default + default + + + \ No newline at end of file diff --git a/tests/integration/test_row_policy/configs/users.d/any_join_distinct_right_table_keys.xml b/tests/integration/test_row_policy/configs/users.d/any_join_distinct_right_table_keys.xml new file mode 100644 index 00000000000..413e64ba3dc --- /dev/null +++ b/tests/integration/test_row_policy/configs/users.d/any_join_distinct_right_table_keys.xml @@ -0,0 +1,8 @@ + + + + + 1 + + + diff --git a/tests/integration/test_row_policy/test.py b/tests/integration/test_row_policy/test.py index 15796ff0c83..dd0495df237 100644 --- a/tests/integration/test_row_policy/test.py +++ b/tests/integration/test_row_policy/test.py @@ -6,8 +6,8 @@ import re import time cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir="configs", with_zookeeper=True) -node2 = cluster.add_instance('node2', config_dir="configs", with_zookeeper=True) +node = cluster.add_instance('node', main_configs=["configs/config.d/remote_servers.xml"], user_configs=["configs/users.d/row_policy.xml", "configs/users.d/another_user.xml", "configs/users.d/any_join_distinct_right_table_keys.xml"], with_zookeeper=True) +node2 = cluster.add_instance('node2', main_configs=["configs/config.d/remote_servers.xml"], user_configs=["configs/users.d/row_policy.xml", "configs/users.d/another_user.xml", "configs/users.d/any_join_distinct_right_table_keys.xml"], with_zookeeper=True) nodes = [node, node2] @@ -42,7 +42,7 @@ def started_cluster(): CREATE TABLE mydb.`.filtered_table4` (a UInt8, b UInt8, c UInt16 ALIAS a + b) ENGINE MergeTree ORDER BY a; INSERT INTO mydb.`.filtered_table4` values (0, 0), (0, 1), (1, 0), (1, 1); - + CREATE TABLE mydb.local (a UInt8, b UInt8) ENGINE MergeTree ORDER BY a; ''') @@ -185,7 +185,7 @@ def test_introspection(): def test_dcl_introspection(): assert node.query("SHOW POLICIES") == TSV(["another ON mydb.filtered_table1", "another ON mydb.filtered_table2", "another ON mydb.filtered_table3", "another ON mydb.local", "default ON mydb.filtered_table1", "default ON mydb.filtered_table2", "default ON mydb.filtered_table3", "default ON mydb.local"]) - + assert node.query("SHOW POLICIES ON mydb.filtered_table1") == TSV([ "another", "default" ]) assert node.query("SHOW POLICIES ON mydb.local") == TSV([ "another", "default" ]) assert node.query("SHOW POLICIES ON mydb.*") == TSV([ "another ON mydb.filtered_table1", "another ON mydb.filtered_table2", "another ON mydb.filtered_table3", "another ON mydb.local", "default ON mydb.filtered_table1", "default ON mydb.filtered_table2", "default ON mydb.filtered_table3", "default ON mydb.local" ]) @@ -195,7 +195,7 @@ def test_dcl_introspection(): assert node.query("SHOW CREATE POLICY default ON mydb.filtered_table2") == "CREATE ROW POLICY default ON mydb.filtered_table2 FOR SELECT USING ((a + b) < 1) OR ((c - d) > 5) TO default\n" assert node.query("SHOW CREATE POLICY default ON mydb.filtered_table3") == "CREATE ROW POLICY default ON mydb.filtered_table3 FOR SELECT USING c = 1 TO default\n" assert node.query("SHOW CREATE POLICY default ON mydb.local") == "CREATE ROW POLICY default ON mydb.local FOR SELECT USING 1 TO default\n" - + assert node.query("SHOW CREATE POLICY default") == TSV([ "CREATE ROW POLICY default ON mydb.filtered_table1 FOR SELECT USING a = 1 TO default", "CREATE ROW POLICY default ON mydb.filtered_table2 FOR SELECT USING ((a + b) < 1) OR ((c - d) > 5) TO default", "CREATE ROW POLICY default ON mydb.filtered_table3 FOR SELECT USING c = 1 TO default", "CREATE ROW POLICY default ON mydb.local FOR SELECT USING 1 TO default" ]) assert node.query("SHOW CREATE POLICIES ON mydb.filtered_table1") == TSV([ "CREATE ROW POLICY another ON mydb.filtered_table1 FOR SELECT USING 1 TO another", "CREATE ROW POLICY default ON mydb.filtered_table1 FOR SELECT USING a = 1 TO default" ]) assert node.query("SHOW CREATE POLICIES ON mydb.*") == TSV([ "CREATE ROW POLICY another ON mydb.filtered_table1 FOR SELECT USING 1 TO another", "CREATE ROW POLICY another ON mydb.filtered_table2 FOR SELECT USING 1 TO another", "CREATE ROW POLICY another ON mydb.filtered_table3 FOR SELECT USING 1 TO another", "CREATE ROW POLICY another ON mydb.local FOR SELECT USING a = 1 TO another", "CREATE ROW POLICY default ON mydb.filtered_table1 FOR SELECT USING a = 1 TO default", "CREATE ROW POLICY default ON mydb.filtered_table2 FOR SELECT USING ((a + b) < 1) OR ((c - d) > 5) TO default", "CREATE ROW POLICY default ON mydb.filtered_table3 FOR SELECT USING c = 1 TO default", "CREATE ROW POLICY default ON mydb.local FOR SELECT USING 1 TO default" ]) diff --git a/tests/integration/test_s3_with_https/configs/config.d/ssl.xml b/tests/integration/test_s3_with_https/configs/config.d/ssl.xml new file mode 100644 index 00000000000..95cdc918bd0 --- /dev/null +++ b/tests/integration/test_s3_with_https/configs/config.d/ssl.xml @@ -0,0 +1,12 @@ + + + + + true + none + + AcceptCertificateHandler + + + + diff --git a/tests/integration/test_s3_with_https/test.py b/tests/integration/test_s3_with_https/test.py index 81e57106afc..2b40e02e701 100644 --- a/tests/integration/test_s3_with_https/test.py +++ b/tests/integration/test_s3_with_https/test.py @@ -18,7 +18,7 @@ def check_proxy_logs(cluster, proxy_instance): def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True, minio_certs_dir='minio_certs') + cluster.add_instance("node", main_configs=["configs/config.d/storage_conf.xml", "configs/config.d/log_conf.xml", "configs/config.d/ssl.xml"], with_minio=True, minio_certs_dir='minio_certs') logging.info("Starting cluster...") cluster.start() logging.info("Cluster started") diff --git a/tests/integration/test_s3_with_proxy/test.py b/tests/integration/test_s3_with_proxy/test.py index 0642cd88fe7..daf53c2e27b 100644 --- a/tests/integration/test_s3_with_proxy/test.py +++ b/tests/integration/test_s3_with_proxy/test.py @@ -21,7 +21,7 @@ def run_resolver(cluster): def cluster(): try: cluster = ClickHouseCluster(__file__) - cluster.add_instance("node", config_dir="configs", with_minio=True) + cluster.add_instance("node", main_configs=["configs/config.d/log_conf.xml", "configs/config.d/storage_conf.xml"], with_minio=True) logging.info("Starting cluster...") cluster.start() logging.info("Cluster started") diff --git a/tests/integration/test_settings_constraints/test.py b/tests/integration/test_settings_constraints/test.py index 1c8e91484ca..b2dcd80448f 100644 --- a/tests/integration/test_settings_constraints/test.py +++ b/tests/integration/test_settings_constraints/test.py @@ -2,8 +2,7 @@ import pytest from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -instance = cluster.add_instance('instance', - config_dir="configs") +instance = cluster.add_instance('instance', user_configs=["configs/users.xml"]) diff --git a/tests/integration/test_settings_constraints_distributed/test.py b/tests/integration/test_settings_constraints_distributed/test.py index 7f0f8868bcf..94afa0d6d2d 100644 --- a/tests/integration/test_settings_constraints_distributed/test.py +++ b/tests/integration/test_settings_constraints_distributed/test.py @@ -8,9 +8,9 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', config_dir="configs") -node2 = cluster.add_instance('node2', config_dir="configs") -distributed = cluster.add_instance('distributed', config_dir="configs", stay_alive=True) +node1 = cluster.add_instance('node1', main_configs=["configs/config.d/remote_servers.xml"], user_configs=["configs/users.d/allow_introspection_functions.xml"]) +node2 = cluster.add_instance('node2', main_configs=["configs/config.d/remote_servers.xml"], user_configs=["configs/users.d/allow_introspection_functions.xml"]) +distributed = cluster.add_instance('distributed', main_configs=["configs/config.d/remote_servers.xml"], user_configs=["configs/users.d/allow_introspection_functions.xml"], stay_alive=True) @pytest.fixture(scope="module", autouse=True) @@ -56,7 +56,7 @@ def test_select_clamps_settings(): assert distributed.query(query, user = 'normal') == '2\n' assert distributed.query(query, user = 'wasteful') == '2\n' assert distributed.query(query, user = 'readonly') == '2\n' - + assert distributed.query(query, settings={"max_memory_usage": 40000000, "readonly": 2}) == '2\n' assert distributed.query(query, settings={"max_memory_usage": 3000000000, "readonly": 2}) == '2\n' diff --git a/tests/integration/test_storage_hdfs/test.py b/tests/integration/test_storage_hdfs/test.py index d65b0efc334..20613bde1bc 100644 --- a/tests/integration/test_storage_hdfs/test.py +++ b/tests/integration/test_storage_hdfs/test.py @@ -12,7 +12,7 @@ import subprocess SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', with_hdfs=True, config_dir="configs", main_configs=['configs/log_conf.xml']) +node1 = cluster.add_instance('node1', with_hdfs=True, user_configs=[], main_configs=['configs/log_conf.xml']) @pytest.fixture(scope="module") def started_cluster(): diff --git a/tests/integration/test_storage_kafka/test.py b/tests/integration/test_storage_kafka/test.py index 8c4bef4a50e..dbd822476da 100644 --- a/tests/integration/test_storage_kafka/test.py +++ b/tests/integration/test_storage_kafka/test.py @@ -40,7 +40,6 @@ import kafka_pb2 cluster = ClickHouseCluster(__file__) instance = cluster.add_instance('instance', - config_dir='configs', main_configs=['configs/kafka.xml', 'configs/log_conf.xml', 'configs/kafka_macros.xml' ], with_kafka=True, with_zookeeper=True, diff --git a/tests/integration/test_storage_rabbitmq/test.py b/tests/integration/test_storage_rabbitmq/test.py index e437b13a59e..517ed81a979 100644 --- a/tests/integration/test_storage_rabbitmq/test.py +++ b/tests/integration/test_storage_rabbitmq/test.py @@ -20,7 +20,6 @@ from google.protobuf.internal.encoder import _VarintBytes cluster = ClickHouseCluster(__file__) instance = cluster.add_instance('instance', - config_dir='configs', main_configs=['configs/rabbitmq.xml','configs/log_conf.xml'], with_rabbitmq=True) rabbitmq_id = '' diff --git a/tests/integration/test_system_merges/test.py b/tests/integration/test_system_merges/test.py index 8e3714bc23b..15e5b1c0835 100644 --- a/tests/integration/test_system_merges/test.py +++ b/tests/integration/test_system_merges/test.py @@ -6,13 +6,11 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - config_dir='configs', main_configs=['configs/logs_config.xml'], with_zookeeper=True, macros={"shard": 0, "replica": 1} ) node2 = cluster.add_instance('node2', - config_dir='configs', main_configs=['configs/logs_config.xml'], with_zookeeper=True, macros={"shard": 0, "replica": 2} ) diff --git a/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_cache.xml b/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_cache.xml index a149c2ba774..806a59debca 100644 --- a/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_cache.xml +++ b/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_cache.xml @@ -1,4 +1,4 @@ - + clickhouse_cache @@ -34,4 +34,4 @@ - + \ No newline at end of file diff --git a/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_flat.xml b/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_flat.xml index feb01b27d1b..e7d32590a39 100644 --- a/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_flat.xml +++ b/tests/integration/test_system_queries/configs/dictionaries/dictionary_clickhouse_flat.xml @@ -1,4 +1,4 @@ - + clickhouse_flat @@ -34,4 +34,4 @@ - + \ No newline at end of file diff --git a/tests/integration/test_system_queries/test.py b/tests/integration/test_system_queries/test.py index 6f36a13b184..db9cf5ccf3c 100644 --- a/tests/integration/test_system_queries/test.py +++ b/tests/integration/test_system_queries/test.py @@ -18,13 +18,14 @@ def started_cluster(): global instance try: cluster = ClickHouseCluster(__file__) - cluster.add_instance('ch1', config_dir="configs") + cluster.add_instance('ch1', main_configs=["configs/config.d/clusters_config.xml", "configs/config.d/query_log.xml"], + dictionaries=["configs/dictionaries/dictionary_clickhouse_cache.xml", "configs/dictionaries/dictionary_clickhouse_flat.xml"]) cluster.start() instance = cluster.instances['ch1'] instance.query('CREATE DATABASE dictionaries ENGINE = Dictionary') instance.query('CREATE TABLE dictionary_source (id UInt64, value UInt8) ENGINE = Memory') - #print instance.query('SELECT * FROM system.dictionaries FORMAT Vertical') + print instance.query('SELECT * FROM system.dictionaries FORMAT Vertical') print "Started ", instance.ip_address yield cluster @@ -90,7 +91,7 @@ def test_RELOAD_CONFIG_AND_MACROS(started_cluster): instance.exec_in_container(['bash', '-c', create_macros], privileged=True, user='root') instance.query("SYSTEM RELOAD CONFIG") - assert TSV(instance.query("select * from system.macros")) == TSV("mac\tro\n") + assert TSV(instance.query("select * from system.macros")) == TSV("instance\tch1\nmac\tro\n") def test_SYSTEM_FLUSH_LOGS(started_cluster): diff --git a/tests/integration/test_text_log_level/test.py b/tests/integration/test_text_log_level/test.py index d7cf72fd9ea..799ae9021cb 100644 --- a/tests/integration/test_text_log_level/test.py +++ b/tests/integration/test_text_log_level/test.py @@ -8,7 +8,7 @@ from helpers.client import QueryRuntimeException cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir='configs') +node = cluster.add_instance('node', main_configs=["configs/config.d/text_log.xml"]) @pytest.fixture(scope='module') def start_cluster(): diff --git a/tests/integration/test_tmp_policy/test.py b/tests/integration/test_tmp_policy/test.py index 5c5900cc9dc..728c62d82fb 100644 --- a/tests/integration/test_tmp_policy/test.py +++ b/tests/integration/test_tmp_policy/test.py @@ -8,7 +8,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) node = cluster.add_instance('node', - config_dir='configs', + main_configs=["configs/config.d/storage_configuration.xml"], tmpfs=['/disk1:size=100M', '/disk2:size=100M']) @pytest.fixture(scope='module') diff --git a/tests/integration/test_ttl_move/test.py b/tests/integration/test_ttl_move/test.py index eedcb01ee3a..d0db52287ca 100644 --- a/tests/integration/test_ttl_move/test.py +++ b/tests/integration/test_ttl_move/test.py @@ -14,15 +14,13 @@ from helpers.test_tools import TSV cluster = ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', - config_dir='configs', - main_configs=['configs/logs_config.xml'], + main_configs=['configs/logs_config.xml', "configs/config.d/instant_moves.xml", "configs/config.d/storage_configuration.xml", "configs/config.d/cluster.xml",], with_zookeeper=True, tmpfs=['/jbod1:size=40M', '/jbod2:size=40M', '/external:size=200M'], macros={"shard": 0, "replica": 1} ) node2 = cluster.add_instance('node2', - config_dir='configs', - main_configs=['configs/logs_config.xml'], + main_configs=['configs/logs_config.xml', "configs/config.d/instant_moves.xml", "configs/config.d/storage_configuration.xml", "configs/config.d/cluster.xml",], with_zookeeper=True, tmpfs=['/jbod1:size=40M', '/jbod2:size=40M', '/external:size=200M'], macros={"shard": 0, "replica": 2} ) @@ -173,7 +171,7 @@ def test_moves_work_after_storage_policy_change(started_cluster, name, engine): ) ENGINE = {engine} ORDER BY tuple() """.format(name=name, engine=engine)) - + node1.query("""ALTER TABLE {name} MODIFY SETTING storage_policy='default_with_small_jbod_with_external'""".format(name=name)) # Second expression is preferred because d1 > now()-3600. diff --git a/tests/integration/test_user_directories/configs/local_directories.xml b/tests/integration/test_user_directories/configs/local_directories.xml index e2cbcd135df..7b9601da982 100644 --- a/tests/integration/test_user_directories/configs/local_directories.xml +++ b/tests/integration/test_user_directories/configs/local_directories.xml @@ -12,4 +12,6 @@ /var/lib/clickhouse/access3-ro/ + + diff --git a/tests/integration/test_user_directories/configs/memory.xml b/tests/integration/test_user_directories/configs/memory.xml index 6e906d2b1d6..78da38ed0bc 100644 --- a/tests/integration/test_user_directories/configs/memory.xml +++ b/tests/integration/test_user_directories/configs/memory.xml @@ -5,4 +5,7 @@ + + + diff --git a/tests/integration/test_user_directories/configs/mixed_style.xml b/tests/integration/test_user_directories/configs/mixed_style.xml new file mode 100644 index 00000000000..d6ddecf6f5d --- /dev/null +++ b/tests/integration/test_user_directories/configs/mixed_style.xml @@ -0,0 +1,8 @@ + + + + + + /etc/clickhouse-server/users6.xml + /var/lib/clickhouse/access6/ + diff --git a/tests/integration/test_user_directories/configs/old_style.xml b/tests/integration/test_user_directories/configs/old_style.xml index a0ff36edaba..cc753006b22 100644 --- a/tests/integration/test_user_directories/configs/old_style.xml +++ b/tests/integration/test_user_directories/configs/old_style.xml @@ -1,5 +1,6 @@ /etc/clickhouse-server/users2.xml /var/lib/clickhouse/access2/ + diff --git a/tests/integration/test_user_directories/configs/relative_path.xml b/tests/integration/test_user_directories/configs/relative_path.xml index 8906478959e..c4ef3c5fd79 100644 --- a/tests/integration/test_user_directories/configs/relative_path.xml +++ b/tests/integration/test_user_directories/configs/relative_path.xml @@ -4,4 +4,7 @@ users4.xml + + + diff --git a/tests/integration/test_user_directories/test.py b/tests/integration/test_user_directories/test.py index 8b7f34cf999..218330cb1a5 100644 --- a/tests/integration/test_user_directories/test.py +++ b/tests/integration/test_user_directories/test.py @@ -16,6 +16,7 @@ def started_cluster(): node.exec_in_container("cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users3.xml") node.exec_in_container("cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users4.xml") node.exec_in_container("cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users5.xml") + node.exec_in_container("cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users6.xml") yield cluster @@ -49,3 +50,10 @@ def test_memory(): node.restart_clickhouse() assert node.query("SELECT * FROM system.user_directories") == TSV([["users.xml", "users.xml", "/etc/clickhouse-server/users5.xml", 1, 1], ["memory", "memory", "", 0, 2]]) + +def test_mixed_style(): + node.copy_file_to_container(os.path.join(SCRIPT_DIR, "configs/mixed_style.xml"), '/etc/clickhouse-server/config.d/z.xml') + node.restart_clickhouse() + assert node.query("SELECT * FROM system.user_directories") == TSV([["users.xml", "users.xml", "/etc/clickhouse-server/users6.xml", 1, 1], + ["local directory", "local directory", "/var/lib/clickhouse/access6/", 0, 2], + ["memory", "memory", "", 0, 3]]) diff --git a/tests/integration/test_user_ip_restrictions/test.py b/tests/integration/test_user_ip_restrictions/test.py index 731f2bd7fa8..aee0819fe95 100644 --- a/tests/integration/test_user_ip_restrictions/test.py +++ b/tests/integration/test_user_ip_restrictions/test.py @@ -7,16 +7,16 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node_ipv4 = cluster.add_instance('node_ipv4', config_dir="configs", user_configs=['configs/users_ipv4.xml'], ipv4_address='10.5.172.77') -client_ipv4_ok = cluster.add_instance('client_ipv4_ok', config_dir="configs", ipv4_address='10.5.172.10') -client_ipv4_ok_direct = cluster.add_instance('client_ipv4_ok_direct', config_dir="configs", ipv4_address='10.5.173.1') -client_ipv4_ok_full_mask = cluster.add_instance('client_ipv4_ok_full_mask', config_dir="configs", ipv4_address='10.5.175.77') -client_ipv4_bad = cluster.add_instance('client_ipv4_bad', config_dir="configs", ipv4_address='10.5.173.10') +node_ipv4 = cluster.add_instance('node_ipv4', main_configs=[], user_configs=['configs/users_ipv4.xml'], ipv4_address='10.5.172.77') +client_ipv4_ok = cluster.add_instance('client_ipv4_ok', main_configs=[], user_configs=[], ipv4_address='10.5.172.10') +client_ipv4_ok_direct = cluster.add_instance('client_ipv4_ok_direct', main_configs=[], user_configs=[], ipv4_address='10.5.173.1') +client_ipv4_ok_full_mask = cluster.add_instance('client_ipv4_ok_full_mask', main_configs=[], user_configs=[], ipv4_address='10.5.175.77') +client_ipv4_bad = cluster.add_instance('client_ipv4_bad', main_configs=[], user_configs=[], ipv4_address='10.5.173.10') -node_ipv6 = cluster.add_instance('node_ipv6', config_dir="configs", main_configs=["configs/config_ipv6.xml"], user_configs=['configs/users_ipv6.xml'], ipv6_address='2001:3984:3989::1:1000') -client_ipv6_ok = cluster.add_instance('client_ipv6_ok', config_dir="configs", ipv6_address='2001:3984:3989::5555') -client_ipv6_ok_direct = cluster.add_instance('client_ipv6_ok_direct', config_dir="configs", ipv6_address='2001:3984:3989::1:1111') -client_ipv6_bad = cluster.add_instance('client_ipv6_bad', config_dir="configs", ipv6_address='2001:3984:3989::1:1112') +node_ipv6 = cluster.add_instance('node_ipv6', main_configs=["configs/config_ipv6.xml"], user_configs=['configs/users_ipv6.xml'], ipv6_address='2001:3984:3989::1:1000') +client_ipv6_ok = cluster.add_instance('client_ipv6_ok', main_configs=[], user_configs=[], ipv6_address='2001:3984:3989::5555') +client_ipv6_ok_direct = cluster.add_instance('client_ipv6_ok_direct', main_configs=[], user_configs=[], ipv6_address='2001:3984:3989::1:1111') +client_ipv6_bad = cluster.add_instance('client_ipv6_bad', main_configs=[], user_configs=[], ipv6_address='2001:3984:3989::1:1112') @pytest.fixture(scope="module") diff --git a/tests/integration/test_user_zero_database_access/test_user_zero_database_access.py b/tests/integration/test_user_zero_database_access/test_user_zero_database_access.py index f3d57e2e174..3af5c18544a 100644 --- a/tests/integration/test_user_zero_database_access/test_user_zero_database_access.py +++ b/tests/integration/test_user_zero_database_access/test_user_zero_database_access.py @@ -5,7 +5,7 @@ from helpers.cluster import ClickHouseCluster cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', config_dir="configs") +node = cluster.add_instance('node', user_configs=["configs/users.xml"]) @pytest.fixture(scope="module") diff --git a/tests/integration/test_version_update_after_mutation/test.py b/tests/integration/test_version_update_after_mutation/test.py index 839664638ff..f78dbf18c0d 100644 --- a/tests/integration/test_version_update_after_mutation/test.py +++ b/tests/integration/test_version_update_after_mutation/test.py @@ -5,9 +5,9 @@ from helpers.test_tools import assert_eq_with_retry cluster = ClickHouseCluster(__file__) -node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server:20.1.10.70', with_installed_binary=True, stay_alive=True) -node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server:20.1.10.70', with_installed_binary=True, stay_alive=True) -node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server:20.1.10.70', with_installed_binary=True, stay_alive=True) +node1 = cluster.add_instance('node1', with_zookeeper=True, image='yandex/clickhouse-server', tag='20.1.10.70', with_installed_binary=True, stay_alive=True) +node2 = cluster.add_instance('node2', with_zookeeper=True, image='yandex/clickhouse-server', tag='20.1.10.70', with_installed_binary=True, stay_alive=True) +node3 = cluster.add_instance('node3', with_zookeeper=True, image='yandex/clickhouse-server', tag='20.1.10.70', with_installed_binary=True, stay_alive=True) @pytest.fixture(scope="module") def start_cluster(): diff --git a/tests/integration/test_zookeeper_config/configs_secure/conf.d/ssl_conf.xml b/tests/integration/test_zookeeper_config/configs_secure/conf.d/ssl_conf.xml index 5e6f5f37624..50303fb70cc 100644 --- a/tests/integration/test_zookeeper_config/configs_secure/conf.d/ssl_conf.xml +++ b/tests/integration/test_zookeeper_config/configs_secure/conf.d/ssl_conf.xml @@ -1,8 +1,8 @@ - /etc/clickhouse-server/client.crt - /etc/clickhouse-server/client.key + /etc/clickhouse-server/config.d/client.crt + /etc/clickhouse-server/config.d/client.key true true sslv2,sslv3 diff --git a/tests/integration/test_zookeeper_config/test.py b/tests/integration/test_zookeeper_config/test.py index 5ee6a8af021..086b9ac0c73 100644 --- a/tests/integration/test_zookeeper_config/test.py +++ b/tests/integration/test_zookeeper_config/test.py @@ -12,8 +12,8 @@ def test_chroot_with_same_root(): cluster_1 = ClickHouseCluster(__file__, zookeeper_config_path='configs/zookeeper_config_root_a.xml') cluster_2 = ClickHouseCluster(__file__, zookeeper_config_path='configs/zookeeper_config_root_a.xml') - node1 = cluster_1.add_instance('node1', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) - node2 = cluster_2.add_instance('node2', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) + node1 = cluster_1.add_instance('node1', main_configs=["configs/remote_servers.xml", "configs/zookeeper_config_root_a.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) + node2 = cluster_2.add_instance('node2', main_configs=["configs/remote_servers.xml", "configs/zookeeper_config_root_a.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) nodes = [node1, node2] def create_zk_root(zk): @@ -51,8 +51,8 @@ def test_chroot_with_different_root(): cluster_1 = ClickHouseCluster(__file__, zookeeper_config_path='configs/zookeeper_config_root_a.xml') cluster_2 = ClickHouseCluster(__file__, zookeeper_config_path='configs/zookeeper_config_root_b.xml') - node1 = cluster_1.add_instance('node1', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) - node2 = cluster_2.add_instance('node2', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) + node1 = cluster_1.add_instance('node1', main_configs=["configs/remote_servers.xml", "configs/zookeeper_config_root_a.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) + node2 = cluster_2.add_instance('node2', main_configs=["configs/remote_servers.xml", "configs/zookeeper_config_root_b.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) nodes = [node1, node2] def create_zk_roots(zk): @@ -90,8 +90,8 @@ def test_identity(): cluster_1 = ClickHouseCluster(__file__, zookeeper_config_path='configs/zookeeper_config_with_password.xml') cluster_2 = ClickHouseCluster(__file__) - node1 = cluster_1.add_instance('node1', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) - node2 = cluster_2.add_instance('node2', config_dir='configs', with_zookeeper=True, zookeeper_use_tmpfs=False) + node1 = cluster_1.add_instance('node1', main_configs=["configs/remote_servers.xml", "configs/zookeeper_config_with_password.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) + node2 = cluster_2.add_instance('node2', main_configs=["configs/remote_servers.xml"], with_zookeeper=True, zookeeper_use_tmpfs=False) try: cluster_1.start() @@ -145,10 +145,12 @@ def test_secure_connection(): ) docker_compose.close() - node1 = cluster.add_instance('node1', config_dir='configs_secure', with_zookeeper=True, - zookeeper_docker_compose_path=docker_compose.name, zookeeper_use_tmpfs=False) - node2 = cluster.add_instance('node2', config_dir='configs_secure', with_zookeeper=True, - zookeeper_docker_compose_path=docker_compose.name, zookeeper_use_tmpfs=False) + node1 = cluster.add_instance('node1', main_configs=["configs_secure/client.crt", "configs_secure/client.key", + "configs_secure/conf.d/remote_servers.xml", "configs_secure/conf.d/ssl_conf.xml"], + with_zookeeper=True, zookeeper_docker_compose_path=docker_compose.name, zookeeper_use_tmpfs=False) + node2 = cluster.add_instance('node2', main_configs=["configs_secure/client.crt", "configs_secure/client.key", + "configs_secure/conf.d/remote_servers.xml", "configs_secure/conf.d/ssl_conf.xml"], + with_zookeeper=True, zookeeper_docker_compose_path=docker_compose.name, zookeeper_use_tmpfs=False) try: cluster.start() diff --git a/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.reference b/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.reference index e3dbeb81c7c..453c7fb5af0 100644 --- a/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.reference +++ b/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.reference @@ -1,6 +1,41 @@ +distributed_group_by_no_merge=1 1 1 1 1 1 1 1 1 1 1 1 1 +1 1 +1 1 +distributed_group_by_no_merge=2 +LIMIT +1 1 1 +OFFSET +2 1 1 +ALIAS +0 +0 +ORDER BY +1 +1 +0 +0 +ORDER BY LIMIT +1 +LIMIT BY +0 +1 +LIMIT BY LIMIT +0 +GROUP BY ORDER BY +1 +1 +1 +1 +GROUP BY w/ ALIAS +0 +1 +0 +1 +ORDER BY w/ ALIAS +0 diff --git a/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.sql b/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.sql index 78f1fb68385..e7174c5b56b 100644 --- a/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.sql +++ b/tests/queries/0_stateless/00184_shard_distributed_group_by_no_merge.sql @@ -1,2 +1,42 @@ -SELECT count(), uniq(dummy) FROM remote('127.0.0.{2,3}', system.one) SETTINGS distributed_group_by_no_merge = 1; -SELECT count(), uniq(dummy) FROM remote('127.0.0.{2,3,4,5}', system.one) SETTINGS distributed_group_by_no_merge = 1; +SELECT 'distributed_group_by_no_merge=1'; +SELECT count(), uniq(dummy) FROM remote('127.0.0.{2,3}', system.one) SETTINGS distributed_group_by_no_merge=1; +SELECT count(), uniq(dummy) FROM remote('127.0.0.{2,3,4,5}', system.one) SETTINGS distributed_group_by_no_merge=1; +SELECT count(), uniq(dummy) FROM remote('127.0.0.{2,3}', system.one) LIMIT 1 SETTINGS distributed_group_by_no_merge=1; + +SELECT 'distributed_group_by_no_merge=2'; +SET max_distributed_connections=1; +SET max_threads=1; +-- breaks any(_shard_num) +SET optimize_move_functions_out_of_any=0; + +SELECT 'LIMIT'; +SELECT any(_shard_num) shard_num, count(), uniq(dummy) FROM remote('127.0.0.{2,3}', system.one) LIMIT 1 SETTINGS distributed_group_by_no_merge=2; +SELECT 'OFFSET'; +SELECT any(_shard_num) shard_num, count(), uniq(dummy) FROM remote('127.0.0.{2,3}', system.one) LIMIT 1, 1 SETTINGS distributed_group_by_no_merge=2; + +SELECT 'ALIAS'; +SELECT dummy AS d FROM remote('127.0.0.{2,3}', system.one) ORDER BY d SETTINGS distributed_group_by_no_merge=2; + +DROP TABLE IF EXISTS data_00184; +CREATE TABLE data_00184 Engine=Memory() AS SELECT * FROM numbers(2); +SELECT 'ORDER BY'; +SELECT number FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) ORDER BY number DESC SETTINGS distributed_group_by_no_merge=2; +SELECT 'ORDER BY LIMIT'; +SELECT number FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) ORDER BY number DESC LIMIT 1 SETTINGS distributed_group_by_no_merge=2; + +SELECT 'LIMIT BY'; +SELECT number FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) LIMIT 1 BY number SETTINGS distributed_group_by_no_merge=2; +SELECT 'LIMIT BY LIMIT'; +SELECT number FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) LIMIT 1 BY number LIMIT 1 SETTINGS distributed_group_by_no_merge=2; + +SELECT 'GROUP BY ORDER BY'; +SELECT uniq(number) u FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) GROUP BY number ORDER BY u DESC SETTINGS distributed_group_by_no_merge=2; + +-- cover possible tricky issues +SELECT 'GROUP BY w/ ALIAS'; +SELECT n FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) GROUP BY number AS n SETTINGS distributed_group_by_no_merge=2; + +SELECT 'ORDER BY w/ ALIAS'; +SELECT n FROM remote('127.0.0.{2,3}', currentDatabase(), data_00184) ORDER BY number AS n LIMIT 1 SETTINGS distributed_group_by_no_merge=2; + +drop table data_00184; diff --git a/tests/queries/0_stateless/00652_mutations_alter_update.sh b/tests/queries/0_stateless/00652_mutations_alter_update.sh index f551872da6c..83a5e18d4ae 100755 --- a/tests/queries/0_stateless/00652_mutations_alter_update.sh +++ b/tests/queries/0_stateless/00652_mutations_alter_update.sh @@ -3,8 +3,6 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib - ${CLICKHOUSE_CLIENT} --query="DROP TABLE IF EXISTS alter_update" ${CLICKHOUSE_CLIENT} --query="CREATE TABLE alter_update \ @@ -26,8 +24,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO alter_update VALUES \ ('2000-01-01', 123, 'abc', 1), \ ('2000-01-01', 234, 'cde', 2)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update UPDATE value1 = 'aaa', value2 = value2 + 100 WHERE key < 200" -wait_for_mutation "alter_update" "mutation_2.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update UPDATE value1 = 'aaa', value2 = value2 + 100 WHERE key < 200" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT * FROM alter_update ORDER BY key" @@ -40,8 +37,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO alter_update VALUES ('2000-01-01', 123 ${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update \ UPDATE value2 = (value2 + 1) / 2 WHERE 1, \ - UPDATE value2 = value2 + 1 WHERE 1" -wait_for_mutation "alter_update" "mutation_4.txt" + UPDATE value2 = value2 + 1 WHERE 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT * FROM alter_update ORDER BY key" @@ -59,8 +55,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO alter_update VALUES \ ${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update \ DELETE WHERE key IN (SELECT toUInt32(arrayJoin([121, 122, 123]))), \ UPDATE value1 = concat(value1, 'ccc') WHERE value2 IN (20, 30), \ - UPDATE value1 = 'iii' WHERE value2 IN (SELECT toUInt64(40))" -wait_for_mutation "alter_update" "mutation_6.txt" + UPDATE value1 = 'iii' WHERE value2 IN (SELECT toUInt64(40))" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT * FROM alter_update ORDER BY key" @@ -75,8 +70,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO alter_update VALUES \ ${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update \ UPDATE value2 = value2 + 10 WHERE 1, \ - DELETE WHERE value2 = 20" -wait_for_mutation "alter_update" "mutation_8.txt" + DELETE WHERE value2 = 20" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT * FROM alter_update ORDER BY key" @@ -96,8 +90,7 @@ ${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update \ UPDATE value2 = value2 + 10 WHERE value2 <= 10, \ DELETE WHERE length(value1) + value2 = 23, \ DELETE WHERE materialized_value = 'materialized_37', \ - UPDATE value1 = concat(value1, '_', materialized_value) WHERE key = 456" -wait_for_mutation "alter_update" "mutation_10.txt" + UPDATE value1 = concat(value1, '_', materialized_value) WHERE key = 456" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT * FROM alter_update ORDER BY key" @@ -123,8 +116,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO alter_update VALUES \ ('2000-01-01', 456, 'ijk', 40)" ${CLICKHOUSE_CLIENT} --query="ALTER TABLE alter_update \ - UPDATE value2 = value2 + 7 WHERE value2 <= 20" -wait_for_mutation "alter_update" "mutation_12.txt" + UPDATE value2 = value2 + 7 WHERE value2 <= 20" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT value2, materialized_value FROM alter_update ORDER BY key" diff --git a/tests/queries/0_stateless/00652_mutations_default_database.sh b/tests/queries/0_stateless/00652_mutations_default_database.sh index c5b406b8c8e..78aa0e88c36 100755 --- a/tests/queries/0_stateless/00652_mutations_default_database.sh +++ b/tests/queries/0_stateless/00652_mutations_default_database.sh @@ -3,9 +3,7 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib - -${CLICKHOUSE_CLIENT} --multiquery << EOF +${CLICKHOUSE_CLIENT} --multiquery --mutations_sync=1 << EOF DROP TABLE IF EXISTS mutations; DROP TABLE IF EXISTS for_subquery; @@ -19,8 +17,6 @@ ALTER TABLE mutations UPDATE y = y + 1 WHERE x IN for_subquery; ALTER TABLE mutations UPDATE y = y + 1 WHERE x IN (SELECT x FROM for_subquery); EOF -wait_for_mutation "mutations" "mutation_3.txt" - ${CLICKHOUSE_CLIENT} --query="SELECT * FROM mutations" ${CLICKHOUSE_CLIENT} --query="DROP TABLE mutations" diff --git a/tests/queries/0_stateless/00682_empty_parts_merge.sh b/tests/queries/0_stateless/00682_empty_parts_merge.sh index ce9e0ab95eb..03915e3f5e4 100755 --- a/tests/queries/0_stateless/00682_empty_parts_merge.sh +++ b/tests/queries/0_stateless/00682_empty_parts_merge.sh @@ -3,17 +3,13 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib - - ${CLICKHOUSE_CLIENT} --query="DROP TABLE IF EXISTS ordinary_00682" ${CLICKHOUSE_CLIENT} --query="CREATE TABLE ordinary_00682(k UInt32) ENGINE MergeTree ORDER BY k" ${CLICKHOUSE_CLIENT} --query="INSERT INTO ordinary_00682(k) VALUES (1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO ordinary_00682(k) VALUES (1)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE ordinary_00682 DELETE WHERE k = 1" -wait_for_mutation "ordinary_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE ordinary_00682 DELETE WHERE k = 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE ordinary_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM ordinary_00682" @@ -32,8 +28,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE vertical_00682(k UInt32, v UInt32) EN ${CLICKHOUSE_CLIENT} --query="INSERT INTO vertical_00682(k, v) VALUES (1, 1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO vertical_00682(k, v) VALUES (2, 2)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE vertical_00682 DELETE WHERE k = 1" -wait_for_mutation "vertical_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE vertical_00682 DELETE WHERE k = 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE vertical_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM vertical_00682" @@ -47,8 +42,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE summing_00682(k UInt32, v UInt32) ENG ${CLICKHOUSE_CLIENT} --query="INSERT INTO summing_00682(k, v) VALUES (1, 1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO summing_00682(k, v) VALUES (1, 2)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE summing_00682 DELETE WHERE k = 1" -wait_for_mutation "summing_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE summing_00682 DELETE WHERE k = 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE summing_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM summing_00682" @@ -62,8 +56,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE aggregating_00682(k UInt32, v Aggrega ${CLICKHOUSE_CLIENT} --query="INSERT INTO aggregating_00682(k) VALUES (1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO aggregating_00682(k) VALUES (1)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE aggregating_00682 DELETE WHERE k = 1" -wait_for_mutation "aggregating_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE aggregating_00682 DELETE WHERE k = 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE aggregating_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM aggregating_00682" @@ -77,8 +70,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE replacing_00682(k UInt32, v String) E ${CLICKHOUSE_CLIENT} --query="INSERT INTO replacing_00682(k, v) VALUES (1, 'a')" ${CLICKHOUSE_CLIENT} --query="INSERT INTO replacing_00682(k, v) VALUES (1, 'b')" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE replacing_00682 DELETE WHERE k = 1" -wait_for_mutation "replacing_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE replacing_00682 DELETE WHERE k = 1" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE replacing_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM replacing_00682" @@ -92,8 +84,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE collapsing_00682(k UInt32, v String, ${CLICKHOUSE_CLIENT} --query="INSERT INTO collapsing_00682(k, v, s) VALUES (1, 'a', 1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO collapsing_00682(k, v, s) VALUES (2, 'b', 1)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE collapsing_00682 DELETE WHERE k IN (1, 2)" -wait_for_mutation "collapsing_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE collapsing_00682 DELETE WHERE k IN (1, 2)" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE collapsing_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM collapsing_00682" @@ -107,8 +98,7 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE versioned_collapsing_00682(k UInt32, ${CLICKHOUSE_CLIENT} --query="INSERT INTO versioned_collapsing_00682(k, val, ver, s) VALUES (1, 'a', 0, 1)" ${CLICKHOUSE_CLIENT} --query="INSERT INTO versioned_collapsing_00682(k, val, ver, s) VALUES (2, 'b', 0, 1)" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE versioned_collapsing_00682 DELETE WHERE k IN (1, 2)" -wait_for_mutation "versioned_collapsing_00682" "mutation_3.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE versioned_collapsing_00682 DELETE WHERE k IN (1, 2)" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="OPTIMIZE TABLE versioned_collapsing_00682 PARTITION tuple() FINAL" ${CLICKHOUSE_CLIENT} --query="SELECT * FROM versioned_collapsing_00682" diff --git a/tests/queries/0_stateless/00699_materialized_view_mutations.sh b/tests/queries/0_stateless/00699_materialized_view_mutations.sh index 0356c24b863..a8166ca29c0 100755 --- a/tests/queries/0_stateless/00699_materialized_view_mutations.sh +++ b/tests/queries/0_stateless/00699_materialized_view_mutations.sh @@ -4,8 +4,6 @@ set -e CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib - ${CLICKHOUSE_CLIENT} --multiquery --query=" DROP TABLE IF EXISTS view_00699; @@ -18,18 +16,14 @@ INSERT INTO null_00699 SELECT * FROM numbers(100); SELECT count(), min(x), max(x) FROM null_00699; SELECT count(), min(x), max(x) FROM view_00699; -ALTER TABLE null_00699 DELETE WHERE x % 2 = 0;" - -wait_for_mutation null_00699 mutation_2.txt +ALTER TABLE null_00699 DELETE WHERE x % 2 = 0;" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --multiquery --query=" SELECT count(), min(x), max(x) FROM null_00699; SELECT count(), min(x), max(x) FROM view_00699; ALTER TABLE view_00699 DELETE WHERE x % 2 = 0; -" - -wait_for_mutation .inner.view_00699 mutation_2.txt +" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --multiquery --query=" SELECT count(), min(x), max(x) FROM null_00699; @@ -37,10 +31,7 @@ SELECT count(), min(x), max(x) FROM view_00699; ALTER TABLE null_00699 DELETE WHERE x % 2 = 1; ALTER TABLE view_00699 DELETE WHERE x % 2 = 1; -" - -wait_for_mutation null_00699 mutation_3.txt -wait_for_mutation .inner.view_00699 mutation_3.txt +" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --multiquery --query=" SELECT count(), min(x), max(x) FROM null_00699; diff --git a/tests/queries/0_stateless/00942_mutate_index.sh b/tests/queries/0_stateless/00942_mutate_index.sh index 783ebbecd37..df02361af78 100755 --- a/tests/queries/0_stateless/00942_mutate_index.sh +++ b/tests/queries/0_stateless/00942_mutate_index.sh @@ -2,11 +2,9 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib $CLICKHOUSE_CLIENT --query="DROP TABLE IF EXISTS minmax_idx;" - $CLICKHOUSE_CLIENT -n --query=" CREATE TABLE minmax_idx ( @@ -18,7 +16,6 @@ CREATE TABLE minmax_idx ORDER BY u64 SETTINGS index_granularity = 2;" - $CLICKHOUSE_CLIENT --query="INSERT INTO minmax_idx VALUES (0, 1, 1), (1, 1, 2), @@ -34,8 +31,7 @@ $CLICKHOUSE_CLIENT --query="INSERT INTO minmax_idx VALUES $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 1;" $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 5;" -$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx UPDATE i64 = 5 WHERE i64 = 1;" -wait_for_mutation "minmax_idx" "mutation_2.txt" "$CLICKHOUSE_DATABASE" +$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx UPDATE i64 = 5 WHERE i64 = 1;" --mutations_sync=1 $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 1;" $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 5;" diff --git a/tests/queries/0_stateless/00944_clear_index_in_partition.sh b/tests/queries/0_stateless/00944_clear_index_in_partition.sh index 7ecbf3b4c85..8687e2044f0 100755 --- a/tests/queries/0_stateless/00944_clear_index_in_partition.sh +++ b/tests/queries/0_stateless/00944_clear_index_in_partition.sh @@ -2,7 +2,6 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib $CLICKHOUSE_CLIENT --query="DROP TABLE IF EXISTS minmax_idx;" @@ -33,18 +32,16 @@ $CLICKHOUSE_CLIENT --query="INSERT INTO minmax_idx VALUES (9, 1, 2)" $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2;" -$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" +$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" # Returns 4 -$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx CLEAR INDEX idx IN PARTITION 1;" --replication_alter_partitions_sync=2 +$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx CLEAR INDEX idx IN PARTITION 1;" --mutations_sync=1 $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2;" -$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" +$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" # Returns 6 -$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx MATERIALIZE INDEX idx IN PARTITION 1;" -wait_for_mutation "minmax_idx" "mutation_3.txt" "$CLICKHOUSE_DATABASE" +$CLICKHOUSE_CLIENT --query="ALTER TABLE minmax_idx MATERIALIZE INDEX idx IN PARTITION 1;" --mutations_sync=1 $CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2;" -$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" - +$CLICKHOUSE_CLIENT --query="SELECT count() FROM minmax_idx WHERE i64 = 2 FORMAT JSON" | grep "rows_read" # Returns 4 $CLICKHOUSE_CLIENT --query="DROP TABLE minmax_idx" diff --git a/tests/queries/0_stateless/01031_mutations_interpreter_and_context.sh b/tests/queries/0_stateless/01031_mutations_interpreter_and_context.sh index 56c95754611..7e77d58a6eb 100755 --- a/tests/queries/0_stateless/01031_mutations_interpreter_and_context.sh +++ b/tests/queries/0_stateless/01031_mutations_interpreter_and_context.sh @@ -3,8 +3,6 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib - ${CLICKHOUSE_CLIENT} --query="DROP TABLE IF EXISTS json_test" @@ -14,9 +12,7 @@ ${CLICKHOUSE_CLIENT} --query="INSERT INTO json_test VALUES (1, '{\"date\": \"201 ${CLICKHOUSE_CLIENT} --query="SELECT COUNT() FROM json_test" -${CLICKHOUSE_CLIENT} --query="ALTER TABLE json_test DELETE WHERE JSONExtractString(metadata, 'date') = '2018-01-01'" - -wait_for_mutation "json_test" "mutation_2.txt" +${CLICKHOUSE_CLIENT} --query="ALTER TABLE json_test DELETE WHERE JSONExtractString(metadata, 'date') = '2018-01-01'" --mutations_sync=1 ${CLICKHOUSE_CLIENT} --query="SELECT COUNT() FROM json_test" diff --git a/tests/queries/0_stateless/01035_lc_empty_part_bug.reference b/tests/queries/0_stateless/01035_lc_empty_part_bug.reference index 1ca0ea26354..e29f51890e2 100644 --- a/tests/queries/0_stateless/01035_lc_empty_part_bug.reference +++ b/tests/queries/0_stateless/01035_lc_empty_part_bug.reference @@ -1,3 +1,3 @@ -Waiting for mutation to finish +Waited for mutation to finish still alive 100 diff --git a/tests/queries/0_stateless/01035_lc_empty_part_bug.sh b/tests/queries/0_stateless/01035_lc_empty_part_bug.sh index f40ec14dfa8..e5bf2197157 100755 --- a/tests/queries/0_stateless/01035_lc_empty_part_bug.sh +++ b/tests/queries/0_stateless/01035_lc_empty_part_bug.sh @@ -2,7 +2,6 @@ CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$CURDIR"/../shell_config.sh -. "$CURDIR"/mergetree_mutations.lib # that test is failing on versions <= 19.11.12 @@ -11,11 +10,9 @@ ${CLICKHOUSE_CLIENT} --multiquery --query=" create table lc_empty_part_bug (id UInt64, s String) Engine=MergeTree ORDER BY id; insert into lc_empty_part_bug select number as id, toString(rand()) from numbers(100); alter table lc_empty_part_bug delete where id < 100; -" +" --mutations_sync=1 -wait_for_mutation 'lc_empty_part_bug' 'mutation_2.txt' - -echo 'Waiting for mutation to finish' +echo 'Waited for mutation to finish' ${CLICKHOUSE_CLIENT} --multiquery --query=" alter table lc_empty_part_bug modify column s LowCardinality(String); @@ -23,4 +20,4 @@ ${CLICKHOUSE_CLIENT} --multiquery --query=" insert into lc_empty_part_bug select number+100 as id, toString(rand()) from numbers(100); SELECT count() FROM lc_empty_part_bug WHERE not ignore(*); DROP TABLE lc_empty_part_bug; -" +" --mutations_sync=1 diff --git a/tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.reference b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.reference similarity index 62% rename from tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.reference rename to tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.reference index c299e3be7a0..bd01c335399 100644 --- a/tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.reference +++ b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.reference @@ -26,8 +26,10 @@ GROUP BY number, 1 GROUP BY 1 4 0 GROUP BY number ORDER BY number DESC -2 1 -2 0 +1 1 +1 1 +1 0 +1 0 GROUP BY toString(number) 1 0 1 1 @@ -49,12 +51,17 @@ DISTINCT 0 1 HAVING +HAVING LIMIT +1 0 LIMIT -2 0 -2 1 -LIMIT BY -2 0 -2 1 +1 0 +LIMIT OFFSET +1 1 +WHERE LIMIT OFFSET +1 1 +LIMIT BY 1 +1 0 +1 1 GROUP BY (Distributed-over-Distributed) 4 0 4 1 @@ -67,24 +74,48 @@ GROUP BY (Distributed-over-Distributed) distributed_group_by_no_merge 1 1 1 0 1 1 -extremes -1 0 -1 1 -1 0 -1 1 +GROUP BY (extemes) +2 0 +2 1 -1 0 -1 1 -WITH TOTALS +2 0 +2 1 +LIMIT (extemes) +2 0 + +2 0 +2 0 +GROUP BY WITH TOTALS 2 0 2 1 4 0 -WITH ROLLUP +GROUP BY WITH ROLLUP 2 0 2 1 4 0 -WITH CUBE +GROUP BY WITH CUBE 2 0 2 1 4 0 +GROUP BY WITH TOTALS ORDER BY +2 0 +2 1 + +4 0 +GROUP BY WITH TOTALS ORDER BY LIMIT +2 0 + +4 0 +GROUP BY WITH TOTALS LIMIT +2 0 + +4 0 +GROUP BY sharding_key, ... +0 0 +1 0 +0 0 +1 0 +GROUP BY ..., sharding_key +0 0 +1 0 diff --git a/tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.sql b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.sql similarity index 63% rename from tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.sql rename to tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.sql index 4a8842ca71f..41311a9d3a7 100644 --- a/tests/queries/0_stateless/01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key.sql +++ b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key.sql @@ -54,26 +54,58 @@ select 'DISTINCT'; select DISTINCT number from dist_01247; select 'HAVING'; -select count() cnt, * from dist_01247 group by number having cnt < 0; +select count() cnt, * from dist_01247 group by number having cnt == 2; + +select 'HAVING LIMIT'; +select count() cnt, * from dist_01247 group by number having cnt == 1 limit 1; select 'LIMIT'; select count(), * from dist_01247 group by number limit 1; +select 'LIMIT OFFSET'; select count(), * from dist_01247 group by number limit 1 offset 1; +-- this will emulate different data on for different shards +select 'WHERE LIMIT OFFSET'; +select count(), * from dist_01247 where number = _shard_num-1 group by number limit 1 offset 1; -select 'LIMIT BY'; -select count(), * from dist_01247 group by number limit 0 by number; -select count(), * from dist_01247 group by number limit 1 by number; +select 'LIMIT BY 1'; +select count(), * from dist_01247 group by number order by number limit 1 by number; select 'GROUP BY (Distributed-over-Distributed)'; select count(), * from cluster(test_cluster_two_shards, currentDatabase(), dist_01247) group by number; select 'GROUP BY (Distributed-over-Distributed) distributed_group_by_no_merge'; select count(), * from cluster(test_cluster_two_shards, currentDatabase(), dist_01247) group by number settings distributed_group_by_no_merge=1; -select 'extremes'; +select 'GROUP BY (extemes)'; select count(), * from dist_01247 group by number settings extremes=1; -select 'WITH TOTALS'; + +select 'LIMIT (extemes)'; +select count(), * from dist_01247 group by number limit 1 settings extremes=1; + +select 'GROUP BY WITH TOTALS'; select count(), * from dist_01247 group by number with totals; -select 'WITH ROLLUP'; +select 'GROUP BY WITH ROLLUP'; select count(), * from dist_01247 group by number with rollup; -select 'WITH CUBE'; +select 'GROUP BY WITH CUBE'; select count(), * from dist_01247 group by number with cube; + +select 'GROUP BY WITH TOTALS ORDER BY'; +select count(), * from dist_01247 group by number with totals order by number; + +select 'GROUP BY WITH TOTALS ORDER BY LIMIT'; +select count(), * from dist_01247 group by number with totals order by number limit 1; + +select 'GROUP BY WITH TOTALS LIMIT'; +select count(), * from dist_01247 group by number with totals limit 1; + +-- GROUP BY (compound) +drop table if exists dist_01247; +drop table if exists data_01247; +create table data_01247 engine=Memory() as select number key, 0 value from numbers(2); +create table dist_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), data_01247, key); +select 'GROUP BY sharding_key, ...'; +select * from dist_01247 group by key, value; +select 'GROUP BY ..., sharding_key'; +select * from dist_01247 group by value, key; + +drop table dist_01247; +drop table data_01247; diff --git a/tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.reference b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.reference similarity index 76% rename from tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.reference rename to tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.reference index 9fc7f7bfcd7..3b8b0b90a2e 100644 --- a/tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.reference +++ b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.reference @@ -1,17 +1,6 @@ Distributed(number)-over-Distributed(number) 1 0 -1 1 -1 0 -1 1 -1 0 -1 1 -1 0 -1 1 Distributed(rand)-over-Distributed(number) 4 0 -4 1 Distributed(rand)-over-Distributed(rand) 2 0 -2 1 -2 0 -2 1 diff --git a/tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.sql b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql similarity index 88% rename from tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.sql rename to tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql index b4852793e7c..7654ba71cc9 100644 --- a/tests/queries/0_stateless/01247_dist_on_dist_group_by_sharding_key_optimization.sql +++ b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql @@ -21,20 +21,21 @@ set optimize_skip_unused_shards=1; select 'Distributed(number)-over-Distributed(number)'; create table dist_layer_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), data_01247, number); create table dist_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), dist_layer_01247, number); -select count(), * from dist_01247 group by number; +select count(), * from dist_01247 group by number order by number limit 1; drop table if exists dist_01247; drop table if exists dist_layer_01247; select 'Distributed(rand)-over-Distributed(number)'; create table dist_layer_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), data_01247, number); create table dist_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), dist_layer_01247, rand()); -select count(), * from dist_01247 group by number; +select count(), * from dist_01247 group by number order by number limit 1; drop table if exists dist_01247; drop table if exists dist_layer_01247; select 'Distributed(rand)-over-Distributed(rand)'; create table dist_layer_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), data_01247, rand()); create table dist_01247 as data_01247 engine=Distributed(test_cluster_two_shards, currentDatabase(), dist_layer_01247, number); -select count(), * from dist_01247 group by number; +select count(), * from dist_01247 group by number order by number limit 1; + drop table dist_01247; drop table dist_layer_01247; diff --git a/tests/queries/0_stateless/01280_ttl_where_group_by.reference b/tests/queries/0_stateless/01280_ttl_where_group_by.reference index dead0a5aac3..ad20d38f2e6 100644 --- a/tests/queries/0_stateless/01280_ttl_where_group_by.reference +++ b/tests/queries/0_stateless/01280_ttl_where_group_by.reference @@ -1,8 +1,8 @@ 1 1 0 4 1 2 3 7 1 3 0 5 -2 1 20 1 2 1 0 1 +2 1 20 1 1 1 [0,2,3] 4 1 1 [5,4,1] 13 1 3 [1,0,1,0] 17 diff --git a/tests/queries/0_stateless/01280_ttl_where_group_by.sh b/tests/queries/0_stateless/01280_ttl_where_group_by.sh index 673e8dbe6d6..9b05606f928 100755 --- a/tests/queries/0_stateless/01280_ttl_where_group_by.sh +++ b/tests/queries/0_stateless/01280_ttl_where_group_by.sh @@ -25,7 +25,7 @@ insert into ttl_01280_1 values (3, 1, 0, 8, now());" sleep 2 optimize "ttl_01280_1" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_1" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_1 ORDER BY a, b, x, y" $CLICKHOUSE_CLIENT --query "drop table if exists ttl_01280_2" @@ -43,7 +43,7 @@ insert into ttl_01280_2 values (3, 1, array(2, 4, 5), 8, now());" sleep 2 optimize "ttl_01280_2" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_2" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_2 ORDER BY a, b, x, y" $CLICKHOUSE_CLIENT --query "drop table if exists ttl_01280_3" @@ -61,7 +61,7 @@ insert into ttl_01280_3 values (3, 5, 5, 8, now());" sleep 2 optimize "ttl_01280_3" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_3" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_3 ORDER BY a, b, x, y" $CLICKHOUSE_CLIENT --query "drop table if exists ttl_01280_4" @@ -75,7 +75,7 @@ insert into ttl_01280_4 values (1, 5, 4, 9, now())" sleep 2 optimize "ttl_01280_4" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_4" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_4 ORDER BY a, b, x, y" $CLICKHOUSE_CLIENT --query "drop table if exists ttl_01280_5" @@ -87,7 +87,7 @@ insert into ttl_01280_5 values (1, 5, 4, 5, now());" sleep 2 optimize "ttl_01280_5" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_5" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_5 ORDER BY a, b, x, y" $CLICKHOUSE_CLIENT --query "drop table if exists ttl_01280_6" @@ -100,4 +100,4 @@ insert into ttl_01280_6 values (1, 5, 3, 5, now())" sleep 2 optimize "ttl_01280_6" -$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_6" +$CLICKHOUSE_CLIENT --query "select a, b, x, y from ttl_01280_6 ORDER BY a, b, x, y" diff --git a/tests/queries/0_stateless/01407_lambda_arrayJoin.reference b/tests/queries/0_stateless/01407_lambda_arrayJoin.reference new file mode 100644 index 00000000000..fe51488c706 --- /dev/null +++ b/tests/queries/0_stateless/01407_lambda_arrayJoin.reference @@ -0,0 +1 @@ +[] diff --git a/tests/queries/0_stateless/01407_lambda_arrayJoin.sql b/tests/queries/0_stateless/01407_lambda_arrayJoin.sql new file mode 100644 index 00000000000..4f34bb59527 --- /dev/null +++ b/tests/queries/0_stateless/01407_lambda_arrayJoin.sql @@ -0,0 +1,6 @@ +SELECT arrayFilter((a) -> ((a, arrayJoin([])) IN (Null, [Null])), []); +SELECT arrayFilter((a) -> ((a, arrayJoin([[]])) IN (Null, [Null])), []); + +-- simplified from the https://clickhouse-test-reports.s3.yandex.net/10373/6c4748a63e7acde2cc3283d96ffec590aae1e724/fuzzer/fuzzer.log#fail1 +SELECT * FROM system.one ARRAY JOIN arrayFilter((a) -> ((a, arrayJoin([])) IN (NULL)), []) AS arr_x; -- { serverError 43; } +SELECT * FROM numbers(1) LEFT ARRAY JOIN arrayFilter((x_0, x_1) -> (arrayJoin([]) IN (NULL)), [], []) AS arr_x; diff --git a/tests/queries/0_stateless/01415_table_function_view.reference b/tests/queries/0_stateless/01415_table_function_view.reference new file mode 100644 index 00000000000..2b5eef0300e --- /dev/null +++ b/tests/queries/0_stateless/01415_table_function_view.reference @@ -0,0 +1,10 @@ +1 +1 +SELECT `1` +FROM view( + SELECT 1 +) +SELECT `1` +FROM remote(\'127.0.0.1\', view( + SELECT 1 +)) diff --git a/tests/queries/0_stateless/01415_table_function_view.sql b/tests/queries/0_stateless/01415_table_function_view.sql new file mode 100644 index 00000000000..0beeb64c02d --- /dev/null +++ b/tests/queries/0_stateless/01415_table_function_view.sql @@ -0,0 +1,5 @@ +SELECT * FROM view(SELECT 1); +SELECT * FROM remote('127.0.0.1', view(SELECT 1)); + +EXPLAIN SYNTAX SELECT * FROM view(SELECT 1); +EXPLAIN SYNTAX SELECT * FROM remote('127.0.0.1', view(SELECT 1)); diff --git a/tests/queries/0_stateless/01440_big_int_shift.reference b/tests/queries/0_stateless/01440_big_int_shift.reference new file mode 100644 index 00000000000..392a24aa7bc --- /dev/null +++ b/tests/queries/0_stateless/01440_big_int_shift.reference @@ -0,0 +1,638 @@ +1 1 Int128 Int128 +2 1 Int128 Int128 +4 1 Int128 Int128 +8 1 Int128 Int128 +16 1 Int128 Int128 +32 1 Int128 Int128 +64 1 Int128 Int128 +128 1 Int128 Int128 +256 1 Int128 Int128 +512 1 Int128 Int128 +1024 1 Int128 Int128 +2048 1 Int128 Int128 +4096 1 Int128 Int128 +8192 1 Int128 Int128 +16384 1 Int128 Int128 +32768 1 Int128 Int128 +65536 1 Int128 Int128 +131072 1 Int128 Int128 +262144 1 Int128 Int128 +524288 1 Int128 Int128 +1048576 1 Int128 Int128 +2097152 1 Int128 Int128 +4194304 1 Int128 Int128 +8388608 1 Int128 Int128 +16777216 1 Int128 Int128 +33554432 1 Int128 Int128 +67108864 1 Int128 Int128 +134217728 1 Int128 Int128 +268435456 1 Int128 Int128 +536870912 1 Int128 Int128 +1073741824 1 Int128 Int128 +2147483648 1 Int128 Int128 +4294967296 1 Int128 Int128 +8589934592 1 Int128 Int128 +17179869184 1 Int128 Int128 +34359738368 1 Int128 Int128 +68719476736 1 Int128 Int128 +137438953472 1 Int128 Int128 +274877906944 1 Int128 Int128 +549755813888 1 Int128 Int128 +1099511627776 1 Int128 Int128 +2199023255552 1 Int128 Int128 +4398046511104 1 Int128 Int128 +8796093022208 1 Int128 Int128 +17592186044416 1 Int128 Int128 +35184372088832 1 Int128 Int128 +70368744177664 1 Int128 Int128 +140737488355328 1 Int128 Int128 +281474976710656 1 Int128 Int128 +562949953421312 1 Int128 Int128 +1125899906842624 1 Int128 Int128 +2251799813685248 1 Int128 Int128 +4503599627370496 1 Int128 Int128 +9007199254740992 1 Int128 Int128 +18014398509481984 1 Int128 Int128 +36028797018963968 1 Int128 Int128 +72057594037927936 1 Int128 Int128 +144115188075855872 1 Int128 Int128 +288230376151711744 1 Int128 Int128 +576460752303423488 1 Int128 Int128 +1152921504606846976 1 Int128 Int128 +2305843009213693952 1 Int128 Int128 +4611686018427387904 1 Int128 Int128 +9223372036854775808 1 Int128 Int128 +18446744073709551616 1 Int128 Int128 +36893488147419103232 1 Int128 Int128 +73786976294838206464 1 Int128 Int128 +147573952589676412928 1 Int128 Int128 +295147905179352825856 1 Int128 Int128 +590295810358705651712 1 Int128 Int128 +1180591620717411303424 1 Int128 Int128 +2361183241434822606848 1 Int128 Int128 +4722366482869645213696 1 Int128 Int128 +9444732965739290427392 1 Int128 Int128 +18889465931478580854784 1 Int128 Int128 +37778931862957161709568 1 Int128 Int128 +75557863725914323419136 1 Int128 Int128 +151115727451828646838272 1 Int128 Int128 +302231454903657293676544 1 Int128 Int128 +604462909807314587353088 1 Int128 Int128 +1208925819614629174706176 1 Int128 Int128 +2417851639229258349412352 1 Int128 Int128 +4835703278458516698824704 1 Int128 Int128 +9671406556917033397649408 1 Int128 Int128 +19342813113834066795298816 1 Int128 Int128 +38685626227668133590597632 1 Int128 Int128 +77371252455336267181195264 1 Int128 Int128 +154742504910672534362390528 1 Int128 Int128 +309485009821345068724781056 1 Int128 Int128 +618970019642690137449562112 1 Int128 Int128 +1237940039285380274899124224 1 Int128 Int128 +2475880078570760549798248448 1 Int128 Int128 +4951760157141521099596496896 1 Int128 Int128 +9903520314283042199192993792 1 Int128 Int128 +19807040628566084398385987584 1 Int128 Int128 +39614081257132168796771975168 1 Int128 Int128 +79228162514264337593543950336 1 Int128 Int128 +158456325028528675187087900672 1 Int128 Int128 +316912650057057350374175801344 1 Int128 Int128 +633825300114114700748351602688 1 Int128 Int128 +1267650600228229401496703205376 1 Int128 Int128 +2535301200456458802993406410752 1 Int128 Int128 +5070602400912917605986812821504 1 Int128 Int128 +10141204801825835211973625643008 1 Int128 Int128 +20282409603651670423947251286016 1 Int128 Int128 +40564819207303340847894502572032 1 Int128 Int128 +81129638414606681695789005144064 1 Int128 Int128 +162259276829213363391578010288128 1 Int128 Int128 +324518553658426726783156020576256 1 Int128 Int128 +649037107316853453566312041152512 1 Int128 Int128 +1298074214633706907132624082305024 1 Int128 Int128 +2596148429267413814265248164610048 1 Int128 Int128 +5192296858534827628530496329220096 1 Int128 Int128 +10384593717069655257060992658440192 1 Int128 Int128 +20769187434139310514121985316880384 1 Int128 Int128 +41538374868278621028243970633760768 1 Int128 Int128 +83076749736557242056487941267521536 1 Int128 Int128 +166153499473114484112975882535043072 1 Int128 Int128 +332306998946228968225951765070086144 1 Int128 Int128 +664613997892457936451903530140172288 1 Int128 Int128 +1329227995784915872903807060280344576 1 Int128 Int128 +2658455991569831745807614120560689152 1 Int128 Int128 +5316911983139663491615228241121378304 1 Int128 Int128 +10633823966279326983230456482242756608 1 Int128 Int128 +21267647932558653966460912964485513216 1 Int128 Int128 +42535295865117307932921825928971026432 1 Int128 Int128 +85070591730234615865843651857942052864 1 Int128 Int128 +1 1 Int256 Int256 +2 1 Int256 Int256 +4 1 Int256 Int256 +8 1 Int256 Int256 +16 1 Int256 Int256 +32 1 Int256 Int256 +64 1 Int256 Int256 +128 1 Int256 Int256 +256 1 Int256 Int256 +512 1 Int256 Int256 +1024 1 Int256 Int256 +2048 1 Int256 Int256 +4096 1 Int256 Int256 +8192 1 Int256 Int256 +16384 1 Int256 Int256 +32768 1 Int256 Int256 +65536 1 Int256 Int256 +131072 1 Int256 Int256 +262144 1 Int256 Int256 +524288 1 Int256 Int256 +1048576 1 Int256 Int256 +2097152 1 Int256 Int256 +4194304 1 Int256 Int256 +8388608 1 Int256 Int256 +16777216 1 Int256 Int256 +33554432 1 Int256 Int256 +67108864 1 Int256 Int256 +134217728 1 Int256 Int256 +268435456 1 Int256 Int256 +536870912 1 Int256 Int256 +1073741824 1 Int256 Int256 +2147483648 1 Int256 Int256 +4294967296 1 Int256 Int256 +8589934592 1 Int256 Int256 +17179869184 1 Int256 Int256 +34359738368 1 Int256 Int256 +68719476736 1 Int256 Int256 +137438953472 1 Int256 Int256 +274877906944 1 Int256 Int256 +549755813888 1 Int256 Int256 +1099511627776 1 Int256 Int256 +2199023255552 1 Int256 Int256 +4398046511104 1 Int256 Int256 +8796093022208 1 Int256 Int256 +17592186044416 1 Int256 Int256 +35184372088832 1 Int256 Int256 +70368744177664 1 Int256 Int256 +140737488355328 1 Int256 Int256 +281474976710656 1 Int256 Int256 +562949953421312 1 Int256 Int256 +1125899906842624 1 Int256 Int256 +2251799813685248 1 Int256 Int256 +4503599627370496 1 Int256 Int256 +9007199254740992 1 Int256 Int256 +18014398509481984 1 Int256 Int256 +36028797018963968 1 Int256 Int256 +72057594037927936 1 Int256 Int256 +144115188075855872 1 Int256 Int256 +288230376151711744 1 Int256 Int256 +576460752303423488 1 Int256 Int256 +1152921504606846976 1 Int256 Int256 +2305843009213693952 1 Int256 Int256 +4611686018427387904 1 Int256 Int256 +9223372036854775808 1 Int256 Int256 +18446744073709551616 1 Int256 Int256 +36893488147419103232 1 Int256 Int256 +73786976294838206464 1 Int256 Int256 +147573952589676412928 1 Int256 Int256 +295147905179352825856 1 Int256 Int256 +590295810358705651712 1 Int256 Int256 +1180591620717411303424 1 Int256 Int256 +2361183241434822606848 1 Int256 Int256 +4722366482869645213696 1 Int256 Int256 +9444732965739290427392 1 Int256 Int256 +18889465931478580854784 1 Int256 Int256 +37778931862957161709568 1 Int256 Int256 +75557863725914323419136 1 Int256 Int256 +151115727451828646838272 1 Int256 Int256 +302231454903657293676544 1 Int256 Int256 +604462909807314587353088 1 Int256 Int256 +1208925819614629174706176 1 Int256 Int256 +2417851639229258349412352 1 Int256 Int256 +4835703278458516698824704 1 Int256 Int256 +9671406556917033397649408 1 Int256 Int256 +19342813113834066795298816 1 Int256 Int256 +38685626227668133590597632 1 Int256 Int256 +77371252455336267181195264 1 Int256 Int256 +154742504910672534362390528 1 Int256 Int256 +309485009821345068724781056 1 Int256 Int256 +618970019642690137449562112 1 Int256 Int256 +1237940039285380274899124224 1 Int256 Int256 +2475880078570760549798248448 1 Int256 Int256 +4951760157141521099596496896 1 Int256 Int256 +9903520314283042199192993792 1 Int256 Int256 +19807040628566084398385987584 1 Int256 Int256 +39614081257132168796771975168 1 Int256 Int256 +79228162514264337593543950336 1 Int256 Int256 +158456325028528675187087900672 1 Int256 Int256 +316912650057057350374175801344 1 Int256 Int256 +633825300114114700748351602688 1 Int256 Int256 +1267650600228229401496703205376 1 Int256 Int256 +2535301200456458802993406410752 1 Int256 Int256 +5070602400912917605986812821504 1 Int256 Int256 +10141204801825835211973625643008 1 Int256 Int256 +20282409603651670423947251286016 1 Int256 Int256 +40564819207303340847894502572032 1 Int256 Int256 +81129638414606681695789005144064 1 Int256 Int256 +162259276829213363391578010288128 1 Int256 Int256 +324518553658426726783156020576256 1 Int256 Int256 +649037107316853453566312041152512 1 Int256 Int256 +1298074214633706907132624082305024 1 Int256 Int256 +2596148429267413814265248164610048 1 Int256 Int256 +5192296858534827628530496329220096 1 Int256 Int256 +10384593717069655257060992658440192 1 Int256 Int256 +20769187434139310514121985316880384 1 Int256 Int256 +41538374868278621028243970633760768 1 Int256 Int256 +83076749736557242056487941267521536 1 Int256 Int256 +166153499473114484112975882535043072 1 Int256 Int256 +332306998946228968225951765070086144 1 Int256 Int256 +664613997892457936451903530140172288 1 Int256 Int256 +1329227995784915872903807060280344576 1 Int256 Int256 +2658455991569831745807614120560689152 1 Int256 Int256 +5316911983139663491615228241121378304 1 Int256 Int256 +10633823966279326983230456482242756608 1 Int256 Int256 +21267647932558653966460912964485513216 1 Int256 Int256 +42535295865117307932921825928971026432 1 Int256 Int256 +85070591730234615865843651857942052864 1 Int256 Int256 +170141183460469231731687303715884105728 1 Int256 Int256 +340282366920938463463374607431768211456 1 Int256 Int256 +680564733841876926926749214863536422912 1 Int256 Int256 +1361129467683753853853498429727072845824 1 Int256 Int256 +2722258935367507707706996859454145691648 1 Int256 Int256 +5444517870735015415413993718908291383296 1 Int256 Int256 +10889035741470030830827987437816582766592 1 Int256 Int256 +21778071482940061661655974875633165533184 1 Int256 Int256 +43556142965880123323311949751266331066368 1 Int256 Int256 +87112285931760246646623899502532662132736 1 Int256 Int256 +174224571863520493293247799005065324265472 1 Int256 Int256 +348449143727040986586495598010130648530944 1 Int256 Int256 +696898287454081973172991196020261297061888 1 Int256 Int256 +1393796574908163946345982392040522594123776 1 Int256 Int256 +2787593149816327892691964784081045188247552 1 Int256 Int256 +5575186299632655785383929568162090376495104 1 Int256 Int256 +11150372599265311570767859136324180752990208 1 Int256 Int256 +22300745198530623141535718272648361505980416 1 Int256 Int256 +44601490397061246283071436545296723011960832 1 Int256 Int256 +89202980794122492566142873090593446023921664 1 Int256 Int256 +178405961588244985132285746181186892047843328 1 Int256 Int256 +356811923176489970264571492362373784095686656 1 Int256 Int256 +713623846352979940529142984724747568191373312 1 Int256 Int256 +1427247692705959881058285969449495136382746624 1 Int256 Int256 +2854495385411919762116571938898990272765493248 1 Int256 Int256 +5708990770823839524233143877797980545530986496 1 Int256 Int256 +11417981541647679048466287755595961091061972992 1 Int256 Int256 +22835963083295358096932575511191922182123945984 1 Int256 Int256 +45671926166590716193865151022383844364247891968 1 Int256 Int256 +91343852333181432387730302044767688728495783936 1 Int256 Int256 +182687704666362864775460604089535377456991567872 1 Int256 Int256 +365375409332725729550921208179070754913983135744 1 Int256 Int256 +730750818665451459101842416358141509827966271488 1 Int256 Int256 +1461501637330902918203684832716283019655932542976 1 Int256 Int256 +2923003274661805836407369665432566039311865085952 1 Int256 Int256 +5846006549323611672814739330865132078623730171904 1 Int256 Int256 +11692013098647223345629478661730264157247460343808 1 Int256 Int256 +23384026197294446691258957323460528314494920687616 1 Int256 Int256 +46768052394588893382517914646921056628989841375232 1 Int256 Int256 +93536104789177786765035829293842113257979682750464 1 Int256 Int256 +187072209578355573530071658587684226515959365500928 1 Int256 Int256 +374144419156711147060143317175368453031918731001856 1 Int256 Int256 +748288838313422294120286634350736906063837462003712 1 Int256 Int256 +1496577676626844588240573268701473812127674924007424 1 Int256 Int256 +2993155353253689176481146537402947624255349848014848 1 Int256 Int256 +5986310706507378352962293074805895248510699696029696 1 Int256 Int256 +11972621413014756705924586149611790497021399392059392 1 Int256 Int256 +23945242826029513411849172299223580994042798784118784 1 Int256 Int256 +47890485652059026823698344598447161988085597568237568 1 Int256 Int256 +95780971304118053647396689196894323976171195136475136 1 Int256 Int256 +191561942608236107294793378393788647952342390272950272 1 Int256 Int256 +383123885216472214589586756787577295904684780545900544 1 Int256 Int256 +766247770432944429179173513575154591809369561091801088 1 Int256 Int256 +1532495540865888858358347027150309183618739122183602176 1 Int256 Int256 +3064991081731777716716694054300618367237478244367204352 1 Int256 Int256 +6129982163463555433433388108601236734474956488734408704 1 Int256 Int256 +12259964326927110866866776217202473468949912977468817408 1 Int256 Int256 +24519928653854221733733552434404946937899825954937634816 1 Int256 Int256 +49039857307708443467467104868809893875799651909875269632 1 Int256 Int256 +98079714615416886934934209737619787751599303819750539264 1 Int256 Int256 +196159429230833773869868419475239575503198607639501078528 1 Int256 Int256 +392318858461667547739736838950479151006397215279002157056 1 Int256 Int256 +784637716923335095479473677900958302012794430558004314112 1 Int256 Int256 +1569275433846670190958947355801916604025588861116008628224 1 Int256 Int256 +3138550867693340381917894711603833208051177722232017256448 1 Int256 Int256 +6277101735386680763835789423207666416102355444464034512896 1 Int256 Int256 +12554203470773361527671578846415332832204710888928069025792 1 Int256 Int256 +25108406941546723055343157692830665664409421777856138051584 1 Int256 Int256 +50216813883093446110686315385661331328818843555712276103168 1 Int256 Int256 +100433627766186892221372630771322662657637687111424552206336 1 Int256 Int256 +200867255532373784442745261542645325315275374222849104412672 1 Int256 Int256 +401734511064747568885490523085290650630550748445698208825344 1 Int256 Int256 +803469022129495137770981046170581301261101496891396417650688 1 Int256 Int256 +1606938044258990275541962092341162602522202993782792835301376 1 Int256 Int256 +3213876088517980551083924184682325205044405987565585670602752 1 Int256 Int256 +6427752177035961102167848369364650410088811975131171341205504 1 Int256 Int256 +12855504354071922204335696738729300820177623950262342682411008 1 Int256 Int256 +25711008708143844408671393477458601640355247900524685364822016 1 Int256 Int256 +51422017416287688817342786954917203280710495801049370729644032 1 Int256 Int256 +102844034832575377634685573909834406561420991602098741459288064 1 Int256 Int256 +205688069665150755269371147819668813122841983204197482918576128 1 Int256 Int256 +411376139330301510538742295639337626245683966408394965837152256 1 Int256 Int256 +822752278660603021077484591278675252491367932816789931674304512 1 Int256 Int256 +1645504557321206042154969182557350504982735865633579863348609024 1 Int256 Int256 +3291009114642412084309938365114701009965471731267159726697218048 1 Int256 Int256 +6582018229284824168619876730229402019930943462534319453394436096 1 Int256 Int256 +13164036458569648337239753460458804039861886925068638906788872192 1 Int256 Int256 +26328072917139296674479506920917608079723773850137277813577744384 1 Int256 Int256 +52656145834278593348959013841835216159447547700274555627155488768 1 Int256 Int256 +105312291668557186697918027683670432318895095400549111254310977536 1 Int256 Int256 +210624583337114373395836055367340864637790190801098222508621955072 1 Int256 Int256 +421249166674228746791672110734681729275580381602196445017243910144 1 Int256 Int256 +842498333348457493583344221469363458551160763204392890034487820288 1 Int256 Int256 +1684996666696914987166688442938726917102321526408785780068975640576 1 Int256 Int256 +3369993333393829974333376885877453834204643052817571560137951281152 1 Int256 Int256 +6739986666787659948666753771754907668409286105635143120275902562304 1 Int256 Int256 +13479973333575319897333507543509815336818572211270286240551805124608 1 Int256 Int256 +26959946667150639794667015087019630673637144422540572481103610249216 1 Int256 Int256 +53919893334301279589334030174039261347274288845081144962207220498432 1 Int256 Int256 +107839786668602559178668060348078522694548577690162289924414440996864 1 Int256 Int256 +215679573337205118357336120696157045389097155380324579848828881993728 1 Int256 Int256 +431359146674410236714672241392314090778194310760649159697657763987456 1 Int256 Int256 +862718293348820473429344482784628181556388621521298319395315527974912 1 Int256 Int256 +1725436586697640946858688965569256363112777243042596638790631055949824 1 Int256 Int256 +3450873173395281893717377931138512726225554486085193277581262111899648 1 Int256 Int256 +6901746346790563787434755862277025452451108972170386555162524223799296 1 Int256 Int256 +13803492693581127574869511724554050904902217944340773110325048447598592 1 Int256 Int256 +27606985387162255149739023449108101809804435888681546220650096895197184 1 Int256 Int256 +55213970774324510299478046898216203619608871777363092441300193790394368 1 Int256 Int256 +110427941548649020598956093796432407239217743554726184882600387580788736 1 Int256 Int256 +220855883097298041197912187592864814478435487109452369765200775161577472 1 Int256 Int256 +441711766194596082395824375185729628956870974218904739530401550323154944 1 Int256 Int256 +883423532389192164791648750371459257913741948437809479060803100646309888 1 Int256 Int256 +1766847064778384329583297500742918515827483896875618958121606201292619776 1 Int256 Int256 +3533694129556768659166595001485837031654967793751237916243212402585239552 1 Int256 Int256 +7067388259113537318333190002971674063309935587502475832486424805170479104 1 Int256 Int256 +14134776518227074636666380005943348126619871175004951664972849610340958208 1 Int256 Int256 +28269553036454149273332760011886696253239742350009903329945699220681916416 1 Int256 Int256 +56539106072908298546665520023773392506479484700019806659891398441363832832 1 Int256 Int256 +113078212145816597093331040047546785012958969400039613319782796882727665664 1 Int256 Int256 +226156424291633194186662080095093570025917938800079226639565593765455331328 1 Int256 Int256 +452312848583266388373324160190187140051835877600158453279131187530910662656 1 Int256 Int256 +904625697166532776746648320380374280103671755200316906558262375061821325312 1 Int256 Int256 +1809251394333065553493296640760748560207343510400633813116524750123642650624 1 Int256 Int256 +3618502788666131106986593281521497120414687020801267626233049500247285301248 1 Int256 Int256 +7237005577332262213973186563042994240829374041602535252466099000494570602496 1 Int256 Int256 +14474011154664524427946373126085988481658748083205070504932198000989141204992 1 Int256 Int256 +28948022309329048855892746252171976963317496166410141009864396001978282409984 1 Int256 Int256 +1 1 UInt256 UInt256 +2 1 UInt256 UInt256 +4 1 UInt256 UInt256 +8 1 UInt256 UInt256 +16 1 UInt256 UInt256 +32 1 UInt256 UInt256 +64 1 UInt256 UInt256 +128 1 UInt256 UInt256 +256 1 UInt256 UInt256 +512 1 UInt256 UInt256 +1024 1 UInt256 UInt256 +2048 1 UInt256 UInt256 +4096 1 UInt256 UInt256 +8192 1 UInt256 UInt256 +16384 1 UInt256 UInt256 +32768 1 UInt256 UInt256 +65536 1 UInt256 UInt256 +131072 1 UInt256 UInt256 +262144 1 UInt256 UInt256 +524288 1 UInt256 UInt256 +1048576 1 UInt256 UInt256 +2097152 1 UInt256 UInt256 +4194304 1 UInt256 UInt256 +8388608 1 UInt256 UInt256 +16777216 1 UInt256 UInt256 +33554432 1 UInt256 UInt256 +67108864 1 UInt256 UInt256 +134217728 1 UInt256 UInt256 +268435456 1 UInt256 UInt256 +536870912 1 UInt256 UInt256 +1073741824 1 UInt256 UInt256 +2147483648 1 UInt256 UInt256 +4294967296 1 UInt256 UInt256 +8589934592 1 UInt256 UInt256 +17179869184 1 UInt256 UInt256 +34359738368 1 UInt256 UInt256 +68719476736 1 UInt256 UInt256 +137438953472 1 UInt256 UInt256 +274877906944 1 UInt256 UInt256 +549755813888 1 UInt256 UInt256 +1099511627776 1 UInt256 UInt256 +2199023255552 1 UInt256 UInt256 +4398046511104 1 UInt256 UInt256 +8796093022208 1 UInt256 UInt256 +17592186044416 1 UInt256 UInt256 +35184372088832 1 UInt256 UInt256 +70368744177664 1 UInt256 UInt256 +140737488355328 1 UInt256 UInt256 +281474976710656 1 UInt256 UInt256 +562949953421312 1 UInt256 UInt256 +1125899906842624 1 UInt256 UInt256 +2251799813685248 1 UInt256 UInt256 +4503599627370496 1 UInt256 UInt256 +9007199254740992 1 UInt256 UInt256 +18014398509481984 1 UInt256 UInt256 +36028797018963968 1 UInt256 UInt256 +72057594037927936 1 UInt256 UInt256 +144115188075855872 1 UInt256 UInt256 +288230376151711744 1 UInt256 UInt256 +576460752303423488 1 UInt256 UInt256 +1152921504606846976 1 UInt256 UInt256 +2305843009213693952 1 UInt256 UInt256 +4611686018427387904 1 UInt256 UInt256 +9223372036854775808 1 UInt256 UInt256 +18446744073709551616 1 UInt256 UInt256 +36893488147419103232 1 UInt256 UInt256 +73786976294838206464 1 UInt256 UInt256 +147573952589676412928 1 UInt256 UInt256 +295147905179352825856 1 UInt256 UInt256 +590295810358705651712 1 UInt256 UInt256 +1180591620717411303424 1 UInt256 UInt256 +2361183241434822606848 1 UInt256 UInt256 +4722366482869645213696 1 UInt256 UInt256 +9444732965739290427392 1 UInt256 UInt256 +18889465931478580854784 1 UInt256 UInt256 +37778931862957161709568 1 UInt256 UInt256 +75557863725914323419136 1 UInt256 UInt256 +151115727451828646838272 1 UInt256 UInt256 +302231454903657293676544 1 UInt256 UInt256 +604462909807314587353088 1 UInt256 UInt256 +1208925819614629174706176 1 UInt256 UInt256 +2417851639229258349412352 1 UInt256 UInt256 +4835703278458516698824704 1 UInt256 UInt256 +9671406556917033397649408 1 UInt256 UInt256 +19342813113834066795298816 1 UInt256 UInt256 +38685626227668133590597632 1 UInt256 UInt256 +77371252455336267181195264 1 UInt256 UInt256 +154742504910672534362390528 1 UInt256 UInt256 +309485009821345068724781056 1 UInt256 UInt256 +618970019642690137449562112 1 UInt256 UInt256 +1237940039285380274899124224 1 UInt256 UInt256 +2475880078570760549798248448 1 UInt256 UInt256 +4951760157141521099596496896 1 UInt256 UInt256 +9903520314283042199192993792 1 UInt256 UInt256 +19807040628566084398385987584 1 UInt256 UInt256 +39614081257132168796771975168 1 UInt256 UInt256 +79228162514264337593543950336 1 UInt256 UInt256 +158456325028528675187087900672 1 UInt256 UInt256 +316912650057057350374175801344 1 UInt256 UInt256 +633825300114114700748351602688 1 UInt256 UInt256 +1267650600228229401496703205376 1 UInt256 UInt256 +2535301200456458802993406410752 1 UInt256 UInt256 +5070602400912917605986812821504 1 UInt256 UInt256 +10141204801825835211973625643008 1 UInt256 UInt256 +20282409603651670423947251286016 1 UInt256 UInt256 +40564819207303340847894502572032 1 UInt256 UInt256 +81129638414606681695789005144064 1 UInt256 UInt256 +162259276829213363391578010288128 1 UInt256 UInt256 +324518553658426726783156020576256 1 UInt256 UInt256 +649037107316853453566312041152512 1 UInt256 UInt256 +1298074214633706907132624082305024 1 UInt256 UInt256 +2596148429267413814265248164610048 1 UInt256 UInt256 +5192296858534827628530496329220096 1 UInt256 UInt256 +10384593717069655257060992658440192 1 UInt256 UInt256 +20769187434139310514121985316880384 1 UInt256 UInt256 +41538374868278621028243970633760768 1 UInt256 UInt256 +83076749736557242056487941267521536 1 UInt256 UInt256 +166153499473114484112975882535043072 1 UInt256 UInt256 +332306998946228968225951765070086144 1 UInt256 UInt256 +664613997892457936451903530140172288 1 UInt256 UInt256 +1329227995784915872903807060280344576 1 UInt256 UInt256 +2658455991569831745807614120560689152 1 UInt256 UInt256 +5316911983139663491615228241121378304 1 UInt256 UInt256 +10633823966279326983230456482242756608 1 UInt256 UInt256 +21267647932558653966460912964485513216 1 UInt256 UInt256 +42535295865117307932921825928971026432 1 UInt256 UInt256 +85070591730234615865843651857942052864 1 UInt256 UInt256 +170141183460469231731687303715884105728 1 UInt256 UInt256 +340282366920938463463374607431768211456 1 UInt256 UInt256 +680564733841876926926749214863536422912 1 UInt256 UInt256 +1361129467683753853853498429727072845824 1 UInt256 UInt256 +2722258935367507707706996859454145691648 1 UInt256 UInt256 +5444517870735015415413993718908291383296 1 UInt256 UInt256 +10889035741470030830827987437816582766592 1 UInt256 UInt256 +21778071482940061661655974875633165533184 1 UInt256 UInt256 +43556142965880123323311949751266331066368 1 UInt256 UInt256 +87112285931760246646623899502532662132736 1 UInt256 UInt256 +174224571863520493293247799005065324265472 1 UInt256 UInt256 +348449143727040986586495598010130648530944 1 UInt256 UInt256 +696898287454081973172991196020261297061888 1 UInt256 UInt256 +1393796574908163946345982392040522594123776 1 UInt256 UInt256 +2787593149816327892691964784081045188247552 1 UInt256 UInt256 +5575186299632655785383929568162090376495104 1 UInt256 UInt256 +11150372599265311570767859136324180752990208 1 UInt256 UInt256 +22300745198530623141535718272648361505980416 1 UInt256 UInt256 +44601490397061246283071436545296723011960832 1 UInt256 UInt256 +89202980794122492566142873090593446023921664 1 UInt256 UInt256 +178405961588244985132285746181186892047843328 1 UInt256 UInt256 +356811923176489970264571492362373784095686656 1 UInt256 UInt256 +713623846352979940529142984724747568191373312 1 UInt256 UInt256 +1427247692705959881058285969449495136382746624 1 UInt256 UInt256 +2854495385411919762116571938898990272765493248 1 UInt256 UInt256 +5708990770823839524233143877797980545530986496 1 UInt256 UInt256 +11417981541647679048466287755595961091061972992 1 UInt256 UInt256 +22835963083295358096932575511191922182123945984 1 UInt256 UInt256 +45671926166590716193865151022383844364247891968 1 UInt256 UInt256 +91343852333181432387730302044767688728495783936 1 UInt256 UInt256 +182687704666362864775460604089535377456991567872 1 UInt256 UInt256 +365375409332725729550921208179070754913983135744 1 UInt256 UInt256 +730750818665451459101842416358141509827966271488 1 UInt256 UInt256 +1461501637330902918203684832716283019655932542976 1 UInt256 UInt256 +2923003274661805836407369665432566039311865085952 1 UInt256 UInt256 +5846006549323611672814739330865132078623730171904 1 UInt256 UInt256 +11692013098647223345629478661730264157247460343808 1 UInt256 UInt256 +23384026197294446691258957323460528314494920687616 1 UInt256 UInt256 +46768052394588893382517914646921056628989841375232 1 UInt256 UInt256 +93536104789177786765035829293842113257979682750464 1 UInt256 UInt256 +187072209578355573530071658587684226515959365500928 1 UInt256 UInt256 +374144419156711147060143317175368453031918731001856 1 UInt256 UInt256 +748288838313422294120286634350736906063837462003712 1 UInt256 UInt256 +1496577676626844588240573268701473812127674924007424 1 UInt256 UInt256 +2993155353253689176481146537402947624255349848014848 1 UInt256 UInt256 +5986310706507378352962293074805895248510699696029696 1 UInt256 UInt256 +11972621413014756705924586149611790497021399392059392 1 UInt256 UInt256 +23945242826029513411849172299223580994042798784118784 1 UInt256 UInt256 +47890485652059026823698344598447161988085597568237568 1 UInt256 UInt256 +95780971304118053647396689196894323976171195136475136 1 UInt256 UInt256 +191561942608236107294793378393788647952342390272950272 1 UInt256 UInt256 +383123885216472214589586756787577295904684780545900544 1 UInt256 UInt256 +766247770432944429179173513575154591809369561091801088 1 UInt256 UInt256 +1532495540865888858358347027150309183618739122183602176 1 UInt256 UInt256 +3064991081731777716716694054300618367237478244367204352 1 UInt256 UInt256 +6129982163463555433433388108601236734474956488734408704 1 UInt256 UInt256 +12259964326927110866866776217202473468949912977468817408 1 UInt256 UInt256 +24519928653854221733733552434404946937899825954937634816 1 UInt256 UInt256 +49039857307708443467467104868809893875799651909875269632 1 UInt256 UInt256 +98079714615416886934934209737619787751599303819750539264 1 UInt256 UInt256 +196159429230833773869868419475239575503198607639501078528 1 UInt256 UInt256 +392318858461667547739736838950479151006397215279002157056 1 UInt256 UInt256 +784637716923335095479473677900958302012794430558004314112 1 UInt256 UInt256 +1569275433846670190958947355801916604025588861116008628224 1 UInt256 UInt256 +3138550867693340381917894711603833208051177722232017256448 1 UInt256 UInt256 +6277101735386680763835789423207666416102355444464034512896 1 UInt256 UInt256 +12554203470773361527671578846415332832204710888928069025792 1 UInt256 UInt256 +25108406941546723055343157692830665664409421777856138051584 1 UInt256 UInt256 +50216813883093446110686315385661331328818843555712276103168 1 UInt256 UInt256 +100433627766186892221372630771322662657637687111424552206336 1 UInt256 UInt256 +200867255532373784442745261542645325315275374222849104412672 1 UInt256 UInt256 +401734511064747568885490523085290650630550748445698208825344 1 UInt256 UInt256 +803469022129495137770981046170581301261101496891396417650688 1 UInt256 UInt256 +1606938044258990275541962092341162602522202993782792835301376 1 UInt256 UInt256 +3213876088517980551083924184682325205044405987565585670602752 1 UInt256 UInt256 +6427752177035961102167848369364650410088811975131171341205504 1 UInt256 UInt256 +12855504354071922204335696738729300820177623950262342682411008 1 UInt256 UInt256 +25711008708143844408671393477458601640355247900524685364822016 1 UInt256 UInt256 +51422017416287688817342786954917203280710495801049370729644032 1 UInt256 UInt256 +102844034832575377634685573909834406561420991602098741459288064 1 UInt256 UInt256 +205688069665150755269371147819668813122841983204197482918576128 1 UInt256 UInt256 +411376139330301510538742295639337626245683966408394965837152256 1 UInt256 UInt256 +822752278660603021077484591278675252491367932816789931674304512 1 UInt256 UInt256 +1645504557321206042154969182557350504982735865633579863348609024 1 UInt256 UInt256 +3291009114642412084309938365114701009965471731267159726697218048 1 UInt256 UInt256 +6582018229284824168619876730229402019930943462534319453394436096 1 UInt256 UInt256 +13164036458569648337239753460458804039861886925068638906788872192 1 UInt256 UInt256 +26328072917139296674479506920917608079723773850137277813577744384 1 UInt256 UInt256 +52656145834278593348959013841835216159447547700274555627155488768 1 UInt256 UInt256 +105312291668557186697918027683670432318895095400549111254310977536 1 UInt256 UInt256 +210624583337114373395836055367340864637790190801098222508621955072 1 UInt256 UInt256 +421249166674228746791672110734681729275580381602196445017243910144 1 UInt256 UInt256 +842498333348457493583344221469363458551160763204392890034487820288 1 UInt256 UInt256 +1684996666696914987166688442938726917102321526408785780068975640576 1 UInt256 UInt256 +3369993333393829974333376885877453834204643052817571560137951281152 1 UInt256 UInt256 +6739986666787659948666753771754907668409286105635143120275902562304 1 UInt256 UInt256 +13479973333575319897333507543509815336818572211270286240551805124608 1 UInt256 UInt256 +26959946667150639794667015087019630673637144422540572481103610249216 1 UInt256 UInt256 +53919893334301279589334030174039261347274288845081144962207220498432 1 UInt256 UInt256 +107839786668602559178668060348078522694548577690162289924414440996864 1 UInt256 UInt256 +215679573337205118357336120696157045389097155380324579848828881993728 1 UInt256 UInt256 +431359146674410236714672241392314090778194310760649159697657763987456 1 UInt256 UInt256 +862718293348820473429344482784628181556388621521298319395315527974912 1 UInt256 UInt256 +1725436586697640946858688965569256363112777243042596638790631055949824 1 UInt256 UInt256 +3450873173395281893717377931138512726225554486085193277581262111899648 1 UInt256 UInt256 +6901746346790563787434755862277025452451108972170386555162524223799296 1 UInt256 UInt256 +13803492693581127574869511724554050904902217944340773110325048447598592 1 UInt256 UInt256 +27606985387162255149739023449108101809804435888681546220650096895197184 1 UInt256 UInt256 +55213970774324510299478046898216203619608871777363092441300193790394368 1 UInt256 UInt256 +110427941548649020598956093796432407239217743554726184882600387580788736 1 UInt256 UInt256 +220855883097298041197912187592864814478435487109452369765200775161577472 1 UInt256 UInt256 +441711766194596082395824375185729628956870974218904739530401550323154944 1 UInt256 UInt256 +883423532389192164791648750371459257913741948437809479060803100646309888 1 UInt256 UInt256 +1766847064778384329583297500742918515827483896875618958121606201292619776 1 UInt256 UInt256 +3533694129556768659166595001485837031654967793751237916243212402585239552 1 UInt256 UInt256 +7067388259113537318333190002971674063309935587502475832486424805170479104 1 UInt256 UInt256 +14134776518227074636666380005943348126619871175004951664972849610340958208 1 UInt256 UInt256 +28269553036454149273332760011886696253239742350009903329945699220681916416 1 UInt256 UInt256 +56539106072908298546665520023773392506479484700019806659891398441363832832 1 UInt256 UInt256 +113078212145816597093331040047546785012958969400039613319782796882727665664 1 UInt256 UInt256 +226156424291633194186662080095093570025917938800079226639565593765455331328 1 UInt256 UInt256 +452312848583266388373324160190187140051835877600158453279131187530910662656 1 UInt256 UInt256 +904625697166532776746648320380374280103671755200316906558262375061821325312 1 UInt256 UInt256 +1809251394333065553493296640760748560207343510400633813116524750123642650624 1 UInt256 UInt256 +3618502788666131106986593281521497120414687020801267626233049500247285301248 1 UInt256 UInt256 +7237005577332262213973186563042994240829374041602535252466099000494570602496 1 UInt256 UInt256 +14474011154664524427946373126085988481658748083205070504932198000989141204992 1 UInt256 UInt256 +28948022309329048855892746252171976963317496166410141009864396001978282409984 1 UInt256 UInt256 +57896044618658097711785492504343953926634992332820282019728792003956564819968 1 UInt256 UInt256 diff --git a/tests/queries/0_stateless/01440_big_int_shift.sql b/tests/queries/0_stateless/01440_big_int_shift.sql new file mode 100644 index 00000000000..e24ae1ba911 --- /dev/null +++ b/tests/queries/0_stateless/01440_big_int_shift.sql @@ -0,0 +1,3 @@ +SELECT bitShiftLeft(toInt128(1), number) x, bitShiftRight(x, number) y, toTypeName(x), toTypeName(y) FROM numbers(127) ORDER BY number; +SELECT bitShiftLeft(toInt256(1), number) x, bitShiftRight(x, number) y, toTypeName(x), toTypeName(y) FROM numbers(255) ORDER BY number; +SELECT bitShiftLeft(toUInt256(1), number) x, bitShiftRight(x, number) y, toTypeName(x), toTypeName(y) FROM numbers(256) ORDER BY number; diff --git a/tests/queries/0_stateless/01442_date_time_with_params.reference b/tests/queries/0_stateless/01442_date_time_with_params.reference new file mode 100644 index 00000000000..f55d095d164 --- /dev/null +++ b/tests/queries/0_stateless/01442_date_time_with_params.reference @@ -0,0 +1,44 @@ +2020-01-01 00:00:00 DateTime 2020-01-01 00:01:00 DateTime 2020-01-01 00:02:00.11 DateTime64(2) 2020-01-01 00:03:00 DateTime(\'Europe/Moscow\') 2020-01-01 00:04:00.220 DateTime64(3, \'Europe/Moscow\') 2020-01-01 00:05:00 DateTime 2020-01-01 00:06:00 DateTime(\'Europe/Moscow\') 2020-01-01 00:06:00 DateTime +2020-01-01 00:00:00 DateTime 2020-01-01 00:02:00.11 DateTime64(2) 2020-01-01 00:03:00 DateTime(\'Europe/Moscow\') 2020-01-01 00:04:00.220 DateTime64(3, \'Europe/Moscow\') 2020-01-01 00:05:00 DateTime +2020-01-01 00:00:00 DateTime 2020-01-01 00:02:00.11 DateTime64(2) 2020-01-01 00:03:00 DateTime(\'Europe/Moscow\') 2020-01-01 00:04:00.220 DateTime64(3, \'Europe/Moscow\') 2020-01-01 00:05:00 DateTime +2020-01-01 00:00:00 DateTime +2020-05-14 03:37:03.000 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.000 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +2020-05-14 06:37:03.253 DateTime64(3, \'Europe/Minsk\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +\N Nullable(DateTime64(3)) +2020-05-14 03:37:03.000 Nullable(DateTime64(3, \'UTC\')) +2020-05-14 03:37:03.000 Nullable(DateTime64(3, \'UTC\')) +2020-05-14 03:37:03.253 Nullable(DateTime64(3, \'UTC\')) +2020-05-14 03:37:03.253 Nullable(DateTime64(3, \'UTC\')) +2020-05-14 06:37:03.253 Nullable(DateTime64(3, \'Europe/Minsk\')) +2020-05-14 03:37:03.253 Nullable(DateTime64(3, \'UTC\')) +1970-01-01 03:00:00.000 DateTime64(3) +2020-05-14 03:37:03.000 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.000 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +2020-05-14 06:37:03.253 DateTime64(3, \'Europe/Minsk\') +2020-05-14 03:37:03.253 DateTime64(3, \'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 06:37:03 DateTime(\'Europe/Minsk\') +2020-05-14 03:37:03 DateTime(\'UTC\') +\N Nullable(DateTime) +2020-05-14 03:37:03 Nullable(DateTime(\'UTC\')) +2020-05-14 03:37:03 Nullable(DateTime(\'UTC\')) +2020-05-14 03:37:03 Nullable(DateTime(\'UTC\')) +2020-05-14 03:37:03 Nullable(DateTime(\'UTC\')) +2020-05-14 06:37:03 Nullable(DateTime(\'Europe/Minsk\')) +2020-05-14 03:37:03 Nullable(DateTime(\'UTC\')) +1970-01-01 03:00:00 DateTime +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 03:37:03 DateTime(\'UTC\') +2020-05-14 06:37:03 DateTime(\'Europe/Minsk\') +2020-05-14 03:37:03 DateTime(\'UTC\') diff --git a/tests/queries/0_stateless/01442_date_time_with_params.sql b/tests/queries/0_stateless/01442_date_time_with_params.sql new file mode 100644 index 00000000000..5ae7fe22699 --- /dev/null +++ b/tests/queries/0_stateless/01442_date_time_with_params.sql @@ -0,0 +1,65 @@ +DROP TABLE IF EXISTS test; + +CREATE TABLE test (a DateTime, b DateTime(), c DateTime(2), d DateTime('Europe/Moscow'), e DateTime(3, 'Europe/Moscow'), f DateTime32, g DateTime32('Europe/Moscow'), h DateTime(0)) ENGINE = MergeTree ORDER BY a; + +INSERT INTO test VALUES('2020-01-01 00:00:00', '2020-01-01 00:01:00', '2020-01-01 00:02:00.11', '2020-01-01 00:03:00', '2020-01-01 00:04:00.22', '2020-01-01 00:05:00', '2020-01-01 00:06:00', '2020-01-01 00:06:00'); + +SELECT a, toTypeName(a), b, toTypeName(b), c, toTypeName(c), d, toTypeName(d), e, toTypeName(e), f, toTypeName(f), g, toTypeName(g), h, toTypeName(h) FROM test; + +SELECT toDateTime('2020-01-01 00:00:00') AS a, toTypeName(a), toDateTime('2020-01-01 00:02:00.11', 2) AS b, toTypeName(b), toDateTime('2020-01-01 00:03:00', 'Europe/Moscow') AS c, toTypeName(c), toDateTime('2020-01-01 00:04:00.22', 3, 'Europe/Moscow') AS d, toTypeName(d), toDateTime('2020-01-01 00:05:00', 0) AS e, toTypeName(e); + +SELECT CAST('2020-01-01 00:00:00', 'DateTime') AS a, toTypeName(a), CAST('2020-01-01 00:02:00.11', 'DateTime(2)') AS b, toTypeName(b), CAST('2020-01-01 00:03:00', 'DateTime(\'Europe/Moscow\')') AS c, toTypeName(c), CAST('2020-01-01 00:04:00.22', 'DateTime(3, \'Europe/Moscow\')') AS d, toTypeName(d), CAST('2020-01-01 00:05:00', 'DateTime(0)') AS e, toTypeName(e); + +SELECT toDateTime32('2020-01-01 00:00:00') AS a, toTypeName(a); + +SELECT parseDateTimeBestEffort('', 3) AS a, toTypeName(a); -- {serverError 6} +SELECT parseDateTimeBestEffort('2020-05-14T03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffort('2020-05-14 03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffort('2020-05-14T03:37:03.253184', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffort('2020-05-14T03:37:03.253184Z', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffort('2020-05-14T03:37:03.253184Z', 3, 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTimeBestEffort(materialize('2020-05-14T03:37:03.253184Z'), 3, 'UTC') AS a, toTypeName(a); + +SELECT parseDateTimeBestEffortOrNull('', 3) AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull('2020-05-14T03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull('2020-05-14 03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull('2020-05-14T03:37:03.253184', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull('2020-05-14T03:37:03.253184Z', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull('2020-05-14T03:37:03.253184Z', 3, 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrNull(materialize('2020-05-14T03:37:03.253184Z'), 3, 'UTC') AS a, toTypeName(a); + +SELECT parseDateTimeBestEffortOrZero('', 3) AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero('2020-05-14T03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero('2020-05-14 03:37:03', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero('2020-05-14T03:37:03.253184', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero('2020-05-14T03:37:03.253184Z', 3, 'UTC') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero('2020-05-14T03:37:03.253184Z', 3, 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTimeBestEffortOrZero(materialize('2020-05-14T03:37:03.253184Z'), 3, 'UTC') AS a, toTypeName(a); + + +SELECT parseDateTime32BestEffort('') AS a, toTypeName(a); -- {serverError 6} +SELECT parseDateTime32BestEffort('2020-05-14T03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffort('2020-05-14 03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffort('2020-05-14T03:37:03.253184', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffort('2020-05-14T03:37:03.253184Z', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffort('2020-05-14T03:37:03.253184Z', 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTime32BestEffort(materialize('2020-05-14T03:37:03.253184Z'), 'UTC') AS a, toTypeName(a); + +SELECT parseDateTime32BestEffortOrNull('') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull('2020-05-14T03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull('2020-05-14 03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull('2020-05-14T03:37:03.253184', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull('2020-05-14T03:37:03.253184Z', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull('2020-05-14T03:37:03.253184Z', 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrNull(materialize('2020-05-14T03:37:03.253184Z'), 'UTC') AS a, toTypeName(a); + +SELECT parseDateTime32BestEffortOrZero('') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero('2020-05-14T03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero('2020-05-14 03:37:03', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero('2020-05-14T03:37:03.253184', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero('2020-05-14T03:37:03.253184Z', 'UTC') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero('2020-05-14T03:37:03.253184Z', 'Europe/Minsk') AS a, toTypeName(a); +SELECT parseDateTime32BestEffortOrZero(materialize('2020-05-14T03:37:03.253184Z'), 'UTC') AS a, toTypeName(a); + + +DROP TABLE IF EXISTS test; diff --git a/tests/queries/0_stateless/01457_int256_hashing.reference b/tests/queries/0_stateless/01457_int256_hashing.reference new file mode 100644 index 00000000000..e1a6a2ad341 --- /dev/null +++ b/tests/queries/0_stateless/01457_int256_hashing.reference @@ -0,0 +1,24 @@ +0 +123 +123 +[1,1,2] +0 +123 +123 +[1,1,2] +0 +123 +123 +[1,1,2] +0 +321 +321 +[1,1,2] +0 +321 +321 +[1,1,2] +0 +321 +321 +[1,1,2] diff --git a/tests/queries/0_stateless/01457_int256_hashing.sql b/tests/queries/0_stateless/01457_int256_hashing.sql new file mode 100644 index 00000000000..d6d655e80f3 --- /dev/null +++ b/tests/queries/0_stateless/01457_int256_hashing.sql @@ -0,0 +1,39 @@ +SELECT toUInt256(123) IN (NULL); +SELECT toUInt256(123) AS k GROUP BY k; +SELECT toUInt256(123) AS k FROM system.one INNER JOIN (SELECT toUInt256(123) AS k) t USING k; +SELECT arrayEnumerateUniq([toUInt256(123), toUInt256(456), toUInt256(123)]); + +SELECT toInt256(123) IN (NULL); +SELECT toInt256(123) AS k GROUP BY k; +SELECT toInt256(123) AS k FROM system.one INNER JOIN (SELECT toInt256(123) AS k) t USING k; +SELECT arrayEnumerateUniq([toInt256(123), toInt256(456), toInt256(123)]); + +-- SELECT toUInt128(123) IN (NULL); +-- SELECT toUInt128(123) AS k GROUP BY k; +-- SELECT toUInt128(123) AS k FROM system.one INNER JOIN (SELECT toUInt128(123) AS k) t USING k; +-- SELECT arrayEnumerateUniq([toUInt128(123), toUInt128(456), toUInt128(123)]); + +SELECT toInt128(123) IN (NULL); +SELECT toInt128(123) AS k GROUP BY k; +SELECT toInt128(123) AS k FROM system.one INNER JOIN (SELECT toInt128(123) AS k) t USING k; +SELECT arrayEnumerateUniq([toInt128(123), toInt128(456), toInt128(123)]); + +SELECT toNullable(toUInt256(321)) IN (NULL); +SELECT toNullable(toUInt256(321)) AS k GROUP BY k; +SELECT toNullable(toUInt256(321)) AS k FROM system.one INNER JOIN (SELECT toUInt256(321) AS k) t USING k; +SELECT arrayEnumerateUniq([toNullable(toUInt256(321)), toNullable(toUInt256(456)), toNullable(toUInt256(321))]); + +SELECT toNullable(toInt256(321)) IN (NULL); +SELECT toNullable(toInt256(321)) AS k GROUP BY k; +SELECT toNullable(toInt256(321)) AS k FROM system.one INNER JOIN (SELECT toInt256(321) AS k) t USING k; +SELECT arrayEnumerateUniq([toNullable(toInt256(321)), toNullable(toInt256(456)), toNullable(toInt256(321))]); + +-- SELECT toNullable(toUInt128(321)) IN (NULL); +-- SELECT toNullable(toUInt128(321)) AS k GROUP BY k; +-- SELECT toNullable(toUInt128(321)) AS k FROM system.one INNER JOIN (SELECT toUInt128(321) AS k) t USING k; +-- SELECT arrayEnumerateUniq([toNullable(toUInt128(321)), toNullable(toUInt128(456)), toNullable(toUInt128(321))]); + +SELECT toNullable(toInt128(321)) IN (NULL); +SELECT toNullable(toInt128(321)) AS k GROUP BY k; +SELECT toNullable(toInt128(321)) AS k FROM system.one INNER JOIN (SELECT toInt128(321) AS k) t USING k; +SELECT arrayEnumerateUniq([toNullable(toInt128(321)), toNullable(toInt128(456)), toNullable(toInt128(321))]); diff --git a/tests/queries/0_stateless/01463_test_alter_live_view_refresh.reference b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.reference new file mode 100644 index 00000000000..4d98c7b6838 --- /dev/null +++ b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.reference @@ -0,0 +1 @@ +ALTER LIVE VIEW live1 REFRESH diff --git a/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql new file mode 100644 index 00000000000..ab316a377fd --- /dev/null +++ b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql @@ -0,0 +1,10 @@ +CREATE TABLE test0 ( + c0 UInt64 + ) ENGINE = MergeTree() PARTITION BY c0 ORDER BY c0; + +SET allow_experimental_live_view=1; + +CREATE LIVE VIEW live1 AS SELECT * FROM test0; + +select 'ALTER LIVE VIEW live1 REFRESH'; +ALTER LIVE VIEW live1 REFRESH; -- success diff --git a/tests/queries/0_stateless/01470_columns_transformers.reference b/tests/queries/0_stateless/01470_columns_transformers.reference new file mode 100644 index 00000000000..595d99b917f --- /dev/null +++ b/tests/queries/0_stateless/01470_columns_transformers.reference @@ -0,0 +1,63 @@ +220 18 347 +110 9 173.5 +1970-04-11 1970-01-11 1970-11-21 +2 3 +1 2 +18 347 +110 173.5 +1970-04-11 1970-01-11 1970-11-21 +222 18 347 +111 11 173.5 +1970-04-11 1970-01-11 1970-11-21 +SELECT + sum(i), + sum(j), + sum(k) +FROM columns_transformers +SELECT + avg(i), + avg(j), + avg(k) +FROM columns_transformers +SELECT + toDate(any(i)), + toDate(any(j)), + toDate(any(k)) +FROM columns_transformers AS a +SELECT + length(toString(j)), + length(toString(k)) +FROM columns_transformers +SELECT + sum(j), + sum(k) +FROM columns_transformers +SELECT + avg(i), + avg(k) +FROM columns_transformers +SELECT + toDate(any(i)), + toDate(any(j)), + toDate(any(k)) +FROM columns_transformers AS a +SELECT + sum(i + 1 AS i), + sum(j), + sum(k) +FROM columns_transformers +SELECT + avg(i + 1 AS i), + avg(j + 2 AS j), + avg(k) +FROM columns_transformers +SELECT + toDate(any(i)), + toDate(any(j)), + toDate(any(k)) +FROM columns_transformers AS a +SELECT + (i + 1) + 1 AS i, + j, + k +FROM columns_transformers diff --git a/tests/queries/0_stateless/01470_columns_transformers.sql b/tests/queries/0_stateless/01470_columns_transformers.sql new file mode 100644 index 00000000000..de6a1a89d81 --- /dev/null +++ b/tests/queries/0_stateless/01470_columns_transformers.sql @@ -0,0 +1,36 @@ +DROP TABLE IF EXISTS columns_transformers; + +CREATE TABLE columns_transformers (i Int64, j Int16, k Int64) Engine=TinyLog; +INSERT INTO columns_transformers VALUES (100, 10, 324), (120, 8, 23); + +SELECT * APPLY(sum) from columns_transformers; +SELECT columns_transformers.* APPLY(avg) from columns_transformers; +SELECT a.* APPLY(toDate) APPLY(any) from columns_transformers a; +SELECT COLUMNS('[jk]') APPLY(toString) APPLY(length) from columns_transformers; + +SELECT * EXCEPT(i) APPLY(sum) from columns_transformers; +SELECT columns_transformers.* EXCEPT(j) APPLY(avg) from columns_transformers; +-- EXCEPT after APPLY will not match anything +SELECT a.* APPLY(toDate) EXCEPT(i, j) APPLY(any) from columns_transformers a; + +SELECT * REPLACE(i + 1 AS i) APPLY(sum) from columns_transformers; +SELECT columns_transformers.* REPLACE(j + 2 AS j, i + 1 AS i) APPLY(avg) from columns_transformers; +SELECT columns_transformers.* REPLACE(j + 1 AS j, j + 2 AS j) APPLY(avg) from columns_transformers; -- { serverError 43 } +-- REPLACE after APPLY will not match anything +SELECT a.* APPLY(toDate) REPLACE(i + 1 AS i) APPLY(any) from columns_transformers a; + +EXPLAIN SYNTAX SELECT * APPLY(sum) from columns_transformers; +EXPLAIN SYNTAX SELECT columns_transformers.* APPLY(avg) from columns_transformers; +EXPLAIN SYNTAX SELECT a.* APPLY(toDate) APPLY(any) from columns_transformers a; +EXPLAIN SYNTAX SELECT COLUMNS('[jk]') APPLY(toString) APPLY(length) from columns_transformers; +EXPLAIN SYNTAX SELECT * EXCEPT(i) APPLY(sum) from columns_transformers; +EXPLAIN SYNTAX SELECT columns_transformers.* EXCEPT(j) APPLY(avg) from columns_transformers; +EXPLAIN SYNTAX SELECT a.* APPLY(toDate) EXCEPT(i, j) APPLY(any) from columns_transformers a; +EXPLAIN SYNTAX SELECT * REPLACE(i + 1 AS i) APPLY(sum) from columns_transformers; +EXPLAIN SYNTAX SELECT columns_transformers.* REPLACE(j + 2 AS j, i + 1 AS i) APPLY(avg) from columns_transformers; +EXPLAIN SYNTAX SELECT a.* APPLY(toDate) REPLACE(i + 1 AS i) APPLY(any) from columns_transformers a; + +-- Multiple REPLACE in a row +EXPLAIN SYNTAX SELECT * REPLACE(i + 1 AS i) REPLACE(i + 1 AS i) from columns_transformers; + +DROP TABLE columns_transformers; diff --git a/tests/queries/0_stateless/01470_explain.reference b/tests/queries/0_stateless/01470_explain.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/01470_explain.sql b/tests/queries/0_stateless/01470_explain.sql new file mode 100644 index 00000000000..8fd145e7f65 --- /dev/null +++ b/tests/queries/0_stateless/01470_explain.sql @@ -0,0 +1,6 @@ +-- +-- regressions +-- + +-- SIGSEGV regression due to QueryPlan lifetime +EXPLAIN PIPELINE graph=1 SELECT * FROM remote('127.{1,2}', system.one) FORMAT Null; diff --git a/tests/queries/0_stateless/01471_calculate_ttl_during_merge.reference b/tests/queries/0_stateless/01471_calculate_ttl_during_merge.reference new file mode 100644 index 00000000000..1e682ec38a9 --- /dev/null +++ b/tests/queries/0_stateless/01471_calculate_ttl_during_merge.reference @@ -0,0 +1,5 @@ +3000 +3000 +2000 +2000 +1001 diff --git a/tests/queries/0_stateless/01471_calculate_ttl_during_merge.sql b/tests/queries/0_stateless/01471_calculate_ttl_during_merge.sql new file mode 100644 index 00000000000..901c47bc10f --- /dev/null +++ b/tests/queries/0_stateless/01471_calculate_ttl_during_merge.sql @@ -0,0 +1,37 @@ +DROP TABLE IF EXISTS table_for_ttl; + +CREATE TABLE table_for_ttl( + d DateTime, + key UInt64, + value String) +ENGINE = MergeTree() +ORDER BY tuple() +PARTITION BY key; + +INSERT INTO table_for_ttl SELECT now() - INTERVAL 2 YEAR, 1, toString(number) from numbers(1000); + +INSERT INTO table_for_ttl SELECT now() - INTERVAL 2 DAY, 3, toString(number) from numbers(2000, 1000); + +INSERT INTO table_for_ttl SELECT now(), 4, toString(number) from numbers(3000, 1000); + +SELECT count() FROM table_for_ttl; + +ALTER TABLE table_for_ttl MODIFY TTL d + INTERVAL 1 YEAR SETTINGS materialize_ttl_after_modify = 0; + +SELECT count() FROM table_for_ttl; + +OPTIMIZE TABLE table_for_ttl FINAL; + +SELECT count() FROM table_for_ttl; + +ALTER TABLE table_for_ttl MODIFY COLUMN value String TTL d + INTERVAL 1 DAY SETTINGS materialize_ttl_after_modify = 0; + +SELECT count(distinct value) FROM table_for_ttl; + +OPTIMIZE TABLE table_for_ttl FINAL; + +SELECT count(distinct value) FROM table_for_ttl; + +OPTIMIZE TABLE table_for_ttl FINAL; -- Just check in logs, that it doesn't run with force again + +DROP TABLE IF EXISTS table_for_ttl; diff --git a/tests/queries/0_stateless/01471_top_k_range_check.reference b/tests/queries/0_stateless/01471_top_k_range_check.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/01471_top_k_range_check.sql b/tests/queries/0_stateless/01471_top_k_range_check.sql new file mode 100644 index 00000000000..1e7ac04bbc5 --- /dev/null +++ b/tests/queries/0_stateless/01471_top_k_range_check.sql @@ -0,0 +1 @@ +SELECT length(topKWeighted(2, -9223372036854775808)(number, 1025)) FROM system.numbers; -- { serverError 69 } diff --git a/tests/queries/0_stateless/arcadia_skip_list.txt b/tests/queries/0_stateless/arcadia_skip_list.txt index 357c42b3664..707f91b0c93 100644 --- a/tests/queries/0_stateless/arcadia_skip_list.txt +++ b/tests/queries/0_stateless/arcadia_skip_list.txt @@ -1,6 +1,7 @@ # Add testcase here to skip it in Arcadia CI (Yandex synchronization check) # It is useful for tests with not supported features in Arcadia build 00105_shard_collations +00184_shard_distributed_group_by_no_merge 00436_convert_charset 00490_special_line_separators_and_characters_outside_of_bmp 00506_union_distributed @@ -101,8 +102,8 @@ 01236_distributed_over_live_view_over_distributed 01236_graphite_mt 01237_live_view_over_distributed_with_subquery_select_table_alias -01247_dist_on_dist_group_by_sharding_key_optimization -01247_distributed_group_by_no_merge_GROUP_BY_injective_sharding_key +01247_optimize_distributed_group_by_sharding_key +01247_optimize_distributed_group_by_sharding_key_dist_on_dist 01251_dict_is_in_infinite_loop 01253_subquery_in_aggregate_function_JustStranger 01254_dict_create_without_db diff --git a/tests/testflows/rbac/tests/syntax/grant_privilege.py b/tests/testflows/rbac/tests/syntax/grant_privilege.py index cabb3a3780b..82c459f546d 100755 --- a/tests/testflows/rbac/tests/syntax/grant_privilege.py +++ b/tests/testflows/rbac/tests/syntax/grant_privilege.py @@ -20,30 +20,30 @@ def setup(node): node.query("DROP ROLE IF EXISTS role1") @TestOutline(Scenario) -@Examples("privilege on allow_introspection", [ - ("dictGet", ("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_DictGet("1.0"))), - ("INTROSPECTION", ("*.*",), True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Introspection("1.0"))), - ("SELECT", ("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Select("1.0"))), - ("INSERT",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Insert("1.0"))), - ("ALTER",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Alter("1.0"))), - ("CREATE",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Create("1.0"))), - ("DROP",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Drop("1.0"))), - ("TRUNCATE",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Truncate("1.0"))), - ("OPTIMIZE",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Optimize("1.0"))), - ("SHOW",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Show("1.0"))), - ("KILL QUERY",("*.*",), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_KillQuery("1.0"))), - ("ACCESS MANAGEMENT",("*.*",), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_AccessManagement("1.0"))), - ("SYSTEM",("db0.table0","db0.*","*.*","tb0","*"), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_System("1.0"))), - ("SOURCES",("*.*",), False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Sources("1.0"))), - ("ALL",("*.*",), True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_All("1.0"))), - ("ALL PRIVILEGES",("*.*",), True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_All("1.0"))), #alias for all +@Examples("privilege on allow_column allow_introspection", [ + ("dictGet", ("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_DictGet("1.0"))), + ("INTROSPECTION", ("*.*",), False, True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Introspection("1.0"))), + ("SELECT", ("db0.table0","db0.*","*.*","tb0","*"), True, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Select("1.0"))), + ("INSERT",("db0.table0","db0.*","*.*","tb0","*"), True, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Insert("1.0"))), + ("ALTER",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Alter("1.0"))), + ("CREATE",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Create("1.0"))), + ("DROP",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Drop("1.0"))), + ("TRUNCATE",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Truncate("1.0"))), + ("OPTIMIZE",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Optimize("1.0"))), + ("SHOW",("db0.table0","db0.*","*.*","tb0","*"), True, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Show("1.0"))), + ("KILL QUERY",("*.*",), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_KillQuery("1.0"))), + ("ACCESS MANAGEMENT",("*.*",), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_AccessManagement("1.0"))), + ("SYSTEM",("db0.table0","db0.*","*.*","tb0","*"), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_System("1.0"))), + ("SOURCES",("*.*",), False, False, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_Sources("1.0"))), + ("ALL",("*.*",), True, True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_All("1.0"))), + ("ALL PRIVILEGES",("*.*",), True, True, Requirements(RQ_SRS_006_RBAC_Grant_Privilege_All("1.0"))), #alias for all ],) -def grant_privileges(self, privilege, on, allow_introspection, node="clickhouse1"): - grant_privilege(privilege=privilege, on=on, allow_introspection=allow_introspection, node=node) +def grant_privileges(self, privilege, on, allow_column, allow_introspection, node="clickhouse1"): + grant_privilege(privilege=privilege, on=on, allow_column=allow_column, allow_introspection=allow_introspection, node=node) @TestOutline(Scenario) @Requirements(RQ_SRS_006_RBAC_Grant_Privilege_GrantOption("1.0")) -def grant_privilege(self, privilege, on, allow_introspection, node="clickhouse1"): +def grant_privilege(self, privilege, on, allow_column, allow_introspection, node="clickhouse1"): node = self.context.cluster.node(node) for on_ in on: @@ -58,9 +58,10 @@ def grant_privilege(self, privilege, on, allow_introspection, node="clickhouse1" with When("I grant privilege with grant option"): node.query(f"GRANT {privilege} ON {on_} TO user1 WITH GRANT OPTION", settings=settings) - #grant column specific for some column 'x' - with When("I grant privilege with columns"): - node.query(f"GRANT {privilege}(x) ON {on_} TO user0", settings=settings) + if allow_column and ('*' not in on_): + #grant column specific for some column 'x' + with When("I grant privilege with columns"): + node.query(f"GRANT {privilege}(x) ON {on_} TO user0", settings=settings) @TestFeature @Name("grant privilege") diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index 0dd95388e7d..b4408a298c3 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -28,6 +28,7 @@ if (NOT DEFINED ENABLE_UTILS OR ENABLE_UTILS) add_subdirectory (test-data-generator) add_subdirectory (convert-month-partitioned-parts) add_subdirectory (checksum-for-compressed-block) + add_subdirectory (db-generator) add_subdirectory (wal-dump) endif () diff --git a/utils/build/build_no_submodules.sh b/utils/build/build_no_submodules.sh index 4bcbe0b2a17..f9e2b6032a5 100755 --- a/utils/build/build_no_submodules.sh +++ b/utils/build/build_no_submodules.sh @@ -11,7 +11,7 @@ ROOT_DIR=${CUR_DIR}/../../build_no_submodules mkdir -p $ROOT_DIR cd $ROOT_DIR URL=`git remote get-url origin | sed 's/.git$//'` -wget -O ch.zip $URL/archive/${BRANCH}.zip +wget -nv -O ch.zip $URL/archive/${BRANCH}.zip unzip -ou ch.zip # TODO: make disableable lz4 zstd diff --git a/utils/ci/build-gcc-from-sources.sh b/utils/ci/build-gcc-from-sources.sh index 06d9820a022..8886bb7afd7 100755 --- a/utils/ci/build-gcc-from-sources.sh +++ b/utils/ci/build-gcc-from-sources.sh @@ -18,7 +18,7 @@ THREADS=$(grep -c ^processor /proc/cpuinfo) mkdir "${WORKSPACE}/gcc" pushd "${WORKSPACE}/gcc" -wget https://ftpmirror.gnu.org/gcc/${GCC_SOURCES_VERSION}/${GCC_SOURCES_VERSION}.tar.xz +wget -nv https://ftpmirror.gnu.org/gcc/${GCC_SOURCES_VERSION}/${GCC_SOURCES_VERSION}.tar.xz tar xf ${GCC_SOURCES_VERSION}.tar.xz pushd ${GCC_SOURCES_VERSION} ./contrib/download_prerequisites diff --git a/utils/ci/docker-multiarch/update.sh b/utils/ci/docker-multiarch/update.sh index 6abcf339607..1348631bdcf 100755 --- a/utils/ci/docker-multiarch/update.sh +++ b/utils/ci/docker-multiarch/update.sh @@ -29,7 +29,7 @@ baseUrl="https://partner-images.canonical.com/core/$VERSION" # install qemu-user-static if [ -n "${QEMU_ARCH}" ]; then if [ ! -f x86_64_qemu-${QEMU_ARCH}-static.tar.gz ]; then - wget -N https://github.com/multiarch/qemu-user-static/releases/download/${QEMU_VER}/x86_64_qemu-${QEMU_ARCH}-static.tar.gz + wget -nv -N https://github.com/multiarch/qemu-user-static/releases/download/${QEMU_VER}/x86_64_qemu-${QEMU_ARCH}-static.tar.gz fi tar -xvf x86_64_qemu-${QEMU_ARCH}-static.tar.gz -C $ROOTFS/usr/bin/ fi @@ -37,13 +37,13 @@ fi # get the image if \ - wget -q --spider "$baseUrl/current" \ - && wget -q --spider "$baseUrl/current/$thisTar" \ + wget -nv --spider "$baseUrl/current" \ + && wget -nv --spider "$baseUrl/current/$thisTar" \ ; then baseUrl+='/current' fi -wget -qN "$baseUrl/"{{MD5,SHA{1,256}}SUMS{,.gpg},"$thisTarBase.manifest",'unpacked/build-info.txt'} || true -wget -N "$baseUrl/$thisTar" +wget -nv -N "$baseUrl/"{{MD5,SHA{1,256}}SUMS{,.gpg},"$thisTarBase.manifest",'unpacked/build-info.txt'} || true +wget -nv -N "$baseUrl/$thisTar" # check checksum if [ -f SHA256SUMS ]; then diff --git a/utils/clickhouse-docker b/utils/clickhouse-docker index a3354aadacb..383a82e6d2c 100755 --- a/utils/clickhouse-docker +++ b/utils/clickhouse-docker @@ -24,7 +24,7 @@ param="$1" if [ "${param}" = "list" ] then # https://stackoverflow.com/a/39454426/1555175 - wget -q https://registry.hub.docker.com/v1/repositories/yandex/clickhouse-server/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}' + wget -nv https://registry.hub.docker.com/v1/repositories/yandex/clickhouse-server/tags -O - | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n' | awk -F: '{print $3}' else docker pull yandex/clickhouse-server:${param} tmp_dir=$(mktemp -d -t ci-XXXXXXXXXX) # older version require /nonexistent folder to exist to run clickhouse client :D diff --git a/utils/db-generator/CMakeLists.txt b/utils/db-generator/CMakeLists.txt new file mode 100644 index 00000000000..f1a86a7f8af --- /dev/null +++ b/utils/db-generator/CMakeLists.txt @@ -0,0 +1,2 @@ +add_executable (query_db_generator query_db_generator.cpp) +target_link_libraries(query_db_generator PRIVATE clickhouse_parsers) \ No newline at end of file diff --git a/utils/db-generator/README.md b/utils/db-generator/README.md new file mode 100644 index 00000000000..1d276063bf9 --- /dev/null +++ b/utils/db-generator/README.md @@ -0,0 +1,34 @@ +**Анализ запроса в Clickhouse** + +В данной работе мы будем рассматривать только select запросы, то есть те запросы, которые из таблицы достают данные. +Встроенный парсер Clickhouse принимает на вход строку, которая является запросом. В select запросе выделяется 14 основных частей: WITH, SELECT, TABLES, PREWHERE, WHERE, GROUP_BY, HAVING, ORDER_BY, LIMIT_BY_OFFSET, LIMIT_BY_LENGTH, LIMIT_BY, LIMIT_OFFSET, LIMIT_LENGTH, SETTINGS. Мы будем рассматривать части SELECT, TABLES, WHERE, GROUP_BY, HAVING, ORDER_BY так как именно в них находятся основные данные, нужные нам для анализа структуры и выявления значений. После анализа запроса парсер выдает нам древовидную структуру, где каждая вершина является определенной операцией выполнения запроса, функцией над значениями, константой, обозначением и тому подобное. Вершины также имеют свои поддеревья, в которых находятся их аргументы или подоперации. Обходя данное дерево мы будем пытаться выявить необходимые нам данные. + +**Анализ схемы** + +По запросу необходимо определить возможные таблицы. Имея строку запроса можно понять, какие его части обозначают названия таблиц, таким образом можно определить их количество в нашей базе данных. +В парсере Clickhouse поддеревом запроса, отвечающее за таблицы из которых мы берем данные, является TABLES (Рисунок 1), в нем лежит основная таблица, из которой берутся колонки, а также операции JOIN, которые совершаются в запросе. Обходя все вершины в поддереве мы берем названия таблиц и баз данных в которых они лежат, а также их алиас, то есть укороченные названия, выбранные автором запроса. Эти названия могут понадобиться нам для определения принадлежности колонки в дальнейшем. +Таким образом для запроса мы получаем набор баз данных, а также таблиц и их условных обозначений (алиасов), по которым делается запрос. + +Затем нам необходимо определить множество столбцов, которые присутствуют в запросе и таблицы, к которым они могут относиться. Во время исполнения запроса уже известно множество столбцов в каждой таблице, поэтому при исполнении программа автоматически связывает столбец и таблицу, однако в нашем случае нельзя однозначно трактовать принадлежность столбца к определенной таблице, например в следующем запросе: “SELECT column1, column2, column3 FROM table1 JOIN table2 on table1.column2 = table2.column3 ”. Здесь мы однозначно можем сказать, к какой таблице относятся колонки column2 и column3, однако column1 может принадлежать как первой, так и второй таблице. Для однозначности трактовки таких случаев, мы будем относить данную неопределенные колонки к основной таблице, по которой делается запрос, например в данном случае это будет таблица table1. +Все столбцы в дереве лежат в вершинах типа INDENTIFIER, которые находятся в поддеревьях SELECT, TABLES, WHERE, GROUP_BY, HAVING, ORDER_BY. Рекурсивно обходя поддеревья мы формируем множество всех таблиц, затем мы разделяем колонку на составляющие: таблица (если она явно указана через точку) и само название, затем, так как таблица может являться алиасом, мы заменяем алиас на оригинальное название таблицы. Теперь у нас есть список всех столбцов и таблиц, к которым они относятся, для столбцов без таблиц определяем основную таблицу запроса. + +**Анализ столбцов** + +Продолжением является точное определение типов данных для столбцов, у которых в запросе присутствует значение. Примером являются логические условие WHERE, в котором у определенного набора атрибутов проверяется логическое выражение. Если в запросе указано column > 5, то можно сделать вывод, что в данном столбце содержится численное значение, либо если на атрибут применяется выражение LIKE, то атрибут представляет собой строковый тип. +В данной части необходимо научится вычленять из запроса все таки выражения и сопоставлять типы данных для тех столбцов, для которых это возможно сделать. При этом, понятно, что из присутствующих значений не всегда можно сделать однозначное решение о типе конкретного атрибута, например column > 5 может означать множество численных типов таких как UINT8, UINT32, INT32, INT64 и тому подобных. Здесь нужно определиться с трактовкой определенных значений, так как перебор всех возможных может быть достаточно большим, а поэтому занимать продолжительное время. +Для числовых значений было решено использовать INT64(целочисленный тип 64 битности) для целочисленных значений и FLOAT64(число с плавающей точкой 64 битности) для нецелых значений. Также используются типы STRING для строковых значений, DATE для дат, DATETIME для времени. Стоит заметить, что существует еще тип ARRAY, который является оберткой над предыдущими типами и представлять собой массив из значений определенного типа. +Определить значения столбцов мы можем используя логический, арифметические и другие функции над значениями столбцов, которые указаны в запросе. Такие функции лежат в поддеревьях SELECT и WHERE. Параметром функции может быть константа, колонка либо другая функция (Рисунок 2). Таким образом для понимания типа колонки могут помочь следующие параметры: 1) Типы аргументов, которые может принимать функция, например функция TOSTARTOFMINUTE(округляет время до кратного 5 минутам вниз) может принимать только DATETIME, таким образом если аргументом данной функции является колонка, то данная колонка имеет тип DATETIME. 2) типы остальных аргументов в данной функции, например функция EQUALS(равенство), она подразумевает собой равенство типов ее аргументов, таким образом если в данной функции присутствует константа и столбец, то мы можем определить тип столбца как тип константы. + +Таким образом, для каждой функции мы определяем возможные типы аргументов, тип возвращаемого значения, а также параметр, являются ли аргументы функции одинакового типа. Рекурсивный обработчик функций будет определять возможные типы столбцов использующихся в данных функциях по значениям аргументов и возвращать возможные типы результата выполнения функции. +Теперь для каждого столбца мы имеем множество возможных типов его значений. Для однозначной трактовки запроса мы выберем один конкретный тип из этого множества. + +**Определение значений столбцов** + +На этом этапе мы уже имеем определенную структуру таблиц базы данных, нам необходимо заполнить эту таблицу значениям. Нам необходимо понять, какие столбцы зависят друг от друга при исполнении функции (например по двум столбцами делается join, значит они должны иметь одинаковые значения), а также какие значения должны принимать столбцы, чтобы выполнялись различные условия при исполнении. +Для достижения цели ищем все операции сравнения в нашем запросе, если аргументами операции являются два столбца, то мы считаем их связанными, если аргументами являются столбец и значение, то присваиваем данное значение возможным значением данного столбца, а также добавляем данное значение + определенный шум. Для числового типа шумом является случайное число, для даты - случайное количество дней и т.п. При этом для каждой операции сравнения необходим свой обработчик этой операции, который генерирует хотя бы два значения, одно из которых условие операции, а другое нет. Например, для операции column1 > 5, column1 должно присваиваться значение большее 5 и меньшее, либо равное 5, аналогично для операции “column2 LIKE some%string”, столбцу column2 должно присваиваться значение удовлетворяющее выражение, а также не удовлетворяющее. +Теперь для некоторых колонок мы имеем множество связанных с ними колонок и множество значений. Мы знаем, что связность колонок симметрична, но для полноценного определения связности колонок нам необходимо добавить транзитивность, т.к если “column1 = column2” и “column2 = column3”, то “column1 = column3”, но это не вытекает из построения. Соответственно нам необходимо распространить связность по всем колонкам. Затем мы для каждой колонки объединяем множество ее значений со значениями всех связанных с ней. Теперь если у нас остались колонки без значений, мы просто генерируем случайные значения. + +**Генерация записей** + +Теперь у нас есть полноценной представление схемы базы данных, а также множество значений каждой таблицы. Мы будем генерировать данные посредством декартова произведения множества значений каждого столбца для определенной таблицы. Таким образом мы получаем для каждой таблицы множество, состоящее из множеств значений каждого столбца. +По этим данным мы начинаем генерировать запросы, создающие данную таблицу и заполняет ее данными. По структуре таблицы и типам ее столбцов мы генерируем CREATE QUERY, которая создает данную таблицу. Затем по множеству значений мы генерируем INSERT QUERY, которая заполняет данную таблицу данными. \ No newline at end of file diff --git a/utils/db-generator/query_db_generator.cpp b/utils/db-generator/query_db_generator.cpp new file mode 100644 index 00000000000..ccef60e7ef2 --- /dev/null +++ b/utils/db-generator/query_db_generator.cpp @@ -0,0 +1,1307 @@ +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using ColumnType = uint32_t; +using TableAndColumn = std::pair; +pcg64 rng; + +std::string randomString(size_t length) +{ + auto randchar = []() -> char + { + const char charset[] = "0123456789" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz"; + const size_t max_index = (sizeof(charset) - 1); + return charset[rng() % max_index]; + }; + std::string str(length, 0); + std::generate_n(str.begin(), length, randchar); + return str; +} +std::string randomInteger(unsigned int min = 0, unsigned int max = 4294967295) +{ + int r = rng() % (max - min) + min; + return std::to_string(r); +} + +std::string randomFloat(unsigned int min = 0, unsigned int max = 4294967295) +{ + float r = static_cast(rng() % max) / (static_cast(rng() % 100)) + min; + return std::to_string(r); +} + +std::string randomDate() +{ + int32_t year = rng() % 136 + 1970; + int32_t month = rng() % 12 + 1; + int32_t day = rng() % 12 + 1; + char ans[13]; + sprintf(ans, "'%04u-%02u-%02u'", year, month, day); + return std::string(ans); +} + +std::string randomDatetime() +{ + int32_t year = rng() % 136 + 1970; + int32_t month = rng() % 12 + 1; + int32_t day = rng() % 12 + 1; + int32_t hours = rng() % 24; + int32_t minutes = rng() % 60; + int32_t seconds = rng() % 60; + char ans[22]; + sprintf( + ans, + "'%04u-%02u-%02u %02u:%02u:%02u'", + year, + month, + day, + hours, + minutes, + seconds); + return std::string(ans); +} +TableAndColumn get_table_a_column(const std::string & c) +{ + auto point_place = c.rfind('.'); + std::string db{}; + std::string column{}; + if (point_place != std::string::npos) + { + db = c.substr(0, point_place); + column = c.substr(point_place + 1); + } + else + { + column = c; + } + return { db, column }; +} + + +enum type : ColumnType +{ + i = 1, + // int + f = 2, + // float + s = 4, + // string + d = 8, + // date + dt = 16, + // datetime + b = 32, + // bool + all = 63, + a = 64, + // array + t = 128, + // tuple +}; + + +std::map type_definition = { + {type::i, "Int64"}, {type::f, "Float64"}, {type::s, "String"}, {type::d, "Date"}, {type::dt, "DateTime"}, {type::b, "UInt8"} +}; +ColumnType time_type(std::string value) +{ + if (value.length() == 12) + { + for (size_t i : {5, 8}) + { + if (value[i] != '-') + return type::s; + } + for (size_t i : {1, 2, 3, 4, 6, 7, 9, 10}) + { + if (!isdigit(value[i])) + return type::s; + } + return type::d; + } + + if (value.length() == 21) + { + for (size_t i : {5, 8}) + { + if (value[i] != '-') + return type::s; + } + for (size_t i : {14, 17}) + { + if (value[i] != '-') + return type::s; + } + if (value[11] != '-') + return type::s; + return type::dt; + } + return type::s; +} +// Casting inner clickhouse parser type to our type +ColumnType type_cast(int t) +{ + switch (t) + { + case 1: + case 2: + case 4: + case 5: + case 19: + case 20: + case 21: + return type::i; + + case 3: + return type::f; + + case 16: + return type::s; + + case 17: + return type::a | type::all; + + case 18: + return type::t | type::all; + } + return type::all; +} + + +class FuncRet +{ +public: + FuncRet() = default; + + FuncRet(ColumnType t, std::string v) + : value(v) + , type(t) {} + + FuncRet(ColumnType t, std::string v, bool is_a) + : value(v) + , type(t) + , is_array(is_a) {} + + std::string value{}; + ColumnType type = type::all; + bool is_array = false; +}; + + +std::map func_to_return_type = { + {"divide", FuncRet(type::f, "")}, {"e", FuncRet(type::f, "e()")}, {"pi", FuncRet(type::f, "pi()")}, {"exp", FuncRet(type::f, "")}, + {"log", FuncRet(type::f,"")}, {"exp2", FuncRet(type::f, "")}, {"log2", FuncRet(type::f, "")}, {"exp10", FuncRet(type::f, "")}, + {"log10", FuncRet(type::f, "")}, {"sqrt", FuncRet(type::f, "")}, {"cbrt", FuncRet(type::f, "")}, {"erf", FuncRet(type::f, "")}, + {"erfc", FuncRet(type::f, "")}, {"lgamma", FuncRet(type::f, "")}, {"tgamma", FuncRet(type::f, "")}, {"sin", FuncRet(type::f, "")}, + {"cos", FuncRet(type::f, "")}, {"tan", FuncRet(type::f, "")}, {"asin", FuncRet(type::f, "")}, {"acos", FuncRet(type::f, "")}, + {"atan", FuncRet(type::f, "")}, {"pow", FuncRet(type::f, "")}, {"splitbystring", FuncRet(type::s | type::a,"")}, + {"splitbychar", FuncRet(type::s | type::a, "")}, {"alphatokens", FuncRet(type::s | type::a, "")}, {"toyear", FuncRet(type::i, "")}, + {"tomonth", FuncRet(type::i, "")}, {"todayofmonth", FuncRet(type::i, "")}, {"tohour", FuncRet(type::dt, "")}, {"tominute", FuncRet(type::dt, "")}, + {"toseconds", FuncRet(type::dt, "")}, {"tounixtimestamp", FuncRet(type::i, "")}, {"tostartofyear", FuncRet(type::dt | type::d, "")}, + {"tostartofquater",FuncRet(type::dt | type::d, "")}, {"tostartofmonth", FuncRet(type::dt | type::d, "")}, {"tomonday", FuncRet(type::dt | type::d, "")}, + {"tostartoffiveminutes", FuncRet(type::dt, "")}, {"tostartoftenminutes", FuncRet(type::dt, "")}, {"tostartoffifteenminutes", FuncRet(type::dt, "")}, + {"tostartofinterval", FuncRet(type::dt, "")}, {"totime", FuncRet(type::dt, "")}, {"torelativemonthnum", FuncRet(type::i, "")}, + {"torelativeweeknum", FuncRet(type::i, "")}, {"torelativedaynum", FuncRet(type::i, "")}, {"torelativehournum", FuncRet(type::i, "")}, + {"torelativeminutenum", FuncRet(type::i, "")}, {"torelativesecondsnum", FuncRet(type::i, "")}, {"datediff", FuncRet(type::d | type::dt, "")}, + {"formatdatetime", FuncRet(type::s, "")}, {"now", FuncRet(type::dt | type::d, "now()")}, {"today", FuncRet(type::d | type::dt, "today()")}, + {"yesterday", FuncRet(type::d | type::dt, "yesterday()")} +}; + +std::set func_args_same_types = { + "equals", "notequals", "less", "greater", "lessorequals", "greaterorequals", "multiply" +}; + +std::map func_to_param_type = { + {"tostartofminute", type::dt}, {"plus", type::i | type::f | type::d | type::dt}, {"multiply", type::i | type::f}, + {"minus", type::i | type::f | type::d | type::dt}, {"negate", type::i | type::f}, {"divide", type::i | type::f}, + {"abs", type::i | type::f}, {"gcd", type::i | type::f}, {"lcm", type::i | type::f}, {"bitnot", type::i}, {"bitshiftleft", type::i}, + {"bitshiftright", type::i}, {"bittest", type::i}, {"exp", type::i | type::f}, {"log", type::i | type::f}, + {"exp2", type::i | type::f}, {"log2", type::i | type::f}, {"exp10", type::i | type::f}, {"log10", type::i | type::f}, + {"sqrt", type::i | type::f}, {"cbrt", type::i | type::f}, {"erf", type::i | type::f}, {"erfc", type::i | type::f}, + {"lgamma", type::i | type::f}, {"tgamma", type::i | type::f}, {"sin", type::i | type::f}, {"cos", type::i | type::f}, + {"tan", type::i | type::f}, {"asin", type::i | type::f}, {"acos", type::i | type::f}, {"atan", type::i | type::f}, + {"pow", type::i | type::f}, {"arrayjoin", type::all | type::a}, {"substring", type::s}, {"splitbystring", type::s}, {"splitbychar", type::s}, + {"alphatokens", type::s}, {"toyear", type::d | type::dt}, {"tomonth", type::d | type::dt}, {"todayofmonth", type::d | type::dt}, {"tohour", type::dt}, + {"tominute", type::dt}, {"tosecond", type::dt}, {"touixtimestamp", type::dt}, {"tostartofyear", type::d | type::dt}, + {"tostartofquarter", type::d | type::dt}, {"tostartofmonth", type::d | type::dt}, {"tomonday", type::d | type::dt}, + {"tostartoffiveminute", type::dt}, {"tostartoftenminutes", type::dt}, {"tostartoffifteenminutes", type::d | type::dt}, + {"tostartofinterval", type::d | type::dt}, {"totime", type::d | type::dt}, {"torelativehonthnum", type::d | type::dt}, + {"torelativeweeknum", type::d | type::dt}, {"torelativedaynum", type::d | type::dt}, {"torelativehournum", type::d | type::dt}, + {"torelativeminutenum", type::d | type::dt}, {"torelativesecondnum", type::d | type::dt}, {"datediff", type::d | type::dt}, + {"formatdatetime", type::dt} +}; + + +class Column +{ +public: + TableAndColumn name; + std::set equals; + std::set values; + ColumnType type = type::all; + bool is_array = false; + + Column() = default; + + explicit Column(const std::string & column_name) + { + name = std::make_pair("", column_name); + type = type::all; + } + + void merge(Column other) + { + if (name.second.empty()) + name = other.name; + equals.insert(other.equals.begin(), other.equals.end()); + values.insert(other.values.begin(), other.values.end()); + type &= other.type; + is_array |= other.is_array; + } + + void printType() const + { + if (type & type::i) + std::cout << "I"; + if (type & type::f) + std::cout << "F"; + if (type & type::s) + std::cout << "S"; + if (type & type::d) + std::cout << "D"; + if (type & type::dt) + std::cout << "DT"; + if (is_array) + std::cout << "ARR"; + std::cout << "\n"; + } + + void print() + { + std::cout << name.first << "." << name.second << "\n"; + std::cout << "type: "; + printType(); + std::cout << "values:"; + for (const auto & val : values) + std::cout << " " << val; + std::cout << "\n"; + std::cout << "equal:"; + for (const auto & col : equals) + std::cout << " " << col.first << "." << col.second; + std::cout << "\n"; + } + + std::string generateOneValue() const + { + if (type & type::i) + return randomInteger(); + + if (type & type::f) + return randomFloat(); + + if (type & type::d) + return randomDate(); + + if (type & type::dt) + return randomDatetime(); + + if (type & type::s) + return "'" + randomString(rng() % 40) + "'"; + + if (type & type::b) + return "0"; + + return ""; + } + + bool generateValues(int amount = 0) + { + if (values.size() > 2 && amount == 0) + return false; + while (values.size() < 1 or amount > 0) + { + amount -= 1; + if (is_array) + { + std::string v = "["; + for (unsigned int i = 0; i < static_cast(rng()) % 10 + 1; ++i) + { + if (i != 0) + v += ", "; + v += generateOneValue(); + } + v += "]"; + values.insert(v); + } + else + { + values.insert(generateOneValue()); + } + } + return true; + } + + void unifyType() + { + if (type & type::i) + type = type::i; + else if (type & type::f) + type = type::f; + else if (type & type::d) + type = type::d; + else if (type & type::dt) + type = type::dt; + else if (type & type::s) + type = type::s; + else if (type & type::b) + type = type::b; + else + throw std::runtime_error("Error in determination column type " + name.first + '.' + name.second); + } +}; + + +std::set> +decartMul( + std::set> & prev, + std::set & mul) +{ + std::set> result; + for (auto v : prev) + for (auto m : mul) + { + std::vector tmp = v; + tmp.push_back(m); + result.insert(tmp); + } + return result; +} + + +class Table +{ +public: + Table() = default; + + explicit Table(std::string table_name) + : name(table_name) {} + + std::string name; + std::set columns; + std::map column_description; + + bool columnExists(const std::string & column_name) const + { + return columns.count(column_name); // || columns_maybe.count(column_name); + } + + void addColumn(const std::string & column_name) + { + columns.insert(column_name); + } + + void setDescription(Column other) + { + column_description[other.name.second].merge(other); + } + + void print() + { + std::cout << "Table\n"; + std::cout << name << "\n"; + std::cout << "Columns:\n\n"; + for (const auto & column : columns) + { + std::cout << column << "\n"; + if (column_description.count(column)) + column_description[column].print(); + std::cout << "\n"; + } + std::cout << "\n"; + } + + void merge(Table other) + { + name = other.name; + columns.insert(other.columns.begin(), other.columns.end()); + for (auto desc : other.column_description) + column_description[desc.first].merge(desc.second); + } + + std::string createQuery() + { + std::string create; + std::string db, _; + std::tie(db, _) = get_table_a_column(name); + create = "CREATE DATABASE IF NOT EXISTS " + db + ";\n\n"; + create += "CREATE TABLE IF NOT EXISTS " + name + " (\n"; + for (auto column = columns.begin(); column != columns.end(); ++column) + { + if (column != columns.begin()) + create += ", \n"; + create += *column + " "; + create += column_description[*column].is_array ? "Array(" : ""; + create += type_definition[column_description[*column].type]; + create += column_description[*column].is_array ? ")" : ""; + } + create += "\n) ENGINE = Log;\n\n"; + return create; + } + + std::string insertQuery() + { + std::string insert = "INSERT INTO " + name + "\n"; + insert += "("; + std::set> values = {std::vector(0)}; + for (auto column = columns.begin(); column != columns.end(); ++column) + { + if (column != columns.begin()) + insert += ", "; + insert += *column; + values = decartMul(values, column_description[*column].values); + } + insert += ") VALUES \n"; + for (auto val_set_iter = values.begin(); val_set_iter != values.end(); + ++val_set_iter) + { + if (val_set_iter != values.begin()) + insert += ",\n"; + auto val_set = *val_set_iter; + insert += "("; + for (auto val = val_set.begin(); val != val_set.end(); ++val) + { + if (val != val_set.begin()) + insert += ", "; + insert += *val; + } + insert += ")"; + } + insert += ";\n\n"; + return insert; + } +}; + + +class TableList +{ +public: + std::string main_table; + std::map aliases; + std::unordered_map tables; + std::set nested; + + bool tableExists(const std::string & table_name) const + { + return tables.count(table_name); + } + + void addColumn(std::string full_column) + { + std::string table, column; + std::tie(table, column) = get_table_a_column(full_column); + if (!table.empty()) + { + if (tables.count(table)) + { + tables[table].addColumn(column); + return; + } + if (aliases.count(table)) + { + tables[aliases[table]].addColumn(column); + return; + } + nested.insert(table); + } + tables[main_table].addColumn(full_column); + } + + void addTable(std::string table_name) + { + if (tables.count(table_name)) + return; + + tables[table_name] = Table(table_name); + if (main_table.empty()) + main_table = table_name; + } + + void addDescription(const Column & description) + { + std::string table = description.name.first; + if (tables.count(table)) + tables[table].setDescription(description); + } + + TableAndColumn getTable(std::string full_column) const + { + std::string table, column; + std::tie(table, column) = get_table_a_column(full_column); + if (!table.empty()) + { + if (tables.count(table)) + return std::make_pair(table, column); + + if (aliases.count(table)) + { + table = aliases.find(table)->second; + return std::make_pair(table, column); + } + } + return std::make_pair(main_table, full_column); + } + + void print() + { + for (auto & table : tables) + { + table.second.print(); + std::cout << "\n"; + } + } + + void merge(TableList other) + { + for (auto table : other.tables) + tables[table.first].merge(table.second); + nested.insert(other.nested.begin(), other.nested.end()); + if (main_table.empty()) + main_table = other.main_table; + } +}; + +std::string getAlias(DB::ASTPtr ch) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + return x->alias; + + for (const auto & child : (*ch).children) + { + auto alias = getAlias(child); + if (!alias.empty()) + return alias; + } + return ""; +} + +using FuncHandler = std::function &)>; +std::map handlers = {}; + +FuncRet arrayJoinFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + std::set indents = {}; + for (auto & arg : x->arguments->children) + { + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + indents.insert(ident->name); + } + for (const auto & indent : indents) + { + auto c = Column(indent); + c.type = type::all; + c.is_array = true; + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + FuncRet r(type::all, ""); + return r; + } + return FuncRet(); +} + +FuncRet inFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + std::set indents{}; + std::set values{}; + ColumnType type_value = type::all; + + for (auto & arg : x->arguments->children) + { + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + { + indents.insert(ident->name); + } + auto literal = std::dynamic_pointer_cast(arg); + if (literal) + { + ColumnType type = type_cast(literal->value.getType()); + + /// C++20 + auto routine = [&] (const T & arr_values) + { + for (auto val : arr_values) + { + type = type_cast(val.getType()); + if (type == type::s || type == type::d || type == type::dt) + type = time_type(applyVisitor(DB::FieldVisitorToString(), val)); + type_value &= type; + values.insert(applyVisitor(DB::FieldVisitorToString(), val)); + } + }; + + if (type & type::a) + { + auto arr_values = literal->value.get(); + routine(arr_values); + } + + if (type & type::a) + { + auto arr_values = literal->value.get(); + routine(arr_values); + } + } + auto subfunc = std::dynamic_pointer_cast(arg); + if (subfunc) + { + FuncHandler f; + auto arg_func_name = std::dynamic_pointer_cast(arg)->name; + if (handlers.count(arg_func_name)) + f = handlers[arg_func_name]; + else + f = handlers[""]; + FuncRet ret = f(arg, columns); + if (ret.value != "") + { + values.insert(ret.value); + } + type_value &= ret.type; + } + } + for (const auto & indent : indents) + { + auto c = Column(indent); + c.type = type_value; + c.values.insert(values.begin(), values.end()); + c.generateValues(1); + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + FuncRet r(type::b | type::i, ""); + return r; + } + return FuncRet(); +} + +FuncRet arrayFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + std::set indents = {}; + std::string value = "["; + ColumnType type_value = type::i | type::f | type::d | type::dt | type::s; + bool no_indent = true; + for (const auto & arg : x->arguments->children) + { + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + { + no_indent = false; + indents.insert(ident->name); + } + auto literal = std::dynamic_pointer_cast(arg); + if (literal) + { + ColumnType type = type_cast(literal->value.getType()); + if (type == type::s || type == type::d || type == type::dt) + type = time_type(value); + type_value &= type; + + if (value != "[") + value += ", "; + value += applyVisitor(DB::FieldVisitorToString(), literal->value); + } + } + for (const auto & indent : indents) + { + auto c = Column(indent); + c.type = type_value; + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + value += ']'; + FuncRet r(type_value, ""); + r.is_array = true; + if (no_indent) + r.value = value; + return r; + } + return FuncRet(); +} +FuncRet arithmeticFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + std::set indents = {}; + std::set values = {}; + ColumnType type_value = type::i | type::f | type::d | type::dt; + ColumnType args_types = 0; + bool no_indent = true; + for (auto & arg : x->arguments->children) + { + ColumnType type = 0; + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + { + no_indent = false; + indents.insert(ident->name); + } + auto literal = std::dynamic_pointer_cast(arg); + if (literal) + type = type_cast(literal->value.getType()); + auto subfunc = std::dynamic_pointer_cast(arg); + if (subfunc) + { + FuncHandler f; + auto arg_func_name = std::dynamic_pointer_cast(arg)->name; + if (handlers.count(arg_func_name)) + f = handlers[arg_func_name]; + else + f = handlers[""]; + FuncRet ret = f(arg, columns); + type = ret.type; + } + args_types |= type; + } + if (args_types & (type::d | type::dt)) + type_value -= type::f; + if (args_types & type::f) + type_value -= type::d | type::dt; + for (auto indent : indents) + { + auto c = Column(indent); + c.type = type_value; + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + ColumnType ret_type = 0; + if (args_types & type::dt) + ret_type = type::dt; + else if (args_types & type::d) + ret_type = type::d | type::dt; + else if (args_types & type::f) + ret_type = type::f; + else + ret_type = type::d | type::f | type::dt | type::i; + FuncRet r(ret_type, ""); + if (no_indent) + { + std::ostringstream ss; + formatAST(*ch, ss); + r.value = ss.str(); + } + return r; + } + return FuncRet(); +} +FuncRet likeFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + std::set indents = {}; + std::set values = {}; + ColumnType type_value = type::s; + for (auto & arg : x->arguments->children) + { + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + indents.insert(ident->name); + auto literal = std::dynamic_pointer_cast(arg); + if (literal) + { + std::string value = applyVisitor(DB::FieldVisitorToString(), literal->value); + std::string example{}; + for (size_t i = 0; i != value.size(); ++i) + { + if (value[i] == '%') + example += randomString(rng() % 10); + else if (value[i] == '_') + example += randomString(1); + else + example += value[i]; + } + values.insert(example); + } + } + for (const auto & indent : indents) + { + auto c = Column(indent); + c.type = type_value; + c.values.insert(values.begin(), values.end()); + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + FuncRet r(type::b, ""); + return r; + } + return FuncRet(); +} + +FuncRet simpleFunc(DB::ASTPtr ch, std::map & columns) +{ + auto X = std::dynamic_pointer_cast(ch); + if (X) + { + std::set indents = {}; + std::set values = {}; + ColumnType type_value = type::all; + bool is_array = false; + bool no_indent = true; + if (func_to_param_type.count(boost::algorithm::to_lower_copy(X->name))) + { + type_value &= func_to_param_type[boost::algorithm::to_lower_copy(X->name)]; + is_array = func_to_param_type[boost::algorithm::to_lower_copy(X->name)] & type::a; + } + for (auto arg : X->arguments->children) + { + ColumnType type = type::all; + std::string value; + auto ident = std::dynamic_pointer_cast(arg); + if (ident) + { + no_indent = false; + indents.insert(ident->name); + } + auto literal = std::dynamic_pointer_cast(arg); + if (literal) + { + value = applyVisitor(DB::FieldVisitorToString(), literal->value); + type = type_cast(literal->value.getType()); + is_array |= type & type::a; + } + auto subfunc = std::dynamic_pointer_cast(arg); + if (subfunc) + { + FuncHandler f; + auto arg_func_name = std::dynamic_pointer_cast(arg)->name; + if (handlers.count(arg_func_name)) + f = handlers[arg_func_name]; + else + f = handlers[""]; + FuncRet ret = f(arg, columns); + is_array |= ret.is_array; + type = ret.type; + value = ret.value; + if (value.empty()) + no_indent = false; + } + if (!value.empty()) + { + if (type == type::i) + { + values.insert(value); + values.insert(value + " + " + randomInteger(1, 10)); + values.insert(value + " - " + randomInteger(1, 10)); + } + if (type == type::f) + { + values.insert(value); + values.insert(value + " + " + randomFloat(1, 10)); + values.insert(value + " - " + randomFloat(1, 10)); + } + if (type & type::s || type & type::d || type & type::dt) + { + if (type == type::s) + type = time_type(value); + if (type == type::s) + values.insert(value); + if (type & type::d) + { + values.insert(value); + values.insert("toDate(" + value + ") + " + randomInteger(1, 10)); + values.insert("toDate(" + value + ") - " + randomInteger(1, 10)); + } + else if (type & type::dt) + { + values.insert(value); + values.insert( + "toDateTime(" + value + ") + " + randomInteger(1, 10000)); + values.insert( + "toDateTime(" + value + ") - " + randomInteger(1, 10000)); + } + } + } + if (func_args_same_types.count(boost::algorithm::to_lower_copy(X->name))) + type_value &= type; + } + for (const auto & indent : indents) + { + auto c = Column(indent); + c.type = type_value; + c.is_array = is_array; + if (func_args_same_types.count( + boost::algorithm::to_lower_copy(X->name))) + c.values = values; + for (const auto & ind : indents) + if (ind != indent) + c.equals.insert(std::make_pair("", ind)); + + if (columns.count(indent)) + columns[indent].merge(c); + else + columns[indent] = c; + } + if (func_to_return_type.count(boost::algorithm::to_lower_copy(X->name))) + { + if (no_indent) + { + std::ostringstream ss; + formatAST(*ch, ss); + auto r = func_to_return_type[boost::algorithm::to_lower_copy(X->name)]; + r.value = ss.str(); + return r; + } + return func_to_return_type[boost::algorithm::to_lower_copy(X->name)]; + } + else if (func_to_param_type.count( + boost::algorithm::to_lower_copy(X->name))) + { + if (no_indent) + { + std::ostringstream ss; + formatAST(*ch, ss); + return FuncRet( + func_to_param_type[boost::algorithm::to_lower_copy(X->name)], + ss.str()); + } + return FuncRet( + func_to_param_type[boost::algorithm::to_lower_copy(X->name)], + ""); + } + } + return FuncRet(); +} + +void processFunc(DB::ASTPtr ch, std::map & columns) +{ + auto x = std::dynamic_pointer_cast(ch); + if (x) + { + FuncHandler f; + auto arg_func_name = x->name; + if (handlers.count(arg_func_name)) + f = handlers[arg_func_name]; + else + f = handlers[""]; + f(ch, columns); + } + else + { + for (const auto & child : (*ch).children) + processFunc(child, columns); + } +} + + +std::set getIndent(DB::ASTPtr ch) +{ + if (!ch) + return {}; + + std::set ret = {}; + auto x = std::dynamic_pointer_cast(ch); + if (x) + ret.insert(x->name); + for (const auto & child : (*ch).children) + { + auto child_ind = getIndent(child); + ret.insert(child_ind.begin(), child_ind.end()); + } + return ret; +} + + +std::set getSelectIndent( + DB::ASTPtr asp, + std::set & column_alias) +{ + std::set ret = {}; + for (auto & ch : asp->children) + { + auto alias = getAlias(ch); + auto columns = getIndent(ch); + if (alias.empty()) + column_alias.insert(alias); + ret.insert(columns.begin(), columns.end()); + } + return ret; +} + + +std::set +connectedEqualityFind( + const Column & now, + std::map & columns_descriptions, + std::set & visited) +{ + std::set result; + for (auto & column : now.equals) + if (!visited.count(column)) + { + visited.insert(column); + auto sub_r = connectedEqualityFind( + columns_descriptions[column.first + "." + column.second], + columns_descriptions, + visited); + result.insert(sub_r.begin(), sub_r.end()); + } + result.insert(now.name); + return result; +} + + +std::map +unificateColumns( + std::map columns_descriptions, + const TableList & all_tables) +{ + for (auto & column : columns_descriptions) + { + std::set changed_equals; + for (const auto & eq : column.second.equals) + { + std::string t, c; + std::tie(t, c) = all_tables.getTable(eq.second); + changed_equals.insert(std::make_pair(t, c)); + } + column.second.equals = changed_equals; + } + std::map result; + for (auto & column : columns_descriptions) + { + std::string t, c; + std::tie(t, c) = all_tables.getTable(column.first); + column.second.name = std::make_pair(t, c); + result[t + "." + c].merge(column.second); + } + std::set visited; + for (auto & column : result) + if (!visited.count(column.second.name)) + { + auto equal = connectedEqualityFind( + result[column.second.name.first + "." + column.second.name.second], + result, + visited); + for (auto c : equal) + result[c.first + "." + c.second].equals = equal; + } + for (auto & column : result) + for (auto e : column.second.equals) + column.second.merge(result[e.first + "." + e.second]); + + for (auto & column : result) + { + column.second.unifyType(); + if (column.second.generateValues()) + for (auto e : column.second.equals) + result[e.first + "." + e.second].merge(column.second); + + } + return result; +} + +std::vector getSelect(DB::ASTPtr vertex) +{ + auto z = std::dynamic_pointer_cast(vertex); + std::vector result; + if (z) + { + result.push_back(vertex); + return result; + } + + for (const auto & child : (*vertex).children) + { + auto v = getSelect(child); + result.insert(result.end(), v.begin(), v.end()); + } + return result; +} + + +void parseSelectQuery(DB::ASTPtr ast, TableList & all_tables) +{ + if (!ast) + throw std::runtime_error("Bad ASTPtr in parseSelectQuery" + StackTrace().toString()); + + auto select_ast = std::dynamic_pointer_cast(ast); + if (!select_ast) + { + std::cerr << "not select query"; + return; + } + std::set columns = {}; + + auto x = select_ast->tables(); + if (!x) + throw std::runtime_error("There is no tables in query. Nothing to generate."); + + for (auto & child : x->children) + { + auto ch = std::dynamic_pointer_cast(child); + auto TEast = std::dynamic_pointer_cast(ch->table_expression); + if (TEast && TEast->database_and_table_name) + { + auto table_name = *(getIndent(TEast->database_and_table_name).begin()); + all_tables.addTable(table_name); + auto alias = getAlias(ch); + if (!alias.empty()) + all_tables.aliases[alias] = table_name; + } + if (TEast && TEast->subquery) + { + for (auto select : getSelect(TEast->subquery)) + { + TableList local; + parseSelectQuery(select, local); + all_tables.merge(local); + } + } + + if (ch->table_join) + { + auto jch = std::dynamic_pointer_cast(ch->table_join); + if (jch->using_expression_list) + { + auto join_columns = getIndent(jch->using_expression_list); + columns.insert(join_columns.begin(), join_columns.end()); + } + else if (jch->on_expression) + { + auto join_columns = getIndent(jch->on_expression); + columns.insert(join_columns.begin(), join_columns.end()); + } + } + } + + std::set column_aliases; + auto select_columns = getSelectIndent(select_ast->select(), column_aliases); + columns.insert(select_columns.begin(), select_columns.end()); + + auto where_columns = getIndent(select_ast->where()); + columns.insert(where_columns.begin(), where_columns.end()); + + auto groupby_columns = getIndent(select_ast->groupBy()); + columns.insert(groupby_columns.begin(), groupby_columns.end()); + + auto orderby_columns = getIndent(select_ast->orderBy()); + columns.insert(orderby_columns.begin(), orderby_columns.end()); + + auto having_columns = getIndent(select_ast->having()); + columns.insert(having_columns.begin(), having_columns.end()); + + std::map columns_descriptions; + processFunc(ast, columns_descriptions); + + for (const auto & column : columns) + if (!column_aliases.count(column)) + { + if (!columns_descriptions.count(column)) + columns_descriptions[column] = Column(column); + all_tables.addColumn(column); + } + + columns_descriptions = unificateColumns(columns_descriptions, all_tables); + for (auto & column : columns_descriptions) + all_tables.addDescription(column.second); +} + + +TableList getTablesFromSelect(std::vector queries) +{ + DB::ParserQueryWithOutput parser; + TableList result; + for (std::string & query : queries) + { + DB::ASTPtr ast = parseQuery(parser, query.data(), query.data() + query.size(), "", 0, 0); + for (auto & select : getSelect(ast)) + { + TableList local; + parseSelectQuery(select, local); + result.merge(local); + } + } + return result; +} + +int main(int, char **) +{ + handlers["plus"] = arithmeticFunc; + handlers["minus"] = arithmeticFunc; + handlers["like"] = likeFunc; + handlers["array"] = arrayFunc; + handlers["in"] = inFunc; + handlers[""] = simpleFunc; + + std::vector queries; + std::string in; + std::string query{}; + while (getline(std::cin, in)) + { + /// Skip comments + if (in.find("--") != std::string::npos) + continue; + + query += in + " "; + + if (in.find(';') != std::string::npos) + { + queries.push_back(query); + query = ""; + } + } + + try + { + auto result = getTablesFromSelect(queries); + + for (auto & table : result.tables) + { + std::cout << table.second.createQuery(); + std::cout << table.second.insertQuery(); + } + + for (auto & q: queries) + std::cout << q << std::endl; + } + catch (std::string & e) + { + std::cerr << "Exception: " << e << std::endl; + } +} diff --git a/utils/github/backport.py b/utils/github/backport.py index 36f40d08623..3a33c3a563f 100644 --- a/utils/github/backport.py +++ b/utils/github/backport.py @@ -20,7 +20,7 @@ class Backport: def getPullRequests(self, from_commit): return self._gh.get_pull_requests(from_commit) - def execute(self, repo, til, number, run_cherrypick): + def execute(self, repo, until_commit, number, run_cherrypick): repo = LocalRepo(repo, 'origin', self.default_branch_name) branches = repo.get_release_branches()[-number:] # [(branch_name, base_commit)] @@ -31,9 +31,9 @@ class Backport: for branch in branches: logging.info('Found release branch: %s', branch[0]) - if not til: - til = branches[0][1] - prs = self.getPullRequests(til) + if not until_commit: + until_commit = branches[0][1] + pull_requests = self.getPullRequests(until_commit) backport_map = {} @@ -42,7 +42,7 @@ class Backport: RE_BACKPORTED = re.compile(r'^v(\d+\.\d+)-backported$') # pull-requests are sorted by ancestry from the least recent. - for pr in prs: + for pr in pull_requests: while repo.comparator(branches[-1][1]) >= repo.comparator(pr['mergeCommit']['oid']): logging.info("PR #{} is already inside {}. Dropping this branch for further PRs".format(pr['number'], branches[-1][0])) branches.pop() @@ -55,28 +55,28 @@ class Backport: # First pass. Find all must-backports for label in pr['labels']['nodes']: - if label['name'].startswith('pr-') and label['color'] == 'ff0000': + if label['name'] == 'pr-bugfix': backport_map[pr['number']] = branch_set.copy() continue - m = RE_MUST_BACKPORT.match(label['name']) - if m: + matched = RE_MUST_BACKPORT.match(label['name']) + if matched: if pr['number'] not in backport_map: backport_map[pr['number']] = set() - backport_map[pr['number']].add(m.group(1)) + backport_map[pr['number']].add(matched.group(1)) # Second pass. Find all no-backports for label in pr['labels']['nodes']: if label['name'] == 'pr-no-backport' and pr['number'] in backport_map: del backport_map[pr['number']] break - m1 = RE_NO_BACKPORT.match(label['name']) - m2 = RE_BACKPORTED.match(label['name']) - if m1 and pr['number'] in backport_map and m1.group(1) in backport_map[pr['number']]: - backport_map[pr['number']].remove(m1.group(1)) - logging.info('\tskipping %s because of forced no-backport', m1.group(1)) - elif m2 and pr['number'] in backport_map and m2.group(1) in backport_map[pr['number']]: - backport_map[pr['number']].remove(m2.group(1)) - logging.info('\tskipping %s because it\'s already backported manually', m2.group(1)) + matched_no_backport = RE_NO_BACKPORT.match(label['name']) + matched_backported = RE_BACKPORTED.match(label['name']) + if matched_no_backport and pr['number'] in backport_map and matched_no_backport.group(1) in backport_map[pr['number']]: + backport_map[pr['number']].remove(matched_no_backport.group(1)) + logging.info('\tskipping %s because of forced no-backport', matched_no_backport.group(1)) + elif matched_backported and pr['number'] in backport_map and matched_backported.group(1) in backport_map[pr['number']]: + backport_map[pr['number']].remove(matched_backported.group(1)) + logging.info('\tskipping %s because it\'s already backported manually', matched_backported.group(1)) for pr, branches in backport_map.items(): logging.info('PR #%s needs to be backported to:', pr) diff --git a/utils/list-versions/version_date.tsv b/utils/list-versions/version_date.tsv index 461d591b3c3..1b24b0349c2 100644 --- a/utils/list-versions/version_date.tsv +++ b/utils/list-versions/version_date.tsv @@ -1,4 +1,5 @@ v20.7.2.30-stable 2020-08-31 +v20.6.5.8-stable 2020-09-03 v20.6.4.44-stable 2020-08-20 v20.6.3.28-stable 2020-08-07 v20.5.5.74-stable 2020-08-20 diff --git a/website/README.md b/website/README.md index 5652cbcc6a1..c8881ad5eb9 100644 --- a/website/README.md +++ b/website/README.md @@ -1,2 +1,12 @@ ClickHouse website is built alongside it's documentation via [docs/tools](https://github.com/ClickHouse/ClickHouse/tree/master/docs/tools), see [README.md there](https://github.com/ClickHouse/ClickHouse/tree/master/docs/tools/README.md). +# How to quickly test the main page of the website + +``` +cd ../docs/tools +sudo apt install python-3 pip +pip3 install -r requirements.txt +./build.py --skip-multi-page --skip-single-page --skip-amp --skip-pdf --skip-blog --skip-git-log --skip-docs --skip-test-templates --livereload 8080 + +# Open the web browser and go to http://localhost:8080/ +``` diff --git a/website/templates/index/quickstart.html b/website/templates/index/quickstart.html index add2b66d981..454fc68151d 100644 --- a/website/templates/index/quickstart.html +++ b/website/templates/index/quickstart.html @@ -20,14 +20,14 @@
-
- {% highlight "bash" %}{% include "install/deb.sh" %}{% endhighlight %} +
+ 
{% include "install/deb.sh" %}
-
- {% highlight "bash" %}{% include "install/rpm.sh" %}{% endhighlight %} +
+ 
{% include "install/rpm.sh" %}
-
- {% highlight "bash" %}{% include "install/tgz.sh" %}{% endhighlight %} +
+ 
{% include "install/tgz.sh" %}
diff --git a/website/templates/index/use.html b/website/templates/index/use.html index 1f345186d71..6fa27782acc 100644 --- a/website/templates/index/use.html +++ b/website/templates/index/use.html @@ -42,8 +42,5 @@
- - -