From c4b9c700c516132471586bff36fcac6f63d5de10 Mon Sep 17 00:00:00 2001 From: Olga Revyakina Date: Tue, 26 Jan 2021 02:09:17 +0300 Subject: [PATCH 01/24] Map type and map function. Data type description template also added. --- .../template-data-type.md | 29 +++++++++ docs/en/sql-reference/data-types/map.md | 56 ++++++++++++++++ .../functions/tuple-map-functions.md | 64 ++++++++++++++++++- 3 files changed, 148 insertions(+), 1 deletion(-) create mode 100644 docs/_description_templates/template-data-type.md create mode 100644 docs/en/sql-reference/data-types/map.md diff --git a/docs/_description_templates/template-data-type.md b/docs/_description_templates/template-data-type.md new file mode 100644 index 00000000000..edb6586ee7d --- /dev/null +++ b/docs/_description_templates/template-data-type.md @@ -0,0 +1,29 @@ +--- +toc_priority: +toc_title: +--- + +# data_type_name {#data_type-name} + +Description. + +**Parameters** (Optional) + +- `x` — Description. [Type name](relative/path/to/type/dscr.md#type). +- `y` — Description. [Type name](relative/path/to/type/dscr.md#type). + +**Examples** + +```sql + +``` + +## Additional Info {#additional-info} (Optional) + +The name of an additional section can be any, for example, **Usage**. + +**See Also** (Optional) + +- [link](#) + +[Original article](https://clickhouse.tech/docs/en/data_types//) diff --git a/docs/en/sql-reference/data-types/map.md b/docs/en/sql-reference/data-types/map.md new file mode 100644 index 00000000000..5f1300896e8 --- /dev/null +++ b/docs/en/sql-reference/data-types/map.md @@ -0,0 +1,56 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +`Map(key, value)` data type stores `key:value` pairs in structures like JSON. + +**Parameters** +- `key` — Key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — Value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +!!! warning "Warning" + Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. + +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. + +**Example** + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Result: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Map() and Tuple() Types {#map-and-tuple} + +You can cast `Tuple()` as `Map()`: + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**See Also** + +- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function + +[Original article](https://clickhouse.tech/docs/en/data_types/map/) diff --git a/docs/en/sql-reference/functions/tuple-map-functions.md b/docs/en/sql-reference/functions/tuple-map-functions.md index a46c36395b8..3de570e6dcc 100644 --- a/docs/en/sql-reference/functions/tuple-map-functions.md +++ b/docs/en/sql-reference/functions/tuple-map-functions.md @@ -5,6 +5,68 @@ toc_title: Working with maps # Functions for maps {#functions-for-working-with-tuple-maps} +## map {#function-map} + +Arranges `key:value` pairs into a JSON data structure. + +**Syntax** + +``` sql +map(key1, value1[, key2, value2, ...]) +``` + +**Parameters** + +- `key` — Key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — Value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +**Returned value** + +- JSON with `key:value` pairs. + +Type: [Map(key, value)](../../sql-reference/data-types/map.md). + +**Examples** + +Query: + +``` sql +SELECT map('key1', number, 'key2', number * 2) FROM numbers(3); +``` + +Result: + +``` text +┌─map('key1', number, 'key2', multiply(number, 2))─┐ +│ {'key1':0,'key2':0} │ +│ {'key1':1,'key2':2} │ +│ {'key1':2,'key2':4} │ +└──────────────────────────────────────────────────┘ +``` + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE = MergeTree() ORDER BY a; +INSERT INTO table_map SELECT map('key1', number, 'key2', number * 2) FROM numbers(3); +SELECT a['key2'] FROM table_map; +``` + +Result: + +``` text +┌─arrayElement(a, 'key2')─┐ +│ 0 │ +│ 2 │ +│ 4 │ +└─────────────────────────┘ +``` + +**See Also** + +- [Map(key, value)](../../sql-reference/data-types/map.md) data type + + ## mapAdd {#function-mapadd} Collect all the keys and sum corresponding values. @@ -112,4 +174,4 @@ Result: └──────────────────────────────┴───────────────────────────────────┘ ``` -[Original article](https://clickhouse.tech/docs/en/query_language/functions/tuple-map-functions/) +[Original article](https://clickhouse.tech/docs/en/sql-reference/functions/tuple-map-functions/) From 297d106f1446790a1a065e2b0ccc416eda658bb8 Mon Sep 17 00:00:00 2001 From: Olga Revyakina Date: Tue, 2 Feb 2021 03:24:01 +0300 Subject: [PATCH 02/24] Fixed, translated. Symbolic links added. --- docs/en/sql-reference/data-types/map.md | 11 ++-- .../functions/tuple-map-functions.md | 4 +- docs/es/sql-reference/data-types/map.md | 57 ++++++++++++++++++ docs/fr/sql-reference/data-types/map.md | 57 ++++++++++++++++++ docs/ja/sql-reference/data-types/map.md | 57 ++++++++++++++++++ docs/ru/sql-reference/data-types/map.md | 57 ++++++++++++++++++ .../functions/tuple-map-functions.md | 60 +++++++++++++++++++ docs/zh/sql-reference/data-types/map.md | 57 ++++++++++++++++++ 8 files changed, 353 insertions(+), 7 deletions(-) create mode 100644 docs/es/sql-reference/data-types/map.md create mode 100644 docs/fr/sql-reference/data-types/map.md create mode 100644 docs/ja/sql-reference/data-types/map.md create mode 100644 docs/ru/sql-reference/data-types/map.md create mode 100644 docs/zh/sql-reference/data-types/map.md diff --git a/docs/en/sql-reference/data-types/map.md b/docs/en/sql-reference/data-types/map.md index 5f1300896e8..0f0f69d421d 100644 --- a/docs/en/sql-reference/data-types/map.md +++ b/docs/en/sql-reference/data-types/map.md @@ -8,8 +8,8 @@ toc_title: Map(key, value) `Map(key, value)` data type stores `key:value` pairs in structures like JSON. **Parameters** -- `key` — Key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — Value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). !!! warning "Warning" Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. @@ -35,9 +35,9 @@ Result: └─────────────────────────┘ ``` -## Map() and Tuple() Types {#map-and-tuple} +## Convert Tuple to Map Type {#map-and-tuple} -You can cast `Tuple()` as `Map()`: +You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: ``` sql SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; @@ -52,5 +52,6 @@ SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map **See Also** - [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function +- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function -[Original article](https://clickhouse.tech/docs/en/data_types/map/) +[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/en/sql-reference/functions/tuple-map-functions.md b/docs/en/sql-reference/functions/tuple-map-functions.md index 3de570e6dcc..b81f971196a 100644 --- a/docs/en/sql-reference/functions/tuple-map-functions.md +++ b/docs/en/sql-reference/functions/tuple-map-functions.md @@ -17,8 +17,8 @@ map(key1, value1[, key2, value2, ...]) **Parameters** -- `key` — Key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — Value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). **Returned value** diff --git a/docs/es/sql-reference/data-types/map.md b/docs/es/sql-reference/data-types/map.md new file mode 100644 index 00000000000..0f0f69d421d --- /dev/null +++ b/docs/es/sql-reference/data-types/map.md @@ -0,0 +1,57 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +`Map(key, value)` data type stores `key:value` pairs in structures like JSON. + +**Parameters** +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +!!! warning "Warning" + Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. + +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. + +**Example** + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Result: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Convert Tuple to Map Type {#map-and-tuple} + +You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**See Also** + +- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function +- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function + +[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/fr/sql-reference/data-types/map.md b/docs/fr/sql-reference/data-types/map.md new file mode 100644 index 00000000000..0f0f69d421d --- /dev/null +++ b/docs/fr/sql-reference/data-types/map.md @@ -0,0 +1,57 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +`Map(key, value)` data type stores `key:value` pairs in structures like JSON. + +**Parameters** +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +!!! warning "Warning" + Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. + +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. + +**Example** + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Result: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Convert Tuple to Map Type {#map-and-tuple} + +You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**See Also** + +- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function +- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function + +[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/ja/sql-reference/data-types/map.md b/docs/ja/sql-reference/data-types/map.md new file mode 100644 index 00000000000..0f0f69d421d --- /dev/null +++ b/docs/ja/sql-reference/data-types/map.md @@ -0,0 +1,57 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +`Map(key, value)` data type stores `key:value` pairs in structures like JSON. + +**Parameters** +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +!!! warning "Warning" + Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. + +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. + +**Example** + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Result: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Convert Tuple to Map Type {#map-and-tuple} + +You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**See Also** + +- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function +- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function + +[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/ru/sql-reference/data-types/map.md b/docs/ru/sql-reference/data-types/map.md new file mode 100644 index 00000000000..c1391e37133 --- /dev/null +++ b/docs/ru/sql-reference/data-types/map.md @@ -0,0 +1,57 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +Тип данных `Map(key, value)` хранит пары `ключ:значение` в структурах типа JSON. + +**Параметры** +- `key` — ключ. [String](../../sql-reference/data-types/string.md) или [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — значение. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) или [Array](../../sql-reference/data-types/array.md). + +!!! warning "Предупреждение" + Сейчас использование типа данных `Map` является экспериментальной возможностью. Чтобы использовать этот тип данных, включите настройку `allow_experimental_map_type = 1`. + +Чтобы получить значение из колонки `a Map('key', 'value')`, используйте синтаксис `a['key']`. + +**Пример** + +Запрос: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Результат: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Преобразование типа данных Tuple в Map {#map-and-tuple} + +Для преобразования данных с типом `Tuple()` в тип `Map()` можно использовать функцию [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast): + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**См. также** + +- функция [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) +- функция [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) + +[Original article](https://clickhouse.tech/docs/ru/data-types/map/) diff --git a/docs/ru/sql-reference/functions/tuple-map-functions.md b/docs/ru/sql-reference/functions/tuple-map-functions.md index a2b25e68fe5..65e44698008 100644 --- a/docs/ru/sql-reference/functions/tuple-map-functions.md +++ b/docs/ru/sql-reference/functions/tuple-map-functions.md @@ -5,6 +5,66 @@ toc_title: Работа с контейнерами map # Функции для работы с контейнерами map {#functions-for-working-with-tuple-maps} +## map {#function-map} + +Преобразовывает пары `ключ:значение` в структуру JSON. + +**Синтаксис** + +``` sql +map(key1, value1[, key2, value2, ...]) +``` + +**Параметры** + +- `key` — ключ. [String](../../sql-reference/data-types/string.md) или [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — значение. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) или [Array](../../sql-reference/data-types/array.md). + +**Возвращаемое значение** + +- Структура JSON с парами `ключ:значение`. + +Тип: [Map(key, value)](../../sql-reference/data-types/map.md). + +**Примеры** + +Запрос: + +``` sql +SELECT map('key1', number, 'key2', number * 2) FROM numbers(3); +``` + +Результат: + +``` text +┌─map('key1', number, 'key2', multiply(number, 2))─┐ +│ {'key1':0,'key2':0} │ +│ {'key1':1,'key2':2} │ +│ {'key1':2,'key2':4} │ +└──────────────────────────────────────────────────┘ +``` + +Запрос: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE = MergeTree() ORDER BY a; +INSERT INTO table_map SELECT map('key1', number, 'key2', number * 2) FROM numbers(3); +SELECT a['key2'] FROM table_map; +``` + +Результат: + +``` text +┌─arrayElement(a, 'key2')─┐ +│ 0 │ +│ 2 │ +│ 4 │ +└─────────────────────────┘ +``` + +**См. также** + +- тип данных [Map(key, value)](../../sql-reference/data-types/map.md) ## mapAdd {#function-mapadd} Собирает все ключи и суммирует соответствующие значения. diff --git a/docs/zh/sql-reference/data-types/map.md b/docs/zh/sql-reference/data-types/map.md new file mode 100644 index 00000000000..0f0f69d421d --- /dev/null +++ b/docs/zh/sql-reference/data-types/map.md @@ -0,0 +1,57 @@ +--- +toc_priority: 65 +toc_title: Map(key, value) +--- + +# Map(key, value) {#data_type-map} + +`Map(key, value)` data type stores `key:value` pairs in structures like JSON. + +**Parameters** +- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). +- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). + +!!! warning "Warning" + Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. + +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. + +**Example** + +Query: + +``` sql +CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; +INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +SELECT a['key2'] FROM table_map; +``` +Result: + +```text +┌─arrayElement(a, 'key2')─┐ +│ 100 │ +│ 200 │ +│ 300 │ +└─────────────────────────┘ +``` + +## Convert Tuple to Map Type {#map-and-tuple} + +You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: + +``` sql +SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +``` + +``` text +┌─map───────────────────────────┐ +│ {1:'Ready',2:'Steady',3:'Go'} │ +└───────────────────────────────┘ +``` + +**See Also** + +- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function +- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function + +[Original article](https://clickhouse.tech/docs/en/data-types/map/) From abeeebc66156b85a690e0bc33f7759f26dcd61da Mon Sep 17 00:00:00 2001 From: Olga Revyakina Date: Tue, 2 Feb 2021 22:01:06 +0300 Subject: [PATCH 03/24] Symbolic links deleted --- docs/es/sql-reference/data-types/map.md | 57 ------------------------- docs/fr/sql-reference/data-types/map.md | 57 ------------------------- docs/ja/sql-reference/data-types/map.md | 57 ------------------------- docs/zh/sql-reference/data-types/map.md | 57 ------------------------- 4 files changed, 228 deletions(-) delete mode 100644 docs/es/sql-reference/data-types/map.md delete mode 100644 docs/fr/sql-reference/data-types/map.md delete mode 100644 docs/ja/sql-reference/data-types/map.md delete mode 100644 docs/zh/sql-reference/data-types/map.md diff --git a/docs/es/sql-reference/data-types/map.md b/docs/es/sql-reference/data-types/map.md deleted file mode 100644 index 0f0f69d421d..00000000000 --- a/docs/es/sql-reference/data-types/map.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -toc_priority: 65 -toc_title: Map(key, value) ---- - -# Map(key, value) {#data_type-map} - -`Map(key, value)` data type stores `key:value` pairs in structures like JSON. - -**Parameters** -- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). - -!!! warning "Warning" - Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. - -To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. - -**Example** - -Query: - -``` sql -CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); -SELECT a['key2'] FROM table_map; -``` -Result: - -```text -┌─arrayElement(a, 'key2')─┐ -│ 100 │ -│ 200 │ -│ 300 │ -└─────────────────────────┘ -``` - -## Convert Tuple to Map Type {#map-and-tuple} - -You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: - -``` sql -SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; -``` - -``` text -┌─map───────────────────────────┐ -│ {1:'Ready',2:'Steady',3:'Go'} │ -└───────────────────────────────┘ -``` - -**See Also** - -- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function -- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function - -[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/fr/sql-reference/data-types/map.md b/docs/fr/sql-reference/data-types/map.md deleted file mode 100644 index 0f0f69d421d..00000000000 --- a/docs/fr/sql-reference/data-types/map.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -toc_priority: 65 -toc_title: Map(key, value) ---- - -# Map(key, value) {#data_type-map} - -`Map(key, value)` data type stores `key:value` pairs in structures like JSON. - -**Parameters** -- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). - -!!! warning "Warning" - Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. - -To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. - -**Example** - -Query: - -``` sql -CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); -SELECT a['key2'] FROM table_map; -``` -Result: - -```text -┌─arrayElement(a, 'key2')─┐ -│ 100 │ -│ 200 │ -│ 300 │ -└─────────────────────────┘ -``` - -## Convert Tuple to Map Type {#map-and-tuple} - -You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: - -``` sql -SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; -``` - -``` text -┌─map───────────────────────────┐ -│ {1:'Ready',2:'Steady',3:'Go'} │ -└───────────────────────────────┘ -``` - -**See Also** - -- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function -- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function - -[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/ja/sql-reference/data-types/map.md b/docs/ja/sql-reference/data-types/map.md deleted file mode 100644 index 0f0f69d421d..00000000000 --- a/docs/ja/sql-reference/data-types/map.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -toc_priority: 65 -toc_title: Map(key, value) ---- - -# Map(key, value) {#data_type-map} - -`Map(key, value)` data type stores `key:value` pairs in structures like JSON. - -**Parameters** -- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). - -!!! warning "Warning" - Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. - -To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. - -**Example** - -Query: - -``` sql -CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); -SELECT a['key2'] FROM table_map; -``` -Result: - -```text -┌─arrayElement(a, 'key2')─┐ -│ 100 │ -│ 200 │ -│ 300 │ -└─────────────────────────┘ -``` - -## Convert Tuple to Map Type {#map-and-tuple} - -You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: - -``` sql -SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; -``` - -``` text -┌─map───────────────────────────┐ -│ {1:'Ready',2:'Steady',3:'Go'} │ -└───────────────────────────────┘ -``` - -**See Also** - -- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function -- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function - -[Original article](https://clickhouse.tech/docs/en/data-types/map/) diff --git a/docs/zh/sql-reference/data-types/map.md b/docs/zh/sql-reference/data-types/map.md deleted file mode 100644 index 0f0f69d421d..00000000000 --- a/docs/zh/sql-reference/data-types/map.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -toc_priority: 65 -toc_title: Map(key, value) ---- - -# Map(key, value) {#data_type-map} - -`Map(key, value)` data type stores `key:value` pairs in structures like JSON. - -**Parameters** -- `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). -- `value` — The value part of the pair. [String](../../sql-reference/data-types/string.md), [Integer](../../sql-reference/data-types/int-uint.md) or [Array](../../sql-reference/data-types/array.md). - -!!! warning "Warning" - Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. - -To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. - -**Example** - -Query: - -``` sql -CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); -SELECT a['key2'] FROM table_map; -``` -Result: - -```text -┌─arrayElement(a, 'key2')─┐ -│ 100 │ -│ 200 │ -│ 300 │ -└─────────────────────────┘ -``` - -## Convert Tuple to Map Type {#map-and-tuple} - -You can cast `Tuple()` as `Map()` using [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function: - -``` sql -SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; -``` - -``` text -┌─map───────────────────────────┐ -│ {1:'Ready',2:'Steady',3:'Go'} │ -└───────────────────────────────┘ -``` - -**See Also** - -- [map()](../../sql-reference/functions/tuple-map-functions.md#function-map) function -- [CAST()](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast) function - -[Original article](https://clickhouse.tech/docs/en/data-types/map/) From 409ff2f6b3f7b16cd9c15cca48b3332574bd8cd5 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Thu, 4 Feb 2021 22:13:55 +0300 Subject: [PATCH 04/24] Document system.opentelemetry_span_log system table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Задокументировал системную таблицу system.opentelemetry_span_log. --- .../system-tables/opentelemetry_span_log.md | 49 +++++++++++++++++++ .../system-tables/opentelemetry_span_log.md | 45 +++++++++++++++++ 2 files changed, 94 insertions(+) create mode 100644 docs/en/operations/system-tables/opentelemetry_span_log.md create mode 100644 docs/ru/operations/system-tables/opentelemetry_span_log.md diff --git a/docs/en/operations/system-tables/opentelemetry_span_log.md b/docs/en/operations/system-tables/opentelemetry_span_log.md new file mode 100644 index 00000000000..64fd549458a --- /dev/null +++ b/docs/en/operations/system-tables/opentelemetry_span_log.md @@ -0,0 +1,49 @@ +# system.opentelemetry_span_log {#system_tables-opentelemetry_span_log} + +Contains information about [trace spans](https://opentracing.io/docs/overview/spans/) for executed queries. + +Columns: + +- `trace_id` ([UUID](../../sql-reference/data-types/uuid.md) — ID of the trace for executed query. + +- `span_id` ([UInt64](../../sql-reference/data-types/int-uint.md)) — ID of the `trace span`. + +- `parent_span_id` ([UInt64](../../sql-reference/data-types/int-uint.md)) — ID of the parent `trace span`. + +- `operation_name` ([String](../../sql-reference/data-types/string.md)) — The name of the operation. + +- `start_time_us` ([UInt64](../../sql-reference/data-types/int-uint.md)) — The start time of the `trace span` (in microseconds). + +- `finish_time_us` ([UInt64](../../sql-reference/data-types/int-uint.md)) — The finish time of the `trace span` (in microseconds). + +- `finish_date` ([Date](../../sql-reference/data-types/date.md)) — The finish date of the `trace span`. + +- `attribute.names` ([Array(String)](../../sql-reference/data-types/array.md)) — [Attribute](https://opentelemetry.io/docs/go/instrumentation/#attributes) names depending on the `trace span`. They are filled in according to the recommendations in the [OpenTelemetry](https://opentelemetry.io/) standard. + +- `attribute.values` ([Array(String)](../../sql-reference/data-types/array.md)) — Attribute values depending on the `trace span`. They are filled in according to the recommendations in the `OpenTelemetry` standard. + +**Example** + +``` sql +SELECT * FROM system.opentelemetry_span_log LIMIT 1 FORMAT Vertical; +``` + +``` text +Row 1: +────── +trace_id: cdab0847-0d62-61d5-4d38-dd65b19a1914 +span_id: 701487461015578150 +parent_span_id: 2991972114672045096 +operation_name: DB::Block DB::InterpreterSelectQuery::getSampleBlockImpl() +start_time_us: 1612374594529090 +finish_time_us: 1612374594529108 +finish_date: 2021-02-03 +attribute.names: [] +attribute.values: [] +``` + +**See Also** + +- [OpenTelemetry](../../operations/opentelemetry.md) + +[Original article](https://clickhouse.tech/docs/en/operations/system_tables/opentelemetry_span_log) diff --git a/docs/ru/operations/system-tables/opentelemetry_span_log.md b/docs/ru/operations/system-tables/opentelemetry_span_log.md new file mode 100644 index 00000000000..5c577eb691d --- /dev/null +++ b/docs/ru/operations/system-tables/opentelemetry_span_log.md @@ -0,0 +1,45 @@ +# system.opentelemetry_span_log {#system_tables-opentelemetry_span_log} + +Содержит информацию о [trace spans](https://opentracing.io/docs/overview/spans/) для выполненных запросов. + +Столбцы: + +- `trace_id` ([UUID](../../sql-reference/data-types/uuid.md) — идентификатор трассировки для выполненного запроса. + +- `span_id` ([UInt64](../../sql-reference/data-types/int-uint.md)) — идентификатор `trace span`. + +- `parent_span_id` ([UInt64](../../sql-reference/data-types/int-uint.md)) — идентификатор родительского `trace span`. + +- `operation_name` ([String](../../sql-reference/data-types/string.md)) — имя операции. + +- `start_time_us` ([UInt64](../../sql-reference/data-types/int-uint.md)) — время начала `trace span` (в микросекундах). + +- `finish_time_us` ([UInt64](../../sql-reference/data-types/int-uint.md)) — время окончания `trace span` (в микросекундах). + +- `finish_date` ([Date](../../sql-reference/data-types/date.md)) — дата окончания `trace span`. + +- `attribute.names` ([Array(String)](../../sql-reference/data-types/array.md)) — имена [атрибутов](https://opentelemetry.io/docs/go/instrumentation/#attributes) в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте [OpenTelemetry](https://opentelemetry.io/). + +- `attribute.values` ([Array(String)](../../sql-reference/data-types/array.md)) — значения атрибутов в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте `OpenTelemetry`. + +**Пример** + +``` sql +SELECT * FROM system.opentelemetry_span_log LIMIT 1 FORMAT Vertical; +``` + +``` text +Row 1: +────── +trace_id: cdab0847-0d62-61d5-4d38-dd65b19a1914 +span_id: 701487461015578150 +parent_span_id: 2991972114672045096 +operation_name: DB::Block DB::InterpreterSelectQuery::getSampleBlockImpl() +start_time_us: 1612374594529090 +finish_time_us: 1612374594529108 +finish_date: 2021-02-03 +attribute.names: [] +attribute.values: [] +``` + +[Оригинальная статья](https://clickhouse.tech/docs/ru/operations/system_tables/opentelemetry_span_log) From 27933e714b956e34a404f1519b7397f3f93d2d7c Mon Sep 17 00:00:00 2001 From: Marquitos Date: Fri, 5 Feb 2021 17:39:05 +0100 Subject: [PATCH 05/24] Add 'access_management' configuration to initial setup --- docker/server/README.md | 8 ++++---- docker/server/entrypoint.sh | 2 ++ 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docker/server/README.md b/docker/server/README.md index d8e9204dffa..6f799d68185 100644 --- a/docker/server/README.md +++ b/docker/server/README.md @@ -56,7 +56,7 @@ $ echo 'SELECT version()' | curl 'http://localhost:8123/' --data-binary @- 20.12.3.3 ``` -### Volumes +### Volumes Typically you may want to mount the following folders inside your container to archieve persistency: @@ -76,7 +76,7 @@ You may also want to mount: * `/etc/clickhouse-server/usert.d/*.xml` - files with use settings adjustmenets * `/docker-entrypoint-initdb.d/` - folder with database initialization scripts (see below). -### Linux capabilities +### Linux capabilities ClickHouse has some advanced functionality which requite enabling several [linux capabilities](https://man7.org/linux/man-pages/man7/capabilities.7.html). @@ -113,10 +113,10 @@ $ docker run --rm -e CLICKHOUSE_UID=0 -e CLICKHOUSE_GID=0 --name clickhouse-serv ### How to create default database and user on starting -Sometimes you may want to create user (user named `default` is used by default) and database on image starting. You can do it using environment variables `CLICKHOUSE_DB`, `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD`: +Sometimes you may want to create user (user named `default` is used by default) and database on image starting. You can do it using environment variables `CLICKHOUSE_DB`, `CLICKHOUSE_USER`, `CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT` and `CLICKHOUSE_PASSWORD`: ``` -$ docker run --rm -e CLICKHOUSE_DB=my_database -e CLICKHOUSE_USER=username -e CLICKHOUSE_PASSWORD=password -p 9000:9000/tcp yandex/clickhouse-server +$ docker run --rm -e CLICKHOUSE_DB=my_database -e CLICKHOUSE_USER=username -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 -e CLICKHOUSE_PASSWORD=password -p 9000:9000/tcp yandex/clickhouse-server ``` ## How to extend this image diff --git a/docker/server/entrypoint.sh b/docker/server/entrypoint.sh index 549ff601c59..0138a165505 100755 --- a/docker/server/entrypoint.sh +++ b/docker/server/entrypoint.sh @@ -54,6 +54,7 @@ FORMAT_SCHEMA_PATH="$(clickhouse extract-from-config --config-file "$CLICKHOUSE_ CLICKHOUSE_USER="${CLICKHOUSE_USER:-default}" CLICKHOUSE_PASSWORD="${CLICKHOUSE_PASSWORD:-}" CLICKHOUSE_DB="${CLICKHOUSE_DB:-}" +CLICKHOUSE_ACCESS_MANAGEMENT="${CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT:-0}" for dir in "$DATA_DIR" \ "$ERROR_LOG_DIR" \ @@ -97,6 +98,7 @@ if [ -n "$CLICKHOUSE_USER" ] && [ "$CLICKHOUSE_USER" != "default" ] || [ -n "$CL ${CLICKHOUSE_PASSWORD} default + ${CLICKHOUSE_ACCESS_MANAGEMENT} From f0370b241c341ce961bac516afbd909631ec6b3d Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Sat, 6 Feb 2021 20:17:25 +0300 Subject: [PATCH 06/24] Document the opentelemetry_start_trace_probability setting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Задокументировал настройку. --- docs/en/operations/settings/settings.md | 11 +++++++++++ docs/ru/operations/settings/settings.md | 11 +++++++++++ 2 files changed, 22 insertions(+) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index edfd391c71e..869c76fb975 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2592,4 +2592,15 @@ Possible values: Default value: `16`. +## opentelemetry_start_trace_probability {#opentelemetry-start-trace-probability} + +Enables a trace for executed queries. + +Possible values: + +- 0 — The trace for a executed query is disabled. +- 1 — The trace for a executed query is enabled. + +Default value: `0`. + [Original article](https://clickhouse.tech/docs/en/operations/settings/settings/) diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index bacc97dfd14..2aa81daa0b0 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -2473,4 +2473,15 @@ SELECT SUM(-1), MAX(0) FROM system.one WHERE 0; Значение по умолчанию: `16`. +## opentelemetry_start_trace_probability {#opentelemetry-start-trace-probability} + +Включает трассировку для выполненных запросов. + +Возможные значения: + +- 0 — трассировка для выполненного запроса отключена. +- 1 — трассировка для выполненного запроса включена. + +Значение по умолчанию: `0`. + [Оригинальная статья](https://clickhouse.tech/docs/ru/operations/settings/settings/) From ed59b355c0dba42da612546a584b0645ef463019 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Tue, 9 Feb 2021 20:34:16 +0300 Subject: [PATCH 07/24] Update the description of the opentelemetry_start_trace_probability setting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Обновил документацию настройки. --- docs/en/operations/settings/settings.md | 7 ++++--- docs/ru/operations/settings/settings.md | 7 ++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 869c76fb975..0554ea79ecd 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2594,12 +2594,13 @@ Default value: `16`. ## opentelemetry_start_trace_probability {#opentelemetry-start-trace-probability} -Enables a trace for executed queries. +Sets the probability that the ClickHouse can start a trace for executed queries (if no parent [trace context](https://www.w3.org/TR/trace-context/) is supplied). Possible values: -- 0 — The trace for a executed query is disabled. -- 1 — The trace for a executed query is enabled. +- 0 — The trace for a executed queries is disabled (if no parent trace context is supplied). +- (0, 1) — The probability with which the ClickHouse can start a trace for executed queries (if no parent trace context is supplied). For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. +- 1 — The trace for all executed queries is enabled. Default value: `0`. diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 2aa81daa0b0..47e2666e652 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -2475,12 +2475,13 @@ SELECT SUM(-1), MAX(0) FROM system.one WHERE 0; ## opentelemetry_start_trace_probability {#opentelemetry-start-trace-probability} -Включает трассировку для выполненных запросов. +Задает вероятность того, что ClickHouse начнет трассировку для выполненных запросов (если не указан [родительский контекст](https://www.w3.org/TR/trace-context/) трассировки). Возможные значения: -- 0 — трассировка для выполненного запроса отключена. -- 1 — трассировка для выполненного запроса включена. +- 0 — трассировка для выполненных запросов отключена (если не указан родительский контекст трассировки). +- (0, 1) — вероятность, с которой ClickHouse начнет трассировку для выполненных запросов (если не указан родительский контекст трассировки). Например, при значении настройки, равной `0,5`, ClickHouse начнет трассировку в среднем для половины запросов. +- 1 — трассировка для всех выполненных запросов включена. Значение по умолчанию: `0`. From 2905df831f9119d414c44a8eedd8df9012825889 Mon Sep 17 00:00:00 2001 From: Olga Revyakina Date: Thu, 11 Feb 2021 10:15:18 +0300 Subject: [PATCH 08/24] JSON deteted --- docs/en/sql-reference/data-types/map.md | 40 +++++++++++++--- .../functions/tuple-map-functions.md | 4 +- docs/ru/sql-reference/data-types/map.md | 46 ++++++++++++------- .../functions/tuple-map-functions.md | 4 +- 4 files changed, 66 insertions(+), 28 deletions(-) diff --git a/docs/en/sql-reference/data-types/map.md b/docs/en/sql-reference/data-types/map.md index 0f0f69d421d..58634e5b669 100644 --- a/docs/en/sql-reference/data-types/map.md +++ b/docs/en/sql-reference/data-types/map.md @@ -5,7 +5,7 @@ toc_title: Map(key, value) # Map(key, value) {#data_type-map} -`Map(key, value)` data type stores `key:value` pairs in structures like JSON. +`Map(key, value)` data type stores `key:value` pairs. **Parameters** - `key` — The key part of the pair. [String](../../sql-reference/data-types/string.md) or [Integer](../../sql-reference/data-types/int-uint.md). @@ -14,24 +14,50 @@ toc_title: Map(key, value) !!! warning "Warning" Currently `Map` data type is an experimental feature. To work with it you must set `allow_experimental_map_type = 1`. -To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. +To get the value from an `a Map('key', 'value')` column, use `a['key']` syntax. This lookup works now with a linear complexity. -**Example** +**Examples** -Query: +Consider the table: ``` sql CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +INSERT INTO table_map VALUES ({'key1':1, 'key2':10}), ({'key1':2,'key2':20}), ({'key1':3,'key2':30}); +``` + +Select all `key2` values: + +```sql SELECT a['key2'] FROM table_map; ``` Result: ```text ┌─arrayElement(a, 'key2')─┐ +│ 10 │ +│ 20 │ +│ 30 │ +└─────────────────────────┘ +``` + +If there's no such `key` in the `Map()` column, the query returns zeros for numerical values, empty strings or empty arrays. + +```sql +INSERT INTO table_map VALUES ({'key3':100}), ({}); +SELECT a['key3'] FROM table_map; +``` + +Result: + +```text +┌─arrayElement(a, 'key3')─┐ │ 100 │ -│ 200 │ -│ 300 │ +│ 0 │ +└─────────────────────────┘ +┌─arrayElement(a, 'key3')─┐ +│ 0 │ +│ 0 │ +│ 0 │ └─────────────────────────┘ ``` diff --git a/docs/en/sql-reference/functions/tuple-map-functions.md b/docs/en/sql-reference/functions/tuple-map-functions.md index b81f971196a..18d008f11f2 100644 --- a/docs/en/sql-reference/functions/tuple-map-functions.md +++ b/docs/en/sql-reference/functions/tuple-map-functions.md @@ -7,7 +7,7 @@ toc_title: Working with maps ## map {#function-map} -Arranges `key:value` pairs into a JSON data structure. +Arranges `key:value` pairs into [Map(key, value)](../../sql-reference/data-types/map.md) data type. **Syntax** @@ -22,7 +22,7 @@ map(key1, value1[, key2, value2, ...]) **Returned value** -- JSON with `key:value` pairs. +- Data structure as `key:value` pairs. Type: [Map(key, value)](../../sql-reference/data-types/map.md). diff --git a/docs/ru/sql-reference/data-types/map.md b/docs/ru/sql-reference/data-types/map.md index c1391e37133..9c2ffedc4a9 100644 --- a/docs/ru/sql-reference/data-types/map.md +++ b/docs/ru/sql-reference/data-types/map.md @@ -5,7 +5,7 @@ toc_title: Map(key, value) # Map(key, value) {#data_type-map} -Тип данных `Map(key, value)` хранит пары `ключ:значение` в структурах типа JSON. +Тип данных `Map(key, value)` хранит пары `ключ:значение`. **Параметры** - `key` — ключ. [String](../../sql-reference/data-types/string.md) или [Integer](../../sql-reference/data-types/int-uint.md). @@ -14,39 +14,51 @@ toc_title: Map(key, value) !!! warning "Предупреждение" Сейчас использование типа данных `Map` является экспериментальной возможностью. Чтобы использовать этот тип данных, включите настройку `allow_experimental_map_type = 1`. -Чтобы получить значение из колонки `a Map('key', 'value')`, используйте синтаксис `a['key']`. +Чтобы получить значение из колонки `a Map('key', 'value')`, используйте синтаксис `a['key']`. В настоящее время такая подстановка работает по алгоритму с линейной сложностью. -**Пример** +**Примеры** -Запрос: +Рассмотрим таблицу: ``` sql CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; -INSERT INTO table_map VALUES ({'key1':1, 'key2':100}), ({'key1':2,'key2':200}), ({'key1':3,'key2':300}); +INSERT INTO table_map VALUES ({'key1':1, 'key2':10}), ({'key1':2,'key2':20}), ({'key1':3,'key2':30}); +``` + +Выборка всем значений ключа `key2`: + +```sql SELECT a['key2'] FROM table_map; ``` Результат: ```text ┌─arrayElement(a, 'key2')─┐ -│ 100 │ -│ 200 │ -│ 300 │ +│ 10 │ +│ 20 │ +│ 30 │ └─────────────────────────┘ ``` -## Преобразование типа данных Tuple в Map {#map-and-tuple} +Если для какого-то ключа `key` в колонке с типом `Map()` нет значения, запрос возвращает нули для числовых колонок, пустые строки или пустые массивы. -Для преобразования данных с типом `Tuple()` в тип `Map()` можно использовать функцию [CAST](../../sql-reference/functions/type-conversion-functions.md#type_conversion_function-cast): - -``` sql -SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map; +```sql +INSERT INTO table_map VALUES ({'key3':100}), ({}); +SELECT a['key3'] FROM table_map; ``` -``` text -┌─map───────────────────────────┐ -│ {1:'Ready',2:'Steady',3:'Go'} │ -└───────────────────────────────┘ +Результат: + +```text +┌─arrayElement(a, 'key3')─┐ +│ 100 │ +│ 0 │ +└─────────────────────────┘ +┌─arrayElement(a, 'key3')─┐ +│ 0 │ +│ 0 │ +│ 0 │ +└─────────────────────────┘ ``` **См. также** diff --git a/docs/ru/sql-reference/functions/tuple-map-functions.md b/docs/ru/sql-reference/functions/tuple-map-functions.md index 65e44698008..a36613280a1 100644 --- a/docs/ru/sql-reference/functions/tuple-map-functions.md +++ b/docs/ru/sql-reference/functions/tuple-map-functions.md @@ -7,7 +7,7 @@ toc_title: Работа с контейнерами map ## map {#function-map} -Преобразовывает пары `ключ:значение` в структуру JSON. +Преобразовывает пары `ключ:значение` в тип данных [Map(key, value)](../../sql-reference/data-types/map.md). **Синтаксис** @@ -22,7 +22,7 @@ map(key1, value1[, key2, value2, ...]) **Возвращаемое значение** -- Структура JSON с парами `ключ:значение`. +- Структура данных в виде пар `ключ:значение`. Тип: [Map(key, value)](../../sql-reference/data-types/map.md). From d3549aca95c1bcdc2b65617afd35f71ee51be4a9 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Thu, 11 Feb 2021 21:42:15 +0300 Subject: [PATCH 09/24] Fix the description of the table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Поправил описание таблицы. --- .../en/operations/system-tables/opentelemetry_span_log.md | 8 ++++++-- .../ru/operations/system-tables/opentelemetry_span_log.md | 8 ++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/en/operations/system-tables/opentelemetry_span_log.md b/docs/en/operations/system-tables/opentelemetry_span_log.md index 64fd549458a..e45a989742c 100644 --- a/docs/en/operations/system-tables/opentelemetry_span_log.md +++ b/docs/en/operations/system-tables/opentelemetry_span_log.md @@ -18,16 +18,20 @@ Columns: - `finish_date` ([Date](../../sql-reference/data-types/date.md)) — The finish date of the `trace span`. -- `attribute.names` ([Array(String)](../../sql-reference/data-types/array.md)) — [Attribute](https://opentelemetry.io/docs/go/instrumentation/#attributes) names depending on the `trace span`. They are filled in according to the recommendations in the [OpenTelemetry](https://opentelemetry.io/) standard. +- `attribute.names` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) — [Attribute](https://opentelemetry.io/docs/go/instrumentation/#attributes) names depending on the `trace span`. They are filled in according to the recommendations in the [OpenTelemetry](https://opentelemetry.io/) standard. -- `attribute.values` ([Array(String)](../../sql-reference/data-types/array.md)) — Attribute values depending on the `trace span`. They are filled in according to the recommendations in the `OpenTelemetry` standard. +- `attribute.values` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) — Attribute values depending on the `trace span`. They are filled in according to the recommendations in the `OpenTelemetry` standard. **Example** +Query: + ``` sql SELECT * FROM system.opentelemetry_span_log LIMIT 1 FORMAT Vertical; ``` +Result: + ``` text Row 1: ────── diff --git a/docs/ru/operations/system-tables/opentelemetry_span_log.md b/docs/ru/operations/system-tables/opentelemetry_span_log.md index 5c577eb691d..96555064b0e 100644 --- a/docs/ru/operations/system-tables/opentelemetry_span_log.md +++ b/docs/ru/operations/system-tables/opentelemetry_span_log.md @@ -18,16 +18,20 @@ - `finish_date` ([Date](../../sql-reference/data-types/date.md)) — дата окончания `trace span`. -- `attribute.names` ([Array(String)](../../sql-reference/data-types/array.md)) — имена [атрибутов](https://opentelemetry.io/docs/go/instrumentation/#attributes) в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте [OpenTelemetry](https://opentelemetry.io/). +- `attribute.names` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) — имена [атрибутов](https://opentelemetry.io/docs/go/instrumentation/#attributes) в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте [OpenTelemetry](https://opentelemetry.io/). -- `attribute.values` ([Array(String)](../../sql-reference/data-types/array.md)) — значения атрибутов в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте `OpenTelemetry`. +- `attribute.values` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) — значения атрибутов в зависимости от `trace span`. Заполняются согласно рекомендациям в стандарте `OpenTelemetry`. **Пример** +Запрос: + ``` sql SELECT * FROM system.opentelemetry_span_log LIMIT 1 FORMAT Vertical; ``` +Результат: + ``` text Row 1: ────── From ce1524c4ebaca545feeaa1493d5ae8e66af8dab9 Mon Sep 17 00:00:00 2001 From: sevirov <72220289+sevirov@users.noreply.github.com> Date: Thu, 11 Feb 2021 22:06:30 +0300 Subject: [PATCH 10/24] Update docs/en/operations/settings/settings.md Co-authored-by: Anna <42538400+adevyatova@users.noreply.github.com> --- docs/en/operations/settings/settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 0554ea79ecd..8f1cb186449 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2599,7 +2599,7 @@ Sets the probability that the ClickHouse can start a trace for executed queries Possible values: - 0 — The trace for a executed queries is disabled (if no parent trace context is supplied). -- (0, 1) — The probability with which the ClickHouse can start a trace for executed queries (if no parent trace context is supplied). For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. +- Positive floating-point number in the range [0..1]. For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. - 1 — The trace for all executed queries is enabled. Default value: `0`. From 6271709efacad598431127808dae44cd1ac6e0bb Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Thu, 11 Feb 2021 22:23:19 +0300 Subject: [PATCH 11/24] Fix the description of the setting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Поправил описание настройки. --- docs/en/operations/settings/settings.md | 2 +- docs/ru/operations/settings/settings.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 8f1cb186449..6f028b00a5b 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2599,7 +2599,7 @@ Sets the probability that the ClickHouse can start a trace for executed queries Possible values: - 0 — The trace for a executed queries is disabled (if no parent trace context is supplied). -- Positive floating-point number in the range [0..1]. For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. +- Positive floating-point number in the range [0..1]. For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. - 1 — The trace for all executed queries is enabled. Default value: `0`. diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 47e2666e652..434157401fa 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -2475,12 +2475,12 @@ SELECT SUM(-1), MAX(0) FROM system.one WHERE 0; ## opentelemetry_start_trace_probability {#opentelemetry-start-trace-probability} -Задает вероятность того, что ClickHouse начнет трассировку для выполненных запросов (если не указан [родительский контекст](https://www.w3.org/TR/trace-context/) трассировки). +Задает вероятность того, что ClickHouse начнет трассировку для выполненных запросов (если не указан [входящий контекст](https://www.w3.org/TR/trace-context/) трассировки). Возможные значения: -- 0 — трассировка для выполненных запросов отключена (если не указан родительский контекст трассировки). -- (0, 1) — вероятность, с которой ClickHouse начнет трассировку для выполненных запросов (если не указан родительский контекст трассировки). Например, при значении настройки, равной `0,5`, ClickHouse начнет трассировку в среднем для половины запросов. +- 0 — трассировка для выполненных запросов отключена (если не указан входящий контекст трассировки). +- Положительное число с плавающей точкой в диапазоне [0..1]. Например, при значении настройки, равной `0,5`, ClickHouse начнет трассировку в среднем для половины запросов. - 1 — трассировка для всех выполненных запросов включена. Значение по умолчанию: `0`. From bb2a11bcfd94c525238a768ac10bdeaa1fb1d2b5 Mon Sep 17 00:00:00 2001 From: Olga Revyakina Date: Fri, 12 Feb 2021 07:43:33 +0300 Subject: [PATCH 12/24] Misspelling --- docs/ru/sql-reference/data-types/map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/data-types/map.md b/docs/ru/sql-reference/data-types/map.md index 9c2ffedc4a9..6cb8ccf1143 100644 --- a/docs/ru/sql-reference/data-types/map.md +++ b/docs/ru/sql-reference/data-types/map.md @@ -25,7 +25,7 @@ CREATE TABLE table_map (a Map(String, UInt64)) ENGINE=Memory; INSERT INTO table_map VALUES ({'key1':1, 'key2':10}), ({'key1':2,'key2':20}), ({'key1':3,'key2':30}); ``` -Выборка всем значений ключа `key2`: +Выборка всех значений ключа `key2`: ```sql SELECT a['key2'] FROM table_map; From 69d4120982fa2b7cae35da83532c8318f44bfc8f Mon Sep 17 00:00:00 2001 From: Amos Bird Date: Fri, 12 Feb 2021 10:22:18 +0800 Subject: [PATCH 13/24] Disable table function view in expression --- src/Parsers/ASTFunction.cpp | 8 ++++++++ .../0_stateless/01715_table_function_view_fix.reference | 0 .../queries/0_stateless/01715_table_function_view_fix.sql | 1 + 3 files changed, 9 insertions(+) create mode 100644 tests/queries/0_stateless/01715_table_function_view_fix.reference create mode 100644 tests/queries/0_stateless/01715_table_function_view_fix.sql diff --git a/src/Parsers/ASTFunction.cpp b/src/Parsers/ASTFunction.cpp index 806b8e6c5b9..29ac01eefc5 100644 --- a/src/Parsers/ASTFunction.cpp +++ b/src/Parsers/ASTFunction.cpp @@ -15,8 +15,16 @@ namespace DB { +namespace ErrorCodes +{ + extern const int UNEXPECTED_EXPRESSION; +} + void ASTFunction::appendColumnNameImpl(WriteBuffer & ostr) const { + if (name == "view") + throw Exception("Table function view cannot be used as an expression", ErrorCodes::UNEXPECTED_EXPRESSION); + writeString(name, ostr); if (parameters) diff --git a/tests/queries/0_stateless/01715_table_function_view_fix.reference b/tests/queries/0_stateless/01715_table_function_view_fix.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/01715_table_function_view_fix.sql b/tests/queries/0_stateless/01715_table_function_view_fix.sql new file mode 100644 index 00000000000..21da116f6ba --- /dev/null +++ b/tests/queries/0_stateless/01715_table_function_view_fix.sql @@ -0,0 +1 @@ +SELECT view(SELECT 1); -- { serverError 183 } From a551edd8d6e308569433a9158df1ee31a60844de Mon Sep 17 00:00:00 2001 From: Amos Bird Date: Sat, 13 Feb 2021 13:18:14 +0800 Subject: [PATCH 14/24] Do not parse view function in expression --- src/Parsers/ASTFunction.cpp | 8 -- src/Parsers/ExpressionElementParsers.cpp | 81 ++++++++++++------- src/Parsers/ExpressionElementParsers.h | 16 +++- src/Parsers/ExpressionListParsers.cpp | 17 +++- src/Parsers/ExpressionListParsers.h | 22 ++++- src/Parsers/ParserTablesInSelectQuery.cpp | 2 +- .../01715_table_function_view_fix.sql | 2 +- 7 files changed, 98 insertions(+), 50 deletions(-) diff --git a/src/Parsers/ASTFunction.cpp b/src/Parsers/ASTFunction.cpp index 29ac01eefc5..806b8e6c5b9 100644 --- a/src/Parsers/ASTFunction.cpp +++ b/src/Parsers/ASTFunction.cpp @@ -15,16 +15,8 @@ namespace DB { -namespace ErrorCodes -{ - extern const int UNEXPECTED_EXPRESSION; -} - void ASTFunction::appendColumnNameImpl(WriteBuffer & ostr) const { - if (name == "view") - throw Exception("Table function view cannot be used as an expression", ErrorCodes::UNEXPECTED_EXPRESSION); - writeString(name, ostr); if (parameters) diff --git a/src/Parsers/ExpressionElementParsers.cpp b/src/Parsers/ExpressionElementParsers.cpp index e7cd85798b9..3d868812304 100644 --- a/src/Parsers/ExpressionElementParsers.cpp +++ b/src/Parsers/ExpressionElementParsers.cpp @@ -266,7 +266,7 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) ParserIdentifier id_parser; ParserKeyword distinct("DISTINCT"); ParserKeyword all("ALL"); - ParserExpressionList contents(false); + ParserExpressionList contents(false, is_table_function); ParserSelectWithUnionQuery select; ParserKeyword over("OVER"); @@ -278,6 +278,12 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) ASTPtr expr_list_args; ASTPtr expr_list_params; + if (is_table_function) + { + if (ParserTableFunctionView().parse(pos, node, expected)) + return true; + } + if (!id_parser.parse(pos, identifier, expected)) return false; @@ -312,36 +318,6 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) } } - if (!has_distinct && !has_all) - { - auto old_pos = pos; - auto maybe_an_subquery = pos->type == TokenType::OpeningRoundBracket; - - if (select.parse(pos, query, expected)) - { - auto & select_ast = query->as(); - if (select_ast.list_of_selects->children.size() == 1 && maybe_an_subquery) - { - // It's an subquery. Bail out. - pos = old_pos; - } - else - { - if (pos->type != TokenType::ClosingRoundBracket) - return false; - ++pos; - auto function_node = std::make_shared(); - tryGetIdentifierNameInto(identifier, function_node->name); - auto expr_list_with_single_query = std::make_shared(); - expr_list_with_single_query->children.push_back(query); - function_node->arguments = expr_list_with_single_query; - function_node->children.push_back(function_node->arguments); - node = function_node; - return true; - } - } - } - const char * contents_begin = pos->begin; if (!contents.parse(pos, expr_list_args, expected)) return false; @@ -477,6 +453,49 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) return true; } +bool ParserTableFunctionView::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) +{ + ParserIdentifier id_parser; + ParserKeyword view("VIEW"); + ParserSelectWithUnionQuery select; + + ASTPtr identifier; + ASTPtr query; + + if (!view.ignore(pos, expected)) + return false; + + if (pos->type != TokenType::OpeningRoundBracket) + return false; + + ++pos; + + bool maybe_an_subquery = pos->type == TokenType::OpeningRoundBracket; + + if (!select.parse(pos, query, expected)) + return false; + + auto & select_ast = query->as(); + if (select_ast.list_of_selects->children.size() == 1 && maybe_an_subquery) + { + // It's an subquery. Bail out. + return false; + } + + if (pos->type != TokenType::ClosingRoundBracket) + return false; + ++pos; + auto function_node = std::make_shared(); + tryGetIdentifierNameInto(identifier, function_node->name); + auto expr_list_with_single_query = std::make_shared(); + expr_list_with_single_query->children.push_back(query); + function_node->name = "view"; + function_node->arguments = expr_list_with_single_query; + function_node->children.push_back(function_node->arguments); + node = function_node; + return true; +} + bool ParserWindowReference::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) { ASTFunction * function = dynamic_cast(node.get()); diff --git a/src/Parsers/ExpressionElementParsers.h b/src/Parsers/ExpressionElementParsers.h index ba18fc2cddd..b6194f981fe 100644 --- a/src/Parsers/ExpressionElementParsers.h +++ b/src/Parsers/ExpressionElementParsers.h @@ -149,11 +149,25 @@ protected: class ParserFunction : public IParserBase { public: - ParserFunction(bool allow_function_parameters_ = true) : allow_function_parameters(allow_function_parameters_) {} + ParserFunction(bool allow_function_parameters_ = true, bool is_table_function_ = false) + : allow_function_parameters(allow_function_parameters_), is_table_function(is_table_function_) + { + } + protected: const char * getName() const override { return "function"; } bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; bool allow_function_parameters; + bool is_table_function; +}; + +// A special function parser for view table function. +// It parses an SELECT query as its argument and doesn't support getColumnName(). +class ParserTableFunctionView : public IParserBase +{ +protected: + const char * getName() const override { return "function"; } + bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; }; // Window reference (the thing that goes after OVER) for window function. diff --git a/src/Parsers/ExpressionListParsers.cpp b/src/Parsers/ExpressionListParsers.cpp index afe85f069c7..e9ad65af471 100644 --- a/src/Parsers/ExpressionListParsers.cpp +++ b/src/Parsers/ExpressionListParsers.cpp @@ -468,6 +468,14 @@ bool ParserLambdaExpression::parseImpl(Pos & pos, ASTPtr & node, Expected & expe } +bool ParserTableFunctionExpression::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) +{ + if (ParserTableFunctionView().parse(pos, node, expected)) + return true; + return elem_parser.parse(pos, node, expected); +} + + bool ParserPrefixUnaryOperatorExpression::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) { /// try to find any of the valid operators @@ -570,9 +578,10 @@ bool ParserTupleElementExpression::parseImpl(Pos & pos, ASTPtr & node, Expected } -ParserExpressionWithOptionalAlias::ParserExpressionWithOptionalAlias(bool allow_alias_without_as_keyword) - : impl(std::make_unique(std::make_unique(), - allow_alias_without_as_keyword)) +ParserExpressionWithOptionalAlias::ParserExpressionWithOptionalAlias(bool allow_alias_without_as_keyword, bool is_table_function) + : impl(std::make_unique( + is_table_function ? ParserPtr(std::make_unique()) : ParserPtr(std::make_unique()), + allow_alias_without_as_keyword)) { } @@ -580,7 +589,7 @@ ParserExpressionWithOptionalAlias::ParserExpressionWithOptionalAlias(bool allow_ bool ParserExpressionList::parseImpl(Pos & pos, ASTPtr & node, Expected & expected) { return ParserList( - std::make_unique(allow_alias_without_as_keyword), + std::make_unique(allow_alias_without_as_keyword, is_table_function), std::make_unique(TokenType::Comma)) .parse(pos, node, expected); } diff --git a/src/Parsers/ExpressionListParsers.h b/src/Parsers/ExpressionListParsers.h index 90b27950873..2371e006c09 100644 --- a/src/Parsers/ExpressionListParsers.h +++ b/src/Parsers/ExpressionListParsers.h @@ -436,13 +436,26 @@ protected: }; +// It's used to parse expressions in table function. +class ParserTableFunctionExpression : public IParserBase +{ +private: + ParserLambdaExpression elem_parser; + +protected: + const char * getName() const override { return "table function expression"; } + + bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; +}; + + using ParserExpression = ParserLambdaExpression; class ParserExpressionWithOptionalAlias : public IParserBase { public: - ParserExpressionWithOptionalAlias(bool allow_alias_without_as_keyword); + explicit ParserExpressionWithOptionalAlias(bool allow_alias_without_as_keyword, bool is_table_function = false); protected: ParserPtr impl; @@ -459,11 +472,12 @@ protected: class ParserExpressionList : public IParserBase { public: - ParserExpressionList(bool allow_alias_without_as_keyword_) - : allow_alias_without_as_keyword(allow_alias_without_as_keyword_) {} + explicit ParserExpressionList(bool allow_alias_without_as_keyword_, bool is_table_function_ = false) + : allow_alias_without_as_keyword(allow_alias_without_as_keyword_), is_table_function(is_table_function_) {} protected: bool allow_alias_without_as_keyword; + bool is_table_function; // This expression list is used by a table function const char * getName() const override { return "list of expressions"; } bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override; @@ -473,7 +487,7 @@ protected: class ParserNotEmptyExpressionList : public IParserBase { public: - ParserNotEmptyExpressionList(bool allow_alias_without_as_keyword) + explicit ParserNotEmptyExpressionList(bool allow_alias_without_as_keyword) : nested_parser(allow_alias_without_as_keyword) {} private: ParserExpressionList nested_parser; diff --git a/src/Parsers/ParserTablesInSelectQuery.cpp b/src/Parsers/ParserTablesInSelectQuery.cpp index 1264acefe64..2e20279dbe1 100644 --- a/src/Parsers/ParserTablesInSelectQuery.cpp +++ b/src/Parsers/ParserTablesInSelectQuery.cpp @@ -22,7 +22,7 @@ bool ParserTableExpression::parseImpl(Pos & pos, ASTPtr & node, Expected & expec auto res = std::make_shared(); if (!ParserWithOptionalAlias(std::make_unique(), true).parse(pos, res->subquery, expected) - && !ParserWithOptionalAlias(std::make_unique(), true).parse(pos, res->table_function, expected) + && !ParserWithOptionalAlias(std::make_unique(true, true), true).parse(pos, res->table_function, expected) && !ParserWithOptionalAlias(std::make_unique(false, true), true).parse(pos, res->database_and_table_name, expected)) return false; diff --git a/tests/queries/0_stateless/01715_table_function_view_fix.sql b/tests/queries/0_stateless/01715_table_function_view_fix.sql index 21da116f6ba..de5150b7b70 100644 --- a/tests/queries/0_stateless/01715_table_function_view_fix.sql +++ b/tests/queries/0_stateless/01715_table_function_view_fix.sql @@ -1 +1 @@ -SELECT view(SELECT 1); -- { serverError 183 } +SELECT view(SELECT 1); -- { clientError 62 } From c1550814ca770a0ecb9aec0de8eeb77dee266ca4 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 15 Feb 2021 10:30:08 +0300 Subject: [PATCH 15/24] Disable snapshots for tests --- src/Coordination/NuKeeperStateMachine.cpp | 2 ++ tests/config/config.d/test_keeper_port.xml | 2 ++ tests/queries/skip_list.json | 14 ++------------ 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/src/Coordination/NuKeeperStateMachine.cpp b/src/Coordination/NuKeeperStateMachine.cpp index d282f57ce73..0061645c75c 100644 --- a/src/Coordination/NuKeeperStateMachine.cpp +++ b/src/Coordination/NuKeeperStateMachine.cpp @@ -166,6 +166,8 @@ void NuKeeperStateMachine::create_snapshot( } } + + LOG_DEBUG(log, "Created snapshot {}", s.get_last_log_idx()); nuraft::ptr except(nullptr); bool ret = true; when_done(ret, except); diff --git a/tests/config/config.d/test_keeper_port.xml b/tests/config/config.d/test_keeper_port.xml index 6ca00a972d4..97c6d7c2e33 100644 --- a/tests/config/config.d/test_keeper_port.xml +++ b/tests/config/config.d/test_keeper_port.xml @@ -6,6 +6,8 @@ 10000 30000 + 0 + 0 diff --git a/tests/queries/skip_list.json b/tests/queries/skip_list.json index ee25bee6a0a..e4e7504ba41 100644 --- a/tests/queries/skip_list.json +++ b/tests/queries/skip_list.json @@ -10,7 +10,6 @@ "00152_insert_different_granularity", "00151_replace_partition_with_different_granularity", "00157_cache_dictionary", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01193_metadata_loading", "01473_event_time_microseconds", "01526_max_untracked_memory", /// requires TraceCollector, does not available under sanitizers @@ -26,7 +25,6 @@ "memory_profiler", "odbc_roundtrip", "01103_check_cpu_instructions_at_startup", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01473_event_time_microseconds", "01526_max_untracked_memory", /// requires TraceCollector, does not available under sanitizers "01193_metadata_loading" @@ -37,7 +35,6 @@ "memory_profiler", "01103_check_cpu_instructions_at_startup", "00900_orc_load", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01473_event_time_microseconds", "01526_max_untracked_memory", /// requires TraceCollector, does not available under sanitizers "01193_metadata_loading" @@ -49,7 +46,6 @@ "01103_check_cpu_instructions_at_startup", "01086_odbc_roundtrip", /// can't pass because odbc libraries are not instrumented "00877_memory_limit_for_new_delete", /// memory limits don't work correctly under msan because it replaces malloc/free - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01473_event_time_microseconds", "01526_max_untracked_memory", /// requires TraceCollector, does not available under sanitizers "01193_metadata_loading" @@ -61,7 +57,6 @@ "00980_alter_settings_race", "00834_kill_mutation_replicated_zookeeper", "00834_kill_mutation", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01200_mutations_memory_consumption", "01103_check_cpu_instructions_at_startup", "01037_polygon_dicts_", @@ -87,7 +82,6 @@ "00505_secure", "00505_shard_secure", "odbc_roundtrip", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "01103_check_cpu_instructions_at_startup", "01114_mysql_database_engine_segfault", "00834_cancel_http_readonly_queries_on_client_close", @@ -101,19 +95,16 @@ "01455_time_zones" ], "release-build": [ - "00992_system_parts_race_condition_zookeeper" /// TODO remove me (alesapin) ], "database-ordinary": [ "00604_show_create_database", "00609_mv_index_in_in", "00510_materizlized_view_and_deduplication_zookeeper", - "00738_lock_for_inner_table", - "00992_system_parts_race_condition_zookeeper" /// TODO remove me (alesapin) + "00738_lock_for_inner_table" ], "polymorphic-parts": [ "01508_partition_pruning_long", /// bug, shoud be fixed - "01482_move_to_prewhere_and_cast", /// bug, shoud be fixed - "00992_system_parts_race_condition_zookeeper" /// TODO remove me (alesapin) + "01482_move_to_prewhere_and_cast" /// bug, shoud be fixed ], "antlr": [ "00186_very_long_arrays", @@ -153,7 +144,6 @@ "00982_array_enumerate_uniq_ranked", "00984_materialized_view_to_columns", "00988_constraints_replication_zookeeper", - "00992_system_parts_race_condition_zookeeper", /// TODO remove me (alesapin) "00995_order_by_with_fill", "01001_enums_in_in_section", "01011_group_uniq_array_memsan", From 9686649b0229cc4f492dbf646d6342d587f02657 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 15 Feb 2021 12:42:50 +0300 Subject: [PATCH 16/24] Fix non-zero session reconnect in integration test --- tests/integration/test_testkeeper_back_to_back/test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/integration/test_testkeeper_back_to_back/test.py b/tests/integration/test_testkeeper_back_to_back/test.py index 0f2c1ed19a5..8ec54f1a883 100644 --- a/tests/integration/test_testkeeper_back_to_back/test.py +++ b/tests/integration/test_testkeeper_back_to_back/test.py @@ -29,8 +29,8 @@ def get_fake_zk(): def reset_last_zxid_listener(state): print("Fake zk callback called for state", state) global _fake_zk_instance - # reset last_zxid -- fake server doesn't support it - _fake_zk_instance.last_zxid = 0 + if state != KazooState.CONNECTED: + _fake_zk_instance._reset() _fake_zk_instance.add_listener(reset_last_zxid_listener) _fake_zk_instance.start() From ac476ad83e526d8afec591189f10c5933edf68e7 Mon Sep 17 00:00:00 2001 From: Nikita Mikhailov Date: Mon, 15 Feb 2021 14:27:16 +0300 Subject: [PATCH 17/24] done --- .../1_stateful/00158_cache_dictionary_has.reference | 6 +++--- tests/queries/1_stateful/00158_cache_dictionary_has.sql | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/queries/1_stateful/00158_cache_dictionary_has.reference b/tests/queries/1_stateful/00158_cache_dictionary_has.reference index f8d5cd4f53d..ad4bce6bec5 100644 --- a/tests/queries/1_stateful/00158_cache_dictionary_has.reference +++ b/tests/queries/1_stateful/00158_cache_dictionary_has.reference @@ -1,6 +1,6 @@ +100 6410 -6410 -25323 +100 25323 -1774655 +100 1774655 diff --git a/tests/queries/1_stateful/00158_cache_dictionary_has.sql b/tests/queries/1_stateful/00158_cache_dictionary_has.sql index 063e7843fd4..8461728c58e 100644 --- a/tests/queries/1_stateful/00158_cache_dictionary_has.sql +++ b/tests/queries/1_stateful/00158_cache_dictionary_has.sql @@ -6,15 +6,15 @@ CREATE DICTIONARY db_dict.cache_hits PRIMARY KEY WatchID SOURCE(CLICKHOUSE(HOST 'localhost' PORT tcpPort() USER 'default' TABLE 'hits' PASSWORD '' DB 'test')) LIFETIME(MIN 300 MAX 600) -LAYOUT(CACHE(SIZE_IN_CELLS 100000 QUERY_WAIT_TIMEOUT_MILLISECONDS 600000)); +LAYOUT(CACHE(SIZE_IN_CELLS 100 QUERY_WAIT_TIMEOUT_MILLISECONDS 600000)); -SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 1400 == 0); +SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 1400 == 0 LIMIT 100); SELECT count() from test.hits PREWHERE WatchID % 1400 == 0; -SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 350 == 0); +SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 350 == 0 LIMIT 100); SELECT count() from test.hits PREWHERE WatchID % 350 == 0; -SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 5 == 0); +SELECT sum(flag) FROM (SELECT dictHas('db_dict.cache_hits', toUInt64(WatchID)) as flag FROM test.hits PREWHERE WatchID % 5 == 0 LIMIT 100); SELECT count() from test.hits PREWHERE WatchID % 5 == 0; DROP DICTIONARY IF EXISTS db_dict.cache_hits; From d615b8e516569ddf69ad92cd3b73f6591c0b7248 Mon Sep 17 00:00:00 2001 From: Nikita Mikhailov Date: Mon, 15 Feb 2021 16:10:14 +0300 Subject: [PATCH 18/24] more checks (cherry picked from commit b45168ecaf37d0061edfd12c67a8c5300d45d2e3) --- src/Formats/JSONEachRowUtils.cpp | 11 ++++++++--- src/IO/BufferWithOwnMemory.h | 6 +++--- src/Processors/Formats/Impl/CSVRowInputFormat.cpp | 13 +++++++++---- .../Formats/Impl/RegexpRowInputFormat.cpp | 5 ++++- .../Formats/Impl/TabSeparatedRowInputFormat.cpp | 8 +++++--- 5 files changed, 29 insertions(+), 14 deletions(-) diff --git a/src/Formats/JSONEachRowUtils.cpp b/src/Formats/JSONEachRowUtils.cpp index 56bef9e09ea..28ba625d9fb 100644 --- a/src/Formats/JSONEachRowUtils.cpp +++ b/src/Formats/JSONEachRowUtils.cpp @@ -6,6 +6,7 @@ namespace DB namespace ErrorCodes { extern const int INCORRECT_DATA; + extern const int LOGICAL_ERROR; } std::pair fileSegmentationEngineJSONEachRowImpl(ReadBuffer & in, DB::Memory<> & memory, size_t min_chunk_size) @@ -28,7 +29,9 @@ std::pair fileSegmentationEngineJSONEachRowImpl(ReadBuffer & in, D if (quotes) { pos = find_first_symbols<'\\', '"'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; if (*pos == '\\') { @@ -45,9 +48,11 @@ std::pair fileSegmentationEngineJSONEachRowImpl(ReadBuffer & in, D else { pos = find_first_symbols<'{', '}', '\\', '"'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; - if (*pos == '{') + else if (*pos == '{') { ++balance; ++pos; diff --git a/src/IO/BufferWithOwnMemory.h b/src/IO/BufferWithOwnMemory.h index 782eea84ed7..f8cc8b7febb 100644 --- a/src/IO/BufferWithOwnMemory.h +++ b/src/IO/BufferWithOwnMemory.h @@ -35,10 +35,10 @@ struct Memory : boost::noncopyable, Allocator char * m_data = nullptr; size_t alignment = 0; - Memory() {} + Memory() = default; /// If alignment != 0, then allocate memory aligned to specified value. - Memory(size_t size_, size_t alignment_ = 0) : m_capacity(size_), m_size(m_capacity), alignment(alignment_) + explicit Memory(size_t size_, size_t alignment_ = 0) : m_capacity(size_), m_size(m_capacity), alignment(alignment_) { alloc(); } @@ -140,7 +140,7 @@ protected: Memory<> memory; public: /// If non-nullptr 'existing_memory' is passed, then buffer will not create its own memory and will use existing_memory without ownership. - BufferWithOwnMemory(size_t size = DBMS_DEFAULT_BUFFER_SIZE, char * existing_memory = nullptr, size_t alignment = 0) + explicit BufferWithOwnMemory(size_t size = DBMS_DEFAULT_BUFFER_SIZE, char * existing_memory = nullptr, size_t alignment = 0) : Base(nullptr, 0), memory(existing_memory ? 0 : size, alignment) { Base::set(existing_memory ? existing_memory : memory.data(), size); diff --git a/src/Processors/Formats/Impl/CSVRowInputFormat.cpp b/src/Processors/Formats/Impl/CSVRowInputFormat.cpp index 8422f09e364..f7f08411dfa 100644 --- a/src/Processors/Formats/Impl/CSVRowInputFormat.cpp +++ b/src/Processors/Formats/Impl/CSVRowInputFormat.cpp @@ -15,6 +15,7 @@ namespace ErrorCodes { extern const int BAD_ARGUMENTS; extern const int INCORRECT_DATA; + extern const int LOGICAL_ERROR; } @@ -436,9 +437,11 @@ static std::pair fileSegmentationEngineCSVImpl(ReadBuffer & in, DB if (quotes) { pos = find_first_symbols<'"'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; - if (*pos == '"') + else if (*pos == '"') { ++pos; if (loadAtPosition(in, memory, pos) && *pos == '"') @@ -450,9 +453,11 @@ static std::pair fileSegmentationEngineCSVImpl(ReadBuffer & in, DB else { pos = find_first_symbols<'"', '\r', '\n'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; - if (*pos == '"') + else if (*pos == '"') { quotes = true; ++pos; diff --git a/src/Processors/Formats/Impl/RegexpRowInputFormat.cpp b/src/Processors/Formats/Impl/RegexpRowInputFormat.cpp index 6e14a1dc3c8..108f4d9d321 100644 --- a/src/Processors/Formats/Impl/RegexpRowInputFormat.cpp +++ b/src/Processors/Formats/Impl/RegexpRowInputFormat.cpp @@ -11,6 +11,7 @@ namespace ErrorCodes { extern const int INCORRECT_DATA; extern const int BAD_ARGUMENTS; + extern const int LOGICAL_ERROR; } RegexpRowInputFormat::RegexpRowInputFormat( @@ -182,7 +183,9 @@ static std::pair fileSegmentationEngineRegexpImpl(ReadBuffer & in, while (loadAtPosition(in, memory, pos) && need_more_data) { pos = find_first_symbols<'\n', '\r'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; // Support DOS-style newline ("\r\n") diff --git a/src/Processors/Formats/Impl/TabSeparatedRowInputFormat.cpp b/src/Processors/Formats/Impl/TabSeparatedRowInputFormat.cpp index 69a5e61caf2..96b01a5bd9b 100644 --- a/src/Processors/Formats/Impl/TabSeparatedRowInputFormat.cpp +++ b/src/Processors/Formats/Impl/TabSeparatedRowInputFormat.cpp @@ -15,6 +15,7 @@ namespace DB namespace ErrorCodes { extern const int INCORRECT_DATA; + extern const int LOGICAL_ERROR; } @@ -433,10 +434,11 @@ static std::pair fileSegmentationEngineTabSeparatedImpl(ReadBuffer { pos = find_first_symbols<'\\', '\r', '\n'>(pos, in.buffer().end()); - if (pos == in.buffer().end()) + if (pos > in.buffer().end()) + throw Exception("Position in buffer is out of bounds. There must be a bug.", ErrorCodes::LOGICAL_ERROR); + else if (pos == in.buffer().end()) continue; - - if (*pos == '\\') + else if (*pos == '\\') { ++pos; if (loadAtPosition(in, memory, pos)) From d08dcb1958a565ad62d2e688413c3942c20e91f6 Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com> Date: Mon, 15 Feb 2021 22:35:49 +0300 Subject: [PATCH 19/24] Update docs/en/operations/settings/settings.md --- docs/en/operations/settings/settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index f64c623415b..963f9fa18bd 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2598,7 +2598,7 @@ Sets the probability that the ClickHouse can start a trace for executed queries Possible values: -- 0 — The trace for a executed queries is disabled (if no parent trace context is supplied). +- 0 — The trace for all executed queries is disabled (if no parent trace context is supplied). - Positive floating-point number in the range [0..1]. For example, if the setting value is `0,5`, ClickHouse can start a trace on average for half of the queries. - 1 — The trace for all executed queries is enabled. From 7f21a216941ae6557e8ac5f75d9093635ec71919 Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com> Date: Mon, 15 Feb 2021 22:40:55 +0300 Subject: [PATCH 20/24] Update index.md --- docs/en/sql-reference/window-functions/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en/sql-reference/window-functions/index.md b/docs/en/sql-reference/window-functions/index.md index 46f7ed3824e..07a7f2f6978 100644 --- a/docs/en/sql-reference/window-functions/index.md +++ b/docs/en/sql-reference/window-functions/index.md @@ -15,6 +15,7 @@ ClickHouse supports the standard grammar for defining windows and window functio | Feature | Support or workaround | | --------| ----------| | ad hoc window specification (`count(*) over (partition by id order by time desc)`) | yes | +| expressions involving window functions, e.g. `(count(*) over ()) / 2)` | not supported, wrap in a subquery ([feature request](https://github.com/ClickHouse/ClickHouse/issues/19857)) | | `WINDOW` clause (`select ... from table window w as (partiton by id)`) | yes | | `ROWS` frame | yes | | `RANGE` frame | yes, it is the default | From 2de6d550cc04d62c8189ca225c4016efe8c1847a Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com> Date: Mon, 15 Feb 2021 22:42:10 +0300 Subject: [PATCH 21/24] Update index.md --- docs/en/sql-reference/window-functions/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/en/sql-reference/window-functions/index.md b/docs/en/sql-reference/window-functions/index.md index 07a7f2f6978..0a19b4a8da4 100644 --- a/docs/en/sql-reference/window-functions/index.md +++ b/docs/en/sql-reference/window-functions/index.md @@ -14,15 +14,15 @@ ClickHouse supports the standard grammar for defining windows and window functio | Feature | Support or workaround | | --------| ----------| -| ad hoc window specification (`count(*) over (partition by id order by time desc)`) | yes | +| ad hoc window specification (`count(*) over (partition by id order by time desc)`) | supported | | expressions involving window functions, e.g. `(count(*) over ()) / 2)` | not supported, wrap in a subquery ([feature request](https://github.com/ClickHouse/ClickHouse/issues/19857)) | -| `WINDOW` clause (`select ... from table window w as (partiton by id)`) | yes | -| `ROWS` frame | yes | -| `RANGE` frame | yes, it is the default | -| `GROUPS` frame | no | +| `WINDOW` clause (`select ... from table window w as (partiton by id)`) | supported | +| `ROWS` frame | supported | +| `RANGE` frame | supported, the default | +| `GROUPS` frame | not supported | | Calculating aggregate functions over a frame (`sum(value) over (order by time)`) | all aggregate functions are supported | -| `rank()`, `dense_rank()`, `row_number()` | yes | -| `lag/lead(value, offset)` | no, replace with `any(value) over (.... rows between preceding and preceding)`, or `following` for `lead`| +| `rank()`, `dense_rank()`, `row_number()` | supported | +| `lag/lead(value, offset)` | not supported, replace with `any(value) over (.... rows between preceding and preceding)`, or `following` for `lead`| ## References From c9dd1aa58b831835a801bb886c77ccc712febcd9 Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov <36882414+akuzm@users.noreply.github.com> Date: Mon, 15 Feb 2021 22:56:26 +0300 Subject: [PATCH 22/24] Update index.md --- docs/en/sql-reference/window-functions/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en/sql-reference/window-functions/index.md b/docs/en/sql-reference/window-functions/index.md index 0a19b4a8da4..cbf03a44d46 100644 --- a/docs/en/sql-reference/window-functions/index.md +++ b/docs/en/sql-reference/window-functions/index.md @@ -19,6 +19,7 @@ ClickHouse supports the standard grammar for defining windows and window functio | `WINDOW` clause (`select ... from table window w as (partiton by id)`) | supported | | `ROWS` frame | supported | | `RANGE` frame | supported, the default | +| `INTERVAL` syntax for `DateTime` `RANGE OFFSET` frame | not supported, specify the number of seconds instead | | `GROUPS` frame | not supported | | Calculating aggregate functions over a frame (`sum(value) over (order by time)`) | all aggregate functions are supported | | `rank()`, `dense_rank()`, `row_number()` | supported | From e485d4cad8e21e721ad250f9117b5717a6d64fd7 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Tue, 16 Feb 2021 09:27:48 +0300 Subject: [PATCH 23/24] Fix SIGSEGV on Unknown packet for Distributed queries On Unknown packet disconnect() will be called, which will reset the input stream, so no need to call setAsyncCallback(): [ 42015 ] {} BaseDaemon: (version 21.3.1.1, build id: 4F9644AF560F6BB6) (from thread 45051) (no query) Received signal Segmentation fault (11) [ 42015 ] {} BaseDaemon: Address: 0x90 Access: read. Address not mapped to object. [ 42015 ] {} BaseDaemon: Stack trace: 0xf82e0f4 0xf82de19 0xf83b9a5 0xf83c0e0 0xe9a6fa7 0xf95016c 0xf950939 0xf95020c 0xf950939 0xf95020c 0xf950939 0xf95020c 0xf9508f9 0xf953e40 0xf958376 0x88056af 0x8809143 0x7f4b3e1aaf27 0x7f4b3e0dc31f [ 42015 ] {} BaseDaemon: 2. ext::basic_scope_guard)::$_3>::~basic_scope_guard() @ 0xf82e0f4 in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug [ 42015 ] {} BaseDaemon: 3. DB::Connection::receivePacket(std::__1::function) @ 0xf82de19 in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug [ 42015 ] {} BaseDaemon: 4. DB::MultiplexedConnections::receivePacketUnlocked(std::__1::function) @ 0xf83b9a5 in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug [ 42015 ] {} BaseDaemon: 5. DB::MultiplexedConnections::drain() @ 0xf83c0e0 in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug [ 42015 ] {} BaseDaemon: 6. DB::RemoteQueryExecutor::finish(std::__1::unique_ptr >*) @ 0xe9a6fa7 in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug [ 42015 ] {} BaseDaemon: 7. DB::PipelineExecutor::tryAddProcessorToStackIfUpdated() @ 0xf95016c in /usr/lib/debug/.build-id/4f/9644af560f6bb6.debug ... --- src/Client/Connection.cpp | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/Client/Connection.cpp b/src/Client/Connection.cpp index e38a6b240a6..164b9565633 100644 --- a/src/Client/Connection.cpp +++ b/src/Client/Connection.cpp @@ -756,7 +756,11 @@ std::optional Connection::checkPacket(size_t timeout_microseconds) Packet Connection::receivePacket(std::function async_callback) { in->setAsyncCallback(std::move(async_callback)); - SCOPE_EXIT(in->setAsyncCallback({})); + SCOPE_EXIT({ + /// disconnect() will reset "in". + if (in) + in->setAsyncCallback({}); + }); try { From 0b0b481825ba2e71074823d2d0bbce043e6e9b4f Mon Sep 17 00:00:00 2001 From: gyuton <40863448+gyuton@users.noreply.github.com> Date: Tue, 16 Feb 2021 14:02:33 +0300 Subject: [PATCH 24/24] DOCSUP-5602: Edited and translated to russian (#20302) * Edited and added translation * Minor fixes * Fix typo Co-authored-by: George Co-authored-by: Ivan <5627721+abyss7@users.noreply.github.com> --- .../sql-reference/functions/url-functions.md | 146 +++++++++++++++- .../sql-reference/functions/url-functions.md | 162 ++++++++++++++++++ 2 files changed, 299 insertions(+), 9 deletions(-) diff --git a/docs/en/sql-reference/functions/url-functions.md b/docs/en/sql-reference/functions/url-functions.md index 006542f494a..975695f40b3 100644 --- a/docs/en/sql-reference/functions/url-functions.md +++ b/docs/en/sql-reference/functions/url-functions.md @@ -133,10 +133,9 @@ For example: ### cutToFirstSignificantSubdomainCustom {#cuttofirstsignificantsubdomaincustom} -Same as `cutToFirstSignificantSubdomain` but accept custom TLD list name, useful if: +Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom [TLD list](https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains) name. -- you need fresh TLD list, -- or you have custom. +Can be useful if you need fresh TLD list or you have custom. Configuration example: @@ -149,21 +148,150 @@ Configuration example: ``` -Example: +**Syntax** -- `cutToFirstSignificantSubdomain('https://news.yandex.com.tr/', 'public_suffix_list') = 'yandex.com.tr'`. +``` sql +cutToFirstSignificantSubdomain(URL, TLD) +``` + +**Parameters** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — Custom TLD list name. [String](../../sql-reference/data-types/string.md). + +**Returned value** + +- Part of the domain that includes top-level subdomains up to the first significant subdomain. + +Type: [String](../../sql-reference/data-types/string.md). + +**Example** + +Query: + +```sql +SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +``` + +Result: + +```text +┌─cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐ +│ foo.there-is-no-such-domain │ +└───────────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +**See Also** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). ### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} -Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. +Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www`. Accepts custom TLD list name. + +Can be useful if you need fresh TLD list or you have custom. + +Configuration example: + +```xml + + + + public_suffix_list.dat + + +``` + +**Syntax** + +```sql +cutToFirstSignificantSubdomainCustomWithWWW(URL, TLD) +``` + +**Parameters** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — Custom TLD list name. [String](../../sql-reference/data-types/string.md). + +**Returned value** + +- Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www`. + +Type: [String](../../sql-reference/data-types/string.md). + +**Example** + +Query: + +```sql +SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list'); +``` + +Result: + +```text +┌─cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐ +│ www.foo │ +└──────────────────────────────────────────────────────────────────────────────┘ +``` + +**See Also** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). ### firstSignificantSubdomainCustom {#firstsignificantsubdomaincustom} -Same as `firstSignificantSubdomain` but accept custom TLD list name. +Returns the first significant subdomain. Accepts customs TLD list name. -### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} +Can be useful if you need fresh TLD list or you have custom. -Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. +Configuration example: + +```xml + + + + public_suffix_list.dat + + +``` + +**Syntax** + +```sql +firstSignificantSubdomainCustom(URL, TLD) +``` + +**Parameters** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — Custom TLD list name. [String](../../sql-reference/data-types/string.md). + +**Returned value** + +- First significant subdomain. + +Type: [String](../../sql-reference/data-types/string.md). + +**Example** + +Query: + +```sql +SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +``` + +Result: + +```text +┌─firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐ +│ foo │ +└──────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +**See Also** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). ### port(URL\[, default_port = 0\]) {#port} diff --git a/docs/ru/sql-reference/functions/url-functions.md b/docs/ru/sql-reference/functions/url-functions.md index 1008e2a359c..7541e16bed4 100644 --- a/docs/ru/sql-reference/functions/url-functions.md +++ b/docs/ru/sql-reference/functions/url-functions.md @@ -115,6 +115,168 @@ SELECT topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk') Например, `cutToFirstSignificantSubdomain('https://news.yandex.com.tr/') = 'yandex.com.tr'`. +### cutToFirstSignificantSubdomainCustom {#cuttofirstsignificantsubdomaincustom} + +Возвращает часть домена, включающую поддомены верхнего уровня до первого существенного поддомена. Принимает имя пользовательского [списка доменов верхнего уровня](https://ru.wikipedia.org/wiki/Список_доменов_верхнего_уровня). + +Полезно, если требуется актуальный список доменов верхнего уровня или если есть пользовательский. + +Пример конфигурации: + +```xml + + + + public_suffix_list.dat + + +``` + +**Синтаксис** + +``` sql +cutToFirstSignificantSubdomain(URL, TLD) +``` + +**Parameters** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — имя пользовательского списка доменов верхнего уровня. [String](../../sql-reference/data-types/string.md). + +**Возвращаемое значение** + +- Часть домена, включающая поддомены верхнего уровня до первого существенного поддомена. + +Тип: [String](../../sql-reference/data-types/string.md). + +**Пример** + +Запрос: + +```sql +SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +``` + +Результат: + +```text +┌─cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐ +│ foo.there-is-no-such-domain │ +└───────────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +**Смотрите также** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). + +### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} + +Возвращает часть домена, включающую поддомены верхнего уровня до первого существенного поддомена, не опуская "www". Принимает имя пользовательского списка доменов верхнего уровня. + +Полезно, если требуется актуальный список доменов верхнего уровня или если есть пользовательский. + +Пример конфигурации: + +```xml + + + + public_suffix_list.dat + + +``` + +**Синтаксис** + +```sql +cutToFirstSignificantSubdomainCustomWithWWW(URL, TLD) +``` + +**Параметры** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — имя пользовательского списка доменов верхнего уровня. [String](../../sql-reference/data-types/string.md). + +**Возвращаемое значение** + +- Часть домена, включающая поддомены верхнего уровня до первого существенного поддомена, без удаления `www`. + +Тип: [String](../../sql-reference/data-types/string.md). + +**Пример** + +Запрос: + +```sql +SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list'); +``` + +Результат: + +```text +┌─cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐ +│ www.foo │ +└──────────────────────────────────────────────────────────────────────────────┘ +``` + +**Смотрите также** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). + +### firstSignificantSubdomainCustom {#firstsignificantsubdomaincustom} + +Возвращает первый существенный поддомен. Принимает имя пользовательского списка доменов верхнего уровня. + +Полезно, если требуется актуальный список доменов верхнего уровня или если есть пользовательский. + +Пример конфигурации: + +```xml + + + + public_suffix_list.dat + + +``` + +**Синтаксис** + +```sql +firstSignificantSubdomainCustom(URL, TLD) +``` + +**Параметры** + +- `URL` — URL. [String](../../sql-reference/data-types/string.md). +- `TLD` — имя пользовательского списка доменов верхнего уровня. [String](../../sql-reference/data-types/string.md). + +**Возвращаемое значение** + +- Первый существенный поддомен. + +Тип: [String](../../sql-reference/data-types/string.md). + +**Пример** + +Запрос: + +```sql +SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +``` + +Результат: + +```text +┌─firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐ +│ foo │ +└──────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +**Смотрите также** + +- [firstSignificantSubdomain](#firstsignificantsubdomain). + ### port(URL[, default_port = 0]) {#port} Возвращает порт или значение `default_port`, если в URL-адресе нет порта (или передан невалидный URL)