thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-23 08:02:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into azure-profile-events

Browse Source
This commit is contained in:
Anton Popov 2024-04-11 16:38:08 +00:00
commit 17b3357e1d
142 changed files with 3327 additions and 822 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.github/workflows/master.yml vendored
View File

@ -23,6 +23,10 @@ jobs:
clear-repository: true # to ensure correct digests
fetch-depth: 0 # to get version
filter: tree:0
- name: Merge sync PR
run: |
cd "$GITHUB_WORKSPACE/tests/ci"
python3 sync_pr.py --merge || :
- name: Python unit tests
run: |
cd "$GITHUB_WORKSPACE/tests/ci"

13
.github/workflows/pull_request.yml vendored
View File

@ -157,16 +157,25 @@ jobs:
################################# Stage Final #################################
#
FinishCheck:
if: ${{ !failure() && !cancelled() && github.event_name != 'merge_group' }}
needs: [Tests_1, Tests_2]
if: ${{ !failure() && !cancelled() }}
needs: [Tests_1, Tests_2, Builds_1_Report, Builds_2_Report]
runs-on: [self-hosted, style-checker]
steps:
- name: Check out repository code
uses: ClickHouse/checkout@v1
- name: Check sync status
if: ${{ github.event_name == 'merge_group' }}
run: |
cd "$GITHUB_WORKSPACE/tests/ci"
python3 sync_pr.py --status
- name: Finish label
run: |
cd "$GITHUB_WORKSPACE/tests/ci"
python3 finish_check.py
- name: Auto merge if approved
if: ${{ github.event_name != 'merge_group' }}
run: |
cd "$GITHUB_WORKSPACE/tests/ci"
python3 merge_pr.py --check-approved

8
base/base/scope_guard.h
View File

@ -29,11 +29,13 @@ public:
requires std::is_convertible_v<G, F>
constexpr BasicScopeGuard & operator=(BasicScopeGuard<G> && src) // NOLINT(cppcoreguidelines-rvalue-reference-param-not-moved, cppcoreguidelines-noexcept-move-operations)
{
if (this != &src)
if constexpr (std::is_same_v<G, F>)
{
invoke();
function = src.release();
if (this == &src)
return *this;
}
invoke();
function = src.release();
return *this;
}

5
docker/test/fuzzer/query-fuzzer-tweaks-users.xml
View File

@ -26,6 +26,11 @@
<table_function_remote_max_addresses>
<max>200</max>
</table_function_remote_max_addresses>
<!-- Don't waste cycles testing the old interpreter. Spend time in the new analyzer instead -->
<allow_experimental_analyzer>
<readonly/>
</allow_experimental_analyzer>
</constraints>
</default>
</profiles>

13
docs/en/operations/server-configuration-parameters/settings.md
View File

@ -42,6 +42,19 @@ Type: UInt32
Default: 1
## auth_use_forwarded_address
Use originating address for authentication for clients connected through proxy.
:::note
This setting should be used with extra caution since forwarded address can be easily spoofed - server accepting such authentication should not be accessed directly but rather exclusively through a trusted proxy.
:::
Type: Bool
Default: 0
## background_buffer_flush_schedule_pool_size
The maximum number of threads that will be used for performing flush operations for Buffer-engine tables in the background.

155
docs/en/operations/settings/composable-protocols.md Normal file
View File

@ -0,0 +1,155 @@
---
slug: /en/operations/settings/composable-protocols
sidebar_position: 64
sidebar_label: Composable Protocols
---
# Composable Protocols
Composable protocols allows more flexible configuration of TCP access to the ClickHouse server. This configuration can co-exist with or replace conventional configuration.
## Composable protocols section is denoted as `protocols` in configuration xml
**Example:**
``` xml
<protocols>
</protocols>
```
## Basic modules define protocol layers
**Example:**
``` xml
<protocols>
<!-- plain_http module -->
<plain_http>
<type>http</type>
</plain_http>
</protocols>
```
where:
- `plain_http` - name which can be referred by another layer
- `type` - denotes protocol handler which will be instantiated to process data, set of protocol handlers is predefined:
* `tcp` - native clickhouse protocol handler
* `http` - http clickhouse protocol handler
* `tls` - TLS encryption layer
* `proxy1` - PROXYv1 layer
* `mysql` - MySQL compatibility protocol handler
* `postgres` - PostgreSQL compatibility protocol handler
* `prometheus` - Prometheus protocol handler
* `interserver` - clickhouse interserver handler
:::note
`gRPC` protocol handler is not implemented for `Composable protocols`
:::
## Endpoint (i.e. listening port) is denoted by `<port>` and (optional) `<host>` tags
**Example:**
``` xml
<protocols>
<plain_http>
<type>http</type>
<!-- endpoint -->
<host>127.0.0.1</host>
<port>8123</port>
</plain_http>
</protocols>
```
If `<host>` is omitted, then `<listen_host>` from root config is used.
## Layers sequence is defined by `<impl>` tag, referencing another module
**Example:** definition for HTTPS protocol
``` xml
<protocols>
<!-- http module -->
<plain_http>
<type>http</type>
</plain_http>
<!-- https module configured as a tls layer on top of plain_http module -->
<https>
<type>tls</type>
<impl>plain_http</impl>
<host>127.0.0.1</host>
<port>8443</port>
</https>
</protocols>
```
## Endpoint can be attached to any layer
**Example:** definition for HTTP (port 8123) and HTTPS (port 8443) endpoints
``` xml
<protocols>
<plain_http>
<type>http</type>
<host>127.0.0.1</host>
<port>8123</port>
</plain_http>
<https>
<type>tls</type>
<impl>plain_http</impl>
<host>127.0.0.1</host>
<port>8443</port>
</https>
</protocols>
```
## Additional endpoints can be defined by referencing any module and omitting `<type>` tag
**Example:** `another_http` endpoint is defined for `plain_http` module
``` xml
<protocols>
<plain_http>
<type>http</type>
<host>127.0.0.1</host>
<port>8123</port>
</plain_http>
<https>
<type>tls</type>
<impl>plain_http</impl>
<host>127.0.0.1</host>
<port>8443</port>
</https>
<another_http>
<impl>plain_http</impl>
<host>127.0.0.1</host>
<port>8223</port>
</another_http>
</protocols>
```
## Some modules can contain specific for its layer parameters
**Example:** for TLS layer private key (`privateKeyFile`) and certificate files (`certificateFile`) can be specified
``` xml
<protocols>
<plain_http>
<type>http</type>
<host>127.0.0.1</host>
<port>8123</port>
</plain_http>
<https>
<type>tls</type>
<impl>plain_http</impl>
<host>127.0.0.1</host>
<port>8443</port>
<privateKeyFile>another_server.key</privateKeyFile>
<certificateFile>another_server.crt</certificateFile>
</https>
</protocols>
```

2
docs/en/operations/system-tables/backup_log.md
View File

@ -9,6 +9,7 @@ Columns:
- `hostname` ([LowCardinality(String)](../../sql-reference/data-types/string.md)) — Hostname of the server executing the query.
- `event_date` ([Date](../../sql-reference/data-types/date.md)) — Date of the entry.
- `event_time` ([DateTime](../../sql-reference/data-types/datetime.md)) — The date and time of the entry.
- `event_time_microseconds` ([DateTime64](../../sql-reference/data-types/datetime64.md)) — Time of the entry with microseconds precision.
- `id` ([String](../../sql-reference/data-types/string.md)) — Identifier of the backup or restore operation.
- `name` ([String](../../sql-reference/data-types/string.md)) — Name of the backup storage (the contents of the `FROM` or `TO` clause).
@ -67,6 +68,7 @@ Row 2:
──────
hostname: clickhouse.eu-central1.internal
event_date: 2023-08-19
event_time: 2023-08-19 11:08:56
event_time_microseconds: 2023-08-19 11:08:56.916192
id: e5b74ecb-f6f1-426a-80be-872f90043885
name: Disk('backups_disk', '1.zip')

420
docs/en/sql-reference/functions/array-functions.md
View File

@ -774,6 +774,59 @@ Returns the number of elements for which `func(arr1[i], …, arrN[i])` returns s
Note that the `arrayCount` is a [higher-order function](../../sql-reference/functions/index.md#higher-order-functions). You can pass a lambda function to it as the first argument.
## arrayDotProduct
Returns the dot product of two arrays.
**Syntax**
```sql
arrayDotProduct(vector1, vector2)
```
Alias: `scalarProduct`, `dotProduct`
**Parameters**
- `vector1`: First vector. [Array](../data-types/array.md) or [Tuple](../data-types/tuple.md) of numeric values.
- `vector2`: Second vector. [Array](../data-types/array.md) or [Tuple](../data-types/tuple.md) of numeric values.
:::note
The sizes of the two vectors must be equal. Arrays and Tuples may also contain mixed element types.
:::
**Returned value**
- The dot product of the two vectors.
Type: numeric - determined by the type of the arguments. If Arrays or Tuples contain mixed element types then the result type is the supertype.
**Examples**
Query:
```sql
SELECT arrayDotProduct([1, 2, 3], [4, 5, 6]) AS res, toTypeName(res);
```
Result:
```response
32 UInt16
```
Query:
```sql
SELECT dotProduct((1::UInt16, 2::UInt8, 3::Float32),(4::Int16, 5::Float32, 6::UInt8)) AS res, toTypeName(res);
```
Result:
```response
32 Float64
```
## countEqual(arr, x)
Returns the number of elements in the array equal to x. Equivalent to arrayCount (elem -\> elem = x, arr).
@ -888,6 +941,66 @@ SELECT arrayEnumerateUniq([1, 1, 1, 2, 2, 2], [1, 1, 2, 1, 1, 2]) AS res
This is necessary when using ARRAY JOIN with a nested data structure and further aggregation across multiple elements in this structure.
## arrayEnumerateUniqRanked
Returns an array the same size as the source array, indicating for each element what its position is among elements with the same value. It allows for enumeration of a multidimensional array with the ability to specify how deep to look inside the array.
**Syntax**
```sql
arrayEnumerateUniqRanked(clear_depth, arr, max_array_depth)
```
**Parameters**
- `clear_depth`: Enumerate elements at the specified level separately. Positive [Integer](../data-types/int-uint.md) less than or equal to `max_arr_depth`.
- `arr`: N-dimensional array to enumerate. [Array](../data-types/array.md).
- `max_array_depth`: The maximum effective depth. Positive [Integer](../data-types/int-uint.md) less than or equal to the depth of `arr`.
**Example**
With `clear_depth=1` and `max_array_depth=1`, the result of `arrayEnumerateUniqRanked` is identical to that which [`arrayEnumerateUniq`](#arrayenumerateuniqarr) would give for the same array.
Query:
``` sql
SELECT arrayEnumerateUniqRanked(1, [1,2,1], 1);
```
Result:
``` text
[1,1,2]
```
In this example, `arrayEnumerateUniqRanked` is used to obtain an array indicating, for each element of the multidimensional array, what its position is among elements of the same value. For the first row of the passed array,`[1,2,3]`, the corresponding result is `[1,1,1]`, indicating that this is the first time `1`,`2` and `3` are encountered. For the second row of the provided array,`[2,2,1]`, the corresponding result is `[2,3,3]`, indicating that `2` is encountered for a second and third time, and `1` is encountered for the second time. Likewise, for the third row of the provided array `[3]` the corresponding result is `[2]` indicating that `3` is encountered for the second time.
Query:
``` sql
SELECT arrayEnumerateUniqRanked(1, [[1,2,3],[2,2,1],[3]], 2);
```
Result:
``` text
[[1,1,1],[2,3,2],[2]]
```
Changing `clear_depth=2`, results in elements being enumerated separately for each row.
Query:
``` sql
SELECT arrayEnumerateUniqRanked(2, [[1,2,3],[2,2,1],[3]], 2);
```
Result:
``` text
[[1,1,1],[1,2,1],[1]]
```
## arrayPopBack
Removes the last item from the array.
@ -1303,6 +1416,125 @@ SELECT arrayReverseSort((x, y) -> -y, [4, 3, 5], [1, 2, 3]) AS res;
Same as `arrayReverseSort` with additional `limit` argument allowing partial sorting. Returns an array of the same size as the original array where elements in range `[1..limit]` are sorted in descending order. Remaining elements `(limit..N]` shall contain elements in unspecified order.
## arrayShuffle
Returns an array of the same size as the original array containing the elements in shuffled order.
Elements are reordered in such a way that each possible permutation of those elements has equal probability of appearance.
**Syntax**
```sql
arrayShuffle(arr[, seed])
```
**Parameters**
- `arr`: The array to partially shuffle. [Array](../data-types/array.md).
- `seed` (optional): seed to be used with random number generation. If not provided a random one is used. [UInt or Int](../data-types/int-uint.md).
**Returned value**
- Array with elements shuffled.
**Implementation details**
:::note
This function will not materialize constants.
:::
**Examples**
In this example, `arrayShuffle` is used without providing a `seed` and will therefore generate one randomly itself.
Query:
```sql
SELECT arrayShuffle([1, 2, 3, 4]);
```
Note: when using [ClickHouse Fiddle](https://fiddle.clickhouse.com/), the exact response may differ due to random nature of the function.
Result:
```response
[1,4,2,3]
```
In this example, `arrayShuffle` is provided a `seed` and will produce stable results.
Query:
```sql
SELECT arrayShuffle([1, 2, 3, 4], 41);
```
Result:
```response
[3,2,1,4]
```
## arrayPartialShuffle
Given an input array of cardinality `N`, returns an array of size N where elements in the range `[1...limit]` are shuffled and the remaining elements in the range `(limit...n]` are unshuffled.
**Syntax**
```sql
arrayPartialShuffle(arr[, limit[, seed]])
```
**Parameters**
- `arr`: The array size `N` to partially shuffle. [Array](../data-types/array.md).
- `limit` (optional): The number to limit element swaps to, in the range `[1..N]`. [UInt or Int](../data-types/int-uint.md).
- `seed` (optional): The seed value to be used with random number generation. If not provided a random one is used. [UInt or Int](../data-types/int-uint.md)
**Returned value**
- Array with elements partially shuffled.
**Implementation details**
:::note
This function will not materialize constants.
The value of `limit` should be in the range `[1..N]`. Values outside of that range are equivalent to performing full [arrayShuffle](#arrayshuffle).
:::
**Examples**
Note: when using [ClickHouse Fiddle](https://fiddle.clickhouse.com/), the exact response may differ due to random nature of the function.
Query:
```sql
SELECT arrayPartialShuffle([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 1)
```
Result:
The order of elements is preserved (`[2,3,4,5], [7,8,9,10]`) except for the two shuffled elements `[1, 6]`. No `seed` is provided so the function selects its own randomly.
```response
[6,2,3,4,5,1,7,8,9,10]
```
In this example, the `limit` is increased to `2` and a `seed` value is provided. The order
Query:
```sql
SELECT arrayPartialShuffle([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 2);
```
The order of elements is preserved (`[4, 5, 6, 7, 8], [10]`) except for the four shuffled elements `[1, 2, 3, 9]`.
Result:
```response
[3,9,1,4,5,6,7,8,2,10]
```
## arrayUniq(arr, …)
If one argument is passed, it counts the number of different elements in the array.
@ -1400,21 +1632,91 @@ Result:
└────────────────────────────────┘
```
## arrayEnumerateDense(arr)
## arrayEnumerateDense
Returns an array of the same size as the source array, indicating where each element first appears in the source array.
Example:
**Syntax**
```sql
arrayEnumerateDense(arr)
```
**Example**
Query:
``` sql
SELECT arrayEnumerateDense([10, 20, 10, 30])
```
Result:
``` text
┌─arrayEnumerateDense([10, 20, 10, 30])─┐
│ [1,2,1,3] │
└───────────────────────────────────────┘
```
## arrayEnumerateDenseRanked
Returns an array the same size as the source array, indicating where each element first appears in the source array. It allows for enumeration of a multidimensional array with the ability to specify how deep to look inside the array.
**Syntax**
```sql
arrayEnumerateDenseRanked(clear_depth, arr, max_array_depth)
```
**Parameters**
- `clear_depth`: Enumerate elements at the specified level separately. Positive [Integer](../data-types/int-uint.md) less than or equal to `max_arr_depth`.
- `arr`: N-dimensional array to enumerate. [Array](../data-types/array.md).
- `max_array_depth`: The maximum effective depth. Positive [Integer](../data-types/int-uint.md) less than or equal to the depth of `arr`.
**Example**
With `clear_depth=1` and `max_array_depth=1`, the result is identical to what [arrayEnumerateDense](#arrayenumeratedense) would give.
Query:
``` sql
SELECT arrayEnumerateDenseRanked(1,[10, 20, 10, 30],1);
```
Result:
``` text
[1,2,1,3]
```
In this example, `arrayEnumerateDenseRanked` is used to obtain an array indicating, for each element of the multidimensional array, what its position is among elements of the same value. For the first row of the passed array,`[10,10,30,20]`, the corresponding first row of the result is `[1,1,2,3]`, indicating that `10` is the first number encountered in position 1 and 2, `30` the second number encountered in position 3 and `20` is the third number encountered in position 4. For the second row, `[40, 50, 10, 30]`, the corresponding second row of the result is `[4,5,1,2]`, indicating that `40` and `50` are the fourth and fifth numbers encountered in position 1 and 2 of that row, that another `10` (the first encountered number) is in position 3 and `30` (the second number encountered) is in the last position.
Query:
``` sql
SELECT arrayEnumerateDenseRanked(1,[[10,10,30,20],[40,50,10,30]],2);
```
Result:
``` text
[[1,1,2,3],[4,5,1,2]]
```
Changing `clear_depth=2` results in the enumeration occurring separately for each row anew.
Query:
``` sql
SELECT arrayEnumerateDenseRanked(2,[[10,10,30,20],[40,50,10,30]],2);
```
Result:
``` text
[[1,1,2,3],[1,2,3,4]]
```
## arrayIntersect(arr)
@ -1652,7 +1954,7 @@ flatten(array_of_arrays)
Alias: `flatten`.
**Arguments**
**Parameters**
- `array_of_arrays` — [Array](../../sql-reference/data-types/array.md) of arrays. For example, `[[1,2,3], [4,5]]`.
@ -1928,7 +2230,67 @@ Note that the `arrayAll` is a [higher-order function](../../sql-reference/functi
Returns the first element in the `arr1` array for which `func(arr1[i], …, arrN[i])` returns something other than 0.
Note that the `arrayFirst` is a [higher-order function](../../sql-reference/functions/index.md#higher-order-functions). You must pass a lambda function to it as the first argument, and it can’t be omitted.
## arrayFirstOrNull
Returns the first element in the `arr1` array for which `func(arr1[i], …, arrN[i])` returns something other than 0, otherwise it returns `NULL`.
**Syntax**
```sql
arrayFirstOrNull(func, arr1, …)
```
**Parameters**
- `func`: Lambda function. [Lambda function](../functions/#higher-order-functions---operator-and-lambdaparams-expr-function).
- `arr1`: Array to operate on. [Array](../data-types/array.md).
**Returned value**
- The first element in the passed array.
- Otherwise, returns `NULL`
**Implementation details**
Note that the `arrayFirstOrNull` is a [higher-order function](../../sql-reference/functions/index.md#higher-order-functions). You must pass a lambda function to it as the first argument, and it can’t be omitted.
**Example**
Query:
```sql
SELECT arrayFirstOrNull(x -> x >= 2, [1, 2, 3]);
```
Result:
```response
2
```
Query:
```sql
SELECT arrayFirstOrNull(x -> x >= 2, emptyArrayUInt8());
```
Result:
```response
\N
```
Query:
```sql
SELECT arrayLastOrNull((x,f) -> f, [1,2,3,NULL], [0,1,0,1]);
```
Result:
```response
\N
```
## arrayLast(func, arr1, …)
@ -1936,6 +2298,56 @@ Returns the last element in the `arr1` array for which `func(arr1[i], …, arrN[
Note that the `arrayLast` is a [higher-order function](../../sql-reference/functions/index.md#higher-order-functions). You must pass a lambda function to it as the first argument, and it can’t be omitted.
## arrayLastOrNull
Returns the last element in the `arr1` array for which `func(arr1[i], …, arrN[i])` returns something other than 0, otherwise returns `NULL`.
**Syntax**
```sql
arrayLastOrNull(func, arr1, …)
```
**Parameters**
- `func`: Lambda function. [Lambda function](../functions/#higher-order-functions---operator-and-lambdaparams-expr-function).
- `arr1`: Array to operate on. [Array](../data-types/array.md).
**Returned value**
- The last element in the passed array.
- Otherwise, returns `NULL`
**Implementation details**
Note that the `arrayLastOrNull` is a [higher-order function](../../sql-reference/functions/index.md#higher-order-functions). You must pass a lambda function to it as the first argument, and it can’t be omitted.
**Example**
Query:
```sql
SELECT arrayLastOrNull(x -> x >= 2, [1, 2, 3]);
```
Result:
```response
3
```
Query:
```sql
SELECT arrayLastOrNull(x -> x >= 2, emptyArrayUInt8());
```
Result:
```response
\N
```
## arrayFirstIndex(func, arr1, …)
Returns the index of the first element in the `arr1` array for which `func(arr1[i], …, arrN[i])` returns something other than 0.

4
docs/en/sql-reference/functions/date-time-functions.md
View File

@ -1906,7 +1906,7 @@ Aliases: `dateAdd`, `DATE_ADD`.
**Arguments**
- `unit` — The type of interval to add. [String](../../sql-reference/data-types/string.md).
- `unit` — The type of interval to add. Note: This is not a [String](../../sql-reference/data-types/string.md) and must therefore not be quoted.
Possible values:
- `second`
@ -1961,7 +1961,7 @@ Aliases: `dateSub`, `DATE_SUB`.
**Arguments**
- `unit` — The type of interval to subtract. Note: The unit should be unquoted.
- `unit` — The type of interval to subtract. Note: This is not a [String](../../sql-reference/data-types/string.md) and must therefore not be quoted.
Possible values:

36
docs/en/sql-reference/functions/string-functions.md
View File

@ -254,8 +254,36 @@ Result:
Converts the ASCII Latin symbols in a string to lowercase.
*Syntax**
``` sql
lower(input)
```
Alias: `lcase`
**Parameters**
- `input`: A string type [String](/docs/en/sql-reference/data-types/string.md).
**Returned value**
- A [String](/docs/en/sql-reference/data-types/string.md) data type value.
**Example**
Query:
```sql
SELECT lower('CLICKHOUSE');
```
```response
┌─lower('CLICKHOUSE')─┐
│ clickhouse │
└─────────────────────┘
```
## upper
Converts the ASCII Latin symbols in a string to uppercase.
@ -281,13 +309,13 @@ Alias: `ucase`
Query:
``` sql
SELECT upper('value') as Upper;
SELECT upper('clickhouse');
```
``` response
┌─Upper─┐
│ VALUE │
└───────┘
┌─upper('clickhouse')─┐
│ CLICKHOUSE │
└─────────────────────┘
```
## lowerUTF8

39
docs/en/sql-reference/functions/tuple-functions.md
View File

@ -521,45 +521,6 @@ Result:
└──────────────────────────────────┘
```
## dotProduct
Calculates the scalar product of two tuples of the same size.
**Syntax**
```sql
dotProduct(tuple1, tuple2)
```
Alias: `scalarProduct`.
**Arguments**
- `tuple1` — First tuple. [Tuple](../../sql-reference/data-types/tuple.md).
- `tuple2` — Second tuple. [Tuple](../../sql-reference/data-types/tuple.md).
**Returned value**
- Scalar product.
Type: [Int/UInt](../../sql-reference/data-types/int-uint.md) or [Float](../../sql-reference/data-types/float.md).
**Example**
Query:
```sql
SELECT dotProduct((1, 2), (2, 3));
```
Result:
```text
┌─dotProduct((1, 2), (2, 3))─┐
│ 8 │
└────────────────────────────┘
```
## tupleConcat
Combines tuples passed as arguments.

6
programs/client/Client.cpp
View File

@ -687,7 +687,11 @@ bool Client::processWithFuzzing(const String & full_query)
try
{
const char * begin = full_query.data();
orig_ast = parseQuery(begin, begin + full_query.size(), true);
orig_ast = parseQuery(begin, begin + full_query.size(),
global_context->getSettingsRef(),
/*allow_multi_statements=*/ true,
/*is_interactive=*/ is_interactive,
/*ignore_error=*/ ignore_error);
}
catch (const Exception & e)
{

21
programs/install/Install.cpp
View File

@ -46,12 +46,12 @@ INCBIN(resource_users_xml, SOURCE_DIR "/programs/server/users.xml");
*
* The following steps are performed:
*
* - copying the binary to binary directory (/usr/bin).
* - copying the binary to binary directory (/usr/bin/)
* - creation of symlinks for tools.
* - creation of clickhouse user and group.
* - creation of config directory (/etc/clickhouse-server).
* - creation of config directory (/etc/clickhouse-server/).
* - creation of default configuration files.
* - creation of a directory for logs (/var/log/clickhouse-server).
* - creation of a directory for logs (/var/log/clickhouse-server/).
* - creation of a data directory if not exists.
* - setting a password for default user.
* - choose an option to listen connections.
@ -226,7 +226,12 @@ int mainEntryClickHouseInstall(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
/// https://stackoverflow.com/a/36734569/22422288
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "where to install binaries")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "where to install binaries")
#endif
("config-path", po::value<std::string>()->default_value("etc/clickhouse-server"), "where to install configs")
("log-path", po::value<std::string>()->default_value("var/log/clickhouse-server"), "where to create log directory")
("data-path", po::value<std::string>()->default_value("var/lib/clickhouse"), "directory for data")
@ -1216,7 +1221,12 @@ int mainEntryClickHouseStart(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
/// https://stackoverflow.com/a/36734569/22422288
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "directory with binary")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "directory with binary")
#endif
("config-path", po::value<std::string>()->default_value("etc/clickhouse-server"), "directory with configs")
("pid-path", po::value<std::string>()->default_value("var/run/clickhouse-server"), "directory for pid file")
("user", po::value<std::string>()->default_value(DEFAULT_CLICKHOUSE_SERVER_USER), "clickhouse user")
@ -1332,7 +1342,12 @@ int mainEntryClickHouseRestart(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
/// https://stackoverflow.com/a/36734569/22422288
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "directory with binary")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "directory with binary")
#endif
("config-path", po::value<std::string>()->default_value("etc/clickhouse-server"), "directory with configs")
("pid-path", po::value<std::string>()->default_value("var/run/clickhouse-server"), "directory for pid file")
("user", po::value<std::string>()->default_value(DEFAULT_CLICKHOUSE_SERVER_USER), "clickhouse user")

14
programs/local/LocalServer.cpp
View File

@ -413,8 +413,20 @@ void LocalServer::setupUsers()
void LocalServer::connect()
{
connection_parameters = ConnectionParameters(config(), "localhost");
ReadBuffer * in;
auto table_file = config().getString("table-file", "-");
if (table_file == "-" || table_file == "stdin")
{
in = &std_in;
}
else
{
input = std::make_unique<ReadBufferFromFile>(table_file);
in = input.get();
}
connection = LocalConnection::createConnection(
connection_parameters, global_context, need_render_progress, need_render_profile_events, server_display_name);
connection_parameters, global_context, in, need_render_progress, need_render_profile_events, server_display_name);
}

2
programs/local/LocalServer.h
View File

@ -65,6 +65,8 @@ private:
std::optional<StatusFile> status;
std::optional<std::filesystem::path> temporary_directory_to_delete;
std::unique_ptr<ReadBufferFromFile> input;
};
}

4
src/Analyzer/ArrayJoinNode.cpp
View File

@ -31,13 +31,13 @@ void ArrayJoinNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_stat
getJoinExpressionsNode()->dumpTreeImpl(buffer, format_state, indent + 4);
}
bool ArrayJoinNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool ArrayJoinNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const ArrayJoinNode &>(rhs);
return is_left == rhs_typed.is_left;
}
void ArrayJoinNode::updateTreeHashImpl(HashState & state) const
void ArrayJoinNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(is_left);
}

4
src/Analyzer/ArrayJoinNode.h
View File

@ -93,9 +93,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

18
src/Analyzer/ColumnNode.cpp
View File

@ -68,20 +68,26 @@ void ColumnNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & state, size_t
}
}
bool ColumnNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool ColumnNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const
{
const auto & rhs_typed = assert_cast<const ColumnNode &>(rhs);
return column == rhs_typed.column;
if (column.name != rhs_typed.column.name)
return false;
return !compare_options.compare_types || column.type->equals(*rhs_typed.column.type);
}
void ColumnNode::updateTreeHashImpl(HashState & hash_state) const
void ColumnNode::updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const
{
hash_state.update(column.name.size());
hash_state.update(column.name);
const auto & column_type_name = column.type->getName();
hash_state.update(column_type_name.size());
hash_state.update(column_type_name);
if (compare_options.compare_types)
{
const auto & column_type_name = column.type->getName();
hash_state.update(column_type_name.size());
hash_state.update(column_type_name);
}
}
QueryTreeNodePtr ColumnNode::cloneImpl() const

4
src/Analyzer/ColumnNode.h
View File

@ -131,9 +131,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

12
src/Analyzer/ColumnTransformers.cpp
View File

@ -74,13 +74,13 @@ void ApplyColumnTransformerNode::dumpTreeImpl(WriteBuffer & buffer, FormatState
expression_node->dumpTreeImpl(buffer, format_state, indent + 4);
}
bool ApplyColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool ApplyColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const ApplyColumnTransformerNode &>(rhs);
return apply_transformer_type == rhs_typed.apply_transformer_type;
}
void ApplyColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const
void ApplyColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const
{
hash_state.update(static_cast<size_t>(getTransformerType()));
hash_state.update(static_cast<size_t>(getApplyTransformerType()));
@ -178,7 +178,7 @@ void ExceptColumnTransformerNode::dumpTreeImpl(WriteBuffer & buffer, FormatState
}
}
bool ExceptColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool ExceptColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const ExceptColumnTransformerNode &>(rhs);
if (except_transformer_type != rhs_typed.except_transformer_type ||
@ -198,7 +198,7 @@ bool ExceptColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs) const
return column_matcher->pattern() == rhs_column_matcher->pattern();
}
void ExceptColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const
void ExceptColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const
{
hash_state.update(static_cast<size_t>(getTransformerType()));
hash_state.update(static_cast<size_t>(getExceptTransformerType()));
@ -302,13 +302,13 @@ void ReplaceColumnTransformerNode::dumpTreeImpl(WriteBuffer & buffer, FormatStat
}
}
bool ReplaceColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool ReplaceColumnTransformerNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const ReplaceColumnTransformerNode &>(rhs);
return is_strict == rhs_typed.is_strict && replacements_names == rhs_typed.replacements_names;
}
void ReplaceColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const
void ReplaceColumnTransformerNode::updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const
{
hash_state.update(static_cast<size_t>(getTransformerType()));

12
src/Analyzer/ColumnTransformers.h
View File

@ -137,9 +137,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;
@ -214,9 +214,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;
@ -290,9 +290,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state) const override;
void updateTreeHashImpl(IQueryTreeNode::HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

26
src/Analyzer/ConstantNode.cpp
View File

@ -126,17 +126,29 @@ void ConstantNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state
}
}
bool ConstantNode::isEqualImpl(const IQueryTreeNode & rhs) const
void ConstantNode::convertToNullable()
{
const auto & rhs_typed = assert_cast<const ConstantNode &>(rhs);
return *constant_value == *rhs_typed.constant_value && value_string == rhs_typed.value_string;
constant_value = std::make_shared<ConstantValue>(constant_value->getValue(), makeNullableSafe(constant_value->getType()));
}
void ConstantNode::updateTreeHashImpl(HashState & hash_state) const
bool ConstantNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const
{
auto type_name = constant_value->getType()->getName();
hash_state.update(type_name.size());
hash_state.update(type_name);
const auto & rhs_typed = assert_cast<const ConstantNode &>(rhs);
if (value_string != rhs_typed.value_string || constant_value->getValue() != rhs_typed.constant_value->getValue())
return false;
return !compare_options.compare_types || constant_value->getType()->equals(*rhs_typed.constant_value->getType());
}
void ConstantNode::updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const
{
if (compare_options.compare_types)
{
auto type_name = constant_value->getType()->getName();
hash_state.update(type_name.size());
hash_state.update(type_name);
}
hash_state.update(value_string.size());
hash_state.update(value_string);

9
src/Analyzer/ConstantNode.h
View File

@ -87,17 +87,14 @@ public:
mask_id = id;
}
void convertToNullable() override
{
constant_value = std::make_shared<ConstantValue>(constant_value->getValue(), makeNullableSafe(constant_value->getType()));
}
void convertToNullable() override;
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const override;
QueryTreeNodePtr cloneImpl() const override;

10
src/Analyzer/ConstantValue.h
View File

@ -34,14 +34,4 @@ private:
DataTypePtr data_type;
};
inline bool operator==(const ConstantValue & lhs, const ConstantValue & rhs)
{
return lhs.getValue() == rhs.getValue() && lhs.getType()->equals(*rhs.getType());
}
inline bool operator!=(const ConstantValue & lhs, const ConstantValue & rhs)
{
return !(lhs == rhs);
}
}

10
src/Analyzer/FunctionNode.cpp
View File

@ -142,7 +142,7 @@ void FunctionNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state
}
}
bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const
{
const auto & rhs_typed = assert_cast<const FunctionNode &>(rhs);
if (function_name != rhs_typed.function_name || isAggregateFunction() != rhs_typed.isAggregateFunction()
@ -150,6 +150,9 @@ bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs) const
|| nulls_action != rhs_typed.nulls_action)
return false;
if (!compare_options.compare_types)
return true;
if (isResolved() != rhs_typed.isResolved())
return false;
if (!isResolved())
@ -168,7 +171,7 @@ bool FunctionNode::isEqualImpl(const IQueryTreeNode & rhs) const
return true;
}
void FunctionNode::updateTreeHashImpl(HashState & hash_state) const
void FunctionNode::updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const
{
hash_state.update(function_name.size());
hash_state.update(function_name);
@ -177,6 +180,9 @@ void FunctionNode::updateTreeHashImpl(HashState & hash_state) const
hash_state.update(isWindowFunction());
hash_state.update(nulls_action);
if (!compare_options.compare_types)
return;
if (!isResolved())
return;

4
src/Analyzer/FunctionNode.h
View File

@ -208,9 +208,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const override;
QueryTreeNodePtr cloneImpl() const override;

24
src/Analyzer/HashUtils.h
View File

@ -11,37 +11,37 @@ namespace DB
* Example of usage:
* std::unordered_map<QueryTreeNodeConstRawPtrWithHash, std::string> map;
*/
template <typename QueryTreeNodePtrType, bool compare_aliases = true>
template <typename QueryTreeNodePtrType, bool compare_aliases = true, bool compare_types = true>
struct QueryTreeNodeWithHash
{
QueryTreeNodeWithHash(QueryTreeNodePtrType node_) /// NOLINT
: node(std::move(node_))
, hash(node->getTreeHash({.compare_aliases = compare_aliases}))
, hash(node->getTreeHash({.compare_aliases = compare_aliases, .compare_types = compare_types}))
{}
QueryTreeNodePtrType node = nullptr;
CityHash_v1_0_2::uint128 hash;
};
template <typename T, bool compare_aliases>
inline bool operator==(const QueryTreeNodeWithHash<T, compare_aliases> & lhs, const QueryTreeNodeWithHash<T, compare_aliases> & rhs)
template <typename T, bool compare_aliases, bool compare_types>
inline bool operator==(const QueryTreeNodeWithHash<T, compare_aliases, compare_types> & lhs, const QueryTreeNodeWithHash<T, compare_aliases, compare_types> & rhs)
{
return lhs.hash == rhs.hash && lhs.node->isEqual(*rhs.node, {.compare_aliases = compare_aliases});
return lhs.hash == rhs.hash && lhs.node->isEqual(*rhs.node, {.compare_aliases = compare_aliases, .compare_types = compare_types});
}
template <typename T, bool compare_aliases>
inline bool operator!=(const QueryTreeNodeWithHash<T, compare_aliases> & lhs, const QueryTreeNodeWithHash<T, compare_aliases> & rhs)
template <typename T, bool compare_aliases, bool compare_types>
inline bool operator!=(const QueryTreeNodeWithHash<T, compare_aliases, compare_types> & lhs, const QueryTreeNodeWithHash<T, compare_aliases, compare_types> & rhs)
{
return !(lhs == rhs);
}
using QueryTreeNodePtrWithHash = QueryTreeNodeWithHash<QueryTreeNodePtr>;
using QueryTreeNodePtrWithHashWithoutAlias = QueryTreeNodeWithHash<QueryTreeNodePtr, /*compare_aliases*/ false>;
using QueryTreeNodePtrWithHashIgnoreTypes = QueryTreeNodeWithHash<QueryTreeNodePtr, /*compare_aliases*/ false, /*compare_types*/ false>;
using QueryTreeNodeRawPtrWithHash = QueryTreeNodeWithHash<IQueryTreeNode *>;
using QueryTreeNodeConstRawPtrWithHash = QueryTreeNodeWithHash<const IQueryTreeNode *>;
using QueryTreeNodePtrWithHashSet = std::unordered_set<QueryTreeNodePtrWithHash>;
using QueryTreeNodePtrWithHashWithoutAliasSet = std::unordered_set<QueryTreeNodePtrWithHashWithoutAlias>;
using QueryTreeNodePtrWithHashIgnoreTypesSet = std::unordered_set<QueryTreeNodePtrWithHashIgnoreTypes>;
using QueryTreeNodeConstRawPtrWithHashSet = std::unordered_set<QueryTreeNodeConstRawPtrWithHash>;
template <typename Value>
@ -52,10 +52,10 @@ using QueryTreeNodeConstRawPtrWithHashMap = std::unordered_map<QueryTreeNodeCons
}
template <typename T, bool compare_aliases>
struct std::hash<DB::QueryTreeNodeWithHash<T, compare_aliases>>
template <typename T, bool compare_aliases, bool compare_types>
struct std::hash<DB::QueryTreeNodeWithHash<T, compare_aliases, compare_types>>
{
size_t operator()(const DB::QueryTreeNodeWithHash<T, compare_aliases> & node_with_hash) const
size_t operator()(const DB::QueryTreeNodeWithHash<T, compare_aliases, compare_types> & node_with_hash) const
{
return node_with_hash.hash.low64;
}

4
src/Analyzer/IQueryTreeNode.cpp
View File

@ -107,7 +107,7 @@ bool IQueryTreeNode::isEqual(const IQueryTreeNode & rhs, CompareOptions compare_
}
if (lhs_node_to_compare->getNodeType() != rhs_node_to_compare->getNodeType() ||
!lhs_node_to_compare->isEqualImpl(*rhs_node_to_compare))
!lhs_node_to_compare->isEqualImpl(*rhs_node_to_compare, compare_options))
return false;
if (compare_options.compare_aliases && lhs_node_to_compare->alias != rhs_node_to_compare->alias)
@ -207,7 +207,7 @@ IQueryTreeNode::Hash IQueryTreeNode::getTreeHash(CompareOptions compare_options)
hash_state.update(node_to_process->alias);
}
node_to_process->updateTreeHashImpl(hash_state);
node_to_process->updateTreeHashImpl(hash_state, compare_options);
hash_state.update(node_to_process->children.size());

9
src/Analyzer/IQueryTreeNode.h
View File

@ -97,6 +97,7 @@ public:
struct CompareOptions
{
bool compare_aliases = true;
bool compare_types = true;
};
/** Is tree equal to other tree with node root.
@ -104,7 +105,7 @@ public:
* With default compare options aliases of query tree nodes are compared during isEqual call.
* Original ASTs of query tree nodes are not compared during isEqual call.
*/
bool isEqual(const IQueryTreeNode & rhs, CompareOptions compare_options = { .compare_aliases = true }) const;
bool isEqual(const IQueryTreeNode & rhs, CompareOptions compare_options = { .compare_aliases = true, .compare_types = true }) const;
using Hash = CityHash_v1_0_2::uint128;
using HashState = SipHash;
@ -114,7 +115,7 @@ public:
* Alias of query tree node is part of query tree hash.
* Original AST is not part of query tree hash.
*/
Hash getTreeHash(CompareOptions compare_options = { .compare_aliases = true }) const;
Hash getTreeHash(CompareOptions compare_options = { .compare_aliases = true, .compare_types = true }) const;
/// Get a deep copy of the query tree
QueryTreeNodePtr clone() const;
@ -264,12 +265,12 @@ protected:
/** Subclass must compare its internal state with rhs node internal state and do not compare children or weak pointers to other
* query tree nodes.
*/
virtual bool isEqualImpl(const IQueryTreeNode & rhs) const = 0;
virtual bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions compare_options) const = 0;
/** Subclass must update tree hash with its internal state and do not update tree hash for children or weak pointers to other
* query tree nodes.
*/
virtual void updateTreeHashImpl(HashState & hash_state) const = 0;
virtual void updateTreeHashImpl(HashState & hash_state, CompareOptions compare_options) const = 0;
/** Subclass must clone its internal state and do not clone children or weak pointers to other
* query tree nodes.

4
src/Analyzer/IdentifierNode.cpp
View File

@ -38,13 +38,13 @@ void IdentifierNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_sta
}
}
bool IdentifierNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool IdentifierNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const IdentifierNode &>(rhs);
return identifier == rhs_typed.identifier && table_expression_modifiers == rhs_typed.table_expression_modifiers;
}
void IdentifierNode::updateTreeHashImpl(HashState & state) const
void IdentifierNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
const auto & identifier_name = identifier.getFullName();
state.update(identifier_name.size());

4
src/Analyzer/IdentifierNode.h
View File

@ -53,9 +53,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/InterpolateNode.cpp
View File

@ -28,13 +28,13 @@ void InterpolateNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_st
getInterpolateExpression()->dumpTreeImpl(buffer, format_state, indent + 4);
}
bool InterpolateNode::isEqualImpl(const IQueryTreeNode &) const
bool InterpolateNode::isEqualImpl(const IQueryTreeNode &, CompareOptions) const
{
/// No state in interpolate node
return true;
}
void InterpolateNode::updateTreeHashImpl(HashState &) const
void InterpolateNode::updateTreeHashImpl(HashState &, CompareOptions) const
{
/// No state in interpolate node
}

4
src/Analyzer/InterpolateNode.h
View File

@ -53,9 +53,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/JoinNode.cpp
View File

@ -79,13 +79,13 @@ void JoinNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, si
}
}
bool JoinNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool JoinNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const JoinNode &>(rhs);
return locality == rhs_typed.locality && strictness == rhs_typed.strictness && kind == rhs_typed.kind;
}
void JoinNode::updateTreeHashImpl(HashState & state) const
void JoinNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(locality);
state.update(strictness);

4
src/Analyzer/JoinNode.h
View File

@ -142,9 +142,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/LambdaNode.cpp
View File

@ -46,13 +46,13 @@ void LambdaNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state,
getExpression()->dumpTreeImpl(buffer, format_state, indent + 4);
}
bool LambdaNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool LambdaNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const LambdaNode &>(rhs);
return argument_names == rhs_typed.argument_names;
}
void LambdaNode::updateTreeHashImpl(HashState & state) const
void LambdaNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(argument_names.size());
for (const auto & argument_name : argument_names)

4
src/Analyzer/LambdaNode.h
View File

@ -97,9 +97,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/ListNode.cpp
View File

@ -38,13 +38,13 @@ void ListNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, si
}
}
bool ListNode::isEqualImpl(const IQueryTreeNode &) const
bool ListNode::isEqualImpl(const IQueryTreeNode &, CompareOptions) const
{
/// No state
return true;
}
void ListNode::updateTreeHashImpl(HashState &) const
void ListNode::updateTreeHashImpl(HashState &, CompareOptions) const
{
/// No state
}

4
src/Analyzer/ListNode.h
View File

@ -51,9 +51,9 @@ public:
const_iterator end() const { return children.end(); }
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState &) const override;
void updateTreeHashImpl(HashState &, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/MatcherNode.cpp
View File

@ -160,7 +160,7 @@ void MatcherNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state,
}
}
bool MatcherNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool MatcherNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const MatcherNode &>(rhs);
if (matcher_type != rhs_typed.matcher_type ||
@ -181,7 +181,7 @@ bool MatcherNode::isEqualImpl(const IQueryTreeNode & rhs) const
return columns_matcher->pattern() == rhs_columns_matcher->pattern();
}
void MatcherNode::updateTreeHashImpl(HashState & hash_state) const
void MatcherNode::updateTreeHashImpl(HashState & hash_state, CompareOptions) const
{
hash_state.update(static_cast<size_t>(matcher_type));

4
src/Analyzer/MatcherNode.h
View File

@ -135,9 +135,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

50
src/Analyzer/Passes/CrossToInnerJoinPass.cpp
View File

@ -15,6 +15,7 @@
#include <Functions/logical.h>
#include <Common/logger_useful.h>
#include <Analyzer/Utils.h>
namespace DB
@ -61,47 +62,7 @@ const QueryTreeNodePtr & getEquiArgument(const QueryTreeNodePtr & cond, size_t i
return func->getArguments().getNodes()[index];
}
/// Check that node has only one source and return it.
/// {_, false} - multiple sources
/// {nullptr, true} - no sources
/// {source, true} - single source
std::pair<const IQueryTreeNode *, bool> getExpressionSource(const QueryTreeNodePtr & node)
{
if (const auto * column = node->as<ColumnNode>())
{
auto source = column->getColumnSourceOrNull();
if (!source)
return {nullptr, false};
return {source.get(), true};
}
if (const auto * func = node->as<FunctionNode>())
{
const IQueryTreeNode * source = nullptr;
const auto & args = func->getArguments().getNodes();
for (const auto & arg : args)
{
auto [arg_source, is_ok] = getExpressionSource(arg);
if (!is_ok)
return {nullptr, false};
if (!source)
source = arg_source;
else if (arg_source && !source->isEqual(*arg_source))
return {nullptr, false};
}
return {source, true};
}
if (node->as<ConstantNode>())
return {nullptr, true};
return {nullptr, false};
}
bool findInTableExpression(const IQueryTreeNode * source, const QueryTreeNodePtr & table_expression)
bool findInTableExpression(const QueryTreeNodePtr & source, const QueryTreeNodePtr & table_expression)
{
if (!source)
return true;
@ -115,7 +76,6 @@ bool findInTableExpression(const IQueryTreeNode * source, const QueryTreeNodePtr
|| findInTableExpression(source, join_node->getRightTableExpression());
}
return false;
}
@ -169,10 +129,10 @@ public:
auto left_src = getExpressionSource(lhs_equi_argument);
auto right_src = getExpressionSource(rhs_equi_argument);
if (left_src.second && right_src.second && left_src.first && right_src.first)
if (left_src && right_src)
{
if ((findInTableExpression(left_src.first, left_table) && findInTableExpression(right_src.first, right_table)) ||
(findInTableExpression(left_src.first, right_table) && findInTableExpression(right_src.first, left_table)))
if ((findInTableExpression(left_src, left_table) && findInTableExpression(right_src, right_table)) ||
(findInTableExpression(left_src, right_table) && findInTableExpression(right_src, left_table)))
{
can_convert_cross_to_inner = true;
continue;

23
src/Analyzer/Passes/LogicalExpressionOptimizerPass.cpp
View File

@ -25,8 +25,9 @@ class JoinOnLogicalExpressionOptimizerVisitor : public InDepthQueryTreeVisitorWi
public:
using Base = InDepthQueryTreeVisitorWithContext<JoinOnLogicalExpressionOptimizerVisitor>;
explicit JoinOnLogicalExpressionOptimizerVisitor(ContextPtr context)
explicit JoinOnLogicalExpressionOptimizerVisitor(const JoinNode * join_node_, ContextPtr context)
: Base(std::move(context))
, join_node(join_node_)
{}
void enterImpl(QueryTreeNodePtr & node)
@ -55,10 +56,11 @@ public:
}
private:
const JoinNode * join_node;
bool need_rerun_resolve = false;
/// Returns true if type of some operand is changed and parent function needs to be re-resolved
static bool tryOptimizeIsNotDistinctOrIsNull(QueryTreeNodePtr & node, const ContextPtr & context)
bool tryOptimizeIsNotDistinctOrIsNull(QueryTreeNodePtr & node, const ContextPtr & context)
{
auto & function_node = node->as<FunctionNode &>();
chassert(function_node.getFunctionName() == "or");
@ -93,6 +95,21 @@ private:
const auto & func_name = argument_function->getFunctionName();
if (func_name == "equals" || func_name == "isNotDistinctFrom")
{
const auto & argument_nodes = argument_function->getArguments().getNodes();
if (argument_nodes.size() != 2)
continue;
/// We can rewrite to a <=> b only if we are joining on a and b,
/// because the function is not yet implemented for other cases.
auto first_src = getExpressionSource(argument_nodes[0]);
auto second_src = getExpressionSource(argument_nodes[1]);
if (!first_src || !second_src)
continue;
const auto & lhs_join = *join_node->getLeftTableExpression();
const auto & rhs_join = *join_node->getRightTableExpression();
bool arguments_from_both_sides = (first_src->isEqual(lhs_join) && second_src->isEqual(rhs_join)) ||
(first_src->isEqual(rhs_join) && second_src->isEqual(lhs_join));
if (!arguments_from_both_sides)
continue;
equals_functions_indices.push_back(or_operands.size() - 1);
}
else if (func_name == "and")
@ -231,7 +248,7 @@ public:
/// Operator <=> is not supported outside of JOIN ON section
if (join_node->hasJoinExpression())
{
JoinOnLogicalExpressionOptimizerVisitor join_on_visitor(getContext());
JoinOnLogicalExpressionOptimizerVisitor join_on_visitor(join_node, getContext());
join_on_visitor.visit(join_node->getJoinExpression());
}
return;

57
src/Analyzer/Passes/QueryAnalysisPass.cpp
View File

@ -776,7 +776,7 @@ struct IdentifierResolveScope
/// Table expression node to data
std::unordered_map<QueryTreeNodePtr, TableExpressionData> table_expression_node_to_data;
QueryTreeNodePtrWithHashWithoutAliasSet nullable_group_by_keys;
QueryTreeNodePtrWithHashIgnoreTypesSet nullable_group_by_keys;
/// Here we count the number of nullable GROUP BY keys we met resolving expression.
/// E.g. for a query `SELECT tuple(tuple(number)) FROM numbers(10) GROUP BY (number, tuple(number)) with cube`
/// both `number` and `tuple(number)` would be in nullable_group_by_keys.
@ -6166,12 +6166,6 @@ ProjectionNames QueryAnalyzer::resolveExpressionNode(QueryTreeNodePtr & node, Id
return resolved_expression_it->second;
}
bool is_nullable_group_by_key = scope.nullable_group_by_keys.contains(node) && !scope.expressions_in_resolve_process_stack.hasAggregateFunction();
if (is_nullable_group_by_key)
++scope.found_nullable_group_by_key_in_scope;
SCOPE_EXIT(scope.found_nullable_group_by_key_in_scope -= is_nullable_group_by_key);
String node_alias = node->getAlias();
ProjectionNames result_projection_names;
@ -6463,10 +6457,14 @@ ProjectionNames QueryAnalyzer::resolveExpressionNode(QueryTreeNodePtr & node, Id
validateTreeSize(node, scope.context->getSettingsRef().max_expanded_ast_elements, node_to_tree_size);
if (is_nullable_group_by_key && scope.found_nullable_group_by_key_in_scope == 1)
if (!scope.expressions_in_resolve_process_stack.hasAggregateFunction())
{
node = node->clone();
node->convertToNullable();
auto it = scope.nullable_group_by_keys.find(node);
if (it != scope.nullable_group_by_keys.end())
{
node = it->node->clone();
node->convertToNullable();
}
}
/** Update aliases after expression node was resolved.
@ -6688,46 +6686,43 @@ void QueryAnalyzer::resolveGroupByNode(QueryNode & query_node_typed, IdentifierR
{
if (query_node_typed.isGroupByWithGroupingSets())
{
QueryTreeNodes nullable_group_by_keys;
for (auto & grouping_sets_keys_list_node : query_node_typed.getGroupBy().getNodes())
{
replaceNodesWithPositionalArguments(grouping_sets_keys_list_node, query_node_typed.getProjection().getNodes(), scope);
resolveExpressionNodeList(grouping_sets_keys_list_node, scope, false /*allow_lambda_expression*/, false /*allow_table_expression*/);
// Remove redundant calls to `tuple` function. It simplifies checking if expression is an aggregation key.
// It's required to support queries like: SELECT number FROM numbers(3) GROUP BY (number, number % 2)
auto & group_by_list = grouping_sets_keys_list_node->as<ListNode &>().getNodes();
expandTuplesInList(group_by_list);
if (scope.group_by_use_nulls)
for (const auto & group_by_elem : group_by_list)
nullable_group_by_keys.push_back(group_by_elem->clone());
resolveExpressionNodeList(grouping_sets_keys_list_node, scope, false /*allow_lambda_expression*/, false /*allow_table_expression*/);
}
for (auto & nullable_group_by_key : nullable_group_by_keys)
scope.nullable_group_by_keys.insert(std::move(nullable_group_by_key));
if (scope.group_by_use_nulls)
{
for (const auto & grouping_set : query_node_typed.getGroupBy().getNodes())
{
for (const auto & group_by_elem : grouping_set->as<ListNode>()->getNodes())
scope.nullable_group_by_keys.insert(group_by_elem);
}
}
}
else
{
replaceNodesWithPositionalArguments(query_node_typed.getGroupByNode(), query_node_typed.getProjection().getNodes(), scope);
resolveExpressionNodeList(query_node_typed.getGroupByNode(), scope, false /*allow_lambda_expression*/, false /*allow_table_expression*/);
// Remove redundant calls to `tuple` function. It simplifies checking if expression is an aggregation key.
// It's required to support queries like: SELECT number FROM numbers(3) GROUP BY (number, number % 2)
auto & group_by_list = query_node_typed.getGroupBy().getNodes();
expandTuplesInList(group_by_list);
QueryTreeNodes nullable_group_by_keys;
if (scope.group_by_use_nulls)
{
for (const auto & group_by_elem : query_node_typed.getGroupBy().getNodes())
nullable_group_by_keys.push_back(group_by_elem->clone());
scope.nullable_group_by_keys.insert(group_by_elem);
}
resolveExpressionNodeList(query_node_typed.getGroupByNode(), scope, false /*allow_lambda_expression*/, false /*allow_table_expression*/);
for (auto & nullable_group_by_key : nullable_group_by_keys)
scope.nullable_group_by_keys.insert(std::move(nullable_group_by_key));
}
}
@ -8036,6 +8031,16 @@ void QueryAnalyzer::resolveQuery(const QueryTreeNodePtr & query_node, Identifier
if (query_node_typed.hasGroupBy())
resolveGroupByNode(query_node_typed, scope);
if (scope.group_by_use_nulls)
{
resolved_expressions.clear();
/// Clone is needed cause aliases share subtrees.
/// If not clone, the same (shared) subtree could be resolved again with different (Nullable) type
/// See 03023_group_by_use_nulls_analyzer_crashes
for (auto & [_, node] : scope.alias_name_to_expression_node)
node = node->clone();
}
if (query_node_typed.hasHaving())
resolveExpressionNode(query_node_typed.getHaving(), scope, false /*allow_lambda_expression*/, false /*allow_table_expression*/);

4
src/Analyzer/QueryNode.cpp
View File

@ -247,7 +247,7 @@ void QueryNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, s
}
}
bool QueryNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool QueryNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const QueryNode &>(rhs);
@ -266,7 +266,7 @@ bool QueryNode::isEqualImpl(const IQueryTreeNode & rhs) const
settings_changes == rhs_typed.settings_changes;
}
void QueryNode::updateTreeHashImpl(HashState & state) const
void QueryNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(is_subquery);
state.update(is_cte);

4
src/Analyzer/QueryNode.h
View File

@ -589,9 +589,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState &) const override;
void updateTreeHashImpl(HashState &, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/SortNode.cpp
View File

@ -71,7 +71,7 @@ void SortNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, si
}
}
bool SortNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool SortNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const SortNode &>(rhs);
if (sort_direction != rhs_typed.sort_direction ||
@ -89,7 +89,7 @@ bool SortNode::isEqualImpl(const IQueryTreeNode & rhs) const
return collator->getLocale() == rhs_typed.collator->getLocale();
}
void SortNode::updateTreeHashImpl(HashState & hash_state) const
void SortNode::updateTreeHashImpl(HashState & hash_state, CompareOptions) const
{
hash_state.update(sort_direction);
/// use some determined value if `nulls_sort_direction` is `nullopt`

4
src/Analyzer/SortNode.h
View File

@ -131,9 +131,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/TableFunctionNode.cpp
View File

@ -82,7 +82,7 @@ void TableFunctionNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_
}
}
bool TableFunctionNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool TableFunctionNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const TableFunctionNode &>(rhs);
if (table_function_name != rhs_typed.table_function_name)
@ -97,7 +97,7 @@ bool TableFunctionNode::isEqualImpl(const IQueryTreeNode & rhs) const
return table_expression_modifiers == rhs_typed.table_expression_modifiers;
}
void TableFunctionNode::updateTreeHashImpl(HashState & state) const
void TableFunctionNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(table_function_name.size());
state.update(table_function_name);

4
src/Analyzer/TableFunctionNode.h
View File

@ -155,9 +155,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/TableNode.cpp
View File

@ -52,14 +52,14 @@ void TableNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, s
}
}
bool TableNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool TableNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const TableNode &>(rhs);
return storage_id == rhs_typed.storage_id && table_expression_modifiers == rhs_typed.table_expression_modifiers &&
temporary_table_name == rhs_typed.temporary_table_name;
}
void TableNode::updateTreeHashImpl(HashState & state) const
void TableNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
if (!temporary_table_name.empty())
{

4
src/Analyzer/TableNode.h
View File

@ -100,9 +100,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & state) const override;
void updateTreeHashImpl(HashState & state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/UnionNode.cpp
View File

@ -145,7 +145,7 @@ void UnionNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, s
getQueriesNode()->dumpTreeImpl(buffer, format_state, indent + 4);
}
bool UnionNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool UnionNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const UnionNode &>(rhs);
@ -153,7 +153,7 @@ bool UnionNode::isEqualImpl(const IQueryTreeNode & rhs) const
union_mode == rhs_typed.union_mode;
}
void UnionNode::updateTreeHashImpl(HashState & state) const
void UnionNode::updateTreeHashImpl(HashState & state, CompareOptions) const
{
state.update(is_subquery);
state.update(is_cte);

4
src/Analyzer/UnionNode.h
View File

@ -143,9 +143,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState &) const override;
void updateTreeHashImpl(HashState &, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

48
src/Analyzer/Utils.cpp
View File

@ -760,6 +760,54 @@ QueryTreeNodePtr createCastFunction(QueryTreeNodePtr node, DataTypePtr result_ty
return function_node;
}
/** Returns:
* {_, false} - multiple sources
* {nullptr, true} - no sources (for constants)
* {source, true} - single source
*/
std::pair<QueryTreeNodePtr, bool> getExpressionSourceImpl(const QueryTreeNodePtr & node)
{
if (const auto * column = node->as<ColumnNode>())
{
auto source = column->getColumnSourceOrNull();
if (!source)
return {nullptr, false};
return {source, true};
}
if (const auto * func = node->as<FunctionNode>())
{
QueryTreeNodePtr source = nullptr;
const auto & args = func->getArguments().getNodes();
for (const auto & arg : args)
{
auto [arg_source, is_ok] = getExpressionSourceImpl(arg);
if (!is_ok)
return {nullptr, false};
if (!source)
source = arg_source;
else if (arg_source && !source->isEqual(*arg_source))
return {nullptr, false};
}
return {source, true};
}
if (node->as<ConstantNode>())
return {nullptr, true};
return {nullptr, false};
}
QueryTreeNodePtr getExpressionSource(const QueryTreeNodePtr & node)
{
auto [source, is_ok] = getExpressionSourceImpl(node);
if (!is_ok)
return nullptr;
return source;
}
QueryTreeNodePtr buildSubqueryToReadColumnsFromTableExpression(QueryTreeNodePtr table_node, const ContextPtr & context)
{
const auto & storage_snapshot = table_node->as<TableNode>()->getStorageSnapshot();

3
src/Analyzer/Utils.h
View File

@ -105,6 +105,9 @@ NameSet collectIdentifiersFullNames(const QueryTreeNodePtr & node);
/// Wrap node into `_CAST` function
QueryTreeNodePtr createCastFunction(QueryTreeNodePtr node, DataTypePtr result_type, ContextPtr context);
/// Checks that node has only one source and returns it
QueryTreeNodePtr getExpressionSource(const QueryTreeNodePtr & node);
/// Build subquery which we execute for `IN table` function.
QueryTreeNodePtr buildSubqueryToReadColumnsFromTableExpression(QueryTreeNodePtr table_node, const ContextPtr & context);

4
src/Analyzer/WindowNode.cpp
View File

@ -78,14 +78,14 @@ void WindowNode::dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state,
}
}
bool WindowNode::isEqualImpl(const IQueryTreeNode & rhs) const
bool WindowNode::isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const
{
const auto & rhs_typed = assert_cast<const WindowNode &>(rhs);
return window_frame == rhs_typed.window_frame && parent_window_name == rhs_typed.parent_window_name;
}
void WindowNode::updateTreeHashImpl(HashState & hash_state) const
void WindowNode::updateTreeHashImpl(HashState & hash_state, CompareOptions) const
{
hash_state.update(window_frame.is_default);
hash_state.update(window_frame.type);

4
src/Analyzer/WindowNode.h
View File

@ -169,9 +169,9 @@ public:
void dumpTreeImpl(WriteBuffer & buffer, FormatState & format_state, size_t indent) const override;
protected:
bool isEqualImpl(const IQueryTreeNode & rhs) const override;
bool isEqualImpl(const IQueryTreeNode & rhs, CompareOptions) const override;
void updateTreeHashImpl(HashState & hash_state) const override;
void updateTreeHashImpl(HashState & hash_state, CompareOptions) const override;
QueryTreeNodePtr cloneImpl() const override;

4
src/Analyzer/tests/gtest_query_tree_node.cpp
View File

@ -22,12 +22,12 @@ public:
{
}
bool isEqualImpl(const IQueryTreeNode &) const override
bool isEqualImpl(const IQueryTreeNode &, CompareOptions) const override
{
return true;
}
void updateTreeHashImpl(HashState &) const override
void updateTreeHashImpl(HashState &, CompareOptions) const override
{
}

13
src/Backups/BackupEntriesCollector.cpp
View File

@ -125,7 +125,7 @@ BackupEntries BackupEntriesCollector::run()
= BackupSettings::Util::filterHostIDs(backup_settings.cluster_host_ids, backup_settings.shard_num, backup_settings.replica_num);
/// Do renaming in the create queries according to the renaming config.
renaming_map = makeRenamingMapFromBackupQuery(backup_query_elements);
renaming_map = BackupUtils::makeRenamingMap(backup_query_elements);
/// Calculate the root path for collecting backup entries, it's either empty or has the format "shards/<shard_num>/replicas/<replica_num>/".
calculateRootPathInBackup();
@ -570,17 +570,16 @@ std::vector<std::pair<ASTPtr, StoragePtr>> BackupEntriesCollector::findTablesInD
checkIsQueryCancelled();
auto filter_by_table_name = [my_database_info = &database_info](const String & table_name)
auto filter_by_table_name = [&](const String & table_name)
{
/// We skip inner tables of materialized views.
if (table_name.starts_with(".inner_id."))
if (BackupUtils::isInnerTable(database_name, table_name))
return false;
if (my_database_info->tables.contains(table_name))
if (database_info.tables.contains(table_name))
return true;
if (my_database_info->all_tables)
return !my_database_info->except_table_names.contains(table_name);
if (database_info.all_tables)
return !database_info.except_table_names.contains(table_name);
return false;
};

15
src/Backups/BackupUtils.cpp
View File

@ -8,10 +8,10 @@
#include <Common/setThreadName.h>
namespace DB
namespace DB::BackupUtils
{
DDLRenamingMap makeRenamingMapFromBackupQuery(const ASTBackupQuery::Elements & elements)
DDLRenamingMap makeRenamingMap(const ASTBackupQuery::Elements & elements)
{
DDLRenamingMap map;
@ -120,4 +120,15 @@ bool compareRestoredDatabaseDef(const IAST & restored_database_create_query, con
return compareRestoredTableDef(restored_database_create_query, create_query_from_backup, global_context);
}
bool isInnerTable(const QualifiedTableName & table_name)
{
return isInnerTable(table_name.database, table_name.table);
}
bool isInnerTable(const String & /* database_name */, const String & table_name)
{
/// We skip inner tables of materialized views.
return table_name.starts_with(".inner.") || table_name.starts_with(".inner_id.");
}
}

12
src/Backups/BackupUtils.h
View File

@ -9,9 +9,13 @@ namespace DB
class IBackup;
class AccessRightsElements;
class DDLRenamingMap;
struct QualifiedTableName;
namespace BackupUtils
{
/// Initializes a DDLRenamingMap from a BACKUP or RESTORE query.
DDLRenamingMap makeRenamingMapFromBackupQuery(const ASTBackupQuery::Elements & elements);
DDLRenamingMap makeRenamingMap(const ASTBackupQuery::Elements & elements);
/// Returns access required to execute BACKUP query.
AccessRightsElements getRequiredAccessToBackup(const ASTBackupQuery::Elements & elements);
@ -20,4 +24,10 @@ AccessRightsElements getRequiredAccessToBackup(const ASTBackupQuery::Elements &
bool compareRestoredTableDef(const IAST & restored_table_create_query, const IAST & create_query_from_backup, const ContextPtr & global_context);
bool compareRestoredDatabaseDef(const IAST & restored_database_create_query, const IAST & create_query_from_backup, const ContextPtr & global_context);
/// Returns true if this table should be skipped while making a backup because it's an inner table.
bool isInnerTable(const QualifiedTableName & table_name);
bool isInnerTable(const String & database_name, const String & table_name);
}
}

2
src/Backups/BackupsWorker.cpp
View File

@ -564,7 +564,7 @@ void BackupsWorker::doBackup(
/// Checks access rights if this is not ON CLUSTER query.
/// (If this is ON CLUSTER query executeDDLQueryOnCluster() will check access rights later.)
auto required_access = getRequiredAccessToBackup(backup_query->elements);
auto required_access = BackupUtils::getRequiredAccessToBackup(backup_query->elements);
if (!on_cluster)
context->checkAccess(required_access);

24
src/Backups/RestorerFromBackup.cpp
View File

@ -124,7 +124,7 @@ void RestorerFromBackup::run(Mode mode)
restore_settings.cluster_host_ids, restore_settings.shard_num, restore_settings.replica_num);
/// Do renaming in the create queries according to the renaming config.
renaming_map = makeRenamingMapFromBackupQuery(restore_query_elements);
renaming_map = BackupUtils::makeRenamingMap(restore_query_elements);
/// Calculate the root path in the backup for restoring, it's either empty or has the format "shards/<shard_num>/replicas/<replica_num>/".
findRootPathsInBackup();
@ -346,12 +346,12 @@ void RestorerFromBackup::findDatabasesAndTablesInBackup()
{
case ASTBackupQuery::ElementType::TABLE:
{
findTableInBackup({element.database_name, element.table_name}, element.partitions);
findTableInBackup({element.database_name, element.table_name}, /* skip_if_inner_table= */ false, element.partitions);
break;
}
case ASTBackupQuery::ElementType::TEMPORARY_TABLE:
{
findTableInBackup({DatabaseCatalog::TEMPORARY_DATABASE, element.table_name}, element.partitions);
findTableInBackup({DatabaseCatalog::TEMPORARY_DATABASE, element.table_name}, /* skip_if_inner_table= */ false, element.partitions);
break;
}
case ASTBackupQuery::ElementType::DATABASE:
@ -370,14 +370,14 @@ void RestorerFromBackup::findDatabasesAndTablesInBackup()
LOG_INFO(log, "Will restore {} databases and {} tables", getNumDatabases(), getNumTables());
}
void RestorerFromBackup::findTableInBackup(const QualifiedTableName & table_name_in_backup, const std::optional<ASTs> & partitions)
void RestorerFromBackup::findTableInBackup(const QualifiedTableName & table_name_in_backup, bool skip_if_inner_table, const std::optional<ASTs> & partitions)
{
schedule(
[this, table_name_in_backup, partitions]() { findTableInBackupImpl(table_name_in_backup, partitions); },
[this, table_name_in_backup, skip_if_inner_table, partitions]() { findTableInBackupImpl(table_name_in_backup, skip_if_inner_table, partitions); },
"Restore_FindTbl");
}
void RestorerFromBackup::findTableInBackupImpl(const QualifiedTableName & table_name_in_backup, const std::optional<ASTs> & partitions)
void RestorerFromBackup::findTableInBackupImpl(const QualifiedTableName & table_name_in_backup, bool skip_if_inner_table, const std::optional<ASTs> & partitions)
{
bool is_temporary_table = (table_name_in_backup.database == DatabaseCatalog::TEMPORARY_DATABASE);
@ -422,6 +422,10 @@ void RestorerFromBackup::findTableInBackupImpl(const QualifiedTableName & table_
= *root_path_in_use / "data" / escapeForFileName(table_name_in_backup.database) / escapeForFileName(table_name_in_backup.table);
}
QualifiedTableName table_name = renaming_map.getNewTableName(table_name_in_backup);
if (skip_if_inner_table && BackupUtils::isInnerTable(table_name))
return;
auto read_buffer = backup->readFile(*metadata_path);
String create_query_str;
readStringUntilEOF(create_query_str, *read_buffer);
@ -432,8 +436,6 @@ void RestorerFromBackup::findTableInBackupImpl(const QualifiedTableName & table_
renameDatabaseAndTableNameInCreateQuery(create_table_query, renaming_map, context->getGlobalContext());
String create_table_query_str = serializeAST(*create_table_query);
QualifiedTableName table_name = renaming_map.getNewTableName(table_name_in_backup);
bool is_predefined_table = DatabaseCatalog::instance().isPredefinedTable(StorageID{table_name.database, table_name.table});
auto table_dependencies = getDependenciesFromCreateQuery(context, table_name, create_table_query);
bool table_has_data = backup->hasFiles(data_path_in_backup);
@ -568,7 +570,7 @@ void RestorerFromBackup::findDatabaseInBackupImpl(const String & database_name_i
if (except_table_names.contains({database_name_in_backup, table_name_in_backup}))
continue;
findTableInBackup({database_name_in_backup, table_name_in_backup}, /* partitions= */ {});
findTableInBackup({database_name_in_backup, table_name_in_backup}, /* skip_if_inner_table= */ true, /* partitions= */ {});
}
}
@ -767,7 +769,7 @@ void RestorerFromBackup::checkDatabase(const String & database_name)
ASTPtr existing_database_def = database->getCreateDatabaseQuery();
ASTPtr database_def_from_backup = database_info.create_database_query;
if (!compareRestoredDatabaseDef(*existing_database_def, *database_def_from_backup, context->getGlobalContext()))
if (!BackupUtils::compareRestoredDatabaseDef(*existing_database_def, *database_def_from_backup, context->getGlobalContext()))
{
throw Exception(
ErrorCodes::CANNOT_RESTORE_DATABASE,
@ -938,7 +940,7 @@ void RestorerFromBackup::checkTable(const QualifiedTableName & table_name)
{
ASTPtr existing_table_def = database->getCreateTableQuery(resolved_id.table_name, context);
ASTPtr table_def_from_backup = table_info.create_table_query;
if (!compareRestoredTableDef(*existing_table_def, *table_def_from_backup, context->getGlobalContext()))
if (!BackupUtils::compareRestoredTableDef(*existing_table_def, *table_def_from_backup, context->getGlobalContext()))
{
throw Exception(
ErrorCodes::CANNOT_RESTORE_TABLE,

4
src/Backups/RestorerFromBackup.h
View File

@ -92,8 +92,8 @@ private:
void findRootPathsInBackup();
void findDatabasesAndTablesInBackup();
void findTableInBackup(const QualifiedTableName & table_name_in_backup, const std::optional<ASTs> & partitions);
void findTableInBackupImpl(const QualifiedTableName & table_name_in_backup, const std::optional<ASTs> & partitions);
void findTableInBackup(const QualifiedTableName & table_name_in_backup, bool skip_if_inner_table, const std::optional<ASTs> & partitions);
void findTableInBackupImpl(const QualifiedTableName & table_name_in_backup, bool skip_if_inner_table, const std::optional<ASTs> & partitions);
void findDatabaseInBackup(const String & database_name_in_backup, const std::set<DatabaseAndTableName> & except_table_names);
void findDatabaseInBackupImpl(const String & database_name_in_backup, const std::set<DatabaseAndTableName> & except_table_names);
void findEverythingInBackup(const std::set<String> & except_database_names, const std::set<DatabaseAndTableName> & except_table_names);

27
src/Client/ClientBase.cpp
View File

@ -329,12 +329,11 @@ void ClientBase::setupSignalHandler()
}
ASTPtr ClientBase::parseQuery(const char *& pos, const char * end, bool allow_multi_statements) const
ASTPtr ClientBase::parseQuery(const char *& pos, const char * end, const Settings & settings, bool allow_multi_statements, bool is_interactive, bool ignore_error)
{
std::unique_ptr<IParserBase> parser;
ASTPtr res;
const auto & settings = global_context->getSettingsRef();
size_t max_length = 0;
if (!allow_multi_statements)
@ -343,11 +342,11 @@ ASTPtr ClientBase::parseQuery(const char *& pos, const char * end, bool allow_mu
const Dialect & dialect = settings.dialect;
if (dialect == Dialect::kusto)
parser = std::make_unique<ParserKQLStatement>(end, global_context->getSettings().allow_settings_after_format_in_insert);
parser = std::make_unique<ParserKQLStatement>(end, settings.allow_settings_after_format_in_insert);
else if (dialect == Dialect::prql)
parser = std::make_unique<ParserPRQLQuery>(max_length, settings.max_parser_depth, settings.max_parser_backtracks);
else
parser = std::make_unique<ParserQuery>(end, global_context->getSettings().allow_settings_after_format_in_insert);
parser = std::make_unique<ParserQuery>(end, settings.allow_settings_after_format_in_insert);
if (is_interactive || ignore_error)
{
@ -916,7 +915,11 @@ void ClientBase::processTextAsSingleQuery(const String & full_query)
/// Some parts of a query (result output and formatting) are executed
/// client-side. Thus we need to parse the query.
const char * begin = full_query.data();
auto parsed_query = parseQuery(begin, begin + full_query.size(), false);
auto parsed_query = parseQuery(begin, begin + full_query.size(),
global_context->getSettingsRef(),
/*allow_multi_statements=*/ false,
is_interactive,
ignore_error);
if (!parsed_query)
return;
@ -2061,7 +2064,7 @@ MultiQueryProcessingStage ClientBase::analyzeMultiQueryText(
return MultiQueryProcessingStage::QUERIES_END;
// Remove leading empty newlines and other whitespace, because they
// are annoying to filter in the query log. This is mostly relevant for
// are annoying to filter in query log. This is mostly relevant for
// the tests.
while (this_query_begin < all_queries_end && isWhitespaceASCII(*this_query_begin))
++this_query_begin;
@ -2089,9 +2092,13 @@ MultiQueryProcessingStage ClientBase::analyzeMultiQueryText(
this_query_end = this_query_begin;
try
{
parsed_query = parseQuery(this_query_end, all_queries_end, true);
parsed_query = parseQuery(this_query_end, all_queries_end,
global_context->getSettingsRef(),
/*allow_multi_statements=*/ true,
is_interactive,
ignore_error);
}
catch (const Exception & e)
catch (Exception & e)
{
current_exception.reset(e.clone());
return MultiQueryProcessingStage::PARSING_EXCEPTION;
@ -2116,9 +2123,9 @@ MultiQueryProcessingStage ClientBase::analyzeMultiQueryText(
// INSERT queries may have the inserted data in the query text
// that follow the query itself, e.g. "insert into t format CSV 1;2".
// They need special handling. First of all, here we find where the
// inserted data ends. In multi-query mode, it is delimited by a
// inserted data ends. In multy-query mode, it is delimited by a
// newline.
// The VALUES format needs even more handling - we also allow the
// The VALUES format needs even more handling -- we also allow the
// data to be delimited by semicolon. This case is handled later by
// the format parser itself.
// We can't do multiline INSERTs with inline data, because most

2
src/Client/ClientBase.h
View File

@ -73,6 +73,7 @@ public:
void init(int argc, char ** argv);
std::vector<String> getAllRegisteredNames() const override { return cmd_options; }
static ASTPtr parseQuery(const char *& pos, const char * end, const Settings & settings, bool allow_multi_statements, bool is_interactive, bool ignore_error);
protected:
void runInteractive();
@ -98,7 +99,6 @@ protected:
ASTPtr parsed_query, std::optional<bool> echo_query_ = {}, bool report_error = false);
static void adjustQueryEnd(const char *& this_query_end, const char * all_queries_end, uint32_t max_parser_depth, uint32_t max_parser_backtracks);
ASTPtr parseQuery(const char *& pos, const char * end, bool allow_multi_statements) const;
static void setupSignalHandler();
bool executeMultiQuery(const String & all_queries_text);

154
src/Client/ClientBaseHelpers.cpp
View File

@ -1,14 +1,11 @@
#include "ClientBaseHelpers.h"
#include <Common/DateLUT.h>
#include <Common/LocalDate.h>
#include <Parsers/ParserQuery.h>
#include <Parsers/parseQuery.h>
#include <Parsers/Lexer.h>
#include <Common/UTF8Helpers.h>
#include <iostream>
namespace DB
{
@ -99,102 +96,77 @@ void highlight(const String & query, std::vector<replxx::Replxx::Color> & colors
{
using namespace replxx;
/// The `colors` array maps to a Unicode code point position in a string into a color.
/// A color is set for every position individually (not for a range).
static const std::unordered_map<TokenType, Replxx::Color> token_to_color
= {{TokenType::Whitespace, Replxx::Color::DEFAULT},
{TokenType::Comment, Replxx::Color::GRAY},
{TokenType::BareWord, Replxx::Color::DEFAULT},
{TokenType::Number, Replxx::Color::GREEN},
{TokenType::StringLiteral, Replxx::Color::CYAN},
{TokenType::QuotedIdentifier, Replxx::Color::MAGENTA},
{TokenType::OpeningRoundBracket, Replxx::Color::BROWN},
{TokenType::ClosingRoundBracket, Replxx::Color::BROWN},
{TokenType::OpeningSquareBracket, Replxx::Color::BROWN},
{TokenType::ClosingSquareBracket, Replxx::Color::BROWN},
{TokenType::DoubleColon, Replxx::Color::BROWN},
{TokenType::OpeningCurlyBrace, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::ClosingCurlyBrace, replxx::color::bold(Replxx::Color::DEFAULT)},
/// Empty input.
if (colors.empty())
return;
{TokenType::Comma, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Semicolon, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::VerticalDelimiter, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Dot, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Asterisk, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::HereDoc, Replxx::Color::CYAN},
{TokenType::Plus, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Minus, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Slash, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Percent, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Arrow, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::QuestionMark, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Colon, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Equals, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::NotEquals, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Less, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Greater, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::LessOrEquals, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::GreaterOrEquals, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Spaceship, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::Concatenation, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::At, replxx::color::bold(Replxx::Color::DEFAULT)},
{TokenType::DoubleAt, Replxx::Color::MAGENTA},
/// The colors should be legible (and look gorgeous) in both dark and light themes.
/// When modifying this, check it in both themes.
{TokenType::EndOfStream, Replxx::Color::DEFAULT},
static const std::unordered_map<Highlight, Replxx::Color> type_to_color =
{
{Highlight::keyword, replxx::color::bold(Replxx::Color::DEFAULT)},
{Highlight::identifier, Replxx::Color::CYAN},
{Highlight::function, Replxx::Color::BROWN},
{Highlight::alias, replxx::color::rgb666(0, 4, 4)},
{Highlight::substitution, Replxx::Color::MAGENTA},
{Highlight::number, replxx::color::rgb666(0, 4, 0)},
{Highlight::string, Replxx::Color::GREEN},
};
{TokenType::Error, Replxx::Color::RED},
{TokenType::ErrorMultilineCommentIsNotClosed, Replxx::Color::RED},
{TokenType::ErrorSingleQuoteIsNotClosed, Replxx::Color::RED},
{TokenType::ErrorDoubleQuoteIsNotClosed, Replxx::Color::RED},
{TokenType::ErrorSinglePipeMark, Replxx::Color::RED},
{TokenType::ErrorWrongNumber, Replxx::Color::RED},
{TokenType::ErrorMaxQuerySizeExceeded, Replxx::Color::RED}};
/// We set reasonably small limits for size/depth, because we don't want the CLI to be slow.
/// While syntax highlighting is unneeded for long queries, which the user couldn't read anyway.
const char * begin = query.data();
const char * end = begin + query.size();
Tokens tokens(begin, end, 1000, true);
IParser::Pos token_iterator(tokens, static_cast<uint32_t>(1000), static_cast<uint32_t>(10000));
Expected expected;
/// We don't do highlighting for foreign dialects, such as PRQL and Kusto.
/// Only normal ClickHouse SQL queries are highlighted.
/// Currently we highlight only the first query in the multi-query mode.
ParserQuery parser(end);
ASTPtr ast;
bool parse_res = false;
try
{
parse_res = parser.parse(token_iterator, ast, expected);
}
catch (...)
{
/// Skip highlighting in the case of exceptions during parsing.
/// It is ok to ignore unknown exceptions here.
return;
}
const Replxx::Color unknown_token_color = Replxx::Color::RED;
Lexer lexer(query.data(), query.data() + query.size());
size_t pos = 0;
const char * prev = begin;
for (const auto & range : expected.highlights)
for (Token token = lexer.nextToken(); !token.isEnd(); token = lexer.nextToken())
{
auto it = type_to_color.find(range.highlight);
if (it != type_to_color.end())
if (token.type == TokenType::Semicolon || token.type == TokenType::VerticalDelimiter)
ReplxxLineReader::setLastIsDelimiter(true);
else if (token.type != TokenType::Whitespace)
ReplxxLineReader::setLastIsDelimiter(false);
size_t utf8_len = UTF8::countCodePoints(reinterpret_cast<const UInt8 *>(token.begin), token.size());
for (size_t code_point_index = 0; code_point_index < utf8_len; ++code_point_index)
{
/// We have to map from byte positions to Unicode positions.
pos += UTF8::countCodePoints(reinterpret_cast<const UInt8 *>(prev), range.begin - prev);
size_t utf8_len = UTF8::countCodePoints(reinterpret_cast<const UInt8 *>(range.begin), range.end - range.begin);
for (size_t code_point_index = 0; code_point_index < utf8_len; ++code_point_index)
colors[pos + code_point_index] = it->second;
pos += utf8_len;
prev = range.end;
if (token_to_color.find(token.type) != token_to_color.end())
colors[pos + code_point_index] = token_to_color.at(token.type);
else
colors[pos + code_point_index] = unknown_token_color;
}
}
Token last_token = token_iterator.max();
/// Raw data in INSERT queries, which is not necessarily tokenized.
const char * insert_data = ast ? getInsertData(ast) : nullptr;
/// Highlight the last error in red. If the parser failed or the lexer found an invalid token,
/// or if it didn't parse all the data (except, the data for INSERT query, which is legitimately unparsed)
if ((!parse_res || last_token.isError() || (!token_iterator->isEnd() && token_iterator->type != TokenType::Semicolon))
&& !(insert_data && expected.max_parsed_pos >= insert_data)
&& expected.max_parsed_pos >= prev)
{
pos += UTF8::countCodePoints(reinterpret_cast<const UInt8 *>(prev), expected.max_parsed_pos - prev);
if (pos >= colors.size())
pos = colors.size() - 1;
colors[pos] = Replxx::Color::BRIGHTRED;
}
/// This is a callback for the client/local app to better find query end. Note: this is a kludge, remove it.
if (last_token.type == TokenType::Semicolon || last_token.type == TokenType::VerticalDelimiter
|| query.ends_with(';') || query.ends_with("\\G")) /// This is for raw data in INSERT queries, which is not necessarily tokenized.
{
ReplxxLineReader::setLastIsDelimiter(true);
}
else if (last_token.type != TokenType::Whitespace)
{
ReplxxLineReader::setLastIsDelimiter(false);
pos += utf8_len;
}
}
#endif

64
src/Client/LocalConnection.cpp
View File

@ -1,11 +1,18 @@
#include "LocalConnection.h"
#include <memory>
#include <Client/ClientBase.h>
#include <Core/Protocol.h>
#include <Interpreters/DatabaseCatalog.h>
#include <Interpreters/executeQuery.h>
#include <Processors/Formats/IInputFormat.h>
#include <Processors/Executors/CompletedPipelineExecutor.h>
#include <Processors/Executors/PullingAsyncPipelineExecutor.h>
#include <Processors/Executors/PushingAsyncPipelineExecutor.h>
#include <Processors/Executors/PushingPipelineExecutor.h>
#include <Processors/Transforms/AddingDefaultsTransform.h>
#include <QueryPipeline/QueryPipeline.h>
#include <QueryPipeline/Pipe.h>
#include <Parsers/ASTInsertQuery.h>
#include <Storages/IStorage.h>
#include <Common/ConcurrentBoundedQueue.h>
#include <Common/CurrentThread.h>
@ -22,12 +29,13 @@ namespace ErrorCodes
extern const int LOGICAL_ERROR;
}
LocalConnection::LocalConnection(ContextPtr context_, bool send_progress_, bool send_profile_events_, const String & server_display_name_)
LocalConnection::LocalConnection(ContextPtr context_, ReadBuffer * in_, bool send_progress_, bool send_profile_events_, const String & server_display_name_)
: WithContext(context_)
, session(getContext(), ClientInfo::Interface::LOCAL)
, send_progress(send_progress_)
, send_profile_events(send_profile_events_)
, server_display_name(server_display_name_)
, in(in_)
{
/// Authenticate and create a context to execute queries.
session.authenticate("default", "", Poco::Net::SocketAddress{});
@ -130,6 +138,57 @@ void LocalConnection::sendQuery(
next_packet_type.reset();
/// Prepare input() function
query_context->setInputInitializer([this] (ContextPtr context, const StoragePtr & input_storage)
{
if (context != query_context)
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected context in Input initializer");
auto metadata_snapshot = input_storage->getInMemoryMetadataPtr();
Block sample = metadata_snapshot->getSampleBlock();
next_packet_type = Protocol::Server::Data;
state->block = sample;
String current_format = "Values";
const char * begin = state->query.data();
auto parsed_query = ClientBase::parseQuery(begin, begin + state->query.size(),
context->getSettingsRef(),
/*allow_multi_statements=*/ false,
/*is_interactive=*/ false,
/*ignore_error=*/ false);
if (const auto * insert = parsed_query->as<ASTInsertQuery>())
{
if (!insert->format.empty())
current_format = insert->format;
}
auto source = context->getInputFormat(current_format, *in, sample, context->getSettingsRef().max_insert_block_size);
Pipe pipe(source);
auto columns_description = metadata_snapshot->getColumns();
if (columns_description.hasDefaults())
{
pipe.addSimpleTransform([&](const Block & header)
{
return std::make_shared<AddingDefaultsTransform>(header, columns_description, *source, context);
});
}
state->input_pipeline = std::make_unique<QueryPipeline>(std::move(pipe));
state->input_pipeline_executor = std::make_unique<PullingAsyncPipelineExecutor>(*state->input_pipeline);
});
query_context->setInputBlocksReaderCallback([this] (ContextPtr context) -> Block
{
if (context != query_context)
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected context in InputBlocksReader");
Block block;
state->input_pipeline_executor->pull(block);
return block;
});
try
{
state->io = executeQuery(state->query, query_context, QueryFlags{}, state->stage).second;
@ -537,11 +596,12 @@ void LocalConnection::sendMergeTreeReadTaskResponse(const ParallelReadResponse &
ServerConnectionPtr LocalConnection::createConnection(
const ConnectionParameters &,
ContextPtr current_context,
ReadBuffer * in,
bool send_progress,
bool send_profile_events,
const String & server_display_name)
{
return std::make_unique<LocalConnection>(current_context, send_progress, send_profile_events, server_display_name);
return std::make_unique<LocalConnection>(current_context, in, send_progress, send_profile_events, server_display_name);
}

15
src/Client/LocalConnection.h
View File

@ -15,6 +15,8 @@ namespace DB
class PullingAsyncPipelineExecutor;
class PushingAsyncPipelineExecutor;
class PushingPipelineExecutor;
class QueryPipeline;
class ReadBuffer;
/// State of query processing.
struct LocalQueryState
@ -31,6 +33,10 @@ struct LocalQueryState
std::unique_ptr<PullingAsyncPipelineExecutor> executor;
std::unique_ptr<PushingPipelineExecutor> pushing_executor;
std::unique_ptr<PushingAsyncPipelineExecutor> pushing_async_executor;
/// For sending data for input() function.
std::unique_ptr<QueryPipeline> input_pipeline;
std::unique_ptr<PullingAsyncPipelineExecutor> input_pipeline_executor;
InternalProfileEventsQueuePtr profile_queue;
std::unique_ptr<Exception> exception;
@ -64,7 +70,11 @@ class LocalConnection : public IServerConnection, WithContext
{
public:
explicit LocalConnection(
ContextPtr context_, bool send_progress_ = false, bool send_profile_events_ = false, const String & server_display_name_ = "");
ContextPtr context_,
ReadBuffer * in_,
bool send_progress_,
bool send_profile_events_,
const String & server_display_name_);
~LocalConnection() override;
@ -73,6 +83,7 @@ public:
static ServerConnectionPtr createConnection(
const ConnectionParameters & connection_parameters,
ContextPtr current_context,
ReadBuffer * in = nullptr,
bool send_progress = false,
bool send_profile_events = false,
const String & server_display_name = "");
@ -158,5 +169,7 @@ private:
String current_database;
ProfileEvents::ThreadIdToCountersSnapshot last_sent_snapshots;
ReadBuffer * in;
};
}

4
src/Common/ZooKeeper/ZooKeeperCommon.cpp
View File

@ -403,7 +403,7 @@ void ZooKeeperSetACLRequest::readImpl(ReadBuffer & in)
std::string ZooKeeperSetACLRequest::toStringImpl() const
{
return fmt::format("path = {}\n", "version = {}", path, version);
return fmt::format("path = {}\nversion = {}", path, version);
}
void ZooKeeperSetACLResponse::writeImpl(WriteBuffer & out) const
@ -457,7 +457,7 @@ void ZooKeeperCheckRequest::readImpl(ReadBuffer & in)
std::string ZooKeeperCheckRequest::toStringImpl() const
{
return fmt::format("path = {}\n", "version = {}", path, version);
return fmt::format("path = {}\nversion = {}", path, version);
}
void ZooKeeperErrorResponse::readImpl(ReadBuffer & in)

10
src/Common/ZooKeeper/ZooKeeperConstants.h
View File

@ -2,6 +2,7 @@
#include <string>
#include <cstdint>
#include <magic_enum.hpp>
namespace Coordination
@ -64,3 +65,12 @@ static constexpr int32_t DEFAULT_OPERATION_TIMEOUT_MS = 10000;
static constexpr int32_t DEFAULT_CONNECTION_TIMEOUT_MS = 1000;
}
/// This is used by fmt::format to print OpNum as strings.
/// All OpNum values should be in range [min, max] to be printed.
template <>
struct magic_enum::customize::enum_range<Coordination::OpNum>
{
static constexpr int min = -100;
static constexpr int max = 1000;
};

4
src/Common/examples/encrypt_decrypt.cpp
View File

@ -35,9 +35,9 @@ int main(int argc, char ** argv)
DB::CompressionCodecEncrypted::Configuration::instance().load(*loaded_config.configuration, "encryption_codecs");
if (action == "-e")
std::cout << processor.encryptValue(codec_name, value) << std::endl;
std::cout << DB::ConfigProcessor::encryptValue(codec_name, value) << std::endl;
else if (action == "-d")
std::cout << processor.decryptValue(codec_name, value) << std::endl;
std::cout << DB::ConfigProcessor::decryptValue(codec_name, value) << std::endl;
else
std::cerr << "Unknown action: " << action << std::endl;
}

8
src/Core/Field.h
View File

@ -898,11 +898,13 @@ NearestFieldType<std::decay_t<T>> & Field::get()
template <typename T>
auto & Field::safeGet()
{
const Types::Which requested = TypeToEnum<NearestFieldType<std::decay_t<T>>>::value;
const Types::Which target = TypeToEnum<NearestFieldType<std::decay_t<T>>>::value;
if (which != requested)
/// We allow converting int64 <-> uint64, int64 <-> bool, uint64 <-> bool in safeGet().
if (target != which
&& (!isInt64OrUInt64orBoolFieldType(target) || !isInt64OrUInt64orBoolFieldType(which)))
throw Exception(ErrorCodes::BAD_GET,
"Bad get: has {}, requested {}", getTypeName(), requested);
"Bad get: has {}, requested {}", getTypeName(), target);
return get<T>();
}

4
src/Core/SettingsChangesHistory.h
View File

@ -89,7 +89,8 @@ static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> sett
{"ignore_drop_queries_probability", 0, 0, "Allow to ignore drop queries in server with specified probability for testing purposes"},
{"lightweight_deletes_sync", 2, 2, "The same as 'mutation_sync', but controls only execution of lightweight deletes"},
{"query_cache_system_table_handling", "save", "throw", "The query cache no longer caches results of queries against system tables"},
}},
{"input_format_hive_text_allow_variable_number_of_columns", false, true, "Ignore extra columns in Hive Text input (if file has more columns than expected) and treat missing fields in Hive Text input as default values."},
}},
{"24.3", {{"s3_connect_timeout_ms", 1000, 1000, "Introduce new dedicated setting for s3 connection timeout"},
{"allow_experimental_shared_merge_tree", false, true, "The setting is obsolete"},
{"use_page_cache_for_disks_without_file_cache", false, false, "Added userspace page cache"},
@ -129,7 +130,6 @@ static std::map<ClickHouseVersion, SettingsChangesHistory::SettingsChanges> sett
{"azure_max_upload_part_size", 5ull*1024*1024*1024, 5ull*1024*1024*1024, "The maximum size of part to upload during multipart upload to Azure blob storage."},
{"azure_upload_part_size_multiply_factor", 2, 2, "Multiply azure_min_upload_part_size by this factor each time azure_multiply_parts_count_threshold parts were uploaded from a single write to Azure blob storage."},
{"azure_upload_part_size_multiply_parts_count_threshold", 500, 500, "Each time this number of parts was uploaded to Azure blob storage, azure_min_upload_part_size is multiplied by azure_upload_part_size_multiply_factor."},
{"input_format_hive_text_allow_variable_number_of_columns", false, true, "Ignore extra columns in Hive Text input (if file has more columns than expected) and treat missing fields in Hive Text input as default values."},
}},
{"24.2", {{"allow_suspicious_variant_types", true, false, "Don't allow creating Variant type with suspicious variants by default"},
{"validate_experimental_and_suspicious_types_inside_nested_types", false, true, "Validate usage of experimental and suspicious types inside nested types"},

247
src/Interpreters/ActionsDAG.cpp
View File

@ -2135,13 +2135,6 @@ ConjunctionNodes getConjunctionNodes(ActionsDAG::Node * predicate, std::unordere
}
}
// std::cerr << "Allowed " << conjunction.allowed.size() << std::endl;
// for (const auto & node : conjunction.allowed)
// std::cerr << node->result_name << std::endl;
// std::cerr << "Rejected " << conjunction.rejected.size() << std::endl;
// for (const auto & node : conjunction.rejected)
// std::cerr << node->result_name << std::endl;
return conjunction;
}
@ -2170,7 +2163,7 @@ ColumnsWithTypeAndName prepareFunctionArguments(const ActionsDAG::NodeRawConstPt
///
/// Result actions add single column with conjunction result (it is always first in outputs).
/// No other columns are added or removed.
ActionsDAGPtr ActionsDAG::cloneActionsForConjunction(NodeRawConstPtrs conjunction, const ColumnsWithTypeAndName & all_inputs)
ActionsDAGPtr ActionsDAG::createActionsForConjunction(NodeRawConstPtrs conjunction, const ColumnsWithTypeAndName & all_inputs)
{
if (conjunction.empty())
return nullptr;
@ -2265,9 +2258,9 @@ ActionsDAGPtr ActionsDAG::cloneActionsForConjunction(NodeRawConstPtrs conjunctio
return actions;
}
ActionsDAGPtr ActionsDAG::cloneActionsForFilterPushDown(
ActionsDAGPtr ActionsDAG::splitActionsForFilterPushDown(
const std::string & filter_name,
bool can_remove_filter,
bool removes_filter,
const Names & available_inputs,
const ColumnsWithTypeAndName & all_inputs)
{
@ -2321,16 +2314,232 @@ ActionsDAGPtr ActionsDAG::cloneActionsForFilterPushDown(
}
}
auto actions = cloneActionsForConjunction(conjunction.allowed, all_inputs);
auto actions = createActionsForConjunction(conjunction.allowed, all_inputs);
if (!actions)
return nullptr;
/// Now, when actions are created, update the current DAG.
removeUnusedConjunctions(std::move(conjunction.rejected), predicate, removes_filter);
if (conjunction.rejected.empty())
return actions;
}
ActionsDAG::ActionsForJOINFilterPushDown ActionsDAG::splitActionsForJOINFilterPushDown(
const std::string & filter_name,
bool removes_filter,
const Names & left_stream_available_columns_to_push_down,
const Block & left_stream_header,
const Names & right_stream_available_columns_to_push_down,
const Block & right_stream_header,
const Names & equivalent_columns_to_push_down,
const std::unordered_map<std::string, ColumnWithTypeAndName> & equivalent_left_stream_column_to_right_stream_column,
const std::unordered_map<std::string, ColumnWithTypeAndName> & equivalent_right_stream_column_to_left_stream_column)
{
Node * predicate = const_cast<Node *>(tryFindInOutputs(filter_name));
if (!predicate)
throw Exception(ErrorCodes::LOGICAL_ERROR,
"Output nodes for ActionsDAG do not contain filter column name {}. DAG:\n{}",
filter_name,
dumpDAG());
/// If condition is constant let's do nothing.
/// It means there is nothing to push down or optimization was already applied.
if (predicate->type == ActionType::COLUMN)
return {};
auto get_input_nodes = [this](const Names & inputs_names)
{
std::unordered_set<const Node *> allowed_nodes;
std::unordered_map<std::string_view, std::list<const Node *>> inputs_map;
for (const auto & input_node : inputs)
inputs_map[input_node->result_name].emplace_back(input_node);
for (const auto & name : inputs_names)
{
auto & inputs_list = inputs_map[name];
if (inputs_list.empty())
continue;
allowed_nodes.emplace(inputs_list.front());
inputs_list.pop_front();
}
return allowed_nodes;
};
auto left_stream_allowed_nodes = get_input_nodes(left_stream_available_columns_to_push_down);
auto right_stream_allowed_nodes = get_input_nodes(right_stream_available_columns_to_push_down);
auto both_streams_allowed_nodes = get_input_nodes(equivalent_columns_to_push_down);
auto left_stream_push_down_conjunctions = getConjunctionNodes(predicate, left_stream_allowed_nodes);
auto right_stream_push_down_conjunctions = getConjunctionNodes(predicate, right_stream_allowed_nodes);
auto both_streams_push_down_conjunctions = getConjunctionNodes(predicate, both_streams_allowed_nodes);
NodeRawConstPtrs left_stream_allowed_conjunctions = std::move(left_stream_push_down_conjunctions.allowed);
NodeRawConstPtrs right_stream_allowed_conjunctions = std::move(right_stream_push_down_conjunctions.allowed);
std::unordered_set<const Node *> left_stream_allowed_conjunctions_set(left_stream_allowed_conjunctions.begin(), left_stream_allowed_conjunctions.end());
std::unordered_set<const Node *> right_stream_allowed_conjunctions_set(right_stream_allowed_conjunctions.begin(), right_stream_allowed_conjunctions.end());
for (const auto * both_streams_push_down_allowed_conjunction_node : both_streams_push_down_conjunctions.allowed)
{
if (!left_stream_allowed_conjunctions_set.contains(both_streams_push_down_allowed_conjunction_node))
left_stream_allowed_conjunctions.push_back(both_streams_push_down_allowed_conjunction_node);
if (!right_stream_allowed_conjunctions_set.contains(both_streams_push_down_allowed_conjunction_node))
right_stream_allowed_conjunctions.push_back(both_streams_push_down_allowed_conjunction_node);
}
std::unordered_set<const Node *> rejected_conjunctions_set;
rejected_conjunctions_set.insert(left_stream_push_down_conjunctions.rejected.begin(), left_stream_push_down_conjunctions.rejected.end());
rejected_conjunctions_set.insert(right_stream_push_down_conjunctions.rejected.begin(), right_stream_push_down_conjunctions.rejected.end());
rejected_conjunctions_set.insert(both_streams_push_down_conjunctions.rejected.begin(), both_streams_push_down_conjunctions.rejected.end());
for (const auto & left_stream_allowed_conjunction : left_stream_allowed_conjunctions)
rejected_conjunctions_set.erase(left_stream_allowed_conjunction);
for (const auto & right_stream_allowed_conjunction : right_stream_allowed_conjunctions)
rejected_conjunctions_set.erase(right_stream_allowed_conjunction);
NodeRawConstPtrs rejected_conjunctions(rejected_conjunctions_set.begin(), rejected_conjunctions_set.end());
if (rejected_conjunctions.size() == 1)
{
chassert(rejected_conjunctions.front()->result_type);
bool left_stream_push_constant = !left_stream_allowed_conjunctions.empty() && left_stream_allowed_conjunctions[0]->type == ActionType::COLUMN;
bool right_stream_push_constant = !right_stream_allowed_conjunctions.empty() && right_stream_allowed_conjunctions[0]->type == ActionType::COLUMN;
if ((left_stream_push_constant || right_stream_push_constant) && !rejected_conjunctions.front()->result_type->equals(*predicate->result_type))
{
/// No further optimization can be done
return {};
}
}
auto left_stream_filter_to_push_down = createActionsForConjunction(left_stream_allowed_conjunctions, left_stream_header.getColumnsWithTypeAndName());
auto right_stream_filter_to_push_down = createActionsForConjunction(right_stream_allowed_conjunctions, right_stream_header.getColumnsWithTypeAndName());
auto replace_equivalent_columns_in_filter = [](const ActionsDAGPtr & filter,
const Block & stream_header,
const std::unordered_map<std::string, ColumnWithTypeAndName> & columns_to_replace)
{
auto updated_filter = ActionsDAG::buildFilterActionsDAG({filter->getOutputs()[0]}, columns_to_replace);
chassert(updated_filter->getOutputs().size() == 1);
/** If result filter to left or right stream has column that is one of the stream inputs, we need distinguish filter column from
* actual input column. It is necessary because after filter step, filter column became constant column with value 1, and
* not all JOIN algorithms properly work with constants.
*
* Example: SELECT key FROM ( SELECT key FROM t1 ) AS t1 JOIN ( SELECT key FROM t1 ) AS t2 ON t1.key = t2.key WHERE key;
*/
const auto * stream_filter_node = updated_filter->getOutputs()[0];
if (stream_header.has(stream_filter_node->result_name))
{
const auto & alias_node = updated_filter->addAlias(*stream_filter_node, "__filter" + stream_filter_node->result_name);
updated_filter->getOutputs()[0] = &alias_node;
}
std::unordered_map<std::string, std::list<const Node *>> updated_filter_inputs;
for (const auto & input : updated_filter->getInputs())
updated_filter_inputs[input->result_name].push_back(input);
for (const auto & input : filter->getInputs())
{
if (updated_filter_inputs.contains(input->result_name))
continue;
const Node * updated_filter_input_node = nullptr;
auto it = columns_to_replace.find(input->result_name);
if (it != columns_to_replace.end())
updated_filter_input_node = &updated_filter->addInput(it->second);
else
updated_filter_input_node = &updated_filter->addInput({input->column, input->result_type, input->result_name});
updated_filter_inputs[input->result_name].push_back(updated_filter_input_node);
}
for (const auto & input_column : stream_header.getColumnsWithTypeAndName())
{
const Node * input;
auto & list = updated_filter_inputs[input_column.name];
if (list.empty())
{
input = &updated_filter->addInput(input_column);
}
else
{
input = list.front();
list.pop_front();
}
if (input != updated_filter->getOutputs()[0])
updated_filter->outputs.push_back(input);
}
return updated_filter;
};
if (left_stream_filter_to_push_down)
left_stream_filter_to_push_down = replace_equivalent_columns_in_filter(left_stream_filter_to_push_down,
left_stream_header,
equivalent_right_stream_column_to_left_stream_column);
if (right_stream_filter_to_push_down)
right_stream_filter_to_push_down = replace_equivalent_columns_in_filter(right_stream_filter_to_push_down,
right_stream_header,
equivalent_left_stream_column_to_right_stream_column);
/*
* We should check the presence of a split filter column name in stream columns to avoid removing the required column.
*
* Example:
* A filter expression is `a AND b = c`, but `b` and `c` belong to another side of the join and not in allowed columns to push down,
* so the final split filter is just `a`.
* In this case `a` can be in stream columns but not `and(a, equals(b, c))`.
*/
bool left_stream_filter_removes_filter = true;
bool right_stream_filter_removes_filter = true;
if (left_stream_filter_to_push_down)
{
const auto & left_stream_filter_column_name = left_stream_filter_to_push_down->getOutputs()[0]->result_name;
left_stream_filter_removes_filter = !left_stream_header.has(left_stream_filter_column_name);
}
if (right_stream_filter_to_push_down)
{
const auto & right_stream_filter_column_name = right_stream_filter_to_push_down->getOutputs()[0]->result_name;
right_stream_filter_removes_filter = !right_stream_header.has(right_stream_filter_column_name);
}
ActionsDAG::ActionsForJOINFilterPushDown result
{
.left_stream_filter_to_push_down = std::move(left_stream_filter_to_push_down),
.left_stream_filter_removes_filter = left_stream_filter_removes_filter,
.right_stream_filter_to_push_down = std::move(right_stream_filter_to_push_down),
.right_stream_filter_removes_filter = right_stream_filter_removes_filter
};
if (!result.left_stream_filter_to_push_down && !result.right_stream_filter_to_push_down)
return result;
/// Now, when actions are created, update the current DAG.
removeUnusedConjunctions(std::move(rejected_conjunctions), predicate, removes_filter);
return result;
}
void ActionsDAG::removeUnusedConjunctions(NodeRawConstPtrs rejected_conjunctions, Node * predicate, bool removes_filter)
{
if (rejected_conjunctions.empty())
{
/// The whole predicate was split.
if (can_remove_filter)
if (removes_filter)
{
/// If filter column is not needed, remove it from output nodes.
std::erase_if(outputs, [&](const Node * node) { return node == predicate; });
@ -2362,7 +2571,7 @@ ActionsDAGPtr ActionsDAG::cloneActionsForFilterPushDown(
{
/// Predicate is conjunction, where both allowed and rejected sets are not empty.
NodeRawConstPtrs new_children = std::move(conjunction.rejected);
NodeRawConstPtrs new_children = std::move(rejected_conjunctions);
if (new_children.size() == 1 && new_children.front()->result_type->equals(*predicate->result_type))
{
@ -2403,13 +2612,12 @@ ActionsDAGPtr ActionsDAG::cloneActionsForFilterPushDown(
std::unordered_set<const Node *> used_inputs;
for (const auto * input : inputs)
{
if (can_remove_filter && input == predicate)
if (removes_filter && input == predicate)
continue;
used_inputs.insert(input);
}
removeUnusedActions(used_inputs);
return actions;
}
static bool isColumnSortingPreserved(const ActionsDAG::Node * start_node, const String & sorted_column)
@ -2557,8 +2765,11 @@ ActionsDAGPtr ActionsDAG::buildFilterActionsDAG(
auto input_node_it = node_name_to_input_node_column.find(node->result_name);
if (input_node_it != node_name_to_input_node_column.end())
{
result_node = &result_dag->addInput(input_node_it->second);
node_to_result_node.emplace(node, result_node);
auto & result_input = result_inputs[input_node_it->second.name];
if (!result_input)
result_input = &result_dag->addInput(input_node_it->second);
node_to_result_node.emplace(node, result_input);
nodes_to_process.pop_back();
continue;
}

42
src/Interpreters/ActionsDAG.h
View File

@ -372,12 +372,46 @@ public:
/// columns will be transformed like `x, y, z` -> `z > 0, z, x, y` -(remove filter)-> `z, x, y`.
/// To avoid it, add inputs from `all_inputs` list,
/// so actions `x, y, z -> z > 0, x, y, z` -(remove filter)-> `x, y, z` will not change columns order.
ActionsDAGPtr cloneActionsForFilterPushDown(
ActionsDAGPtr splitActionsForFilterPushDown(
const std::string & filter_name,
bool can_remove_filter,
bool removes_filter,
const Names & available_inputs,
const ColumnsWithTypeAndName & all_inputs);
struct ActionsForJOINFilterPushDown
{
ActionsDAGPtr left_stream_filter_to_push_down;
bool left_stream_filter_removes_filter;
ActionsDAGPtr right_stream_filter_to_push_down;
bool right_stream_filter_removes_filter;
};
/** Split actions for JOIN filter push down.
*
* @param filter_name - name of filter node in current DAG.
* @param removes_filter - if filter is removed after it is applied.
* @param left_stream_available_columns_to_push_down - columns from left stream that are safe to use in push down conditions
* to left stream.
* @param left_stream_header - left stream header.
* @param right_stream_available_columns_to_push_down - columns from right stream that are safe to use in push down conditions
* to right stream.
* @param right_stream_header - right stream header.
* @param equivalent_columns_to_push_down - columns from left and right streams that are safe to use in push down conditions
* to left and right streams.
* @param equivalent_left_stream_column_to_right_stream_column - equivalent left stream column name to right stream column map.
* @param equivalent_right_stream_column_to_left_stream_column - equivalent right stream column name to left stream column map.
*/
ActionsForJOINFilterPushDown splitActionsForJOINFilterPushDown(
const std::string & filter_name,
bool removes_filter,
const Names & left_stream_available_columns_to_push_down,
const Block & left_stream_header,
const Names & right_stream_available_columns_to_push_down,
const Block & right_stream_header,
const Names & equivalent_columns_to_push_down,
const std::unordered_map<std::string, ColumnWithTypeAndName> & equivalent_left_stream_column_to_right_stream_column,
const std::unordered_map<std::string, ColumnWithTypeAndName> & equivalent_right_stream_column_to_left_stream_column);
bool
isSortingPreserved(const Block & input_header, const SortDescription & sort_description, const String & ignore_output_column = "") const;
@ -429,7 +463,9 @@ private:
void compileFunctions(size_t min_count_to_compile_expression, const std::unordered_set<const Node *> & lazy_executed_nodes = {});
#endif
static ActionsDAGPtr cloneActionsForConjunction(NodeRawConstPtrs conjunction, const ColumnsWithTypeAndName & all_inputs);
static ActionsDAGPtr createActionsForConjunction(NodeRawConstPtrs conjunction, const ColumnsWithTypeAndName & all_inputs);
void removeUnusedConjunctions(NodeRawConstPtrs rejected_conjunctions, Node * predicate, bool removes_filter);
};
class FindOriginalNodeForOutputName

2
src/Interpreters/BackupLog.cpp
View File

@ -24,6 +24,7 @@ ColumnsDescription BackupLogElement::getColumnsDescription()
{
{"hostname", std::make_shared<DataTypeLowCardinality>(std::make_shared<DataTypeString>()), "Hostname of the server executing the query."},
{"event_date", std::make_shared<DataTypeDate>(), "Date of the entry."},
{"event_time", std::make_shared<DataTypeDateTime>(), "Time of the entry."},
{"event_time_microseconds", std::make_shared<DataTypeDateTime64>(6), "Time of the entry with microseconds precision."},
{"id", std::make_shared<DataTypeString>(), "Identifier of the backup or restore operation."},
{"name", std::make_shared<DataTypeString>(), "Name of the backup storage (the contents of the FROM or TO clause)."},
@ -48,6 +49,7 @@ void BackupLogElement::appendToBlock(MutableColumns & columns) const
size_t i = 0;
columns[i++]->insert(getFQDNOrHostName());
columns[i++]->insert(DateLUT::instance().toDayNum(std::chrono::system_clock::to_time_t(event_time)).toUnderType());
columns[i++]->insert(std::chrono::system_clock::to_time_t(event_time));
columns[i++]->insert(event_time_usec);
columns[i++]->insert(info.id);
columns[i++]->insert(info.name);

1
src/Parsers/ASTOrderByElement.cpp
View File

@ -1,3 +1,4 @@
#include <Columns/Collator.h>
#include <Parsers/ASTOrderByElement.h>
#include <Common/SipHash.h>
#include <IO/Operators.h>

2
src/Parsers/CommonParsers.h
View File

@ -601,8 +601,6 @@ public:
constexpr const char * getName() const override { return s.data(); }
Highlight highlight() const override { return Highlight::keyword; }
protected:
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
};

4
src/Parsers/ExpressionElementParsers.cpp
View File

@ -278,7 +278,7 @@ bool ParserTableAsStringLiteralIdentifier::parseImpl(Pos & pos, ASTPtr & node, E
bool ParserCompoundIdentifier::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
ASTPtr id_list;
if (!ParserList(std::make_unique<ParserIdentifier>(allow_query_parameter, highlight_type), std::make_unique<ParserToken>(TokenType::Dot), false)
if (!ParserList(std::make_unique<ParserIdentifier>(allow_query_parameter), std::make_unique<ParserToken>(TokenType::Dot), false)
.parse(pos, id_list, expected))
return false;
@ -1491,7 +1491,7 @@ const char * ParserAlias::restricted_keywords[] =
bool ParserAlias::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
ParserKeyword s_as(Keyword::AS);
ParserIdentifier id_p(false, Highlight::alias);
ParserIdentifier id_p;
bool has_as_word = s_as.ignore(pos, expected);
if (!allow_alias_without_as_keyword && !has_as_word)

13
src/Parsers/ExpressionElementParsers.h
View File

@ -25,15 +25,12 @@ protected:
class ParserIdentifier : public IParserBase
{
public:
explicit ParserIdentifier(bool allow_query_parameter_ = false, Highlight highlight_type_ = Highlight::identifier)
: allow_query_parameter(allow_query_parameter_), highlight_type(highlight_type_) {}
Highlight highlight() const override { return highlight_type; }
explicit ParserIdentifier(bool allow_query_parameter_ = false) : allow_query_parameter(allow_query_parameter_) {}
protected:
const char * getName() const override { return "identifier"; }
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
bool allow_query_parameter;
Highlight highlight_type;
};
@ -56,8 +53,8 @@ protected:
class ParserCompoundIdentifier : public IParserBase
{
public:
explicit ParserCompoundIdentifier(bool table_name_with_optional_uuid_ = false, bool allow_query_parameter_ = false, Highlight highlight_type_ = Highlight::identifier)
: table_name_with_optional_uuid(table_name_with_optional_uuid_), allow_query_parameter(allow_query_parameter_), highlight_type(highlight_type_)
explicit ParserCompoundIdentifier(bool table_name_with_optional_uuid_ = false, bool allow_query_parameter_ = false)
: table_name_with_optional_uuid(table_name_with_optional_uuid_), allow_query_parameter(allow_query_parameter_)
{
}
@ -66,7 +63,6 @@ protected:
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
bool table_name_with_optional_uuid;
bool allow_query_parameter;
Highlight highlight_type;
};
/** *, t.*, db.table.*, COLUMNS('<regular expression>') APPLY(...) or EXCEPT(...) or REPLACE(...)
@ -257,7 +253,6 @@ class ParserNumber : public IParserBase
protected:
const char * getName() const override { return "number"; }
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
Highlight highlight() const override { return Highlight::number; }
};
/** Unsigned integer, used in right hand side of tuple access operator (x.1).
@ -278,7 +273,6 @@ class ParserStringLiteral : public IParserBase
protected:
const char * getName() const override { return "string literal"; }
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
Highlight highlight() const override { return Highlight::string; }
};
@ -391,7 +385,6 @@ class ParserSubstitution : public IParserBase
protected:
const char * getName() const override { return "substitution"; }
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override;
Highlight highlight() const override { return Highlight::substitution; }
};

91
src/Parsers/ExpressionListParsers.cpp
View File

@ -441,21 +441,6 @@ bool ParserKeyValuePairsList::parseImpl(Pos & pos, ASTPtr & node, Expected & exp
return parser.parse(pos, node, expected);
}
namespace
{
/// This wrapper is needed to highlight function names differently.
class ParserFunctionName : public IParserBase
{
protected:
const char * getName() const override { return "function name"; }
bool parseImpl(Pos & pos, ASTPtr & node, Expected & expected) override
{
ParserCompoundIdentifier parser(false, true, Highlight::function);
return parser.parse(pos, node, expected);
}
};
}
enum class Action
{
@ -824,7 +809,6 @@ struct ParserExpressionImpl
static const Operator finish_between_operator;
ParserFunctionName function_name_parser;
ParserCompoundIdentifier identifier_parser{false, true};
ParserNumber number_parser;
ParserAsterisk asterisk_parser;
@ -2375,7 +2359,7 @@ bool ParserFunction::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
ASTPtr identifier;
if (ParserFunctionName().parse(pos, identifier, expected)
if (ParserCompoundIdentifier(false,true).parse(pos, identifier, expected)
&& ParserToken(TokenType::OpeningRoundBracket).ignore(pos, expected))
{
auto start = getFunctionLayer(identifier, is_table_function, allow_function_parameters);
@ -2513,7 +2497,7 @@ Action ParserExpressionImpl::tryParseOperand(Layers & layers, IParser::Pos & pos
{
if (typeid_cast<ViewLayer *>(layers.back().get()) || typeid_cast<KustoLayer *>(layers.back().get()))
{
if (function_name_parser.parse(pos, tmp, expected)
if (identifier_parser.parse(pos, tmp, expected)
&& ParserToken(TokenType::OpeningRoundBracket).ignore(pos, expected))
{
layers.push_back(getFunctionLayer(tmp, layers.front()->is_table_function));
@ -2645,53 +2629,50 @@ Action ParserExpressionImpl::tryParseOperand(Layers & layers, IParser::Pos & pos
{
layers.back()->pushOperand(std::move(tmp));
}
else
else if (identifier_parser.parse(pos, tmp, expected))
{
old_pos = pos;
if (function_name_parser.parse(pos, tmp, expected) && pos->type == TokenType::OpeningRoundBracket)
if (pos->type == TokenType::OpeningRoundBracket)
{
++pos;
layers.push_back(getFunctionLayer(tmp, layers.front()->is_table_function));
return Action::OPERAND;
}
pos = old_pos;
if (identifier_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
}
else if (substitution_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
}
else if (pos->type == TokenType::OpeningRoundBracket)
{
if (subquery_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
return Action::OPERATOR;
}
++pos;
layers.push_back(std::make_unique<RoundBracketsLayer>());
return Action::OPERAND;
}
else if (pos->type == TokenType::OpeningSquareBracket)
{
++pos;
layers.push_back(std::make_unique<ArrayLayer>());
return Action::OPERAND;
}
else if (mysql_global_variable_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
}
else
{
return Action::NONE;
layers.back()->pushOperand(std::move(tmp));
}
}
else if (substitution_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
}
else if (pos->type == TokenType::OpeningRoundBracket)
{
if (subquery_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
return Action::OPERATOR;
}
++pos;
layers.push_back(std::make_unique<RoundBracketsLayer>());
return Action::OPERAND;
}
else if (pos->type == TokenType::OpeningSquareBracket)
{
++pos;
layers.push_back(std::make_unique<ArrayLayer>());
return Action::OPERAND;
}
else if (mysql_global_variable_parser.parse(pos, tmp, expected))
{
layers.back()->pushOperand(std::move(tmp));
}
else
{
return Action::NONE;
}
return Action::OPERATOR;
}

23
src/Parsers/IParser.cpp
View File

@ -9,7 +9,6 @@ namespace ErrorCodes
extern const int TOO_SLOW_PARSING;
}
IParser::Pos & IParser::Pos::operator=(const IParser::Pos & rhs)
{
depth = rhs.depth;
@ -33,26 +32,4 @@ IParser::Pos & IParser::Pos::operator=(const IParser::Pos & rhs)
return *this;
}
template <typename T>
static bool intersects(T a_begin, T a_end, T b_begin, T b_end)
{
return (a_begin <= b_begin && b_begin < a_end)
|| (b_begin <= a_begin && a_begin < b_end);
}
void Expected::highlight(HighlightedRange range)
{
auto it = highlights.lower_bound(range);
while (it != highlights.end() && range.begin < it->end)
{
if (intersects(range.begin, range.end, it->begin, it->end))
it = highlights.erase(it);
else
++it;
}
highlights.insert(range);
}
}

39
src/Parsers/IParser.h
View File

@ -1,7 +1,6 @@
#pragma once
#include <absl/container/inlined_vector.h>
#include <set>
#include <algorithm>
#include <memory>
@ -22,42 +21,14 @@ namespace ErrorCodes
extern const int LOGICAL_ERROR;
}
enum class Highlight
{
none = 0,
keyword,
identifier,
function,
alias,
substitution,
number,
string,
};
struct HighlightedRange
{
const char * begin;
const char * end;
Highlight highlight;
auto operator<=>(const HighlightedRange & other) const
{
return begin <=> other.begin;
}
};
/** Collects variants, how parser could proceed further at rightmost position.
* Also collects a mapping of parsed ranges for highlighting,
* which is accumulated through the parsing.
*/
struct Expected
{
absl::InlinedVector<const char *, 7> variants;
const char * max_parsed_pos = nullptr;
std::set<HighlightedRange> highlights;
/// 'description' should be statically allocated string.
ALWAYS_INLINE void add(const char * current_pos, const char * description)
{
@ -77,8 +48,6 @@ struct Expected
{
add(it->begin, description);
}
void highlight(HighlightedRange range);
};
@ -189,14 +158,6 @@ public:
return parse(pos, node, expected);
}
/** If the parsed fragment should be highlighted in the query editor,
* which type of highlighting to use?
*/
virtual Highlight highlight() const
{
return Highlight::none;
}
virtual ~IParser() = default;
};

19
src/Parsers/IParserBase.cpp
View File

@ -10,25 +10,8 @@ bool IParserBase::parse(Pos & pos, ASTPtr & node, Expected & expected)
return wrapParseImpl(pos, IncreaseDepthTag{}, [&]
{
const char * begin = pos->begin;
bool res = parseImpl(pos, node, expected);
if (res)
{
Highlight type = highlight();
if (pos->begin > begin && type != Highlight::none)
{
Pos prev_token = pos;
--prev_token;
HighlightedRange range;
range.begin = begin;
range.end = prev_token->end;
range.highlight = type;
expected.highlight(range);
}
}
else
if (!res)
node = nullptr;
return res;
});

4
src/Parsers/ParserInsertQuery.cpp
View File

@ -40,6 +40,7 @@ bool ParserInsertQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
ParserKeyword s_with(Keyword::WITH);
ParserToken s_lparen(TokenType::OpeningRoundBracket);
ParserToken s_rparen(TokenType::ClosingRoundBracket);
ParserToken s_semicolon(TokenType::Semicolon);
ParserIdentifier name_p(true);
ParserList columns_p(std::make_unique<ParserInsertElement>(), std::make_unique<ParserToken>(TokenType::Comma), false);
ParserFunction table_function_p{false};
@ -146,9 +147,8 @@ bool ParserInsertQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expected)
{
/// If VALUES is defined in query, everything except setting will be parsed as data,
/// and if values followed by semicolon, the data should be null.
if (pos->type != TokenType::Semicolon)
if (!s_semicolon.checkWithoutMoving(pos, expected))
data = pos->begin;
format_str = "Values";
}
else if (s_format.ignore(pos, expected))

12
src/Parsers/ParserSetQuery.cpp
View File

@ -210,12 +210,8 @@ bool ParserSetQuery::parseNameValuePair(SettingChange & change, IParser::Pos & p
if (!s_eq.ignore(pos, expected))
return false;
if (ParserKeyword(Keyword::TRUE_KEYWORD).ignore(pos, expected))
value = std::make_shared<ASTLiteral>(Field(static_cast<UInt64>(1)));
else if (ParserKeyword(Keyword::FALSE_KEYWORD).ignore(pos, expected))
value = std::make_shared<ASTLiteral>(Field(static_cast<UInt64>(0)));
/// for SETTINGS disk=disk(type='s3', path='', ...)
else if (function_p.parse(pos, function_ast, expected) && function_ast->as<ASTFunction>()->name == "disk")
if (function_p.parse(pos, function_ast, expected) && function_ast->as<ASTFunction>()->name == "disk")
{
tryGetIdentifierNameInto(name, change.name);
change.value = createFieldFromAST(function_ast);
@ -276,11 +272,7 @@ bool ParserSetQuery::parseNameValuePairWithParameterOrDefault(
}
/// Setting
if (ParserKeyword(Keyword::TRUE_KEYWORD).ignore(pos, expected))
node = std::make_shared<ASTLiteral>(Field(static_cast<UInt64>(1)));
else if (ParserKeyword(Keyword::FALSE_KEYWORD).ignore(pos, expected))
node = std::make_shared<ASTLiteral>(Field(static_cast<UInt64>(0)));
else if (function_p.parse(pos, function_ast, expected) && function_ast->as<ASTFunction>()->name == "disk")
if (function_p.parse(pos, function_ast, expected) && function_ast->as<ASTFunction>()->name == "disk")
{
change.name = name;
change.value = createFieldFromAST(function_ast);

15
src/Parsers/parseDatabaseAndTableName.cpp
View File

@ -60,6 +60,21 @@ bool parseDatabaseAndTableAsAST(IParser::Pos & pos, Expected & expected, ASTPtr
}
bool parseDatabase(IParser::Pos & pos, Expected & expected, String & database_str)
{
ParserToken s_dot(TokenType::Dot);
ParserIdentifier identifier_parser;
ASTPtr database;
database_str = "";
if (!identifier_parser.parse(pos, database, expected))
return false;
tryGetIdentifierNameInto(database, database_str);
return true;
}
bool parseDatabaseAsAST(IParser::Pos & pos, Expected & expected, ASTPtr & database)
{
ParserIdentifier identifier_parser(/* allow_query_parameter */true);

58
src/Parsers/parseQuery.cpp
View File

@ -226,32 +226,6 @@ std::string getUnmatchedParenthesesErrorMessage(
}
static ASTInsertQuery * getInsertAST(const ASTPtr & ast)
{
/// Either it is INSERT or EXPLAIN INSERT.
if (auto * explain = ast->as<ASTExplainQuery>())
{
if (auto explained_query = explain->getExplainedQuery())
{
return explained_query->as<ASTInsertQuery>();
}
}
else
{
return ast->as<ASTInsertQuery>();
}
return nullptr;
}
const char * getInsertData(const ASTPtr & ast)
{
if (const ASTInsertQuery * insert = getInsertAST(ast))
return insert->data;
return nullptr;
}
ASTPtr tryParseQuery(
IParser & parser,
const char * & _out_query_end, /* also query begin as input parameter */
@ -296,11 +270,29 @@ ASTPtr tryParseQuery(
if (res && max_parser_depth)
res->checkDepth(max_parser_depth);
/// If parsed query ends at data for insertion. Data for insertion could be
/// in any format and not necessary be lexical correct, so we can't perform
/// most of the checks.
if (res && getInsertData(res))
ASTInsertQuery * insert = nullptr;
if (parse_res)
{
if (auto * explain = res->as<ASTExplainQuery>())
{
if (auto explained_query = explain->getExplainedQuery())
{
insert = explained_query->as<ASTInsertQuery>();
}
}
else
{
insert = res->as<ASTInsertQuery>();
}
}
// If parsed query ends at data for insertion. Data for insertion could be
// in any format and not necessary be lexical correct, so we can't perform
// most of the checks.
if (insert && insert->data)
{
return res;
}
// More granular checks for queries other than INSERT w/inline data.
/// Lexical error
@ -442,9 +434,11 @@ std::pair<const char *, bool> splitMultipartQuery(
ast = parseQueryAndMovePosition(parser, pos, end, "", true, max_query_size, max_parser_depth, max_parser_backtracks);
if (ASTInsertQuery * insert = getInsertAST(ast))
auto * insert = ast->as<ASTInsertQuery>();
if (insert && insert->data)
{
/// Data for INSERT is broken on the new line
/// Data for INSERT is broken on new line
pos = insert->data;
while (*pos && *pos != '\n')
++pos;

5
src/Parsers/parseQuery.h
View File

@ -71,9 +71,4 @@ std::pair<const char *, bool> splitMultipartQuery(
size_t max_parser_backtracks,
bool allow_settings_after_format_in_insert);
/** If the query contains raw data part, such as INSERT ... FORMAT ..., return a pointer to it.
* The SQL parser stops at the raw data part, which is parsed by a separate parser.
*/
const char * getInsertData(const ASTPtr & ast);
}

76
src/Planner/PlannerActionsVisitor.cpp
View File

@ -39,6 +39,7 @@ namespace ErrorCodes
extern const int UNSUPPORTED_METHOD;
extern const int LOGICAL_ERROR;
extern const int BAD_ARGUMENTS;
extern const int INCORRECT_QUERY;
}
namespace
@ -500,7 +501,41 @@ public:
ActionsDAG::NodeRawConstPtrs visit(QueryTreeNodePtr expression_node);
private:
using NodeNameAndNodeMinLevel = std::pair<std::string, size_t>;
class Levels
{
public:
explicit Levels(size_t level) { set(level); }
void set(size_t level)
{
check(level);
if (level)
mask |= (uint64_t(1) << (level - 1));
}
void reset(size_t level)
{
check(level);
if (level)
mask &= ~(uint64_t(1) << (level - 1));
}
void add(Levels levels) { mask |= levels.mask; }
size_t max() const { return 64 - getLeadingZeroBits(mask); }
private:
uint64_t mask = 0;
void check(size_t level)
{
if (level > 64)
throw Exception(ErrorCodes::INCORRECT_QUERY, "Maximum lambda depth exceeded. Maximum 64.");
}
};
using NodeNameAndNodeMinLevel = std::pair<std::string, Levels>;
NodeNameAndNodeMinLevel visitImpl(QueryTreeNodePtr node);
@ -586,11 +621,11 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
column_source->getNodeType() == QueryTreeNodeType::LAMBDA &&
actions_stack[i].getScopeNode().get() == column_source.get())
{
return {column_node_name, i};
return {column_node_name, Levels(i)};
}
}
return {column_node_name, 0};
return {column_node_name, Levels(0)};
}
PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::visitConstant(const QueryTreeNodePtr & node)
@ -660,7 +695,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
actions_stack_node.addInputConstantColumnIfNecessary(constant_node_name, column);
}
return {constant_node_name, 0};
return {constant_node_name, Levels(0)};
}
@ -688,7 +723,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
auto lambda_actions_dag = std::make_shared<ActionsDAG>();
actions_stack.emplace_back(lambda_actions_dag, node);
auto [lambda_expression_node_name, level] = visitImpl(lambda_node.getExpression());
auto [lambda_expression_node_name, levels] = visitImpl(lambda_node.getExpression());
lambda_actions_dag->getOutputs().push_back(actions_stack.back().getNodeOrThrow(lambda_expression_node_name));
lambda_actions_dag->removeUnusedActions(Names(1, lambda_expression_node_name));
@ -699,8 +734,9 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
ActionsDAG::NodeRawConstPtrs lambda_children;
Names required_column_names = lambda_actions->getRequiredColumns();
if (level == actions_stack.size() - 1)
--level;
actions_stack.pop_back();
levels.reset(actions_stack.size());
size_t level = levels.max();
const auto & lambda_argument_names = lambda_node.getArgumentNames();
@ -718,7 +754,6 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
auto lambda_node_name = calculateActionNodeName(node, *planner_context);
auto function_capture = std::make_shared<FunctionCaptureOverloadResolver>(
lambda_actions, captured_column_names, lambda_arguments_names_and_types, lambda_node.getExpression()->getResultType(), lambda_expression_node_name);
actions_stack.pop_back();
// TODO: Pass IFunctionBase here not FunctionCaptureOverloadResolver.
const auto * actions_node = actions_stack[level].addFunctionIfNecessary(lambda_node_name, std::move(lambda_children), function_capture);
@ -735,7 +770,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
actions_stack_node.addInputColumnIfNecessary(lambda_node_name, result_type);
}
return {lambda_node_name, level};
return {lambda_node_name, levels};
}
PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::makeSetForInFunction(const QueryTreeNodePtr & node)
@ -799,7 +834,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::ma
actions_stack_node.addInputConstantColumnIfNecessary(column.name, column);
}
return {column.name, 0};
return {column.name, Levels(0)};
}
PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::visitIndexHintFunction(const QueryTreeNodePtr & node)
@ -833,7 +868,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
size_t index_hint_function_level = actions_stack.size() - 1;
actions_stack[index_hint_function_level].addFunctionIfNecessary(function_node_name, {}, index_hint_function_overload_resolver);
return {function_node_name, index_hint_function_level};
return {function_node_name, Levels(index_hint_function_level)};
}
PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::visitFunction(const QueryTreeNodePtr & node)
@ -868,7 +903,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
actions_stack_node.addInputColumnIfNecessary(function_node_name, function_node.getResultType());
}
return {function_node_name, 0};
return {function_node_name, Levels(0)};
}
const auto & function_arguments = function_node.getArguments().getNodes();
@ -877,14 +912,14 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
Names function_arguments_node_names;
function_arguments_node_names.reserve(function_arguments_size);
size_t level = 0;
Levels levels(0);
for (size_t function_argument_index = 0; function_argument_index < function_arguments_size; ++function_argument_index)
{
if (in_function_second_argument_node_name_with_level && function_argument_index == 1)
{
auto & [node_name, node_min_level] = *in_function_second_argument_node_name_with_level;
auto & [node_name, node_levels] = *in_function_second_argument_node_name_with_level;
function_arguments_node_names.push_back(std::move(node_name));
level = std::max(level, node_min_level);
levels.add(node_levels);
continue;
}
@ -892,20 +927,21 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
if (argument->getNodeType() == QueryTreeNodeType::LAMBDA)
{
auto [node_name, node_min_level] = visitLambda(argument);
auto [node_name, node_levels] = visitLambda(argument);
function_arguments_node_names.push_back(std::move(node_name));
level = std::max(level, node_min_level);
levels.add(node_levels);
continue;
}
auto [node_name, node_min_level] = visitImpl(argument);
auto [node_name, node_levels] = visitImpl(argument);
function_arguments_node_names.push_back(std::move(node_name));
level = std::max(level, node_min_level);
levels.add(node_levels);
}
ActionsDAG::NodeRawConstPtrs children;
children.reserve(function_arguments_size);
size_t level = levels.max();
for (auto & function_argument_node_name : function_arguments_node_names)
children.push_back(actions_stack[level].getNodeOrThrow(function_argument_node_name));
@ -930,7 +966,7 @@ PlannerActionsVisitorImpl::NodeNameAndNodeMinLevel PlannerActionsVisitorImpl::vi
actions_stack_node.addInputColumnIfNecessary(function_node_name, function_node.getResultType());
}
return {function_node_name, level};
return {function_node_name, levels};
}
}

1
src/Processors/QueryPlan/FilterStep.h
View File

@ -24,6 +24,7 @@ public:
void describeActions(FormatSettings & settings) const override;
const ActionsDAGPtr & getExpression() const { return actions_dag; }
ActionsDAGPtr & getExpression() { return actions_dag; }
const String & getFilterColumnName() const { return filter_column_name; }
bool removesFilterColumn() const { return remove_filter_column; }

319
src/Processors/QueryPlan/Optimizations/filterPushDown.cpp
View File

@ -100,7 +100,7 @@ static NameSet findIdentifiersOfNode(const ActionsDAG::Node * node)
return res;
}
static ActionsDAGPtr splitFilter(QueryPlan::Node * parent_node, const Names & allowed_inputs, size_t child_idx = 0)
static ActionsDAGPtr splitFilter(QueryPlan::Node * parent_node, const Names & available_inputs, size_t child_idx = 0)
{
QueryPlan::Node * child_node = parent_node->children.front();
checkChildrenSize(child_node, child_idx + 1);
@ -114,14 +114,12 @@ static ActionsDAGPtr splitFilter(QueryPlan::Node * parent_node, const Names & al
bool removes_filter = filter->removesFilterColumn();
const auto & all_inputs = child->getInputStreams()[child_idx].header.getColumnsWithTypeAndName();
auto split_filter = expression->cloneActionsForFilterPushDown(filter_column_name, removes_filter, allowed_inputs, all_inputs);
return split_filter;
return expression->splitActionsForFilterPushDown(filter_column_name, removes_filter, available_inputs, all_inputs);
}
static size_t
tryAddNewFilterStep(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, const ActionsDAGPtr & split_filter,
bool can_remove_filter = true, size_t child_idx = 0)
addNewFilterStepOrThrow(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, const ActionsDAGPtr & split_filter,
bool can_remove_filter = true, size_t child_idx = 0, bool update_parent_filter = true)
{
QueryPlan::Node * child_node = parent_node->children.front();
checkChildrenSize(child_node, child_idx + 1);
@ -134,21 +132,18 @@ tryAddNewFilterStep(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, con
const auto & filter_column_name = filter->getFilterColumnName();
const auto * filter_node = expression->tryFindInOutputs(filter_column_name);
if (!filter_node && !filter->removesFilterColumn())
if (update_parent_filter && !filter_node && !filter->removesFilterColumn())
throw Exception(ErrorCodes::LOGICAL_ERROR,
"Filter column {} was removed from ActionsDAG but it is needed in result. DAG:\n{}",
filter_column_name, expression->dumpDAG());
/// Filter column was replaced to constant.
const bool filter_is_constant = filter_node && filter_node->column && isColumnConst(*filter_node->column);
/// Add new Filter step before Aggregating.
/// Expression/Filter -> Aggregating -> Something
/// Add new Filter step before Child.
/// Expression/Filter -> Child -> Something
auto & node = nodes.emplace_back();
node.children.emplace_back(&node);
std::swap(node.children[0], child_node->children[child_idx]);
/// Expression/Filter -> Aggregating -> Filter -> Something
/// Expression/Filter -> Child -> Filter -> Something
/// New filter column is the first one.
String split_filter_column_name = split_filter->getOutputs().front()->result_name;
@ -171,12 +166,22 @@ tryAddNewFilterStep(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, con
ErrorCodes::LOGICAL_ERROR, "We are trying to push down a filter through a step for which we cannot update input stream");
}
if (!filter_node || filter_is_constant)
/// This means that all predicates of filter were pushed down.
/// Replace current actions to expression, as we don't need to filter anything.
parent = std::make_unique<ExpressionStep>(child->getOutputStream(), expression);
else
filter->updateInputStream(child->getOutputStream());
if (update_parent_filter)
{
/// Filter column was replaced to constant.
const bool filter_is_constant = filter_node && filter_node->column && isColumnConst(*filter_node->column);
if (!filter_node || filter_is_constant)
{
/// This means that all predicates of filter were pushed down.
/// Replace current actions to expression, as we don't need to filter anything.
parent = std::make_unique<ExpressionStep>(child->getOutputStream(), expression);
}
else
{
filter->updateInputStream(child->getOutputStream());
}
}
return 3;
}
@ -186,7 +191,7 @@ tryAddNewFilterStep(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, con
bool can_remove_filter = true, size_t child_idx = 0)
{
if (auto split_filter = splitFilter(parent_node, allowed_inputs, child_idx))
return tryAddNewFilterStep(parent_node, nodes, split_filter, can_remove_filter, child_idx);
return addNewFilterStepOrThrow(parent_node, nodes, split_filter, can_remove_filter, child_idx);
return 0;
}
@ -204,6 +209,204 @@ static size_t simplePushDownOverStep(QueryPlan::Node * parent_node, QueryPlan::N
return 0;
}
static size_t tryPushDownOverJoinStep(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes, QueryPlanStepPtr & child)
{
auto & parent = parent_node->step;
auto * filter = assert_cast<FilterStep *>(parent.get());
auto * join = typeid_cast<JoinStep *>(child.get());
auto * filled_join = typeid_cast<FilledJoinStep *>(child.get());
if (!join && !filled_join)
return 0;
/** For equivalent JOIN with condition `ON lhs.x_1 = rhs.y_1 AND lhs.x_2 = rhs.y_2 ...`, we can build equivalent sets of columns and this
* will allow to push conditions that only use columns from equivalent sets to both sides of JOIN, without considering JOIN type.
*
* For example: `FROM lhs INNER JOIN rhs ON lhs.id = rhs.id AND lhs.value = rhs.value`
* In this example columns `id` and `value` from both tables are equivalent.
*
* During filter push down for different JOIN types filter push down logic is different:
*
* 1. For INNER JOIN we can push all valid conditions to both sides of JOIN. We also can push all valid conditions that use columns from
* equivalent sets to both sides of JOIN.
* 2. For LEFT/RIGHT JOIN we can push conditions that use columns from LEFT/RIGHT stream to LEFT/RIGHT JOIN side. We can also push conditions
* that use columns from LEFT/RIGHT equivalent sets to RIGHT/LEFT JOIN side.
*
* Additional filter push down optimizations:
* 1. TODO: Support building equivalent sets for more than 2 JOINS. It is possible, but will require more complex analysis step.
* 2. TODO: Support building equivalent sets for JOINs with more than 1 clause.
* 3. TODO: For LEFT/RIGHT JOIN, we can assume that RIGHT/LEFT columns used in filter will be default/NULL constants and
* check if filter will always be false, in those scenario we can transform LEFT/RIGHT JOIN into INNER JOIN and push conditions to both tables.
* 4. TODO: It is possible to pull up filter conditions from LEFT/RIGHT stream and push conditions that use columns from LEFT/RIGHT equivalent sets
* to RIGHT/LEFT JOIN side.
*/
const auto & join_header = child->getOutputStream().header;
const auto & table_join = join ? join->getJoin()->getTableJoin() : filled_join->getJoin()->getTableJoin();
const auto & left_stream_input_header = child->getInputStreams().front().header;
const auto & right_stream_input_header = child->getInputStreams().back().header;
if (table_join.kind() == JoinKind::Full)
return 0;
std::unordered_map<std::string, ColumnWithTypeAndName> equivalent_left_stream_column_to_right_stream_column;
std::unordered_map<std::string, ColumnWithTypeAndName> equivalent_right_stream_column_to_left_stream_column;
bool has_single_clause = table_join.getClauses().size() == 1;
if (has_single_clause)
{
const auto & join_clause = table_join.getClauses()[0];
size_t key_names_size = join_clause.key_names_left.size();
for (size_t i = 0; i < key_names_size; ++i)
{
const auto & left_table_key_name = join_clause.key_names_left[i];
const auto & right_table_key_name = join_clause.key_names_right[i];
if (!join_header.has(left_table_key_name) || !join_header.has(right_table_key_name))
continue;
const auto & left_table_column = left_stream_input_header.getByName(left_table_key_name);
const auto & right_table_column = right_stream_input_header.getByName(right_table_key_name);
if (!left_table_column.type->equals(*right_table_column.type))
continue;
equivalent_left_stream_column_to_right_stream_column[left_table_key_name] = right_table_column;
equivalent_right_stream_column_to_left_stream_column[right_table_key_name] = left_table_column;
}
}
auto get_available_columns_for_filter = [&](bool push_to_left_stream, bool filter_push_down_input_columns_available)
{
Names available_input_columns_for_filter;
if (!filter_push_down_input_columns_available)
return available_input_columns_for_filter;
const auto & input_header = push_to_left_stream ? left_stream_input_header : right_stream_input_header;
const auto & input_columns_names = input_header.getNames();
for (const auto & name : input_columns_names)
{
if (!join_header.has(name))
continue;
/// Skip if type is changed. Push down expression expect equal types.
if (!input_header.getByName(name).type->equals(*join_header.getByName(name).type))
continue;
available_input_columns_for_filter.push_back(name);
}
return available_input_columns_for_filter;
};
bool left_stream_filter_push_down_input_columns_available = true;
bool right_stream_filter_push_down_input_columns_available = true;
if (table_join.kind() == JoinKind::Left)
right_stream_filter_push_down_input_columns_available = false;
else if (table_join.kind() == JoinKind::Right)
left_stream_filter_push_down_input_columns_available = false;
/** We disable push down to right table in cases:
* 1. Right side is already filled. Example: JOIN with Dictionary.
* 2. ASOF Right join is not supported.
*/
bool allow_push_down_to_right = join && join->allowPushDownToRight() && table_join.strictness() != JoinStrictness::Asof;
if (!allow_push_down_to_right)
right_stream_filter_push_down_input_columns_available = false;
Names equivalent_columns_to_push_down;
if (left_stream_filter_push_down_input_columns_available)
{
for (const auto & [name, _] : equivalent_left_stream_column_to_right_stream_column)
equivalent_columns_to_push_down.push_back(name);
}
if (right_stream_filter_push_down_input_columns_available)
{
for (const auto & [name, _] : equivalent_right_stream_column_to_left_stream_column)
equivalent_columns_to_push_down.push_back(name);
}
Names left_stream_available_columns_to_push_down = get_available_columns_for_filter(true /*push_to_left_stream*/, left_stream_filter_push_down_input_columns_available);
Names right_stream_available_columns_to_push_down = get_available_columns_for_filter(false /*push_to_left_stream*/, right_stream_filter_push_down_input_columns_available);
auto join_filter_push_down_actions = filter->getExpression()->splitActionsForJOINFilterPushDown(filter->getFilterColumnName(),
filter->removesFilterColumn(),
left_stream_available_columns_to_push_down,
left_stream_input_header.getColumnsWithTypeAndName(),
right_stream_available_columns_to_push_down,
right_stream_input_header.getColumnsWithTypeAndName(),
equivalent_columns_to_push_down,
equivalent_left_stream_column_to_right_stream_column,
equivalent_right_stream_column_to_left_stream_column);
size_t updated_steps = 0;
if (join_filter_push_down_actions.left_stream_filter_to_push_down)
{
updated_steps += addNewFilterStepOrThrow(parent_node,
nodes,
join_filter_push_down_actions.left_stream_filter_to_push_down,
join_filter_push_down_actions.left_stream_filter_removes_filter,
0 /*child_idx*/,
false /*update_parent_filter*/);
LOG_DEBUG(&Poco::Logger::get("QueryPlanOptimizations"),
"Pushed down filter {} to the {} side of join",
join_filter_push_down_actions.left_stream_filter_to_push_down->getOutputs()[0]->result_name,
JoinKind::Left);
}
if (join_filter_push_down_actions.right_stream_filter_to_push_down)
{
updated_steps += addNewFilterStepOrThrow(parent_node,
nodes,
join_filter_push_down_actions.right_stream_filter_to_push_down,
join_filter_push_down_actions.right_stream_filter_removes_filter,
1 /*child_idx*/,
false /*update_parent_filter*/);
LOG_DEBUG(&Poco::Logger::get("QueryPlanOptimizations"),
"Pushed down filter {} to the {} side of join",
join_filter_push_down_actions.right_stream_filter_to_push_down->getOutputs()[0]->result_name,
JoinKind::Right);
}
if (updated_steps > 0)
{
const auto & filter_column_name = filter->getFilterColumnName();
const auto & filter_expression = filter->getExpression();
const auto * filter_node = filter_expression->tryFindInOutputs(filter_column_name);
if (!filter_node && !filter->removesFilterColumn())
throw Exception(ErrorCodes::LOGICAL_ERROR,
"Filter column {} was removed from ActionsDAG but it is needed in result. DAG:\n{}",
filter_column_name, filter_expression->dumpDAG());
/// Filter column was replaced to constant.
const bool filter_is_constant = filter_node && filter_node->column && isColumnConst(*filter_node->column);
if (!filter_node || filter_is_constant)
{
/// This means that all predicates of filter were pushed down.
/// Replace current actions to expression, as we don't need to filter anything.
parent = std::make_unique<ExpressionStep>(child->getOutputStream(), filter_expression);
}
else
{
filter->updateInputStream(child->getOutputStream());
}
}
return updated_steps;
}
size_t tryPushDownFilter(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes)
{
if (parent_node->children.size() != 1)
@ -317,9 +520,6 @@ size_t tryPushDownFilter(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes
if (!keys.contains(column.name))
allowed_inputs.push_back(column.name);
// for (const auto & name : allowed_inputs)
// std::cerr << name << std::endl;
if (auto updated_steps = tryAddNewFilterStep(parent_node, nodes, allowed_inputs))
return updated_steps;
}
@ -327,77 +527,8 @@ size_t tryPushDownFilter(QueryPlan::Node * parent_node, QueryPlan::Nodes & nodes
if (auto updated_steps = simplePushDownOverStep<DistinctStep>(parent_node, nodes, child))
return updated_steps;
auto * join = typeid_cast<JoinStep *>(child.get());
auto * filled_join = typeid_cast<FilledJoinStep *>(child.get());
if (join || filled_join)
{
auto join_push_down = [&](JoinKind kind) -> size_t
{
const auto & table_join = join ? join->getJoin()->getTableJoin() : filled_join->getJoin()->getTableJoin();
/// Only inner, cross and left(/right) join are supported. Other types may generate default values for left table keys.
/// So, if we push down a condition like `key != 0`, not all rows may be filtered.
if (table_join.kind() != JoinKind::Inner && table_join.kind() != JoinKind::Cross && table_join.kind() != kind)
return 0;
/// There is no ASOF Right join, so we're talking about pushing to the right side
if (kind == JoinKind::Right && table_join.strictness() == JoinStrictness::Asof)
return 0;
bool is_left = kind == JoinKind::Left;
const auto & input_header = is_left ? child->getInputStreams().front().header : child->getInputStreams().back().header;
const auto & res_header = child->getOutputStream().header;
Names allowed_keys;
const auto & source_columns = input_header.getNames();
for (const auto & name : source_columns)
{
/// Skip key if it is renamed.
/// I don't know if it is possible. Just in case.
if (!input_header.has(name) || !res_header.has(name))
continue;
/// Skip if type is changed. Push down expression expect equal types.
if (!input_header.getByName(name).type->equals(*res_header.getByName(name).type))
continue;
allowed_keys.push_back(name);
}
/// For left JOIN, push down to the first child; for right - to the second one.
const auto child_idx = is_left ? 0 : 1;
ActionsDAGPtr split_filter = splitFilter(parent_node, allowed_keys, child_idx);
if (!split_filter)
return 0;
/*
* We should check the presence of a split filter column name in `source_columns` to avoid removing the required column.
*
* Example:
* A filter expression is `a AND b = c`, but `b` and `c` belong to another side of the join and not in `allowed_keys`, so the final split filter is just `a`.
* In this case `a` can be in `source_columns` but not `and(a, equals(b, c))`.
*
* New filter column is the first one.
*/
const String & split_filter_column_name = split_filter->getOutputs().front()->result_name;
bool can_remove_filter = source_columns.end() == std::find(source_columns.begin(), source_columns.end(), split_filter_column_name);
const size_t updated_steps = tryAddNewFilterStep(parent_node, nodes, split_filter, can_remove_filter, child_idx);
if (updated_steps > 0)
{
LOG_DEBUG(getLogger("QueryPlanOptimizations"), "Pushed down filter {} to the {} side of join", split_filter_column_name, kind);
}
return updated_steps;
};
if (size_t updated_steps = join_push_down(JoinKind::Left))
return updated_steps;
/// For full sorting merge join we push down both to the left and right tables, because left and right streams are not independent.
if (join && join->allowPushDownToRight())
{
if (size_t updated_steps = join_push_down(JoinKind::Right))
return updated_steps;
}
}
if (auto updated_steps = tryPushDownOverJoinStep(parent_node, nodes, child))
return updated_steps;
/// TODO.
/// We can filter earlier if expression does not depend on WITH FILL columns.

69
src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
View File

@ -8,10 +8,8 @@
#include <IO/WriteHelpers.h>
#include <Common/StringUtils/StringUtils.h>
#include <Common/CurrentMetrics.h>
#include "Storages/MutationCommands.h"
#include <Parsers/formatAST.h>
#include <base/sort.h>
#include <ranges>
#include <Poco/Timestamp.h>
@ -221,6 +219,43 @@ void ReplicatedMergeTreeQueue::createLogEntriesToFetchBrokenParts()
broken_parts_to_enqueue_fetches_on_loading.clear();
}
void ReplicatedMergeTreeQueue::addDropReplaceIntent(const MergeTreePartInfo & intent)
{
std::lock_guard lock{state_mutex};
drop_replace_range_intents.push_back(intent);
}
void ReplicatedMergeTreeQueue::removeDropReplaceIntent(const MergeTreePartInfo & intent)
{
std::lock_guard lock{state_mutex};
auto it = std::find(drop_replace_range_intents.begin(), drop_replace_range_intents.end(), intent);
chassert(it != drop_replace_range_intents.end());
drop_replace_range_intents.erase(it);
}
bool ReplicatedMergeTreeQueue::isIntersectingWithDropReplaceIntent(
const LogEntry & entry, const String & part_name, String & out_reason, std::unique_lock<std::mutex> & /*state_mutex lock*/) const
{
const auto part_info = MergeTreePartInfo::fromPartName(part_name, format_version);
for (const auto & intent : drop_replace_range_intents)
{
if (!intent.isDisjoint(part_info))
{
constexpr auto fmt_string = "Not executing {} of type {} for part {} (actual part {})"
"because there is a drop or replace intent with part name {}.";
LOG_INFO(
LogToStr(out_reason, log),
fmt_string,
entry.znode_name,
entry.type,
entry.new_part_name,
part_name,
intent.getPartNameForLogs());
return true;
}
}
return false;
}
void ReplicatedMergeTreeQueue::insertUnlocked(
const LogEntryPtr & entry, std::optional<time_t> & min_unprocessed_insert_time_changed,
@ -1175,6 +1210,33 @@ void ReplicatedMergeTreeQueue::removePartProducingOpsInRange(
entry->execution_complete.wait(lock, [&entry] { return !entry->currently_executing; });
}
void ReplicatedMergeTreeQueue::waitForCurrentlyExecutingOpsInRange(const MergeTreePartInfo & part_info) const
{
Queue to_wait;
std::unique_lock lock(state_mutex);
for (const auto& entry : queue)
{
if (!entry->currently_executing)
continue;
const auto virtual_part_names = entry->getVirtualPartNames(format_version);
for (const auto & virtual_part_name : virtual_part_names)
{
if (!part_info.isDisjoint(MergeTreePartInfo::fromPartName(virtual_part_name, format_version)))
{
to_wait.push_back(entry);
break;
}
}
}
LOG_DEBUG(log, "Waiting for {} entries that are currently executing.", to_wait.size());
for (LogEntryPtr & entry : to_wait)
entry->execution_complete.wait(lock, [&entry] { return !entry->currently_executing; });
}
bool ReplicatedMergeTreeQueue::isCoveredByFuturePartsImpl(const LogEntry & entry, const String & new_part_name,
String & out_reason, std::unique_lock<std::mutex> & /* queue_lock */,
@ -1303,6 +1365,9 @@ bool ReplicatedMergeTreeQueue::shouldExecuteLogEntry(
/// We can wait in worker threads, but not in scheduler.
if (isCoveredByFuturePartsImpl(entry, new_part_name, out_postpone_reason, state_lock, /* covered_entries_to_wait */ nullptr))
return false;
if (isIntersectingWithDropReplaceIntent(entry, new_part_name, out_postpone_reason, state_lock))
return false;
}
if (entry.type != LogEntry::DROP_RANGE && entry.type != LogEntry::DROP_PART)

15
src/Storages/MergeTree/ReplicatedMergeTreeQueue.h
View File

@ -107,6 +107,8 @@ private:
*/
ActiveDataPartSet virtual_parts;
/// Used to prevent operations to start in ranges which will be affected by DROP_RANGE/REPLACE_RANGE
std::vector<MergeTreePartInfo> drop_replace_range_intents;
/// We do not add DROP_PARTs to virtual_parts because they can intersect,
/// so we store them separately in this structure.
@ -251,6 +253,10 @@ private:
std::optional<time_t> min_unprocessed_insert_time_changed,
std::optional<time_t> max_processed_insert_time_changed) const;
bool isIntersectingWithDropReplaceIntent(
const LogEntry & entry,
const String & part_name, String & out_reason, std::unique_lock<std::mutex> & /*state_mutex lock*/) const;
/// Marks the element of the queue as running.
class CurrentlyExecuting
{
@ -349,6 +355,9 @@ public:
const MergeTreePartInfo & part_info,
const std::optional<ReplicatedMergeTreeLogEntryData> & covering_entry);
/// Wait for the execution of currently executing actions with virtual parts intersecting with part_info
void waitForCurrentlyExecutingOpsInRange(const MergeTreePartInfo & part_info) const;
/** In the case where there are not enough parts to perform the merge in part_name
* - move actions with merged parts to the end of the queue
* (in order to download a already merged part from another replica).
@ -490,6 +499,12 @@ public:
void setBrokenPartsToEnqueueFetchesOnLoading(Strings && parts_to_fetch);
/// Must be called right after queue loading.
void createLogEntriesToFetchBrokenParts();
/// Add an intent to block operations to start in the range. All intents must be removed by calling
/// removeDropReplaceIntent(). The same intent can be added multiple times, but it has to be removed exactly
/// the same amount of times.
void addDropReplaceIntent(const MergeTreePartInfo& intent);
void removeDropReplaceIntent(const MergeTreePartInfo& intent);
};
using CommittingBlocks = std::unordered_map<String, std::set<Int64>>;

36
src/Storages/StorageReplicatedMergeTree.cpp
View File

@ -8027,6 +8027,20 @@ void StorageReplicatedMergeTree::replacePartitionFrom(
assert(replace == !LogEntry::ReplaceRangeEntry::isMovePartitionOrAttachFrom(drop_range));
scope_guard intent_guard;
if (replace)
{
queue.addDropReplaceIntent(drop_range);
intent_guard = scope_guard{[this, my_drop_range = drop_range]() { queue.removeDropReplaceIntent(my_drop_range); }};
getContext()->getMergeList().cancelInPartition(getStorageID(), drop_range.partition_id, drop_range.max_block);
queue.waitForCurrentlyExecutingOpsInRange(drop_range);
{
auto pause_checking_parts = part_check_thread.pausePartsCheck();
part_check_thread.cancelRemovedPartsCheck(drop_range);
}
}
String drop_range_fake_part_name = getPartNamePossiblyFake(format_version, drop_range);
std::set<String> replaced_parts;
@ -8195,8 +8209,11 @@ void StorageReplicatedMergeTree::replacePartitionFrom(
lock2.reset();
lock1.reset();
/// We need to pull the DROP_RANGE before cleaning the replaced parts (otherwise CHeckThread may decide that parts are lost)
/// We need to pull the REPLACE_RANGE before cleaning the replaced parts (otherwise CHeckThread may decide that parts are lost)
queue.pullLogsToQueue(getZooKeeperAndAssertNotReadonly(), {}, ReplicatedMergeTreeQueue::SYNC);
// No need to block operations further, especially that in case we have to wait for mutation to finish, the intent would block
// the execution of REPLACE_RANGE
intent_guard.reset();
parts_holder.clear();
cleanup_thread.wakeup();
@ -8248,11 +8265,23 @@ void StorageReplicatedMergeTree::movePartitionToTable(const StoragePtr & dest_ta
Coordination::Stat alter_partition_version_stat;
zookeeper->get(alter_partition_version_path, &alter_partition_version_stat);
MergeTreePartInfo drop_range;
std::optional<EphemeralLockInZooKeeper> delimiting_block_lock;
MergeTreePartInfo drop_range;
getFakePartCoveringAllPartsInPartition(partition_id, drop_range, delimiting_block_lock, true);
String drop_range_fake_part_name = getPartNamePossiblyFake(format_version, drop_range);
queue.addDropReplaceIntent(drop_range);
// Let's copy drop_range to make sure it doesn't get modified, otherwise we might run into issue on removal
scope_guard intent_guard{[this, my_drop_range = drop_range]() { queue.removeDropReplaceIntent(my_drop_range); }};
getContext()->getMergeList().cancelInPartition(getStorageID(), drop_range.partition_id, drop_range.max_block);
queue.waitForCurrentlyExecutingOpsInRange(drop_range);
{
auto pause_checking_parts = part_check_thread.pausePartsCheck();
part_check_thread.cancelRemovedPartsCheck(drop_range);
}
DataPartPtr covering_part;
DataPartsVector src_all_parts;
{
@ -8457,6 +8486,9 @@ void StorageReplicatedMergeTree::movePartitionToTable(const StoragePtr & dest_ta
/// We need to pull the DROP_RANGE before cleaning the replaced parts (otherwise CHeckThread may decide that parts are lost)
queue.pullLogsToQueue(getZooKeeperAndAssertNotReadonly(), {}, ReplicatedMergeTreeQueue::SYNC);
// No need to block operations further, especially that in case we have to wait for mutation to finish, the intent would block
// the execution of DROP_RANGE
intent_guard.reset();
parts_holder.clear();
cleanup_thread.wakeup();

12
tests/ci/ci.py
View File

@ -2121,11 +2121,13 @@ def main() -> int:
pr_info,
dump_to_file=True,
)
update_mergeable_check(
commit,
pr_info,
job_report.check_name or _get_ext_check_name(args.job_name),
)
if not pr_info.is_merge_queue():
# in the merge queue mergeable status must be set only in FinishCheck (last job in wf)
update_mergeable_check(
commit,
pr_info,
job_report.check_name or _get_ext_check_name(args.job_name),
)
print(f"Job report url: [{check_url}]")
prepared_events = prepare_tests_results_for_clickhouse(

31
tests/ci/finish_check.py
View File

@ -28,21 +28,22 @@ def main():
statuses = get_commit_filtered_statuses(commit)
trigger_mergeable_check(commit, statuses)
statuses = [s for s in statuses if s.context == CI_STATUS_NAME]
if not statuses:
return
# Take the latest status
status = statuses[-1]
if status.state == PENDING:
post_commit_status(
commit,
SUCCESS,
status.target_url,
"All checks finished",
CI_STATUS_NAME,
pr_info,
dump_to_file=True,
)
if not pr_info.is_merge_queue():
statuses = [s for s in statuses if s.context == CI_STATUS_NAME]
if not statuses:
return
# Take the latest status
status = statuses[-1]
if status.state == PENDING:
post_commit_status(
commit,
SUCCESS,
status.target_url,
"All checks finished",
CI_STATUS_NAME,
pr_info,
dump_to_file=True,
)
if __name__ == "__main__":

5
tests/ci/pr_info.py
View File

@ -199,7 +199,6 @@ class PRInfo:
EventType.MERGE_QUEUE in github_event
): # pull request and other similar events
self.event_type = EventType.MERGE_QUEUE
# FIXME: need pr? we can parse it from ["head_ref": "refs/heads/gh-readonly-queue/test-merge-queue/pr-6751-4690229995a155e771c52e95fbd446d219c069bf"]
self.number = 0
self.sha = github_event[EventType.MERGE_QUEUE]["head_sha"]
self.base_ref = github_event[EventType.MERGE_QUEUE]["base_ref"]
@ -208,6 +207,8 @@ class PRInfo:
self.base_name = github_event["repository"]["full_name"]
# any_branch-name - the name of working branch name
self.head_ref = github_event[EventType.MERGE_QUEUE]["head_ref"]
# parse underlying pr from ["head_ref": "refs/heads/gh-readonly-queue/test-merge-queue/pr-6751-4690229995a155e771c52e95fbd446d219c069bf"]
self.merged_pr = int(self.head_ref.split("/pr-")[-1].split("-")[0])
# UserName/ClickHouse or ClickHouse/ClickHouse
self.head_name = self.base_name
self.user_login = github_event["sender"]["login"]
@ -235,6 +236,8 @@ class PRInfo:
if pull_request is None or pull_request["state"] == "closed":
# it's merged PR to master
self.number = 0
if pull_request:
self.merged_pr = pull_request["number"]
self.labels = set()
self.pr_html_url = f"{repo_prefix}/commits/{ref}"
self.base_ref = ref

Some files were not shown because too many files have changed in this diff Show More