add normalizeQueryKeepNames and normalizedQueryHashKeepNames

This commit is contained in:
Blargian 2024-07-07 18:46:54 +02:00
parent 2bca1963a7
commit 126417cc61

View File

@ -1556,7 +1556,8 @@ The result type is UInt64.
## normalizeQuery ## normalizeQuery
Replaces literals, sequences of literals and complex aliases with placeholders. Replaces literals, sequences of literals and complex aliases (containing whitespace, more than two digits
or at least 36 bytes long such as UUIDs) with placeholder `?`.
**Syntax** **Syntax**
@ -1574,6 +1575,8 @@ normalizeQuery(x)
**Example** **Example**
Query:
``` sql ``` sql
SELECT normalizeQuery('[1, 2, 3, x]') AS query; SELECT normalizeQuery('[1, 2, 3, x]') AS query;
``` ```
@ -1586,9 +1589,44 @@ Result:
└──────────┘ └──────────┘
``` ```
## normalizeQueryKeepNames
Replaces literals, sequences of literals with placeholder `?` but does not replace complex aliases (containing whitespace, more than two digits
or at least 36 bytes long such as UUIDs). This helps better analyze complex query logs.
**Syntax**
``` sql
normalizeQueryKeepNames(x)
```
**Arguments**
- `x` — Sequence of characters. [String](../data-types/string.md).
**Returned value**
- Sequence of characters with placeholders. [String](../data-types/string.md).
**Example**
Query:
``` sql
SELECT normalizeQuery('SELECT 1 AS aComplexName123'), normalizeQueryKeepNames('SELECT 1 AS aComplexName123');
```
Result:
```result
┌─normalizeQuery('SELECT 1 AS aComplexName123')─┬─normalizeQueryKeepNames('SELECT 1 AS aComplexName123')─┐
│ SELECT ? AS `?` │ SELECT ? AS aComplexName123 │
└───────────────────────────────────────────────┴────────────────────────────────────────────────────────┘
```
## normalizedQueryHash ## normalizedQueryHash
Returns identical 64bit hash values without the values of literals for similar queries. Can be helpful to analyze query log. Returns identical 64bit hash values without the values of literals for similar queries. Can be helpful to analyze query logs.
**Syntax** **Syntax**
@ -1606,6 +1644,8 @@ normalizedQueryHash(x)
**Example** **Example**
Query:
``` sql ``` sql
SELECT normalizedQueryHash('SELECT 1 AS `xyz`') != normalizedQueryHash('SELECT 1 AS `abc`') AS res; SELECT normalizedQueryHash('SELECT 1 AS `xyz`') != normalizedQueryHash('SELECT 1 AS `abc`') AS res;
``` ```
@ -1618,6 +1658,43 @@ Result:
└─────┘ └─────┘
``` ```
## normalizedQueryHashKeepNames
Like [normalizedQueryHash](#normalizedqueryhash) it returns identical 64bit hash values without the values of literals for similar queries but it does not replace complex aliases (containing whitespace, more than two digits
or at least 36 bytes long such as UUIDs) with a placeholder before hashing. Can be helpful to analyze query logs.
**Syntax**
``` sql
normalizedQueryHashKeepNames(x)
```
**Arguments**
- `x` — Sequence of characters. [String](../data-types/string.md).
**Returned value**
- Hash value. [UInt64](../data-types/int-uint.md#uint-ranges).
**Example**
``` sql
SELECT normalizedQueryHash('SELECT 1 AS `xyz123`') != normalizedQueryHash('SELECT 1 AS `abc123`') AS normalizedQueryHash;
SELECT normalizedQueryHashKeepNames('SELECT 1 AS `xyz123`') != normalizedQueryHashKeepNames('SELECT 1 AS `abc123`') AS normalizedQueryHashKeepNames;
```
Result:
```result
┌─normalizedQueryHash─┐
│ 0 │
└─────────────────────┘
┌─normalizedQueryHashKeepNames─┐
│ 1 │
└──────────────────────────────┘
```
## normalizeUTF8NFC ## normalizeUTF8NFC
Converts a string to [NFC normalized form](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms), assuming the string is valid UTF8-encoded text. Converts a string to [NFC normalized form](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms), assuming the string is valid UTF8-encoded text.