2020-04-03 13:23:32 +00:00
---
toc_priority: 40
2020-06-19 10:05:00 +00:00
toc_title: Strings
2020-04-03 13:23:32 +00:00
---
2020-04-30 18:19:18 +00:00
# Functions for Working with Strings {#functions-for-working-with-strings}
2017-12-28 15:13:23 +00:00
2020-06-19 10:09:45 +00:00
!!! note "Note"
2020-07-09 15:10:35 +00:00
Functions for [searching ](../../sql-reference/functions/string-search-functions.md ) and [replacing ](../../sql-reference/functions/string-replace-functions.md ) in strings are described separately.
2020-06-19 10:09:45 +00:00
2020-03-19 15:32:53 +00:00
## empty {#empty}
2017-04-03 19:49:50 +00:00
2021-08-03 13:07:46 +00:00
Checks whether the input string is empty.
**Syntax**
``` sql
empty(x)
```
2017-04-26 19:16:38 +00:00
A string is considered non-empty if it contains at least one byte, even if this is a space or a null byte.
2021-08-03 13:07:46 +00:00
2021-08-10 10:53:35 +00:00
The function also works for [arrays ](array-functions.md#function-empty ) or [UUID ](uuid-functions.md#empty ).
2021-08-03 13:07:46 +00:00
**Arguments**
2021-08-10 10:53:35 +00:00
- `x` — Input value. [String ](../data-types/string.md ).
2021-08-03 13:07:46 +00:00
**Returned value**
2021-10-14 10:15:45 +00:00
- Returns `1` for an empty string or `0` for a non-empty string.
2021-08-03 13:07:46 +00:00
Type: [UInt8 ](../data-types/int-uint.md ).
**Example**
Query:
```sql
SELECT empty('');
```
Result:
```text
┌─empty('')─┐
│ 1 │
└───────────┘
```
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## notEmpty {#notempty}
2017-12-28 15:13:23 +00:00
2021-08-10 10:53:35 +00:00
Checks whether the input string is non-empty.
2021-08-06 09:04:55 +00:00
**Syntax**
``` sql
2021-08-10 10:53:35 +00:00
notEmpty(x)
2021-08-06 09:04:55 +00:00
```
2021-08-12 06:45:23 +00:00
A string is considered non-empty if it contains at least one byte, even if this is a space or a null byte.
2021-08-06 09:04:55 +00:00
2021-08-10 10:53:35 +00:00
The function also works for [arrays ](array-functions.md#function-notempty ) or [UUID ](uuid-functions.md#notempty ).
2021-08-06 09:04:55 +00:00
**Arguments**
2021-08-10 10:53:35 +00:00
- `x` — Input value. [String ](../data-types/string.md ).
2021-08-06 09:04:55 +00:00
**Returned value**
2021-10-14 10:15:45 +00:00
- Returns `1` for a non-empty string or `0` for an empty string string.
2021-08-06 09:04:55 +00:00
Type: [UInt8 ](../data-types/int-uint.md ).
**Example**
Query:
```sql
2021-08-10 10:53:35 +00:00
SELECT notEmpty('text');
2021-08-06 09:04:55 +00:00
```
Result:
```text
2021-08-10 10:53:35 +00:00
┌─notEmpty('text')─┐
│ 1 │
└──────────────────┘
2021-08-06 09:04:55 +00:00
```
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## length {#length}
2017-12-28 15:13:23 +00:00
2017-04-26 19:16:38 +00:00
Returns the length of a string in bytes (not in characters, and not in code points).
The result type is UInt64.
The function also works for arrays.
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## lengthUTF8 {#lengthutf8}
2017-12-28 15:13:23 +00:00
2021-05-27 19:44:11 +00:00
Returns the length of a string in Unicode code points (not in characters), assuming that the string contains a set of bytes that make up UTF-8 encoded text. If this assumption is not met, it returns some result (it does not throw an exception).
2017-04-26 19:16:38 +00:00
The result type is UInt64.
2017-04-03 19:49:50 +00:00
2020-10-13 17:23:29 +00:00
## char_length, CHAR_LENGTH {#char-length}
2019-01-30 10:39:46 +00:00
2021-05-27 19:44:11 +00:00
Returns the length of a string in Unicode code points (not in characters), assuming that the string contains a set of bytes that make up UTF-8 encoded text. If this assumption is not met, it returns some result (it does not throw an exception).
2019-01-30 10:39:46 +00:00
The result type is UInt64.
2020-10-13 17:23:29 +00:00
## character_length, CHARACTER_LENGTH {#character-length}
2019-01-30 10:39:46 +00:00
2021-05-27 19:44:11 +00:00
Returns the length of a string in Unicode code points (not in characters), assuming that the string contains a set of bytes that make up UTF-8 encoded text. If this assumption is not met, it returns some result (it does not throw an exception).
2019-01-30 10:39:46 +00:00
The result type is UInt64.
2021-08-03 18:58:32 +00:00
## leftPad {#leftpad}
2021-08-09 13:43:17 +00:00
Pads the current string from the left with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Similarly to the MySQL `LPAD` function.
2021-08-03 18:58:32 +00:00
**Syntax**
``` sql
2021-08-09 13:19:31 +00:00
leftPad('string', 'length'[, 'pad_string'])
2021-08-03 18:58:32 +00:00
```
**Arguments**
2021-08-12 16:09:27 +00:00
- `string` — Input string that needs to be padded. [String ](../data-types/string.md ).
2021-08-09 13:19:39 +00:00
- `length` — The length of the resulting string. [UInt ](../data-types/int-uint.md ). If the value is less than the input string length, then the input string is returned as-is.
2021-08-09 13:19:49 +00:00
- `pad_string` — The string to pad the input string with. [String ](../data-types/string.md ). Optional. If not specified, then the input string is padded with spaces.
2021-08-03 18:58:32 +00:00
2021-08-09 14:29:26 +00:00
**Returned value**
2021-08-03 18:58:32 +00:00
2021-08-09 13:20:13 +00:00
- The resulting string of the given length.
2021-08-03 18:58:32 +00:00
Type: [String ](../data-types/string.md ).
**Example**
Query:
``` sql
2021-08-09 13:20:20 +00:00
SELECT leftPad('abc', 7, '*'), leftPad('def', 7);
2021-08-03 18:58:32 +00:00
```
Result:
``` text
2021-08-09 13:20:26 +00:00
┌─leftPad('abc', 7, '*')─┬─leftPad('def', 7)─┐
│ ** **abc │ def │
└────────────────────────┴───────────────────┘
2021-08-03 18:58:32 +00:00
```
2021-08-06 15:03:31 +00:00
## leftPadUTF8 {#leftpadutf8}
2021-08-09 13:43:17 +00:00
Pads the current string from the left with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Similarly to the MySQL `LPAD` function. While in the [leftPad ](#leftpad ) function the length is measured in bytes, here in the `leftPadUTF8` function it is measured in code points.
2021-08-06 15:03:31 +00:00
**Syntax**
``` sql
2021-08-09 13:20:32 +00:00
leftPadUTF8('string','length'[, 'pad_string'])
2021-08-06 15:03:31 +00:00
```
**Arguments**
2021-08-12 16:09:51 +00:00
- `string` — Input string that needs to be padded. [String ](../data-types/string.md ).
2021-08-09 13:21:09 +00:00
- `length` — The length of the resulting string. [UInt ](../data-types/int-uint.md ). If the value is less than the input string length, then the input string is returned as-is.
2021-08-09 13:21:14 +00:00
- `pad_string` — The string to pad the input string with. [String ](../data-types/string.md ). Optional. If not specified, then the input string is padded with spaces.
2021-08-06 15:03:31 +00:00
2021-08-09 14:29:26 +00:00
**Returned value**
2021-08-06 15:03:31 +00:00
2021-08-09 14:29:26 +00:00
- The resulting string of the given length.
2021-08-06 15:03:31 +00:00
Type: [String ](../data-types/string.md ).
**Example**
Query:
``` sql
2021-08-09 13:21:22 +00:00
SELECT leftPadUTF8('абвг', 7, '*'), leftPadUTF8('дежз', 7);
2021-08-06 15:03:31 +00:00
```
Result:
``` text
2021-08-09 13:21:28 +00:00
┌─leftPadUTF8('абвг', 7, '*')─┬─leftPadUTF8('дежз', 7)─┐
│ ** *абвг │ дежз │
└─────────────────────────────┴────────────────────────┘
2021-08-06 15:03:31 +00:00
```
2021-08-03 18:58:32 +00:00
## rightPad {#rightpad}
2021-08-09 13:43:17 +00:00
Pads the current string from the right with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Similarly to the MySQL `RPAD` function.
2021-08-03 18:58:32 +00:00
**Syntax**
``` sql
2021-08-09 13:43:17 +00:00
rightPad('string', 'length'[, 'pad_string'])
2021-08-03 18:58:32 +00:00
```
**Arguments**
2021-08-12 16:09:36 +00:00
- `string` — Input string that needs to be padded. [String ](../data-types/string.md ).
2021-08-09 13:43:17 +00:00
- `length` — The length of the resulting string. [UInt ](../data-types/int-uint.md ). If the value is less than the input string length, then the input string is returned as-is.
- `pad_string` — The string to pad the input string with. [String ](../data-types/string.md ). Optional. If not specified, then the input string is padded with spaces.
2021-08-03 18:58:32 +00:00
2021-08-09 14:29:26 +00:00
**Returned value**
2021-08-03 18:58:32 +00:00
2021-08-09 13:43:17 +00:00
- The resulting string of the given length.
2021-08-03 18:58:32 +00:00
Type: [String ](../data-types/string.md ).
**Example**
Query:
``` sql
2021-08-09 13:43:17 +00:00
SELECT rightPad('abc', 7, '*'), rightPad('abc', 7);
2021-08-03 18:58:32 +00:00
```
Result:
``` text
2021-08-09 13:43:17 +00:00
┌─rightPad('abc', 7, '*')─┬─rightPad('abc', 7)─┐
│ abc**** │ abc │
└─────────────────────────┴────────────────────┘
2021-08-03 18:58:32 +00:00
```
2021-08-06 15:03:31 +00:00
## rightPadUTF8 {#rightpadutf8}
2021-08-09 14:29:26 +00:00
Pads the current string from the right with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Similarly to the MySQL `RPAD` function. While in the [rightPad ](#rightpad ) function the length is measured in bytes, here in the `rightPadUTF8` function it is measured in code points.
2021-08-06 15:03:31 +00:00
**Syntax**
``` sql
2021-08-09 13:43:17 +00:00
rightPadUTF8('string','length'[, 'pad_string'])
2021-08-06 15:03:31 +00:00
```
**Arguments**
2021-08-12 16:09:43 +00:00
- `string` — Input string that needs to be padded. [String ](../data-types/string.md ).
2021-08-09 13:43:17 +00:00
- `length` — The length of the resulting string. [UInt ](../data-types/int-uint.md ). If the value is less than the input string length, then the input string is returned as-is.
- `pad_string` — The string to pad the input string with. [String ](../data-types/string.md ). Optional. If not specified, then the input string is padded with spaces.
2021-08-06 15:03:31 +00:00
2021-08-09 14:29:26 +00:00
**Returned value**
2021-08-06 15:03:31 +00:00
2021-08-09 14:29:26 +00:00
- The resulting string of the given length.
2021-08-06 15:03:31 +00:00
Type: [String ](../data-types/string.md ).
**Example**
Query:
``` sql
2021-08-09 13:43:17 +00:00
SELECT rightPadUTF8('абвг', 7, '*'), rightPadUTF8('абвг', 7);
2021-08-06 15:03:31 +00:00
```
Result:
``` text
2021-08-09 13:43:17 +00:00
┌─rightPadUTF8('абвг', 7, '*')─┬─rightPadUTF8('абвг', 7)─┐
│ абвг*** │ абвг │
└──────────────────────────────┴─────────────────────────┘
2021-08-06 15:03:31 +00:00
```
2020-03-19 15:32:53 +00:00
## lower, lcase {#lower}
2017-12-28 15:13:23 +00:00
2017-04-26 19:16:38 +00:00
Converts ASCII Latin symbols in a string to lowercase.
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## upper, ucase {#upper}
2017-12-28 15:13:23 +00:00
2021-08-06 15:44:46 +00:00
Converts ASCII Latin symbols in a string to uppercase.
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## lowerUTF8 {#lowerutf8}
2017-12-28 15:13:23 +00:00
Converts a string to lowercase, assuming the string contains a set of bytes that make up a UTF-8 encoded text.
2021-05-27 19:44:11 +00:00
It does not detect the language. So for Turkish the result might not be exactly correct.
2017-12-28 15:13:23 +00:00
If the length of the UTF-8 byte sequence is different for upper and lower case of a code point, the result may be incorrect for this code point.
If the string contains a set of bytes that is not UTF-8, then the behavior is undefined.
2020-03-19 15:32:53 +00:00
## upperUTF8 {#upperutf8}
2017-04-03 19:49:50 +00:00
2017-12-28 15:13:23 +00:00
Converts a string to uppercase, assuming the string contains a set of bytes that make up a UTF-8 encoded text.
2021-05-27 19:44:11 +00:00
It does not detect the language. So for Turkish the result might not be exactly correct.
2017-12-28 15:13:23 +00:00
If the length of the UTF-8 byte sequence is different for upper and lower case of a code point, the result may be incorrect for this code point.
If the string contains a set of bytes that is not UTF-8, then the behavior is undefined.
2020-03-19 15:32:53 +00:00
## isValidUTF8 {#isvalidutf8}
2019-04-07 18:58:13 +00:00
2019-04-07 18:59:53 +00:00
Returns 1, if the set of bytes is valid UTF-8 encoded, otherwise 0.
2019-04-07 18:58:13 +00:00
2020-03-19 15:32:53 +00:00
## toValidUTF8 {#tovalidutf8}
2019-05-17 12:55:21 +00:00
2019-05-20 14:41:10 +00:00
Replaces invalid UTF-8 characters by the `<60> ` (U+FFFD) character. All running in a row invalid characters are collapsed into the one replacement character.
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
toValidUTF8(input_string)
2019-05-20 14:41:10 +00:00
```
2021-02-16 11:21:23 +00:00
**Arguments**
2019-05-20 14:41:10 +00:00
2021-03-13 18:18:45 +00:00
- `input_string` — Any set of bytes represented as the [String ](../../sql-reference/data-types/string.md ) data type object.
2019-05-20 14:41:10 +00:00
Returned value: Valid UTF-8 string.
2020-03-19 15:32:53 +00:00
**Example**
2019-05-20 14:41:10 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT toValidUTF8('\x61\xF0\x80\x80\x80b');
2019-05-20 14:41:10 +00:00
```
2020-03-20 10:10:48 +00:00
``` text
2019-05-23 11:37:05 +00:00
┌─toValidUTF8('a<> <61> <EFBFBD> <EFBFBD> b')─┐
│ a<> b │
└───────────────────────┘
2021-10-14 10:15:45 +00:00
```
2020-03-18 18:43:51 +00:00
## repeat {#repeat}
2019-10-24 14:59:00 +00:00
Repeats a string as many times as specified and concatenates the replicated values as a single string.
2021-02-16 11:31:24 +00:00
Alias: `REPEAT` .
2019-10-24 14:59:00 +00:00
**Syntax**
2020-03-20 10:10:48 +00:00
``` sql
2019-10-24 14:59:00 +00:00
repeat(s, n)
```
2021-02-15 21:22:10 +00:00
**Arguments**
2019-10-24 14:59:00 +00:00
2020-04-30 18:19:18 +00:00
- `s` — The string to repeat. [String ](../../sql-reference/data-types/string.md ).
- `n` — The number of times to repeat the string. [UInt ](../../sql-reference/data-types/int-uint.md ).
2019-10-24 14:59:00 +00:00
**Returned value**
2020-03-20 10:10:48 +00:00
The single string, which contains the string `s` repeated `n` times. If `n` \< 1, the function returns empty string.
2019-10-24 14:59:00 +00:00
Type: `String` .
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT repeat('abc', 10);
2019-10-24 14:59:00 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-10-24 14:59:00 +00:00
┌─repeat('abc', 10)──────────────┐
│ abcabcabcabcabcabcabcabcabcabc │
└────────────────────────────────┘
```
2019-05-21 09:08:43 +00:00
2020-03-19 15:32:53 +00:00
## reverse {#reverse}
2017-04-03 19:49:50 +00:00
2017-04-26 19:16:38 +00:00
Reverses the string (as a sequence of bytes).
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## reverseUTF8 {#reverseutf8}
2017-12-28 15:13:23 +00:00
2021-05-27 19:44:11 +00:00
Reverses a sequence of Unicode code points, assuming that the string contains a set of bytes representing a UTF-8 text. Otherwise, it does something else (it does not throw an exception).
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
## format(pattern, s0, s1, …) {#format}
2019-05-18 11:30:36 +00:00
2020-03-20 10:10:48 +00:00
Formatting constant pattern with the string listed in the arguments. `pattern` is a simplified Python format pattern. Format string contains “replacement fields” surrounded by curly braces `{}` . Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: `{{ '{{' }}` and `{{ '}}' }}` . Field names can be numbers (starting from zero) or empty (then they are treated as consequence numbers).
2019-05-18 11:30:36 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2019-05-18 11:30:36 +00:00
SELECT format('{1} {0} {1}', 'World', 'Hello')
2019-09-23 15:31:46 +00:00
```
2020-03-20 10:10:48 +00:00
``` text
2019-05-18 11:30:36 +00:00
┌─format('{1} {0} {1}', 'World', 'Hello')─┐
│ Hello World Hello │
└─────────────────────────────────────────┘
2019-09-23 15:31:46 +00:00
```
2020-03-20 10:10:48 +00:00
``` sql
2019-05-18 11:30:36 +00:00
SELECT format('{} {}', 'Hello', 'World')
2019-09-23 15:31:46 +00:00
```
2020-03-20 10:10:48 +00:00
``` text
2019-05-18 11:30:36 +00:00
┌─format('{} {}', 'Hello', 'World')─┐
│ Hello World │
└───────────────────────────────────┘
```
2020-03-18 18:43:51 +00:00
## concat {#concat}
2017-12-28 15:13:23 +00:00
Concatenates the strings listed in the arguments, without a separator.
2020-03-20 10:10:48 +00:00
**Syntax**
2019-01-30 10:39:46 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2019-11-08 19:08:55 +00:00
concat(s1, s2, ...)
```
2021-02-15 21:22:10 +00:00
**Arguments**
2019-11-08 19:08:55 +00:00
2020-02-02 21:59:23 +00:00
Values of type String or FixedString.
2019-11-08 19:08:55 +00:00
**Returned values**
2020-03-20 10:10:48 +00:00
Returns the String that results from concatenating the arguments.
2019-11-08 19:08:55 +00:00
2020-03-20 10:10:48 +00:00
If any of argument values is `NULL` , `concat` returns `NULL` .
2019-11-08 19:08:55 +00:00
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT concat('Hello, ', 'World!');
2019-11-08 19:08:55 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-12-26 12:51:48 +00:00
┌─concat('Hello, ', 'World!')─┐
│ Hello, World! │
└─────────────────────────────┘
2019-11-08 19:08:55 +00:00
```
2020-03-18 18:43:51 +00:00
## concatAssumeInjective {#concatassumeinjective}
2019-11-08 19:08:55 +00:00
2019-12-26 12:51:48 +00:00
Same as [concat ](#concat ), the difference is that you need to ensure that `concat(s1, s2, ...) → sn` is injective, it will be used for optimization of GROUP BY.
2019-11-08 19:08:55 +00:00
2020-03-20 10:10:48 +00:00
The function is named “injective” if it always returns different result for different values of arguments. In other words: different arguments never yield identical result.
2019-11-08 19:08:55 +00:00
2020-03-20 10:10:48 +00:00
**Syntax**
2020-03-19 15:32:53 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2020-03-19 15:32:53 +00:00
concatAssumeInjective(s1, s2, ...)
```
2021-02-15 21:22:10 +00:00
**Arguments**
2020-03-19 15:32:53 +00:00
Values of type String or FixedString.
**Returned values**
2020-03-20 10:10:48 +00:00
Returns the String that results from concatenating the arguments.
2020-03-19 15:32:53 +00:00
If any of argument values is `NULL` , `concatAssumeInjective` returns `NULL` .
2019-11-08 19:08:55 +00:00
**Example**
2019-12-26 12:51:48 +00:00
Input table:
2020-03-20 10:10:48 +00:00
``` sql
2020-02-02 21:59:23 +00:00
CREATE TABLE key_val(`key1` String, `key2` String, `value` UInt32) ENGINE = TinyLog;
INSERT INTO key_val VALUES ('Hello, ','World',1), ('Hello, ','World',2), ('Hello, ','World!',3), ('Hello',', World!',2);
SELECT * from key_val;
2019-12-26 12:51:48 +00:00
```
2020-03-20 10:10:48 +00:00
``` text
2019-12-26 12:51:48 +00:00
┌─key1────┬─key2─────┬─value─┐
│ Hello, │ World │ 1 │
│ Hello, │ World │ 2 │
│ Hello, │ World! │ 3 │
│ Hello │ , World! │ 2 │
└─────────┴──────────┴───────┘
```
2019-11-08 19:08:55 +00:00
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT concat(key1, key2), sum(value) FROM key_val GROUP BY concatAssumeInjective(key1, key2);
2019-11-08 19:08:55 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-11-08 19:08:55 +00:00
┌─concat(key1, key2)─┬─sum(value)─┐
│ Hello, World! │ 3 │
│ Hello, World! │ 2 │
│ Hello, World │ 3 │
└────────────────────┴────────────┘
```
2019-01-30 10:39:46 +00:00
2020-03-19 15:32:53 +00:00
## substring(s, offset, length), mid(s, offset, length), substr(s, offset, length) {#substring}
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
Returns a substring starting with the byte from the ‘ offset’ index that is ‘ length’ bytes long. Character indexing starts from one (as in standard SQL). The ‘ offset’ and ‘ length’ arguments must be constants.
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## substringUTF8(s, offset, length) {#substringutf8}
2017-12-28 15:13:23 +00:00
2021-05-27 19:44:11 +00:00
The same as ‘ substring’ , but for Unicode code points. Works under the assumption that the string contains a set of bytes representing a UTF-8 encoded text. If this assumption is not met, it returns some result (it does not throw an exception).
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## appendTrailingCharIfAbsent(s, c) {#appendtrailingcharifabsent}
2017-12-28 15:13:23 +00:00
2020-03-20 10:10:48 +00:00
If the ‘ s’ string is non-empty and does not contain the ‘ c’ character at the end, it appends the ‘ c’ character to the end.
2017-12-28 15:13:23 +00:00
2020-03-19 15:32:53 +00:00
## convertCharset(s, from, to) {#convertcharset}
2017-12-28 15:13:23 +00:00
2020-03-20 10:10:48 +00:00
Returns the string ‘ s’ that was converted from the encoding in ‘ from’ to the encoding in ‘ to’ .
2017-04-03 19:49:50 +00:00
2020-03-19 15:32:53 +00:00
## base64Encode(s) {#base64encode}
2020-03-20 10:10:48 +00:00
Encodes ‘ s’ string into base64
2018-11-02 19:06:05 +00:00
2021-02-16 11:13:01 +00:00
Alias: `TO_BASE64` .
2020-03-19 15:32:53 +00:00
## base64Decode(s) {#base64decode}
2020-03-20 10:10:48 +00:00
Decode base64-encoded string ‘ s’ into original string. In case of failure raises an exception.
2018-11-02 19:06:05 +00:00
2021-02-16 11:13:01 +00:00
Alias: `FROM_BASE64` .
2020-03-19 15:32:53 +00:00
## tryBase64Decode(s) {#trybase64decode}
2018-11-14 09:32:42 +00:00
Similar to base64Decode, but in case of error an empty string would be returned.
2020-03-19 15:32:53 +00:00
## endsWith(s, suffix) {#endswith}
2019-01-30 10:39:46 +00:00
Returns whether to end with the specified suffix. Returns 1 if the string ends with the specified suffix, otherwise it returns 0.
2020-03-19 15:32:53 +00:00
## startsWith(str, prefix) {#startswith}
2019-01-30 10:39:46 +00:00
2019-09-26 11:39:06 +00:00
Returns 1 whether string starts with the specified prefix, otherwise it returns 0.
2020-03-20 10:10:48 +00:00
``` sql
2019-09-30 07:24:02 +00:00
SELECT startsWith('Spider-Man', 'Spi');
2019-09-26 11:39:06 +00:00
```
**Returned values**
2020-03-21 04:11:51 +00:00
- 1, if the string starts with the specified prefix.
2021-05-27 19:44:11 +00:00
- 0, if the string does not start with the specified prefix.
2019-09-26 11:39:06 +00:00
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2019-09-26 11:39:06 +00:00
SELECT startsWith('Hello, world!', 'He');
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-09-26 11:39:06 +00:00
┌─startsWith('Hello, world!', 'He')─┐
│ 1 │
└───────────────────────────────────┘
```
2019-01-30 10:39:46 +00:00
2020-03-18 18:43:51 +00:00
## trim {#trim}
2020-01-05 22:50:16 +00:00
Removes all specified characters from the start or end of a string.
By default removes all consecutive occurrences of common whitespace (ASCII character 32) from both ends of a string.
**Syntax**
2020-03-20 10:10:48 +00:00
``` sql
2020-01-05 22:50:16 +00:00
trim([[LEADING|TRAILING|BOTH] trim_character FROM] input_string)
```
2021-02-15 21:22:10 +00:00
**Arguments**
2020-01-05 22:50:16 +00:00
2021-03-13 18:18:45 +00:00
- `trim_character` — Specified characters for trim. [String ](../../sql-reference/data-types/string.md ).
- `input_string` — String for trim. [String ](../../sql-reference/data-types/string.md ).
2020-01-05 22:50:16 +00:00
**Returned value**
A string without leading and (or) trailing specified characters.
Type: `String` .
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT trim(BOTH ' ()' FROM '( Hello, world! )');
2020-01-05 22:50:16 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2020-01-05 22:50:16 +00:00
┌─trim(BOTH ' ()' FROM '( Hello, world! )')─┐
│ Hello, world! │
└───────────────────────────────────────────────┘
```
2020-03-18 18:43:51 +00:00
## trimLeft {#trimleft}
2019-01-30 10:39:46 +00:00
2021-05-27 19:44:11 +00:00
Removes all consecutive occurrences of common whitespace (ASCII character 32) from the beginning of a string. It does not remove other kinds of whitespace characters (tab, no-break space, etc.).
2019-01-30 10:39:46 +00:00
2020-03-20 10:10:48 +00:00
**Syntax**
2019-01-30 10:39:46 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2020-01-05 22:50:16 +00:00
trimLeft(input_string)
2019-12-05 10:03:35 +00:00
```
2020-01-05 22:50:16 +00:00
Alias: `ltrim(input_string)` .
2019-12-05 10:03:35 +00:00
2021-02-15 21:22:10 +00:00
**Arguments**
2019-12-05 10:03:35 +00:00
2020-04-30 18:19:18 +00:00
- `input_string` — string to trim. [String ](../../sql-reference/data-types/string.md ).
2019-12-05 10:03:35 +00:00
**Returned value**
A string without leading common whitespaces.
Type: `String` .
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT trimLeft(' Hello, world! ');
2019-12-05 10:03:35 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-12-05 10:03:35 +00:00
┌─trimLeft(' Hello, world! ')─┐
│ Hello, world! │
└─────────────────────────────────────┘
```
2020-03-18 18:43:51 +00:00
## trimRight {#trimright}
2019-12-05 10:03:35 +00:00
2021-05-27 19:44:11 +00:00
Removes all consecutive occurrences of common whitespace (ASCII character 32) from the end of a string. It does not remove other kinds of whitespace characters (tab, no-break space, etc.).
2019-12-05 10:03:35 +00:00
2020-03-20 10:10:48 +00:00
**Syntax**
2019-12-05 10:03:35 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2020-01-05 22:50:16 +00:00
trimRight(input_string)
2019-12-05 10:03:35 +00:00
```
2020-01-05 22:50:16 +00:00
Alias: `rtrim(input_string)` .
2019-12-05 10:03:35 +00:00
2021-02-15 21:22:10 +00:00
**Arguments**
2019-12-05 10:03:35 +00:00
2020-04-30 18:19:18 +00:00
- `input_string` — string to trim. [String ](../../sql-reference/data-types/string.md ).
2019-12-05 10:03:35 +00:00
**Returned value**
A string without trailing common whitespaces.
Type: `String` .
2019-01-30 10:39:46 +00:00
2019-12-05 10:03:35 +00:00
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT trimRight(' Hello, world! ');
2019-12-05 10:03:35 +00:00
```
Result:
2019-01-30 10:39:46 +00:00
2020-03-20 10:10:48 +00:00
``` text
2019-12-05 10:03:35 +00:00
┌─trimRight(' Hello, world! ')─┐
│ Hello, world! │
└──────────────────────────────────────┘
```
2020-03-20 10:10:48 +00:00
## trimBoth {#trimboth}
2019-12-05 10:03:35 +00:00
2021-05-27 19:44:11 +00:00
Removes all consecutive occurrences of common whitespace (ASCII character 32) from both ends of a string. It does not remove other kinds of whitespace characters (tab, no-break space, etc.).
2019-12-05 10:03:35 +00:00
2020-03-20 10:10:48 +00:00
**Syntax**
2019-12-05 10:03:35 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2020-01-05 22:50:16 +00:00
trimBoth(input_string)
2019-12-05 10:03:35 +00:00
```
2020-01-05 22:50:16 +00:00
Alias: `trim(input_string)` .
2019-12-05 10:03:35 +00:00
2021-02-15 21:22:10 +00:00
**Arguments**
2019-12-05 10:03:35 +00:00
2020-04-30 18:19:18 +00:00
- `input_string` — string to trim. [String ](../../sql-reference/data-types/string.md ).
2019-12-05 10:03:35 +00:00
**Returned value**
A string without leading and trailing common whitespaces.
Type: `String` .
**Example**
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT trimBoth(' Hello, world! ');
2019-12-05 10:03:35 +00:00
```
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-12-05 10:03:35 +00:00
┌─trimBoth(' Hello, world! ')─┐
│ Hello, world! │
└─────────────────────────────────────┘
```
2019-01-30 10:39:46 +00:00
2020-03-19 15:32:53 +00:00
## CRC32(s) {#crc32}
2019-06-17 21:49:37 +00:00
2019-10-20 20:04:52 +00:00
Returns the CRC32 checksum of a string, using CRC-32-IEEE 802.3 polynomial and initial value `0xffffffff` (zlib implementation).
The result type is UInt32.
2020-03-19 15:32:53 +00:00
## CRC32IEEE(s) {#crc32ieee}
2019-10-20 20:04:52 +00:00
Returns the CRC32 checksum of a string, using CRC-32-IEEE 802.3 polynomial.
2019-06-17 21:49:37 +00:00
The result type is UInt32.
2020-03-19 15:32:53 +00:00
## CRC64(s) {#crc64}
2019-10-20 20:04:52 +00:00
Returns the CRC64 checksum of a string, using CRC-64-ECMA polynomial.
The result type is UInt64.
2020-09-26 07:07:16 +00:00
## normalizeQuery {#normalized-query}
Replaces literals, sequences of literals and complex aliases with placeholders.
2021-03-13 18:18:45 +00:00
**Syntax**
2020-09-26 07:07:16 +00:00
``` sql
normalizeQuery(x)
```
2021-07-29 15:20:55 +00:00
**Arguments**
2020-09-26 07:07:16 +00:00
- `x` — Sequence of characters. [String ](../../sql-reference/data-types/string.md ).
2020-09-28 20:58:08 +00:00
**Returned value**
2020-09-26 07:07:16 +00:00
- Sequence of characters with placeholders.
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
SELECT normalizeQuery('[1, 2, 3, x]') AS query;
```
Result:
``` text
┌─query────┐
│ [?.., x] │
└──────────┘
```
## normalizedQueryHash {#normalized-query-hash}
Returns identical 64bit hash values without the values of literals for similar queries. It helps to analyze query log.
2021-07-29 15:20:55 +00:00
**Syntax**
2020-09-26 07:07:16 +00:00
``` sql
normalizedQueryHash(x)
```
2021-07-29 15:20:55 +00:00
**Arguments**
2020-09-26 07:07:16 +00:00
- `x` — Sequence of characters. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
- Hash value.
Type: [UInt64 ](../../sql-reference/data-types/int-uint.md#uint-ranges ).
**Example**
Query:
``` sql
SELECT normalizedQueryHash('SELECT 1 AS `xyz` ') != normalizedQueryHash('SELECT 1 AS `abc` ') AS res;
```
Result:
``` text
┌─res─┐
│ 1 │
└─────┘
```
2021-10-19 22:31:39 +00:00
## normalizeUTF8NFC {#normalizeutf8nfc}
Converts a string to [NFC normalized form ](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms ), assuming the string contains a set of bytes that make up a UTF-8 encoded text.
**Syntax**
``` sql
2021-10-25 10:44:50 +00:00
normalizeUTF8NFC(words)
2021-10-19 22:31:39 +00:00
```
**Arguments**
- `words` — Input string that contains UTF-8 encoded text. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
2021-10-22 20:10:02 +00:00
- String transformed to NFC normalization form.
2021-10-19 22:31:39 +00:00
Type: [String ](../../sql-reference/data-types/string.md ).
2021-10-22 20:10:02 +00:00
**Example**
Query:
``` sql
2021-10-25 09:46:50 +00:00
SELECT length('â'), normalizeUTF8NFC('â') AS nfc, length(nfc) AS nfc_len;
2021-10-22 20:10:02 +00:00
```
Result:
``` text
2021-10-25 09:46:50 +00:00
┌─length('â')─┬─nfc─┬─nfc_len─┐
│ 2 │ â │ 2 │
2021-10-22 20:10:02 +00:00
└─────────────┴─────┴─────────┘
```
## normalizeUTF8NFD {#normalizeutf8nfd}
Converts a string to [NFD normalized form ](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms ), assuming the string contains a set of bytes that make up a UTF-8 encoded text.
**Syntax**
``` sql
normalizeUTF8NFD(words)
```
**Arguments**
- `words` — Input string that contains UTF-8 encoded text. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
- String transformed to NFD normalization form.
2021-10-19 22:31:39 +00:00
2021-10-22 20:10:02 +00:00
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
2021-10-25 09:46:50 +00:00
SELECT length('â'), normalizeUTF8NFD('â') AS nfd, length(nfd) AS nfd_len;
2021-10-22 20:10:02 +00:00
```
Result:
``` text
2021-10-25 09:46:50 +00:00
┌─length('â')─┬─nfd─┬─nfd_len─┐
│ 2 │ â │ 3 │
2021-10-22 20:10:02 +00:00
└─────────────┴─────┴─────────┘
```
## normalizeUTF8NFKC {#normalizeutf8nfkc}
Converts a string to [NFKC normalized form ](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms ), assuming the string contains a set of bytes that make up a UTF-8 encoded text.
**Syntax**
``` sql
normalizeUTF8NFKC(words)
```
**Arguments**
- `words` — Input string that contains UTF-8 encoded text. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
- String transformed to NFKC normalization form.
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
2021-10-25 09:46:50 +00:00
SELECT length('â'), normalizeUTF8NFKC('â') AS nfkc, length(nfkc) AS nfkc_len;
2021-10-22 20:10:02 +00:00
```
Result:
``` text
2021-10-25 09:46:50 +00:00
┌─length('â')─┬─nfkc─┬─nfkc_len─┐
│ 2 │ â │ 2 │
2021-10-22 20:10:02 +00:00
└─────────────┴──────┴──────────┘
```
## normalizeUTF8NFKD {#normalizeutf8nfkd}
Converts a string to [NFKD normalized form ](https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms ), assuming the string contains a set of bytes that make up a UTF-8 encoded text.
**Syntax**
``` sql
normalizeUTF8NFKD(words)
```
**Arguments**
- `words` — Input string that contains UTF-8 encoded text. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
- String transformed to NFKD normalization form.
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
2021-10-25 09:46:50 +00:00
SELECT length('â'), normalizeUTF8NFKD('â') AS nfkd, length(nfkd) AS nfkd_len;
2021-10-22 20:10:02 +00:00
```
Result:
``` text
2021-10-25 09:46:50 +00:00
┌─length('â')─┬─nfkd─┬─nfkd_len─┐
│ 2 │ â │ 3 │
2021-10-22 20:10:02 +00:00
└─────────────┴──────┴──────────┘
```
2021-10-19 22:31:39 +00:00
2020-12-27 11:21:58 +00:00
## encodeXMLComponent {#encode-xml-component}
2020-12-29 13:27:47 +00:00
Escapes characters to place string into XML text node or attribute.
2020-12-27 11:21:58 +00:00
2020-12-29 13:27:47 +00:00
The following five XML predefined entities will be replaced: `<` , `&` , `>` , `"` , `'` .
2020-12-27 20:49:05 +00:00
2021-07-29 15:20:55 +00:00
**Syntax**
2020-12-27 11:21:58 +00:00
``` sql
encodeXMLComponent(x)
```
2021-07-29 15:20:55 +00:00
**Arguments**
2020-12-27 11:21:58 +00:00
2020-12-27 20:49:05 +00:00
- `x` — The sequence of characters. [String ](../../sql-reference/data-types/string.md ).
2020-12-27 11:21:58 +00:00
2021-01-18 15:24:17 +00:00
**Returned value**
2020-12-27 11:21:58 +00:00
2020-12-27 20:49:05 +00:00
- The sequence of characters with escape characters.
2020-12-27 11:21:58 +00:00
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
2020-12-27 20:49:05 +00:00
SELECT encodeXMLComponent('Hello, "world"!');
SELECT encodeXMLComponent('< 123 > ');
SELECT encodeXMLComponent('&clickhouse');
SELECT encodeXMLComponent('\'foo\'');
2020-12-27 11:21:58 +00:00
```
Result:
``` text
2020-12-27 20:49:05 +00:00
Hello, " world" !
< 123>
& clickhouse
' foo'
2020-12-27 11:21:58 +00:00
```
2021-02-12 19:28:03 +00:00
## decodeXMLComponent {#decode-xml-component}
2021-02-15 18:25:32 +00:00
Replaces XML predefined entities with characters. Predefined entities are `"` `&` `'` `>` `<`
This function also replaces numeric character references with Unicode characters. Both decimal (like `✓` ) and hexadecimal (`✓ `) forms are supported.
2021-02-12 19:28:03 +00:00
**Syntax**
``` sql
decodeXMLComponent(x)
```
2021-03-13 18:18:45 +00:00
**Arguments**
2021-02-12 19:28:03 +00:00
- `x` — A sequence of characters. [String ](../../sql-reference/data-types/string.md ).
**Returned value**
- The sequence of characters after replacement.
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
Query:
``` sql
SELECT decodeXMLComponent('' foo' ');
SELECT decodeXMLComponent('< Σ > ');
```
Result:
``` text
2021-07-29 15:20:55 +00:00
'foo'
2021-02-12 19:28:03 +00:00
< Σ >
```
**See Also**
- [List of XML and HTML character entity references ](https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references )
2021-03-28 18:42:04 +00:00
## extractTextFromHTML {#extracttextfromhtml}
2021-04-06 18:52:06 +00:00
A function to extract text from HTML or XHTML.
2021-04-07 18:35:11 +00:00
It does not necessarily 100% conform to any of the HTML, XML or XHTML standards, but the implementation is reasonably accurate and it is fast. The rules are the following:
2021-04-06 18:52:06 +00:00
1. Comments are skipped. Example: `<!-- test -->` . Comment must end with `-->` . Nested comments are not possible.
2021-04-07 18:35:11 +00:00
Note: constructions like `<!-->` and `<!--->` are not valid comments in HTML but they are skipped by other rules.
2. CDATA is pasted verbatim. Note: CDATA is XML/XHTML specific. But it is processed for "best-effort" approach.
3. `script` and `style` elements are removed with all their content. Note: it is assumed that closing tag cannot appear inside content. For example, in JS string literal has to be escaped like `"<\/script>"` .
2021-04-07 19:23:53 +00:00
Note: comments and CDATA are possible inside `script` or `style` - then closing tags are not searched inside CDATA. Example: `<script><![CDATA[</script>]]></script>` . But they are still searched inside comments. Sometimes it becomes complicated: `<script>var x = "<!--"; </script> var y = "-->"; alert(x + y);</script>`
2021-04-06 19:10:22 +00:00
Note: `script` and `style` can be the names of XML namespaces - then they are not treated like usual `script` or `style` elements. Example: `<script:a>Hello</script:a>` .
2021-04-06 18:52:06 +00:00
Note: whitespaces are possible after closing tag name: `</script >` but not before: `< / script>` .
4. Other tags or tag-like elements are skipped without inner content. Example: `<a>.</a>`
2021-04-07 18:35:11 +00:00
Note: it is expected that this HTML is illegal: `<a test=">"></a>`
Note: it also skips something like tags: `<>` , `<!>` , etc.
Note: tag without end is skipped to the end of input: `<hello `
2021-04-06 19:10:22 +00:00
5. HTML and XML entities are not decoded. They must be processed by separate function.
6. Whitespaces in the text are collapsed or inserted by specific rules.
- Whitespaces at the beginning and at the end are removed.
- Consecutive whitespaces are collapsed.
- But if the text is separated by other elements and there is no whitespace, it is inserted.
2021-04-07 18:35:11 +00:00
- It may cause unnatural examples: `Hello<b>world</b>` , `Hello<!-- -->world` - there is no whitespace in HTML, but the function inserts it. Also consider: `Hello<p>world</p>` , `Hello<br>world` . This behavior is reasonable for data analysis, e.g. to convert HTML to a bag of words.
7. Also note that correct handling of whitespaces requires the support of `<pre></pre>` and CSS `display` and `white-space` properties.
2021-03-28 18:42:04 +00:00
**Syntax**
``` sql
extractTextFromHTML(x)
```
2021-04-03 16:16:56 +00:00
**Arguments**
2021-03-28 18:42:04 +00:00
2021-07-29 15:20:55 +00:00
- `x` — input text. [String ](../../sql-reference/data-types/string.md ).
2021-03-28 18:42:04 +00:00
**Returned value**
- Extracted text.
Type: [String ](../../sql-reference/data-types/string.md ).
**Example**
2021-04-03 19:31:24 +00:00
The first example contains several tags and a comment and also shows whitespace processing.
2021-04-05 20:19:18 +00:00
The second example shows `CDATA` and `script` tag processing.
2021-04-07 18:35:11 +00:00
In the third example text is extracted from the full HTML response received by the [url ](../../sql-reference/table-functions/url.md ) function.
2021-04-03 16:16:56 +00:00
2021-03-28 18:42:04 +00:00
Query:
``` sql
2021-04-05 20:53:55 +00:00
SELECT extractTextFromHTML(' < p > A text < i > with< / i > < b > tags< / b > . <!-- comments --> < / p > ');
2021-04-03 16:16:56 +00:00
SELECT extractTextFromHTML('<![CDATA[The content within <b>CDATA</b>]]> < script > alert ( "Script" ) ; < / script > ');
2021-04-05 19:37:01 +00:00
SELECT extractTextFromHTML(html) FROM url('http://www.donothingfor2minutes.com/', RawBLOB, 'html String');
2021-03-28 18:42:04 +00:00
```
Result:
``` text
2021-04-05 20:53:55 +00:00
A text with tags .
2021-04-03 16:16:56 +00:00
The content within < b > CDATA< / b >
2021-04-05 19:37:01 +00:00
Do Nothing for 2 Minutes 2:00
2021-04-03 16:16:56 +00:00
```