2020-04-03 13:23:32 +00:00
---
2022-08-28 14:53:34 +00:00
slug: /en/sql-reference/functions/url-functions
2023-04-19 17:05:55 +00:00
sidebar_position: 200
2022-04-09 13:29:05 +00:00
sidebar_label: URLs
2020-04-03 13:23:32 +00:00
---
2022-06-02 10:55:18 +00:00
# Functions for Working with URLs
2017-12-28 15:13:23 +00:00
2024-05-29 11:50:45 +00:00
:::note
2024-05-29 13:49:46 +00:00
The functions mentioned in this section are optimized for maximum performance and for the most part do not follow the RFC-3986 standard. Functions which implement RFC-3986 have `RFC` appended to their function name and are generally slower.
2024-05-29 11:50:45 +00:00
:::
2017-12-28 15:13:23 +00:00
2024-05-29 13:49:46 +00:00
You can generally use the non-`RFC` function variants when working with publicly registered domains that contain neither user strings nor `@` symbols.
2024-05-29 18:26:52 +00:00
The table below details which symbols in a URL can (`✔`) or cannot (`✗`) be parsed by the respective `RFC` and non-`RFC` variants:
2024-05-29 13:04:02 +00:00
|Symbol | non-`RFC`| `RFC` |
|-------|----------|-------|
2024-05-29 13:43:15 +00:00
| ' ' | ✗ |✗ |
| \t | ✗ |✗ |
| < | ✗ |✗ |
| > | ✗ |✗ |
| % | ✗ |✔* |
| { | ✗ |✗ |
| } | ✗ |✗ |
| \| | ✗ |✗ |
| \\\ | ✗ |✗ |
| ^ | ✗ |✗ |
| ~ | ✗ |✔* |
| [ | ✗ |✗ |
| ] | ✗ |✔ |
| ; | ✗ |✔* |
| = | ✗ |✔* |
2024-05-29 13:04:02 +00:00
| & | ✗ |✔* |
2024-05-29 13:49:46 +00:00
symbols marked `*` are sub-delimiters in RFC 3986 and allowed for user info following the `@` symbol.
2024-05-29 13:04:02 +00:00
2022-06-02 10:55:18 +00:00
## Functions that Extract Parts of a URL
2017-12-28 15:13:23 +00:00
2020-03-20 10:10:48 +00:00
If the relevant part isn’ t present in a URL, an empty string is returned.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### protocol
2017-12-28 15:13:23 +00:00
2019-08-26 16:09:30 +00:00
Extracts the protocol from a URL.
2024-05-29 13:50:24 +00:00
Examples of typical returned values: http, https, ftp, mailto, tel, magnet.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### domain
2017-12-28 15:13:23 +00:00
2019-08-26 16:09:30 +00:00
Extracts the hostname from a URL.
2019-07-31 03:53:36 +00:00
2024-05-29 11:50:45 +00:00
**Syntax**
2020-03-20 10:10:48 +00:00
``` sql
2024-05-29 13:52:54 +00:00
domain(url)
2019-07-31 03:53:36 +00:00
```
2021-02-15 21:22:10 +00:00
**Arguments**
2019-07-31 03:53:36 +00:00
2024-05-29 18:26:52 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2019-07-31 03:53:36 +00:00
2024-05-29 11:50:45 +00:00
The URL can be specified with or without a protocol. Examples:
2019-07-31 03:53:36 +00:00
2020-03-20 10:10:48 +00:00
``` text
2019-07-31 03:53:36 +00:00
svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
2022-03-12 06:24:31 +00:00
https://clickhouse.com/time/
2019-07-31 03:53:36 +00:00
```
2019-08-26 16:09:30 +00:00
For these examples, the `domain` function returns the following results:
2020-03-20 10:10:48 +00:00
``` text
2019-08-26 16:09:30 +00:00
some.svn-hosting.com
some.svn-hosting.com
2022-03-12 06:24:31 +00:00
clickhouse.com
2019-08-26 16:09:30 +00:00
```
2019-07-31 03:53:36 +00:00
**Returned values**
2024-05-29 14:15:46 +00:00
- Host name if the input string can be parsed as a URL, otherwise an empty string. [String ](../data-types/string.md ).
2019-07-31 03:53:36 +00:00
**Example**
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT domain('svn+ssh://some.svn-hosting.com:80/repo/trunk');
2019-07-31 03:53:36 +00:00
```
2020-03-20 10:10:48 +00:00
``` text
2019-07-31 03:53:36 +00:00
┌─domain('svn+ssh://some.svn-hosting.com:80/repo/trunk')─┐
│ some.svn-hosting.com │
└────────────────────────────────────────────────────────┘
```
2017-12-28 15:13:23 +00:00
2024-05-29 11:50:45 +00:00
### domainRFC
Extracts the hostname from a URL. Similar to [domain ](#domain ), but RFC 3986 conformant.
**Syntax**
``` sql
2024-05-29 13:52:54 +00:00
domainRFC(url)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 18:26:52 +00:00
- `url` — URL. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned values**
2024-05-29 14:15:46 +00:00
- Host name if the input string can be parsed as a URL, otherwise an empty string. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
``` sql
SELECT
domain('http://user:password@example.com:8080/path?query=value#fragment'),
domainRFC('http://user:password@example.com:8080/path?query=value#fragment');
```
``` text
┌─domain('http://user:password@example.com:8080/path?query=value#fragment')─┬─domainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│ │ example.com │
└───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────┘
```
2022-06-02 10:55:18 +00:00
### domainWithoutWWW
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the domain without leading `www.` if present.
2017-12-28 15:13:23 +00:00
2024-05-29 18:26:52 +00:00
**Syntax**
```sql
domainWithoutWWW(url)
```
**Arguments**
- `url` — URL. [String ](../data-types/string.md ).
**Returned values**
- Domain name if the input string can be parsed as a URL (without leading `www.` ), otherwise an empty string. [String ](../data-types/string.md ).
**Example**
``` sql
SELECT domainWithoutWWW('http://paul@www.example.com:80/');
```
``` text
┌─domainWithoutWWW('http://paul@www.example.com:80/')─┐
│ example.com │
└─────────────────────────────────────────────────────┘
```
### domainWithoutWWWRFC
Returns the domain without leading `www.` if present. Similar to [domainWithoutWWW ](#domainwithoutwww ) but conforms to RFC 3986.
**Syntax**
```sql
domainWithoutWWWRFC(url)
```
**Arguments**
- `url` — URL. [String ](../data-types/string.md ).
**Returned values**
- Domain name if the input string can be parsed as a URL (without leading `www.` ), otherwise an empty string. [String ](../data-types/string.md ).
**Example**
Query:
```sql
SELECT
domainWithoutWWW('http://user:password@www.example.com:8080/path?query=value#fragment'),
domainWithoutWWWRFC('http://user:password@www.example.com:8080/path?query=value#fragment');
```
Result:
```response
┌─domainWithoutWWW('http://user:password@www.example.com:8080/path?query=value#fragment')─┬─domainWithoutWWWRFC('http://user:password@www.example.com:8080/path?query=value#fragment')─┐
│ │ example.com │
└─────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────┘
```
2022-06-02 10:55:18 +00:00
### topLevelDomain
2017-12-28 15:13:23 +00:00
2019-08-26 16:09:30 +00:00
Extracts the the top-level domain from a URL.
2019-07-31 03:53:36 +00:00
2020-03-20 10:10:48 +00:00
``` sql
2024-05-29 13:52:54 +00:00
topLevelDomain(url)
2019-07-31 03:53:36 +00:00
```
2021-02-15 21:22:10 +00:00
**Arguments**
2019-07-31 03:53:36 +00:00
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2019-07-31 03:53:36 +00:00
2024-05-28 07:02:21 +00:00
:::note
2024-05-29 11:50:45 +00:00
The URL can be specified with or without a protocol. Examples:
2019-07-31 03:53:36 +00:00
2020-03-20 10:10:48 +00:00
``` text
2019-07-31 03:53:36 +00:00
svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
2022-03-12 06:24:31 +00:00
https://clickhouse.com/time/
2019-07-31 03:53:36 +00:00
```
2024-05-28 07:02:21 +00:00
:::
2019-07-31 03:53:36 +00:00
**Returned values**
2024-05-29 14:15:46 +00:00
- Domain name if the input string can be parsed as a URL. Otherwise, an empty string. [String ](../../sql-reference/data-types/string.md ).
2019-07-31 03:53:36 +00:00
**Example**
2024-05-28 07:02:21 +00:00
Query:
2020-03-20 10:10:48 +00:00
``` sql
2021-03-13 18:18:45 +00:00
SELECT topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk');
2019-07-31 03:53:36 +00:00
```
2020-03-20 10:10:48 +00:00
2024-05-28 07:02:21 +00:00
Result:
2020-03-20 10:10:48 +00:00
``` text
2019-07-31 03:53:36 +00:00
┌─topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk')─┐
│ com │
└────────────────────────────────────────────────────────────────────┘
```
2017-12-28 15:13:23 +00:00
2024-05-28 07:02:21 +00:00
### topLevelDomainRFC
2024-05-29 14:15:46 +00:00
Extracts the the top-level domain from a URL.
Similar to [topLevelDomain ](#topleveldomain ), but conforms to RFC 3986.
2024-05-28 07:02:21 +00:00
``` sql
2024-05-29 13:52:54 +00:00
topLevelDomainRFC(url)
2024-05-28 07:02:21 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-28 07:02:21 +00:00
:::note
2024-05-29 11:50:45 +00:00
The URL can be specified with or without a protocol. Examples:
2024-05-28 07:02:21 +00:00
``` text
svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/
```
:::
**Returned values**
2024-05-29 14:15:46 +00:00
- Domain name if the input string can be parsed as a URL. Otherwise, an empty string. [String ](../../sql-reference/data-types/string.md ).
2024-05-28 07:02:21 +00:00
**Example**
Query:
``` sql
2024-05-29 11:50:45 +00:00
SELECT topLevelDomain('http://foo:foo%41bar@foo.com'), topLevelDomainRFC('http://foo:foo%41bar@foo.com');
2024-05-28 07:02:21 +00:00
```
Result:
``` text
2024-05-29 11:50:45 +00:00
┌─topLevelDomain('http://foo:foo%41bar@foo.com')─┬─topLevelDomainRFC('http://foo:foo%41bar@foo.com')─┐
│ │ com │
└────────────────────────────────────────────────┴───────────────────────────────────────────────────┘
2024-05-28 07:02:21 +00:00
```
2022-06-02 10:55:18 +00:00
### firstSignificantSubdomain
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the “first significant subdomain”.
The first significant subdomain is a second-level domain for `com` , `net` , `org` , or `co` , otherwise it is a third-level domain.
For example, `firstSignificantSubdomain (‘ https://news.clickhouse.com/’ ) = ‘ clickhouse’ , firstSignificantSubdomain (‘ https://news.clickhouse.com.tr/’ ) = ‘ clickhouse’ ` .
The list of "insignificant" second-level domains and other implementation details may change in the future.
2017-12-28 15:13:23 +00:00
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 14:15:46 +00:00
firstSignificantSubdomain(url)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 13:26:10 +00:00
- The first significant subdomain. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')
```
Result:
```reference
┌─firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')─┐
│ example │
└───────────────────────────────────────────────────────────────┘
```
### firstSignificantSubdomainRFC
2024-05-29 14:15:46 +00:00
Returns the “first significant subdomain”.
The first significant subdomain is a second-level domain for `com` , `net` , `org` , or `co` , otherwise it is a third-level domain.
For example, `firstSignificantSubdomain (‘ https://news.clickhouse.com/’ ) = ‘ clickhouse’ , firstSignificantSubdomain (‘ https://news.clickhouse.com.tr/’ ) = ‘ clickhouse’ ` .
The list of "insignificant" second-level domains and other implementation details may change in the future.
Similar to [firstSignficantSubdomain ](#firstsignificantsubdomain ) but conforms to RFC 1034.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
firstSignificantSubdomainRFC(url)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 13:26:10 +00:00
- The first significant subdomain. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT
firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment'),
firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment');
```
Result:
```reference
┌─firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment')─┬─firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│ │ example │
└──────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘
```
2022-06-02 10:55:18 +00:00
### cutToFirstSignificantSubdomain
2017-12-28 15:13:23 +00:00
2024-05-29 11:50:45 +00:00
Returns the part of the domain that includes top-level subdomains up to the [“first significant subdomain” ](#firstsignificantsubdomain ).
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomain(url)
2024-05-29 11:50:45 +00:00
```
2017-12-28 15:13:23 +00:00
2024-05-29 11:50:45 +00:00
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 13:26:10 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT
cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/'),
cutToFirstSignificantSubdomain('www.tr'),
cutToFirstSignificantSubdomain('tr');
```
Result:
```response
┌─cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomain('www.tr')─┬─cutToFirstSignificantSubdomain('tr')─┐
│ clickhouse.com.tr │ tr │ │
└───────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┴──────────────────────────────────────┘
```
### cutToFirstSignificantSubdomainRFC
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the [“first significant subdomain” ](#firstsignificantsubdomain ).
Similar to [cutToFirstSignificantSubdomain ](#cuttofirstsignificantsubdomain ) but conforms to RFC 3986.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainRFC(url)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 13:26:10 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT
cutToFirstSignificantSubdomain('http://user:password@example.com:8080'),
cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080');
```
Result:
```response
┌─cutToFirstSignificantSubdomain('http://user:password@example.com:8080')─┬─cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080')─┐
│ │ example.com │
└─────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────┘
```
2020-11-10 22:04:59 +00:00
2022-06-02 10:55:18 +00:00
### cutToFirstSignificantSubdomainWithWWW
2020-11-10 22:04:59 +00:00
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain", without stripping `www` .
2020-11-10 22:04:59 +00:00
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainWithWWW(url)
2024-05-29 11:50:45 +00:00
```
2020-11-10 22:04:59 +00:00
2024-05-29 11:50:45 +00:00
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 14:15:46 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain (with `www` ) if possible, otherwise returns an empty string. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT
cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/'),
cutToFirstSignificantSubdomainWithWWW('www.tr'),
cutToFirstSignificantSubdomainWithWWW('tr');
```
Result:
```response
┌─cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomainWithWWW('www.tr')─┬─cutToFirstSignificantSubdomainWithWWW('tr')─┐
│ clickhouse.com.tr │ www.tr │ │
└──────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┴─────────────────────────────────────────────┘
```
### cutToFirstSignificantSubdomainWithWWWRFC
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain", without stripping `www` .
Similar to [cutToFirstSignificantSubdomainWithWWW ](#cuttofirstsignificantsubdomaincustomwithwww ) but conforms to RFC 3986.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainWithWWW(url)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 13:26:10 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain (with "www") if possible, otherwise returns an empty string. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Example**
Query:
```sql
SELECT
cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy'),
cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy');
```
Result:
```response
┌─cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┬─cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┐
│ │ mail.ru │
└───────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
```
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### cutToFirstSignificantSubdomainCustom
2020-12-03 21:11:38 +00:00
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the first significant subdomain.
Accepts custom [TLD list ](https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains ) name.
This function can be useful if you need a fresh TLD list or if you have a custom list.
2020-12-03 21:11:38 +00:00
2024-05-29 11:50:45 +00:00
**Configuration example**
2020-12-03 21:11:38 +00:00
```xml
2020-12-08 20:54:03 +00:00
<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
2020-12-03 21:11:38 +00:00
< top_level_domains_lists >
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
2020-12-08 20:54:03 +00:00
< public_suffix_list > public_suffix_list.dat< / public_suffix_list >
<!-- NOTE: path is under top_level_domains_path -->
2020-12-03 21:11:38 +00:00
< / top_level_domains_lists >
```
2021-02-16 11:02:33 +00:00
**Syntax**
2020-12-03 21:11:38 +00:00
2021-02-16 11:02:33 +00:00
``` sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomain(url, tld)
2021-02-16 11:02:33 +00:00
```
2024-03-11 08:34:01 +00:00
**Arguments**
2021-02-16 11:02:33 +00:00
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Returned value**
2024-05-29 13:04:02 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain. [String ](../../sql-reference/data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Example**
Query:
```sql
SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');
```
Result:
```text
┌─cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo.there-is-no-such-domain │
└───────────────────────────────────────────────────────────────────────────────────────────────┘
```
**See Also**
2023-04-19 15:55:29 +00:00
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
2020-12-03 21:11:38 +00:00
2024-05-29 11:50:45 +00:00
### cutToFirstSignificantSubdomainCustomRFC
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the first significant subdomain.
Accepts custom [TLD list ](https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains ) name.
This function can be useful if you need a fresh TLD list or if you have a custom list.
Similar to [cutToFirstSignificantSubdomainCustom ](#cuttofirstsignificantsubdomaincustom ) but conforms to RFC 3986.
2024-05-29 11:50:45 +00:00
**Syntax**
``` sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainRFC(url, tld)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
- Part of the domain that includes top-level subdomains up to the first significant subdomain. [String ](../../sql-reference/data-types/string.md ).
**See Also**
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
2022-06-02 10:55:18 +00:00
### cutToFirstSignificantSubdomainCustomWithWWW
2020-12-03 21:11:38 +00:00
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www` .
Accepts custom TLD list name.
2024-05-29 11:50:45 +00:00
It can be useful if you need a fresh TLD list or if you have a custom list.
2021-02-16 11:02:33 +00:00
2024-05-29 11:50:45 +00:00
**Configuration example**
2021-02-16 11:02:33 +00:00
```xml
<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
< top_level_domains_lists >
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
< public_suffix_list > public_suffix_list.dat< / public_suffix_list >
<!-- NOTE: path is under top_level_domains_path -->
< / top_level_domains_lists >
```
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainCustomWithWWW(url, tld)
2021-02-16 11:02:33 +00:00
```
2024-03-11 08:34:01 +00:00
**Arguments**
2021-02-16 11:02:33 +00:00
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Returned value**
2024-05-29 18:26:52 +00:00
- Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www` . [String ](../data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Example**
Query:
```sql
SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list');
```
Result:
```text
┌─cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐
│ www.foo │
└──────────────────────────────────────────────────────────────────────────────┘
```
**See Also**
2023-04-19 15:55:29 +00:00
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
2020-12-03 21:11:38 +00:00
2024-05-29 11:50:45 +00:00
### cutToFirstSignificantSubdomainCustomWithWWWRFC
2024-05-29 14:15:46 +00:00
Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www` .
Accepts custom TLD list name.
It can be useful if you need a fresh TLD list or if you have a custom list.
Similar to [cutToFirstSignificantSubdomainCustomWithWWW ](#cuttofirstsignificantsubdomaincustomwithwww ) but conforms to RFC 3986.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
cutToFirstSignificantSubdomainCustomWithWWWRFC(url, tld)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
- Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping `www` . [String ](../../sql-reference/data-types/string.md ).
**See Also**
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
2022-06-02 10:55:18 +00:00
### firstSignificantSubdomainCustom
2020-12-03 21:11:38 +00:00
2024-05-29 14:15:46 +00:00
Returns the first significant subdomain.
Accepts customs TLD list name.
2021-02-16 11:02:33 +00:00
Can be useful if you need fresh TLD list or you have custom.
Configuration example:
```xml
<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
< top_level_domains_lists >
<!-- https://publicsuffix.org/list/public_suffix_list.dat -->
< public_suffix_list > public_suffix_list.dat< / public_suffix_list >
<!-- NOTE: path is under top_level_domains_path -->
< / top_level_domains_lists >
```
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
firstSignificantSubdomainCustom(url, tld)
2021-02-16 11:02:33 +00:00
```
2024-03-11 08:34:01 +00:00
**Arguments**
2021-02-16 11:02:33 +00:00
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Returned value**
2024-05-29 18:26:52 +00:00
- First significant subdomain. [String ](../../sql-reference/data-types/string.md ).
2021-02-16 11:02:33 +00:00
**Example**
Query:
```sql
SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');
```
Result:
2021-07-29 15:20:55 +00:00
```text
2021-02-16 11:02:33 +00:00
┌─firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo │
└──────────────────────────────────────────────────────────────────────────────────────────┘
```
**See Also**
2020-12-03 21:11:38 +00:00
2023-04-19 15:55:29 +00:00
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
2020-12-03 21:11:38 +00:00
2024-05-29 11:50:45 +00:00
### firstSignificantSubdomainCustomRFC
2020-05-21 21:37:47 +00:00
2024-05-29 14:15:46 +00:00
Returns the first significant subdomain.
Accepts customs TLD list name.
Can be useful if you need fresh TLD list or you have custom.
Similar to [firstSignificantSubdomainCustom ](#firstsignificantsubdomaincustom ) but conforms to RFC 3986.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
firstSignificantSubdomainCustomRFC(url, tld)
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
- `tld` — Custom TLD list name. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**Returned value**
2024-05-29 18:26:52 +00:00
- First significant subdomain. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
**See Also**
- [firstSignificantSubdomain ](#firstsignificantsubdomain ).
### port
2024-05-29 14:15:46 +00:00
Returns the port or `default_port` if the URL contains no port or cannot be parsed.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
port(url [, default_port = 0])
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../data-types/string.md ).
2024-05-29 11:50:45 +00:00
- `default_port` — The default port number to be returned. [UInt16 ](../data-types/int-uint.md ).
**Returned value**
- Port or the default port if there is no port in the URL or in case of a validation error. [UInt16 ](../data-types/int-uint.md ).
**Example**
Query:
```sql
SELECT port('http://paul@www.example.com:80/');
```
Result:
```response
┌─port('http://paul@www.example.com:80/')─┐
│ 80 │
└─────────────────────────────────────────┘
```
### portRFC
2024-05-29 14:15:46 +00:00
Returns the port or `default_port` if the URL contains no port or cannot be parsed.
Similar to [port ](#port ), but RFC 3986 conformant.
2024-05-29 11:50:45 +00:00
**Syntax**
```sql
2024-05-29 13:52:54 +00:00
portRFC(url [, default_port = 0])
2024-05-29 11:50:45 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 11:50:45 +00:00
- `default_port` — The default port number to be returned. [UInt16 ](../data-types/int-uint.md ).
**Returned value**
- Port or the default port if there is no port in the URL or in case of a validation error. [UInt16 ](../data-types/int-uint.md ).
**Example**
Query:
```sql
SELECT
port('http://user:password@example.com:8080'),
portRFC('http://user:password@example.com:8080');
```
Result:
```resposne
┌─port('http://user:password@example.com:8080')─┬─portRFC('http://user:password@example.com:8080')─┐
│ 0 │ 8080 │
└───────────────────────────────────────────────┴──────────────────────────────────────────────────┘
```
2020-05-21 21:37:47 +00:00
2022-06-02 10:55:18 +00:00
### path
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the path without query string.
Example: `/top/news.html` .
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### pathFull
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
The same as above, but including query string and fragment.
Example: `/top/news.html?page=2#comments` .
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### queryString
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the query string without the initial question mark, `#` and everything after `#` .
Example: `page=1&lr=213` .
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### fragment
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the fragment identifier without the initial hash symbol.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### queryStringAndFragment
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the query string and fragment identifier.
Example: `page=1#29390` .
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### extractURLParameter(url, name)
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns the value of the `name` parameter in the URL, if present, otherwise an empty string is returned.
If there are multiple parameters with this name, the first occurrence is returned.
The function assumes that the parameter in the `url` parameter is encoded in the same way as in the `name` argument.
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### extractURLParameters(url)
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns an array of `name=value` strings corresponding to the URL parameters.
The values are not decoded.
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### extractURLParameterNames(url)
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns an array of name strings corresponding to the names of URL parameters.
The values are not decoded.
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### URLHierarchy(url)
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Returns an array containing the URL, truncated at the end by the symbols /,? in the path and query-string.
Consecutive separator characters are counted as one.
The cut is made in the position after all the consecutive separator characters.
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### URLPathHierarchy(url)
2017-12-28 15:13:23 +00:00
2022-03-12 06:24:31 +00:00
The same as above, but without the protocol and host in the result. The / element (root) is not included.
2017-12-28 15:13:23 +00:00
2020-03-20 10:10:48 +00:00
``` text
2017-12-28 15:13:23 +00:00
URLPathHierarchy('https://example.com/browse/CONV-6788') =
[
'/browse/',
'/browse/CONV-6788'
]
```
2024-05-29 13:52:54 +00:00
### encodeURLComponent(url)
2022-02-16 02:19:20 +00:00
Returns the encoded URL.
2024-05-29 14:15:46 +00:00
2022-02-16 02:19:20 +00:00
Example:
``` sql
SELECT encodeURLComponent('http://127.0.0.1:8123/?query=SELECT 1;') AS EncodedURL;
```
``` text
┌─EncodedURL───────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT%201%3B │
└──────────────────────────────────────────────────────────┘
```
2024-05-29 13:52:54 +00:00
### decodeURLComponent(url)
2017-12-28 15:13:23 +00:00
Returns the decoded URL.
2024-05-29 14:15:46 +00:00
2017-12-28 15:13:23 +00:00
Example:
2020-03-20 10:10:48 +00:00
``` sql
2017-12-28 15:13:23 +00:00
SELECT decodeURLComponent('http://127.0.0.1:8123/?query=SELECT%201%3B') AS DecodedURL;
```
2020-03-20 10:10:48 +00:00
``` text
2017-12-28 15:13:23 +00:00
┌─DecodedURL─────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1; │
└────────────────────────────────────────┘
2020-07-09 08:50:53 +00:00
```
2024-05-29 13:52:54 +00:00
### encodeURLFormComponent(url)
2022-02-16 02:19:20 +00:00
Returns the encoded URL. Follows rfc-1866, space(` `) is encoded as plus(` +`).
2024-05-29 14:15:46 +00:00
2022-02-16 02:19:20 +00:00
Example:
``` sql
SELECT encodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT 1 2+3') AS EncodedURL;
```
``` text
┌─EncodedURL────────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT+1+2%2B3 │
└───────────────────────────────────────────────────────────┘
```
2024-05-29 13:52:54 +00:00
### decodeURLFormComponent(url)
2022-01-07 12:51:30 +00:00
Returns the decoded URL. Follows rfc-1866, plain plus(`+`) is decoded as space(` `).
2024-05-29 14:15:46 +00:00
2022-01-07 12:51:30 +00:00
Example:
``` sql
SELECT decodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT%201+2%2B3') AS DecodedURL;
```
``` text
┌─DecodedURL────────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1 2+3 │
└───────────────────────────────────────────┘
```
2022-06-02 10:55:18 +00:00
### netloc
2020-07-09 08:50:53 +00:00
Extracts network locality (`username:password@host:port`) from a URL.
**Syntax**
2020-07-11 11:05:49 +00:00
``` sql
2024-05-29 13:52:54 +00:00
netloc(url)
2020-07-09 08:50:53 +00:00
```
2021-02-15 21:22:10 +00:00
**Arguments**
2020-07-09 08:50:53 +00:00
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2020-07-09 08:50:53 +00:00
**Returned value**
2024-05-29 18:26:52 +00:00
- `username:password@host:port` . [String ](../data-types/string.md ).
2020-07-09 08:50:53 +00:00
**Example**
Query:
``` sql
SELECT netloc('http://paul@www.example.com:80/');
```
Result:
``` text
┌─netloc('http://paul@www.example.com:80/')─┐
│ paul@www.example.com:80 │
└───────────────────────────────────────────┘
2017-12-28 15:13:23 +00:00
```
2024-05-29 14:15:46 +00:00
## Functions that remove part of a URL
2017-12-28 15:13:23 +00:00
2021-05-27 19:44:11 +00:00
If the URL does not have anything similar, the URL remains unchanged.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### cutWWW
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Removes leading `www.` (if present) from the URL’ s domain.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### cutQueryString
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Removes query string, including the question mark.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### cutFragment
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Removes the fragment identifier, including the number sign.
2017-12-28 15:13:23 +00:00
2022-06-02 10:55:18 +00:00
### cutQueryStringAndFragment
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Removes the query string and fragment identifier, including the question mark and number sign.
2017-12-28 15:13:23 +00:00
2024-05-29 13:52:54 +00:00
### cutURLParameter(url, name)
2017-12-28 15:13:23 +00:00
2024-05-29 14:15:46 +00:00
Removes the `name` parameter from a URL, if present.
This function does not encode or decode characters in parameter names, e.g. `Client ID` and `Client%20ID` are treated as different parameter names.
2018-10-16 10:47:17 +00:00
2022-11-29 07:10:31 +00:00
**Syntax**
``` sql
2024-05-29 13:52:54 +00:00
cutURLParameter(url, name)
2022-11-29 07:10:31 +00:00
```
**Arguments**
2024-05-29 13:52:54 +00:00
- `url` — URL. [String ](../../sql-reference/data-types/string.md ).
2024-05-29 13:04:02 +00:00
- `name` — name of URL parameter. [String ](../../sql-reference/data-types/string.md ) or [Array ](../../sql-reference/data-types/array.md ) of Strings.
2022-11-29 07:10:31 +00:00
**Returned value**
2024-05-29 18:26:52 +00:00
- url with `name` URL parameter removed. [String ](../data-types/string.md ).
2022-11-29 07:10:31 +00:00
**Example**
Query:
``` sql
SELECT
2022-12-01 13:47:44 +00:00
cutURLParameter('http://bigmir.net/?a=b& c=d& e=f#g', 'a') as url_without_a,
cutURLParameter('http://bigmir.net/?a=b& c=d& e=f#g', ['c', 'e']) as url_without_c_and_e;
2022-11-29 07:10:31 +00:00
```
Result:
``` text
2022-12-01 13:47:44 +00:00
┌─url_without_a────────────────┬─url_without_c_and_e──────┐
│ http://bigmir.net/?c=d& e=f#g │ http://bigmir.net/?a=b#g │
└──────────────────────────────┴──────────────────────────┘
2022-11-29 07:10:31 +00:00
```