mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-16 20:53:27 +00:00

2024-05-29 13:56:16 +00:00

32 KiB

Raw Blame History

slug	sidebar_position	sidebar_label
/en/sql-reference/functions/url-functions	200	URLs

Functions for Working with URLs

:::note The functions mentioned in this section are optimized for maximum performance and for the most part do not follow the RFC-3986 standard. Functions which implement RFC-3986 have RFC appended to their function name and are generally slower. :::

You can generally use the non-RFC function variants when working with publicly registered domains that contain neither user strings nor @ symbols. Below table below details which symbols in an URL can (✔) or cannot (✗) be parsed by the respective RFC and non-RFC variants:

Symbol	non-`RFC`	`RFC`
' '	✗	✗
\t	✗	✗
<	✗	✗
>	✗	✗
%	✗	✔*
{	✗	✗
}	✗	✗
\|	✗	✗
\\	✗	✗
^	✗	✗
~	✗	✔*
[	✗	✗
]	✗	✔
;	✗	✔*
=	✗	✔*
&	✗	✔*

symbols marked * are sub-delimiters in RFC 3986 and allowed for user info following the @ symbol.

Functions that Extract Parts of a URL

If the relevant part isn’t present in a URL, an empty string is returned.

protocol

Extracts the protocol from a URL.

Examples of typical returned values: http, https, ftp, mailto, tel, magnet.

domain

Extracts the hostname from a URL.

Syntax

domain(url)

Arguments

url — URL. Type: String.

The URL can be specified with or without a protocol. Examples:

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

For these examples, the domain function returns the following results:

some.svn-hosting.com
some.svn-hosting.com
clickhouse.com

Returned values

Host name if ClickHouse can parse the input string as a URL, otherwise an empty string. String.

Example

SELECT domain('svn+ssh://some.svn-hosting.com:80/repo/trunk');

┌─domain('svn+ssh://some.svn-hosting.com:80/repo/trunk')─┐
│ some.svn-hosting.com                                   │
└────────────────────────────────────────────────────────┘

domainRFC

Extracts the hostname from a URL. Similar to domain, but RFC 3986 conformant.

Syntax

domainRFC(url)

Arguments

url — URL. Type: String.

Returned values

Host name if ClickHouse can parse the input string as a URL, otherwise an empty string. String.

Type: String.

Example

SELECT
    domain('http://user:password@example.com:8080/path?query=value#fragment'),
    domainRFC('http://user:password@example.com:8080/path?query=value#fragment');

┌─domain('http://user:password@example.com:8080/path?query=value#fragment')─┬─domainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│                                                                           │ example.com                                                                  │
└───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────┘

domainWithoutWWW

Returns the domain and removes no more than one ‘www.’ from the beginning of it, if present.

topLevelDomain

Extracts the the top-level domain from a URL.

topLevelDomain(url)

Arguments

url — URL. String.

:::note The URL can be specified with or without a protocol. Examples:

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

:::

Returned values

Domain name if ClickHouse can parse the input string as a URL. Otherwise, an empty string. String.

Example

Query:

SELECT topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk');

Result:

┌─topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk')─┐
│ com                                                                │
└────────────────────────────────────────────────────────────────────┘

topLevelDomainRFC

Extracts the the top-level domain from a URL. It is similar to topLevelDomain, but conforms to RFC 3986.

topLevelDomainRFC(url)

Arguments

url — URL. String.

:::note The URL can be specified with or without a protocol. Examples:

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

:::

Returned values

Domain name if ClickHouse can parse the input string as a URL. Otherwise, an empty string. String.

Example

Query:

SELECT topLevelDomain('http://foo:foo%41bar@foo.com'), topLevelDomainRFC('http://foo:foo%41bar@foo.com');

Result:

┌─topLevelDomain('http://foo:foo%41bar@foo.com')─┬─topLevelDomainRFC('http://foo:foo%41bar@foo.com')─┐
│                                                │ com                                               │
└────────────────────────────────────────────────┴───────────────────────────────────────────────────┘

firstSignificantSubdomain

Returns the “first significant subdomain”. The first significant subdomain is a second-level domain if it is ‘com’, ‘net’, ‘org’, or ‘co’. Otherwise, it is a third-level domain. For example, firstSignificantSubdomain (‘https://news.clickhouse.com/’) = ‘clickhouse’, firstSignificantSubdomain (‘https://news.clickhouse.com.tr/’) = ‘clickhouse’. The list of “insignificant” second-level domains and other implementation details may change in the future.

Syntax

firstSignificantSubdomain(URL)

Arguments

url — URL. String.

Returned value

The first significant subdomain. String.

Example

Query:

SELECT firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')

Result:

┌─firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')─┐
│ example                                                       │
└───────────────────────────────────────────────────────────────┘

firstSignificantSubdomainRFC

Returns the “first significant subdomain”. Similar to firstSignficantSubdomain but according to RFC 1034.

Syntax

firstSignificantSubdomainRFC(url)

Arguments

url — URL. String.

Returned value

The first significant subdomain. String.

Example

Query:

SELECT
    firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment'),
    firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment');

Result:

┌─firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment')─┬─firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│                                                                                              │ example                                                                                         │
└──────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomain

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”.

Syntax

cutToFirstSignificantSubdomain(url)

Arguments

url — URL. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query:

SELECT
    cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/'),
    cutToFirstSignificantSubdomain('www.tr'),
    cutToFirstSignificantSubdomain('tr');

Result:

┌─cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomain('www.tr')─┬─cutToFirstSignificantSubdomain('tr')─┐
│ clickhouse.com.tr                                                 │ tr                                       │                                      │
└───────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┴──────────────────────────────────────┘

cutToFirstSignificantSubdomainRFC

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”. It is similar to cutToFirstSignificantSubdomain but follows stricter rules to be compatible with RFC 3986 and is less performant.

Syntax

cutToFirstSignificantSubdomainRFC(url)

Arguments

url — URL. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query:

SELECT
    cutToFirstSignificantSubdomain('http://user:password@example.com:8080'),
    cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080');

Result:

┌─cutToFirstSignificantSubdomain('http://user:password@example.com:8080')─┬─cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080')─┐
│                                                                         │ example.com                                                                │
└─────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomainWithWWW

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”, without stripping "www".

Syntax

cutToFirstSignificantSubdomainWithWWW(url)

Arguments

url — URL. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain (with "www") if possible, otherwise returns an empty string. String.

Example

Query:

SELECT
    cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/'),
    cutToFirstSignificantSubdomainWithWWW('www.tr'),
    cutToFirstSignificantSubdomainWithWWW('tr');

Result:

┌─cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomainWithWWW('www.tr')─┬─cutToFirstSignificantSubdomainWithWWW('tr')─┐
│ clickhouse.com.tr                                                        │ www.tr                                          │                                             │
└──────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┴─────────────────────────────────────────────┘

cutToFirstSignificantSubdomainWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”, without stripping "www". Similar to cutToFirstSignificantSubdomainWithWWW but follows stricter rules to be compatible with RFC 3986 and is less performant.

Syntax

cutToFirstSignificantSubdomainWithWWW(url)

Arguments

url — URL. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain (with "www") if possible, otherwise returns an empty string. String.

Example

Query:

SELECT
    cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy'),
    cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy');

Result:

┌─cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┬─cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┐
│                                                                                       │ mail.ru                                                                                  │
└───────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomainCustom

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name.

This can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

cutToFirstSignificantSubdomain(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

Example

Query:

SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result:

┌─cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo.there-is-no-such-domain                                                                   │
└───────────────────────────────────────────────────────────────────────────────────────────────┘

See Also

firstSignificantSubdomain.

cutToFirstSignificantSubdomainCustomRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name. It is similar to cutToFirstSignificantSubdomainCustom but follows stricter rules according to RFC 3986 and is generally less performant as a result.

Syntax

cutToFirstSignificantSubdomainRFC(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

See Also

firstSignificantSubdomain.

cutToFirstSignificantSubdomainCustomWithWWW

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name.

It can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

cutToFirstSignificantSubdomainCustomWithWWW(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www.

Type: String.

Example

Query:

SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list');

Result:

┌─cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐
│ www.foo                                                                      │
└──────────────────────────────────────────────────────────────────────────────┘

See Also

firstSignificantSubdomain.

cutToFirstSignificantSubdomainCustomWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name. It is similar to cutToFirstSignificantSubdomainCustomWithWWW but follows stricter rules according to RFC 3986 and is generally less performant as a result.

Syntax

cutToFirstSignificantSubdomainCustomWithWWWRFC(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. String.

See Also

firstSignificantSubdomain.

firstSignificantSubdomainCustom

Returns the first significant subdomain. Accepts customs TLD list name.

Can be useful if you need fresh TLD list or you have custom.

Configuration example:

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax

firstSignificantSubdomainCustom(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

First significant subdomain.

Type: String.

Example

Query:

SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result:

┌─firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────────┘

See Also

firstSignificantSubdomain.

firstSignificantSubdomainCustomRFC

Returns the first significant subdomain. Accepts customs TLD list name. Similar to firstSignificantSubdomainCustom but follows stricter rules according to RFC 3986 and is generally less performant as a result.

Syntax

firstSignificantSubdomainCustomRFC(url, tld)

Arguments

url — URL. String.
tld — Custom TLD list name. String.

Returned value

First significant subdomain.

Type: String.

See Also

firstSignificantSubdomain.

port

Returns the port or default_port if there is no port in the url (or in case of validation error).

Syntax

port(url [, default_port = 0])

Arguments

url — URL. String.
default_port — The default port number to be returned. UInt16.

Returned value

Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query:

SELECT port('http://paul@www.example.com:80/');

Result:

┌─port('http://paul@www.example.com:80/')─┐
│                                      80 │
└─────────────────────────────────────────┘

portRFC

Returns the port or default_port if there is no port in the url (or in case of validation error). Similar to port, but RFC 3986 conformant.

Syntax

portRFC(url [, default_port = 0])

Arguments

url — URL. String.
default_port — The default port number to be returned. UInt16.

Returned value

Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query:

SELECT
    port('http://user:password@example.com:8080'),
    portRFC('http://user:password@example.com:8080');

Result:

┌─port('http://user:password@example.com:8080')─┬─portRFC('http://user:password@example.com:8080')─┐
│                                             0 │                                             8080 │
└───────────────────────────────────────────────┴──────────────────────────────────────────────────┘

path

Returns the path. Example: /top/news.html The path does not include the query string.

pathFull

The same as above, but including query string and fragment. Example: /top/news.html?page=2#comments

queryString

Returns the query string. Example: page=1&lr=213. query-string does not include the initial question mark, as well as # and everything after #.

fragment

Returns the fragment identifier. fragment does not include the initial hash symbol.

queryStringAndFragment

Returns the query string and fragment identifier. Example: page=1#29390.

extractURLParameter(url, name)

Returns the value of the ‘name’ parameter in the URL, if present. Otherwise, an empty string. If there are many parameters with this name, it returns the first occurrence. This function works under the assumption that the parameter name is encoded in the URL exactly the same way as in the passed argument.

extractURLParameters(url)

Returns an array of name=value strings corresponding to the URL parameters. The values are not decoded in any way.

extractURLParameterNames(url)

Returns an array of name strings corresponding to the names of URL parameters. The values are not decoded in any way.

URLHierarchy(url)

Returns an array containing the URL, truncated at the end by the symbols /,? in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters.

URLPathHierarchy(url)

The same as above, but without the protocol and host in the result. The / element (root) is not included.

URLPathHierarchy('https://example.com/browse/CONV-6788') =
[
    '/browse/',
    '/browse/CONV-6788'
]

encodeURLComponent(url)

Returns the encoded URL. Example:

SELECT encodeURLComponent('http://127.0.0.1:8123/?query=SELECT 1;') AS EncodedURL;

┌─EncodedURL───────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT%201%3B │
└──────────────────────────────────────────────────────────┘

decodeURLComponent(url)

Returns the decoded URL. Example:

SELECT decodeURLComponent('http://127.0.0.1:8123/?query=SELECT%201%3B') AS DecodedURL;

┌─DecodedURL─────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1; │
└────────────────────────────────────────┘

encodeURLFormComponent(url)

Returns the encoded URL. Follows rfc-1866, space( ) is encoded as plus(+). Example:

SELECT encodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT 1 2+3') AS EncodedURL;

┌─EncodedURL────────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT+1+2%2B3 │
└───────────────────────────────────────────────────────────┘

decodeURLFormComponent(url)

Returns the decoded URL. Follows rfc-1866, plain plus(+) is decoded as space( ). Example:

SELECT decodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT%201+2%2B3') AS DecodedURL;

┌─DecodedURL────────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1 2+3 │
└───────────────────────────────────────────┘

netloc

Extracts network locality (username:password@host:port) from a URL.

Syntax

netloc(url)

Arguments

url — URL. String.

Returned value

username:password@host:port.

Type: String.

Example

Query:

SELECT netloc('http://paul@www.example.com:80/');

Result:

┌─netloc('http://paul@www.example.com:80/')─┐
│ paul@www.example.com:80                   │
└───────────────────────────────────────────┘

Functions that Remove Part of a URL

If the URL does not have anything similar, the URL remains unchanged.

cutWWW

Removes no more than one ‘www.’ from the beginning of the URL’s domain, if present.

cutQueryString

Removes query string. The question mark is also removed.

cutFragment

Removes the fragment identifier. The number sign is also removed.

cutQueryStringAndFragment

Removes the query string and fragment identifier. The question mark and number sign are also removed.

cutURLParameter(url, name)

Removes the name parameter from URL, if present. This function does not encode or decode characters in parameter names, e.g. Client ID and Client%20ID are treated as different parameter names.

Syntax

cutURLParameter(url, name)

Arguments

url — URL. String.
name — name of URL parameter. String or Array of Strings.

Returned value

url with name URL parameter removed.

Type: String.

Example

Query:

SELECT
    cutURLParameter('http://bigmir.net/?a=b&c=d&e=f#g', 'a') as url_without_a,
    cutURLParameter('http://bigmir.net/?a=b&c=d&e=f#g', ['c', 'e']) as url_without_c_and_e;

Result:

┌─url_without_a────────────────┬─url_without_c_and_e──────┐
│ http://bigmir.net/?c=d&e=f#g │ http://bigmir.net/?a=b#g │
└──────────────────────────────┴──────────────────────────┘

32 KiB Raw Blame History Unescape Escape

Functions for Working with URLs

Functions that Extract Parts of a URL

protocol

domain

domainRFC

domainWithoutWWW

topLevelDomain

topLevelDomainRFC

firstSignificantSubdomain

firstSignificantSubdomainRFC

cutToFirstSignificantSubdomain

cutToFirstSignificantSubdomainRFC

cutToFirstSignificantSubdomainWithWWW

cutToFirstSignificantSubdomainWithWWWRFC

cutToFirstSignificantSubdomainCustom

cutToFirstSignificantSubdomainCustomRFC

cutToFirstSignificantSubdomainCustomWithWWW

cutToFirstSignificantSubdomainCustomWithWWWRFC

firstSignificantSubdomainCustom

firstSignificantSubdomainCustomRFC

port

portRFC

path

pathFull

queryString

fragment

queryStringAndFragment

extractURLParameter(url, name)

extractURLParameters(url)

extractURLParameterNames(url)

URLHierarchy(url)

URLPathHierarchy(url)

encodeURLComponent(url)

decodeURLComponent(url)

encodeURLFormComponent(url)

decodeURLFormComponent(url)

netloc

Functions that Remove Part of a URL

cutWWW

cutQueryString

cutFragment

cutQueryStringAndFragment

cutURLParameter(url, name)

32 KiB

Raw Blame History