Functions for Working with URLs

Note

The functions mentioned in this section are optimized for maximum performance and for the most part do not follow the RFC-3986 standard. Functions which implement RFC-3986 have RFC appended to their function name and are generally slower.

You can generally use the non-RFC function variants when working with publicly registered domains that contain neither user strings nor @ symbols. The table below details which symbols in a URL can (✔) or cannot (✗) be parsed by the respective RFC and non-RFC variants:

Symbol	non-RFC	RFC
' '	✗	✗
\t	✗	✗
<	✗	✗
>	✗	✗
%	✗	✔*
{	✗	✗
}	✗	✗
|	✗	✗
\\	✗	✗
^	✗	✗
~	✗	✔*
[	✗	✗
]	✗	✔
;	✗	✔*
=	✗	✔*
&	✗	✔*

symbols marked * are sub-delimiters in RFC 3986 and allowed for user info following the @ symbol.

Functions that Extract Parts of a URL

If the relevant part isn't present in a URL, an empty string is returned.

protocol

Extracts the protocol from a URL.

Examples of typical returned values: http, https, ftp, mailto, tel, magnet.

domain

Extracts the hostname from a URL.

Syntax

Arguments

• url — URL. String.

The URL can be specified with or without a protocol. Examples:

For these examples, the domain function returns the following results:

Returned values

• Host name if the input string can be parsed as a URL, otherwise an empty string. String.

Example

domainRFC

Extracts the hostname from a URL. Similar to domain, but RFC 3986 conformant.

Syntax

Arguments

• url — URL. String.

Returned values

• Host name if the input string can be parsed as a URL, otherwise an empty string. String.

Example

domainWithoutWWW

Returns the domain without leading www. if present.

Syntax

Arguments

• url — URL. String.

Returned values

• Domain name if the input string can be parsed as a URL (without leading www.), otherwise an empty string. String.

Example

domainWithoutWWWRFC

Returns the domain without leading www. if present. Similar to domainWithoutWWW but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.

Returned values

• Domain name if the input string can be parsed as a URL (without leading www.), otherwise an empty string. String.

Example

Query:

Result:

topLevelDomain

Extracts the the top-level domain from a URL.

Arguments

• url — URL. String.

Note

The URL can be specified with or without a protocol. Examples:

Returned values

• Domain name if the input string can be parsed as a URL. Otherwise, an empty string. String.

Example

Query:

Result:

topLevelDomainRFC

Extracts the the top-level domain from a URL. Similar to topLevelDomain, but conforms to RFC 3986.

Arguments

• url — URL. String.

Note

The URL can be specified with or without a protocol. Examples:

Returned values

• Domain name if the input string can be parsed as a URL. Otherwise, an empty string. String.

Example

Query:

Result:

firstSignificantSubdomain

Returns the "first significant subdomain". The first significant subdomain is a second-level domain for com, net, org, or co, otherwise it is a third-level domain. For example, firstSignificantSubdomain ('https://news.clickhouse.com/') = 'clickhouse', firstSignificantSubdomain ('https://news.clickhouse.com.tr/') = 'clickhouse'. The list of "insignificant" second-level domains and other implementation details may change in the future.

Syntax

Arguments

• url — URL. String.

Returned value

• The first significant subdomain. String.

Example

Query:

Result:

firstSignificantSubdomainRFC

Returns the "first significant subdomain". The first significant subdomain is a second-level domain for com, net, org, or co, otherwise it is a third-level domain. For example, firstSignificantSubdomain ('https://news.clickhouse.com/') = 'clickhouse', firstSignificantSubdomain ('https://news.clickhouse.com.tr/') = 'clickhouse'. The list of "insignificant" second-level domains and other implementation details may change in the future. Similar to firstSignficantSubdomain but conforms to RFC 1034.

Syntax

Arguments

• url — URL. String.

Returned value

• The first significant subdomain. String.

Example

Query:

Result:

cutToFirstSignificantSubdomain

Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain".

Syntax

Arguments

• url — URL. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query:

Result:

cutToFirstSignificantSubdomainRFC

Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain". Similar to cutToFirstSignificantSubdomain but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query:

Result:

cutToFirstSignificantSubdomainWithWWW

Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain", without stripping www.

Syntax

Arguments

• url — URL. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain (with www) if possible, otherwise returns an empty string. String.

Example

Query:

Result:

cutToFirstSignificantSubdomainWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain", without stripping www. Similar to cutToFirstSignificantSubdomainWithWWW but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain (with "www") if possible, otherwise returns an empty string. String.

Example

Query:

Result:

cutToFirstSignificantSubdomainCustom

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name. This function can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

Example

Query:

Result:

See Also

• firstSignificantSubdomain.

cutToFirstSignificantSubdomainCustomRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name. This function can be useful if you need a fresh TLD list or if you have a custom list. Similar to cutToFirstSignificantSubdomainCustom but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

See Also

• firstSignificantSubdomain.

cutToFirstSignificantSubdomainCustomWithWWW

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name. It can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. String.

Example

Query:

Result:

See Also

• firstSignificantSubdomain.
• top_level_domains_list

cutToFirstSignificantSubdomainCustomWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name. It can be useful if you need a fresh TLD list or if you have a custom list. Similar to cutToFirstSignificantSubdomainCustomWithWWW but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. String.

See Also

• firstSignificantSubdomain.
• top_level_domains_list

firstSignificantSubdomainCustom

Returns the first significant subdomain. Accepts customs TLD list name. Can be useful if you need fresh TLD list or you have custom.

Configuration example:

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• First significant subdomain. String.

Example

Query:

Result:

See Also

• firstSignificantSubdomain.
• top_level_domains_list

firstSignificantSubdomainCustomRFC

Returns the first significant subdomain. Accepts customs TLD list name. Can be useful if you need fresh TLD list or you have custom. Similar to firstSignificantSubdomainCustom but conforms to RFC 3986.

Syntax

Arguments

• url — URL. String.
• tld — Custom TLD list name. String.

Returned value

• First significant subdomain. String.

See Also

• firstSignificantSubdomain.
• top_level_domains_list

port

Returns the port or default_port if the URL contains no port or cannot be parsed.

Syntax

Arguments

• url — URL. String.
• default_port — The default port number to be returned. UInt16.

Returned value

• Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query:

Result:

portRFC

Returns the port or default_port if the URL contains no port or cannot be parsed. Similar to port, but RFC 3986 conformant.

Syntax

Arguments

• url — URL. String.
• default_port — The default port number to be returned. UInt16.

Returned value

• Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query:

Result:

path

Returns the path without query string.

Example: /top/news.html.

pathFull

The same as above, but including query string and fragment.

Example: /top/news.html?page=2#comments.

protocol

Extracts the protocol from a URL.

Syntax

Arguments

• url — URL to extract protocol from. String.

Returned value

• Protocol, or an empty string if it cannot be determined. String.

Example

Query:

Result:

queryString

Returns the query string without the initial question mark, # and everything after #.

Example: page=1&lr=213.

fragment

Returns the fragment identifier without the initial hash symbol.

queryStringAndFragment

Returns the query string and fragment identifier.

Example: page=1#29390.

extractURLParameter(url, name)

Returns the value of the name parameter in the URL, if present, otherwise an empty string is returned. If there are multiple parameters with this name, the first occurrence is returned. The function assumes that the parameter in the url parameter is encoded in the same way as in the name argument.

extractURLParameters(url)

Returns an array of name=value strings corresponding to the URL parameters. The values are not decoded.

extractURLParameterNames(url)

Returns an array of name strings corresponding to the names of URL parameters. The values are not decoded.

URLHierarchy(url)

Returns an array containing the URL, truncated at the end by the symbols /,? in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters.

URLPathHierarchy(url)

The same as above, but without the protocol and host in the result. The / element (root) is not included.

encodeURLComponent(url)

Returns the encoded URL.

Example:

decodeURLComponent(url)

Returns the decoded URL.

Example:

encodeURLFormComponent(url)

Returns the encoded URL. Follows rfc-1866, space( ) is encoded as plus(+).

Example:

decodeURLFormComponent(url)

Returns the decoded URL. Follows rfc-1866, plain plus(+) is decoded as space( ).

Example:

netloc

Extracts network locality (username:password@host:port) from a URL.

Syntax

Arguments

• url — URL. String.

Returned value

• username:password@host:port. String.

Example

Query:

Result:

Functions that remove part of a URL

If the URL does not have anything similar, the URL remains unchanged.

cutWWW

Removes leading www. (if present) from the URL's domain.

cutQueryString

Removes query string, including the question mark.

cutFragment

Removes the fragment identifier, including the number sign.

cutQueryStringAndFragment

Removes the query string and fragment identifier, including the question mark and number sign.

cutURLParameter(url, name)

Removes the name parameter from a URL, if present. This function does not encode or decode characters in parameter names, e.g. Client ID and Client%20ID are treated as different parameter names.

Syntax

Arguments

• url — URL. String.
• name — name of URL parameter. String or Array of Strings.

Returned value

• url with name URL parameter removed. String.

Example

Query:

Result: