ClickHouse/docs/en/sql-reference/functions/index.md

150 lines
9.5 KiB
Markdown
Raw Normal View History

2020-04-03 13:23:32 +00:00
---
toc_folder_title: Functions
toc_priority: 32
toc_title: Introduction
---
2020-03-20 10:10:48 +00:00
# Functions {#functions}
2017-04-03 19:49:50 +00:00
2021-05-27 19:44:11 +00:00
There are at least\* two types of functions - regular functions (they are just called “functions”) and aggregate functions. These are completely different concepts. Regular functions work as if they are applied to each row separately (for each row, the result of the function does not depend on the other rows). Aggregate functions accumulate a set of values from various rows (i.e. they depend on the entire set of rows).
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
In this section we discuss regular functions. For aggregate functions, see the section “Aggregate functions”.
CLICKHOUSE-2720: progress on website and reference (#886) * update presentations * CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com * update submodule * lost files * CLICKHOUSE-2981: prefer sphinx docs over original reference * CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links * update presentations * Less confusing directory structure (docs -> doc/reference/) * Minify sphinx docs too * Website release script: fail fast + pass docker hash on deploy * Do not underline links in docs * shorter * cleanup docker images * tune nginx config * CLICKHOUSE-3043: get rid of habrastorage links * Lost translation * CLICKHOUSE-2936: temporary client-side redirect * behaves weird in test * put redirect back * CLICKHOUSE-3047: copy docs txts to public too * move to proper file * remove old pages to avoid confusion * Remove reference redirect warning for now * Refresh README.md * Yellow buttons in docs * Use svg flags instead of unicode ones in docs * fix test website instance * Put flags to separate files * wrong flag * Copy Yandex.Metrica introduction from main page to docs * Yet another home page structure change, couple new blocks (CLICKHOUSE-3045) * Update Contacts section * CLICKHOUSE-2849: more detailed legal information * CLICKHOUSE-2978 preparation - split by files * More changes in Contacts block * Tune texts on index page * update presentations * One more benchmark * Add usage sections to index page, adapted from slides * Get the roadmap started, based on slides from last ClickHouse Meetup * CLICKHOUSE-2977: some rendering tuning * Get rid of excessive section in the end of getting started * Make headers linkable * CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849 * CLICKHOUSE-2981: fix mobile styles in docs * Ban crawling of duplicating docs * Open some external links in new tab * Ban old docs too * Lots of trivial fixes in english docs * Lots of trivial fixes in russian docs * Remove getting started copies in markdown * Add Yandex.Webmaster * Fix some sphinx warnings * More warnings fixed in english docs * More sphinx warnings fixed * Add code-block:: text * More code-block:: text * These headers look not that well * Better switch between documentation languages * merge use_case.rst into ya_metrika_task.rst * Edit the agg_functions.rst texts * Add lost empty lines * Lost blank lines * Add new logo sizes * update presentations * Next step in migrating to new documentation * Fix all warnings in en reference * Fix all warnings in ru reference * Re-arrange existing reference * Move operation tips to main reference * Fix typos noticed by milovidov@ * Get rid of zookeeper.md * Looks like duplicate of tutorial.html * Fix some mess with html tags in tutorial * No idea why nobody noticed this before, but it was completely not clear whet to get the data * Match code block styling between main and tutorial pages (in favor of the latter) * Get rid of some copypaste in tutorial * Normalize header styles * Move example_datasets to sphinx * Move presentations submodule to website * Move and update README.md * No point in duplicating articles from habrahabr here * Move development-related docs as is for now * doc/reference/ -> docs/ (to match the URL on website) * Adapt links to match the previous commit * Adapt development docs to rst (still lacks translation and strikethrough support) * clean on release * blacklist presentations in gulp * strikethrough support in sphinx * just copy development folder for now * fix weird introduction in style article * Style guide translation (WIP) * Finish style guide translation to English * gulp clean separately * Update year in LICENSE * Initial CONTRIBUTING.md * Fix remaining links to old docs in tutorial * Some tutorial fixes * Typo * Another typo * Update list of authors from yandex-team accoding to git log
2017-06-20 14:19:03 +00:00
2020-03-20 10:10:48 +00:00
\* - There is a third type of function that the arrayJoin function belongs to; table functions can also be mentioned separately.\*
2017-04-03 19:49:50 +00:00
## Strong Typing {#strong-typing}
2017-04-03 19:49:50 +00:00
2021-05-27 19:44:11 +00:00
In contrast to standard SQL, ClickHouse has strong typing. In other words, it does not make implicit conversions between types. Each function works for a specific set of types. This means that sometimes you need to use type conversion functions.
2017-04-03 19:49:50 +00:00
## Common Subexpression Elimination {#common-subexpression-elimination}
2017-04-26 19:16:38 +00:00
All expressions in a query that have the same AST (the same record or same result of syntactic parsing) are considered to have identical values. Such expressions are concatenated and executed once. Identical subqueries are also eliminated this way.
2017-04-03 19:49:50 +00:00
## Types of Results {#types-of-results}
2017-04-26 19:16:38 +00:00
All functions return a single return as the result (not several values, and not zero values). The type of result is usually defined only by the types of arguments, not by the values. Exceptions are the tupleElement function (the a.N operator), and the toFixedString function.
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
## Constants {#constants}
2017-04-26 19:16:38 +00:00
For simplicity, certain functions can only work with constants for some arguments. For example, the right argument of the LIKE operator must be a constant.
Almost all functions return a constant for constant arguments. The exception is functions that generate random numbers.
2020-03-20 10:10:48 +00:00
The now function returns different values for queries that were run at different times, but the result is considered a constant, since constancy is only important within a single query.
2017-04-26 19:16:38 +00:00
A constant expression is also considered a constant (for example, the right half of the LIKE operator can be constructed from multiple constants).
2017-04-03 19:49:50 +00:00
2017-04-26 19:16:38 +00:00
Functions can be implemented in different ways for constant and non-constant arguments (different code is executed). But the results for a constant and for a true column containing only the same value should match each other.
2017-04-03 19:49:50 +00:00
## NULL Processing {#null-processing}
Functions have the following behaviors:
- If at least one of the arguments of the function is `NULL`, the function result is also `NULL`.
- Special behavior that is specified individually in the description of each function. In the ClickHouse source code, these functions have `UseDefaultImplementationForNulls=false`.
2020-03-20 10:10:48 +00:00
## Constancy {#constancy}
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
Functions cant change the values of their arguments any changes are returned as the result. Thus, the result of calculating separate functions does not depend on the order in which the functions are written in the query.
2017-04-03 19:49:50 +00:00
## Higher-order functions, `->` operator and lambda(params, expr) function {#higher-order-functions}
Higher-order functions can only accept lambda functions as their functional argument. To pass a lambda function to a higher-order function use `->` operator. The left side of the arrow has a formal parameter, which is any ID, or multiple formal parameters any IDs in a tuple. The right side of the arrow has an expression that can use these formal parameters, as well as any table columns.
2021-07-29 15:20:55 +00:00
Examples:
```
x -> 2 * x
str -> str != Referer
```
A lambda function that accepts multiple arguments can also be passed to a higher-order function. In this case, the higher-order function is passed several arrays of identical length that these arguments will correspond to.
For some functions the first argument (the lambda function) can be omitted. In this case, identical mapping is assumed.
2021-09-26 19:44:53 +00:00
## SQL User Defined Functions {#user-defined-functions}
2021-09-24 11:44:29 +00:00
Custom functions from lambda expressions can be created using the [CREATE FUNCTION](../statements/create/function.md) statement. To delete these functions use the [DROP FUNCTION](../statements/drop.md#drop-function) statement.
2021-09-26 19:44:53 +00:00
## Executable User Defined Functions {#executable-user-defined-functions}
2021-09-27 20:15:15 +00:00
ClickHouse can call any external executable program or script to process data. Describe such functions in a [configuration file](../../operations/configuration-files.md) and add the path of that file to the main configuration in `user_defined_executable_functions_config` setting. If a wildcard symbol `*` is used in the path, then all files matching the pattern are loaded. Example:
2021-09-24 11:44:29 +00:00
``` xml
<user_defined_executable_functions_config>*_function.xml</user_defined_executable_functions_config>
```
2021-09-27 20:15:15 +00:00
User defined function configurations are searched relative to the path specified in the `user_files_path` setting.
2021-09-24 11:44:29 +00:00
2021-09-26 19:44:53 +00:00
A function configuration contains the following settings:
2021-09-24 11:44:29 +00:00
- `name` - a function name.
- `command` - a command or a script to execute.
2021-09-26 19:44:53 +00:00
- `argument` - argument description with the `type` of an argument. Each argument is described in a separate setting.
- `format` - a [format](../../interfaces/formats.md) in which arguments are passed to the command.
- `return_type` - the type of a returned value.
- `type` - an executable type. If `type` is set to `executable` then single command is started. If it is set to `executable_pool` then a pool of commands is created.
2021-09-27 20:32:38 +00:00
- `max_command_execution_time` - maximum execution time in seconds for processing block of data. This setting is valid for `executable_pool` commands only. Optional. Default value is `10`.
- `command_termination_timeout` - time in seconds during which a command should finish after its pipe is closed. After that time `SIGTERM` is sent to the process executing the command. This setting is valid for `executable_pool` commands only. Optional. Default value is `10`.
2021-09-27 20:15:15 +00:00
- `pool_size` - the size of a command pool. Optional. Default value is `16`.
- `lifetime` - the reload interval of a function in seconds. If it is set to `0` then the function is not reloaded.
- `send_chunk_header` - controls whether to send row count before sending a chunk of data to process. Optional. Default value is `false`.
2021-09-26 19:44:53 +00:00
The command must read arguments from `STDIN` and must output the result to `STDOUT`. The command must process arguments iteratively. That is after processing a chunk of arguments it must wait for the next chunk.
2021-09-24 11:44:29 +00:00
**Example**
Creating `test_function` using XML configuration:
2021-09-24 11:44:29 +00:00
```
<functions>
<function>
2021-09-27 20:15:15 +00:00
<type>executable</type>
<name>test_function</name>
<return_type>UInt64</return_type>
<argument>
<type>UInt64</type>
</argument>
2021-09-24 11:44:29 +00:00
<argument>
2021-09-27 20:15:15 +00:00
<type>UInt64</type>
2021-09-24 11:44:29 +00:00
</argument>
<format>TabSeparated</format>
2021-09-27 20:15:15 +00:00
<command>cd /; clickhouse-local --input-format TabSeparated --output-format TabSeparated --structure 'x UInt64, y UInt64' --query "SELECT x + y FROM table"</command>
2021-09-24 11:44:29 +00:00
<lifetime>0</lifetime>
</function>
</functions>
```
Query:
``` sql
2021-09-27 20:15:15 +00:00
SELECT test_function(toUInt64(2), toUInt64(2));
2021-09-24 11:44:29 +00:00
```
Result:
``` text
2021-09-27 20:15:15 +00:00
┌─test_function(toUInt64(2), toUInt64(2))─┐
│ 4 │
└─────────────────────────────────────────┘
2021-09-24 11:44:29 +00:00
```
## Error Handling {#error-handling}
2017-04-03 19:49:50 +00:00
2017-04-26 19:16:38 +00:00
Some functions might throw an exception if the data is invalid. In this case, the query is canceled and an error text is returned to the client. For distributed processing, when an exception occurs on one of the servers, the other servers also attempt to abort the query.
2017-04-03 19:49:50 +00:00
## Evaluation of Argument Expressions {#evaluation-of-argument-expressions}
2017-04-03 19:49:50 +00:00
In almost all programming languages, one of the arguments might not be evaluated for certain operators. This is usually the operators `&&`, `||`, and `?:`.
2017-04-26 19:16:38 +00:00
But in ClickHouse, arguments of functions (operators) are always evaluated. This is because entire parts of columns are evaluated at once, instead of calculating each row separately.
2017-04-03 19:49:50 +00:00
## Performing Functions for Distributed Query Processing {#performing-functions-for-distributed-query-processing}
2017-04-03 19:49:50 +00:00
2017-04-26 19:16:38 +00:00
For distributed query processing, as many stages of query processing as possible are performed on remote servers, and the rest of the stages (merging intermediate results and everything after that) are performed on the requestor server.
2017-04-03 19:49:50 +00:00
2017-04-26 19:16:38 +00:00
This means that functions can be performed on different servers.
For example, in the query `SELECT f(sum(g(x))) FROM distributed_table GROUP BY h(y),`
- if a `distributed_table` has at least two shards, the functions g and h are performed on remote servers, and the function f is performed on the requestor server.
- if a `distributed_table` has only one shard, all the f, g, and h functions are performed on this shards server.
2017-04-03 19:49:50 +00:00
2021-05-27 19:44:11 +00:00
The result of a function usually does not depend on which server it is performed on. However, sometimes this is important.
2017-04-26 19:16:38 +00:00
For example, functions that work with dictionaries use the dictionary that exists on the server they are running on.
Another example is the `hostName` function, which returns the name of the server it is running on in order to make `GROUP BY` by servers in a `SELECT` query.
2020-03-20 10:10:48 +00:00
If a function in a query is performed on the requestor server, but you need to perform it on remote servers, you can wrap it in an any aggregate function or add it to a key in `GROUP BY`.