thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-20 00:30:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Sources for english documentation switched to Markdown.

Browse Source 
Edit page link is fixed too for both language versions of documentation.
This commit is contained in:
BayoNet 2017-12-28 18:13:23 +03:00
parent 6d95729960
commit 13d9a4eebe
284 changed files with 11680 additions and 9831 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/_static/custom.js vendored
View File

@ -4,7 +4,7 @@ $(function() {
var pathname = window.location.pathname;
var url;
if (pathname.indexOf('html') >= 0) {
url = pathname.replace('/docs/', 'https://github.com/yandex/ClickHouse/edit/master/docs/').replace('html', 'rst');
url = pathname.replace('/docs/', 'https://github.com/yandex/ClickHouse/edit/master/docs/').replace('html', 'md');
} else {
if (pathname.indexOf('/single/') >= 0) {
if (pathname.indexOf('ru') >= 0) {
@ -14,9 +14,9 @@ $(function() {
}
} else {
if (pathname.indexOf('ru') >= 0) {
url = 'https://github.com/yandex/ClickHouse/edit/master/docs/ru/index.rst';
url = 'https://github.com/yandex/ClickHouse/edit/master/docs/ru/index.md';
} else {
url = 'https://github.com/yandex/ClickHouse/edit/master/docs/en/index.rst';
url = 'https://github.com/yandex/ClickHouse/edit/master/docs/en/index.md';
}
}
}

40
docs/en/agg_functions/combinators.md Normal file
View File

@ -0,0 +1,40 @@
<a name="aggregate_functions_combinators"></a>
# Aggregate function combinators
The name of an aggregate function can have a suffix appended to it. This changes the way the aggregate function works.
## -If
The suffix -If can be appended to the name of any aggregate function. In this case, the aggregate function accepts an extra argument – a condition (Uint8 type). The aggregate function processes only the rows that trigger the condition. If the condition was not triggered even once, it returns a default value (usually zeros or empty strings).
Examples: `sumIf(column, cond)`, `countIf(cond)`, `avgIf(x, cond)`, `quantilesTimingIf(level1, level2)(x, cond)`, `argMinIf(arg, val, cond)` and so on.
With conditional aggregate functions, you can calculate aggregates for several conditions at once, without using subqueries and `JOIN`s. For example, in Yandex.Metrica, conditional aggregate functions are used to implement the segment comparison functionality.
## -Array
The -Array suffix can be appended to any aggregate function. In this case, the aggregate function takes arguments of the 'Array(T)' type (arrays) instead of 'T' type arguments. If the aggregate function accepts multiple arguments, this must be arrays of equal lengths. When processing arrays, the aggregate function works like the original aggregate function across all array elements.
Example 1: `sumArray(arr)` - Totals all the elements of all 'arr' arrays. In this example, it could have been written more simply: `sum(arraySum(arr))`.
Example 2: `uniqArray(arr)` – Count the number of unique elements in all 'arr' arrays. This could be done an easier way: `uniq(arrayJoin(arr))`, but it's not always possible to add 'arrayJoin' to a query.
-If and -Array can be combined. However, 'Array' must come first, then 'If'. Examples: `uniqArrayIf(arr, cond)`, `quantilesTimingArrayIf(level1, level2)(arr, cond)`. Due to this order, the 'cond' argument can't be an array.
## -State
If you apply this combinator, the aggregate function doesn't return the resulting value (such as the number of unique values for the 'uniq' function), but an intermediate state of the aggregation (for ` uniq`, this is the hash table for calculating the number of unique values). This is an AggregateFunction(...) that can be used for further processing or stored in a table to finish aggregating later. See the sections "AggregatingMergeTree" and "Functions for working with intermediate aggregation states".
## -Merge
If you apply this combinator, the aggregate function takes the intermediate aggregation state as an argument, combines the states to finish aggregation, and returns the resulting value.
## -MergeState.
Merges the intermediate aggregation states in the same way as the -Merge combinator. However, it doesn't return the resulting value, but an intermediate aggregation state, similar to the -State combinator.
## -ForEach
Converts an aggregate function for tables into an aggregate function for arrays that aggregates the corresponding array items and returns an array of results. For example, `sumForEach` for the arrays `[1, 2]`, `[3, 4, 5]`and`[6, 7]`returns the result `[10, 13, 5]` after adding together the corresponding array items.

21
docs/en/agg_functions/index.md Normal file
View File

@ -0,0 +1,21 @@
<a name="aggregate_functions"></a>
# Aggregate functions
Aggregate functions work in the [normal](http://www.sql-tutorial.com/sql-aggregate-functions-sql-tutorial) way as expected by database experts.
ClickHouse also supports:
- [Parametric aggregate functions](parametric_functions.md#aggregate_functions_parametric), which accept other parameters in addition to columns.
- [Combinators](combinators.md#aggregate_functions_combinators), which change the behavior of aggregate functions.
**Table of Contents**
```eval_rst
.. toctree::
reference
parametric_functions
combinators
```

331
docs/en/agg_functions/index.rst
View File

@ -1,331 +0,0 @@
Aggregate functions
===================
count()
-------
Counts the number of rows. Accepts zero arguments and returns UInt64.
The syntax ``COUNT(DISTINCT x)`` is not supported. The separate ``uniq`` aggregate function exists for this purpose.
A ``SELECT count() FROM table`` query is not optimized, because the number of entries in the table is not stored separately. It will select some small column from the table and count the number of values in it.
any(x)
------
Selects the first encountered value.
The query can be executed in any order and even in a different order each time, so the result of this function is indeterminate.
To get a determinate result, you can use the ``min`` or ``max`` function instead of ``any``.
In some cases, you can rely on the order of execution. This applies to cases when ``SELECT`` comes from a subquery that uses ``ORDER BY``.
When a SELECT query has the GROUP BY clause or at least one aggregate function, ClickHouse (in contrast to for example MySQL) requires that all expressions in the ``SELECT``, ``HAVING`` and ``ORDER BY`` clauses be calculated from keys or from aggregate functions. That is, each column selected from the table must be used either in keys, or inside aggregate functions. To get behavior like in MySQL, you can put the other columns in the ``any`` aggregate function.
anyLast(x)
----------
Selects the last value encountered.
The result is just as indeterminate as for the 'any' function.
min(x)
------
Calculates the minimum.
max(x)
------
Calculates the maximum
argMin(arg, val)
----------------
Calculates the 'arg' value for a minimal 'val' value. If there are several different values of 'arg' for minimal values of 'val', the first of these values encountered is output.
argMax(arg, val)
----------------
Calculates the 'arg' value for a maximum 'val' value. If there are several different values of 'arg' for maximum values of 'val', the first of these values encountered is output.
sum(x)
------
Calculates the sum.
Only works for numbers.
sumWithOverflow(x)
------------------
Calculates the sum of numbers using the same data type as the input. If the sum of values is larger than the maximum value for given type, it will overflow. Only works for numbers.
sumMap(key, value)
------
Performs summation of array 'value' by corresponding keys of array 'key'.
Number of elements in 'key' and 'value' arrays should be the same for each row, on which summation is being performed.
Returns a tuple of two arrays - sorted keys and values, summed up by corresponding keys.
Example:
.. code-block:: sql
CREATE TABLE sum_map(
date Date,
timeslot DateTime,
statusMap Nested(
status UInt16,
requests UInt64
)
) ENGINE = Log;
INSERT INTO sum_map VALUES
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10]);
SELECT
timeslot,
sumMap(statusMap.status, statusMap.requests)
FROM sum_map
GROUP BY timeslot
.. code-block:: text
┌────────────timeslot─┬─sumMap(statusMap.status, statusMap.requests)─┐
│ 2000-01-01 00:00:00 │ ([1,2,3,4,5],[10,10,20,10,10]) │
│ 2000-01-01 00:01:00 │ ([4,5,6,7,8],[10,10,20,10,10]) │
└─────────────────────┴──────────────────────────────────────────────┘
avg(x)
------
Calculates the average.
Only works for numbers.
The result is always Float64.
uniq(x)
-------
Calculates the approximate number of different values of the argument. Works for numbers, strings, dates, and dates with times.
Uses an adaptive sampling algorithm: for the calculation state, it uses a sample of element hash values with a size up to 65535.
Compared with the widely known `HyperLogLog <https://en.wikipedia.org/wiki/HyperLogLog>`_ algorithm, this algorithm is less effective in terms of accuracy and memory consumption (even up to proportionality), but it is adaptive. This means that with fairly high accuracy, it consumes less memory during simultaneous computation of cardinality for a large number of data sets whose cardinality has power law distribution (i.e. in cases when most of the data sets are small). This algorithm is also very accurate for data sets with small cardinality (up to 65536) and very efficient on CPU (when computing not too many of these functions, using ``uniq`` is almost as fast as using other aggregate functions).
There is no compensation for the bias of an estimate, so for large data sets the results are systematically deflated. This function is normally used for computing the number of unique visitors in Yandex.Metrica, so this bias does not play a role.
The result is deterministic (it does not depend on the order of query execution).
uniqCombined(x)
---------------
Approximately computes the number of different values ​​of the argument. Works for numbers, strings, dates, date-with-time, for several arguments and arguments-tuples.
A combination of three algorithms is used: an array, a hash table and `HyperLogLog <https://en.wikipedia.org/wiki/HyperLogLog>`_ with an error correction table. The memory consumption is several times smaller than the ``uniq`` function, and the accuracy is several times higher. The speed of operation is slightly lower than that of the ``uniq`` function, but sometimes it can be even higher - in the case of distributed requests, in which a large number of aggregation states are transmitted over the network. The maximum state size is 96 KiB (HyperLogLog of 217 6-bit cells).
The result is deterministic (it does not depend on the order of query execution).
The ``uniqCombined`` function is a good default choice for calculating the number of different values.
uniqHLL12(x)
------------
Uses the `HyperLogLog <https://en.wikipedia.org/wiki/HyperLogLog>`_ algorithm to approximate the number of different values of the argument. It uses 212 5-bit cells. The size of the state is slightly more than 2.5 KB.
The result is deterministic (it does not depend on the order of query execution).
In most cases, use the 'uniq' function. You should only use this function if you understand its advantages well.
uniqExact(x)
------------
Calculates the number of different values of the argument, exactly.
There is no reason to fear approximations, so it's better to use the ``uniq`` function.
You should use the ``uniqExact`` function if you definitely need an exact result.
The ``uniqExact`` function uses more memory than the ``uniq`` function, because the size of the state has unbounded growth as the number of different values increases.
groupArray(x), groupArray(max_size)(x)
--------------------------------------
Creates an array of argument values.
Values can be added to the array in any (indeterminate) order.
The second version (with ``max_size`` parameter) limits the size of resulting array to ``max_size`` elements.
For example, ``groupArray(1)(x)`` is equivalent to ``[any(x)]``.
In some cases, you can rely on the order of execution. This applies to cases when ``SELECT`` comes from a subquery that uses ``ORDER BY``.
groupUniqArray(x)
-----------------
Creates an array from different argument values. Memory consumption is the same as for the ``uniqExact`` function.
quantile(level)(x)
------------------
Approximates the 'level' quantile. 'level' is a constant, a floating-point number from 0 to 1. We recommend using a 'level' value in the range of 0.01..0.99.
Don't use a 'level' value equal to 0 or 1 - use the 'min' and 'max' functions for these cases.
The algorithm is the same as for the ``median`` function. Actually, ``quantile`` and ``median`` are internally the same function. You can use the ``quantile`` function without parameters - in this case, it calculates the median, and you can use the ``median`` function with parameters - in this case, it calculates the quantile of the set level.
When using multiple ``quantile` and ``median`` functions with different levels in a query, the internal states are not combined (that is, the query works less efficiently than it could). In this case, use the ``quantiles`` function.
quantileDeterministic(level)(x, determinator)
---------------------------------------------
Calculates the quantile of 'level' using the same algorithm as the ``medianDeterministic`` function.
quantileTiming(level)(x)
------------------------
Calculates the quantile of 'level' using the same algorithm as the ``medianTiming`` function.
quantileTimingWeighted(level)(x, weight)
----------------------------------------
Calculates the quantile of 'level' using the same algorithm as the ``medianTimingWeighted`` function.
quantileExact(level)(x)
-----------------------
Computes the level quantile exactly. To do this, all transferred values are added to an array, which is then partially sorted. Therefore, the function consumes O(n) memory, where n is the number of transferred values. However, for a small number of values, the function is very effective.
quantileExactWeighted(level)(x, weight)
---------------------------------------
Computes the level quantile exactly. In this case, each value is taken into account with the weight weight - as if it is present weight once. The arguments of the function can be considered as histograms, where the value "x" corresponds to the "column" of the histogram of the height weight, and the function itself can be considered as the summation of histograms.
The algorithm is a hash table. Because of this, in case the transmitted values ​​are often repeated, the function consumes less RAM than the quantileExact. You can use this function instead of quantileExact, specifying the number 1 as the weight.
quantileTDigest(level)(x)
-------------------------
Computes the level quantile approximately, using the `t-digest <https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf>`_ algorithm. The maximum error is 1%. The memory consumption per state is proportional to the logarithm of the number of transmitted values.
The performance of the function is below quantile, quantileTiming. By the ratio of state size and accuracy, the function is significantly better than quantile.
The result depends on the order in which the query is executed, and is nondeterministic.
median
------
Approximates the median. Also see the similar ``quantile`` function.
Works for numbers, dates, and dates with times.
For numbers it returns Float64, for dates - a date, and for dates with times - a date with time.
Uses `reservoir sampling <https://en.wikipedia.org/wiki/Reservoir_sampling>`_ with a reservoir size up to 8192.
If necessary, the result is output with linear approximation from the two neighboring values.
This algorithm proved to be more practical than another well-known algorithm - QDigest.
The result depends on the order of running the query, and is nondeterministic.
quantiles(level1, level2, ...)(x)
---------------------------------
Approximates quantiles of all specified levels.
The result is an array containing the corresponding number of values.
varSamp(x)
----------
Calculates the amount ``Σ((x - x̅)2) / (n - 1)``, where 'n' is the sample size and 'x̅' is the average value of 'x'.
It represents an unbiased estimate of the variance of a random variable, if the values passed to the function are a sample of this random amount.
Returns Float64. If n <= 1, it returns +∞.
varPop(x)
---------
Calculates the amount ``Σ((x - x̅)2) / n``, where 'n' is the sample size and 'x̅' is the average value of 'x'.
In other words, dispersion for a set of values. Returns Float64.
stddevSamp(x)
-------------
The result is equal to the square root of ``varSamp(x)``.
stddevPop(x)
------------
The result is equal to the square root of ``varPop(x)``.
covarSamp(x, y)
---------------
Calculates the value of ``Σ((x - x̅)(y - y̅)) / (n - 1)``.
Returns Float64. If n <= 1, it returns +∞.
covarPop(x, y)
--------------
Calculates the value of ``Σ((x - x̅)(y - y̅)) / n``.
corr(x, y)
----------
Calculates the Pearson correlation coefficient: ``Σ((x - x̅)(y - y̅)) / sqrt(Σ((x - x̅)2) * Σ((y - y̅)2))``.
Parametric aggregate functions
==============================
Some aggregate functions can accept not only argument columns (used for compression), but a set of parameters - constants for initialization. The syntax is two pairs of brackets instead of one. The first is for parameters, and the second is for arguments.
sequenceMatch(pattern)(time, cond1, cond2, ...)
-----------------------------------------------
Pattern matching for event chains.
'pattern' is a string containing a pattern to match. The pattern is similar to a regular expression.
'time' is the event time of the DateTime type.
'cond1, cond2 ...' are from one to 32 arguments of the UInt8 type that indicate whether an event condition was met.
The function collects a sequence of events in RAM. Then it checks whether this sequence matches the pattern.
It returns UInt8 - 0 if the pattern isn't matched, or 1 if it matches.
Example: ``sequenceMatch('(?1).*(?2)')(EventTime, URL LIKE '%company%', URL LIKE '%cart%')``
- whether there was a chain of events in which pages with the address in company were visited earlier than pages with the address in cart.
This is a simple example. You could write it using other aggregate functions:
.. code-block:: sql
minIf(EventTime, URL LIKE '%company%') < maxIf(EventTime, URL LIKE '%cart%').
However, there is no such solution for more complex situations.
Pattern syntax:
``(?1)`` - Reference to a condition (any number in place of 1).
``.*`` - Any number of events.
``(?t>=1800)`` - Time condition.
Any quantity of any type of events is allowed over the specified time.
The operators <, >, <= may be used instead of >=.
Any number may be specified in place of 1800.
Events that occur during the same second may be put in the chain in any order. This may affect the result of the function.
sequenceCount(pattern)(time, cond1, cond2, ...)
-----------------------------------------------
Similar to the sequenceMatch function, but it does not return the fact that there is a chain of events, and UInt64 is the number of strings found.
Chains are searched without overlapping. That is, the following chain can start only after the end of the previous one.
uniqUpTo(N)(x)
--------------
Calculates the number of different argument values, if it is less than or equal to N.
If the number of different argument values is greater than N, it returns N + 1.
Recommended for use with small Ns, up to 10. The maximum N value is 100.
For the state of an aggregate function, it uses the amount of memory equal to 1 + N * the size of one value of bytes.
For strings, it stores a non-cryptographic hash of 8 bytes. That is, the calculation is approximated for strings.
It works as fast as possible, except for cases when a large N value is used and the number of unique values is slightly less than N.
Usage example:
Problem: Generate a report that shows only keywords that produced at least 5 unique users.
Solution: Write in the query ``GROUP BY SearchPhrase HAVING uniqUpTo(4)(UserID) >= 5``
Aggregate function combinators
==============================
The name of an aggregate function can have a suffix appended to it. This changes the way the aggregate function works.
There are ``If`` and ``Array`` combinators. See the sections below.
If combinator. Conditional aggregate functions
----------------------------------------------
The suffix ``-If`` can be appended to the name of any aggregate function. In this case, the aggregate function accepts an extra argument - a condition (Uint8 type). The aggregate function processes only the rows that trigger the condition. If the condition was not triggered even once, it returns a default value (usually zeros or empty strings).
Examples: ``sumIf(column, cond)``, ``countIf(cond)``, ``avgIf(x, cond)``, ``quantilesTimingIf(level1, level2)(x, cond)``, ``argMinIf(arg, val, cond)`` and so on.
You can use aggregate functions to calculate aggregates for multiple conditions at once, without using subqueries and JOINs.
For example, in Yandex.Metrica, we use conditional aggregate functions for implementing segment comparison functionality.
Array combinator. Aggregate functions for array arguments
---------------------------------------------------------
The -Array suffix can be appended to any aggregate function. In this case, the aggregate function takes arguments of the 'Array(T)' type (arrays) instead of 'T' type arguments. If the aggregate function accepts multiple arguments, this must be arrays of equal lengths. When processing arrays, the aggregate function works like the original aggregate function across all array elements.
Example 1: ``sumArray(arr)`` - Totals all the elements of all 'arr' arrays. In this example, it could have been written more simply: sum(arraySum(arr)).
Example 2: ``uniqArray(arr)`` - Count the number of unique elements in all 'arr' arrays. This could be done an easier way: ``uniq(arrayJoin(arr))``, but it's not always possible to add 'arrayJoin' to a query.
The ``-If`` and ``-Array`` combinators can be used together. However, 'Array' must come first, then 'If'.
Examples: ``uniqArrayIf(arr, cond)``, ``quantilesTimingArrayIf(level1, level2)(arr, cond)``. Due to this order, the 'cond' argument can't be an array.
State combinator
----------------
If this combinator is used, the aggregate function returns intermediate aggregation state (for example, in the case of the ``uniqCombined`` function, a HyperLogLog structure for calculating the number of unique values), which has type of ``AggregateFunction(...)`` and can be used for further processing or can be saved to a table for subsequent pre-aggregation - see the sections "AggregatingMergeTree" and "functions for working with intermediate aggregation states".
Merge combinator
----------------
In the case of using this combinator, the aggregate function will take as an argument the intermediate state of an aggregation, pre-aggregate (combine together) these states, and return the finished/complete value.
MergeState combinator
---------------------
Merges the intermediate aggregation states, similar to the -Merge combinator, but returns a non-complete value, but an intermediate aggregation state, similar to the -State combinator.

72
docs/en/agg_functions/parametric_functions.md Normal file
View File

@ -0,0 +1,72 @@
<a name="aggregate_functions_parametric"></a>
# Parametric aggregate functions
Some aggregate functions can accept not only argument columns (used for compression), but a set of parameters – constants for initialization. The syntax is two pairs of brackets instead of one. The first is for parameters, and the second is for arguments.
## sequenceMatch(pattern)(time, cond1, cond2, ...)
Pattern matching for event chains.
`pattern` is a string containing a pattern to match. The pattern is similar to a regular expression.
`time` is the time of the event with the DateTime type.
`cond1`, `cond2` ... is from one to 32 arguments of type UInt8 that indicate whether a certain condition was met for the event.
The function collects a sequence of events in RAM. Then it checks whether this sequence matches the pattern.
It returns UInt8: 0 if the pattern isn't matched, or 1 if it matches.
Example: `sequenceMatch ('(?1).*(?2)')(EventTime, URL LIKE '%company%', URL LIKE '%cart%')`
- whether there was a chain of events in which a pageview with 'company' in the address occurred earlier than a pageview with 'cart' in the address.
This is a singular example. You could write it using other aggregate functions:
```text
minIf(EventTime, URL LIKE '%company%') < maxIf(EventTime, URL LIKE '%cart%').
```
However, there is no such solution for more complex situations.
Pattern syntax:
`(?1)` refers to the condition (any number can be used in place of 1).
`.*` is any number of any events.
`(?t>=1800)` is a time condition.
Any quantity of any type of events is allowed over the specified time.
Instead of `>=`, the following operators can be used:`<`, `>`, `<=`.
Any number may be specified in place of 1800.
Events that occur during the same second can be put in the chain in any order. This may affect the result of the function.
## sequenceCount(pattern)(time, cond1, cond2, ...)
Works the same way as the sequenceMatch function, but instead of returning whether there is an event chain, it returns UInt64 with the number of event chains found.
Chains are searched for without overlapping. In other words, the next chain can start only after the end of the previous one.
## uniqUpTo(N)(x)
Calculates the number of different argument values ​​if it is less than or equal to N. If the number of different argument values is greater than N, it returns N + 1.
Recommended for use with small Ns, up to 10. The maximum value of N is 100.
For the state of an aggregate function, it uses the amount of memory equal to 1 + N \* the size of one value of bytes.
For strings, it stores a non-cryptographic hash of 8 bytes. That is, the calculation is approximated for strings.
The function also works for several arguments.
It works as fast as possible, except for cases when a large N value is used and the number of unique values is slightly less than N.
Usage example:
```text
Problem: Generate a report that shows only keywords that produced at least 5 unique users.
Solution: Write in the GROUP BY query SearchPhrase HAVING uniqUpTo(4)(UserID) >= 5
```

335
docs/en/agg_functions/reference.md Normal file
View File

@ -0,0 +1,335 @@
<a name="aggregate_functions_reference"></a>
# Function reference
## count()
Counts the number of rows. Accepts zero arguments and returns UInt64.
The syntax `COUNT(DISTINCT x)` is not supported. The separate `uniq` aggregate function exists for this purpose.
A `SELECT count() FROM table` query is not optimized, because the number of entries in the table is not stored separately. It will select some small column from the table and count the number of values in it.
## any(x)
Selects the first encountered value.
The query can be executed in any order and even in a different order each time, so the result of this function is indeterminate.
To get a determinate result, you can use the 'min' or 'max' function instead of 'any'.
In some cases, you can rely on the order of execution. This applies to cases when SELECT comes from a subquery that uses ORDER BY.
When a `SELECT` query has the `GROUP BY` clause or at least one aggregate function, ClickHouse (in contrast to MySQL) requires that all expressions in the `SELECT`, `HAVING`, and `ORDER BY` clauses be calculated from keys or from aggregate functions. In other words, each column selected from the table must be used either in keys or inside aggregate functions. To get behavior like in MySQL, you can put the other columns in the `any` aggregate function.
## anyHeavy
Selects a frequently occurring value using the [heavy hitters](http://www.cs.umd.edu/~samir/498/karp.pdf) algorithm. If there is a value that occurs more than in half the cases in each of the query's execution threads, this value is returned. Normally, the result is nondeterministic.
```
anyHeavy(column)
```
**Arguments**
- `column` – The column name.
**Example**
Take the [OnTime](../getting_started/example_datasets/ontime.md#example_datasets-ontime)data set and select any frequently occurring value in the `AirlineID` column.
```sql
SELECT anyHeavy(AirlineID) AS res
FROM ontime
```
```
┌───res─┐
│ 19690 │
└───────┘
```
## anyLast(x)
Selects the last value encountered.
The result is just as indeterminate as for the `any` function.
## min(x)
Calculates the minimum.
## max(x)
Calculates the maximum.
## argMin(arg, val)
Calculates the 'arg' value for a minimal 'val' value. If there are several different values of 'arg' for minimal values of 'val', the first of these values encountered is output.
## argMax(arg, val)
Calculates the 'arg' value for a maximum 'val' value. If there are several different values of 'arg' for maximum values of 'val', the first of these values encountered is output.
## sum(x)
Calculates the sum.
Only works for numbers.
## sumWithOverflow(x)
Computes the sum of the numbers, using the same data type for the result as for the input parameters. If the sum exceeds the maximum value for this data type, the function returns an error.
Only works for numbers.
## sumMap(key, value)
Totals the 'value' array according to the keys specified in the 'key' array.
The number of elements in 'key' and 'value' must be the same for each row that is totaled.
Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys.
Example:
```sql
CREATE TABLE sum_map(
date Date,
timeslot DateTime,
statusMap Nested(
status UInt16,
requests UInt64
)
) ENGINE = Log;
INSERT INTO sum_map VALUES
('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10]);
SELECT
timeslot,
sumMap(statusMap.status, statusMap.requests)
FROM sum_map
GROUP BY timeslot
```
```text
┌────────────timeslot─┬─sumMap(statusMap.status, statusMap.requests)─┐
│ 2000-01-01 00:00:00 │ ([1,2,3,4,5],[10,10,20,10,10]) │
│ 2000-01-01 00:01:00 │ ([4,5,6,7,8],[10,10,20,10,10]) │
└─────────────────────┴──────────────────────────────────────────────┘
```
## avg(x)
Calculates the average.
Only works for numbers.
The result is always Float64.
## uniq(x)
Calculates the approximate number of different values of the argument. Works for numbers, strings, dates, date-with-time, and for multiple arguments and tuple arguments.
Uses an adaptive sampling algorithm: for the calculation state, it uses a sample of element hash values with a size up to 65536.
This algorithm is also very accurate for data sets with small cardinality (up to 65536) and very efficient on CPU (when computing not too many of these functions, using `uniq` is almost as fast as using other aggregate functions).
The result is determinate (it doesn't depend on the order of query processing).
## uniqCombined(x)
Calculates the approximate number of different values of the argument. Works for numbers, strings, dates, date-with-time, and for multiple arguments and tuple arguments.
A combination of three algorithms is used: array, hash table and [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) with an error correction table. The memory consumption is several times smaller than for the `uniq` function, and the accuracy is several times higher. Performance is slightly lower than for the `uniq` function, but sometimes it can be even higher than it, such as with distributed queries that transmit a large number of aggregation states over the network. The maximum state size is 96 KiB (HyperLogLog of 217 6-bit cells).
The result is determinate (it doesn't depend on the order of query processing).
The `uniqCombined` function is a good default choice for calculating the number of different values.
## uniqHLL12(x)
Uses the [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) algorithm to approximate the number of different values of the argument.
212 5-bit cells are used. The size of the state is slightly more than 2.5 KB.
The result is determinate (it doesn't depend on the order of query processing).
In most cases, use the `uniq` or `uniqCombined` function.
## uniqExact(x)
Calculates the number of different values of the argument, exactly.
There is no reason to fear approximations. It's better to use the `uniq` function.
Use the `uniqExact` function if you definitely need an exact result.
The `uniqExact` function uses more memory than the `uniq` function, because the size of the state has unbounded growth as the number of different values increases.
## groupArray(x), groupArray(max_size)(x)
Creates an array of argument values.
Values can be added to the array in any (indeterminate) order.
The second version (with the `max_size` parameter) limits the size of the resulting array to `max_size` elements.
For example, `groupArray (1) (x)` is equivalent to `[any (x)]`.
In some cases, you can still rely on the order of execution. This applies to cases when `SELECT` comes from a subquery that uses `ORDER BY`.
<a name="agg_functions_groupArrayInsertAt"></a>
## groupArrayInsertAt
Inserts a value into the array in the specified position.
Accepts the value and position as input. If several values ​​are inserted into the same position, any of them might end up in the resulting array (the first one will be used in the case of single-threaded execution). If no value is inserted into a position, the position is assigned the default value.
Optional parameters:
- The default value for substituting in empty positions.
- The length of the resulting array. This allows you to receive arrays of the same size for all the aggregate keys. When using this parameter, the default value must be specified.
## groupUniqArray(x)
Creates an array from different argument values. Memory consumption is the same as for the `uniqExact` function.
## quantile(level)(x)
Approximates the 'level' quantile. 'level' is a constant, a floating-point number from 0 to 1.
We recommend using a 'level' value in the range of 0.01..0.99
Don't use a 'level' value equal to 0 or 1 – use the 'min' and 'max' functions for these cases.
In this function, as well as in all functions for calculating quantiles, the 'level' parameter can be omitted. In this case, it is assumed to be equal to 0.5 (in other words, the function will calculate the median).
Works for numbers, dates, and dates with times.
Returns: for numbers – Float64; for dates – a date; for dates with times – a date with time.
Uses [reservoir sampling](https://ru.wikipedia.org/wiki/Reservoir_sampling) with a reservoir size up to 8192.
If necessary, the result is output with linear approximation from the two neighboring values.
This algorithm provides very low accuracy. See also: `quantileTiming`, `quantileTDigest`, `quantileExact`.
The result depends on the order of running the query, and is nondeterministic.
When using multiple `quantile` (and similar) functions with different levels in a query, the internal states are not combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` (and similar) functions.
## quantileDeterministic(level)(x, determinator)
Works the same way as the `quantile` function, but the result is deterministic and does not depend on the order of query execution.
To achieve this, the function takes a second argument – the "determinator". This is a number whose hash is used instead of a random number generator in the reservoir sampling algorithm. For the function to work correctly, the same determinator value should not occur too often. For the determinator, you can use an event ID, user ID, and so on.
Don't use this function for calculating timings. There is a more suitable function for this purpose: `quantileTiming`.
## quantileTiming(level)(x)
Computes the quantile of 'level' with a fixed precision.
Works for numbers. Intended for calculating quantiles of page loading time in milliseconds.
If the value is greater than 30,000 (a page loading time of more than 30 seconds), the result is equated to 30,000.
If the total value is not more than about 5670, then the calculation is accurate.
Otherwise:
- if the time is less than 1024 ms, then the calculation is accurate.
- otherwise the calculation is rounded to a multiple of 16 ms.
When passing negative values to the function, the behavior is undefined.
The returned value has the Float32 type. If no values were passed to the function (when using `quantileTimingIf`), 'nan' is returned. The purpose of this is to differentiate these instances from zeros. See the note on sorting NaNs in "ORDER BY clause".
The result is determinate (it doesn't depend on the order of query processing).
For its purpose (calculating quantiles of page loading times), using this function is more effective and the result is more accurate than for the `quantile` function.
## quantileTimingWeighted(level)(x, weight)
Differs from the 'medianTiming' function in that it has a second argument, "weights". Weight is a non-negative integer.
The result is calculated as if the 'x' value were passed 'weight' number of times to the 'medianTiming\` function.
## quantileExact(level)(x)
Computes the quantile of 'level' exactly. To do this, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes O(n) memory, where 'n' is the number of values that were passed. However, for a small number of values, the function is very effective.
## quantileExactWeighted(level)(x, weight)
Computes the quantile of 'level' exactly. In addition, each value is counted with its weight, as if it is present 'weight' times. The arguments of the function can be considered as histograms, where the value 'x' corresponds to a histogram "column" of the height 'weight', and the function itself can be considered as a summation of histograms.
A hash table is used as the algorithm. Because of this, if the passed values ​​are frequently repeated, the function consumes less RAM than `quantileExact`. You can use this function instead of `quantileExact` and specify the weight as 1.
## quantileTDigest(level)(x)
Approximates the quantile level using the [t-digest](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf) algorithm. The maximum error is 1%. Memory consumption by State is proportional to the logarithm of the number of passed values.
The performance of the function is lower than for ` quantile`, ` quantileTiming`. In terms of the ratio of State size to precision, this function is much better than `quantile`.
The result depends on the order of running the query, and is nondeterministic.
## median
All the quantile functions have corresponding median functions: `median`, `medianDeterministic`, `medianTiming`, `medianTimingWeighted`, `medianExact`, `medianExactWeighted`, `medianTDigest`. They are synonyms and their behavior is identical.
## quantiles(level1, level2, ...)(x)
All the quantile functions also have corresponding quantiles functions: `quantiles`, `quantilesDeterministic`, `quantilesTiming`, `quantilesTimingWeighted`, `quantilesExact`, `quantilesExactWeighted`, `quantilesTDigest`. These functions calculate all the quantiles of the listed levels in one pass, and return an array of the resulting values.
## varSamp(x)
Calculates the amount `Σ((x - x̅)^2) / (n - 1)`, where `n` is the sample size and `x̅`is the average value of `x`.
It represents an unbiased estimate of the variance of a random variable, if the values passed to the function are a sample of this random amount.
Returns `Float64`. When `n <= 1`, returns `+∞`.
## varPop(x)
Calculates the amount `Σ((x - x̅)^2) / (n - 1)`, where `n` is the sample size and `x̅`is the average value of `x`.
In other words, dispersion for a set of values. Returns `Float64`.
## stddevSamp(x)
The result is equal to the square root of `varSamp(x)`.
## stddevPop(x)
The result is equal to the square root of `varPop(x)`.
## topK
Returns an array of the most frequent values in the specified column. The resulting array is sorted in descending order of frequency of values (not by the values themselves).
Implements the [ Filtered Space-Saving](http://www.l2f.inesc-id.pt/~fmmb/wiki/uploads/Work/misnis.ref0a.pdf) algorithm for analyzing TopK, based on the reduce-and-combine algorithm from [Parallel Space Saving](https://arxiv.org/pdf/1401.0702.pdf).
```
topK(N)(column)
```
This function doesn't provide a guaranteed result. In certain situations, errors might occur and it might return frequent values that aren't the most frequent values.
We recommend using the `N < 10 ` value; performance is reduced with large `N` values. Maximum value of ` N = 65536`.
**Arguments**
- 'N' – The number of values.
- ' x ' – The column.
**Example**
Take the [OnTime](../getting_started/example_datasets/ontime.md#example_datasets-ontime)data set and select the three most frequently occurring values in the `AirlineID` column.
```sql
SELECT topK(3)(AirlineID) AS res
FROM ontime
```
```
┌─res─────────────────┐
│ [19393,19790,19805] │
└─────────────────────┘
```
## covarSamp(x, y)
Calculates the value of `Σ((x - x̅)(y - y̅)) / (n - 1)`.
Returns Float64. When `n <= 1`, returns +∞.
## covarPop(x, y)
Calculates the value of `Σ((x - x̅)(y - y̅)) / n`.
## corr(x, y)
Calculates the Pearson correlation coefficient: `Σ((x - x̅)(y - y̅)) / sqrt(Σ((x - x̅)^2) * Σ((y - y̅)^2))`.

11
docs/en/conf.py
View File

@ -17,6 +17,13 @@ import collections
import os
import sys
from recommonmark.parser import CommonMarkParser
from recommonmark.transform import AutoStructify
source_parsers = {
'.md': CommonMarkParser,
}
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
@ -41,7 +48,7 @@ templates_path = [
]
# The suffix of source filenames.
source_suffix = '.rst'
source_suffix = [ '.rst', '.md' ]
# The encoding of source files.
#source_encoding = 'utf-8-sig'
@ -299,3 +306,5 @@ def add_filters(app):
def setup(app):
app.add_javascript('custom.js')
app.connect(str('builder-inited'), add_filters)
app.add_config_value('recommonmark_config', {'enable_eval_rst': True}, True)
app.add_transform(AutoStructify)

5
docs/en/data_types/array.md Normal file
View File

@ -0,0 +1,5 @@
# Array(T)
An array of elements of type T. The T type can be any type, including an array.
We don't recommend using multidimensional arrays, because they are not well supported (for example, you can't store multidimensional arrays in tables with a MergeTree engine).

6
docs/en/data_types/array.rst
View File

@ -1,6 +0,0 @@
Array(T)
--------
Array of T-type items. The T type can be any type, including an array.
We don't recommend using multidimensional arrays, because they are not well supported (for example, you can't store multidimensional arrays in tables with engines from MergeTree family).

4
docs/en/data_types/boolean.md Normal file
View File

@ -0,0 +1,4 @@
# Boolean values
There isn't a separate type for boolean values. They use the UInt8 type, restricted to the values 0 or 1.

4
docs/en/data_types/boolean.rst
View File

@ -1,4 +0,0 @@
Boolean
-------
There is no separate type for boolean values. For them, the type UInt8 is used, in which only the values 0 and 1 are used.

7
docs/en/data_types/date.md Normal file
View File

@ -0,0 +1,7 @@
# Date
Date. Stored in two bytes as the number of days since 1970-01-01 (unsigned). Allows storing values from just after the beginning of the Unix Epoch to the upper threshold defined by a constant at the compilation stage (currently, this is until the year 2038, but it may be expanded to 2106).
The minimum value is output as 0000-00-00.
The date is stored without the time zone.

7
docs/en/data_types/date.rst
View File

@ -1,7 +0,0 @@
Date
----
A date. Stored in two bytes as the number of days since 1970-01-01 (unsigned). Allows storing values from just after the beginning of the Unix Epoch to the upper threshold defined by a constant at the compilation stage. Currently, this is until the year 2106 (the year 2106 is not covered).
The minimum value is output as 0000-00-00.
The date is stored without the time zone.

13
docs/en/data_types/datetime.rst → docs/en/data_types/datetime.md
View File

@ -1,16 +1,15 @@
DateTime
--------
# DateTime
Date with time. Stored in four bytes as a Unix timestamp (unsigned). Allows storing values in the same range as for the Date type. The minimal value is output as 0000-00-00 00:00:00. The time is stored with accuracy up to one second (without leap seconds).
Date with time. Stored in four bytes as a Unix timestamp (unsigned). Allows storing values in the same range as for the Date type. The minimal value is output as 0000-00-00 00:00:00.
The time is stored with accuracy up to one second (without leap seconds).
Time zones
~~~~~~~~~~
## Time zones
The date with time is converted from text (divided into component parts) to binary and back, using the system's time zone at the time the client or server starts. In text format, information about daylight savings is lost.
Note that by default the client adopts the server time zone at the beginning of the session. You can change this behaviour with the --use_client_time_zone command line switch.
By default, the client switches to the timezone of the server when it connects. You can change this behavior by enabling the client command-line option `--use_client_time_zone`.
Supports only those time zones that never had the time differ from UTC for a partial number of hours (without leap seconds) over the entire time range you will be working with.
So when working with a textual date (for example, when saving text dumps), keep in mind that there may be ambiguity during changes for daylight savings time, and there may be problems matching data if the time zone changed.

31
docs/en/data_types/enum.md Normal file
View File

@ -0,0 +1,31 @@
# Enum
Enum8 or Enum16. A finite set of string values that can be stored more efficiently than the `String` data type.
Example:
```text
Enum8('hello' = 1, 'world' = 2)
```
- A data type with two possible values: 'hello' and 'world'.
Each of the values is assigned a number in the range `-128 ... 127` for `Enum8` or in the range `-32768 ... 32767` for `Enum16`. All the strings and numbers must be different. An empty string is allowed. If this type is specified (in a table definition), numbers can be in an arbitrary order. However, the order does not matter.
In RAM, this type of column is stored in the same way as `Int8` or `Int16` of the corresponding numerical values.
When reading in text form, ClickHouse parses the value as a string and searches for the corresponding string from the set of Enum values. If it is not found, an exception is thrown. When reading in text format, the string is read and the corresponding numeric value is looked up. An exception will be thrown if it is not found.
When writing in text form, it writes the value as the corresponding string. If column data contains garbage (numbers that are not from the valid set), an exception is thrown. When reading and writing in binary form, it works the same way as for Int8 and Int16 data types.
The implicit default value is the value with the lowest number.
During `ORDER BY`, `GROUP BY`, `IN`, `DISTINCT` and so on, Enums behave the same way as the corresponding numbers. For example, ORDER BY sorts them numerically. Equality and comparison operators work the same way on Enums as they do on the underlying numeric values.
Enum values cannot be compared with numbers. Enums can be compared to a constant string. If the string compared to is not a valid value for the Enum, an exception will be thrown. The IN operator is supported with the Enum on the left hand side and a set of strings on the right hand side. The strings are the values of the corresponding Enum.
Most numeric and string operations are not defined for Enum values, e.g. adding a number to an Enum or concatenating a string to an Enum.
However, the Enum has a natural `toString` function that returns its string value.
Enum values are also convertible to numeric types using the `toT` function, where T is a numeric type. When T corresponds to the enum’s underlying numeric type, this conversion is zero-cost.
The Enum type can be changed without cost using ALTER, if only the set of values is changed. It is possible to both add and remove members of the Enum using ALTER (removing is safe only if the removed value has never been used in the table). As a safeguard, changing the numeric value of a previously defined Enum member will throw an exception.
Using ALTER, it is possible to change an Enum8 to an Enum16 or vice versa, just like changing an Int8 to Int16.

30
docs/en/data_types/enum.rst
View File

@ -1,30 +0,0 @@
Enum
----
Enum8 or Enum16. A set of enumerated string values that are stored as Int8 or Int16.
Example:
.. code-block:: sql
Enum8('hello' = 1, 'world' = 2)
This data type has two possible values - 'hello' and 'world'.
The numeric values must be within -128..127 for ``Enum8`` and -32768..32767 for ``Enum16``. Every member of the enum must also have different numbers. The empty string is a valid value. The numbers do not need to be sequential and can be in any order. The order does not matter.
In memory, the data is stored in the same way as the numeric types ``Int8`` and ``Int16``.
When reading in text format, the string is read and the corresponding numeric value is looked up. An exception will be thrown if it is not found.
When writing in text format, the stored number is looked up and the corresponding string is written out. An exception will be thrown if the number does not correspond to a known value.
In binary format, the information is saved in the same way as ``Int8`` and ``Int16``.
The implicit default value for an Enum is the value having the smallest numeric value.
In ORDER BY, GROUP BY, IN, DISTINCT, etc. Enums behave like the numeric value. e.g. they will be sorted by the numeric value in an ``ORDER BY``. Equality and comparison operators behave like they do on the underlying numeric value.
Enum values cannot be compared to numbers, they must be compared to a string. If the string compared to is not a valid value for the Enum, an exception will be thrown. The ``IN`` operator is supported with the Enum on the left hand side and a set of strings on the right hand side.
Most numeric and string operations are not defined for Enum values, e.g. adding a number to an Enum or concatenating a string to an Enum. However, the toString function can be used to convert the Enum to its string value. Enum values are also convertible to numeric types using the ``toT`` function where ``T`` is a numeric type. When ``T`` corresponds to the enum's underlying numeric type, this conversion is zero-cost.
It is possible to add new members to the ``Enum`` using ``ALTER``. If the only change is to the set of values, the operation will be almost instant. It is also possible to remove members of the Enum using ALTER. Removing members is only safe if the removed value has never been used in the table. As a safeguard, changing the numeric value of a previously defined Enum member will throw an exception.
Using ``ALTER``, it is possible to change an ``Enum8`` to an ``Enum16`` or vice versa - just like changing an ``Int8`` to ``Int16``.

10
docs/en/data_types/fixedstring.md Normal file
View File

@ -0,0 +1,10 @@
# FixedString(N)
A fixed-length string of N bytes (not characters or code points). N must be a strictly positive natural number.
When the server reads a string that contains fewer bytes (such as when parsing INSERT data), the string is padded to N bytes by appending null bytes at the right.
When the server reads a string that contains more bytes, an error message is returned.
When the server writes a string (such as when outputting the result of a SELECT query), null bytes are not trimmed off of the end of the string, but are output.
Note that this behavior differs from MySQL behavior for the CHAR type (where strings are padded with spaces, and the spaces are removed for output).
Fewer functions can work with the FixedString(N) type than with String, so it is less convenient to use.

10
docs/en/data_types/fixedstring.rst
View File

@ -1,10 +0,0 @@
FixedString(N)
--------------
A fixed-length string of N bytes (not characters or code points). N must be a strictly positive natural number.
When server reads a string (as an input passed in INSERT query, for example) that contains fewer bytes, the string is padded to N bytes by appending null bytes at the right.
When server reads a string that contains more bytes, an error message is returned.
When server writes a string (as an output of SELECT query, for example), null bytes are not trimmed off of the end of the string, but are output.
Note that this behavior differs from MySQL behavior for the CHAR type (where strings are padded with spaces, and the spaces are removed for output).
Fewer functions can work with the FixedString(N) type than with String, so it is less convenient to use.

71
docs/en/data_types/float.md Normal file
View File

@ -0,0 +1,71 @@
# Float32, Float64
[Floating point numbers](https://en.wikipedia.org/wiki/IEEE_754).
Types are equivalent to types of C:
- `Float32` - `float`
- `Float64` - ` double`
We recommend that you store data in integer form whenever possible. For example, convert fixed precision numbers to integer values, such as monetary amounts or page load times in milliseconds.
## Using floating-point numbers
- Computations with floating-point numbers might produce a rounding error.
```sql
SELECT 1 - 0.9
```
```
┌───────minus(1, 0.9)─┐
│ 0.09999999999999998 │
└─────────────────────┘
```
- The result of the calculation depends on the calculation method (the processor type and architecture of the computer system).
- Floating-point calculations might result in numbers such as infinity (`Inf`) and "not-a-number" (`NaN`). This should be taken into account when processing the results of calculations.
- When reading floating point numbers from rows, the result might not be the nearest machine-representable number.
## NaN and Inf
In contrast to standard SQL, ClickHouse supports the following categories of floating-point numbers:
- `Inf` – Infinity.
```sql
SELECT 0.5 / 0
```
```
┌─divide(0.5, 0)─┐
│ inf │
└────────────────┘
```
- `-Inf` – Negative infinity.
```sql
SELECT -0.5 / 0
```
```
┌─divide(-0.5, 0)─┐
│ -inf │
└─────────────────┘
```
- `NaN` – Not a number.
```
SELECT 0 / 0
```
```
┌─divide(0, 0)─┐
│ nan │
└──────────────┘
```
See the rules for ` NaN` sorting in the section [ORDER BY clause](../query_language/queries.md#query_language-queries-order_by).

7
docs/en/data_types/float.rst
View File

@ -1,7 +0,0 @@
Float32, Float64
----------------
Floating-point numbers are just like 'float' and 'double' in the C language.
In contrast to standard SQL, floating-point numbers support 'inf', '-inf', and even 'nan's.
See the notes on sorting nans in "ORDER BY clause".
We do not recommend storing floating-point numbers in tables.

13
docs/en/data_types/index.md Normal file
View File

@ -0,0 +1,13 @@
<a name="data_types"></a>
# Data types
```eval_rst
.. toctree::
:glob:
*
*/index
```

18
docs/en/data_types/int_uint.md Normal file
View File

@ -0,0 +1,18 @@
# UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
Fixed-length integers, with or without a sign.
## Int ranges
- Int8 - [-128 : 127]
- Int16 - [-32768 : 32767]
- Int32 - [-2147483648 : 2147483647]
- Int64 - [-9223372036854775808 : 9223372036854775807]
## Uint ranges
- UInt8 - [0 : 255]
- UInt16 - [0 : 65535]
- UInt32 - [0 : 4294967295]
- UInt64 - [0 : 18446744073709551615]

40
docs/en/data_types/int_uint.rst
View File

@ -1,40 +0,0 @@
UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
--------------------------------------------------------
Fixed-length integers, with or without a sign.
Int ranges
""""""""""
.. table::
+--------+----------------------+-----------------------+
| Тип | From | To |
+========+======================+=======================+
| Int8 | -128 | 127 |
+--------+----------------------+-----------------------+
| Int16 | -32768 | 32767 |
+--------+----------------------+-----------------------+
| Int32 | -2147483648 | 2147483647 |
+--------+----------------------+-----------------------+
| Int64 | -9223372036854775808 | 9223372036854775807 |
+--------+----------------------+-----------------------+
Uint ranges
"""""""""""
.. table::
+--------+----------------------+-----------------------+
| Тип | From | To |
+========+======================+=======================+
| UInt8 | 0 | 255 |
+--------+----------------------+-----------------------+
| UInt16 | 0 | 65535 |
+--------+----------------------+-----------------------+
| UInt32 | 0 | 4294967295 |
+--------+----------------------+-----------------------+
| UInt64 | 0 | 18446744073709551615 |
+--------+----------------------+-----------------------+

4
docs/en/data_types/nested_data_structures/aggregatefunction.rst → docs/en/data_types/nested_data_structures/aggregatefunction.md
View File

@ -1,4 +1,4 @@
AggregateFunction(name, types_of_arguments...)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# AggregateFunction(name, types_of_arguments...)
The intermediate state of an aggregate function. To get it, use aggregate functions with the '-State' suffix. For more information, see "AggregatingMergeTree".

9
docs/en/data_types/nested_data_structures/index.md Normal file
View File

@ -0,0 +1,9 @@
# Nested data structures
```eval_rst
.. toctree::
:glob:
*
```

7
docs/en/data_types/nested_data_structures/index.rst
View File

@ -1,7 +0,0 @@
Nested data structures
----------------------
.. toctree::
:glob:
*

98
docs/en/data_types/nested_data_structures/nested.md Normal file
View File

@ -0,0 +1,98 @@
# Nested(Name1 Type1, Name2 Type2, ...)
A nested data structure is like a nested table. The parameters of a nested data structure – the column names and types – are specified the same way as in a CREATE query. Each table row can correspond to any number of rows in a nested data structure.
Example:
```sql
CREATE TABLE test.visits
(
CounterID UInt32,
StartDate Date,
Sign Int8,
IsNew UInt8,
VisitID UInt64,
UserID UInt64,
...
Goals Nested
(
ID UInt32,
Serial UInt32,
EventTime DateTime,
Price Int64,
OrderID String,
CurrencyID UInt32
),
...
) ENGINE = CollapsingMergeTree(StartDate, intHash32(UserID), (CounterID, StartDate, intHash32(UserID), VisitID), 8192, Sign)
```
This example declares the `Goals` nested data structure, which contains data about conversions (goals reached). Each row in the 'visits' table can correspond to zero or any number of conversions.
Only a single nesting level is supported. Columns of nested structures containing arrays are equivalent to multidimensional arrays, so they have limited support (there is no support for storing these columns in tables with the MergeTree engine).
In most cases, when working with a nested data structure, its individual columns are specified. To do this, the column names are separated by a dot. These columns make up an array of matching types. All the column arrays of a single nested data structure have the same length.
Example:
```sql
SELECT
Goals.ID,
Goals.EventTime
FROM test.visits
WHERE CounterID = 101500 AND length(Goals.ID) < 5
LIMIT 10
```
```text
┌─Goals.ID───────────────────────┬─Goals.EventTime───────────────────────────────────────────────────────────────────────────┐
│ [1073752,591325,591325] │ ['2014-03-17 16:38:10','2014-03-17 16:38:48','2014-03-17 16:42:27'] │
│ [1073752] │ ['2014-03-17 00:28:25'] │
│ [1073752] │ ['2014-03-17 10:46:20'] │
│ [1073752,591325,591325,591325] │ ['2014-03-17 13:59:20','2014-03-17 22:17:55','2014-03-17 22:18:07','2014-03-17 22:18:51'] │
│ [] │ [] │
│ [1073752,591325,591325] │ ['2014-03-17 11:37:06','2014-03-17 14:07:47','2014-03-17 14:36:21'] │
│ [] │ [] │
│ [] │ [] │
│ [591325,1073752] │ ['2014-03-17 00:46:05','2014-03-17 00:46:05'] │
│ [1073752,591325,591325,591325] │ ['2014-03-17 13:28:33','2014-03-17 13:30:26','2014-03-17 18:51:21','2014-03-17 18:51:45'] │
└────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────┘
```
It is easiest to think of a nested data structure as a set of multiple column arrays of the same length.
The only place where a SELECT query can specify the name of an entire nested data structure instead of individual columns is the ARRAY JOIN clause. For more information, see "ARRAY JOIN clause". Example:
```sql
SELECT
Goal.ID,
Goal.EventTime
FROM test.visits
ARRAY JOIN Goals AS Goal
WHERE CounterID = 101500 AND length(Goals.ID) < 5
LIMIT 10
```
```text
┌─Goal.ID─┬──────Goal.EventTime─┐
│ 1073752 │ 2014-03-17 16:38:10 │
│ 591325 │ 2014-03-17 16:38:48 │
│ 591325 │ 2014-03-17 16:42:27 │
│ 1073752 │ 2014-03-17 00:28:25 │
│ 1073752 │ 2014-03-17 10:46:20 │
│ 1073752 │ 2014-03-17 13:59:20 │
│ 591325 │ 2014-03-17 22:17:55 │
│ 591325 │ 2014-03-17 22:18:07 │
│ 591325 │ 2014-03-17 22:18:51 │
│ 1073752 │ 2014-03-17 11:37:06 │
└─────────┴─────────────────────┘
```
You can't perform SELECT for an entire nested data structure. You can only explicitly list individual columns that are part of it.
For an INSERT query, you should pass all the component column arrays of a nested data structure separately (as if they were individual column arrays). During insertion, the system checks that they have the same length.
For a DESCRIBE query, the columns in a nested data structure are listed separately in the same way.
The ALTER query is very limited for elements in a nested data structure.

99
docs/en/data_types/nested_data_structures/nested.rst
View File

@ -1,99 +0,0 @@
Nested(Name1 Type1, Name2 Type2, ...)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A nested data structure is like a nested table. The parameters of a nested data structure - the column names and types - are specified the same way as in a CREATE query. Each table row can correspond to any number of rows in a nested data structure.
Example:
.. code-block:: sql
CREATE TABLE test.visits
(
CounterID UInt32,
StartDate Date,
Sign Int8,
IsNew UInt8,
VisitID UInt64,
UserID UInt64,
...
Goals Nested
(
ID UInt32,
Serial UInt32,
EventTime DateTime,
Price Int64,
OrderID String,
CurrencyID UInt32
),
...
) ENGINE = CollapsingMergeTree(StartDate, intHash32(UserID), (CounterID, StartDate, intHash32(UserID), VisitID), 8192, Sign)
This example declares the 'Goals' nested data structure, which contains data about conversions (goals reached). Each row in the 'visits' table can correspond to zero or any number of conversions.
Only a single nesting level is supported. Nested structure columns with array type are equivalent to multidimensional arrays and thus their support is limited (storing such columns in tables with engines from MergeTree family is not supported).
In most cases, when working with a nested data structure, its individual columns are specified. To do this, the column names are separated by a dot. These columns make up an array of matching types. All the column arrays of a single nested data structure have the same length.
Example:
.. code-block:: sql
SELECT
Goals.ID,
Goals.EventTime
FROM test.visits
WHERE CounterID = 101500 AND length(Goals.ID) < 5
LIMIT 10
.. code-block:: text
┌─Goals.ID───────────────────────┬─Goals.EventTime───────────────────────────────────────────────────────────────────────────┐
│ [1073752,591325,591325] │ ['2014-03-17 16:38:10','2014-03-17 16:38:48','2014-03-17 16:42:27'] │
│ [1073752] │ ['2014-03-17 00:28:25'] │
│ [1073752] │ ['2014-03-17 10:46:20'] │
│ [1073752,591325,591325,591325] │ ['2014-03-17 13:59:20','2014-03-17 22:17:55','2014-03-17 22:18:07','2014-03-17 22:18:51'] │
│ [] │ [] │
│ [1073752,591325,591325] │ ['2014-03-17 11:37:06','2014-03-17 14:07:47','2014-03-17 14:36:21'] │
│ [] │ [] │
│ [] │ [] │
│ [591325,1073752] │ ['2014-03-17 00:46:05','2014-03-17 00:46:05'] │
│ [1073752,591325,591325,591325] │ ['2014-03-17 13:28:33','2014-03-17 13:30:26','2014-03-17 18:51:21','2014-03-17 18:51:45'] │
└────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────┘
It is easiest to think of a nested data structure as a set of multiple column arrays of the same length.
The only place where a SELECT query can specify the name of an entire nested data structure instead of individual columns is the ARRAY JOIN clause. For more information, see "ARRAY JOIN clause". Example:
.. code-block:: sql
SELECT
Goal.ID,
Goal.EventTime
FROM test.visits
ARRAY JOIN Goals AS Goal
WHERE CounterID = 101500 AND length(Goals.ID) < 5
LIMIT 10
.. code-block:: text
┌─Goal.ID─┬──────Goal.EventTime─┐
│ 1073752 │ 2014-03-17 16:38:10 │
│ 591325 │ 2014-03-17 16:38:48 │
│ 591325 │ 2014-03-17 16:42:27 │
│ 1073752 │ 2014-03-17 00:28:25 │
│ 1073752 │ 2014-03-17 10:46:20 │
│ 1073752 │ 2014-03-17 13:59:20 │
│ 591325 │ 2014-03-17 22:17:55 │
│ 591325 │ 2014-03-17 22:18:07 │
│ 591325 │ 2014-03-17 22:18:51 │
│ 1073752 │ 2014-03-17 11:37:06 │
└─────────┴─────────────────────┘
You can't perform SELECT for an entire nested data structure. You can only explicitly list individual columns that are part of it.
For an INSERT query, you should pass all the component column arrays of a nested data structure separately (as if they were individual column arrays). During insertion, the system checks that they have the same length.
For a DESCRIBE query, the columns in a nested data structure are listed separately in the same way.
The ALTER query is very limited for elements in a nested data structure.

4
docs/en/data_types/special_data_types/expression.rst → docs/en/data_types/special_data_types/expression.md
View File

@ -1,4 +1,4 @@
Expression
~~~~~~~~~~
# Expression
Used for representing lambda expressions in high-order functions.

6
docs/en/data_types/special_data_types/index.rst → docs/en/data_types/special_data_types/index.md
View File

@ -1,9 +1,11 @@
Special data types
------------------
# Special data types
Special data type values can't be saved to a table or output in results, but are used as the intermediate result of running a query.
```eval_rst
.. toctree::
:glob:
*
```

4
docs/en/data_types/special_data_types/set.rst → docs/en/data_types/special_data_types/set.md
View File

@ -1,4 +1,4 @@
Set
~~~
# Set
Used for the right half of an IN expression.

8
docs/en/data_types/string.rst → docs/en/data_types/string.md
View File

@ -1,14 +1,12 @@
String
------
# String
Strings of an arbitrary length. The length is not limited. The value can contain an arbitrary set of bytes, including null bytes.
The String type replaces the types VARCHAR, BLOB, CLOB, and others from other DBMSs.
Encodings
~~~~~~~~~
## Encodings
ClickHouse doesn't have the concept of encodings. Strings can contain an arbitrary set of bytes, which are stored and output as-is.
If you need to store texts, we recommend using UTF-8 encoding. At the very least, if your terminal uses UTF-8 (as recommended), you can read and write your values without making conversions.
Similarly, certain functions for working with strings have separate variations that work under the assumption that the string contains a set of bytes representing a UTF-8 encoded text.
For example, the 'length' function calculates the string length in bytes, while the 'lengthUTF8' function calculates the string length in Unicode code points, assuming that the value is UTF-8 encoded.

6
docs/en/data_types/tuple.rst → docs/en/data_types/tuple.md
View File

@ -1,6 +1,6 @@
Tuple(T1, T2, ...)
------------------
# Tuple(T1, T2, ...)
Tuples can't be written to tables (other than Memory tables). They are used for temporary column grouping. Columns can be grouped when an IN expression is used in a query, and for specifying certain formal parameters of lambda functions. For more information, see "IN operators" and "Higher order functions".
Tuples can be output as the result of running a query. In this case, for text formats other than JSON*, values are comma-separated in brackets. In JSON* formats, tuples are output as arrays (in square brackets).
Tuples can be output as the result of running a query. In this case, for text formats other than JSON\*, values are comma-separated in brackets. In JSON\* formats, tuples are output as arrays (in square brackets).

195
docs/en/development/architecture.md Normal file
View File

@ -0,0 +1,195 @@
# Overview of ClickHouse architecture
ClickHouse is a true column-oriented DBMS. Data is stored by columns, and during the execution of arrays (vectors or chunks of columns). Whenever possible, operations are dispatched on arrays, rather than on individual values. This is called "vectorized query execution," and it helps lower the cost of actual data processing.
> This idea is nothing new. It dates back to the `APL` programming language and its descendants: `A +`, `J`, `K`, and `Q`. Array programming is used in scientific data processing. Neither is this idea something new in relational databases: for example, it is used in the `Vectorwise` system.
There are two different approaches for speeding up the query processing: vectorized query execution and runtime code generation. In the latter, the code is generated for every kind of query on the fly, removing all indirection and dynamic dispatch. Neither of these approaches is strictly better than the other. Runtime code generation can be better when it's fuses many operations together, thus fully utilizing CPU execution units and the pipeline. Vectorized query execution can be less practical, because it involves the temporary vectors that must be written to the cache and read back. If the temporary data does not fit in the L2 cache, this becomes an issue. But vectorized query execution more easily utilizes the SIMD capabilities of the CPU. A [research paper](http://15721.courses.cs.cmu.edu/spring2016/papers/p5-sompolski.pdf) written by our friends shows that it is better to combine both approaches. ClickHouse uses vectorized query execution and has limited initial support for runtime code.
## Columns
To represent columns in memory (actually, chunks of columns), the `IColumn` interface is used. This interface provides helper methods for implementation of various relational operators. Almost all operations are immutable: they do not modify the original column, but create a new modified one. For example, the `IColumn :: filter` method accepts a filter byte mask. It is used for the `WHERE` and `HAVING` relational operators. Additional examples: the `IColumn :: permute` method to support `ORDER BY`, the `IColumn :: cut` method to support `LIMIT`, and so on.
Various `IColumn` implementations (`ColumnUInt8`, `ColumnString` and so on) are responsible for the memory layout of columns. Memory layout is usually a contiguous array. For the integer type of columns it is just one contiguous array, like `std :: vector`. For `String` and `Array` columns, it is two vectors: one for all array elements, placed contiguously, and a second one for offsets to the beginning of each array. There is also `ColumnConst` that stores just one value in memory, but looks like a column.
## Field
Nevertheless, it is possible to work with individual values as well. To represent an individual value, the `Field` is used. `Field` is just a discriminated union of `UInt64`, `Int64`, `Float64`, `String` and `Array`. `IColumn` has the `operator[]` method to get the n-th value as a `Field`, and the `insert` method to append a `Field` to the end of a column. These methods are not very efficient, because they require dealing with temporary `Field` objects representing an individual value. There are more efficient methods, such as `insertFrom`, `insertRangeFrom`, and so on.
`Field` doesn't have enough information about a specific data type for a table. For example, `UInt8`, `UInt16`, `UInt32`, and `UInt64` are all represented as `UInt64` in a `Field`.
## Leaky abstractions
`IColumn` has methods for common relational transformations of data, but they don't meet all needs. For example, `ColumnUInt64` doesn't have a method to calculate the sum of two columns, and `ColumnString` doesn't have a method to run a substring search. These countless routines are implemented outside of `IColumn`.
Various functions on columns can be implemented in a generic, non-efficient way using `IColumn` methods to extract `Field` values, or in a specialized way using knowledge of inner memory layout of data in a specific `IColumn` implementation. To do this, functions are cast to a specific `IColumn` type and deal with internal representation directly. For example, `ColumnUInt64` has the `getData` method that returns a reference to an internal array, then a separate routine reads or fills that array directly. In fact, we have "leaky abstractions" to allow efficient specializations of various routines.
## Data types
`IDataType` is responsible for serialization and deserialization: for reading and writing chunks of columns or individual values in binary or text form.
`IDataType` directly corresponds to data types in tables. For example, there are `DataTypeUInt32`, `DataTypeDateTime`, `DataTypeString` and so on.
`IDataType` and `IColumn` are only loosely related to each other. Different data types can be represented in memory by the same `IColumn` implementations. For example, `DataTypeUInt32` and `DataTypeDateTime` are both represented by `ColumnUInt32` or `ColumnConstUInt32`. In addition, the same data type can be represented by different `IColumn` implementations. For example, `DataTypeUInt8` can be represented by `ColumnUInt8` or `ColumnConstUInt8`.
`IDataType` only stores metadata. For instance, `DataTypeUInt8` doesn't store anything at all (except vptr) and `DataTypeFixedString` stores just `N` (the size of fixed-size strings).
`IDataType` has helper methods for various data formats. Examples are methods to serialize a value with possible quoting, to serialize a value for JSON, and to serialize a value as part of XML format. There is no direct correspondence to data formats. For example, the different data formats `Pretty` and `TabSeparated` can use the same `serializeTextEscaped` helper method from the `IDataType` interface.
## Block
A `Block` is a container that represents a subset (chunk) of a table in memory. It is just a set of triples: `(IColumn, IDataType, column name)`. During query execution, data is processed by `Block`s. If we have a `Block`, we have data (in the `IColumn` object), we have information about its type (in `IDataType`) that tells us how to deal with that column, and we have the column name (either the original column name from the table, or some artificial name assigned for getting temporary results of calculations).
When we calculate some function over columns in a block, we add another column with its result to the block, and we don't touch columns for arguments of the function because operations are immutable. Later, unneeded columns can be removed from the block, but not modified. This is convenient for elimination of common subexpressions.
Blocks are created for every processed chunk of data. Note that for the same type of calculation, the column names and types remain the same for different blocks, and only column data changes. It is better to split block data from the block header, because small block sizes will have a high overhead of temporary strings for copying shared_ptrs and column names.
## Block Streams
Block streams are for processing data. We use streams of blocks to read data from somewhere, perform data transformations, or write data to somewhere. `IBlockInputStream` has the `read` method to fetch the next block while available. `IBlockOutputStream` has the `write` method to push the block somewhere.
Streams are responsible for:
1. Reading or writing to a table. The table just returns a stream for reading or writing blocks.
2. Implementing data formats. For example, if you want to output data to a terminal in `Pretty` format, you create a block output stream where you push blocks, and it formats them.
3. Performing data transformations. Let's say you have `IBlockInputStream` and want to create a filtered stream. You create `FilterBlockInputStream` and initialize it with your stream. Then when you pull a block from `FilterBlockInputStream`, it pulls a block from your stream, filters it, and returns the filtered block to you. Query execution pipelines are represented this way.
There are more sophisticated transformations. For example, when you pull from `AggregatingBlockInputStream`, it reads all data from its source, aggregates it, and then returns a stream of aggregated data for you. Another example: `UnionBlockInputStream` accepts many input sources in the constructor and also a number of threads. It launches multiple threads and reads from multiple sources in parallel.
> Block streams use the "pull" approach to control flow: when you pull a block from the first stream, it consequently pulls the required blocks from nested streams, and the entire execution pipeline will work. Neither "pull" nor "push" is the best solution, because control flow is implicit, and that limits implementation of various features like simultaneous execution of multiple queries (merging many pipelines together). This limitation could be overcome with coroutines or just running extra threads that wait for each other. We may have more possibilities if we make control flow explicit: if we locate the logic for passing data from one calculation unit to another outside of those calculation units. Read this [article](http://journal.stuffwithstuff.com/2013/01/13/iteration-inside-and-out/) for more thoughts.
We should note that the query execution pipeline creates temporary data at each step. We try to keep block size small enough so that temporary data fits in the CPU cache. With that assumption, writing and reading temporary data is almost free in comparison with other calculations. We could consider an alternative, which is to fuse many operations in the pipeline together, to make the pipeline as short as possible and remove much of the temporary data. This could be an advantage, but it also has drawbacks. For example, a split pipeline makes it easy to implement caching intermediate data, stealing intermediate data from similar queries running at the same time, and merging pipelines for similar queries.
## Formats
Data formats are implemented with block streams. There are "presentational" formats only suitable for output of data to the client, such as `Pretty` format, which provides only `IBlockOutputStream`. And there are input/output formats, such as `TabSeparated` or `JSONEachRow`.
There are also row streams: `IRowInputStream` and `IRowOutputStream`. They allow you to pull/push data by individual rows, not by blocks. And they are only needed to simplify implementation of row-oriented formats. The wrappers `BlockInputStreamFromRowInputStream` and `BlockOutputStreamFromRowOutputStream` allow you to convert row-oriented streams to regular block-oriented streams.
## I/O
For byte-oriented input/output, there are `ReadBuffer` and `WriteBuffer` abstract classes. They are used instead of C++ `iostream`'s. Don't worry: every mature C++ project is using something other than `iostream`'s for good reasons.
`ReadBuffer` and `WriteBuffer` are just a contiguous buffer and a cursor pointing to the position in that buffer. Implementations may own or not own the memory for the buffer. There is a virtual method to fill the buffer with the following data (for `ReadBuffer`) or to flush the buffer somewhere (for `WriteBuffer`). The virtual methods are rarely called.
Implementations of `ReadBuffer`/`WriteBuffer` are used for working with files and file descriptors and network sockets, for implementing compression (`CompressedWriteBuffer` is initialized with another WriteBuffer and performs compression before writing data to it), and for other purposes – the names `ConcatReadBuffer`, `LimitReadBuffer`, and `HashingWriteBuffer` speak for themselves.
Read/WriteBuffers only deal with bytes. To help with formatted input/output (for instance, to write a number in decimal format), there are functions from `ReadHelpers` and `WriteHelpers` header files.
Let's look at what happens when you want to write a result set in `JSON` format to stdout. You have a result set ready to be fetched from `IBlockInputStream`. You create `WriteBufferFromFileDescriptor(STDOUT_FILENO)` to write bytes to stdout. You create `JSONRowOutputStream`, initialized with that `WriteBuffer`, to write rows in `JSON` to stdout. You create `BlockOutputStreamFromRowOutputStream` on top of it, to represent it as `IBlockOutputStream`. Then you call `copyData` to transfer data from `IBlockInputStream` to `IBlockOutputStream`, and everything works. Internally, `JSONRowOutputStream` will write various JSON delimiters and call the `IDataType::serializeTextJSON` method with a reference to `IColumn` and the row number as arguments. Consequently, `IDataType::serializeTextJSON` will call a method from `WriteHelpers.h`: for example, `writeText` for numeric types and `writeJSONString` for `DataTypeString`.
## Tables
Tables are represented by the `IStorage` interface. Different implementations of that interface are different table engines. Examples are `StorageMergeTree`, `StorageMemory`, and so on. Instances of these classes are just tables.
The most important `IStorage` methods are `read` and `write`. There are also `alter`, `rename`, `drop`, and so on. The `read` method accepts the following arguments: the set of columns to read from a table, the `AST` query to consider, and the desired number of streams to return. It returns one or multiple `IBlockInputStream` objects and information about the stage of data processing that was completed inside a table engine during query execution.
In most cases, the read method is only responsible for reading the specified columns from a table, not for any further data processing. All further data processing is done by the query interpreter and is outside the responsibility of `IStorage`.
But there are notable exceptions:
- The AST query is passed to the `read` method and the table engine can use it to derive index usage and to read less data from a table.
- Sometimes the table engine can process data itself to a specific stage. For example, `StorageDistributed` can send a query to remote servers, ask them to process data to a stage where data from different remote servers can be merged, and return that preprocessed data.
The query interpreter then finishes processing the data.
The table's `read` method can return multiple `IBlockInputStream` objects to allow parallel data processing. These multiple block input streams can read from a table in parallel. Then you can wrap these streams with various transformations (such as expression evaluation or filtering) that can be calculated independently and create a `UnionBlockInputStream` on top of them, to read from multiple streams in parallel.
There are also `TableFunction`s. These are functions that return a temporary `IStorage` object to use in the `FROM` clause of a query.
To get a quick idea of how to implement your own table engine, look at something simple, like `StorageMemory` or `StorageTinyLog`.
> As the result of the `read` method, `IStorage` returns `QueryProcessingStage` – information about what parts of the query were already calculated inside storage. Currently we have only very coarse granularity for that information. There is no way for the storage to say "I have already processed this part of the expression in WHERE, for this range of data". We need to work on that.
## Parsers
A query is parsed by a hand-written recursive descent parser. For example, `ParserSelectQuery` just recursively calls the underlying parsers for various parts of the query. Parsers create an `AST`. The `AST` is represented by nodes, which are instances of `IAST`.
> Parser generators are not used for historical reasons.
## Interpreters
Interpreters are responsible for creating the query execution pipeline from an `AST`. There are simple interpreters, such as `InterpreterExistsQuery`and `InterpreterDropQuery`, or the more sophisticated `InterpreterSelectQuery`. The query execution pipeline is a combination of block input or output streams. For example, the result of interpreting the `SELECT` query is the `IBlockInputStream` to read the result set from; the result of the INSERT query is the `IBlockOutputStream` to write data for insertion to; and the result of interpreting the `INSERT SELECT` query is the `IBlockInputStream` that returns an empty result set on the first read, but that copies data from `SELECT` to `INSERT` at the same time.
`InterpreterSelectQuery` uses `ExpressionAnalyzer` and `ExpressionActions` machinery for query analysis and transformations. This is where most rule-based query optimizations are done. `ExpressionAnalyzer` is quite messy and should be rewritten: various query transformations and optimizations should be extracted to separate classes to allow modular transformations or query.
## Functions
There are ordinary functions and aggregate functions. For aggregate functions, see the next section.
Ordinary functions don't change the number of rows – they work as if they are processing each row independently. In fact, functions are not called for individual rows, but for `Block`'s of data to implement vectorized query execution.
There are some miscellaneous functions, like `blockSize`, `rowNumberInBlock`, and `runningAccumulate`, that exploit block processing and violate the independence of rows.
ClickHouse has strong typing, so implicit type conversion doesn't occur. If a function doesn't support a specific combination of types, an exception will be thrown. But functions can work (be overloaded) for many different combinations of types. For example, the `plus` function (to implement the `+` operator) works for any combination of numeric types: `UInt8` + `Float32`, `UInt16` + `Int8`, and so on. Also, some variadic functions can accept any number of arguments, such as the `concat` function.
Implementing a function may be slightly inconvenient because a function explicitly dispatches supported data types and supported `IColumns`. For example, the `plus` function has code generated by instantiation of a C++ template for each combination of numeric types, and for constant or non-constant left and right arguments.
> This is a nice place to implement runtime code generation to avoid template code bloat. Also, it will make it possible to add fused functions like fused multiply-add, or to make multiple comparisons in one loop iteration.
Due to vectorized query execution, functions are not short-circuit. For example, if you write `WHERE f(x) AND g(y)`, both sides will be calculated, even for rows, when `f(x)` is zero (except when `f(x)` is a zero constant expression). But if selectivity of the `f(x)` condition is high, and calculation of `f(x)` is much cheaper than `g(y)`, it's better to implement multi-pass calculation: first calculate `f(x)`, then filter columns by the result, and then calculate `g(y)` only for smaller, filtered chunks of data.
## Aggregate Functions
Aggregate functions are stateful functions. They accumulate passed values into some state, and allow you to get results from that state. They are managed with the `IAggregateFunction` interface. States can be rather simple (the state for `AggregateFunctionCount` is just a single `UInt64` value) or quite complex (the state of `AggregateFunctionUniqCombined` is a combination of a linear array, a hash table and a `HyperLogLog` probabilistic data structure).
To deal with multiple states while executing a high-cardinality `GROUP BY` query, states are allocated in `Arena` (a memory pool), or they could be allocated in any suitable piece of memory. States can have a non-trivial constructor and destructor: for example, complex aggregation states can allocate additional memory themselves. This requires some attention to creating and destroying states and properly passing their ownership, to keep track of who and when will destroy states.
Aggregation states can be serialized and deserialized to pass over the network during distributed query execution or to write them on disk where there is not enough RAM. They can even be stored in a table with the `DataTypeAggregateFunction` to allow incremental aggregation of data.
> The serialized data format for aggregate function states is not versioned right now. This is ok if aggregate states are only stored temporarily. But we have the `AggregatingMergeTree` table engine for incremental aggregation, and people are already using it in production. This is why we should add support for backward compatibility when changing the serialized format for any aggregate function in the future.
## Server
The server implements several different interfaces:
- An HTTP interface for any foreign clients.
- A TCP interface for the native ClickHouse client and for cross-server communication during distributed query execution.
- An interface for transferring data for replication.
Internally, it is just a basic multithreaded server without coroutines, fibers, etc. Since the server is not designed to process a high rate of simple queries but is intended to process a relatively low rate of complex queries, each of them can process a vast amount of data for analytics.
The server initializes the `Context` class with the necessary environment for query execution: the list of available databases, users and access rights, settings, clusters, the process list, the query log, and so on. This environment is used by interpreters.
We maintain full backward and forward compatibility for the server TCP protocol: old clients can talk to new servers and new clients can talk to old servers. But we don't want to maintain it eternally, and we are removing support for old versions after about one year.
> For all external applications, we recommend using the HTTP interface because it is simple and easy to use. The TCP protocol is more tightly linked to internal data structures: it uses an internal format for passing blocks of data and it uses custom framing for compressed data. We haven't released a C library for that protocol because it requires linking most of the ClickHouse codebase, which is not practical.
## Distributed query execution
Servers in a cluster setup are mostly independent. You can create a `Distributed` table on one or all servers in a cluster. The `Distributed` table does not store data itself – it only provides a "view" to all local tables on multiple nodes of a cluster. When you SELECT from a `Distributed` table, it rewrites that query, chooses remote nodes according to load balancing settings, and sends the query to them. The `Distributed` table requests remote servers to process a query just up to a stage where intermediate results from different servers can be merged. Then it receives the intermediate results and merges them. The distributed table tries to distribute as much work as possible to remote servers, and does not send much intermediate data over the network.
> Things become more complicated when you have subqueries in IN or JOIN clauses and each of them uses a `Distributed` table. We have different strategies for execution of these queries.
There is no global query plan for distributed query execution. Each node has its own local query plan for its part of the job. We only have simple one-pass distributed query execution: we send queries for remote nodes and then merge the results. But this is not feasible for difficult queries with high cardinality GROUP BYs or with a large amount of temporary data for JOIN: in such cases, we need to "reshuffle" data between servers, which requires additional coordination. ClickHouse does not support that kind of query execution, and we need to work on it.
## Merge Tree
`MergeTree` is a family of storage engines that supports indexing by primary key. The primary key can be an arbitary tuple of columns or expressions. Data in a `MergeTree` table is stored in "parts". Each part stores data in the primary key order (data is ordered lexicographically by the primary key tuple). All the table columns are stored in separate `column.bin` files in these parts. The files consist of compressed blocks. Each block is usually from 64 KB to 1 MB of uncompressed data, depending on the average value size. The blocks consist of column values placed contiguously one after the other. Column values are in the same order for each column (the order is defined by the primary key), so when you iterate by many columns, you get values for the corresponding rows.
The primary key itself is "sparse". It doesn't address each single row, but only some ranges of data. A separate `primary.idx` file has the value of the primary key for each N-th row, where N is called `index_granularity` (usually, N = 8192). Also, for each column, we have `column.mrk` files with "marks," which are offsets to each N-th row in the data file. Each mark is a pair: the offset in the file to the beginning of the compressed block, and the offset in the decompressed block to the beginning of data. Usually compressed blocks are aligned by marks, and the offset in the decompressed block is zero. Data for `primary.idx` always resides in memory and data for `column.mrk` files is cached.
When we are going to read something from a part in `MergeTree`, we look at `primary.idx` data and locate ranges that could possibly contain requested data, then look at `column.mrk` data and calculate offsets for where to start reading those ranges. Because of sparseness, excess data may be read. ClickHouse is not suitable for a high load of simple point queries, because the entire range with `index_granularity` rows must be read for each key, and the entire compressed block must be decompressed for each column. We made the index sparse because we must be able to maintain trillions of rows per single server without noticeable memory consumption for the index. Also, because the primary key is sparse, it is not unique: it cannot check the existence of the key in the table at INSERT time. You could have many rows with the same key in a table.
When you `INSERT` a bunch of data into `MergeTree`, that bunch is sorted by primary key order and forms a new part. To keep the number of parts relatively low, there are background threads that periodically select some parts and merge them to a single sorted part. That's why it is called `MergeTree`. Of course, merging leads to "write amplification". All parts are immutable: they are only created and deleted, but not modified. When SELECT is run, it holds a snapshot of the table (a set of parts). After merging, we also keep old parts for some time to make recovery after failure easier, so if we see that some merged part is probably broken, we can replace it with its source parts.
`MergeTree` is not an LSM tree because it doesn't contain "memtable" and "log": inserted data is written directly to the filesystem. This makes it suitable only to INSERT data in batches, not by individual row and not very frequently – about once per second is ok, but a thousand times a second is not. We did it this way for simplicity's sake, and because we are already inserting data in batches in our applications.
> MergeTree tables can only have one (primary) index: there aren't any secondary indices. It would be nice to allow multiple physical representations under one logical table, for example, to store data in more than one physical order or even to allow representations with pre-aggregated data along with original data.
There are MergeTree engines that are doing additional work during background merges. Examples are `CollapsingMergeTree` and `AggregatingMergeTree`. This could be treated as special support for updates. Keep in mind that these are not real updates because users usually have no control over the time when background merges will be executed, and data in a `MergeTree` table is almost always stored in more than one part, not in completely merged form.
## Replication
Replication in ClickHouse is implemented on a per-table basis. You could have some replicated and some non-replicated tables on the same server. You could also have tables replicated in different ways, such as one table with two-factor replication and another with three-factor.
Replication is implemented in the `ReplicatedMergeTree` storage engine. The path in `ZooKeeper` is specified as a parameter for the storage engine. All tables with the same path in `ZooKeeper` become replicas of each other: they synchronize their data and maintain consistency. Replicas can be added and removed dynamically simply by creating or dropping a table.
Replication uses an asynchronous multi-master scheme. You can insert data into any replica that has a session with `ZooKeeper`, and data is replicated to all other replicas asynchronously. Because ClickHouse doesn't support UPDATEs, replication is conflict-free. As there is no quorum acknowledgment of inserts, just-inserted data might be lost if one node fails.
Metadata for replication is stored in ZooKeeper. There is a replication log that lists what actions to do. Actions are: get part; merge parts; drop partition, etc. Each replica copies the replication log to its queue and then executes the actions from the queue. For example, on insertion, the "get part" action is created in the log, and every replica downloads that part. Merges are coordinated between replicas to get byte-identical results. All parts are merged in the same way on all replicas. To achieve this, one replica is elected as the leader, and that replica initiates merges and writes "merge parts" actions to the log.
Replication is physical: only compressed parts are transferred between nodes, not queries. To lower the network cost (to avoid network amplification), merges are processed on each replica independently in most cases. Large merged parts are sent over the network only in cases of significant replication lag.
In addition, each replica stores its state in ZooKeeper as the set of parts and its checksums. When the state on the local filesystem diverges from the reference state in ZooKeeper, the replica restores its consistency by downloading missing and broken parts from other replicas. When there is some unexpected or broken data in the local filesystem, ClickHouse does not remove it, but moves it to a separate directory and forgets it.
> The ClickHouse cluster consists of independent shards, and each shard consists of replicas. The cluster is not elastic, so after adding a new shard, data is not rebalanced between shards automatically. Instead, the cluster load will be uneven. This implementation gives you more control, and it is fine for relatively small clusters such as tens of nodes. But for clusters with hundreds of nodes that we are using in production, this approach becomes a significant drawback. We should implement a table engine that will span its data across the cluster with dynamically replicated regions that could be split and balanced between clusters automatically.

227
docs/en/development/architecture.rst
View File

@ -1,227 +0,0 @@
Overview of ClickHouse architecture
===================================
ClickHouse is a true column-oriented DBMS. Data is stored by columns, and during query execution data is processed by arrays (vectors or chunks of columns). Whenever possible, operations are dispatched on arrays, rather than on individual values. This is called "vectorized query execution," and it helps lower dispatch cost relative to the cost of actual data processing.
This idea is nothing new. It dates back to the ``APL`` programming language and its descendants: ``A+``, ``J``, ``K``, and ``Q``. Array programming is widely used in scientific data processing. Neither is this idea something new in relational databases: for example, it is used in the ``Vectorwise`` system.
There are two different approaches for speeding up query processing: vectorized query execution and runtime code generation. In the latter, the code is generated for every kind of query on the fly, removing all indirection and dynamic dispatch. Neither of these approaches is strictly better than the other. Runtime code generation can be better when it fuses many operations together, thus fully utilizing CPU execution units and the pipeline. Vectorized query execution can be less practical, because it involves temporary vectors that must be written to cache and read back. If the temporary data does not fit in the L2 cache, this becomes an issue. But vectorized query execution more easily utilizes the SIMD capabilities of the CPU. A `research paper <http://15721.courses.cs.cmu.edu/spring2016/papers/p5-sompolski.pdf>`_ written by our friends shows that it is better to combine both approaches. ClickHouse mainly uses vectorized query execution and has limited initial support for runtime code generation (only the inner loop of the first stage of GROUP BY can be compiled).
Columns
-------
To represent columns in memory (actually, chunks of columns), the ``IColumn`` interface is used. This interface provides helper methods for implementation of various relational operators. Almost all operations are immutable: they do not modify the original column, but create a new modified one. For example, the ``IColumn::filter`` method accepts a filter byte mask and creates a new filtered column. It is used for the ``WHERE`` and ``HAVING`` relational operators. Additional examples: the ``IColumn::permute`` method to support ``ORDER BY``, the ``IColumn::cut`` method to support ``LIMIT``, and so on.
Various ``IColumn`` implementations (``ColumnUInt8``, ``ColumnString`` and so on) are responsible for the memory layout of columns. Memory layout is usually a contiguous array. For the integer type of columns it is just one contiguous array, like ``std::vector``. For ``String`` and ``Array`` columns, it is two vectors: one for all array elements, placed contiguously, and a second one for offsets to the beginning of each array. There is also ``ColumnConst`` that stores just one value in memory, but looks like a column.
Field
-----
Nevertheless, it is possible to work with individual values as well. To represent an individual value, ``Field`` is used. ``Field`` is just a discriminated union of ``UInt64``, ``Int64``, ``Float64``, ``String`` and ``Array``. ``IColumn`` has the ``operator[]`` method to get the n-th value as a ``Field``, and the ``insert`` method to append a ``Field`` to the end of a column. These methods are not very efficient, because they require dealing with temporary ``Field`` objects representing an individual value. There are more efficient methods, such as ``insertFrom``, ``insertRangeFrom``, and so on.
``Field`` doesn't have enough information about a specific data type for a table. For example, ``UInt8``, ``UInt16``, ``UInt32``, and ``UInt64`` are all represented as ``UInt64`` in a ``Field``.
Leaky abstractions
------------------
``IColumn`` has methods for common relational transformations of data, but they don't meet all needs. For example, ``ColumnUInt64`` doesn't have a method to calculate the sum of two columns, and ``ColumnString`` doesn't have a method to run a substring search. These countless routines are implemented outside of ``IColumn``.
Various functions on columns can be implemented in a generic, non-efficient way using ``IColumn`` methods to extract ``Field`` values, or in a specialized way using knowledge of inner memory layout of data in a specific ``IColumn`` implementation. To do this, functions are cast to a specific ``IColumn`` type and deal with internal representation directly. For example, ``ColumnUInt64`` has the ``getData`` method that returns a reference to an internal array, then a separate routine reads or fills that array directly. In fact, we have "leaky abstractions" to allow efficient specializations of various routines.
Data types
----------
``IDataType`` is responsible for serialization and deserialization: for reading and writing chunks of columns or individual values in binary or text form.
``IDataType`` directly corresponds to data types in tables. For example, there are ``DataTypeUInt32``, ``DataTypeDateTime``, ``DataTypeString`` and so on.
``IDataType`` and ``IColumn`` are only loosely related to each other. Different data types can be represented in memory by the same ``IColumn`` implementations. For example, ``DataTypeUInt32`` and ``DataTypeDateTime`` are both represented by ``ColumnUInt32`` or ``ColumnConstUInt32``. In addition, the same data type can be represented by different ``IColumn`` implementations. For example, ``DataTypeUInt8`` can be represented by ``ColumnUInt8`` or ``ColumnConstUInt8``.
``IDataType`` only stores metadata. For instance, ``DataTypeUInt8`` doesn't store anything at all (except vptr) and ``DataTypeFixedString`` stores just ``N`` (the size of fixed-size strings).
``IDataType`` has helper methods for various data formats. Examples are methods to serialize a value with possible quoting, to serialize a value for JSON, and to serialize a value as part of XML format. There is no direct correspondence to data formats. For example, the different data formats ``Pretty`` and ``TabSeparated`` can use the same ``serializeTextEscaped`` helper method from the ``IDataType`` interface.
Block
-----
A ``Block`` is a container that represents a subset (chunk) of a table in memory. It is just a set of triples: ``(IColumn, IDataType, column name)``. During query execution, data is processed by ``Block`` s. If we have a ``Block``, we have data (in the ``IColumn`` object), we have information about its type (in ``IDataType``) that tells us how to deal with that column, and we have the column name (either the original column name from the table, or some artificial name assigned for getting temporary results of calculations).
When we calculate some function over columns in a block, we add another column with its result to the block, and we don't touch columns for arguments of the function because operations are immutable. Later, unneeded columns can be removed from the block, but not modified. This is convenient for elimination of common subexpressions.
Blocks are created for every processed chunk of data. Note that for the same type of calculation, the column names and types remain the same for different blocks, and only column data changes. It is better to split block data from the block header, because small block sizes will have a high overhead of temporary strings for copying shared_ptrs and column names.
Block Streams
-------------
Block streams are for processing data. We use streams of blocks to read data from somewhere, perform data transformations, or write data to somewhere. ``IBlockInputStream`` has the ``read`` method to fetch the next block while available. ``IBlockOutputStream`` has the ``write`` method to push the block somewhere.
Streams are responsible for:
#. Reading or writing to a table. The table just returns a stream for reading or writing blocks.
#. Implementing data formats. For example, if you want to output data to a terminal in ``Pretty`` format, you create a block output stream where you push blocks, and it formats them.
#. Performing data transformations. Let's say you have ``IBlockInputStream`` and want to create a filtered stream. You create ``FilterBlockInputStream`` and initialize it with your stream. Then when you pull a block from ``FilterBlockInputStream``, it pulls a block from your stream, filters it, and returns the filtered block to you. Query execution pipelines are represented this way.
There are more sophisticated transformations. For example, when you pull from ``AggregatingBlockInputStream``, it reads all data from its source, aggregates it, and then returns a stream of aggregated data for you. Another example: ``UnionBlockInputStream`` accepts many input sources in the constructor and also a number of threads. It launches multiple threads and reads from multiple sources in parallel.
Block streams use the "pull" approach to control flow: when you pull a block from the first stream, it consequently pulls the required blocks from nested streams, and the entire execution pipeline will work. Neither "pull" nor "push" is the best solution, because control flow is implicit, and that limits implementation of various features like simultaneous execution of multiple queries (merging many pipelines together). This limitation could be overcome with coroutines or just running extra threads that wait for each other. We may have more possibilities if we make control flow explicit: if we locate the logic for passing data from one calculation unit to another outside of those calculation units. Read this `nice article <http://journal.stuffwithstuff.com/2013/01/13/iteration-inside-and-out/>`_ for more thoughts.
We should note that the query execution pipeline creates temporary data at each step. We try to keep block size small enough so that temporary data fits in the CPU cache. With that assumption, writing and reading temporary data is almost free in comparison with other calculations. We could consider an alternative, which is to fuse many operations in the pipeline together, to make the pipeline as short as possible and remove much of the temporary data. This could be an advantage, but it also has drawbacks. For example, a split pipeline makes it easy to implement caching intermediate data, stealing intermediate data from similar queries running at the same time, and merging pipelines for similar queries.
Formats
-------
Data formats are implemented with block streams. There are "presentational" formats only suitable for output of data to the client, such as ``Pretty`` format, which provides only ``IBlockOutputStream``. And there are input/output formats, such as ``TabSeparated`` or ``JSONEachRow``.
There are also row streams: ``IRowInputStream`` and ``IRowOutputStream``. They allow you to pull/push data by individual rows, not by blocks. And they are only needed to simplify implementation of row-oriented formats. The wrappers ``BlockInputStreamFromRowInputStream`` and ``BlockOutputStreamFromRowOutputStream`` allow you to convert row-oriented streams to regular block-oriented streams.
I/O
---
For byte-oriented input/output, there are ``ReadBuffer`` and ``WriteBuffer`` abstract classes. They are used instead of C++ ``iostream``'s. Don't worry: every mature C++ project is using something other than ``iostream``'s for good reasons.
``ReadBuffer`` and ``WriteBuffer`` are just a contiguous buffer and a cursor pointing to the position in that buffer. Implementations may own or not own the memory for the buffer. There is a virtual method to fill the buffer with the following data (for ``ReadBuffer``) or to flush the buffer somewhere (for ``WriteBuffer``). The virtual methods are rarely called.
Implementations of ``ReadBuffer``/``WriteBuffer`` are used for working with files and file descriptors and network sockets, for implementing compression (``CompressedWriteBuffer`` is initialized with another WriteBuffer and performs compression before writing data to it), and for other purposes – the names ``ConcatReadBuffer``, ``LimitReadBuffer``, and ``HashingWriteBuffer`` speak for themselves.
Read/WriteBuffers only deal with bytes. To help with formatted input/output (for instance, to write a number in decimal format), there are functions from ``ReadHelpers`` and ``WriteHelpers`` header files.
Let's look at what happens when you want to write a result set in ``JSON`` format to stdout. You have a result set ready to be fetched from ``IBlockInputStream``. You create ``WriteBufferFromFileDescriptor(STDOUT_FILENO)`` to write bytes to stdout. You create ``JSONRowOutputStream``, initialized with that ``WriteBuffer``, to write rows in ``JSON`` to stdout. You create ``BlockOutputStreamFromRowOutputStream`` on top of it, to represent it as ``IBlockOutputStream``. Then you call ``copyData`` to transfer data from ``IBlockInputStream`` to ``IBlockOutputStream``, and everything works. Internally, ``JSONRowOutputStream`` will write various JSON delimiters and call the ``IDataType::serializeTextJSON`` method with a reference to ``IColumn`` and the row number as arguments. Consequently, ``IDataType::serializeTextJSON`` will call a method from ``WriteHelpers.h``: for example, ``writeText`` for numeric types and ``writeJSONString`` for ``DataTypeString``.
Tables
------
Tables are represented by the ``IStorage`` interface. Different implementations of that interface are different table engines. Examples are ``StorageMergeTree``, ``StorageMemory``, and so on. Instances of these classes are just tables.
The most important ``IStorage`` methods are ``read`` and ``write``. There are also ``alter``, ``rename``, ``drop``, and so on. The ``read`` method accepts the following arguments: the set of columns to read from a table, the ``AST`` query to consider, and the desired number of streams to return. It returns one or multiple ``IBlockInputStream`` objects and information about the stage of data processing that was completed inside a table engine during query execution.
In most cases, the read method is only responsible for reading the specified columns from a table, not for any further data processing. All further data processing is done by the query interpreter and is outside the responsibility of ``IStorage``.
But there are notable exceptions:
- The AST query is passed to the ``read`` method and the table engine can use it to derive index usage and to read less data from a table.
- Sometimes the table engine can process data itself to a specific stage. For example, ``StorageDistributed`` can send a query to remote servers, ask them to process data to a stage where data from different remote servers can be merged, and return that preprocessed data.
The query interpreter then finishes processing the data.
The table's ``read`` method can return multiple ``IBlockInputStream`` objects to allow parallel data processing. These multiple block input streams can read from a table in parallel. Then you can wrap these streams with various transformations (such as expression evaluation or filtering) that can be calculated independently and create a ``UnionBlockInputStream`` on top of them, to read from multiple streams in parallel.
There are also ``TableFunction`` s. These are functions that return a temporary ``IStorage`` object to use in the ``FROM`` clause of a query.
To get a quick idea of how to implement your own table engine, look at something simple, like ``StorageMemory`` or ``StorageTinyLog``.
As the result of the ``read`` method, ``IStorage`` returns ``QueryProcessingStage`` – information about what parts of the query were already calculated inside storage. Currently we have only very coarse granularity for that information. There is no way for the storage to say "I have already processed this part of the expression in WHERE, for this range of data". We need to work on that.
Parsers
-------
A query is parsed by a hand-written recursive descent parser. For example, ``ParserSelectQuery`` just recursively calls the underlying parsers for various parts of the query. Parsers create an ``AST``. The ``AST`` is represented by nodes, which are instances of ``IAST``.
Parser generators are not used for historical reasons.
Interpreters
------------
Interpreters are responsible for creating the query execution pipeline from an ``AST``. There are simple interpreters, such as ``InterpreterExistsQuery`` and ``InterpreterDropQuery``, or the more sophisticated ``InterpreterSelectQuery``. The query execution pipeline is a combination of block input or output streams. For example, the result of interpreting the ``SELECT`` query is the ``IBlockInputStream`` to read the result set from; the result of the INSERT query is the ``IBlockOutputStream`` to write data for insertion to; and the result of interpreting the ``INSERT SELECT`` query is the ``IBlockInputStream`` that returns an empty result set on the first read, but that copies data from ``SELECT`` to ``INSERT`` at the same time.
``InterpreterSelectQuery`` uses ``ExpressionAnalyzer`` and ``ExpressionActions`` machinery for query analysis and transformations. This is where most rule-based query optimizations are done. ``ExpressionAnalyzer`` is quite messy and should be rewritten: various query transformations and optimizations should be extracted to separate classes to allow modular transformations or query.
Functions
---------
There are ordinary functions and aggregate functions. For aggregate functions, see the next section.
Ordinary functions don't change the number of rows – they work as if they are processing each row independently. In fact, functions are not called for individual rows, but for ``Block``'s of data to implement vectorized query execution.
There are some miscellaneous functions, like ``blockSize``, ``rowNumberInBlock``, and ``runningAccumulate``, that exploit block processing and violate the independence of rows.
ClickHouse has strong typing, so implicit type conversion doesn't occur. If a function doesn't support a specific combination of types, an exception will be thrown. But functions can work (be overloaded) for many different combinations of types. For example, the ``plus`` function (to implement the ``+`` operator) works for any combination of numeric types: ``UInt8`` + ``Float32``, ``UInt16`` + ``Int8``, and so on. Also, some variadic functions can accept any number of arguments, such as the ``concat`` function.
Implementing a function may be slightly inconvenient because a function explicitly dispatches supported data types and supported ``IColumns``. For example, the ``plus`` function has code generated by instantiation of a C++ template for each combination of numeric types, and for constant or non-constant left and right arguments.
This is a nice place to implement runtime code generation to avoid template code bloat. Also, it will make it possible to add fused functions like fused multiply-add, or to make multiple comparisons in one loop iteration.
Due to vectorized query execution, functions are not short-circuit. For example, if you write ``WHERE f(x) AND g(y)``, both sides will be calculated, even for rows, when ``f(x)`` is zero (except when ``f(x)`` is a zero constant expression). But if selectivity of the ``f(x)`` condition is high, and calculation of ``f(x)`` is much cheaper than ``g(y)``, it's better to implement multi-pass calculation: first calculate ``f(x)``, then filter columns by the result, and then calculate ``g(y)`` only for smaller, filtered chunks of data.
Aggregate Functions
-------------------
Aggregate functions are stateful functions. They accumulate passed values into some state, and allow you to get results from that state. They are managed with the ``IAggregateFunction`` interface. States can be rather simple (the state for ``AggregateFunctionCount`` is just a single ``UInt64`` value) or quite complex (the state of ``AggregateFunctionUniqCombined`` is a combination of a linear array, a hash table and a ``HyperLogLog`` probabilistic data structure).
To deal with multiple states while executing a high-cardinality ``GROUP BY`` query, states are allocated in ``Arena`` (a memory pool), or they could be allocated in any suitable piece of memory. States can have a non-trivial constructor and destructor: for example, complex aggregation states can allocate additional memory themselves. This requires some attention to creating and destroying states and properly passing their ownership, to keep track of who and when will destroy states.
Aggregation states can be serialized and deserialized to pass over the network during distributed query execution or to write them on disk where there is not enough RAM. They can even be stored in a table with the ``DataTypeAggregateFunction`` to allow incremental aggregation of data.
The serialized data format for aggregate function states is not versioned right now. This is ok if aggregate states are only stored temporarily. But we have the ``AggregatingMergeTree`` table engine for incremental aggregation, and people are already using it in production. This is why we should add support for backward compatibility when changing the serialized format for any aggregate function in the future.
Server
------
The server implements several different interfaces:
- An HTTP interface for any foreign clients.
- A TCP interface for the native ClickHouse client and for cross-server communication during distributed query execution.
- An interface for transferring data for replication.
Internally, it is just a basic multithreaded server without coroutines, fibers, etc. Since the server is not designed to process a high rate of simple queries but is intended to process a relatively low rate of complex queries, each of them can process a vast amount of data for analytics.
The server initializes the ``Context`` class with the necessary environment for query execution: the list of available databases, users and access rights, settings, clusters, the process list, the query log, and so on. This environment is used by interpreters.
We maintain full backward and forward compatibility for the server TCP protocol: old clients can talk to new servers and new clients can talk to old servers. But we don't want to maintain it eternally, and we are removing support for old versions after about one year.
For all external applications, we recommend using the HTTP interface because it is simple and easy to use. The TCP protocol is more tightly linked to internal data structures: it uses an internal format for passing blocks of data and it uses custom framing for compressed data. We haven't released a C library for that protocol because it requires linking most of the ClickHouse codebase, which is not practical.
Distributed query execution
---------------------------
Servers in a cluster setup are mostly independent. You can create a ``Distributed`` table on one or all servers in a cluster. The ``Distributed`` table does not store data itself – it only provides a "view" to all local tables on multiple nodes of a cluster. When you SELECT from a ``Distributed`` table, it rewrites that query, chooses remote nodes according to load balancing settings, and sends the query to them. The ``Distributed`` table requests remote servers to process a query just up to a stage where intermediate results from different servers can be merged. Then it receives the intermediate results and merges them. The distributed table tries to distribute as much work as possible to remote servers, and does not send much intermediate data over the network.
Things become more complicated when you have subqueries in IN or JOIN clauses and each of them uses a ``Distributed`` table. We have different strategies for execution of these queries.
There is no global query plan for distributed query execution. Each node has its own local query plan for its part of the job. We only have simple one-pass distributed query execution: we send queries for remote nodes and then merge the results. But this is not feasible for difficult queries with high cardinality GROUP BYs or with a large amount of temporary data for JOIN: in such cases, we need to "reshuffle" data between servers, which requires additional coordination. ClickHouse does not support that kind of query execution, and we need to work on it.
Merge Tree
----------
``MergeTree`` is a family of storage engines that supports indexing by primary key. The primary key can be an arbitary tuple of columns or expressions. Data in a ``MergeTree`` table is stored in "parts". Each part stores data in the primary key order (data is ordered lexicographically by the primary key tuple). All the table columns are stored in separate ``column.bin`` files in these parts. The files consist of compressed blocks. Each block is usually from 64 KB to 1 MB of uncompressed data, depending on the average value size. The blocks consist of column values placed contiguously one after the other. Column values are in the same order for each column (the order is defined by the primary key), so when you iterate by many columns, you get values for the corresponding rows.
The primary key itself is "sparse". It doesn't address each single row, but only some ranges of data. A separate ``primary.idx`` file has the value of the primary key for each N-th row, where N is called ``index_granularity`` (usually, N = 8192). Also, for each column, we have ``column.mrk`` files with "marks," which are offsets to each N-th row in the data file. Each mark is a pair: the offset in the file to the beginning of the compressed block, and the offset in the decompressed block to the beginning of data. Usually compressed blocks are aligned by marks, and the offset in the decompressed block is zero. Data for ``primary.idx`` always resides in memory and data for ``column.mrk`` files is cached.
When we are going to read something from a part in ``MergeTree``, we look at ``primary.idx`` data and locate ranges that could possibly contain requested data, then look at ``column.mrk`` data and calculate offsets for where to start reading those ranges. Because of sparseness, excess data may be read. ClickHouse is not suitable for a high load of simple point queries, because the entire range with ``index_granularity`` rows must be read for each key, and the entire compressed block must be decompressed for each column. We made the index sparse because we must be able to maintain trillions of rows per single server without noticeable memory consumption for the index. Also, because the primary key is sparse, it is not unique: it cannot check the existence of the key in the table at INSERT time. You could have many rows with the same key in a table.
When you ``INSERT`` a bunch of data into ``MergeTree``, that bunch is sorted by primary key order and forms a new part. To keep the number of parts relatively low, there are background threads that periodically select some parts and merge them to a single sorted part. That's why it is called ``MergeTree``. Of course, merging leads to "write amplification". All parts are immutable: they are only created and deleted, but not modified. When SELECT is run, it holds a snapshot of the table (a set of parts). After merging, we also keep old parts for some time to make recovery after failure easier, so if we see that some merged part is probably broken, we can replace it with its source parts.
``MergeTree`` is not an LSM tree because it doesn't contain "memtable" and "log": inserted data is written directly to the filesystem. This makes it suitable only to INSERT data in batches, not by individual row and not very frequently – about once per second is ok, but a thousand times a second is not. We did it this way for simplicity's sake, and because we are already inserting data in batches in our applications.
MergeTree tables can only have one (primary) index: there aren't any secondary indices. It would be nice to allow multiple physical representations under one logical table, for example, to store data in more than one physical order or even to allow representations with pre-aggregated data along with original data.
There are MergeTree engines that are doing additional work during background merges. Examples are ``CollapsingMergeTree`` and ``AggregatingMergeTree``. This could be treated as special support for updates. Keep in mind that these are not real updates because users usually have no control over the time when background merges will be executed, and data in a ``MergeTree`` table is almost always stored in more than one part, not in completely merged form.
Replication
-----------
Replication in ClickHouse is implemented on a per-table basis. You could have some replicated and some non-replicated tables on the same server. You could also have tables replicated in different ways, such as one table with two-factor replication and another with three-factor.
Replication is implemented in the ``ReplicatedMergeTree`` storage engine. The path in ``ZooKeeper`` is specified as a parameter for the storage engine. All tables with the same path in ``ZooKeeper`` become replicas of each other: they synchronise their data and maintain consistency. Replicas can be added and removed dynamically simply by creating or dropping a table.
Replication uses an asynchronous multi-master scheme. You can insert data into any replica that has a session with ``ZooKeeper``, and data is replicated to all other replicas asynchronously. Because ClickHouse doesn't support UPDATEs, replication is conflict-free. As there is no quorum acknowledgment of inserts, just-inserted data might be lost if one node fails.
Metadata for replication is stored in ZooKeeper. There is a replication log that lists what actions to do. Actions are: get part; merge parts; drop partition, etc. Each replica copies the replication log to its queue and then executes the actions from the queue. For example, on insertion, the "get part" action is created in the log, and every replica downloads that part. Merges are coordinated between replicas to get byte-identical results. All parts are merged in the same way on all replicas. To achieve this, one replica is elected as the leader, and that replica initiates merges and writes "merge parts" actions to the log.
Replication is physical: only compressed parts are transferred between nodes, not queries. To lower the network cost (to avoid network amplification), merges are processed on each replica independently in most cases. Large merged parts are sent over the network only in cases of significant replication lag.
In addition, each replica stores its state in ZooKeeper as the set of parts and its checksums. When the state on the local filesystem diverges from the reference state in ZooKeeper, the replica restores its consistency by downloading missing and broken parts from other replicas. When there is some unexpected or broken data in the local filesystem, ClickHouse does not remove it, but moves it to a separate directory and forgets it.
The ClickHouse cluster consists of independent shards, and each shard consists of replicas. The cluster is not elastic, so after adding a new shard, data is not rebalanced between shards automatically. Instead, the cluster load will be uneven. This implementation gives you more control, and it is fine for relatively small clusters such as tens of nodes. But for clusters with hundreds of nodes that we are using in production, this approach becomes a significant drawback. We should implement a table engine that will span its data across the cluster with dynamically replicated regions that could be split and balanced between clusters automatically.

128
docs/en/development/build.md Normal file
View File

@ -0,0 +1,128 @@
# How to build ClickHouse on Linux
Build should work on Linux Ubuntu 12.04, 14.04 or newer.
With appropriate changes, it should also work on any other Linux distribution.
The build process is not intended to work on Mac OS X.
Only x86_64 with SSE 4.2 is supported. Support for AArch64 is experimental.
To test for SSE 4.2, do
```bash
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
```
## Install Git and CMake
```bash
sudo apt-get install git cmake3
```
Or just cmake on newer systems.
## Detect the number of threads
```bash
export THREADS=$(grep -c ^processor /proc/cpuinfo)
```
## Install GCC 7
There are several ways to do this.
### Install from a PPA package
```bash
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-7 g++-7
```
### Install from sources
Look at [https://github.com/yandex/ClickHouse/blob/master/utils/prepare-environment/install-gcc.sh]
## Use GCC 7 for builds
```bash
export CC=gcc-7
export CXX=g++-7
```
## Install required libraries from packages
```bash
sudo apt-get install libicu-dev libreadline-dev libmysqlclient-dev libssl-dev unixodbc-dev
```
## Checkout ClickHouse sources
To get the latest stable version:
```bash
git clone -b stable --recursive git@github.com:yandex/ClickHouse.git
# or: git clone -b stable --recursive https://github.com/yandex/ClickHouse.git
cd ClickHouse
```
For development, switch to the `master` branch.
For the latest release candidate, switch to the `testing` branch.
## Build ClickHouse
There are two build variants.
### Build release package
Install prerequisites to build Debian packages.
```bash
sudo apt-get install devscripts dupload fakeroot debhelper
```
Install the most recent version of Clang.
Clang is embedded into the ClickHouse package and used at runtime. The minimum version is 5.0. It is optional.
To install clang, see `utils/prepare-environment/install-clang.sh`
You may also build ClickHouse with Clang for development purposes.
For production releases, GCC is used.
Run the release script:
```bash
rm -f ../clickhouse*.deb
./release
```
You will find built packages in the parent directory:
```bash
ls -l ../clickhouse*.deb
```
Note that usage of debian packages is not required.
ClickHouse has no runtime dependencies except libc, so it could work on almost any Linux.
Installing freshly built packages on a development server:
```bash
sudo dpkg -i ../clickhouse*.deb
sudo service clickhouse-server start
```
### Build to work with code
```bash
mkdir build
cd build
cmake ..
make -j $THREADS
cd ..
```
To create an executable, run `make clickhouse`.
This will create the `dbms/src/Server/clickhouse` executable, which can be used with `client` or `server` arguments.

147
docs/en/development/build.rst
View File

@ -1,147 +0,0 @@
How to build ClickHouse on Linux
================================
Build should work on Linux Ubuntu 12.04, 14.04 or newer.
With appropriate changes, build should work on any other Linux distribution.
Build is not intended to work on Mac OS X.
Only x86_64 with SSE 4.2 is supported. Support for AArch64 is experimental.
To test for SSE 4.2, do
.. code-block:: bash
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
Install Git and CMake
---------------------
.. code-block:: bash
sudo apt-get install git cmake3
Or just cmake on newer systems.
Detect number of threads
------------------------
.. code-block:: bash
export THREADS=$(grep -c ^processor /proc/cpuinfo)
Install GCC 7
-------------
There are several ways to do it.
Install from PPA package
~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-7 g++-7
Install from sources
~~~~~~~~~~~~~~~~~~~~
Look at [https://github.com/yandex/ClickHouse/blob/master/utils/prepare-environment/install-gcc.sh]
Use GCC 7 for builds
--------------------
.. code-block:: bash
export CC=gcc-7
export CXX=g++-7
Install required libraries from packages
----------------------------------------
.. code-block:: bash
sudo apt-get install libicu-dev libreadline-dev libmysqlclient-dev libssl-dev unixodbc-dev
Checkout ClickHouse sources
---------------------------
To get latest stable version:
.. code-block:: bash
git clone -b stable --recursive git@github.com:yandex/ClickHouse.git
# or: git clone -b stable --recursive https://github.com/yandex/ClickHouse.git
cd ClickHouse
For development, switch to the ``master`` branch.
For latest release candidate, switch to the ``testing`` branch.
Build ClickHouse
----------------
There are two variants of build.
Build release package
~~~~~~~~~~~~~~~~~~~~~
Install prerequisites to build debian packages.
.. code-block:: bash
sudo apt-get install devscripts dupload fakeroot debhelper
Install recent version of clang.
Clang is embedded into ClickHouse package and used at runtime. Minimum version is 5.0. It is optional.
To install clang, look at ``utils/prepare-environment/install-clang.sh``
You may also build ClickHouse with clang for development purposes.
For production releases, GCC is used.
Run release script:
.. code-block:: bash
rm -f ../clickhouse*.deb
./release
You will find built packages in parent directory:
.. code-block:: bash
ls -l ../clickhouse*.deb
Note that usage of debian packages is not required.
ClickHouse has no runtime dependencies except libc, so it could work on almost any Linux.
Installing just built packages on development server:
.. code-block:: bash
sudo dpkg -i ../clickhouse*.deb
sudo service clickhouse-server start
Build to work with code
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
mkdir build
cd build
cmake ..
make -j $THREADS
cd ..
To create an executable, run ``make clickhouse``.
This will create the ``dbms/src/Server/clickhouse`` executable, which can be used with ``client`` or ``server`` arguments.

45
docs/en/development/build_osx.md Normal file
View File

@ -0,0 +1,45 @@
# How to build ClickHouse on Mac OS X
Build should work on Mac OS X 10.12. If you're using earlier version, you can try to build ClickHouse using Gentoo Prefix and clang sl in this instruction.
With appropriate changes, it should also work on any other Linux distribution.
## Install Homebrew
```bash
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
## Install required compilers, tools, and libraries
```bash
brew install cmake gcc icu4c mysql openssl unixodbc libtool gettext homebrew/dupes/zlib readline boost --cc=gcc-7
```
## Checkout ClickHouse sources
To get the latest stable version:
```bash
git clone -b stable --recursive --depth=10 git@github.com:yandex/ClickHouse.git
# or: git clone -b stable --recursive --depth=10 https://github.com/yandex/ClickHouse.git
cd ClickHouse
```
For development, switch to the `master` branch.
For the latest release candidate, switch to the `testing` branch.
## Build ClickHouse
```bash
mkdir build
cd build
cmake .. -DCMAKE_CXX_COMPILER=`which g++-7` -DCMAKE_C_COMPILER=`which gcc-7`
make -j `sysctl -n hw.ncpu`
cd ..
```
## Caveats
If you intend to run clickhouse-server, make sure to increase the system's maxfiles variable. See [MacOS.md](https://github.com/yandex/ClickHouse/blob/master/MacOS.md) for more details.

63
docs/en/development/build_osx.rst
View File

@ -1,63 +0,0 @@
How to build ClickHouse on Mac OS X
===================================
Build should work on Mac OS X 10.12. If you're using earlier version, you can try to build ClickHouse using Gentoo Prefix and clang sl in this instruction.
With appropriate changes, build should work on any other OS X distribution.
Install Homebrew
----------------
.. code-block:: bash
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Install required compilers, tools, libraries
--------------------------------------------
.. code-block:: bash
brew install cmake gcc icu4c mysql openssl unixodbc libtool gettext homebrew/dupes/zlib readline boost --cc=gcc-7
Checkout ClickHouse sources
---------------------------
To get the latest stable version:
.. code-block:: bash
git clone -b stable --recursive --depth=10 git@github.com:yandex/ClickHouse.git
# or: git clone -b stable --recursive --depth=10 https://github.com/yandex/ClickHouse.git
cd ClickHouse
For development, switch to the ``master`` branch.
For the latest release candidate, switch to the ``testing`` branch.
Build ClickHouse
----------------
.. code-block:: bash
mkdir build
cd build
cmake .. -DCMAKE_CXX_COMPILER=`which g++-7` -DCMAKE_C_COMPILER=`which gcc-7`
make -j `sysctl -n hw.ncpu`
cd ..
If you're using macOS 10.13 High Sierra, it's not possible to build it with GCC-7 due to `a bug <https://github.com/yandex/ClickHouse/issues/1474>`_. Build it with Clang 5.0 instead:
.. code-block:: bash
brew install llvm
mkdir build
cd build
export PATH="/usr/local/opt/llvm/bin:$PATH"
cmake .. -DCMAKE_CXX_COMPILER=`which clang++` -DCMAKE_C_COMPILER=`which clang` -DLINKER_NAME=ld -DUSE_STATIC_LIBRARIES=OFF
make -j `sysctl -n hw.ncpu` clickhouse
Caveats
-------
If you intend to run clickhouse-server, make sure to increase system's maxfiles variable. See `MacOS.md <https://github.com/yandex/ClickHouse/blob/master/MacOS.md>`_ for more details.

9
docs/en/development/index.md Normal file
View File

@ -0,0 +1,9 @@
# ClickHouse Development
```eval_rst
.. toctree::
:glob:
*
```

7
docs/en/development/index.rst
View File

@ -1,7 +0,0 @@
ClickHouse Development
======================
.. toctree::
:glob:
*

813
docs/en/development/style.md Normal file
View File

@ -0,0 +1,813 @@
# How to write C++ code
## General recommendations
1. The following are recommendations, not requirements.
2. If you are editing code, it makes sense to follow the formatting of the existing code.
3. Code style is needed for consistency. Consistency makes it easier to read the code. and it also makes it easier to search the code.
4. Many of the rules do not have logical reasons; they are dictated by established practices.
## Formatting
1. Most of the formatting will be done automatically by `clang-format`.
1. Offsets are 4 spaces. Configure your development environment so that a tab adds four spaces.
1. A left curly bracket must be separated on a new line. (And the right one, as well.)
```cpp
inline void readBoolText(bool & x, ReadBuffer & buf)
{
char tmp = '0';
readChar(tmp, buf);
x = tmp != '0';
}
```
1. But if the entire function body is quite short (a single statement), you can place it entirely on one line if you wish. Place spaces around curly braces (besides the space at the end of the line).
```cpp
inline size_t mask() const { return buf_size() - 1; }
inline size_t place(HashValue x) const { return x & mask(); }
```
1. For functions, don't put spaces around brackets.
```cpp
void reinsert(const Value & x)
```
```cpp
memcpy(&buf[place_value], &x, sizeof(x));
```
1. When using statements such as if, for, and while (unlike function calls), put a space before the opening bracket.
```cpp
for (size_t i = 0; i < rows; i += storage.index_granularity)
```
1. Put spaces around binary operators (+,-, *,/,%, ...), as well as the ternary operator?:.
```cpp
UInt16 year = (s[0] - '0') * 1000 + (s[1] - '0') * 100 + (s[2] - '0') * 10 + (s[3] - '0');
UInt8 month = (s[5] - '0') * 10 + (s[6] - '0');
UInt8 day = (s[8] - '0') * 10 + (s[9] - '0');
```
1. If a line feed is entered, put the operator on a new line and increase the indent before it.
```cpp
if (elapsed_ns)
message << " ("
<< rows_read_on_server * 1000000000 / elapsed_ns << " rows/s., "
<< bytes_read_on_server * 1000.0 / elapsed_ns << " MB/s.) ";
```
1. You can use spaces for alignment within a line, if desired.
```cpp
dst.ClickLogID = click.LogID;
dst.ClickEventID = click.EventID;
dst.ClickGoodEvent = click.GoodEvent;
```
9. Don't use spaces around the operators `.`, `->` .
If necessary, the operator can be wrapped to the next line. In this case, the offset in front of it is increased.
10. Do not use a space to separate unary operators (`-, +, +, *, &`, ...) from the argument.
11. Put a space after a comma, but not before it. The same rule goes for a semicolon inside a for expression.
12. Do not use spaces to separate the `[]` operator.
13. In a `template <...>` expression, use a space between `template`and`<`; no spaces after `<` or before `>`.
```cpp
template <typename TKey, typename TValue>
struct AggregatedStatElement
{}
```
14. In classes and structures, public, private, and protected are written on the same level as the class/struct, but all other internal elements should be deeper.
```cpp
template <typename T, typename Ptr = std::shared_ptr<T>>
class MultiVersion
{
public:
/// The specific version of the object to use.
using Version = Ptr;
...
}
```
15. If the same namespace is used for the entire file, and there isn't anything else significant, an offset is not necessary inside namespace.
16. If the block for if, for, while... expressions consists of a single statement, you don't need to use curly brackets. Place the statement on a separate line, instead. The same is true for a nested if, for, while... statement. But if the inner statement contains curly brackets or else, the external block should be written in curly brackets.
```cpp
/// Если файлы не открыты, то открываем их.
if (streams.empty())
for (const auto & name : column_names)
streams.emplace(name, std::make_unique<Stream>(
storage.files[name].data_file.path(),
storage.files[name].marks[mark_number].offset));
```
17. There should be any spaces at the ends of lines.
18. Sources are UTF-8 encoded.
19. Non-ASCII characters can be used in string literals.
```cpp
<< ", " << (timer.elapsed() / chunks_stats.hits) << " μsec/hit.";
```
20. Do not write multiple expressions in a single line.
21. Group sections of code inside functions and separate them with no more than one empty line.
22. Separate functions, classes, and so on with at least one empty line (maximum – two empty lines).
23. A const (related to a value) must be written before the type name.
```
//correct
const char * pos
const std::string & s
//incorrect
char const * pos
```
24. When declaring a pointer or reference, the \* and & symbols should be separated by spaces on both sides.
```
//correct
const char * pos
//incorrect
const char* pos
const char *pos
```
25. When using template types, alias them with the `using` keyword (except in the simplest cases).
In other words, the template parameters are specified only in `using` and aren't repeated in the code.
`using` can be declared locally, such as inside a function.
```
//correct
using FileStreams = std::map<std::string, std::shared_ptr<Stream>>;
FileStreams streams;
//incorrect
std::map<std::string, std::shared_ptr<Stream>> streams;
```
26. Do not declare several variables of different types in one statement.
```
//incorrect
int x, *y;
```
27. Do not use C-style casts.
```cpp
//incorrect
std::cerr << (int)c <<; std::endl;
//correct
std::cerr << static_cast<int>(c) << std::endl;
```
28. In classes and structs, group members and functions separately inside each visibility scope.
29. For small classes and structs, it is not necessary to separate the method declaration from the implementation.
The same is true for small methods in any classes or structs.
For templated classes and structs, don't separate the method declarations from the implementation (because otherwise they must be defined in the same translation unit).
30. You can wrap lines at 140 characters, instead of 80.
31. Always use the prefix increment/decrement operators if postfix is not required.
```cpp
for (Names::const_iterator it = column_names.begin(); it != column_names.end(); ++it)
```
## Comments
1. Be sure to add comments for all non-trivial parts of code.
This is very important. Writing the comment might help you realize that the code isn't necessary, or that it is designed wrong.
```cpp
/** How much of the piece of memory can be used.
* For example, if internal_buffer is 1 MB, and only 10 bytes were loaded to the buffer from the file for reading,
* then working_buffer will have a size of only 10 bytes
* (working_buffer.end() will point to the position right after those 10 bytes available for read).
*/
```
2. Comments can be as detailed as necessary.
3. Place comments before the code they describe. In rare cases, comments can come after the code, on the same line.
```cpp
/** Parses and executes the query.
*/
void executeQuery(
ReadBuffer & istr, /// Where to read the query from (and data for INSERT, if applicable)
WriteBuffer & ostr, /// Where to write the result
Context & context, /// DB, tables, data types, engines, functions, aggregate functions...
BlockInputStreamPtr & query_plan, /// A description of query processing can be included here
QueryProcessingStage::Enum stage = QueryProcessingStage::Complete /// The last stage to process the SELECT query to
)
```
4. Comments should be written in English only.
5. If you are writing a library, include detailed comments explaining it in the main header file.
6. Do not add comments that do not provide additional information. In particular, do not leave empty comments like this:
```cpp
/*
* Procedure Name:
* Original procedure name:
* Author:
* Date of creation:
* Dates of modification:
* Modification authors:
* Original file name:
* Purpose:
* Intent:
* Designation:
* Classes used:
* Constants:
* Local variables:
* Parameters:
* Date of creation:
* Purpose:
*/
```
(Example taken from: [http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/)](http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/)
7. Do not write garbage comments (author, creation date ..) at the beginning of each file.
8. Single-line comments begin with three slashes: `///` and multi-line comments begin with `/**`. These comments are considered "documentation".
Note: You can use Doxygen to generate documentation from these comments. But Doxygen is not generally used because it is more convenient to navigate the code in the IDE.
9. Multi-line comments must not have empty lines at the beginning and end (except the line that closes a multi-line comment).
10. For commenting out code, use basic comments, not “documenting” comments.
1. Delete the commented out parts of the code before commiting.
11. Do not use profanity in comments or code.
12. Do not use uppercase letters. Do not use excessive punctuation.
```cpp
/// WHAT THE FAIL???
```
13. Do not make delimeters from comments.
```
///******************************************************
```
14. Do not start discussions in comments.
```
/// Зачем ты сделал эту фигню?
```
15. There's no need to write a comment at the end of a block describing what it was about.
```
/// for
```
## Names
1. The names of variables and class members use lowercase letters with underscores.
```cpp
size_t max_block_size;
```
2. The names of functions (methods) use camelCase beginning with a lowercase letter.
```cpp
std::string getName() const override { return "Memory"; }
```
3. The names of classes (structures) use CamelCase beginning with an uppercase letter. Prefixes other than I are not used for interfaces.
```cpp
class StorageMemory : public IStorage
```
4. The names of usings follow the same rules as classes, or you can add _t at the end.
5. Names of template type arguments for simple cases: T; T, U; T1, T2.
For more complex cases, either follow the rules for class names, or add the prefix T.
```cpp
template <typename TKey, typename TValue>
struct AggregatedStatElement
```
6. Names of template constant arguments: either follow the rules for variable names, or use N in simple cases.
```cpp
template <bool without_www>
struct ExtractDomain
```
7. For abstract classes (interfaces) you can add the I prefix.
```cpp
class IBlockInputStream
```
8. If you use a variable locally, you can use the short name.
In other cases, use a descriptive name that conveys the meaning.
```cpp
bool info_successfully_loaded = false;
```
9. define‘s should be in ALL_CAPS with underscores. The same is true for global constants.
```cpp
#define MAX_SRC_TABLE_NAMES_TO_STORE 1000
```
10. File names should use the same style as their contents.
If a file contains a single class, name the file the same way as the class, in CamelCase.
If the file contains a single function, name the file the same way as the function, in camelCase.
11. If the name contains an abbreviation, then:
- For variable names, the abbreviation should use lowercase letters `mysql_connection` (not `mySQL_connection`).
- For names of classes and functions, keep the uppercase letters in the abbreviation`MySQLConnection` (not `MySqlConnection`).
12. Constructor arguments that are used just to initialize the class members should be named the same way as the class members, but with an underscore at the end.
```cpp
FileQueueProcessor(
const std::string & path_,
const std::string & prefix_,
std::shared_ptr<FileHandler> handler_)
: path(path_),
prefix(prefix_),
handler(handler_),
log(&Logger::get("FileQueueProcessor"))
{
}
```
The underscore suffix can be omitted if the argument is not used in the constructor body.
13. There is no difference in the names of local variables and class members (no prefixes required).
```
timer (не m_timer)
```
14. Constants in enums use CamelCase beginning with an uppercase letter. ALL_CAPS is also allowed. If the enum is not local, use enum class.
```cpp
enum class CompressionMethod
{
QuickLZ = 0,
LZ4 = 1,
};
```
15. All names must be in English. Transliteration of Russian words is not allowed.
```
not Stroka
```
16. Abbreviations are acceptable if they are well known (when you can easily find the meaning of the abbreviation in Wikipedia or in a search engine).
`AST`, `SQL`.
Not `NVDH` (some random letters)
Incomplete words are acceptable if the shortened version is common use.
You can also use an abbreviation if the full name is included next to it in the comments.
17. File names with C++ source code must have the .cpp extension. Header files must have the .h extension.
## How to write code
1. Memory management.
Manual memory deallocation (delete) can only be used in library code.
In library code, the delete operator can only be used in destructors.
In application code, memory must be freed by the object that owns it.
Examples:
- The easiest way is to place an object on the stack, or make it a member of another class.
- For a large number of small objects, use containers.
- For automatic deallocation of a small number of objects that reside in the heap, use shared_ptr/unique_ptr.
2. Resource management.
Use RAII and see the previous point.
3. Error handling.
Use exceptions. In most cases, you only need to throw an exception, and don't need to catch it (because of RAII).
In offline data processing applications, it's often acceptable to not catch exceptions.
In servers that handle user requests, it's usually enough to catch exceptions at the top level of the connection handler.
In thread functions, you should catch and keep all exceptions to rethrow them in the main thread after join.
```cpp
/// If there were no other calculations yet, do it synchronously
if (!started)
{
calculate();
started = true;
}
else /// If the calculations are already in progress, wait for results
pool.wait();
if (exception)
exception->rethrow();
```
Never hide exceptions without handling. Never just blindly put all exceptions to log.
Not `catch (...) {}`.
If you need to ignore some exceptions, do so only for specific ones and rethrow the rest.
```cpp
catch (const DB::Exception & e)
{
if (e.code() == ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION)
return nullptr;
else
throw;
}
```
When using functions with response codes or errno, always check the result and throw an exception in case of error.
```cpp
if (0 != close(fd))
throwFromErrno("Cannot close file " + file_name, ErrorCodes::CANNOT_CLOSE_FILE);
```
Asserts are not used.
4. Exception types.
There is no need to use complex exception hierarchy in application code. The exception text should be understandable to a system administrator.
5. Throwing exceptions from destructors.
This is not recommended, but it is allowed.
Use the following options:
- Create a (done() or finalize()) function that will do all the work in advance that might lead to an exception. If that function was called, there should be no exceptions in the destructor later.
- Tasks that are too complex (such as sending messages over the network) can be put in separate method that the class user will have to call before destruction.
- If there is an exception in the destructor, it’s better to log it than to hide it (if the logger is available).
- In simple applications, it is acceptable to rely on std::terminate (for cases of noexcept by default in C++11) to handle exceptions.
6. Anonymous code blocks.
You can create a separate code block inside a single function in order to make certain variables local, so that the destructors are called when exiting the block.
```cpp
Block block = data.in->read();{ std::lock_guard<std::mutex> lock(mutex); data.ready = true; data.block = block;}ready_any.set();
```
7. Multithreading.
For offline data processing applications:
- Try to get the best possible performance on a single CPU core. You can then parallelize your code if necessary.
In server applications:
- Use the thread pool to process requests. At this point, we haven't had any tasks that required userspace context switching.
Fork is not used for parallelization.
8. Synchronizing threads.
Often it is possible to make different threads use different memory cells (even better: different cache lines,) and to not use any thread synchronization (except joinAll).
If synchronization is required, in most cases, it is sufficient to use mutex under lock_guard.
In other cases use system synchronization primitives. Do not use busy wait.
Atomic operations should be used only in the simplest cases.
Do not try to implement lock-free data structures unless it is your primary area of expertise.
9. Pointers vs references.
In most cases, prefer references.
10. const.
Use constant references, pointers to constants, const_iterator, const methods.
Consider const to be default and use non-const only when necessary.
When passing variable by value, using const usually does not make sense.
11. unsigned.
Use unsigned, if needed.
12. Numeric types
Use UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64, and size_t, ssize_t, ptrdiff_t.
Don't use signed/unsigned long, long long, short; signed char, unsigned char, or char types for numbers.
13. Passing arguments.
Pass complex values by reference (including std::string).
If a function captures ownership of an objected created in the heap, make the argument type shared_ptr or unique_ptr.
14. Returning values.
In most cases, just use return. Do not write [return std::move(res)]{.strike}.
If the function allocates an object on heap and returns it, use shared_ptr or unique_ptr.
In rare cases you might need to return the value via an argument. In this case, the argument should be a reference.
```cpp
using AggregateFunctionPtr = std::shared_ptr<IAggregateFunction>;
/** Creates an aggregate function by name.
*/
class AggregateFunctionFactory
{
public:
AggregateFunctionFactory();
AggregateFunctionPtr get(const String & name, const DataTypes & argument_types) const;
```
15. namespace.
There is no need to use a separate namespace for application code or small libraries.
or small libraries.
For medium to large libraries, put everything in the namespace.
You can use the additional detail namespace in a library's .h file to hide implementation details.
In a .cpp file, you can use the static or anonymous namespace to hide symbols.
You can also use namespace for enums to prevent its names from polluting the outer namespace, but it’s better to use the enum class.
16. Delayed initialization.
If arguments are required for initialization then do not write a default constructor.
If later you’ll need to delay initialization, you can add a default constructor that will create an invalid object. Or, for a small number of objects, you can use shared_ptr/unique_ptr.
```cpp
Loader(DB::Connection * connection_, const std::string & query, size_t max_block_size_);
/// For delayed initialization
Loader() {}
```
17. Virtual functions.
If the class is not intended for polymorphic use, you do not need to make functions virtual. This also applies to the destructor.
18. Encodings.
Use UTF-8 everywhere. Use `std::string`and`char *`. Do not use `std::wstring`and`wchar_t`.
19. Logging.
See the examples everywhere in the code.
Before committing, delete all meaningless and debug logging, and any other types of debug output.
Logging in cycles should be avoided, even on the Trace level.
Logs must be readable at any logging level.
Logging should only be used in application code, for the most part.
Log messages must be written in English.
The log should preferably be understandable for the system administrator.
Do not use profanity in the log.
Use UTF-8 encoding in the log. In rare cases you can use non-ASCII characters in the log.
20. I/O.
Don't use iostreams in internal cycles that are critical for application performance (and never use stringstream).
Use the DB/IO library instead.
21. Date and time.
See the DateLUT library.
22. include.
Always use `#pragma once` instead of include guards.
23. using.
The using namespace is not used.
It's fine if you are 'using' something specific, but make it local inside a class or function.
24. Do not use trailing return type for functions unless necessary.
[auto f() -&gt; void;]{.strike}
25. Do not declare and init variables like this:
```cpp
auto s = std::string{"Hello"};
```
Do it like this:
```cpp
std::string s = "Hello";
std::string s{"Hello"};
```
26. For virtual functions, write 'virtual' in the base class, but write 'override' in descendent classes.
## Unused features of C++
1. Virtual inheritance is not used.
2. Exception specifiers from C++03 are not used.
3. Function try block is not used, except for the main function in tests.
## Platform
1. We write code for a specific platform.
But other things being equal, cross-platform or portable code is preferred.
2. The language is C++17.
3. The compiler is gcc. At this time (December 2017), the code is compiled using version 7.2. (It can also be compiled using clang 5.)
The standard library is used (implementation of libstdc++ or libc++).
4. OS: Linux Ubuntu, not older than Precise.
5. Code is written for x86_64 CPU architecture.
The CPU instruction set is the minimum supported set among our servers. Currently, it is SSE 4.2.
6. Use `-Wall -Werror` compilation flags.
7. Use static linking with all libraries except those that are difficult to connect to statically (see the output of the 'ldd' command).
8. Code is developed and debugged with release settings.
## Tools
1. KDevelop is a good IDE.
2. For debugging, use gdb, valgrind (memcheck), strace,-fsanitize=..., tcmalloc_minimal_debug.
3. For profiling, use Linux Perf valgrind (callgrind), strace-cf.
4. Sources are in Git.
5. Compilation is managed by CMake.
6. Releases are in deb packages.
7. Commits to master must not break the build.
Though only selected revisions are considered workable.
8. Make commits as often as possible, even if the code is only partially ready.
Use branches for this purpose.
If your code is not buildable yet, exclude it from the build before pushing to master. You'll need to finish it or remove it from master within a few days.
9. For non-trivial changes, used branches and publish them on the server.
10. Unused code is removed from the repository.
## Libraries
1. The C++14 standard library is used (experimental extensions are fine), as well as boost and Poco frameworks.
2. If necessary, you can use any well-known libraries available in the OS package.
If there is a good solution already available, then use it, even if it means you have to install another library.
(But be prepared to remove bad libraries from code.)
3. You can install a library that isn't in the packages, if the packages don't have what you need or have an outdated version or the wrong type of compilation.
4. If the library is small and doesn't have its own complex build system, put the source files in the contrib folder.
5. Preference is always given to libraries that are already used.
## General recommendations
1. Write as little code as possible.
2. Try the simplest solution.
3. Don't write code until you know how it's going to work and how the inner loop will function.
4. In the simplest cases, use 'using' instead of classes or structs.
5. If possible, do not write copy constructors, assignment operators, destructors (other than a virtual one, if the class contains at least one virtual function), mpve-constructors and move assignment operators. In other words, the compiler-generated functions must work correctly. You can use 'default'.
6. Code simplification is encouraged. Reduce the size of your code where possible.
## Additional recommendations
1. Explicit std:: for types from stddef.h
is not recommended. We recommend writing size_t instead std::size_t because it's shorter.
But if you prefer, std:: is acceptable.
2. Explicit std:: for functions from the standard C library.
is not recommended. Write memcpy instead of std::memcpy.
The reason is that there are similar non-standard functions, such as memmem. We do use these functions on occasion. These functions do not exist in namespace std.
If you write std::memcpy instead of memcpy everywhere, then memmem without std:: will look awkward.
Nevertheless, std:: is allowed if you prefer it.
3. Using functions from C when the ones are available in the standard C++ library.
This is acceptable if it is more efficient.
For example, use memcpy instead of std::copy for copying large chunks of memory.
4. Multiline function arguments.
Any of the following wrapping styles are allowed:
```cpp
function(
T1 x1,
T2 x2)
```
```cpp
function(
size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
```
```cpp
function(size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
```
```cpp
function(size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
```
```cpp
function(
size_t left,
size_t right,
const & RangesInDataParts ranges,
size_t limit)
```

731
docs/en/development/style.rst
View File

@ -1,731 +0,0 @@
.. role:: strike
:class: strike
How to write C++ code
=====================
General
-------
#. This text should be considered as recommendations.
#. If you edit some code, it makes sense to keep style in changes consistent with the rest of it.
#. Style is needed to keep code consistent. Consistency is required to make it easier (more convenient) to read the code. And also for code navigation.
#. Many rules do not have any logical explanation and just come from existing practice.
Formatting
----------
#. Most of formatting is done automatically by ``clang-format``.
#. Indent is 4 spaces wide. Configure your IDE to insert 4 spaces on pressing Tab.
#. Curly braces on separate lines.
.. code-block:: cpp
inline void readBoolText(bool & x, ReadBuffer & buf)
{
char tmp = '0';
readChar(tmp, buf);
x = tmp != '0';
}
#. But if function body is short enough (one statement) you can put the whole thing on one line. In this case put spaces near curly braces, except for the last one in the end.
.. code-block:: cpp
inline size_t mask() const { return buf_size() - 1; }
inline size_t place(HashValue x) const { return x & mask(); }
#. For functions there are no spaces near brackets.
.. code-block:: cpp
void reinsert(const Value & x)
.. code-block:: cpp
memcpy(&buf[place_value], &x, sizeof(x));
#. When using expressions if, for, while, ... (in contrast to function calls) there should be space before opening bracket.
.. code-block:: cpp
for (size_t i = 0; i < rows; i += storage.index_granularity)
#. There should be spaces around binary operators (+, -, \*, /, %, ...) and ternary operator ?:.
.. code-block:: cpp
UInt16 year = (s[0] - '0') * 1000 + (s[1] - '0') * 100 + (s[2] - '0') * 10 + (s[3] - '0');
UInt8 month = (s[5] - '0') * 10 + (s[6] - '0');
UInt8 day = (s[8] - '0') * 10 + (s[9] - '0');
#. If there's a line break, operator is written on new line and it has additional indent.
.. code-block:: cpp
if (elapsed_ns)
message << " ("
<< rows_read_on_server * 1000000000 / elapsed_ns << " rows/s., "
<< bytes_read_on_server * 1000.0 / elapsed_ns << " MB/s.) ";
#. It is ok to insert additional spaces to align the code.
.. code-block:: cpp
dst.ClickLogID = click.LogID;
dst.ClickEventID = click.EventID;
dst.ClickGoodEvent = click.GoodEvent;
#. No spaces around ``.``, ``->`` operators.
If necessary these operators can be moved to next line with additional indent.
#. Unary operators (``--, ++, *, &``, ...) are not delimited from argument.
#. Space is put after comma or semicolon, not before.
#. Operator ``[]`` is not delimited with spaces.
#. In ``template <...>``, put space between ``template`` and ``<``; after ``<`` and before ``>`` - do not.
.. code-block:: cpp
template <typename TKey, typename TValue>
struct AggregatedStatElement
#. In classes and structs keywords public, private, protected are written on same indention level as class/struct, while other contents - deeper.
.. code-block:: cpp
template <typename T>
class MultiVersion
{
public:
/// Version of object for usage. shared_ptr manage lifetime of version.
using Version = std::shared_ptr<const T>;
#. If there's only one namespace in a file and there's nothing else significant - no need to indent the namespace.
#. If ``if, for, while...`` block consists of only one statement, it's not required to wrap it in curly braces. Instead you can put the statement on separate line. This statements can also be a ``if, for, while...`` block. But if internal statement contains curly braces or else, this option should not be used.
.. code-block:: cpp
/// Finish write.
for (auto & stream : streams)
stream.second->finalize();
#. No spaces before end of line.
#. Source code should be in UTF-8 encoding.
#. It's ok to have non-ASCII characters in string literals.
.. code-block:: cpp
<< ", " << (timer.elapsed() / chunks_stats.hits) << " μsec/hit.";
#. Don't put multiple statements on single line.
#. Inside functions do not delimit logical blocks by more than one empty line.
#. Functions, classes and similar constructs are delimited by one or two empty lines.
#. const (related to value) is written before type name.
.. code-block:: cpp
const char * pos
.. code-block:: cpp
const std::string & s
:strike:`char const * pos`
#. When declaring pointer or reference symbols \* and & should be surrounded by spaces.
.. code-block:: cpp
const char * pos
:strike:`const char\* pos`
:strike:`const char \*pos`
#. Alias template types with ``using`` keyword (except the most simple cases). It can be declared even locally, for example inside functions.
.. code-block:: cpp
using FileStreams = std::map<std::string, std::shared_ptr<Stream>>;
FileStreams streams;
:strike:`std::map<std::string, std::shared_ptr<Stream>> streams;`
#. Do not declare several variables of different types in one statements.
:strike:`int x, *y;`
#. C-style casts should be avoided.
:strike:`std::cerr << (int)c << std::endl;`
.. code-block:: cpp
std::cerr << static_cast<int>(c) << std::endl;
#. In classes and structs group members and functions separately inside each visibility scope.
#. For small classes and structs, it is not necessary to split method declaration and implementation.
The same for small methods.
For templated classes and structs it is better not to split declaration and implementations (because anyway they should be defined in the same translation unit).
#. Lines should be wrapped at 140 symbols, not 80.
#. Always use prefix increment/decrement if postfix is not required.
.. code-block:: cpp
for (Names::const_iterator it = column_names.begin(); it != column_names.end(); ++it)
Comments
--------
#. You shoud write comments in all not trivial places.
It is very important. While writing comment you could even understand that code does the wrong thing or is completely unnecessary.
.. code-block:: cpp
/** Part of piece of memory, that can be used.
* For example, if internal_buffer is 1MB, and there was only 10 bytes loaded to buffer from file for reading,
* then working_buffer will have size of only 10 bytes
* (working_buffer.end() will point to position right after those 10 bytes available for read).
*/
#. Comments can be as detailed as necessary.
#. Comments are written before the relevant code. In rare cases - after on the same line.
.. code-block:: text
/** Parses and executes the query.
*/
void executeQuery(
ReadBuffer & istr, /// Where to read the query from (and data for INSERT, if applicable)
WriteBuffer & ostr, /// Where to write the result
Context & context, /// DB, tables, data types, engines, functions, aggregate functions...
BlockInputStreamPtr & query_plan, /// Here could be written the description on how query was executed
QueryProcessingStage::Enum stage = QueryProcessingStage::Complete); /// Up to which stage process the SELECT query
#. Comments should be written only in english
#. When writing a library, put it's detailed description in it's main header file.
#. You shouldn't write comments not providing additional information. For instance, you *CAN'T* write empty comments like this one:
.. code-block:: cpp
/*
* Procedure Name:
* Original procedure name:
* Author:
* Date of creation:
* Dates of modification:
* Modification authors:
* Original file name:
* Purpose:
* Intent:
* Designation:
* Classes used:
* Constants:
* Local variables:
* Parameters:
* Date of creation:
* Purpose:
*/
(example is borrowed from here: http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/)
#. You shouldn't write garbage comments (author, creation date...) in the beginning of each file.
#. One line comments should start with three slashes: ``///``, multiline - with ``/**``. This comments are considered "documenting".
Note: such comments could be used to generate docs using Doxygen. But in reality Doxygen is not used because it is way more convenient to use IDE for code navigation.
#. In beginning and end of multiline comments there should be no empty lines (except the one where the comment ends).
#. For commented out code use simple, not "documenting" comments. Delete commented out code before commits.
#. Do not use profanity in comments or code.
#. Do not use too many question signs, exclamation points or capital letters.
:strike:`/// WHAT THE FAIL???`
#. Do not make delimeters from comments.
:strike:`/*******************************************************/`
#. Do not create discussions in comments.
:strike:`/// Why you did this?`
#. Do not comment end of block describing what kind of block it was.
:strike:`} /// for`
Names
-----
#. Names of variables and class members — in lowercase with underscores.
.. code-block:: cpp
size_t max_block_size;
#. Names of functions (methids) - in camelCase starting with lowercase letter.
.. code-block:: cpp
std::string getName() const override { return "Memory"; }
#. Names of classes (structs) - CamelCase starting with uppercase letter. Prefixes are not used, except I for interfaces.
.. code-block:: cpp
class StorageMemory : public IStorage
#. Names of ``using``'s - same as classes and can have _t suffix.
#. Names of template type arguments: in simple cases - T; T, U; T1, T2.
In complex cases - like class names or can have T prefix.
.. code-block:: cpp
template <typename TKey, typename TValue>
struct AggregatedStatElement
#. Names of template constant arguments: same as variable names or N in simple cases.
.. code-block:: cpp
template <bool without_www>
struct ExtractDomain
#. For abstract classes (interfaces) you can add I to the start of name.
.. code-block:: cpp
class IBlockInputStream
#. If variable is used pretty locally, you can use short name.
In other cases - use descriptive name.
.. code-block:: cpp
bool info_successfully_loaded = false;
#. ``define``'s should be in ALL_CAPS with underlines. Global constants - too.
.. code-block:: cpp
#define MAX_SRC_TABLE_NAMES_TO_STORE 1000
#. Names of files should match it's contents.
If file contains one class - name it like class in CamelCase.
If file contains one function - name it like function in camelCase.
#. If name contains an abbreviation:
* for variables names it should be all lowercase;
``mysql_connection``
:strike:`mySQL_connection`
* for class and function names it should be all uppercase;
``MySQLConnection``
:strike:`MySqlConnection`
#. Constructor arguments used just to initialize the class members, should have the matching name, but with underscore suffix.
.. code-block:: cpp
FileQueueProcessor(
const std::string & path_,
const std::string & prefix_,
std::shared_ptr<FileHandler> handler_)
: path(path_),
prefix(prefix_),
handler(handler_),
log(&Logger::get("FileQueueProcessor"))
{
}
The underscore suffix can be omitted if argument is not used in constructor body.
#. Naming of local variables and class members do not have any differences (no prefixes required).
``timer``
:strike:`m_timer`
#. Constants in enums - CamelCase starting with uppercase letter. ALL_CAPS is also ok. If enum is not local, use enum class.
.. code-block:: cpp
enum class CompressionMethod
{
QuickLZ = 0,
LZ4 = 1,
};
#. All names - in English. Transliteration from Russian is not allowed.
:strike:`Stroka`
#. Abbreviations are fine only if they are well known (when you can find what it means in wikipedia or with web search query).
``AST`` ``SQL``
:strike:`NVDH (some random letters)`
Using incomplete words is ok if it is commonly used. Also you can put the whole word in comments.
#. C++ source code extensions should be .cpp. Header files - only .h.
:strike:`.hpp` :strike:`.cc` :strike:`.C` :strike:`.inl`
``.inl.h`` is ok, but not :strike:`.h.inl:strike:`
How to write code
-----------------
#. Memory management.
Manual memory deallocation (delete) is ok only in destructors in library code.
In application code memory should be freed by some object that owns it.
Examples:
* you can put object on stack or make it a member of another class.
* use containers for many small objects.
* for automatic deallocation of not that many objects residing in heap, use shared_ptr/unique_ptr.
#. Resource management.
Use RAII and see abovee.
#. Error handling.
Use exceptions. In most cases you should only throw exception, but not catch (because of RAII).
In offline data processing applications it's often ok not to catch exceptions.
In server code serving user requests usually you should catch exceptions only on top level of connection handler.
In thread functions you should catch and keep all exceptions to rethrow it in main thread after join.
.. code-block:: cpp
/// If there were no other calculations yet - lets do it synchronously
if (!started)
{
calculate();
started = true;
}
else /// If the calculations are already in progress - lets wait
pool.wait();
if (exception)
exception->rethrow();
Never hide exceptions without handling. Never just blindly put all exceptions to log.
:strike:`catch (...) {}`
If you need to ignore some exceptions, do so only for specific ones and rethrow the rest..
.. code-block:: cpp
catch (const DB::Exception & e)
{
if (e.code() == ErrorCodes::UNKNOWN_AGGREGATE_FUNCTION)
return nullptr;
else
throw;
}
When using functions with error codes - always check it and throw exception in case of error.
.. code-block:: cpp
if (0 != close(fd))
throwFromErrno("Cannot close file " + file_name, ErrorCodes::CANNOT_CLOSE_FILE);
Asserts are not used.
#. Exception types.
No need to use complex exception hierarchy in application code. Exception code should be understandable by operations engineer.
#. Throwing exception from destructors.
Not recommended, but allowed.
Use the following options:
* Create function (done() or finalize()) that will in advance do all the work that might lead to exception. If that function was called, later there should be no exceptions in destructor.
* Too complex work (for example, sending messages via network) can be put in separate method that class user will have to call before destruction.
* If nevertheless there's an exception in destructor it's better to log it that to hide it.
* In simple applications it is ok to rely on std::terminate (in case of noexcept by default in C++11) to handle exception.
#. Anonymous code blocks.
It is ok to declare anonymous code block to make some variables local to it and make them be destroyed earlier than they otherwise would.
.. code-block:: cpp
Block block = data.in->read();
{
std::lock_guard<std::mutex> lock(mutex);
data.ready = true;
data.block = block;
}
ready_any.set();
#. Multithreading.
In case of offline data processing applications:
* Try to make code as fast as possible on single core.
* Make it parallel only if single core performance appeared to be not enough.
In server application:
* use thread pool for request handling;
* for now there were no tasks where userspace context switching was really necessary.
Fork is not used to parallelize code.
#. Synchronizing threads.
Often it is possible to make different threads use different memory cells (better - different cache lines) and do not use any synchronization (except joinAll).
If synchronization is necessary in most cases mutex under lock_guard is enough.
In other cases use system synchronization primitives. Do not use busy wait.
Atomic operations should be used only in the most simple cases.
Do not try to implement lock-free data structures unless it is your primary area of expertise.
#. Pointers vs reference.
Prefer references.
#. const.
Use constant references, pointers to constants, const_iterator, const methods.
Consider const to be default and use non-const only when necessary.
When passing variable by value using const usually do not make sense.
#. unsigned.
unsinged is ok if necessary.
#. Numeric types.
Use types UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64, as well as size_t, ssize_t, ptrdiff_t.
Do not use signed/unsigned long, long long, short; signed char, unsigned char, аnd char.
#. Passing arguments.
Pass complex values by reference (including std::string).
If functions captures object ownership created in heap, make an argument to be shared_ptr or unique_ptr.
#. Returning values.
In most cases just use return. Do not write :strike:`return std::move(res)`.
If function allocates an object on heap and returns it, use shared_ptr or unique_ptr.
In rare cases you might need to return value via argument, in this cases the argument should be a reference.
.. code-block:: cpp
using AggregateFunctionPtr = std::shared_ptr<IAggregateFunction>;
/** Creates aggregate function by it's name
*/
class AggregateFunctionFactory
{
public:
AggregateFunctionFactory();
AggregateFunctionPtr get(const String & name, const DataTypes & argument_types) const;
#. namespace.
No need to use separate namespace for application code or small libraries.
For medium to large libraries - put everything in namespace.
You can use additional detail namespace in .h file to hide implementation details.
In .cpp you can use static or anonymous namespace to hide symbols.
You can also use namespace for enums to prevent it's names to pollute outer namespace, but it's better to use enum class.
#. Delayed initialization.
If arguments are required for initialization then do not write default constructor.
If later you'll need to delay initialization you can add default constructor creating invalid object.
For small number of object you could use shared_ptr/unique_ptr.
.. code-block:: cpp
Loader(DB::Connection * connection_, const std::string & query, size_t max_block_size_);
/// For delayed initialization
Loader() {}
#. Virtual methods.
Do not mark methods or destructor as virtual if class is not intended for polymorph usage.
#. Encoding.
Always use UTF-8. Use ``std::string``, ``char *``. Do not use ``std::wstring``, ``wchar_t``.
#. Logging.
See examples in code.
Remove debug logging before commit.
Logging in cycles should be avoided, even on Trace level.
Logs should be readable even with most detailed settings.
Log mostly in application code.
Log messages should be in English and understandable by operations engineers.
#. I/O.
In internal cycle (critical for application performance) you can't use iostreams (especially stringstream).
Instead use DB/IO library.
#. Date and time.
See DateLUT library.
#. include.
Always use ``#pragma once`` instead of include guards.
#. using.
Don't use ``using namespace``. ``using`` something specific is fine, try to put it as locally as possible.
#. Do not use trailing return unless necessary.
:strike:`auto f() -> void;`
#. Do not delcare and init variables like this:
:strike:`auto s = std::string{"Hello"};`
Do it like this instead:
``std::string s = "Hello";``
``std::string s{"Hello"};``
#. For virtual functions write virtual in base class and don't forget to write override in descendant classes.
Unused C++ features
-------------------
#. Virtual inheritance.
#. Exception specifiers from C++03.
#. Function try block, except main function in tests.
Platform
--------
#. We write code for specific platform. But other things equal cross platform and portable code is preferred.
#. Language is C++17.
#. Compiler is gcc. As of December 2017 version 7.2 is used. It is also compilable with clang 5.
Standard library is used (libstdc++ or libc++).
#. OS - Linux Ubuntu, not older than Precise.
#. Code is written for x86_64 CPU architecture.
CPU instruction set is SSE4.2 is currently required.
#. ``-Wall -Werror`` compilation flags are used.
#. Static linking is used by default. See ldd output for list of exceptions from this rule.
#. Code is developed and debugged with release settings.
Tools
-----
#. Good IDE - KDevelop.
#. For debugging gdb, valgrind (memcheck), strace, -fsanitize=..., tcmalloc_minimal_debug are used.
#. For profiling - Linux Perf, valgrind (callgrind), strace -cf.
#. Source code is in Git.
#. Compilation is managed by CMake.
#. Releases are in deb packages.
#. Commits to master should not break build.
Though only selected revisions are considered workable.
#. Commit as often as possible, even if code is not quite ready yet.
Use branches for this.
If your code is not buildable yet, exclude it from build before pushing to master;
you'll have few days to fix or delete it from master after that.
#. For non-trivial changes use branches and publish them on server.
#. Unused code is removed from repository.
Libraries
---------
#. C++14 standard library is used (experimental extensions are fine), as well as boost and Poco frameworks.
#. If necessary you can use any well known library available in OS with packages.
If there's a good ready solution it is used even if it requires to install one more dependency.
(Buy be prepared to remove bad libraries from code.)
#. It is ok to use library not from packages if it is not available or too old.
#. If the library is small and does not have it's own complex build system, you should put it's sources in contrib folder.
#. Already used libraries are preferred.
General
-------
#. Write as short code as possible.
#. Try the most simple solution.
#. Do not write code if you do not know how it will work yet.
#. In the most simple cases prefer using over classes and structs.
#. Write copy constructors, assignment operators, destructor (except virtual), mpve-constructor and move assignment operators only if there's no other option. You can use ``default``.
#. It is encouraged to simplify code.
Additional
----------
#. Explicit std:: for types from stddef.h.
Not recommended, but allowed.
#. Explicit std:: for functions from C standard library.
Not recommended. For example, write memcpy instead of std::memcpy.
Sometimes there are non standard functions with similar names, like memmem. It will look weird to have memcpy with std:: prefix near memmem without. Though specifying std:: is not prohibited.
#. Usage of C functions when there are alternatives in C++ standard library.
Allowed if they are more effective. For example, use memcpy instead of std::copy for copying large chunks of memory.
#. Multiline function arguments.
All of the following styles are allowed:
.. code-block:: cpp
function(
T1 x1,
T2 x2)
.. code-block:: cpp
function(
size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
.. code-block:: cpp
function(size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
.. code-block:: cpp
function(size_t left, size_t right,
const & RangesInDataParts ranges,
size_t limit)
.. code-block:: cpp
function(
size_t left,
size_t right,
const & RangesInDataParts ranges,
size_t limit)

32
docs/en/development/tests.md Normal file
View File

@ -0,0 +1,32 @@
# How to run ClickHouse tests
The `clickhouse-test` utility that is used for functional testing is written using Python 2.x.It also requires you to have some third-party packages:
```bash
$ pip install lxml termcolor
```
In a nutshell:
- Put the `clickhouse` program to `/usr/bin` (or `PATH`)
- Create a `clickhouse-client` symlink in `/usr/bin` pointing to `clickhouse`
- Start the `clickhouse` server
- `cd dbms/tests/`
- Run `./clickhouse-test`
## Example usage
Run `./clickhouse-test --help` to see available options.
To run tests without having to create a symlink or mess with `PATH`:
```bash
./clickhouse-test -c "../../build/dbms/src/Server/clickhouse --client"
```
To run a single test, i.e. `00395_nullable`:
```bash
./clickhouse-test 00395
```

38
docs/en/development/tests.rst
View File

@ -1,38 +0,0 @@
How to run ClickHouse tests
===========================
The ``clickhouse-test`` utility that is used for functional testing is written using Python 2.x.
It also requires you to have some third-party packages:
.. code-block:: bash
$ pip install lxml termcolor
In a nutshell:
- Put the ``clickhouse`` program to ``/usr/bin`` (or ``PATH``)
- Create a ``clickhouse-client`` symlink in ``/usr/bin`` pointing to ``clickhouse``
- Start the ``clickhouse`` server
- ``cd dbms/tests/``
- Run ``./clickhouse-test``
Example usage
-------------
Run ``./clickhouse-test --help`` to see available options.
To run tests without having to create a symlink or mess with ``PATH``:
.. code-block:: bash
./clickhouse-test -c "../../build/dbms/src/Server/clickhouse --client"
To run a single test, i.e. ``00395_nullable``:
.. code-block:: bash
./clickhouse-test 00395

54
docs/en/dicts/external_dicts.md Normal file
View File

@ -0,0 +1,54 @@
<a name="dicts-external_dicts"></a>
# External dictionaries
You can add your own dictionaries from various data sources. The data source for a dictionary can be a local text or executable file, an HTTP(s) resource, or another DBMS. For more information, see "[Sources for external dictionaries](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources)".
ClickHouse:
> - Fully or partially stores dictionaries in RAM.
- Periodically updates dictionaries and dynamically loads missing values. In other words, dictionaries can be loaded dynamically.
The configuration of external dictionaries is located in one or more files. The path to the configuration is specified in the [dictionaries_config](../operations/server_settings/settings.md#server_settings-dictionaries_config) parameter.
Dictionaries can be loaded at server startup or at first use, depending on the [dictionaries_lazy_load](../operations/server_settings/settings.md#server_settings-dictionaries_lazy_load) setting.
The dictionary config file has the following format:
```xml
<yandex>
<comment>An optional element with any content. Ignored by the ClickHouse server.</comment>
<!--Optional element. File name with substitutions-->
<include_from>/etc/metrika.xml</include_from>
<dictionary>
<!-- Dictionary configuration -->
</dictionary>
...
<dictionary>
<!-- Dictionary configuration -->
</dictionary>
</yandex>
```
You can [configure](external_dicts_dict.md#dicts-external_dicts_dict) any number of dictionaries in the same file. The file format is preserved even if there is only one dictionary (i.e. `<yandex><dictionary> <!--configuration -> </dictionary></yandex>` ).
See also "[Functions for working with external dictionaries](../functions/ext_dict_functions.md#ext_dict_functions)".
<div class="admonition attention">
You can convert values ​​for a small dictionary by describing it in a `SELECT` query (see the [transform](../functions/other_functions.md#other_functions-transform) function). This functionality is not related to external dictionaries.
</div>
```eval_rst
.. toctree::
:glob:
external_dicts_dict*
```

368
docs/en/dicts/external_dicts.rst
View File

@ -1,368 +0,0 @@
External dictionaries
=====================
It is possible to add your own dictionaries from various data sources. The data source for a dictionary can be a file in the local file system, the ClickHouse server, or a MySQL server.
A dictionary can be stored completely in RAM and updated regularly, or it can be partially cached in RAM and dynamically load missing values.
The configuration of external dictionaries is in a separate file or files specified in the ``dictionaries_config`` configuration parameter.
This parameter contains the absolute or relative path to the file with the dictionary configuration. A relative path is relative to the directory with the server config file. The path can contain wildcards ``*`` and ``?``, in which case all matching files are found. Example: ``dictionaries/*.xml``.
The dictionary configuration, as well as the set of files with the configuration, can be updated without restarting the server. The server checks updates every 5 seconds. This means that dictionaries can be enabled dynamically.
Dictionaries can be created when starting the server, or at first use. This is defined by the ``dictionaries_lazy_load`` parameter in the main server config file. This parameter is optional, 'true' by default. If set to 'true', each dictionary is created at first use. If dictionary creation failed, the function that was using the dictionary throws an exception. If 'false', all dictionaries are created when the server starts, and if there is an error, the server shuts down.
The dictionary config file has the following format:
.. code-block:: xml
<yandex>
<comment>Optional element with any content; completely ignored.</comment>
<!--Optional element to specify include file-->
<include_from>/etc/metrika.xml</include_from>
<!--You can set any number of different dictionaries. -->
<dictionary>
<!-- Dictionary name. The dictionary will be accessed for use by this name. -->
<name>os</name>
<!-- Data source. -->
<source>
<!-- Source is a file in the local file system. -->
<file>
<!-- Path on the local file system. -->
<path>/opt/dictionaries/os.tsv</path>
<!-- Which format to use for reading the file. -->
<format>TabSeparated</format>
</file>
<!-- or the source is a table on a MySQL server.
<mysql>
<!- - These parameters can be specified outside (common for all replicas) or inside a specific replica - ->
<port>3306</port>
<user>clickhouse</user>
<password>qwerty</password>
<!- - Specify from one to any number of replicas for fault tolerance. - ->
<replica>
<host>example01-1</host>
<priority>1</priority> <!- - The lower the value, the higher the priority. - ->
</replica>
<replica>
<host>example01-2</host>
<priority>1</priority>
</replica>
<db>conv_main</db>
<table>counters</table>
</mysql>
-->
<!-- or the source is a table on the ClickHouse server.
<clickhouse>
<host>example01-01-1</host>
<port>9000</port>
<user>default</user>
<password></password>
<db>default</db>
<table>counters</table>
</clickhouse>
<!- - If the address is similar to localhost, the request is made without network interaction. For fault tolerance, you can create a Distributed table on localhost and enter it. - ->
-->
<!-- or the source is a executable. If layout.complex_key_cache - list of needed keys will be written in STDIN of program -->
<executable>
<!-- Path on the local file system or name located in one of env PATH dirs. -->
<command>cat /opt/dictionaries/os.tsv</command>
<!-- Which format to use for reading/writing stream. -->
<format>TabSeparated</format>
</executable>
<!-- or the source is a http server. If layout.complex_key_cache - list of needed keys will be sent as POST -->
<http>
<!-- Host. -->
<url>http://[::1]/os.tsv</url>
<!-- Which format to use for reading answer and making POST. -->
<format>TabSeparated</format>
</http>
</source>
<!-- Update interval for fully loaded dictionaries. 0 - never update. -->
<lifetime>
<min>300</min>
<max>360</max>
<!-- The update interval is selected uniformly randomly between min and max, in order to spread out the load when updating dictionaries on a large number of servers. -->
</lifetime>
<!-- or <!- - The update interval for fully loaded dictionaries or invalidation time for cached dictionaries. 0 - never update. - ->
<lifetime>300</lifetime>
-->
<layout> <!-- Method for storing in memory. -->
<flat />
<!-- or <hashed />
or
<cache>
<!- - Cache size in number of cells; rounded up to a degree of two. - ->
<size_in_cells>1000000000</size_in_cells>
</cache>
or
<ip_trie />
-->
</layout>
<!-- Structure. -->
<structure>
<!-- Description of the column that serves as the dictionary identifier (key). -->
<id>
<!-- Column name with ID. -->
<name>Id</name>
</id>
<attribute>
<!-- Column name. -->
<name>Name</name>
<!-- Column type. (How the column is understood when loading. For MySQL, a table can have TEXT, VARCHAR, and BLOB, but these are all loaded as String) -->
<type>String</type>
<!-- Value to use for a non-existing element. In the example, an empty string. -->
<null_value></null_value>
</attribute>
<!-- Any number of attributes can be specified. -->
<attribute>
<name>ParentID</name>
<type>UInt64</type>
<null_value>0</null_value>
<!-- Whether it defines a hierarchy - mapping to the parent ID (by default, false). -->
<hierarchical>true</hierarchical>
<!-- The mapping id -> attribute can be considered injective, in order to optimize GROUP BY. (by default, false) -->
<injective>true</injective>
</attribute>
</structure>
</dictionary>
</yandex>
The dictionary identifier (key attribute) should be a number that fits into UInt64. Also, you can use arbitrary tuples as keys (see section "Dictionaries with complex keys"). Note: you can use complex keys consisting of just one element. This allows using e.g. Strings as dictionary keys.
There are six ways to store dictionaries in memory.
flat
----
This is the most effective method. It works if all keys are smaller than ``500,000``. If a larger key is discovered when creating the dictionary, an exception is thrown and the dictionary is not created. The dictionary is loaded to RAM in its entirety. The dictionary uses the amount of memory proportional to maximum key value. With the limit of 500,000, memory consumption is not likely to be high. All types of sources are supported. When updating, data (from a file or from a table) is read in its entirety.
hashed
------
This method is slightly less effective than the first one. The dictionary is also loaded to RAM in its entirety, and can contain any number of items with any identifiers. In practice, it makes sense to use up to tens of millions of items, while there is enough RAM.
All types of sources are supported. When updating, data (from a file or from a table) is read in its entirety.
cache
-----
This is the least effective method. It is appropriate if the dictionary doesn't fit in RAM. It is a cache of a fixed number of cells, where frequently-used data can be located. MySQL, ClickHouse, executable, http sources are supported, but file sources are not supported.
When searching a dictionary, the cache is searched first. For each data block, all keys not found in the cache (or expired keys) are collected in a package, which is sent to the source with the query ``SELECT attrs... FROM db.table WHERE id IN (k1, k2, ...)``. The received data is then written to the cache.
range_hashed
------------
The table lists some data for date ranges, for each key. To give the possibility to extract this data for a given key, for a given date.
Example: in the table there are discounts for each advertiser in the form:
.. code-block:: text
advertiser id discount start date end date value
123 2015-01-01 2015-01-15 0.15
123 2015-01-16 2015-01-31 0.25
456 2015-01-01 2015-01-15 0.05
Adding layout = range_hashed.
When using such a layout, the structure should have the elements range_min, range_max.
Example:
.. code-block:: xml
<structure>
<id>
<name>Id</name>
</id>
<range_min>
<name>first</name>
</range_min>
<range_max>
<name>last</name>
</range_max>
...
These columns must be of type Date. Other types are not yet supported.
The columns indicate a closed date range.
To work with such dictionaries, dictGetT functions must take one more argument - the date:
``dictGetT('dict_name', 'attr_name', id, date)``
The function takes out the value for this id and for the date range, which includes the transmitted date. If no id is found or the range found is not found for the found id, the default value for the dictionary is returned.
If there are overlapping ranges, then any suitable one can be used.
If the range boundary is NULL or is an incorrect date (1900-01-01, 2039-01-01), then the range should be considered open. The range can be open on both sides.
In the RAM, the data is presented as a hash table with a value in the form of an ordered array of ranges and their corresponding values.
Example of a dictionary by ranges:
.. code-block:: xml
<yandex>
<dictionary>
<name>xxx</name>
<source>
<mysql>
<password>xxx</password>
<port>3306</port>
<user>xxx</user>
<replica>
<host>xxx</host>
<priority>1</priority>
</replica>
<db>dicts</db>
<table>xxx</table>
</mysql>
</source>
<lifetime>
<min>300</min>
<max>360</max>
</lifetime>
<layout>
<range_hashed />
</layout>
<structure>
<id>
<name>Abcdef</name>
</id>
<range_min>
<name>StartDate</name>
</range_min>
<range_max>
<name>EndDate</name>
</range_max>
<attribute>
<name>XXXType</name>
<type>String</type>
<null_value />
</attribute>
</structure>
</dictionary>
</yandex>
ip_trie
-------
The table stores IP prefixes for each key (IP address), which makes it possible to map IP addresses to metadata such as ASN or threat score.
Example: in the table there are prefixes matches to AS number and country:
.. code-block:: text
prefix asn cca2
202.79.32.0/20 17501 NP
2620:0:870::/48 3856 US
2a02:6b8:1::/48 13238 RU
2001:db8::/32 65536 ZZ
When using such a layout, the structure should have the "key" element.
Example:
.. code-block:: xml
<structure>
<key>
<attribute>
<name>prefix</name>
<type>String</type>
</attribute>
</key>
<attribute>
<name>asn</name>
<type>UInt32</type>
<null_value />
</attribute>
<attribute>
<name>cca2</name>
<type>String</type>
<null_value>??</null_value>
</attribute>
...
These key must have only one attribute of type String, containing a valid IP prefix. Other types are not yet supported.
For querying, same functions (dictGetT with tuple) as for complex key dictionaries have to be used:
``dictGetT('dict_name', 'attr_name', tuple(ip))``
The function accepts either UInt32 for IPv4 address or FixedString(16) for IPv6 address in wire format:
``dictGetString('prefix', 'asn', tuple(IPv6StringToNum('2001:db8::1')))``
No other type is supported. The function returns attribute for a prefix matching the given IP address. If there are overlapping prefixes, the most specific one is returned.
The data is stored currently in a bitwise trie, it has to fit in memory.
complex_key_hashed
------------------
The same as ``hashed``, but for complex keys.
complex_key_cache
-----------------
The same as ``cache``, but for complex keys.
Notes
-----
We recommend using the ``flat`` method when possible, or ``hashed``. The speed of the dictionaries is impeccable with this type of memory storage.
Use the cache method only in cases when it is unavoidable. The speed of the cache depends strongly on correct settings and the usage scenario. A cache type dictionary only works normally for high enough hit rates (recommended 99% and higher). You can view the average hit rate in the system.dictionaries table. Set a large enough cache size. You will need to experiment to find the right number of cells - select a value, use a query to get the cache completely full, look at the memory consumption (this information is in the system.dictionaries table), then proportionally increase the number of cells so that a reasonable amount of memory is consumed. We recommend MySQL as the source for the cache, because ClickHouse doesn't handle requests with random reads very well.
In all cases, performance is better if you call the function for working with a dictionary after ``GROUP BY``, and if the attribute being fetched is marked as injective. For a dictionary cache, performance improves if you call the function after LIMIT. To do this, you can use a subquery with LIMIT, and call the function with the dictionary from the outside.
An attribute is called injective if different attribute values correspond to different keys. So when ``GROUP BY`` uses a function that fetches an attribute value by the key, this function is automatically taken out of ``GROUP BY``.
When updating dictionaries from a file, first the file modification time is checked, and it is loaded only if the file has changed.
When updating from MySQL, for flat and hashed dictionaries, first a ``SHOW TABLE STATUS`` query is made, and the table update time is checked. If it is not NULL, it is compared to the stored time. This works for MyISAM tables, but for InnoDB tables the update time is unknown, so loading from InnoDB is performed on each update.
For cache dictionaries, the expiration (lifetime) of data in the cache can be set. If more time than 'lifetime' has passed since loading the data in a cell, the cell's value is not used, and it is re-requested the next time it needs to be used.
If a dictionary couldn't be loaded even once, an attempt to use it throws an exception.
If an error occurred during a request to a cached source, an exception is thrown.
Dictionary updates (other than loading for first use) do not block queries. During updates, the old version of a dictionary is used. If an error occurs during an update, the error is written to the server log, and queries continue using the old version of dictionaries.
You can view the list of external dictionaries and their status in the system.dictionaries table.
To use external dictionaries, see the section "Functions for working with external dictionaries".
Note that you can convert values for a small dictionary by specifying all the contents of the dictionary directly in a ``SELECT`` query (see the section "transform function"). This functionality is not related to external dictionaries.
Dictionaries with complex keys
------------------------------
You can use tuples consisting of fields of arbitrary types as keys. Configure your dictionary with ``complex_key_hashed`` or ``complex_key_cache`` layout in this case.
Key structure is configured not in the ``<id>`` element but in the ``<key>`` element. Fields of the key tuple are configured analogously to dictionary attributes. Example:
.. code-block:: xml
<structure>
<key>
<attribute>
<name>field1</name>
<type>String</type>
</attribute>
<attribute>
<name>field2</name>
<type>UInt32</type>
</attribute>
...
</key>
...
When using such dictionary, use a Tuple of field values as a key in dictGet* functions. Example: ``dictGetString('dict_name', 'attr_name', tuple('field1_value', 123))``.

34
docs/en/dicts/external_dicts_dict.md Normal file
View File

@ -0,0 +1,34 @@
<a name="dicts-external_dicts_dict"></a>
# Configuring an external dictionary
The dictionary configuration has the following structure:
```xml
<dictionary>
<name>dict_name</name>
<source>
<!-- Source configuration -->
</source>
<layout>
<!-- Memory layout configuration -->
</layout>
<structure>
<!-- Complex key configuration -->
</structure>
<lifetime>
<!-- Lifetime of dictionary in memory -->
</lifetime>
</dictionary>
```
- name – The identifier that can be used to access the dictionary. Use the characters `[a-zA-Z0-9_\-]`.
- [source](external_dicts_dict_sources.html/#dicts-external_dicts_dict_sources) – Source of the dictionary.
- [layout](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout) – Location of the dictionary in memory.
- [structure](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure) – Structure of the dictionary. A key and attributes that can be retrieved by this key.
- [lifetime](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime) – How frequently to update dictionaries.

222
docs/en/dicts/external_dicts_dict_layout.md Normal file
View File

@ -0,0 +1,222 @@
<a name="dicts-external_dicts_dict_layout"></a>
# Storing dictionaries in memory
There are [many different ways](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout-manner) to store dictionaries in memory.
We recommend [flat](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout-flat), [hashed](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout-hashed), and [complex_key_hashed](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout-complex_key_hashed). which provide optimal processing speed.
Caching is not recommended because of potentially poor performance and difficulties in selecting optimal parameters. Read more about this in the "[cache](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout-cache)" section.
There are several ways to improve dictionary performance:
- Call the function for working with the dictionary after `GROUP BY`.
- Mark attributes to extract as injective. An attribute is called injective if different attribute values correspond to different keys. So when `GROUP BY` uses a function that fetches an attribute value by the key, this function is automatically taken out of `GROUP BY`.
ClickHouse generates an exception for errors with dictionaries. Examples of errors:
- The dictionary being accessed could not be loaded.
- Error querying a `cached` dictionary.
You can view the list of external dictionaries and their statuses in the `system.dictionaries` table.
The configuration looks like this:
```xml
<yandex>
<dictionary>
...
<layout>
<layout_type>
<!-- layout settings -->
</layout_type>
</layout>
...
</dictionary>
</yandex>
```
<a name="dicts-external_dicts_dict_layout-manner"></a>
## Ways to store dictionaries in memory
- [flat](#dicts-external_dicts_dict_layout-flat)
- [hashed](#dicts-external_dicts_dict_layout-hashed)
- [cache](#dicts-external_dicts_dict_layout-cache)
- [range_hashed](#dicts-external_dicts_dict_layout-range_hashed)
- [complex_key_hashed](#dicts-external_dicts_dict_layout-complex_key_hashed)
- [complex_key_cache](#dicts-external_dicts_dict_layout-complex_key_cache)
<a name="dicts-external_dicts_dict_layout-flat"></a>
### flat
The dictionary is completely stored in memory in the form of flat arrays. How much memory does the dictionary use? The amount is proportional to the size of the largest key (in space used).
The dictionary key has the ` UInt64` type and the value is limited to 500,000. If a larger key is discovered when creating the dictionary, ClickHouse throws an exception and does not create the dictionary.
All types of sources are supported. When updating, data (from a file or from a table) is read in its entirety.
This method provides the best performance among all available methods of storing the dictionary.
Configuration example:
```xml
<layout>
<flat />
</layout>
```
<a name="dicts-external_dicts_dict_layout-hashed"></a>
### hashed
The dictionary is completely stored in memory in the form of a hash table. The dictionary can contain any number of elements with any identifiers In practice, the number of keys can reach tens of millions of items.
All types of sources are supported. When updating, data (from a file or from a table) is read in its entirety.
Configuration example:
```xml
<layout>
<hashed />
</layout>
```
<a name="dicts-external_dicts_dict_layout-complex_key_hashed"></a>
### complex_key_hashed
This type of storage is designed for use with compound [keys](..external_dicts_dict_structure.html/#dicts-external_dicts_dict_structure). It is similar to hashed.
Configuration example:
```xml
<layout>
<complex_key_hashed />
</layout>
```
<a name="dicts-external_dicts_dict_layout-range_hashed"></a>
### range_hashed
The dictionary is stored in memory in the form of a hash table with an ordered array of ranges and their corresponding values.
This storage method works the same way as hashed and allows using date/time ranges in addition to the key, if they appear in the dictionary.
Example: The table contains discounts for each advertiser in the format:
```
+------------------+-----------------------------+------------+----------+ | advertiser id | discount start date | discount end date | amount | +==================+=============================+============+==========+ | 123 | 2015-01-01 | 2015-01-15 | 0.15 | +------------------+-----------------------------+------------+----------+ | 123 | 2015-01-16 | 2015-01-31 | 0.25 | +------------------+-----------------------------+------------+----------+ | 456 | 2015-01-01 | 2015-01-15 | 0.05 | +------------------+-----------------------------+------------+----------+
```
To use a sample for date ranges, define `range_min` and `range_max` in [structure](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure).
Example:
```xml
<structure>
<id>
<name>Id</name>
</id>
<range_min>
<name>first</name>
</range_min>
<range_max>
<name>last</name>
</range_max>
...
```
To work with these dictionaries, you need to pass an additional date argument to the `dictGetT` function:
dictGetT('dict_name', 'attr_name', id, date)
This function returns the value for the specified `id`s and the date range that includes the passed date.
Details of the algorithm:
- If the ` id` is not found or a range is not found for the ` id`, it returns the default value for the dictionary.
- If there are overlapping ranges, you can use any.
- If the range delimiter is `NULL` or an invalid date (such as 1900-01-01 or 2039-01-01), the range is left open. The range can be open on both sides.
Configuration example:
```xml
<yandex>
<dictionary>
...
<layout>
<range_hashed />
</layout>
<structure>
<id>
<name>Abcdef</name>
</id>
<range_min>
<name>StartDate</name>
</range_min>
<range_max>
<name>EndDate</name>
</range_max>
<attribute>
<name>XXXType</name>
<type>String</type>
<null_value />
</attribute>
</structure>
</dictionary>
</yandex>
```
<a name="dicts-external_dicts_dict_layout-cache"></a>
### cache
The dictionary is stored in a cache that has a fixed number of cells. These cells contain frequently used elements.
When searching for a dictionary, the cache is searched first. For each block of data, all keys that are not found in the cache or are outdated are requested from the source using ` SELECT attrs... FROM db.table WHERE id IN (k1, k2, ...)`. The received data is then written to the cache.
For cache dictionaries, the expiration (lifetime &lt;dicts-external_dicts_dict_lifetime&gt;) of data in the cache can be set. If more time than `lifetime` has passed since loading the data in a cell, the cell's value is not used, and it is re-requested the next time it needs to be used.
This is the least effective of all the ways to store dictionaries. The speed of the cache depends strongly on correct settings and the usage scenario. A cache type dictionary performs well only when the hit rates are high enough (recommended 99% and higher). You can view the average hit rate in the `system.dictionaries` table.
To improve cache performance, use a subquery with ` LIMIT`, and call the function with the dictionary externally.
Supported [sources](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources): MySQL, ClickHouse, executable, HTTP.
Example of settings:
```xml
<layout>
<cache>
<!-- The size of the cache, in number of cells. Rounded up to a power of two. -->
<size_in_cells>1000000000</size_in_cells>
</cache>
</layout>
```
Set a large enough cache size. You need to experiment to select the number of cells:
1. Set some value.
2. Run queries until the cache is completely full.
3. Assess memory consumption using the `system.dictionaries` table.
4. Increase or decrease the number of cells until the required memory consumption is reached.
<div class="admonition warning">
Do not use ClickHouse as a source, because it is slow to process queries with random reads.
</div>
<a name="dicts-external_dicts_dict_layout-complex_key_cache"></a>
### complex_key_cache
This type of storage is designed for use with compound [keys](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure). Similar to `cache`.

59
docs/en/dicts/external_dicts_dict_lifetime.md Normal file
View File

@ -0,0 +1,59 @@
<a name="dicts-external_dicts_dict_lifetime"></a>
# Dictionary updates
ClickHouse periodically updates the dictionaries. The update interval for fully downloaded dictionaries and the invalidation interval for cached dictionaries are defined in the `<lifetime>` tag in seconds.
Dictionary updates (other than loading for first use) do not block queries. During updates, the old version of a dictionary is used. If an error occurs during an update, the error is written to the server log, and queries continue using the old version of dictionaries.
Example of settings:
```xml
<dictionary>
...
<lifetime>300</lifetime>
...
</dictionary>
```
Setting ` <lifetime> 0</lifetime> ` prevents updating dictionaries.
You can set a time interval for upgrades, and ClickHouse will choose a uniformly random time within this range. This is necessary in order to distribute the load on the dictionary source when upgrading on a large number of servers.
Example of settings:
```xml
<dictionary>
...
<lifetime>
<min>300</min>
<max>360</max>
</lifetime>
...
</dictionary>
```
When upgrading the dictionaries, the ClickHouse server applies different logic depending on the type of [ source](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources):
> - For a text file, it checks the time of modification. If the time differs from the previously recorded time, the dictionary is updated.
> - For MyISAM tables, the time of modification is checked using a `SHOW TABLE STATUS` query.
> - Dictionaries from other sources are updated every time by default.
For MySQL (InnoDB) and ODBC sources, you can set up a query that will update the dictionaries only if they really changed, rather than each time. To do this, follow these steps:
> - The dictionary table must have a field that always changes when the source data is updated.
> - The settings of the source must specify a query that retrieves the changing field. The ClickHouse server interprets the query result as a row, and if this row has changed relative to its previous state, the dictionary is updated. The query must be specified in the `<invalidate_query>` field in the [ source](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources) settings.
Example of settings:
```xml
<dictionary>
...
<odbc>
...
<invalidate_query>SELECT update_time FROM dictionary_source where id = 1</invalidate_query>
</odbc>
...
</dictionary>
```

397
docs/en/dicts/external_dicts_dict_sources.md Normal file
View File

@ -0,0 +1,397 @@
<a name="dicts-external_dicts_dict_sources"></a>
# Sources of external dictionaries
An external dictionary can be connected from many different sources.
The configuration looks like this:
```xml
<yandex>
<dictionary>
...
<source>
<source_type>
<!-- Source configuration -->
</source_type>
</source>
...
</dictionary>
...
</yandex>
```
The source is configured in the `source` section.
Types of sources (`source_type`):
- [Local file](#dicts-external_dicts_dict_sources-local_file)
- [Executable file](#dicts-external_dicts_dict_sources-executable)
- [HTTP(s)](#dicts-external_dicts_dict_sources-http)
- [ODBC](#dicts-external_dicts_dict_sources-odbc)
- DBMS
- [MySQL](#dicts-external_dicts_dict_sources-mysql)
- [ClickHouse](#dicts-external_dicts_dict_sources-clickhouse)
- [MongoDB](#dicts-external_dicts_dict_sources-mongodb)
<a name="dicts-external_dicts_dict_sources-local_file"></a>
## Local file
Example of settings:
```xml
<source>
<file>
<path>/opt/dictionaries/os.tsv</path>
<format>TabSeparated</format>
</file>
</source>
```
Setting fields:
- `path` – The absolute path to the file.
- `format` – The file format. All the formats described in "[Formats](../formats/index.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-executable"></a>
## Executable file
Working with executable files depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file's `STDIN`.
Example of settings:
```xml
<source>
<executable>
<command>cat /opt/dictionaries/os.tsv</command>
<format>TabSeparated</format>
</executable>
</source>
```
Setting fields:
- `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`).
- `format` – The file format. All the formats described in "[Formats](../formats/index.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-http"></a>
## HTTP(s)
Working with executable files depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method.
Example of settings:
```xml
<source>
<http>
<url>http://[::1]/os.tsv</url>
<format>TabSeparated</format>
</http>
</source>
```
In order for ClickHouse to access an HTTPS resource, you must [configure openSSL](../operations/server_settings/settings.md#server_settings-openSSL) in the server configuration.
Setting fields:
- `url` – The source URL.
- `format` – The file format. All the formats described in "[Formats](../formats/index.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-odbc"></a>
## ODBC
You can use this method to connect any database that has an ODBC driver.
Example of settings:
```xml
<odbc>
<db>DatabaseName</db>
<table>TableName</table>
<connection_string>DSN=some_parameters</connection_string>
<invalidate_query>SQL_QUERY</invalidate_query>
</odbc>
```
Setting fields:
- `db` – Name of the database. Omit it if the database name is set in the `<connection_string>` parameters.
- `table` – Name of the table.
- `connection_string` – Connection string.
- `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime).
## Example of connecting PostgreSQL
Ubuntu OS.
Installing unixODBC and the ODBC driver for PostgreSQL:
sudo apt-get install -y unixodbc odbcinst odbc-postgresql
Configuring `/etc/odbc.ini` (или `~/.odbc.ini`): :
[DEFAULT]
Driver = myconnection
[myconnection]
Description = PostgreSQL connection to my_db
Driver = PostgreSQL Unicode
Database = my_db
Servername = 127.0.0.1
UserName = username
Password = password
Port = 5432
Protocol = 9.3
ReadOnly = No
RowVersioning = No
ShowSystemTables = No
ConnSettings =
The dictionary configuration in ClickHouse:
```xml
<dictionary>
<name>table_name</name>
<source>
<odbc>
<!-- You can specifiy the following parameters in connection_string: -->
<!-- DSN=myconnection;UID=username;PWD=password;HOST=127.0.0.1;PORT=5432;DATABASE=my_db -->
<connection_string>DSN=myconnection</connection_string>
<table>postgresql_table</table>
</odbc>
</source>
<lifetime>
<min>300</min>
<max>360</max>
</lifetime>
<layout>
<hashed/>
</layout>
<structure>
<id>
<name>id</name>
</id>
<attribute>
<name>some_column</name>
<type>UInt64</type>
<null_value>0</null_value>
</attribute>
</structure>
</dictionary>
```
You may need to edit `odbc.ini` to specify the full path to the library with the driver `DRIVER=/usr/local/lib/psqlodbcw.so`.
### Example of connecting MS SQL Server
Ubuntu OS.
Installing the driver: :
```
sudo apt-get install tdsodbc freetds-bin sqsh
```
Configuring the driver: :
```
$ cat /etc/freetds/freetds.conf
...
[MSSQL]
host = 192.168.56.101
port = 1433
tds version = 7.0
client charset = UTF-8
$ cat /etc/odbcinst.ini
...
[FreeTDS]
Description = FreeTDS
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage = 1
UsageCount = 5
$ cat ~/.odbc.ini
...
[MSSQL]
Description = FreeTDS
Driver = FreeTDS
Servername = MSSQL
Database = test
UID = test
PWD = test
Port = 1433
```
Configuring the dictionary in ClickHouse:
```xml
<yandex>
<dictionary>
<name>test</name>
<source>
<odbc>
<table>dict</table>
<connection_string>DSN=MSSQL;UID=test;PWD=test</connection_string>
</odbc>
</source>
<lifetime>
<min>300</min>
<max>360</max>
</lifetime>
<layout>
<flat />
</layout>
<structure>
<id>
<name>k</name>
</id>
<attribute>
<name>s</name>
<type>String</type>
<null_value></null_value>
</attribute>
</structure>
</dictionary>
</yandex>
```
## DBMS
<a name="dicts-external_dicts_dict_sources-mysql"></a>
### MySQL
Example of settings:
```xml
<source>
<mysql>
<port>3306</port>
<user>clickhouse</user>
<password>qwerty</password>
<replica>
<host>example01-1</host>
<priority>1</priority>
</replica>
<replica>
<host>example01-2</host>
<priority>1</priority>
</replica>
<db>db_name</db>
<table>table_name</table>
<where>id=10</where>
<invalidate_query>SQL_QUERY</invalidate_query>
</mysql>
</source>
```
Setting fields:
- `port` – The port on the MySQL server. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `user` – Name of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `password` – Password of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `replica` – Section of replica configurations. There can be multiple sections.
- `replica/host` – The MySQL host.
\* `replica/priority` – The replica priority. When attempting to connect, ClickHouse traverses the replicas in order of priority. The lower the number, the higher the priority.
- `db` – Name of the database.
- `table` – Name of the table.
- `where ` – The selection criteria. Optional parameter.
- `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime).
MySQL can be connected on a local host via sockets. To do this, set `host` and `socket`.
Example of settings:
```xml
<source>
<mysql>
<host>localhost</host>
<socket>/path/to/socket/file.sock</socket>
<user>clickhouse</user>
<password>qwerty</password>
<db>db_name</db>
<table>table_name</table>
<where>id=10</where>
<invalidate_query>SQL_QUERY</invalidate_query>
</mysql>
</source>
```
<a name="dicts-external_dicts_dict_sources-clickhouse"></a>
### ClickHouse
Example of settings:
```xml
<source>
<clickhouse>
<host>example01-01-1</host>
<port>9000</port>
<user>default</user>
<password></password>
<db>default</db>
<table>ids</table>
<where>id=10</where>
</clickhouse>
</source>
```
Setting fields:
- `host` – The ClickHouse host. If it is a local host, the query is processed without any network activity. To improve fault tolerance, you can create a [Distributed](../table_engines/distributed.md#table_engines-distributed) table and enter it in subsequent configurations.
- `port` – The port on the ClickHouse server.
- `user` – Name of the ClickHouse user.
- `password` – Password of the ClickHouse user.
- `db` – Name of the database.
- `table` – Name of the table.
- `where ` – The selection criteria. May be omitted.
<a name="dicts-external_dicts_dict_sources-mongodb"></a>
### MongoDB
Example of settings:
```xml
<source>
<mongodb>
<host>localhost</host>
<port>27017</port>
<user></user>
<password></password>
<db>test</db>
<collection>dictionary_source</collection>
</mongodb>
</source>
```
Setting fields:
- `host` – The MongoDB host.
- `port` – The port on the MongoDB server.
- `user` – Name of the MongoDB user.
- `password` – Password of the MongoDB user.
- `db` – Name of the database.
- `collection` – Name of the collection.

122
docs/en/dicts/external_dicts_dict_structure.md Normal file
View File

@ -0,0 +1,122 @@
<a name="dicts-external_dicts_dict_structure"></a>
# Dictionary key and fields
The `<structure>` clause describes the dictionary key and fields available for queries.
Overall structure:
```xml
<dictionary>
<structure>
<id>
<name>Id</name>
</id>
<attribute>
<!-- Attribute parameters -->
</attribute>
...
</structure>
</dictionary>
```
Columns are described in the structure:
- `<id>` – [Key column](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure-key).
- `<attribute>` – [Data column](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure-attributes). There can be a large number of columns.
<a name="dicts-external_dicts_dict_structure-key"></a>
## Key
ClickHouse supports the following types of keys:
- Numeric key. UInt64. Defined in the tag `<id>` .
- Composite key. Set of values of different types. Defined in the tag `<key>` .
A structure can contain either `<id>` or `<key>` .
<div class="admonition attention">
The key doesn't need to be defined separately in attributes.
</div>
### Numeric key
Format: `UInt64`.
Configuration example:
```xml
<id>
<name>Id</name>
</id>
```
Configuration fields:
- name – The name of the column with keys.
### Composite key
The key can be a `tuple` from any types of fields. The [ layout](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout) in this case must be `complex_key_hashed` or `complex_key_cache`.
<div class="admonition tip">
A composite key can also consist of a single element, which makes it possible to use a string as the key, for instance.
</div>
The key structure is set in the element `<key>`. Key fields are specified in the same format as the dictionary [attributes](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure-attributes). Example:
```xml
<structure>
<key>
<attribute>
<name>field1</name>
<type>String</type>
</attribute>
<attribute>
<name>field2</name>
<type>UInt32</type>
</attribute>
...
</key>
...
```
For a query to the `dictGet*` function, a tuple is passed as the key. Example: `dictGetString('dict_name', 'attr_name', tuple('string for field1', num_for_field2))`.
<a name="dicts-external_dicts_dict_structure-attributes"></a>
## Attributes
Configuration example:
```xml
<structure>
...
<attribute>
<name>Name</name>
<type>Type</type>
<null_value></null_value>
<expression>rand64()</expression>
<hierarchical>true</hierarchical>
<injective>true</injective>
</attribute>
</structure>
```
Configuration fields:
- `name` – The column name.
- `type` – The column type. Sets the method for interpreting data in the source. For example, for MySQL, the field might be `TEXT`, `VARCHAR`, or `BLOB` in the source table, but it can be uploaded as `String`.
- `null_value` – The default value for a non-existing element. In the example, it is an empty string.
- `expression` – The attribute can be an expression. The tag is not required.
- `hierarchical` – Hierarchical support. Mirrored to the parent identifier. By default, ` false`.
- `injective` Whether the `id -> attribute` image is injective. If ` true`, then you can optimize the ` GROUP BY` clause. By default, `false`.

15
docs/en/dicts/index.md Normal file
View File

@ -0,0 +1,15 @@
# Dictionaries
`A dictionary` is a mapping (key `->` attributes) that can be used in a query as functions.
You can think of this as a more convenient and efficient type of JOIN with dimension tables.
There are built-in (internal) and add-on (external) dictionaries.
```eval_rst
.. toctree::
external_dicts
internal_dicts
```

11
docs/en/dicts/index.rst
View File

@ -1,11 +0,0 @@
Dictionaries
============
A dictionary is a mapping (key -> attributes) that can be used in a query as functions. You can think of this as a more convenient and efficient type of JOIN with dimension tables.
There are built-in (internal) and add-on (external) dictionaries.
.. toctree::
:glob:
*

49
docs/en/dicts/internal_dicts.md Normal file
View File

@ -0,0 +1,49 @@
# Internal dictionaries
ClickHouse contains a built-in feature for working with a geobase.
This allows you to:
- Use a region's ID to get its name in the desired language.
- Use a region's ID to get the ID of a city, area, federal district, country, or continent.
- Check whether a region is part of another region.
- Get a chain of parent regions.
All the functions support "translocality," the ability to simultaneously use different perspectives on region ownership. For more information, see the section "Functions for working with Yandex.Metrica dictionaries".
The internal dictionaries are disabled in the default package.
To enable them, uncomment the parameters `path_to_regions_hierarchy_file` and `path_to_regions_names_files` in the server configuration file.
The geobase is loaded from text files.
If you work at Yandex, you can follow these instructions to create them:
<https://github.yandex-team.ru/raw/Metrika/ClickHouse_private/master/doc/create_embedded_geobase_dictionaries.txt>
Put the regions_hierarchy\*.txt files in the path_to_regions_hierarchy_file directory. This configuration parameter must contain the path to the regions_hierarchy.txt file (the default regional hierarchy), and the other files (regions_hierarchy_ua.txt) must be located in the same directory.
Put the `regions_names_*.txt` files in the path_to_regions_names_files directory.
You can also create these files yourself. The file format is as follows:
`regions_hierarchy*.txt`: TabSeparated (no header), columns:
- Region ID (UInt32)
- Parent region ID (UInt32)
- Region type (UInt8): 1 - continent, 3 - country, 4 - federal district, 5 - region, 6 - city; other types don't have values.
- Population (UInt32) - Optional column.
`regions_names_*.txt`: TabSeparated (no header), columns:
- Region ID (UInt32)
- Region name (String) - Can't contain tabs or line feeds, even escaped ones.
A flat array is used for storing in RAM. For this reason, IDs shouldn't be more than a million.
Dictionaries can be updated without restarting the server. However, the set of available dictionaries is not updated.
For updates, the file modification times are checked. If a file has changed, the dictionary is updated.
The interval to check for changes is configured in the 'builtin_dictionaries_reload_interval' parameter.
Dictionary updates (other than loading at first use) do not block queries. During updates, queries use the old versions of dictionaries. If an error occurs during an update, the error is written to the server log, and queries continue using the old version of dictionaries.
We recommend periodically updating the dictionaries with the geobase. During an update, generate new files and write them to a separate location. When everything is ready, rename them to the files used by the server.
There are also functions for working with OS identifiers and Yandex.Metrica search engines, but they shouldn't be used.

45
docs/en/dicts/internal_dicts.rst
View File

@ -1,45 +0,0 @@
Internal dictionaries
---------------------
ClickHouse contains a built-in feature for working with a geobase.
This allows you to:
* Use a region's ID to get its name in the desired language.
* Use a region's ID to get the ID of a city, area, federal district, country, or continent.
* Check whether a region is part of another region.
* Get a chain of parent regions.
All the functions support "translocality", the ability to simultaneously use different perspectives on region ownership. For more information, see the section "Functions for working with Yandex.Metrica dictionaries".
The internal dictionaries are disabled in the default package.
To enable them, uncomment the parameters ``path_to_regions_hierarchy_file`` and ``path_to_regions_names_files`` in the server config file.
The geobase is loaded from text files.
If you are Yandex employee, to create them, use the following instructions:
https://github.yandex-team.ru/raw/Metrika/ClickHouse_private/master/doc/create_embedded_geobase_dictionaries.txt
Put the regions_hierarchy*.txt files in the path_to_regions_hierarchy_file directory. This configuration parameter must contain the path to the regions_hierarchy.txt file (the default regional hierarchy), and the other files (regions_hierarchy_ua.txt) must be located in the same directory.
Put the regions_names_*.txt files in the path_to_regions_names_files directory.
You can also create these files yourself. The file format is as follows:
``regions_hierarchy*.txt``: TabSeparated (no header), columns:
* Region ID (UInt32)
* Parent region ID (UInt32)
* Region type (UInt8): 1 - continent, 3 - country, 4 - federal district, 5 - region, 6 - city; other types don't have values.
* Population (UInt32) - Optional column.
``regions_names_*.txt``: TabSeparated (no header), columns:
* Region ID (UInt32)
* Region name (String) - Can't contain tabs or line breaks, even escaped ones.
A flat array is used for storing in RAM. For this reason, IDs shouldn't be more than a million.
Dictionaries can be updated without the server restart. However, the set of available dictionaries is not updated. For updates, the file modification times are checked. If a file has changed, the dictionary is updated.
The interval to check for changes is configured in the 'builtin_dictionaries_reload_interval' parameter.
Dictionary updates (other than loading at first use) do not block queries. During updates, queries use the old versions of dictionaries. If an error occurs during an update, the error is written to the server log, while queries continue using the old version of dictionaries.
We recommend periodically updating the dictionaries with the geobase. During an update, generate new files and write them to a separate location. When everything is ready, rename them to the files used by the server.
There are also functions for working with OS identifiers and Yandex.Metrica search engines, but they shouldn't be used.

26
docs/en/formats/capnproto.md Normal file
View File

@ -0,0 +1,26 @@
<a name="format_capnproto"></a>
# CapnProto
Cap'n Proto is a binary message format similar to Protocol Buffers and Thrift, but not like JSON or MessagePack.
Cap'n Proto messages are strictly typed and not self-describing, meaning they need an external schema description. The schema is applied on the fly and cached for each query.
```sql
SELECT SearchPhrase, count() AS c FROM test.hits
GROUP BY SearchPhrase FORMAT CapnProto SETTINGS schema = 'schema:Message'
```
Where `schema.capnp` looks like this:
```
struct Message {
SearchPhrase @0 :Text;
c @1 :Uint64;
}
```
Schema files are in the file that is located in the directory specified in [ format_schema_path](../operations/server_settings/settings.md#server_settings-format_schema_path) in the server configuration.
Deserialization is effective and usually doesn't increase the system load.

29
docs/en/formats/capnproto.rst
View File

@ -1,29 +0,0 @@
CapnProto
---------
Cap'n Proto is a binary message format. Like Protocol Buffers and Thrift (but unlike JSON or MessagePack), Cap'n Proto messages are strongly-typed and not self-describing. Due to this, it requires a ``schema`` setting to specify schema file and the root object. The schema is parsed on runtime and cached for each SQL statement.
.. code-block:: sql
SELECT SearchPhrase, count() AS c FROM test.hits
GROUP BY SearchPhrase FORMAT CapnProto SETTINGS schema = 'schema:Message'
When the `schema.capnp` schema file looks like:
.. code-block:: text
struct Message {
SearchPhrase @0 :Text;
c @1 :Uint64;
}
The schema files are located in the path specified in the configuration file:
.. code-block:: xml
<!-- Directory containing schema files for various input formats. -->
<format_schema_path>format_schemas/</format_schema_path>
Deserialization is almost as efficient as the binary rows format, with typically zero allocation overhead per message.
You can use this format as an efficient exchange message format in your data processing pipeline.

10
docs/en/formats/csv.md Normal file
View File

@ -0,0 +1,10 @@
# CSV -15
Comma Separated Values format ([RFC](https://tools.ietf.org/html/rfc4180)).
When formatting, rows are enclosed in double quotes. A double quote inside a string is output as two double quotes in a row. There are no other rules for escaping characters. Date and date-time are enclosed in double quotes. Numbers are output without quotes. Values ​​are separated by commas. Rows are separated using the Unix line feed (LF). Arrays are serialized in CSV as follows: first the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double quotes. Tuples in CSV format are serialized as separate columns (that is, their nesting in the tuple is lost).
When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to a comma or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) are all supported.
The CSV format supports the output of totals and extremes the same way as `TabSeparated`.

10
docs/en/formats/csv.rst
View File

@ -1,10 +0,0 @@
CSV
---
Comma separated values (`RFC <https://tools.ietf.org/html/rfc4180>`_).
String values are output in double quotes. Double quote inside a string is output as two consecutive double quotes. That's all escaping rules. Date and DateTime values are output in double quotes. Numbers are output without quotes. Fields are delimited by commas. Rows are delimited by unix newlines (LF). Arrays are output in following way: first, array are serialized to String (as in TabSeparated or Values formats), and then the String value are output in double quotes. Tuples are narrowed and serialized as separate columns.
During parsing, values could be enclosed or not enclosed in quotes. Supported both single and double quotes. In particular, Strings could be represented without quotes - in that case, they are parsed up to comma or newline (CR or LF). Contrary to RFC, in case of parsing strings without quotes, leading and trailing spaces and tabs are ignored. As line delimiter, both Unix (LF), Windows (CR LF) or Mac OS Classic (LF CR) variants are supported.
CSV format supports output of totals and extremes similar to TabSeparated format.

4
docs/en/formats/csvwithnames.md Normal file
View File

@ -0,0 +1,4 @@
# CSVWithNames
Also prints the header row, similar to `TabSeparatedWithNames`.

4
docs/en/formats/csvwithnames.rst
View File

@ -1,4 +0,0 @@
CSVWithNames
------------
Also contains header, similar to ``TabSeparatedWithNames``.

13
docs/en/formats/index.md Normal file
View File

@ -0,0 +1,13 @@
<a name="formats"></a>
# Formats
The format determines how data is returned to you after SELECTs (how it is written and formatted by the server), and how it is accepted for INSERTs (how it is read and parsed by the server).
```eval_rst
.. toctree::
:glob:
*
```

9
docs/en/formats/index.rst
View File

@ -1,9 +0,0 @@
Formats
=======
The format determines how data is given (written by server as output) to you after SELECTs, and how it is accepted (read by server as input) for INSERTs.
.. toctree::
:glob:
*

86
docs/en/formats/json.md Normal file
View File

@ -0,0 +1,86 @@
# JSON
Outputs data in JSON format. Besides data tables, it also outputs column names and types, along with some additional information: the total number of output rows, and the number of rows that could have been output if there weren't a LIMIT. Example:
```sql
SELECT SearchPhrase, count() AS c FROM test.hits GROUP BY SearchPhrase WITH TOTALS ORDER BY c DESC LIMIT 5 FORMAT JSON
```
```json
{
"meta":
[
{
"name": "SearchPhrase",
"type": "String"
},
{
"name": "c",
"type": "UInt64"
}
],
"data":
[
{
"SearchPhrase": "",
"c": "8267016"
},
{
"SearchPhrase": "bathroom interior design",
"c": "2166"
},
{
"SearchPhrase": "yandex",
"c": "1655"
},
{
"SearchPhrase": "spring 2014 fashion",
"c": "1549"
},
{
"SearchPhrase": "freeform photo",
"c": "1480"
}
],
"totals":
{
"SearchPhrase": "",
"c": "8873898"
},
"extremes":
{
"min":
{
"SearchPhrase": "",
"c": "1480"
},
"max":
{
"SearchPhrase": "",
"c": "8267016"
}
},
"rows": 5,
"rows_before_limit_at_least": 141137
}
```
The JSON is compatible with JavaScript. To ensure this, some characters are additionally escaped: the slash ` /` is escaped as ` \/`; alternative line breaks ` U+2028` and ` U+2029`, which break some browsers, are escaped as ` \uXXXX`. ASCII control characters are escaped: backspace, form feed, line feed, carriage return, and horizontal tab are replaced with `\b`, `\f`, `\n`, `\r`, `\t` , as well as the remaining bytes in the 00-1F range using `\uXXXX` sequences. Invalid UTF-8 sequences are changed to the replacement character <20> so the output text will consist of valid UTF-8 sequences. For compatibility with JavaScript, Int64 and UInt64 integers are enclosed in double quotes by default. To remove the quotes, you can set the configuration parameter output_format_json_quote_64bit_integers to 0.
`rows` – The total number of output rows.
`rows_before_limit_at_least` The minimal number of rows there would have been without LIMIT. Output only if the query contains LIMIT.
If the query contains GROUP BY, rows_before_limit_at_least is the exact number of rows there would have been without a LIMIT.
`totals` – Total values (when using WITH TOTALS).
`extremes` – Extreme values (when extremes is set to 1).
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
See also the JSONEachRow format.

87
docs/en/formats/json.rst
View File

@ -1,87 +0,0 @@
JSON
----
Outputs data in JSON format. Besides data tables, it also outputs column names and types, along with some additional information - the total number of output rows, and the number of rows that could have been output if there weren't a LIMIT. Example:
.. code-block:: sql
SELECT SearchPhrase, count() AS c FROM test.hits GROUP BY SearchPhrase WITH TOTALS ORDER BY c DESC LIMIT 5 FORMAT JSON
.. code-block:: json
{
"meta":
[
{
"name": "SearchPhrase",
"type": "String"
},
{
"name": "c",
"type": "UInt64"
}
],
"data":
[
{
"SearchPhrase": "",
"c": "8267016"
},
{
"SearchPhrase": "интерьер ванной комнаты",
"c": "2166"
},
{
"SearchPhrase": "яндекс",
"c": "1655"
},
{
"SearchPhrase": "весна 2014 мода",
"c": "1549"
},
{
"SearchPhrase": "фриформ фото",
"c": "1480"
}
],
"totals":
{
"SearchPhrase": "",
"c": "8873898"
},
"extremes":
{
"min":
{
"SearchPhrase": "",
"c": "1480"
},
"max":
{
"SearchPhrase": "",
"c": "8267016"
}
},
"rows": 5,
"rows_before_limit_at_least": 141137
}
JSON is compatible with JavaScript. For this purpose, certain symbols are additionally escaped: the forward slash ``/`` is escaped as ``\/``; alternative line breaks ``U+2028`` and ``U+2029``, which don't work in some browsers, are escaped as \uXXXX-sequences. ASCII control characters are escaped: backspace, form feed, line feed, carriage return, and horizontal tab as ``\b``, ``\f``, ``\n``, ``\r``, and ``\t`` respectively, along with the rest of the bytes from the range 00-1F using \uXXXX-sequences. Invalid UTF-8 sequences are changed to the replacement character ``<EFBFBD>`` and, thus, the output text will consist of valid UTF-8 sequences. UInt64 and Int64 numbers are output in double quotes for compatibility with JavaScript.
``rows`` - The total number of output rows.
``rows_before_limit_at_least`` - The minimal number of rows there would have been without a LIMIT. Output only if the query contains LIMIT.
If the query contains GROUP BY, ``rows_before_limit_at_least`` is the exact number of rows there would have been without a LIMIT.
``totals`` - Total values (when using WITH TOTALS).
``extremes`` - Extreme values (when extremes is set to 1).
This format is only appropriate for outputting a query result, not for parsing.
See JSONEachRow format for INSERT queries.

46
docs/en/formats/jsoncompact.md Normal file
View File

@ -0,0 +1,46 @@
# JSONCompact
Differs from JSON only in that data rows are output in arrays, not in objects.
Example:
```json
{
"meta":
[
{
"name": "SearchPhrase",
"type": "String"
},
{
"name": "c",
"type": "UInt64"
}
],
"data":
[
["", "8267016"],
["bathroom interior design", "2166"],
["yandex", "1655"],
["spring 2014 fashion", "1549"],
["freeform photos", "1480"]
],
"totals": ["","8873898"],
"extremes":
{
"min": ["","1480"],
"max": ["","8267016"]
},
"rows": 5,
"rows_before_limit_at_least": 141137
}
```
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
See also the `JSONEachRow` format.

46
docs/en/formats/jsoncompact.rst
View File

@ -1,46 +0,0 @@
JSONCompact
-----------
Differs from ``JSON`` only in that data rows are output in arrays, not in objects.
Example:
.. code-block:: text
{
"meta":
[
{
"name": "SearchPhrase",
"type": "String"
},
{
"name": "c",
"type": "UInt64"
}
],
"data":
[
["", "8267016"],
["bath interiors", "2166"],
["yandex", "1655"],
["spring 2014 fashion", "1549"],
["freeform photo", "1480"]
],
"totals": ["","8873898"],
"extremes":
{
"min": ["","1480"],
"max": ["","8267016"]
},
"rows": 5,
"rows_before_limit_at_least": 141137
}
This format is only appropriate for outputting a query result, not for parsing.
See ``JSONEachRow`` format for INSERT queries.

21
docs/en/formats/jsoneachrow.md Normal file
View File

@ -0,0 +1,21 @@
# JSONEachRow
Outputs data as separate JSON objects for each row (newline delimited JSON).
```json
{"SearchPhrase":"","count()":"8267016"}
{"SearchPhrase":"bathroom interior design","count()":"2166"}
{"SearchPhrase":"yandex","count()":"1655"}
{"SearchPhrase":"spring 2014 fashion","count()":"1549"}
{"SearchPhrase":"freeform photo","count()":"1480"}
{"SearchPhrase":"angelina jolie","count()":"1245"}
{"SearchPhrase":"omsk","count()":"1112"}
{"SearchPhrase":"photos of dog breeds","count()":"1091"}
{"SearchPhrase":"curtain design","count()":"1064"}
{"SearchPhrase":"baku","count()":"1000"}
```
Unlike the JSON format, there is no substitution of invalid UTF-8 sequences. Any set of bytes can be output in the rows. This is necessary so that data can be formatted without losing any information. Values are escaped in the same way as for JSON.
For parsing, any order is supported for the values of different columns. It is acceptable for some values to be omitted – they are treated as equal to their default values. In this case, zeros and blank rows are used as default values. Complex values that could be specified in the table are not supported as defaults. Whitespace between elements is ignored. If a comma is placed after the objects, it is ignored. Objects don't necessarily have to be separated by new lines.

25
docs/en/formats/jsoneachrow.rst
View File

@ -1,25 +0,0 @@
JSONEachRow
-----------
If put in SELECT query, displays data in newline delimited JSON (JSON objects separated by \\n character) format.
If put in INSERT query, expects this kind of data as input.
.. code-block:: text
{"SearchPhrase":"","count()":"8267016"}
{"SearchPhrase":"bathroom interior","count()":"2166"}
{"SearchPhrase":"yandex","count()":"1655"}
{"SearchPhrase":"spring 2014 fashion","count()":"1549"}
{"SearchPhrase":"free-form photo","count()":"1480"}
{"SearchPhrase":"Angelina Jolie","count()":"1245"}
{"SearchPhrase":"omsk","count()":"1112"}
{"SearchPhrase":"photos of dog breeds","count()":"1091"}
{"SearchPhrase":"curtain design","count()":"1064"}
{"SearchPhrase":"baku","count()":"1000"}
Unlike JSON format, there are no replacements of invalid UTF-8 sequences. There can be arbitrary amount of bytes in a line.
This is done in order to avoid data loss during formatting. Values are displayed analogous to JSON format.
In INSERT queries JSON data can be supplied with arbitrary order of columns (JSON key-value pairs). It is also possible to omit values in which case the default value of the column is inserted. N.B. when using JSONEachRow format, complex default values are not supported, so when omitting a column its value will be zeros or empty string depending on its type.
Space characters between JSON objects are skipped. Between objects there can be a comma which is ignored. Newline character is not a mandatory separator for objects.

6
docs/en/formats/native.rst → docs/en/formats/native.md
View File

@ -1,6 +1,6 @@
Native
------
# Native
The most efficient format. Data is written and read by blocks in binary format. For each block, the number of rows, number of columns, column names and types, and parts of columns in this block are recorded one after another. In other words, this format is "columnar" - it doesn't convert columns to rows. This is the format used in the native interface for interaction between servers, for using the command-line client, and for C++ clients.
The most efficient format. Data is written and read by blocks in binary format. For each block, the number of rows, number of columns, column names and types, and parts of columns in this block are recorded one after another. In other words, this format is "columnar" – it doesn't convert columns to rows. This is the format used in the native interface for interaction between servers, for using the command-line client, and for C++ clients.
You can use this format to quickly generate dumps that can only be read by the ClickHouse DBMS. It doesn't make sense to work with this format yourself.

5
docs/en/formats/null.md Normal file
View File

@ -0,0 +1,5 @@
# Null
Nothing is output. However, the query is processed, and when using the command-line client, data is transmitted to the client. This is used for tests, including productivity testing.
Obviously, this format is only appropriate for output, not for parsing.

4
docs/en/formats/null.rst
View File

@ -1,4 +0,0 @@
Null
----
Nothing is output. However, the query is processed, and when using the command-line client, data is transmitted to the client. This is used for tests, including productivity testing. Obviously, this format is only appropriate for outputting a query result, not for parsing.

37
docs/en/formats/pretty.md Normal file
View File

@ -0,0 +1,37 @@
# Pretty
Outputs data as Unicode-art tables, also using ANSI-escape sequences for setting colors in the terminal.
A full grid of the table is drawn, and each row occupies two lines in the terminal.
Each result block is output as a separate table. This is necessary so that blocks can be output without buffering results (buffering would be necessary in order to pre-calculate the visible width of all the values).
To avoid dumping too much data to the terminal, only the first 10,000 rows are printed. If the number of rows is greater than or equal to 10,000, the message "Showed first 10 000" is printed.
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
The Pretty format supports outputting total values (when using WITH TOTALS) and extremes (when 'extremes' is set to 1). In these cases, total values and extreme values are output after the main data, in separate tables. Example (shown for the PrettyCompact format):
```sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT PrettyCompact
```
```text
┌──EventDate─┬───────c─┐
│ 2014-03-17 │ 1406958 │
│ 2014-03-18 │ 1383658 │
│ 2014-03-19 │ 1405797 │
│ 2014-03-20 │ 1353623 │
│ 2014-03-21 │ 1245779 │
│ 2014-03-22 │ 1031592 │
│ 2014-03-23 │ 1046491 │
└────────────┴─────────┘
Totals:
┌──EventDate─┬───────c─┐
│ 0000-00-00 │ 8873898 │
└────────────┴─────────┘
Extremes:
┌──EventDate─┬───────c─┐
│ 2014-03-17 │ 1031592 │
│ 2014-03-23 │ 1406958 │
└────────────┴─────────┘
```

36
docs/en/formats/pretty.rst
View File

@ -1,36 +0,0 @@
Pretty
------
Writes data as Unicode-art tables, also using ANSI-escape sequences for setting colors in the terminal.
A full grid of the table is drawn, and each row occupies two lines in the terminal. Each result block is output as a separate table. This is necessary so that blocks can be output without buffering results (buffering would be necessary in order to pre-calculate the visible width of all the values).
To avoid dumping too much data to the terminal, only the first 10,000 rows are printed. If the number of rows is greater than or equal to 10,000, the message "Showed first 10,000" is printed.
This format is only appropriate for outputting a query result, not for parsing.
The Pretty format supports outputting total values (when using WITH TOTALS) and extremes (when 'extremes' is set to 1). In these cases, total values and extreme values are output after the main data, in separate tables. Example (shown for the PrettyCompact format):
.. code-block:: sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT PrettyCompact
.. code-block:: text
┌──EventDate─┬───────c─┐
│ 2014-03-17 │ 1406958 │
│ 2014-03-18 │ 1383658 │
│ 2014-03-19 │ 1405797 │
│ 2014-03-20 │ 1353623 │
│ 2014-03-21 │ 1245779 │
│ 2014-03-22 │ 1031592 │
│ 2014-03-23 │ 1046491 │
└────────────┴─────────┘
Totals:
┌──EventDate─┬───────c─┐
│ 0000-00-00 │ 8873898 │
└────────────┴─────────┘
Extremes:
┌──EventDate─┬───────c─┐
│ 2014-03-17 │ 1031592 │
│ 2014-03-23 │ 1406958 │
└────────────┴─────────┘

5
docs/en/formats/prettycompact.md Normal file
View File

@ -0,0 +1,5 @@
# PrettyCompact
Differs from `Pretty` in that the grid is drawn between rows and the result is more compact.
This format is used by default in the command-line client in interactive mode.

4
docs/en/formats/prettycompact.rst
View File

@ -1,4 +0,0 @@
PrettyCompact
-------------
Differs from ``Pretty`` in that the grid is drawn between rows and the result is more compact. This format is used by default in the command-line client in interactive mode.

4
docs/en/formats/prettycompactmonoblock.md Normal file
View File

@ -0,0 +1,4 @@
# PrettyCompactMonoBlock
Differs from `PrettyCompact` in that up to 10,000 rows are buffered, then output as a single table, not by blocks.

4
docs/en/formats/prettycompactmonoblock.rst
View File

@ -1,4 +0,0 @@
PrettyCompactMonoBlock
----------------------
Differs from ``PrettyCompact`` in that up to 10,000 rows are buffered, then output as a single table, not by blocks.

20
docs/en/formats/prettynoescapes.md Normal file
View File

@ -0,0 +1,20 @@
# PrettyNoEscapes
Differs from Pretty in that ANSI-escape sequences aren't used. This is necessary for displaying this format in a browser, as well as for using the 'watch' command-line utility.
Example:
```bash
watch -n1 "clickhouse-client --query='SELECT * FROM system.events FORMAT PrettyCompactNoEscapes'"
```
You can use the HTTP interface for displaying in the browser.
## PrettyCompactNoEscapes
The same as the previous setting.
## PrettySpaceNoEscapes
The same as the previous setting.

20
docs/en/formats/prettynoescapes.rst
View File

@ -1,20 +0,0 @@
PrettyNoEscapes
---------------
Differs from Pretty in that ANSI-escape sequences aren't used. This is necessary for displaying this format in a browser, as well as for using the 'watch' command-line utility.
Example:
.. code-block:: text
watch -n1 "clickhouse-client --query='SELECT * FROM system.events FORMAT PrettyCompactNoEscapes'"
You can use the HTTP interface for displaying in the browser.
PrettyCompactNoEscapes
----------------------
The same.
PrettySpaceNoEscapes
--------------------
The same.

4
docs/en/formats/prettyspace.md Normal file
View File

@ -0,0 +1,4 @@
# PrettySpace
Differs from `PrettyCompact` in that whitespace (space characters) is used instead of the grid.

4
docs/en/formats/prettyspace.rst
View File

@ -1,4 +0,0 @@
PrettySpace
-----------
Differs from ``PrettyCompact`` in that whitespace (space characters) is used instead of the grid.

13
docs/en/formats/rowbinary.md Normal file
View File

@ -0,0 +1,13 @@
# RowBinary
Formats and parses data by row in binary format. Rows and values are listed consecutively, without separators.
This format is less efficient than the Native format, since it is row-based.
Integers use fixed-length little endian representation. For example, UInt64 uses 8 bytes.
DateTime is represented as UInt32 containing the Unix timestamp as the value.
Date is represented as a UInt16 object that contains the number of days since 1970-01-01 as the value.
String is represented as a varint length (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string.
FixedString is represented simply as a sequence of bytes.
Arrays are represented as a varint length (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), followed by the array elements in order.

13
docs/en/formats/rowbinary.rst
View File

@ -1,13 +0,0 @@
RowBinary
---------
Writes data by row in binary format. Rows and values are listed consecutively, without separators.
This format is less efficient than the Native format, since it is row-based.
Numbers is written in little endian, fixed width. For example, UInt64 takes 8 bytes.
DateTime is written as UInt32 with unix timestamp value.
Date is written as UInt16 with number of days since 1970-01-01 in value.
String is written as length in varint (unsigned `LEB128 <https://en.wikipedia.org/wiki/LEB128>`_) format and then bytes of string.
FixedString is written as just its bytes.
Array is written as length in varint (unsigned `LEB128 <https://en.wikipedia.org/wiki/LEB128>`_) format and then all elements, contiguously

59
docs/en/formats/tabseparated.md Normal file
View File

@ -0,0 +1,59 @@
# TabSeparated
In TabSeparated format, data is written by row. Each row contains values separated by tabs. Each value is follow by a tab, except the last value in the row, which is followed by a line feed. Strictly Unix line feeds are assumed everywhere. The last row also must contain a line feed at the end. Values are written in text format, without enclosing quotation marks, and with special characters escaped.
Integer numbers are written in decimal form. Numbers can contain an extra "+" character at the beginning (ignored when parsing, and not recorded when formatting). Non-negative numbers can't contain the negative sign. When reading, it is allowed to parse an empty string as a zero, or (for signed types) a string consisting of just a minus sign as a zero. Numbers that do not fit into the corresponding data type may be parsed as a different number, without an error message.
Floating-point numbers are written in decimal form. The dot is used as the decimal separator. Exponential entries are supported, as are 'inf', '+inf', '-inf', and 'nan'. An entry of floating-point numbers may begin or end with a decimal point.
During formatting, accuracy may be lost on floating-point numbers.
During parsing, it is not strictly required to read the nearest machine-representable number.
Dates are written in YYYY-MM-DD format and parsed in the same format, but with any characters as separators.
Dates with times are written in the format YYYY-MM-DD hh:mm:ss and parsed in the same format, but with any characters as separators.
This all occurs in the system time zone at the time the client or server starts (depending on which one formats data). For dates with times, daylight saving time is not specified. So if a dump has times during daylight saving time, the dump does not unequivocally match the data, and parsing will select one of the two times.
During a read operation, incorrect dates and dates with times can be parsed with natural overflow or as null dates and times, without an error message.
As an exception, parsing dates with times is also supported in Unix timestamp format, if it consists of exactly 10 decimal digits. The result is not time zone-dependent. The formats YYYY-MM-DD hh:mm:ss and NNNNNNNNNN are differentiated automatically.
Strings are output with backslash-escaped special characters. The following escape sequences are used for output: `\b`, `\f`, `\r`, `\n`, `\t`, `\0`, `\'`, `\\`. Parsing also supports the sequences `\a`, `\v`, and `\xHH` (hex escape sequences) and any `\c` sequences, where `c` is any character (these sequences are converted to `c`). Thus, reading data supports formats where a line feed can be written as `\n` or `\`, or as a line feed. For example, the string `Hello world` with a line feed between the words instead of a space can be parsed in any of the following variations:
```text
Hello\nworld
Hello\
world
```
The second variant is supported because MySQL uses it when writing tab-separated dumps.
The minimum set of characters that you need to escape when passing data in TabSeparated format: tab, line feed (LF) and backslash.
Only a small set of symbols are escaped. You can easily stumble onto a string value that your terminal will ruin in output.
Arrays are written as a list of comma-separated values in square brackets. Number items in the array are fomratted as normally, but dates, dates with times, and strings are written in single quotes with the same escaping rules as above.
The TabSeparated format is convenient for processing data using custom programs and scripts. It is used by default in the HTTP interface, and in the command-line client's batch mode. This format also allows transferring data between different DBMSs. For example, you can get a dump from MySQL and upload it to ClickHouse, or vice versa.
The TabSeparated format supports outputting total values (when using WITH TOTALS) and extreme values (when 'extremes' is set to 1). In these cases, the total values and extremes are output after the main data. The main result, total values, and extremes are separated from each other by an empty line. Example:
```sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated``
```
```text
2014-03-17 1406958
2014-03-18 1383658
2014-03-19 1405797
2014-03-20 1353623
2014-03-21 1245779
2014-03-22 1031592
2014-03-23 1046491
0000-00-00 8873898
2014-03-17 1031592
2014-03-23 1406958
```
This format is also available under the name `TSV`.

59
docs/en/formats/tabseparated.rst
View File

@ -1,59 +0,0 @@
TabSeparated
------------
In TabSeparated format, data is written by row. Each row contains values separated by tabs. Each value is follow by a tab, except the last value in the row, which is followed by a line break. Strictly Unix line breaks are assumed everywhere. The last row also must contain a line break at the end. Values are written in text format, without enclosing quotation marks, and with special characters escaped.
Numbers are written in decimal form. Numbers may contain an extra "+" symbol at the beginning (but it is not recorded during an output). Non-negative numbers can't contain the negative sign. When parsing, it is allowed to parse an empty string as a zero, or (for signed types) a string consisting of just a minus sign as a zero. Numbers that do not fit into the corresponding data type may be parsed as a different number, without an error message.
Floating-point numbers are formatted in decimal form. The dot is used as the decimal separator. Exponential entries are supported, as are 'inf', '+inf', '-inf', and 'nan'. An entry of floating-point numbers may begin or end with a decimal point.
During formatting, accuracy may be lost on floating-point numbers.
During parsing, a result is not necessarily the nearest machine-representable number.
Dates are formatted in YYYY-MM-DD format and parsed in the same format, but with any characters as separators.
DateTimes are formatted in the format YYYY-MM-DD hh:mm:ss and parsed in the same format, but with any characters as separators.
This all occurs in the system time zone at the time the client or server starts (depending on which one formats data). For DateTimes, daylight saving time is not specified. So if a dump has times during daylight saving time, the dump does not unequivocally match the data, and parsing will select one of the two times.
During a parsing operation, incorrect dates and dates with times can be parsed with natural overflow or as null dates and times, without an error message.
As an exception, parsing DateTime is also supported in Unix timestamp format, if it consists of exactly 10 decimal digits. The result is not time zone-dependent. The formats ``YYYY-MM-DD hh:mm:ss`` and ``NNNNNNNNNN`` are differentiated automatically.
Strings are parsed and formatted with backslash-escaped special characters. The following escape sequences are used while formatting: ``\b``, ``\f``, ``\r``, ``\n``, ``\t``, ``\0``, ``\'``, and ``\\``. For parsing, also supported \a, \v and \xHH (hex escape sequence) and any sequences of the type \c where c is any character (these sequences are converted to c). This means that parsing supports formats where a line break can be written as \n or as \ and a line break. For example, the string 'Hello world' with a line break between the words instead of a space can be retrieved in any of the following variations:
.. code-block:: text
Hello\nworld
Hello\
world
The second variant is supported because MySQL uses it when writing tab-separated dumps.
Only a small set of symbols are escaped. You can easily stumble onto a string value that your terminal will ruin in output.
Minimum set of symbols that you must escape in TabSeparated format is tab, newline (LF) and backslash.
Arrays are formatted as a list of comma-separated values in square brackets. Number items in the array are formatted as normally, but dates, dates with times, and strings are formatted in single quotes with the same escaping rules as above.
The TabSeparated format is convenient for processing data using custom programs and scripts. It is used by default in the HTTP interface, and in the command-line client's batch mode. This format also allows transferring data between different DBMSs. For example, you can get a dump from MySQL and upload it to ClickHouse, or vice versa.
The TabSeparated format supports outputting total values (when using WITH TOTALS) and extreme values (when 'extremes' is set to 1). In these cases, the total values and extremes are output after the main data. The main result, total values, and extremes are separated from each other by an empty line. Example:
.. code-block:: sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated
.. code-block:: text
2014-03-17 1406958
2014-03-18 1383658
2014-03-19 1405797
2014-03-20 1353623
2014-03-21 1245779
2014-03-22 1031592
2014-03-23 1046491
0000-00-00 8873898
2014-03-17 1031592
2014-03-23 1406958
It's also available as ``TSV``.

7
docs/en/formats/tabseparatedraw.md Normal file
View File

@ -0,0 +1,7 @@
# TabSeparatedRaw
Differs from `TabSeparated` format in that the rows are written without escaping.
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
This format is also available under the name ` TSVRaw`.

7
docs/en/formats/tabseparatedraw.rst
View File

@ -1,7 +0,0 @@
TabSeparatedRaw
---------------
Differs from the ``TabSeparated`` format in that the rows are formatted without escaping.
This format is only appropriate for outputting a query result, but not for parsing data to insert into a table.
It's also available as ``TSVRaw``.

8
docs/en/formats/tabseparatedwithnames.md Normal file
View File

@ -0,0 +1,8 @@
# TabSeparatedWithNames
Differs from the `TabSeparated` format in that the column names are written in the first row.
During parsing, the first row is completely ignored. You can't use column names to determine their position or to check their correctness.
(Support for parsing the header row may be added in the future.)
This format is also available under the name ` TSVWithNames`.

8
docs/en/formats/tabseparatedwithnames.rst
View File

@ -1,8 +0,0 @@
TabSeparatedWithNames
---------------------
Differs from the TabSeparated format in that the column names are output in the first row.
For parsing, the first row is completely ignored. You can't use column names to determine their position or to check their correctness.
(Support for using header while parsing could be added in future.)
It's also available as ``TSVWithNames``.

7
docs/en/formats/tabseparatedwithnamesandtypes.md Normal file
View File

@ -0,0 +1,7 @@
# TabSeparatedWithNamesAndTypes
Differs from the `TabSeparated` format in that the column names are written to the first row, while the column types are in the second row.
During parsing, the first and second rows are completely ignored.
This format is also available under the name ` TSVWithNamesAndTypes`.

7
docs/en/formats/tabseparatedwithnamesandtypes.rst
View File

@ -1,7 +0,0 @@
TabSeparatedWithNamesAndTypes
-----------------------------
Differs from the ``TabSeparated`` format in that the column names are output to the first row, while the column types are in the second row.
For parsing, the first and second rows are completely ignored.
It's also available as ``TSVWithNamesAndTypes``.

23
docs/en/formats/tskv.md Normal file
View File

@ -0,0 +1,23 @@
# TSKV
Similar to TabSeparated, but outputs a value in name=value format. Names are escaped the same way as in TabSeparated format, and the = symbol is also escaped.
```text
SearchPhrase= count()=8267016
SearchPhrase=bathroom interior design count()=2166
SearchPhrase=yandex count()=1655
SearchPhrase=spring 2014 fashion count()=1549
SearchPhrase=freeform photos count()=1480
SearchPhrase=angelina jolia count()=1245
SearchPhrase=omsk count()=1112
SearchPhrase=photos of dog breeds count()=1091
SearchPhrase=curtain design count()=1064
SearchPhrase=baku count()=1000
```
When there is a large number of small columns, this format is ineffective, and there is generally no reason to use it. It is used in some departments of Yandex.
Both data output and parsing are supported in this format. For parsing, any order is supported for the values of different columns. It is acceptable for some values to be omitted – they are treated as equal to their default values. In this case, zeros and blank rows are used as default values. Complex values that could be specified in the table are not supported as defaults.
Parsing allows the presence of the additional field `tskv` without the equal sign or a value. This field is ignored.

21
docs/en/formats/tskv.rst
View File

@ -1,21 +0,0 @@
TSKV
----
Similar to TabSeparated, but displays data in name=value format. Names are displayed just as in TabSeparated. Additionally, a ``=`` symbol is displayed.
.. code-block:: text
SearchPhrase= count()=8267016
SearchPhrase=bathroom interior count()=2166
SearchPhrase=yandex count()=1655
SearchPhrase=spring 2014 fashion count()=1549
SearchPhrase=free-form photo count()=1480
SearchPhrase=Angelina Jolie count()=1245
SearchPhrase=omsk count()=1112
SearchPhrase=photos of dog breeds count()=1091
SearchPhrase=curtain design count()=1064
SearchPhrase=baku count()=1000
In case of many small columns this format is obviously not effective and there usually is no reason to use it. This format is supported because it is used for some cases in Yandex.
Format is supported both for input and output. In INSERT queries data can be supplied with arbitrary order of columns. It is also possible to omit values in which case the default value of the column is inserted. N.B. when using TSKV format, complex default values are not supported, so when omitting a column its value will be zeros or empty string depending on its type.

9
docs/en/formats/values.md Normal file
View File

@ -0,0 +1,9 @@
# Values
Prints every row in brackets. Rows are separated by commas. There is no comma after the last row. The values inside the brackets are also comma-separated. Numbers are output in decimal format without quotes. Arrays are output in square brackets. Strings, dates, and dates with times are output in quotes. Escaping rules and parsing are similar to the TabSeparated format. During formatting, extra spaces aren't inserted, but during parsing, they are allowed and skipped (except for spaces inside array values, which are not allowed).
The minimum set of characters that you need to escape when passing data in Values ​​format: single quotes and backslashes.
This is the format that is used in `INSERT INTO t VALUES ...`
Но вы также можете использовать его для форматирования результатов запросов.

9
docs/en/formats/values.rst
View File

@ -1,9 +0,0 @@
Values
------
Prints every row in parentheses. Rows are separated by commas. There is no comma after the last row. The values inside the parentheses are also comma-separated. Numbers are output in decimal format without quotes. Arrays are output in square brackets. Strings, dates, and dates with times are output in quotes. Escaping rules and parsing are same as in the TabSeparated format. During formatting, extra spaces aren't inserted, but during parsing, they are allowed and skipped (except for spaces inside array values, which are not allowed).
Minimum set of symbols that you must escape in Values format is single quote and backslash.
This is the format that is used in ``INSERT INTO t VALUES ...``
But you can also use it for query result.

6
docs/en/formats/vertical.rst → docs/en/formats/vertical.md
View File

@ -1,5 +1,5 @@
Vertical
--------
# Vertical
Prints each value on a separate line with the column name specified. This format is convenient for printing just one or a few rows, if each row consists of a large number of columns.
This format is only appropriate for outputting a query result, not for parsing.
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).

73
docs/en/formats/xml.md Normal file
View File

@ -0,0 +1,73 @@
# XML
XML format is suitable only for output, not for parsing. Example:
```xml
<?xml version='1.0' encoding='UTF-8' ?>
<result>
<meta>
<columns>
<column>
<name>SearchPhrase</name>
<type>String</type>
</column>
<column>
<name>count()</name>
<type>UInt64</type>
</column>
</columns>
</meta>
<data>
<row>
<SearchPhrase></SearchPhrase>
<field>8267016</field>
</row>
<row>
<SearchPhrase>bathroom interior design</SearchPhrase>
<field>2166</field>
</row>
<row>
<SearchPhrase>yandex</SearchPhrase>
<field>1655</field>
</row>
<row>
<SearchPhrase>spring 2014 fashion</SearchPhrase>
<field>1549</field>
</row>
<row>
<SearchPhrase>freeform photos</SearchPhrase>
<field>1480</field>
</row>
<row>
<SearchPhrase>angelina jolie</SearchPhrase>
<field>1245</field>
</row>
<row>
<SearchPhrase>omsk</SearchPhrase>
<field>1112</field>
</row>
<row>
<SearchPhrase>photos of dog breeds</SearchPhrase>
<field>1091</field>
</row>
<row>
<SearchPhrase>curtain design</SearchPhrase>
<field>1064</field>
</row>
<row>
<SearchPhrase>baku</SearchPhrase>
<field>1000</field>
</row>
</data>
<rows>10</rows>
<rows_before_limit_at_least>141137</rows_before_limit_at_least>
</result>
```
If the column name does not have an acceptable format, just 'field' is used as the element name. In general, the XML structure follows the JSON structure.
Just as for JSON, invalid UTF-8 sequences are changed to the replacement character <20> so the output text will consist of valid UTF-8 sequences.
In string values, the characters `<` and `&` are escaped as `<` and `&`.
Arrays are output as `<array><elem>Hello</elem><elem>World</elem>...</array>`,and tuples as `<tuple><elem>Hello</elem><elem>World</elem>...</tuple>`.

74
docs/en/formats/xml.rst
View File

@ -1,74 +0,0 @@
XML
---
XML format is supported only for displaying data, not for INSERTS. Example:
.. code-block:: xml
<?xml version='1.0' encoding='UTF-8' ?>
<result>
<meta>
<columns>
<column>
<name>SearchPhrase</name>
<type>String</type>
</column>
<column>
<name>count()</name>
<type>UInt64</type>
</column>
</columns>
</meta>
<data>
<row>
<SearchPhrase></SearchPhrase>
<field>8267016</field>
</row>
<row>
<SearchPhrase>bathroom interior</SearchPhrase>
<field>2166</field>
</row>
<row>
<SearchPhrase>yandex>
<field>1655</field>
</row>
<row>
<SearchPhrase>spring 2014 fashion</SearchPhrase>
<field>1549</field>
</row>
<row>
<SearchPhrase>free-form photo</SearchPhrase>
<field>1480</field>
</row>
<row>
<SearchPhrase>Angelina Jolie</SearchPhrase>
<field>1245</field>
</row>
<row>
<SearchPhrase>omsk</SearchPhrase>
<field>1112</field>
</row>
<row>
<SearchPhrase>photos of dog breeds</SearchPhrase>
<field>1091</field>
</row>
<row>
<SearchPhrase>curtain design</SearchPhrase>
<field>1064</field>
</row>
<row>
<SearchPhrase>baku</SearchPhrase>
<field>1000</field>
</row>
</data>
<rows>10</rows>
<rows_before_limit_at_least>141137</rows_before_limit_at_least>
</result>
If name of a column contains some unacceptable character, field is used as a name. In other aspects XML uses JSON structure.
As in case of JSON, invalid UTF-8 sequences are replaced by replacement character <20> so displayed text will only contain valid UTF-8 sequences.
In string values ``<`` and ``&`` are displayed as ``&lt;`` and ``&amp;``.
Arrays are displayed as ``<array><elem>Hello</elem><elem>World</elem>...</array>``,
and tuples as ``<tuple><elem>Hello</elem><elem>World</elem>...</tuple>``.

Some files were not shown because too many files have changed in this diff Show More