thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-20 00:30:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update of english documentation (#2918)

Browse Source 
* Updating of english translation.

* Some bugs are fixed.
This commit is contained in:
BayoNet 2018-09-04 14:18:59 +03:00 committed by Ivan Blinkov
parent 604c64f40d
commit 324fd98de6
96 changed files with 2703 additions and 1134 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

294
CHANGELOG.md
View File

@ -1,3 +1,16 @@
## ClickHouse release 18.6.0, 2018-08-02
### New features:
* Added support for ON expressions for the JOIN ON syntax:
`JOIN ON Expr([table.]column ...) = Expr([table.]column, ...) [AND Expr([table.]column, ...) = Expr([table.]column, ...) ...]`
Выражение должно представлять из себя цепочку равенств, объединенных оператором AND. Каждая часть равенства может являться произвольным выражением над столбцами одной из таблиц. Поддержана возможность использования fully qualified имен столбцов (`table.name`, `database.table.name`, `table_alias.name`, `subquery_alias.name`) for the right table. [#2742](https://github.com/yandex/ClickHouse/pull/2742)
* HTTPS can be enabled for replication. [#2760](https://github.com/yandex/ClickHouse/pull/2760)
### Improvements:
* The server passes the patch component of its version to the client. Data about the patch version component is in `system.processes` and `query_log`. [#2646](https://github.com/yandex/ClickHouse/pull/2646)
## ClickHouse release 18.5.1, 2018-07-31
### New features:
@ -6,7 +19,7 @@
### Improvements:
* Now you can use the `from_env` attribute to set values in config files from environment variables [#2741](https://github.com/yandex/ClickHouse/pull/2741).
* Now you can use the `from_env` [#2741](https://github.com/yandex/ClickHouse/pull/2741) attribute to set values in config files from environment variables.
* Added case-insensitive versions of the `coalesce`, `ifNull`, and `nullIf functions` [#2752](https://github.com/yandex/ClickHouse/pull/2752).
### Bug fixes:
@ -18,21 +31,21 @@
### New features:
* Added system tables: `formats`, `data_type_families`, `aggregate_function_combinators`, `table_functions`, `table_engines`, `collations` [#2721](https://github.com/yandex/ClickHouse/pull/2721).
* Added the ability to use a table function instead of a table as an argument of a `remote` or `cluster` table function [#2708](https://github.com/yandex/ClickHouse/pull/2708).
* Added the ability to use a table function instead of a table as an argument of a `remote` or `cluster table function` [#2708](https://github.com/yandex/ClickHouse/pull/2708).
* Support for `HTTP Basic` authentication in the replication protocol [#2727](https://github.com/yandex/ClickHouse/pull/2727).
* The `has` function now allows searching for a numeric value in an array of `Enum` values [Maxim Khrisanfov](https://github.com/yandex/ClickHouse/pull/2699).
* Support for adding arbitrary message separators when reading from `Kafka` [Amos Bird](https://github.com/yandex/ClickHouse/pull/2701).
### Improvements:
* The `ALTER TABLE t DELETE WHERE` query does not rewrite data chunks that were not affected by the WHERE condition [#2694](https://github.com/yandex/ClickHouse/pull/2694).
* The `ALTER TABLE t DELETE WHERE` query does not rewrite data parts that were not affected by the WHERE condition [#2694](https://github.com/yandex/ClickHouse/pull/2694).
* The `use_minimalistic_checksums_in_zookeeper` option for `ReplicatedMergeTree` tables is enabled by default. This setting was added in version 1.1.54378, 2018-04-16. Versions that are older than 1.1.54378 can no longer be installed.
* Support for running `KILL` and `OPTIMIZE` queries that specify `ON CLUSTER` [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2689).
### Bug fixes:
* Fixed the error `Column ... is not under an aggregate function and not in GROUP BY` for aggregation with an IN expression. This bug appeared in version 18.1.0. ([bbdd780b](https://github.com/yandex/ClickHouse/commit/bbdd780be0be06a0f336775941cdd536878dd2c2))
* Fixed a bug in the `windowFunnel` aggregate function [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2735).
* Fixed a bug in the `windowFunnel aggregate function` [Winter Zhang](https://github.com/yandex/ClickHouse/pull/2735).
* Fixed a bug in the `anyHeavy` aggregate function ([a2101df2](https://github.com/yandex/ClickHouse/commit/a2101df25a6a0fba99aa71f8793d762af2b801ee))
* Fixed server crash when using the `countArray()` aggregate function.
@ -72,7 +85,6 @@
* Converting a string containing the number zero to DateTime does not work. Example: `SELECT toDateTime('0')`. This is also the reason that `DateTime DEFAULT '0'` does not work in tables, as well as `<null_value>0</null_value>` in dictionaries. Solution: replace `0` with `0000-00-00 00:00:00`.
## ClickHouse release 1.1.54394, 2018-07-12
### New features:
@ -99,7 +111,7 @@
### Improvements:
* Improved performance, reduced memory consumption, and correct tracking of memory consumption with use of the IN operator when a table index could be used ([#2584](https://github.com/yandex/ClickHouse/pull/2584)).
* Improved performance, reduced memory consumption, and correct memory consumption tracking with use of the IN operator when a table index could be used ([#2584](https://github.com/yandex/ClickHouse/pull/2584)).
* Removed redundant checking of checksums when adding a data part. This is important when there are a large number of replicas, because in these cases the total number of checks was equal to N^2.
* Added support for `Array(Tuple(...))` arguments for the `arrayEnumerateUniq` function ([#2573](https://github.com/yandex/ClickHouse/pull/2573)).
* Added `Nullable` support for the `runningDifference` function ([#2594](https://github.com/yandex/ClickHouse/pull/2594)).
@ -126,8 +138,8 @@
### New features:
* Support for the `ALTER TABLE t DELETE WHERE` query for replicated tables. Added the `system.mutations` table to track progress of this type of queries.
* Support for the `ALTER TABLE t [REPLACE|ATTACH] PARTITION` query for MergeTree tables.
* Support for the `TRUNCATE TABLE` query ([Winter Zhang](https://github.com/yandex/ClickHouse/pull/2260)).
* Support for the `ALTER TABLE t [REPLACE|ATTACH] PARTITION` query for \*MergeTree tables.
* Support for the `TRUNCATE TABLE` query ([Winter Zhang](https://github.com/yandex/ClickHouse/pull/2260))
* Several new `SYSTEM` queries for replicated tables (`RESTART REPLICAS`, `SYNC REPLICA`, `[STOP|START] [MERGES|FETCHES|SENDS REPLICATED|REPLICATION QUEUES]`).
* Added the ability to write to a table with the MySQL engine and the corresponding table function ([sundy-li](https://github.com/yandex/ClickHouse/pull/2294)).
* Added the `url()` table function and the `URL` table engine ([Alexander Sapin](https://github.com/yandex/ClickHouse/pull/2501)).
@ -137,13 +149,13 @@
* The password to `clickhouse-client` can be entered interactively.
* Server logs can now be sent to syslog ([Alexander Krasheninnikov](https://github.com/yandex/ClickHouse/pull/2459)).
* Support for logging in dictionaries with a shared library source ([Alexander Sapin](https://github.com/yandex/ClickHouse/pull/2472)).
* Support for custom CSV delimiters ([Ivan Zhukov](https://github.com/yandex/ClickHouse/pull/2263)).
* Support for custom CSV delimiters ([Ivan Zhukov](https://github.com/yandex/ClickHouse/pull/2263))
* Added the `date_time_input_format` setting. If you switch this setting to `'best_effort'`, DateTime values will be read in a wide range of formats.
* Added the `clickhouse-obfuscator` utility for data obfuscation. Usage example: publishing data used in performance tests.
### Experimental features:
* Added the ability to calculate `and` arguments only where they are needed ([Anastasia Tsarkova](https://github.com/yandex/ClickHouse/pull/2272)).
* Added the ability to calculate `and` arguments only where they are needed ([Anastasia Tsarkova](https://github.com/yandex/ClickHouse/pull/2272))
* JIT compilation to native code is now available for some expressions ([pyos](https://github.com/yandex/ClickHouse/pull/2277)).
### Bug fixes:
@ -151,11 +163,11 @@
* Duplicates no longer appear for a query with `DISTINCT` and `ORDER BY`.
* Queries with `ARRAY JOIN` and `arrayFilter` no longer return an incorrect result.
* Fixed an error when reading an array column from a Nested structure ([#2066](https://github.com/yandex/ClickHouse/issues/2066)).
* Fixed an error when analyzing queries with a HAVING section like `HAVING tuple IN (...)`.
* Fixed an error when analyzing queries with a HAVING clause like `HAVING tuple IN (...)`.
* Fixed an error when analyzing queries with recursive aliases.
* Fixed an error when reading from ReplacingMergeTree with a condition in PREWHERE that filters all rows ([#2525](https://github.com/yandex/ClickHouse/issues/2525)).
* User profile settings were not applied when using sessions in the HTTP interface.
* Fixed how settings are applied from the command line parameters in `clickhouse-local`.
* Fixed how settings are applied from the command line parameters in clickhouse-local.
* The ZooKeeper client library now uses the session timeout received from the server.
* Fixed a bug in the ZooKeeper client library when the client waited for the server response longer than the timeout.
* Fixed pruning of parts for queries with conditions on partition key columns ([#2342](https://github.com/yandex/ClickHouse/issues/2342)).
@ -165,7 +177,7 @@
* Fixed syntactic parsing and formatting of the `CAST` operator.
* Fixed insertion into a materialized view for the Distributed table engine ([Babacar Diassé](https://github.com/yandex/ClickHouse/pull/2411)).
* Fixed a race condition when writing data from the `Kafka` engine to materialized views ([Yangkuan Liu](https://github.com/yandex/ClickHouse/pull/2448)).
* Fixed SSRF in the `remote()` table function.
* Fixed SSRF in the remote() table function.
* Fixed exit behavior of `clickhouse-client` in multiline mode ([#2510](https://github.com/yandex/ClickHouse/issues/2510)).
### Improvements:
@ -184,7 +196,7 @@
### Build changes:
* The gcc8 compiler can be used for builds.
* Added the ability to build llvm from a submodule.
* Added the ability to build llvm from submodule.
* The version of the librdkafka library has been updated to v0.11.4.
* Added the ability to use the system libcpuid library. The library version has been updated to 0.4.0.
* Fixed the build using the vectorclass library ([Babacar Diassé](https://github.com/yandex/ClickHouse/pull/2274)).
@ -195,44 +207,52 @@
### Backward incompatible changes:
* Removed escaping in `Vertical` and `Pretty*` formats and deleted the `VerticalRaw` format.
* If servers with version 1.1.54388 (or newer) and servers with older version are used simultaneously in distributed query and the query has `cast(x, 'Type')` expression in the form without `AS` keyword and with `cast` not in uppercase, then the exception with message like `Not found column cast(0, 'UInt8') in block` will be thrown. Solution: update server on all cluster nodes.
* If servers with version 1.1.54388 (or newer) and servers with an older version are used simultaneously in a distributed query and the query has the `cast(x, 'Type')` expression without the `AS` keyword and doesn't have the word `cast` in uppercase, an exception will be thrown with a message like `Not found column cast(0, 'UInt8') in block`. Solution: Update the server on the entire cluster.
## ClickHouse release 1.1.54385, 2018-06-01
### Bug fixes:
* Fixed an error that in some cases caused ZooKeeper operations to block.
## ClickHouse release 1.1.54383, 2018-05-22
### Bug fixes:
* Fixed a slowdown of replication queue if a table has many replicas.
## ClickHouse release 1.1.54381, 2018-05-14
### Bug fixes:
* Fixed a nodes leak in ZooKeeper when ClickHouse loses connection to ZooKeeper server.
## ClickHouse release 1.1.54380, 2018-04-21
### New features:
* Added table function `file(path, format, structure)`. An example reading bytes from `/dev/urandom`: `ln -s /dev/urandom /var/lib/clickhouse/user_files/random` `clickhouse-client -q "SELECT * FROM file('random', 'RowBinary', 'd UInt8') LIMIT 10"`.
* Added the table function `file(path, format, structure)`. An example reading bytes from `/dev/urandom`: `ln -s /dev/urandom /var/lib/clickhouse/user_files/random``clickhouse-client -q "SELECT * FROM file('random', 'RowBinary', 'd UInt8') LIMIT 10"`.
### Improvements:
* Subqueries could be wrapped by `()` braces (to enhance queries readability). For example, `(SELECT 1) UNION ALL (SELECT 1)`.
* Simple `SELECT` queries from table `system.processes` are not counted in `max_concurrent_queries` limit.
* Subqueries can be wrapped in `()` brackets to enhance query readability. For example: `(SELECT 1) UNION ALL (SELECT 1)`.
* Simple `SELECT` queries from the `system.processes` table are not included in the `max_concurrent_queries` limit.
### Bug fixes:
* Fixed incorrect behaviour of `IN` operator when select from `MATERIALIZED VIEW`.
* Fixed incorrect filtering by partition index in expressions like `WHERE partition_key_column IN (...)`
* Fixed inability to execute `OPTIMIZE` query on non-leader replica if the table was `REANAME`d.
* Fixed authorization error when execute `OPTIMIZE` or `ALTER` queries on a non-leader replica.
* Fixed freezing of `KILL QUERY` queries.
* Fixed an error in ZooKeeper client library which led to watches loses, freezing of distributed DDL queue and slowing replication queue if non-empty `chroot` prefix is used in ZooKeeper configuration.
* Fixed incorrect behavior of the `IN` operator when select from `MATERIALIZED VIEW`.
* Fixed incorrect filtering by partition index in expressions like `partition_key_column IN (...)`.
* Fixed inability to execute `OPTIMIZE` query on non-leader replica if `REANAME` was performed on the table.
* Fixed the authorization error when executing `OPTIMIZE` or `ALTER` queries on a non-leader replica.
* Fixed freezing of `KILL QUERY`.
* Fixed an error in ZooKeeper client library which led to loss of watches, freezing of distributed DDL queue, and slowdowns in the replication queue if a non-empty `chroot` prefix is used in the ZooKeeper configuration.
### Backward incompatible changes:
* Removed support of expressions like `(a, b) IN (SELECT (a, b))` (instead of them you can use their equivalent `(a, b) IN (SELECT a, b)`). In previous releases, these expressions led to undetermined data filtering or caused errors.
* Removed support for expressions like `(a, b) IN (SELECT (a, b))` (you can use the equivalent expression `(a, b) IN (SELECT a, b)`). In previous releases, these expressions led to undetermined `WHERE` filtering or caused errors.
## ClickHouse release 1.1.54378, 2018-04-16
### New features:
* Logging level can be changed without restarting the server.
@ -242,10 +262,10 @@
* Added support for `ALTER TABLE ... PARTITION ... ` for `MATERIALIZED VIEW`.
* Added information about the size of data parts in uncompressed form in the system table.
* Server-to-server encryption support for distributed tables (`<secure>1</secure>` in the replica config in `<remote_servers>`).
* Configuration of the table level for the `ReplicatedMergeTree` family in order to minimize the amount of data stored in zookeeper: `use_minimalistic_checksums_in_zookeeper = 1`
* Configuration of the `clickhouse-client` prompt. By default, server names are now output to the prompt. The server's display name can be changed; it's also sent in the `X-ClickHouse-Display-Name` HTTP header (Kirill Shvakov).
* Multiple comma-separated `topics` can be specified for the `Kafka` engine (Tobias Adamson).
* When a query is stopped by `KILL QUERY` or `replace_running_query`, the client receives the `Query was cancelled` exception instead of an incomplete response.
* Configuration of the table level for the `ReplicatedMergeTree` family in order to minimize the amount of data stored in Zookeeper: : `use_minimalistic_checksums_in_zookeeper = 1`
* Configuration of the `clickhouse-client` prompt. By default, server names are now output to the prompt. The server's display name can be changed. It's also sent in the `X-ClickHouse-Display-Name` HTTP header (Kirill Shvakov).
* Multiple comma-separated `topics` can be specified for the `Kafka` engine (Tobias Adamson)
* When a query is stopped by `KILL QUERY` or `replace_running_query`, the client receives the `Query was cancelled` exception instead of an incomplete result.
### Improvements:
@ -264,11 +284,11 @@
* Correct results are now returned when using tuples with `IN` when some of the tuple components are in the table index.
* The `max_execution_time` limit now works correctly with distributed queries.
* Fixed errors when calculating the size of composite columns in the `system.columns` table.
* Fixed an error when creating a temporary table `CREATE TEMPORARY TABLE IF NOT EXISTS`.
* Fixed errors in `StorageKafka` (#2075)
* Fixed an error when creating a temporary table `CREATE TEMPORARY TABLE IF NOT EXISTS.`
* Fixed errors in `StorageKafka` (##2075)
* Fixed server crashes from invalid arguments of certain aggregate functions.
* Fixed the error that prevented the `DETACH DATABASE` query from stopping background tasks for `ReplicatedMergeTree` tables.
* `Too many parts` state is less likely to happen when inserting into aggregated materialized views (#2084).
* `Too many parts` state is less likely to happen when inserting into aggregated materialized views (##2084).
* Corrected recursive handling of substitutions in the config if a substitution must be followed by another substitution on the same level.
* Corrected the syntax in the metadata file when creating a `VIEW` that uses a query with `UNION ALL`.
* `SummingMergeTree` now works correctly for summation of nested data structures with a composite key.
@ -276,15 +296,14 @@
### Build changes:
* The build supports `ninja` instead of `make` and uses it by default for building releases.
* Renamed packages: `clickhouse-server-base` is now `clickhouse-common-static`; `clickhouse-server-common` is now `clickhouse-server`; `clickhouse-common-dbg` is now `clickhouse-common-static-dbg`. To install, use `clickhouse-server clickhouse-client`. Packages with the old names will still load in the repositories for backward compatibility.
* The build supports `ninja` instead of `make` and uses `ninja` by default for building releases.
* Renamed packages: `clickhouse-server-base` in `clickhouse-common-static`; `clickhouse-server-common` in `clickhouse-server`; `clickhouse-common-dbg` in `clickhouse-common-static-dbg`. To install, use `clickhouse-server clickhouse-client`. Packages with the old names will still load in the repositories for backward compatibility.
### Backward-incompatible changes:
### Backward incompatible changes:
* Removed the special interpretation of an IN expression if an array is specified on the left side. Previously, the expression `arr IN (set)` was interpreted as "at least one `arr` element belongs to the `set`". To get the same behavior in the new version, write `arrayExists(x -> x IN (set), arr)`.
* Disabled the incorrect use of the socket option `SO_REUSEPORT`, which was incorrectly enabled by default in the Poco library. Note that on Linux there is no longer any reason to simultaneously specify the addresses `::` and `0.0.0.0` for listen – use just `::`, which allows listening to the connection both over IPv4 and IPv6 (with the default kernel config settings). You can also revert to the behavior from previous versions by specifying `<listen_reuse_port>1</listen_reuse_port>` in the config.
## ClickHouse release 1.1.54370, 2018-03-16
### New features:
@ -296,43 +315,44 @@
### Improvements:
* When inserting data in a `Replicated` table, fewer requests are made to `ZooKeeper` (and most of the user-level errors have disappeared from the `ZooKeeper` log).
* Added the ability to create aliases for sets. Example: `WITH (1, 2, 3) AS set SELECT number IN set FROM system.numbers LIMIT 10`.
* Added the ability to create aliases for data sets. Example: `WITH (1, 2, 3) AS set SELECT number IN set FROM system.numbers LIMIT 10`.
### Bug fixes:
* Fixed the `Illegal PREWHERE` error when reading from `Merge` tables over `Distributed` tables.
* Added fixes that allow you to run `clickhouse-server` in IPv4-only Docker containers.
* Fixed a race condition when reading from system `system.parts_columns` tables.
* Fixed the `Illegal PREWHERE` error when reading from Merge tables for `Distributed`tables.
* Added fixes that allow you to start clickhouse-server in IPv4-only Docker containers.
* Fixed a race condition when reading from system `system.parts_columns tables.`
* Removed double buffering during a synchronous insert to a `Distributed` table, which could have caused the connection to timeout.
* Fixed a bug that caused excessively long waits for an unavailable replica before beginning a `SELECT` query.
* Fixed incorrect dates in the `system.parts` table.
* Fixed a bug that made it impossible to insert data in a `Replicated` table if `chroot` was non-empty in the configuration of the `ZooKeeper` cluster.
* Fixed the vertical merging algorithm for an empty `ORDER BY` table.
* Restored the ability to use dictionaries in queries to remote tables, even if these dictionaries are not present on the requestor server. This functionality was lost in release 1.1.54362.
* Restored the behavior for queries like `SELECT * FROM remote('server2', default.table) WHERE col IN (SELECT col2 FROM default.table)` when the right side argument of the `IN` should use a remote `default.table` instead of a local one. This behavior was broken in version 1.1.54358.
* Restored the behavior for queries like `SELECT * FROM remote('server2', default.table) WHERE col IN (SELECT col2 FROM default.table)` when the right side of the `IN` should use a remote `default.table` instead of a local one. This behavior was broken in version 1.1.54358.
* Removed extraneous error-level logging of `Not found column ... in block`.
## ClickHouse release 1.1.54356, 2018-03-06
## Clickhouse Release 1.1.54362, 2018-03-11
### New features:
* Aggregation without `GROUP BY` for an empty set (such as `SELECT count(*) FROM table WHERE 0`) now returns a result with one row with null values for aggregate functions, in compliance with the SQL standard. To restore the old behavior (return an empty result), set `empty_result_for_aggregation_by_empty_set` to 1.
* Added type conversion for `UNION ALL`. Different alias names are allowed in `SELECT` positions in `UNION ALL`, in compliance with the SQL standard.
* Arbitrary expressions are supported in `LIMIT BY` sections. Previously, it was only possible to use columns resulting from `SELECT`.
* Arbitrary expressions are supported in `LIMIT BY` clauses. Previously, it was only possible to use columns resulting from `SELECT`.
* An index of `MergeTree` tables is used when `IN` is applied to a tuple of expressions from the columns of the primary key. Example: `WHERE (UserID, EventDate) IN ((123, '2000-01-01'), ...)` (Anastasiya Tsarkova).
* Added the `clickhouse-copier` tool for copying between clusters and resharding data (beta).
* Added consistent hashing functions: `yandexConsistentHash`, `jumpConsistentHash`, `sumburConsistentHash`. They can be used as a sharding key in order to reduce the amount of network traffic during subsequent reshardings.
* Added functions: `arrayAny`, `arrayAll`, `hasAny`, `hasAll`, `arrayIntersect`, `arrayResize`.
* Added the `arrayCumSum` function (Javi Santana).
* Added the `parseDateTimeBestEffort`, `parseDateTimeBestEffortOrZero`, and `parseDateTimeBestEffortOrNull` functions to read the DateTime from a string containing text in a wide variety of possible formats.
* It is now possible to change the logging settings without restarting the server.
* Data can be partially reloaded from external dictionaries during updating (load just the records in which the value of the specified field greater than in the previous download) (Arsen Hakobyan).
* Added the `cluster` table function. Example: `cluster(cluster_name, db, table)`. The `remote` table function can accept the cluster name as the first argument, if it is specified as an identifier.
* The `remote` and `cluster` table functions can be used in `INSERT` requests.
* Added the `create_table_query` and `engine_full` virtual columns to the `system.tables`table . The `metadata_modification_time` column is virtual.
* Added the `data_path` and `metadata_path` columns to `system.tables`and` system.databases` tables, and added the `path` column to the `system.parts` and `system.parts_columns` tables.
* Added additional information about merges in the `system.part_log` table.
* An arbitrary partitioning key can be used for the `system.query_log` table (Kirill Shvakov).
* The `SHOW TABLES` query now also shows temporary tables. Added temporary tables and the `is_temporary` column to `system.tables` (zhang2014).
* Added the `DROP TEMPORARY TABLE` query (zhang2014).
* Added `DROP TEMPORARY TABLE` and `EXISTS TEMPORARY TABLE` queries (zhang2014).
* Support for `SHOW CREATE TABLE` for temporary tables (zhang2014).
* Added the `system_profile` configuration parameter for the settings used by internal processes.
* Support for loading `object_id` as an attribute in `MongoDB` dictionaries (Pavel Litvinenko).
@ -347,7 +367,9 @@
* `MergeTree` tables can be used without a primary key (you need to specify `ORDER BY tuple()`).
* A `Nullable` type can be `CAST` to a non-`Nullable` type if the argument is not `NULL`.
* `RENAME TABLE` can be performed for `VIEW`.
* Added the `throwIf` function.
* Added the `odbc_default_field_size` option, which allows you to extend the maximum size of the value loaded from an ODBC source (by default, it is 1024).
* The `system.processes` table and `SHOW PROCESSLIST` now have the `is_cancelled` and `peak_memory_usage` columns.
### Improvements:
@ -371,6 +393,7 @@
* Fixed a bug in merges for `ReplacingMergeTree` tables.
* Fixed synchronous insertions in `Distributed` tables (`insert_distributed_sync = 1`).
* Fixed segfault for certain uses of `FULL` and `RIGHT JOIN` with duplicate columns in subqueries.
* Fixed segfault for certain uses of `replace_running_query` and `KILL QUERY`.
* Fixed the order of the `source` and `last_exception` columns in the `system.dictionaries` table.
* Fixed a bug when the `DROP DATABASE` query did not delete the file with metadata.
* Fixed the `DROP DATABASE` query for `Dictionary` databases.
@ -390,74 +413,77 @@
* Fixed a race condition in the query execution pipeline that occurred in very rare cases when using `Merge` tables with a large number of tables, and when using `GLOBAL` subqueries.
* Fixed a crash when passing arrays of different sizes to an `arrayReduce` function when using aggregate functions from multiple arguments.
* Prohibited the use of queries with `UNION ALL` in a `MATERIALIZED VIEW`.
* Fixed an error during initialization of the `part_log` system table when the server starts (by default, `part_log` is disabled).
### Backward incompatible changes:
* Removed the `distributed_ddl_allow_replicated_alter` option. This behavior is enabled by default.
* Removed the `strict_insert_defaults` setting. If you were using this functionality, write to `clickhouse-feedback@yandex-team.com`.
* Removed the `UnsortedMergeTree` engine.
## ClickHouse release 1.1.54343, 2018-02-05
## Clickhouse Release 1.1.54343, 2018-02-05
* Added macros support for defining cluster names in distributed DDL queries and constructors of Distributed tables: `CREATE TABLE distr ON CLUSTER '{cluster}' (...) ENGINE = Distributed('{cluster}', 'db', 'table')`.
* Now the table index is used for conditions like `expr IN (subquery)`.
* Now queries like `SELECT ... FROM table WHERE expr IN (subquery)` are processed using the `table` index.
* Improved processing of duplicates when inserting to Replicated tables, so they no longer slow down execution of the replication queue.
## ClickHouse release 1.1.54342, 2018-01-22
## Clickhouse Release 1.1.54342, 2018-01-22
This release contains bug fixes for the previous release 1.1.54337:
* Fixed a regression in 1.1.54337: if the default user has readonly access, then the server refuses to start up with the message `Cannot create database in readonly mode`.
* Fixed a regression in 1.1.54337: on systems with `systemd`, logs are always written to syslog regardless of the configuration; the watchdog script still uses `init.d`.
* Fixed a regression in 1.1.54337: on systems with systemd, logs are always written to syslog regardless of the configuration; the watchdog script still uses init.d.
* Fixed a regression in 1.1.54337: wrong default configuration in the Docker image.
* Fixed nondeterministic behaviour of GraphiteMergeTree (you can notice it in log messages `Data after merge is not byte-identical to data on another replicas`).
* Fixed a bug that may lead to inconsistent merges after OPTIMIZE query to Replicated tables (you may notice it in log messages `Part ... intersects previous part`).
* Fixed nondeterministic behavior of GraphiteMergeTree (you can see it in log messages `Data after merge is not byte-identical to the data on another replicas`).
* Fixed a bug that may lead to inconsistent merges after OPTIMIZE query to Replicated tables (you may see it in log messages `Part ... intersects the previous part`).
* Buffer tables now work correctly when MATERIALIZED columns are present in the destination table (by zhang2014).
* Fixed a bug in implementation of NULL.
## ClickHouse release 1.1.54337, 2018-01-18
## Clickhouse Release 1.1.54337, 2018-01-18
### New features:
* Added support for storage of multidimensional arrays and tuples (`Tuple` data type) in tables.
* Added support for table functions in `DESCRIBE` and `INSERT` queries. Added support for subqueries in `DESCRIBE`. Examples: `DESC TABLE remote('host', default.hits)`; `DESC TABLE (SELECT 1)`; `INSERT INTO TABLE FUNCTION remote('host', default.hits)`. Support for `INSERT INTO TABLE` syntax in addition to `INSERT INTO`.
* Improved support for timezones. The `DateTime` data type can be annotated with the timezone that is used for parsing and formatting in text formats. Example: `DateTime('Europe/Moscow')`. When timezones are specified in functions for DateTime arguments, the return type will track the timezone, and the value will be displayed as expected.
* Added the functions `toTimeZone`, `timeDiff`, `toQuarter`, `toRelativeQuarterNum`. The `toRelativeHour`/`Minute`/`Second` functions can take a value of type `Date` as an argument. The name of the `now` function has been made case-insensitive.
* Added support for storage of multi-dimensional arrays and tuples (`Tuple` data type) in tables.
* Support for table functions for `DESCRIBE` and `INSERT` queries. Added support for subqueries in `DESCRIBE`. Examples: `DESC TABLE remote('host', default.hits)`; `DESC TABLE (SELECT 1)`; `INSERT INTO TABLE FUNCTION remote('host', default.hits)`. Support for `INSERT INTO TABLE` in addition to `INSERT INTO`.
* Improved support for time zones. The `DateTime` data type can be annotated with the timezone that is used for parsing and formatting in text formats. Example: `DateTime('Europe/Moscow')`. When timezones are specified in functions for `DateTime` arguments, the return type will track the timezone, and the value will be displayed as expected.
* Added the functions `toTimeZone`, `timeDiff`, `toQuarter`, `toRelativeQuarterNum`. The `toRelativeHour`/`Minute`/`Second` functions can take a value of type `Date` as an argument. The `now` function name is case-sensitive.
* Added the `toStartOfFifteenMinutes` function (Kirill Shvakov).
* Added the `clickhouse format` tool for formatting queries.
* Added the `format_schema_path` configuration parameter (Marek Vavruša). It is used for specifying a schema in `Cap'n'Proto` format. Schema files can be located only in the specified directory.
* Added the `format_schema_path` configuration parameter (Marek Vavruşa). It is used for specifying a schema in `Cap'n Proto` format. Schema files can be located only in the specified directory.
* Added support for config substitutions (`incl` and `conf.d`) for configuration of external dictionaries and models (Pavel Yakunin).
* Added a column with documentation for the `system.settings` table (Kirill Shvakov).
* Added the `system.parts_columns` table with information about column sizes in each data part of `MergeTree` tables.
* Added the `system.models` table with information about loaded `CatBoost` machine learning models.
* Added the `mysql` and `odbc` table functions along with the corresponding `MySQL` and `ODBC` table engines for working with foreign databases. This feature is in the beta stage.
* Added the `mysql` and `odbc` table function and corresponding `MySQL` and `ODBC` table engines for accessing remote databases. This functionality is in the beta stage.
* Added the possibility to pass an argument of type `AggregateFunction` for the `groupArray` aggregate function (so you can create an array of states of some aggregate function).
* Removed restrictions on various combinations of aggregate function combinators. For example, you can use `avgForEachIf` as well as `avgIfForEach` aggregate functions, which have different behaviors.
* The `-ForEach` aggregate function combinator is extended for the case of aggregate functions of multiple arguments.
* Added support for aggregate functions of `Nullable` arguments even for cases when the function returns a non-`Nullable` result (added with the contribution of Silviu Caragea). Examples: `groupArray`, `groupUniqArray`, `topK`.
* Added the `max_client_network_bandwidth` command line parameter for `clickhouse-client` (Kirill Shvakov).
* Added support for aggregate functions of `Nullable` arguments even for cases when the function returns a non-`Nullable` result (added with the contribution of Silviu Caragea). Example: `groupArray`, `groupUniqArray`, `topK`.
* Added the `max_client_network_bandwidth` for `clickhouse-client` (Kirill Shvakov).
* Users with the ` readonly = 2` setting are allowed to work with TEMPORARY tables (CREATE, DROP, INSERT...) (Kirill Shvakov).
* Added support for using multiple consumers with the `Kafka` engine. Extended configuration options for `Kafka` (Marek Vavruša).
* Added the `intExp2` and `intExp10` functions.
* Added the `sumKahan` aggregate function (computationally stable summation of floating point numbers).
* Added to*Number*OrNull functions, where *Number* is a numeric type.
* Added support for the `WITH` clause for an `INSERT SELECT` query (by zhang2014).
* Added the settings `http_connection_timeout`, `http_send_timeout`, and `http_receive_timeout`. In particular, these settings are used for downloading data parts for replication. Changing these settings allows for faster failover if the network is overloaded.
* Added support for the `ALTER` query for tables of type `Null` (Anastasiya Tsarkova). Tables of type `Null` are often used with materialized views.
* Added the `intExp3` and `intExp4` functions.
* Added the `sumKahan` aggregate function.
* Added the to * Number* OrNull functions, where * Number* is a numeric type.
* Added support for `WITH` clauses for an `INSERT SELECT` query (author: zhang2014).
* Added settings: `http_connection_timeout`, `http_send_timeout`, `http_receive_timeout`. In particular, these settings are used for downloading data parts for replication. Changing these settings allows for faster failover if the network is overloaded.
* Added support for `ALTER` for tables of type `Null` (Anastasiya Tsarkova).
* The `reinterpretAsString` function is extended for all data types that are stored contiguously in memory.
* Added the `--silent` option for the `clickhouse-local` tool. It suppresses printing query execution info in stderr.
* Added support for reading values of type `Date` from text in a format where the month and/or day of the month is specified using a single digit instead of two digits (Amos Bird).
### Performance optimizations:
* Improved performance of `min`, `max`, `any`, `anyLast`, `anyHeavy`, `argMin`, `argMax` aggregate functions for String arguments.
* Improved performance of `isInfinite`, `isFinite`, `isNaN`, `roundToExp2` functions.
* Improved performance of parsing and formatting values of type `Date` and `DateTime` in text formats.
* Improved performance of aggregate functions `min`, `max`, `any`, `anyLast`, `anyHeavy`, `argMin`, `argMax` from string arguments.
* Improved performance of the functions `isInfinite`, `isFinite`, `isNaN`, `roundToExp2`.
* Improved performance of parsing and formatting `Date` and `DateTime` type values in text format.
* Improved performance and precision of parsing floating point numbers.
* Lowered memory usage for `JOIN` in the case when the left and right parts have columns with identical names that are not contained in `USING` .
* Improved performance of `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, and `corr` aggregate functions by reducing computational stability. The old functions are available under the names: `varSampStable`, `varPopStable`, `stddevSampStable`, `stddevPopStable`, `covarSampStable`, `covarPopStable`, `corrStable`.
* Improved performance of aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr` by reducing computational stability. The old functions are available under the names `varSampStable`, `varPopStable`, `stddevSampStable`, `stddevPopStable`, `covarSampStable`, `covarPopStable`, `corrStable`.
### Bug fixes:
* Fixed data deduplication after running a `DROP PARTITION` query. In the previous version, dropping a partition and INSERTing the same data again was not working because INSERTed blocks were considered duplicates.
* Fixed data deduplication after running a `DROP` or `DETACH PARTITION` query. In the previous version, dropping a partition and inserting the same data again was not working because inserted blocks were considered duplicates.
* Fixed a bug that could lead to incorrect interpretation of the `WHERE` clause for ` CREATE MATERIALIZED VIEW` queries with `POPULATE` .
* Fixed a bug in using the `root_path` parameter in the `zookeeper_servers` configuration.
* Fixed unexpected results of passing the `Date` argument to `toStartOfDay` .
@ -476,19 +502,19 @@ This release contains bug fixes for the previous release 1.1.54337:
* Fixed a bug in the background check of parts (`MergeTreePartChecker` ) when using a custom partition key.
* Fixed parsing of tuples (values of the `Tuple` data type) in text formats.
* Improved error messages about incompatible types passed to `multiIf` , `array` and some other functions.
* Support for `Nullable` types is completely reworked. Fixed bugs that may lead to a server crash. Fixed almost all other bugs related to NULL support: incorrect type conversions in INSERT SELECT, insufficient support for Nullable in HAVING and PREWHERE, `join_use_nulls` mode, Nullable types as arguments of OR operator, etc.
* Fixed various bugs related to internal semantics of data types. Examples: unnecessary summing of `Enum` type fields in `SummingMergeTree`; alignment of `Enum` types in Pretty formats, etc.
* Stricter checks for allowed combinations of composite columns. Fixed several bugs that could lead to a server crash.
* Redesigned support for `Nullable` types. Fixed bugs that may lead to a server crash. Fixed almost all other bugs related to ` NULL` support: incorrect type conversions in INSERT SELECT, insufficient support for Nullable in HAVING and PREWHERE, `join_use_nulls` mode, Nullable types as arguments of `OR` operator, etc.
* Fixed various bugs related to internal semantics of data types. Examples: unnecessary summing of `Enum` type fields in `SummingMergeTree` ; alignment of `Enum` types in `Pretty` formats, etc.
* Stricter checks for allowed combinations of composite columns.
* Fixed the overflow when specifying a very large parameter for the `FixedString` data type.
* Fixed a bug in the `topK` aggregate function in a generic case.
* Added the missing check for equality of array sizes in arguments of n-ary variants of aggregate functions with an `-Array` combinator.
* Fixed the `--pager` option for `clickhouse-client` (by ks1322).
* Fixed a bug in `--pager` for `clickhouse-client` (author: ks1322).
* Fixed the precision of the `exp10` function.
* Fixed the behavior of the `visitParamExtract` function for better compliance with documentation.
* Fixed the crash when incorrect data types are specified.
* Fixed the behavior of `DISTINCT` in the case when all columns are constants.
* Fixed query formatting in the case of using the `tupleElement` function with a complex constant expression as the tuple element index.
* Fixed the `Dictionary` table engine for dictionaries of type `range_hashed`.
* Fixed a bug in `Dictionary` tables for `range_hashed` dictionaries.
* Fixed a bug that leads to excessive rows in the result of `FULL` and ` RIGHT JOIN` (Amos Bird).
* Fixed a server crash when creating and removing temporary files in `config.d` directories during config reload.
* Fixed the ` SYSTEM DROP DNS CACHE` query: the cache was flushed but addresses of cluster nodes were not updated.
@ -496,41 +522,44 @@ This release contains bug fixes for the previous release 1.1.54337:
### Build improvements:
* Builds use `pbuilder`. The build process is almost completely independent of the build host environment.
* The `pbuilder` tool is used for builds. The build process is almost completely independent of the build host environment.
* A single build is used for different OS versions. Packages and binaries have been made compatible with a wide range of Linux systems.
* Added the `clickhouse-test` package. It can be used to run functional tests.
* The source tarball can now be published to the repository. It can be used to reproduce the build without using GitHub.
* Added limited integration with Travis CI. Due to limits on build time in Travis, only the debug build is tested and a limited subset of tests are run.
* Added support for `Cap'n'Proto` in the default build.
* Changed the format of documentation sources from `Restructured Text` to `Markdown`.
* Changed the format of documentation sources from `Restricted Text` to `Markdown`.
* Added support for `systemd` (Vladimir Smirnov). It is disabled by default due to incompatibility with some OS images and can be enabled manually.
* For dynamic code generation, `clang` and `lld` are embedded into the `clickhouse` binary. They can also be invoked as ` clickhouse clang` and ` clickhouse lld` .
* Removed usage of GNU extensions from the code. Enabled the `-Wextra` option. When building with `clang`, `libc++` is used instead of `libstdc++`.
* Removed usage of GNU extensions from the code. Enabled the `-Wextra` option. When building with `clang` the default is `libc++` instead of `libstdc++`.
* Extracted `clickhouse_parsers` and `clickhouse_common_io` libraries to speed up builds of various tools.
### Backward incompatible changes:
* The format for marks in `Log` type tables that contain `Nullable` columns was changed in a backward incompatible way. If you have these tables, you should convert them to the `TinyLog` type before starting up the new server version. To do this, replace `ENGINE = Log` with `ENGINE = TinyLog` in the corresponding `.sql` file in the `metadata` directory. If your table doesn't have `Nullable` columns or if the type of your table is not `Log`, then you don't need to do anything.
* Removed the `experimental_allow_extended_storage_definition_syntax` setting. Now this feature is enabled by default.
* To avoid confusion, the `runningIncome` function has been renamed to `runningDifferenceStartingWithFirstValue`.
* The `runningIncome` function was renamed to `runningDifferenceStartingWithFirstvalue` to avoid confusion.
* Removed the ` FROM ARRAY JOIN arr` syntax when ARRAY JOIN is specified directly after FROM with no table (Amos Bird).
* Removed the `BlockTabSeparated` format that was used solely for demonstration purposes.
* Changed the serialization format of intermediate states of the aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, and `corr`. If you have stored states of these aggregate functions in tables (using the AggregateFunction data type or materialized views with corresponding states), please write to clickhouse-feedback@yandex-team.com.
* Changed the state format for aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr`. If you have stored states of these aggregate functions in tables (using the `AggregateFunction` data type or materialized views with corresponding states), please write to clickhouse-feedback@yandex-team.com.
* In previous server versions there was an undocumented feature: if an aggregate function depends on parameters, you can still specify it without parameters in the AggregateFunction data type. Example: `AggregateFunction(quantiles, UInt64)` instead of `AggregateFunction(quantiles(0.5, 0.9), UInt64)`. This feature was lost. Although it was undocumented, we plan to support it again in future releases.
* Enum data types cannot be used in min/max aggregate functions. The possibility will be returned back in future release.
* Enum data types cannot be used in min/max aggregate functions. This ability will be returned in the next release.
### Please note when upgrading:
* When doing a rolling update on a cluster, at the point when some of the replicas are running the old version of ClickHouse and some are running the new version, replication is temporarily stopped and the message ` unknown parameter 'shard'` appears in the log. Replication will continue after all replicas of the cluster are updated.
* If you have different ClickHouse versions on the cluster, you can get incorrect results for distributed queries with the aggregate functions `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, and `corr`. You should update all cluster nodes.
* If different versions of ClickHouse are running on the cluster servers, it is possible that distributed queries using the following functions will have incorrect results: `varSamp`, `varPop`, `stddevSamp`, `stddevPop`, `covarSamp`, `covarPop`, `corr`. You should update all cluster nodes.
## ClickHouse release 1.1.54327, 2017-12-21
This release contains bug fixes for the previous release 1.1.54318:
* Fixed bug with possible race condition in replication that could lead to data loss. This issue affects versions 1.1.54310 and 1.1.54318. If you use one of these versions with Replicated tables, the update is strongly recommended. This issue shows in logs in Warning messages like ` Part ... from own log doesn't exist.` The issue is relevant even if you don't see these messages in logs.
## ClickHouse release 1.1.54318, 2017-11-30
This release contains bug fixes for the previous release 1.1.54310:
* Fixed incorrect row deletions during merges in the SummingMergeTree engine
* Fixed a memory leak in unreplicated MergeTree engines
* Fixed performance degradation with frequent inserts in MergeTree engines
@ -540,6 +569,7 @@ This release contains bug fixes for the previous release 1.1.54310:
## ClickHouse release 1.1.54310, 2017-11-01
### New features:
* Custom partitioning key for the MergeTree family of table engines.
* [ Kafka](https://clickhouse.yandex/docs/en/single/index.html#document-table_engines/kafka) table engine.
* Added support for loading [CatBoost](https://catboost.yandex/) models and applying them to data stored in ClickHouse.
@ -550,17 +580,19 @@ This release contains bug fixes for the previous release 1.1.54310:
* Added the `ATTACH TABLE` query without arguments.
* The processing logic for Nested columns with names ending in -Map in a SummingMergeTree table was extracted to the sumMap aggregate function. You can now specify such columns explicitly.
* Max size of the IP trie dictionary is increased to 128M entries.
* Added the `getSizeOfEnumType` function.
* Added the `sumWithOverflow` aggregate function.
* Added the getSizeOfEnumType function.
* Added the sumWithOverflow aggregate function.
* Added support for the Cap'n Proto input format.
* You can now customize compression level when using the zstd algorithm.
### Backward incompatible changes:
* Creation of temporary tables with an engine other than Memory is forbidden.
* Explicit creation of tables with the View or MaterializedView engine is forbidden.
* Creation of temporary tables with an engine other than Memory is not allowed.
* Explicit creation of tables with the View or MaterializedView engine is not allowed.
* During table creation, a new check verifies that the sampling key expression is included in the primary key.
### Bug fixes:
* Fixed hangups when synchronously inserting into a Distributed table.
* Fixed nonatomic adding and removing of parts in Replicated tables.
* Data inserted into a materialized view is not subjected to unnecessary deduplication.
@ -568,39 +600,44 @@ This release contains bug fixes for the previous release 1.1.54310:
* Users don't need access permissions to the `default` database to create temporary tables anymore.
* Fixed crashing when specifying the Array type without arguments.
* Fixed hangups when the disk volume containing server logs is full.
* Fixed an overflow in the `toRelativeWeekNum` function for the first week of the Unix epoch.
* Fixed an overflow in the toRelativeWeekNum function for the first week of the Unix epoch.
### Build improvements:
* Several third-party libraries (notably Poco) were updated and converted to git submodules.
## ClickHouse release 1.1.54304, 2017-10-19
### New features:
* TLS support in the native protocol (to enable, set `tcp_ssl_port` in `config.xml`)
* TLS support in the native protocol (to enable, set `tcp_ssl_port` in `config.xml` ).
### Bug fixes:
* `ALTER` for replicated tables now tries to start running as soon as possible
* Fixed crashing when reading data with the setting `preferred_block_size_bytes=0`
* Fixed crashes of `clickhouse-client` when `Page Down` is pressed
* `ALTER` for replicated tables now tries to start running as soon as possible.
* Fixed crashing when reading data with the setting `preferred_block_size_bytes=0.`
* Fixed crashes of `clickhouse-client` when pressing ` Page Down`
* Correct interpretation of certain complex queries with `GLOBAL IN` and `UNION ALL`
* `FREEZE PARTITION` always works atomically now
* Empty POST requests now return a response with code 411
* Fixed interpretation errors for expressions like `CAST(1 AS Nullable(UInt8))`
* Fixed an error when reading columns like `Array(Nullable(String))` from `MergeTree` tables
* `FREEZE PARTITION` always works atomically now.
* Empty POST requests now return a response with code 411.
* Fixed interpretation errors for expressions like `CAST(1 AS Nullable(UInt8)).`
* Fixed an error when reading `Array(Nullable(String))` columns from `MergeTree` tables.
* Fixed crashing when parsing queries like `SELECT dummy AS dummy, dummy AS b`
* Users are updated correctly when `users.xml` is invalid
* Correct handling when an executable dictionary returns a non-zero response code
* Users are updated correctly with invalid `users.xml`
* Correct handling when an executable dictionary returns a non-zero response code.
## ClickHouse release 1.1.54292, 2017-09-20
### New features:
* Added the `pointInPolygon` function for working with coordinates on a coordinate plane.
* Added the `sumMap` aggregate function for calculating the sum of arrays, similar to `SummingMergeTree`.
* Added the `trunc` function. Improved performance of the rounding functions (`round`, `floor`, `ceil`, `roundToExp2`) and corrected the logic of how they work. Changed the logic of the `roundToExp2` function for fractions and negative numbers.
* The ClickHouse executable file is now less dependent on the libc version. The same ClickHouse executable file can run on a wide variety of Linux systems. Note: There is still a dependency when using compiled queries (with the setting `compile = 1`, which is not used by default).
* The ClickHouse executable file is now less dependent on the libc version. The same ClickHouse executable file can run on a wide variety of Linux systems. There is still a dependency when using compiled queries (with the setting ` compile = 1` , which is not used by default).
* Reduced the time needed for dynamic compilation of queries.
### Bug fixes:
* Fixed an error that sometimes produced ` part ... intersects previous part` messages and weakened replica consistency.
* Fixed an error that caused the server to lock up if ZooKeeper was unavailable during shutdown.
* Removed excessive logging when restoring replicas.
@ -611,12 +648,13 @@ This release contains bug fixes for the previous release 1.1.54310:
## ClickHouse release 1.1.54289, 2017-09-13
### New features:
* `SYSTEM` queries for server administration: `SYSTEM RELOAD DICTIONARY`, `SYSTEM RELOAD DICTIONARIES`, `SYSTEM DROP DNS CACHE`, `SYSTEM SHUTDOWN`, `SYSTEM KILL`.
* Added functions for working with arrays: `concat`, `arraySlice`, `arrayPushBack`, `arrayPushFront`, `arrayPopBack`, `arrayPopFront`.
* Added the `root` and `identity` parameters for the ZooKeeper configuration. This allows you to isolate individual users on the same ZooKeeper cluster.
* Added the aggregate functions `groupBitAnd`, `groupBitOr`, and `groupBitXor` (for compatibility, they can also be accessed with the names `BIT_AND`, `BIT_OR`, and `BIT_XOR`).
* Added `root` and `identity` parameters for the ZooKeeper configuration. This allows you to isolate individual users on the same ZooKeeper cluster.
* Added aggregate functions `groupBitAnd`, `groupBitOr`, and `groupBitXor` (for compatibility, they are also available under the names `BIT_AND`, `BIT_OR`, and `BIT_XOR`).
* External dictionaries can be loaded from MySQL by specifying a socket in the filesystem.
* External dictionaries can be loaded from MySQL over SSL (the `ssl_cert`, `ssl_key`, and `ssl_ca` parameters).
* External dictionaries can be loaded from MySQL over SSL (`ssl_cert`, `ssl_key`, `ssl_ca` parameters).
* Added the `max_network_bandwidth_for_user` setting to restrict the overall bandwidth use for queries per user.
* Support for `DROP TABLE` for temporary tables.
* Support for reading `DateTime` values in Unix timestamp format from the `CSV` and `JSONEachRow` formats.
@ -626,6 +664,7 @@ This release contains bug fixes for the previous release 1.1.54310:
* Improved performance for queries with `DISTINCT` .
### Bug fixes:
* Improved the process for deleting old nodes in ZooKeeper. Previously, old nodes sometimes didn't get deleted if there were very frequent inserts, which caused the server to be slow to shut down, among other things.
* Fixed randomization when choosing hosts for the connection to ZooKeeper.
* Fixed the exclusion of lagging replicas in distributed queries if the replica is localhost.
@ -638,30 +677,33 @@ This release contains bug fixes for the previous release 1.1.54310:
* Resolved the appearance of zombie processes when using a dictionary with an `executable` source.
* Fixed segfault for the HEAD query.
### Improvements to development workflow and ClickHouse build:
### Improved workflow for developing and assembling ClickHouse:
* You can use `pbuilder` to build ClickHouse.
* You can use `libc++` instead of `libstdc++` for builds on Linux.
* Added instructions for using static code analysis tools: `Coverity`, `clang-tidy`, and `cppcheck`.
* Added instructions for using static code analysis tools: `Coverage`, `clang-tidy`, `cppcheck`.
### Please note when upgrading:
* There is now a higher default value for the MergeTree setting `max_bytes_to_merge_at_max_space_in_pool` (the maximum total size of data parts to merge, in bytes): it has increased from 100 GiB to 150 GiB. This might result in large merges running after the server upgrade, which could cause an increased load on the disk subsystem. If the free space available on the server is less than twice the total amount of the merges that are running, this will cause all other merges to stop running, including merges of small data parts. As a result, INSERT requests will fail with the message "Merges are processing significantly slower than inserts." Use the `SELECT * FROM system.merges` request to monitor the situation. You can also check the `DiskSpaceReservedForMerge` metric in the `system.metrics` table, or in Graphite. You don't need to do anything to fix this, since the issue will resolve itself once the large merges finish. If you find this unacceptable, you can restore the previous value for the `max_bytes_to_merge_at_max_space_in_pool` setting (to do this, go to the `<merge_tree>` section in config.xml, set `<max_bytes_to_merge_at_max_space_in_pool>107374182400</max_bytes_to_merge_at_max_space_in_pool>` and restart the server).
* There is now a higher default value for the MergeTree setting `max_bytes_to_merge_at_max_space_in_pool` (the maximum total size of data parts to merge, in bytes): it has increased from 100 GiB to 150 GiB. This might result in large merges running after the server upgrade, which could cause an increased load on the disk subsystem. If the free space available on the server is less than twice the total amount of the merges that are running, this will cause all other merges to stop running, including merges of small data parts. As a result, INSERT requests will fail with the message "Merges are processing significantly slower than inserts." Use the ` SELECT * FROM system.merges` request to monitor the situation. You can also check the `DiskSpaceReservedForMerge` metric in the `system.metrics` table, or in Graphite. You don't need to do anything to fix this, since the issue will resolve itself once the large merges finish. If you find this unacceptable, you can restore the previous value for the `max_bytes_to_merge_at_max_space_in_pool` setting. To do this, go to the <merge_tree> section in config.xml, set `<merge_tree>``<max_bytes_to_merge_at_max_space_in_pool>107374182400</max_bytes_to_merge_at_max_space_in_pool>` and restart the server.
## ClickHouse release 1.1.54284, 2017-08-29
* This is bugfix release for previous 1.1.54282 release. It fixes ZooKeeper nodes leak in `parts/` directory.
* This is a bugfix release for the previous 1.1.54282 release. It fixes leaks in the parts directory in ZooKeeper.
## ClickHouse release 1.1.54282, 2017-08-23
This is a bugfix release. The following bugs were fixed:
* `DB::Exception: Assertion violation: !_path.empty()` error when inserting into a Distributed table.
* Error when parsing inserted data in RowBinary format if the data begins with ';' character.
This release contains bug fixes for the previous release 1.1.54276:
* Fixed `DB::Exception: Assertion violation: !_path.empty()` when inserting into a Distributed table.
* Fixed parsing when inserting in RowBinary format if input data starts with';'.
* Errors during runtime compilation of certain aggregate functions (e.g. `groupArray()`).
## ClickHouse release 1.1.54276, 2017-08-16
## Clickhouse Release 1.1.54276, 2017-08-16
### New features:
* You can use an optional WITH clause in a SELECT query. Example query: `WITH 1+1 AS a SELECT a, a*a`
* Added an optional WITH section for a SELECT query. Example query: `WITH 1+1 AS a SELECT a, a*a`
* INSERT can be performed synchronously in a Distributed table: OK is returned only after all the data is saved on all the shards. This is activated by the setting insert_distributed_sync=1.
* Added the UUID data type for working with 16-byte identifiers.
* Added aliases of CHAR, FLOAT and other types for compatibility with the Tableau.
@ -670,13 +712,13 @@ This is a bugfix release. The following bugs were fixed:
* Added support for non-constant arguments and negative offsets in the function `substring(str, pos, len).`
* Added the max_size parameter for the `groupArray(max_size)(column)` aggregate function, and optimized its performance.
### Major changes:
### Main changes:
* Improved security: all server files are created with 0640 permissions (can be changed via <umask> config parameter).
* Security improvements: all server files are created with 0640 permissions (can be changed via <umask> config parameter).
* Improved error messages for queries with invalid syntax.
* Significantly reduced memory consumption and improved performance when merging large sections of MergeTree data.
* Significantly increased the performance of data merges for the ReplacingMergeTree engine.
* Improved performance for asynchronous inserts from a Distributed table by batching multiple source inserts. To enable this functionality, use the setting distributed_directory_monitor_batch_inserts=1.
* Improved performance for asynchronous inserts from a Distributed table by combining multiple source inserts. To enable this functionality, use the setting distributed_directory_monitor_batch_inserts=1.
### Backward incompatible changes:
@ -685,12 +727,12 @@ This is a bugfix release. The following bugs were fixed:
### Complete list of changes:
* Added the `output_format_json_quote_denormals` setting, which enables outputting nan and inf values in JSON format.
* Optimized thread allocation when reading from a Distributed table.
* Settings can be modified in readonly mode if the value doesn't change.
* Added the ability to read fractional granules of the MergeTree engine in order to meet restrictions on the block size specified in the preferred_block_size_bytes setting. The purpose is to reduce the consumption of RAM and increase cache locality when processing queries from tables with large columns.
* Optimized stream allocation when reading from a Distributed table.
* Settings can be configured in readonly mode if the value doesn't change.
* Added the ability to retrieve non-integer granules of the MergeTree engine in order to meet restrictions on the block size specified in the preferred_block_size_bytes setting. The purpose is to reduce the consumption of RAM and increase cache locality when processing queries from tables with large columns.
* Efficient use of indexes that contain expressions like `toStartOfHour(x)` for conditions like `toStartOfHour(x) op сonstexpr.`
* Added new settings for MergeTree engines (the merge_tree section in config.xml):
- replicated_deduplication_window_seconds sets the size of deduplication window in seconds for Replicated tables.
- replicated_deduplication_window_seconds sets the number of seconds allowed for deduplicating inserts in Replicated tables.
- cleanup_delay_period sets how often to start cleanup to remove outdated data.
- replicated_can_become_leader can prevent a replica from becoming the leader (and assigning merges).
* Accelerated cleanup to remove outdated data from ZooKeeper.
@ -699,11 +741,11 @@ This is a bugfix release. The following bugs were fixed:
* Added the "none" value for the compression method.
* You can use multiple dictionaries_config sections in config.xml.
* It is possible to connect to MySQL through a socket in the file system.
* The `system.parts` table has a new column with information about the size of marks, in bytes.
* The system.parts table has a new column with information about the size of marks, in bytes.
### Bug fixes:
* Distributed tables using a Merge table now work correctly for a SELECT query with a condition on the _table field.
* Distributed tables using a Merge table now work correctly for a SELECT query with a condition on the `_table` field.
* Fixed a rare race condition in ReplicatedMergeTree when checking data parts.
* Fixed possible freezing on "leader election" when starting a server.
* The max_replica_delay_for_distributed_queries setting was ignored when using a local replica of the data source. This has been fixed.
@ -717,11 +759,11 @@ This is a bugfix release. The following bugs were fixed:
* Too many threads were used for parallel aggregation. This has been fixed.
* Fixed how the "if" function works with FixedString arguments.
* SELECT worked incorrectly from a Distributed table for shards with a weight of 0. This has been fixed.
* Crashes no longer occur when running `CREATE VIEW IF EXISTS.`
* Running `CREATE VIEW IF EXISTS no longer causes crashes.`
* Fixed incorrect behavior when input_format_skip_unknown_fields=1 is set and there are negative numbers.
* Fixed an infinite loop in the `dictGetHierarchy()` function if there is some invalid data in the dictionary.
* Fixed `Syntax error: unexpected (...)` errors when running distributed queries with subqueries in an IN or JOIN clause and Merge tables.
* Fixed the incorrect interpretation of a SELECT query from Dictionary tables.
* Fixed an incorrect interpretation of a SELECT query from Dictionary tables.
* Fixed the "Cannot mremap" error when using arrays in IN and JOIN clauses with more than 2 billion elements.
* Fixed the failover for dictionaries with MySQL as the source.
@ -735,7 +777,7 @@ This is a bugfix release. The following bugs were fixed:
### New features:
* Distributed DDL (for example, `CREATE TABLE ON CLUSTER`).
* Distributed DDL (for example, `CREATE TABLE ON CLUSTER`)
* The replicated request `ALTER TABLE CLEAR COLUMN IN PARTITION.`
* The engine for Dictionary tables (access to dictionary data in the form of a table).
* Dictionary database engine (this type of database automatically has Dictionary tables available for all the connected external dictionaries).
@ -751,8 +793,8 @@ This is a bugfix release. The following bugs were fixed:
### Minor changes:
* If an alert is triggered, the full stack trace is printed into the log.
* Relaxed the verification of the number of damaged or extra data parts at startup (there were too many false positives).
* Now after an alert is triggered, the log prints the full stack trace.
* Relaxed the verification of the number of damaged/extra data parts at startup (there were too many false positives).
### Bug fixes:
@ -762,7 +804,7 @@ This is a bugfix release. The following bugs were fixed:
* Changes in how an executable source of cached external dictionaries works.
* Fixed the comparison of strings containing null characters.
* Fixed the comparison of Float32 primary key fields with constants.
* Previously, an incorrect estimate of the size of a field could lead to overly large allocations. This has been fixed.
* Previously, an incorrect estimate of the size of a field could lead to overly large allocations.
* Fixed a crash when querying a Nullable column added to a table using ALTER.
* Fixed a crash when sorting by a Nullable column, if the number of rows is less than LIMIT.
* Fixed an ORDER BY subquery consisting of only constant values.

84
docs/en/data_types/array.md
View File

@ -1,5 +1,85 @@
<a name="data_type-array"></a>
# Array(T)
An array of elements of type T. The T type can be any type, including an array.
We don't recommend using multidimensional arrays, because they are not well supported (for example, you can't store multidimensional arrays in tables with a MergeTree engine).
Array of `T`-type items.
`T` can be anything, including an array. Use multi-dimensional arrays with caution. ClickHouse has limited support for multi-dimensional arrays. For example, they can't be stored in `MergeTree` tables.
## Creating an array
You can use a function to create an array:
```
array(T)
```
You can also use square brackets.
```
[]
```
Example of creating an array:
```
:) SELECT array(1, 2) AS x, toTypeName(x)
SELECT
[1, 2] AS x,
toTypeName(x)
┌─x─────┬─toTypeName(array(1, 2))─┐
│ [1,2] │ Array(UInt8) │
└───────┴─────────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
:) SELECT [1, 2] AS x, toTypeName(x)
SELECT
[1, 2] AS x,
toTypeName(x)
┌─x─────┬─toTypeName([1, 2])─┐
│ [1,2] │ Array(UInt8) │
└───────┴────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
```
## Working with data types
When creating an array on the fly, ClickHouse automatically defines the argument type as the narrowest data type that can store all the listed arguments. If there are any [NULL](../query_language/syntax.md#null-literal) or [Nullable](nullable.md#data_type-nullable) type arguments, the type of array elements is [Nullable](nullable.md#data_type-nullable).
If ClickHouse couldn't determine the data type, it will generate an exception. For instance, this will happen when trying to create an array with strings and numbers simultaneously (`SELECT array(1, 'a')`).
Examples of automatic data type detection:
```
:) SELECT array(1, 2, NULL) AS x, toTypeName(x)
SELECT
[1, 2, NULL] AS x,
toTypeName(x)
┌─x──────────┬─toTypeName(array(1, 2, NULL))─┐
│ [1,2,NULL] │ Array(Nullable(UInt8)) │
└────────────┴───────────────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
```
If you try to create an array of incompatible data types, ClickHouse throws an exception:
```
:) SELECT array(1, 'a')
SELECT [1, 'a']
Received exception from server (version 1.1.54388):
Code: 386. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: There is no supertype for types UInt8, String because some of them are String/FixedString and some of them are not.
0 rows in set. Elapsed: 0.246 sec.
```

2
docs/en/data_types/datetime.md
View File

@ -1,3 +1,5 @@
<a name="data_type-datetime"></a>
# DateTime
Date with time. Stored in four bytes as a Unix timestamp (unsigned). Allows storing values in the same range as for the Date type. The minimal value is output as 0000-00-00 00:00:00.

97
docs/en/data_types/enum.md
View File

@ -1,18 +1,101 @@
# Enum
<a name="data_type-enum"></a>
Enum8 or Enum16. A finite set of string values that can be stored more efficiently than the `String` data type.
# Enum8, Enum16
Example:
Includes the `Enum8` and `Enum16` types. `Enum` saves the final set of pairs of `'string' = integer`. In ClickHouse , all operations with the `Enum` data type are performed as if with numbers, although the user is working with string constants. This is more effective in terms of performance than working with the `String` data type.
```text
Enum8('hello' = 1, 'world' = 2)
- `Enum8` is described by pairs of `'String' = Int8`.
- `Enum16` is described by pairs of `'String' = Int16`.
## Usage examples
Here we create a table with an `Enum8('hello' = 1, 'world' = 2)` type column.
```
CREATE TABLE t_enum
(
x Enum8('hello' = 1, 'world' = 2)
)
ENGINE = TinyLog
```
- A data type with two possible values: 'hello' and 'world'.
This column `x` can only store the values that are listed in the type definition: `'hello'` or `'world'`. If you try to save a different value, ClickHouse generates an exception.
```
:) INSERT INTO t_enum Values('hello'),('world'),('hello')
INSERT INTO t_enum VALUES
Ok.
3 rows in set. Elapsed: 0.002 sec.
:) insert into t_enum values('a')
INSERT INTO t_enum VALUES
Exception on client:
Code: 49. DB::Exception: Unknown element 'a' for type Enum8('hello' = 1, 'world' = 2)
```
When you query data from the table, ClickHouse outputs the string values from `Enum`.
```
SELECT * FROM t_enum
┌─x─────┐
│ hello │
│ world │
│ hello │
└───────┘
```
If you need to see the numeric equivalents of the rows, you must cast the type.
```
SELECT CAST(x, 'Int8') FROM t_enum
┌─CAST(x, 'Int8')─┐
│ 1 │
│ 2 │
│ 1 │
└─────────────────┘
```
To create an Enum value in a query, you also need the `CAST` function.
```
SELECT toTypeName(CAST('a', 'Enum8(\'a\' = 1, \'b\' = 2)'))
┌─toTypeName(CAST('a', 'Enum8(\'a\' = 1, \'b\' = 2)'))─┐
│ Enum8('a' = 1, 'b' = 2) │
└──────────────────────────────────────────────────────┘
```
## General rules and usage
Each of the values is assigned a number in the range `-128 ... 127` for `Enum8` or in the range `-32768 ... 32767` for `Enum16`. All the strings and numbers must be different. An empty string is allowed. If this type is specified (in a table definition), numbers can be in an arbitrary order. However, the order does not matter.
In RAM, this type of column is stored in the same way as `Int8` or `Int16` of the corresponding numerical values.
Neither the string nor the numeric value in an `Enum` can be [NULL](../query_language/syntax.md#null-literal).
`An Enum` can be passed to a [Nullable](nullable.md#data_type-nullable) type. So if you create a table using the query
```
CREATE TABLE t_enum_nullable
(
x Nullable( Enum8('hello' = 1, 'world' = 2) )
)
ENGINE = TinyLog
```
it can store not only `'hello'` and `'world'`, but `NULL`, as well.
```
INSERT INTO t_enum_null Values('hello'),('world'),(NULL)
```
In RAM, an `Enum` column is stored in the same way as `Int8` or `Int16` of the corresponding numerical values.
When reading in text form, ClickHouse parses the value as a string and searches for the corresponding string from the set of Enum values. If it is not found, an exception is thrown. When reading in text format, the string is read and the corresponding numeric value is looked up. An exception will be thrown if it is not found.
When writing in text form, it writes the value as the corresponding string. If column data contains garbage (numbers that are not from the valid set), an exception is thrown. When reading and writing in binary form, it works the same way as for Int8 and Int16 data types.
The implicit default value is the value with the lowest number.

1
docs/en/data_types/float.md
View File

@ -16,6 +16,7 @@ We recommend that you store data in integer form whenever possible. For example,
```sql
SELECT 1 - 0.9
```
```
┌───────minus(1, 0.9)─┐
│ 0.09999999999999998 │

2
docs/en/data_types/int_uint.md
View File

@ -1,3 +1,5 @@
<a name="data_type-int"></a>
# UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
Fixed-length integers, with or without a sign.

63
docs/en/data_types/nullable.md Normal file
View File

@ -0,0 +1,63 @@
<a name="data_type-nullable"></a>
# Nullable(TypeName)
Allows you to work with the `TypeName` value or without it ([NULL](../query_language/syntax.md#null-literal)) in the same variable, including storage of `NULL` tables with the `TypeName` values. For example, a `Nullable(Int8)` type column can store `Int8` type values, and the rows that don't have a value will store `NULL`.
For a `TypeName`, you can't use composite data types [Array](array.md#data_type is array) and [Tuple](tuple.md#data_type-tuple). Composite data types can contain `Nullable` type values, such as `Array(Nullable(Int8))`.
A `Nullable` type field can't be included in indexes.
`NULL` is the default value for the `Nullable` type, unless specified otherwise in the ClickHouse server configuration.
##Storage features
For storing `Nullable` type values, ClickHouse uses:
- A separate file with `NULL` masks (referred to as the mask).
- The file with the values.
The mask determines what is in a data cell: `NULL` or a value.
When the mask indicates that `NULL` is stored in a cell, the file with values stores the default value for the data type. So if the field has the type `Nullable(Int8)`, the cell will store the default value for `Int8`. This feature increases storage capacity.
!!! Note:
Using `Nullable` almost always reduces performance, so keep this in mind when designing your databases.
## Usage example
```
:) CREATE TABLE t_null(x Int8, y Nullable(Int8)) ENGINE TinyLog
CREATE TABLE t_null
(
x Int8,
y Nullable(Int8)
)
ENGINE = TinyLog
Ok.
0 rows in set. Elapsed: 0.012 sec.
:) INSERT INTO t_null VALUES (1, NULL)
INSERT INTO t_null VALUES
Ok.
1 rows in set. Elapsed: 0.007 sec.
:) SELECT x + y from t_null
SELECT x + y
FROM t_null
┌─plus(x, y)─┐
│ ᴺᵁᴸᴸ │
│ 5 │
└────────────┘
2 rows in set. Elapsed: 0.144 sec.
```

20
docs/en/data_types/special_data_types/nothing.md Normal file
View File

@ -0,0 +1,20 @@
<a name="special_data_type-nothing"></a>
# Nothing
The only purpose of this data type is to represent [NULL](../../query_language/syntax.md#null-literal), i.e., no value.
You can't create a `Nothing` type value, because it is used where a value is not expected. For example, `NULL` is written as `Nullable(Nothing)` ([Nullable](../../data_types/nullable.md#data_type-nullable) — this is the data type that allows storing `NULL` in tables.) The `Nothing` type is also used to denote empty arrays:
```bash
:) SELECT toTypeName(Array())
SELECT toTypeName([])
┌─toTypeName(array())─┐
│ Array(Nothing) │
└─────────────────────┘
1 rows in set. Elapsed: 0.062 sec.
```

2
docs/en/data_types/string.md
View File

@ -1,3 +1,5 @@
<a name="data_types-string"></a>
# String
Strings of an arbitrary length. The length is not limited. The value can contain an arbitrary set of bytes, including null bytes.

52
docs/en/data_types/tuple.md
View File

@ -1,6 +1,54 @@
<a name="data_type-tuple"></a>
# Tuple(T1, T2, ...)
Tuples can't be written to tables (other than Memory tables). They are used for temporary column grouping. Columns can be grouped when an IN expression is used in a query, and for specifying certain formal parameters of lambda functions. For more information, see "IN operators" and "Higher order functions".
A tuple of elements of any [type](index.md#data_types). There can be one or more types of elements in a tuple.
Tuples can be output as the result of running a query. In this case, for text formats other than JSON\*, values are comma-separated in brackets. In JSON\* formats, tuples are output as arrays (in square brackets).
You can't store tuples in tables (other than Memory tables). They are used for temporary column grouping. Columns can be grouped when an IN expression is used in a query, and for specifying certain formal parameters of lambda functions. For more information, see the sections [IN operators](../query_language/select.md#in_operators) and [Higher order functions](../query_language/functions/higher_order_functions.md#higher_order_functions).
Tuples can be the result of a query. In this case, for text formats other than JSON, values are comma-separated in brackets. In JSON formats, tuples are output as arrays (in square brackets).
## Creating a tuple
You can use a function to create a tuple
```
tuple(T1, T2, ...)
```
Example of creating a tuple:
```
:) SELECT tuple(1,'a') AS x, toTypeName(x)
SELECT
(1, 'a') AS x,
toTypeName(x)
┌─x───────┬─toTypeName(tuple(1, 'a'))─┐
│ (1,'a') │ Tuple(UInt8, String) │
└─────────┴───────────────────────────┘
1 rows in set. Elapsed: 0.021 sec.
```
## Working with data types
When creating a tuple on the fly, ClickHouse automatically detects the type of each argument as the minimum of the types which can store the argument value. If the argument is [NULL](../query_language/syntax.md#null-literal), the type of the tuple element is [Nullable](nullable.md#data_type-nullable).
Example of automatic data type detection:
```
SELECT tuple(1,NULL) AS x, toTypeName(x)
SELECT
(1, NULL) AS x,
toTypeName(x)
┌─x────────┬─toTypeName(tuple(1, NULL))──────┐
│ (1,NULL) │ Tuple(UInt8, Nullable(Nothing)) │
└──────────┴─────────────────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
```

242
docs/en/development/style.md
View File

@ -16,7 +16,7 @@
**2.** Indents are 4 spaces. Configure your development environment so that a tab adds four spaces.
**3.** A left curly bracket must be separated on a new line. (And the right one, as well.)
**3.** Opening and closing curly brackets must be on a separate line.
```cpp
inline void readBoolText(bool & x, ReadBuffer & buf)
@ -27,28 +27,30 @@ inline void readBoolText(bool & x, ReadBuffer & buf)
}
```
**4.**
But if the entire function body is quite short (a single statement), you can place it entirely on one line if you wish. Place spaces around curly braces (besides the space at the end of the line).
**4.** If the entire function body is a single `statement`, it can be placed on a single line. Place spaces around curly braces (besides the space at the end of the line).
```cpp
inline size_t mask() const { return buf_size() - 1; }
inline size_t place(HashValue x) const { return x & mask(); }
```
**5.** For functions, don't put spaces around brackets.
**5.** For functions. Don't put spaces around brackets.
```cpp
void reinsert(const Value & x)
```
```cpp
memcpy(&buf[place_value], &x, sizeof(x));
```
**6.** When using statements such as `if`, `for`, and `while` (unlike function calls), put a space before the opening bracket.
**6.** In `if`, `for`, `while` and other expressions, a space is inserted in front of the opening bracket (as opposed to function calls).
```cpp
for (size_t i = 0; i < rows; i += storage.index_granularity)
```
**7.** Put spaces around binary operators (`+`, `-`, `*`, `/`, `%`, ...), as well as the ternary operator `?:`.
**7.** Add spaces around binary operators (`+`, `-`, `*`, `/`, `%`, ...) and the ternary operator `?:`.
```cpp
UInt16 year = (s[0] - '0') * 1000 + (s[1] - '0') * 100 + (s[2] - '0') * 10 + (s[3] - '0');
@ -77,13 +79,13 @@ dst.ClickGoodEvent = click.GoodEvent;
If necessary, the operator can be wrapped to the next line. In this case, the offset in front of it is increased.
**11.** Do not use a space to separate unary operators (`-`, `+`, `*`, `&`, ...) from the argument.
**11.** Do not use a space to separate unary operators (`--`, `++`, `*`, `&`, ...) from the argument.
**12.** Put a space after a comma, but not before it. The same rule goes for a semicolon inside a for expression.
**12.** Put a space after a comma, but not before it. The same rule goes for a semicolon inside a `for` expression.
**13.** Do not use spaces to separate the `[]` operator.
**14.** In a `template <...>` expression, use a space between `template` and `<`. No spaces after `<` or before `>`.
**14.** In a `template <...>` expression, use a space between `template` and `<`; no spaces after `<` or before `>`.
```cpp
template <typename TKey, typename TValue>
@ -91,7 +93,7 @@ struct AggregatedStatElement
{}
```
**15.** In classes and structures, public, private, and protected are written on the same level as the `class/struct`, but all other internal elements should be deeper.
**15.** In classes and structures, write `public`, `private`, and `protected` on the same level as `class/struct`, and indent the rest of the code.
```cpp
template <typename T>
@ -104,9 +106,11 @@ public:
}
```
**16.** If the same namespace is used for the entire file, and there isn't anything else significant, an offset is not necessary inside namespace.
**16.** If the same `namespace` is used for the entire file, and there isn't anything else significant, an offset is not necessary inside `namespace`.
**17.** If the block for `if`, `for`, `while`... expressions consists of a single statement, you don't need to use curly brackets. Place the statement on a separate line, instead. The same is true for a nested if, for, while... statement. But if the inner statement contains curly brackets or else, the external block should be written in curly brackets.
**17.** If the block for an `if`, `for`, `while`, or other expression consists of a single `statement`, the curly brackets are optional. Place the `statement` on a separate line, instead. This rule is also valid for nested `if`, `for`, `while`, ...
But if the inner `statement` contains curly brackets or `else`, the external block should be written in curly brackets.
```cpp
/// Finish write.
@ -114,9 +118,9 @@ for (auto & stream : streams)
stream.second->finalize();
```
**18.** There should be any spaces at the ends of lines.
**18.** There shouldn't be any spaces at the ends of lines.
**19.** Sources are UTF-8 encoded.
**19.** Source files are UTF-8 encoded.
**20.** Non-ASCII characters can be used in string literals.
@ -124,13 +128,13 @@ for (auto & stream : streams)
<< ", " << (timer.elapsed() / chunks_stats.hits) << " μsec/hit.";
```
**21.** Do not write multiple expressions in a single line.
**21** Do not write multiple expressions in a single line.
**22.** Group sections of code inside functions and separate them with no more than one empty line.
**23.** Separate functions, classes, and so on with one or two empty lines.
**24.** A `const` (related to a value) must be written before the type name.
**24.** `A const` (related to a value) must be written before the type name.
```cpp
//correct
@ -206,7 +210,7 @@ This is very important. Writing the comment might help you realize that the code
/** Part of piece of memory, that can be used.
* For example, if internal_buffer is 1MB, and there was only 10 bytes loaded to buffer from file for reading,
* then working_buffer will have size of only 10 bytes
* (working_buffer.end() will point to the position right after those 10 bytes available for read).
* (working_buffer.end() will point to position right after those 10 bytes available for read).
*/
```
@ -221,8 +225,8 @@ void executeQuery(
ReadBuffer & istr, /// Where to read the query from (and data for INSERT, if applicable)
WriteBuffer & ostr, /// Where to write the result
Context & context, /// DB, tables, data types, engines, functions, aggregate functions...
BlockInputStreamPtr & query_plan, /// A description of query processing can be included here
QueryProcessingStage::Enum stage = QueryProcessingStage::Complete /// The last stage to process the SELECT query to
BlockInputStreamPtr & query_plan, /// Here could be written the description on how query was executed
QueryProcessingStage::Enum stage = QueryProcessingStage::Complete /// Up to which stage process the SELECT query
)
```
@ -253,7 +257,7 @@ void executeQuery(
*/
```
The example is borrowed from [http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/](http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/).
The example is borrowed from the resource [http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/](http://home.tamk.fi/~jaalto/course/coding-style/doc/unmaintainable-code/).
**7.** Do not write garbage comments (author, creation date ..) at the beginning of each file.
@ -263,9 +267,9 @@ Note: You can use Doxygen to generate documentation from these comments. But Dox
**9.** Multi-line comments must not have empty lines at the beginning and end (except the line that closes a multi-line comment).
**10.** For commenting out code, use basic comments, not "documenting" comments.
**10.** For commenting out code, use basic comments, not “documenting” comments.
**11.** Delete the commented out parts of the code before commiting.
**11.** Delete the commented out parts of the code before committing.
**12.** Do not use profanity in comments or code.
@ -275,63 +279,63 @@ Note: You can use Doxygen to generate documentation from these comments. But Dox
/// WHAT THE FAIL???
```
**14.** Do not make delimeters from comments.
**14.** Do not use comments to make delimeters.
```
```cpp
///******************************************************
```
**15.** Do not start discussions in comments.
```
```cpp
/// Why did you do this stuff?
```
**16.** There's no need to write a comment at the end of a block describing what it was about.
```
```cpp
/// for
```
## Names
**1.** The names of variables and class members use lowercase letters with underscores.
**1.** Use lowercase letters with underscores in the names of variables and class members.
```cpp
size_t max_block_size;
```
**2.** The names of functions (methods) use camelCase beginning with a lowercase letter.
**2.** For the names of functions (methods), use camelCase beginning with a lowercase letter.
```cpp
std::string getName() const override { return "Memory"; }
```
**3.** The names of classes (structures) use CamelCase beginning with an uppercase letter. Prefixes other than I are not used for interfaces.
**3.** For the names of classes (structs), use CamelCase beginning with an uppercase letter. Prefixes other than I are not used for interfaces.
```cpp
class StorageMemory : public IStorage
```
**4.** The names of usings follow the same rules as classes, or you can add _t at the end.
**4.** `using` are named the same way as classes, or with `_t` on the end.
**5.** Names of template type arguments for simple cases: T; T, U; T1, T2.
**5.** Names of template type arguments: in simple cases, use `T`; `T`, `U`; `T1`, `T2`.
For more complex cases, either follow the rules for class names, or add the prefix T.
For more complex cases, either follow the rules for class names, or add the prefix `T`.
```cpp
template <typename TKey, typename TValue>
struct AggregatedStatElement
```
**6.** Names of template constant arguments: either follow the rules for variable names, or use N in simple cases.
**6.** Names of template constant arguments: either follow the rules for variable names, or use `N` in simple cases.
```cpp
template <bool without_www>
struct ExtractDomain
```
**7.** For abstract classes (interfaces) you can add the I prefix.
**7.** For abstract classes (interfaces) you can add the `I` prefix.
```cpp
class IBlockInputStream
@ -339,13 +343,13 @@ class IBlockInputStream
**8.** If you use a variable locally, you can use the short name.
In other cases, use a descriptive name that conveys the meaning.
In all other cases, use a name that describes the meaning.
```cpp
bool info_successfully_loaded = false;
```
**9.** `define`‘s should be in ALL_CAPS with underscores. The same is true for global constants.
**9.** Names of `define`s and global constants use ALL_CAPS with underscores.
```cpp
#define MAX_SRC_TABLE_NAMES_TO_STORE 1000
@ -353,9 +357,9 @@ bool info_successfully_loaded = false;
**10.** File names should use the same style as their contents.
If a file contains a single class, name the file the same way as the class, in CamelCase.
If a file contains a single class, name the file the same way as the class (CamelCase).
If the file contains a single function, name the file the same way as the function, in camelCase.
If the file contains a single function, name the file the same way as the function (camelCase).
**11.** If the name contains an abbreviation, then:
@ -385,7 +389,7 @@ The underscore suffix can be omitted if the argument is not used in the construc
timer (not m_timer)
```
**14.** Constants in enums use CamelCase beginning with an uppercase letter. ALL_CAPS is also allowed. If the enum is not local, use enum class.
**14.** For the constants in an `enum`, use CamelCase with a capital letter. ALL_CAPS is also acceptable. If the `enum` is non-local, use an `enum class`.
```cpp
enum class CompressionMethod
@ -397,7 +401,7 @@ enum class CompressionMethod
**15.** All names must be in English. Transliteration of Russian words is not allowed.
```cpp
```
not Stroka
```
@ -419,9 +423,9 @@ You can also use an abbreviation if the full name is included next to it in the
**1.** Memory management.
Manual memory deallocation (delete) can only be used in library code.
Manual memory deallocation (`delete`) can only be used in library code.
In library code, the delete operator can only be used in destructors.
In library code, the `delete` operator can only be used in destructors.
In application code, memory must be freed by the object that owns it.
@ -429,36 +433,42 @@ Examples:
- The easiest way is to place an object on the stack, or make it a member of another class.
- For a large number of small objects, use containers.
- For automatic deallocation of a small number of objects that reside in the heap, use shared_ptr/unique_ptr.
- For automatic deallocation of a small number of objects that reside in the heap, use `shared_ptr/unique_ptr`.
**2.** Resource management.
Use RAII and see the previous point.
Use `RAII` and see above.
**3.** Error handling.
Use exceptions. In most cases, you only need to throw an exception, and don't need to catch it (because of RAII).
Use exceptions. In most cases, you only need to throw an exception, and don't need to catch it (because of `RAII`).
In offline data processing applications, it's often acceptable to not catch exceptions.
In servers that handle user requests, it's usually enough to catch exceptions at the top level of the connection handler.
In thread functions, you should catch and keep all exceptions to rethrow them in the main thread after `join`.
```cpp
/// If there were no other calculations yet, do it synchronously
/// If there weren't any calculations yet, calculate the first block synchronously
if (!started)
{
calculate();
started = true;
}
else /// If the calculations are already in progress, wait for results
else /// If calculations are already in progress, wait for the result
pool.wait();
if (exception)
exception->rethrow();
```
Never hide exceptions without handling. Never just blindly put all exceptions to log.
Not `catch (...) {}`.
```cpp
//Not correct
catch (...) {}
```
If you need to ignore some exceptions, do so only for specific ones and rethrow the rest.
@ -472,14 +482,14 @@ catch (const DB::Exception & e)
}
```
When using functions with response codes or errno, always check the result and throw an exception in case of error.
When using functions with response codes or `errno`, always check the result and throw an exception in case of error.
```cpp
if (0 != close(fd))
throwFromErrno("Cannot close file " + file_name, ErrorCodes::CANNOT_CLOSE_FILE);
```
Asserts are not used.
`Do not use assert`.
**4.** Exception types.
@ -491,10 +501,10 @@ This is not recommended, but it is allowed.
Use the following options:
- Create a (done() or finalize()) function that will do all the work in advance that might lead to an exception. If that function was called, there should be no exceptions in the destructor later.
- Create a function (`done()` or `finalize()`) that will do all the work in advance that might lead to an exception. If that function was called, there should be no exceptions in the destructor later.
- Tasks that are too complex (such as sending messages over the network) can be put in separate method that the class user will have to call before destruction.
- If there is an exception in the destructor, it’s better to log it than to hide it (if the logger is available).
- In simple applications, it is acceptable to rely on std::terminate (for cases of noexcept by default in C++11) to handle exceptions.
- In simple applications, it is acceptable to rely on `std::terminate` (for cases of `noexcept` by default in C++11) to handle exceptions.
**6.** Anonymous code blocks.
@ -514,7 +524,7 @@ ready_any.set();
**7.** Multithreading.
For offline data processing applications:
In offline data processing programs:
- Try to get the best possible performance on a single CPU core. You can then parallelize your code if necessary.
@ -524,11 +534,11 @@ In server applications:
Fork is not used for parallelization.
**8.** Synchronizing threads.
**8.** Syncing threads.
Often it is possible to make different threads use different memory cells (even better: different cache lines,) and to not use any thread synchronization (except joinAll).
Often it is possible to make different threads use different memory cells (even better: different cache lines,) and to not use any thread synchronization (except `joinAll`).
If synchronization is required, in most cases, it is sufficient to use mutex under lock_guard.
If synchronization is required, in most cases, it is sufficient to use mutex under `lock_guard`.
In other cases use system synchronization primitives. Do not use busy wait.
@ -542,40 +552,40 @@ In most cases, prefer references.
**10.** const.
Use constant references, pointers to constants, `const_iterator`, `const` methods.
Use constant references, pointers to constants, `const_iterator`, and const methods.
Consider `const` to be default and use non-const only when necessary.
Consider `const` to be default and use non-`const` only when necessary.
When passing variable by value, using `const` usually does not make sense.
When passing variables by value, using `const` usually does not make sense.
**11.** unsigned.
Use `unsigned`, if needed.
Use `unsigned` if necessary.
**12.** Numeric types
**12.** Numeric types.
Use `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int8`, `Int16`, `Int32`, `Int64`, and `size_t`, `ssize_t`, `ptrdiff_t`.
Use the types `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int8`, `Int16`, `Int32`, and `Int64`, as well as `size_t`, `ssize_t`, and `ptrdiff_t`.
Don't use `signed/unsigned long`, `long long`, `short`, `signed char`, `unsigned char`, or `char` types for numbers.
Don't use these types for numbers: `signed/unsigned long`, `long long`, `short`, `signed/unsigned char`, `char`.
**13.** Passing arguments.
Pass complex values by reference (including `std::string`).
If a function captures ownership of an objected created in the heap, make the argument type `shared_ptr` or `unique_ptr`.
If a function captures ownership of an object created in the heap, make the argument type `shared_ptr` or `unique_ptr`.
**14.** Returning values.
**14.** Return values.
In most cases, just use return. Do not write `[return std::move(res)]{.strike}`.
In most cases, just use `return`. Do not write `[return std::move(res)]{.strike}`.
If the function allocates an object on heap and returns it, use `shared_ptr` or `unique_ptr`.
In rare cases you might need to return the value via an argument. In this case, the argument should be a reference.
```
```cpp
using AggregateFunctionPtr = std::shared_ptr<IAggregateFunction>;
/** Creates an aggregate function by name.
/** Allows creating an aggregate function by its name.
*/
class AggregateFunctionFactory
{
@ -586,28 +596,28 @@ public:
**15.** namespace.
There is no need to use a separate namespace for application code or small libraries.
There is no need to use a separate `namespace` for application code.
or small libraries.
Small libraries don't need this, either.
For medium to large libraries, put everything in the namespace.
For medium to large libraries, put everything in a `namespace`.
You can use the additional detail namespace in a library's `.h` file to hide implementation details.
In the library's `.h` file, you can use `namespace detail` to hide implementation details not needed for the application code.
In a `.cpp` file, you can use the static or anonymous namespace to hide symbols.
In a `.cpp` file, you can use a `static` or anonymous namespace to hide symbols.
You can also use namespace for enums to prevent its names from polluting the outer namespace, but it’s better to use the enum class.
Also, a `namespace` can be used for an `enum` to prevent the corresponding names from falling into an external `namespace` (but it's better to use an `enum class`).
**16.** Delayed initialization.
**16.** Deferred initialization.
If arguments are required for initialization then do not write a default constructor.
If arguments are required for initialization, then you normally shouldn't write a default constructor.
If later you’ll need to delay initialization, you can add a default constructor that will create an invalid object. Or, for a small number of objects, you can use `shared_ptr/unique_ptr`.
```cpp
Loader(DB::Connection * connection_, const std::string & query, size_t max_block_size_);
/// For delayed initialization
/// For deferred initialization
Loader() {}
```
@ -639,11 +649,11 @@ Do not use profanity in the log.
Use UTF-8 encoding in the log. In rare cases you can use non-ASCII characters in the log.
**20.** I/O.
**20.** Input-output.
Don't use iostreams in internal cycles that are critical for application performance (and never use stringstream).
Don't use `iostreams` in internal cycles that are critical for application performance (and never use `stringstream`).
Use the DB/IO library instead.
Use the `DB/IO` library instead.
**21.** Date and time.
@ -655,30 +665,26 @@ Always use `#pragma once` instead of include guards.
**23.** using.
The `using namespace` is not used.
`using namespace` is not used. You can use `using` with something specific. But make it local inside a class or function.
It's fine if you are 'using' something specific, but make it local inside a class or function.
**24.** Do not use `trailing return type` for functions unless necessary.
**24.** Do not use trailing return type for functions unless necessary.
```
```cpp
[auto f() -&gt; void;]{.strike}
```
**25.** Do not declare and init variables like this:
**25.** Declaration and initialization of variables.
```cpp
//right way
std::string s = "Hello";
std::string s{"Hello"};
//wrong way
auto s = std::string{"Hello"};
```
Do it like this:
```cpp
std::string s = "Hello";
std::string s{"Hello"};
```
**26.** For virtual functions, write `virtual` in the base class, but write `override` in descendent classes.
**26.** For virtual functions, write `virtual` in the base class, but write `override` instead of `virtual` in descendent classes.
## Unused Features of C++
@ -692,11 +698,11 @@ std::string s{"Hello"};
But other things being equal, cross-platform or portable code is preferred.
**2.** The language is C++17.
**2.** Language: C++17.
**3.** The compiler is `gcc`. At this time (December 2017), the code is compiled using version 7.2. (It can also be compiled using clang 5.)
**3.** Compiler: `gcc`. At this time (December 2017), the code is compiled using version 7.2. (It can also be compiled using `clang 4`.)
The standard library is used (implementation of `libstdc++` or `libc++`).
The standard library is used (`libstdc++` or `libc++`).
**4.**OS: Linux Ubuntu, not older than Precise.
@ -712,17 +718,17 @@ The CPU instruction set is the minimum supported set among our servers. Currentl
## Tools
**1.** `KDevelop` is a good IDE.
**1.** KDevelop is a good IDE.
**2.** For debugging, use `gdb`, `valgrind` (`memcheck`), `strace`, `-fsanitize=`, ..., `tcmalloc_minimal_debug`.
**2.** For debugging, use `gdb`, `valgrind` (`memcheck`), `strace`, `-fsanitize=...`, or `tcmalloc_minimal_debug`.
**3.** For profiling, use Linux Perf `valgrind` (`callgrind`), `strace-cf`.
**3.** For profiling, use `Linux Perf`, `valgrind` (`callgrind`), or `strace -cf`.
**4.** Sources are in Git.
**5.** Compilation is managed by `CMake`.
**5.** Assembly uses `CMake`.
**6.** Releases are in `deb` packages.
**6.** Programs are released using `deb` packages.
**7.** Commits to master must not break the build.
@ -732,15 +738,15 @@ Though only selected revisions are considered workable.
Use branches for this purpose.
If your code is not buildable yet, exclude it from the build before pushing to master. You'll need to finish it or remove it from master within a few days.
If your code in the `master` branch is not buildable yet, exclude it from the build before the `push`. You'll need to finish it or remove it within a few days.
**9.** For non-trivial changes, used branches and publish them on the server.
**9.** For non-trivial changes, use branches and publish them on the server.
**10.** Unused code is removed from the repository.
## Libraries
**1.** The C++14 standard library is used (experimental extensions are fine), as well as boost and Poco frameworks.
**1.** The C++14 standard library is used (experimental extensions are allowed), as well as `boost` and `Poco` frameworks.
**2.** If necessary, you can use any well-known libraries available in the OS package.
@ -750,9 +756,9 @@ If there is a good solution already available, then use it, even if it means you
**3.** You can install a library that isn't in the packages, if the packages don't have what you need or have an outdated version or the wrong type of compilation.
**4.** If the library is small and doesn't have its own complex build system, put the source files in the contrib folder.
**4.** If the library is small and doesn't have its own complex build system, put the source files in the `contrib` folder.
**5.** Preference is always given to libraries that are already used.
**5.** Preference is always given to libraries that are already in use.
## General Recommendations
@ -762,31 +768,31 @@ If there is a good solution already available, then use it, even if it means you
**3.** Don't write code until you know how it's going to work and how the inner loop will function.
**4.** In the simplest cases, use 'using' instead of classes or structs.
**4.** In the simplest cases, use `using` instead of classes or structs.
**5.** If possible, do not write copy constructors, assignment operators, destructors (other than a virtual one, if the class contains at least one virtual function), mpve-constructors and move assignment operators. In other words, the compiler-generated functions must work correctly. You can use 'default'.
**5.** If possible, do not write copy constructors, assignment operators, destructors (other than a virtual one, if the class contains at least one virtual function), move constructors or move assignment operators. In other words, the compiler-generated functions must work correctly. You can use `default`.
**6.** Code simplification is encouraged. Reduce the size of your code where possible.
## Additional Recommendations
**1.** Explicit `std::` for types from `stddef.h` is not recommended.
**1.** Explicitly specifying `std::` for types from `stddef.h`
We recommend writing `size_t` instead `std::size_t` because it's shorter.
is not recommended. In other words, we recommend writing `size_t` instead `std::size_t`, because it's shorter.
But if you prefer, `std::` is acceptable.
It is acceptable to add `std::`.
**2.** Explicit `std::` for functions from the standard C library is not recommended.
**2.** Explicitly specifying `std::` for functions from the standard C library
Write `memcpy` instead of `std::memcpy`.
is not recommended. In other words, write `memcpy` instead of `std::memcpy`.
The reason is that there are similar non-standard functions, such as `memmem`. We do use these functions on occasion. These functions do not exist in namespace `std`.
The reason is that there are similar non-standard functions, such as `memmem`. We do use these functions on occasion. These functions do not exist in `namespace std`.
If you write `std::memcpy` instead of `memcpy` everywhere, then `memmem` without `std::` will look awkward.
If you write `std::memcpy` instead of `memcpy` everywhere, then `memmem` without `std::` will look strange.
Nevertheless, `std::` is allowed if you prefer it.
Nevertheless, you can still use `std::` if you prefer it.
**3.** Using functions from C when the ones are available in the standard C++ library.
**3.** Using functions from C when the same ones are available in the standard C++ library.
This is acceptable if it is more efficient.

11
docs/en/faq/general.md
View File

@ -2,11 +2,12 @@
## Why Not Use Something Like MapReduce?
We can refer to systems like MapReduce as distributed computing systems in which the reduce operation is based on a distributed sort. The most common opensource solution of this kind is [Apache Hadoop](http://hadoop.apache.org), while Yandex internally uses it's own MapReduce implementation — YT.
We can refer to systems like MapReduce as distributed computing systems in which the reduce operation is based on distributed sorting. The most common open source solution in this class is [Apache Hadoop](http://hadoop.apache.org). Yandex uses their in-house solution, YT.
The systems of this kind are not suitable for online queries due to their high latency. In other words, they can't be used as the back-end for a web interface.
These systems aren't appropriate for online queries due to their high latency. In other words, they can't be used as the back-end for a web interface.
These types of systems aren't useful for real-time data updates.
Distributed sorting isn't the best way to perform reduce operations if the result of the operation and all the intermediate results (if there are any) are located in the RAM of a single server, which is usually the case for online queries. In such a case, a hash table is the optimal way to perform reduce operations. A common approach to optimizing map-reduce tasks is pre-aggregation (partial reduce) using a hash table in RAM. The user performs this optimization manually.
Distributed sorting is one of the main causes of reduced performance when running simple map-reduce tasks.
Distributed sorting isn't the best way to perform reduce operations if the result of the operation and all the intermediate results (if there are any) are located in the RAM of a single server, which is usually the case for online queries. In such a case, a hash table is the optimal way to perform reduce operations. A common approach to optimizing MapReduce tasks is pre-aggregation (partial reduce) using a hash table in RAM. The user performs this optimization manually.
Distributed sorting is one of the main causes of reduced performance when running simple MapReduce tasks.
Most MapReduce implementations allow you to execute arbitrary code on a cluster. But a declarative query language is better suited to OLAP in order to run experiments quickly. For example, Hadoop has Hive and Pig. Also consider Cloudera Impala or Shark (outdated) for Spark, as well as Spark SQL, Presto, and Apache Drill. Performance when running such tasks is highly sub-optimal compared to specialized systems, but relatively high latency makes it unrealistic to use these systems as the backend for a web interface.
Most MapReduce implementations allow executing any code on the cluster. But a declarative query language is better suited to OLAP in order to run experiments quickly. For example, Hadoop has Hive and Pig. Also consider Cloudera Impala, Shark (outdated) for Spark, and Spark SQL, Presto, and Apache Drill. Performance when running such tasks is highly sub-optimal compared to specialized systems, but relatively high latency makes it unrealistic to use these systems as the backend for a web interface.

1
docs/en/getting_started/example_datasets/amplab_benchmark.md
View File

@ -118,3 +118,4 @@ GROUP BY sourceIP
ORDER BY totalRevenue DESC
LIMIT 1
```

11
docs/en/getting_started/example_datasets/nyc_taxi.md
View File

@ -2,7 +2,7 @@
## How to Import The Raw Data
See <https://github.com/toddwschneider/nyc-taxi-data> and <http://tech.marksblogg.com/billion-nyc-taxi-rides-redshift.html> for the description of the dataset and instructions for downloading.
See <https://github.com/toddwschneider/nyc-taxi-data> and <http://tech.marksblogg.com/billion-nyc-taxi-rides-redshift.html> for the description of a dataset and instructions for downloading.
Downloading will result in about 227 GB of uncompressed data in CSV files. The download takes about an hour over a 1 Gbit connection (parallel downloading from s3.amazonaws.com recovers at least half of a 1 Gbit channel).
Some of the files might not download fully. Check the file sizes and re-download any that seem doubtful.
@ -311,9 +311,7 @@ ORDER BY year, count(*) DESC
The following server was used:
Two Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, 16 physical kernels total,
128 GiB RAM,
8x6 TB HD on hardware RAID-5
Two Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, 16 physical kernels total,128 GiB RAM,8x6 TB HD on hardware RAID-5
Execution time is the best of three runsBut starting from the second run, queries read data from the file system cache. No further caching occurs: the data is read out and processed in each run.
@ -360,8 +358,9 @@ We ran queries using a client located in a Yandex datacenter in Finland on a clu
## Summary
| nodes | Q1 | Q2 | Q3 | Q4 |
| ----- | ----- | ----- | ----- | ----- |
| servers | Q1 | Q2 | Q3 | Q4 |
| ------- | ----- | ----- | ----- | ----- |
| 1 | 0.490 | 1.224 | 2.104 | 3.593 |
| 3 | 0.212 | 0.438 | 0.733 | 1.241 |
| 140 | 0.028 | 0.043 | 0.051 | 0.072 |

3
docs/en/getting_started/example_datasets/ontime.md
View File

@ -308,7 +308,7 @@ SELECT OriginCityName, DestCityName, count() AS c FROM ontime GROUP BY OriginCit
SELECT OriginCityName, count() AS c FROM ontime GROUP BY OriginCityName ORDER BY c DESC LIMIT 10;
```
This performance test was created by Vadim Tkachenko. For mode details see:
This performance test was created by Vadim Tkachenko. See:
- <https://www.percona.com/blog/2009/10/02/analyzing-air-traffic-performance-with-infobright-and-monetdb/>
- <https://www.percona.com/blog/2009/10/26/air-traffic-queries-in-luciddb/>
@ -316,3 +316,4 @@ This performance test was created by Vadim Tkachenko. For mode details see:
- <https://www.percona.com/blog/2014/04/21/using-apache-hadoop-and-impala-together-with-mysql-for-data-analysis/>
- <https://www.percona.com/blog/2016/01/07/apache-spark-with-air-ontime-performance-data/>
- <http://nickmakos.blogspot.ru/2012/08/analyzing-air-traffic-performance-with.html>

1
docs/en/getting_started/example_datasets/star_schema.md
View File

@ -82,3 +82,4 @@ Downloading data (change 'customer' to 'customerd' in the distributed version):
cat customer.tbl | sed 's/$/2000-01-01/' | clickhouse-client --query "INSERT INTO customer FORMAT CSV"
cat lineorder.tbl | clickhouse-client --query "INSERT INTO lineorder FORMAT CSV"
```

1
docs/en/getting_started/example_datasets/wikistat.md
View File

@ -24,3 +24,4 @@ for i in {2007..2016}; do for j in {01..12}; do echo $i-$j >&2; curl -sSL "http:
cat links.txt | while read link; do wget http://dumps.wikimedia.org/other/pagecounts-raw/$(echo $link | sed -r 's/pagecounts-([0-9]{4})([0-9]{2})[0-9]{2}-[0-9]+\.gz/\1/')/$(echo $link | sed -r 's/pagecounts-([0-9]{4})([0-9]{2})[0-9]{2}-[0-9]+\.gz/\1-\2/')/$link; done
ls -1 /opt/wikistat/ | grep gz | while read i; do echo $i; gzip -cd /opt/wikistat/$i | ./wikistat-loader --time="$(echo -n $i | sed -r 's/pagecounts-([0-9]{4})([0-9]{2})([0-9]{2})-([0-9]{2})([0-9]{2})([0-9]{2})\.gz/\1-\2-\3 \4-00-00/')" | clickhouse-client --query="INSERT INTO wikistat FORMAT TabSeparated"; done
```

12
docs/en/getting_started/index.md
View File

@ -2,15 +2,19 @@
## System Requirements
This is not a cross-platform system. It requires Linux Ubuntu Precise (12.04) or newer, with x86_64 architecture and support for the SSE 4.2 instruction set.
Installation from the official repository requires Linux with x86_64 architecture and support for the SSE 4.2 instruction set.
To check for SSE 4.2:
```bash
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
```
We recommend using Ubuntu Trusty, Ubuntu Xenial, or Ubuntu Precise.
The terminal must use UTF-8 encoding (the default in Ubuntu).
We recommend using Ubuntu or Debian. The terminal must use UTF-8 encoding.
For rpm-based systems, you can use 3rd-party packages: https://packagecloud.io/altinity/clickhouse or install debian packages.
ClickHouse also works on FreeBSD and Mac OS X. It can be compiled for x86_64 processors without SSE 4.2 support, and for AArch64 CPUs.
## Installation
@ -62,7 +66,7 @@ For the server, create a catalog with data, such as:
(Configurable in the server config.)
Run 'chown' for the desired user.
Note the path to logs in the server config (src/dbms/programs/config.xml).
Note the path to logs in the server config (src/dbms/programs/server/config.xml).
### Other Installation Methods

61
docs/en/index.md
View File

@ -1,76 +1,75 @@
# What is ClickHouse?
ClickHouse is a columnar database management system (DBMS) for online analytical processing (OLAP).
ClickHouse is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).
In a "normal" row-oriented DBMS, data is stored in this order:
| Row | WatchID | JavaEnable | Title | GoodEvent | EventTime |
| --- | ----------- | ---------- | ------------------ | --------- | ------------------- |
| #0 | 89354350662 | 1 | Investor Relations | 1 | 2016-05-18 05:19:20 |
| #1 | 90329509958 | 0 | Contact us | 1 | 2016-05-18 08:10:20 |
| #2 | 89953706054 | 1 | Mission | 1 | 2016-05-18 07:38:00 |
| ------ | ------------------- | ---------- | ------------------ | --------- | ------------------- |
| #0 | 5385521489354350662 | 1 | Investor Relations | 1 | 2016-05-18 05:19:20 |
| #1 | 5385521490329509958 | 0 | Contact us | 1 | 2016-05-18 08:10:20 |
| #2 | 5385521489953706054 | 1 | Mission | 1 | 2016-05-18 07:38:00 |
| #N | ... | ... | ... | ... | ... |
In order words, all the values related to a row are physically stored next to each other.
Examples of a row-oriented DBMSs are MySQL, Postgres and MS SQL Server.
Examples of a row-oriented DBMS are MySQL, Postgres, and MS SQL Server.
{: .grey }
In a column-oriented DBMS, data is stored like this:
| Row: | #0 | #1 | #2 | #N |
| ----------- | ------------------- | ------------------- | ------------------- | ------------------- |
| WatchID: | 89354350662 | 90329509958 | 89953706054 | ... |
| WatchID: | 5385521489354350662 | 5385521490329509958 | 5385521489953706054 | ... |
| JavaEnable: | 1 | 0 | 1 | ... |
| Title: | Investor Relations | Contact us | Mission | ... |
| GoodEvent: | 1 | 1 | 1 | ... |
| EventTime: | 2016-05-18 05:19:20 | 2016-05-18 08:10:20 | 2016-05-18 07:38:00 | ... |
These examples only show the order that data is arranged in.
The values from different columns are stored separately, and data from the same column is stored together.
Examples of column-oriented DBMSs: Vertica, Paraccel (Actian Matrix, Amazon Redshift), Sybase IQ, Exasol, Infobright, InfiniDB, MonetDB (VectorWise, Actian Vector), LucidDB, SAP HANA, Google Dremel, Google PowerDrill, Druid, kdb+.
Examples of a column-oriented DBMS: Vertica, Paraccel (Actian Matrix and Amazon Redshift), Sybase IQ, Exasol, Infobright, InfiniDB, MonetDB (VectorWise and Actian Vector), LucidDB, SAP HANA, Google Dremel, Google PowerDrill, Druid, and kdb+.
{: .grey }
Different orders for storing data are better suited to different scenarios. The data access scenario refers to which queries are made, how often, and in what proportion; how much data is read for each type of query – rows, columns, and bytes; the relationship between reading and writing data; the size of the actively used dataset and how locally it is used; whether transactions are used, and how isolated they are; requirements for data replication and logical integrity; requirements for latency and throughput for each type of query, and so on.
Different orders for storing data are better suited to different scenarios.
The data access scenario refers to what queries are made, how often, and in what proportion; how much data is read for each type of query – rows, columns, and bytes; the relationship between reading and updating data; the working size of the data and how locally it is used; whether transactions are used, and how isolated they are; requirements for data replication and logical integrity; requirements for latency and throughput for each type of query, and so on.
The higher the load on the system, the more important it is to customize the system set up to match the requirements of the usage scenario, and the more fine grained this customization becomes. There is no system that is equally well-suited to significantly different scenarios. If a system is adaptable to a wide set of scenarios, under a high load, the system will handle all the scenarios equally poorly, or will work well for just one or few of possible scenarios.
The higher the load on the system, the more important it is to customize the system to the scenario, and the more specific this customization becomes. There is no system that is equally well-suited to significantly different scenarios. If a system is adaptable to a wide set of scenarios, under a high load, the system will handle all the scenarios equally poorly, or will work well for just one of the scenarios.
## Key Properties of OLAP Scenario
## Key features of the OLAP scenario
- The vast majority of requests are for read access.
- Data is ingested in fairly large batches (> 1000 rows), not by single rows; or it is not updated at all.
- Data is updated in fairly large batches (> 1000 rows), not by single rows; or it is not updated at all.
- Data is added to the DB but is not modified.
- For reads, quite a large number of rows are extracted from the DB, but only a small subset of columns.
- Tables are "wide", meaning they contain a large number of columns.
- Queries are relatively rare (usually hundreds of queries per second per server or less).
- Tables are "wide," meaning they contain a large number of columns.
- Queries are relatively rare (usually hundreds of queries per server or less per second).
- For simple queries, latencies around 50 ms are allowed.
- Column values are fairly small: numbers and short strings (for example, 60 bytes per URL).
- Requires high throughput when processing a single query (up to billions of rows per second per server).
- Transactions are not necessary.
- There are no transactions.
- Low requirements for data consistency.
- There is one large table per query. All tables are small, except for one.
- A query result is significantly smaller than the source data. In other words, data is filtered or aggregated, so the result fits in a single server's RAM.
- A query result is significantly smaller than the source data. In other words, data is filtered or aggregated. The result fits in a single server's RAM.
It is easy to see that the OLAP scenario is very different from other popular scenarios (such as OLTP or Key-Value access). So it doesn't make sense to try to use OLTP or a Key-Value DB for processing analytical queries if you want to get decent performance. For example, if you try to use MongoDB or Redis for analytics, you will get very poor performance compared to OLAP databases.
## Reasons Why Columnar Databases Are Better Suited for OLAP Scenario
## Why column-oriented databases are better for the OLAP scenario
Column-oriented databases are better suited to OLAP scenarios (at least 100 times better in processing speed for most queries). The reasons for that are explained below in detail, but it's easier to be demonstrated visually:
Column-oriented databases are better suited to OLAP scenarios: they are at least 100 times faster in processing most queries. The reasons are explained in detail below, but the fact is easier to demonstrate visually:
**Row oriented**
**Row-oriented DBMS**
![Row oriented](images/row_oriented.gif#)
![Row-oriented ]( images/row_oriented.gif#)
**Column oriented**
**Column-oriented DBMS**
![Column oriented](images/column_oriented.gif#)
![Column-oriented](images / column_oriented.gif#)
See the difference? Read further to learn why this happens.
See the difference?
### Input/output
### About input/output
1. For an analytical query, only a small number of table columns need to be read. In a column-oriented database, you can read just the data you need. For example, if you need 5 columns out of 100, you can expect a 20-fold reduction in I/O.
2. Since data is read in packets, it is easier to compress. Data in columns is also easier to compress. This further reduces the I/O volume.
@ -121,12 +120,11 @@ LIMIT 20
20 rows in set. Elapsed: 0.153 sec. Processed 1.00 billion rows, 4.00 GB (6.53 billion rows/s., 26.10 GB/s.)
:)
</pre>
</p>
</details>
:)</pre>
### CPU
</p></details>
### By calculation
Since executing a query requires processing a large number of rows, it helps to dispatch all operations for entire vectors instead of for separate rows, or to implement the query engine so that there is almost no dispatching cost. If you don't do this, with any half-decent disk subsystem, the query interpreter inevitably stalls the CPU.
It makes sense to both store data in columns and process it, when possible, by columns.
@ -140,4 +138,3 @@ There are two ways to do this:
This is not done in "normal" databases, because it doesn't make sense when running simple queries. However, there are exceptions. For example, MemSQL uses code generation to reduce latency when processing SQL queries. (For comparison, analytical DBMSs require optimization of throughput, not latency.)
Note that for CPU efficiency, the query language must be declarative (SQL or MDX), or at least a vector (J, K). The query should only contain implicit loops, allowing for optimization.

4
docs/en/interfaces/cli.md
View File

@ -31,6 +31,7 @@ _EOF
cat file.csv | clickhouse-client --database=test --query="INSERT INTO test FORMAT CSV";
```
In batch mode, the default data format is TabSeparated. You can set the format in the FORMAT clause of the query.
By default, you can only process a single query in batch mode. To make multiple queries from a "script," use the --multiquery parameter. This works for all queries except INSERT. Query results are output consecutively without additional separators.
@ -51,7 +52,8 @@ The history is written to `~/.clickhouse-client-history`.
By default, the format used is PrettyCompact. You can change the format in the FORMAT clause of the query, or by specifying `\G` at the end of the query, using the `--format` or `--vertical` argument in the command line, or using the client configuration file.
To exit the client, press Ctrl+D (or Ctrl+C), or enter one of the following instead of a query: "exit", "quit", "logout", "exit;", "quit;", "logout;", "q", "Q", ":q"
To exit the client, press Ctrl+D (or Ctrl+C), or enter one of the following instead of a query:
"exit", "quit", "logout", "exit;", "quit;", "logout;", "q", "Q", ":q"
When processing a query, the client shows:

215
docs/en/interfaces/formats.md
View File

@ -1,36 +1,36 @@
<a name="formats"></a>
# Input and Output Formats
# Formats for input and output data
The format determines how data is returned to you after SELECTs (how it is written and formatted by the server), and how it is accepted for INSERTs (how it is read and parsed by the server).
ClickHouse can accept (`INSERT`) and return (`SELECT`) data in various formats.
See the table below for the list of supported formats for either kinds of queries.
The table below lists supported formats and how they can be used in `INSERT` and `SELECT` queries.
Format | INSERT | SELECT
-------|--------|--------
[TabSeparated](formats.md#tabseparated) | ✔ | ✔ |
[TabSeparatedRaw](formats.md#tabseparatedraw) | ✗ | ✔ |
[TabSeparatedWithNames](formats.md#tabseparatedwithnames) | ✔ | ✔ |
[TabSeparatedWithNamesAndTypes](formats.md#tabseparatedwithnamesandtypes) | ✔ | ✔ |
[CSV](formats.md#csv) | ✔ | ✔ |
[CSVWithNames](formats.md#csvwithnames) | ✔ | ✔ |
[Values](formats.md#values) | ✔ | ✔ |
[Vertical](formats.md#vertical) | ✗ | ✔ |
[VerticalRaw](formats.md#verticalraw) | ✗ | ✔ |
[JSON](formats.md#json) | ✗ | ✔ |
[JSONCompact](formats.md#jsoncompact) | ✗ | ✔ |
[JSONEachRow](formats.md#jsoneachrow) | ✔ | ✔ |
[TSKV](formats.md#tskv) | ✔ | ✔ |
[Pretty](formats.md#pretty) | ✗ | ✔ |
[PrettyCompact](formats.md#prettycompact) | ✗ | ✔ |
[PrettyCompactMonoBlock](formats.md#prettycompactmonoblock) | ✗ | ✔ |
[PrettyNoEscapes](formats.md#prettynoescapes) | ✗ | ✔ |
[PrettySpace](formats.md#prettyspace) | ✗ | ✔ |
[RowBinary](formats.md#rowbinary) | ✔ | ✔ |
[Native](formats.md#native) | ✔ | ✔ |
[Null](formats.md#null) | ✗ | ✔ |
[XML](formats.md#xml) | ✗ | ✔ |
[CapnProto](formats.md#capnproto) | ✔ | ✔ |
| Format | INSERT | SELECT |
| ------- | -------- | -------- |
| [TabSeparated](#tabseparated) | ✔ | ✔ |
| [TabSeparatedRaw](#tabseparatedraw) | ✗ | ✔ |
| [TabSeparatedWithNames](#tabseparatedwithnames) | ✔ | ✔ |
| [TabSeparatedWithNamesAndTypes](#tabseparatedwithnamesandtypes) | ✔ | ✔ |
| [CSV](#csv) | ✔ | ✔ |
| [CSVWithNames](#csvwithnames) | ✔ | ✔ |
| [Values](#values) | ✔ | ✔ |
| [Vertical](#vertical) | ✗ | ✔ |
| [VerticalRaw](#verticalraw) | ✗ | ✔ |
| [JSON](#json) | ✗ | ✔ |
| [JSONCompact](#jsoncompact) | ✗ | ✔ |
| [JSONEachRow](#jsoneachrow) | ✔ | ✔ |
| [TSKV](#tskv) | ✔ | ✔ |
| [Pretty](#pretty) | ✗ | ✔ |
| [PrettyCompact](#prettycompact) | ✗ | ✔ |
| [PrettyCompactMonoBlock](#prettycompactmonoblock) | ✗ | ✔ |
| [PrettyNoEscapes](#prettynoescapes) | ✗ | ✔ |
| [PrettySpace](#prettyspace) | ✗ | ✔ |
| [RowBinary](#rowbinary) | ✔ | ✔ |
| [Native](#native) | ✔ | ✔ |
| [Null](#null) | ✗ | ✔ |
| [XML](#xml) | ✗ | ✔ |
| [CapnProto](#capnproto) | ✔ | ✔ |
<a name="format_capnproto"></a>
@ -57,26 +57,30 @@ struct Message {
Schema files are in the file that is located in the directory specified in [ format_schema_path](../operations/server_settings/settings.md#server_settings-format_schema_path) in the server configuration.
Deserialization is effective and usually doesn't increase the system load.
<a name="csv"></a>
## CSV
Comma Separated Values format ([RFC](https://tools.ietf.org/html/rfc4180)).
When formatting, rows are enclosed in double quotes. A double quote inside a string is output as two double quotes in a row. There are no other rules for escaping characters. Date and date-time are enclosed in double quotes. Numbers are output without quotes. Values ​​are separated by a delimiter&ast;. Rows are separated using the Unix line feed (LF). Arrays are serialized in CSV as follows: first the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double quotes. Tuples in CSV format are serialized as separate columns (that is, their nesting in the tuple is lost).
When formatting, rows are enclosed in double quotes. A double quote inside a string is output as two double quotes in a row. There are no other rules for escaping characters. Date and date-time are enclosed in double quotes. Numbers are output without quotes. Values are separated by a delimiter character, which is `,` by default. The delimiter character is defined in the setting [format_csv_delimiter](../operations/settings/settings.md#format_csv_delimiter). Rows are separated using the Unix line feed (LF). Arrays are serialized in CSV as follows: first the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double quotes. Tuples in CSV format are serialized as separate columns (that is, their nesting in the tuple is lost).
```
clickhouse-client --format_csv_delimiter="|" --query="INSERT INTO test.csv FORMAT CSV" < data.csv
```
&ast;By default — `,`. See a [format_csv_delimiter](/operations/settings/settings/#format_csv_delimiter) setting for additional info.
&ast;By default, the delimiter is `,`. See the [format_csv_delimiter](/operations/settings/settings/#format_csv_delimiter) setting for more information.
When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to a delimiter or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) are all supported.
When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to the delimiter character or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) types are all supported.
`NULL` is formatted as `\N`.
The CSV format supports the output of totals and extremes the same way as `TabSeparated`.
## CSVWithNames
Also prints the header row, similar to `TabSeparatedWithNames`.
<a name="json"></a>
## JSON
@ -162,7 +166,13 @@ If the query contains GROUP BY, rows_before_limit_at_least is the exact number o
`extremes` – Extreme values (when extremes is set to 1).
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
ClickHouse supports [NULL](../query_language/syntax.md#null-literal), which is displayed as `null` in the JSON output.
See also the JSONEachRow format.
<a name="jsoncompact"></a>
## JSONCompact
Differs from JSON only in that data rows are output in arrays, not in objects.
@ -188,8 +198,8 @@ Example:
["", "8267016"],
["bathroom interior design", "2166"],
["yandex", "1655"],
["spring 2014 fashion", "1549"],
["freeform photos", "1480"]
["fashion trends spring 2014", "1549"],
["freeform photo", "1480"]
],
"totals": ["","8873898"],
@ -208,6 +218,7 @@ Example:
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
See also the `JSONEachRow` format.
<a name="jsoneachrow"></a>
## JSONEachRow
@ -217,35 +228,51 @@ Outputs data as separate JSON objects for each row (newline delimited JSON).
{"SearchPhrase":"","count()":"8267016"}
{"SearchPhrase": "bathroom interior design","count()": "2166"}
{"SearchPhrase":"yandex","count()":"1655"}
{"SearchPhrase":"spring 2014 fashion","count()":"1549"}
{"SearchPhrase":"2014 spring fashion","count()":"1549"}
{"SearchPhrase":"freeform photo","count()":"1480"}
{"SearchPhrase":"angelina jolie","count()":"1245"}
{"SearchPhrase":"omsk","count()":"1112"}
{"SearchPhrase":"photos of dog breeds","count()":"1091"}
{"SearchPhrase":"curtain design","count()":"1064"}
{"SearchPhrase":"curtain designs","count()":"1064"}
{"SearchPhrase":"baku","count()":"1000"}
```
Unlike the JSON format, there is no substitution of invalid UTF-8 sequences. Any set of bytes can be output in the rows. This is necessary so that data can be formatted without losing any information. Values are escaped in the same way as for JSON.
For parsing, any order is supported for the values of different columns. It is acceptable for some values to be omitted – they are treated as equal to their default values. In this case, zeros and blank rows are used as default values. Complex values that could be specified in the table are not supported as defaults. Whitespace between elements is ignored. If a comma is placed after the objects, it is ignored. Objects don't necessarily have to be separated by new lines.
<a name="native"></a>
## Native
The most efficient format. Data is written and read by blocks in binary format. For each block, the number of rows, number of columns, column names and types, and parts of columns in this block are recorded one after another. In other words, this format is "columnar" – it doesn't convert columns to rows. This is the format used in the native interface for interaction between servers, for using the command-line client, and for C++ clients.
You can use this format to quickly generate dumps that can only be read by the ClickHouse DBMS. It doesn't make sense to work with this format yourself.
<a name="null"></a>
## Null
Nothing is output. However, the query is processed, and when using the command-line client, data is transmitted to the client. This is used for tests, including productivity testing.
Obviously, this format is only appropriate for output, not for parsing.
<a name="pretty"></a>
## Pretty
Outputs data as Unicode-art tables, also using ANSI-escape sequences for setting colors in the terminal.
A full grid of the table is drawn, and each row occupies two lines in the terminal.
Each result block is output as a separate table. This is necessary so that blocks can be output without buffering results (buffering would be necessary in order to pre-calculate the visible width of all the values).
[NULL](../query_language/syntax.md#null-literal) is output as `ᴺᵁᴸᴸ`.
```sql
SELECT * FROM t_null
```
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
└───┴──────┘
```
To avoid dumping too much data to the terminal, only the first 10,000 rows are printed. If the number of rows is greater than or equal to 10,000, the message "Showed first 10 000" is printed.
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
@ -278,14 +305,18 @@ Extremes:
└────────────┴─────────┘
```
<a name="prettycompact"></a>
## PrettyCompact
Differs from `Pretty` in that the grid is drawn between rows and the result is more compact.
This format is used by default in the command-line client in interactive mode.
<a name="prettycompactmonoblock"></a>
## PrettyCompactMonoBlock
Differs from `PrettyCompact` in that up to 10,000 rows are buffered, then output as a single table, not by blocks.
Differs from [PrettyCompact](#prettycompact) in that up to 10,000 rows are buffered, then output as a single table, not by blocks.
<a name="prettynoescapes"></a>
## PrettyNoEscapes
@ -306,10 +337,12 @@ The same as the previous setting.
### PrettySpaceNoEscapes
The same as the previous setting.
<a name="prettyspace"></a>
## PrettySpace
Differs from `PrettyCompact` in that whitespace (space characters) is used instead of the grid.
Differs from [PrettyCompact](#prettycompact) in that whitespace (space characters) is used instead of the grid.
<a name="rowbinary"></a>
## RowBinary
@ -324,10 +357,41 @@ FixedString is represented simply as a sequence of bytes.
Array is represented as a varint length (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), followed by successive elements of the array.
For [NULL](../query_language/syntax.md#null-literal) support, an additional byte containing 1 or 0 is added before each [Nullable](../data_types/nullable.md#data_type-nullable) value. If 1, then the value is `NULL` and this byte is interpreted as a separate value. If 0, the value after the byte is not `NULL`.
<a name="tabseparated"></a>
## TabSeparated
In TabSeparated format, data is written by row. Each row contains values separated by tabs. Each value is follow by a tab, except the last value in the row, which is followed by a line feed. Strictly Unix line feeds are assumed everywhere. The last row also must contain a line feed at the end. Values are written in text format, without enclosing quotation marks, and with special characters escaped.
This format is also available under the name `TSV`.
The `TabSeparated` format is convenient for processing data using custom programs and scripts. It is used by default in the HTTP interface, and in the command-line client's batch mode. This format also allows transferring data between different DBMSs. For example, you can get a dump from MySQL and upload it to ClickHouse, or vice versa.
The `TabSeparated` format supports outputting total values (when using WITH TOTALS) and extreme values (when 'extremes' is set to 1). In these cases, the total values and extremes are output after the main data. The main result, total values, and extremes are separated from each other by an empty line. Example:
```sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated``
```
```text
2014-03-17 1406958
2014-03-18 1383658
2014-03-19 1405797
2014-03-20 1353623
2014-03-21 1245779
2014-03-22 1031592
2014-03-23 1046491
0000-00-00 8873898
2014-03-17 1031592
2014-03-23 1406958
```
## Data formatting
Integer numbers are written in decimal form. Numbers can contain an extra "+" character at the beginning (ignored when parsing, and not recorded when formatting). Non-negative numbers can't contain the negative sign. When reading, it is allowed to parse an empty string as a zero, or (for signed types) a string consisting of just a minus sign as a zero. Numbers that do not fit into the corresponding data type may be parsed as a different number, without an error message.
Floating-point numbers are written in decimal form. The dot is used as the decimal separator. Exponential entries are supported, as are 'inf', '+inf', '-inf', and 'nan'. An entry of floating-point numbers may begin or end with a decimal point.
@ -358,30 +422,9 @@ Only a small set of symbols are escaped. You can easily stumble onto a string va
Arrays are written as a list of comma-separated values in square brackets. Number items in the array are fomratted as normally, but dates, dates with times, and strings are written in single quotes with the same escaping rules as above.
The TabSeparated format is convenient for processing data using custom programs and scripts. It is used by default in the HTTP interface, and in the command-line client's batch mode. This format also allows transferring data between different DBMSs. For example, you can get a dump from MySQL and upload it to ClickHouse, or vice versa.
[NULL](../query_language/syntax.md#null-literal) is formatted as `\N`.
The TabSeparated format supports outputting total values (when using WITH TOTALS) and extreme values (when 'extremes' is set to 1). In these cases, the total values and extremes are output after the main data. The main result, total values, and extremes are separated from each other by an empty line. Example:
```sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated``
```
```text
2014-03-17 1406958
2014-03-18 1383658
2014-03-19 1405797
2014-03-20 1353623
2014-03-21 1245779
2014-03-22 1031592
2014-03-23 1046491
0000-00-00 8873898
2014-03-17 1031592
2014-03-23 1406958
```
This format is also available under the name `TSV`.
<a name="tabseparatedraw"></a>
## TabSeparatedRaw
@ -389,6 +432,7 @@ Differs from `TabSeparated` format in that the rows are written without escaping
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
This format is also available under the name `TSVRaw`.
<a name="tabseparatedwithnames"></a>
## TabSeparatedWithNames
@ -397,6 +441,7 @@ During parsing, the first row is completely ignored. You can't use column names
(Support for parsing the header row may be added in the future.)
This format is also available under the name `TSVWithNames`.
<a name="tabseparatedwithnamesandtypes"></a>
## TabSeparatedWithNamesAndTypes
@ -404,6 +449,7 @@ Differs from the `TabSeparated` format in that the column names are written to t
During parsing, the first and second rows are completely ignored.
This format is also available under the name `TSVWithNamesAndTypes`.
<a name="tskv"></a>
## TSKV
@ -413,15 +459,25 @@ Similar to TabSeparated, but outputs a value in name=value format. Names are esc
SearchPhrase= count()=8267016
SearchPhrase=bathroom interior design count()=2166
SearchPhrase=yandex count()=1655
SearchPhrase=spring 2014 fashion count()=1549
SearchPhrase=2014 spring fashion count()=1549
SearchPhrase=freeform photos count()=1480
SearchPhrase=angelina jolia count()=1245
SearchPhrase=angelina jolie count()=1245
SearchPhrase=omsk count()=1112
SearchPhrase=photos of dog breeds count()=1091
SearchPhrase=curtain design count()=1064
SearchPhrase=curtain designs count()=1064
SearchPhrase=baku count()=1000
```
[NULL](../query_language/syntax.md#null-literal) is formatted as `\N`.
```sql
SELECT * FROM t_null FORMAT TSKV
```
```
x=1 y=\N
```
When there is a large number of small columns, this format is ineffective, and there is generally no reason to use it. It is used in some departments of Yandex.
Both data output and parsing are supported in this format. For parsing, any order is supported for the values of different columns. It is acceptable for some values to be omitted – they are treated as equal to their default values. In this case, zeros and blank rows are used as default values. Complex values that could be specified in the table are not supported as defaults.
@ -430,17 +486,37 @@ Parsing allows the presence of the additional field `tskv` without the equal sig
## Values
Prints every row in brackets. Rows are separated by commas. There is no comma after the last row. The values inside the brackets are also comma-separated. Numbers are output in decimal format without quotes. Arrays are output in square brackets. Strings, dates, and dates with times are output in quotes. Escaping rules and parsing are similar to the TabSeparated format. During formatting, extra spaces aren't inserted, but during parsing, they are allowed and skipped (except for spaces inside array values, which are not allowed).
Prints every row in brackets. Rows are separated by commas. There is no comma after the last row. The values inside the brackets are also comma-separated. Numbers are output in decimal format without quotes. Arrays are output in square brackets. Strings, dates, and dates with times are output in quotes. Escaping rules and parsing are similar to the [TabSeparated](#tabseparated) format. During formatting, extra spaces aren't inserted, but during parsing, they are allowed and skipped (except for spaces inside array values, which are not allowed). [NULL](../query_language/syntax.md#null-literal) is represented as `NULL`.
The minimum set of characters that you need to escape when passing data in Values ​​format: single quotes and backslashes.
This is the format that is used in `INSERT INTO t VALUES ...`, but you can also use it for formatting query results.
<a name="vertical"></a>
## Vertical
Prints each value on a separate line with the column name specified. This format is convenient for printing just one or a few rows, if each row consists of a large number of columns.
[NULL](../query_language/syntax.md#null-literal) is output as `ᴺᵁᴸᴸ`.
Example:
```sql
SELECT * FROM t_null FORMAT Vertical
```
```
Row 1:
──────
x: 1
y: ᴺᵁᴸᴸ
```
This format is only appropriate for outputting a query result, but not for parsing (retrieving data to insert in a table).
<a name="verticalraw"></a>
## VerticalRaw
Differs from `Vertical` format in that the rows are not escaped.
@ -469,6 +545,9 @@ Row 1:
──────
test: string with \'quotes\' and \t with some special \n characters
```
<a name="xml"></a>
## XML
XML format is suitable only for output, not for parsing. Example:
@ -502,7 +581,7 @@ XML format is suitable only for output, not for parsing. Example:
<field>1655</field>
</row>
<row>
<SearchPhrase>spring 2014 fashion</SearchPhrase>
<SearchPhrase>2014 spring fashion</SearchPhrase>
<field>1549</field>
</row>
<row>
@ -522,7 +601,7 @@ XML format is suitable only for output, not for parsing. Example:
<field>1091</field>
</row>
<row>
<SearchPhrase>curtain design</SearchPhrase>
<SearchPhrase>curtain designs</SearchPhrase>
<field>1064</field>
</row>
<row>
@ -540,6 +619,4 @@ Just as for JSON, invalid UTF-8 sequences are changed to the replacement charact
In string values, the characters `<` and `&` are escaped as `<` and `&`.
Arrays are output as `<array><elem>Hello</elem><elem>World</elem>...</array>`,
and tuples as `<tuple><elem>Hello</elem><elem>World</elem>...</tuple>`.
Arrays are output as `<array><elem>Hello</elem><elem>World</elem>...</array>`,and tuples as `<tuple><elem>Hello</elem><elem>World</elem>...</tuple>`.

6
docs/en/interfaces/http_interface.md
View File

@ -34,7 +34,8 @@ Date: Fri, 16 Nov 2012 19:21:50 GMT
1
```
As you can see, curl is somewhat inconvenient in that spaces must be URL escaped.Although wget escapes everything itself, we don't recommend using it because it doesn't work well over HTTP 1.1 when using keep-alive and Transfer-Encoding: chunked.
As you can see, curl is somewhat inconvenient in that spaces must be URL escaped.
Although wget escapes everything itself, we don't recommend using it because it doesn't work well over HTTP 1.1 when using keep-alive and Transfer-Encoding: chunked.
```bash
$ echo 'SELECT 1' | curl 'http://localhost:8123/' --data-binary @-
@ -170,8 +171,7 @@ echo 'SELECT 1' | curl 'http://localhost:8123/?user=user&password=password' -d @
```
If the user name is not indicated, the username 'default' is used. If the password is not indicated, an empty password is used.
You can also use the URL parameters to specify any settings for processing a single query, or entire profiles of settings. Example:
http://localhost:8123/?profile=web&max_rows_to_read=1000000000&query=SELECT+1
You can also use the URL parameters to specify any settings for processing a single query, or entire profiles of settings. Example:http://localhost:8123/?profile=web&max_rows_to_read=1000000000&query=SELECT+1
For more information, see the section "Settings".

6
docs/en/interfaces/jdbc.md
View File

@ -1,7 +1,5 @@
# JDBC Driver
There is an official JDBC driver for ClickHouse. See [here](https://github.com/yandex/clickhouse-jdbc) .
- [Official driver](https://github.com/yandex/clickhouse-jdbc).
- Third-party driver from [ClickHouse-Native-JDBC](https://github.com/housepower/ClickHouse-Native-JDBC).
JDBC drivers implemented by other organizations:
- [ClickHouse-Native-JDBC](https://github.com/housepower/ClickHouse-Native-JDBC)

3
docs/en/interfaces/third-party_client_libraries.md
View File

@ -1,6 +1,6 @@
# Libraries from Third-party Developers
There are libraries for working with ClickHouse for:
We have not tested the libraries listed below.
- Python
- [infi.clickhouse_orm](https://github.com/Infinidat/infi.clickhouse_orm)
@ -45,4 +45,3 @@ There are libraries for working with ClickHouse for:
- Nim
- [nim-clickhouse](https://github.com/leonardoce/nim-clickhouse)
We have not tested these libraries. They are listed in random order.

38
docs/en/interfaces/third-party_gui.md
View File

@ -4,7 +4,8 @@
Web interface for ClickHouse in the [Tabix](https://github.com/tabixio/tabix) project.
### Features:
Main features:
- Works with ClickHouse directly from the browser, without the need to install additional software.
- Query editor with syntax highlighting.
- Auto-completion of commands.
@ -13,22 +14,25 @@ Web interface for ClickHouse in the [Tabix](https://github.com/tabixio/tabix) pr
[Tabix documentation](https://tabix.io/doc/).
## HouseOps
[HouseOps](https://github.com/HouseOps/HouseOps) is a unique Desktop ClickHouse Ops UI / IDE for OSX, Linux and Windows.
[HouseOps](https://github.com/HouseOps/HouseOps) is a UI/IDE for OSX, Linux and Windows.
Main features:
- Query builder with syntax highlighting. View the response in a table or JSON view.
- Export query results as CSV or JSON.
- List of processes with descriptions. Write mode. Ability to stop (`KILL`) a process.
- Database graph. Shows all tables and their columns with additional information.
- Quick view of the column size.
- Server configuration.
The following features are planned for development:
- Database management.
- User management.
- Real-time data analysis.
- Cluster monitoring.
- Cluster management.
- Monitoring replicated and Kafka tables.
### Features:
- Query builder with syntax highlighting, response viewed in Table and JSON Object.
- Export results in csv and JSON object.
- Processes List with description, Record mode and Kill processes feature.
- Database Graph with all tables and columns with extra informations.
- Easy view your columns size.
- Server settings.
- Database manangement (soon);
- Users manangement (soon);
- Real-Time Data Analytics (soon);
- Cluster/Infra monitoring (soon);
- Cluster manangement (soon);
- Kafka and Replicated tables monitoring (soon);
- And a lot of others features for you take a beautiful implementation of ClickHouse.

49
docs/en/introduction/distinctive_features.md
View File

@ -2,68 +2,61 @@
## True Column-oriented DBMS
In a true column-oriented DBMS, there is no excessive data stored with the values. For example, this means that constant-length values must be supported, to avoid storing their length as additional integer next to the values. In this case, a billion UInt8 values should actually consume around 1 GB uncompressed, or this will strongly affect the CPU use. It is very important to store data compactly even when uncompressed, since the speed of decompression (CPU usage) depends mainly on the volume of uncompressed data.
In a true column-oriented DBMS, no extra data is stored with the values. Among other things, this means that constant-length values must be supported, to avoid storing their length "number" next to the values. As an example, a billion UInt8-type values should actually consume around 1 GB uncompressed, or this will strongly affect the CPU use. It is very important to store data compactly (without any "garbage") even when uncompressed, since the speed of decompression (CPU usage) depends mainly on the volume of uncompressed data.
This is worth noting because there are systems that can store values of different columns separately, but that can't effectively process analytical queries due to their optimization for other scenarios. Examples are HBase, BigTable, Cassandra, and HyperTable. In these systems, you will get throughput around a hundred thousand rows per second, but not hundreds of millions of rows per second.
This is worth noting because there are systems that can store values of separate columns separately, but that can't effectively process analytical queries due to their optimization for other scenarios. Examples are HBase, BigTable, Cassandra, and HyperTable. In these systems, you will get throughput around a hundred thousand rows per second, but not hundreds of millions of rows per second.
Also note that ClickHouse is a database management system, not a single database. ClickHouse allows creating tables and databases in runtime, loading data, and running queries without reconfiguring and restarting the server.
It's also worth noting that ClickHouse is a database management system, not a single database. ClickHouse allows creating tables and databases in runtime, loading data, and running queries without reconfiguring and restarting the server.
## Data Compression
Some column-oriented DBMSs (InfiniDB CE and MonetDB) do not use data compression. However, data compression is crucial to achieve excellent performance.
Some column-oriented DBMSs (InfiniDB CE and MonetDB) do not use data compression. However, data compression does play a key role in achieving excellent performance.
## Disk Storage of Data
Many column-oriented DBMSs (such as SAP HANA and Google PowerDrill) can only work in RAM. This approach stimulates the allocation of a larger hardware budget than is actually necessary for real-time analysis. ClickHouse is designed to work on regular hard drives, which ensures low cost of ownership per gigabyte of data, but SSD and additional RAM are also utilized fully if available.
Many column-oriented DBMSs (such as SAP HANA and Google PowerDrill) can only work in RAM. This approach encourages more budgeting for hardware than is actually needed for real-time analysis. ClickHouse is designed to work on normal hard drives, which means the cost per GB of data storage is low, but SSD b additional RAM is also fully used if available.
## Parallel Processing on Multiple Cores
Large queries are parallelized in a natural way, utilizing all necessary resources that are available on the current server.
Large queries are parallelized in a natural way, taking all the necessary resources from what is available on the server.
## Distributed Processing on Multiple Servers
Almost none of the columnar DBMSs mentioned above have support for distributed query processing.
Almost none of the columnar DBMSs listed above have support for distributed processing.
In ClickHouse, data can reside on different shards. Each shard can be a group of replicas that are used for fault tolerance. The query is processed on all the shards in parallel. This is transparent for the user.
## SQL Support
If you are familiar with standard SQL, we can't really talk about SQL support.
All the functions have different names.
However, this is a declarative query language based on SQL that can't be differentiated from SQL in many instances.
JOINs are supported. Subqueries are supported in FROM, IN, and JOIN clauses, as well as scalar subqueries.
Dependent subqueries are not supported.
ClickHouse supports declarative query language that is based on SQL and complies to SQL standard in many cases.
GROUP BY, ORDER BY, scalar subqueries and subqueries in FROM, IN and JOIN clauses are supported.
Correlated subqueries and window functions are not supported.
ClickHouse supports a declarative query language based on SQL that is identical to the SQL standard in many cases.
Supported queries include GROUP BY, ORDER BY, subqueries in FROM, IN, and JOIN clauses, and scalar subqueries.
Dependent subqueries and window functions are not supported.
## Vector Engine
Data is not only stored by columns, but is also processed by vectors (parts of columns). This allows to achieve high CPU efficiency.
Data is not only stored by columns, but is processed by vectors (parts of columns). This allows us to achieve high CPU performance.
## Real-time Data Updates
ClickHouse supports tables with a primary key. In order to quickly perform queries on the range of the primary key, the data is sorted incrementally using the merge tree. Due to this, data can continually be added to the table. No locks are taken when new data is ingested.
ClickHouse supports primary key tables. In order to quickly perform queries on the range of the primary key, the data is sorted incrementally using the merge tree. Due to this, data can continually be added to the table. There is no locking when adding data.
## Index
Having a data physically sorted by primary key makes it possible to extract data for it's specific values or value ranges with low latency, less than few dozen milliseconds.
Physical sorting of data by primary key allows you to get data for specific key values or ranges of values with low latency of less than several dozen milliseconds.
## Suitable for Online Queries
Low latency means that queries can be processed without delay and without trying to prepare answer in advance, right at the same moment while user interface page is loading. In other words, online.
Low latency means queries can be processed without delay and without preparing the response ahead of time, so a query can be processed while the user interface page is loading. In other words, in online mode.
## Support for Approximated Calculations
ClickHouse provides various ways to trade accuracy for performance:
ClickHouse provides various ways to change the precision of calculations for improved performance:
1. Aggregate functions for approximated calculation of the number of distinct values, medians, and quantiles.
2. Running a query based on a part (sample) of data and getting an approximated result. In this case, proportionally less data is retrieved from the disk.
3. Running an aggregation for a limited number of random keys, instead of for all keys. Under certain conditions for key distribution in the data, this provides a reasonably accurate result while using fewer resources.
1. The system contains aggregate functions for approximated calculation of the number of various values, medians, and quantiles.
2. Supports running a query based on a part (sample) of data and getting an approximated result. In this case, proportionally less data is retrieved from the disk.
3. Supports running an aggregation for a limited number of random keys, instead of for all keys. Under certain conditions for key distribution in the data, this provides a reasonably accurate result while using fewer resources.
## Data Replication and Integrity
## Data replication and data integrity support
ClickHouse uses asynchronous multimaster replication. After being written to any available replica, data is distributed to all the other replicas in background. The system maintains identical data on different replicas. Data is restored automatically after most failures, or semiautomatically in complicated cases.
For more information, see the [Data replication](../operations/table_engines/replication.md#table_engines-replication) section.
Uses asynchronous multimaster replication. After being written to any available replica, data is distributed to all the remaining replicas in the background. The system maintains identical data on different replicas. Recovery after most failures is performed automatically, and in complex cases — semi-automatically.
For more information, see the section [Data replication](../operations/table_engines/replication.md#table_engines-replication).

7
docs/en/introduction/features_considered_disadvantages.md
View File

@ -1,6 +1,7 @@
# ClickHouse Features that Can be Considered Disadvantages
1. No full-fledged transactions.
2. Lack of ability to modify or delete already inserted data with high rate and low latency. There are batch deletes available to clean up data that is not needed anymore or to comply with [GDPR](https://gdpr-info.eu). Batch updates are currently in development as of July 2018.
3. Sparse index makes ClickHouse not really suitable for point queries retrieving single rows by their keys.
1. Lack of full transactions.
2. Previously recorded data can't be changed or deleted with low latency and high query frequency. Mass deletions to clear data that is no longer relevant or falls under [GDPR](https://gdpr-info.eu) regulations. Batch changes to data are in development (as of July, 2018).
3. The sparse index makes ClickHouse ill-suited for point-reading single rows on its own
keys.

13
docs/en/introduction/performance.md
View File

@ -1,24 +1,23 @@
# Performance
According to internal testing results by Yandex, ClickHouse shows the best performance for comparable operating scenarios among systems of its class that were available for testing. This includes the highest throughput for long queries, and the lowest latency on short queries. Testing results are shown on a [separate page](https://clickhouse.yandex/benchmark.html).
According to internal testing results at Yandex, ClickHouse shows the best performance (both the highest throughput for long queries and the lowest latency on short queries) for comparable operating scenarios among systems of its class that were available for testing. You can view the test results on a [separate page](https://clickhouse.yandex/benchmark.html).
There are a lot of independent benchmarks that confirm this as well. You can look it up on your own or here is the small [collection of independent benchmark links](https://clickhouse.yandex/#independent-benchmarks).
This has also been confirmed by numerous independent benchmarks. They are not difficult to find using an internet search, or you can see [our small collection of related links](https://clickhouse.yandex/#independent-bookmarks).
## Throughput for a Single Large Query
Throughput can be measured in rows per second or in megabytes per second. If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of approximately 2-10 GB/s of uncompressed data on a single server (for the simplest cases, the speed may reach 30 GB/s). If data is not placed in the page cache, the speed is bound by the disk subsystem and how well the data has been compressed. For example, if the disk subsystem allows reading data at 400 MB/s, and the data compression rate is 3, the speed will be around 1.2 GB/s. To get the speed in rows per second, divide the speed in bytes per second by the total size of the columns used in the query. For example, if 10 bytes of columns are extracted, the speed will be around 100-200 million rows per second.
Throughput can be measured in rows per second or in megabytes per second. If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of approximately 2-10 GB/s of uncompressed data on a single server (for the simplest cases, the speed may reach 30 GB/s). If data is not placed in the page cache, the speed depends on the disk subsystem and the data compression rate. For example, if the disk subsystem allows reading data at 400 MB/s, and the data compression rate is 3, the speed will be around 1.2 GB/s. To get the speed in rows per second, divide the speed in bytes per second by the total size of the columns used in the query. For example, if 10 bytes of columns are extracted, the speed will be around 100-200 million rows per second.
The processing speed increases almost linearly for distributed processing, but only if the number of rows resulting from aggregation or sorting is not too large.
## Latency When Processing Short Queries
If a query uses a primary key and does not select too many rows to process (hundreds of thousands), and does not use too many columns, we can expect less than 50 milliseconds of latency (single digits of milliseconds in the best case) if data is placed in the page cache. Otherwise, latency is calculated from the number of seeks. If you use rotating drives, for a system that is not overloaded, the approximate latency can be calculated by this formula: seek time (10 ms) \* number of columns queried \* number of data parts.
If a query uses a primary key and does not select too many rows to process (hundreds of thousands), and does not use too many columns, we can expect less than 50 milliseconds of latency (single digits of milliseconds in the best case) if data is placed in the page cache. Otherwise, latency is calculated from the number of seeks. If you use rotating drives, for a system that is not overloaded, the latency is calculated by this formula: seek time (10 ms) \* number of columns queried \* number of data parts.
## Throughput When Processing a Large Quantity of Short Queries
Under the same circumstances, ClickHouse can handle several hundred queries per second on a single server (up to several thousands in the best case). Since this scenario is not typical for analytical DBMSs, it is better to expect a maximum of hundreds of queries per second.
Under the same conditions, ClickHouse can handle several hundred queries per second on a single server (up to several thousand in the best case). Since this scenario is not typical for analytical DBMSs, we recommend expecting a maximum of 100 queries per second.
## Performance When Inserting Data
It is recommended to insert data in batches of at least 1000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed will be from 50 to 200 MB/s. If the inserted rows are around 1 Kb in size, the speed will be from 50,000 to 200,000 rows per second. If the rows are small, the performance will be higher in rows per second (on Banner System data -`>` 500,000 rows per second; on Graphite data -`>` 1,000,000 rows per second). To improve performance, you can make multiple INSERT queries in parallel, and performance will increase linearly.
We recommend inserting data in packets of at least 1000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed will be from 50 to 200 MB/s. If the inserted rows are around 1 Kb in size, the speed will be from 50,000 to 200,000 rows per second. If the rows are small, the performance will be higher in rows per second (on Banner System data -`>` 500,000 rows per second; on Graphite data -`>` 1,000,000 rows per second). To improve performance, you can make multiple INSERT queries in parallel, and performance will increase linearly.

6
docs/en/introduction/ya_metrika_task.md
View File

@ -1,10 +1,10 @@
# Yandex.Metrica Use Case
ClickHouse has been initially developed to power [Yandex.Metrica](https://metrica.yandex.com/), [the second largest web analytics platform in the world](http://w3techs.com/technologies/overview/traffic_analysis/all), and continues to be it's core component. With more than 13 trillion records in the database and more than 20 billion events daily, ClickHouse allows generating custom reports on the fly directly from non-aggregated data. This article gives a historical background on what was the main goal of ClickHouse before it became an opensource product.
ClickHouse was originally developed to power [Yandex.Metrica](https://metrica.yandex.com/), [the second largest web analytics platform in the world](http://w3techs.com/technologies/overview/traffic_analysis/all), and continues to be the core component of this system. With more than 13 trillion records in the database and more than 20 billion events daily, ClickHouse allows generating custom reports on the fly directly from non-aggregated data. This article briefly covers the goals of ClickHouse in the early stages of its development.
Yandex.Metrica generates custom reports based on hits and sessions on the fly, with arbitrary segments and time periods chosen by the end user. Complex aggregates are often required, such as the number of unique visitors. New data for the reports arrives in real-time.
Yandex.Metrica builds customized reports on the fly based on hits and sessions, with arbitrary segments defined by the user. This often requires building complex aggregates, such as the number of unique users. New data for building a report is received in real time.
As of April 2014, Yandex.Metrica received approximately 12 billion events (page views and clicks) daily. All these events must be stored in order to build those custom reports. A single query may require scanning millions of rows in no more than a few hundred milliseconds, or hundreds of millions of rows over a few seconds.
As of April 2014, Yandex.Metrica was tracking about 12 billion events (page views and clicks) daily. All these events must be stored in order to build custom reports. A single query may require scanning millions of rows within a few hundred milliseconds, or hundreds of millions of rows in just a few seconds.
## Usage in Yandex.Metrica and Other Yandex Services

27
docs/en/operations/access_rights.md
View File

@ -11,11 +11,11 @@ Users are recorded in the `users` section. Here is a fragment of the `users.xml`
<default>
<!-- Password could be specified in plaintext or in SHA256 (in hex format).
If you want to specify the password in plain text (not recommended), place it in the 'password' element.
If you want to specify password in plaintext (not recommended), place it in 'password' element.
Example: <password>qwerty</password>.
Password can be empty.
Password could be empty.
If you want to specify SHA256, place it in the 'password_sha256_hex' element.
If you want to specify SHA256, place it in 'password_sha256_hex' element.
Example: <password_sha256_hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</password_sha256_hex>
How to generate decent password:
@ -23,16 +23,17 @@ Users are recorded in the `users` section. Here is a fragment of the `users.xml`
In first line will be password and in second - corresponding SHA256.
-->
<password></password>
<!-- A list of networks that access is allowed from.
Each list item has one of the following forms:
<ip>IP address or subnet mask. For example: 198.51.100.0/24 or 2001:DB8::/32.
<ip> The IP address or subnet mask. For example: 198.51.100.0/24 or 2001:DB8::/32.
<host> Host name. For example: example01. A DNS query is made for verification, and all addresses obtained are compared with the address of the customer.
<host_regexp> Regular expression for host names. For example: ^example\d\d-\d\d-\d\.yandex\.ru$
For verification, a DNS PTR query is made for the customer's address and a regular expression is applied to the result.
<host_regexp> Regular expression for host names. For example, ^example\d\d-\d\d-\d\.yandex\.ru$
To check it, a DNS PTR request is made for the client's address and a regular expression is applied to the result.
Then another DNS query is made for the result of the PTR query, and all received address are compared to the client address.
We strongly recommend that the regex ends with \.yandex\.ru$.
If you are installing ClickHouse yourself, enter:
If you are installing ClickHouse yourself, specify here:
<networks>
<ip>::/0</ip>
</networks>
@ -56,7 +57,6 @@ Users are recorded in the `users` section. Here is a fragment of the `users.xml`
<database>test</database>
</allow_databases>
</web>
</users>
```
You can see a declaration from two users: `default`and`web`. We added the `web` user separately.
@ -67,7 +67,7 @@ The user that is used for exchanging information between servers combined in a c
The password is specified in clear text (not recommended) or in SHA-256. The hash isn't salted. In this regard, you should not consider these passwords as providing security against potential malicious attacks. Rather, they are necessary for protection from employees.
A list of networks is specified that access is allowed from. In this example, the list of networks for both users is loaded from a separate file (/etc/metrika.xml) containing the 'networks' substitution. Here is a fragment of it:
A list of networks is specified that access is allowed from. In this example, the list of networks for both users is loaded from a separate file (`/etc/metrika.xml`) containing the `networks` substitution. Here is a fragment of it:
```xml
<yandex>
@ -81,15 +81,15 @@ A list of networks is specified that access is allowed from. In this example, th
</yandex>
```
We could have defined this list of networks directly in 'users.xml', or in a file in the 'users.d' directory (for more information, see the section "Configuration files").
You could define this list of networks directly in `users.xml`, or in a file in the `users.d` directory (for more information, see the section "[Configuration files](configuration_files.md#configuration_files)").
The config includes comments explaining how to open access from everywhere.
For use in production, only specify IP elements (IP addresses and their masks), since using 'host' and 'hoost_regexp' might cause extra latency.
For use in production, only specify `ip` elements (IP addresses and their masks), since using `host` and `hoost_regexp` might cause extra latency.
Next the user settings profile is specified (see the section "Settings profiles"). You can specify the default profile, `default`. The profile can have any name. You can specify the same profile for different users. The most important thing you can write in the settings profile is 'readonly' set to 1, which provides read-only access.
Next the user settings profile is specified (see the section "[Settings profiles](settings/settings_profiles.md#settings_profiles)"). You can specify the default profile, `default'`. The profile can have any name. You can specify the same profile for different users. The most important thing you can write in the settings profile is `readonly=1`, which ensures read-only access.
After this, the quota is defined (see the section "Quotas"). You can specify the default quota, `default`. It is set in the config by default so that it only counts resource usage, but does not restrict it. The quota can have any name. You can specify the same quota for different users – in this case, resource usage is calculated for each user individually.
Then specify the quota to be used (see the section "[Quotas](quotas.md#quotas)"). You can specify the default quota: `default`. It is set in the config by default to only count resource usage, without restricting it. The quota can have any name. You can specify the same quota for different users – in this case, resource usage is calculated for each user individually.
In the optional `<allow_databases>` section, you can also specify a list of databases that the user can access. By default, all databases are available to the user. You can specify the `default` database. In this case, the user will receive access to the database by default.
@ -98,3 +98,4 @@ Access to the `system` database is always allowed (since this database is used f
The user can get a list of all databases and tables in them by using `SHOW` queries or system tables, even if access to individual databases isn't allowed.
Database access is not related to the [readonly](settings/query_complexity.md#query_complexity_readonly) setting. You can't grant full access to one database and `readonly` access to another one.

3
docs/en/operations/configuration_files.md
View File

@ -14,7 +14,7 @@ If `replace` is specified, it replaces the entire element with the specified one
If `remove` is specified, it deletes the element.
The config can also define "substitutions". If an element has the `incl` attribute, the corresponding substitution from the file will be used as the value. By default, the path to the file with substitutions is `/etc/metrika.xml`. This can be changed in the [include_from](server_settings/settings.md#server_settings-include_from) element in the server config. The substitution values are specified in `/yandex/substitution_name` elements in this file. If a substitution specified in ` incl` does not exist, it is recorded in the log. To prevent ClickHouse from logging missing substitutions, specify the `optional="true"` attribute (for example, settings for [macros](server_settings/settings.md#server_settings-macros)).
The config can also define "substitutions". If an element has the `incl` attribute, the corresponding substitution from the file will be used as the value. By default, the path to the file with substitutions is `/etc/metrika.xml`. This can be changed in the [include_from](server_settings/settings.md#server_settings-include_from) element in the server config. The substitution values are specified in `/yandex/substitution_name` elements in this file. If a substitution specified in `incl` does not exist, it is recorded in the log. To prevent ClickHouse from logging missing substitutions, specify the `optional="true"` attribute (for example, settings for [macros]()server_settings/settings.md#server_settings-macros)).
Substitutions can also be performed from ZooKeeper. To do this, specify the attribute `from_zk = "/path/to/node"`. The element value is replaced with the contents of the node at `/path/to/node` in ZooKeeper. You can also put an entire XML subtree on the ZooKeeper node and it will be fully inserted into the source element.
@ -25,4 +25,3 @@ In addition, `users_config` may have overrides in files from the `users_config.d
For each config file, the server also generates `file-preprocessed.xml` files when starting. These files contain all the completed substitutions and overrides, and they are intended for informational use. If ZooKeeper substitutions were used in the config files but ZooKeeper is not available on the server start, the server loads the configuration from the preprocessed file.
The server tracks changes in config files, as well as files and ZooKeeper nodes that were used when performing substitutions and overrides, and reloads the settings for users and clusters on the fly. This means that you can modify the cluster, users, and their settings without restarting the server.

2
docs/en/operations/index.md
View File

@ -1,2 +1,2 @@
# Operations
# Usage

6
docs/en/operations/quotas.md
View File

@ -1,3 +1,5 @@
<a name="quotas"></a>
# Quotas
Quotas allow you to limit resource usage over a period of time, or simply track the use of resources.
@ -13,7 +15,7 @@ In contrast to query complexity restrictions, quotas:
Let's look at the section of the 'users.xml' file that defines quotas.
```xml
<!-- Quotas. -->
<!-- Quotas -->
<quotas>
<!-- Quota name. -->
<default>
@ -84,7 +86,7 @@ Quotas can use the "quota key" feature in order to report on resources for multi
```xml
<!-- For the global reports designer. -->
<web_global>
<!-- keyed - The quota_key "key" is passed in the query parameter,
<!-- keyed – The quota_key "key" is passed in the query parameter,
and the quota is tracked separately for each key value.
For example, you can pass a Yandex.Metrica username as the key,
so the quota will be counted separately for each username.

66
docs/en/operations/server_settings/settings.md
View File

@ -22,7 +22,7 @@ Default value: 3600.
Data compression settings.
!!! warning "Warning"
!!! Important
Don't use it if you have just started using ClickHouse.
The configuration looks like this:
@ -64,7 +64,7 @@ ClickHouse checks ` min_part_size` and ` min_part_size_ratio` and processes th
The default database.
To get a list of databases, use the [SHOW DATABASES](../../query_language/misc.md#query_language_queries_show_databases).
To get a list of databases, use the [ SHOW DATABASES](../../query_language/misc.md#query_language_queries_show_databases) query.
**Example**
@ -176,7 +176,7 @@ You can configure multiple `<graphite>` clauses. For instance, you can use this
Settings for thinning data for Graphite.
For more information, see [GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#table_engines-graphitemergetree).
For more details, see [GraphiteMergeTree](../../operations/table_engines/graphitemergetree.md#table_engines-graphitemergetree).
**Example**
@ -308,7 +308,7 @@ Logging settings.
Keys:
- level – Logging level. Acceptable values: ``trace``, ``debug``, ``information``, ``warning``, ``error``.
- log – The log file. Contains all the entries according to `` level``.
- log – The log file. Contains all the entries according to `level`.
- errorlog – Error log file.
- size – Size of the file. Applies to ``log``and``errorlog``. Once the file reaches ``size``, ClickHouse archives and renames it, and creates a new log file in its place.
- count – The number of archived log files that ClickHouse stores.
@ -325,7 +325,8 @@ Keys:
</logger>
```
Also, logging to syslog is possible. Configuration example:
Writing to the syslog is also supported. Config example:
```xml
<logger>
<use_syslog>1</use_syslog>
@ -339,13 +340,14 @@ Also, logging to syslog is possible. Configuration example:
```
Keys:
- user_syslog - activation key, turning on syslog logging.
- address - host[:port] of syslogd. If not specified, local one would be used.
- hostname - optional, source host of logs
- facility - [syslog facility](https://en.wikipedia.org/wiki/Syslog#Facility),
in uppercase, prefixed with "LOG_": (``LOG_USER``, ``LOG_DAEMON``, ``LOG_LOCAL3`` etc.).
Default values: when ``address`` is specified, then ``LOG_USER``, otherwise - ``LOG_DAEMON``
- format - message format. Possible values are - ``bsd`` and ``syslog``
- user_syslog — Required setting if you want to write to the syslog.
- address — The host[:порт] of syslogd. If omitted, the local daemon is used.
- hostname — Optional. The name of the host that logs are sent from.
- facility — [The syslog facility keyword](https://en.wikipedia.org/wiki/Syslog#Facility)
in uppercase letters with the "LOG_" prefix: (``LOG_USER``, ``LOG_DAEMON``, ``LOG_LOCAL3``, and so on).
Default value: ``LOG_USER`` if ``address`` is specified, ``LOG_DAEMON otherwise.``
- format – Message format. Possible values: ``bsd`` and ``syslog.``
<a name="server_settings-macros"></a>
@ -367,7 +369,7 @@ For more information, see the section "[Creating replicated tables](../../operat
## mark_cache_size
Approximate size (in bytes) of the cache of "marks" used by [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) engines.
Approximate size (in bytes) of the cache of "marks" used by [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree).
The cache is shared for the server and memory is allocated as needed. The cache size must be at least 5368709120.
@ -423,7 +425,7 @@ We recommend using this option in Mac OS X, since the ` getrlimit()` function re
Restriction on deleting tables.
If the size of a [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) type table exceeds `max_table_size_to_drop` (in bytes), you can't delete it using a DROP query.
If the size of a [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) table exceeds `max_table_size_to_drop` (in bytes), you can't delete it using a DROP query.
If you still need to delete the table without restarting the ClickHouse server, create the `<clickhouse-path>/flags/force_drop_table` file and run the DROP query.
@ -441,7 +443,7 @@ The value 0 means that you can delete all tables without any restrictions.
## merge_tree
Fine tuning for tables in the [ MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) family.
Fine tuning for tables in the [ MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree).
For more information, see the MergeTreeSettings.h header file.
@ -459,25 +461,25 @@ For more information, see the MergeTreeSettings.h header file.
SSL client/server configuration.
Support for SSL is provided by the `` libpoco`` library. The interface is described in the file [SSLManager.h](https://github.com/ClickHouse-Extras/poco/blob/master/NetSSL_OpenSSL/include/Poco/Net/SSLManager.h)
Support for SSL is provided by the `libpoco` library. The interface is described in the file [SSLManager.h](https://github.com/ClickHouse-Extras/poco/blob/master/NetSSL_OpenSSL/include/Poco/Net/SSLManager.h)
Keys for server/client settings:
- privateKeyFile – The path to the file with the secret key of the PEM certificate. The file may contain a key and certificate at the same time.
- certificateFile – The path to the client/server certificate file in PEM format. You can omit it if `` privateKeyFile`` contains the certificate.
- certificateFile – The path to the client/server certificate file in PEM format. You can omit it if `privateKeyFile` contains the certificate.
- caConfig – The path to the file or directory that contains trusted root certificates.
- verificationMode – The method for checking the node's certificates. Details are in the description of the [Context](https://github.com/ClickHouse-Extras/poco/blob/master/NetSSL_OpenSSL/include/Poco/Net/Context.h) class. Possible values: ``none``, ``relaxed``, ``strict``, ``once``.
- verificationDepth – The maximum length of the verification chain. Verification will fail if the certificate chain length exceeds the set value.
- loadDefaultCAFile – Indicates that built-in CA certificates for OpenSSL will be used. Acceptable values: `` true``, `` false``. |
- cipherList – Supported OpenSSL encryptions. For example: `` ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH``.
- cacheSessions – Enables or disables caching sessions. Must be used in combination with ``sessionIdContext``. Acceptable values: `` true``, `` false``.
- loadDefaultCAFile – Indicates that built-in CA certificates for OpenSSL will be used. Acceptable values: `true`, `false`. |
- cipherList – Supported OpenSSL encryptions. For example: `ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH`.
- cacheSessions – Enables or disables caching sessions. Must be used in combination with ``sessionIdContext``. Acceptable values: `true`, `false`.
- sessionIdContext – A unique set of random characters that the server appends to each generated identifier. The length of the string must not exceed ``SSL_MAX_SSL_SESSION_ID_LENGTH``. This parameter is always recommended, since it helps avoid problems both if the server caches the session and if the client requested caching. Default value: ``${application.name}``.
- sessionCacheSize – The maximum number of sessions that the server caches. Default value: 1024\*20. 0 – Unlimited sessions.
- sessionTimeout – Time for caching the session on the server.
- extendedVerification – Automatically extended verification of certificates after the session ends. Acceptable values: `` true``, `` false``.
- requireTLSv1 – Require a TLSv1 connection. Acceptable values: `` true``, `` false``.
- requireTLSv1_1 – Require a TLSv1.1 connection. Acceptable values: `` true``, `` false``.
- requireTLSv1 – Require a TLSv1.2 connection. Acceptable values: `` true``, `` false``.
- extendedVerification – Automatically extended verification of certificates after the session ends. Acceptable values: `true`, `false`.
- requireTLSv1 – Require a TLSv1 connection. Acceptable values: `true`, `false`.
- requireTLSv1_1 – Require a TLSv1.1 connection. Acceptable values: `true`, `false`.
- requireTLSv1 – Require a TLSv1.2 connection. Acceptable values: `true`, `false`.
- fips – Activates OpenSSL FIPS mode. Supported if the library's OpenSSL version supports FIPS.
- privateKeyPassphraseHandler – Class (PrivateKeyPassphraseHandler subclass) that requests the passphrase for accessing the private key. For example: ``<privateKeyPassphraseHandler>``, ``<name>KeyFileHandler</name>``, ``<options><password>test</password></options>``, ``</privateKeyPassphraseHandler>``.
- invalidCertificateHandler – Class (subclass of CertificateHandler) for verifying invalid certificates. For example: `` <invalidCertificateHandler> <name>ConsoleCertificateHandler</name> </invalidCertificateHandler>`` .
@ -518,7 +520,7 @@ Keys for server/client settings:
## part_log
Logging events that are associated with [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) data. For instance, adding or merging data. You can use the log to simulate merge algorithms and compare their characteristics. You can visualize the merge process.
Logging events that are associated with [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree). For instance, adding or merging data. You can use the log to simulate merge algorithms and compare their characteristics. You can visualize the merge process.
Queries are logged in the ClickHouse table, not in a separate file.
@ -558,9 +560,8 @@ Use the following parameters to configure logging:
The path to the directory containing data.
!!! warning "Attention"
The trailing slash is mandatory.
!!! Note:
The end slash is mandatory.
**Example**
@ -646,8 +647,8 @@ Port for communicating with clients over the TCP protocol.
Path to temporary data for processing large queries.
!!! warning "Attention"
The trailing slash is mandatory.
!!! Note:
The end slash is mandatory.
**Example**
@ -659,7 +660,7 @@ Path to temporary data for processing large queries.
## uncompressed_cache_size
Cache size (in bytes) for uncompressed data used by table engines from the [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree) family.
Cache size (in bytes) for uncompressed data used by table engines from the [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree).
There is one shared cache for the server. Memory is allocated on demand. The cache is used if the option [use_uncompressed_cache](../settings/settings.md#settings-use_uncompressed_cache) is enabled.
@ -673,7 +674,7 @@ The uncompressed cache is advantageous for very short queries in individual case
## user_files_path
A catalog with user files. Used in a [file()](../../query_language/table_functions/file.md#table_functions-file) table function.
The directory with user files. Used in the table function [file()](../../query_language/table_functions/file.md#table_functions-file).
**Example**
@ -715,3 +716,4 @@ For more information, see the section "[Replication](../../operations/table_engi
```xml
<zookeeper incl="zookeeper-servers" optional="true" />
```

7
docs/en/operations/settings/index.md
View File

@ -7,17 +7,18 @@ Settings are configured in layers, so each subsequent layer redefines the previo
Ways to configure settings, in order of priority:
- Settings in the server config file `users.xml`.
- Settings in the `users.xml` server configuration file.
Set it in user profile in `<profiles>` element.
Set in the element `<profiles>`.
- Session settings.
Send ` SET setting=value` from the ClickHouse console client in interactive mode.
Similarly, you can use ClickHouse sessions in the HTTP protocol. To do this, you need to specify the `session_id` HTTP parameter.
- For a query.
- Query settings.
- When starting the ClickHouse console client in non-interactive mode, set the startup parameter `--setting=value`.
- When using the HTTP API, pass CGI parameters (`URL?setting_1=value&setting_2=value...`).
Settings that can only be made in the server config file are not covered in this section.

26
docs/en/operations/settings/settings.md
View File

@ -4,18 +4,24 @@
## distributed_product_mode
Changes the behavior of [distributed subqueries](../../query_language/select.md#queries-distributed-subrequests), i.e. in cases when the query contains the product of distributed tables.
Changes the behavior of [distributed subqueries](../../query_language/select.md#queries-distributed-subrequests).
ClickHouse applies the configuration if the subqueries on any level have a distributed table that exists on the local server and has more than one shard.
ClickHouse applies this setting when the query contains the product of distributed tables, i.e. when the query for a distributed table contains a non-GLOBAL subquery for the distributed table.
Restrictions:
- Only applied for IN and JOIN subqueries.
- Used only if a distributed table is used in the FROM clause.
- Not used for a table-valued [ remote](../../query_language/table_functions/remote.md#table_functions-remote) function.
- Only if the FROM section uses a distributed table containing more than one shard.
- If the subquery concerns a distributed table containing more than one shard,
- Not used for a table-valued [ remote](../../query_language/table_functions/remote.md#table_functions-remote).
The possible values ​​are:
- `deny` — Default value. Prohibits using these types of subqueries (returns the "Double-distributed in/JOIN subqueries is denied" exception).
- `local` — Replaces the database and table in the subquery with local ones for the destination server (shard), leaving the normal `IN` / `JOIN.`
- `global` — Replaces the `IN` / `JOIN` query with `GLOBAL IN` / `GLOBAL JOIN.`
- `allow` — Allows the use of these types of subqueries.
<a name="settings-settings-fallback_to_stale_replicas_for_distributed_queries"></a>
## fallback_to_stale_replicas_for_distributed_queries
@ -158,7 +164,7 @@ Don't confuse blocks for compression (a chunk of memory consisting of bytes) and
## min_compress_block_size
For [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree)" tables. In order to reduce latency when processing queries, a block is compressed when writing the next mark if its size is at least 'min_compress_block_size'. By default, 65,536.
For [MergeTree](../../operations/table_engines/mergetree.md#table_engines-mergetree)". In order to reduce latency when processing queries, a block is compressed when writing the next mark if its size is at least 'min_compress_block_size'. By default, 65,536.
The actual size of the block, if the uncompressed data is less than 'max_compress_block_size', is no less than this value and no less than the volume of data for one mark.
@ -343,4 +349,12 @@ If the value is true, integers appear in quotes when using JSON\* Int64 and UInt
## format_csv_delimiter
The character to be considered as a delimiter in CSV data. By default, `,`.
The character interpreted as a delimiter in the CSV data. By default, the delimiter is `,`.
<!--a name="settings-join_use_nulls"></a-->
## join_use_nulls {: #settings-join_use_nulls}
Affects the behavior of [JOIN](../../query_language/select.md#query_language-join).
With `join_use_nulls=1`, `JOIN` behaves like in standard SQL, i.e. if empty cells appear when merging, the type of the corresponding field is converted to [Nullable](../../data_types/nullable.md#data_type-nullable), and empty cells are filled with [NULL](../../query_language/syntax.md#null-literal).

7
docs/en/operations/settings/settings_profiles.md
View File

@ -1,11 +1,13 @@
<a name="settings_profiles"></a>
# Settings profiles
A settings profile is a collection of settings grouped under the same name. Each ClickHouse user has a profile.
To apply all the settings in a profile, set `profile`.
To apply all the settings in a profile, set the `profile` setting.
Example:
Setting `web` profile.
Install the `web` profile.
```sql
SET profile = 'web'
@ -60,3 +62,4 @@ Example:
The example specifies two profiles: `default` and `web`. The `default` profile has a special purpose: it must always be present and is applied when starting the server. In other words, the `default` profile contains default settings. The `web` profile is a regular profile that can be set using the `SET` query or using a URL parameter in an HTTP query.
Settings profiles can inherit from each other. To use inheritance, indicate the `profile` setting before the other settings that are listed in the profile.

189
docs/en/operations/system_tables.md
View File

@ -5,7 +5,6 @@ You can't delete a system table (but you can perform DETACH).
System tables don't have files with data on the disk or files with metadata. The server creates all the system tables when it starts.
System tables are read-only.
They are located in the 'system' database.
<a name="system_tables-system.asynchronous_metrics"></a>
## system.asynchronous_metrics
@ -20,27 +19,28 @@ Contains information about clusters available in the config file and the servers
Columns:
```text
cluster String – Cluster name.
shard_num UInt32 – Number of a shard in the cluster, starting from 1.
shard_weight UInt32 – Relative weight of a shard when writing data.
replica_num UInt32 – Number of a replica in the shard, starting from 1.
host_name String – Host name as specified in the config.
host_address String – Host's IP address obtained from DNS.
port UInt16 – The port used to access the server.
user String – The username to use for connecting to the server.
cluster String — The cluster name.
shard_num UInt32 — The shard number in the cluster, starting from 1.
shard_weight UInt32 — The relative weight of the shard when writing data.
replica_num UInt32 — The replica number in the shard, starting from 1.
host_name String — The host name, as specified in the config.
String host_address — The host IP address obtained from DNS.
port UInt16 — The port to use for connecting to the server.
user String — The name of the user for connecting to the server.
```
## system.columns
Contains information about the columns in all tables.
You can use this table to get information similar to `DESCRIBE TABLE`, but for multiple tables at once.
```text
database String - Name of the database the table is located in.
table String - Table name.
name String - Column name.
type String - Column type.
default_type String - Expression type (DEFAULT, MATERIALIZED, ALIAS) for the default value, or an empty string if it is not defined.
default_expression String - Expression for the default value, or an empty string if it is not defined.
database String — The name of the database the table is in.
table String – Table name.
name String — Column name.
type String — Column type.
default_type String — Expression type (DEFAULT, MATERIALIZED, ALIAS) for the default value, or an empty string if it is not defined.
default_expression String — Expression for the default value, or an empty string if it is not defined.
```
## system.databases
@ -55,19 +55,19 @@ Contains information about external dictionaries.
Columns:
- `name String` – Dictionary name.
- `type String` – Dictionary type: Flat, Hashed, Cache.
- `origin String` – Path to the config file where the dictionary is described.
- `attribute.names Array(String)` – Array of attribute names provided by the dictionary.
- `attribute.types Array(String)` – Corresponding array of attribute types provided by the dictionary.
- `has_hierarchy UInt8` – Whether the dictionary is hierarchical.
- `bytes_allocated UInt64` – The amount of RAM used by the dictionary.
- `hit_rate Float64` – For cache dictionaries, the percent of usage for which the value was in the cache.
- `element_count UInt64` – The number of items stored in the dictionary.
- `load_factor Float64` – The filled percentage of the dictionary (for a hashed dictionary, it is the filled percentage of the hash table).
- `creation_time DateTime` – Time spent for the creation or last successful reload of the dictionary.
- `last_exception String` – Text of an error that occurred when creating or reloading the dictionary, if the dictionary couldn't be created.
- `source String` – Text describing the data source for the dictionary.
- `name String` — Dictionary name.
- `type String` — Dictionary type: Flat, Hashed, Cache.
- `origin String` — Path to the configuration file that describes the dictionary.
- `attribute.names Array(String)` — Array of attribute names provided by the dictionary.
- `attribute.types Array(String)` — Corresponding array of attribute types that are provided by the dictionary.
- `has_hierarchy UInt8` — Whether the dictionary is hierarchical.
- `bytes_allocated UInt64` — The amount of RAM the dictionary uses.
- `hit_rate Float64` — For cache dictionaries, the percentage of uses for which the value was in the cache.
- `element_count UInt64` — The number of items stored in the dictionary.
- `load_factor Float64` — The percentage full of the dictionary (for a hashed dictionary, the percentage filled in the hash table).
- `creation_time DateTime` — The time when the dictionary was created or last successfully reloaded.
- `last_exception String` — Text of the error that occurs when creating or reloading the dictionary if the dictionary couldn't be created.
- `source String` — Text describing the data source for the dictionary.
Note that the amount of memory used by the dictionary is not proportional to the number of items stored in it. So for flat and cached dictionaries, all the memory cells are pre-assigned, regardless of how full the dictionary actually is.
<a name="system_tables-system.events"></a>
@ -84,26 +84,27 @@ Contains information about normal and aggregate functions.
Columns:
- `name` (`String`) – Function name.
- `is_aggregate` (`UInt8`) – Whether it is an aggregate function.
- `name`(`String`) – The name of the function.
- `is_aggregate`(`UInt8`) — Whether the function is aggregate.
## system.merges
Contains information about merges currently in process for tables in the MergeTree family.
Columns:
- `database String` — Name of the database the table is located in.
- `table String` — Name of the table.
- `elapsed Float64` — Time in seconds since the merge started.
- `progress Float64` — Percent of progress made, from 0 to 1.
- `num_parts UInt64` — Number of parts to merge.
- `result_part_name String` — Name of the part that will be formed as the result of the merge.
- `total_size_bytes_compressed UInt64` — Total size of compressed data in the parts being merged.
- `total_size_marks UInt64` — Total number of marks in the parts being merged.
- `bytes_read_uncompressed UInt64` — Amount of bytes read, decompressed.
- `database String` — The name of the database the table is in.
- `table String` — Table name.
- `elapsed Float64` — The time elapsed (in seconds) since the merge started.
- `progress Float64` — The percentage of completed work from 0 to 1.
- `num_parts UInt64` — The number of pieces to be merged.
- `result_part_name String` — The name of the part that will be formed as the result of merging.
- `total_size_bytes_compressed UInt64` — The total size of the compressed data in the merged chunks.
- `total_size_marks UInt64` — The total number of marks in the merged partss.
- `bytes_read_uncompressed UInt64` — Number of bytes read, uncompressed.
- `rows_read UInt64` — Number of rows read.
- `bytes_written_uncompressed UInt64` — Amount of bytes written, uncompressed.
- `rows_written UInt64` — Number of rows written.
- `bytes_written_uncompressed UInt64` — Number of bytes written, uncompressed.
- `rows_written UInt64` — Number of lines rows written.
<a name="system_tables-system.metrics"></a>
## system.metrics
@ -127,31 +128,54 @@ This is similar to the DUAL table found in other DBMSs.
## system.parts
Contains information about parts of a table in the [MergeTree](../operations/table_engines/mergetree.md#table_engines-mergetree) family.
Contains information about parts of [MergeTree](table_engines/mergetree.md#table_engines-mergetree) tables.
Each row describes one part of the data.
Columns:
- partition (String) – The partition name. It's in YYYYMM format in case of old-style partitioning and is arbitary serialized value in case of custom partitioning. To learn what a partition is, see the description of the [ALTER](../query_language/alter.md#query_language_queries_alter) query.
- partition (String) – The partition name. To learn what a partition is, see the description of the [ALTER](../query_language/alter.md#query_language_queries_alter) query.
Formats:
- `YYYYMM` for automatic partitioning by month.
- `any_string` when partitioning manually.
- name (String) – Name of the data part.
- active (UInt8) – Indicates whether the part is active. If a part is active, it is used in a table; otherwise, it will be deleted. Inactive data parts remain after merging.
- marks (UInt64) – The number of marks. To get the approximate number of rows in a data part, multiply ``marks`` by the index granularity (usually 8192).
- marks_size (UInt64) – The size of the file with marks.
- rows (UInt64) – The number of rows.
- bytes (UInt64) – The number of bytes when compressed.
- modification_time (DateTime) – The modification time of the directory with the data part. This usually corresponds to the time of data part creation.|
- remove_time (DateTime) – The time when the data part became inactive.
- refcount (UInt32) – The number of places where the data part is used. A value greater than 2 indicates that the data part is used in queries or merges.
- min_date (Date) – The minimum value of the date key in the data part.
- max_date (Date) – The maximum value of the date key in the data part.
- min_block_number (UInt64) – The minimum number of data parts that make up the current part after merging.
- max_block_number (UInt64) – The maximum number of data parts that make up the current part after merging.
- level (UInt32) – Depth of the merge tree. If a merge was not performed, ``level=0``.
- primary_key_bytes_in_memory (UInt64) – The amount of memory (in bytes) used by primary key values.
- primary_key_bytes_in_memory_allocated (UInt64) – The amount of memory (in bytes) reserved for primary key values.
- database (String) – Name of the database.
- table (String) – Name of the table.
- engine (String) – Name of the table engine without parameters.
## system.processes
@ -162,21 +186,21 @@ Columns:
```text
user String – Name of the user who made the request. For distributed query processing, this is the user who helped the requestor server send the query to this server, not the user who made the distributed request on the requestor server.
address String – The IP address that the query was made from. The same is true for distributed query processing.
address String - The IP address the request was made from. The same for distributed processing.
elapsed Float64 – The time in seconds since request execution started.
elapsed Float64 - The time in seconds since request execution started.
rows_read UInt64 – The number of rows read from the table. For distributed processing, on the requestor server, this is the total for all remote servers.
rows_read UInt64 - The number of rows read from the table. For distributed processing, on the requestor server, this is the total for all remote servers.
bytes_read UInt64 – The number of uncompressed bytes read from the table. For distributed processing, on the requestor server, this is the total for all remote servers.
bytes_read UInt64 - The number of uncompressed bytes read from the table. For distributed processing, on the requestor server, this is the total for all remote servers.
UInt64 total_rows_approx – The approximate total number of rows that must be read. For distributed processing, on the requestor server, this is the total for all remote servers. It can be updated during request processing, when new sources to process become known.
total_rows_approx UInt64 - The approximation of the total number of rows that should be read. For distributed processing, on the requestor server, this is the total for all remote servers. It can be updated during request processing, when new sources to process become known.
memory_usage UInt64 – Memory consumption by the query. It might not include some types of dedicated memory.
memory_usage UInt64 - How much memory the request uses. It might not include some types of dedicated memory.
query String – The query text. For INSERT, it doesn't include the data to insert.
query String - The query text. For INSERT, it doesn't include the data to insert.
query_id – Query ID, if defined.
query_id String - Query ID, if defined.
```
## system.replicas
@ -220,54 +244,54 @@ active_replicas: 2
Columns:
```text
database: database name
table: table name
engine: table engine name
database: Database name
table: Table name
engine: Table engine name
is_leader: whether the replica is the leader
is_leader: Whether the replica is the leader.
Only one replica at a time can be the leader. The leader is responsible for selecting background merges to perform.
Note that writes can be performed to any replica that is available and has a session in ZK, regardless of whether it is a leader.
is_readonly: Whether the replica is in read-only mode.
This mode is turned on if the config doesn't have sections with ZK, if an unknown error occurred when reinitializing sessions in ZK, and during session reinitialization in ZK.
This mode is turned on if the config doesn't have sections with ZooKeeper, if an unknown error occurred when reinitializing sessions in ZooKeeper, and during session reinitialization in ZooKeeper.
is_session_expired: Whether the ZK session expired.
Basically, the same thing as is_readonly.
is_session_expired: Whether the session with ZooKeeper has expired.
Basically the same as 'is_readonly'.
future_parts: The number of data parts that will appear as the result of INSERTs or merges that haven't been done yet.
parts_to_check: The number of data parts in the queue for verification.
A part is put in the verification queue if there is suspicion that it might be damaged.
zookeeper_path: The path to the table data in ZK.
replica_name: Name of the replica in ZK. Different replicas of the same table have different names.
replica_path: The path to the replica data in ZK. The same as concatenating zookeeper_path/replicas/replica_path.
zookeeper_path: Path to table data in ZooKeeper.
replica_name: Replica name in ZooKeeper. Different replicas of the same table have different names.
replica_path: Path to replica data in ZooKeeper. The same as concatenating 'zookeeper_path/replicas/replica_path'.
columns_version: Version number of the table structure.
Indicates how many times ALTER was performed. If replicas have different versions, it means some replicas haven't made all of the ALTERs yet.
queue_size: Size of the queue for operations waiting to be performed.
Operations include inserting blocks of data, merges, and certain other actions.
Normally coincides with future_parts.
It usually coincides with 'future_parts'.
inserts_in_queue: Number of inserts of blocks of data that need to be made.
Insertions are usually replicated fairly quickly. If the number is high, something is wrong.
Insertions are usually replicated fairly quickly. If this number is large, it means something is wrong.
merges_in_queue: The number of merges waiting to be made.
Sometimes merges are lengthy, so this value may be greater than zero for a long time.
The next 4 columns have a non-null value only if the ZK session is active.
The next 4 columns have a non-zero value only where there is an active session with ZK.
log_max_index: Maximum entry number in the log of general activity.
log_pointer: Maximum entry number in the log of general activity that the replica copied to its execution queue, plus one.
If log_pointer is much smaller than log_max_index, something is wrong.
total_replicas: Total number of known replicas of this table.
active_replicas: Number of replicas of this table that have a ZK session (the number of active replicas).
total_replicas: The total number of known replicas of this table.
active_replicas: The number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas).
```
If you request all the columns, the table may work a bit slowly, since several reads from ZK are made for each row.
If you request all the columns, the table may work a bit slowly, since several reads from ZooKeeper are made for each row.
If you don't request the last 4 columns (log_max_index, log_pointer, total_replicas, active_replicas), the table works quickly.
For example, you can check that everything is working correctly like this:
@ -307,14 +331,14 @@ If this query doesn't return anything, it means that everything is fine.
## system.settings
Contains information about settings that are currently in use.
I.e. used for executing the query you are using to read from the system.settings table).
I.e. used for executing the query you are using to read from the system.settings table.
Columns:
```text
name String – Setting name.
value String – Setting value.
changed UInt8 - Whether the setting was explicitly defined in the config or explicitly changed.
name String — Setting name.
value String — Setting value.
changed UInt8 — Whether the setting was explicitly defined in the config or explicitly changed.
```
Example:
@ -343,7 +367,7 @@ This system table is used for implementing SHOW TABLES queries.
## system.zookeeper
This table presents when ZooKeeper is configured. It allows reading data from the ZooKeeper cluster defined in the config.
The table does not exist if ZooKeeper is not configured. Allows reading data from the ZooKeeper cluster defined in the config.
The query must have a 'path' equality condition in the WHERE clause. This is the path in ZooKeeper for the children that you want to get data for.
The query `SELECT * FROM system.zookeeper WHERE path = '/clickhouse'` outputs data for all children on the `/clickhouse` node.
@ -352,21 +376,20 @@ If the path specified in 'path' doesn't exist, an exception will be thrown.
Columns:
- `name String` — Name of the node.
- `path String` — Path to the node.
- `value String` — Value of the node.
- `name String` — The name of the node.
- `path String` — The path to the node.
- `value String` — Node value.
- `dataLength Int32` — Size of the value.
- `numChildren Int32` — Number of children.
- `numChildren Int32` — Number of descendants.
- `czxid Int64` — ID of the transaction that created the node.
- `mzxid Int64` — ID of the transaction that last changed the node.
- `pzxid Int64` — ID of the transaction that last added or removed children.
- `pzxid Int64` — ID of the transaction that last deleted or added descendants.
- `ctime DateTime` — Time of node creation.
- `mtime DateTime` — Time of the last node modification.
- `version Int32` — Node version - the number of times the node was changed.
- `cversion Int32` — Number of added or removed children.
- `aversion Int32` — Number of changes to ACL.
- `ephemeralOwner Int64` — For ephemeral nodes, the ID of the session that owns this node.
- `mtime DateTime` — Time of the last modification of the node.
- `version Int32` — Node version: the number of times the node was changed.
- `cversion Int32` — Number of added or removed descendants.
- `aversion Int32` — Number of changes to the ACL.
- `ephemeralOwner Int64` — For ephemeral nodes, the ID of hte session that owns this node.
Example:

7
docs/en/operations/table_engines/aggregatingmergetree.md
View File

@ -21,15 +21,14 @@ This type of column stores the state of an aggregate function.
To get this type of value, use aggregate functions with the `State` suffix.
Example:
`uniqState(UserID), quantilesState(0.5, 0.9)(SendTiming)`
Example:`uniqState(UserID), quantilesState(0.5, 0.9)(SendTiming)`
In contrast to the corresponding `uniq` and `quantiles` functions, these functions return the state, rather than the prepared value. In other words, they return an `AggregateFunction` type value.
An `AggregateFunction` type value can't be output in Pretty formats. In other formats, these types of values are output as implementation-specific binary data. The `AggregateFunction` type values are not intended for output or saving in a dump.
The only useful thing you can do with `AggregateFunction` type values is combine the states and get a result, which essentially means to finish aggregation. Aggregate functions with the 'Merge' suffix are used for this purpose.
Example: `uniqMerge(UserIDState), where UserIDState has the AggregateFunction` type.
The only useful thing you can do with `AggregateFunction` type values is to combine the states and get a result, which essentially means to finish aggregation. Aggregate functions with the 'Merge' suffix are used for this purpose.
Example: `uniqMerge(UserIDState)`, where `UserIDState` has the `AggregateFunction` type.
In other words, an aggregate function with the 'Merge' suffix takes a set of states, combines them, and returns the result.
As an example, these two queries return the same result:

2
docs/en/operations/table_engines/custom_partitioning_key.md
View File

@ -37,7 +37,7 @@ Note: For old-style tables, the partition can be specified either as a number `2
In the `system.parts` table, the `partition` column specifies the value of the partition expression to use in ALTER queries (if quotas are removed). The `name` column should specify the name of the data part that has a new format.
Was: `20140317_20140323_2_2_0` (minimum date - maximum date - minimum block number - maximum block number - level).
Old: `20140317_20140323_2_2_0` (minimum date - maximum date - minimum block number - maximum block number - level).
Now: `201403_2_2_0` (partition ID - minimum block number - maximum block number - level).

2
docs/en/operations/table_engines/dictionary.md
View File

@ -83,6 +83,7 @@ CREATE TABLE products
)
ENGINE = Dictionary(products)
```
```
Ok.
@ -106,3 +107,4 @@ LIMIT 1
1 rows in set. Elapsed: 0.006 sec.
```

45
docs/en/operations/table_engines/file.md
View File

@ -1,43 +1,43 @@
<a name="table_engines-file"></a>
# File(InputFormat)
# File(Format)
The data source is a file that stores data in one of the supported input formats (TabSeparated, Native, etc.).
Manages data in a single file on disk in the specified format.
Usage examples:
- Data export from ClickHouse to file.
- Convert data from one format to another.
- Updating data in ClickHouse via editing a file on a disk.
- Downloading data from ClickHouse to a file.
- Converting data from one format to another.
- Updating data in ClickHouse by editing the file on disk.
## Usage in ClickHouse Server
## Using the engine in the ClickHouse server
```
File(Format)
```
`Format` should be supported for either `INSERT` and `SELECT`. For the full list of supported formats see [Formats](../../interfaces/formats.md#formats).
`The format` must be one that ClickHouse can use both in `INSERT` queries and in `SELECT` queries. For the full list of supported formats, see [Formats](../../interfaces/formats.md#formats).
ClickHouse does not allow to specify filesystem path for`File`. It will use folder defined by [path](../server_settings/settings.md#server_settings-path) setting in server configuration.
The ClickHouse server does not allow you to specify the path to the file tthat `File` will work with. It uses the path to the storage that is specified by the [path](../server_settings/settings.md#server_settings-path) parameter in the server configuration.
When creating table using `File(Format)` it creates empty subdirectory in that folder. When data is written to that table, it's put into `data.Format` file in that subdirectory.
When creating a table using `File(Format)`, the ClickHouse server creates a directory with the name of the table in the storage, and puts the `data.Format` file there after it is added to the data table.
You may manually create this subfolder and file in server filesystem and then [ATTACH](../../query_language/misc.md#queries-attach) it to table information with matching name, so you can query data from that file.
You can manually create the table's directory in storage, put the file there, and then use [ATTACH](../../query_language/misc.md#queries-attach) the ClickHouse server to add information about the table corresponding to the directory name and read data from the file.
!!! warning
Be careful with this funcionality, because ClickHouse does not keep track of external changes to such files. The result of simultaneous writes via ClickHouse and outside of ClickHouse is undefined.
!!! Warning:
Be careful with this functionality, because the ClickHouse server does not track external data changes. If the file will be written to simultaneously from the ClickHouse server and from an external source, the result is unpredictable.
**Example:**
**1.** Set up the `file_engine_table` table:
**1.** Create a `file_engine_table` table on the server :
```sql
CREATE TABLE file_engine_table (name String, value UInt32) ENGINE=File(TabSeparated)
```
By default ClickHouse will create folder `/var/lib/clickhouse/data/default/file_engine_table`.
In the default configuration, the ClickHouse server creates the directory `/var/lib/clickhouse/data/default/file_engine_table`.
**2.** Manually create `/var/lib/clickhouse/data/default/file_engine_table/data.TabSeparated` containing:
**2.** Manually create the file `/var/lib/clickhouse/data/default/file_engine_table/data.TabSeparated` with the contents:
```bash
$cat data.TabSeparated
@ -45,7 +45,7 @@ one 1
two 2
```
**3.** Query the data:
**3.** Request data:
```sql
SELECT * FROM file_engine_table
@ -58,9 +58,9 @@ SELECT * FROM file_engine_table
└──────┴───────┘
```
## Usage in Clickhouse-local
## Using the engine in clickhouse-local
In [clickhouse-local](../utils/clickhouse-local.md#utils-clickhouse-local) File engine accepts file path in addition to `Format`. Default input/output streams can be specified using numeric or human-readable names like `0` or `stdin`, `1` or `stdout`.
In [clickhouse-local](../utils/clickhouse-local.md#utils-clickhouse-local) the engine takes the file path as a parameter, as well as the format. The standard input/output streams can be specified using numbers or letters: `0` or `stdin`, `1` or `stdout`.
**Example:**
@ -70,9 +70,8 @@ $ echo -e "1,2\n3,4" | clickhouse-local -q "CREATE TABLE table (a Int64, b Int64
## Details of Implementation
- Reads can be parallel, but not writes
- Multi-stream reading and single-stream writing are supported.
- Not supported:
- `ALTER`
- `SELECT ... SAMPLE`
- Indices
- Replication
- `ALTER` and `SELECT...SAMPLE` operations.
- Indexes.
- Replication.

12
docs/en/operations/table_engines/index.md
View File

@ -1,14 +1,16 @@
# Table Engines
<a name="table_engines"></a>
# Table engines
The table engine (type of table) determines:
- How and where data is stored: where to write it to, and where to read it from.
- How and where data is stored, where to write it to, and where to read it from.
- Which queries are supported, and how.
- Concurrent data access.
- Use of indexes, if present.
- Whether multithreaded request execution is possible.
- Data replication.
- Data replication parameters.
When reading data, the engine is only required to extract the necessary set of columns. However, in some cases, the query may be partially processed inside the table engine.
When reading, the engine is only required to output the requested columns, but in some cases the engine can partially process data when responding to the request.
Note that for most serious tasks, you should use engines from the `MergeTree` family.
For most serious tasks, you should use engines from the `MergeTree` family.

5
docs/en/operations/table_engines/kafka.md
View File

@ -115,11 +115,11 @@ To stop receiving topic data or to change the conversion logic, detach the mater
ATTACH MATERIALIZED VIEW consumer;
```
If you want to change the target table by using ` ALTER`materialized view, we recommend disabling the material view to avoid discrepancies between the target table and the data from the view.
If you want to change the target table by using `ALTER`, we recommend disabling the material view to avoid discrepancies between the target table and the data from the view.
## Configuration
Similar to GraphiteMergeTree, the Kafka engine supports extended configuration using the ClickHouse config file. There are two configuration keys that you can use: global (`kafka`) and topic-level (`kafka_topic_*`). The global configuration is applied first, and the topic-level configuration is second (if it exists).
Similar to GraphiteMergeTree, the Kafka engine supports extended configuration using the ClickHouse config file. There are two configuration keys that you can use: global (`kafka`) and topic-level (`kafka_topic_*`). The global configuration is applied first, and then the topic-level configuration is applied (if it exists).
```xml
<!-- Global configuration options for all tables of Kafka engine type -->
@ -136,4 +136,3 @@ Similar to GraphiteMergeTree, the Kafka engine supports extended configuration u
```
For a list of possible configuration options, see the [librdkafka configuration reference](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). Use the underscore (`_`) instead of a dot in the ClickHouse configuration. For example, `check.crcs=true` will be `<check_crcs>true</check_crcs>`.

2
docs/en/operations/table_engines/materializedview.md
View File

@ -1,4 +1,4 @@
# MaterializedView
Used for implementing materialized views (for more information, see the [CREATE TABLE](../../query_language/create.md#query_language-queries-create_table)) query. For storing data, it uses a different engine that was specified when creating the view. When reading from a table, it just uses this engine.
Used for implementing materialized views (for more information, see [CREATE TABLE](../../query_language/create.md#query_language-queries-create_table)). For storing data, it uses a different engine that was specified when creating the view. When reading from a table, it just uses this engine.

23
docs/en/operations/table_engines/merge.md
View File

@ -1,8 +1,8 @@
# Merge
The Merge engine (not to be confused with `MergeTree`) does not store data itself, but allows reading from any number of other tables simultaneously.
The `Merge` engine (not to be confused with `MergeTree`) does not store data itself, but allows reading from any number of other tables simultaneously.
Reading is automatically parallelized. Writing to a table is not supported. When reading, the indexes of tables that are actually being read are used, if they exist.
The Merge engine accepts parameters: the database name and a regular expression for tables.
The `Merge` engine accepts parameters: the database name and a regular expression for tables.
Example:
@ -10,30 +10,31 @@ Example:
Merge(hits, '^WatchLog')
```
Data will be read from the tables in the 'hits' database that have names that match the regular expression '`^WatchLog`'.
Data will be read from the tables in the `hits` database that have names that match the regular expression '`^WatchLog`'.
Instead of the database name, you can use a constant expression that returns a string. For example, `currentDatabase()`.
Regular expressions — [re2](https://github.com/google/re2) (supports a subset of PCRE), case-sensitive.
See the notes about escaping symbols in regular expressions in the "match" section.
When selecting tables to read, the Merge table itself will not be selected, even if it matches the regex. This is to avoid loops.
It is possible to create two Merge tables that will endlessly try to read each others' data, but this is not a good idea.
When selecting tables to read, the `Merge` table itself will not be selected, even if it matches the regex. This is to avoid loops.
It is possible to create two `Merge` tables that will endlessly try to read each others' data, but this is not a good idea.
The typical way to use the Merge engine is for working with a large number of TinyLog tables as if with a single table.
The typical way to use the `Merge` engine is for working with a large number of `TinyLog` tables as if with a single table.
## Virtual Columns
Virtual columns are columns that are provided by the table engine, regardless of the table definition. In other words, these columns are not specified in CREATE TABLE, but they are accessible for SELECT.
Virtual columns are columns that are provided by the table engine, regardless of the table definition. In other words, these columns are not specified in `CREATE TABLE`, but they are accessible for `SELECT`.
Virtual columns differ from normal columns in the following ways:
- They are not specified in table definitions.
- Data can't be added to them with INSERT.
- When using INSERT without specifying the list of columns, virtual columns are ignored.
- Data can't be added to them with `INSERT`.
- When using `INSERT` without specifying the list of columns, virtual columns are ignored.
- They are not selected when using the asterisk (`SELECT *`).
- Virtual columns are not shown in `SHOW CREATE TABLE` and `DESC TABLE` queries.
A Merge type table contains a virtual _table column with the String type. (If the table already has a _table column, the virtual column is named _table1, and if it already has _table1, it is named _table2, and so on.) It contains the name of the table that data was read from.
The `Merge` type table contains a virtual `_table` column of the `String` type. (If the table already has a `_table` column, the virtual column is called `_table1`; if you already have `_table1`, it's called `_table2`, and so on.) It contains the name of the table that data was read from.
If the `WHERE/PREWHERE` clause contains conditions for the `_table` column that do not depend on other table columns (as one of the conjunction elements, or as an entire expression), these conditions are used as an index. The conditions are performed on a data set of table names to read data from, and the read operation will be performed from only those tables that the condition was triggered on.
If the WHERE or PREWHERE clause contains conditions for the '_table' column that do not depend on other table columns (as one of the conjunction elements, or as an entire expression), these conditions are used as an index. The conditions are performed on a data set of table names to read data from, and the read operation will be performed from only those tables that the condition was triggered on.

3
docs/en/operations/table_engines/mergetree.md
View File

@ -56,7 +56,7 @@ In this example, the index can't be used.
SELECT count() FROM table WHERE CounterID = 34 OR URL LIKE '%upyachka%'
```
To check whether ClickHouse can use the index when executing the query, use the settings [force_index_by_date](../settings/settings.md#settings-settings-force_index_by_date)and[force_primary_key](../settings/settings.md#settings-settings-force_primary_key).
To check whether ClickHouse can use the index when running a query, use the settings [force_index_by_date](../settings/settings.md#settings-settings-force_index_by_date) and [force_primary_key](../settings/settings.md#settings-settings-force_primary_key).
The index by date only allows reading those parts that contain dates from the desired range. However, a data part may contain data for many dates (up to an entire month), while within a single part the data is ordered by the primary key, which might not contain the date as the first column. Because of this, using a query with only a date condition that does not specify the primary key prefix will cause more data to be read than for a single date.
@ -69,4 +69,3 @@ The `OPTIMIZE` query is supported, which calls an extra merge step.
You can use a single large table and continually add data to it in small chunks – this is what MergeTree is intended for.
Data replication is possible for all types of tables in the MergeTree family (see the section "Data replication").

25
docs/en/operations/table_engines/mysql.md
View File

@ -2,18 +2,27 @@
# MySQL
The MySQL engine allows you to perform SELECT queries on data that is stored on a remote MySQL server.
The MySQL engine allows you to perform `SELECT` queries on data that is stored on a remote MySQL server.
The engine takes 5-7 parameters: the server address (host and port); the name of the database; the name of the table; the user's name; the user's password; whether to use replace query; the on duplcate clause. Example:
Call format:
```text
```
MySQL('host:port', 'database', 'table', 'user', 'password'[, replace_query, 'on_duplicate_clause']);
```
At this time, simple WHERE clauses such as ```=, !=, >, >=, <, <=``` are executed on the MySQL server.
**Call parameters**
The rest of the conditions and the LIMIT sampling constraint are executed in ClickHouse only after the query to MySQL finishes.
- `host:port` — Address of the MySQL server.
- `database` — Database name on the MySQL server.
- `table` — Name of the table.
- `user` — The MySQL User.
- `password` — User password.
- `replace_query` — Flag that sets query substitution `INSERT INTO` to `REPLACE INTO`. If `replace_query=1`, the query is replaced.
- `'on_duplicate_clause'` — Adds the `ON DUPLICATE KEY UPDATE 'on_duplicate_clause'` expression to the `INSERT` query. For example: `impression = VALUES(impression) + impression`. To specify `'on_duplicate_clause'` you need to pass `0` to the `replace_query` parameter. If you simultaneously pass `replace_query = 1` and `'on_duplicate_clause'`, ClickHouse generates an exception.
At this time, simple `WHERE` clauses such as ` =, !=, >, >=, <, <=` are executed on the MySQL server.
The rest of the conditions and the `LIMIT` sampling constraint are executed in ClickHouse only after the query to MySQL finishes.
The `MySQL` engine does not support the [Nullable](../../data_types/nullable.md#data_type-nullable) data type, so when reading data from MySQL tables, `NULL` is converted to default values for the specified column type (usually 0 or an empty string).
If `replace_query` is specified to 1, then `INSERT INTO` query to this table would be transformed to `REPLACE INTO`.
If `on_duplicate_clause` is specified, eg `update impression = values(impression) + impression`, it would add `on_duplicate_clause` to the end of the MySQL insert sql.
Notice that only one of 'replace_query' and 'on_duplicate_clause' can be specified, or none of them.

5
docs/en/operations/table_engines/replication.md
View File

@ -15,7 +15,7 @@ Replication works at the level of an individual table, not the entire server. A
Replication does not depend on sharding. Each shard has its own independent replication.
Compressed data is replicated for `INSERT` and `ALTER` queries (see the description of the [ALTER](../../query_language/alter.md#query_language_queries_alter) query).
Compressed data for `INSERT` and `ALTER` queries is replicated (for more information, see the documentation for [ALTER](../../query_language/alter.md#query_language_queries_alter)).
`CREATE`, `DROP`, `ATTACH`, `DETACH` and `RENAME` queries are executed on a single server and are not replicated:
@ -178,6 +178,3 @@ After this, you can launch the server, create a `MergeTree` table, move the data
## Recovery When Metadata in The ZooKeeper Cluster is Lost or Damaged
If the data in ZooKeeper was lost or damaged, you can save data by moving it to an unreplicated table as described above.
If exactly the same parts exist on the other replicas, they are added to the working set on them. If not, the parts are downloaded from the replica that has them.

2
docs/en/operations/table_engines/summingmergetree.md
View File

@ -16,7 +16,7 @@ The columns to total are set explicitly (the last parameter – Shows, Clicks, C
If the values were zero in all of these columns, the row is deleted.
For the other columns that are not part of the primary key, the first value that occurs is selected when merging. But if a column is of AggregateFunction type, then it is merged according to that function, which effectively makes this engine behave like `AggregatingMergeTree`.
For the other columns that are not part of the primary key, the first value that occurs is selected when merging. But for the AggregateFunction type of columns, aggregation is performed according to the set function, so this engine actually behaves like `AggregatingMergeTree`.
Summation is not performed for a read operation. If it is necessary, write the appropriate GROUP BY.

46
docs/en/operations/table_engines/url.md
View File

@ -2,36 +2,34 @@
# URL(URL, Format)
This data source operates with data on remote HTTP/HTTPS server. The engine is
similar to [`File`](./file.md#).
Manages data on a remote HTTP/HTTPS server. This engine is similar
to the [`File`](./file.md#) engine.
## Usage in ClickHouse Server
## Using the engine in the ClickHouse server
```
URL(URL, Format)
```
`The format` must be one that ClickHouse can use in
`SELECT` queries and, if necessary, in `INSERTs`. For the full list of supported formats, see
[Formats](../../interfaces/formats.md#formats).
`Format` should be supported for `SELECT` and/or `INSERT`. For the full list of
supported formats see [Formats](../../interfaces/formats.md#formats).
`The URL` must conform to the structure of a Uniform Resource Locator. The specified URL must point to a server
that uses HTTP or HTTPS. This does not require any
additional headers for getting a response from the server.
`URL` must match the format of Uniform Resource Locator. The specified
URL must address a server working with HTTP or HTTPS. The server shouldn't
require any additional HTTP-headers.
`INSERT` and `SELECT` queries are transformed into `POST` and `GET` requests
respectively. For correct `POST`-requests handling the remote server should support
[Chunked transfer encoding](https://ru.wikipedia.org/wiki/Chunked_transfer_encoding).
`INSERT` and `SELECT` queries are transformed to `POST` and `GET` requests,
respectively. For processing `POST` requests, the remote server must support
[Chunked transfer encoding](https://en.wikipedia.org/wiki/Chunked_transfer_encoding).
**Example:**
**1.** Create the `url_engine_table` table:
**1.** Create a `url_engine_table` table on the server :
```sql
CREATE TABLE url_engine_table (word String, value UInt64)
ENGINE=URL('http://127.0.0.1:12345/', CSV)
```
**2.** Implement simple http-server using python3:
**2.** Create a basic HTTP server using the standard Python 3 tools and
start it:
```python3
from http.server import BaseHTTPRequestHandler, HTTPServer
@ -53,7 +51,7 @@ if __name__ == "__main__":
python3 server.py
```
**3.** Query the data:
**3.** Request data:
```sql
SELECT * FROM url_engine_table
@ -66,12 +64,16 @@ SELECT * FROM url_engine_table
└───────┴───────┘
```
## Usage
<<<<<<< HEAD
- Multi-stream reading and writing are supported.
=======
## Details of Implementation
- Reads and writes can be parallel
>>>>>>> upstream/master
- Not supported:
- `ALTER`
- `SELECT ... SAMPLE`
- Indices
- Replication
- `ALTER` and `SELECT...SAMPLE` operations.
- Indexes.
- Replication.

8
docs/en/operations/tips.md
View File

@ -107,6 +107,10 @@ You are probably already using ZooKeeper for other purposes. You can use the sam
It's best to use a fresh version of ZooKeeper – 3.4.9 or later. The version in stable Linux distributions may be outdated.
You should never use manually written scripts to transfer data between different ZooKeeper clusters, because the result will be incorrect for sequential nodes. Never use the "zkcopy" utility for the same reason: https://github.com/ksprojects/zkcopy/issues/15
If you want to divide an existing ZooKeeper cluster into two, the correct way is to increase the number of its replicas and then reconfigure it as two independent clusters.
Do not run ZooKeeper on the same servers as ClickHouse. Because ZooKeeper is very sensitive for latency and ClickHouse may utilize all available system resources.
With the default settings, ZooKeeper is a time bomb:
@ -115,10 +119,6 @@ With the default settings, ZooKeeper is a time bomb:
This bomb must be defused.
If you want to move data between different ZooKeeper clusters, never move it by hand-written script, because it will produce wrong data for sequential nodes. Never use "zkcopy" tool, by the same reason: https://github.com/ksprojects/zkcopy/issues/15
If you want to split ZooKeeper cluster, proper way is to increase number of replicas and then reconfigure it as two independent clusters.
The ZooKeeper (3.5.1) configuration below is used in the Yandex.Metrica production environment as of May 20, 2017:
zoo.cfg:

32
docs/en/operations/utils/clickhouse-copier.md
View File

@ -83,7 +83,7 @@ Parameters:
<readonly>0</readonly>
</settings_push>
<!-- Common setting for fetch (pull) and insert (push) operations. The copier process context also uses it.
<!-- Common setting for fetch (pull) and insert (push) operations. Also, copier process context uses it.
They are overlaid by <settings_pull/> and <settings_push/> respectively. -->
<settings>
<connect_timeout>3</connect_timeout>
@ -91,13 +91,14 @@ Parameters:
<insert_distributed_sync>1</insert_distributed_sync>
</settings>
<!-- Copying description of tasks.
You can specify several table tasks in the same task description (in the same ZooKeeper node), and they will be performed sequentially.
<!-- Copying tasks description.
You could specify several table task in the same task description (in the same ZooKeeper node), they will be performed
sequentially.
-->
<tables>
<!-- A table task that copies one table. -->
<!-- A table task, copies one table. -->
<table_hits>
<!-- Source cluster name (from the <remote_servers/> section) and tables in it that should be copied -->
<!-- Source cluster name (from <remote_servers/> section) and tables in it that should be copied -->
<cluster_pull>source_cluster</cluster_pull>
<database_pull>test</database_pull>
<table_pull>hits</table_pull>
@ -108,11 +109,12 @@ Parameters:
<table_push>hits2</table_push>
<!-- Engine of destination tables.
If the destination tables have not been created yet, workers create them using column definitions from source tables and the engine definition from here.
If destination tables have not be created, workers create them using columns definition from source tables and engine
definition from here.
NOTE: If the first worker starts to insert data and detects that the destination partition is not empty, then the partition will
be dropped and refilled. Take this into account if you already have some data in destination tables. You can directly
specify partitions that should be copied in <enabled_partitions/>. They should be in quoted format like the partition column in the
NOTE: If the first worker starts insert data and detects that destination partition is not empty then the partition will
be dropped and refilled, take it into account if you already have some data in destination tables. You could directly
specify partitions that should be copied in <enabled_partitions/>, they should be in quoted format like partition column of
system.parts table.
-->
<engine>
@ -133,11 +135,11 @@ Parameters:
Since partition key of source and destination cluster could be different,
these partition names specify destination partitions.
Note: Although this section is optional (if it omitted, all partitions will be copied),
it is strongly recommended to specify the partitions explicitly.
If you already have some partitions ready on the destination cluster, they
will be removed at the start of the copying, because they will be interpreted
as unfinished data from the previous copying.
NOTE: In spite of this section is optional (if it is not specified, all partitions will be copied),
it is strictly recommended to specify them explicitly.
If you already have some ready paritions on destination cluster they
will be removed at the start of the copying since they will be interpeted
as unfinished data from the previous copying!!!
-->
<enabled_partitions>
<partition>'2018-02-26'</partition>
@ -146,7 +148,7 @@ Parameters:
</enabled_partitions>
</table_hits>
<!-- Next table to copy. It is not copied until the previous table is copying. -->
<!-- Next table to copy. It is not copied until previous table is copying. -->
</table_visits>
...
</table_visits>

48
docs/en/operations/utils/clickhouse-local.md
View File

@ -2,42 +2,36 @@
# clickhouse-local
The `clickhouse-local` program enables you to perform fast processing on local files, without having to deploy and configure the ClickHouse server.
Accepts data that can be represented as a table and performs the operations that are specified in the ClickHouse [query language](../../query_language/index.md#queries).
Accepts data that represent tables and queries them using [ClickHouse SQL dialect](../../query_language/index.md#queries).
`clickhouse-local` uses the ClickHouse server engine, meaning it supports all date formats and table engines that ClickHouse works with, and operations do not require a running server.
`clickhouse-local` uses the same core as ClickHouse server, so it supports most of the features and the same set of formats and table engines.
`When clickhouse-local` is configured by default, it does not have access to data managed by the ClickHouse server that is installed on the same host, but you can use the `--config-file` key to connect the server configuration.
By default `clickhouse-local` does not have access to data on the same host, but it supports loading server configuration using `--config-file` argument.
!!! Warning:
We do not recommend connecting the server configuration to `clickhouse-local`, since data can be easily damaged by accident.
!!! warning
It is not recommended to load production server configuration into `clickhouse-local` because data can be damaged in case of human error.
## Invoking the program
## Usage
Basic usage:
Basic format of the call:
```bash
clickhouse-local --structure "table_structure" --input-format "format_of_incoming_data" -q "query"
```
Arguments:
- `-S`, `--structure` — table structure for input data.
- `-if`, `--input-format` — input format, `TSV` by default.
- `-f`, `--file` — path to data, `stdin` by default.
- `-q` `--query` — queries to execute with `;` as delimeter.
- `-N`, `--table` — table name where to put output data, `table` by default.
- `-of`, `--format`, `--output-format` — output format, `TSV` by default.
- `--stacktrace` — whether to dump debug output in case of exception.
- `--verbose` — more details on query execution.
- `-s` — disables `stderr` logging.
- `--config-file` — path to configuration file in same format as for ClickHouse server, by default the configuration empty.
- `--help` — arguments references for `clickhouse-local`.
Also there are arguments for each ClickHouse configuration variable which are more commonly used instead of `--config-file`.
Command keys:
- `-S`, `--structure` — The structure of the table where the input data will be placed.
- `-if`, `--input-format` — The input data format. By default, it is `TSV`.
- `-f`, `--file` — The path to the data file. By default, it is `stdin`.
- `-q`, `--query` — Queries to run. The query separator is `;`.
- `-N`, `-- table` — The name of the table where the input data will be placed. By default, it is `table`.
- `-of`, `--format`, `--output-format` — The output data format. By default, it is `TSV`.
- `--stacktrace` — Output debugging information for exceptions.
- `--verbose` — Verbose output when running a query.
- `-s` — Suppresses displaying the system log in `stderr`.
- `--config-file` — The path to the configuration file. By default, `clickhouse-local` starts with an empty configuration. The configuration file has the same format as the ClickHouse server and can use all the server configuration parameters. Typically, the connection configuration is not required. If you want to set a specific parameter, you can use a key with the parameter name.
- `--help` — Output reference information about `clickhouse-local`.
## Examples
@ -48,7 +42,7 @@ Read 2 rows, 32.00 B in 0.000 sec., 5182 rows/sec., 80.97 KiB/sec.
3 4
```
Previous example is the same as:
The above command is equivalent to the following:
```bash
$ echo -e "1,2\n3,4" | clickhouse-local -q "CREATE TABLE table (a Int64, b Int64) ENGINE = File(CSV, stdin); SELECT a, b FROM table; DROP TABLE table"
@ -57,7 +51,7 @@ Read 2 rows, 32.00 B in 0.000 sec., 4987 rows/sec., 77.93 KiB/sec.
3 4
```
Now let's output memory user for each Unix user:
Now let's show the amount of RAM occupied by users (Unix) on the screen:
```bash
$ ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' | clickhouse-local -S "user String, mem Float64" -q "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty"

52
docs/en/query_language/agg_functions/index.md
View File

@ -9,3 +9,55 @@ ClickHouse also supports:
- [Parametric aggregate functions](parametric_functions.md#aggregate_functions_parametric), which accept other parameters in addition to columns.
- [Combinators](combinators.md#aggregate_functions_combinators), which change the behavior of aggregate functions.
## NULL processing
During aggregation, all `NULL`s are skipped.
**Examples:**
Consider this table:
```
┌─x─┬────y─┐
│ 1 │ 2 │
│ 2 │ ᴺᵁᴸᴸ │
│ 3 │ 2 │
│ 3 │ 3 │
│ 3 │ ᴺᵁᴸᴸ │
└───┴──────┘
```
Let's say you need to total the values in the `y` column:
```
:) SELECT sum(y) FROM t_null_big
SELECT sum(y)
FROM t_null_big
┌─sum(y)─┐
│ 7 │
└────────┘
1 rows in set. Elapsed: 0.002 sec.
```
The `sum` function interprets `NULL` as `0`. In particular, this means that if the function receives input of a selection where all the values are `NULL`, then the result will be `0`, not `NULL`.
Now you can use the `groupArray` function to create an array from the `y` column:
```
:) SELECT groupArray(y) FROM t_null_big
SELECT groupArray(y)
FROM t_null_big
┌─groupArray(y)─┐
│ [2,2,3] │
└───────────────┘
1 rows in set. Elapsed: 0.002 sec.
```
`groupArray` does not include `NULL` in the resulting array.

37
docs/en/query_language/agg_functions/parametric_functions.md
View File

@ -52,19 +52,39 @@ Chains are searched for without overlapping. In other words, the next chain can
## windowFunnel(window)(timestamp, cond1, cond2, cond3, ...)
Window funnel matching for event chains, calculates the max event level in a sliding window.
Searches for event chains in a sliding time window and calculates the maximum number of events that occurred from the chain.
`window` is the timestamp window value, such as 3600.
```
windowFunnel(window)(timestamp, cond1, cond2, cond3, ...)
```
`timestamp` is the time of the event with the DateTime type or UInt32 type.
**Parameters:**
`cond1`, `cond2` ... is from one to 32 arguments of type UInt8 that indicate whether a certain condition was met for the event
- `window` — Length of the sliding window in seconds.
- `timestamp` — Name of the column containing the timestamp. Data type: [DateTime](../../data_types/datetime.md#data_type-datetime) or [UInt32](../../data_types/int_uint.md#data_type-int).
- `cond1`, `cond2`... — Conditions or data describing the chain of events. Data type: `UInt8`. Values can be 0 or 1.
Example:
**Algorithm**
Consider you are doing a website analytics, intend to find out the user counts clicked login button (event = 1001), then the user counts followed by searched the phones( event = 1003 and product = 'phone'), then the user counts followed by made an order (event = 1009). And all event chains must be in a 3600 seconds sliding window.
- The function searches for data that triggers the first condition in the chain and sets the event counter to 1. This is the moment when the sliding window starts.
- If events from the chain occur sequentially within the window, the counter is incremented. If the sequence of events is disrupted, the counter isn't incremented.
- If the data has multiple event chains at varying points of completion, the function will only output the size of the longest chain.
This could be easily calculate by `windowFunnel`
**Returned value**
- Integer. The maximum number of consecutive triggered conditions from the chain within the sliding time window. All the chains in the selection are analyzed.
**Example**
Determine if one hour is enough for the user to select a phone and purchase it in the online store.
Set the following chain of events:
1. The user logged in to their account on the store (`eventID=1001`).
2. The user searches for a phone (`eventID = 1003, product = 'phone'`).
3. The user placed an order (`eventID = 1009`).
To find out how far the user `user_id` could get through the chain in an hour in January of 2017, make the query:
```
SELECT
@ -74,7 +94,7 @@ FROM
(
SELECT
user_id,
windowFunnel(3600)(timestamp, event_id = 1001, event_id = 1003 AND product = 'phone', event_id = 1009) AS level
windowFunnel(3600)(timestamp, eventID = 1001, eventID = 1003 AND product = 'phone', eventID = 1009) AS level
FROM trend_event
WHERE (event_date >= '2017-01-01') AND (event_date <= '2017-01-31')
GROUP BY user_id
@ -135,4 +155,3 @@ Usage example:
Problem: Generate a report that shows only keywords that produced at least 5 unique users.
Solution: Write in the GROUP BY query SearchPhrase HAVING uniqUpTo(4)(UserID) >= 5
```

3
docs/en/query_language/agg_functions/reference.md
View File

@ -28,6 +28,7 @@ anyHeavy(column)
```
**Arguments**
- `column` – The column name.
**Example**
@ -301,6 +302,7 @@ This function doesn't provide a guaranteed result. In certain situations, errors
We recommend using the `N < 10 ` value; performance is reduced with large `N` values. Maximum value of ` N = 65536`.
**Arguments**
- 'N' is the number of values.
- ' x ' – The column.
@ -312,6 +314,7 @@ Take the [OnTime](../../getting_started/example_datasets/ontime.md#example_datas
SELECT topK(3)(AirlineID) AS res
FROM ontime
```
```
┌─res─────────────────┐
│ [19393,19790,19805] │

34
docs/en/query_language/alter.md
View File

@ -224,45 +224,45 @@ Possible values: `0` – do not wait; `1` – only wait for own execution (defau
### Mutations
Mutations are an ALTER query variant that allows changing or deleting rows in a table. In contrast to standard `UPDATE` and `DELETE` queries that are intended for point data changes, mutations are intended for heavy operations that change a lot of rows in a table.
A mutation is a type of ALTER query that lets you change or delete data in a table. In contrast to standard `DELETE` and `UPDATE` queries that are calculated for point-based data changes, mutations are used for broad changes that affect many rows in the table.
The functionality is in beta stage and is available starting with the 1.1.54388 version. Currently *MergeTree table engines are supported (both replicated and unreplicated).
The functionality is in beta testing and is available starting from version 1.1.54388. Implemented support for \*MergeTree tables (with and without replication).
Existing tables are ready for mutations as-is (no conversion necessary), but after the first mutation is applied to a table, its metadata format becomes incompatible with previous server versions and falling back to a previous version becomes impossible.
You don't need to convert existing tables to work with mutations. However, after the first mutation is applied, the table data format becomes incompatible with previous versions and it is not possible to roll back to the previous version.
At the moment the `ALTER DELETE` command is available:
The `ALTER DELETE` command is currently available:
```sql
ALTER TABLE [db.]table DELETE WHERE expr
```
The expression `expr` must be of UInt8 type. The query deletes rows for which this expression evaluates to a non-zero value.
The `expr` must be of type UInt8. The query deletes rows in the table for which this expression takes a non-zero value.
One query can contain several commands separated by commas.
A single query can specify multiple comma-separated commands.
For *MergeTree tables mutations execute by rewriting whole data parts. There is no atomicity - parts are substituted for mutated parts as soon as they are ready and a `SELECT` query that started executing during a mutation will see data from parts that have already been mutated along with data from parts that have not been mutated yet.
For \*Merge-tables, the mutations are applied by overwriting the data in chunks (parts). However, there is no atomicity: the parts are replaced with the mutations as they are processed and the `SELECT` query specified during execution of the mutation will see the data for both the changed parts and the parts that have not yet been changed.
Mutations are totally ordered by their creation order and are applied to each part in that order. Mutations are also partially ordered with INSERTs - data that was inserted into the table before the mutation was submitted will be mutated and data that was inserted after that will not be mutated. Note that mutations do not block INSERTs in any way.
The mutations have a linear order and they are applied to each part in the order they were added. Mutations are also ordered with inserts. This guarantees that data inserted in the table before the start of the mutation query will be changed, and data inserted after the query ends will not be changed. However, mutations do not block inserts in any way.
A mutation query returns immediately after the mutation entry is added (in case of replicated tables to ZooKeeper, for nonreplicated tables - to the filesystem). The mutation itself executes asynchronously using the system profile settings. To track the progress of mutations you can use the `system.mutations` table. A mutation that was successfully submitted will continue to execute even if ClickHouse servers are restarted. There is no way to roll back the mutation once it is submitted.
The query is completed immediately after adding information about the mutation (for replicated tables, in Zookeeper; for non-replicated tables, in the file system). The mutation itself is executed asynchronously using the system profile settings. You can monitor the progress in the `system.mutations` table. Added mutations are fully completed even if ClickHouse is restarted. You can't revert a mutation after it has been added.
Entries for finished mutations are not deleted right away (the number of preserved entries is determined by the `finished_mutations_to_keep` storage engine parameter). Older mutation entries are deleted.
#### system.mutations Table
The table contains information about mutations of MergeTree tables and their progress. Each mutation command is represented by a single row. The table has the following columns:
This table contains information about the progress of mutations on MergeTree tables. Each mutation command corresponds to a single row. The table has the following columns:
**database**, **table** - The name of the database and table to which the mutation was applied.
**database**, **table** — The name of the database and the table that the mutation was applied to.
**mutation_id** - The ID of the mutation. For replicated tables these IDs correspond to znode names in the `<table_path_in_zookeeper>/mutations/` directory in ZooKeeper. For unreplicated tables the IDs correspond to file names in the data directory of the table.
**mutation_id** — ID of the query. For replicated tables, these IDs correspond to the names of entries in the `<table_path_in_zookeeper>/mutations/` directory in ZooKeeper. For unreplicated tables, they are the file names in the directory with the table data.
**command** - The mutation command string (the part of the query after `ALTER TABLE [db.]table`).
**command** — The mutation command (the part of the query after `ALTER TABLE [db.]table`).
**create_time** - When this mutation command was submitted for execution.
**create_time** — The time of creation of the mutation.
**block_numbers.partition_id**, **block_numbers.number** - A Nested column. For mutations of replicated tables contains one record for each partition: the partition ID and the block number that was acquired by the mutation (in each partition only parts that contain blocks with numbers less than the block number acquired by the mutation in that partition will be mutated). Because in non-replicated tables blocks numbers in all partitions form a single sequence, for mutatations of non-replicated tables the column will contain one record with a single block number acquired by the mutation.
**block_numbers.partition_id**, **block_numbers.number** — Nested column. For mutations in replicated tables, it contains the block number obtained by this mutation for each partition (each partition will only have changes in the parts that contain blocks with numbers less than the number obtained by the mutation in this partition). For non-replicated tables, the block numbering carries through the partitions, so the column contains a single entry with a single block number that was obtained by the mutation.
**parts_to_do** - The number of data parts that need to be mutated for the mutation to finish.
**parts_to_do** — The number of parts of the table that still need to be changed.
**is_done** - Is the mutation done? Note that even if `parts_to_do = 0` it is possible that a mutation of a replicated table is not done yet because of a long-running INSERT that will create a new data part that will need to be mutated.
**is_done** — Whether the mutation is complete. Note: Even if `parts_to_do = 0`, for a replicated table, there may be a situation when the mutation has not yet finished because of a long insert that is adding data that will need to be mutated.

1
docs/en/query_language/create.md
View File

@ -152,4 +152,3 @@ The execution of `ALTER` queries on materialized views has not been fully develo
Views look the same as normal tables. For example, they are listed in the result of the `SHOW TABLES` query.
There isn't a separate query for deleting views. To delete a view, use `DROP TABLE`.

17
docs/en/query_language/dicts/external_dicts.md
View File

@ -2,11 +2,11 @@
# External Dictionaries
You can add your own dictionaries from various data sources. The data source for a dictionary can be a local text or executable file, an HTTP(s) resource, or another DBMS. For more information, see "[Sources for external dictionaries](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources)".
You can add your own dictionaries from various data sources. The data source for a dictionary can be a local text or executable file, an HTTP(s) resource, or another DBMS. For more information, see "[Sources of external dictionaries](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources)".
ClickHouse:
> - Fully or partially stores dictionaries in RAM.
- Fully or partially stores dictionaries in RAM.
- Periodically updates dictionaries and dynamically loads missing values. In other words, dictionaries can be loaded dynamically.
The configuration of external dictionaries is located in one or more files. The path to the configuration is specified in the [dictionaries_config](../../operations/server_settings/settings.md#server_settings-dictionaries_config) parameter.
@ -37,7 +37,14 @@ The dictionary config file has the following format:
You can [configure](external_dicts_dict.md#dicts-external_dicts_dict) any number of dictionaries in the same file. The file format is preserved even if there is only one dictionary (i.e. `<yandex><dictionary> <!--configuration -> </dictionary></yandex>` ).
See also "[Functions for working with external dictionaries](../functions/ext_dict_functions.md#ext_dict_functions)".
> You can convert values ​​for a small dictionary by describing it in a `SELECT` query (see the [transform](../functions/other_functions.md#other_functions-transform) function). This functionality is not related to external dictionaries.
See also:
- [Configuring an external dictionary](external_dicts_dict.md#dicts-external_dicts_dict)
- [Storing dictionaries in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout)
- [Updating dictionaries](external_dicts_dict_lifetime#dicts-external_dicts_dict_lifetime)
- [Sources of external dictionaries](external_dicts_dict_sources.md#dicts-external_dicts_dict_sources)
- [Dictionary key and fields](external_dicts_dict_structure.md#dicts-external_dicts_dict_dict_structure)
- [Functions for working with external dictionaries](../functions/ext_dict_functions.md#ext_dict_functions)
!!! attention
You can convert values for a small dictionary by describing it in a `SELECT` query (see the [transform](../functions/other_functions.md#other_functions-transform) function). This functionality is not related to external dictionaries.

15
docs/en/query_language/dicts/external_dicts_dict_layout.md
View File

@ -88,7 +88,7 @@ Configuration example:
### complex_key_hashed
This type of storage is for use with composite [keys](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure). Similar to `hashed`.
This type of storage is for use with complex [keys](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure). Similar to `hashed`.
Configuration example:
@ -140,9 +140,7 @@ Example:
To work with these dictionaries, you need to pass an additional date argument to the `dictGetT` function:
```
dictGetT('dict_name', 'attr_name', id, date)
```
This function returns the value for the specified `id`s and the date range that includes the passed date.
@ -193,7 +191,7 @@ The dictionary is stored in a cache that has a fixed number of cells. These cell
When searching for a dictionary, the cache is searched first. For each block of data, all keys that are not found in the cache or are outdated are requested from the source using ` SELECT attrs... FROM db.table WHERE id IN (k1, k2, ...)`. The received data is then written to the cache.
For cache dictionaries, the expiration [lifetime](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime) of data in the cache can be set. If more time than `lifetime` has passed since loading the data in a cell, the cell's value is not used, and it is re-requested the next time it needs to be used.
For cache dictionaries, the expiration ([lifetime](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime)) of data in the cache can be set. If more time than `lifetime` has passed since loading the data in a cell, the cell's value is not used, and it is re-requested the next time it needs to be used.
This is the least effective of all the ways to store dictionaries. The speed of the cache depends strongly on correct settings and the usage scenario. A cache type dictionary performs well only when the hit rates are high enough (recommended 99% and higher). You can view the average hit rate in the `system.dictionaries` table.
@ -219,14 +217,14 @@ Set a large enough cache size. You need to experiment to select the number of ce
3. Assess memory consumption using the `system.dictionaries` table.
4. Increase or decrease the number of cells until the required memory consumption is reached.
!!! warning
!!! Warning:
Do not use ClickHouse as a source, because it is slow to process queries with random reads.
<a name="dicts-external_dicts_dict_layout-complex_key_cache"></a>
### complex_key_cache
This type of storage is for use with composite [keys](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure). Similar to `cache`.
This type of storage is for use with complex [keys](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure). Similar to `cache`.
<a name="dicts-external_dicts_dict_layout-ip_trie"></a>
@ -275,7 +273,7 @@ Example:
...
```
The key must have only one String type attribute that contains an allowed IP prefix. Other types are not supported yet.
The key must have only one `String` type attribute that contains an allowed IP prefix. Other types are not supported yet.
For queries, you must use the same functions (`dictGetT` with a tuple) as for dictionaries with composite keys:
@ -283,7 +281,7 @@ For queries, you must use the same functions (`dictGetT` with a tuple) as for di
dictGetT('dict_name', 'attr_name', tuple(ip))
```
The function takes either `UInt32` for IPv4, or `FixedString(16)` for IPv6:
The function accepts either `UInt32` for IPv4, or `FixedString(16)` for IPv6:
```
dictGetString('prefix', 'asn', tuple(IPv6StringToNum('2001:db8::1')))
@ -292,3 +290,4 @@ dictGetString('prefix', 'asn', tuple(IPv6StringToNum('2001:db8::1')))
Other types are not supported yet. The function returns the attribute for the prefix that corresponds to this IP address. If there are overlapping prefixes, the most specific one is returned.
Data is stored in a `trie` . It must completely fit into RAM.

3
docs/en/query_language/dicts/external_dicts_dict_sources.md
View File

@ -161,7 +161,7 @@ The dictionary configuration in ClickHouse:
<name>table_name</name>
<source>
<odbc>
<!-- You can specifiy the following parameters in connection_string: -->
<!-- You can specify the following parameters in connection_string: -->
<!-- DSN=myconnection;UID=username;PWD=password;HOST=127.0.0.1;PORT=5432;DATABASE=my_db -->
<connection_string>DSN=myconnection</connection_string>
<table>postgresql_table</table>
@ -398,4 +398,3 @@ Setting fields:
- `password` – Password of the MongoDB user.
- `db` – Name of the database.
- `collection` – Name of the collection.

5
docs/en/query_language/dicts/external_dicts_dict_structure.md
View File

@ -39,7 +39,7 @@ ClickHouse supports the following types of keys:
A structure can contain either `<id>` or `<key>` .
!!! warning
!!! Note:
The key doesn't need to be defined separately in attributes.
### Numeric Key
@ -62,7 +62,7 @@ Configuration fields:
The key can be a `tuple` from any types of fields. The [layout](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout) in this case must be `complex_key_hashed` or `complex_key_cache`.
!!! tip
!!! Tip:
A composite key can consist of a single element. This makes it possible to use a string as the key, for instance.
The key structure is set in the element `<key>`. Key fields are specified in the same format as the dictionary [attributes](external_dicts_dict_structure.md#dicts-external_dicts_dict_structure-attributes). Example:
@ -115,3 +115,4 @@ Configuration fields:
- `hierarchical` – Hierarchical support. Mirrored to the parent identifier. By default, `false`.
- `injective` – Whether the `id -> attribute` image is injective. If `true`, then you can optimize the ` GROUP BY` clause. By default, `false`.
- `is_object_id` – Whether the query is executed for a MongoDB document by `ObjectID`.

12
docs/en/query_language/dicts/index.md
View File

@ -1,7 +1,13 @@
# Dictionaries
`A dictionary` is a mapping (key `->` attributes) that can be used in a query as functions.
You can think of this as a more convenient and efficient type of JOIN with dimension tables.
A dictionary is a mapping (`key -> attributes`) that is convenient for various types of reference lists.
There are built-in (internal) and add-on (external) dictionaries.
ClickHouse supports special functions for working with dictionaries that can be used in queries. It is easier and more efficient to use dictionaries with functions than a `JOIN` with reference tables.
[NULL](../syntax.md#null-literal) values can't be stored in a dictionary.
ClickHouse supports:
- [Built-in dictionaries](internal_dicts.md#internal_dicts) with a specific [set of functions](../functions/ym_dict_functions.md#ym_dict_functions).
- [Plug-in (external) dictionaries](external_dicts.md#dicts-external_dicts) with a [set of functions](../functions/ext_dict_functions.md#ext_dict_functions).

26
docs/en/query_language/dicts/internal_dicts.md
View File

@ -1,4 +1,6 @@
# Internal Dictionaries
<a name="internal_dicts"></a>
# Internal dictionaries
ClickHouse contains a built-in feature for working with a geobase.
@ -15,35 +17,33 @@ The internal dictionaries are disabled in the default package.
To enable them, uncomment the parameters `path_to_regions_hierarchy_file` and `path_to_regions_names_files` in the server configuration file.
The geobase is loaded from text files.
If you work at Yandex, you can follow these instructions to create them:
<https://github.yandex-team.ru/raw/Metrika/ClickHouse_private/master/doc/create_embedded_geobase_dictionaries.txt>
If you work at Yandex, you can create them by following [the instructions](https://github.yandex-team.ru/raw/Metrika/ClickHouse_private/master/doc/create_embedded_geobase_dictionaries.txt).
Put the regions_hierarchy\*.txt files in the path_to_regions_hierarchy_file directory. This configuration parameter must contain the path to the regions_hierarchy.txt file (the default regional hierarchy), and the other files (regions_hierarchy_ua.txt) must be located in the same directory.
Place the `regions_hierarchy*.txt` files into the `path_to_regions_hierarchy_file` directory. This configuration parameter must contain the path to the `regions_hierarchy.txt` file (the default regional hierarchy), and the other files (`regions_hierarchy_ua.txt`) must be located in the same directory.
Put the `regions_names_*.txt` files in the path_to_regions_names_files directory.
Put the `regions_names_*.txt` files in the `path_to_regions_names_files` directory.
You can also create these files yourself. The file format is as follows:
`regions_hierarchy*.txt`: TabSeparated (no header), columns:
- Region ID (UInt32)
- Parent region ID (UInt32)
- Region type (UInt8): 1 - continent, 3 - country, 4 - federal district, 5 - region, 6 - city; other types don't have values.
- Population (UInt32) - Optional column.
- region ID (`UInt32`)
- parent region ID (`UInt32`)
- region type (`UInt8`): 1 - continent, 3 - country, 4 - federal district, 5 - region, 6 - city; other types don't have values
- population (`UInt32`) — optional column
`regions_names_*.txt`: TabSeparated (no header), columns:
- Region ID (UInt32)
- Region name (String) - Can't contain tabs or line feeds, even escaped ones.
- region ID (`UInt32`)
- region name (`String`) — Can't contain tabs or line feeds, even escaped ones.
A flat array is used for storing in RAM. For this reason, IDs shouldn't be more than a million.
Dictionaries can be updated without restarting the server. However, the set of available dictionaries is not updated.
For updates, the file modification times are checked. If a file has changed, the dictionary is updated.
The interval to check for changes is configured in the 'builtin_dictionaries_reload_interval' parameter.
The interval to check for changes is configured in the `builtin_dictionaries_reload_interval` parameter.
Dictionary updates (other than loading at first use) do not block queries. During updates, queries use the old versions of dictionaries. If an error occurs during an update, the error is written to the server log, and queries continue using the old version of dictionaries.
We recommend periodically updating the dictionaries with the geobase. During an update, generate new files and write them to a separate location. When everything is ready, rename them to the files used by the server.
There are also functions for working with OS identifiers and Yandex.Metrica search engines, but they shouldn't be used.

98
docs/en/query_language/functions/array_functions.md
View File

@ -62,6 +62,7 @@ arrayConcat(arrays)
```sql
SELECT arrayConcat([1, 2], [3, 4], [5, 6]) AS res
```
```
┌─res───────────┐
│ [1,2,3,4,5,6] │
@ -81,13 +82,49 @@ If the index falls outside of the bounds of an array, it returns some default va
Checks whether the 'arr' array has the 'elem' element.
Returns 0 if the the element is not in the array, or 1 if it is.
`NULL` is processed as a value.
```
SELECT has([1, 2, NULL], NULL)
┌─has([1, 2, NULL], NULL)─┐
│ 1 │
└─────────────────────────┘
```
## indexOf(arr, x)
Returns the index of the 'x' element (starting from 1) if it is in the array, or 0 if it is not.
Returns the index of the first 'x' element (starting from 1) if it is in the array, or 0 if it is not.
Example:
```
:) select indexOf([1,3,NULL,NULL],NULL)
SELECT indexOf([1, 3, NULL, NULL], NULL)
┌─indexOf([1, 3, NULL, NULL], NULL)─┐
│ 3 │
└───────────────────────────────────┘
```
Elements set to `NULL` are handled as normal values.
## countEqual(arr, x)
Returns the number of elements in the array equal to x. Equivalent to arrayCount (elem-&gt; elem = x, arr).
Returns the number of elements in the array equal to x. Equivalent to arrayCount (elem -> elem = x, arr).
`NULL` elements are handled as separate values.
Example:
```
SELECT countEqual([1, 2, NULL, NULL], NULL)
┌─countEqual([1, 2, NULL, NULL], NULL)─┐
│ 2 │
└──────────────────────────────────────┘
```
## arrayEnumerate(arr)
@ -202,6 +239,7 @@ arrayPopBack(array)
```sql
SELECT arrayPopBack([1, 2, 3]) AS res
```
```
┌─res───┐
│ [1,2] │
@ -225,6 +263,7 @@ arrayPopFront(array)
```sql
SELECT arrayPopFront([1, 2, 3]) AS res
```
```
┌─res───┐
│ [2,3] │
@ -242,7 +281,7 @@ arrayPushBack(array, single_value)
**Arguments**
- `array` – Array.
- `single_value` – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. When adding numbers, ClickHouse automatically sets the `single_value` type for the data type of the array. For more information about ClickHouse data types, read the section "[Data types](../../data_types/index.md#data_types)".
- `single_value` – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. When adding numbers, ClickHouse automatically sets the `single_value` type for the data type of the array. For more information about the types of data in ClickHouse, see "[Data types](../../data_types/index.md#data_types)". Can be `NULL`. The function adds a `NULL` element to an array, and the type of array elements converts to `Nullable`.
**Example**
@ -267,19 +306,58 @@ arrayPushFront(array, single_value)
**Arguments**
- `array` – Array.
- `single_value` – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. When adding numbers, ClickHouse automatically sets the `single_value` type for the data type of the array. For more information about ClickHouse data types, read the section "[Data types](../../data_types/index.md#data_types)".
- `single_value` – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. When adding numbers, ClickHouse automatically sets the `single_value` type for the data type of the array. For more information about the types of data in ClickHouse, see "[Data types](../../data_types/index.md#data_types)". Can be `NULL`. The function adds a `NULL` element to an array, and the type of array elements converts to `Nullable`.
**Example**
```sql
SELECT arrayPushBack(['b'], 'a') AS res
```
```
┌─res───────┐
│ ['a','b'] │
└───────────┘
```
## arrayResize
Changes the length of the array.
```
arrayResize(array, size[, extender])
```
**Parameters:**
- `array` — Array.
- `size` — Required length of the array.
- If `size` is less than the original size of the array, the array is truncated from the right.
- If `size` is larger than the initial size of the array, the array is extended to the right with `extender` values or default values for the data type of the array items.
- `extender` — Value for extending an array. Can be `NULL`.
**Returned value:**
An array of length `size`.
**Examples of calls**
```
SELECT arrayResize([1], 3)
┌─arrayResize([1], 3)─┐
│ [1,0,0] │
└─────────────────────┘
```
```
SELECT arrayResize([1], 3, NULL)
┌─arrayResize([1], 3, NULL)─┐
│ [1,NULL,NULL] │
└───────────────────────────┘
```
## arraySlice
Returns a slice of the array.
@ -297,14 +375,17 @@ arraySlice(array, offset[, length])
**Example**
```sql
SELECT arraySlice([1, 2, 3, 4, 5], 2, 3) AS res
SELECT arraySlice([1, 2, NULL, 4, 5], 2, 3) AS res
```
```
┌─res─────┐
│ [2,3,4] │
└─────────┘
┌─res────────┐
│ [2,NULL,4] │
└────────────┘
```
Array elements set to `NULL` are handled as normal values.
## arrayUniq(arr, ...)
If one argument is passed, it counts the number of different elements in the array.
@ -315,4 +396,3 @@ If you want to get a list of unique items in an array, you can use arrayReduce('
## arrayJoin(arr)
A special function. See the section ["ArrayJoin function"](array_join.md#functions_arrayjoin).

44
docs/en/query_language/functions/conditional_functions.md
View File

@ -2,5 +2,47 @@
## if(cond, then, else), cond ? operator then : else
Returns 'then' if cond !or 'else' if cond = 0.'cond' must be UInt 8, and 'then' and 'else' must be a type that has the smallest common type.
Returns `then` if `cond != 0`, or `else` if `cond = 0`.
`cond` must be of type `UInt8`, and `then` and `else` must have the lowest common type.
`then` and `else` can be `NULL`
## multiIf
Allows you to write the [CASE](../operators.md#operator_case) operator more compactly in the query.
```
multiIf(cond_1, then_1, cond_2, then_2...else)
```
**Parameters:**
- `cond_N` — The condition for the function to return `then_N`.
- `then_N` — The result of the function when executed.
- `else` — The result of the function if none of the conditions is met.
The function accepts `2N+1` parameters.
**Returned values**
The function returns one of the values `then_N` or `else`, depending on the conditions `cond_N`.
**Example**
Take the table
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 3 │
└───┴──────┘
```
Run the query `SELECT multiIf(isNull(y) x, y < 3, y, NULL) FROM t_null`. Result:
```
┌─multiIf(isNull(y), x, less(y, 3), y, NULL)─┐
│ 1 │
│ ᴺᵁᴸᴸ │
└────────────────────────────────────────────┘
```

1
docs/en/query_language/functions/ext_dict_functions.md
View File

@ -15,6 +15,7 @@ For information on connecting and configuring external dictionaries, see "[Exter
## dictGetUUID
## dictGetString
`dictGetT('dict_name', 'attr_name', id)`
- Get the value of the attr_name attribute from the dict_name dictionary using the 'id' key.`dict_name` and `attr_name` are constant strings.`id`must be UInt64.

295
docs/en/query_language/functions/functions_for_nulls.md Normal file
View File

@ -0,0 +1,295 @@
# Functions for working with Nullable aggregates
## isNull
Checks whether the argument is [NULL](../syntax.md#null-literal).
```
isNull(x)
```
**Parameters:**
- `x` — A value with a non-compound data type.
**Returned value**
- `1` if `x` is `NULL`.
- `0` if `x` is not `NULL`.
**Example**
Input table
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 3 │
└───┴──────┘
```
Query
```
:) SELECT x FROM t_null WHERE isNull(y)
SELECT x
FROM t_null
WHERE isNull(y)
┌─x─┐
│ 1 │
└───┘
1 rows in set. Elapsed: 0.010 sec.
```
## isNotNull
Checks whether the argument is [NULL](../syntax.md#null-literal).
```
isNotNull(x)
```
**Parameters:**
- `x` — A value with a non-compound data type.
**Returned value**
- `0` if `x` is `NULL`.
- `1` if `x` is not `NULL`.
**Example**
Input table
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 3 │
└───┴──────┘
```
Query
```
:) SELECT x FROM t_null WHERE isNotNull(y)
SELECT x
FROM t_null
WHERE isNotNull(y)
┌─x─┐
│ 2 │
└───┘
1 rows in set. Elapsed: 0.010 sec.
```
## coalesce
Checks from left to right whether `NULL` arguments were passed and returns the first non-`NULL` argument.
```
coalesce(x,...)
```
**Parameters:**
- Any number of parameters of a non-compound type. All parameters must be compatible by data type.
**Returned values**
- The first non-`NULL` argument.
- `NULL`, if all arguments are `NULL`.
**Example**
Consider a list of contacts that may specify multiple ways to contact a customer.
```
┌─name─────┬─mail─┬─phone─────┬──icq─┐
│ client 1 │ ᴺᵁᴸᴸ │ 123-45-67 │ 123 │
│ client 2 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │
└──────────┴──────┴───────────┴──────┘
```
The `mail` and `phone` fields are of type String, but the `icq` field is `UInt32`, so it needs to be converted to `String`.
Get the first available contact method for the customer from the contact list:
```
:) SELECT coalesce(mail, phone, CAST(icq,'Nullable(String)')) FROM aBook
SELECT coalesce(mail, phone, CAST(icq, 'Nullable(String)'))
FROM aBook
┌─name─────┬─coalesce(mail, phone, CAST(icq, 'Nullable(String)'))─┐
│ client 1 │ 123-45-67 │
│ client 2 │ ᴺᵁᴸᴸ │
└──────────┴──────────────────────────────────────────────────────┘
2 rows in set. Elapsed: 0.006 sec.
```
## ifNull
Returns an alternative value if the main argument is `NULL`.
```
ifNull(x,alt)
```
**Parameters:**
- `x` — The value to check for `NULL`.
- `alt` — The value that the function returns if `x` is `NULL`.
**Returned values**
- The value `x`, if `x` is not `NULL`.
- The value `alt`, if `x` is `NULL`.
**Example**
```
SELECT ifNull('a', 'b')
┌─ifNull('a', 'b')─┐
│ a │
└──────────────────┘
```
```
SELECT ifNull(NULL, 'b')
┌─ifNull(NULL, 'b')─┐
│ b │
└───────────────────┘
```
## nullIf
Returns `NULL` if the arguments are equal.
```
nullIf(x, y)
```
**Parameters:**
`x`, `y` — Values for comparison. They must be compatible types, or ClickHouse will generate an exception.
**Returned values**
- `NULL`, if the arguments are equal.
- The `x` value, if the arguments are not equal.
**Example**
```
SELECT nullIf(1, 1)
┌─nullIf(1, 1)─┐
│ ᴺᵁᴸᴸ │
└──────────────┘
```
```
SELECT nullIf(1, 2)
┌─nullIf(1, 2)─┐
│ 1 │
└──────────────┘
```
## assumeNotNull
Results in a value of type [Nullable](../../data_types/nullable.md#data_type-nullable) for a non- `Nullable`, if the value is not `NULL`.
```
assumeNotNull(x)
```
**Parameters:**
- `x` — The original value.
**Returned values**
- The original value from the non-`Nullable` type, if it is not `NULL`.
- The default value for the non-`Nullable` type if the original value was `NULL`.
**Example**
Consider the `t_null` table.
```
SHOW CREATE TABLE t_null
┌─statement─────────────────────────────────────────────────────────────────┐
│ CREATE TABLE default.t_null ( x Int8, y Nullable(Int8)) ENGINE = TinyLog │
└───────────────────────────────────────────────────────────────────────────┘
```
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 3 │
└───┴──────┘
```
Apply the `resumenotnull` function to the `y` column.
```
SELECT assumeNotNull(y) FROM t_null
┌─assumeNotNull(y)─┐
│ 0 │
│ 3 │
└──────────────────┘
```
```
SELECT toTypeName(assumeNotNull(y)) FROM t_null
┌─toTypeName(assumeNotNull(y))─┐
│ Int8 │
│ Int8 │
└──────────────────────────────┘
```
## toNullable
Converts the argument type to `Nullable`.
```
toNullable(x)
```
**Parameters:**
- `x` — The value of any non-compound type.
**Returned value**
- The input value with a non-`Nullable` type.
**Example**
```
SELECT toTypeName(10)
┌─toTypeName(10)─┐
│ UInt8 │
└────────────────┘
SELECT toTypeName(toNullable(10))
┌─toTypeName(toNullable(10))─┐
│ Nullable(UInt8) │
└────────────────────────────┘
```

70
docs/en/query_language/functions/geo.md Normal file
View File

@ -0,0 +1,70 @@
# Functions for working with geographical coordinates
## greatCircleDistance
Calculate the distance between two points on the Earth's surface using [the great-circle formula](https://en.wikipedia.org/wiki/Great-circle_distance).
```
greatCircleDistance(lon1Deg, lat1Deg, lon2Deg, lat2Deg)
```
**Input parameters**
- `lon1Deg` — Latitude of the first point in degrees. Range: `[-90°, 90°]`.
- `lat1Deg` — Longitude of the first point in degrees. Range: `[-180°, 180°]`.
- `lon2Deg` — Latitude of the second point in degrees. Range: `[-90°, 90°]`.
- `lat2Deg` — Longitude of the second point in degrees. Range: `[-180°, 180°]`.
Positive values correspond to North latitude and East longitude, and negative values correspond to South latitude and West longitude.
**Returned value**
The distance between two points on the Earth's surface, in meters.
Generates an exception when the input parameter values fall outside of the range.
**Example**
```sql
SELECT greatCircleDistance(55.755831, 37.617673, -55.755831, -37.617673)
```
```text
┌─greatCircleDistance(55.755831, 37.617673, -55.755831, -37.617673)─┐
│ 14132374.194975413 │
└───────────────────────────────────────────────────────────────────┘
```
## pointInEllipses
Checks whether the point belongs to at least one of the ellipses.
```
pointInEllipses(x, y, x₀, y₀, a₀, b₀,...,xₙ, yₙ, aₙ, bₙ)
```
**Input parameters**
- `x` — Latitude of the point.
- `y` — Longitude of the point.
- `xᵢ, yᵢ` — Coordinates of the center of the `i`-th ellipsis.
- `aᵢ, bᵢ` — Axes of the `i`-th ellipsis in meters.
The input parameters must be `2+4⋅n`, where `n` is the number of ellipses.
**Returned values**
`1` if the point is inside at least one of the ellipses; `0`if it is not.
**Examples:**
```sql
SELECT pointInEllipses(55.755831, 37.617673, 55.755831, 37.617673, 1.0, 2.0)
```
```text
┌─pointInEllipses(55.755831, 37.617673, 55.755831, 37.617673, 1., 2.)─┐
│ 1 │
└─────────────────────────────────────────────────────────────────────┘
```

12
docs/en/query_language/functions/higher_order_functions.md
View File

@ -1,3 +1,5 @@
<a name="higher_order_functions"></a>
# Higher-order functions
## `->` operator, lambda(params, expr) function
@ -89,9 +91,9 @@ SELECT arrayCumSum([1, 1, 1, 1]) AS res
### arraySort(\[func,\] arr1, ...)
Returns an array as result of sorting the elements of `arr1` in ascending order. If the `func` function is specified, sorting order is determined by the result of the function `func` applied to the elements of array (arrays)
Returns the `arr1` array sorted in ascending order. If `func` is set, the sort order is determined by the result of the `func` function on the elements of the array or arrays.
The [Schwartzian transform](https://en.wikipedia.org/wiki/Schwartzian_transform) is used to impove sorting efficiency.
To improve sorting efficiency, we use the [Schwartzian Transform](https://en.wikipedia.org/wiki/%D0%9F%D1%80%D0%B5%D0%BE%D0%B1%D1%80%D0%B0%D0%B7%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%A8%D0%B2%D0%B0%D1%80%D1%86%D0%B0).
Example:
@ -107,9 +109,5 @@ SELECT arraySort((x, y) -> y, ['hello', 'world'], [2, 1]);
### arrayReverseSort(\[func,\] arr1, ...)
Returns an array as result of sorting the elements of `arr1` in descending order. If the `func` function is specified, sorting order is determined by the result of the function `func` applied to the elements of array (arrays)
Returns the `arr1` array sorted in descending order. If `func` is set, the sort order is determined by the result of the `func` function on the elements of the array or arrays.

7
docs/en/query_language/functions/index.md
View File

@ -27,6 +27,13 @@ A constant expression is also considered a constant (for example, the right half
Functions can be implemented in different ways for constant and non-constant arguments (different code is executed). But the results for a constant and for a true column containing only the same value should match each other.
## NULL processing
Functions have the following behaviors:
- If at least one of the arguments of the function is `NULL`, the function result is also `NULL`.
- Special behavior that is specified individually in the description of each function. In the ClickHouse source code, these functions have `UseDefaultImplementationForNulls=false`.
## Constancy
Functions can't change the values of their arguments – any changes are returned as the result. Thus, the result of calculating separate functions does not depend on the order in which the functions are written in the query.

10
docs/en/query_language/functions/math_functions.md
View File

@ -4,11 +4,11 @@ All the functions return a Float64 number. The accuracy of the result is close t
## e()
Returns a Float64 number close to the e number.
Returns a Float64 number that is close to the number e.
## pi()
Returns a Float64 number close to π.
Returns a Float64 number that is close to the number π.
## exp(x)
@ -20,7 +20,7 @@ Accepts a numeric argument and returns a Float64 number close to the natural log
## exp2(x)
Accepts a numeric argument and returns a Float64 number close to 2^x.
Accepts a numeric argument and returns a Float64 number close to 2 to the power of x.
## log2(x)
@ -28,7 +28,7 @@ Accepts a numeric argument and returns a Float64 number close to the binary loga
## exp10(x)
Accepts a numeric argument and returns a Float64 number close to 10^x.
Accepts a numeric argument and returns a Float64 number close to 10 to the power of x.
## log10(x)
@ -96,4 +96,4 @@ The arc tangent.
## pow(x, y)
Accepts two numeric arguments and returns a Float64 number close to x^y.
Takes two numeric arguments x and y. Returns a Float64 number close to x to the power of y.

284
docs/en/query_language/functions/other_functions.md
View File

@ -9,10 +9,22 @@ Returns a string with the name of the host that this function was performed on.
Calculates the approximate width when outputting values to the console in text format (tab-separated).
This function is used by the system for implementing Pretty formats.
`NULL` is represented as a string corresponding to `NULL` in `Pretty` formats.
```
SELECT visibleWidth(NULL)
┌─visibleWidth(NULL)─┐
│ 4 │
└────────────────────┘
```
## toTypeName(x)
Returns a string containing the type name of the passed argument.
If `NULL` is passed to the function as input, then it returns the `Nullable(Nothing)` type, which corresponds to an internal `NULL` representation in ClickHouse.
## blockSize()
Gets the size of the block.
@ -25,7 +37,7 @@ In ClickHouse, full columns and constants are represented differently in memory.
## ignore(...)
Accepts any arguments and always returns 0.
Accepts any arguments, including `NULL`. Always returns 0.
However, the argument is still evaluated. This can be used for benchmarks.
## sleep(seconds)
@ -63,9 +75,9 @@ Allows building a unicode-art diagram.
Parameters:
- `x` – Value to display.
- `min, max` – Integer constants. The value must fit in Int64.
- `width` – Constant, positive number, may be a fraction.
- `x` — Size to display.
- `min, max` — Integer constants. The value must fit in `Int64`.
- `width` — Constant, positive integer, can be fractional.
The band is drawn with accuracy to one eighth of a symbol.
@ -283,3 +295,267 @@ The inverse function of MACNumToString. If the MAC address has an invalid format
## MACStringToOUI(s)
Accepts a MAC address in the format AA:BB:CC:DD:EE:FF (colon-separated numbers in hexadecimal form). Returns the first three octets as a UInt64 number. If the MAC address has an invalid format, it returns 0.
## getSizeOfEnumType
Returns the number of fields in [Enum](../../data_types/enum.md#data_type-enum).
```
getSizeOfEnumType(value)
```
**Parameters:**
- `value` — Value of type `Enum`.
**Returned values**
- The number of fields with `Enum` input values.
- An exception is thrown if the type is not `Enum`.
**Example**
```
SELECT getSizeOfEnumType( CAST('a' AS Enum8('a' = 1, 'b' = 2) ) ) AS x
┌─x─┐
│ 2 │
└───┘
```
## toColumnTypeName
Returns the name of the class that represents the data type of the column in RAM.
```
toColumnTypeName(value)
```
**Parameters:**
- `value` — Any type of value.
**Returned values**
- A string with the name of the class that is used for representing the `value` data type in RAM.
**Example of the difference between` toTypeName ' and ' toColumnTypeName`**
```
:) select toTypeName(cast('2018-01-01 01:02:03' AS DateTime))
SELECT toTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))
┌─toTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ DateTime │
└─────────────────────────────────────────────────────┘
1 rows in set. Elapsed: 0.008 sec.
:) select toColumnTypeName(cast('2018-01-01 01:02:03' AS DateTime))
SELECT toColumnTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))
┌─toColumnTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ Const(UInt32) │
└───────────────────────────────────────────────────────────┘
```
The example shows that the `DateTime` data type is stored in memory as `Const(UInt32)`.
## dumpColumnStructure
Outputs a detailed description of data structures in RAM
```
dumpColumnStructure(value)
```
**Parameters:**
- `value` — Any type of value.
**Returned values**
- A string describing the structure that is used for representing the `value` data type in RAM.
**Example**
```
SELECT dumpColumnStructure(CAST('2018-01-01 01:02:03', 'DateTime'))
┌─dumpColumnStructure(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ DateTime, Const(size = 1, UInt32(size = 1)) │
└──────────────────────────────────────────────────────────────┘
```
## defaultValueOfArgumentType
Outputs the default value for the data type.
Does not include default values for custom columns set by the user.
```
defaultValueOfArgumentType(expression)
```
**Parameters:**
- `expression` — Arbitrary type of value or an expression that results in a value of an arbitrary type.
**Returned values**
- `0` for numbers.
- Empty string for strings.
- `ᴺᵁᴸᴸ` for [Nullable](../../data_types/nullable.md#data_type-nullable).
**Example**
```
:) SELECT defaultValueOfArgumentType( CAST(1 AS Int8) )
SELECT defaultValueOfArgumentType(CAST(1, 'Int8'))
┌─defaultValueOfArgumentType(CAST(1, 'Int8'))─┐
│ 0 │
└─────────────────────────────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
:) SELECT defaultValueOfArgumentType( CAST(1 AS Nullable(Int8) ) )
SELECT defaultValueOfArgumentType(CAST(1, 'Nullable(Int8)'))
┌─defaultValueOfArgumentType(CAST(1, 'Nullable(Int8)'))─┐
│ ᴺᵁᴸᴸ │
└───────────────────────────────────────────────────────┘
1 rows in set. Elapsed: 0.002 sec.
```
## indexHint
Outputs data in the range selected by the index without filtering by the expression specified as an argument.
The expression passed to the function is not calculated, but ClickHouse applies the index to this expression in the same way as if the expression was in the query without `indexHint`.
**Returned value**
- 1.
**Example**
Here is a table with the test data for [ontime](../../getting_started/example_datasets/ontime.md#example_datasets-ontime).
```
SELECT count() FROM ontime
┌─count()─┐
│ 4276457 │
└─────────┘
```
The table has indexes for the fields `(FlightDate, (Year, FlightDate))`.
Create a selection by date like this:
```
:) SELECT FlightDate AS k, count() FROM ontime GROUP BY k ORDER BY k
SELECT
FlightDate AS k,
count()
FROM ontime
GROUP BY k
ORDER BY k ASC
┌──────────k─┬─count()─┐
│ 2017-01-01 │ 13970 │
│ 2017-01-02 │ 15882 │
........................
│ 2017-09-28 │ 16411 │
│ 2017-09-29 │ 16384 │
│ 2017-09-30 │ 12520 │
└────────────┴─────────┘
273 rows in set. Elapsed: 0.072 sec. Processed 4.28 million rows, 8.55 MB (59.00 million rows/s., 118.01 MB/s.)
```
In this selection, the index is not used and ClickHouse processed the entire table (`Processed 4.28 million rows`). To apply the index, select a specific date and run the following query:
```
:) SELECT FlightDate AS k, count() FROM ontime WHERE k = '2017-09-15' GROUP BY k ORDER BY k
SELECT
FlightDate AS k,
count()
FROM ontime
WHERE k = '2017-09-15'
GROUP BY k
ORDER BY k ASC
┌──────────k─┬─count()─┐
│ 2017-09-15 │ 16428 │
└────────────┴─────────┘
1 rows in set. Elapsed: 0.014 sec. Processed 32.74 thousand rows, 65.49 KB (2.31 million rows/s., 4.63 MB/s.)
```
The last line of output shows that by using the index, ClickHouse processed a significantly smaller number of rows (`Processed 32.74 thousand rows`).
Now pass the expression `k = '2017-09-15'` to the `indexHint` function:
```
:) SELECT FlightDate AS k, count() FROM ontime WHERE indexHint(k = '2017-09-15') GROUP BY k ORDER BY k
SELECT
FlightDate AS k,
count()
FROM ontime
WHERE indexHint(k = '2017-09-15')
GROUP BY k
ORDER BY k ASC
┌──────────k─┬─count()─┐
│ 2017-09-14 │ 7071 │
│ 2017-09-15 │ 16428 │
│ 2017-09-16 │ 1077 │
│ 2017-09-30 │ 8167 │
└────────────┴─────────┘
4 rows in set. Elapsed: 0.004 sec. Processed 32.74 thousand rows, 65.49 KB (8.97 million rows/s., 17.94 MB/s.)
```
The response to the request shows that ClickHouse applied the index in the same way as the previous time (`Processed 32.74 thousand rows`). However, the resulting set of rows shows that the expression `k = '2017-09-15'` was not used when generating the result.
Because the index is sparse in ClickHouse, "extra" data ends up in the response when reading a range (in this case, the adjacent dates). Use the `indexHint` function to see it.
## replicate
Creates an array with a single value.
Used for internal implementation of [arrayJoin](array_join.md#functions_arrayjoin).
```
replicate(x, arr)
```
**Parameters:**
- `arr` — Original array. ClickHouse creates a new array of the same length as the original and fills it with the value `x`.
- `x` — The value that the resulting array will be filled with.
**Output value**
- An array filled with the value `x`.
**Example**
```
SELECT replicate(1, ['a', 'b', 'c'])
┌─replicate(1, ['a', 'b', 'c'])─┐
│ [1,1,1] │
└───────────────────────────────┘
```

7
docs/en/query_language/functions/string_search_functions.md
View File

@ -5,16 +5,16 @@ The search substring or regular expression must be a constant in all these funct
## position(haystack, needle)
Search for the `needle` substring in the `haystack` string.
Search for the substring `needle` in the string `haystack`.
Returns the position (in bytes) of the found substring, starting from 1, or returns 0 if the substring was not found.
For case-insensitive search use `positionCaseInsensitive` function.
For a case-insensitive search, use the function `positionCaseInsensitive`.
## positionUTF8(haystack, needle)
The same as `position`, but the position is returned in Unicode code points. Works under the assumption that the string contains a set of bytes representing a UTF-8 encoded text. If this assumption is not met, it returns some result (it doesn't throw an exception).
For case-insensitive search use `positionCaseInsensitiveUTF8` function.
For a case-insensitive search, use the function `positionCaseInsensitiveUTF8`.
## match(haystack, pattern)
@ -51,3 +51,4 @@ For other regular expressions, the code is the same as for the 'match' function.
## notLike(haystack, pattern), haystack NOT LIKE pattern operator
The same thing as 'like', but negative.

18
docs/en/query_language/functions/type_conversion_functions.md
View File

@ -119,3 +119,21 @@ SELECT
Conversion to FixedString(N) only works for arguments of type String or FixedString(N).
Type conversion to [Nullable](../../data_types/nullable.md#data_type-nullable) and back is supported. Example:
```
SELECT toTypeName(x) FROM t_null
┌─toTypeName(x)─┐
│ Int8 │
│ Int8 │
└───────────────┘
SELECT toTypeName(CAST(x, 'Nullable(UInt16)')) FROM t_null
┌─toTypeName(CAST(x, 'Nullable(UInt16)'))─┐
│ Nullable(UInt16) │
│ Nullable(UInt16) │
└─────────────────────────────────────────┘
```

32
docs/en/query_language/functions/ym_dict_functions.md
View File

@ -1,3 +1,5 @@
<a name="ym_dict_functions"></a>
# Functions for working with Yandex.Metrica dictionaries
In order for the functions below to work, the server config must specify the paths and addresses for getting all the Yandex.Metrica dictionaries. The dictionaries are loaded at the first call of any of these functions. If the reference lists can't be loaded, an exception is thrown.
@ -21,9 +23,9 @@ All functions for working with regions have an optional argument at the end –
Example:
```text
regionToCountry(RegionID) – Uses the default dictionary: /opt/geo/regions_hierarchy.txt
regionToCountry(RegionID, '') – Uses the default dictionary: /opt/geo/regions_hierarchy.txt
regionToCountry(RegionID, 'ua') – Uses the dictionary for the 'ua' key: /opt/geo/regions_hierarchy_ua.txt
regionToCountry (RegionID) — Uses the default dictionary: /opt/geo/regions_hierarchy.txt.
regionToCountry (RegionID, '') — Uses the default dictionary: /opt/geo/regions_hierarchy.txt.
regionToCountry (RegionID, 'ua') — Uses the dictionary for the ua key: /opt/geo/regions_hierarchy_ua.txt.
```
### regionToCity(id[, geobase])
@ -45,13 +47,13 @@ LIMIT 15
│ │
│ Moscow and Moscow region │
│ St. Petersburg and Leningrad region │
│ Belgorod region │
│ Ivanovsk region │
│ Belogorod region │
│ Ivanovo region │
│ Kaluga region │
│ Kostroma region │
│ Kursk region │
│ Lipetsk region │
│ Orlov region │
│ Oryol region │
│ Ryazan region │
│ Smolensk region │
│ Tambov region │
@ -73,19 +75,19 @@ LIMIT 15
```text
┌─regionToName(regionToDistrict(toUInt32(number), \'ua\'))─┐
│ │
│ Central federal district │
│ Northwest federal district │
│ South federal district │
│ North Caucases federal district │
│ Privolga federal district │
│ Ural federal district │
│ Siberian federal district │
│ Far East federal district │
│ Central Federal District │
│ Northwest Federal District │
│ Southern Federal District │
│ North Caucasian Federal District │
│ Privolzhsky Federal District │
│ Ural Federal District │
│ Siberian Federal District │
│ Far East Federal District │
│ Scotland │
│ Faroe Islands │
│ Flemish region │
│ Brussels capital region │
│ Wallonia │
│ Walloon │
│ Federation of Bosnia and Herzegovina │
└──────────────────────────────────────────────────────────┘
```

3
docs/en/query_language/index.md
View File

@ -4,4 +4,5 @@
* [INSERT INTO](insert_into.md#queries-insert)
* [CREATE](create.md#create-database)
* [ALTER](alter.md#query_language_queries_alter)
* [Other kinds of queries](misc.md#miscellaneous-queries)
* [Other types of queries](misc.md#miscellanous-queries)

11
docs/en/query_language/misc.md
View File

@ -6,9 +6,10 @@ This query is exactly the same as `CREATE`, but
- instead of the word `CREATE` it uses the word `ATTACH`.
- The query doesn't create data on the disk, but assumes that data is already in the appropriate places, and just adds information about the table to the server.
After executing an ATTACH query, the server will know about the existence of the table.
If the table was previously detached (``DETACH``), meaning that its structure is known, you can use shorthand without defining the structure.
After executing an `ATTACH` query, the server will know about the existence of the table.
If the table was previously detached (`DETACH`), meaning that its structure is known, you can use shorthand without defining the structure.
```sql
ATTACH TABLE [IF NOT EXISTS] [db.]name [ON CLUSTER cluster]
@ -175,8 +176,8 @@ Supported only by `*MergeTree` engines, in which this query initializes a non-sc
If you specify a `PARTITION`, only the specified partition will be optimized.
If you specify `FINAL`, optimization will be performed even when all the data is already in one part.
!!! warning
OPTIMIZE can't fix the "Too many parts" error.
!!! Important:
The OPTIMIZE query can't fix the cause of the "Too many parts" error.
## KILL QUERY
@ -193,7 +194,7 @@ The queries to terminate are selected from the system.processes table using the
Examples:
```sql
-- Forcibly terminates all queries with the specified query_id:
-- Terminates all queries with the specified query_id:
KILL QUERY WHERE query_id='2-857d-4a57-9ee0-327da5d60a90'
-- Synchronously terminates all queries run by 'username':

53
docs/en/query_language/operators.md
View File

@ -81,7 +81,9 @@ Groups of operators are listed in order of priority (the higher it is in the lis
Note:
The conditional operator calculates the values of b and c, then checks whether condition a is met, and then returns the corresponding value. If "b" or "c" is an arrayJoin() function, each row will be replicated regardless of the "a" condition.
The conditional operator calculates the values of b and c, then checks whether condition a is met, and then returns the corresponding value. If `b` or `C` is an [arrayJoin()](functions/array_join.md#functions_arrayjoin) function, each row will be replicated regardless of the "a" condition.
<a name="operator_case"><a>
## Conditional Expression
@ -120,3 +122,52 @@ Sometimes this doesn't work the way you expect. For example, ` SELECT 4 > 2 > 3`
For efficiency, the `and` and `or` functions accept any number of arguments. The corresponding chains of `AND` and `OR` operators are transformed to a single call of these functions.
## Checking for `NULL`
ClickHouse supports the `IS NULL` and `IS NOT NULL` operators.
<a name="operator-is-null"></a>
### IS NULL
- For [Nullable](../data_types/nullable.md#data_type-nullable) type values, the `IS NULL` operator returns:
- `1`, if the value is `NULL`.
- `0` otherwise.
- For other values, the `IS NULL` operator always returns `0`.
```bash
:) SELECT x+100 FROM t_null WHERE y IS NULL
SELECT x + 100
FROM t_null
WHERE isNull(y)
┌─plus(x, 100)─┐
│ 101 │
└──────────────┘
1 rows in set. Elapsed: 0.002 sec.
```
<a name="operator-is-not-null"></a>
### IS NOT NULL
- For [Nullable](../data_types/nullable.md#data_type-nullable) type values, the `IS NOT NULL` operator returns:
- `0`, if the value is `NULL`.
- `1` otherwise.
- For other values, the `IS NOT NULL` operator always returns `1`.
```bash
:) SELECT * FROM t_null WHERE y IS NOT NULL
SELECT *
FROM t_null
WHERE isNotNull(y)
┌─x─┬─y─┐
│ 2 │ 3 │
└───┴───┘
1 rows in set. Elapsed: 0.002 sec.
```

210
docs/en/query_language/select.md
View File

@ -1,6 +1,6 @@
# SELECT Queries Syntax
`SELECT` performs data retrieval.
`SELECT` forms samples of the data.
```sql
SELECT [DISTINCT] expr_list
@ -26,7 +26,9 @@ The clauses below are described in almost the same order as in the query executi
If the query omits the `DISTINCT`, `GROUP BY` and `ORDER BY` clauses and the `IN` and `JOIN` subqueries, the query will be completely stream processed, using O(1) amount of RAM.
Otherwise, the query might consume a lot of RAM if the appropriate restrictions are not specified: `max_memory_usage`, `max_rows_to_group_by`, `max_rows_to_sort`, `max_rows_in_distinct`, `max_bytes_in_distinct`, `max_rows_in_set`, `max_bytes_in_set`, `max_rows_in_join`, `max_bytes_in_join`, `max_bytes_before_external_sort`, `max_bytes_before_external_group_by`. For more information, see the section "Settings". It is possible to use external sorting (saving temporary tables to a disk) and external aggregation. `The system does not have "merge join"`.
### FROM Clause
<a name="query_language-section-from"></a>
### FROM clause
If the FROM clause is omitted, data will be read from the `system.one` table.
The 'system.one' table contains exactly one row (this table fulfills the same purpose as the DUAL table found in other DBMSs).
@ -332,7 +334,9 @@ The query can only specify a single ARRAY JOIN clause.
The corresponding conversion can be performed before the WHERE/PREWHERE clause (if its result is needed in this clause), or after completing WHERE/PREWHERE (to reduce the volume of calculations).
### JOIN Clause
<a name="query_language-join"></a>
### JOIN clause
The normal JOIN, which is not related to ARRAY JOIN described above.
@ -426,29 +430,58 @@ Among the various types of JOINs, the most efficient is ANY LEFT JOIN, then ANY
If you need a JOIN for joining with dimension tables (these are relatively small tables that contain dimension properties, such as names for advertising campaigns), a JOIN might not be very convenient due to the bulky syntax and the fact that the right table is re-accessed for every query. For such cases, there is an "external dictionaries" feature that you should use instead of JOIN. For more information, see the section "External dictionaries".
### WHERE Clause
#### NULL processing
If there is a WHERE clause, it must contain an expression with the UInt8 type. This is usually an expression with comparison and logical operators.
This expression will be used for filtering data before all other transformations.
The JOIN behavior is affected by the [join_use_nulls](../operations/settings/settings.md#settings-join_use_nulls) setting. With `join_use_nulls=1,` `JOIN` works like in standard SQL.
If indexes are supported by the database table engine, the expression is evaluated on the ability to use indexes.
If the JOIN keys are [Nullable](../data_types/nullable.md#data_types-nullable) fields, the rows where at least one of the keys has the value [NULL](syntax.md#null-literal) are not joined.
<a name="query_language-queries-where"></a>
### WHERE clause
Allows you to set an expression that ClickHouse uses to filter data before all other actions in the query, other than the expressions contained in the [PREWHERE](#query_language-queries-prewhere) clause. This is usually an expression with logical operators.
The result of the expression must be of type `UInt8`.
ClickHouse uses indexes in the expression if this is allowed by the [table engine](../operations/table_engines/index.md#table_engines).
If [NULL](syntax.md#null-literal) must be checked in the clause, then use the [IS NULL](operators.md#operator-is-null) and [IS NOT NULL](operators.md#operator-is-not-null) operators and the related `isNull` and `isNotNull` functions. Otherwise, the expression will always be considered as not executed.
Example of checking for `NULL`:
```bash
:) SELECT * FROM t_null WHERE y IS NULL
SELECT *
FROM t_null
WHERE isNull(y)
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
└───┴──────┘
1 rows in set. Elapsed: 0.002 sec.
```
<a name="query_language-queries-prewhere"></a>
### PREWHERE Clause
This clause has the same meaning as the WHERE clause. The difference is in which data is read from the table.
When using PREWHERE, first only the columns necessary for executing PREWHERE are read. Then the other columns are read that are needed for running the query, but only those blocks where the PREWHERE expression is true.
It has the same purpose as the [WHERE](#query_language-queries-where) clause. The difference is in which data is read from the table.
When using `PREWHERE`, first only the columns necessary for executing `PREWHERE` are read. Then the other columns are read that are needed for running the query, but only those blocks where the `PREWHERE` expression is true.
It makes sense to use PREWHERE if there are filtration conditions that are not suitable for indexes that are used by a minority of the columns in the query, but that provide strong data filtration. This reduces the volume of data to read.
`PREWHERE` makes sense if there are filtration conditions that are not suitable for indexes that are used by a minority of the columns in the query, but that provide strong data filtration. This reduces the volume of data to read.
For example, it is useful to write PREWHERE for queries that extract a large number of columns, but that only have filtration for a few columns.
For example, it is useful to write `PREWHERE` for queries that extract a large number of columns, but that only have filtration for a few columns.
PREWHERE is only supported by tables from the `*MergeTree` family.
`PREWHERE` is only supported by tables from the `*MergeTree` family.
A query may simultaneously specify PREWHERE and WHERE. In this case, PREWHERE precedes WHERE.
A query may simultaneously specify `PREWHERE` and `WHERE`. In this case, `PREWHERE` goes before `WHERE`.
Keep in mind that it does not make much sense for PREWHERE to only specify those columns that have an index, because when using an index, only the data blocks that match the index are read.
Keep in mind that it does not make much sense for `PREWHERE` to only specify those columns that have an index, because when using an index, only the data blocks that match the index are read.
If the 'optimize_move_to_prewhere' setting is set to 1 and PREWHERE is omitted, the system uses heuristics to automatically move parts of expressions from WHERE to PREWHERE.
If the setting `optimize_move_to_prewhere` is set to `1`, in the absence of `PREWHERE`, the system will automatically move parts of expressions from `WHERE` to `PREWHERE` according to heuristic analysis.
### GROUP BY Clause
@ -490,7 +523,39 @@ GROUP BY is not supported for array columns.
A constant can't be specified as arguments for aggregate functions. Example: sum(1). Instead of this, you can get rid of the constant. Example: `count()`.
#### WITH TOTALS Modifier
#### NULL processing
For grouping, ClickHouse interprets [NULL](syntax.md#null-literal) as a value, and `NULL=NULL`.
Here's an example to show what this means.
Assume you have this table:
```
┌─x─┬────y─┐
│ 1 │ 2 │
│ 2 │ ᴺᵁᴸᴸ │
│ 3 │ 2 │
│ 3 │ 3 │
│ 3 │ ᴺᵁᴸᴸ │
└───┴──────┘
```
The query `SELECT sum(x), y FROM t_null_big GROUP BY y` results in:
```
┌─sum(x)─┬────y─┐
│ 4 │ 2 │
│ 3 │ 3 │
│ 5 │ ᴺᵁᴸᴸ │
└────────┴──────┘
```
You can see that `GROUP BY` for `У = NULL` summed up `x`, as if `NULL` is this value.
If you pass several keys to `GROUP BY`, the result will give you all the combinations of the selection, as if `NULL` were a specific value.
#### WITH TOTALS modifier
If the WITH TOTALS modifier is specified, another row will be calculated. This row will have key columns containing default values (zeros or empty lines), and columns of aggregate functions with the values calculated across all the rows (the "total" values).
@ -535,7 +600,7 @@ But if the ORDER BY doesn't have LIMIT, don't forget to enable external sorting
### LIMIT N BY Clause
LIMIT N BY COLUMNS selects the top N rows for each group of COLUMNS. LIMIT N BY is not related to LIMIT; they can both be used in the same query. The key for LIMIT N BY can contain any number of columns or expressions.
`LIMIT N BY COLUMNS` selects the top `N` rows for each group of `COLUMNS`. `LIMIT N BY` is not related to `LIMIT`; they can both be used in the same query. The key for `LIMIT N BY` can contain any number of columns or expressions.
Example:
@ -554,7 +619,9 @@ LIMIT 100
The query will select the top 5 referrers for each `domain, device_type` pair, but not more than 100 rows (`LIMIT n BY + LIMIT`).
### HAVING Clause
`LIMIT n BY` works with [NULL](syntax.md#null-literal) as if it were a specific value. This means that as the result of the query, the user will get all the combinations of fields specified in `BY`.
### HAVING clause
Allows filtering the result received after GROUP BY, similar to the WHERE clause.
WHERE and HAVING differ in that WHERE is performed before aggregation (GROUP BY), while HAVING is performed after it.
@ -573,7 +640,47 @@ We only recommend using COLLATE for final sorting of a small number of rows, sin
Rows that have identical values for the list of sorting expressions are output in an arbitrary order, which can also be nondeterministic (different each time).
If the ORDER BY clause is omitted, the order of the rows is also undefined, and may be nondeterministic as well.
When floating point numbers are sorted, NaNs are separate from the other values. Regardless of the sorting order, NaNs come at the end. In other words, for ascending sorting they are placed as if they are larger than all the other numbers, while for descending sorting they are placed as if they are smaller than the rest.
`NaN` and `NULL` sorting order:
- With the modifier `NULLS FIRST` — First `NULL`, then `NaN`, then other values.
- With the modifier `NULLS LAST` — First the values, then `NaN`, then `NULL`.
- Default — The same as with the `NULLS LAST` modifier.
Example:
For the table
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 2 │
│ 1 │ nan │
│ 2 │ 2 │
│ 3 │ 4 │
│ 5 │ 6 │
│ 6 │ nan │
│ 7 │ ᴺᵁᴸᴸ │
│ 6 │ 7 │
│ 8 │ 9 │
└───┴──────┘
```
Run the query `SELECT * FROM t_null_nan ORDER BY y NULLS FIRST` to get:
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 7 │ ᴺᵁᴸᴸ │
│ 1 │ nan │
│ 6 │ nan │
│ 2 │ 2 │
│ 2 │ 2 │
│ 3 │ 4 │
│ 5 │ 6 │
│ 6 │ 7 │
│ 8 │ 9 │
└───┴──────┘
```
Less RAM is used if a small enough LIMIT is specified in addition to ORDER BY. Otherwise, the amount of memory spent is proportional to the volume of data for sorting. For distributed query processing, if GROUP BY is omitted, sorting is partially done on remote servers, and the results are merged on the requestor server. This means that for distributed sorting, the volume of data to sort can be greater than the amount of memory on a single server.
@ -592,14 +699,16 @@ These expressions work as if they are applied to separate rows in the result.
### DISTINCT Clause
If DISTINCT is specified, only a single row will remain out of all the sets of fully matching rows in the result.
The result will be the same as if GROUP BY were specified across all the fields specified in SELECT without aggregate functions. But there are several differences from GROUP BY:
If `DISTINCT` is specified, only a single row will remain out of all the sets of fully matching rows in the result.
The result will be the same as if `GROUP BY` were specified across all the fields specified in `SELECT` without aggregate functions. But there are several differences from `GROUP BY`:
- DISTINCT can be applied together with GROUP BY.
- When ORDER BY is omitted and LIMIT is defined, the query stops running immediately after the required number of different rows has been read.
- `DISTINCT` can be applied together with `GROUP BY`.
- When `ORDER BY` is omitted and `LIMIT` is defined, the query stops running immediately after the required number of different rows has been read.
- Data blocks are output as they are processed, without waiting for the entire query to finish running.
DISTINCT is not supported if SELECT has at least one array column.
`DISTINCT` is not supported if `SELECT` has at least one array column.
`DISTINCT` works with [NULL](syntax.md#null-literal) as if `NULL` were a specific value, and `NULL=NULL`. In other words, in the `DISTINCT` results, different combinations with `NULL` only occur once.
### LIMIT Clause
@ -610,9 +719,11 @@ LIMIT n, m allows you to select the first 'm' rows from the result after skippin
If there isn't an ORDER BY clause that explicitly sorts results, the result may be arbitrary and nondeterministic.
### UNION ALL Clause
`DISTINCT` works with [NULL](syntax.md#null-literal) as if `NULL` were a specific value, and `NULL=NULL`. In other words, in the `DISTINCT` results, different combinations with `NULL` only occur once.
You can use UNION ALL to combine any number of queries. Example:
### UNION ALL clause
You can use `UNION ALL` to combine any number of queries. Example:
```sql
SELECT CounterID, 1 AS table, toInt64(count()) AS c
@ -627,13 +738,13 @@ SELECT CounterID, 2 AS table, sum(Sign) AS c
HAVING c > 0
```
Only UNION ALL is supported. The regular UNION (UNION DISTINCT) is not supported. If you need UNION DISTINCT, you can write SELECT DISTINCT from a subquery containing UNION ALL.
Only `UNION ALL` is supported. The normal `UNION` (`UNION DISTINCT`) is not supported. If you need `UNION DISTINCT`, you can write `SELECT DISTINCT` from a subquery containing `UNION ALL`.
Queries that are parts of UNION ALL can be run simultaneously, and their results can be mixed together.
Queries that are part of a `UNION ALL` can be run in parallel and their results might be mixed together when returned.
The structure of results (the number and type of columns) must match for the queries. But the column names can differ. In this case, the column names for the final result will be taken from the first query.
The structure of results (the number and type of columns) must match for the queries. But the column names can differ. In this case, the column names for the final result will be taken from the first query. Type casting is performed for unions. For example, if two queries being combined have the same field with non-`Nullable` and `Nullable` types from a compatible type, the resulting `UNION ALL` has a `Nullable` type field.
Queries that are parts of UNION ALL can't be enclosed in brackets. ORDER BY and LIMIT are applied to separate queries, not to the final result. If you need to apply a conversion to the final result, you can put all the queries with UNION ALL in a subquery in the FROM clause.
Queries that are parts of `UNION ALL` can't be enclosed in brackets. `ORDER BY` and `LIMIT` are applied to separate queries, not to the final result. If you need to apply a conversion to the final result, you can put all the queries with `UNION ALL` in a subquery in the `FROM` clause.
### INTO OUTFILE Clause
@ -652,7 +763,9 @@ If the FORMAT clause is omitted, the default format is used, which depends on bo
When using the command-line client, data is passed to the client in an internal efficient format. The client independently interprets the FORMAT clause of the query and formats the data itself (thus relieving the network and the server from the load).
### IN Operators
<a name="query_language-in_operators"></a>
### IN operators
The `IN`, `NOT IN`, `GLOBAL IN`, and `GLOBAL NOT IN` operators are covered separately, since their functionality is quite rich.
@ -716,13 +829,46 @@ ORDER BY EventDate ASC
For each day after March 17th, count the percentage of pageviews made by users who visited the site on March 17th.
A subquery in the IN clause is always run just one time on a single server. There are no dependent subqueries.
#### NULL processing
During request processing, the IN operator assumes that the result of an operation with [NULL](syntax.md#null-literal) is always equal to `0`, regardless of whether `NULL` is on the right or left side of the operator. `NULL` values are not included in any dataset, do not correspond to each other and cannot be compared.
Here is an example with the `t_null` table:
```
┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │ 3 │
└───┴──────┘
```
Running the query `SELECT x FROM t_null WHERE y IN (NULL,3)` gives you the following result:
```
┌─x─┐
│ 2 │
└───┘
```
You can see that the row in which `y = NULL` is thrown out of the query results. This is because ClickHouse can't decide whether `NULL` is included in the `(NULL,3)` set, returns `0` as the result of the operation, and `SELECT` excludes this row from the final output.
```
SELECT y IN (NULL, 3)
FROM t_null
┌─in(y, tuple(NULL, 3))─┐
│ 0 │
│ 1 │
└───────────────────────┘
```
<a name="queries-distributed-subrequests"></a>
#### Distributed Subqueries
There are two options for IN-s with subqueries (similar to JOINs): normal `IN` / `OIN` and `IN GLOBAL` / `GLOBAL JOIN`. They differ in how they are run for distributed query processing.
!!! attention
!!! Attention:
Remember that the algorithms described below may work differently depending on the [settings](../operations/settings/settings.md#settings-distributed_product_mode) `distributed_product_mode` setting.
When using the regular IN, the query is sent to remote servers, and each of them runs the subqueries in the `IN` or `JOIN` clause.

35
docs/en/query_language/syntax.md
View File

@ -35,34 +35,32 @@ Keywords (such as `SELECT`) are not case-sensitive. Everything else (column name
Identifiers (column names, functions, and data types) can be quoted or non-quoted.
Non-quoted identifiers start with a Latin letter or underscore, and continue with a Latin letter, underscore, or number. In other words, they must match the regex `^[a-zA-Z_][0-9a-zA-Z_]*$`. Examples: `x, _1, X_y__Z123_.`
Quoted identifiers are placed in reversed quotation marks `` `id` `` (the same as in MySQL), and can indicate any set of bytes (non-empty). In addition, symbols (for example, the reverse quotation mark) inside this type of identifier can be backslash-escaped. Escaping rules are the same as for string literals (see below).
Quoted identifiers are placed in reversed quotation marks ` `id` ` (the same as in MySQL), and can indicate any set of bytes (non-empty). In addition, symbols (for example, the reverse quotation mark) inside this type of identifier can be backslash-escaped. Escaping rules are the same as for string literals (see below).
We recommend using identifiers that do not need to be quoted.
## Literals
There are numeric literals, string literals, and compound literals.
### Numeric Literals
### Numeric
A numeric literal tries to be parsed:
- First as a 64-bit signed number, using the 'strtoull' function.
- If unsuccessful, as a 64-bit unsigned number, using the 'strtoll' function.
- If unsuccessful, as a floating-point number using the 'strtod' function.
- First as a 64-bit signed number, using the `strtoull` function.
- If unsuccessful, as a 64-bit unsigned number, using the `strtoll` function.
- If unsuccessful, as a floating-point number using the `strtod` function.
- Otherwise, an error is returned.
The corresponding value will have the smallest type that the value fits in.
For example, 1 is parsed as UInt8, but 256 is parsed as UInt16. For more information, see "Data types".
For example, 1 is parsed as `UInt8`, and 256 is parsed as `UInt16`. For more information, see the section [Data types](../data_types/index.md#data_types).
Examples: `1`, `18446744073709551615`, `0xDEADBEEF`, `01`, `0.1`, `1e100`, `-1e-100`, `inf`, `nan`.
### String Literals
### String
Only string literals in single quotes are supported. The enclosed characters can be backslash-escaped. The following escape sequences have a corresponding special value: `\b`, `\f`, `\r`, `\n`, `\t`, `\0`, `\a`, `\v`, `\xHH`. In all other cases, escape sequences in the format `\c`, where "c" is any character, are converted to "c". This means that you can use the sequences `\'`and`\\`. The value will have the String type.
Only string literals in single quotes are supported. The enclosed characters can be backslash-escaped. The following escape sequences have a corresponding special value: `\b`, `\f`, `\r`, `\n`, `\t`, `\0`, `\a`, `\v`, `\xHH`. In all other cases, escape sequences in the format `\c`, where "c" is any character, are converted to "c". This means that you can use the sequences `\'`and`\\`. The value will be of type [String](../data_types/string.md#data_types-string).
The minimum set of characters that you need to escape in string literals: `'` and `\`.
### Compound Literals
### Compound
Constructions are supported for arrays: `[1, 2, 3]` and tuples: `(1, 'Hello, world!', 2)`..
Actually, these are not literals, but expressions with the array creation operator and the tuple creation operator, respectively.
@ -70,6 +68,20 @@ For more information, see the section "Operators2".
An array must consist of at least one item, and a tuple must have at least two items.
Tuples have a special purpose for use in the IN clause of a SELECT query. Tuples can be obtained as the result of a query, but they can't be saved to a database (with the exception of Memory-type tables).
<a name="null-literal"></a>
### NULL
Indicates that the value is missing.
In order to store `NULL` in a table field, it must be of the [Nullable](../data_types/nullable.md#data_type-nullable) type.
Depending on the data format (input or output), `NULL` may have a different representation. For more information, see the documentation for [data formats](../interfaces/formats.md#formats).
There are many nuances to processing `NULL`. For example, if at least one of the arguments of a comparison operation is `NULL`, the result of this operation will also be `NULL`. The same is true for multiplication, addition, and other operations. For more information, read the documentation for each operation.
In queries, you can check `NULL` using the [IS NULL](operators.md#operator-is-null) and [IS NOT NULL](operators.md#operator-is-not-null) operators and the related functions `isNull` and `isNotNull`.
## Functions
Functions are written like an identifier with a list of arguments (possibly empty) in brackets. In contrast to standard SQL, the brackets are required, even for an empty arguments list. Example: `now()`.
@ -104,4 +116,3 @@ In a `SELECT` query, an asterisk can replace the expression. For more informatio
An expression is a function, identifier, literal, application of an operator, expression in brackets, subquery, or asterisk. It can also contain a synonym.
A list of expressions is one or more expressions separated by commas.
Functions and operators, in turn, can have expressions as arguments.

11
docs/en/query_language/table_functions/file.md
View File

@ -2,17 +2,18 @@
# file
`file(path, format, structure)` - returns a table created from a path file with a format type, with columns specified in structure.
`file(path, format, structure` — Returns a table with the columns specified in the structure created from the file at path in the specified format.
path - a relative path to a file from [user_files_path](../../operations/server_settings/settings.md#user_files_path).
path — The relative path to the file from [user_files_path](../../operations/server_settings/settings.md#user_files_path).
format - file [format](../../interfaces/formats.md#formats).
format — The file data [format](../../interfaces/formats.md#formats).
structure - table structure in 'UserID UInt64, URL String' format. Determines column names and types.
structure — The structure of the table in the format 'UserID UInt64, URL String'. Defines the column names and types.
**Example**
```sql
-- getting the first 10 lines of a table that contains 3 columns of UInt32 type from a CSV file
-- Get the first 10 rows of a table consisting of three UInt32 columns from a CSV file.
SELECT * FROM file('test.csv', 'CSV', 'column1 UInt32, column2 UInt32, column3 UInt32') LIMIT 10
```

8
docs/en/query_language/table_functions/numbers.md
View File

@ -1,9 +1,9 @@
# numbers
`numbers(N)` – Returns a table with the single 'number' column (UInt64) that contains integers from 0 to N-1.
`numbers(N, M)` - Returns a table with the single 'number' column (UInt64) that contains integers from N to (N + M - 1).
`numbers(N)` — Returns a table with the single `number` column (UInt64) that contains integers from `0` to `N-1`.
`numbers(N, M)` — Returns a table with the single `number` column (UInt64) that contains integers from `N` to `(N + M - 1)`.
Similar to the `system.numbers` table, it can be used for testing and generating successive values, `numbers(N, M)` more efficient than `system.numbers`.
Similar to the `system.numbers` table, it can be used for testing and generating successive values. The `numbers(N, M)` function is more efficient than a selection from `system.numbers`.
The following queries are equivalent:
@ -16,7 +16,7 @@ SELECT * FROM system.numbers LIMIT 10;
Examples:
```sql
-- Generate a sequence of dates from 2010-01-01 to 2010-12-31
-- generate a sequence of all dates from 2010-01-01 to 2010-12-31
select toDate('2010-01-01') + number as d FROM numbers(365);
```

7
docs/en/query_language/table_functions/remote.md
View File

@ -13,8 +13,8 @@ remote('addresses_expr', db.table[, 'user'[, 'password']])
`addresses_expr` – An expression that generates addresses of remote servers. This may be just one server address. The server address is `host:port`, or just `host`. The host can be specified as the server name, or as the IPv4 or IPv6 address. An IPv6 address is specified in square brackets. The port is the TCP port on the remote server. If the port is omitted, it uses `tcp_port` from the server's config file (by default, 9000).
!!! important
The port is required for an IPv6 address.
!!! Important:
With an IPv6 address, you must specify the port.
Examples:
@ -49,7 +49,7 @@ example01-{01..02}-1
If you have multiple pairs of curly brackets, it generates the direct product of the corresponding sets.
Addresses and parts of addresses in curly brackets can be separated by the pipe symbol (|). In this case, the corresponding sets of addresses are interpreted as replicas, and the query will be sent to the first healthy replica. However, the replicas are iterated in the order currently set in the [load_balancing](../../operations/settings/settings.md#settings-load_balancing) setting.
Addresses and parts of addresses in curly brackets can be separated by the pipe symbol (|). In this case, the corresponding sets of addresses are interpreted as replicas, and the query will be sent to the first healthy replica. The replicas are evaluated in the order currently set in the [load_balancing](../../operations/settings/settings.md#settings-load_balancing).
Example:
@ -72,4 +72,3 @@ The `remote` table function can be useful in the following cases:
If the user is not specified, `default` is used.
If the password is not specified, an empty password is used.

14
docs/en/query_language/table_functions/url.md
View File

@ -2,18 +2,20 @@
# url
`url(URL, format, structure)` - returns a table created from the `URL` with given
`format` and `structure`.
`url(URL, format, structure)` — Returns a table with the columns specified in
`structure`, created from data located at `URL` in the specified `format`.
URL - HTTP or HTTPS server address, which can accept `GET` and/or `POST` requests.
URL — URL where the server accepts `GET` and/or `POST` requests
over HTTP or HTTPS.
format - [format](../../interfaces/formats.md#formats) of the data.
format — The data [format](../../interfaces/formats.md#formats).
structure - table structure in `'UserID UInt64, Name String'` format. Determines column names and types.
structure — The structure of the table in the format `'UserID UInt64, Name String'`. Defines the column names and types.
**Example**
```sql
-- getting the first 3 lines of a table that contains columns of String and UInt32 type from HTTP-server which answers in CSV format.
-- Get 3 rows of a table consisting of two columns of type String and UInt32 from the server, which returns the data in CSV format
SELECT * FROM url('http://127.0.0.1:12345/', CSV, 'column1 String, column2 UInt32') LIMIT 3
```

23
docs/en/roadmap.md
View File

@ -2,19 +2,20 @@
## Q3 2018
- `ALTER UPDATE` for batch changing the data with approach similar to `ALTER DELETE`
- Protobuf and Parquet input and output formats
- Improved compatibility with Tableau and other BI tools
- `ALTER UPDATE` for mass changes to data using an approach similar to `ALTER DELETE`
- Adding Protobuf and Parquet to the range of supported input/output formats
- Improved compatibility with Tableau and other business analytics tools
## Q4 2018
- JOIN syntax compatible with SQL standard:
- Mutliple `JOIN`s in single `SELECT`
- Connecting tables with `ON`
- Support table reference instead of subquery
- JOIN syntax that conforms to the SQL standard:
- Multiple `JOIN`s in a single `SELECT`
- Setting the relationship between tables via `ON`
- Ability to refer to the table name instead of using a subquery
- JOIN execution improvements:
- Distributed join not limited by memory
- Predicate pushdown through join
- Improvements in the performance of JOIN:
- Distributed JOIN not limited by RAM
- Transferring predicates that depend on only one side via JOIN
- Resource pools for more accurate distribution of cluster capacity between its users
- Resource pools for more precise distribution of cluster capacity between users

2
docs/ru/operations/system_tables.md
View File

@ -122,7 +122,7 @@ default_expression String - выражение для значения по ум
## system.parts
Содержит информацию о кусках таблиц семейства [MergeTree](../operations/table_engines/mergetree.md#table_engines-mergetree).
Содержит информацию о кусках таблиц семейства [MergeTree](table_engines/mergetree.md#table_engines-mergetree).
Каждая строка описывает один кусок данных.

2
docs/ru/operations/table_engines/file.md
View File

@ -68,7 +68,7 @@ SELECT * FROM file_engine_table
$ echo -e "1,2\n3,4" | clickhouse-local -q "CREATE TABLE table (a Int64, b Int64) ENGINE = File(CSV, stdin); SELECT a, b FROM table; DROP TABLE table"
```
## Особенности использования
## Детали реализации
- Поддерживается многопоточное чтение и однопоточная запись.
- Не поддерживается:

2
docs/ru/operations/utils/clickhouse-local.md
View File

@ -24,7 +24,7 @@ clickhouse-local --structure "table_structure" --input-format "format_of_incomin
- `-S`, `--structure` — структура таблицы, в которую будут помещены входящие данные.
- `-if`, `--input-format` — формат входящих данных. По умолчанию — `TSV`.
- `-f`, `--file` — путь к файлу с данными. По умолчанию — `stdin`.
- `-q` `--query` — запросы на выполнение. Разделитель запросов — `;`.
- `-q`, `--query` — запросы на выполнение. Разделитель запросов — `;`.
- `-N`, `--table` — имя таблицы, в которую будут помещены входящие данные. По умолчанию - `table`.
- `-of`, `--format`, `--output-format` — формат выходных данных. По умолчанию — `TSV`.
- `--stacktrace` — вывод отладочной информации при исключениях.

2
docs/ru/query_language/syntax.md
View File

@ -35,7 +35,7 @@ INSERT INTO t VALUES (1, 'Hello, world'), (2, 'abc'), (3, 'def')
Идентификаторы (имена столбцов, функций, типов данных) могут быть квотированными или не квотированными.
Не квотированные идентификаторы начинаются на букву латинского алфавита или подчёркивание; продолжаются на букву латинского алфавита или подчёркивание или цифру. Короче говоря, должны соответствовать регулярному выражению `^[a-zA-Z_][0-9a-zA-Z_]*$`. Примеры: `x, _1, X_y__Z123_.`
Квотированные идентификаторы расположены в обратных кавычках `` `id` `` (также, как в MySQL), и могут обозначать произвольный (непустой) набор байт. При этом, внутри записи такого идентификатора, символы (например, символ обратной кавычки) могут экранироваться с помощью обратного слеша. Правила экранирования такие же, как в строковых литералах (см. ниже).
Квотированные идентификаторы расположены в обратных кавычках ` `id` ` (также, как в MySQL), и могут обозначать произвольный (непустой) набор байт. При этом, внутри записи такого идентификатора, символы (например, символ обратной кавычки) могут экранироваться с помощью обратного слеша. Правила экранирования такие же, как в строковых литералах (см. ниже).
Рекомендуется использовать идентификаторы, которые не нужно квотировать.
## Литералы

4
docs/toc_en.yml
View File

@ -40,6 +40,7 @@ nav:
- 'Array(T)': 'data_types/array.md'
- 'AggregateFunction(name, types_of_arguments...)': 'data_types/nested_data_structures/aggregatefunction.md'
- 'Tuple(T1, T2, ...)': 'data_types/tuple.md'
- 'Nullable': 'data_types/nullable.md'
- 'Nested data structures':
- 'hidden': 'data_types/nested_data_structures/index.md'
- 'Nested(Name1 Type1, Name2 Type2, ...)': 'data_types/nested_data_structures/nested.md'
@ -47,6 +48,7 @@ nav:
- 'hidden': 'data_types/special_data_types/index.md'
- 'Expression': 'data_types/special_data_types/expression.md'
- 'Set': 'data_types/special_data_types/set.md'
- 'Nothing': 'data_types/special_data_types/nothing.md'
- 'SQL reference':
- 'hidden': 'query_language/index.md'
@ -83,6 +85,8 @@ nav:
- 'Functions for working with Yandex.Metrica dictionaries': 'query_language/functions/ym_dict_functions.md'
- 'Functions for implementing the IN operator': 'query_language/functions/in_functions.md'
- 'arrayJoin function': 'query_language/functions/array_join.md'
- 'Functions for working with geographical coordinates': 'query_language/functions/geo.md'
- 'Functions for working with Nullable arguments': 'query_language/functions/functions_for_nulls.md'
- 'Aggregate functions':
- 'Introduction': 'query_language/agg_functions/index.md'
- 'Function reference': 'query_language/agg_functions/reference.md'