From 77e5a63648ef23cb182385a1059aa58d238536ae Mon Sep 17 00:00:00 2001 From: Kevin Chiang Date: Mon, 12 Oct 2020 12:58:38 -0700 Subject: [PATCH 001/174] Fix a markdown issue that causes a section of the note to appear outside of the Note box --- docs/en/sql-reference/statements/alter/index/index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/en/sql-reference/statements/alter/index/index.md b/docs/en/sql-reference/statements/alter/index/index.md index 4660478551f..6b00696e07f 100644 --- a/docs/en/sql-reference/statements/alter/index/index.md +++ b/docs/en/sql-reference/statements/alter/index/index.md @@ -19,5 +19,4 @@ The first two commands areare lightweight in a sense that they only change metad Also, they are replicated, syncing indices metadata via ZooKeeper. !!! note "Note" - Index manipulation is supported only for tables with [`*MergeTree`](../../../../engines/table-engines/mergetree-family/mergetree.md) engine (including -[replicated](../../../../engines/table-engines/mergetree-family/replication.md) variants). + Index manipulation is supported only for tables with [`*MergeTree`](../../../../engines/table-engines/mergetree-family/mergetree.md) engine (including [replicated](../../../../engines/table-engines/mergetree-family/replication.md) variants). From 74bdc7e645df799d35fdb32fbe77c481c89a390a Mon Sep 17 00:00:00 2001 From: Kevin Chiang Date: Mon, 12 Oct 2020 12:59:18 -0700 Subject: [PATCH 002/174] MergeTrees can have data skipping indexes, removing section from architecture to avoid confusion. --- docs/en/development/architecture.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/en/development/architecture.md b/docs/en/development/architecture.md index da5b4470704..c86818c7a1a 100644 --- a/docs/en/development/architecture.md +++ b/docs/en/development/architecture.md @@ -177,8 +177,6 @@ When you `INSERT` a bunch of data into `MergeTree`, that bunch is sorted by prim `MergeTree` is not an LSM tree because it doesn’t contain “memtable” and “log”: inserted data is written directly to the filesystem. This makes it suitable only to INSERT data in batches, not by individual row and not very frequently – about once per second is ok, but a thousand times a second is not. We did it this way for simplicity’s sake, and because we are already inserting data in batches in our applications. -> MergeTree tables can only have one (primary) index: there aren’t any secondary indices. It would be nice to allow multiple physical representations under one logical table, for example, to store data in more than one physical order or even to allow representations with pre-aggregated data along with original data. - There are MergeTree engines that are doing additional work during background merges. Examples are `CollapsingMergeTree` and `AggregatingMergeTree`. This could be treated as special support for updates. Keep in mind that these are not real updates because users usually have no control over the time when background merges are executed, and data in a `MergeTree` table is almost always stored in more than one part, not in completely merged form. ## Replication {#replication} From ee890e7689305c24118666a4f829ea3b4e6fe1fe Mon Sep 17 00:00:00 2001 From: annvsh Date: Sat, 17 Oct 2020 23:39:01 +0700 Subject: [PATCH 003/174] Added RawBLOB format description --- docs/en/interfaces/formats.md | 38 +++++++++++++++++++++++++++++++++++ docs/ru/interfaces/formats.md | 38 +++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index d96d48bdca3..7b5e887e04b 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -57,6 +57,7 @@ The supported formats are: | [XML](#xml) | ✗ | ✔ | | [CapnProto](#capnproto) | ✔ | ✗ | | [LineAsString](#lineasstring) | ✔ | ✗ | +| [RawBLOB](#rawblob) | ✔ | ✔ | You can control some format processing parameters with the ClickHouse settings. For more information read the [Settings](../operations/settings/settings.md) section. @@ -1338,4 +1339,41 @@ Result: └───────────────────────────────────────────────────┘ ``` +## RawBLOB {#rawblob} + +This format slurps all input data into a single value. This format can only parse a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. +When an empty value is passed to the input, ClickHouse generates an exception: + + ``` text +Code: 108. DB::Exception: No data to insert +``` + +The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. + +**Example** + +``` bash +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CURDIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} -n --query " +DROP TABLE IF EXISTS t; +CREATE TABLE t (a LowCardinality(Nullable(String))) ENGINE = Memory; + +${CLICKHOUSE_CLIENT} --query "INSERT INTO t FORMAT RawBLOB" < ${BASH_SOURCE[0]} + +cat ${BASH_SOURCE[0]} | md5sum + +${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum + +${CLICKHOUSE_CLIENT} --query " +DROP TABLE t; +``` + +Result: + +``` text +f9725a22f9191e064120d718e26862a9 - +``` + [Original article](https://clickhouse.tech/docs/en/interfaces/formats/) diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 2745139998f..b0901e70f20 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -39,6 +39,7 @@ ClickHouse может принимать (`INSERT`) и отдавать (`SELECT | [XML](#xml) | ✗ | ✔ | | [CapnProto](#capnproto) | ✔ | ✗ | | [LineAsString](#lineasstring) | ✔ | ✗ | +| [RawBLOB](#rawblob) | ✔ | ✔ | Вы можете регулировать некоторые параметры работы с форматами с помощью настроек ClickHouse. За дополнительной информацией обращайтесь к разделу [Настройки](../operations/settings/settings.md). @@ -1143,4 +1144,41 @@ SELECT * FROM line_as_string; └───────────────────────────────────────────────────┘ ``` +## RawBLOB {#rawblob} + +Этот формат объединяет все входные данные в одно значение. Этот формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. +При передаче на вход пустого значения ClickHouse сгенерирует исключение: + + ``` text +Code: 108. DB::Exception: No data to insert +``` + +Результат выводится в двоичном формате без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. + +**Пример** + +``` bash +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CURDIR"/../shell_config.sh + +${CLICKHOUSE_CLIENT} -n --query " +DROP TABLE IF EXISTS t; +CREATE TABLE t (a LowCardinality(Nullable(String))) ENGINE = Memory; + +${CLICKHOUSE_CLIENT} --query "INSERT INTO t FORMAT RawBLOB" < ${BASH_SOURCE[0]} + +cat ${BASH_SOURCE[0]} | md5sum + +${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum + +${CLICKHOUSE_CLIENT} --query " +DROP TABLE t; +``` + +Результат: + +``` text +f9725a22f9191e064120d718e26862a9 - +``` + [Оригинальная статья](https://clickhouse.tech/docs/ru/interfaces/formats/) From a9532d611b34b4f427fd44dda719f86fc235a428 Mon Sep 17 00:00:00 2001 From: annvsh Date: Sat, 17 Oct 2020 23:50:19 +0700 Subject: [PATCH 004/174] Fixed --- docs/en/interfaces/formats.md | 1 + docs/ru/interfaces/formats.md | 2 ++ 2 files changed, 3 insertions(+) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 7b5e887e04b..1a96a2e4643 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1348,6 +1348,7 @@ When an empty value is passed to the input, ClickHouse generates an exception: Code: 108. DB::Exception: No data to insert ``` + The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. **Example** diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index b0901e70f20..844c1a7fd59 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1153,6 +1153,7 @@ SELECT * FROM line_as_string; Code: 108. DB::Exception: No data to insert ``` + Результат выводится в двоичном формате без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. **Пример** @@ -1181,4 +1182,5 @@ DROP TABLE t; f9725a22f9191e064120d718e26862a9 - ``` + [Оригинальная статья](https://clickhouse.tech/docs/ru/interfaces/formats/) From 42047068b7c1473632cb9efd508e394c63d7a7ca Mon Sep 17 00:00:00 2001 From: annvsh Date: Sat, 17 Oct 2020 23:51:26 +0700 Subject: [PATCH 005/174] Fixed --- docs/ru/interfaces/formats.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 844c1a7fd59..ad326e2cfef 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1182,5 +1182,4 @@ DROP TABLE t; f9725a22f9191e064120d718e26862a9 - ``` - [Оригинальная статья](https://clickhouse.tech/docs/ru/interfaces/formats/) From a6402336375563bbaa454bfc71316dedf231ea8d Mon Sep 17 00:00:00 2001 From: annvsh Date: Sat, 17 Oct 2020 23:58:53 +0700 Subject: [PATCH 006/174] Fixed --- docs/en/interfaces/formats.md | 2 +- docs/ru/interfaces/formats.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 1a96a2e4643..cccef0725b7 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1344,7 +1344,7 @@ Result: This format slurps all input data into a single value. This format can only parse a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. When an empty value is passed to the input, ClickHouse generates an exception: - ``` text +``` text Code: 108. DB::Exception: No data to insert ``` diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index ad326e2cfef..11fb65fd0c0 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1149,7 +1149,7 @@ SELECT * FROM line_as_string; Этот формат объединяет все входные данные в одно значение. Этот формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. При передаче на вход пустого значения ClickHouse сгенерирует исключение: - ``` text +``` text Code: 108. DB::Exception: No data to insert ``` From ada6f7eb2469fc32c12b65153f2b6789a6b25d89 Mon Sep 17 00:00:00 2001 From: annvsh Date: Sun, 18 Oct 2020 00:05:24 +0700 Subject: [PATCH 007/174] Fixed --- docs/en/interfaces/formats.md | 1 + docs/ru/interfaces/formats.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index cccef0725b7..46b29478fc8 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1369,6 +1369,7 @@ ${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum ${CLICKHOUSE_CLIENT} --query " DROP TABLE t; +" ``` Result: diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 11fb65fd0c0..654ad3de9f4 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1174,6 +1174,7 @@ ${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum ${CLICKHOUSE_CLIENT} --query " DROP TABLE t; +" ``` Результат: From 8519f66555ea3bfb3060ea7c2bdcfe576702184e Mon Sep 17 00:00:00 2001 From: annvsh Date: Wed, 21 Oct 2020 19:46:31 +0700 Subject: [PATCH 008/174] x --- docs/ru/interfaces/formats.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 654ad3de9f4..3519b29c2d4 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1146,7 +1146,7 @@ SELECT * FROM line_as_string; ## RawBLOB {#rawblob} -Этот формат объединяет все входные данные в одно значение. Этот формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. +Этот формат считывает все входные данные в одно значение. Этот формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. При передаче на вход пустого значения ClickHouse сгенерирует исключение: ``` text From 1399f8889e9c4ae703e0da7833106ac9a1b21414 Mon Sep 17 00:00:00 2001 From: annvsh Date: Fri, 23 Oct 2020 22:51:08 +0700 Subject: [PATCH 009/174] Fixed --- docs/en/interfaces/formats.md | 35 ++++++++++++++------------------- docs/ru/interfaces/formats.md | 37 +++++++++++++++-------------------- 2 files changed, 31 insertions(+), 41 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 46b29478fc8..bb96f6c62a9 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1342,34 +1342,29 @@ Result: ## RawBLOB {#rawblob} This format slurps all input data into a single value. This format can only parse a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. -When an empty value is passed to the input, ClickHouse generates an exception: +The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. + +The difference between `RawBLOB` and `TSVRaw`: +- data is output in binary format, no escaping; +- no delimiters between values; +- no newline at the end of each value. + +In `Raw BLOB` unlike `Raw Binary` strings are output without their length. + +When an empty value is passed to the `RawBLOB` input, ClickHouse generates an exception: ``` text Code: 108. DB::Exception: No data to insert ``` - -The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. - **Example** ``` bash -CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -. "$CURDIR"/../shell_config.sh - -${CLICKHOUSE_CLIENT} -n --query " -DROP TABLE IF EXISTS t; -CREATE TABLE t (a LowCardinality(Nullable(String))) ENGINE = Memory; - -${CLICKHOUSE_CLIENT} --query "INSERT INTO t FORMAT RawBLOB" < ${BASH_SOURCE[0]} - -cat ${BASH_SOURCE[0]} | md5sum - -${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum - -${CLICKHOUSE_CLIENT} --query " -DROP TABLE t; -" +$ clickhouse-client --query "DROP TABLE IF EXISTS {some_table};" +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" +$ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum +$ clickhouse-client --query "DROP TABLE {some_table};" ``` Result: diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 3519b29c2d4..e65de3d74e0 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1146,35 +1146,30 @@ SELECT * FROM line_as_string; ## RawBLOB {#rawblob} -Этот формат считывает все входные данные в одно значение. Этот формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. -При передаче на вход пустого значения ClickHouse сгенерирует исключение: +Этот формат считывает все входные данные в одно значение. Формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. +Результат выводится в бинарном виде без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. + +Отличия между `RawBLOB` и `TabSeparatedRaw`: +- данные выводятся в бинарном виде, без экранирования; +- нет разделителей между значениями; +- нет новой строки в конце каждого значения. + +В `RawBLOB`, в отличие от `RowBinary`, строки выводятся без их длины. + +При передаче на вход `RawBLOB` пустого значения ClickHouse сгенерирует исключение: ``` text Code: 108. DB::Exception: No data to insert ``` - -Результат выводится в двоичном формате без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. - **Пример** ``` bash -CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) -. "$CURDIR"/../shell_config.sh - -${CLICKHOUSE_CLIENT} -n --query " -DROP TABLE IF EXISTS t; -CREATE TABLE t (a LowCardinality(Nullable(String))) ENGINE = Memory; - -${CLICKHOUSE_CLIENT} --query "INSERT INTO t FORMAT RawBLOB" < ${BASH_SOURCE[0]} - -cat ${BASH_SOURCE[0]} | md5sum - -${CLICKHOUSE_CLIENT} -n --query "SELECT * FROM t FORMAT RawBLOB" | md5sum - -${CLICKHOUSE_CLIENT} --query " -DROP TABLE t; -" +$ clickhouse-client --query "DROP TABLE IF EXISTS {some_table};" +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" +$ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum +$ clickhouse-client --query "DROP TABLE {some_table};" ``` Результат: From 5e2a3d12d762ee5222b2a70990e06c4255771454 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 29 Oct 2020 17:25:44 +0300 Subject: [PATCH 010/174] Split requests and responses part from zookeeper --- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 623 +++++++++++++++++++++++ src/Common/ZooKeeper/ZooKeeperCommon.h | 326 ++++++++++++ src/Common/ZooKeeper/ZooKeeperImpl.cpp | 455 ----------------- src/Common/ZooKeeper/ZooKeeperImpl.h | 33 +- 4 files changed, 951 insertions(+), 486 deletions(-) create mode 100644 src/Common/ZooKeeper/ZooKeeperCommon.cpp create mode 100644 src/Common/ZooKeeper/ZooKeeperCommon.h diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp new file mode 100644 index 00000000000..eb04536ae00 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -0,0 +1,623 @@ +#include +#include +#include +#include +#include +#include + + +namespace Coordination +{ + +using namespace DB; + +/// ZooKeeper has 1 MB node size and serialization limit by default, +/// but it can be raised up, so we have a slightly larger limit on our side. +#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB + +/// Assuming we are at little endian. + +static void write(int64_t x, WriteBuffer & out) +{ + x = __builtin_bswap64(x); + writeBinary(x, out); +} + +static void write(int32_t x, WriteBuffer & out) +{ + x = __builtin_bswap32(x); + writeBinary(x, out); +} + +static void write(bool x, WriteBuffer & out) +{ + writeBinary(x, out); +} + +static void write(const String & s, WriteBuffer & out) +{ + write(int32_t(s.size()), out); + out.write(s.data(), s.size()); +} + +template void write(std::array s, WriteBuffer & out) +{ + write(int32_t(N), out); + out.write(s.data(), N); +} + +template void write(const std::vector & arr, WriteBuffer & out) +{ + write(int32_t(arr.size()), out); + for (const auto & elem : arr) + write(elem, out); +} + +static void write(const ACL & acl, WriteBuffer & out) +{ + write(acl.permissions, out); + write(acl.scheme, out); + write(acl.id, out); +} + +static void write(const Stat & stat, WriteBuffer & out) +{ + write(stat.czxid, out); + write(stat.mzxid, out); + write(stat.ctime, out); + write(stat.mtime, out); + write(stat.version, out); + write(stat.cversion, out); + write(stat.aversion, out); + write(stat.ephemeralOwner, out); + write(stat.dataLength, out); + write(stat.numChildren, out); + write(stat.pzxid, out); +} + +static void write(const Error & x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +static void read(int64_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap64(x); +} + +static void read(int32_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap32(x); +} + +static void read(Error & x, ReadBuffer & in) +{ + int32_t code; + read(code, in); + x = Error(code); +} + +static void read(bool & x, ReadBuffer & in) +{ + readBinary(x, in); +} + +static void read(String & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + + if (size == -1) + { + /// It means that zookeeper node has NULL value. We will treat it like empty string. + s.clear(); + return; + } + + if (size < 0) + throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); + + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + + s.resize(size); + in.read(s.data(), size); +} + +template void read(std::array & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size != N) + throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + in.read(s.data(), N); +} + +static void read(Stat & stat, ReadBuffer & in) +{ + read(stat.czxid, in); + read(stat.mzxid, in); + read(stat.ctime, in); + read(stat.mtime, in); + read(stat.version, in); + read(stat.cversion, in); + read(stat.aversion, in); + read(stat.ephemeralOwner, in); + read(stat.dataLength, in); + read(stat.numChildren, in); + read(stat.pzxid, in); +} + +template void read(std::vector & arr, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size < 0) + throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + arr.resize(size); + for (auto & elem : arr) + read(elem, in); +} + +static void read(ACL & acl, ReadBuffer & in) +{ + read(acl.permissions, in); + read(acl.scheme, in); + read(acl.id, in); +} + +void ZooKeeperRequest::write(WriteBuffer & out) const +{ + /// Excessive copy to calculate length. + WriteBufferFromOwnString buf; + Coordination::write(xid, buf); + Coordination::write(getOpNum(), buf); + writeImpl(buf); + Coordination::write(buf.str(), out); + out.next(); +} + +void ZooKeeperWatchResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(type, in); + Coordination::read(state, in); + Coordination::read(path, in); +} + +void ZooKeeperWatchResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(type, out); + Coordination::write(state, out); + Coordination::write(path, out); +} + +void ZooKeeperAuthRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(type, out); + Coordination::write(scheme, out); + Coordination::write(data, out); +} + +void ZooKeeperAuthRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(type, in); + Coordination::read(scheme, in); + Coordination::read(data, in); +} + +void ZooKeeperCreateRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(data, out); + Coordination::write(acls, out); + + int32_t flags = 0; + + if (is_ephemeral) + flags |= 1; + if (is_sequential) + flags |= 2; + + Coordination::write(flags, out); +} + +void ZooKeeperCreateRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(data, in); + Coordination::read(acls, in); + + int32_t flags = 0; + Coordination::read(flags, in); + + if (flags & 1) + is_ephemeral = true; + if (flags & 2) + is_sequential = true; +} + +void ZooKeeperCreateResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(path_created, in); +} + +void ZooKeeperCreateResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path_created, out); +} + +void ZooKeeperRemoveRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(version, out); +} + +void ZooKeeperRemoveRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(version, in); +} + +void ZooKeeperExistsRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperExistsRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperExistsResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(stat, in); +} + +void ZooKeeperExistsResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(stat, out); +} + +void ZooKeeperGetRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperGetRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperGetResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(data, in); + Coordination::read(stat, in); +} + +void ZooKeeperGetResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(data, out); + Coordination::write(stat, out); +} + +void ZooKeeperSetRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(data, out); + Coordination::write(version, out); +} + +void ZooKeeperSetRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(data, in); + Coordination::read(version, in); +} + +void ZooKeeperSetResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(stat, in); +} + +void ZooKeeperSetResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(stat, out); +} + +void ZooKeeperListRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperListRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperListResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(names, in); + Coordination::read(stat, in); +} + +void ZooKeeperListResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(names, out); + Coordination::write(stat, out); +} + +void ZooKeeperCheckRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(version, out); +} + +void ZooKeeperCheckRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(version, in); +} + +void ZooKeeperErrorResponse::readImpl(ReadBuffer & in) +{ + Coordination::Error read_error; + Coordination::read(read_error, in); + + if (read_error != error) + throw Exception(fmt::format("Error code in ErrorResponse ({}) doesn't match error code in header ({})", read_error, error), + Error::ZMARSHALLINGERROR); +} + +void ZooKeeperErrorResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(error, out); +} + +ZooKeeperMultiRequest::ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls) +{ + /// Convert nested Requests to ZooKeeperRequests. + /// Note that deep copy is required to avoid modifying path in presence of chroot prefix. + requests.reserve(generic_requests.size()); + + for (const auto & generic_request : generic_requests) + { + if (const auto * concrete_request_create = dynamic_cast(generic_request.get())) + { + auto create = std::make_shared(*concrete_request_create); + if (create->acls.empty()) + create->acls = default_acls; + requests.push_back(create); + } + else if (const auto * concrete_request_remove = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_remove)); + } + else if (const auto * concrete_request_set = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_set)); + } + else if (const auto * concrete_request_check = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_check)); + } + else + throw Exception("Illegal command as part of multi ZooKeeper request", Error::ZBADARGUMENTS); + } +} + +void ZooKeeperMultiRequest::writeImpl(WriteBuffer & out) const +{ + for (const auto & request : requests) + { + const auto & zk_request = dynamic_cast(*request); + + bool done = false; + int32_t error = -1; + + Coordination::write(zk_request.getOpNum(), out); + Coordination::write(done, out); + Coordination::write(error, out); + + zk_request.writeImpl(out); + } + + OpNum op_num = -1; + bool done = true; + int32_t error = -1; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(error, out); +} + +void ZooKeeperMultiRequest::readImpl(ReadBuffer & in) +{ + + while (true) + { + OpNum op_num; + bool done; + int32_t error; + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(error, in); + + if (done) + { + if (op_num != -1) + throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + if (error != -1) + throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + break; + } + + ZooKeeperRequestPtr request = ZooKeeperRequestFactory::instance().get(op_num); + request->readImpl(in); + requests.push_back(request); + + if (in.eof()) + throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); + } +} + +void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) +{ + for (auto & response : responses) + { + OpNum op_num; + bool done; + Error op_error; + + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(op_error, in); + + if (done) + throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); + + /// op_num == -1 is special for multi transaction. + /// For unknown reason, error code is duplicated in header and in response body. + + if (op_num == -1) + response = std::make_shared(); + + if (op_error != Error::ZOK) + { + response->error = op_error; + + /// Set error for whole transaction. + /// If some operations fail, ZK send global error as zero and then send details about each operation. + /// It will set error code for first failed operation and it will set special "runtime inconsistency" code for other operations. + if (error == Error::ZOK && op_error != Error::ZRUNTIMEINCONSISTENCY) + error = op_error; + } + + if (op_error == Error::ZOK || op_num == -1) + dynamic_cast(*response).readImpl(in); + } + + /// Footer. + { + OpNum op_num; + bool done; + int32_t error_read; + + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(error_read, in); + + if (!done) + throw Exception("Too many results received for multi transaction", Error::ZMARSHALLINGERROR); + if (op_num != -1) + throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + if (error_read != -1) + throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + } +} + +void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const +{ + for (auto & response : responses) + { + const ZooKeeperResponse & zk_response = dynamic_cast(*response); + OpNum op_num = zk_response.getOpNum(); + bool done = false; + Error op_error = zk_response.error; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(op_error, out); + zk_response.writeImpl(out); + } + + /// Footer. + { + OpNum op_num = -1; + bool done = true; + int32_t error_read = - 1; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(error_read, out); + } +} + +ZooKeeperResponsePtr ZooKeeperHeartbeatRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperAuthRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperCreateRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperRemoveRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperExistsRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperGetRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperSetRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperListRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperCheckRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperMultiRequest::makeResponse() const { return std::make_shared(requests); } +ZooKeeperResponsePtr ZooKeeperCloseRequest::makeResponse() const { return std::make_shared(); } + +void ZooKeeperRequestFactory::registerRequest(OpNum op_num, Creator creator) +{ + if (!op_num_to_request.try_emplace(op_num, creator).second) + throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Request with op num {} already registered", op_num); +} + +std::shared_ptr ZooKeeperRequest::read(ReadBuffer & in) +{ + XID xid; + OpNum op_num; + + Coordination::read(xid, in); + Coordination::read(op_num, in); + + auto request = ZooKeeperRequestFactory::instance().get(op_num); + request->xid = xid; + request->readImpl(in); + return request; +} + +ZooKeeperRequestPtr ZooKeeperRequestFactory::get(OpNum op_num) const +{ + auto it = op_num_to_request.find(op_num); + if (it == op_num_to_request.end()) + throw Exception("Unknown operation type " + std::to_string(op_num), Error::ZBADARGUMENTS); + + return it->second(); +} + +ZooKeeperRequestFactory & ZooKeeperRequestFactory::instance() +{ + static ZooKeeperRequestFactory factory; + return factory; +} + +template +void registerZooKeeperRequest(ZooKeeperRequestFactory & factory) +{ + factory.registerRequest(num, [] { return std::make_shared(); }); +} + +ZooKeeperRequestFactory::ZooKeeperRequestFactory() +{ + registerZooKeeperRequest<11, ZooKeeperHeartbeatRequest>(*this); + registerZooKeeperRequest<100, ZooKeeperAuthRequest>(*this); + registerZooKeeperRequest<-11, ZooKeeperCloseRequest>(*this); + registerZooKeeperRequest<1, ZooKeeperCreateRequest>(*this); + registerZooKeeperRequest<2, ZooKeeperRemoveRequest>(*this); + registerZooKeeperRequest<3, ZooKeeperExistsRequest>(*this); + registerZooKeeperRequest<4, ZooKeeperGetRequest>(*this); + registerZooKeeperRequest<5, ZooKeeperSetRequest>(*this); + registerZooKeeperRequest<12, ZooKeeperListRequest>(*this); + registerZooKeeperRequest<13, ZooKeeperCheckRequest>(*this); + registerZooKeeperRequest<14, ZooKeeperMultiRequest>(*this); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h new file mode 100644 index 00000000000..0b19869dd5a --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -0,0 +1,326 @@ +#pragma once + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + + + +namespace Coordination +{ + +using XID = int32_t; +using OpNum = int32_t; + + +struct ZooKeeperResponse : virtual Response +{ + virtual ~ZooKeeperResponse() override = default; + virtual void readImpl(ReadBuffer &) = 0; + virtual void writeImpl(WriteBuffer &) const = 0; + virtual OpNum getOpNum() const = 0; +}; + +using ZooKeeperResponsePtr = std::shared_ptr; + +/// Exposed in header file for Yandex.Metrica code. +struct ZooKeeperRequest : virtual Request +{ + XID xid = 0; + bool has_watch = false; + /// If the request was not send and the error happens, we definitely sure, that it has not been processed by the server. + /// If the request was sent and we didn't get the response and the error happens, then we cannot be sure was it processed or not. + bool probably_sent = false; + + ZooKeeperRequest() = default; + ZooKeeperRequest(const ZooKeeperRequest &) = default; + virtual ~ZooKeeperRequest() override = default; + + virtual OpNum getOpNum() const = 0; + + /// Writes length, xid, op_num, then the rest. + void write(WriteBuffer & out) const; + + virtual void writeImpl(WriteBuffer &) const = 0; + virtual void readImpl(ReadBuffer &) = 0; + + static std::shared_ptr read(ReadBuffer & in); + + virtual ZooKeeperResponsePtr makeResponse() const = 0; +}; + +using ZooKeeperRequestPtr = std::shared_ptr; + +struct ZooKeeperHeartbeatRequest final : ZooKeeperRequest +{ + String getPath() const override { return {}; } + OpNum getOpNum() const override { return 11; } + void writeImpl(WriteBuffer &) const override {} + void readImpl(ReadBuffer &) override {} + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperHeartbeatResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return 11; } +}; + +struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + + /// TODO FIXME alesap + OpNum getOpNum() const override { return 0; } +}; + +struct ZooKeeperAuthRequest final : ZooKeeperRequest +{ + int32_t type = 0; /// ignored by the server + String scheme; + String data; + + String getPath() const override { return {}; } + OpNum getOpNum() const override { return 100; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperAuthResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + + OpNum getOpNum() const override { return 100; } +}; + +struct ZooKeeperCloseRequest final : ZooKeeperRequest +{ + String getPath() const override { return {}; } + OpNum getOpNum() const override { return -11; } + void writeImpl(WriteBuffer &) const override {} + void readImpl(ReadBuffer &) override {} + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCloseResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override + { + throw Exception("Received response for close request", Error::ZRUNTIMEINCONSISTENCY); + } + + void writeImpl(WriteBuffer &) const override {} + + OpNum getOpNum() const override { return -11; } +}; + +struct ZooKeeperCreateRequest final : CreateRequest, ZooKeeperRequest +{ + ZooKeeperCreateRequest() = default; + explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} + + OpNum getOpNum() const override { return 1; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCreateResponse final : CreateResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + + OpNum getOpNum() const override { return 1; } +}; + +struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest +{ + ZooKeeperRemoveRequest() = default; + explicit ZooKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} + + OpNum getOpNum() const override { return 2; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperRemoveResponse final : RemoveResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return 2; } +}; + +struct ZooKeeperExistsRequest final : ExistsRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return 3; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperExistsResponse final : ExistsResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return 3; } +}; + +struct ZooKeeperGetRequest final : GetRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return 4; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperGetResponse final : GetResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return 4; } +}; + +struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest +{ + ZooKeeperSetRequest() = default; + explicit ZooKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} + + OpNum getOpNum() const override { return 5; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return 5; } +}; + +struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return 12; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperListResponse final : ListResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return 12; } +}; + +struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest +{ + ZooKeeperCheckRequest() = default; + explicit ZooKeeperCheckRequest(const CheckRequest & base) : CheckRequest(base) {} + + OpNum getOpNum() const override { return 13; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCheckResponse final : CheckResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return 13; } +}; + +/// This response may be received only as an element of responses in MultiResponse. +struct ZooKeeperErrorResponse final : ErrorResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + + OpNum getOpNum() const override { return -1; } +}; + +struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return 14; } + ZooKeeperMultiRequest() = default; + + ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls); + + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperMultiResponse final : MultiResponse, ZooKeeperResponse +{ + OpNum getOpNum() const override { return 14; } + + explicit ZooKeeperMultiResponse(const Requests & requests) + { + responses.reserve(requests.size()); + + for (const auto & request : requests) + responses.emplace_back(dynamic_cast(*request).makeResponse()); + } + + explicit ZooKeeperMultiResponse(const Responses & responses_) + { + responses = responses_; + } + + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + +}; + +class ZooKeeperRequestFactory final : private boost::noncopyable +{ + +public: + using Creator = std::function; + using OpNumToRequest = std::unordered_map; + + static ZooKeeperRequestFactory & instance(); + + ZooKeeperRequestPtr get(OpNum op_num) const; + + void registerRequest(OpNum op_num, Creator creator); + +private: + OpNumToRequest op_num_to_request; + +private: + ZooKeeperRequestFactory(); +}; + +} diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index f5c57781eef..f3c8b537cf2 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -281,17 +281,6 @@ static void write(int32_t x, WriteBuffer & out) writeBinary(x, out); } -static void write(bool x, WriteBuffer & out) -{ - writeBinary(x, out); -} - -static void write(const String & s, WriteBuffer & out) -{ - write(int32_t(s.size()), out); - out.write(s.data(), s.size()); -} - template void write(std::array s, WriteBuffer & out) { write(int32_t(N), out); @@ -305,14 +294,6 @@ template void write(const std::vector & arr, WriteBuffer & out) write(elem, out); } -static void write(const ACL & acl, WriteBuffer & out) -{ - write(acl.permissions, out); - write(acl.scheme, out); - write(acl.id, out); -} - - static void read(int64_t & x, ReadBuffer & in) { readBinary(x, in); @@ -332,33 +313,6 @@ static void read(Error & x, ReadBuffer & in) x = Error(code); } -static void read(bool & x, ReadBuffer & in) -{ - readBinary(x, in); -} - -static void read(String & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - - if (size == -1) - { - /// It means that zookeeper node has NULL value. We will treat it like empty string. - s.clear(); - return; - } - - if (size < 0) - throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); - - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - - s.resize(size); - in.read(s.data(), size); -} - template void read(std::array & s, ReadBuffer & in) { int32_t size = 0; @@ -368,21 +322,6 @@ template void read(std::array & s, ReadBuffer & in) in.read(s.data(), N); } -static void read(Stat & stat, ReadBuffer & in) -{ - read(stat.czxid, in); - read(stat.mzxid, in); - read(stat.ctime, in); - read(stat.mtime, in); - read(stat.version, in); - read(stat.cversion, in); - read(stat.aversion, in); - read(stat.ephemeralOwner, in); - read(stat.dataLength, in); - read(stat.numChildren, in); - read(stat.pzxid, in); -} - template void read(std::vector & arr, ReadBuffer & in) { int32_t size = 0; @@ -396,7 +335,6 @@ template void read(std::vector & arr, ReadBuffer & in) read(elem, in); } - template void ZooKeeper::write(const T & x) { @@ -409,19 +347,6 @@ void ZooKeeper::read(T & x) Coordination::read(x, *in); } - -void ZooKeeperRequest::write(WriteBuffer & out) const -{ - /// Excessive copy to calculate length. - WriteBufferFromOwnString buf; - Coordination::write(xid, buf); - Coordination::write(getOpNum(), buf); - writeImpl(buf); - Coordination::write(buf.str(), out); - out.next(); -} - - static void removeRootPath(String & path, const String & root_path) { if (root_path.empty()) @@ -433,385 +358,6 @@ static void removeRootPath(String & path, const String & root_path) path = path.substr(root_path.size()); } - -struct ZooKeeperResponse : virtual Response -{ - virtual ~ZooKeeperResponse() override = default; - virtual void readImpl(ReadBuffer &) = 0; -}; - - -struct ZooKeeperHeartbeatRequest final : ZooKeeperRequest -{ - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return 11; } - void writeImpl(WriteBuffer &) const override {} - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperHeartbeatResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(type, in); - Coordination::read(state, in); - Coordination::read(path, in); - } -}; - -struct ZooKeeperAuthRequest final : ZooKeeperRequest -{ - int32_t type = 0; /// ignored by the server - String scheme; - String data; - - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return 100; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(type, out); - Coordination::write(scheme, out); - Coordination::write(data, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperAuthResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperCloseRequest final : ZooKeeperRequest -{ - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return -11; } - void writeImpl(WriteBuffer &) const override {} - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCloseResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override - { - throw Exception("Received response for close request", Error::ZRUNTIMEINCONSISTENCY); - } -}; - -struct ZooKeeperCreateRequest final : CreateRequest, ZooKeeperRequest -{ - ZooKeeperCreateRequest() = default; - explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 1; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(data, out); - Coordination::write(acls, out); - - int32_t flags = 0; - - if (is_ephemeral) - flags |= 1; - if (is_sequential) - flags |= 2; - - Coordination::write(flags, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCreateResponse final : CreateResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(path_created, in); - } -}; - -struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest -{ - ZooKeeperRemoveRequest() = default; - explicit ZooKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 2; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperRemoveResponse final : RemoveResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperExistsRequest final : ExistsRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 3; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperExistsResponse final : ExistsResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(stat, in); - } -}; - -struct ZooKeeperGetRequest final : GetRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 4; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperGetResponse final : GetResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(data, in); - Coordination::read(stat, in); - } -}; - -struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest -{ - ZooKeeperSetRequest() = default; - explicit ZooKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 5; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(data, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(stat, in); - } -}; - -struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 12; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperListResponse final : ListResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(names, in); - Coordination::read(stat, in); - } -}; - -struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest -{ - ZooKeeperCheckRequest() = default; - explicit ZooKeeperCheckRequest(const CheckRequest & base) : CheckRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 13; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCheckResponse final : CheckResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -/// This response may be received only as an element of responses in MultiResponse. -struct ZooKeeperErrorResponse final : ErrorResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::Error read_error; - Coordination::read(read_error, in); - - if (read_error != error) - throw Exception(fmt::format("Error code in ErrorResponse ({}) doesn't match error code in header ({})", read_error, error), - Error::ZMARSHALLINGERROR); - } -}; - -struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 14; } - - ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls) - { - /// Convert nested Requests to ZooKeeperRequests. - /// Note that deep copy is required to avoid modifying path in presence of chroot prefix. - requests.reserve(generic_requests.size()); - - for (const auto & generic_request : generic_requests) - { - if (const auto * concrete_request_create = dynamic_cast(generic_request.get())) - { - auto create = std::make_shared(*concrete_request_create); - if (create->acls.empty()) - create->acls = default_acls; - requests.push_back(create); - } - else if (const auto * concrete_request_remove = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_remove)); - } - else if (const auto * concrete_request_set = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_set)); - } - else if (const auto * concrete_request_check = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_check)); - } - else - throw Exception("Illegal command as part of multi ZooKeeper request", Error::ZBADARGUMENTS); - } - } - - void writeImpl(WriteBuffer & out) const override - { - for (const auto & request : requests) - { - const auto & zk_request = dynamic_cast(*request); - - bool done = false; - int32_t error = -1; - - Coordination::write(zk_request.getOpNum(), out); - Coordination::write(done, out); - Coordination::write(error, out); - - zk_request.writeImpl(out); - } - - ZooKeeper::OpNum op_num = -1; - bool done = true; - int32_t error = -1; - - Coordination::write(op_num, out); - Coordination::write(done, out); - Coordination::write(error, out); - } - - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperMultiResponse final : MultiResponse, ZooKeeperResponse -{ - explicit ZooKeeperMultiResponse(const Requests & requests) - { - responses.reserve(requests.size()); - - for (const auto & request : requests) - responses.emplace_back(dynamic_cast(*request).makeResponse()); - } - - void readImpl(ReadBuffer & in) override - { - for (auto & response : responses) - { - ZooKeeper::OpNum op_num; - bool done; - Error op_error; - - Coordination::read(op_num, in); - Coordination::read(done, in); - Coordination::read(op_error, in); - - if (done) - throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); - - /// op_num == -1 is special for multi transaction. - /// For unknown reason, error code is duplicated in header and in response body. - - if (op_num == -1) - response = std::make_shared(); - - if (op_error != Error::ZOK) - { - response->error = op_error; - - /// Set error for whole transaction. - /// If some operations fail, ZK send global error as zero and then send details about each operation. - /// It will set error code for first failed operation and it will set special "runtime inconsistency" code for other operations. - if (error == Error::ZOK && op_error != Error::ZRUNTIMEINCONSISTENCY) - error = op_error; - } - - if (op_error == Error::ZOK || op_num == -1) - dynamic_cast(*response).readImpl(in); - } - - /// Footer. - { - ZooKeeper::OpNum op_num; - bool done; - int32_t error_read; - - Coordination::read(op_num, in); - Coordination::read(done, in); - Coordination::read(error_read, in); - - if (!done) - throw Exception("Too many results received for multi transaction", Error::ZMARSHALLINGERROR); - if (op_num != -1) - throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); - if (error_read != -1) - throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); - } - } -}; - - -ZooKeeperResponsePtr ZooKeeperHeartbeatRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperAuthRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperCreateRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperRemoveRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperExistsRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperGetRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperSetRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperListRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperCheckRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperMultiRequest::makeResponse() const { return std::make_shared(requests); } -ZooKeeperResponsePtr ZooKeeperCloseRequest::makeResponse() const { return std::make_shared(); } - - static constexpr int32_t protocol_version = 0; static constexpr ZooKeeper::XID watch_xid = -1; @@ -1688,5 +1234,4 @@ void ZooKeeper::close() ProfileEvents::increment(ProfileEvents::ZooKeeperClose); } - } diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.h b/src/Common/ZooKeeper/ZooKeeperImpl.h index 085b0e9856a..c96d7d2f0cb 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.h +++ b/src/Common/ZooKeeper/ZooKeeperImpl.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -85,9 +86,6 @@ namespace Coordination using namespace DB; -struct ZooKeeperRequest; - - /** Usage scenario: look at the documentation for IKeeper class. */ class ZooKeeper : public IKeeper @@ -193,7 +191,7 @@ private: struct RequestInfo { - std::shared_ptr request; + ZooKeeperRequestPtr request; ResponseCallback callback; WatchCallback watch; clock::time_point time; @@ -246,31 +244,4 @@ private: CurrentMetrics::Increment active_session_metric_increment{CurrentMetrics::ZooKeeperSession}; }; -struct ZooKeeperResponse; -using ZooKeeperResponsePtr = std::shared_ptr; - -/// Exposed in header file for Yandex.Metrica code. -struct ZooKeeperRequest : virtual Request -{ - ZooKeeper::XID xid = 0; - bool has_watch = false; - /// If the request was not send and the error happens, we definitely sure, that is has not been processed by the server. - /// If the request was sent and we didn't get the response and the error happens, then we cannot be sure was it processed or not. - bool probably_sent = false; - - ZooKeeperRequest() = default; - ZooKeeperRequest(const ZooKeeperRequest &) = default; - virtual ~ZooKeeperRequest() override = default; - - virtual ZooKeeper::OpNum getOpNum() const = 0; - - /// Writes length, xid, op_num, then the rest. - void write(WriteBuffer & out) const; - - virtual void writeImpl(WriteBuffer &) const = 0; - - virtual ZooKeeperResponsePtr makeResponse() const = 0; -}; - - } From c2525ef211a5cc1b93914cc67a7a6a36a2f2df5e Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 30 Oct 2020 17:16:47 +0300 Subject: [PATCH 011/174] Server and client pinging each other --- programs/server/Server.cpp | 16 + src/Common/ZooKeeper/TestKeeperStorage.cpp | 6 + src/Common/ZooKeeper/TestKeeperStorage.h | 77 +++++ src/Common/ZooKeeper/ZooKeeperCommon.h | 1 - src/Interpreters/Context.cpp | 12 + src/Interpreters/Context.h | 3 + src/Server/TCPHandlerFactory.h | 11 +- src/Server/TestKeeperTCPHandler.cpp | 339 +++++++++++++++++++++ src/Server/TestKeeperTCPHandler.h | 48 +++ 9 files changed, 510 insertions(+), 3 deletions(-) create mode 100644 src/Common/ZooKeeper/TestKeeperStorage.cpp create mode 100644 src/Common/ZooKeeper/TestKeeperStorage.h create mode 100644 src/Server/TestKeeperTCPHandler.cpp create mode 100644 src/Server/TestKeeperTCPHandler.h diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index b85cb5e75f2..f855a76e362 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -949,6 +949,22 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for connections with native protocol (tcp): {}", address.toString()); }); + /// TCP TestKeeper + create_server("test_keeper_tcp_port", [&](UInt16 port) + { + Poco::Net::ServerSocket socket; + auto address = socket_bind_listen(socket, listen_host, port); + socket.setReceiveTimeout(settings.receive_timeout); + socket.setSendTimeout(settings.send_timeout); + servers.emplace_back(std::make_unique( + new TCPHandlerFactory(*this, false, true), + server_pool, + socket, + new Poco::Net::TCPServerParams)); + + LOG_INFO(log, "Listening for connections to fake zookeeper (tcp): {}", address.toString()); + }); + /// TCP with SSL create_server("tcp_port_secure", [&](UInt16 port) { diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp new file mode 100644 index 00000000000..c238fa2620f --- /dev/null +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -0,0 +1,6 @@ +#include + +namespace DB +{ + +} diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h new file mode 100644 index 00000000000..f0c8a942dff --- /dev/null +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -0,0 +1,77 @@ +#pragma once + +#include +#include +#include +#include + +namespace zkutil +{ + +using namespace DB; + +class TestKeeperStorage +{ + struct TestKeeperRequest; + using TestKeeperRequestPtr = std::shared_ptr; + + std::atomic session_id_counter{0}; + + struct Node + { + String data; + Coordination::ACLs acls; + bool is_ephemeral = false; + bool is_sequental = false; + Coordination::Stat stat{}; + int32_t seq_num = 0; + }; + + using Container = std::map; + + using WatchCallbacks = std::vector; + using Watches = std::map; + + Container container; + + String root_path; + + std::atomic zxid{0}; + + Watches watches; + Watches list_watches; /// Watches for 'list' request (watches on children). + + using clock = std::chrono::steady_clock; + + struct RequestInfo + { + TestKeeperRequestPtr request; + Coordination::ResponseCallback callback; + Coordination::WatchCallback watch; + clock::time_point time; + }; + using RequestsQueue = ConcurrentBoundedQueue; + RequestsQueue requests_queue{1}; + + void pushRequest(RequestInfo && request); + + void finalize(); + + ThreadFromGlobalPool processing_thread; + + void processingThread(); + +public: + void putRequest(const Coordination::ZooKeeperRequestPtr & request, std::shared_ptr response_out); + + int64_t getSessionID() + { + return session_id_counter.fetch_add(1); + } + int64_t getZXID() + { + return zxid.fetch_add(1); + } +}; + +} diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index 0b19869dd5a..6293cbb09fe 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -25,7 +25,6 @@ namespace Coordination using XID = int32_t; using OpNum = int32_t; - struct ZooKeeperResponse : virtual Response { virtual ~ZooKeeperResponse() override = default; diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index b75e9ae9d58..863b179df90 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -300,6 +301,8 @@ struct ContextShared mutable zkutil::ZooKeeperPtr zookeeper; /// Client for ZooKeeper. + mutable std::mutex test_keeper_storage_mutex; + mutable std::shared_ptr test_keeper_storage; mutable std::mutex auxiliary_zookeepers_mutex; mutable std::map auxiliary_zookeepers; /// Map for auxiliary ZooKeeper clients. @@ -1492,6 +1495,15 @@ zkutil::ZooKeeperPtr Context::getZooKeeper() const return shared->zookeeper; } +std::shared_ptr & Context::getTestKeeperStorage() const +{ + std::lock_guard lock(shared->test_keeper_storage_mutex); + if (!shared->test_keeper_storage) + shared->test_keeper_storage = std::make_shared(); + + return shared->test_keeper_storage; +} + zkutil::ZooKeeperPtr Context::getAuxiliaryZooKeeper(const String & name) const { std::lock_guard lock(shared->auxiliary_zookeepers_mutex); diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index fbf578494ed..22e90d8eb5a 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -40,6 +40,7 @@ namespace Poco namespace zkutil { class ZooKeeper; + class TestKeeperStorage; } @@ -483,6 +484,8 @@ public: std::shared_ptr getZooKeeper() const; /// Same as above but return a zookeeper connection from auxiliary_zookeepers configuration entry. std::shared_ptr getAuxiliaryZooKeeper(const String & name) const; + + std::shared_ptr & getTestKeeperStorage() const; /// Has ready or expired ZooKeeper bool hasZooKeeper() const; /// Reset current zookeeper session. Do not create a new one. diff --git a/src/Server/TCPHandlerFactory.h b/src/Server/TCPHandlerFactory.h index 5ecd427bf8b..945a2350508 100644 --- a/src/Server/TCPHandlerFactory.h +++ b/src/Server/TCPHandlerFactory.h @@ -5,6 +5,7 @@ #include #include #include +#include namespace Poco { class Logger; } @@ -16,6 +17,7 @@ class TCPHandlerFactory : public Poco::Net::TCPServerConnectionFactory private: IServer & server; Poco::Logger * log; + bool test_keeper; class DummyTCPHandler : public Poco::Net::TCPServerConnection { @@ -25,9 +27,10 @@ private: }; public: - explicit TCPHandlerFactory(IServer & server_, bool secure_ = false) + explicit TCPHandlerFactory(IServer & server_, bool secure_ = false, bool test_keeper_ = false) : server(server_) , log(&Poco::Logger::get(std::string("TCP") + (secure_ ? "S" : "") + "HandlerFactory")) + , test_keeper(test_keeper_) { } @@ -36,7 +39,11 @@ public: try { LOG_TRACE(log, "TCP Request. Address: {}", socket.peerAddress().toString()); - return new TCPHandler(server, socket); + + if (test_keeper) + return new TestKeeperTCPHandler(server, socket); + else + return new TCPHandler(server, socket); } catch (const Poco::Net::NetException &) { diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp new file mode 100644 index 00000000000..a0679554f64 --- /dev/null +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -0,0 +1,339 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int LOGICAL_ERROR; + +} + +#ifdef __clang__ +#pragma clang diagnostic push +#pragma clang diagnostic ignored "-Wunused-function" +#endif + +/// ZooKeeper has 1 MB node size and serialization limit by default, +/// but it can be raised up, so we have a slightly larger limit on our side. +#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB + +/// Assuming we are at little endian. + +static void write(int64_t x, WriteBuffer & out) +{ + x = __builtin_bswap64(x); + writeBinary(x, out); +} + +static void write(int32_t x, WriteBuffer & out) +{ + x = __builtin_bswap32(x); + writeBinary(x, out); +} + +static void write(bool x, WriteBuffer & out) +{ + writeBinary(x, out); +} + +static void write(const String & s, WriteBuffer & out) +{ + write(int32_t(s.size()), out); + out.write(s.data(), s.size()); +} + +template void write(std::array s, WriteBuffer & out) +{ + write(int32_t(N), out); + out.write(s.data(), N); +} + +template void write(const std::vector & arr, WriteBuffer & out) +{ + write(int32_t(arr.size()), out); + for (const auto & elem : arr) + write(elem, out); +} + +static void write(const Coordination::ACL & acl, WriteBuffer & out) +{ + write(acl.permissions, out); + write(acl.scheme, out); + write(acl.id, out); +} + +static void write(const Coordination::Stat & stat, WriteBuffer & out) +{ + write(stat.czxid, out); + write(stat.mzxid, out); + write(stat.ctime, out); + write(stat.mtime, out); + write(stat.version, out); + write(stat.cversion, out); + write(stat.aversion, out); + write(stat.ephemeralOwner, out); + write(stat.dataLength, out); + write(stat.numChildren, out); + write(stat.pzxid, out); +} + +static void write(const Coordination::Error & x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +static void read(int64_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap64(x); +} + +static void read(int32_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap32(x); +} + +static void read(Coordination::Error & x, ReadBuffer & in) +{ + int32_t code; + read(code, in); + x = Coordination::Error(code); +} + +static void read(bool & x, ReadBuffer & in) +{ + readBinary(x, in); +} + +static void read(String & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + + if (size == -1) + { + /// It means that zookeeper node has NULL value. We will treat it like empty string. + s.clear(); + return; + } + + if (size < 0) + throw Exception("Negative size while reading string from ZooKeeper", ErrorCodes::LOGICAL_ERROR); + + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large string size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); + + s.resize(size); + in.read(s.data(), size); +} + +template void read(std::array & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size != N) + throw Exception("Unexpected array size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); + in.read(s.data(), N); +} + +static void read(Coordination::Stat & stat, ReadBuffer & in) +{ + read(stat.czxid, in); + read(stat.mzxid, in); + read(stat.ctime, in); + read(stat.mtime, in); + read(stat.version, in); + read(stat.cversion, in); + read(stat.aversion, in); + read(stat.ephemeralOwner, in); + read(stat.dataLength, in); + read(stat.numChildren, in); + read(stat.pzxid, in); +} + +template void read(std::vector & arr, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size < 0) + throw Exception("Negative size while reading array from ZooKeeper", ErrorCodes::LOGICAL_ERROR); + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large array size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); + arr.resize(size); + for (auto & elem : arr) + read(elem, in); +} + +static void read(Coordination::ACL & acl, ReadBuffer & in) +{ + read(acl.permissions, in); + read(acl.scheme, in); + read(acl.id, in); +} + +#ifdef __clang__ +#pragma clang diagnostic pop +#endif + +void TestKeeperTCPHandler::sendHandshake() +{ + static constexpr int32_t handshake_length = 36; + static constexpr int32_t protocol_version = 0; + static constexpr int32_t DEFAULT_SESSION_TIMEOUT = 30000; + + write(handshake_length, *out); + write(protocol_version, *out); + write(DEFAULT_SESSION_TIMEOUT, *out); + write(test_keeper_storage->getSessionID(), *out); + constexpr int32_t passwd_len = 16; + std::array passwd{}; + write(passwd, *out); + out->next(); +} + +void TestKeeperTCPHandler::run() +{ + runImpl(); +} + +void TestKeeperTCPHandler::receiveHandshake() +{ + int32_t handshake_length; + int32_t protocol_version; + int64_t last_zxid_seen; + int32_t timeout; + int64_t previous_session_id = 0; /// We don't support session restore. So previous session_id is always zero. + constexpr int32_t passwd_len = 16; + std::array passwd {}; + + read(handshake_length, *in); + if (handshake_length != 44) + throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::LOGICAL_ERROR); + + read(protocol_version, *in); + + if (protocol_version != 0) + throw Exception("Unexpected protocol version: " + toString(protocol_version), ErrorCodes::LOGICAL_ERROR); + + read(last_zxid_seen, *in); + + if (last_zxid_seen != 0) + throw Exception("Non zero last_zxid_seen is not supported", ErrorCodes::LOGICAL_ERROR); + + read(timeout, *in); + read(previous_session_id, *in); + + if (previous_session_id != 0) + throw Exception("Non zero previous session id is not supported", ErrorCodes::LOGICAL_ERROR); + + read(passwd, *in); +} + + +void TestKeeperTCPHandler::runImpl() +{ + setThreadName("TstKprHandler"); + ThreadStatus thread_status; + auto global_receive_timeout = global_context.getSettingsRef().receive_timeout; + auto global_send_timeout = global_context.getSettingsRef().send_timeout; + + socket().setReceiveTimeout(global_receive_timeout); + socket().setSendTimeout(global_send_timeout); + socket().setNoDelay(true); + + in = std::make_shared(socket()); + out = std::make_shared(socket()); + + if (in->eof()) + { + LOG_WARNING(log, "Client has not sent any data."); + return; + } + + try + { + receiveHandshake(); + } + catch (const Exception & e) /// Typical for an incorrect username, password, or address. + { + LOG_DEBUG(log, "Cannot receive handshake {}", e.displayText()); + } + + sendHandshake(); + + while (true) + { + UInt64 max_wait = operation_timeout.totalMicroseconds(); + if (in->poll(max_wait)) + { + receiveHeartbeatRequest(); + sendHeartbeatResponse(); + } + } +} + + +void TestKeeperTCPHandler::receiveHeartbeatRequest() +{ + LOG_DEBUG(log, "Receiving heartbeat event"); + int32_t length; + read(length, *in); + int32_t total_count = in->count(); + LOG_DEBUG(log, "RECEIVED LENGTH {}", length); + int32_t xid; + LOG_DEBUG(log, "READING XID"); + read(xid, *in); + + LOG_DEBUG(log, "Received xid {}", xid); + + if (xid == -2) + { + int32_t opnum; + read(opnum, *in); + LOG_DEBUG(log, "RRECEIVED OP NUM {}", opnum); + auto request = std::make_shared(); + request->readImpl(*in); + int32_t readed = in->count() - total_count; + if (readed != length) + LOG_DEBUG(log, "EXPECTED TO READ {}, BUT GOT {}", length, readed); + } + else + { + LOG_INFO(log, "UNKNOWN EVENT xid:{}", xid); + } + + LOG_DEBUG(log, "Event received"); +} + + +void TestKeeperTCPHandler::sendHeartbeatResponse() +{ + LOG_DEBUG(log, "Sending heartbeat event"); + int32_t length = sizeof(int32_t) + sizeof(int64_t) + sizeof(Coordination::Error); + write(length, *out); + int64_t zxid = test_keeper_storage->getZXID(); + int32_t xid = -2; + write(xid, *out); + write(zxid, *out); + write(Coordination::Error::ZOK, *out); + auto response = std::make_shared(); + response->writeImpl(*out); + out->next(); +} + +} diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h new file mode 100644 index 00000000000..967ea1d29e7 --- /dev/null +++ b/src/Server/TestKeeperTCPHandler.h @@ -0,0 +1,48 @@ +#pragma once + +#include +#include "IServer.h" +#include +#include +#include +#include +#include + +namespace DB +{ + +class TestKeeperTCPHandler : public Poco::Net::TCPServerConnection +{ +public: + TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) + : Poco::Net::TCPServerConnection(socket_) + , server(server_) + , log(&Poco::Logger::get("TestKeeperTCPHandler")) + , global_context(server.context()) + , test_keeper_storage(global_context.getTestKeeperStorage()) + , operation_timeout(10000) + { + } + + void run() override; +private: + IServer & server; + Poco::Logger * log; + Context global_context; + std::shared_ptr test_keeper_storage; + Poco::Timespan operation_timeout; + + /// Streams for reading/writing from/to client connection socket. + std::shared_ptr in; + std::shared_ptr out; + + void runImpl(); + + void sendHandshake(); + void receiveHandshake(); + + void receiveHeartbeatRequest(); + void sendHeartbeatResponse(); +}; + +} From 1fc763328965d7cab2890f15b073b3824ca5c068 Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 30 Oct 2020 22:57:30 +0300 Subject: [PATCH 012/174] Heartbeats working --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 107 ++++++++++++++++++++- src/Common/ZooKeeper/TestKeeperStorage.h | 4 + src/Common/ZooKeeper/ZooKeeperCommon.h | 2 +- src/Server/TestKeeperTCPHandler.h | 2 + 4 files changed, 112 insertions(+), 3 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index c238fa2620f..96593b291d6 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -1,6 +1,109 @@ #include +#include -namespace DB +namespace zkutil { - +using namespace DB; + +static String parentPath(const String & path) +{ + auto rslash_pos = path.rfind('/'); + if (rslash_pos > 0) + return path.substr(0, rslash_pos); + return "/"; +} + +using Undo = std::function; + +struct TestKeeperStorageRequest +{ + Coordination::ZooKeeperRequestPtr zk_request; + + TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + : zk_request(zk_request_) + {} + virtual bool isMutable() const { return false; } + virtual std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const = 0; + virtual ~TestKeeperStorageRequest() {} +}; + + +struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest +{ + std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Undo undo; + Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperCreateRequest & request = dynamic_cast(*zk_request); + + if (container.count(request.path)) + { + response.error = Coordination::Error::ZNODEEXISTS; + } + else + { + auto it = container.find(parentPath(request.path)); + + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (it->second.is_ephemeral) + { + response.error = Coordination::Error::ZNOCHILDRENFOREPHEMERALS; + } + else + { + TestKeeperStorage::Node created_node; + created_node.seq_num = 0; + created_node.stat.czxid = zxid; + created_node.stat.mzxid = zxid; + created_node.stat.ctime = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1); + created_node.stat.mtime = created_node.stat.ctime; + created_node.stat.numChildren = 0; + created_node.stat.dataLength = request.data.length(); + created_node.data = request.data; + created_node.is_ephemeral = request.is_ephemeral; + created_node.is_sequental = request.is_sequential; + std::string path_created = request.path; + + if (request.is_sequential) + { + auto seq_num = it->second.seq_num; + ++it->second.seq_num; + + std::stringstream seq_num_str; + seq_num_str << std::setw(10) << std::setfill('0') << seq_num; + + path_created += seq_num_str.str(); + } + + response.path_created = path_created; + container.emplace(path_created, std::move(created_node)); + + undo = [&container, path_created, is_sequential = request.is_sequential, parent_path = it->first] + { + container.erase(path_created); + auto & undo_parent = container.at(parent_path); + --undo_parent.stat.cversion; + --undo_parent.stat.numChildren; + + if (is_sequential) + --undo_parent.seq_num; + }; + + ++it->second.stat.cversion; + ++it->second.stat.numChildren; + + response.error = Coordination::Error::ZOK; + } + } + + return { response_ptr, undo }; + } +}; + + + } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index f0c8a942dff..893f9f2842c 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -12,6 +12,8 @@ using namespace DB; class TestKeeperStorage { + +public: struct TestKeeperRequest; using TestKeeperRequestPtr = std::shared_ptr; @@ -61,6 +63,8 @@ class TestKeeperStorage void processingThread(); + void writeResponse(const Coordination::ZooKeeperResponsePtr & response); + public: void putRequest(const Coordination::ZooKeeperRequestPtr & request, std::shared_ptr response_out); diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index 6293cbb09fe..05886fc0468 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -133,7 +133,7 @@ struct ZooKeeperCloseResponse final : ZooKeeperResponse OpNum getOpNum() const override { return -11; } }; -struct ZooKeeperCreateRequest final : CreateRequest, ZooKeeperRequest +struct ZooKeeperCreateRequest final : public CreateRequest, ZooKeeperRequest { ZooKeeperCreateRequest() = default; explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 967ea1d29e7..695f5f04382 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -43,6 +43,8 @@ private: void receiveHeartbeatRequest(); void sendHeartbeatResponse(); + + void receiveCreateRequest(); }; } From 93c2ad603426e7de3deea8c5226bbf4884c4e2a1 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 3 Nov 2020 17:49:30 +0300 Subject: [PATCH 013/174] Something work --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 180 ++++++++++++++++++++- src/Common/ZooKeeper/TestKeeperStorage.h | 22 +-- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 18 +++ src/Common/ZooKeeper/ZooKeeperCommon.h | 4 + src/Common/ZooKeeper/ZooKeeperImpl.cpp | 6 +- src/Server/TestKeeperTCPHandler.cpp | 33 +++- src/Server/TestKeeperTCPHandler.h | 6 +- 7 files changed, 252 insertions(+), 17 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 96593b291d6..2024daab7ee 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -1,5 +1,9 @@ #include #include +#include +#include +#include +#include namespace zkutil { @@ -13,6 +17,14 @@ static String parentPath(const String & path) return "/"; } + +TestKeeperStorage::TestKeeperStorage() +{ + container.emplace("/", Node()); + + processing_thread = ThreadFromGlobalPool([this] { processingThread(); }); +} + using Undo = std::function; struct TestKeeperStorageRequest @@ -27,11 +39,22 @@ struct TestKeeperStorageRequest virtual ~TestKeeperStorageRequest() {} }; +struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & /* container */, int64_t /* zxid */) const override + { + return {zk_request->makeResponse(), {}}; + } +}; + struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest { + using TestKeeperStorageRequest::TestKeeperStorageRequest; std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override { + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING CREATE REQUEST"); Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Undo undo; Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr); @@ -104,6 +127,161 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest } }; +struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t /* zxid */) const override + { + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING GET REQUEST"); + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request); + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else + { + response.stat = it->second.stat; + response.data = it->second.data; + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +void TestKeeperStorage::processingThread() +{ + setThreadName("TestKeeperSProc"); + + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "LOOPING IN THREAD"); + try + { + while (!shutdown) + { + RequestInfo info; + + UInt64 max_wait = UInt64(operation_timeout.totalMilliseconds()); + + if (requests_queue.tryPop(info, max_wait)) + { + if (shutdown) + break; + + ++zxid; + + auto zk_request = info.request->zk_request; + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "GOT REQUEST {}", zk_request->getOpNum()); + Coordination::ZooKeeperResponsePtr response; + if (zk_request->xid == -2) + { + response = std::make_shared(); + response->xid = zk_request->xid; + response->zxid = zxid; + } + else + { + zk_request->addRootPath(root_path); + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PROCESSING REQUEST"); + std::tie(response, std::ignore) = info.request->process(container, zxid); + response->xid = zk_request->xid; + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "SENDING XID {}", response->xid); + response->zxid = zxid; + + response->removeRootPath(root_path); + } + + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "SENDING RESPONSE"); + info.response_callback(response); + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "DONE"); + } + } + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + finalize(); + } +} + + +void TestKeeperStorage::finalize() +{ + { + std::lock_guard lock(push_request_mutex); + + if (shutdown) + return; + shutdown = true; + } + try + { + RequestInfo info; + while (requests_queue.tryPop(info)) + { + auto response = info.request->zk_request->makeResponse(); + response->error = Coordination::Error::ZSESSIONEXPIRED; + try + { + info.response_callback(response); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } + } + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } + +} + +TestKeeperStorage::AsyncResponse TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) +{ + auto promise = std::make_shared>(); + auto future = promise->get_future(); + TestKeeperStorageRequestPtr storage_request; + if (request->xid == -2) + storage_request = std::make_shared(request); + else if (request->getOpNum() == 1) + storage_request = std::make_shared(request); + else if (request->getOpNum() == 4) + storage_request = std::make_shared(request); + else + throw Exception(ErrorCodes::LOGICAL_ERROR, "UNKNOWN EVENT WITH OPNUM {}", request->getOpNum()) +; + RequestInfo request_info; + request_info.time = clock::now(); + request_info.request = storage_request; + request_info.response_callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); }; + + std::lock_guard lock(push_request_mutex); + if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); + LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PUSHED"); + return future; +} + + +TestKeeperStorage::~TestKeeperStorage() +{ + try + { + finalize(); + if (processing_thread.joinable()) + processing_thread.join(); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } +} + + - } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index 893f9f2842c..86be2e0eeaf 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -4,19 +4,21 @@ #include #include #include +#include namespace zkutil { using namespace DB; +struct TestKeeperStorageRequest; +using TestKeeperStorageRequestPtr = std::shared_ptr; class TestKeeperStorage { public: - struct TestKeeperRequest; - using TestKeeperRequestPtr = std::shared_ptr; + Poco::Timespan operation_timeout{10000}; std::atomic session_id_counter{0}; struct Node @@ -39,6 +41,7 @@ public: String root_path; std::atomic zxid{0}; + std::atomic shutdown{false}; Watches watches; Watches list_watches; /// Watches for 'list' request (watches on children). @@ -47,26 +50,25 @@ public: struct RequestInfo { - TestKeeperRequestPtr request; - Coordination::ResponseCallback callback; - Coordination::WatchCallback watch; + TestKeeperStorageRequestPtr request; + std::function response_callback; clock::time_point time; }; + std::mutex push_request_mutex; using RequestsQueue = ConcurrentBoundedQueue; RequestsQueue requests_queue{1}; - void pushRequest(RequestInfo && request); - void finalize(); ThreadFromGlobalPool processing_thread; void processingThread(); - void writeResponse(const Coordination::ZooKeeperResponsePtr & response); - public: - void putRequest(const Coordination::ZooKeeperRequestPtr & request, std::shared_ptr response_out); + using AsyncResponse = std::future; + TestKeeperStorage(); + ~TestKeeperStorage(); + AsyncResponse putRequest(const Coordination::ZooKeeperRequestPtr & request); int64_t getSessionID() { diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index eb04536ae00..7d7bbbd8dc3 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include @@ -36,6 +37,8 @@ static void write(bool x, WriteBuffer & out) static void write(const String & s, WriteBuffer & out) { + + LOG_DEBUG(&Poco::Logger::get("LOG"), "S SIZE {}", s.size()); write(int32_t(s.size()), out); out.write(s.data(), s.size()); } @@ -170,6 +173,20 @@ static void read(ACL & acl, ReadBuffer & in) read(acl.id, in); } +void ZooKeeperResponse::write(WriteBuffer & out) const +{ + /// Excessive copy to calculate length. + WriteBufferFromOwnString buf; + LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITING {}", xid); + Coordination::write(xid, buf); + Coordination::write(zxid, buf); + Coordination::write(error, buf); + writeImpl(buf); + LOG_DEBUG(&Poco::Logger::get("LOG"), "BUFFER LENGTH {}", buf.str().length()); + Coordination::write(buf.str(), out); + out.next(); +} + void ZooKeeperRequest::write(WriteBuffer & out) const { /// Excessive copy to calculate length. @@ -247,6 +264,7 @@ void ZooKeeperCreateResponse::readImpl(ReadBuffer & in) void ZooKeeperCreateResponse::writeImpl(WriteBuffer & out) const { + LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITE IMPL ON: {}", path_created); Coordination::write(path_created, out); } diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index 05886fc0468..f40de116da2 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -27,9 +27,13 @@ using OpNum = int32_t; struct ZooKeeperResponse : virtual Response { + XID xid = 0; + int64_t zxid; + virtual ~ZooKeeperResponse() override = default; virtual void readImpl(ReadBuffer &) = 0; virtual void writeImpl(WriteBuffer &) const = 0; + void write(WriteBuffer & out) const; virtual OpNum getOpNum() const = 0; }; diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index f3c8b537cf2..30b30695eea 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #if !defined(ARCADIA_BUILD) # include @@ -750,9 +751,12 @@ void ZooKeeper::receiveEvent() Error err; read(length); + std::cerr << "RECEIVED LENGTH " << length << std::endl; size_t count_before_event = in->count(); read(xid); + std::cerr << "RECEIVED XID " << xid << std::endl; read(zxid); + std::cerr << "RECEIVED ZXID " << zxid << std::endl; read(err); RequestInfo request_info; @@ -806,7 +810,7 @@ void ZooKeeper::receiveEvent() auto it = operations.find(xid); if (it == operations.end()) - throw Exception("Received response for unknown xid", Error::ZRUNTIMEINCONSISTENCY); + throw Exception("Received response for unknown xid " + toString(xid), Error::ZRUNTIMEINCONSISTENCY); /// After this point, we must invoke callback, that we've grabbed from 'operations'. /// Invariant: all callbacks are invoked either in case of success or in case of error. diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index a0679554f64..baf78e1fc59 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -12,6 +12,7 @@ #include #include #include +#include namespace DB { @@ -279,16 +280,23 @@ void TestKeeperTCPHandler::runImpl() while (true) { UInt64 max_wait = operation_timeout.totalMicroseconds(); + using namespace std::chrono_literals; + if (!responses.empty() && responses.front().wait_for(100ms) == std::future_status::ready) + { + auto response = responses.front().get(); + response->write(*out); + responses.pop(); + } + if (in->poll(max_wait)) { receiveHeartbeatRequest(); - sendHeartbeatResponse(); } } } -void TestKeeperTCPHandler::receiveHeartbeatRequest() +bool TestKeeperTCPHandler::receiveHeartbeatRequest() { LOG_DEBUG(log, "Receiving heartbeat event"); int32_t length; @@ -301,12 +309,14 @@ void TestKeeperTCPHandler::receiveHeartbeatRequest() LOG_DEBUG(log, "Received xid {}", xid); + Coordination::ZooKeeperRequestPtr request; if (xid == -2) { int32_t opnum; read(opnum, *in); LOG_DEBUG(log, "RRECEIVED OP NUM {}", opnum); - auto request = std::make_shared(); + request = std::make_shared(); + request->xid = xid; request->readImpl(*in); int32_t readed = in->count() - total_count; if (readed != length) @@ -314,10 +324,25 @@ void TestKeeperTCPHandler::receiveHeartbeatRequest() } else { - LOG_INFO(log, "UNKNOWN EVENT xid:{}", xid); + int32_t opnum; + read(opnum, *in); + LOG_DEBUG(log, "RRECEIVED OP NUM {}", opnum); + if (opnum == 1) + request = std::make_shared(); + else if (opnum == 4) + request = std::make_shared(); + request->readImpl(*in); + request->xid = xid; + int32_t readed = in->count() - total_count; + if (readed != length) + LOG_DEBUG(log, "EXPECTED TO READ {}, BUT GOT {}", length, readed); + LOG_DEBUG(log, "REQUEST PUTTED TO STORAGE"); } + responses.push(test_keeper_storage->putRequest(request)); + LOG_DEBUG(log, "Event received"); + return false; } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 695f5f04382..fd2c6227e73 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -7,6 +7,7 @@ #include #include #include +#include namespace DB { @@ -32,6 +33,9 @@ private: std::shared_ptr test_keeper_storage; Poco::Timespan operation_timeout; + std::queue responses; + + /// Streams for reading/writing from/to client connection socket. std::shared_ptr in; std::shared_ptr out; @@ -41,7 +45,7 @@ private: void sendHandshake(); void receiveHandshake(); - void receiveHeartbeatRequest(); + bool receiveHeartbeatRequest(); void sendHeartbeatResponse(); void receiveCreateRequest(); From 5700e5e46d3c495342149a6484798e4157766fa7 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 3 Nov 2020 17:57:40 +0300 Subject: [PATCH 014/174] Fix response serialization --- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index 7d7bbbd8dc3..5ab22474d72 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -181,7 +181,8 @@ void ZooKeeperResponse::write(WriteBuffer & out) const Coordination::write(xid, buf); Coordination::write(zxid, buf); Coordination::write(error, buf); - writeImpl(buf); + if (error == Error::ZOK) + writeImpl(buf); LOG_DEBUG(&Poco::Logger::get("LOG"), "BUFFER LENGTH {}", buf.str().length()); Coordination::write(buf.str(), out); out.next(); From 598532b58216d1ab1cf9f3ee766778a55476801f Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 3 Nov 2020 18:01:56 +0300 Subject: [PATCH 015/174] Remove strange method --- src/Common/ZooKeeper/TestKeeper.cpp | 3 --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 1 - 2 files changed, 4 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeper.cpp b/src/Common/ZooKeeper/TestKeeper.cpp index 4f7beadef5f..b25e20afeda 100644 --- a/src/Common/ZooKeeper/TestKeeper.cpp +++ b/src/Common/ZooKeeper/TestKeeper.cpp @@ -31,7 +31,6 @@ using Undo = std::function; struct TestKeeperRequest : virtual Request { - virtual bool isMutable() const { return false; } virtual ResponsePtr createResponse() const = 0; virtual std::pair process(TestKeeper::Container & container, int64_t zxid) const = 0; virtual void processWatches(TestKeeper::Watches & /*watches*/, TestKeeper::Watches & /*list_watches*/) const {} @@ -85,7 +84,6 @@ struct TestKeeperRemoveRequest final : RemoveRequest, TestKeeperRequest { TestKeeperRemoveRequest() = default; explicit TestKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} - bool isMutable() const override { return true; } ResponsePtr createResponse() const override; std::pair process(TestKeeper::Container & container, int64_t zxid) const override; @@ -112,7 +110,6 @@ struct TestKeeperSetRequest final : SetRequest, TestKeeperRequest { TestKeeperSetRequest() = default; explicit TestKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} - bool isMutable() const override { return true; } ResponsePtr createResponse() const override; std::pair process(TestKeeper::Container & container, int64_t zxid) const override; diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 2024daab7ee..50564e6ae54 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -34,7 +34,6 @@ struct TestKeeperStorageRequest TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) : zk_request(zk_request_) {} - virtual bool isMutable() const { return false; } virtual std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const = 0; virtual ~TestKeeperStorageRequest() {} }; From a9529e8d6558f54f171906c7f906c7cea91b49fb Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 4 Nov 2020 21:54:55 +0300 Subject: [PATCH 016/174] Bad code but all events --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 347 +++++++++++++++++++-- src/Server/TestKeeperTCPHandler.cpp | 58 +--- src/Server/TestKeeperTCPHandler.h | 5 +- 3 files changed, 328 insertions(+), 82 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 50564e6ae54..a3e637a4ce8 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -4,6 +4,7 @@ #include #include #include +#include namespace zkutil { @@ -17,6 +18,11 @@ static String parentPath(const String & path) return "/"; } +static String baseName(const String & path) +{ + auto rslash_pos = path.rfind('/'); + return path.substr(rslash_pos + 1); +} TestKeeperStorage::TestKeeperStorage() { @@ -152,6 +158,255 @@ struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest } }; +struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperRemoveResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperRemoveRequest & request = dynamic_cast(*zk_request); + Undo undo; + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version != -1 && request.version != it->second.stat.version) + { + response.error = Coordination::Error::ZBADVERSION; + } + else if (it->second.stat.numChildren) + { + response.error = Coordination::Error::ZNOTEMPTY; + } + else + { + auto prev_node = it->second; + container.erase(it); + auto & parent = container.at(parentPath(request.path)); + --parent.stat.numChildren; + ++parent.stat.cversion; + response.error = Coordination::Error::ZOK; + + undo = [prev_node, &container, path = request.path] + { + container.emplace(path, prev_node); + auto & undo_parent = container.at(parentPath(path)); + ++undo_parent.stat.numChildren; + --undo_parent.stat.cversion; + }; + } + + return { response_ptr, undo }; + } +}; + +struct TestKeeperStorageExistsRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperExistsResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperExistsRequest & request = dynamic_cast(*zk_request); + + auto it = container.find(request.path); + if (it != container.end()) + { + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + } + else + { + response.error = Coordination::Error::ZNONODE; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperSetResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperSetRequest & request = dynamic_cast(*zk_request); + Undo undo; + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version == -1 || request.version == it->second.stat.version) + { + auto prev_node = it->second; + + it->second.data = request.data; + ++it->second.stat.version; + it->second.stat.mzxid = zxid; + it->second.stat.mtime = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1); + it->second.data = request.data; + ++container.at(parentPath(request.path)).stat.cversion; + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + + undo = [prev_node, &container, path = request.path] + { + container.at(path) = prev_node; + --container.at(parentPath(path)).stat.cversion; + }; + } + else + { + response.error = Coordination::Error::ZBADVERSION; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperListResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperListRequest & request = dynamic_cast(*zk_request); + Undo undo; + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else + { + auto path_prefix = request.path; + if (path_prefix.empty()) + throw Coordination::Exception("Logical error: path cannot be empty", Coordination::Error::ZSESSIONEXPIRED); + + if (path_prefix.back() != '/') + path_prefix += '/'; + + /// Fairly inefficient. + for (auto child_it = container.upper_bound(path_prefix); + child_it != container.end() && startsWith(child_it->first, path_prefix); + ++child_it) + { + if (parentPath(child_it->first) == request.path) + response.names.emplace_back(baseName(child_it->first)); + } + + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageCheckRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperCheckResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperCheckRequest & request = dynamic_cast(*zk_request); + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version != -1 && request.version != it->second.stat.version) + { + response.error = Coordination::Error::ZBADVERSION; + } + else + { + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest +{ + std::vector concrete_requests; + TestKeeperStorageMultiRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + : TestKeeperStorageRequest(zk_request_) + { + Coordination::ZooKeeperMultiRequest & request = dynamic_cast(*zk_request); + concrete_requests.reserve(request.requests.size()); + + for (const auto & zk_request : request.requests) + { + if (const auto * concrete_request_create = dynamic_cast(zk_request.get())) + { + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + } + else if (const auto * concrete_request_remove = dynamic_cast(zk_request.get())) + { + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + } + else if (const auto * concrete_request_set = dynamic_cast(zk_request.get())) + { + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + } + else if (const auto * concrete_request_check = dynamic_cast(zk_request.get())) + { + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + } + else + throw Coordination::Exception("Illegal command as part of multi ZooKeeper request", Coordination::Error::ZBADARGUMENTS); + } + } + + std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperMultiResponse & response = dynamic_cast(*response_ptr); + std::vector undo_actions; + + try + { + for (const auto & concrete_request : concrete_requests) + { + auto [ cur_response, undo_action ] = concrete_request->process(container, zxid); + response.responses.emplace_back(cur_response); + if (cur_response->error != Coordination::Error::ZOK) + { + response.error = cur_response->error; + + for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) + if (*it) + (*it)(); + + return { response_ptr, {} }; + } + else + undo_actions.emplace_back(std::move(undo_action)); + } + + response.error = Coordination::Error::ZOK; + return { response_ptr, {} }; + } + catch (...) + { + for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) + if (*it) + (*it)(); + throw; + } + } +}; + void TestKeeperStorage::processingThread() { setThreadName("TestKeeperSProc"); @@ -174,24 +429,11 @@ void TestKeeperStorage::processingThread() auto zk_request = info.request->zk_request; LOG_DEBUG(&Poco::Logger::get("STORAGE"), "GOT REQUEST {}", zk_request->getOpNum()); - Coordination::ZooKeeperResponsePtr response; - if (zk_request->xid == -2) - { - response = std::make_shared(); - response->xid = zk_request->xid; - response->zxid = zxid; - } - else - { - zk_request->addRootPath(root_path); - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PROCESSING REQUEST"); - std::tie(response, std::ignore) = info.request->process(container, zxid); - response->xid = zk_request->xid; - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "SENDING XID {}", response->xid); - response->zxid = zxid; - response->removeRootPath(root_path); - } + auto [response, _] = info.request->process(container, zxid); + response->xid = zk_request->xid; + response->zxid = zxid; + response->removeRootPath(root_path); LOG_DEBUG(&Poco::Logger::get("STORAGE"), "SENDING RESPONSE"); info.response_callback(response); @@ -237,23 +479,74 @@ void TestKeeperStorage::finalize() { tryLogCurrentException(__PRETTY_FUNCTION__); } +} + + + + +class TestKeeperWrapperFactory final : private boost::noncopyable +{ + +public: + using Creator = std::function; + using OpNumToRequest = std::unordered_map; + + static TestKeeperWrapperFactory & instance() + { + static TestKeeperWrapperFactory factory; + return factory; + } + + TestKeeperStorageRequestPtr get(const Coordination::ZooKeeperRequestPtr & zk_request) const + { + auto it = op_num_to_request.find(zk_request->getOpNum()); + if (it == op_num_to_request.end()) + throw Coordination::Exception("Unknown operation type " + std::to_string(zk_request->getOpNum()), Coordination::Error::ZBADARGUMENTS); + + return it->second(zk_request); + } + + void registerRequest(int32_t op_num, Creator creator) + { + if (!op_num_to_request.try_emplace(op_num, creator).second) + throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Request with op num {} already registered", op_num); + } + +private: + OpNumToRequest op_num_to_request; + +private: + TestKeeperWrapperFactory(); +}; + +template +void registerTestKeeperRequestWrapper(TestKeeperWrapperFactory & factory) +{ + factory.registerRequest(num, [] (const Coordination::ZooKeeperRequestPtr & zk_request) { return std::make_shared(zk_request); }); +} + + +TestKeeperWrapperFactory::TestKeeperWrapperFactory() +{ + registerTestKeeperRequestWrapper<11, TestKeeperStorageHeartbeatRequest>(*this); + //registerTestKeeperRequestWrapper<100, TestKeeperStorageAuthRequest>(*this); + //registerTestKeeperRequestWrapper<-11, TestKeeperStorageCloseRequest>(*this); + registerTestKeeperRequestWrapper<1, TestKeeperStorageCreateRequest>(*this); + registerTestKeeperRequestWrapper<2, TestKeeperStorageRemoveRequest>(*this); + registerTestKeeperRequestWrapper<3, TestKeeperStorageExistsRequest>(*this); + registerTestKeeperRequestWrapper<4, TestKeeperStorageGetRequest>(*this); + registerTestKeeperRequestWrapper<5, TestKeeperStorageSetRequest>(*this); + registerTestKeeperRequestWrapper<12, TestKeeperStorageListRequest>(*this); + registerTestKeeperRequestWrapper<13, TestKeeperStorageCheckRequest>(*this); + registerTestKeeperRequestWrapper<14, TestKeeperStorageMultiRequest>(*this); } TestKeeperStorage::AsyncResponse TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) { auto promise = std::make_shared>(); auto future = promise->get_future(); - TestKeeperStorageRequestPtr storage_request; - if (request->xid == -2) - storage_request = std::make_shared(request); - else if (request->getOpNum() == 1) - storage_request = std::make_shared(request); - else if (request->getOpNum() == 4) - storage_request = std::make_shared(request); - else - throw Exception(ErrorCodes::LOGICAL_ERROR, "UNKNOWN EVENT WITH OPNUM {}", request->getOpNum()) -; + TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); RequestInfo request_info; request_info.time = clock::now(); request_info.request = storage_request; diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index baf78e1fc59..0ca106d0e32 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -289,19 +289,16 @@ void TestKeeperTCPHandler::runImpl() } if (in->poll(max_wait)) - { - receiveHeartbeatRequest(); - } + receiveRequest(); } } -bool TestKeeperTCPHandler::receiveHeartbeatRequest() +void TestKeeperTCPHandler::receiveRequest() { LOG_DEBUG(log, "Receiving heartbeat event"); int32_t length; read(length, *in); - int32_t total_count = in->count(); LOG_DEBUG(log, "RECEIVED LENGTH {}", length); int32_t xid; LOG_DEBUG(log, "READING XID"); @@ -309,56 +306,15 @@ bool TestKeeperTCPHandler::receiveHeartbeatRequest() LOG_DEBUG(log, "Received xid {}", xid); - Coordination::ZooKeeperRequestPtr request; - if (xid == -2) - { - int32_t opnum; - read(opnum, *in); - LOG_DEBUG(log, "RRECEIVED OP NUM {}", opnum); - request = std::make_shared(); - request->xid = xid; - request->readImpl(*in); - int32_t readed = in->count() - total_count; - if (readed != length) - LOG_DEBUG(log, "EXPECTED TO READ {}, BUT GOT {}", length, readed); - } - else - { - int32_t opnum; - read(opnum, *in); - LOG_DEBUG(log, "RRECEIVED OP NUM {}", opnum); - if (opnum == 1) - request = std::make_shared(); - else if (opnum == 4) - request = std::make_shared(); - request->readImpl(*in); - request->xid = xid; - int32_t readed = in->count() - total_count; - if (readed != length) - LOG_DEBUG(log, "EXPECTED TO READ {}, BUT GOT {}", length, readed); - LOG_DEBUG(log, "REQUEST PUTTED TO STORAGE"); - } - + int32_t opnum; + read(opnum, *in); + Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); + request->xid = xid; + request->readImpl(*in); responses.push(test_keeper_storage->putRequest(request)); LOG_DEBUG(log, "Event received"); - return false; } -void TestKeeperTCPHandler::sendHeartbeatResponse() -{ - LOG_DEBUG(log, "Sending heartbeat event"); - int32_t length = sizeof(int32_t) + sizeof(int64_t) + sizeof(Coordination::Error); - write(length, *out); - int64_t zxid = test_keeper_storage->getZXID(); - int32_t xid = -2; - write(xid, *out); - write(zxid, *out); - write(Coordination::Error::ZOK, *out); - auto response = std::make_shared(); - response->writeImpl(*out); - out->next(); -} - } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index fd2c6227e73..2a796daa4e9 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -45,10 +45,7 @@ private: void sendHandshake(); void receiveHandshake(); - bool receiveHeartbeatRequest(); - void sendHeartbeatResponse(); - - void receiveCreateRequest(); + void receiveRequest(); }; } From 456b0b94c1661a37d52b2261e6725710c693f393 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 9 Nov 2020 09:54:35 +0300 Subject: [PATCH 017/174] Remove some debug --- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 2 -- src/Common/ZooKeeper/ZooKeeperImpl.cpp | 3 --- 2 files changed, 5 deletions(-) diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index 5ab22474d72..b1a11b2d473 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -37,8 +37,6 @@ static void write(bool x, WriteBuffer & out) static void write(const String & s, WriteBuffer & out) { - - LOG_DEBUG(&Poco::Logger::get("LOG"), "S SIZE {}", s.size()); write(int32_t(s.size()), out); out.write(s.data(), s.size()); } diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 30b30695eea..33204686dce 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -751,12 +751,9 @@ void ZooKeeper::receiveEvent() Error err; read(length); - std::cerr << "RECEIVED LENGTH " << length << std::endl; size_t count_before_event = in->count(); read(xid); - std::cerr << "RECEIVED XID " << xid << std::endl; read(zxid); - std::cerr << "RECEIVED ZXID " << zxid << std::endl; read(err); RequestInfo request_info; From 45b089425250edcafc4ef3ce982134c0b448e7db Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 9 Nov 2020 20:50:39 +0300 Subject: [PATCH 018/174] Some fixes --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 8 ++++---- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 2 ++ src/Common/ZooKeeper/ZooKeeperImpl.cpp | 4 ++++ 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index a3e637a4ce8..22a06f67988 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -376,12 +376,14 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest try { + size_t i = 0; for (const auto & concrete_request : concrete_requests) { auto [ cur_response, undo_action ] = concrete_request->process(container, zxid); - response.responses.emplace_back(cur_response); + response.responses[i] = cur_response; if (cur_response->error != Coordination::Error::ZOK) { + std::cerr << "GOT ERROR ON: " << i << " error" << static_cast(cur_response->error) << std::endl; response.error = cur_response->error; for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) @@ -392,6 +394,7 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest } else undo_actions.emplace_back(std::move(undo_action)); + ++i; } response.error = Coordination::Error::ZOK; @@ -482,9 +485,6 @@ void TestKeeperStorage::finalize() } - - - class TestKeeperWrapperFactory final : private boost::noncopyable { diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index b1a11b2d473..f26a72a3d55 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -544,12 +544,14 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const { + std::cerr << "WRITING MULTIRESPONSE " << responses.size() << std::endl; for (auto & response : responses) { const ZooKeeperResponse & zk_response = dynamic_cast(*response); OpNum op_num = zk_response.getOpNum(); bool done = false; Error op_error = zk_response.error; + std::cerr << "WRITING OP ERROR:" << static_cast(op_error) << std::endl; Coordination::write(op_num, out); Coordination::write(done, out); diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 33204686dce..4d81748ca3d 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -824,11 +824,15 @@ void ZooKeeper::receiveEvent() try { + std::cerr << "READING RESPONSE FOR REQUEST ID:" << request_info.request->getOpNum() << std::endl; if (!response) response = request_info.request->makeResponse(); if (err != Error::ZOK) + { + std::cerr << "GOT ERROR:" << static_cast(err) << std::endl; response->error = err; + } else { response->readImpl(*in); From ba449d7c929a8be63d09290e8f790b5733738f00 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 9 Nov 2020 20:51:22 +0300 Subject: [PATCH 019/174] Add small test --- utils/zookeeper-test/CMakeLists.txt | 3 + utils/zookeeper-test/main.cpp | 129 ++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) create mode 100644 utils/zookeeper-test/CMakeLists.txt create mode 100644 utils/zookeeper-test/main.cpp diff --git a/utils/zookeeper-test/CMakeLists.txt b/utils/zookeeper-test/CMakeLists.txt new file mode 100644 index 00000000000..aa26c840ba3 --- /dev/null +++ b/utils/zookeeper-test/CMakeLists.txt @@ -0,0 +1,3 @@ +add_executable(zk-test main.cpp) +target_link_libraries(zk-test PRIVATE clickhouse_common_zookeeper) +INSTALL(TARGETS zk-test RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse-utils) diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp new file mode 100644 index 00000000000..694a6205adb --- /dev/null +++ b/utils/zookeeper-test/main.cpp @@ -0,0 +1,129 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +using namespace std; + +void checkEq(zkutil::ZooKeeper & zk, const std::string & path, const std::string & expected) +{ + auto result = zk.get(path); + if (result != expected) + throw std::runtime_error(fmt::format("Data on path '{}' = '{}' doesn't match expected '{}'", + path, result, expected)); +} + +void checkExists(zkutil::ZooKeeper & zk, const std::string & path) +{ + if (!zk.exists(path)) + throw std::runtime_error(fmt::format("Path '{}' doesn't exists", path)); +} + +void testCreateGetExistsNode(zkutil::ZooKeeper & zk) +{ + zk.create("/data", "test_string", zkutil::CreateMode::Persistent); + zk.create("/data/seq-", "another_string", zkutil::CreateMode::PersistentSequential); + checkEq(zk, "/data", "test_string"); + checkExists(zk, "/data/seq-0000000000"); + checkEq(zk, "/data/seq-0000000000", "another_string"); +} + +void testCreateSetNode(zkutil::ZooKeeper & zk) +{ + zk.create("/data/set", "sssss", zkutil::CreateMode::Persistent); + checkEq(zk, "/data/set", "sssss"); + zk.set("/data/set", "qqqqq"); + checkEq(zk, "/data/set", "qqqqq"); +} + +void testCreateList(zkutil::ZooKeeper & zk) +{ + zk.create("/data/lst", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d1", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d2", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d3", "", zkutil::CreateMode::Persistent); + auto children = zk.getChildren("/data/lst"); + if (children.size() != 3) + throw std::runtime_error("Children of /data/lst doesn't equal to three"); + for (size_t i = 0; i < children.size(); ++i) + { + if (children[i] != "d" + std::to_string(i + 1)) + throw std::runtime_error(fmt::format("Incorrect children #{} got {}, expected {}", i, children[i], "d" + std::to_string(i + 1))); + } +} + +void testCreateSetVersionRequest(zkutil::ZooKeeper & zk) +{ + zk.create("/data/check_data", "d", zkutil::CreateMode::Persistent); + Coordination::Stat stat; + std::string result = zk.get("/data/check_data", &stat); + try + { + zk.set("/data/check_data", "e", stat.version + 2); + std::terminate(); + } + catch (...) + { + std::cerr << "Got exception on incorrect version (it's ok)\n"; + } + + checkEq(zk, "/data/check_data", "d"); + zk.set("/data/check_data", "e", stat.version); + + checkEq(zk, "/data/check_data", "e"); +} + +void testMultiRequest(zkutil::ZooKeeper & zk) +{ + Coordination::Requests requests; + requests.push_back(zkutil::makeCreateRequest("/data/multirequest", "aaa", zkutil::CreateMode::Persistent)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "bbb", -1)); + zk.multi(requests); + + try + { + requests.clear(); + requests.push_back(zkutil::makeCreateRequest("/data/multirequest", "qweqwe", zkutil::CreateMode::Persistent)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "bbb", -1)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "ccc", -1)); + zk.multi(requests); + std::terminate(); + } + catch(...) + { + std::cerr << "Got exception on multy request (it's ok)\n"; + } + + checkEq(zk, "/data/multirequest", "bbb"); +} + +int main(int argc, char *argv[]) { + if (argc != 2) + { + std::cerr << "usage: " << argv[0] << " hosts" << std::endl; + return 2; + } + Poco::AutoPtr channel = new Poco::ConsoleChannel(std::cerr); + Poco::Logger::root().setChannel(channel); + Poco::Logger::root().setLevel("trace"); + + zkutil::ZooKeeper zk(argv[1]); + + testCreateGetExistsNode(zk); + testCreateSetNode(zk); + testCreateList(zk); + testCreateSetVersionRequest(zk); + testMultiRequest(zk); + + //zk.removeRecursive("/data"); + return 0; +} From 09fac0da9d37b3365bd9a574535816cb4123277b Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 9 Nov 2020 21:16:02 +0300 Subject: [PATCH 020/174] Better --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 1 - src/Common/ZooKeeper/ZooKeeperCommon.cpp | 3 ++- src/Common/ZooKeeper/ZooKeeperImpl.cpp | 1 + utils/CMakeLists.txt | 1 + utils/zookeeper-test/main.cpp | 22 +++++++++++++++------- 5 files changed, 19 insertions(+), 9 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 22a06f67988..5ac8daf2e09 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -384,7 +384,6 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest if (cur_response->error != Coordination::Error::ZOK) { std::cerr << "GOT ERROR ON: " << i << " error" << static_cast(cur_response->error) << std::endl; - response.error = cur_response->error; for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) if (*it) diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index f26a72a3d55..604bb2137c4 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -556,7 +556,8 @@ void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const Coordination::write(op_num, out); Coordination::write(done, out); Coordination::write(op_error, out); - zk_response.writeImpl(out); + if (op_error == Error::ZOK) + zk_response.writeImpl(out); } /// Footer. diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 4d81748ca3d..fef0b7063f5 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -835,6 +835,7 @@ void ZooKeeper::receiveEvent() } else { + std::cerr << "NO ERROR RECEIVED\n"; response->readImpl(*in); response->removeRootPath(root_path); } diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index 0987d64abed..9e1872ded7b 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -19,6 +19,7 @@ if (NOT DEFINED ENABLE_UTILS OR ENABLE_UTILS) add_subdirectory (iotest) add_subdirectory (corrector_utf8) add_subdirectory (zookeeper-cli) + add_subdirectory (zookeeper-test) add_subdirectory (zookeeper-dump-tree) add_subdirectory (zookeeper-remove-by-list) add_subdirectory (zookeeper-create-entry-to-download-part) diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp index 694a6205adb..802fbc17708 100644 --- a/utils/zookeeper-test/main.cpp +++ b/utils/zookeeper-test/main.cpp @@ -107,6 +107,7 @@ void testMultiRequest(zkutil::ZooKeeper & zk) } int main(int argc, char *argv[]) { + if (argc != 2) { std::cerr << "usage: " << argv[0] << " hosts" << std::endl; @@ -118,12 +119,19 @@ int main(int argc, char *argv[]) { zkutil::ZooKeeper zk(argv[1]); - testCreateGetExistsNode(zk); - testCreateSetNode(zk); - testCreateList(zk); - testCreateSetVersionRequest(zk); - testMultiRequest(zk); - - //zk.removeRecursive("/data"); + try + { + zk.tryRemoveRecursive("/data"); + testCreateGetExistsNode(zk); + testCreateSetNode(zk); + testCreateList(zk); + testCreateSetVersionRequest(zk); + testMultiRequest(zk); + } + catch(...) + { + zk.tryRemoveRecursive("/data"); + throw; + } return 0; } From d8f515c4f31026fb4089ce147eed95fde4f42053 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 10 Nov 2020 16:43:10 +0300 Subject: [PATCH 021/174] Almost watches --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 85 +++++++++++++++++++++- src/Common/ZooKeeper/TestKeeperStorage.h | 13 +++- src/Common/ZooKeeper/ZooKeeperImpl.cpp | 9 ++- src/Server/TestKeeperTCPHandler.cpp | 30 ++++++-- src/Server/TestKeeperTCPHandler.h | 3 +- 5 files changed, 122 insertions(+), 18 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 5ac8daf2e09..9ddb7a7e82f 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -24,6 +24,37 @@ static String baseName(const String & path) return path.substr(rslash_pos + 1); } +static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) +{ + Coordination::ZooKeeperWatchResponse watch_response; + watch_response.path = path; + watch_response.xid = -1; + + auto it = watches.find(watch_response.path); + if (it != watches.end()) + { + for (auto & callback : it->second) + if (callback) + callback(std::make_shared(watch_response)); + + watches.erase(it); + } + + Coordination::ZooKeeperWatchResponse watch_list_response; + watch_list_response.path = parentPath(path); + watch_list_response.xid = -1; + + it = list_watches.find(watch_list_response.path); + if (it != list_watches.end()) + { + for (auto & callback : it->second) + if (callback) + callback(std::make_shared(watch_list_response)); + + list_watches.erase(it); + } +} + TestKeeperStorage::TestKeeperStorage() { container.emplace("/", Node()); @@ -41,6 +72,8 @@ struct TestKeeperStorageRequest : zk_request(zk_request_) {} virtual std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const = 0; + virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} + virtual ~TestKeeperStorageRequest() {} }; @@ -57,6 +90,12 @@ struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches); + } + std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override { LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING CREATE REQUEST"); @@ -201,6 +240,11 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest return { response_ptr, undo }; } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches); + } }; struct TestKeeperStorageExistsRequest final : public TestKeeperStorageRequest @@ -268,6 +312,12 @@ struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest return { response_ptr, {} }; } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches); + } + }; struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest @@ -407,6 +457,12 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest throw; } } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + for (const auto & generic_request : concrete_requests) + generic_request->processWatches(watches, list_watches); + } }; void TestKeeperStorage::processingThread() @@ -427,12 +483,26 @@ void TestKeeperStorage::processingThread() if (shutdown) break; - ++zxid; + if (info.watch_callback) + { + auto & watches_type = dynamic_cast(info.request->zk_request.get()) + ? list_watches + : watches; + + watches_type[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + } auto zk_request = info.request->zk_request; LOG_DEBUG(&Poco::Logger::get("STORAGE"), "GOT REQUEST {}", zk_request->getOpNum()); + info.request->zk_request->addRootPath(root_path); auto [response, _] = info.request->process(container, zxid); + if (response->error == Coordination::Error::ZOK) + { + info.request->processWatches(watches, list_watches); + } + + ++zxid; response->xid = zk_request->xid; response->zxid = zxid; response->removeRootPath(root_path); @@ -541,7 +611,7 @@ TestKeeperWrapperFactory::TestKeeperWrapperFactory() registerTestKeeperRequestWrapper<14, TestKeeperStorageMultiRequest>(*this); } -TestKeeperStorage::AsyncResponse TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) +TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) { auto promise = std::make_shared>(); auto future = promise->get_future(); @@ -550,12 +620,19 @@ TestKeeperStorage::AsyncResponse TestKeeperStorage::putRequest(const Coordinatio request_info.time = clock::now(); request_info.request = storage_request; request_info.response_callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); }; + std::optional watch_future; + if (request->has_watch) + { + auto watch_promise = std::make_shared>(); + watch_future.emplace(watch_promise->get_future()); + request_info.watch_callback = [watch_promise] (const Coordination::ZooKeeperResponsePtr & response) { watch_promise->set_value(response); }; + } std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PUSHED"); - return future; + //LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PUSHED"); + return ResponsePair{std::move(future), std::move(watch_future)}; } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index 86be2e0eeaf..edcae28af85 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -12,6 +12,7 @@ namespace zkutil using namespace DB; struct TestKeeperStorageRequest; using TestKeeperStorageRequestPtr = std::shared_ptr; +using ResponseCallback = std::function; class TestKeeperStorage { @@ -33,7 +34,7 @@ public: using Container = std::map; - using WatchCallbacks = std::vector; + using WatchCallbacks = std::vector; using Watches = std::map; Container container; @@ -51,7 +52,8 @@ public: struct RequestInfo { TestKeeperStorageRequestPtr request; - std::function response_callback; + ResponseCallback response_callback; + ResponseCallback watch_callback; clock::time_point time; }; std::mutex push_request_mutex; @@ -68,7 +70,12 @@ public: using AsyncResponse = std::future; TestKeeperStorage(); ~TestKeeperStorage(); - AsyncResponse putRequest(const Coordination::ZooKeeperRequestPtr & request); + struct ResponsePair + { + AsyncResponse response; + std::optional watch_response; + }; + ResponsePair putRequest(const Coordination::ZooKeeperRequestPtr & request); int64_t getSessionID() { diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index fef0b7063f5..093b2bec7df 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -654,11 +654,14 @@ void ZooKeeper::sendThread() } if (expired) + { break; + } info.request->addRootPath(root_path); info.request->probably_sent = true; + std::cerr << "SENDING GENERAL REQUEST\n"; info.request->write(*out); if (info.request->xid == close_xid) @@ -899,7 +902,6 @@ void ZooKeeper::finalize(bool error_send, bool error_receive) if (expired) return; - expired = true; } active_session_metric_increment.destroy(); @@ -922,6 +924,11 @@ void ZooKeeper::finalize(bool error_send, bool error_receive) send_thread.join(); } + { + std::lock_guard lock(push_request_mutex); + expired = true; + } + try { /// This will also wakeup the receiving thread. diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 0ca106d0e32..38d9f7e54c5 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -289,31 +289,45 @@ void TestKeeperTCPHandler::runImpl() } if (in->poll(max_wait)) - receiveRequest(); + { + bool close_received = receiveRequest(); + if (close_received) + { + LOG_DEBUG(log, "Received close request"); + break; + } + } } } -void TestKeeperTCPHandler::receiveRequest() +bool TestKeeperTCPHandler::receiveRequest() { - LOG_DEBUG(log, "Receiving heartbeat event"); + LOG_DEBUG(log, "Receiving event"); int32_t length; read(length, *in); - LOG_DEBUG(log, "RECEIVED LENGTH {}", length); + //LOG_DEBUG(log, "RECEIVED LENGTH {}", length); int32_t xid; - LOG_DEBUG(log, "READING XID"); + //LOG_DEBUG(log, "READING XID"); read(xid, *in); - LOG_DEBUG(log, "Received xid {}", xid); + //LOG_DEBUG(log, "Received xid {}", xid); int32_t opnum; read(opnum, *in); + if (opnum == -11) + return true; + Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); request->xid = xid; request->readImpl(*in); - responses.push(test_keeper_storage->putRequest(request)); + auto request_future_responses = test_keeper_storage->putRequest(request); + responses.push(std::move(request_future_responses.response)); + if (request_future_responses.watch_response) + responses.push(std::move(*request_future_responses.watch_response)); - LOG_DEBUG(log, "Event received"); + return false; + //LOG_DEBUG(log, "Event received"); } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 2a796daa4e9..b50af3363c1 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -35,7 +35,6 @@ private: std::queue responses; - /// Streams for reading/writing from/to client connection socket. std::shared_ptr in; std::shared_ptr out; @@ -45,7 +44,7 @@ private: void sendHandshake(); void receiveHandshake(); - void receiveRequest(); + bool receiveRequest(); }; } From bb3b420057609620fa39d3f7468b929f0da3e65b Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 11:51:54 +0300 Subject: [PATCH 022/174] Ugly working code --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 12 +++++++ src/Common/ZooKeeper/ZooKeeperImpl.cpp | 16 ++++++--- src/Server/TestKeeperTCPHandler.cpp | 29 +++++++++++++--- src/Server/TestKeeperTCPHandler.h | 1 + utils/zookeeper-test/main.cpp | 39 ++++++++++++++++++++++ 5 files changed, 88 insertions(+), 9 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 9ddb7a7e82f..8f80fa1a0c5 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -30,12 +30,18 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watch_response.path = path; watch_response.xid = -1; + std::cerr << "WATCHES SIZE:" << watches.size() << " path:" << path << std::endl; auto it = watches.find(watch_response.path); if (it != watches.end()) { for (auto & callback : it->second) + { if (callback) + { + std::cerr << "CALLING WATCH CALLBACK\n"; callback(std::make_shared(watch_response)); + } + } watches.erase(it); } @@ -44,12 +50,18 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watch_list_response.path = parentPath(path); watch_list_response.xid = -1; + std::cerr << "LIST WATCHES SIZE:" << list_watches.size() << " path:" << path << std::endl; it = list_watches.find(watch_list_response.path); if (it != list_watches.end()) { for (auto & callback : it->second) + { if (callback) + { + std::cerr << "Calling list watch callback\n" << std::endl; callback(std::make_shared(watch_list_response)); + } + } list_watches.erase(it); } diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 17999044a77..8e10adb4500 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -650,6 +650,7 @@ void ZooKeeper::sendThread() if (info.watch) { info.request->has_watch = true; + std::cerr << "REQUEST" << info.request->getOpNum() << " HAS WATCH" << std::endl; CurrentMetrics::add(CurrentMetrics::ZooKeeperWatch); } @@ -661,7 +662,7 @@ void ZooKeeper::sendThread() info.request->addRootPath(root_path); info.request->probably_sent = true; - std::cerr << "SENDING GENERAL REQUEST\n"; + std::cerr << "SENDING GENERAL REQUEST:" << info.request->getOpNum() << std::endl; info.request->write(*out); /// We sent close request, exit @@ -729,7 +730,9 @@ void ZooKeeper::receiveThread() else { if (earliest_operation) - throw Exception("Operation timeout (no response) for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); + { + throw Exception("Operation timeout (no response) for request " + std::to_string(earliest_operation->request->getOpNum()) + " for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); + } waited += max_wait; if (waited >= session_timeout.totalMicroseconds()) throw Exception("Nothing is received in session timeout", Error::ZOPERATIONTIMEOUT); @@ -772,6 +775,7 @@ void ZooKeeper::receiveEvent() } else if (xid == watch_xid) { + std::cerr << "Receiving watch\n"; ProfileEvents::increment(ProfileEvents::ZooKeeperWatchResponse); response = std::make_shared(); @@ -828,21 +832,23 @@ void ZooKeeper::receiveEvent() try { - std::cerr << "READING RESPONSE FOR REQUEST ID:" << request_info.request->getOpNum() << std::endl; + std::cerr << "READING RESPONSE FOR REQUEST\n"; if (!response) response = request_info.request->makeResponse(); if (err != Error::ZOK) { - std::cerr << "GOT ERROR:" << static_cast(err) << std::endl; + //std::cerr << "GOT ERROR:" << static_cast(err) << std::endl; response->error = err; } else { - std::cerr << "NO ERROR RECEIVED\n"; + //std::cerr << "NO ERROR RECEIVED\n"; response->readImpl(*in); response->removeRootPath(root_path); } + if (request_info.request) + std::cerr << "Response Request ID" << request_info.request->getOpNum() << std::endl; /// Instead of setting the watch in sendEvent, set it in receiveEvent because need to check the response. /// The watch shouldn't be set if the node does not exist and it will never exist like sequential ephemeral nodes. diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 38d9f7e54c5..82949dfc547 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -279,16 +279,33 @@ void TestKeeperTCPHandler::runImpl() while (true) { - UInt64 max_wait = operation_timeout.totalMicroseconds(); + //UInt64 max_wait = operation_timeout.totalMicroseconds(); using namespace std::chrono_literals; + LOG_DEBUG(log, "TRYING TO GET RESPONSE (size {})", responses.size()); if (!responses.empty() && responses.front().wait_for(100ms) == std::future_status::ready) { auto response = responses.front().get(); + + LOG_DEBUG(log, "Writing response bytes to socket {}", response->getOpNum()); response->write(*out); responses.pop(); + LOG_DEBUG(log, "Responses size {}", responses.size()); + } + for (auto it = watch_responses.begin(); it != watch_responses.end();) + { + if (it->wait_for(0s) == std::future_status::ready) + { + it->get()->write(*out); + it = watch_responses.erase(it); + } + else + { + ++it; + } } - if (in->poll(max_wait)) + LOG_DEBUG(log, "WAITING ON POLL"); + if (in->poll(100 * 1000)) { bool close_received = receiveRequest(); if (close_received) @@ -297,6 +314,10 @@ void TestKeeperTCPHandler::runImpl() break; } } + else + { + //LOG_DEBUG(log, "NOTHING POLLED"); + } } } @@ -324,10 +345,10 @@ bool TestKeeperTCPHandler::receiveRequest() auto request_future_responses = test_keeper_storage->putRequest(request); responses.push(std::move(request_future_responses.response)); if (request_future_responses.watch_response) - responses.push(std::move(*request_future_responses.watch_response)); + watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); + LOG_DEBUG(log, "Responses size {}", responses.size()); return false; - //LOG_DEBUG(log, "Event received"); } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index b50af3363c1..7eea5419006 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -34,6 +34,7 @@ private: Poco::Timespan operation_timeout; std::queue responses; + std::vector watch_responses; /// Streams for reading/writing from/to client connection socket. std::shared_ptr in; diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp index 802fbc17708..a924465d02a 100644 --- a/utils/zookeeper-test/main.cpp +++ b/utils/zookeeper-test/main.cpp @@ -82,6 +82,43 @@ void testCreateSetVersionRequest(zkutil::ZooKeeper & zk) checkEq(zk, "/data/check_data", "e"); } +void testCreateSetWatchEvent(zkutil::ZooKeeper & zk) +{ + + std::shared_ptr event = std::make_shared(); + zk.create("/data/nodeforwatch", "", zkutil::CreateMode::Persistent); + Coordination::Stat stat; + zk.get("/data/nodeforwatch", &stat, event); + + if (event->tryWait(300)) + throw std::runtime_error(fmt::format("Event for path {} was set without any actions", "/data/nodeforwatch")); + + zk.set("/data/nodeforwatch", "x"); + if (!event->tryWait(300)) + throw std::runtime_error(fmt::format("Event for path {} was not set after set", "/data/nodeforwatch")); + else + std::cerr << "Event was set well\n"; +} + +void testCreateListWatchEvent(zkutil::ZooKeeper & zk) +{ + std::shared_ptr event = std::make_shared(); + std::string path = "/data/pathforwatch"; + zk.create(path, "", zkutil::CreateMode::Persistent); + zk.create(path + "/n1", "", zkutil::CreateMode::Persistent); + zk.create(path + "/n2", "", zkutil::CreateMode::Persistent); + zk.getChildren(path, nullptr, event); + + if (event->tryWait(300)) + throw std::runtime_error(fmt::format("ListEvent for path {} was set without any actions", path)); + + zk.create(path + "/n3", "", zkutil::CreateMode::Persistent); + if (!event->tryWait(300)) + throw std::runtime_error(fmt::format("ListEvent for path {} was not set after create", path)); + else + std::cerr << "ListEvent was set well\n"; +} + void testMultiRequest(zkutil::ZooKeeper & zk) { Coordination::Requests requests; @@ -127,6 +164,8 @@ int main(int argc, char *argv[]) { testCreateList(zk); testCreateSetVersionRequest(zk); testMultiRequest(zk); + testCreateSetWatchEvent(zk); + testCreateListWatchEvent(zk); } catch(...) { From 0388006ed672a570925993ca43c5a60860ab63de Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 12:05:45 +0300 Subject: [PATCH 023/174] Less garbage --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 21 +-------------------- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 6 ++---- src/Common/ZooKeeper/ZooKeeperImpl.cpp | 9 ++------- src/Server/TestKeeperTCPHandler.cpp | 13 +++++++------ 4 files changed, 12 insertions(+), 37 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 8f80fa1a0c5..b60144f4a87 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -30,18 +30,12 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watch_response.path = path; watch_response.xid = -1; - std::cerr << "WATCHES SIZE:" << watches.size() << " path:" << path << std::endl; auto it = watches.find(watch_response.path); if (it != watches.end()) { for (auto & callback : it->second) - { if (callback) - { - std::cerr << "CALLING WATCH CALLBACK\n"; callback(std::make_shared(watch_response)); - } - } watches.erase(it); } @@ -50,18 +44,12 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watch_list_response.path = parentPath(path); watch_list_response.xid = -1; - std::cerr << "LIST WATCHES SIZE:" << list_watches.size() << " path:" << path << std::endl; it = list_watches.find(watch_list_response.path); if (it != list_watches.end()) { for (auto & callback : it->second) - { if (callback) - { - std::cerr << "Calling list watch callback\n" << std::endl; callback(std::make_shared(watch_list_response)); - } - } list_watches.erase(it); } @@ -110,7 +98,6 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override { - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING CREATE REQUEST"); Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Undo undo; Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr); @@ -188,7 +175,7 @@ struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest using TestKeeperStorageRequest::TestKeeperStorageRequest; std::pair process(TestKeeperStorage::Container & container, int64_t /* zxid */) const override { - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING GET REQUEST"); + //LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING GET REQUEST"); Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr); Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request); @@ -445,8 +432,6 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest response.responses[i] = cur_response; if (cur_response->error != Coordination::Error::ZOK) { - std::cerr << "GOT ERROR ON: " << i << " error" << static_cast(cur_response->error) << std::endl; - for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) if (*it) (*it)(); @@ -481,7 +466,6 @@ void TestKeeperStorage::processingThread() { setThreadName("TestKeeperSProc"); - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "LOOPING IN THREAD"); try { while (!shutdown) @@ -505,7 +489,6 @@ void TestKeeperStorage::processingThread() } auto zk_request = info.request->zk_request; - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "GOT REQUEST {}", zk_request->getOpNum()); info.request->zk_request->addRootPath(root_path); auto [response, _] = info.request->process(container, zxid); @@ -519,9 +502,7 @@ void TestKeeperStorage::processingThread() response->zxid = zxid; response->removeRootPath(root_path); - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "SENDING RESPONSE"); info.response_callback(response); - LOG_DEBUG(&Poco::Logger::get("STORAGE"), "DONE"); } } } diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index 604bb2137c4..cb61f22fb3c 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -175,13 +175,13 @@ void ZooKeeperResponse::write(WriteBuffer & out) const { /// Excessive copy to calculate length. WriteBufferFromOwnString buf; - LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITING {}", xid); + //LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITING {}", xid); Coordination::write(xid, buf); Coordination::write(zxid, buf); Coordination::write(error, buf); if (error == Error::ZOK) writeImpl(buf); - LOG_DEBUG(&Poco::Logger::get("LOG"), "BUFFER LENGTH {}", buf.str().length()); + //LOG_DEBUG(&Poco::Logger::get("LOG"), "BUFFER LENGTH {}", buf.str().length()); Coordination::write(buf.str(), out); out.next(); } @@ -544,14 +544,12 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const { - std::cerr << "WRITING MULTIRESPONSE " << responses.size() << std::endl; for (auto & response : responses) { const ZooKeeperResponse & zk_response = dynamic_cast(*response); OpNum op_num = zk_response.getOpNum(); bool done = false; Error op_error = zk_response.error; - std::cerr << "WRITING OP ERROR:" << static_cast(op_error) << std::endl; Coordination::write(op_num, out); Coordination::write(done, out); diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 8e10adb4500..23a5c6c4301 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -650,7 +650,7 @@ void ZooKeeper::sendThread() if (info.watch) { info.request->has_watch = true; - std::cerr << "REQUEST" << info.request->getOpNum() << " HAS WATCH" << std::endl; + //std::cerr << "REQUEST" << info.request->getOpNum() << " HAS WATCH" << std::endl; CurrentMetrics::add(CurrentMetrics::ZooKeeperWatch); } @@ -662,7 +662,7 @@ void ZooKeeper::sendThread() info.request->addRootPath(root_path); info.request->probably_sent = true; - std::cerr << "SENDING GENERAL REQUEST:" << info.request->getOpNum() << std::endl; + //std::cerr << "SENDING GENERAL REQUEST:" << info.request->getOpNum() << std::endl; info.request->write(*out); /// We sent close request, exit @@ -775,7 +775,6 @@ void ZooKeeper::receiveEvent() } else if (xid == watch_xid) { - std::cerr << "Receiving watch\n"; ProfileEvents::increment(ProfileEvents::ZooKeeperWatchResponse); response = std::make_shared(); @@ -832,7 +831,6 @@ void ZooKeeper::receiveEvent() try { - std::cerr << "READING RESPONSE FOR REQUEST\n"; if (!response) response = request_info.request->makeResponse(); @@ -847,9 +845,6 @@ void ZooKeeper::receiveEvent() response->readImpl(*in); response->removeRootPath(root_path); } - if (request_info.request) - std::cerr << "Response Request ID" << request_info.request->getOpNum() << std::endl; - /// Instead of setting the watch in sendEvent, set it in receiveEvent because need to check the response. /// The watch shouldn't be set if the node does not exist and it will never exist like sequential ephemeral nodes. /// By using getData() instead of exists(), a watch won't be set if the node doesn't exist. diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 82949dfc547..1e7f69dc14f 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -273,6 +273,7 @@ void TestKeeperTCPHandler::runImpl() catch (const Exception & e) /// Typical for an incorrect username, password, or address. { LOG_DEBUG(log, "Cannot receive handshake {}", e.displayText()); + return; } sendHandshake(); @@ -281,15 +282,15 @@ void TestKeeperTCPHandler::runImpl() { //UInt64 max_wait = operation_timeout.totalMicroseconds(); using namespace std::chrono_literals; - LOG_DEBUG(log, "TRYING TO GET RESPONSE (size {})", responses.size()); + //LOG_DEBUG(log, "TRYING TO GET RESPONSE (size {})", responses.size()); if (!responses.empty() && responses.front().wait_for(100ms) == std::future_status::ready) { auto response = responses.front().get(); - LOG_DEBUG(log, "Writing response bytes to socket {}", response->getOpNum()); + //LOG_DEBUG(log, "Writing response bytes to socket {}", response->getOpNum()); response->write(*out); responses.pop(); - LOG_DEBUG(log, "Responses size {}", responses.size()); + //LOG_DEBUG(log, "Responses size {}", responses.size()); } for (auto it = watch_responses.begin(); it != watch_responses.end();) { @@ -304,7 +305,7 @@ void TestKeeperTCPHandler::runImpl() } } - LOG_DEBUG(log, "WAITING ON POLL"); + //LOG_DEBUG(log, "WAITING ON POLL"); if (in->poll(100 * 1000)) { bool close_received = receiveRequest(); @@ -324,7 +325,7 @@ void TestKeeperTCPHandler::runImpl() bool TestKeeperTCPHandler::receiveRequest() { - LOG_DEBUG(log, "Receiving event"); + //LOG_DEBUG(log, "Receiving event"); int32_t length; read(length, *in); //LOG_DEBUG(log, "RECEIVED LENGTH {}", length); @@ -347,7 +348,7 @@ bool TestKeeperTCPHandler::receiveRequest() if (request_future_responses.watch_response) watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); - LOG_DEBUG(log, "Responses size {}", responses.size()); + //LOG_DEBUG(log, "Responses size {}", responses.size()); return false; } From 66236d6ebb8af22eed790ddea1aa2f83d5ef404d Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 16:07:06 +0300 Subject: [PATCH 024/174] Refactored a little --- programs/server/Server.cpp | 274 ++++++++++---------- programs/server/Server.h | 14 + src/Common/ZooKeeper/TestKeeperStorage.cpp | 70 +++-- src/Common/ZooKeeper/TestKeeperStorage.h | 3 +- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 197 ++------------ src/Common/ZooKeeper/ZooKeeperCommon.h | 56 ++-- src/Common/ZooKeeper/ZooKeeperConstants.cpp | 64 +++++ src/Common/ZooKeeper/ZooKeeperConstants.h | 47 ++++ src/Common/ZooKeeper/ZooKeeperIO.cpp | 135 ++++++++++ src/Common/ZooKeeper/ZooKeeperIO.h | 73 ++++++ src/Common/ZooKeeper/ZooKeeperImpl.cpp | 134 ++-------- src/Common/ZooKeeper/ZooKeeperImpl.h | 3 - src/Server/TestKeeperTCPHandler.cpp | 246 +++--------------- src/Server/TestKeeperTCPHandler.h | 5 +- 14 files changed, 625 insertions(+), 696 deletions(-) create mode 100644 src/Common/ZooKeeper/ZooKeeperConstants.cpp create mode 100644 src/Common/ZooKeeper/ZooKeeperConstants.h create mode 100644 src/Common/ZooKeeper/ZooKeeperIO.cpp create mode 100644 src/Common/ZooKeeper/ZooKeeperIO.h diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 344082086e9..597b6c43fe5 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -180,6 +180,85 @@ static std::string getUserName(uid_t user_id) return toString(user_id); } +Poco::Net::SocketAddress makeSocketAddress(const std::string & host, UInt16 port, Poco::Logger * log) +{ + Poco::Net::SocketAddress socket_address; + try + { + socket_address = Poco::Net::SocketAddress(host, port); + } + catch (const Poco::Net::DNSException & e) + { + const auto code = e.code(); + if (code == EAI_FAMILY +#if defined(EAI_ADDRFAMILY) + || code == EAI_ADDRFAMILY +#endif + ) + { + LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " + "If it is an IPv6 address and your host has disabled IPv6, then consider to " + "specify IPv4 address to listen in element of configuration " + "file. Example: 0.0.0.0", + host, e.code(), e.message()); + } + + throw; + } + return socket_address; +} + +Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) +{ + auto address = makeSocketAddress(host, port, &logger()); +#if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100 + if (secure) + /// Bug in old (<1.9.1) poco, listen() after bind() with reusePort param will fail because have no implementation in SecureServerSocketImpl + /// https://github.com/pocoproject/poco/pull/2257 + socket.bind(address, /* reuseAddress = */ true); + else +#endif +#if POCO_VERSION < 0x01080000 + socket.bind(address, /* reuseAddress = */ true); +#else + socket.bind(address, /* reuseAddress = */ true, /* reusePort = */ config().getBool("listen_reuse_port", false)); +#endif + + socket.listen(/* backlog = */ config().getUInt("listen_backlog", 64)); + + return address; +} + +void Server::createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) +{ + /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file. + if (!config().has(port_name)) + return; + + auto port = config().getInt(port_name); + try + { + func(port); + } + catch (const Poco::Exception &) + { + std::string message = "Listen [" + listen_host + "]:" + std::to_string(port) + " failed: " + getCurrentExceptionMessage(false); + + if (listen_try) + { + LOG_WARNING(&logger(), "{}. If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to " + "specify not disabled IPv4 or IPv6 address to listen in element of configuration " + "file. Example for disabled IPv6: 0.0.0.0 ." + " Example for disabled IPv4: ::", + message); + } + else + { + throw Exception{message, ErrorCodes::NETWORK_ERROR}; + } + } +} + void Server::uninitialize() { logger().information("shutting down"); @@ -667,6 +746,47 @@ int Server::main(const std::vector & /*args*/) total_memory_tracker.setDescription("(total)"); total_memory_tracker.setMetric(CurrentMetrics::MemoryTracking); + Poco::Timespan keep_alive_timeout(config().getUInt("keep_alive_timeout", 10), 0); + + Poco::ThreadPool server_pool(3, config().getUInt("max_connections", 1024)); + Poco::Net::HTTPServerParams::Ptr http_params = new Poco::Net::HTTPServerParams; + http_params->setTimeout(settings.http_receive_timeout); + http_params->setKeepAliveTimeout(keep_alive_timeout); + + std::vector> servers; + + std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); + + bool listen_try = config().getBool("listen_try", false); + if (listen_hosts.empty()) + { + listen_hosts.emplace_back("::1"); + listen_hosts.emplace_back("127.0.0.1"); + listen_try = true; + } + + for (const auto & listen_host : listen_hosts) + { + /// TCP TestKeeper + createServer(listen_host, "test_keeper_tcp_port", listen_try, [&](UInt16 port) + { + Poco::Net::ServerSocket socket; + auto address = socketBindListen(socket, listen_host, port); + socket.setReceiveTimeout(settings.receive_timeout); + socket.setSendTimeout(settings.send_timeout); + servers.emplace_back(std::make_unique( + new TCPHandlerFactory(*this, false, true), + server_pool, + socket, + new Poco::Net::TCPServerParams)); + + LOG_INFO(log, "Listening for connections to fake zookeeper (tcp): {}", address.toString()); + }); + } + + for (auto & server : servers) + server->start(); + /// Set current database name before loading tables and databases because /// system logs may copy global context. global_context->setCurrentDatabaseNameInGlobalContext(default_database); @@ -797,74 +917,6 @@ int Server::main(const std::vector & /*args*/) #endif { - Poco::Timespan keep_alive_timeout(config().getUInt("keep_alive_timeout", 10), 0); - - Poco::ThreadPool server_pool(3, config().getUInt("max_connections", 1024)); - Poco::Net::HTTPServerParams::Ptr http_params = new Poco::Net::HTTPServerParams; - http_params->setTimeout(settings.http_receive_timeout); - http_params->setKeepAliveTimeout(keep_alive_timeout); - - std::vector> servers; - - std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); - - bool listen_try = config().getBool("listen_try", false); - if (listen_hosts.empty()) - { - listen_hosts.emplace_back("::1"); - listen_hosts.emplace_back("127.0.0.1"); - listen_try = true; - } - - auto make_socket_address = [&](const std::string & host, UInt16 port) - { - Poco::Net::SocketAddress socket_address; - try - { - socket_address = Poco::Net::SocketAddress(host, port); - } - catch (const Poco::Net::DNSException & e) - { - const auto code = e.code(); - if (code == EAI_FAMILY -#if defined(EAI_ADDRFAMILY) - || code == EAI_ADDRFAMILY -#endif - ) - { - LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " - "If it is an IPv6 address and your host has disabled IPv6, then consider to " - "specify IPv4 address to listen in element of configuration " - "file. Example: 0.0.0.0", - host, e.code(), e.message()); - } - - throw; - } - return socket_address; - }; - - auto socket_bind_listen = [&](auto & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false) - { - auto address = make_socket_address(host, port); -#if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100 - if (secure) - /// Bug in old (<1.9.1) poco, listen() after bind() with reusePort param will fail because have no implementation in SecureServerSocketImpl - /// https://github.com/pocoproject/poco/pull/2257 - socket.bind(address, /* reuseAddress = */ true); - else -#endif -#if POCO_VERSION < 0x01080000 - socket.bind(address, /* reuseAddress = */ true); -#else - socket.bind(address, /* reuseAddress = */ true, /* reusePort = */ config().getBool("listen_reuse_port", false)); -#endif - - socket.listen(/* backlog = */ config().getUInt("listen_backlog", 64)); - - return address; - }; - /// This object will periodically calculate some metrics. AsynchronousMetrics async_metrics(*global_context, config().getUInt("asynchronous_metrics_update_period_s", 60)); @@ -872,41 +924,11 @@ int Server::main(const std::vector & /*args*/) for (const auto & listen_host : listen_hosts) { - auto create_server = [&](const char * port_name, auto && func) - { - /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file. - if (!config().has(port_name)) - return; - - auto port = config().getInt(port_name); - try - { - func(port); - } - catch (const Poco::Exception &) - { - std::string message = "Listen [" + listen_host + "]:" + std::to_string(port) + " failed: " + getCurrentExceptionMessage(false); - - if (listen_try) - { - LOG_WARNING(log, "{}. If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to " - "specify not disabled IPv4 or IPv6 address to listen in element of configuration " - "file. Example for disabled IPv6: 0.0.0.0 ." - " Example for disabled IPv4: ::", - message); - } - else - { - throw Exception{message, ErrorCodes::NETWORK_ERROR}; - } - } - }; - /// HTTP - create_server("http_port", [&](UInt16 port) + createServer(listen_host, "http_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); @@ -917,11 +939,11 @@ int Server::main(const std::vector & /*args*/) }); /// HTTPS - create_server("https_port", [&](UInt16 port) + createServer(listen_host, "https_port", listen_try, [&](UInt16 port) { #if USE_SSL Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -936,10 +958,10 @@ int Server::main(const std::vector & /*args*/) }); /// TCP - create_server("tcp_port", [&](UInt16 port) + createServer(listen_host, "tcp_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -951,28 +973,12 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for connections with native protocol (tcp): {}", address.toString()); }); - /// TCP TestKeeper - create_server("test_keeper_tcp_port", [&](UInt16 port) - { - Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); - socket.setReceiveTimeout(settings.receive_timeout); - socket.setSendTimeout(settings.send_timeout); - servers.emplace_back(std::make_unique( - new TCPHandlerFactory(*this, false, true), - server_pool, - socket, - new Poco::Net::TCPServerParams)); - - LOG_INFO(log, "Listening for connections to fake zookeeper (tcp): {}", address.toString()); - }); - /// TCP with SSL - create_server("tcp_port_secure", [&](UInt16 port) + createServer(listen_host, "tcp_port_secure", listen_try, [&](UInt16 port) { #if USE_SSL Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -989,10 +995,10 @@ int Server::main(const std::vector & /*args*/) }); /// Interserver IO HTTP - create_server("interserver_http_port", [&](UInt16 port) + createServer(listen_host, "interserver_http_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -1001,11 +1007,11 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for replica communication (interserver): http://{}", address.toString()); }); - create_server("interserver_https_port", [&](UInt16 port) + createServer(listen_host, "interserver_https_port", listen_try, [&](UInt16 port) { #if USE_SSL Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -1019,10 +1025,10 @@ int Server::main(const std::vector & /*args*/) #endif }); - create_server("mysql_port", [&](UInt16 port) + createServer(listen_host, "mysql_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(Poco::Timespan()); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -1034,10 +1040,10 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for MySQL compatibility protocol: {}", address.toString()); }); - create_server("postgresql_port", [&](UInt16 port) + createServer(listen_host, "postgresql_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(Poco::Timespan()); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -1050,10 +1056,10 @@ int Server::main(const std::vector & /*args*/) }); /// Prometheus (if defined and not setup yet with http_port) - create_server("prometheus.port", [&](UInt16 port) + createServer(listen_host, "prometheus.port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -1069,8 +1075,8 @@ int Server::main(const std::vector & /*args*/) global_context->enableNamedSessions(); - for (auto & server : servers) - server->start(); + for (size_t i = 1; i < servers.size(); ++i) + servers[i]->start(); { String level_str = config().getString("text_log.level", ""); diff --git a/programs/server/Server.h b/programs/server/Server.h index ad9e51c881c..57fcea3cee8 100644 --- a/programs/server/Server.h +++ b/programs/server/Server.h @@ -14,6 +14,13 @@ * 3. Interserver HTTP - for replication. */ +namespace Poco +{ + namespace Net + { + class ServerSocket; + } +} namespace DB { @@ -57,6 +64,13 @@ protected: private: Context * global_context_ptr = nullptr; + +private: + + Poco::Net::SocketAddress socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false); + + using CreateServerFunc = std::function; + void createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func); }; } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index b60144f4a87..4e291679b78 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -175,7 +175,6 @@ struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest using TestKeeperStorageRequest::TestKeeperStorageRequest; std::pair process(TestKeeperStorage::Container & container, int64_t /* zxid */) const override { - //LOG_DEBUG(&Poco::Logger::get("STORAGE"), "EXECUTING GET REQUEST"); Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr); Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request); @@ -309,7 +308,7 @@ struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest response.error = Coordination::Error::ZBADVERSION; } - return { response_ptr, {} }; + return { response_ptr, undo }; } void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override @@ -327,7 +326,6 @@ struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperListResponse & response = dynamic_cast(*response_ptr); Coordination::ZooKeeperListRequest & request = dynamic_cast(*zk_request); - Undo undo; auto it = container.find(request.path); if (it == container.end()) { @@ -490,17 +488,14 @@ void TestKeeperStorage::processingThread() auto zk_request = info.request->zk_request; - info.request->zk_request->addRootPath(root_path); auto [response, _] = info.request->process(container, zxid); if (response->error == Coordination::Error::ZOK) { info.request->processWatches(watches, list_watches); } - ++zxid; response->xid = zk_request->xid; - response->zxid = zxid; - response->removeRootPath(root_path); + response->zxid = getZXID(); info.response_callback(response); } @@ -525,6 +520,36 @@ void TestKeeperStorage::finalize() } try { + { + auto finish_watch = [] (const auto & watch_pair) + { + Coordination::ZooKeeperWatchResponse response; + response.type = Coordination::SESSION; + response.state = Coordination::EXPIRED_SESSION; + response.error = Coordination::Error::ZSESSIONEXPIRED; + + for (auto & callback : watch_pair.second) + { + if (callback) + { + try + { + callback(std::make_shared(response)); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } + } + } + }; + for (auto & path_watch : watches) + finish_watch(path_watch); + watches.clear(); + for (auto & path_watch : list_watches) + finish_watch(path_watch); + list_watches.clear(); + } RequestInfo info; while (requests_queue.tryPop(info)) { @@ -552,7 +577,7 @@ class TestKeeperWrapperFactory final : private boost::noncopyable public: using Creator = std::function; - using OpNumToRequest = std::unordered_map; + using OpNumToRequest = std::unordered_map; static TestKeeperWrapperFactory & instance() { @@ -564,12 +589,12 @@ public: { auto it = op_num_to_request.find(zk_request->getOpNum()); if (it == op_num_to_request.end()) - throw Coordination::Exception("Unknown operation type " + std::to_string(zk_request->getOpNum()), Coordination::Error::ZBADARGUMENTS); + throw Coordination::Exception("Unknown operation type " + toString(zk_request->getOpNum()), Coordination::Error::ZBADARGUMENTS); return it->second(zk_request); } - void registerRequest(int32_t op_num, Creator creator) + void registerRequest(Coordination::OpNum op_num, Creator creator) { if (!op_num_to_request.try_emplace(op_num, creator).second) throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Request with op num {} already registered", op_num); @@ -582,7 +607,7 @@ private: TestKeeperWrapperFactory(); }; -template +template void registerTestKeeperRequestWrapper(TestKeeperWrapperFactory & factory) { factory.registerRequest(num, [] (const Coordination::ZooKeeperRequestPtr & zk_request) { return std::make_shared(zk_request); }); @@ -591,17 +616,17 @@ void registerTestKeeperRequestWrapper(TestKeeperWrapperFactory & factory) TestKeeperWrapperFactory::TestKeeperWrapperFactory() { - registerTestKeeperRequestWrapper<11, TestKeeperStorageHeartbeatRequest>(*this); - //registerTestKeeperRequestWrapper<100, TestKeeperStorageAuthRequest>(*this); - //registerTestKeeperRequestWrapper<-11, TestKeeperStorageCloseRequest>(*this); - registerTestKeeperRequestWrapper<1, TestKeeperStorageCreateRequest>(*this); - registerTestKeeperRequestWrapper<2, TestKeeperStorageRemoveRequest>(*this); - registerTestKeeperRequestWrapper<3, TestKeeperStorageExistsRequest>(*this); - registerTestKeeperRequestWrapper<4, TestKeeperStorageGetRequest>(*this); - registerTestKeeperRequestWrapper<5, TestKeeperStorageSetRequest>(*this); - registerTestKeeperRequestWrapper<12, TestKeeperStorageListRequest>(*this); - registerTestKeeperRequestWrapper<13, TestKeeperStorageCheckRequest>(*this); - registerTestKeeperRequestWrapper<14, TestKeeperStorageMultiRequest>(*this); + registerTestKeeperRequestWrapper(*this); + //registerTestKeeperRequestWrapper(*this); + //registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); } TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) @@ -624,7 +649,6 @@ TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); - //LOG_DEBUG(&Poco::Logger::get("STORAGE"), "PUSHED"); return ResponsePair{std::move(future), std::move(watch_future)}; } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index edcae28af85..875cb9f0253 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -39,8 +39,6 @@ public: Container container; - String root_path; - std::atomic zxid{0}; std::atomic shutdown{false}; @@ -56,6 +54,7 @@ public: ResponseCallback watch_callback; clock::time_point time; }; + std::mutex push_request_mutex; using RequestsQueue = ConcurrentBoundedQueue; RequestsQueue requests_queue{1}; diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index cb61f22fb3c..7b162492d1a 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -1,4 +1,5 @@ #include +#include #include #include #include @@ -12,165 +13,6 @@ namespace Coordination using namespace DB; -/// ZooKeeper has 1 MB node size and serialization limit by default, -/// but it can be raised up, so we have a slightly larger limit on our side. -#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB - -/// Assuming we are at little endian. - -static void write(int64_t x, WriteBuffer & out) -{ - x = __builtin_bswap64(x); - writeBinary(x, out); -} - -static void write(int32_t x, WriteBuffer & out) -{ - x = __builtin_bswap32(x); - writeBinary(x, out); -} - -static void write(bool x, WriteBuffer & out) -{ - writeBinary(x, out); -} - -static void write(const String & s, WriteBuffer & out) -{ - write(int32_t(s.size()), out); - out.write(s.data(), s.size()); -} - -template void write(std::array s, WriteBuffer & out) -{ - write(int32_t(N), out); - out.write(s.data(), N); -} - -template void write(const std::vector & arr, WriteBuffer & out) -{ - write(int32_t(arr.size()), out); - for (const auto & elem : arr) - write(elem, out); -} - -static void write(const ACL & acl, WriteBuffer & out) -{ - write(acl.permissions, out); - write(acl.scheme, out); - write(acl.id, out); -} - -static void write(const Stat & stat, WriteBuffer & out) -{ - write(stat.czxid, out); - write(stat.mzxid, out); - write(stat.ctime, out); - write(stat.mtime, out); - write(stat.version, out); - write(stat.cversion, out); - write(stat.aversion, out); - write(stat.ephemeralOwner, out); - write(stat.dataLength, out); - write(stat.numChildren, out); - write(stat.pzxid, out); -} - -static void write(const Error & x, WriteBuffer & out) -{ - write(static_cast(x), out); -} - -static void read(int64_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap64(x); -} - -static void read(int32_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap32(x); -} - -static void read(Error & x, ReadBuffer & in) -{ - int32_t code; - read(code, in); - x = Error(code); -} - -static void read(bool & x, ReadBuffer & in) -{ - readBinary(x, in); -} - -static void read(String & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - - if (size == -1) - { - /// It means that zookeeper node has NULL value. We will treat it like empty string. - s.clear(); - return; - } - - if (size < 0) - throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); - - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - - s.resize(size); - in.read(s.data(), size); -} - -template void read(std::array & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size != N) - throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - in.read(s.data(), N); -} - -static void read(Stat & stat, ReadBuffer & in) -{ - read(stat.czxid, in); - read(stat.mzxid, in); - read(stat.ctime, in); - read(stat.mtime, in); - read(stat.version, in); - read(stat.cversion, in); - read(stat.aversion, in); - read(stat.ephemeralOwner, in); - read(stat.dataLength, in); - read(stat.numChildren, in); - read(stat.pzxid, in); -} - -template void read(std::vector & arr, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size < 0) - throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - arr.resize(size); - for (auto & elem : arr) - read(elem, in); -} - -static void read(ACL & acl, ReadBuffer & in) -{ - read(acl.permissions, in); - read(acl.scheme, in); - read(acl.id, in); -} - void ZooKeeperResponse::write(WriteBuffer & out) const { /// Excessive copy to calculate length. @@ -263,7 +105,6 @@ void ZooKeeperCreateResponse::readImpl(ReadBuffer & in) void ZooKeeperCreateResponse::writeImpl(WriteBuffer & out) const { - LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITE IMPL ON: {}", path_created); Coordination::write(path_created, out); } @@ -448,7 +289,7 @@ void ZooKeeperMultiRequest::writeImpl(WriteBuffer & out) const zk_request.writeImpl(out); } - OpNum op_num = -1; + OpNum op_num = OpNum::Error; bool done = true; int32_t error = -1; @@ -471,7 +312,7 @@ void ZooKeeperMultiRequest::readImpl(ReadBuffer & in) if (done) { - if (op_num != -1) + if (op_num != OpNum::Error) throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); if (error != -1) throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); @@ -505,7 +346,7 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) /// op_num == -1 is special for multi transaction. /// For unknown reason, error code is duplicated in header and in response body. - if (op_num == -1) + if (op_num == OpNum::Error) response = std::make_shared(); if (op_error != Error::ZOK) @@ -519,7 +360,7 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) error = op_error; } - if (op_error == Error::ZOK || op_num == -1) + if (op_error == Error::ZOK || op_num == OpNum::Error) dynamic_cast(*response).readImpl(in); } @@ -535,7 +376,7 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) if (!done) throw Exception("Too many results received for multi transaction", Error::ZMARSHALLINGERROR); - if (op_num != -1) + if (op_num != OpNum::Error) throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); if (error_read != -1) throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); @@ -560,7 +401,7 @@ void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const /// Footer. { - OpNum op_num = -1; + OpNum op_num = OpNum::Error; bool done = true; int32_t error_read = - 1; @@ -606,7 +447,7 @@ ZooKeeperRequestPtr ZooKeeperRequestFactory::get(OpNum op_num) const { auto it = op_num_to_request.find(op_num); if (it == op_num_to_request.end()) - throw Exception("Unknown operation type " + std::to_string(op_num), Error::ZBADARGUMENTS); + throw Exception("Unknown operation type " + toString(op_num), Error::ZBADARGUMENTS); return it->second(); } @@ -625,17 +466,17 @@ void registerZooKeeperRequest(ZooKeeperRequestFactory & factory) ZooKeeperRequestFactory::ZooKeeperRequestFactory() { - registerZooKeeperRequest<11, ZooKeeperHeartbeatRequest>(*this); - registerZooKeeperRequest<100, ZooKeeperAuthRequest>(*this); - registerZooKeeperRequest<-11, ZooKeeperCloseRequest>(*this); - registerZooKeeperRequest<1, ZooKeeperCreateRequest>(*this); - registerZooKeeperRequest<2, ZooKeeperRemoveRequest>(*this); - registerZooKeeperRequest<3, ZooKeeperExistsRequest>(*this); - registerZooKeeperRequest<4, ZooKeeperGetRequest>(*this); - registerZooKeeperRequest<5, ZooKeeperSetRequest>(*this); - registerZooKeeperRequest<12, ZooKeeperListRequest>(*this); - registerZooKeeperRequest<13, ZooKeeperCheckRequest>(*this); - registerZooKeeperRequest<14, ZooKeeperMultiRequest>(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); } } diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index f40de116da2..286330bd769 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -1,6 +1,7 @@ #pragma once #include +#include #include #include @@ -22,9 +23,6 @@ namespace Coordination { -using XID = int32_t; -using OpNum = int32_t; - struct ZooKeeperResponse : virtual Response { XID xid = 0; @@ -70,7 +68,7 @@ using ZooKeeperRequestPtr = std::shared_ptr; struct ZooKeeperHeartbeatRequest final : ZooKeeperRequest { String getPath() const override { return {}; } - OpNum getOpNum() const override { return 11; } + OpNum getOpNum() const override { return OpNum::Heartbeat; } void writeImpl(WriteBuffer &) const override {} void readImpl(ReadBuffer &) override {} ZooKeeperResponsePtr makeResponse() const override; @@ -80,7 +78,7 @@ struct ZooKeeperHeartbeatResponse final : ZooKeeperResponse { void readImpl(ReadBuffer &) override {} void writeImpl(WriteBuffer &) const override {} - OpNum getOpNum() const override { return 11; } + OpNum getOpNum() const override { return OpNum::Heartbeat; } }; struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse @@ -89,8 +87,10 @@ struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse void writeImpl(WriteBuffer & out) const override; - /// TODO FIXME alesap - OpNum getOpNum() const override { return 0; } + OpNum getOpNum() const override + { + throw Exception("OpNum for watch response doesn't exist", Error::ZRUNTIMEINCONSISTENCY); + } }; struct ZooKeeperAuthRequest final : ZooKeeperRequest @@ -100,7 +100,7 @@ struct ZooKeeperAuthRequest final : ZooKeeperRequest String data; String getPath() const override { return {}; } - OpNum getOpNum() const override { return 100; } + OpNum getOpNum() const override { return OpNum::Auth; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -112,13 +112,13 @@ struct ZooKeeperAuthResponse final : ZooKeeperResponse void readImpl(ReadBuffer &) override {} void writeImpl(WriteBuffer &) const override {} - OpNum getOpNum() const override { return 100; } + OpNum getOpNum() const override { return OpNum::Auth; } }; struct ZooKeeperCloseRequest final : ZooKeeperRequest { String getPath() const override { return {}; } - OpNum getOpNum() const override { return -11; } + OpNum getOpNum() const override { return OpNum::Close; } void writeImpl(WriteBuffer &) const override {} void readImpl(ReadBuffer &) override {} @@ -134,7 +134,7 @@ struct ZooKeeperCloseResponse final : ZooKeeperResponse void writeImpl(WriteBuffer &) const override {} - OpNum getOpNum() const override { return -11; } + OpNum getOpNum() const override { return OpNum::Close; } }; struct ZooKeeperCreateRequest final : public CreateRequest, ZooKeeperRequest @@ -142,7 +142,7 @@ struct ZooKeeperCreateRequest final : public CreateRequest, ZooKeeperRequest ZooKeeperCreateRequest() = default; explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} - OpNum getOpNum() const override { return 1; } + OpNum getOpNum() const override { return OpNum::Create; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -155,7 +155,7 @@ struct ZooKeeperCreateResponse final : CreateResponse, ZooKeeperResponse void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return 1; } + OpNum getOpNum() const override { return OpNum::Create; } }; struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest @@ -163,7 +163,7 @@ struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest ZooKeeperRemoveRequest() = default; explicit ZooKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} - OpNum getOpNum() const override { return 2; } + OpNum getOpNum() const override { return OpNum::Remove; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -174,12 +174,12 @@ struct ZooKeeperRemoveResponse final : RemoveResponse, ZooKeeperResponse { void readImpl(ReadBuffer &) override {} void writeImpl(WriteBuffer &) const override {} - OpNum getOpNum() const override { return 2; } + OpNum getOpNum() const override { return OpNum::Remove; } }; struct ZooKeeperExistsRequest final : ExistsRequest, ZooKeeperRequest { - OpNum getOpNum() const override { return 3; } + OpNum getOpNum() const override { return OpNum::Exists; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -190,12 +190,12 @@ struct ZooKeeperExistsResponse final : ExistsResponse, ZooKeeperResponse { void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return 3; } + OpNum getOpNum() const override { return OpNum::Exists; } }; struct ZooKeeperGetRequest final : GetRequest, ZooKeeperRequest { - OpNum getOpNum() const override { return 4; } + OpNum getOpNum() const override { return OpNum::Get; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -206,7 +206,7 @@ struct ZooKeeperGetResponse final : GetResponse, ZooKeeperResponse { void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return 4; } + OpNum getOpNum() const override { return OpNum::Get; } }; struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest @@ -214,7 +214,7 @@ struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest ZooKeeperSetRequest() = default; explicit ZooKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} - OpNum getOpNum() const override { return 5; } + OpNum getOpNum() const override { return OpNum::Set; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; ZooKeeperResponsePtr makeResponse() const override; @@ -224,12 +224,12 @@ struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse { void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return 5; } + OpNum getOpNum() const override { return OpNum::Set; } }; struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest { - OpNum getOpNum() const override { return 12; } + OpNum getOpNum() const override { return OpNum::List; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; ZooKeeperResponsePtr makeResponse() const override; @@ -239,7 +239,7 @@ struct ZooKeeperListResponse final : ListResponse, ZooKeeperResponse { void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return 12; } + OpNum getOpNum() const override { return OpNum::List; } }; struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest @@ -247,7 +247,7 @@ struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest ZooKeeperCheckRequest() = default; explicit ZooKeeperCheckRequest(const CheckRequest & base) : CheckRequest(base) {} - OpNum getOpNum() const override { return 13; } + OpNum getOpNum() const override { return OpNum::Check; } void writeImpl(WriteBuffer & out) const override; void readImpl(ReadBuffer & in) override; @@ -258,7 +258,7 @@ struct ZooKeeperCheckResponse final : CheckResponse, ZooKeeperResponse { void readImpl(ReadBuffer &) override {} void writeImpl(WriteBuffer &) const override {} - OpNum getOpNum() const override { return 13; } + OpNum getOpNum() const override { return OpNum::Check; } }; /// This response may be received only as an element of responses in MultiResponse. @@ -267,12 +267,12 @@ struct ZooKeeperErrorResponse final : ErrorResponse, ZooKeeperResponse void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; - OpNum getOpNum() const override { return -1; } + OpNum getOpNum() const override { return OpNum::Error; } }; struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest { - OpNum getOpNum() const override { return 14; } + OpNum getOpNum() const override { return OpNum::Multi; } ZooKeeperMultiRequest() = default; ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls); @@ -285,7 +285,7 @@ struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest struct ZooKeeperMultiResponse final : MultiResponse, ZooKeeperResponse { - OpNum getOpNum() const override { return 14; } + OpNum getOpNum() const override { return OpNum::Multi; } explicit ZooKeeperMultiResponse(const Requests & requests) { diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.cpp b/src/Common/ZooKeeper/ZooKeeperConstants.cpp new file mode 100644 index 00000000000..9a1a43d43a7 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperConstants.cpp @@ -0,0 +1,64 @@ +#include +#include +#include + +namespace Coordination +{ + +static const std::unordered_set VALID_OPERATIONS = +{ + static_cast(OpNum::Close), + static_cast(OpNum::Error), + static_cast(OpNum::Create), + static_cast(OpNum::Remove), + static_cast(OpNum::Exists), + static_cast(OpNum::Get), + static_cast(OpNum::Set), + static_cast(OpNum::Heartbeat), + static_cast(OpNum::List), + static_cast(OpNum::Check), + static_cast(OpNum::Multi), + static_cast(OpNum::Auth), +}; + +std::string toString(OpNum op_num) +{ + switch (op_num) + { + case OpNum::Close: + return "Close"; + case OpNum::Error: + return "Error"; + case OpNum::Create: + return "Create"; + case OpNum::Remove: + return "Remove"; + case OpNum::Exists: + return "Exists"; + case OpNum::Get: + return "Get"; + case OpNum::Set: + return "Set"; + case OpNum::List: + return "List"; + case OpNum::Check: + return "Check"; + case OpNum::Multi: + return "Multi"; + case OpNum::Heartbeat: + return "Heartbeat"; + case OpNum::Auth: + return "Auth"; + } + int32_t raw_op = static_cast(op_num); + throw Exception("Operation " + std::to_string(raw_op) + " is unknown", Error::ZUNIMPLEMENTED); +} + +OpNum getOpNum(int32_t raw_op_num) +{ + if (!VALID_OPERATIONS.count(raw_op_num)) + throw Exception("Operation " + std::to_string(raw_op_num) + " is unknown", Error::ZUNIMPLEMENTED); + return static_cast(raw_op_num); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.h b/src/Common/ZooKeeper/ZooKeeperConstants.h new file mode 100644 index 00000000000..47b597f1c9d --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperConstants.h @@ -0,0 +1,47 @@ +#pragma once + +#include +#include + + +namespace Coordination +{ + +using XID = int32_t; + +static constexpr XID WATCH_XID = -1; +static constexpr XID PING_XID = -2; +static constexpr XID AUTH_XID = -4; +static constexpr XID CLOSE_XID = 0x7FFFFFFF; + +enum class OpNum : int32_t +{ + Close = -11, + Error = -1, + Create = 1, + Remove = 2, + Exists = 3, + Get = 4, + Set = 5, + Heartbeat = 11, + List = 12, + Check = 13, + Multi = 14, + Auth = 100, +}; + +std::string toString(OpNum op_num); +OpNum getOpNum(int32_t raw_op_num); + +static constexpr int32_t ZOOKEEPER_PROTOCOL_VERSION = 0; +static constexpr int32_t CLIENT_HANDSHAKE_LENGTH = 44; +static constexpr int32_t SERVER_HANDSHAKE_LENGTH = 36; +static constexpr int32_t PASSWORD_LENGTH = 16; + +/// ZooKeeper has 1 MB node size and serialization limit by default, +/// but it can be raised up, so we have a slightly larger limit on our side. +static constexpr int32_t MAX_STRING_OR_ARRAY_SIZE = 1 << 28; /// 256 MiB +static constexpr int32_t DEFAULT_SESSION_TIMEOUT = 30000; +static constexpr int32_t DEFAULT_OPERATION_TIMEOUT = 10000; + +} diff --git a/src/Common/ZooKeeper/ZooKeeperIO.cpp b/src/Common/ZooKeeper/ZooKeeperIO.cpp new file mode 100644 index 00000000000..157325cb8e6 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperIO.cpp @@ -0,0 +1,135 @@ +#include + +namespace Coordination +{ + +void write(int64_t x, WriteBuffer & out) +{ + x = __builtin_bswap64(x); + writeBinary(x, out); +} +void write(int32_t x, WriteBuffer & out) +{ + x = __builtin_bswap32(x); + writeBinary(x, out); +} + +void write(OpNum x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +void write(bool x, WriteBuffer & out) +{ + writeBinary(x, out); +} + +void write(const std::string & s, WriteBuffer & out) +{ + write(int32_t(s.size()), out); + out.write(s.data(), s.size()); +} + +void write(const ACL & acl, WriteBuffer & out) +{ + write(acl.permissions, out); + write(acl.scheme, out); + write(acl.id, out); +} + +void write(const Stat & stat, WriteBuffer & out) +{ + write(stat.czxid, out); + write(stat.mzxid, out); + write(stat.ctime, out); + write(stat.mtime, out); + write(stat.version, out); + write(stat.cversion, out); + write(stat.aversion, out); + write(stat.ephemeralOwner, out); + write(stat.dataLength, out); + write(stat.numChildren, out); + write(stat.pzxid, out); +} + +void write(const Error & x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +void read(int64_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap64(x); +} + +void read(int32_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap32(x); +} + +void read(OpNum & x, ReadBuffer & in) +{ + int32_t raw_op_num; + read(raw_op_num, in); + x = getOpNum(raw_op_num); +} + +void read(bool & x, ReadBuffer & in) +{ + readBinary(x, in); +} + +void read(std::string & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + + if (size == -1) + { + /// It means that zookeeper node has NULL value. We will treat it like empty string. + s.clear(); + return; + } + + if (size < 0) + throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); + + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + + s.resize(size); + in.read(s.data(), size); +} + +void read(ACL & acl, ReadBuffer & in) +{ + read(acl.permissions, in); + read(acl.scheme, in); + read(acl.id, in); +} + +void read(Stat & stat, ReadBuffer & in) +{ + read(stat.czxid, in); + read(stat.mzxid, in); + read(stat.ctime, in); + read(stat.mtime, in); + read(stat.version, in); + read(stat.cversion, in); + read(stat.aversion, in); + read(stat.ephemeralOwner, in); + read(stat.dataLength, in); + read(stat.numChildren, in); + read(stat.pzxid, in); +} + +void read(Error & x, ReadBuffer & in) +{ + int32_t code; + read(code, in); + x = Coordination::Error(code); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperIO.h b/src/Common/ZooKeeper/ZooKeeperIO.h new file mode 100644 index 00000000000..bd6d6a6f849 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperIO.h @@ -0,0 +1,73 @@ +#pragma once +#include +#include +#include +#include +#include +#include +#include +#include + +namespace Coordination +{ + +using namespace DB; + +void write(int64_t x, WriteBuffer & out); +void write(int32_t x, WriteBuffer & out); +void write(OpNum x, WriteBuffer & out); +void write(bool x, WriteBuffer & out); +void write(const std::string & s, WriteBuffer & out); +void write(const ACL & acl, WriteBuffer & out); +void write(const Stat & stat, WriteBuffer & out); +void write(const Error & x, WriteBuffer & out); + +template +void write(const std::array s, WriteBuffer & out) +{ + write(int32_t(N), out); + out.write(s.data(), N); +} + +template +void write(const std::vector & arr, WriteBuffer & out) +{ + write(int32_t(arr.size()), out); + for (const auto & elem : arr) + write(elem, out); +} + +void read(int64_t & x, ReadBuffer & in); +void read(int32_t & x, ReadBuffer & in); +void read(OpNum & x, ReadBuffer & in); +void read(bool & x, ReadBuffer & in); +void read(std::string & s, ReadBuffer & in); +void read(ACL & acl, ReadBuffer & in); +void read(Stat & stat, ReadBuffer & in); +void read(Error & x, ReadBuffer & in); + +template +void read(std::array & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size != N) + throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + in.read(s.data(), N); +} + +template +void read(std::vector & arr, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size < 0) + throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + arr.resize(size); + for (auto & elem : arr) + read(elem, in); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 23a5c6c4301..c9ef2a26edd 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -2,7 +2,7 @@ #include #include #include - +#include #include #include #include @@ -20,11 +20,6 @@ #include -/// ZooKeeper has 1 MB node size and serialization limit by default, -/// but it can be raised up, so we have a slightly larger limit on our side. -#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB - - namespace ProfileEvents { extern const Event ZooKeeperInit; @@ -267,75 +262,6 @@ namespace Coordination using namespace DB; - -/// Assuming we are at little endian. - -static void write(int64_t x, WriteBuffer & out) -{ - x = __builtin_bswap64(x); - writeBinary(x, out); -} - -static void write(int32_t x, WriteBuffer & out) -{ - x = __builtin_bswap32(x); - writeBinary(x, out); -} - -template void write(std::array s, WriteBuffer & out) -{ - write(int32_t(N), out); - out.write(s.data(), N); -} - -template void write(const std::vector & arr, WriteBuffer & out) -{ - write(int32_t(arr.size()), out); - for (const auto & elem : arr) - write(elem, out); -} - -static void read(int64_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap64(x); -} - -static void read(int32_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap32(x); -} - -static void read(Error & x, ReadBuffer & in) -{ - int32_t code; - read(code, in); - x = Error(code); -} - -template void read(std::array & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size != N) - throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - in.read(s.data(), N); -} - -template void read(std::vector & arr, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size < 0) - throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - arr.resize(size); - for (auto & elem : arr) - read(elem, in); -} - template void ZooKeeper::write(const T & x) { @@ -359,15 +285,6 @@ static void removeRootPath(String & path, const String & root_path) path = path.substr(root_path.size()); } -static constexpr int32_t protocol_version = 0; - -static constexpr ZooKeeper::XID watch_xid = -1; -static constexpr ZooKeeper::XID ping_xid = -2; -static constexpr ZooKeeper::XID auth_xid = -4; - -static constexpr ZooKeeper::XID close_xid = 0x7FFFFFFF; - - ZooKeeper::~ZooKeeper() { try @@ -542,7 +459,7 @@ void ZooKeeper::sendHandshake() std::array passwd {}; write(handshake_length); - write(protocol_version); + write(ZOOKEEPER_PROTOCOL_VERSION); write(last_zxid_seen); write(timeout); write(previous_session_id); @@ -557,16 +474,15 @@ void ZooKeeper::receiveHandshake() int32_t handshake_length; int32_t protocol_version_read; int32_t timeout; - constexpr int32_t passwd_len = 16; - std::array passwd; + std::array passwd; read(handshake_length); - if (handshake_length != 36) - throw Exception("Unexpected handshake length received: " + toString(handshake_length), Error::ZMARSHALLINGERROR); + if (handshake_length != SERVER_HANDSHAKE_LENGTH) + throw Exception("Unexpected handshake length received: " + DB::toString(handshake_length), Error::ZMARSHALLINGERROR); read(protocol_version_read); - if (protocol_version_read != protocol_version) - throw Exception("Unexpected protocol version: " + toString(protocol_version_read), Error::ZMARSHALLINGERROR); + if (protocol_version_read != ZOOKEEPER_PROTOCOL_VERSION) + throw Exception("Unexpected protocol version: " + DB::toString(protocol_version_read), Error::ZMARSHALLINGERROR); read(timeout); if (timeout != session_timeout.totalMilliseconds()) @@ -583,7 +499,7 @@ void ZooKeeper::sendAuth(const String & scheme, const String & data) ZooKeeperAuthRequest request; request.scheme = scheme; request.data = data; - request.xid = auth_xid; + request.xid = AUTH_XID; request.write(*out); int32_t length; @@ -597,17 +513,17 @@ void ZooKeeper::sendAuth(const String & scheme, const String & data) read(zxid); read(err); - if (read_xid != auth_xid) - throw Exception("Unexpected event received in reply to auth request: " + toString(read_xid), + if (read_xid != AUTH_XID) + throw Exception("Unexpected event received in reply to auth request: " + DB::toString(read_xid), Error::ZMARSHALLINGERROR); int32_t actual_length = in->count() - count_before_event; if (length != actual_length) - throw Exception("Response length doesn't match. Expected: " + toString(length) + ", actual: " + toString(actual_length), + throw Exception("Response length doesn't match. Expected: " + DB::toString(length) + ", actual: " + DB::toString(actual_length), Error::ZMARSHALLINGERROR); if (err != Error::ZOK) - throw Exception("Error received in reply to auth request. Code: " + toString(int32_t(err)) + ". Message: " + String(errorMessage(err)), + throw Exception("Error received in reply to auth request. Code: " + DB::toString(int32_t(err)) + ". Message: " + String(errorMessage(err)), Error::ZMARSHALLINGERROR); } @@ -640,7 +556,7 @@ void ZooKeeper::sendThread() /// After we popped element from the queue, we must register callbacks (even in the case when expired == true right now), /// because they must not be lost (callbacks must be called because the user will wait for them). - if (info.request->xid != close_xid) + if (info.request->xid != CLOSE_XID) { CurrentMetrics::add(CurrentMetrics::ZooKeeperRequest); std::lock_guard lock(operations_mutex); @@ -650,7 +566,6 @@ void ZooKeeper::sendThread() if (info.watch) { info.request->has_watch = true; - //std::cerr << "REQUEST" << info.request->getOpNum() << " HAS WATCH" << std::endl; CurrentMetrics::add(CurrentMetrics::ZooKeeperWatch); } @@ -662,11 +577,10 @@ void ZooKeeper::sendThread() info.request->addRootPath(root_path); info.request->probably_sent = true; - //std::cerr << "SENDING GENERAL REQUEST:" << info.request->getOpNum() << std::endl; info.request->write(*out); /// We sent close request, exit - if (info.request->xid == close_xid) + if (info.request->xid == CLOSE_XID) break; } } @@ -676,7 +590,7 @@ void ZooKeeper::sendThread() prev_heartbeat_time = clock::now(); ZooKeeperHeartbeatRequest request; - request.xid = ping_xid; + request.xid = PING_XID; request.write(*out); } @@ -731,7 +645,7 @@ void ZooKeeper::receiveThread() { if (earliest_operation) { - throw Exception("Operation timeout (no response) for request " + std::to_string(earliest_operation->request->getOpNum()) + " for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); + throw Exception("Operation timeout (no response) for request " + toString(earliest_operation->request->getOpNum()) + " for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); } waited += max_wait; if (waited >= session_timeout.totalMicroseconds()) @@ -766,14 +680,14 @@ void ZooKeeper::receiveEvent() RequestInfo request_info; ZooKeeperResponsePtr response; - if (xid == ping_xid) + if (xid == PING_XID) { if (err != Error::ZOK) throw Exception("Received error in heartbeat response: " + String(errorMessage(err)), Error::ZRUNTIMEINCONSISTENCY); response = std::make_shared(); } - else if (xid == watch_xid) + else if (xid == WATCH_XID) { ProfileEvents::increment(ProfileEvents::ZooKeeperWatchResponse); response = std::make_shared(); @@ -814,7 +728,7 @@ void ZooKeeper::receiveEvent() auto it = operations.find(xid); if (it == operations.end()) - throw Exception("Received response for unknown xid " + toString(xid), Error::ZRUNTIMEINCONSISTENCY); + throw Exception("Received response for unknown xid " + DB::toString(xid), Error::ZRUNTIMEINCONSISTENCY); /// After this point, we must invoke callback, that we've grabbed from 'operations'. /// Invariant: all callbacks are invoked either in case of success or in case of error. @@ -836,12 +750,10 @@ void ZooKeeper::receiveEvent() if (err != Error::ZOK) { - //std::cerr << "GOT ERROR:" << static_cast(err) << std::endl; response->error = err; } else { - //std::cerr << "NO ERROR RECEIVED\n"; response->readImpl(*in); response->removeRootPath(root_path); } @@ -854,7 +766,7 @@ void ZooKeeper::receiveEvent() /// 3 indicates the ZooKeeperExistsRequest. // For exists, we set the watch on both node exist and nonexist case. // For other case like getData, we only set the watch when node exists. - if (request_info.request->getOpNum() == 3) + if (request_info.request->getOpNum() == OpNum::Exists) add_watch = (response->error == Error::ZOK || response->error == Error::ZNONODE); else add_watch = response->error == Error::ZOK; @@ -871,7 +783,7 @@ void ZooKeeper::receiveEvent() int32_t actual_length = in->count() - count_before_event; if (length != actual_length) - throw Exception("Response length doesn't match. Expected: " + toString(length) + ", actual: " + toString(actual_length), Error::ZMARSHALLINGERROR); + throw Exception("Response length doesn't match. Expected: " + DB::toString(length) + ", actual: " + DB::toString(actual_length), Error::ZMARSHALLINGERROR); } catch (...) { @@ -1058,7 +970,7 @@ void ZooKeeper::pushRequest(RequestInfo && info) if (!info.request->xid) { info.request->xid = next_xid.fetch_add(1); - if (info.request->xid == close_xid) + if (info.request->xid == CLOSE_XID) throw Exception("xid equal to close_xid", Error::ZSESSIONEXPIRED); if (info.request->xid < 0) throw Exception("XID overflow", Error::ZSESSIONEXPIRED); @@ -1238,7 +1150,7 @@ void ZooKeeper::multi( void ZooKeeper::close() { ZooKeeperCloseRequest request; - request.xid = close_xid; + request.xid = CLOSE_XID; RequestInfo request_info; request_info.request = std::make_shared(std::move(request)); diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.h b/src/Common/ZooKeeper/ZooKeeperImpl.h index c96d7d2f0cb..65f8fe91793 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.h +++ b/src/Common/ZooKeeper/ZooKeeperImpl.h @@ -99,9 +99,6 @@ public: using Nodes = std::vector; - using XID = int32_t; - using OpNum = int32_t; - /** Connection to nodes is performed in order. If you want, shuffle them manually. * Operation timeout couldn't be greater than session timeout. * Operation timeout applies independently for network read, network write, waiting for events and synchronization. diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 1e7f69dc14f..7afd1af1aa0 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -1,9 +1,6 @@ #include +#include #include -#include -#include -#include -#include #include #include #include @@ -23,187 +20,15 @@ namespace ErrorCodes } -#ifdef __clang__ -#pragma clang diagnostic push -#pragma clang diagnostic ignored "-Wunused-function" -#endif - -/// ZooKeeper has 1 MB node size and serialization limit by default, -/// but it can be raised up, so we have a slightly larger limit on our side. -#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB - -/// Assuming we are at little endian. - -static void write(int64_t x, WriteBuffer & out) -{ - x = __builtin_bswap64(x); - writeBinary(x, out); -} - -static void write(int32_t x, WriteBuffer & out) -{ - x = __builtin_bswap32(x); - writeBinary(x, out); -} - -static void write(bool x, WriteBuffer & out) -{ - writeBinary(x, out); -} - -static void write(const String & s, WriteBuffer & out) -{ - write(int32_t(s.size()), out); - out.write(s.data(), s.size()); -} - -template void write(std::array s, WriteBuffer & out) -{ - write(int32_t(N), out); - out.write(s.data(), N); -} - -template void write(const std::vector & arr, WriteBuffer & out) -{ - write(int32_t(arr.size()), out); - for (const auto & elem : arr) - write(elem, out); -} - -static void write(const Coordination::ACL & acl, WriteBuffer & out) -{ - write(acl.permissions, out); - write(acl.scheme, out); - write(acl.id, out); -} - -static void write(const Coordination::Stat & stat, WriteBuffer & out) -{ - write(stat.czxid, out); - write(stat.mzxid, out); - write(stat.ctime, out); - write(stat.mtime, out); - write(stat.version, out); - write(stat.cversion, out); - write(stat.aversion, out); - write(stat.ephemeralOwner, out); - write(stat.dataLength, out); - write(stat.numChildren, out); - write(stat.pzxid, out); -} - -static void write(const Coordination::Error & x, WriteBuffer & out) -{ - write(static_cast(x), out); -} - -static void read(int64_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap64(x); -} - -static void read(int32_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap32(x); -} - -static void read(Coordination::Error & x, ReadBuffer & in) -{ - int32_t code; - read(code, in); - x = Coordination::Error(code); -} - -static void read(bool & x, ReadBuffer & in) -{ - readBinary(x, in); -} - -static void read(String & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - - if (size == -1) - { - /// It means that zookeeper node has NULL value. We will treat it like empty string. - s.clear(); - return; - } - - if (size < 0) - throw Exception("Negative size while reading string from ZooKeeper", ErrorCodes::LOGICAL_ERROR); - - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large string size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); - - s.resize(size); - in.read(s.data(), size); -} - -template void read(std::array & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size != N) - throw Exception("Unexpected array size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); - in.read(s.data(), N); -} - -static void read(Coordination::Stat & stat, ReadBuffer & in) -{ - read(stat.czxid, in); - read(stat.mzxid, in); - read(stat.ctime, in); - read(stat.mtime, in); - read(stat.version, in); - read(stat.cversion, in); - read(stat.aversion, in); - read(stat.ephemeralOwner, in); - read(stat.dataLength, in); - read(stat.numChildren, in); - read(stat.pzxid, in); -} - -template void read(std::vector & arr, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size < 0) - throw Exception("Negative size while reading array from ZooKeeper", ErrorCodes::LOGICAL_ERROR); - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large array size while reading from ZooKeeper", ErrorCodes::LOGICAL_ERROR); - arr.resize(size); - for (auto & elem : arr) - read(elem, in); -} - -static void read(Coordination::ACL & acl, ReadBuffer & in) -{ - read(acl.permissions, in); - read(acl.scheme, in); - read(acl.id, in); -} - -#ifdef __clang__ -#pragma clang diagnostic pop -#endif - void TestKeeperTCPHandler::sendHandshake() { - static constexpr int32_t handshake_length = 36; - static constexpr int32_t protocol_version = 0; - static constexpr int32_t DEFAULT_SESSION_TIMEOUT = 30000; - write(handshake_length, *out); - write(protocol_version, *out); - write(DEFAULT_SESSION_TIMEOUT, *out); - write(test_keeper_storage->getSessionID(), *out); - constexpr int32_t passwd_len = 16; - std::array passwd{}; - write(passwd, *out); + Coordination::write(Coordination::SERVER_HANDSHAKE_LENGTH, *out); + Coordination::write(Coordination::ZOOKEEPER_PROTOCOL_VERSION, *out); + Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT, *out); + Coordination::write(test_keeper_storage->getSessionID(), *out); + std::array passwd{}; + Coordination::write(passwd, *out); out->next(); } @@ -219,30 +44,29 @@ void TestKeeperTCPHandler::receiveHandshake() int64_t last_zxid_seen; int32_t timeout; int64_t previous_session_id = 0; /// We don't support session restore. So previous session_id is always zero. - constexpr int32_t passwd_len = 16; - std::array passwd {}; + std::array passwd {}; - read(handshake_length, *in); - if (handshake_length != 44) + Coordination::read(handshake_length, *in); + if (handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH) throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::LOGICAL_ERROR); - read(protocol_version, *in); + Coordination::read(protocol_version, *in); - if (protocol_version != 0) + if (protocol_version != Coordination::ZOOKEEPER_PROTOCOL_VERSION) throw Exception("Unexpected protocol version: " + toString(protocol_version), ErrorCodes::LOGICAL_ERROR); - read(last_zxid_seen, *in); + Coordination::read(last_zxid_seen, *in); if (last_zxid_seen != 0) throw Exception("Non zero last_zxid_seen is not supported", ErrorCodes::LOGICAL_ERROR); - read(timeout, *in); - read(previous_session_id, *in); + Coordination::read(timeout, *in); + Coordination::read(previous_session_id, *in); if (previous_session_id != 0) throw Exception("Non zero previous session id is not supported", ErrorCodes::LOGICAL_ERROR); - read(passwd, *in); + Coordination::read(passwd, *in); } @@ -272,7 +96,7 @@ void TestKeeperTCPHandler::runImpl() } catch (const Exception & e) /// Typical for an incorrect username, password, or address. { - LOG_DEBUG(log, "Cannot receive handshake {}", e.displayText()); + LOG_WARNING(log, "Cannot receive handshake {}", e.displayText()); return; } @@ -280,18 +104,17 @@ void TestKeeperTCPHandler::runImpl() while (true) { - //UInt64 max_wait = operation_timeout.totalMicroseconds(); using namespace std::chrono_literals; - //LOG_DEBUG(log, "TRYING TO GET RESPONSE (size {})", responses.size()); - if (!responses.empty() && responses.front().wait_for(100ms) == std::future_status::ready) + while(!responses.empty()) { - auto response = responses.front().get(); + if (responses.front().wait_for(10ms) != std::future_status::ready) + break; - //LOG_DEBUG(log, "Writing response bytes to socket {}", response->getOpNum()); + auto response = responses.front().get(); response->write(*out); responses.pop(); - //LOG_DEBUG(log, "Responses size {}", responses.size()); } + for (auto it = watch_responses.begin(); it != watch_responses.end();) { if (it->wait_for(0s) == std::future_status::ready) @@ -305,8 +128,9 @@ void TestKeeperTCPHandler::runImpl() } } - //LOG_DEBUG(log, "WAITING ON POLL"); - if (in->poll(100 * 1000)) + long poll_wait = responses.empty() ? session_timeout.totalMicroseconds() : 10000; + + if (in->poll(poll_wait)) { bool close_received = receiveRequest(); if (close_received) @@ -315,29 +139,20 @@ void TestKeeperTCPHandler::runImpl() break; } } - else - { - //LOG_DEBUG(log, "NOTHING POLLED"); - } } } bool TestKeeperTCPHandler::receiveRequest() { - //LOG_DEBUG(log, "Receiving event"); int32_t length; - read(length, *in); - //LOG_DEBUG(log, "RECEIVED LENGTH {}", length); + Coordination::read(length, *in); int32_t xid; - //LOG_DEBUG(log, "READING XID"); - read(xid, *in); + Coordination::read(xid, *in); - //LOG_DEBUG(log, "Received xid {}", xid); - - int32_t opnum; - read(opnum, *in); - if (opnum == -11) + Coordination::OpNum opnum; + Coordination::read(opnum, *in); + if (opnum == Coordination::OpNum::Close) return true; Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); @@ -348,7 +163,6 @@ bool TestKeeperTCPHandler::receiveRequest() if (request_future_responses.watch_response) watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); - //LOG_DEBUG(log, "Responses size {}", responses.size()); return false; } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 7eea5419006..06b3c102df5 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -4,6 +4,7 @@ #include "IServer.h" #include #include +#include #include #include #include @@ -21,7 +22,8 @@ public: , log(&Poco::Logger::get("TestKeeperTCPHandler")) , global_context(server.context()) , test_keeper_storage(global_context.getTestKeeperStorage()) - , operation_timeout(10000) + , operation_timeout(Coordination::DEFAULT_OPERATION_TIMEOUT) + , session_timeout(Coordination::DEFAULT_SESSION_TIMEOUT) { } @@ -32,6 +34,7 @@ private: Context global_context; std::shared_ptr test_keeper_storage; Poco::Timespan operation_timeout; + Poco::Timespan session_timeout; std::queue responses; std::vector watch_responses; From d83c68fca806af5c40060c857ba47951905b0330 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 16:55:28 +0300 Subject: [PATCH 025/174] Fix timeouts --- programs/server/Server.cpp | 20 ++++++++++---------- src/Common/ZooKeeper/TestKeeperStorage.cpp | 12 ++++++++++++ src/Common/ZooKeeper/TestKeeperStorage.h | 4 ++-- src/Common/ZooKeeper/ZooKeeper.cpp | 4 ++-- src/Common/ZooKeeper/ZooKeeper.h | 8 +++----- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 6 ++---- src/Common/ZooKeeper/ZooKeeperConstants.h | 4 ++-- src/Common/ZooKeeper/ZooKeeperIO.cpp | 2 +- src/Common/ya.make | 4 ++++ src/Server/TestKeeperTCPHandler.cpp | 4 ++-- src/Server/TestKeeperTCPHandler.h | 4 ++-- src/Server/ya.make | 1 + utils/zookeeper-test/main.cpp | 7 ++++--- 13 files changed, 47 insertions(+), 33 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 597b6c43fe5..064fa391813 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -195,17 +195,17 @@ Poco::Net::SocketAddress makeSocketAddress(const std::string & host, UInt16 port || code == EAI_ADDRFAMILY #endif ) - { - LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " - "If it is an IPv6 address and your host has disabled IPv6, then consider to " - "specify IPv4 address to listen in element of configuration " - "file. Example: 0.0.0.0", - host, e.code(), e.message()); - } + { + LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " + "If it is an IPv6 address and your host has disabled IPv6, then consider to " + "specify IPv4 address to listen in element of configuration " + "file. Example: 0.0.0.0", + host, e.code(), e.message()); + } - throw; - } - return socket_address; + throw; + } + return socket_address; } Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 4e291679b78..19d5fe6ad22 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -6,10 +6,22 @@ #include #include +namespace DB +{ + namespace ErrorCodes + { + extern const int LOGICAL_ERROR; + } +} + namespace zkutil { + + using namespace DB; + + static String parentPath(const String & path) { auto rslash_pos = path.rfind('/'); diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index 875cb9f0253..02fdc21cc58 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -19,7 +19,7 @@ class TestKeeperStorage public: - Poco::Timespan operation_timeout{10000}; + Poco::Timespan operation_timeout{0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000}; std::atomic session_id_counter{0}; struct Node @@ -85,5 +85,5 @@ public: return zxid.fetch_add(1); } }; - + } diff --git a/src/Common/ZooKeeper/ZooKeeper.cpp b/src/Common/ZooKeeper/ZooKeeper.cpp index bee875d1c74..b3a0a082a9f 100644 --- a/src/Common/ZooKeeper/ZooKeeper.cpp +++ b/src/Common/ZooKeeper/ZooKeeper.cpp @@ -129,8 +129,8 @@ struct ZooKeeperArgs std::vector hosts_strings; - session_timeout_ms = DEFAULT_SESSION_TIMEOUT; - operation_timeout_ms = DEFAULT_OPERATION_TIMEOUT; + session_timeout_ms = Coordination::DEFAULT_SESSION_TIMEOUT_MS; + operation_timeout_ms = Coordination::DEFAULT_OPERATION_TIMEOUT_MS; implementation = "zookeeper"; for (const auto & key : keys) { diff --git a/src/Common/ZooKeeper/ZooKeeper.h b/src/Common/ZooKeeper/ZooKeeper.h index b1a69646db5..0d9dc104c48 100644 --- a/src/Common/ZooKeeper/ZooKeeper.h +++ b/src/Common/ZooKeeper/ZooKeeper.h @@ -11,6 +11,7 @@ #include #include #include +#include #include @@ -28,9 +29,6 @@ namespace CurrentMetrics namespace zkutil { -const UInt32 DEFAULT_SESSION_TIMEOUT = 30000; -const UInt32 DEFAULT_OPERATION_TIMEOUT = 10000; - /// Preferred size of multi() command (in number of ops) constexpr size_t MULTI_BATCH_SIZE = 100; @@ -53,8 +51,8 @@ public: using Ptr = std::shared_ptr; ZooKeeper(const std::string & hosts_, const std::string & identity_ = "", - int32_t session_timeout_ms_ = DEFAULT_SESSION_TIMEOUT, - int32_t operation_timeout_ms_ = DEFAULT_OPERATION_TIMEOUT, + int32_t session_timeout_ms_ = Coordination::DEFAULT_SESSION_TIMEOUT_MS, + int32_t operation_timeout_ms_ = Coordination::DEFAULT_OPERATION_TIMEOUT_MS, const std::string & chroot_ = "", const std::string & implementation_ = "zookeeper"); diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index 7b162492d1a..d1c6b1ee80b 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -17,13 +17,11 @@ void ZooKeeperResponse::write(WriteBuffer & out) const { /// Excessive copy to calculate length. WriteBufferFromOwnString buf; - //LOG_DEBUG(&Poco::Logger::get("LOG"), "WRITING {}", xid); Coordination::write(xid, buf); Coordination::write(zxid, buf); Coordination::write(error, buf); if (error == Error::ZOK) writeImpl(buf); - //LOG_DEBUG(&Poco::Logger::get("LOG"), "BUFFER LENGTH {}", buf.str().length()); Coordination::write(buf.str(), out); out.next(); } @@ -426,7 +424,7 @@ ZooKeeperResponsePtr ZooKeeperCloseRequest::makeResponse() const { return std::m void ZooKeeperRequestFactory::registerRequest(OpNum op_num, Creator creator) { if (!op_num_to_request.try_emplace(op_num, creator).second) - throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Request with op num {} already registered", op_num); + throw Coordination::Exception("Request type " + toString(op_num) + " already registered", Coordination::Error::ZRUNTIMEINCONSISTENCY); } std::shared_ptr ZooKeeperRequest::read(ReadBuffer & in) @@ -478,5 +476,5 @@ ZooKeeperRequestFactory::ZooKeeperRequestFactory() registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); } - + } diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.h b/src/Common/ZooKeeper/ZooKeeperConstants.h index 47b597f1c9d..292bccd86b4 100644 --- a/src/Common/ZooKeeper/ZooKeeperConstants.h +++ b/src/Common/ZooKeeper/ZooKeeperConstants.h @@ -41,7 +41,7 @@ static constexpr int32_t PASSWORD_LENGTH = 16; /// ZooKeeper has 1 MB node size and serialization limit by default, /// but it can be raised up, so we have a slightly larger limit on our side. static constexpr int32_t MAX_STRING_OR_ARRAY_SIZE = 1 << 28; /// 256 MiB -static constexpr int32_t DEFAULT_SESSION_TIMEOUT = 30000; -static constexpr int32_t DEFAULT_OPERATION_TIMEOUT = 10000; +static constexpr int32_t DEFAULT_SESSION_TIMEOUT_MS = 30000; +static constexpr int32_t DEFAULT_OPERATION_TIMEOUT_MS = 10000; } diff --git a/src/Common/ZooKeeper/ZooKeeperIO.cpp b/src/Common/ZooKeeper/ZooKeeperIO.cpp index 157325cb8e6..07227669366 100644 --- a/src/Common/ZooKeeper/ZooKeeperIO.cpp +++ b/src/Common/ZooKeeper/ZooKeeperIO.cpp @@ -16,7 +16,7 @@ void write(int32_t x, WriteBuffer & out) void write(OpNum x, WriteBuffer & out) { - write(static_cast(x), out); + write(static_cast(x), out); } void write(bool x, WriteBuffer & out) diff --git a/src/Common/ya.make b/src/Common/ya.make index 0d6caa22f3a..2d8110b13d0 100644 --- a/src/Common/ya.make +++ b/src/Common/ya.make @@ -78,7 +78,11 @@ SRCS( WeakHash.cpp ZooKeeper/IKeeper.cpp ZooKeeper/TestKeeper.cpp + ZooKeeper/TestKeeperStorage.cpp ZooKeeper/ZooKeeper.cpp + ZooKeeper/ZooKeeperCommon.cpp + ZooKeeper/ZooKeeperConstants.cpp + ZooKeeper/ZooKeeperIO.cpp ZooKeeper/ZooKeeperImpl.cpp ZooKeeper/ZooKeeperNodeCache.cpp checkStackSize.cpp diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 7afd1af1aa0..cdac4fdd9b2 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -25,7 +25,7 @@ void TestKeeperTCPHandler::sendHandshake() Coordination::write(Coordination::SERVER_HANDSHAKE_LENGTH, *out); Coordination::write(Coordination::ZOOKEEPER_PROTOCOL_VERSION, *out); - Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT, *out); + Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT_MS, *out); Coordination::write(test_keeper_storage->getSessionID(), *out); std::array passwd{}; Coordination::write(passwd, *out); @@ -105,7 +105,7 @@ void TestKeeperTCPHandler::runImpl() while (true) { using namespace std::chrono_literals; - while(!responses.empty()) + while (!responses.empty()) { if (responses.front().wait_for(10ms) != std::future_status::ready) break; diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 06b3c102df5..ec44a481522 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -22,8 +22,8 @@ public: , log(&Poco::Logger::get("TestKeeperTCPHandler")) , global_context(server.context()) , test_keeper_storage(global_context.getTestKeeperStorage()) - , operation_timeout(Coordination::DEFAULT_OPERATION_TIMEOUT) - , session_timeout(Coordination::DEFAULT_SESSION_TIMEOUT) + , operation_timeout(0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000) + , session_timeout(0, Coordination::DEFAULT_SESSION_TIMEOUT_MS * 1000) { } diff --git a/src/Server/ya.make b/src/Server/ya.make index 8a9bbd3bbc2..a83b642dc8b 100644 --- a/src/Server/ya.make +++ b/src/Server/ya.make @@ -21,6 +21,7 @@ SRCS( ReplicasStatusHandler.cpp StaticRequestHandler.cpp TCPHandler.cpp + TestKeeperTCPHandler.cpp WebUIRequestHandler.cpp ) diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp index a924465d02a..1bf17a035bf 100644 --- a/utils/zookeeper-test/main.cpp +++ b/utils/zookeeper-test/main.cpp @@ -135,7 +135,7 @@ void testMultiRequest(zkutil::ZooKeeper & zk) zk.multi(requests); std::terminate(); } - catch(...) + catch (...) { std::cerr << "Got exception on multy request (it's ok)\n"; } @@ -143,7 +143,8 @@ void testMultiRequest(zkutil::ZooKeeper & zk) checkEq(zk, "/data/multirequest", "bbb"); } -int main(int argc, char *argv[]) { +int main(int argc, char *argv[]) +{ if (argc != 2) { @@ -167,7 +168,7 @@ int main(int argc, char *argv[]) { testCreateSetWatchEvent(zk); testCreateListWatchEvent(zk); } - catch(...) + catch (...) { zk.tryRemoveRecursive("/data"); throw; From df02bb23151af67364dc6a3695cc703b698b7a6a Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 17:00:28 +0300 Subject: [PATCH 026/174] Change test config --- tests/config/config.d/zookeeper.xml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/config/config.d/zookeeper.xml b/tests/config/config.d/zookeeper.xml index 68c85788c98..06ed7fcd39f 100644 --- a/tests/config/config.d/zookeeper.xml +++ b/tests/config/config.d/zookeeper.xml @@ -1,5 +1,8 @@ - testkeeper + + localhost + 9181 + From d57b57156ecda30fcdb01aa7cf2f0ff07311bbbf Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 17:22:11 +0300 Subject: [PATCH 027/174] Fix style check --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 5 ----- src/Common/ZooKeeper/ZooKeeperCommon.h | 1 - 2 files changed, 6 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 19d5fe6ad22..70702dbecc9 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -17,11 +17,8 @@ namespace DB namespace zkutil { - using namespace DB; - - static String parentPath(const String & path) { auto rslash_pos = path.rfind('/'); @@ -679,6 +676,4 @@ TestKeeperStorage::~TestKeeperStorage() } } - - } diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index 286330bd769..fbf3e13fbfe 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -19,7 +19,6 @@ #include - namespace Coordination { From ade14da63484a8c44c034f8c4bdb8a48f4a471e4 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 18:45:10 +0300 Subject: [PATCH 028/174] Remove some redundant files --- src/Common/ZooKeeper/TestKeeper.h | 2 -- utils/zookeeper-test/main.cpp | 3 ++- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeper.h b/src/Common/ZooKeeper/TestKeeper.h index 01c92c98778..ca9f584304f 100644 --- a/src/Common/ZooKeeper/TestKeeper.h +++ b/src/Common/ZooKeeper/TestKeeper.h @@ -125,8 +125,6 @@ private: Watches watches; Watches list_watches; /// Watches for 'list' request (watches on children). - void createWatchCallBack(const String & path); - using RequestsQueue = ConcurrentBoundedQueue; RequestsQueue requests_queue{1}; diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp index 1bf17a035bf..fe7bf93fd15 100644 --- a/utils/zookeeper-test/main.cpp +++ b/utils/zookeeper-test/main.cpp @@ -1,4 +1,3 @@ -#include #include #include #include @@ -14,6 +13,8 @@ using namespace std; +/// TODO: Remove ME + void checkEq(zkutil::ZooKeeper & zk, const std::string & path, const std::string & expected) { auto result = zk.get(path); From 7d32873cb9a75ed3ad730bba85317d8cbe50e761 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 11 Nov 2020 19:59:13 +0300 Subject: [PATCH 029/174] Add missed file --- tests/config/config.d/test_keeper_port.xml | 3 +++ tests/config/install.sh | 1 + 2 files changed, 4 insertions(+) create mode 100644 tests/config/config.d/test_keeper_port.xml diff --git a/tests/config/config.d/test_keeper_port.xml b/tests/config/config.d/test_keeper_port.xml new file mode 100644 index 00000000000..23b744630f2 --- /dev/null +++ b/tests/config/config.d/test_keeper_port.xml @@ -0,0 +1,3 @@ + + 9181 + diff --git a/tests/config/install.sh b/tests/config/install.sh index f6fae181ac8..4eb665b01ba 100755 --- a/tests/config/install.sh +++ b/tests/config/install.sh @@ -28,6 +28,7 @@ ln -sf $SRC_PATH/config.d/clusters.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/graphite.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/database_atomic.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/test_cluster_with_incorrect_pw.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/test_keeper_port.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/users.d/log_queries.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/readonly.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/access_management.xml $DEST_SERVER_PATH/users.d/ From eb46951e75f49776a096073ef1f45699c10d7fb1 Mon Sep 17 00:00:00 2001 From: feng lv Date: Thu, 12 Nov 2020 15:55:26 +0000 Subject: [PATCH 030/174] aggregate_functions_null_for_empty move OrNull ahead --- src/Interpreters/TreeRewriter.cpp | 27 ++++++++++++++ .../01562_agg_null_for_empty_ahead.reference | 20 +++++++++++ .../01562_agg_null_for_empty_ahead.sql | 35 +++++++++++++++++++ 3 files changed, 82 insertions(+) create mode 100644 tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference create mode 100644 tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index b2dbd027191..8724d2979c6 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -125,7 +125,34 @@ struct CustomizeAggregateFunctionsSuffixData { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) + { + auto name_size = func.name.size(); + if (endsWith(func.name, "MergeState")) + { + func.name = func.name.substr(0, name_size - 10) + customized_func_suffix + "MergeState"; + return; + } + + if (endsWith(func.name, "Merge")) + { + func.name = func.name.substr(0, name_size - 5) + customized_func_suffix + "Merge"; + return; + } + + if (endsWith(func.name, "State")) + { + func.name = func.name.substr(0, name_size - 5) + customized_func_suffix + "State"; + return; + } + + if (endsWith(func.name, "If")) + { + func.name = func.name.substr(0, name_size - 2) + customized_func_suffix + "If"; + return; + } + func.name = func.name + customized_func_suffix; + } } } }; diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference new file mode 100644 index 00000000000..31133cfb3a1 --- /dev/null +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference @@ -0,0 +1,20 @@ +0 +0 +0 +0 +0 +1 +0 +\N +1 +\N +\N +0 +\N +0 +\N +1 +0 +\N +1 +\N diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql new file mode 100644 index 00000000000..50ce4a84ee0 --- /dev/null +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql @@ -0,0 +1,35 @@ +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(0)); +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(1)); + +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(0))); +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(1))); + +SELECT sumIf(1, 0); + +SELECT sumIf(1, 1); + +SELECT sumIfOrNull(1, 0); +SELECT sumOrNullIf(1, 0); + +SELECT nullIf(1, 0); + +SELECT nullIf(1, 1); + +SET aggregate_functions_null_for_empty=1; + +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(0)); +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(1)); + +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(0))); +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(1))); + +SELECT sumIf(1, 0); + +SELECT sumIf(1, 1); + +SELECT sumIfOrNull(1, 0); +SELECT sumOrNullIf(1, 0); + +SELECT nullIf(1, 0); + +SELECT nullIf(1, 1); From 24910a61be78524d88299ad157d632dfac0bb3a9 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 18 Nov 2020 18:25:59 +0300 Subject: [PATCH 031/174] Fix stringstream --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 70702dbecc9..a374f752b18 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -5,6 +5,8 @@ #include #include #include +#include +#include namespace DB { @@ -148,7 +150,8 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest auto seq_num = it->second.seq_num; ++it->second.seq_num; - std::stringstream seq_num_str; + std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + seq_num_str.exceptions(std::ios::failbit); seq_num_str << std::setw(10) << std::setfill('0') << seq_num; path_created += seq_num_str.str(); From 00186ece72d35eeec0e644c5eb1efc7bbbf18d45 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 18 Nov 2020 23:36:25 +0300 Subject: [PATCH 032/174] Join processing thread --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index a374f752b18..262a0b372a2 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -528,8 +528,12 @@ void TestKeeperStorage::finalize() if (shutdown) return; + shutdown = true; + + processing_thread.join(); } + try { { From 991386e37b535d5278bba687c290a3d2cc70bfb7 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 19 Nov 2020 13:56:36 +0300 Subject: [PATCH 033/174] Fix stupid error --- programs/server/Server.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 8ba09c3cdd0..f93e88cca4a 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -789,6 +789,8 @@ int Server::main(const std::vector & /*args*/) for (auto & server : servers) server->start(); + size_t already_started_servers = servers.size(); + /// Set current database name before loading tables and databases because /// system logs may copy global context. global_context->setCurrentDatabaseNameInGlobalContext(default_database); @@ -1077,7 +1079,7 @@ int Server::main(const std::vector & /*args*/) global_context->enableNamedSessions(); - for (size_t i = 1; i < servers.size(); ++i) + for (size_t i = already_started_servers; i < servers.size(); ++i) servers[i]->start(); { From 8e3f7e4dbd8365667ee4743ef03c90cb26d152d8 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 19 Nov 2020 19:06:19 +0300 Subject: [PATCH 034/174] Some non working code --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 118 +++++++++++++++------ src/Common/ZooKeeper/TestKeeperStorage.h | 8 +- src/Server/TestKeeperTCPHandler.cpp | 50 ++++++--- src/Server/TestKeeperTCPHandler.h | 6 +- 4 files changed, 132 insertions(+), 50 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 262a0b372a2..20eba6b6198 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -82,7 +82,7 @@ struct TestKeeperStorageRequest TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) : zk_request(zk_request_) {} - virtual std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const = 0; + virtual std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const = 0; virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} virtual ~TestKeeperStorageRequest() {} @@ -91,7 +91,7 @@ struct TestKeeperStorageRequest struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & /* container */, int64_t /* zxid */) const override + std::pair process(TestKeeperStorage::Container & /* container */, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /* zxid */, int64_t /* session_id */) const override { return {zk_request->makeResponse(), {}}; } @@ -107,7 +107,7 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest processWatchesImpl(zk_request->getPath(), watches, list_watches); } - std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Undo undo; @@ -160,9 +160,13 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest response.path_created = path_created; container.emplace(path_created, std::move(created_node)); - undo = [&container, path_created, is_sequential = request.is_sequential, parent_path = it->first] + if (request.is_ephemeral) + ephemerals[session_id].emplace(path_created); + + undo = [&container, &ephemerals, session_id, path_created, is_sequential = request.is_sequential, parent_path = it->first] { container.erase(path_created); + ephemerals[session_id].erase(path_created); auto & undo_parent = container.at(parent_path); --undo_parent.stat.cversion; --undo_parent.stat.numChildren; @@ -185,7 +189,7 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t /* zxid */) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /* zxid */, int64_t /* session_id */) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr); @@ -210,7 +214,7 @@ struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t /*zxid*/, int64_t session_id) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperRemoveResponse & response = dynamic_cast(*response_ptr); @@ -233,6 +237,9 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest else { auto prev_node = it->second; + if (it->second.is_ephemeral) + ephemerals[session_id].erase(request.path); + container.erase(it); auto & parent = container.at(parentPath(request.path)); --parent.stat.numChildren; @@ -260,7 +267,7 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest struct TestKeeperStorageExistsRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /* session_id */) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperExistsResponse & response = dynamic_cast(*response_ptr); @@ -284,7 +291,7 @@ struct TestKeeperStorageExistsRequest final : public TestKeeperStorageRequest struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t zxid, int64_t /* session_id */) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperSetResponse & response = dynamic_cast(*response_ptr); @@ -333,7 +340,7 @@ struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /*session_id*/) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperListResponse & response = dynamic_cast(*response_ptr); @@ -372,7 +379,7 @@ struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest struct TestKeeperStorageCheckRequest final : public TestKeeperStorageRequest { using TestKeeperStorageRequest::TestKeeperStorageRequest; - std::pair process(TestKeeperStorage::Container & container, int64_t /*zxid*/) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /*session_id*/) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperCheckResponse & response = dynamic_cast(*response_ptr); @@ -427,7 +434,7 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest } } - std::pair process(TestKeeperStorage::Container & container, int64_t zxid) const override + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const override { Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); Coordination::ZooKeeperMultiResponse & response = dynamic_cast(*response_ptr); @@ -438,7 +445,7 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest size_t i = 0; for (const auto & concrete_request : concrete_requests) { - auto [ cur_response, undo_action ] = concrete_request->process(container, zxid); + auto [ cur_response, undo_action ] = concrete_request->process(container, ephemerals, zxid, session_id); response.responses[i] = cur_response; if (cur_response->error != Coordination::Error::ZOK) { @@ -472,6 +479,15 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest } }; +struct TestKeeperStorageCloseRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container &, TestKeeperStorage::Ephemerals &, int64_t, int64_t) const override + { + throw Coordination::Exception("Called process on close request", Coordination::Error::ZRUNTIMEINCONSISTENCY); + } +}; + void TestKeeperStorage::processingThread() { setThreadName("TestKeeperSProc"); @@ -489,27 +505,48 @@ void TestKeeperStorage::processingThread() if (shutdown) break; - if (info.watch_callback) - { - auto & watches_type = dynamic_cast(info.request->zk_request.get()) - ? list_watches - : watches; - - watches_type[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); - } - auto zk_request = info.request->zk_request; - - auto [response, _] = info.request->process(container, zxid); - if (response->error == Coordination::Error::ZOK) + if (dynamic_cast(zk_request.get())) { - info.request->processWatches(watches, list_watches); + auto it = ephemerals.find(info.session_id); + if (it != ephemerals.end()) + { + for (const auto & ephemeral_path : it->second) + { + container.erase(ephemeral_path); + processWatchesImpl(ephemeral_path, watches, list_watches); + } + ephemerals.erase(it); + } } + else + { + auto [response, _] = info.request->process(container, ephemerals, zxid, info.session_id); - response->xid = zk_request->xid; - response->zxid = getZXID(); + if (info.watch_callback) + { + if (response->error == Coordination::Error::ZOK) + { + auto & watches_type = dynamic_cast(zk_request.get()) + ? list_watches + : watches; - info.response_callback(response); + watches_type[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + } + else if (response->error == Coordination::Error::ZNONODE && dynamic_cast(zk_request.get())) + { + watches[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + } + } + + if (response->error == Coordination::Error::ZOK) + info.request->processWatches(watches, list_watches); + + response->xid = zk_request->xid; + response->zxid = getZXID(); + + info.response_callback(response); + } } } } @@ -531,7 +568,8 @@ void TestKeeperStorage::finalize() shutdown = true; - processing_thread.join(); + if (processing_thread.joinable()) + processing_thread.join(); } try @@ -634,7 +672,7 @@ TestKeeperWrapperFactory::TestKeeperWrapperFactory() { registerTestKeeperRequestWrapper(*this); //registerTestKeeperRequestWrapper(*this); - //registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); @@ -645,7 +683,20 @@ TestKeeperWrapperFactory::TestKeeperWrapperFactory() registerTestKeeperRequestWrapper(*this); } -TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request) + +void TestKeeperStorage::putCloseRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id) +{ + TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); + RequestInfo request_info; + request_info.time = clock::now(); + request_info.request = storage_request; + request_info.session_id = session_id; + std::lock_guard lock(push_request_mutex); + if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); +} + +TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id) { auto promise = std::make_shared>(); auto future = promise->get_future(); @@ -653,6 +704,7 @@ TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination RequestInfo request_info; request_info.time = clock::now(); request_info.request = storage_request; + request_info.session_id = session_id; request_info.response_callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); }; std::optional watch_future; if (request->has_watch) @@ -665,17 +717,15 @@ TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); + return ResponsePair{std::move(future), std::move(watch_future)}; } - TestKeeperStorage::~TestKeeperStorage() { try { finalize(); - if (processing_thread.joinable()) - processing_thread.join(); } catch (...) { diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index 02fdc21cc58..f1383b7b367 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -5,6 +5,8 @@ #include #include #include +#include +#include namespace zkutil { @@ -33,11 +35,13 @@ public: }; using Container = std::map; + using Ephemerals = std::unordered_map>; using WatchCallbacks = std::vector; using Watches = std::map; Container container; + Ephemerals ephemerals; std::atomic zxid{0}; std::atomic shutdown{false}; @@ -53,6 +57,7 @@ public: ResponseCallback response_callback; ResponseCallback watch_callback; clock::time_point time; + int64_t session_id; }; std::mutex push_request_mutex; @@ -74,7 +79,8 @@ public: AsyncResponse response; std::optional watch_response; }; - ResponsePair putRequest(const Coordination::ZooKeeperRequestPtr & request); + ResponsePair putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id); + void putCloseRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id); int64_t getSessionID() { diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index cdac4fdd9b2..1c607ea0a9c 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -22,11 +22,11 @@ namespace ErrorCodes void TestKeeperTCPHandler::sendHandshake() { - + session_id = test_keeper_storage->getSessionID(); Coordination::write(Coordination::SERVER_HANDSHAKE_LENGTH, *out); Coordination::write(Coordination::ZOOKEEPER_PROTOCOL_VERSION, *out); Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT_MS, *out); - Coordination::write(test_keeper_storage->getSessionID(), *out); + Coordination::write(session_id, *out); std::array passwd{}; Coordination::write(passwd, *out); out->next(); @@ -101,6 +101,7 @@ void TestKeeperTCPHandler::runImpl() } sendHandshake(); + session_stopwatch.start(); while (true) { @@ -128,22 +129,38 @@ void TestKeeperTCPHandler::runImpl() } } - long poll_wait = responses.empty() ? session_timeout.totalMicroseconds() : 10000; + Int64 poll_wait = responses.empty() ? session_timeout.totalMicroseconds() - session_stopwatch.elapsedMicroseconds() : 10000; if (in->poll(poll_wait)) { - bool close_received = receiveRequest(); - if (close_received) + auto received_op = receiveRequest(); + if (received_op == Coordination::OpNum::Close) { - LOG_DEBUG(log, "Received close request"); + LOG_DEBUG(log, "Received close request for session #{}", session_id); break; } + else if (received_op == Coordination::OpNum::Heartbeat) + { + session_stopwatch.restart(); + } + } + + if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) + { + LOG_DEBUG(log, "Session #{} expired", session_id); + putCloseRequest(); + break; } } } +void TestKeeperTCPHandler::putCloseRequest() +{ + Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(Coordination::OpNum::Close); + test_keeper_storage->putCloseRequest(request, session_id); +} -bool TestKeeperTCPHandler::receiveRequest() +Coordination::OpNum TestKeeperTCPHandler::receiveRequest() { int32_t length; Coordination::read(length, *in); @@ -152,18 +169,23 @@ bool TestKeeperTCPHandler::receiveRequest() Coordination::OpNum opnum; Coordination::read(opnum, *in); - if (opnum == Coordination::OpNum::Close) - return true; Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); request->xid = xid; request->readImpl(*in); - auto request_future_responses = test_keeper_storage->putRequest(request); - responses.push(std::move(request_future_responses.response)); - if (request_future_responses.watch_response) - watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); + if (opnum != Coordination::OpNum::Close) + { + auto request_future_responses = test_keeper_storage->putRequest(request, session_id); + responses.push(std::move(request_future_responses.response)); + if (request_future_responses.watch_response) + watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); + } + else + { + test_keeper_storage->putCloseRequest(request, session_id); + } - return false; + return opnum; } diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index ec44a481522..76da1bad89b 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -2,6 +2,7 @@ #include #include "IServer.h" +#include #include #include #include @@ -35,6 +36,8 @@ private: std::shared_ptr test_keeper_storage; Poco::Timespan operation_timeout; Poco::Timespan session_timeout; + int64_t session_id; + Stopwatch session_stopwatch; std::queue responses; std::vector watch_responses; @@ -48,7 +51,8 @@ private: void sendHandshake(); void receiveHandshake(); - bool receiveRequest(); + Coordination::OpNum receiveRequest(); + void putCloseRequest(); }; } From 5925760892072b34fbf9bef047ae21cc714c09e8 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 19 Nov 2020 19:31:55 +0300 Subject: [PATCH 035/174] Fix handler --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 12 ++++++++++-- src/Server/TestKeeperTCPHandler.cpp | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 20eba6b6198..5fb188ac7a0 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -531,11 +531,19 @@ void TestKeeperStorage::processingThread() ? list_watches : watches; - watches_type[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + watches_type[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); } else if (response->error == Coordination::Error::ZNONODE && dynamic_cast(zk_request.get())) { - watches[info.request->zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + watches[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + } + else + { + Coordination::ZooKeeperWatchResponse watch_response; + watch_response.path = zk_request->getPath(); + watch_response.xid = -1; + watch_response.error = response->error; + info.watch_callback(std::make_shared(watch_response)); } } diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 1c607ea0a9c..261e9576ec5 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -120,7 +120,9 @@ void TestKeeperTCPHandler::runImpl() { if (it->wait_for(0s) == std::future_status::ready) { - it->get()->write(*out); + auto response = it->get(); + if (response->error == Coordination::Error::ZOK) + response->write(*out); it = watch_responses.erase(it); } else From fdf3b569f4a47a3072bdc9413014f96c5bf24b4c Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 19 Nov 2020 19:48:49 +0300 Subject: [PATCH 036/174] Fix bug --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 5fb188ac7a0..9f88774455f 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -163,10 +163,11 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest if (request.is_ephemeral) ephemerals[session_id].emplace(path_created); - undo = [&container, &ephemerals, session_id, path_created, is_sequential = request.is_sequential, parent_path = it->first] + undo = [&container, &ephemerals, session_id, path_created, is_sequential = request.is_sequential, is_ephemeral = request.is_ephemeral, parent_path = it->first] { container.erase(path_created); - ephemerals[session_id].erase(path_created); + if (is_ephemeral) + ephemerals[session_id].erase(path_created); auto & undo_parent = container.at(parent_path); --undo_parent.stat.cversion; --undo_parent.stat.numChildren; From 73ed05492bf21f12f68bfc910abd7d8db0aaa0ee Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 11:37:16 +0300 Subject: [PATCH 037/174] More correct initialization --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 9 +++++++-- src/Server/TestKeeperTCPHandler.cpp | 2 -- src/Server/TestKeeperTCPHandler.h | 1 + 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 9f88774455f..02649dccbff 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -40,6 +40,7 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & Coordination::ZooKeeperWatchResponse watch_response; watch_response.path = path; watch_response.xid = -1; + watch_response.zxid = -1; auto it = watches.find(watch_response.path); if (it != watches.end()) @@ -54,6 +55,7 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & Coordination::ZooKeeperWatchResponse watch_list_response; watch_list_response.path = parentPath(path); watch_list_response.xid = -1; + watch_list_response.zxid = -1; it = list_watches.find(watch_list_response.path); if (it != list_watches.end()) @@ -238,7 +240,7 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest else { auto prev_node = it->second; - if (it->second.is_ephemeral) + if (prev_node.is_ephemeral) ephemerals[session_id].erase(request.path); container.erase(it); @@ -247,8 +249,11 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest ++parent.stat.cversion; response.error = Coordination::Error::ZOK; - undo = [prev_node, &container, path = request.path] + undo = [prev_node, &container, &ephemerals, session_id, path = request.path] { + if (prev_node.is_ephemeral) + ephemerals[session_id].emplace(path); + container.emplace(path, prev_node); auto & undo_parent = container.at(parentPath(path)); ++undo_parent.stat.numChildren; diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 261e9576ec5..935e392325d 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -17,12 +17,10 @@ namespace DB namespace ErrorCodes { extern const int LOGICAL_ERROR; - } void TestKeeperTCPHandler::sendHandshake() { - session_id = test_keeper_storage->getSessionID(); Coordination::write(Coordination::SERVER_HANDSHAKE_LENGTH, *out); Coordination::write(Coordination::ZOOKEEPER_PROTOCOL_VERSION, *out); Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT_MS, *out); diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index 76da1bad89b..ca72b9711bb 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -25,6 +25,7 @@ public: , test_keeper_storage(global_context.getTestKeeperStorage()) , operation_timeout(0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000) , session_timeout(0, Coordination::DEFAULT_SESSION_TIMEOUT_MS * 1000) + , session_id(test_keeper_storage->getSessionID()) { } From 29631aaf2e9e57fdeba2022b0d4187c2034fc8e2 Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 14:48:48 +0300 Subject: [PATCH 038/174] Fix build warnings --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 25 +++++++++++----------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 02649dccbff..31f2b602b4f 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -81,14 +81,13 @@ struct TestKeeperStorageRequest { Coordination::ZooKeeperRequestPtr zk_request; - TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + explicit TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) : zk_request(zk_request_) {} virtual std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const = 0; virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} - virtual ~TestKeeperStorageRequest() {} -}; + virtual ~TestKeeperStorageRequest() = default struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest { @@ -411,29 +410,29 @@ struct TestKeeperStorageCheckRequest final : public TestKeeperStorageRequest struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest { std::vector concrete_requests; - TestKeeperStorageMultiRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + explicit TestKeeperStorageMultiRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) : TestKeeperStorageRequest(zk_request_) { Coordination::ZooKeeperMultiRequest & request = dynamic_cast(*zk_request); concrete_requests.reserve(request.requests.size()); - for (const auto & zk_request : request.requests) + for (const auto & sub_zk_request : request.requests) { - if (const auto * concrete_request_create = dynamic_cast(zk_request.get())) + if (dynamic_cast(sub_zk_request.get())) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); } - else if (const auto * concrete_request_remove = dynamic_cast(zk_request.get())) + else if (dynamic_cast(sub_zk_request.get())) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); } - else if (const auto * concrete_request_set = dynamic_cast(zk_request.get())) + else if (dynamic_cast(sub_zk_request.get())) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); } - else if (const auto * concrete_request_check = dynamic_cast(zk_request.get())) + else if (dynamic_cast(sub_zk_request.get())) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(zk_request))); + concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); } else throw Coordination::Exception("Illegal command as part of multi ZooKeeper request", Coordination::Error::ZBADARGUMENTS); From ab5f3b0b41fdcaae6872aff1cfd7c86c56977da1 Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 15:31:40 +0300 Subject: [PATCH 039/174] Fix build --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 31f2b602b4f..3a16aadf2f7 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -87,7 +87,7 @@ struct TestKeeperStorageRequest virtual std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const = 0; virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} - virtual ~TestKeeperStorageRequest() = default + virtual ~TestKeeperStorageRequest() = default; struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest { From 20eeab45027e4e947f1192c73f868a57e1a7e4cf Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 15:36:10 +0300 Subject: [PATCH 040/174] Fix build #2 --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 3a16aadf2f7..e0ca5326dd1 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -88,6 +88,7 @@ struct TestKeeperStorageRequest virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} virtual ~TestKeeperStorageRequest() = default; +}; struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest { From 0208751a7de6ad5117a516ce6610bb00416186e8 Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 15:46:05 +0300 Subject: [PATCH 041/174] Add dirty hack for connections close --- programs/server/Server.cpp | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index f93e88cca4a..2657475b764 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -1110,10 +1110,12 @@ int Server::main(const std::vector & /*args*/) is_cancelled = true; int current_connections = 0; - for (auto & server : servers) + for (size_t i = 0; i < servers.size(); ++i) { - server->stop(); - current_connections += server->currentConnections(); + servers[i]->stop(); + /// TODO (alesap) + if (i >= already_started_servers) + current_connections += servers[i]->currentConnections(); } if (current_connections) From 63464df3eb46cb8cdc2b35b9e4314d37ef8a9c39 Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 20 Nov 2020 18:00:23 +0300 Subject: [PATCH 042/174] Fix clang-tidy --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index e0ca5326dd1..43311cdae3d 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -670,8 +670,6 @@ public: private: OpNumToRequest op_num_to_request; - -private: TestKeeperWrapperFactory(); }; From 5a48076802837af3d82df6d3ae06bd8ad10e2ec6 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 23 Nov 2020 13:55:00 +0300 Subject: [PATCH 043/174] Trying to get rid of leak --- programs/server/Server.cpp | 51 +++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 2657475b764..4593b5b1a6d 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -472,27 +472,6 @@ int Server::main(const std::vector & /*args*/) StatusFile status{path + "status", StatusFile::write_full_info}; - SCOPE_EXIT({ - /** Ask to cancel background jobs all table engines, - * and also query_log. - * It is important to do early, not in destructor of Context, because - * table engines could use Context on destroy. - */ - LOG_INFO(log, "Shutting down storages."); - - global_context->shutdown(); - - LOG_DEBUG(log, "Shut down storages."); - - /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. - * At this moment, no one could own shared part of Context. - */ - global_context_ptr = nullptr; - global_context.reset(); - shared_context.reset(); - LOG_DEBUG(log, "Destroyed global context."); - }); - /// Try to increase limit on number of open files. { rlimit rlim; @@ -791,6 +770,30 @@ int Server::main(const std::vector & /*args*/) size_t already_started_servers = servers.size(); + SCOPE_EXIT({ + /** Ask to cancel background jobs all table engines, + * and also query_log. + * It is important to do early, not in destructor of Context, because + * table engines could use Context on destroy. + */ + LOG_INFO(log, "Shutting down storages."); + + global_context->shutdown(); + + LOG_DEBUG(log, "Shut down storages."); + + for (size_t i = 0; i < already_started_servers; ++i) + servers[i]->stop(); + + /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. + * At this moment, no one could own shared part of Context. + */ + global_context_ptr = nullptr; + global_context.reset(); + shared_context.reset(); + LOG_DEBUG(log, "Destroyed global context."); + }); + /// Set current database name before loading tables and databases because /// system logs may copy global context. global_context->setCurrentDatabaseNameInGlobalContext(default_database); @@ -1110,12 +1113,10 @@ int Server::main(const std::vector & /*args*/) is_cancelled = true; int current_connections = 0; - for (size_t i = 0; i < servers.size(); ++i) + for (size_t i = already_started_servers; i < servers.size(); ++i) { servers[i]->stop(); - /// TODO (alesap) - if (i >= already_started_servers) - current_connections += servers[i]->currentConnections(); + current_connections += servers[i]->currentConnections(); } if (current_connections) From 2da250164ede5997677008f0cb693c68e2368e1e Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 23 Nov 2020 17:03:57 +0300 Subject: [PATCH 044/174] Fix clang tidy message --- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index d1c6b1ee80b..7f7e5f4abaf 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -383,7 +383,7 @@ void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const { - for (auto & response : responses) + for (const auto & response : responses) { const ZooKeeperResponse & zk_response = dynamic_cast(*response); OpNum op_num = zk_response.getOpNum(); From 1de175257bc49724f6a6ebc75819f4a25bbd4a74 Mon Sep 17 00:00:00 2001 From: alesapin Date: Mon, 23 Nov 2020 23:22:04 +0300 Subject: [PATCH 045/174] More const --- programs/server/Server.cpp | 4 ++-- programs/server/Server.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 4593b5b1a6d..2c58b22bd2a 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -208,7 +208,7 @@ Poco::Net::SocketAddress makeSocketAddress(const std::string & host, UInt16 port return socket_address; } -Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) +Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) const { auto address = makeSocketAddress(host, port, &logger()); #if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100 @@ -229,7 +229,7 @@ Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & sock return address; } -void Server::createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) +void Server::createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const { /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file. if (!config().has(port_name)) diff --git a/programs/server/Server.h b/programs/server/Server.h index 57fcea3cee8..c582e475308 100644 --- a/programs/server/Server.h +++ b/programs/server/Server.h @@ -67,10 +67,10 @@ private: private: - Poco::Net::SocketAddress socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false); + Poco::Net::SocketAddress socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false) const; using CreateServerFunc = std::function; - void createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func); + void createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const; }; } From 0d52cfb1be5fbd45235e544999d6006d3d08617c Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 24 Nov 2020 17:02:55 +0300 Subject: [PATCH 046/174] Less clumsy code with interruptable code --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 29 +-- src/Common/ZooKeeper/TestKeeperStorage.h | 4 +- src/Server/TestKeeperTCPHandler.cpp | 201 ++++++++++++++++++--- src/Server/TestKeeperTCPHandler.h | 17 +- 4 files changed, 196 insertions(+), 55 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 43311cdae3d..299a6606058 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -708,29 +708,34 @@ void TestKeeperStorage::putCloseRequest(const Coordination::ZooKeeperRequestPtr throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); } -TestKeeperStorage::ResponsePair TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id) + +void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback) { - auto promise = std::make_shared>(); - auto future = promise->get_future(); TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); RequestInfo request_info; request_info.time = clock::now(); request_info.request = storage_request; request_info.session_id = session_id; - request_info.response_callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); }; - std::optional watch_future; + request_info.response_callback = callback; + std::lock_guard lock(push_request_mutex); + if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); +} + +void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback) +{ + TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); + RequestInfo request_info; + request_info.time = clock::now(); + request_info.request = storage_request; + request_info.session_id = session_id; + request_info.response_callback = callback; if (request->has_watch) - { - auto watch_promise = std::make_shared>(); - watch_future.emplace(watch_promise->get_future()); - request_info.watch_callback = [watch_promise] (const Coordination::ZooKeeperResponsePtr & response) { watch_promise->set_value(response); }; - } + request_info.watch_callback = watch_callback; std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); - - return ResponsePair{std::move(future), std::move(watch_future)}; } TestKeeperStorage::~TestKeeperStorage() diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index f1383b7b367..ee8bc1ae713 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -79,7 +79,9 @@ public: AsyncResponse response; std::optional watch_response; }; - ResponsePair putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id); + void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback); + void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback); + void putCloseRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id); int64_t getSessionID() diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 935e392325d..c5c0d17ecba 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -11,12 +11,126 @@ #include #include +#include +#include + namespace DB { namespace ErrorCodes { extern const int LOGICAL_ERROR; + extern const int SYSTEM_ERROR; +} + +struct SocketInterruptablePollWrapper +{ + int sockfd; + int efd; + enum class PollStatus + { + HAS_DATA, + TIMEOUT, + INTERRUPTED, + ERROR, + }; + + using InterruptCallback = std::function; + + SocketInterruptablePollWrapper(const Poco::Net::StreamSocket & poco_socket_) + : sockfd(poco_socket_.impl()->sockfd()) + , efd(eventfd(0, EFD_NONBLOCK)) + { + if (efd < 0) + throwFromErrno("Cannot create eventfd file descriptor", ErrorCodes::SYSTEM_ERROR); + } + + ~SocketInterruptablePollWrapper() + { + if (efd >= 0) + { + close(efd); + efd = -1; + } + } + + void interruptPoll() + { + UInt64 bytes = 1; + int ret = write(efd, &bytes, sizeof(bytes)); + if (ret < 0) + throwFromErrno("Cannot write into eventfd descriptor", ErrorCodes::SYSTEM_ERROR); + } + + PollStatus poll(Poco::Timespan& remainingTime) + { + int epollfd = epoll_create(2); + + if (epollfd < 0) + throwFromErrno("Cannot epoll_create", ErrorCodes::SYSTEM_ERROR); + + epoll_event socket_event{}; + socket_event.events = EPOLLIN | EPOLLERR; + socket_event.data.fd = sockfd; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &socket_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } + epoll_event efd_event{}; + efd_event.events = EPOLLIN | EPOLLERR; + efd_event.data.fd = efd; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, efd, &efd_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } + + int rc; + epoll_event evout{}; + do + { + Poco::Timestamp start; + rc = epoll_wait(epollfd, &evout, 1, remainingTime.totalMilliseconds()); + if (rc < 0 && errno == EINTR) + { + Poco::Timestamp end; + Poco::Timespan waited = end - start; + if (waited < remainingTime) + remainingTime -= waited; + else + remainingTime = 0; + } + } + while (rc < 0 && errno == EINTR); + + ::close(epollfd); + if (rc < 0) + return PollStatus::ERROR; + else if (rc == 0) + return PollStatus::TIMEOUT; + else if (evout.data.fd == efd) + { + UInt64 bytes; + if (read(efd, &bytes, sizeof(bytes)) < 0) + throwFromErrno("Cannot read from eventfd", ErrorCodes::SYSTEM_ERROR); + return PollStatus::INTERRUPTED; + } + return PollStatus::HAS_DATA; + } +}; + +TestKeeperTCPHandler::TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) + : Poco::Net::TCPServerConnection(socket_) + , server(server_) + , log(&Poco::Logger::get("TestKeeperTCPHandler")) + , global_context(server.context()) + , test_keeper_storage(global_context.getTestKeeperStorage()) + , operation_timeout(0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000) + , session_timeout(0, Coordination::DEFAULT_SESSION_TIMEOUT_MS * 1000) + , session_id(test_keeper_storage->getSessionID()) + , poll_wrapper(std::make_unique(socket_)) +{ } void TestKeeperTCPHandler::sendHandshake() @@ -104,34 +218,11 @@ void TestKeeperTCPHandler::runImpl() while (true) { using namespace std::chrono_literals; - while (!responses.empty()) - { - if (responses.front().wait_for(10ms) != std::future_status::ready) - break; - auto response = responses.front().get(); - response->write(*out); - responses.pop(); - } + Poco::Timespan poll_wait = responses.empty() ? session_timeout.totalMicroseconds() - session_stopwatch.elapsedMicroseconds() : session_timeout; - for (auto it = watch_responses.begin(); it != watch_responses.end();) - { - if (it->wait_for(0s) == std::future_status::ready) - { - auto response = it->get(); - if (response->error == Coordination::Error::ZOK) - response->write(*out); - it = watch_responses.erase(it); - } - else - { - ++it; - } - } - - Int64 poll_wait = responses.empty() ? session_timeout.totalMicroseconds() - session_stopwatch.elapsedMicroseconds() : 10000; - - if (in->poll(poll_wait)) + auto state = poll_wrapper->poll(poll_wait); + if (state == SocketInterruptablePollWrapper::PollStatus::HAS_DATA) { auto received_op = receiveRequest(); if (received_op == Coordination::OpNum::Close) @@ -144,6 +235,36 @@ void TestKeeperTCPHandler::runImpl() session_stopwatch.restart(); } } + else if (state == SocketInterruptablePollWrapper::PollStatus::INTERRUPTED) + { + while (!responses.empty()) + { + if (responses.front().wait_for(0ms) != std::future_status::ready) + break; + + auto response = responses.front().get(); + response->write(*out); + responses.pop(); + } + for (auto it = watch_responses.begin(); it != watch_responses.end();) + { + if (it->wait_for(0s) == std::future_status::ready) + { + auto response = it->get(); + if (response->error == Coordination::Error::ZOK) + response->write(*out); + it = watch_responses.erase(it); + } + else + { + ++it; + } + } + } + else if (state == SocketInterruptablePollWrapper::PollStatus::ERROR) + { + throw Exception("Exception happened while reading from socket", ErrorCodes::SYSTEM_ERROR); + } if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) { @@ -175,10 +296,30 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() request->readImpl(*in); if (opnum != Coordination::OpNum::Close) { - auto request_future_responses = test_keeper_storage->putRequest(request, session_id); - responses.push(std::move(request_future_responses.response)); - if (request_future_responses.watch_response) - watch_responses.emplace_back(std::move(*request_future_responses.watch_response)); + auto promise = std::make_shared>(); + zkutil::ResponseCallback callback = [this, promise] (const Coordination::ZooKeeperResponsePtr & response) + { + promise->set_value(response); + poll_wrapper->interruptPoll(); + }; + + if (request->has_watch) + { + auto watch_promise = std::make_shared>(); + zkutil::ResponseCallback watch_callback = [this, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) + { + watch_promise->set_value(response); + poll_wrapper->interruptPoll(); + }; + test_keeper_storage->putRequest(request, session_id, callback, watch_callback); + responses.push(promise->get_future()); + watch_responses.emplace_back(watch_promise->get_future()); + } + else + { + test_keeper_storage->putRequest(request, session_id, callback); + responses.push(promise->get_future()); + } } else { diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index ca72b9711bb..f2674ff0f75 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -14,21 +14,13 @@ namespace DB { +struct SocketInterruptablePollWrapper; +using SocketInterruptablePollWrapperPtr = std::unique_ptr; + class TestKeeperTCPHandler : public Poco::Net::TCPServerConnection { public: - TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) - : Poco::Net::TCPServerConnection(socket_) - , server(server_) - , log(&Poco::Logger::get("TestKeeperTCPHandler")) - , global_context(server.context()) - , test_keeper_storage(global_context.getTestKeeperStorage()) - , operation_timeout(0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000) - , session_timeout(0, Coordination::DEFAULT_SESSION_TIMEOUT_MS * 1000) - , session_id(test_keeper_storage->getSessionID()) - { - } - + TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_); void run() override; private: IServer & server; @@ -39,6 +31,7 @@ private: Poco::Timespan session_timeout; int64_t session_id; Stopwatch session_stopwatch; + SocketInterruptablePollWrapperPtr poll_wrapper; std::queue responses; std::vector watch_responses; From a74235397eadf2e555014448ed9d9d2db33250f8 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Tue, 24 Nov 2020 23:31:43 +0800 Subject: [PATCH 047/174] ISSUES-16835 try fix miss match header with MySQL SHOW statement --- base/mysqlxx/Row.cpp | 8 +++ base/mysqlxx/Row.h | 2 + src/Databases/MySQL/MaterializeMetadata.cpp | 36 ++++-------- src/Databases/MySQL/MaterializeMetadata.h | 4 +- .../MySQL/MaterializeMySQLSyncThread.cpp | 28 +++------ .../MySQL/MaterializeMySQLSyncThread.h | 4 +- src/Formats/MySQLBlockInputStream.cpp | 57 ++++++++++++++----- src/Formats/MySQLBlockInputStream.h | 10 +++- 8 files changed, 84 insertions(+), 65 deletions(-) diff --git a/base/mysqlxx/Row.cpp b/base/mysqlxx/Row.cpp index aecec46e519..bfc18689403 100644 --- a/base/mysqlxx/Row.cpp +++ b/base/mysqlxx/Row.cpp @@ -21,4 +21,12 @@ Value Row::operator[] (const char * name) const throw Exception(std::string("Unknown column ") + name); } +std::string Row::getFieldName(size_t n) const +{ + if (res->getNumFields() <= n) + throw Exception(std::string("Unknown column position ") + std::to_string(n)); + + return res->getFields()[n].name; +} + } diff --git a/base/mysqlxx/Row.h b/base/mysqlxx/Row.h index a0b88638546..547ee35e52e 100644 --- a/base/mysqlxx/Row.h +++ b/base/mysqlxx/Row.h @@ -79,6 +79,8 @@ public: */ operator private_bool_type() const { return row == nullptr ? nullptr : &Row::row; } + std::string getFieldName(size_t n) const; + private: MYSQL_ROW row{}; ResultBase * res{}; diff --git a/src/Databases/MySQL/MaterializeMetadata.cpp b/src/Databases/MySQL/MaterializeMetadata.cpp index c001955a8ae..591527474ca 100644 --- a/src/Databases/MySQL/MaterializeMetadata.cpp +++ b/src/Databases/MySQL/MaterializeMetadata.cpp @@ -36,7 +36,7 @@ static std::unordered_map fetchTablesCreateQuery( MySQLBlockInputStream show_create_table( connection, "SHOW CREATE TABLE " + backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(fetch_table_name), - show_create_table_header, DEFAULT_BLOCK_SIZE); + show_create_table_header, DEFAULT_BLOCK_SIZE, false, true); Block create_query_block = show_create_table.read(); if (!create_query_block || create_query_block.rows() != 1) @@ -77,7 +77,7 @@ void MaterializeMetadata::fetchMasterStatus(mysqlxx::PoolWithFailover::Entry & c {std::make_shared(), "Executed_Gtid_Set"}, }; - MySQLBlockInputStream input(connection, "SHOW MASTER STATUS;", header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream input(connection, "SHOW MASTER STATUS;", header, DEFAULT_BLOCK_SIZE, false, true); Block master_status = input.read(); if (!master_status || master_status.rows() != 1) @@ -99,7 +99,7 @@ void MaterializeMetadata::fetchMasterVariablesValue(const mysqlxx::PoolWithFailo }; const String & fetch_query = "SHOW VARIABLES WHERE Variable_name = 'binlog_checksum'"; - MySQLBlockInputStream variables_input(connection, fetch_query, variables_header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream variables_input(connection, fetch_query, variables_header, DEFAULT_BLOCK_SIZE, false, true); while (Block variables_block = variables_input.read()) { @@ -114,23 +114,6 @@ void MaterializeMetadata::fetchMasterVariablesValue(const mysqlxx::PoolWithFailo } } -static Block getShowMasterLogHeader(const String & mysql_version) -{ - if (startsWith(mysql_version, "5.")) - { - return Block { - {std::make_shared(), "Log_name"}, - {std::make_shared(), "File_size"} - }; - } - - return Block { - {std::make_shared(), "Log_name"}, - {std::make_shared(), "File_size"}, - {std::make_shared(), "Encrypted"} - }; -} - static bool checkSyncUserPrivImpl(mysqlxx::PoolWithFailover::Entry & connection, WriteBuffer & out) { Block sync_user_privs_header @@ -174,9 +157,14 @@ static void checkSyncUserPriv(mysqlxx::PoolWithFailover::Entry & connection) "But the SYNC USER grant query is: " + out.str(), ErrorCodes::SYNC_MYSQL_USER_ACCESS_ERROR); } -bool MaterializeMetadata::checkBinlogFileExists(mysqlxx::PoolWithFailover::Entry & connection, const String & mysql_version) const +bool MaterializeMetadata::checkBinlogFileExists(mysqlxx::PoolWithFailover::Entry & connection) const { - MySQLBlockInputStream input(connection, "SHOW MASTER LOGS", getShowMasterLogHeader(mysql_version), DEFAULT_BLOCK_SIZE); + Block logs_header { + {std::make_shared(), "Log_name"}, + {std::make_shared(), "File_size"} + }; + + MySQLBlockInputStream input(connection, "SHOW MASTER LOGS", logs_header, DEFAULT_BLOCK_SIZE, false, true); while (Block block = input.read()) { @@ -233,7 +221,7 @@ void MaterializeMetadata::transaction(const MySQLReplication::Position & positio MaterializeMetadata::MaterializeMetadata( mysqlxx::PoolWithFailover::Entry & connection, const String & path_, - const String & database, bool & opened_transaction, const String & mysql_version) + const String & database, bool & opened_transaction) : persistent_path(path_) { checkSyncUserPriv(connection); @@ -251,7 +239,7 @@ MaterializeMetadata::MaterializeMetadata( assertString("\nData Version:\t", in); readIntText(data_version, in); - if (checkBinlogFileExists(connection, mysql_version)) + if (checkBinlogFileExists(connection)) return; } diff --git a/src/Databases/MySQL/MaterializeMetadata.h b/src/Databases/MySQL/MaterializeMetadata.h index 94dfc73e5df..336dc85d899 100644 --- a/src/Databases/MySQL/MaterializeMetadata.h +++ b/src/Databases/MySQL/MaterializeMetadata.h @@ -41,13 +41,13 @@ struct MaterializeMetadata void fetchMasterVariablesValue(const mysqlxx::PoolWithFailover::Entry & connection); - bool checkBinlogFileExists(mysqlxx::PoolWithFailover::Entry & connection, const String & mysql_version) const; + bool checkBinlogFileExists(mysqlxx::PoolWithFailover::Entry & connection) const; void transaction(const MySQLReplication::Position & position, const std::function & fun); MaterializeMetadata( mysqlxx::PoolWithFailover::Entry & connection, const String & path - , const String & database, bool & opened_transaction, const String & mysql_version); + , const String & database, bool & opened_transaction); }; } diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp index b8f13f4ed18..ceea22091b6 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp @@ -93,7 +93,7 @@ MaterializeMySQLSyncThread::~MaterializeMySQLSyncThread() } } -static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection) +static void checkMySQLVariables(const mysqlxx::Pool::Entry & connection) { Block variables_header{ {std::make_shared(), "Variable_name"}, @@ -106,7 +106,7 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection "OR (Variable_name = 'binlog_row_image' AND upper(Value) = 'FULL') " "OR (Variable_name = 'default_authentication_plugin' AND upper(Value) = 'MYSQL_NATIVE_PASSWORD');"; - MySQLBlockInputStream variables_input(connection, check_query, variables_header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream variables_input(connection, check_query, variables_header, DEFAULT_BLOCK_SIZE, false, true); Block variables_block = variables_input.read(); if (!variables_block || variables_block.rows() != 4) @@ -140,15 +140,6 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection throw Exception(error_message.str(), ErrorCodes::ILLEGAL_MYSQL_VARIABLE); } - - Block version_header{{std::make_shared(), "version"}}; - MySQLBlockInputStream version_input(connection, "SELECT version() AS version;", version_header, DEFAULT_BLOCK_SIZE); - - Block version_block = version_input.read(); - if (!version_block || version_block.rows() != 1) - throw Exception("LOGICAL ERROR: cannot get mysql version.", ErrorCodes::LOGICAL_ERROR); - - return version_block.getByPosition(0).column->getDataAt(0).toString(); } MaterializeMySQLSyncThread::MaterializeMySQLSyncThread( @@ -160,13 +151,13 @@ MaterializeMySQLSyncThread::MaterializeMySQLSyncThread( query_prefix = "EXTERNAL DDL FROM MySQL(" + backQuoteIfNeed(database_name) + ", " + backQuoteIfNeed(mysql_database_name) + ") "; } -void MaterializeMySQLSyncThread::synchronization(const String & mysql_version) +void MaterializeMySQLSyncThread::synchronization() { setThreadName(MYSQL_BACKGROUND_THREAD_NAME); try { - if (std::optional metadata = prepareSynchronized(mysql_version)) + if (std::optional metadata = prepareSynchronized()) { Stopwatch watch; Buffers buffers(database_name); @@ -217,10 +208,8 @@ void MaterializeMySQLSyncThread::startSynchronization() { try { - const auto & mysql_server_version = checkVariableAndGetVersion(pool.get()); - - background_thread_pool = std::make_unique( - [this, mysql_server_version = mysql_server_version]() { synchronization(mysql_server_version); }); + checkMySQLVariables(pool.get()); + background_thread_pool = std::make_unique([this]() { synchronization(); }); } catch (...) { @@ -324,7 +313,7 @@ static inline UInt32 randomNumber() return dist6(rng); } -std::optional MaterializeMySQLSyncThread::prepareSynchronized(const String & mysql_version) +std::optional MaterializeMySQLSyncThread::prepareSynchronized() { bool opened_transaction = false; mysqlxx::PoolWithFailover::Entry connection; @@ -337,8 +326,7 @@ std::optional MaterializeMySQLSyncThread::prepareSynchroniz opened_transaction = false; MaterializeMetadata metadata( - connection, getDatabase(database_name).getMetadataPath() + "/.metadata", - mysql_database_name, opened_transaction, mysql_version); + connection, getDatabase(database_name).getMetadataPath() + "/.metadata", mysql_database_name, opened_transaction); if (!metadata.need_dumping_tables.empty()) { diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.h b/src/Databases/MySQL/MaterializeMySQLSyncThread.h index 323ae5beb80..54f148026ad 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.h +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.h @@ -95,11 +95,11 @@ private: BufferAndSortingColumnsPtr getTableDataBuffer(const String & table, const Context & context); }; - void synchronization(const String & mysql_version); + void synchronization(); bool isCancelled() { return sync_quit.load(std::memory_order_relaxed); } - std::optional prepareSynchronized(const String & mysql_version); + std::optional prepareSynchronized(); void flushBuffersData(Buffers & buffers, MaterializeMetadata & metadata); diff --git a/src/Formats/MySQLBlockInputStream.cpp b/src/Formats/MySQLBlockInputStream.cpp index 73def337240..d38713879e7 100644 --- a/src/Formats/MySQLBlockInputStream.cpp +++ b/src/Formats/MySQLBlockInputStream.cpp @@ -37,12 +37,14 @@ MySQLBlockInputStream::MySQLBlockInputStream( const std::string & query_str, const Block & sample_block, const UInt64 max_block_size_, - const bool auto_close_) + const bool auto_close_, + const bool fetch_by_name_) : connection{std::make_unique(entry, query_str)} , max_block_size{max_block_size_} , auto_close{auto_close_} + , fetch_by_name(fetch_by_name_) { - if (sample_block.columns() != connection->result.getNumFields()) + if (!fetch_by_name && sample_block.columns() != connection->result.getNumFields()) throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " + toString(sample_block.columns()) + " expected", ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; @@ -132,27 +134,51 @@ Block MySQLBlockInputStream::readImpl() for (const auto i : ext::range(0, columns.size())) columns[i] = description.sample_block.getByPosition(i).column->cloneEmpty(); + if (unlikely(position_mapping.size() != description.sample_block.columns())) + { + position_mapping.resize(description.sample_block.columns()); + + if (!fetch_by_name) + { + for (const auto idx : ext::range(0, row.size())) + position_mapping[idx] = idx; + } + else + { + for (const auto idx : ext::range(0, row.size())) + { + const auto & field_name = row.getFieldName(idx); + if (description.sample_block.has(field_name)) + { + const auto & position = description.sample_block.getPositionByName(field_name); + position_mapping[position] = idx; + } + } + } + } + size_t num_rows = 0; while (row) { - for (const auto idx : ext::range(0, row.size())) + for (size_t index = 0; index < position_mapping.size(); ++index) { - const auto value = row[idx]; - const auto & sample = description.sample_block.getByPosition(idx); + const auto value = row[position_mapping[index]]; + const auto & sample = description.sample_block.getByPosition(index); + if (!value.isNull()) { - if (description.types[idx].second) + if (description.types[index].second) { - ColumnNullable & column_nullable = assert_cast(*columns[idx]); + ColumnNullable & column_nullable = assert_cast(*columns[index]); const auto & data_type = assert_cast(*sample.type); - insertValue(*data_type.getNestedType(), column_nullable.getNestedColumn(), description.types[idx].first, value); + insertValue(*data_type.getNestedType(), column_nullable.getNestedColumn(), description.types[index].first, value); column_nullable.getNullMapData().emplace_back(0); } else - insertValue(*sample.type, *columns[idx], description.types[idx].first, value); + insertValue(*sample.type, *columns[index], description.types[index].first, value); } else - insertDefaultValue(*columns[idx], *sample.column); + insertDefaultValue(*columns[index], *sample.column); } ++num_rows; @@ -167,9 +193,11 @@ Block MySQLBlockInputStream::readImpl() MySQLBlockInputStream::MySQLBlockInputStream( const Block & sample_block_, UInt64 max_block_size_, - bool auto_close_) + bool auto_close_, + bool fetch_by_name_) : max_block_size(max_block_size_) , auto_close(auto_close_) + , fetch_by_name(fetch_by_name_) { description.init(sample_block_); } @@ -179,8 +207,9 @@ MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( const std::string & query_str_, const Block & sample_block_, const UInt64 max_block_size_, - const bool auto_close_) - : MySQLBlockInputStream(sample_block_, max_block_size_, auto_close_) + const bool auto_close_, + const bool fetch_by_name_) + : MySQLBlockInputStream(sample_block_, max_block_size_, auto_close_, fetch_by_name_) , pool(pool_) , query_str(query_str_) { @@ -189,7 +218,7 @@ MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( void MySQLLazyBlockInputStream::readPrefix() { connection = std::make_unique(pool.get(), query_str); - if (description.sample_block.columns() != connection->result.getNumFields()) + if (!fetch_by_name && description.sample_block.columns() != connection->result.getNumFields()) throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " + toString(description.sample_block.columns()) + " expected", ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; diff --git a/src/Formats/MySQLBlockInputStream.h b/src/Formats/MySQLBlockInputStream.h index 2eaeb5b8d59..a516e81ac6c 100644 --- a/src/Formats/MySQLBlockInputStream.h +++ b/src/Formats/MySQLBlockInputStream.h @@ -20,14 +20,15 @@ public: const std::string & query_str, const Block & sample_block, const UInt64 max_block_size_, - const bool auto_close_ = false); + const bool auto_close_ = false, + const bool fetch_by_name_ = false); String getName() const override { return "MySQL"; } Block getHeader() const override { return description.sample_block.cloneEmpty(); } protected: - MySQLBlockInputStream(const Block & sample_block_, UInt64 max_block_size_, bool auto_close_); + MySQLBlockInputStream(const Block & sample_block_, UInt64 max_block_size_, bool auto_close_, bool fetch_by_name_); Block readImpl() override; struct Connection @@ -43,6 +44,8 @@ protected: const UInt64 max_block_size; const bool auto_close; + const bool fetch_by_name; + std::vector position_mapping; ExternalResultDescription description; }; @@ -56,7 +59,8 @@ public: const std::string & query_str_, const Block & sample_block_, const UInt64 max_block_size_, - const bool auto_close_ = false); + const bool auto_close_ = false, + const bool fetch_by_name_ = false); private: void readPrefix() override; From 6701dfe369d892070dcf774ba01c97d30b7af522 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Wed, 25 Nov 2020 13:47:11 +0800 Subject: [PATCH 048/174] ISSUES-16835 add missing name check --- src/Formats/MySQLBlockInputStream.cpp | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/Formats/MySQLBlockInputStream.cpp b/src/Formats/MySQLBlockInputStream.cpp index d38713879e7..f2c8b3b24bc 100644 --- a/src/Formats/MySQLBlockInputStream.cpp +++ b/src/Formats/MySQLBlockInputStream.cpp @@ -12,6 +12,7 @@ # include # include # include +# include # include # include # include "MySQLBlockInputStream.h" @@ -145,6 +146,8 @@ Block MySQLBlockInputStream::readImpl() } else { + Names missing_names = description.sample_block.getNames(); + for (const auto idx : ext::range(0, row.size())) { const auto & field_name = row.getFieldName(idx); @@ -152,8 +155,19 @@ Block MySQLBlockInputStream::readImpl() { const auto & position = description.sample_block.getPositionByName(field_name); position_mapping[position] = idx; + missing_names.erase(missing_names.begin() + position); } } + + if (!missing_names.empty()) + { + WriteBufferFromOwnString exception_message; + writeText(missing_names, exception_message); + + throw Exception("mysqlxx::UseQueryResult must be contain the" + exception_message.str() + " columns.", + ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); + } + } } From 310307fa727b3ae189dd5333599b7eaf59f02899 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 11:18:15 +0300 Subject: [PATCH 049/174] Replace eventfd with pipes --- programs/server/Server.cpp | 19 +++++++------ src/Interpreters/Context.cpp | 2 ++ src/Server/TestKeeperTCPHandler.cpp | 42 +++++++++++------------------ 3 files changed, 28 insertions(+), 35 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index ea27278dc16..2d30453fb81 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -740,7 +740,7 @@ int Server::main(const std::vector & /*args*/) http_params->setTimeout(settings.http_receive_timeout); http_params->setKeepAliveTimeout(keep_alive_timeout); - std::vector> servers; + std::vector servers; std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); @@ -772,7 +772,7 @@ int Server::main(const std::vector & /*args*/) } for (auto & server : servers) - server->start(); + server.start(); size_t already_started_servers = servers.size(); @@ -789,7 +789,7 @@ int Server::main(const std::vector & /*args*/) LOG_DEBUG(log, "Shut down storages."); for (size_t i = 0; i < already_started_servers; ++i) - servers[i]->stop(); + servers[i].stop(); /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. * At this moment, no one could own shared part of Context. @@ -1098,7 +1098,7 @@ int Server::main(const std::vector & /*args*/) global_context->enableNamedSessions(); for (size_t i = already_started_servers; i < servers.size(); ++i) - servers[i]->start(); + servers[i].start(); { String level_str = config().getString("text_log.level", ""); @@ -1131,8 +1131,8 @@ int Server::main(const std::vector & /*args*/) int current_connections = 0; for (size_t i = already_started_servers; i < servers.size(); ++i) { - servers[i]->stop(); - current_connections += servers[i]->currentConnections(); + servers[i].stop(); + current_connections += servers[i].currentConnections(); } if (current_connections) @@ -1151,8 +1151,11 @@ int Server::main(const std::vector & /*args*/) while (sleep_current_ms < sleep_max_ms) { current_connections = 0; - for (auto & server : servers) - current_connections += server.currentConnections(); + for (size_t i = already_started_servers; i < servers.size(); ++i) + { + servers[i].stop(); + current_connections += servers[i].currentConnections(); + } if (!current_connections) break; sleep_current_ms += sleep_one_ms; diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index be60d755d8f..7b6bbdb2b0e 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -445,6 +445,8 @@ struct ContextShared /// Stop trace collector if any trace_collector.reset(); + /// Stop test_keeper storage + test_keeper_storage.reset(); } bool hasTraceCollector() const diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index c5c0d17ecba..882b46fca70 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -10,8 +10,7 @@ #include #include #include - -#include +#include #include namespace DB @@ -26,7 +25,7 @@ namespace ErrorCodes struct SocketInterruptablePollWrapper { int sockfd; - int efd; + PipeFDs pipe; enum class PollStatus { HAS_DATA, @@ -37,32 +36,21 @@ struct SocketInterruptablePollWrapper using InterruptCallback = std::function; - SocketInterruptablePollWrapper(const Poco::Net::StreamSocket & poco_socket_) + explicit SocketInterruptablePollWrapper(const Poco::Net::StreamSocket & poco_socket_) : sockfd(poco_socket_.impl()->sockfd()) - , efd(eventfd(0, EFD_NONBLOCK)) { - if (efd < 0) - throwFromErrno("Cannot create eventfd file descriptor", ErrorCodes::SYSTEM_ERROR); + pipe.setNonBlocking(); } - ~SocketInterruptablePollWrapper() - { - if (efd >= 0) - { - close(efd); - efd = -1; - } - } - - void interruptPoll() + void interruptPoll() const { UInt64 bytes = 1; - int ret = write(efd, &bytes, sizeof(bytes)); + int ret = write(pipe.fds_rw[1], &bytes, sizeof(bytes)); if (ret < 0) - throwFromErrno("Cannot write into eventfd descriptor", ErrorCodes::SYSTEM_ERROR); + throwFromErrno("Cannot write into pipe descriptor", ErrorCodes::SYSTEM_ERROR); } - PollStatus poll(Poco::Timespan& remainingTime) + PollStatus poll(Poco::Timespan remainingTime) { int epollfd = epoll_create(2); @@ -77,10 +65,10 @@ struct SocketInterruptablePollWrapper ::close(epollfd); throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); } - epoll_event efd_event{}; - efd_event.events = EPOLLIN | EPOLLERR; - efd_event.data.fd = efd; - if (epoll_ctl(epollfd, EPOLL_CTL_ADD, efd, &efd_event) < 0) + epoll_event pipe_event{}; + pipe_event.events = EPOLLIN | EPOLLERR; + pipe_event.data.fd = pipe.fds_rw[0]; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pipe.fds_rw[0], &pipe_event) < 0) { ::close(epollfd); throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); @@ -109,11 +97,11 @@ struct SocketInterruptablePollWrapper return PollStatus::ERROR; else if (rc == 0) return PollStatus::TIMEOUT; - else if (evout.data.fd == efd) + else if (evout.data.fd == pipe.fds_rw[0]) { UInt64 bytes; - if (read(efd, &bytes, sizeof(bytes)) < 0) - throwFromErrno("Cannot read from eventfd", ErrorCodes::SYSTEM_ERROR); + if (read(pipe.fds_rw[0], &bytes, sizeof(bytes)) < 0) + throwFromErrno("Cannot read from pipe", ErrorCodes::SYSTEM_ERROR); return PollStatus::INTERRUPTED; } return PollStatus::HAS_DATA; From 8acbf75dc9d487e0ee3b693bdaa29e71471695fd Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 11:56:40 +0300 Subject: [PATCH 050/174] Fix segfault --- src/Server/TestKeeperTCPHandler.cpp | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 882b46fca70..05bb1123aff 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -26,6 +26,7 @@ struct SocketInterruptablePollWrapper { int sockfd; PipeFDs pipe; + enum class PollStatus { HAS_DATA, @@ -42,12 +43,9 @@ struct SocketInterruptablePollWrapper pipe.setNonBlocking(); } - void interruptPoll() const + int getInterruptFD() const { - UInt64 bytes = 1; - int ret = write(pipe.fds_rw[1], &bytes, sizeof(bytes)); - if (ret < 0) - throwFromErrno("Cannot write into pipe descriptor", ErrorCodes::SYSTEM_ERROR); + return pipe.fds_rw[1]; } PollStatus poll(Poco::Timespan remainingTime) @@ -282,22 +280,25 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); request->xid = xid; request->readImpl(*in); + int interrupt_fd = poll_wrapper->getInterruptFD(); if (opnum != Coordination::OpNum::Close) { auto promise = std::make_shared>(); - zkutil::ResponseCallback callback = [this, promise] (const Coordination::ZooKeeperResponsePtr & response) + zkutil::ResponseCallback callback = [interrupt_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); - poll_wrapper->interruptPoll(); + UInt64 bytes = 1; + write(interrupt_fd, &bytes, sizeof(bytes)); }; if (request->has_watch) { auto watch_promise = std::make_shared>(); - zkutil::ResponseCallback watch_callback = [this, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) + zkutil::ResponseCallback watch_callback = [interrupt_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) { watch_promise->set_value(response); - poll_wrapper->interruptPoll(); + UInt64 bytes = 1; + write(interrupt_fd, &bytes, sizeof(bytes)); }; test_keeper_storage->putRequest(request, session_id, callback, watch_callback); responses.push(promise->get_future()); From d277db52fd7b6f343d01324f8b5ec76d67359049 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 15:14:31 +0300 Subject: [PATCH 051/174] Maybe unused --- src/Server/TestKeeperTCPHandler.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 05bb1123aff..0dd0f9b2f45 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -288,7 +288,7 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() { promise->set_value(response); UInt64 bytes = 1; - write(interrupt_fd, &bytes, sizeof(bytes)); + [[maybe_unused]] int result = write(interrupt_fd, &bytes, sizeof(bytes)); }; if (request->has_watch) @@ -298,7 +298,7 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() { watch_promise->set_value(response); UInt64 bytes = 1; - write(interrupt_fd, &bytes, sizeof(bytes)); + [[maybe_unused]] int result = write(interrupt_fd, &bytes, sizeof(bytes)); }; test_keeper_storage->putRequest(request, session_id, callback, watch_callback); responses.push(promise->get_future()); From f3555ad2d4c39d39464789e4a612ec4fd3fac85e Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 15:36:46 +0300 Subject: [PATCH 052/174] Fix build on non-linux platforms --- src/Server/TestKeeperTCPHandler.cpp | 51 +++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 0dd0f9b2f45..09b04e81d42 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -11,7 +11,12 @@ #include #include #include -#include + +#ifdef POCO_HAVE_FD_EPOLL + #include +#else + #include +#endif namespace DB { @@ -48,8 +53,10 @@ struct SocketInterruptablePollWrapper return pipe.fds_rw[1]; } - PollStatus poll(Poco::Timespan remainingTime) + PollStatus poll(Poco::Timespan remaining_time) { + +#if defined(POCO_HAVE_FD_EPOLL) int epollfd = epoll_create(2); if (epollfd < 0) @@ -77,25 +84,55 @@ struct SocketInterruptablePollWrapper do { Poco::Timestamp start; - rc = epoll_wait(epollfd, &evout, 1, remainingTime.totalMilliseconds()); + rc = epoll_wait(epollfd, &evout, 1, remaining_time.totalMilliseconds()); if (rc < 0 && errno == EINTR) { Poco::Timestamp end; Poco::Timespan waited = end - start; - if (waited < remainingTime) - remainingTime -= waited; + if (waited < remaining_time) + remaining_time -= waited; else - remainingTime = 0; + remaining_time = 0; } } while (rc < 0 && errno == EINTR); ::close(epollfd); + int out_fd = evout.data.fd; +#else + pollfd poll_buf[2]; + poll_buf[0].fd = sockfd; + poll_buf[0].events = POLLIN; + poll_buf[1].fd = pipe.fds_rw[0]; + poll_buf[1].events = POLLIN; + + int rc; + do + { + Poco::Timestamp start; + rc = ::poll(poll_buf, 2, remaining_time.totalMilliseconds()); + if (rc < 0 && errno == POCO_EINTR) + { + Poco::Timestamp end; + Poco::Timespan waited = end - start; + if (waited < remaining_time) + remaining_time -= waited; + else + remaining_time = 0; + } + } + while (rc < 0 && errno == POCO_EINTR); + int out_fd = -1; + if (poll_buf[0].revents & POLLIN) + out_fd = sockfd; + else if (poll_buf[1].revents & POLLIN) + out_fd = pipe.fds_rw[0]; +#endif if (rc < 0) return PollStatus::ERROR; else if (rc == 0) return PollStatus::TIMEOUT; - else if (evout.data.fd == pipe.fds_rw[0]) + else if (out_fd == pipe.fds_rw[0]) { UInt64 bytes; if (read(pipe.fds_rw[0], &bytes, sizeof(bytes)) < 0) From fb86eaf6fcf2d51b07d9fb146149e1d5eb024c92 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 16:19:09 +0300 Subject: [PATCH 053/174] Better errors and better config --- programs/server/Server.cpp | 2 +- src/Common/ZooKeeper/TestKeeperStorage.cpp | 51 ++++++++++++---------- src/Server/TestKeeperTCPHandler.cpp | 16 ++++--- tests/config/config.d/test_keeper_port.xml | 6 ++- 4 files changed, 43 insertions(+), 32 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 2d30453fb81..d22d077f6c0 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -755,7 +755,7 @@ int Server::main(const std::vector & /*args*/) for (const auto & listen_host : listen_hosts) { /// TCP TestKeeper - createServer(listen_host, "test_keeper_tcp_port", listen_try, [&](UInt16 port) + createServer(listen_host, "test_keeper_server.tcp_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; auto address = socketBindListen(socket, listen_host, port); diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 299a6606058..a2f7b1a8b2d 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -10,10 +10,14 @@ namespace DB { - namespace ErrorCodes - { - extern const int LOGICAL_ERROR; - } + +namespace ErrorCodes +{ + extern const int LOGICAL_ERROR; + extern const int TIMEOUT_EXCEEDED; + extern const int BAD_ARGUMENTS; +} + } namespace zkutil @@ -360,7 +364,7 @@ struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest { auto path_prefix = request.path; if (path_prefix.empty()) - throw Coordination::Exception("Logical error: path cannot be empty", Coordination::Error::ZSESSIONEXPIRED); + throw DB::Exception("Logical error: path cannot be empty", ErrorCodes::LOGICAL_ERROR); if (path_prefix.back() != '/') path_prefix += '/'; @@ -417,26 +421,27 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest Coordination::ZooKeeperMultiRequest & request = dynamic_cast(*zk_request); concrete_requests.reserve(request.requests.size()); - for (const auto & sub_zk_request : request.requests) + for (const auto & sub_request : request.requests) { - if (dynamic_cast(sub_zk_request.get())) + auto sub_zk_request = dynamic_pointer_cast(sub_request); + if (sub_zk_request->getOpNum() == Coordination::OpNum::Create) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); + concrete_requests.push_back(std::make_shared(sub_zk_request)); } - else if (dynamic_cast(sub_zk_request.get())) + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Remove) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); + concrete_requests.push_back(std::make_shared(sub_zk_request)); } - else if (dynamic_cast(sub_zk_request.get())) + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Set) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); + concrete_requests.push_back(std::make_shared(sub_zk_request)); } - else if (dynamic_cast(sub_zk_request.get())) + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Check) { - concrete_requests.push_back(std::make_shared(dynamic_pointer_cast(sub_zk_request))); + concrete_requests.push_back(std::make_shared(sub_zk_request)); } else - throw Coordination::Exception("Illegal command as part of multi ZooKeeper request", Coordination::Error::ZBADARGUMENTS); + throw DB::Exception(ErrorCodes::BAD_ARGUMENTS, "Illegal command as part of multi ZooKeeper request {}", sub_zk_request->getOpNum()); } } @@ -490,7 +495,7 @@ struct TestKeeperStorageCloseRequest final : public TestKeeperStorageRequest using TestKeeperStorageRequest::TestKeeperStorageRequest; std::pair process(TestKeeperStorage::Container &, TestKeeperStorage::Ephemerals &, int64_t, int64_t) const override { - throw Coordination::Exception("Called process on close request", Coordination::Error::ZRUNTIMEINCONSISTENCY); + throw DB::Exception("Called process on close request", ErrorCodes::LOGICAL_ERROR); } }; @@ -512,7 +517,7 @@ void TestKeeperStorage::processingThread() break; auto zk_request = info.request->zk_request; - if (dynamic_cast(zk_request.get())) + if (zk_request->getOpNum() == Coordination::OpNum::Close) { auto it = ephemerals.find(info.session_id); if (it != ephemerals.end()) @@ -533,13 +538,13 @@ void TestKeeperStorage::processingThread() { if (response->error == Coordination::Error::ZOK) { - auto & watches_type = dynamic_cast(zk_request.get()) + auto & watches_type = zk_request->getOpNum() == Coordination::OpNum::List ? list_watches : watches; watches_type[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); } - else if (response->error == Coordination::Error::ZNONODE && dynamic_cast(zk_request.get())) + else if (response->error == Coordination::Error::ZNONODE && zk_request->getOpNum() == Coordination::OpNum::Exists) { watches[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); } @@ -657,7 +662,7 @@ public: { auto it = op_num_to_request.find(zk_request->getOpNum()); if (it == op_num_to_request.end()) - throw Coordination::Exception("Unknown operation type " + toString(zk_request->getOpNum()), Coordination::Error::ZBADARGUMENTS); + throw DB::Exception("Unknown operation type " + toString(zk_request->getOpNum()), ErrorCodes::LOGICAL_ERROR); return it->second(zk_request); } @@ -705,7 +710,7 @@ void TestKeeperStorage::putCloseRequest(const Coordination::ZooKeeperRequestPtr request_info.session_id = session_id; std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) - throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); } @@ -719,7 +724,7 @@ void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & req request_info.response_callback = callback; std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) - throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); } void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback) @@ -735,7 +740,7 @@ void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & req std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) - throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::LOGICAL_ERROR); + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); } TestKeeperStorage::~TestKeeperStorage() diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 09b04e81d42..f266eae2093 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -11,6 +11,8 @@ #include #include #include +#include + #ifdef POCO_HAVE_FD_EPOLL #include @@ -23,8 +25,8 @@ namespace DB namespace ErrorCodes { - extern const int LOGICAL_ERROR; extern const int SYSTEM_ERROR; + extern const int UNEXPECTED_PACKET_FROM_CLIENT; } struct SocketInterruptablePollWrapper @@ -149,8 +151,8 @@ TestKeeperTCPHandler::TestKeeperTCPHandler(IServer & server_, const Poco::Net::S , log(&Poco::Logger::get("TestKeeperTCPHandler")) , global_context(server.context()) , test_keeper_storage(global_context.getTestKeeperStorage()) - , operation_timeout(0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000) - , session_timeout(0, Coordination::DEFAULT_SESSION_TIMEOUT_MS * 1000) + , operation_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.operation_timeout_ms", Coordination::DEFAULT_OPERATION_TIMEOUT_MS) * 1000) + , session_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.session_timeout_ms", Coordination::DEFAULT_SESSION_TIMEOUT_MS) * 1000) , session_id(test_keeper_storage->getSessionID()) , poll_wrapper(std::make_unique(socket_)) { @@ -183,23 +185,23 @@ void TestKeeperTCPHandler::receiveHandshake() Coordination::read(handshake_length, *in); if (handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH) - throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::LOGICAL_ERROR); + throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(protocol_version, *in); if (protocol_version != Coordination::ZOOKEEPER_PROTOCOL_VERSION) - throw Exception("Unexpected protocol version: " + toString(protocol_version), ErrorCodes::LOGICAL_ERROR); + throw Exception("Unexpected protocol version: " + toString(protocol_version), ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(last_zxid_seen, *in); if (last_zxid_seen != 0) - throw Exception("Non zero last_zxid_seen is not supported", ErrorCodes::LOGICAL_ERROR); + throw Exception("Non zero last_zxid_seen is not supported", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(timeout, *in); Coordination::read(previous_session_id, *in); if (previous_session_id != 0) - throw Exception("Non zero previous session id is not supported", ErrorCodes::LOGICAL_ERROR); + throw Exception("Non zero previous session id is not supported", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(passwd, *in); } diff --git a/tests/config/config.d/test_keeper_port.xml b/tests/config/config.d/test_keeper_port.xml index 23b744630f2..79e993b41f7 100644 --- a/tests/config/config.d/test_keeper_port.xml +++ b/tests/config/config.d/test_keeper_port.xml @@ -1,3 +1,7 @@ - 9181 + + 9181 + 10000 + 30000 + From fd67864e153263b20fe01db9805b76006735a58a Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 17:34:13 +0300 Subject: [PATCH 054/174] Remove double space --- src/Server/TestKeeperTCPHandler.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index f266eae2093..a178b5e43e6 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -152,7 +152,7 @@ TestKeeperTCPHandler::TestKeeperTCPHandler(IServer & server_, const Poco::Net::S , global_context(server.context()) , test_keeper_storage(global_context.getTestKeeperStorage()) , operation_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.operation_timeout_ms", Coordination::DEFAULT_OPERATION_TIMEOUT_MS) * 1000) - , session_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.session_timeout_ms", Coordination::DEFAULT_SESSION_TIMEOUT_MS) * 1000) + , session_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.session_timeout_ms", Coordination::DEFAULT_SESSION_TIMEOUT_MS) * 1000) , session_id(test_keeper_storage->getSessionID()) , poll_wrapper(std::make_unique(socket_)) { From 9e5253c9fa4820607a5146baca72f33516d4aed9 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Wed, 25 Nov 2020 23:52:30 +0800 Subject: [PATCH 055/174] ISSUES-16835 try fix test --- src/Formats/MySQLBlockInputStream.cpp | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/Formats/MySQLBlockInputStream.cpp b/src/Formats/MySQLBlockInputStream.cpp index f2c8b3b24bc..6cbec0650af 100644 --- a/src/Formats/MySQLBlockInputStream.cpp +++ b/src/Formats/MySQLBlockInputStream.cpp @@ -146,7 +146,8 @@ Block MySQLBlockInputStream::readImpl() } else { - Names missing_names = description.sample_block.getNames(); + const auto & sample_names = description.sample_block.getNames(); + std::unordered_set missing_names(sample_names.begin(), sample_names.end()); for (const auto idx : ext::range(0, row.size())) { @@ -155,19 +156,23 @@ Block MySQLBlockInputStream::readImpl() { const auto & position = description.sample_block.getPositionByName(field_name); position_mapping[position] = idx; - missing_names.erase(missing_names.begin() + position); + missing_names.erase(field_name); } } if (!missing_names.empty()) { WriteBufferFromOwnString exception_message; - writeText(missing_names, exception_message); + for (auto iter = missing_names.begin(); iter != missing_names.end(); ++iter) + { + if (iter != missing_names.begin()) + exception_message << ", "; + exception_message << *iter; + } throw Exception("mysqlxx::UseQueryResult must be contain the" + exception_message.str() + " columns.", ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); } - } } From 77651dd239f37c5dac85d5c4074dae07ab36cab7 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 25 Nov 2020 23:42:23 +0300 Subject: [PATCH 056/174] Don't epoll_create each poll --- src/Server/TestKeeperTCPHandler.cpp | 59 +++++++++++++++++------------ 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index a178b5e43e6..fdff814e6e0 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -13,7 +13,6 @@ #include #include - #ifdef POCO_HAVE_FD_EPOLL #include #else @@ -34,6 +33,12 @@ struct SocketInterruptablePollWrapper int sockfd; PipeFDs pipe; +#if defined(POCO_HAVE_FD_EPOLL) + int epollfd; + epoll_event socket_event{}; + epoll_event pipe_event{}; +#endif + enum class PollStatus { HAS_DATA, @@ -48,6 +53,27 @@ struct SocketInterruptablePollWrapper : sockfd(poco_socket_.impl()->sockfd()) { pipe.setNonBlocking(); + +#if defined(POCO_HAVE_FD_EPOLL) + epollfd = epoll_create(2); + if (epollfd < 0) + throwFromErrno("Cannot epoll_create", ErrorCodes::SYSTEM_ERROR); + + socket_event.events = EPOLLIN | EPOLLERR; + socket_event.data.fd = sockfd; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &socket_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } + pipe_event.events = EPOLLIN | EPOLLERR; + pipe_event.data.fd = pipe.fds_rw[0]; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pipe.fds_rw[0], &pipe_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } +#endif } int getInterruptFD() const @@ -57,30 +83,7 @@ struct SocketInterruptablePollWrapper PollStatus poll(Poco::Timespan remaining_time) { - #if defined(POCO_HAVE_FD_EPOLL) - int epollfd = epoll_create(2); - - if (epollfd < 0) - throwFromErrno("Cannot epoll_create", ErrorCodes::SYSTEM_ERROR); - - epoll_event socket_event{}; - socket_event.events = EPOLLIN | EPOLLERR; - socket_event.data.fd = sockfd; - if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &socket_event) < 0) - { - ::close(epollfd); - throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); - } - epoll_event pipe_event{}; - pipe_event.events = EPOLLIN | EPOLLERR; - pipe_event.data.fd = pipe.fds_rw[0]; - if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pipe.fds_rw[0], &pipe_event) < 0) - { - ::close(epollfd); - throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); - } - int rc; epoll_event evout{}; do @@ -99,7 +102,6 @@ struct SocketInterruptablePollWrapper } while (rc < 0 && errno == EINTR); - ::close(epollfd); int out_fd = evout.data.fd; #else pollfd poll_buf[2]; @@ -143,6 +145,13 @@ struct SocketInterruptablePollWrapper } return PollStatus::HAS_DATA; } + +#if defined(POCO_HAVE_FD_EPOLL) + ~SocketInterruptablePollWrapper() + { + ::close(epollfd); + } +#endif }; TestKeeperTCPHandler::TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) From 6f6476ad958c74617bc2f17818ad54078c08a89c Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 26 Nov 2020 00:00:51 +0300 Subject: [PATCH 057/174] Remove complex code on session timeout --- src/Server/TestKeeperTCPHandler.cpp | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index fdff814e6e0..8964217a802 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -253,9 +253,7 @@ void TestKeeperTCPHandler::runImpl() { using namespace std::chrono_literals; - Poco::Timespan poll_wait = responses.empty() ? session_timeout.totalMicroseconds() - session_stopwatch.elapsedMicroseconds() : session_timeout; - - auto state = poll_wrapper->poll(poll_wait); + auto state = poll_wrapper->poll(session_timeout); if (state == SocketInterruptablePollWrapper::PollStatus::HAS_DATA) { auto received_op = receiveRequest(); From 0b079cdeb87ff106dbdb3f282ea6d1798f43676c Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 26 Nov 2020 11:59:23 +0300 Subject: [PATCH 058/174] Better polling --- src/Server/TestKeeperTCPHandler.cpp | 104 ++++++++++++++++++---------- 1 file changed, 68 insertions(+), 36 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 8964217a802..5f36decad12 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -28,6 +28,9 @@ namespace ErrorCodes extern const int UNEXPECTED_PACKET_FROM_CLIENT; } +static constexpr UInt8 RESPONSE_BYTE = 1; +static constexpr UInt8 WATCH_RESPONSE_BYTE = 2; + struct SocketInterruptablePollWrapper { int sockfd; @@ -39,13 +42,12 @@ struct SocketInterruptablePollWrapper epoll_event pipe_event{}; #endif - enum class PollStatus - { - HAS_DATA, - TIMEOUT, - INTERRUPTED, - ERROR, - }; + using PollStatus = size_t; + static constexpr PollStatus TIMEOUT = 0x0; + static constexpr PollStatus HAS_REQUEST = 0x1; + static constexpr PollStatus HAS_RESPONSE = 0x2; + static constexpr PollStatus HAS_WATCH_RESPONSE = 0x4; + static constexpr PollStatus ERROR = 0x8; using InterruptCallback = std::function; @@ -76,20 +78,21 @@ struct SocketInterruptablePollWrapper #endif } - int getInterruptFD() const + int getResponseFD() const { return pipe.fds_rw[1]; } PollStatus poll(Poco::Timespan remaining_time) { + std::array outputs = {-1, -1}; #if defined(POCO_HAVE_FD_EPOLL) int rc; - epoll_event evout{}; + epoll_event evout[2]; do { Poco::Timestamp start; - rc = epoll_wait(epollfd, &evout, 1, remaining_time.totalMilliseconds()); + rc = epoll_wait(epollfd, evout, 2, remaining_time.totalMilliseconds()); if (rc < 0 && errno == EINTR) { Poco::Timestamp end; @@ -102,7 +105,10 @@ struct SocketInterruptablePollWrapper } while (rc < 0 && errno == EINTR); - int out_fd = evout.data.fd; + if (rc >= 1 && evout[0].events & EPOLLIN) + outputs[0] = evout[0].data.fd; + if (rc == 2 && evout[1].events & EPOLLIN) + outputs[1] = evout[1].data.fd; #else pollfd poll_buf[2]; poll_buf[0].fd = sockfd; @@ -126,24 +132,46 @@ struct SocketInterruptablePollWrapper } } while (rc < 0 && errno == POCO_EINTR); - int out_fd = -1; - if (poll_buf[0].revents & POLLIN) - out_fd = sockfd; - else if (poll_buf[1].revents & POLLIN) - out_fd = pipe.fds_rw[0]; + if (rc >= 1 && poll_buf[0].revents & POLLIN) + outputs[0] = sockfd; + if (rc == 2 && poll_buf[1].revents & POLLIN) + outputs[1] = pipe.fds_rw[0]; #endif + + PollStatus result = TIMEOUT; if (rc < 0) - return PollStatus::ERROR; - else if (rc == 0) - return PollStatus::TIMEOUT; - else if (out_fd == pipe.fds_rw[0]) { - UInt64 bytes; - if (read(pipe.fds_rw[0], &bytes, sizeof(bytes)) < 0) - throwFromErrno("Cannot read from pipe", ErrorCodes::SYSTEM_ERROR); - return PollStatus::INTERRUPTED; + return ERROR; } - return PollStatus::HAS_DATA; + else if (rc == 0) + { + return result; + } + else + { + for (size_t i = 0; i < outputs.size(); ++i) + { + int fd = outputs[i]; + if (fd != -1) + { + if (fd == sockfd) + result |= HAS_REQUEST; + else + { + UInt8 byte; + if (read(pipe.fds_rw[0], &byte, sizeof(byte)) < 0) + throwFromErrno("Cannot read from pipe", ErrorCodes::SYSTEM_ERROR); + if (byte == WATCH_RESPONSE_BYTE) + result |= HAS_WATCH_RESPONSE; + else if (byte == RESPONSE_BYTE) + result |= HAS_RESPONSE; + else + throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + } + } + } + } + return result; } #if defined(POCO_HAVE_FD_EPOLL) @@ -254,7 +282,7 @@ void TestKeeperTCPHandler::runImpl() using namespace std::chrono_literals; auto state = poll_wrapper->poll(session_timeout); - if (state == SocketInterruptablePollWrapper::PollStatus::HAS_DATA) + if (state & SocketInterruptablePollWrapper::HAS_REQUEST) { auto received_op = receiveRequest(); if (received_op == Coordination::OpNum::Close) @@ -267,17 +295,22 @@ void TestKeeperTCPHandler::runImpl() session_stopwatch.restart(); } } - else if (state == SocketInterruptablePollWrapper::PollStatus::INTERRUPTED) + + if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) { while (!responses.empty()) { - if (responses.front().wait_for(0ms) != std::future_status::ready) + if (responses.front().wait_for(0s) != std::future_status::ready) break; auto response = responses.front().get(); response->write(*out); responses.pop(); } + } + + if (state & SocketInterruptablePollWrapper::HAS_WATCH_RESPONSE) + { for (auto it = watch_responses.begin(); it != watch_responses.end();) { if (it->wait_for(0s) == std::future_status::ready) @@ -293,7 +326,8 @@ void TestKeeperTCPHandler::runImpl() } } } - else if (state == SocketInterruptablePollWrapper::PollStatus::ERROR) + + if (state == SocketInterruptablePollWrapper::ERROR) { throw Exception("Exception happened while reading from socket", ErrorCodes::SYSTEM_ERROR); } @@ -326,25 +360,23 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); request->xid = xid; request->readImpl(*in); - int interrupt_fd = poll_wrapper->getInterruptFD(); + int response_fd = poll_wrapper->getResponseFD(); if (opnum != Coordination::OpNum::Close) { auto promise = std::make_shared>(); - zkutil::ResponseCallback callback = [interrupt_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) + zkutil::ResponseCallback callback = [response_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) { promise->set_value(response); - UInt64 bytes = 1; - [[maybe_unused]] int result = write(interrupt_fd, &bytes, sizeof(bytes)); + [[maybe_unused]] int result = write(response_fd, &RESPONSE_BYTE, sizeof(RESPONSE_BYTE)); }; if (request->has_watch) { auto watch_promise = std::make_shared>(); - zkutil::ResponseCallback watch_callback = [interrupt_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) + zkutil::ResponseCallback watch_callback = [response_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) { watch_promise->set_value(response); - UInt64 bytes = 1; - [[maybe_unused]] int result = write(interrupt_fd, &bytes, sizeof(bytes)); + [[maybe_unused]] int result = write(response_fd, &WATCH_RESPONSE_BYTE, sizeof(WATCH_RESPONSE_BYTE)); }; test_keeper_storage->putRequest(request, session_id, callback, watch_callback); responses.push(promise->get_future()); From fc0bca1ff25b699cb3561ab45e801e357ee8bfdf Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 26 Nov 2020 15:26:41 +0300 Subject: [PATCH 059/174] Fix bug with buffered read --- src/Server/TestKeeperTCPHandler.cpp | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 5f36decad12..b9bf549fa7b 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -89,6 +89,7 @@ struct SocketInterruptablePollWrapper #if defined(POCO_HAVE_FD_EPOLL) int rc; epoll_event evout[2]; + memset(evout, 0, sizeof(evout)); do { Poco::Timestamp start; @@ -284,16 +285,20 @@ void TestKeeperTCPHandler::runImpl() auto state = poll_wrapper->poll(session_timeout); if (state & SocketInterruptablePollWrapper::HAS_REQUEST) { - auto received_op = receiveRequest(); - if (received_op == Coordination::OpNum::Close) + do { - LOG_DEBUG(log, "Received close request for session #{}", session_id); - break; - } - else if (received_op == Coordination::OpNum::Heartbeat) - { - session_stopwatch.restart(); - } + auto received_op = receiveRequest(); + if (received_op == Coordination::OpNum::Close) + { + LOG_DEBUG(log, "Received close request for session #{}", session_id); + break; + } + else if (received_op == Coordination::OpNum::Heartbeat) + { + LOG_TRACE(log, "Received heartbeat for session #{}", session_id); + session_stopwatch.restart(); + } + } while (in->available()); } if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) @@ -360,7 +365,7 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); request->xid = xid; request->readImpl(*in); - int response_fd = poll_wrapper->getResponseFD(); + int response_fd = poll_wrapper->getResponseFD(); if (opnum != Coordination::OpNum::Close) { auto promise = std::make_shared>(); From 15a76b476378232bb1f9c72930fbe07decfa2b85 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 26 Nov 2020 15:28:54 +0300 Subject: [PATCH 060/174] Fix clang-tidy --- src/Server/TestKeeperTCPHandler.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index b9bf549fa7b..f1ce8ab29ee 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -150,9 +150,8 @@ struct SocketInterruptablePollWrapper } else { - for (size_t i = 0; i < outputs.size(); ++i) + for (auto fd : outputs) { - int fd = outputs[i]; if (fd != -1) { if (fd == sockfd) From 0c6ce276b2b979822308f958cb365bd2cd156f40 Mon Sep 17 00:00:00 2001 From: alesapin Date: Thu, 26 Nov 2020 17:57:32 +0300 Subject: [PATCH 061/174] Add dead watches cleaner --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 52 +++++++++++++++++----- src/Common/ZooKeeper/TestKeeperStorage.h | 11 ++++- 2 files changed, 51 insertions(+), 12 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index a2f7b1a8b2d..146eb374e28 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -49,9 +49,9 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & auto it = watches.find(watch_response.path); if (it != watches.end()) { - for (auto & callback : it->second) - if (callback) - callback(std::make_shared(watch_response)); + for (auto & watcher : it->second) + if (watcher.watch_callback) + watcher.watch_callback(std::make_shared(watch_response)); watches.erase(it); } @@ -64,9 +64,9 @@ static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & it = list_watches.find(watch_list_response.path); if (it != list_watches.end()) { - for (auto & callback : it->second) - if (callback) - callback(std::make_shared(watch_list_response)); + for (auto & watcher : it->second) + if (watcher.watch_callback) + watcher.watch_callback(std::make_shared(watch_list_response)); list_watches.erase(it); } @@ -529,6 +529,7 @@ void TestKeeperStorage::processingThread() } ephemerals.erase(it); } + clearDeadWatches(info.session_id); } else { @@ -542,11 +543,13 @@ void TestKeeperStorage::processingThread() ? list_watches : watches; - watches_type[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + watches_type[zk_request->getPath()].emplace_back(Watcher{info.session_id, info.watch_callback}); + sessions_and_watchers[info.session_id].emplace(zk_request->getPath()); } else if (response->error == Coordination::Error::ZNONODE && zk_request->getOpNum() == Coordination::OpNum::Exists) { - watches[zk_request->getPath()].emplace_back(std::move(info.watch_callback)); + watches[zk_request->getPath()].emplace_back(Watcher{info.session_id, info.watch_callback}); + sessions_and_watchers[info.session_id].emplace(zk_request->getPath()); } else { @@ -601,13 +604,13 @@ void TestKeeperStorage::finalize() response.state = Coordination::EXPIRED_SESSION; response.error = Coordination::Error::ZSESSIONEXPIRED; - for (auto & callback : watch_pair.second) + for (auto & watcher : watch_pair.second) { - if (callback) + if (watcher.watch_callback) { try { - callback(std::make_shared(response)); + watcher.watch_callback(std::make_shared(response)); } catch (...) { @@ -622,6 +625,7 @@ void TestKeeperStorage::finalize() for (auto & path_watch : list_watches) finish_watch(path_watch); list_watches.clear(); + sessions_and_watchers.clear(); } RequestInfo info; while (requests_queue.tryPop(info)) @@ -755,4 +759,30 @@ TestKeeperStorage::~TestKeeperStorage() } } +void TestKeeperStorage::clearDeadWatches(int64_t session_id) +{ + auto watches_it = sessions_and_watchers.find(session_id); + if (watches_it != sessions_and_watchers.end()) + { + for (const auto & watch_path : watches_it->second) + { + auto watch = watches.find(watch_path); + if (watch != watches.end()) + { + auto & watches_for_path = watch->second; + for (auto w_it = watches_for_path.begin(); w_it != watches_for_path.end();) + { + if (w_it->session_id == session_id) + w_it = watches_for_path.erase(w_it); + else + ++w_it; + } + if (watches_for_path.empty()) + watches.erase(watch); + } + } + sessions_and_watchers.erase(watches_it); + } +} + } diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index ee8bc1ae713..80b871554d2 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -34,14 +34,22 @@ public: int32_t seq_num = 0; }; + struct Watcher + { + int64_t session_id; + ResponseCallback watch_callback; + }; + using Container = std::map; using Ephemerals = std::unordered_map>; + using SessionAndWatcher = std::unordered_map>; - using WatchCallbacks = std::vector; + using WatchCallbacks = std::vector; using Watches = std::map; Container container; Ephemerals ephemerals; + SessionAndWatcher sessions_and_watchers; std::atomic zxid{0}; std::atomic shutdown{false}; @@ -69,6 +77,7 @@ public: ThreadFromGlobalPool processing_thread; void processingThread(); + void clearDeadWatches(int64_t session_id); public: using AsyncResponse = std::future; From 89c292f5b786093b61963e2db7b51d94745c8a8f Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 27 Nov 2020 13:12:11 +0300 Subject: [PATCH 062/174] Fix response pipe read --- base/daemon/BaseDaemon.cpp | 2 +- src/Common/PipeFDs.cpp | 17 +++++++++++++++- src/Common/PipeFDs.h | 7 ++++++- src/Common/TraceCollector.cpp | 2 +- src/Server/TestKeeperTCPHandler.cpp | 30 ++++++++++++++++++----------- 5 files changed, 43 insertions(+), 15 deletions(-) diff --git a/base/daemon/BaseDaemon.cpp b/base/daemon/BaseDaemon.cpp index 22455d09cf2..65840eeeb2e 100644 --- a/base/daemon/BaseDaemon.cpp +++ b/base/daemon/BaseDaemon.cpp @@ -761,7 +761,7 @@ void BaseDaemon::initializeTerminationAndSignalProcessing() static KillingErrorHandler killing_error_handler; Poco::ErrorHandler::set(&killing_error_handler); - signal_pipe.setNonBlocking(); + signal_pipe.setNonBlockingWrite(); signal_pipe.tryIncreaseSize(1 << 20); signal_listener = std::make_unique(*this); diff --git a/src/Common/PipeFDs.cpp b/src/Common/PipeFDs.cpp index d91917c23a4..a5c21e3d872 100644 --- a/src/Common/PipeFDs.cpp +++ b/src/Common/PipeFDs.cpp @@ -70,7 +70,7 @@ LazyPipeFDs::~LazyPipeFDs() } -void LazyPipeFDs::setNonBlocking() +void LazyPipeFDs::setNonBlockingWrite() { int flags = fcntl(fds_rw[1], F_GETFL, 0); if (-1 == flags) @@ -79,6 +79,21 @@ void LazyPipeFDs::setNonBlocking() throwFromErrno("Cannot set non-blocking mode of pipe", ErrorCodes::CANNOT_FCNTL); } +void LazyPipeFDs::setNonBlockingRead() +{ + int flags = fcntl(fds_rw[0], F_GETFL, 0); + if (-1 == flags) + throwFromErrno("Cannot get file status flags of pipe", ErrorCodes::CANNOT_FCNTL); + if (-1 == fcntl(fds_rw[0], F_SETFL, flags | O_NONBLOCK)) + throwFromErrno("Cannot set non-blocking mode of pipe", ErrorCodes::CANNOT_FCNTL); +} + +void LazyPipeFDs::setNonBlockingReadWrite() +{ + setNonBlockingRead(); + setNonBlockingWrite(); +} + void LazyPipeFDs::tryIncreaseSize(int desired_size) { #if defined(OS_LINUX) diff --git a/src/Common/PipeFDs.h b/src/Common/PipeFDs.h index fe76740da70..20bd847c077 100644 --- a/src/Common/PipeFDs.h +++ b/src/Common/PipeFDs.h @@ -17,7 +17,12 @@ struct LazyPipeFDs void open(); void close(); - void setNonBlocking(); + /// Set O_NONBLOCK to different ends of pipe preserving existing flags. + /// Throws an exception if fcntl was not successful. + void setNonBlockingWrite(); + void setNonBlockingRead(); + void setNonBlockingReadWrite(); + void tryIncreaseSize(int desired_size); ~LazyPipeFDs(); diff --git a/src/Common/TraceCollector.cpp b/src/Common/TraceCollector.cpp index 1548af50d98..cbac9cd1a19 100644 --- a/src/Common/TraceCollector.cpp +++ b/src/Common/TraceCollector.cpp @@ -36,7 +36,7 @@ TraceCollector::TraceCollector(std::shared_ptr trace_log_) /** Turn write end of pipe to non-blocking mode to avoid deadlocks * when QueryProfiler is invoked under locks and TraceCollector cannot pull data from pipe. */ - pipe.setNonBlocking(); + pipe.setNonBlockingWrite(); pipe.tryIncreaseSize(1 << 20); thread = ThreadFromGlobalPool(&TraceCollector::run, this); diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index f1ce8ab29ee..9343ada2c55 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -54,7 +54,7 @@ struct SocketInterruptablePollWrapper explicit SocketInterruptablePollWrapper(const Poco::Net::StreamSocket & poco_socket_) : sockfd(poco_socket_.impl()->sockfd()) { - pipe.setNonBlocking(); + pipe.setNonBlockingReadWrite(); #if defined(POCO_HAVE_FD_EPOLL) epollfd = epoll_create(2); @@ -158,15 +158,22 @@ struct SocketInterruptablePollWrapper result |= HAS_REQUEST; else { - UInt8 byte; - if (read(pipe.fds_rw[0], &byte, sizeof(byte)) < 0) - throwFromErrno("Cannot read from pipe", ErrorCodes::SYSTEM_ERROR); - if (byte == WATCH_RESPONSE_BYTE) - result |= HAS_WATCH_RESPONSE; - else if (byte == RESPONSE_BYTE) - result |= HAS_RESPONSE; - else - throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + int read_result; + do + { + UInt8 byte; + read_result = read(pipe.fds_rw[0], &byte, sizeof(byte)); + if (byte == WATCH_RESPONSE_BYTE) + result |= HAS_WATCH_RESPONSE; + else if (byte == RESPONSE_BYTE) + result |= HAS_RESPONSE; + else + throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + } + while (read_result > 0 || (read_result < 0 && errno == EINTR)); + + if (read_result < 0 && errno != EAGAIN) + throwFromErrno("Got error reading from pipe", ErrorCodes::SYSTEM_ERROR); } } } @@ -297,7 +304,8 @@ void TestKeeperTCPHandler::runImpl() LOG_TRACE(log, "Received heartbeat for session #{}", session_id); session_stopwatch.restart(); } - } while (in->available()); + } + while (in->available()); } if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) From 00c190aeed65eeef235bd502434ed36c23aa932b Mon Sep 17 00:00:00 2001 From: alesapin Date: Fri, 27 Nov 2020 14:57:48 +0300 Subject: [PATCH 063/174] Fix unitialized memory --- src/Server/TestKeeperTCPHandler.cpp | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 9343ada2c55..fff6c344b4e 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -163,12 +163,15 @@ struct SocketInterruptablePollWrapper { UInt8 byte; read_result = read(pipe.fds_rw[0], &byte, sizeof(byte)); - if (byte == WATCH_RESPONSE_BYTE) - result |= HAS_WATCH_RESPONSE; - else if (byte == RESPONSE_BYTE) - result |= HAS_RESPONSE; - else - throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + if (read_result > 0) + { + if (byte == WATCH_RESPONSE_BYTE) + result |= HAS_WATCH_RESPONSE; + else if (byte == RESPONSE_BYTE) + result |= HAS_RESPONSE; + else + throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + } } while (read_result > 0 || (read_result < 0 && errno == EINTR)); From 6791558668ed151249ede4f1400ddc882d6a99b7 Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Tue, 1 Dec 2020 14:22:28 -0400 Subject: [PATCH 064/174] Update date-time-functions.md --- .../functions/date-time-functions.md | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 3c9bd99de57..48d9aee8a49 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -63,10 +63,18 @@ int32samoa: 1546300800 Переводит дату или дату-с-временем в число типа UInt16, содержащее номер года (AD). +## toQuarter {#toquarter} + +Переводит дату или дату-с-временем в число типа UInt8, содержащее номер квартала. + ## toMonth {#tomonth} Переводит дату или дату-с-временем в число типа UInt8, содержащее номер месяца (1-12). +## toDayOfYear {#todayofyear} + +Переводит дату или дату-с-временем в число типа UInt16, содержащее номер дня года (1-366). + ## toDayOfMonth {#todayofmonth} Переводит дату или дату-с-временем в число типа UInt8, содержащее номер дня в месяце (1-31). @@ -128,6 +136,22 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp Округляет дату или дату-с-временем вниз до первого дня года. Возвращается дата. +## toStartOfISOYear {#tostartofisoyear} + +Округляет дату или дату-с-временем вниз до первого дня ISO года. Возвращается дата. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года это неделя с 4-мя или более днями в этом году. + +1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это 2016 ISO-год, который начался 2016-01-04. + +```sql +SELECT toStartOfISOYear(toDate('2017-01-01')) AS ISOYear20170101; +``` +```text +┌─ISOYear20170101─┐ +│ 2016-01-04 │ +└─────────────────┘ +``` + ## toStartOfQuarter {#tostartofquarter} Округляет дату или дату-с-временем вниз до первого дня квартала. @@ -147,6 +171,12 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp Округляет дату или дату-с-временем вниз до ближайшего понедельника. Возвращается дата. +## toStartOfWeek(t[,mode]) {#tostartofweek} + +Округляет дату или дату со временем до ближайшего воскресенья или понедельника в соответствии с mode. +Возвращается дата. +Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0. + ## toStartOfDay {#tostartofday} Округляет дату-с-временем вниз до начала дня. Возвращается дата-с-временем. @@ -243,6 +273,10 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d Переводит дату-с-временем или дату в номер года, начиная с некоторого фиксированного момента в прошлом. +## toRelativeQuarterNum {#torelativequarternum} + +Переводит дату-с-временем или дату в номер квартала, начиная с некоторого фиксированного момента в прошлом. + ## toRelativeMonthNum {#torelativemonthnum} Переводит дату-с-временем или дату в номер месяца, начиная с некоторого фиксированного момента в прошлом. @@ -267,6 +301,102 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d Переводит дату-с-временем в номер секунды, начиная с некоторого фиксированного момента в прошлом. +## toISOYear {toisoyear} + +Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого 1 января. + +Пример: + +```sql +SELECT + toDate('2017-01-01') AS date, + toYear(date), + toISOYear(date) +``` +```text +┌───────date─┬─toYear(toDate('2017-01-01'))─┬─toISOYear(toDate('2017-01-01'))─┐ +│ 2017-01-01 │ 2017 │ 2016 │ +└────────────┴──────────────────────────────┴─────────────────────────────────┘ +``` + +## toISOWeek {toisoweek} + +Переводит дату-с-временем или дату в число типа UInt8, содержащее номер ISO недели. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года это неделя с 4-мя или более днями в этом году. + +1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это последняя неделя 2016 года. + +```sql +SELECT + toISOWeek(toDate('2017-01-01')) AS ISOWeek20170101, + toISOWeek(toDate('2017-01-02')) AS ISOWeek20170102 +``` + +```text +┌─ISOWeek20170101─┬─ISOWeek20170102─┐ +│ 52 │ 1 │ +└─────────────────┴─────────────────┘ +``` + +## toWeek(date\[, mode\]\[, timezone\]) {#toweek} +Переводит дату-с-временем или дату в число UInt8, содержащее номер недели. Второй аргументам mode задает режим, начинается ли неделя с воскресенья или с понедельника и должно ли возвращаемое значение находиться в диапазоне от 0 до 53 или от 1 до 53. Если аргумент mode опущен, то используется режим 0. + +`toISOWeek() ` эквивалентно `toWeek(date,3)`. + +Описание режимов (mode): + +| Mode | Первый день недели | Диапазон | Неделя 1 это первая неделя … | +| ----------- | -------- | -------- | ------------------ | +|0|Воскресенье|0-53|с Воскресеньем в этом году +|1|Понедельник|0-53|с 4-мя или более днями в этом году +|2|Воскресенье|1-53|с Воскресеньем в этом году +|3|Понедельник|1-53|с 4-мя или более днями в этом году +|4|Воскресенье|0-53|с 4-мя или более днями в этом году +|5|Понедельник|0-53|с Понедельником в этом году +|6|Воскресенье|1-53|с 4-мя или более днями в этом году +|7|Понедельник|1-53|с Понедельником в этом году +|8|Воскресенье|1-53|содержащая 1 Января +|9|Понедельник|1-53|содержащая 1 Января + +Для режимов со значением «с 4 или более днями в этом году» недели нумеруются в соответствии с ISO 8601:1988: + +- Если неделя, содержащая 1 января, имеет 4 или более дней в новом году, это неделя 1. + +- В противном случае это последняя неделя предыдущего года, а следующая неделя - неделя 1. + +Для режимов со значением «содержит 1 января», неделя 1 – это неделя содержащая 1 января. Не имеет значения, сколько дней в новом году содержала неделя, даже если она содержала только один день. + +**Пример** + +```sql +SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS week1, toWeek(date,9) AS week9; +``` + +```text +┌───────date─┬─week0─┬─week1─┬─week9─┐ +│ 2016-12-27 │ 52 │ 52 │ 1 │ +└────────────┴───────┴───────┴───────┘ +``` + +## toYearWeek(date[,mode]) {#toyearweek} +Возвращает год и неделю для даты. Год в результате может отличаться от года в аргументе даты для первой и последней недели года. + +Аргумент mode работает точно так же, как аргумент mode [toWeek()](#date_time_functions-toweek). Если mode не задан, используется режим 0. + +`toISOYear() ` эквивалентно `intDiv(toYearWeek(date,3),100)`. + +**Пример** + +```sql +SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(date,1) AS yearWeek1, toYearWeek(date,9) AS yearWeek9; +``` + +```text +┌───────date─┬─yearWeek0─┬─yearWeek1─┬─yearWeek9─┐ +│ 2016-12-27 │ 201652 │ 201652 │ 201701 │ +└────────────┴───────────┴───────────┴───────────┘ +``` + ## date_trunc {#date_trunc} Отсекает от даты и времени части, меньшие чем указанная часть. From 12f0a3feaf36df3b3ad4b9e4210e69bfbbd63c0e Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Tue, 1 Dec 2020 14:45:00 -0400 Subject: [PATCH 065/174] Update date-time-functions.md --- docs/en/sql-reference/functions/date-time-functions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index 75db3fafe36..a5b22f073f6 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -366,7 +366,7 @@ SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(d └────────────┴───────────┴───────────┴───────────┘ ``` -## date_trunc {#date_trunc} +## date\_trunc {#date_trunc} Truncates date and time data to the specified part of date. @@ -435,7 +435,7 @@ Result: - [toStartOfInterval](#tostartofintervaltime-or-data-interval-x-unit-time-zone) -# now {#now} +## now {#now} Returns the current date and time. @@ -662,7 +662,7 @@ Result: [Original article](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) -## FROM_UNIXTIME +## FROM_UNIXTIME {#fromunixfime} When there is only single argument of integer type, it act in the same way as `toDateTime` and return [DateTime](../../sql-reference/data-types/datetime.md). type. From 03a74e9d624fc33d989e7630df19c5edf5bed93f Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Tue, 1 Dec 2020 14:47:56 -0400 Subject: [PATCH 066/174] Update date-time-functions.md --- docs/en/sql-reference/functions/date-time-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index a5b22f073f6..f20f09789eb 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -662,7 +662,7 @@ Result: [Original article](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) -## FROM_UNIXTIME {#fromunixfime} +## FROM\_UNIXTIME {#fromunixfime} When there is only single argument of integer type, it act in the same way as `toDateTime` and return [DateTime](../../sql-reference/data-types/datetime.md). type. From 7a0fcaa8694c018c7f4f7bfd6d6e8ed051b3902c Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Tue, 1 Dec 2020 22:25:57 +0300 Subject: [PATCH 067/174] Update math-functions.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Выполнил описание некоторых математических функций. --- .../sql-reference/functions/math-functions.md | 302 ++++++++++++++++++ 1 file changed, 302 insertions(+) diff --git a/docs/en/sql-reference/functions/math-functions.md b/docs/en/sql-reference/functions/math-functions.md index acb88afd219..8373205e6b9 100644 --- a/docs/en/sql-reference/functions/math-functions.md +++ b/docs/en/sql-reference/functions/math-functions.md @@ -111,4 +111,306 @@ Accepts a numeric argument and returns a UInt64 number close to 2 to the power o Accepts a numeric argument and returns a UInt64 number close to 10 to the power of x. +## cosh(x) {#coshx} + +[Hyperbolic cosine](https://in.mathworks.com/help/matlab/ref/cosh.html). + +**Syntax** + +``` sql +cosh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `1 <= cosh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT cosh(0); +``` + +Result: + +``` text +┌─cosh(0)──┐ +│ 1 │ +└──────────┘ +``` + +## acosh(x) {#acoshx} + +[Inverse hyperbolic cosine](https://www.mathworks.com/help/matlab/ref/acosh.html). + +**Syntax** + +``` sql +acosh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values are from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `0 <= acosh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT acosh(1); +``` + +Result: + +``` text +┌─acosh(1)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [cosh(x)](../../sql-reference/functions/math-functions.md#coshx) + +## sinh(x) {#sinhx} + +[Hyperbolic sine](https://www.mathworks.com/help/matlab/ref/sinh.html). + +**Syntax** + +``` sql +sinh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `-∞ < sinh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT sinh(0); +``` + +Result: + +``` text +┌─sinh(0)──┐ +│ 0 │ +└──────────┘ +``` + +## asinh(x) {#asinhx} + +[Inverse hyperbolic sine](https://www.mathworks.com/help/matlab/ref/asinh.html). + +**Syntax** + +``` sql +asinh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `-∞ < asinh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT asinh(0); +``` + +Result: + +``` text +┌─asinh(0)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [sinh(x)](../../sql-reference/functions/math-functions.md#sinhx) + +## atanh(x) {#atanhx} + +[Inverse hyperbolic tangent](https://www.mathworks.com/help/matlab/ref/atanh.html). + +**Syntax** + +``` sql +atanh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values are from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `-∞ < atanh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT atanh(0); +``` + +Result: + +``` text +┌─atanh(0)─┐ +│ 0 │ +└──────────┘ +``` + +## atan2(y, x) {#atan2yx} + +The [function](https://en.wikipedia.org/wiki/Atan2) is defined as the angle in the Euclidean plane, given in radians, between the positive x axis and the ray(`r`) to the point `(x, y) ≠ (0, 0)`. + +**Syntax** + +``` sql +atan2(y, x) +``` + +**Parameters** + +- `y` — y axis coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — x axis coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The angle `θ` such that `−π < θ ≤ π` and, for some `r > 0`, in radians. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT atan2(1, 1); +``` + +Result: + +``` text +┌────────atan2(1, 1)─┐ +│ 0.7853981633974483 │ +└────────────────────┘ +``` + +## hypot(x, y) {#hypotxy} + +The [function](https://en.wikipedia.org/wiki/Hypot) is defined to calculate the length of the hypotenuse of a right-angle triangle. It was designed to avoid errors arising due to limited-precision calculations performed on computers. The function avoids problems that occur when squaring very large or very small numbers. + +**Syntax** + +``` sql +hypot(x, y) +``` + +**Parameters** + +- `x` — The first cathetus of a right-angle triangle. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `y` — The second cathetus of a right-angle triangle. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The length of the hypotenuse of a right-angle triangle. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT hypot(1, 1); +``` + +Result: + +``` text +┌────────hypot(1, 1)─┐ +│ 1.4142135623730951 │ +└────────────────────┘ +``` + +## log1p(x) {#log1px} + +The [function](https://en.wikipedia.org/wiki/Natural_logarithm#lnp1) calculates `log(1 + x)`, compensating for the roundoff in `1+x`. `log1p(x)` is more accurate than `log(1+x)` for small values of `x`. For small `x`, `log1p(x)` is approximately `x`, whereas `log(1+x)` can be zero. + +**Syntax** + +``` sql +log1p(x) +``` + +**Parameters** + +- `x` — Values are from the interval: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values are from the interval: `-∞ < log1p(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT log1p(0); +``` + +Result: + +``` text +┌─log1p(0)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [log(x)](../../sql-reference/functions/math-functions.md#logx-lnx) + [Original article](https://clickhouse.tech/docs/en/query_language/functions/math_functions/) From c1b989c45e7b8fe40b54001d8778f6a26a5af921 Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Tue, 1 Dec 2020 15:38:53 -0400 Subject: [PATCH 068/174] Update date-time-functions.md --- docs/ru/sql-reference/functions/date-time-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 48d9aee8a49..27bef02f2ee 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -301,7 +301,7 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d Переводит дату-с-временем в номер секунды, начиная с некоторого фиксированного момента в прошлом. -## toISOYear {toisoyear} +## toISOYear {#toisoyear} Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого 1 января. @@ -319,7 +319,7 @@ SELECT └────────────┴──────────────────────────────┴─────────────────────────────────┘ ``` -## toISOWeek {toisoweek} +## toISOWeek {#toisoweek} Переводит дату-с-временем или дату в число типа UInt8, содержащее номер ISO недели. Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года это неделя с 4-мя или более днями в этом году. From 49c9793d51c8d7257d67faa658da63c45686f87b Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Tue, 1 Dec 2020 17:02:21 -0400 Subject: [PATCH 069/174] Update date-time-functions.md --- docs/ru/sql-reference/functions/date-time-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 27bef02f2ee..50152001072 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -381,7 +381,7 @@ SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS we ## toYearWeek(date[,mode]) {#toyearweek} Возвращает год и неделю для даты. Год в результате может отличаться от года в аргументе даты для первой и последней недели года. -Аргумент mode работает точно так же, как аргумент mode [toWeek()](#date_time_functions-toweek). Если mode не задан, используется режим 0. +Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если mode не задан, используется режим 0. `toISOYear() ` эквивалентно `intDiv(toYearWeek(date,3),100)`. From 79d68958cb13823fbf9249100bedd52b7dc944c5 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Thu, 3 Dec 2020 23:59:17 +0300 Subject: [PATCH 070/174] Update math-functions.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Выполнил перевод на русский язык. --- .../sql-reference/functions/math-functions.md | 36 +- .../sql-reference/functions/math-functions.md | 310 ++++++++++++++++++ 2 files changed, 328 insertions(+), 18 deletions(-) diff --git a/docs/en/sql-reference/functions/math-functions.md b/docs/en/sql-reference/functions/math-functions.md index 8373205e6b9..668d770a95f 100644 --- a/docs/en/sql-reference/functions/math-functions.md +++ b/docs/en/sql-reference/functions/math-functions.md @@ -123,11 +123,11 @@ cosh(x) **Parameters** -- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `1 <= cosh(x) < +∞`. +- Values from the interval: `1 <= cosh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -159,11 +159,11 @@ acosh(x) **Parameters** -- `x` — The angle, in radians. Values are from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — The angle, in radians. Values from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `0 <= acosh(x) < +∞`. +- Values from the interval: `0 <= acosh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -199,11 +199,11 @@ sinh(x) **Parameters** -- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `-∞ < sinh(x) < +∞`. +- Values from the interval: `-∞ < sinh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -235,11 +235,11 @@ asinh(x) **Parameters** -- `x` — The angle, in radians. Values are from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `-∞ < asinh(x) < +∞`. +- Values from the interval: `-∞ < asinh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -275,11 +275,11 @@ atanh(x) **Parameters** -- `x` — The angle, in radians. Values are from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — The angle, in radians. Values from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `-∞ < atanh(x) < +∞`. +- Values from the interval: `-∞ < atanh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -301,7 +301,7 @@ Result: ## atan2(y, x) {#atan2yx} -The [function](https://en.wikipedia.org/wiki/Atan2) is defined as the angle in the Euclidean plane, given in radians, between the positive x axis and the ray(`r`) to the point `(x, y) ≠ (0, 0)`. +The [function](https://en.wikipedia.org/wiki/Atan2) calculates the angle in the Euclidean plane, given in radians, between the positive x axis and the ray to the point `(x, y) ≠ (0, 0)`. **Syntax** @@ -311,12 +311,12 @@ atan2(y, x) **Parameters** -- `y` — y axis coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). -- `x` — x axis coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `y` — y-coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — x-coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- The angle `θ` such that `−π < θ ≤ π` and, for some `r > 0`, in radians. +- The angle `θ` such that `−π < θ ≤ π`, in radians. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -338,7 +338,7 @@ Result: ## hypot(x, y) {#hypotxy} -The [function](https://en.wikipedia.org/wiki/Hypot) is defined to calculate the length of the hypotenuse of a right-angle triangle. It was designed to avoid errors arising due to limited-precision calculations performed on computers. The function avoids problems that occur when squaring very large or very small numbers. +Calculates the length of the hypotenuse of a right-angle triangle. The [function](https://en.wikipedia.org/wiki/Hypot) avoids problems that occur when squaring very large or very small numbers. **Syntax** @@ -375,7 +375,7 @@ Result: ## log1p(x) {#log1px} -The [function](https://en.wikipedia.org/wiki/Natural_logarithm#lnp1) calculates `log(1 + x)`, compensating for the roundoff in `1+x`. `log1p(x)` is more accurate than `log(1+x)` for small values of `x`. For small `x`, `log1p(x)` is approximately `x`, whereas `log(1+x)` can be zero. +Calculates `log(1+x)`. The [function](https://en.wikipedia.org/wiki/Natural_logarithm#lnp1) `log1p(x)` is more accurate than `log(1+x)` for small values of x. **Syntax** @@ -385,11 +385,11 @@ log1p(x) **Parameters** -- `x` — Values are from the interval: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — Values from the interval: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values are from the interval: `-∞ < log1p(x) < +∞`. +- Values from the interval: `-∞ < log1p(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). diff --git a/docs/ru/sql-reference/functions/math-functions.md b/docs/ru/sql-reference/functions/math-functions.md index 6df366d129f..70a154b13d9 100644 --- a/docs/ru/sql-reference/functions/math-functions.md +++ b/docs/ru/sql-reference/functions/math-functions.md @@ -103,4 +103,314 @@ SELECT erf(3 / sqrt(2)) Принимает два числовых аргумента x и y. Возвращает число типа Float64, близкое к x в степени y. +## intExp2 {#intexp2} + +Принимает числовой аргумент. Возвращает число типа UInt64, близкое к 2 в степени x. + +## intExp10 {#intexp10} + +Принимает числовой аргумент. Возвращает число типа UInt64, близкое к 10 в степени x. + +## cosh(x) {#coshx} + +[Гиперболический косинус](https://help.scilab.org/docs/5.4.0/ru_RU/cosh.html). + +**Синтаксис** + +``` sql +cosh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `1 <= cosh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT cosh(0); +``` + +Результат: + +``` text +┌─cosh(0)──┐ +│ 1 │ +└──────────┘ +``` + +## acosh(x) {#acoshx} + +[Обратный гиперболический косинус](https://help.scilab.org/docs/5.4.0/ru_RU/acosh.html). + +**Синтаксис** + +``` sql +acosh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `0 <= acosh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT acosh(1); +``` + +Результат: + +``` text +┌─acosh(1)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [cosh(x)](../../sql-reference/functions/math-functions.md#coshx) + +## sinh(x) {#sinhx} + +[Гиперболический синус](https://help.scilab.org/docs/5.4.0/ru_RU/sinh.html). + +**Синтаксис** + +``` sql +sinh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < sinh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT sinh(0); +``` + +Результат: + +``` text +┌─sinh(0)──┐ +│ 0 │ +└──────────┘ +``` + +## asinh(x) {#asinhx} + +[Обратный гиперболический синус](https://help.scilab.org/docs/5.4.0/ru_RU/asinh.html). + +**Синтаксис** + +``` sql +asinh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < asinh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT asinh(0); +``` + +Результат: + +``` text +┌─asinh(0)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [sinh(x)](../../sql-reference/functions/math-functions.md#sinhx) + +## atanh(x) {#atanhx} + +[Обратный гиперболический тангенс](https://help.scilab.org/docs/5.4.0/ru_RU/atanh.html). + +**Синтаксис** + +``` sql +atanh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < atanh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT atanh(0); +``` + +Результат: + +``` text +┌─atanh(0)─┐ +│ 0 │ +└──────────┘ +``` + +## atan2(y, x) {#atan2yx} + +[Функция](https://msoffice-prowork.com/ref/excel/excelfunc/math/atan2/) вычисляет угол в радианах между положительной осью x и линией, проведенной из начала координат в точку `(x, y) ≠ (0, 0)`. + +**Синтаксис** + +``` sql +atan2(y, x) +``` + +**Параметры** + +- `y` — координата y точки, в которую проведена линия. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — координата х точки, в которую проведена линия. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Угол `θ` в радианах из интервала: `−π < θ ≤ π`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT atan2(1, 1); +``` + +Результат: + +``` text +┌────────atan2(1, 1)─┐ +│ 0.7853981633974483 │ +└────────────────────┘ +``` + +## hypot(x, y) {#hypotxy} + +Вычисляет длину гипотенузы прямоугольного треугольника. При использовании этой [функции](https://php.ru/manual/function.hypot.html) не возникает проблем при возведении в квадрат очень больших или очень малых чисел. + +**Синтаксис** + +``` sql +hypot(x, y) +``` + +**Параметры** + +- `x` — первый катет прямоугольного треугольника. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `y` — второй катет прямоугольного треугольника. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Длина гипотенузы прямоугольного треугольника. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT hypot(1, 1); +``` + +Результат: + +``` text +┌────────hypot(1, 1)─┐ +│ 1.4142135623730951 │ +└────────────────────┘ +``` + +## log1p(x) {#log1px} + +Вычисляет `log(1+x)`. [Функция](https://help.scilab.org/docs/6.0.1/ru_RU/log1p.html) `log1p(x)` является более точной, чем функция `log(1+x)` для малых значений x. + +**Синтаксис** + +``` sql +log1p(x) +``` + +**Параметры** + +- `x` — значения из интервала: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < log1p(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT log1p(0); +``` + +Результат: + +``` text +┌─log1p(0)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [log(x)](../../sql-reference/functions/math-functions.md#logx-lnx) + [Оригинальная статья](https://clickhouse.tech/docs/ru/query_language/functions/math_functions/) From b0724569324911404e5fa87d2ba7a949133951a8 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Fri, 4 Dec 2020 01:08:10 +0300 Subject: [PATCH 071/174] Update math-functions.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Правлю битую ссылку. --- docs/ru/sql-reference/functions/math-functions.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/ru/sql-reference/functions/math-functions.md b/docs/ru/sql-reference/functions/math-functions.md index 70a154b13d9..afcce14dd74 100644 --- a/docs/ru/sql-reference/functions/math-functions.md +++ b/docs/ru/sql-reference/functions/math-functions.md @@ -103,14 +103,6 @@ SELECT erf(3 / sqrt(2)) Принимает два числовых аргумента x и y. Возвращает число типа Float64, близкое к x в степени y. -## intExp2 {#intexp2} - -Принимает числовой аргумент. Возвращает число типа UInt64, близкое к 2 в степени x. - -## intExp10 {#intexp10} - -Принимает числовой аргумент. Возвращает число типа UInt64, близкое к 10 в степени x. - ## cosh(x) {#coshx} [Гиперболический косинус](https://help.scilab.org/docs/5.4.0/ru_RU/cosh.html). From 3a265cf5a0bcfd092bccfbf5fc5f1feb36fee7b1 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Fri, 4 Dec 2020 23:24:31 +0300 Subject: [PATCH 072/174] Update math-functions.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Исправляю битую ссылку. --- docs/ru/sql-reference/functions/math-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/functions/math-functions.md b/docs/ru/sql-reference/functions/math-functions.md index afcce14dd74..3ed1afffb3a 100644 --- a/docs/ru/sql-reference/functions/math-functions.md +++ b/docs/ru/sql-reference/functions/math-functions.md @@ -403,6 +403,6 @@ SELECT log1p(0); **Смотрите также** -- [log(x)](../../sql-reference/functions/math-functions.md#logx-lnx) +- [log(x)](../../sql-reference/functions/math-functions.md#logx) [Оригинальная статья](https://clickhouse.tech/docs/ru/query_language/functions/math_functions/) From fadf764feffa82abf19819c3cc9ae2c581deb36b Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Sun, 6 Dec 2020 12:55:09 +0300 Subject: [PATCH 073/174] Update math-functions.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Внес изменения в описание обратных гиперболических функций. --- docs/en/sql-reference/functions/math-functions.md | 12 ++++++------ docs/ru/sql-reference/functions/math-functions.md | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/en/sql-reference/functions/math-functions.md b/docs/en/sql-reference/functions/math-functions.md index 668d770a95f..1f45b69f7a1 100644 --- a/docs/en/sql-reference/functions/math-functions.md +++ b/docs/en/sql-reference/functions/math-functions.md @@ -159,11 +159,11 @@ acosh(x) **Parameters** -- `x` — The angle, in radians. Values from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — Hyperbolic cosine of angle. Values from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values from the interval: `0 <= acosh(x) < +∞`. +- The angle, in radians. Values from the interval: `0 <= acosh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -235,11 +235,11 @@ asinh(x) **Parameters** -- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — Hyperbolic sine of angle. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values from the interval: `-∞ < asinh(x) < +∞`. +- The angle, in radians. Values from the interval: `-∞ < asinh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -275,11 +275,11 @@ atanh(x) **Parameters** -- `x` — The angle, in radians. Values from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — Hyperbolic tangent of angle. Values from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Returned value** -- Values from the interval: `-∞ < atanh(x) < +∞`. +- The angle, in radians. Values from the interval: `-∞ < atanh(x) < +∞`. Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). diff --git a/docs/ru/sql-reference/functions/math-functions.md b/docs/ru/sql-reference/functions/math-functions.md index 3ed1afffb3a..2e57aca6a0a 100644 --- a/docs/ru/sql-reference/functions/math-functions.md +++ b/docs/ru/sql-reference/functions/math-functions.md @@ -151,11 +151,11 @@ acosh(x) **Параметры** -- `x` — угол в радианах. Значения из интервала: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — гиперболический косинус угла. Значения из интервала: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Возвращаемое значение** -- Значения из интервала: `0 <= acosh(x) < +∞`. +- Угол в радианах. Значения из интервала: `0 <= acosh(x) < +∞`. Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -227,11 +227,11 @@ asinh(x) **Параметры** -- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — гиперболический синус угла. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Возвращаемое значение** -- Значения из интервала: `-∞ < asinh(x) < +∞`. +- Угол в радианах. Значения из интервала: `-∞ < asinh(x) < +∞`. Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). @@ -267,11 +267,11 @@ atanh(x) **Параметры** -- `x` — угол в радианах. Значения из интервала: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — гиперболический тангенс угла. Значения из интервала: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). **Возвращаемое значение** -- Значения из интервала: `-∞ < atanh(x) < +∞`. +- Угол в радианах. Значения из интервала: `-∞ < atanh(x) < +∞`. Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). From 99ba006be080b635978a2d0c492ec54602096397 Mon Sep 17 00:00:00 2001 From: annvsh Date: Sun, 6 Dec 2020 23:10:02 +0700 Subject: [PATCH 074/174] Fixed --- docs/en/interfaces/formats.md | 21 +++++++++++++++------ docs/ru/interfaces/formats.md | 17 ++++++++++++----- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index bb96f6c62a9..60d3c4e9285 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1341,15 +1341,24 @@ Result: ## RawBLOB {#rawblob} -This format slurps all input data into a single value. This format can only parse a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. +In this format, all input data is read to a single value. It is possible to parse щтдн a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. -The difference between `RawBLOB` and `TSVRaw`: -- data is output in binary format, no escaping; -- no delimiters between values; -- no newline at the end of each value. +Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](# tabseparatedraw). +`RawBLOB`: +- data is output in binary format, no escaping; +- there are no delimiters between values; +- no newline at the end of each value. +[TabSeparatedRaw] (# tabseparatedraw): +- data is output without screening; +- the string contains values ​​separated by tabs; +- there is a line feed after the last value in the line. -In `Raw BLOB` unlike `Raw Binary` strings are output without their length. +The following is a comparison of the `Raw BLOB` and [RowBinary](#rowbinary) formats. +`Raw BLOB`: +- lines are output without their length. +[TabSeparatedRaw](#tabseparatedraw): +- strings are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. When an empty value is passed to the `RawBLOB` input, ClickHouse generates an exception: diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index e65de3d74e0..82d4f70e178 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1146,15 +1146,24 @@ SELECT * FROM line_as_string; ## RawBLOB {#rawblob} -Этот формат считывает все входные данные в одно значение. Формат может парсить только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. +В этом формате все входные данные считываются в одно значение. Парсить можно только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. Результат выводится в бинарном виде без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. -Отличия между `RawBLOB` и `TabSeparatedRaw`: +Ниже приведено сравнение форматов `RawBLOB` и [TabSeparatedRaw](#tabseparatedraw). +`RawBLOB`: - данные выводятся в бинарном виде, без экранирования; - нет разделителей между значениями; - нет новой строки в конце каждого значения. +[TabSeparatedRaw](#tabseparatedraw): +- данные выводятся без экранирования; +- строка содержит значения, разделённые табуляцией; +- после последнего значения в строке есть перевод строки. -В `RawBLOB`, в отличие от `RowBinary`, строки выводятся без их длины. +Далее рассмотрено сравнение форматов `RawBLOB` и [RowBinary](#rowbinary). +`RawBLOB`: +- строки выводятся без их длины. +[TabSeparatedRaw](#tabseparatedraw): +- строки представлены как длина в формате varint (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), а затем байты строки. При передаче на вход `RawBLOB` пустого значения ClickHouse сгенерирует исключение: @@ -1165,11 +1174,9 @@ Code: 108. DB::Exception: No data to insert **Пример** ``` bash -$ clickhouse-client --query "DROP TABLE IF EXISTS {some_table};" $ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" $ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" $ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum -$ clickhouse-client --query "DROP TABLE {some_table};" ``` Результат: From b12b35a386b4cba4b4ee756d4857392c163e2927 Mon Sep 17 00:00:00 2001 From: annvsh Date: Sun, 6 Dec 2020 23:22:09 +0700 Subject: [PATCH 075/174] Fixed --- docs/en/interfaces/formats.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 6791ebce4c7..c72925811c0 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1383,8 +1383,8 @@ Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](# tabsepara - the string contains values ​​separated by tabs; - there is a line feed after the last value in the line. -The following is a comparison of the `Raw BLOB` and [RowBinary](#rowbinary) formats. -`Raw BLOB`: +The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) formats. +`RawBLOB`: - lines are output without their length. [TabSeparatedRaw](#tabseparatedraw): - strings are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. From 8ecd5654410dd940edbff8a730f8d9e305d222e3 Mon Sep 17 00:00:00 2001 From: jianmei zhang Date: Mon, 7 Dec 2020 13:28:09 +0800 Subject: [PATCH 076/174] Fix translated errors and update misc to latest --- docs/en/sql-reference/statements/kill.md | 2 +- docs/zh/sql-reference/statements/misc.md | 137 +++++++++++++---------- 2 files changed, 77 insertions(+), 62 deletions(-) diff --git a/docs/en/sql-reference/statements/kill.md b/docs/en/sql-reference/statements/kill.md index d3f2d9bb5c6..6aa09cca4ef 100644 --- a/docs/en/sql-reference/statements/kill.md +++ b/docs/en/sql-reference/statements/kill.md @@ -53,7 +53,7 @@ KILL MUTATION [ON CLUSTER cluster] Tries to cancel and remove [mutations](../../sql-reference/statements/alter/index.md#alter-mutations) that are currently executing. Mutations to cancel are selected from the [`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) table using the filter specified by the `WHERE` clause of the `KILL` query. -A test query (`TEST`) only checks the user’s rights and displays a list of queries to stop. +A test query (`TEST`) only checks the user’s rights and displays a list of mutations to stop. Examples: diff --git a/docs/zh/sql-reference/statements/misc.md b/docs/zh/sql-reference/statements/misc.md index a736ed2af5b..b4297d1ed4f 100644 --- a/docs/zh/sql-reference/statements/misc.md +++ b/docs/zh/sql-reference/statements/misc.md @@ -41,25 +41,25 @@ CHECK TABLE [db.]name 该 `CHECK TABLE` 查询支持下表引擎: -- [日志](../../engines/table-engines/log-family/log.md) +- [Log](../../engines/table-engines/log-family/log.md) - [TinyLog](../../engines/table-engines/log-family/tinylog.md) - [StripeLog](../../engines/table-engines/log-family/stripelog.md) -- [梅树家族](../../engines/table-engines/mergetree-family/mergetree.md) +- [MergeTree 家族](../../engines/table-engines/mergetree-family/mergetree.md) -使用另一个表引擎对表执行会导致异常。 +对其他不支持的表引擎的表执行会导致异常。 -从发动机 `*Log` 家庭不提供故障自动数据恢复。 使用 `CHECK TABLE` 查询以及时跟踪数据丢失。 +来自 `*Log` 家族的引擎不提供故障自动数据恢复。 使用 `CHECK TABLE` 查询及时跟踪数据丢失。 -为 `MergeTree` 家庭发动机， `CHECK TABLE` 查询显示本地服务器上表的每个单独数据部分的检查状态。 +对于 `MergeTree` 家族引擎， `CHECK TABLE` 查询显示本地服务器上表的每个单独数据部分的检查状态。 **如果数据已损坏** 如果表已损坏，则可以将未损坏的数据复制到另一个表。 要做到这一点: -1. 创建具有与损坏的表相同结构的新表。 要执行此操作，请执行查询 `CREATE TABLE AS `. -2. 设置 [max_threads](../../operations/settings/settings.md#settings-max_threads) 值为1以在单个线程中处理下一个查询。 要执行此操作，请运行查询 `SET max_threads = 1`. +1. 创建一个与损坏的表结构相同的新表。 要做到这一点，请执行查询 `CREATE TABLE AS `. +2. 将 [max_threads](../../operations/settings/settings.md#settings-max_threads) 值设置为1，以在单个线程中处理下一个查询。 要这样做，请运行查询 `SET max_threads = 1`. 3. 执行查询 `INSERT INTO SELECT * FROM `. 此请求将未损坏的数据从损坏的表复制到另一个表。 只有损坏部分之前的数据才会被复制。 -4. 重新启动 `clickhouse-client` 要重置 `max_threads` 价值。 +4. 重新启动 `clickhouse-client` 以重置 `max_threads` 值。 ## DESCRIBE TABLE {#misc-describe-table} @@ -67,57 +67,65 @@ CHECK TABLE [db.]name DESC|DESCRIBE TABLE [db.]table [INTO OUTFILE filename] [FORMAT format] ``` -返回以下内容 `String` 类型列: +返回以下 `String` 类型列: -- `name` — Column name. -- `type`— Column type. -- `default_type` — Clause that is used in [默认表达式](create.md#create-default-values) (`DEFAULT`, `MATERIALIZED` 或 `ALIAS`). 如果未指定默认表达式，则Column包含一个空字符串。 -- `default_expression` — Value specified in the `DEFAULT` 条款 -- `comment_expression` — Comment text. +- `name` — 列名。 +- `type`— 列的类型。 +- `default_type` — [默认表达式](create.md#create-default-values) (`DEFAULT`, `MATERIALIZED` 或 `ALIAS`)中使用的子句。 如果没有指定默认表达式，则列包含一个空字符串。 +- `default_expression` — `DEFAULT` 子句中指定的值。 +- `comment_expression` — 注释。 -嵌套的数据结构输出 “expanded” 格式。 每列分别显示，名称后面有一个点。 +嵌套数据结构以 “expanded” 格式输出。 每列分别显示，列名后加点号。 ## DETACH {#detach} -删除有关 ‘name’ 表从服务器。 服务器停止了解表的存在。 +从服务器中删除有关 ‘name’ 表的信息。 服务器停止了解该表的存在。 ``` sql DETACH TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -这不会删除表的数据或元数据。 在下一次服务器启动时，服务器将读取元数据并再次查找有关表的信息。 -同样，一个 “detached” 表可以使用重新连接 `ATTACH` 查询（系统表除外，它们没有为它们存储元数据）。 - -没有 `DETACH DATABASE` 查询。 +这不会删除表的数据或元数据。 在下一次服务器启动时，服务器将读取元数据并再次查找该表。 +同样，可以使用 `ATTACH` 查询重新连接一个 “detached” 的表（系统表除外，没有为它们存储元数据）。 ## DROP {#drop} -此查询有两种类型: `DROP DATABASE` 和 `DROP TABLE`. +删除已经存在的实体。如果指定 `IF EXISTS`， 则如果实体不存在，则不返回错误。 + +## DROP DATABASE {#drop-database} + +删除 `db` 数据库中的所有表，然后删除 `db` 数据库本身。 + +语法: ``` sql DROP DATABASE [IF EXISTS] db [ON CLUSTER cluster] ``` +## DROP TABLE {#drop-table} -删除内部的所有表 ‘db’ 数据库，然后删除 ‘db’ 数据库本身。 -如果 `IF EXISTS` 如果数据库不存在，则不会返回错误。 +删除表。 + +语法: ``` sql DROP [TEMPORARY] TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -删除表。 -如果 `IF EXISTS` 如果表不存在或数据库不存在，则不会返回错误。 - - DROP DICTIONARY [IF EXISTS] [db.]name +## DROP DICTIONARY {#drop-dictionary} 删除字典。 -如果 `IF EXISTS` 如果表不存在或数据库不存在，则不会返回错误。 + +语法: + +``` sql +DROP DICTIONARY [IF EXISTS] [db.]name +``` ## DROP USER {#drop-user-statement} 删除用户。 -### 语法 {#drop-user-syntax} +语法: ``` sql DROP USER [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -129,7 +137,7 @@ DROP USER [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除的角色将从授予该角色的所有实体撤销。 -### 语法 {#drop-role-syntax} +语法: ``` sql DROP ROLE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -141,7 +149,7 @@ DROP ROLE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除行策略将从分配该策略的所有实体撤销。 -### 语法 {#drop-row-policy-syntax} +语法: ``` sql DROP [ROW] POLICY [IF EXISTS] name [,...] ON [database.]table [,...] [ON CLUSTER cluster_name] @@ -153,7 +161,7 @@ DROP [ROW] POLICY [IF EXISTS] name [,...] ON [database.]table [,...] [ON CLUSTER 已删除的配额将从分配该配额的所有实体撤销。 -### 语法 {#drop-quota-syntax} +语法: ``` sql DROP QUOTA [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -165,12 +173,22 @@ DROP QUOTA [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除的settings配置将从分配该settings配置的所有实体撤销。 -### 语法 {#drop-settings-profile-syntax} +语法: ``` sql DROP [SETTINGS] PROFILE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] ``` +## DROP VIEW {#drop-view} + +删除视图。视图也可以通过 `DROP TABLE` 删除，但是 `DROP VIEW` 检查 `[db.]name` 是视图。 + +语法: + +``` sql +DROP VIEW [IF EXISTS] [db.]name [ON CLUSTER cluster] +``` + ## EXISTS {#exists-statement} ``` sql @@ -189,7 +207,7 @@ KILL QUERY [ON CLUSTER cluster] ``` 尝试强制终止当前正在运行的查询。 -要终止的查询是从系统中选择的。使用在定义的标准进程表 `WHERE` 《公约》条款 `KILL` 查询。 +要终止的查询是使用 `KILL` 查询的 `WHERE` 子句定义的标准从system.processes表中选择的。 例: @@ -206,13 +224,13 @@ KILL QUERY WHERE user='username' SYNC 默认情况下，使用异步版本的查询 (`ASYNC`），不等待确认查询已停止。 同步版本 (`SYNC`）等待所有查询停止，并在停止时显示有关每个进程的信息。 -响应包含 `kill_status` 列，它可以采用以下值: +响应包含 `kill_status` 列，该列可以采用以下值: -1. ‘finished’ – The query was terminated successfully. -2. ‘waiting’ – Waiting for the query to end after sending it a signal to terminate. -3. The other values ​​explain why the query can't be stopped. +1. ‘finished’ – 查询已成功终止。 +2. ‘waiting’ – 发送查询信号终止后，等待查询结束。 +3. 其他值解释为什么查询不能停止。 -测试查询 (`TEST`）仅检查用户的权限并显示要停止的查询列表。 +测试查询 (`TEST`）仅检查用户的权限，并显示要停止的查询列表。 ## KILL MUTATION {#kill-mutation} @@ -223,9 +241,9 @@ KILL MUTATION [ON CLUSTER cluster] [FORMAT format] ``` -尝试取消和删除 [突变](alter.md#alter-mutations) 当前正在执行。 要取消的突变选自 [`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) 表使用由指定的过滤器 `WHERE` 《公约》条款 `KILL` 查询。 +尝试取消和删除当前正在执行的 [mutations](alter.md#alter-mutations) 。 要取消的mutation是使用 `KILL` 查询的WHERE子句指定的过滤器从[`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) 表中选择的。 -测试查询 (`TEST`）仅检查用户的权限并显示要停止的查询列表。 +测试查询 (`TEST`）仅检查用户的权限并显示要停止的mutations列表。 例: @@ -237,9 +255,9 @@ KILL MUTATION WHERE database = 'default' AND table = 'table' KILL MUTATION WHERE database = 'default' AND table = 'table' AND mutation_id = 'mutation_3.txt' ``` -The query is useful when a mutation is stuck and cannot finish (e.g. if some function in the mutation query throws an exception when applied to the data contained in the table). +当mutation卡住且无法完成时，该查询是有用的(例如，当mutation查询中的某些函数在应用于表中包含的数据时抛出异常)。 -已经由突变所做的更改不会回滚。 +Mutation已经做出的更改不会回滚。 ## OPTIMIZE {#misc_operations-optimize} @@ -247,19 +265,19 @@ The query is useful when a mutation is stuck and cannot finish (e.g. if some fu OPTIMIZE TABLE [db.]name [ON CLUSTER cluster] [PARTITION partition | PARTITION ID 'partition_id'] [FINAL] [DEDUPLICATE] ``` -此查询尝试使用来自表引擎的表初始化表的数据部分的非计划合并 [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md) 家人 +此查询尝试初始化 [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md)家族的表引擎的表中未计划合并数据部分。 -该 `OPTMIZE` 查询也支持 [MaterializedView](../../engines/table-engines/special/materializedview.md) 和 [缓冲区](../../engines/table-engines/special/buffer.md) 引擎 不支持其他表引擎。 +该 `OPTMIZE` 查询也支持 [MaterializedView](../../engines/table-engines/special/materializedview.md) 和 [Buffer](../../engines/table-engines/special/buffer.md) 引擎。 不支持其他表引擎。 -当 `OPTIMIZE` 与使用 [ReplicatedMergeTree](../../engines/table-engines/mergetree-family/replication.md) 表引擎的家族，ClickHouse创建合并任务，并等待在所有节点上执行（如果 `replication_alter_partitions_sync` 设置已启用）。 +当 `OPTIMIZE` 与 [ReplicatedMergeTree](../../engines/table-engines/mergetree-family/replication.md) 家族的表引擎一起使用时，ClickHouse将创建一个合并任务，并等待所有节点上的执行（如果 `replication_alter_partitions_sync` 设置已启用）。 - 如果 `OPTIMIZE` 出于任何原因不执行合并，它不通知客户端。 要启用通知，请使用 [optimize_throw_if_noop](../../operations/settings/settings.md#setting-optimize_throw_if_noop) 设置。 - 如果您指定 `PARTITION`，仅优化指定的分区。 [如何设置分区表达式](alter.md#alter-how-to-specify-part-expr). - 如果您指定 `FINAL`，即使所有数据已经在一个部分中，也会执行优化。 -- 如果您指定 `DEDUPLICATE`，然后完全相同的行将被重复数据删除（所有列进行比较），这仅适用于MergeTree引擎。 +- 如果您指定 `DEDUPLICATE`，则将对完全相同的行进行重复数据删除（所有列进行比较），这仅适用于MergeTree引擎。 !!! warning "警告" - `OPTIMIZE` 无法修复 “Too many parts” 错误 + `OPTIMIZE` 无法修复 “Too many parts” 错误。 ## RENAME {#misc_operations-rename} @@ -270,6 +288,7 @@ RENAME TABLE [db11.]name11 TO [db12.]name12, [db21.]name21 TO [db22.]name22, ... ``` 所有表都在全局锁定下重命名。 重命名表是一个轻型操作。 如果您在TO之后指定了另一个数据库，则表将被移动到此数据库。 但是，包含数据库的目录必须位于同一文件系统中（否则，将返回错误）。 +如果您在一个查询中重命名多个表，这是一个非原子操作，它可能被部分执行，其他会话中的查询可能会接收错误 Table ... doesn't exist ...。 ## SET {#query-set} @@ -277,9 +296,9 @@ RENAME TABLE [db11.]name11 TO [db12.]name12, [db21.]name21 TO [db22.]name22, ... SET param = value ``` -分配 `value` 到 `param` [设置](../../operations/settings/index.md) 对于当前会话。 你不能改变 [服务器设置](../../operations/server-configuration-parameters/index.md) 这边 +为当前会话的 [设置](../../operations/settings/index.md) `param` 分配值 `value`。 您不能以这种方式更改 [服务器设置](../../operations/server-configuration-parameters/index.md)。 -您还可以在单个查询中设置指定设置配置文件中的所有值。 +您还可以在单个查询中从指定的设置配置文件中设置所有值。 ``` sql SET profile = 'profile-name-from-the-settings-file' @@ -291,8 +310,6 @@ SET profile = 'profile-name-from-the-settings-file' 激活当前用户的角色。 -### 语法 {#set-role-syntax} - ``` sql SET ROLE {DEFAULT | NONE | role [,...] | ALL | ALL EXCEPT role [,...]} ``` @@ -301,15 +318,13 @@ SET ROLE {DEFAULT | NONE | role [,...] | ALL | ALL EXCEPT role [,...]} 将默认角色设置为用户。 -默认角色在用户登录时自动激活。 您只能将以前授予的角色设置为默认值。 如果未向用户授予角色，ClickHouse将引发异常。 - -### 语法 {#set-default-role-syntax} +默认角色在用户登录时自动激活。 您只能将以前授予的角色设置为默认值。 如果角色没有授予用户，ClickHouse会抛出异常。 ``` sql SET DEFAULT ROLE {NONE | role [,...] | ALL | ALL EXCEPT role [,...]} TO {user|CURRENT_USER} [,...] ``` -### 例 {#set-default-role-examples} +### 示例 {#set-default-role-examples} 为用户设置多个默认角色: @@ -317,19 +332,19 @@ SET DEFAULT ROLE {NONE | role [,...] | ALL | ALL EXCEPT role [,...]} TO {user|CU SET DEFAULT ROLE role1, role2, ... TO user ``` -将所有授予的角色设置为用户的默认值: +将所有授予的角色设置为用户的默认角色: ``` sql SET DEFAULT ROLE ALL TO user ``` -从用户清除默认角色: +清除用户的默认角色: ``` sql SET DEFAULT ROLE NONE TO user ``` -将所有授予的角色设置为默认角色，其中一些角色除外: +将所有授予的角色设置为默认角色，但其中一些角色除外: ``` sql SET DEFAULT ROLE ALL EXCEPT role1, role2 TO user @@ -341,9 +356,9 @@ SET DEFAULT ROLE ALL EXCEPT role1, role2 TO user TRUNCATE TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -从表中删除所有数据。 当条款 `IF EXISTS` 如果该表不存在，则查询返回错误。 +从表中删除所有数据。 当省略 `IF EXISTS`子句时，如果该表不存在，则查询返回错误。 -该 `TRUNCATE` 查询不支持 [查看](../../engines/table-engines/special/view.md), [文件](../../engines/table-engines/special/file.md), [URL](../../engines/table-engines/special/url.md) 和 [Null](../../engines/table-engines/special/null.md) 表引擎. +该 `TRUNCATE` 查询不支持 [View](../../engines/table-engines/special/view.md), [File](../../engines/table-engines/special/file.md), [URL](../../engines/table-engines/special/url.md) 和 [Null](../../engines/table-engines/special/null.md) 表引擎. ## USE {#use} From e07374331a97bebd1181069aecec391afc17c89a Mon Sep 17 00:00:00 2001 From: root Date: Mon, 7 Dec 2020 15:23:58 +0800 Subject: [PATCH 077/174] add diagnostic information when two merge tables try to read each other's data --- src/Storages/StorageMerge.cpp | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/Storages/StorageMerge.cpp b/src/Storages/StorageMerge.cpp index f15ef8a5784..97f4ccd0bba 100644 --- a/src/Storages/StorageMerge.cpp +++ b/src/Storages/StorageMerge.cpp @@ -430,7 +430,15 @@ StorageMerge::StorageListWithLocks StorageMerge::getSelectedTables( DatabaseTablesIteratorPtr StorageMerge::getDatabaseIterator(const Context & context) const { - checkStackSize(); + try + { + checkStackSize(); + } + catch (Exception & e) + { + e.addMessage("while getting table iterator of Merge table. Maybe caused by two Merge tables that will endlessly try to read each other's data"); + throw; + } auto database = DatabaseCatalog::instance().getDatabase(source_database); auto table_name_match = [this](const String & table_name_) { return table_name_regexp.match(table_name_); }; return database->getTablesIterator(context, table_name_match); From e2ddb3f72306164f56f811a5fc9664a9bb6f19cb Mon Sep 17 00:00:00 2001 From: Evgeniia Sudarikova Date: Mon, 7 Dec 2020 13:01:04 +0300 Subject: [PATCH 078/174] edited after review --- docs/en/operations/settings/settings.md | 2 +- docs/ru/operations/settings/settings.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 29c74b76c29..f1c281ef2aa 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2360,7 +2360,7 @@ Default value: `1`. ## output_format_tsv_null_representation {#output_format_tsv_null_representation} -Allows configurable `NULL` representation for [TSV](../../interfaces/formats.md#tabseparated) output format. The setting only controls output format and `\N` is the only supported `NULL` representation for TSV input format. +Defines the representation of `NULL` for [TSV](../../interfaces/formats.md#tabseparated) output format. User can set any string as a value, for example, `My NULL`. Default value: `\N`. diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 9750f777990..06f061793af 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -2231,7 +2231,7 @@ SELECT CAST(toNullable(toInt32(0)) AS Int32) as x, toTypeName(x); ## output_format_tsv_null_representation {#output_format_tsv_null_representation} -Позволяет настраивать представление `NULL` для формата выходных данных [TSV](../../interfaces/formats.md#tabseparated). Настройка управляет форматом выходных данных, `\N` является единственным поддерживаемым представлением для формата входных данных TSV. +Определяет представление `NULL` для формата выходных данных [TSV](../../interfaces/formats.md#tabseparated). Пользователь может установить в качестве значения любую строку. Значение по умолчанию: `\N`. From d7b30f74d1b76e004c077e14ec3dcf470593ce6b Mon Sep 17 00:00:00 2001 From: feng lv Date: Mon, 7 Dec 2020 12:47:51 +0000 Subject: [PATCH 079/174] add min-max_compress_block_size to mergetree settings --- src/Storages/MergeTree/MergeTreeIOSettings.h | 20 ++++++++++++++----- src/Storages/MergeTree/MergeTreeSettings.h | 2 ++ .../MergeTree/MergedBlockOutputStream.cpp | 1 + .../MergedColumnOnlyOutputStream.cpp | 3 +++ 4 files changed, 21 insertions(+), 5 deletions(-) diff --git a/src/Storages/MergeTree/MergeTreeIOSettings.h b/src/Storages/MergeTree/MergeTreeIOSettings.h index 42554e51d83..41d9be07c70 100644 --- a/src/Storages/MergeTree/MergeTreeIOSettings.h +++ b/src/Storages/MergeTree/MergeTreeIOSettings.h @@ -1,6 +1,7 @@ #pragma once #include #include +#include namespace DB { @@ -19,13 +20,22 @@ struct MergeTreeWriterSettings { MergeTreeWriterSettings() = default; - MergeTreeWriterSettings(const Settings & global_settings, bool can_use_adaptive_granularity_, - size_t aio_threshold_, bool blocks_are_granules_size_ = false) - : min_compress_block_size(global_settings.min_compress_block_size) - , max_compress_block_size(global_settings.min_compress_block_size) + MergeTreeWriterSettings( + const Settings & global_settings, + const MergeTreeSettingsPtr & storage_settings, + bool can_use_adaptive_granularity_, + size_t aio_threshold_, + bool blocks_are_granules_size_ = false) + : min_compress_block_size( + storage_settings->min_compress_block_size ? storage_settings->min_compress_block_size : global_settings.min_compress_block_size) + , max_compress_block_size( + storage_settings->max_compress_block_size ? storage_settings->max_compress_block_size + : global_settings.max_compress_block_size) , aio_threshold(aio_threshold_) , can_use_adaptive_granularity(can_use_adaptive_granularity_) - , blocks_are_granules_size(blocks_are_granules_size_) {} + , blocks_are_granules_size(blocks_are_granules_size_) + { + } size_t min_compress_block_size; size_t max_compress_block_size; diff --git a/src/Storages/MergeTree/MergeTreeSettings.h b/src/Storages/MergeTree/MergeTreeSettings.h index b2889af4f11..a0466052122 100644 --- a/src/Storages/MergeTree/MergeTreeSettings.h +++ b/src/Storages/MergeTree/MergeTreeSettings.h @@ -17,6 +17,8 @@ struct Settings; #define LIST_OF_MERGE_TREE_SETTINGS(M) \ + M(UInt64, min_compress_block_size, 0, "The actual size of the block to compress of each table, if this setting doesn't set, using corresponding global setting.", 0) \ + M(UInt64, max_compress_block_size, 0, "The maximum size of blocks of uncompressed data of each table before compressing for writing to a table, is this setting doesn't set, using correspondin global setting.", 0) \ M(UInt64, index_granularity, 8192, "How many rows correspond to one primary key value.", 0) \ \ /** Data storing format settings. */ \ diff --git a/src/Storages/MergeTree/MergedBlockOutputStream.cpp b/src/Storages/MergeTree/MergedBlockOutputStream.cpp index 42d310da485..00a4c37c60d 100644 --- a/src/Storages/MergeTree/MergedBlockOutputStream.cpp +++ b/src/Storages/MergeTree/MergedBlockOutputStream.cpp @@ -48,6 +48,7 @@ MergedBlockOutputStream::MergedBlockOutputStream( { MergeTreeWriterSettings writer_settings( storage.global_context.getSettings(), + storage.getSettings(), data_part->index_granularity_info.is_adaptive, aio_threshold, blocks_are_granules_size); diff --git a/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp b/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp index 8ce4ea126ed..12b14d13656 100644 --- a/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp +++ b/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp @@ -22,8 +22,11 @@ MergedColumnOnlyOutputStream::MergedColumnOnlyOutputStream( , header(header_) { const auto & global_settings = data_part->storage.global_context.getSettings(); + const auto & storage_settings = data_part->storage.getSettings(); + MergeTreeWriterSettings writer_settings( global_settings, + storage_settings, index_granularity_info ? index_granularity_info->is_adaptive : data_part->storage.canUseAdaptiveGranularity(), global_settings.min_bytes_to_use_direct_io); From f4204b10ead7cfa3c9c27098b4a8a05cdfaff38c Mon Sep 17 00:00:00 2001 From: PHO Date: Wed, 4 Nov 2020 16:40:06 +0900 Subject: [PATCH 080/174] Fix build with GCC 9.x Use -std=gnu++2a regardless of whether UNBUNDLED or not, because __int128 is a non-standard builtin and GCC, unlike Clang, only considers it to be an integral type when GNU extensions are enabled. --- CMakeLists.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e3ddfb0c7ad..1f9b700a7d2 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -223,8 +223,8 @@ if (ARCH_NATIVE) set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native") endif () -if (UNBUNDLED AND (COMPILER_GCC OR COMPILER_CLANG)) - # to make numeric_limits<__int128> works for unbundled build +if (COMPILER_GCC OR COMPILER_CLANG) + # to make numeric_limits<__int128> works with GCC set (_CXX_STANDARD "-std=gnu++2a") else() set (_CXX_STANDARD "-std=c++2a") From d014ac4ce276574c1f0284b3f58ad92b096a95aa Mon Sep 17 00:00:00 2001 From: PHO Date: Wed, 4 Nov 2020 16:46:24 +0900 Subject: [PATCH 081/174] Provide a default constructor for wide::integer The implicit one provided by compiler does not initialize its member variable "items" because it's a POD array. GCC emits a warning about this. --- base/common/wide_integer.h | 3 +-- base/common/wide_integer_impl.h | 5 +++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/base/common/wide_integer.h b/base/common/wide_integer.h index 61d88bdcaf3..c9d1eaa32aa 100644 --- a/base/common/wide_integer.h +++ b/base/common/wide_integer.h @@ -58,8 +58,7 @@ public: using signed_base_type = int64_t; // ctors - integer() = default; - + constexpr integer() noexcept; template constexpr integer(T rhs) noexcept; template diff --git a/base/common/wide_integer_impl.h b/base/common/wide_integer_impl.h index 2a889819c11..9a80660ea85 100644 --- a/base/common/wide_integer_impl.h +++ b/base/common/wide_integer_impl.h @@ -916,6 +916,11 @@ public: // Members +template +constexpr integer::integer() noexcept + : items{} +{} + template template constexpr integer::integer(T rhs) noexcept From 8ce936af2c15d4f7bf98cea8ad8206fa3099fbea Mon Sep 17 00:00:00 2001 From: PHO Date: Wed, 4 Nov 2020 16:49:41 +0900 Subject: [PATCH 082/174] Add a missing #include GCC fails to compile this due to std::unique_ptr not being declared. --- src/Common/RadixSort.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/Common/RadixSort.h b/src/Common/RadixSort.h index f490df3e4bb..f3a19068f5a 100644 --- a/src/Common/RadixSort.h +++ b/src/Common/RadixSort.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include From 3c900651586e93435e36a97c86b828ae1e25357c Mon Sep 17 00:00:00 2001 From: PHO Date: Mon, 16 Nov 2020 12:29:29 +0900 Subject: [PATCH 083/174] Add functions toMJD and fromMJD These functions convert between Proleptic Gregorian calendar date and Modified Julian Day number. --- .../functions/date-time-functions.md | 144 ++++++ src/Functions/GregorianDate.h | 431 ++++++++++++++++++ src/Functions/fromMJD.cpp | 248 ++++++++++ src/Functions/registerFunctionsDateTime.cpp | 4 + src/Functions/toMJD.cpp | 241 ++++++++++ .../queries/0_stateless/01543_toMJD.reference | 18 + tests/queries/0_stateless/01543_toMJD.sql | 38 ++ .../0_stateless/01544_fromMJD.reference | 14 + tests/queries/0_stateless/01544_fromMJD.sql | 26 ++ 9 files changed, 1164 insertions(+) create mode 100644 src/Functions/GregorianDate.h create mode 100644 src/Functions/fromMJD.cpp create mode 100644 src/Functions/toMJD.cpp create mode 100644 tests/queries/0_stateless/01543_toMJD.reference create mode 100644 tests/queries/0_stateless/01543_toMJD.sql create mode 100644 tests/queries/0_stateless/01544_fromMJD.reference create mode 100644 tests/queries/0_stateless/01544_fromMJD.sql diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index 75db3fafe36..f7504d9efcb 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -692,3 +692,147 @@ SELECT FROM_UNIXTIME(1234334543, '%Y-%m-%d %R:%S') AS DateTime │ 2009-02-11 14:42:23 │ └─────────────────────┘ ``` + +## toMJD {#tomjd} + +Converts a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD` to a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number in Int32. This function supports date from `0000-01-01` to `9999-12-31`. It raises an exception if the argument cannot be parsed as a date, or the date is invalid. + +**Syntax** + +``` sql +toMJD(date) +``` + +**Parameters** + +- `date` — Date in text form. [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md). + +**Returned value** + +- Modified Julian Day number. + +Type: [Int32](../../sql-reference/data-types/int-uint.md). + +**Example** + +Query: + +``` sql +SELECT toMJD('2020-01-01'); +``` + +Result: + +``` text +┌─toMJD('2020-01-01')─┐ +│ 58849 │ +└─────────────────────┘ +``` + +## toMJDOrNull {#tomjdornull} + +Similar to [toMJD()](#tomjd), but instead of raising exceptions it returns `NULL`. + +**Syntax** + +``` sql +toMJDOrNull(date) +``` + +**Parameters** + +- `date` — Date in text form. [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md). + +**Returned value** + +- Modified Julian Day number. + +Type: [Nullable(Int32)](../../sql-reference/data-types/int-uint.md). + +**Example** + +Query: + +``` sql +SELECT toMJDOrNull('2020-01-01'); +``` + +Result: + +``` text +┌─toMJDOrNull('2020-01-01')─┐ +│ 58849 │ +└───────────────────────────┘ +``` + +## fromMJD {#frommjd} + +Converts a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number to a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD`. This function supports day number from `-678941` to `2973119` (which represent 0000-01-01 and 9999-12-31 respectively). It raises an exception if the day number is outside of the supported range. + +**Syntax** + +``` sql +fromMJD(day) +``` + +**Parameters** + +- `day` — Modified Julian Day number. [Any integral types](../../sql-reference/data-types/int-uint.md). + +**Returned value** + +- Date in text form. + +Type: [String](../../sql-reference/data-types/string.md) + +**Example** + +Query: + +``` sql +SELECT fromMJD(58849); +``` + +Result: + +``` text +┌─fromMJD(58849)─┐ +│ 2020-01-01 │ +└────────────────┘ +``` + +## fromMJDOrNull {#frommjdornull} + +Similar to [fromMJD()](#frommjd), but instead of raising exceptions it returns `NULL`. + +**Syntax** + +``` sql +fromMJDOrNull(day) +``` + +**Parameters** + +- `day` — Modified Julian Day number. [Any integral types](../../sql-reference/data-types/int-uint.md). + +**Returned value** + +- Date in text form. + +Type: [Nullable(String)](../../sql-reference/data-types/string.md) + +**Example** + +Query: + +``` sql +SELECT fromMJDOrNull(58849); +``` + +Result: + +``` text +┌─fromMJDOrNull(58849)─┐ +│ 2020-01-01 │ +└──────────────────────┘ +``` diff --git a/src/Functions/GregorianDate.h b/src/Functions/GregorianDate.h new file mode 100644 index 00000000000..ae548ae128f --- /dev/null +++ b/src/Functions/GregorianDate.h @@ -0,0 +1,431 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#include + +namespace DB +{ + namespace ErrorCodes + { + extern const int CANNOT_PARSE_INPUT_ASSERTION_FAILED; + extern const int CANNOT_PARSE_DATE; + extern const int CANNOT_FORMAT_DATETIME; + extern const int LOGICAL_ERROR; + } + + /** Proleptic Gregorian calendar date. YearT is an integral type + * which should be at least 32 bits wide, and should preferably + * be signed. + */ + template + class GregorianDate + { + public: + GregorianDate() = delete; + + /** Construct from date in text form 'YYYY-MM-DD' by reading from + * ReadBuffer. + */ + GregorianDate(ReadBuffer & in); + + /** Construct from Modified Julian Day. The type T is an + * integral type which should be at least 32 bits wide, and + * should preferably signed. + */ + template ()> * = nullptr> + GregorianDate(T mjd); + + /** Convert to Modified Julian Day. The type T is an integral type + * which should be at least 32 bits wide, and should preferably + * signed. + */ + template ()> * = nullptr> + T toMJD() const; + + /** Write the date in text form 'YYYY-MM-DD' to a buffer. + */ + void write(WriteBuffer & buf) const; + + /** Convert to a string in text form 'YYYY-MM-DD'. + */ + std::string toString() const; + + YearT year() const noexcept + { + return year_; + } + + uint8_t month() const noexcept + { + return month_; + } + + uint8_t dayOfMonth() const noexcept + { + return dayOfMonth_; + } + + private: + YearT year_; + uint8_t month_; + uint8_t dayOfMonth_; + }; + + /** ISO 8601 Ordinal Date. YearT is an integral type which should + * be at least 32 bits wide, and should preferably signed. + */ + template + class OrdinalDate + { + public: + OrdinalDate(YearT year, uint16_t dayOfYear); + + /** Construct from Modified Julian Day. The type T is an + * integral type which should be at least 32 bits wide, and + * should preferably signed. + */ + template ()> * = nullptr> + OrdinalDate(T mjd); + + /** Convert to Modified Julian Day. The type T is an integral + * type which should be at least 32 bits wide, and should + * preferably be signed. + */ + template ()> * = nullptr> + T toMJD() const noexcept; + + YearT year() const noexcept + { + return year_; + } + + uint16_t dayOfYear() const noexcept + { + return dayOfYear_; + } + + private: + YearT year_; + uint16_t dayOfYear_; + }; + + class MonthDay + { + public: + /** Construct from month and day. */ + MonthDay(uint8_t month, uint8_t dayOfMonth); + + /** Construct from day of year in Gregorian or Julian + * calendars to month and day. + */ + MonthDay(bool isLeapYear, uint16_t dayOfYear); + + /** Convert month and day in Gregorian or Julian calendars to + * day of year. + */ + uint16_t dayOfYear(bool isLeapYear) const; + + uint8_t month() const noexcept + { + return month_; + } + + uint8_t dayOfMonth() const noexcept + { + return dayOfMonth_; + } + + private: + uint8_t month_; + uint8_t dayOfMonth_; + }; +} + +/* Implementation */ + +namespace gd { + using namespace DB; + + template + static inline constexpr bool isLeapYear(YearT year) + { + return ( (year % 4 == 0) + && + ( (year % 400 == 0) + || + (! (year % 100 == 0)) ) ); + } + + static inline constexpr uint8_t monthLength(bool isLeapYear, uint8_t month) + { + switch (month) + { + case 1: return 31; + case 2: return isLeapYear ? 29 : 28; + case 3: return 31; + case 4: return 30; + case 5: return 31; + case 6: return 30; + case 7: return 31; + case 8: return 31; + case 9: return 30; + case 10: return 31; + case 11: return 30; + case 12: return 31; + default: + std::terminate(); + } + } + + /** Integer division truncated toward negative infinity. + */ + template + static inline constexpr I div(I x, J y) + { + const auto y_ = static_cast(y); + if (x > 0 && y_ < 0) { + return ((x - 1) / y_) - 1; + } + else if (x < 0 && y_ > 0) { + return ((x + 1) / y_) - 1; + } + else { + return x / y_; + } + } + + /** Integer modulus, satisfying div(x, y)*y + mod(x, y) == x. + */ + template + static inline constexpr I mod(I x, J y) + { + const auto y_ = static_cast(y); + const auto r = x % y_; + if ((x > 0 && y_ < 0) || (x < 0 && y_ > 0)) { + return r == 0 ? static_cast(0) : r + y_; + } + else { + return r; + } + } + + /** Like std::min(), but the type of operands may differ. + */ + template + static inline constexpr I min(I x, J y) + { + const auto y_ = static_cast(y); + return x < y_ ? x : y_; + } + + static inline char readDigit(ReadBuffer & in) + { + char c; + if (!in.read(c)) { + throw Exception( + "Cannot parse input: expected a digit at the end of stream", + ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); + } + else if (c < '0' || c > '9') { + throw Exception( + "Cannot read input: expected a digit but got something else", + ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); + } + else { + return c - '0'; + } + } +} + +namespace DB +{ + template + GregorianDate::GregorianDate(ReadBuffer & in) + { + year_ = gd::readDigit(in) * 1000 + + gd::readDigit(in) * 100 + + gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertChar('-', in); + + month_ = gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertChar('-', in); + + dayOfMonth_ = gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertEOF(in); + + if (month_ < 1 || month_ > 12 || + dayOfMonth_ < 1 || dayOfMonth_ > gd::monthLength(gd::isLeapYear(year_), month_)) + { + throw Exception( + "Invalid date: " + toString(), ErrorCodes::CANNOT_PARSE_DATE); + } + } + + template + template ()> *> + GregorianDate::GregorianDate(T mjd) + { + const OrdinalDate ord(mjd); + const MonthDay md(gd::isLeapYear(ord.year()), ord.dayOfYear()); + year_ = ord.year(); + month_ = md.month(); + dayOfMonth_ = md.dayOfMonth(); + } + + template + template ()> *> + T GregorianDate::toMJD() const + { + const MonthDay md(month_, dayOfMonth_); + const auto dayOfYear = md.dayOfYear(gd::isLeapYear(year_)); + const OrdinalDate ord(year_, dayOfYear); + return ord.template toMJD(); + } + + template + void GregorianDate::write(WriteBuffer & buf) const + { + if (year_ < 0 || year_ > 9999) + { + throw Exception( + "Impossible to stringify: year too big or small: " + DB::toString(year_), + ErrorCodes::CANNOT_FORMAT_DATETIME); + } + else { + auto y = year_; + writeChar('0' + y / 1000, buf); y %= 1000; + writeChar('0' + y / 100, buf); y %= 100; + writeChar('0' + y / 10, buf); y %= 10; + writeChar('0' + y , buf);; + + writeChar('-', buf); + + auto m = month_; + writeChar('0' + m / 10, buf); m %= 10; + writeChar('0' + m , buf); + + writeChar('-', buf); + + auto d = dayOfMonth_; + writeChar('0' + d / 10, buf); d %= 10; + writeChar('0' + d , buf); + } + } + + template + std::string GregorianDate::toString() const + { + WriteBufferFromOwnString buf; + write(buf); + return buf.str(); + } + + template + OrdinalDate::OrdinalDate(YearT year, uint16_t dayOfYear) + : year_(year) + , dayOfYear_(dayOfYear) + { + if (dayOfYear < 1 || dayOfYear > (gd::isLeapYear(year) ? 366 : 365)) + { + throw Exception( + "Invalid ordinal date: " + toString(year) + "-" + toString(dayOfYear), + ErrorCodes::LOGICAL_ERROR); + } + } + + template + template ()> *> + OrdinalDate::OrdinalDate(T mjd) + { + const auto a = mjd + 678575; + const auto quad_cent = gd::div(a, 146097); + const auto b = gd::mod(a, 146097); + const auto cent = gd::min(gd::div(b, 36524), 3); + const auto c = b - cent * 36524; + const auto quad = gd::div(c, 1461); + const auto d = gd::mod(c, 1461); + const auto y = gd::min(gd::div(d, 365), 3); + dayOfYear_ = d - y * 365 + 1; + year_ = quad_cent * 400 + cent * 100 + quad * 4 + y + 1; + } + + template + template ()> *> + T OrdinalDate::toMJD() const noexcept + { + const auto y = year_ - 1; + return dayOfYear_ + + 365 * y + + gd::div(y, 4) + - gd::div(y, 100) + + gd::div(y, 400) + - 678576; + } + + inline MonthDay::MonthDay(uint8_t month, uint8_t dayOfMonth) + : month_(month) + , dayOfMonth_(dayOfMonth) + { + if (month < 1 || month > 12) { + throw Exception( + "Invalid month: " + DB::toString(month), + ErrorCodes::LOGICAL_ERROR); + } + /* We can't validate dayOfMonth here, because we don't know if + * it's a leap year. */ + } + + inline MonthDay::MonthDay(bool isLeapYear, uint16_t dayOfYear) + { + if (dayOfYear < 1 || dayOfYear > (isLeapYear ? 366 : 365)) + { + throw Exception( + std::string("Invalid day of year: ") + + (isLeapYear ? "leap, " : "non-leap, ") + DB::toString(dayOfYear), + ErrorCodes::LOGICAL_ERROR); + } + + month_ = 1; + uint16_t d = dayOfYear; + while (true) { + const auto len = gd::monthLength(isLeapYear, month_); + if (d > len) + { + month_++; + d -= len; + } + else { + break; + } + } + dayOfMonth_ = d; + } + + inline uint16_t MonthDay::dayOfYear(bool isLeapYear) const + { + if (dayOfMonth_ < 1 || dayOfMonth_ > gd::monthLength(isLeapYear, month_)) + { + throw Exception( + std::string("Invalid day of month: ") + + (isLeapYear ? "leap, " : "non-leap, ") + DB::toString(month_) + + "-" + DB::toString(dayOfMonth_), + ErrorCodes::LOGICAL_ERROR); + } + const auto k = month_ <= 2 ? 0 + : isLeapYear ? -1 + : -2; + return (367 * month_ - 362) / 12 + k + dayOfMonth_; + } +} diff --git a/src/Functions/fromMJD.cpp b/src/Functions/fromMJD.cpp new file mode 100644 index 00000000000..f6858c7fb25 --- /dev/null +++ b/src/Functions/fromMJD.cpp @@ -0,0 +1,248 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + template + class ExecutableFunctionFromMJD : public IExecutableFunctionImpl + { + public: + String getName() const override + { + return Name::name; + } + + ColumnPtr execute(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override + { + using ColVecType = typename FromDataType::ColumnType; + const ColVecType * col_from = checkAndGetColumn(arguments[0].column.get()); + const typename ColVecType::Container & vec_from = col_from->getData(); + + auto col_to = ColumnString::create(); + ColumnString::Chars & data_to = col_to->getChars(); + ColumnString::Offsets & offsets_to = col_to->getOffsets(); + data_to.resize(input_rows_count * strlen("YYYY-MM-DD") + 1); + offsets_to.resize(input_rows_count); + + ColumnUInt8::MutablePtr col_null_map_to; + ColumnUInt8::Container * vec_null_map_to [[maybe_unused]] = nullptr; + if constexpr (nullOnErrors) + { + col_null_map_to = ColumnUInt8::create(input_rows_count); + vec_null_map_to = &col_null_map_to->getData(); + } + + WriteBufferFromVector write_buffer(data_to); + for (size_t i = 0; i < input_rows_count; ++i) + { + if constexpr (nullOnErrors) + { + try + { + const GregorianDate<> gd(vec_from[i]); + gd.write(write_buffer); + (*vec_null_map_to)[i] = false; + } + catch (const Exception & e) + { + if (e.code() == ErrorCodes::CANNOT_FORMAT_DATETIME) + { + (*vec_null_map_to)[i] = true; + } + else + { + throw; + } + } + writeChar(0, write_buffer); + offsets_to[i] = write_buffer.count(); + } + else { + const GregorianDate<> gd(vec_from[i]); + gd.write(write_buffer); + writeChar(0, write_buffer); + offsets_to[i] = write_buffer.count(); + } + } + write_buffer.finalize(); + + if constexpr (nullOnErrors) + { + return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); + } + else + { + return col_to; + } + } + + bool useDefaultImplementationForConstants() const override + { + return true; + } + }; + + template + class FunctionBaseFromMJD : public IFunctionBaseImpl + { + public: + explicit FunctionBaseFromMJD(DataTypes argument_types_, DataTypePtr return_type_) + : argument_types(std::move(argument_types_)) + , return_type(std::move(return_type_)) {} + + String getName() const override + { + return Name::name; + } + + const DataTypes & getArgumentTypes() const override + { + return argument_types; + } + + const DataTypePtr & getResultType() const override + { + return return_type; + } + + ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override + { + return std::make_unique>(); + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + + bool hasInformationAboutMonotonicity() const override + { + return true; + } + + Monotonicity getMonotonicityForRange(const IDataType &, const Field &, const Field &) const override + { + return Monotonicity( + true, // is_monotonic + true, // is_positive + true); // is_always_monotonic + } + + private: + DataTypes argument_types; + DataTypePtr return_type; + }; + + template + class FromMJDOverloadResolver : public IFunctionOverloadResolverImpl + { + public: + static constexpr auto name = Name::name; + + static FunctionOverloadResolverImplPtr create(const Context &) + { + return std::make_unique>(); + } + + String getName() const override + { + return Name::name; + } + + FunctionBaseImplPtr build(const ColumnsWithTypeAndName & arguments, const DataTypePtr & return_type) const override + { + const DataTypePtr & from_type = arguments[0].type; + DataTypes argument_types = { from_type }; + FunctionBaseImplPtr base; + auto call = [&](const auto & types) -> bool + { + using Types = std::decay_t; + using FromIntType = typename Types::RightType; + using FromDataType = DataTypeNumber; + + base = std::make_unique>(argument_types, return_type); + return true; + }; + bool built = callOnBasicType(from_type->getTypeId(), call); + if (built) + { + return base; + } + else + { + /* When the argument is a NULL constant, the resulting + * function base will not be actually called but it + * will still be inspected. Returning a NULL pointer + * here causes a SEGV. So we must somehow create a + * dummy implementation and return it. + */ + if (WhichDataType(from_type).isNullable()) // Nullable(Nothing) + { + return std::make_unique>(argument_types, return_type); + } + else { + // Should not happen. + throw Exception( + "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + } + } + + DataTypePtr getReturnType(const DataTypes & arguments) const override + { + if (!isInteger(arguments[0])) + { + throw Exception( + "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + + DataTypePtr baseType = std::make_shared(); + if constexpr (nullOnErrors) + { + return std::make_shared(baseType); + } + else + { + return baseType; + } + } + + size_t getNumberOfArguments() const override + { + return 1; + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + }; + + struct NameFromMJD + { + static constexpr auto name = "fromMJD"; + }; + + struct NameFromMJDOrNull + { + static constexpr auto name = "fromMJDOrNull"; + }; + + void registerFunctionFromMJD(FunctionFactory & factory) + { + factory.registerFunction>(); + factory.registerFunction>(); + } +} diff --git a/src/Functions/registerFunctionsDateTime.cpp b/src/Functions/registerFunctionsDateTime.cpp index e95ab230087..cc329241ead 100644 --- a/src/Functions/registerFunctionsDateTime.cpp +++ b/src/Functions/registerFunctionsDateTime.cpp @@ -18,6 +18,7 @@ void registerFunctionToMonday(FunctionFactory &); void registerFunctionToISOWeek(FunctionFactory &); void registerFunctionToISOYear(FunctionFactory &); void registerFunctionToCustomWeek(FunctionFactory &); +void registerFunctionToMJD(FunctionFactory &); void registerFunctionToStartOfMonth(FunctionFactory &); void registerFunctionToStartOfQuarter(FunctionFactory &); void registerFunctionToStartOfYear(FunctionFactory &); @@ -65,6 +66,7 @@ void registerFunctionSubtractYears(FunctionFactory &); void registerFunctionDateDiff(FunctionFactory &); void registerFunctionToTimeZone(FunctionFactory &); void registerFunctionFormatDateTime(FunctionFactory &); +void registerFunctionFromMJD(FunctionFactory &); void registerFunctionDateTrunc(FunctionFactory &); void registerFunctionsDateTime(FunctionFactory & factory) @@ -83,6 +85,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionToISOWeek(factory); registerFunctionToISOYear(factory); registerFunctionToCustomWeek(factory); + registerFunctionToMJD(factory); registerFunctionToStartOfMonth(factory); registerFunctionToStartOfQuarter(factory); registerFunctionToStartOfYear(factory); @@ -131,6 +134,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionDateDiff(factory); registerFunctionToTimeZone(factory); registerFunctionFormatDateTime(factory); + registerFunctionFromMJD(factory); registerFunctionDateTrunc(factory); } diff --git a/src/Functions/toMJD.cpp b/src/Functions/toMJD.cpp new file mode 100644 index 00000000000..841b477774b --- /dev/null +++ b/src/Functions/toMJD.cpp @@ -0,0 +1,241 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + namespace ErrorCodes + { + extern const int ILLEGAL_COLUMN; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + } + + template + class ExecutableFunctionToMJD : public IExecutableFunctionImpl + { + public: + String getName() const override + { + return Name::name; + } + + ColumnPtr execute(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override + { + const IColumn * col_from = arguments[0].column.get(); + const ColumnString * col_from_string = checkAndGetColumn(col_from); + const ColumnFixedString * col_from_fixed_string = checkAndGetColumn(col_from); + + const ColumnString::Chars * chars = nullptr; + const IColumn::Offsets * offsets = nullptr; + size_t fixed_string_size = 0; + + if (col_from_string) + { + chars = &col_from_string->getChars(); + offsets = &col_from_string->getOffsets(); + } + else if (col_from_fixed_string) + { + chars = &col_from_fixed_string->getChars(); + fixed_string_size = col_from_fixed_string->getN(); + } + else + { + throw Exception("Illegal column " + col_from->getName() + + " of first argument of function " + Name::name, + ErrorCodes::ILLEGAL_COLUMN); + } + + using ColVecTo = typename ToDataType::ColumnType; + typename ColVecTo::MutablePtr col_to = ColVecTo::create(input_rows_count); + typename ColVecTo::Container & vec_to = col_to->getData(); + + ColumnUInt8::MutablePtr col_null_map_to; + ColumnUInt8::Container * vec_null_map_to [[maybe_unused]] = nullptr; + if constexpr (nullOnErrors) + { + col_null_map_to = ColumnUInt8::create(input_rows_count); + vec_null_map_to = &col_null_map_to->getData(); + } + + size_t current_offset = 0; + for (size_t i = 0; i < input_rows_count; ++i) + { + const size_t next_offset = offsets ? (*offsets)[i] : current_offset + fixed_string_size; + const size_t string_size = offsets ? next_offset - current_offset - 1 : fixed_string_size; + ReadBufferFromMemory read_buffer(&(*chars)[current_offset], string_size); + current_offset = next_offset; + + if constexpr (nullOnErrors) + { + try + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toMJD(); + (*vec_null_map_to)[i] = false; + } + catch (const Exception & e) + { + if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || + e.code() == ErrorCodes::CANNOT_PARSE_DATE) + { + (*vec_null_map_to)[i] = true; + } + else + { + throw; + } + } + } + else + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toMJD(); + } + } + + if constexpr (nullOnErrors) + { + return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); + } + else + { + return col_to; + } + } + + bool useDefaultImplementationForConstants() const override + { + return true; + } + }; + + template + class FunctionBaseToMJD : public IFunctionBaseImpl + { + public: + explicit FunctionBaseToMJD(DataTypes argument_types_, DataTypePtr return_type_) + : argument_types(std::move(argument_types_)) + , return_type(std::move(return_type_)) {} + + String getName() const override + { + return Name::name; + } + + const DataTypes & getArgumentTypes() const override + { + return argument_types; + } + + const DataTypePtr & getResultType() const override + { + return return_type; + } + + ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override + { + return std::make_unique>(); + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + + bool hasInformationAboutMonotonicity() const override + { + return true; + } + + Monotonicity getMonotonicityForRange(const IDataType &, const Field &, const Field &) const override + { + return Monotonicity( + true, // is_monotonic + true, // is_positive + true); // is_always_monotonic + } + + private: + DataTypes argument_types; + DataTypePtr return_type; + }; + + template + class ToMJDOverloadResolver : public IFunctionOverloadResolverImpl + { + public: + static constexpr auto name = Name::name; + + static FunctionOverloadResolverImplPtr create(const Context &) + { + return std::make_unique>(); + } + + String getName() const override + { + return Name::name; + } + + FunctionBaseImplPtr build(const ColumnsWithTypeAndName & arguments, const DataTypePtr & return_type) const override + { + DataTypes argument_types = { arguments[0].type }; + + return std::make_unique>(argument_types, return_type); + } + + DataTypePtr getReturnType(const DataTypes & arguments) const override + { + if (!isStringOrFixedString(arguments[0])) + { + throw Exception( + "The argument of function " + getName() + " must be String or FixedString", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + + DataTypePtr baseType = std::make_shared(); + if constexpr (nullOnErrors) + { + return std::make_shared(baseType); + } + else + { + return baseType; + } + } + + size_t getNumberOfArguments() const override + { + return 1; + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + }; + + struct NameToMJD + { + static constexpr auto name = "toMJD"; + }; + + struct NameToMJDOrNull + { + static constexpr auto name = "toMJDOrNull"; + }; + + void registerFunctionToMJD(FunctionFactory & factory) + { + factory.registerFunction>(); + factory.registerFunction>(); + } +} diff --git a/tests/queries/0_stateless/01543_toMJD.reference b/tests/queries/0_stateless/01543_toMJD.reference new file mode 100644 index 00000000000..fbb668d09e5 --- /dev/null +++ b/tests/queries/0_stateless/01543_toMJD.reference @@ -0,0 +1,18 @@ +Invocation with constant +-1 +0 +59154 +\N +or null +59154 +\N +\N +\N +Invocation with String column +-1 +0 +59154 +Invocation with FixedString column +-1 +0 +59154 diff --git a/tests/queries/0_stateless/01543_toMJD.sql b/tests/queries/0_stateless/01543_toMJD.sql new file mode 100644 index 00000000000..4dcfab98000 --- /dev/null +++ b/tests/queries/0_stateless/01543_toMJD.sql @@ -0,0 +1,38 @@ +-- +SELECT 'Invocation with constant'; + +SELECT toMJD('1858-11-16'); +SELECT toMJD('1858-11-17'); +SELECT toMJD('2020-11-01'); +SELECT toMJD(NULL); +SELECT toMJD('unparsable'); -- { serverError 27 } +SELECT toMJD('1999-02-29'); -- { serverError 38 } +SELECT toMJD('1999-13-32'); -- { serverError 38 } + +SELECT 'or null'; +SELECT toMJDOrNull('2020-11-01'); +SELECT toMJDOrNull('unparsable'); +SELECT toMJDOrNull('1999-02-29'); +SELECT toMJDOrNull('1999-13-32'); + +-- +SELECT 'Invocation with String column'; + +DROP TABLE IF EXISTS toMJD_test; +CREATE TABLE toMJD_test (d String) ENGINE = Memory; + +INSERT INTO toMJD_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toMJD(d) FROM toMJD_test; + +DROP TABLE toMJD_test; + +-- +SELECT 'Invocation with FixedString column'; + +DROP TABLE IF EXISTS toMJD_test; +CREATE TABLE toMJD_test (d FixedString(10)) ENGINE = Memory; + +INSERT INTO toMJD_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toMJD(d) FROM toMJD_test; + +DROP TABLE toMJD_test; diff --git a/tests/queries/0_stateless/01544_fromMJD.reference b/tests/queries/0_stateless/01544_fromMJD.reference new file mode 100644 index 00000000000..6f71b6263c0 --- /dev/null +++ b/tests/queries/0_stateless/01544_fromMJD.reference @@ -0,0 +1,14 @@ +Invocation with constant +1858-11-16 +1858-11-17 +2020-11-01 +\N +or null +2020-11-01 +\N +\N +\N +Invocation with Int32 column +1858-11-16 +1858-11-17 +2020-11-01 diff --git a/tests/queries/0_stateless/01544_fromMJD.sql b/tests/queries/0_stateless/01544_fromMJD.sql new file mode 100644 index 00000000000..4fa2788245b --- /dev/null +++ b/tests/queries/0_stateless/01544_fromMJD.sql @@ -0,0 +1,26 @@ +-- +SELECT 'Invocation with constant'; + +SELECT fromMJD(-1); +SELECT fromMJD(0); +SELECT fromMJD(59154); +SELECT fromMJD(NULL); +SELECT fromMJD(-678942); -- { serverError 490 } +SELECT fromMJD(2973484); -- { serverError 490 } + +SELECT 'or null'; +SELECT fromMJDOrNull(59154); +SELECT fromMJDOrNull(NULL); +SELECT fromMJDOrNull(-678942); +SELECT fromMJDOrNull(2973484); + +-- +SELECT 'Invocation with Int32 column'; + +DROP TABLE IF EXISTS fromMJD_test; +CREATE TABLE fromMJD_test (d Int32) ENGINE = Memory; + +INSERT INTO fromMJD_test VALUES (-1), (0), (59154); +SELECT fromMJD(d) FROM fromMJD_test; + +DROP TABLE fromMJD_test; From af9d9a7b9c3a4b89d408775679866ba2ffb27ac4 Mon Sep 17 00:00:00 2001 From: filimonov <1549571+filimonov@users.noreply.github.com> Date: Mon, 7 Dec 2020 14:01:12 +0100 Subject: [PATCH 084/174] dict notes sync docs with https://github.com/ClickHouse/ClickHouse/pull/8127 --- .../external-dicts-dict-sources.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md index 957f2b6ae53..e86ac7fe105 100644 --- a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md +++ b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md @@ -93,6 +93,8 @@ Setting fields: - `path` – The absolute path to the file. - `format` – The file format. All the formats described in “[Formats](../../../interfaces/formats.md#formats)” are supported. +When dictionary with FILE source is created via DDL command (`CREATE DICTIONARY ...`), source of the dictionary have to be located in `user_files` directory, to prevent DB users accessing arbitrary file on clickhouse node. + ## Executable File {#dicts-external_dicts_dict_sources-executable} Working with executable files depends on [how the dictionary is stored in memory](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file’s STDIN. Otherwise, ClickHouse starts executable file and treats its output as dictionary data. @@ -108,17 +110,13 @@ Example of settings: ``` -or - -``` sql -SOURCE(EXECUTABLE(command 'cat /opt/dictionaries/os.tsv' format 'TabSeparated')) -``` - Setting fields: - `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`). - `format` – The file format. All the formats described in “[Formats](../../../interfaces/formats.md#formats)” are supported. +That dictionary source can be configured only via XML configuration. Creating dictionaries with executable source via DDL is disabled, otherwise, the DB user would be able to execute arbitrary binary on clickhouse node. + ## Http(s) {#dicts-external_dicts_dict_sources-http} Working with an HTTP(s) server depends on [how the dictionary is stored in memory](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method. @@ -169,6 +167,8 @@ Setting fields: - `name` – Identifiant name used for the header send on the request. - `value` – Value set for a specific identifiant name. +When creating a dictionary using the DDL command (`CREATE DICTIONARY ...`) remote hosts for HTTP dictionaries checked with the `remote_url_allow_hosts` section from config to prevent database users to access arbitrary HTTP server. + ## ODBC {#dicts-external_dicts_dict_sources-odbc} You can use this method to connect any database that has an ODBC driver. From 50b64ba1f84e5f117cc306b6c6d0901b35dada70 Mon Sep 17 00:00:00 2001 From: feng lv Date: Mon, 7 Dec 2020 13:18:31 +0000 Subject: [PATCH 085/174] fix --- src/Storages/MergeTree/MergeTreeSettings.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Storages/MergeTree/MergeTreeSettings.h b/src/Storages/MergeTree/MergeTreeSettings.h index a0466052122..5941e18506b 100644 --- a/src/Storages/MergeTree/MergeTreeSettings.h +++ b/src/Storages/MergeTree/MergeTreeSettings.h @@ -18,7 +18,7 @@ struct Settings; #define LIST_OF_MERGE_TREE_SETTINGS(M) \ M(UInt64, min_compress_block_size, 0, "The actual size of the block to compress of each table, if this setting doesn't set, using corresponding global setting.", 0) \ - M(UInt64, max_compress_block_size, 0, "The maximum size of blocks of uncompressed data of each table before compressing for writing to a table, is this setting doesn't set, using correspondin global setting.", 0) \ + M(UInt64, max_compress_block_size, 0, "The maximum size of blocks of uncompressed data of each table before compressing for writing to a table, if this setting doesn't set, using correspondin global setting.", 0) \ M(UInt64, index_granularity, 8192, "How many rows correspond to one primary key value.", 0) \ \ /** Data storing format settings. */ \ From b10db16d3b5253b78cbbd97c81292a205cdb5876 Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Mon, 7 Dec 2020 16:23:10 +0300 Subject: [PATCH 086/174] done --- contrib/dragonbox-cmake/CMakeLists.txt | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 contrib/dragonbox-cmake/CMakeLists.txt diff --git a/contrib/dragonbox-cmake/CMakeLists.txt b/contrib/dragonbox-cmake/CMakeLists.txt new file mode 100644 index 00000000000..3422c037837 --- /dev/null +++ b/contrib/dragonbox-cmake/CMakeLists.txt @@ -0,0 +1,5 @@ +set(DRAGONBOX_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/dragonbox/include) + +add_library(dragonbox INTERFACE) + +target_include_directories(dragonbox SYSTEM INTERFACE ${DRAGONBOX_INCLUDE_DIR}) \ No newline at end of file From 303d27f82b74dd26402d57ce36d185580da590e8 Mon Sep 17 00:00:00 2001 From: annvsh Date: Mon, 7 Dec 2020 21:21:37 +0700 Subject: [PATCH 087/174] Fixed --- docs/en/interfaces/formats.md | 6 ++---- docs/ru/interfaces/formats.md | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index c72925811c0..4c5b586935c 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1390,7 +1390,7 @@ The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) forma - strings are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. When an empty value is passed to the `RawBLOB` input, ClickHouse generates an exception: - + ``` text Code: 108. DB::Exception: No data to insert ``` @@ -1398,11 +1398,9 @@ Code: 108. DB::Exception: No data to insert **Example** ``` bash -$ clickhouse-client --query "DROP TABLE IF EXISTS {some_table};" -$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" $ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" $ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum -$ clickhouse-client --query "DROP TABLE {some_table};" ``` Result: diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 92e3be9588f..bd9ca6bb1ad 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1277,7 +1277,7 @@ Code: 108. DB::Exception: No data to insert **Пример** ``` bash -$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" $ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" $ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum ``` From 88435efc1c7efcf6e77d60b5b61fe7b6667c61ed Mon Sep 17 00:00:00 2001 From: annvsh Date: Mon, 7 Dec 2020 22:33:48 +0700 Subject: [PATCH 088/174] Fixed --- docs/en/interfaces/formats.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 4c5b586935c..4d9bea89278 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1373,12 +1373,12 @@ Result: In this format, all input data is read to a single value. It is possible to parse щтдн a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. -Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](# tabseparatedraw). +Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparatedraw). `RawBLOB`: - data is output in binary format, no escaping; - there are no delimiters between values; - no newline at the end of each value. -[TabSeparatedRaw] (# tabseparatedraw): +[TabSeparatedRaw] (#tabseparatedraw): - data is output without screening; - the string contains values ​​separated by tabs; - there is a line feed after the last value in the line. From 50297f7012b43076adb00236def9f32736a9d3fb Mon Sep 17 00:00:00 2001 From: annvsh Date: Mon, 7 Dec 2020 22:35:31 +0700 Subject: [PATCH 089/174] Fixed --- docs/en/interfaces/formats.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 4d9bea89278..9b6164c67e7 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1370,7 +1370,7 @@ Result: ## RawBLOB {#rawblob} -In this format, all input data is read to a single value. It is possible to parse щтдн a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. +In this format, all input data is read to a single value. It is possible to parse only a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparatedraw). From a8616ea2629addf25a1d7c4165e0ac6eb7de4fc9 Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Mon, 7 Dec 2020 18:47:57 +0300 Subject: [PATCH 090/174] better --- src/Functions/GregorianDate.h | 2 +- src/Functions/fromMJD.cpp | 30 +++---- src/Functions/registerFunctionsDateTime.cpp | 8 +- .../{toMJD.cpp => toModifiedJulianDay.cpp} | 88 +++++++++---------- tests/queries/0_stateless/01543_toMJD.sql | 38 -------- ...ce => 01543_toModifiedJulianDay.reference} | 0 .../0_stateless/01543_toModifiedJulianDay.sql | 38 ++++++++ tests/queries/0_stateless/01544_fromMJD.sql | 26 ------ ... => 01544_fromModifiedJulianDay.reference} | 0 .../01544_fromModifiedJulianDay.sql | 26 ++++++ 10 files changed, 128 insertions(+), 128 deletions(-) rename src/Functions/{toMJD.cpp => toModifiedJulianDay.cpp} (69%) delete mode 100644 tests/queries/0_stateless/01543_toMJD.sql rename tests/queries/0_stateless/{01543_toMJD.reference => 01543_toModifiedJulianDay.reference} (100%) create mode 100644 tests/queries/0_stateless/01543_toModifiedJulianDay.sql delete mode 100644 tests/queries/0_stateless/01544_fromMJD.sql rename tests/queries/0_stateless/{01544_fromMJD.reference => 01544_fromModifiedJulianDay.reference} (100%) create mode 100644 tests/queries/0_stateless/01544_fromModifiedJulianDay.sql diff --git a/src/Functions/GregorianDate.h b/src/Functions/GregorianDate.h index ae548ae128f..e5caff3ae46 100644 --- a/src/Functions/GregorianDate.h +++ b/src/Functions/GregorianDate.h @@ -308,7 +308,7 @@ namespace DB writeChar('0' + y / 1000, buf); y %= 1000; writeChar('0' + y / 100, buf); y %= 100; writeChar('0' + y / 10, buf); y %= 10; - writeChar('0' + y , buf);; + writeChar('0' + y , buf); writeChar('-', buf); diff --git a/src/Functions/fromMJD.cpp b/src/Functions/fromMJD.cpp index f6858c7fb25..3f8251e2caf 100644 --- a/src/Functions/fromMJD.cpp +++ b/src/Functions/fromMJD.cpp @@ -16,7 +16,7 @@ namespace DB { template - class ExecutableFunctionFromMJD : public IExecutableFunctionImpl + class ExecutableFunctionFromModifiedJulianDay : public IExecutableFunctionImpl { public: String getName() const override @@ -95,10 +95,10 @@ namespace DB }; template - class FunctionBaseFromMJD : public IFunctionBaseImpl + class FunctionBaseFromModifiedJulianDay : public IFunctionBaseImpl { public: - explicit FunctionBaseFromMJD(DataTypes argument_types_, DataTypePtr return_type_) + explicit FunctionBaseFromModifiedJulianDay(DataTypes argument_types_, DataTypePtr return_type_) : argument_types(std::move(argument_types_)) , return_type(std::move(return_type_)) {} @@ -119,7 +119,7 @@ namespace DB ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override { - return std::make_unique>(); + return std::make_unique>(); } bool isInjective(const ColumnsWithTypeAndName &) const override @@ -146,14 +146,14 @@ namespace DB }; template - class FromMJDOverloadResolver : public IFunctionOverloadResolverImpl + class FromModifiedJulianDayOverloadResolver : public IFunctionOverloadResolverImpl { public: static constexpr auto name = Name::name; static FunctionOverloadResolverImplPtr create(const Context &) { - return std::make_unique>(); + return std::make_unique>(); } String getName() const override @@ -172,7 +172,7 @@ namespace DB using FromIntType = typename Types::RightType; using FromDataType = DataTypeNumber; - base = std::make_unique>(argument_types, return_type); + base = std::make_unique>(argument_types, return_type); return true; }; bool built = callOnBasicType(from_type->getTypeId(), call); @@ -190,7 +190,7 @@ namespace DB */ if (WhichDataType(from_type).isNullable()) // Nullable(Nothing) { - return std::make_unique>(argument_types, return_type); + return std::make_unique>(argument_types, return_type); } else { // Should not happen. @@ -230,19 +230,19 @@ namespace DB } }; - struct NameFromMJD + struct NameFromModifiedJulianDay { - static constexpr auto name = "fromMJD"; + static constexpr auto name = "fromModifiedJulianDay"; }; - struct NameFromMJDOrNull + struct NameFromModifiedJulianDayOrNull { - static constexpr auto name = "fromMJDOrNull"; + static constexpr auto name = "fromModifiedJulianDayOrNull"; }; - void registerFunctionFromMJD(FunctionFactory & factory) + void registerFunctionFromModifiedJulianDay(FunctionFactory & factory) { - factory.registerFunction>(); - factory.registerFunction>(); + factory.registerFunction>(); + factory.registerFunction>(); } } diff --git a/src/Functions/registerFunctionsDateTime.cpp b/src/Functions/registerFunctionsDateTime.cpp index cc329241ead..2cb737b8c75 100644 --- a/src/Functions/registerFunctionsDateTime.cpp +++ b/src/Functions/registerFunctionsDateTime.cpp @@ -18,7 +18,7 @@ void registerFunctionToMonday(FunctionFactory &); void registerFunctionToISOWeek(FunctionFactory &); void registerFunctionToISOYear(FunctionFactory &); void registerFunctionToCustomWeek(FunctionFactory &); -void registerFunctionToMJD(FunctionFactory &); +void registerFunctionToModifiedJulianDay(FunctionFactory &); void registerFunctionToStartOfMonth(FunctionFactory &); void registerFunctionToStartOfQuarter(FunctionFactory &); void registerFunctionToStartOfYear(FunctionFactory &); @@ -66,7 +66,7 @@ void registerFunctionSubtractYears(FunctionFactory &); void registerFunctionDateDiff(FunctionFactory &); void registerFunctionToTimeZone(FunctionFactory &); void registerFunctionFormatDateTime(FunctionFactory &); -void registerFunctionFromMJD(FunctionFactory &); +void registerFunctionFromModifiedJulianDay(FunctionFactory &); void registerFunctionDateTrunc(FunctionFactory &); void registerFunctionsDateTime(FunctionFactory & factory) @@ -85,7 +85,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionToISOWeek(factory); registerFunctionToISOYear(factory); registerFunctionToCustomWeek(factory); - registerFunctionToMJD(factory); + registerFunctionToModifiedJulianDay(factory); registerFunctionToStartOfMonth(factory); registerFunctionToStartOfQuarter(factory); registerFunctionToStartOfYear(factory); @@ -134,7 +134,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionDateDiff(factory); registerFunctionToTimeZone(factory); registerFunctionFormatDateTime(factory); - registerFunctionFromMJD(factory); + registerFunctionFromModifiedJulianDay(factory); registerFunctionDateTrunc(factory); } diff --git a/src/Functions/toMJD.cpp b/src/Functions/toModifiedJulianDay.cpp similarity index 69% rename from src/Functions/toMJD.cpp rename to src/Functions/toModifiedJulianDay.cpp index 841b477774b..bd1ff7183ff 100644 --- a/src/Functions/toMJD.cpp +++ b/src/Functions/toModifiedJulianDay.cpp @@ -20,7 +20,7 @@ namespace DB } template - class ExecutableFunctionToMJD : public IExecutableFunctionImpl + class ExecutableFunctionToModifiedJulianDay : public IExecutableFunctionImpl { public: String getName() const override @@ -70,37 +70,37 @@ namespace DB size_t current_offset = 0; for (size_t i = 0; i < input_rows_count; ++i) { - const size_t next_offset = offsets ? (*offsets)[i] : current_offset + fixed_string_size; - const size_t string_size = offsets ? next_offset - current_offset - 1 : fixed_string_size; - ReadBufferFromMemory read_buffer(&(*chars)[current_offset], string_size); - current_offset = next_offset; + const size_t next_offset = offsets ? (*offsets)[i] : current_offset + fixed_string_size; + const size_t string_size = offsets ? next_offset - current_offset - 1 : fixed_string_size; + ReadBufferFromMemory read_buffer(&(*chars)[current_offset], string_size); + current_offset = next_offset; - if constexpr (nullOnErrors) - { - try - { - const GregorianDate<> date(read_buffer); - vec_to[i] = date.toMJD(); - (*vec_null_map_to)[i] = false; - } - catch (const Exception & e) - { - if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || - e.code() == ErrorCodes::CANNOT_PARSE_DATE) - { - (*vec_null_map_to)[i] = true; - } - else - { - throw; - } - } - } - else - { - const GregorianDate<> date(read_buffer); - vec_to[i] = date.toMJD(); - } + if constexpr (nullOnErrors) + { + try + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toMJD(); + (*vec_null_map_to)[i] = false; + } + catch (const Exception & e) + { + if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || + e.code() == ErrorCodes::CANNOT_PARSE_DATE) + { + (*vec_null_map_to)[i] = true; + } + else + { + throw; + } + } + } + else + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toMJD(); + } } if constexpr (nullOnErrors) @@ -120,10 +120,10 @@ namespace DB }; template - class FunctionBaseToMJD : public IFunctionBaseImpl + class FunctionBaseToModifiedJulianDay : public IFunctionBaseImpl { public: - explicit FunctionBaseToMJD(DataTypes argument_types_, DataTypePtr return_type_) + explicit FunctionBaseToModifiedJulianDay(DataTypes argument_types_, DataTypePtr return_type_) : argument_types(std::move(argument_types_)) , return_type(std::move(return_type_)) {} @@ -144,7 +144,7 @@ namespace DB ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override { - return std::make_unique>(); + return std::make_unique>(); } bool isInjective(const ColumnsWithTypeAndName &) const override @@ -171,14 +171,14 @@ namespace DB }; template - class ToMJDOverloadResolver : public IFunctionOverloadResolverImpl + class ToModifiedJulianDayOverloadResolver : public IFunctionOverloadResolverImpl { public: static constexpr auto name = Name::name; static FunctionOverloadResolverImplPtr create(const Context &) { - return std::make_unique>(); + return std::make_unique>(); } String getName() const override @@ -190,7 +190,7 @@ namespace DB { DataTypes argument_types = { arguments[0].type }; - return std::make_unique>(argument_types, return_type); + return std::make_unique>(argument_types, return_type); } DataTypePtr getReturnType(const DataTypes & arguments) const override @@ -223,19 +223,19 @@ namespace DB } }; - struct NameToMJD + struct NameToModifiedJulianDay { - static constexpr auto name = "toMJD"; + static constexpr auto name = "toModifiedJulianDay"; }; - struct NameToMJDOrNull + struct NameToModifiedJulianDayOrNull { - static constexpr auto name = "toMJDOrNull"; + static constexpr auto name = "toModifiedJulianDayOrNull"; }; - void registerFunctionToMJD(FunctionFactory & factory) + void registerFunctionToModifiedJulianDay(FunctionFactory & factory) { - factory.registerFunction>(); - factory.registerFunction>(); + factory.registerFunction>(); + factory.registerFunction>(); } } diff --git a/tests/queries/0_stateless/01543_toMJD.sql b/tests/queries/0_stateless/01543_toMJD.sql deleted file mode 100644 index 4dcfab98000..00000000000 --- a/tests/queries/0_stateless/01543_toMJD.sql +++ /dev/null @@ -1,38 +0,0 @@ --- -SELECT 'Invocation with constant'; - -SELECT toMJD('1858-11-16'); -SELECT toMJD('1858-11-17'); -SELECT toMJD('2020-11-01'); -SELECT toMJD(NULL); -SELECT toMJD('unparsable'); -- { serverError 27 } -SELECT toMJD('1999-02-29'); -- { serverError 38 } -SELECT toMJD('1999-13-32'); -- { serverError 38 } - -SELECT 'or null'; -SELECT toMJDOrNull('2020-11-01'); -SELECT toMJDOrNull('unparsable'); -SELECT toMJDOrNull('1999-02-29'); -SELECT toMJDOrNull('1999-13-32'); - --- -SELECT 'Invocation with String column'; - -DROP TABLE IF EXISTS toMJD_test; -CREATE TABLE toMJD_test (d String) ENGINE = Memory; - -INSERT INTO toMJD_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); -SELECT toMJD(d) FROM toMJD_test; - -DROP TABLE toMJD_test; - --- -SELECT 'Invocation with FixedString column'; - -DROP TABLE IF EXISTS toMJD_test; -CREATE TABLE toMJD_test (d FixedString(10)) ENGINE = Memory; - -INSERT INTO toMJD_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); -SELECT toMJD(d) FROM toMJD_test; - -DROP TABLE toMJD_test; diff --git a/tests/queries/0_stateless/01543_toMJD.reference b/tests/queries/0_stateless/01543_toModifiedJulianDay.reference similarity index 100% rename from tests/queries/0_stateless/01543_toMJD.reference rename to tests/queries/0_stateless/01543_toModifiedJulianDay.reference diff --git a/tests/queries/0_stateless/01543_toModifiedJulianDay.sql b/tests/queries/0_stateless/01543_toModifiedJulianDay.sql new file mode 100644 index 00000000000..4cc0813c8cc --- /dev/null +++ b/tests/queries/0_stateless/01543_toModifiedJulianDay.sql @@ -0,0 +1,38 @@ +-- +SELECT 'Invocation with constant'; + +SELECT toModifiedJulianDay('1858-11-16'); +SELECT toModifiedJulianDay('1858-11-17'); +SELECT toModifiedJulianDay('2020-11-01'); +SELECT toModifiedJulianDay(NULL); +SELECT toModifiedJulianDay('unparsable'); -- { serverError 27 } +SELECT toModifiedJulianDay('1999-02-29'); -- { serverError 38 } +SELECT toModifiedJulianDay('1999-13-32'); -- { serverError 38 } + +SELECT 'or null'; +SELECT toModifiedJulianDayOrNull('2020-11-01'); +SELECT toModifiedJulianDayOrNull('unparsable'); +SELECT toModifiedJulianDayOrNull('1999-02-29'); +SELECT toModifiedJulianDayOrNull('1999-13-32'); + +-- +SELECT 'Invocation with String column'; + +DROP TABLE IF EXISTS toModifiedJulianDay_test; +CREATE TABLE toModifiedJulianDay_test (d String) ENGINE = Memory; + +INSERT INTO toModifiedJulianDay_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toModifiedJulianDay(d) FROM toModifiedJulianDay_test; + +DROP TABLE toModifiedJulianDay_test; + +-- +SELECT 'Invocation with FixedString column'; + +DROP TABLE IF EXISTS toModifiedJulianDay_test; +CREATE TABLE toModifiedJulianDay_test (d FixedString(10)) ENGINE = Memory; + +INSERT INTO toModifiedJulianDay_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toModifiedJulianDay(d) FROM toModifiedJulianDay_test; + +DROP TABLE toModifiedJulianDay_test; diff --git a/tests/queries/0_stateless/01544_fromMJD.sql b/tests/queries/0_stateless/01544_fromMJD.sql deleted file mode 100644 index 4fa2788245b..00000000000 --- a/tests/queries/0_stateless/01544_fromMJD.sql +++ /dev/null @@ -1,26 +0,0 @@ --- -SELECT 'Invocation with constant'; - -SELECT fromMJD(-1); -SELECT fromMJD(0); -SELECT fromMJD(59154); -SELECT fromMJD(NULL); -SELECT fromMJD(-678942); -- { serverError 490 } -SELECT fromMJD(2973484); -- { serverError 490 } - -SELECT 'or null'; -SELECT fromMJDOrNull(59154); -SELECT fromMJDOrNull(NULL); -SELECT fromMJDOrNull(-678942); -SELECT fromMJDOrNull(2973484); - --- -SELECT 'Invocation with Int32 column'; - -DROP TABLE IF EXISTS fromMJD_test; -CREATE TABLE fromMJD_test (d Int32) ENGINE = Memory; - -INSERT INTO fromMJD_test VALUES (-1), (0), (59154); -SELECT fromMJD(d) FROM fromMJD_test; - -DROP TABLE fromMJD_test; diff --git a/tests/queries/0_stateless/01544_fromMJD.reference b/tests/queries/0_stateless/01544_fromModifiedJulianDay.reference similarity index 100% rename from tests/queries/0_stateless/01544_fromMJD.reference rename to tests/queries/0_stateless/01544_fromModifiedJulianDay.reference diff --git a/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql b/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql new file mode 100644 index 00000000000..4e50351d191 --- /dev/null +++ b/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql @@ -0,0 +1,26 @@ +-- +SELECT 'Invocation with constant'; + +SELECT fromModifiedJulianDay(-1); +SELECT fromModifiedJulianDay(0); +SELECT fromModifiedJulianDay(59154); +SELECT fromModifiedJulianDay(NULL); +SELECT fromModifiedJulianDay(-678942); -- { serverError 490 } +SELECT fromModifiedJulianDay(2973484); -- { serverError 490 } + +SELECT 'or null'; +SELECT fromModifiedJulianDayOrNull(59154); +SELECT fromModifiedJulianDayOrNull(NULL); +SELECT fromModifiedJulianDayOrNull(-678942); +SELECT fromModifiedJulianDayOrNull(2973484); + +-- +SELECT 'Invocation with Int32 column'; + +DROP TABLE IF EXISTS fromModifiedJulianDay_test; +CREATE TABLE fromModifiedJulianDay_test (d Int32) ENGINE = Memory; + +INSERT INTO fromModifiedJulianDay_test VALUES (-1), (0), (59154); +SELECT fromModifiedJulianDay(d) FROM fromModifiedJulianDay_test; + +DROP TABLE fromModifiedJulianDay_test; From ecb424a773f5c4b2f7a0a22c0661511a9c9dacec Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Mon, 7 Dec 2020 21:26:02 +0300 Subject: [PATCH 091/174] better --- contrib/CMakeLists.txt | 3 +-- contrib/dragonbox-cmake/CMakeLists.txt | 6 +++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/contrib/CMakeLists.txt b/contrib/CMakeLists.txt index a674d542c78..eb80138d5c2 100644 --- a/contrib/CMakeLists.txt +++ b/contrib/CMakeLists.txt @@ -26,6 +26,7 @@ add_subdirectory (boost-cmake) add_subdirectory (cctz-cmake) add_subdirectory (consistent-hashing-sumbur) add_subdirectory (consistent-hashing) +add_subdirectory (dragonbox-cmake) add_subdirectory (FastMemcpy) add_subdirectory (hyperscan-cmake) add_subdirectory (jemalloc-cmake) @@ -321,5 +322,3 @@ endif() if (USE_INTERNAL_ROCKSDB_LIBRARY) add_subdirectory(rocksdb-cmake) endif() - -add_subdirectory(dragonbox) diff --git a/contrib/dragonbox-cmake/CMakeLists.txt b/contrib/dragonbox-cmake/CMakeLists.txt index 3422c037837..604394c6dce 100644 --- a/contrib/dragonbox-cmake/CMakeLists.txt +++ b/contrib/dragonbox-cmake/CMakeLists.txt @@ -1,5 +1,5 @@ -set(DRAGONBOX_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/dragonbox/include) +set(LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/dragonbox") -add_library(dragonbox INTERFACE) +add_library(dragonbox_to_chars "${LIBRARY_DIR}/source/dragonbox_to_chars.cpp") -target_include_directories(dragonbox SYSTEM INTERFACE ${DRAGONBOX_INCLUDE_DIR}) \ No newline at end of file +target_include_directories(dragonbox_to_chars SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}/include/") From acd8eada4a8469618fc8b012bab5d349caf03c52 Mon Sep 17 00:00:00 2001 From: Denny Crane Date: Mon, 7 Dec 2020 20:40:04 -0400 Subject: [PATCH 092/174] Update system.md SYSTEM DROP REPLICA Russian doc --- docs/ru/sql-reference/statements/system.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/ru/sql-reference/statements/system.md b/docs/ru/sql-reference/statements/system.md index 4f7ac98807d..e64248096d2 100644 --- a/docs/ru/sql-reference/statements/system.md +++ b/docs/ru/sql-reference/statements/system.md @@ -12,6 +12,7 @@ toc_title: SYSTEM - [DROP MARK CACHE](#query_language-system-drop-mark-cache) - [DROP UNCOMPRESSED CACHE](#query_language-system-drop-uncompressed-cache) - [DROP COMPILED EXPRESSION CACHE](#query_language-system-drop-compiled-expression-cache) +- [DROP REPLICA](#query_language-system-drop-replica) - [FLUSH LOGS](#query_language-system-flush_logs) - [RELOAD CONFIG](#query_language-system-reload-config) - [SHUTDOWN](#query_language-system-shutdown) @@ -66,6 +67,24 @@ SELECT name, status FROM system.dictionaries; Сбрасывает кеш «засечек» (`mark cache`). Используется при разработке ClickHouse и тестах производительности. +## DROP REPLICA {#query_language-system-drop-replica} + +Мертвые реплики можно удалить, используя следующий синтаксис: + +``` sql +SYSTEM DROP REPLICA 'replica_name' FROM TABLE database.table; +SYSTEM DROP REPLICA 'replica_name' FROM DATABASE database; +SYSTEM DROP REPLICA 'replica_name'; +SYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/path/to/table/in/zk'; +``` + +Удаляет путь реплики из ZooKeeper-а. Это полезно, когда реплика мертва и ее метаданные не могут быть удалены из ZooKeeper с помощью `DROP TABLE`, потому что такой таблицы больше нет. `DROP REPLICA` может удалить только неактивную / устаревшую реплику и не может удалить локальную реплику, используйте для этого `DROP TABLE`. `DROP REPLICA` не удаляет таблицы и не удаляет данные или метаданные с диска. + +Первая команда удаляет метаданные реплики `'replica_name'` для таблицы `database.table`. +Вторая команда удаляет метаданные реплики `'replica_name'` для всех таблиц базы данных `database`. +Треться команда удаляет метаданные реплики `'replica_name'` для всех таблиц существующих на локальной реплике (список таблиц генерируется из локальной реплики). +Четверая команда полезна для удаления метадданных мертвой реплики когда все другие реплики таблицы уже были удалены ранее, поэтому необходимо явно указать Zookeeper путь таблицы. Zookeeper путь это первый аргумент для `ReplicatedMergeTree` движка при создании таблицы. + ## DROP UNCOMPRESSED CACHE {#query_language-system-drop-uncompressed-cache} Сбрасывает кеш не сжатых данных. Используется при разработке ClickHouse и тестах производительности. From e7376f45f42064dfc5ce9f1d877372f509125e65 Mon Sep 17 00:00:00 2001 From: "Mr.General" Date: Tue, 8 Dec 2020 09:29:00 +0800 Subject: [PATCH 093/174] update and fix over-translation on date-time-functions --- .../functions/date-time-functions.md | 534 +++++++++++++++--- 1 file changed, 463 insertions(+), 71 deletions(-) diff --git a/docs/zh/sql-reference/functions/date-time-functions.md b/docs/zh/sql-reference/functions/date-time-functions.md index 65d331a7846..00dab5ee680 100644 --- a/docs/zh/sql-reference/functions/date-time-functions.md +++ b/docs/zh/sql-reference/functions/date-time-functions.md @@ -20,7 +20,37 @@ SELECT ## toTimeZone {#totimezone} -将Date或DateTime转换为指定的时区。 +将Date或DateTime转换为指定的时区。 时区是Date/DateTime类型的属性。 表字段或结果集的列的内部值（秒数）不会更改，列的类型会更改，并且其字符串表示形式也会相应更改。 + +```sql +SELECT + toDateTime('2019-01-01 00:00:00', 'UTC') AS time_utc, + toTypeName(time_utc) AS type_utc, + toInt32(time_utc) AS int32utc, + toTimeZone(time_utc, 'Asia/Yekaterinburg') AS time_yekat, + toTypeName(time_yekat) AS type_yekat, + toInt32(time_yekat) AS int32yekat, + toTimeZone(time_utc, 'US/Samoa') AS time_samoa, + toTypeName(time_samoa) AS type_samoa, + toInt32(time_samoa) AS int32samoa +FORMAT Vertical; +``` + +```text +Row 1: +────── +time_utc: 2019-01-01 00:00:00 +type_utc: DateTime('UTC') +int32utc: 1546300800 +time_yekat: 2019-01-01 05:00:00 +type_yekat: DateTime('Asia/Yekaterinburg') +int32yekat: 1546300800 +time_samoa: 2018-12-31 13:00:00 +type_samoa: DateTime('US/Samoa') +int32samoa: 1546300800 +``` + +`toTimeZone(time_utc, 'Asia/Yekaterinburg')` 把 `DateTime('UTC')` 类型转换为 `DateTime('Asia/Yekaterinburg')`. 内部值 (Unixtimestamp) 1546300800 保持不变, 但是字符串表示(toString() 函数的结果值) 由 `time_utc: 2019-01-01 00:00:00` 转换为o `time_yekat: 2019-01-01 05:00:00`. ## toYear {#toyear} @@ -34,15 +64,15 @@ SELECT 将Date或DateTime转换为包含月份编号（1-12）的UInt8类型的数字。 -## 今天一年 {#todayofyear} +## toDayOfYear {#todayofyear} 将Date或DateTime转换为包含一年中的某一天的编号的UInt16（1-366）类型的数字。 -## 今天月 {#todayofmonth} +## toDayOfMonth {#todayofmonth} 将Date或DateTime转换为包含一月中的某一天的编号的UInt8（1-31）类型的数字。 -## 今天一周 {#todayofweek} +## toDayOfWeek {#todayofweek} 将Date或DateTime转换为包含一周中的某一天的编号的UInt8（周一是1, 周日是7）类型的数字。 @@ -55,31 +85,61 @@ SELECT 将DateTime转换为包含一小时中分钟数（0-59）的UInt8数字。 -## 秒 {#tosecond} +## toSecond {#tosecond} 将DateTime转换为包含一分钟中秒数（0-59）的UInt8数字。 闰秒不计算在内。 -## toUnixTimestamp {#tounixtimestamp} +## toUnixTimestamp {#to-unix-timestamp} -将DateTime转换为unix时间戳。 +对于DateTime参数：将值转换为UInt32类型的数字-Unix时间戳（https://en.wikipedia.org/wiki/Unix_time）。 +对于String参数：根据时区将输入字符串转换为日期时间（可选的第二个参数，默认使用服务器时区），并返回相应的unix时间戳。 -## 开始一年 {#tostartofyear} +**语法** + +``` sql +toUnixTimestamp(datetime) +toUnixTimestamp(str, [timezone]) +``` + +**返回值** + +- 返回 unix timestamp. + +类型: `UInt32`. + +**示例** + +查询: + +``` sql +SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp +``` + +结果: + +``` text +┌─unix_timestamp─┐ +│ 1509836867 │ +└────────────────┘ +``` + +## toStartOfYear {#tostartofyear} 将Date或DateTime向前取整到本年的第一天。 返回Date类型。 -## 今年开始 {#tostartofisoyear} +## toStartOfISOYear {#tostartofisoyear} 将Date或DateTime向前取整到ISO本年的第一天。 返回Date类型。 -## 四分之一开始 {#tostartofquarter} +## toStartOfQuarter {#tostartofquarter} 将Date或DateTime向前取整到本季度的第一天。 返回Date类型。 -## 到月份开始 {#tostartofmonth} +## toStartOfMonth {#tostartofmonth} 将Date或DateTime向前取整到本月的第一天。 返回Date类型。 @@ -92,27 +152,90 @@ SELECT 将Date或DateTime向前取整到本周的星期一。 返回Date类型。 -## 今天开始 {#tostartofday} +## toStartOfWeek(t\[,mode\]) {#tostartofweek} -将DateTime向前取整到当日的开始。 +按mode将Date或DateTime向前取整到最近的星期日或星期一。 +返回Date类型。 +mode参数的工作方式与toWeek()的mode参数完全相同。 对于单参数语法，mode使用默认值0。 -## 开始一小时 {#tostartofhour} +## toStartOfDay {#tostartofday} + +将DateTime向前取整到今天的开始。 + +## toStartOfHour {#tostartofhour} 将DateTime向前取整到当前小时的开始。 -## to startofminute {#tostartofminute} +## toStartOfMinute {#tostartofminute} 将DateTime向前取整到当前分钟的开始。 -## to startoffiveminute {#tostartoffiveminute} +## toStartOfSecond {#tostartofsecond} + +将DateTime向前取整到当前秒数的开始。 + +**语法** + +``` sql +toStartOfSecond(value[, timezone]) +``` + +**参数** + +- `value` — 时间和日期[DateTime64](../../sql-reference/data-types/datetime64.md). +- `timezone` — 返回值的[Timezone](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (可选参数)。 如果未指定将使用 `value` 参数的时区。 [String](../../sql-reference/data-types/string.md)。 + +**返回值** + +- 输入值毫秒部分为零。 + +类型: [DateTime64](../../sql-reference/data-types/datetime64.md). + +**示例** + +不指定时区查询: + +``` sql +WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 +SELECT toStartOfSecond(dt64); +``` + +结果: + +``` text +┌───toStartOfSecond(dt64)─┐ +│ 2020-01-01 10:20:30.000 │ +└─────────────────────────┘ +``` + +指定时区查询: + +``` sql +WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 +SELECT toStartOfSecond(dt64, 'Europe/Moscow'); +``` + +结果: + +``` text +┌─toStartOfSecond(dt64, 'Europe/Moscow')─┐ +│ 2020-01-01 13:20:30.000 │ +└────────────────────────────────────────┘ +``` + +**参考** + +- [Timezone](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 服务器配置选项。 + +## toStartOfFiveMinute {#tostartoffiveminute} 将DateTime以五分钟为单位向前取整到最接近的时间点。 -## 开始分钟 {#tostartoftenminutes} +## toStartOfTenMinutes {#tostartoftenminutes} 将DateTime以十分钟为单位向前取整到最接近的时间点。 -## 开始几分钟 {#tostartoffifteenminutes} +## toStartOfFifteenMinutes {#tostartoffifteenminutes} 将DateTime以十五分钟为单位向前取整到最接近的时间点。 @@ -168,31 +291,214 @@ SELECT 将Date或DateTime转换为包含ISO周数的UInt8类型的编号。 -## 现在 {#now} +## toWeek(date\[,mode\]) {#toweekdatemode} -不接受任何参数并在请求执行时的某一刻返回当前时间（DateTime）。 -此函数返回一个常量，即时请求需要很长时间能够完成。 +返回Date或DateTime的周数。两个参数形式可以指定星期是从星期日还是星期一开始，以及返回值应在0到53还是从1到53的范围内。如果省略了mode参数，则默认 模式为0。 +`toISOWeek()`是一个兼容函数，等效于`toWeek(date,3)`。 +下表描述了mode参数的工作方式。 -## 今天 {#today} +| Mode | First day of week | Range | Week 1 is the first week … | +|------|-------------------|-------|-------------------------------| +| 0 | Sunday | 0-53 | with a Sunday in this year | +| 1 | Monday | 0-53 | with 4 or more days this year | +| 2 | Sunday | 1-53 | with a Sunday in this year | +| 3 | Monday | 1-53 | with 4 or more days this year | +| 4 | Sunday | 0-53 | with 4 or more days this year | +| 5 | Monday | 0-53 | with a Monday in this year | +| 6 | Sunday | 1-53 | with 4 or more days this year | +| 7 | Monday | 1-53 | with a Monday in this year | +| 8 | Sunday | 1-53 | contains January 1 | +| 9 | Monday | 1-53 | contains January 1 | + +对于象“with 4 or more days this year,”的mode值，根据ISO 8601：1988对周进行编号： + +- 如果包含1月1日的一周在后一年度中有4天或更多天，则为第1周。 + +- 否则，它是上一年的最后一周，下周是第1周。 + +对于像“contains January 1”的mode值, 包含1月1日的那周为本年度的第1周。 + +``` sql +toWeek(date, [, mode][, Timezone]) +``` + +**参数** + +- `date` – Date 或 DateTime. +- `mode` – 可选参数, 取值范围 \[0,9\]， 默认0。 +- `Timezone` – 可选参数， 可其他时间日期转换参数的行为一致。 + +**示例** + +``` sql +SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS week1, toWeek(date,9) AS week9; +``` + +``` text +┌───────date─┬─week0─┬─week1─┬─week9─┐ +│ 2016-12-27 │ 52 │ 52 │ 1 │ +└────────────┴───────┴───────┴───────┘ +``` + +## toYearWeek(date\[,mode\]) {#toyearweekdatemode} + +返回Date的年和周。 结果中的年份可能因为Date为该年份的第一周和最后一周而于Date的年份不同。 + +mode参数的工作方式与toWeek()的mode参数完全相同。 对于单参数语法，mode使用默认值0。 + +`toISOYear()`是一个兼容函数，等效于`intDiv(toYearWeek(date,3),100)`. + +**示例** + +``` sql +SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(date,1) AS yearWeek1, toYearWeek(date,9) AS yearWeek9; +``` + +``` text +┌───────date─┬─yearWeek0─┬─yearWeek1─┬─yearWeek9─┐ +│ 2016-12-27 │ 201652 │ 201652 │ 201701 │ +└────────────┴───────────┴───────────┴───────────┘ +``` + +## date_trunc {#date_trunc} + +将Date或DateTime按指定的单位向前取整到最接近的时间点。 + +**语法** + +``` sql +date_trunc(unit, value[, timezone]) +``` + +别名: `dateTrunc`. + +**参数** + +- `unit` — 单位. [String](../syntax.md#syntax-string-literal). + 可选值: + + - `second` + - `minute` + - `hour` + - `day` + - `week` + - `month` + - `quarter` + - `year` + +- `value` — [DateTime](../../sql-reference/data-types/datetime.md) 或者 [DateTime64](../../sql-reference/data-types/datetime64.md). +- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 返回值的时区(可选值)。如果未指定将使用`value`的时区。 [String](../../sql-reference/data-types/string.md). + +**返回值** + +- 按指定的单位向前取整后的DateTime。 + +类型: [Datetime](../../sql-reference/data-types/datetime.md). + +**示例** + +不指定时区查询: + +``` sql +SELECT now(), date_trunc('hour', now()); +``` + +结果: + +``` text +┌───────────────now()─┬─date_trunc('hour', now())─┐ +│ 2020-09-28 10:40:45 │ 2020-09-28 10:00:00 │ +└─────────────────────┴───────────────────────────┘ +``` + +指定时区查询: + +```sql +SELECT now(), date_trunc('hour', now(), 'Europe/Moscow'); +``` + +结果: + +```text +┌───────────────now()─┬─date_trunc('hour', now(), 'Europe/Moscow')─┐ +│ 2020-09-28 10:46:26 │ 2020-09-28 13:00:00 │ +└─────────────────────┴────────────────────────────────────────────┘ +``` + +**参考** + +- [toStartOfInterval](#tostartofintervaltime-or-data-interval-x-unit-time-zone) + +# now {#now} + +返回当前日期和时间。 + +**语法** + +``` sql +now([timezone]) +``` + +**参数** + +- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 返回结果的时区(可先参数). [String](../../sql-reference/data-types/string.md). + +**返回值** + +- 当前日期和时间。 + +类型: [Datetime](../../sql-reference/data-types/datetime.md). + +**示例** + +不指定时区查询: + +``` sql +SELECT now(); +``` + +结果: + +``` text +┌───────────────now()─┐ +│ 2020-10-17 07:42:09 │ +└─────────────────────┘ +``` + +指定时区查询: + +``` sql +SELECT now('Europe/Moscow'); +``` + +结果: + +``` text +┌─now('Europe/Moscow')─┐ +│ 2020-10-17 10:42:23 │ +└──────────────────────┘ +``` + +## today {#today} 不接受任何参数并在请求执行时的某一刻返回当前日期(Date)。 -其功能与’toDate（now（））’相同。 +其功能与’toDate（now()）’相同。 -## 昨天 {#yesterday} +## yesterday {#yesterday} 不接受任何参数并在请求执行时的某一刻返回昨天的日期(Date)。 -其功能与’today（） - 1’相同。 +其功能与’today() - 1’相同。 -## 时隙 {#timeslot} +## timeSlot {#timeslot} 将时间向前取整半小时。 此功能用于Yandex.Metrica，因为如果跟踪标记显示单个用户的连续综合浏览量在时间上严格超过此数量，则半小时是将会话分成两个会话的最短时间。这意味着（tag id，user id，time slot）可用于搜索相应会话中包含的综合浏览量。 -## toyyymm {#toyyyymm} +## toYYYMM {#toyyyymm} 将Date或DateTime转换为包含年份和月份编号的UInt32类型的数字（YYYY \* 100 + MM）。 -## toyyymmdd {#toyyyymmdd} +## toYYYMMDD {#toyyyymmdd} 将Date或DateTime转换为包含年份和月份编号的UInt32类型的数字（YYYY \* 10000 + MM \* 100 + DD）。 @@ -200,7 +506,7 @@ SELECT 将Date或DateTime转换为包含年份和月份编号的UInt64类型的数字（YYYY \* 10000000000 + MM \* 100000000 + DD \* 1000000 + hh \* 10000 + mm \* 100 + ss）。 -## 隆隆隆隆路虏脢,,陇,貌,垄拢卢虏禄quar陇,貌路,隆拢脳枚脢虏,麓脢,脱,,,录,禄庐戮,utes, {#addyears-addmonths-addweeks-adddays-addhours-addminutes-addseconds-addquarters} +## addYears, addMonths, addWeeks, addDays, addHours, addMinutes, addSeconds, addQuarters {#addyears-addmonths-addweeks-adddays-addhours-addminutes-addseconds-addquarters} 函数将一段时间间隔添加到Date/DateTime，然后返回Date/DateTime。例如： @@ -234,59 +540,145 @@ SELECT │ 2018-01-01 │ 2018-01-01 00:00:00 │ └──────────────────────────┴───────────────────────────────┘ -## dateDiff(‘unit’,t1,t2,\[时区\]) {#datediffunit-t1-t2-timezone} +## dateDiff {#datediff} -返回以’unit’为单位表示的两个时间之间的差异，例如`'hours'`。 ‘t1’和’t2’可以是Date或DateTime，如果指定’timezone’，它将应用于两个参数。如果不是，则使用来自数据类型’t1’和’t2’的时区。如果时区不相同，则结果将是未定义的。 +返回两个Date或DateTime类型之间的时差。 -支持的单位值: +**语法** -| 单位 | -|------| -| 第二 | -| 分钟 | -| 小时 | -| 日 | -| 周 | -| 月 | -| 季 | -| 年 | +``` sql +dateDiff('unit', startdate, enddate, [timezone]) +``` -## 时隙（开始时间，持续时间，\[，大小\]) {#timeslotsstarttime-duration-size} +**参数** + +- `unit` — 返回结果的时间单位。 [String](../../sql-reference/syntax.md#syntax-string-literal). + + 支持的时间单位: second, minute, hour, day, week, month, quarter, year. + +- `startdate` — 第一个待比较值。 [Date](../../sql-reference/data-types/date.md) 或 [DateTime](../../sql-reference/data-types/datetime.md). + +- `enddate` — 第二个待比较值。 [Date](../../sql-reference/data-types/date.md) 或 [DateTime](../../sql-reference/data-types/datetime.md). + +- `timezone` — 可选参数。 如果指定了，则同时适用于`startdate`和`enddate`。如果不指定，则使用`startdate`和`enddate`的时区。如果两个时区不一致，则结果不可预料。 + +**返回值** + +以`unit`为单位的`startdate`和`enddate`之间的时差。 + +类型: `int`. + +**示例** + +查询: + +``` sql +SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00')); +``` + +结果: + +``` text +┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐ +│ 25 │ +└────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## timeSlots(StartTime, Duration,\[, Size\]) {#timeslotsstarttime-duration-size} 它返回一个时间数组，其中包括从从«StartTime»开始到«StartTime + Duration 秒»内的所有符合«size»（以秒为单位）步长的时间点。其中«size»是一个可选参数，默认为1800。 例如，`timeSlots(toDateTime('2012-01-01 12:20:00')，600) = [toDateTime（'2012-01-01 12:00:00'），toDateTime（'2012-01-01 12:30:00' ）]`。 这对于搜索在相应会话中综合浏览量是非常有用的。 -## formatDateTime（时间，格式\[，时区\]) {#formatdatetimetime-format-timezone} +## formatDateTime {#formatdatetime} 函数根据给定的格式字符串来格式化时间。请注意：格式字符串必须是常量表达式，例如：单个结果列不能有多种格式字符串。 -支持的格式修饰符： -(«Example» 列是对`2018-01-02 22:33:44`的格式化结果) +**语法** -| 修饰符 | 产品描述 | 示例 | -|--------|-------------------------------------------|------------| -| %C | 年除以100并截断为整数(00-99) | 20 | -| %d | 月中的一天，零填充（01-31) | 02 | -| %D | 短MM/DD/YY日期，相当于%m/%d/%y | 01/02/2018 | -| %e | 月中的一天，空格填充（1-31) | 2 | -| %F | 短YYYY-MM-DD日期，相当于%Y-%m-%d | 2018-01-02 | -| %H | 24小时格式（00-23) | 22 | -| %I | 小时12h格式（01-12) | 10 | -| %j | 一年(001-366) | 002 | -| %m | 月份为十进制数（01-12) | 01 | -| %M | 分钟(00-59) | 33 | -| %n | 换行符(") | | -| %p | AM或PM指定 | PM | -| %R | 24小时HH:MM时间，相当于%H:%M | 22:33 | -| %S | 第二(00-59) | 44 | -| %t | 水平制表符(’) | | -| %T | ISO8601时间格式(HH:MM:SS)，相当于%H:%M:%S | 22:33:44 | -| %u | ISO8601平日as编号，星期一为1(1-7) | 2 | -| %V | ISO8601周编号(01-53) | 01 | -| %w | 周日为十进制数，周日为0(0-6) | 2 | -| %y | 年份，最后两位数字（00-99) | 18 | -| %Y | 年 | 2018 | -| %% | %符号 | % | +``` sql +formatDateTime(Time, Format\[, Timezone\]) +``` -[来源文章](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) +**返回值** + +根据指定格式返回的日期和时间。 + +**支持的格式修饰符** + +使用格式修饰符来指定结果字符串的样式。«Example» 列是对`2018-01-02 22:33:44`的格式化结果。 + +| 修饰符 | 描述 | 示例 | +|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| +| %C | 年除以100并截断为整数(00-99) | 20 | +| %d | 月中的一天，零填充（01-31) | 02 | +| %D | 短MM/DD/YY日期，相当于%m/%d/%y | 01/02/2018 | +| %e | 月中的一天，空格填充（1-31) | 2 | +| %F | 短YYYY-MM-DD日期，相当于%Y-%m-%d | 2018-01-02 | +| %G | ISO周号的四位数年份格式， 从基于周的年份[由ISO 8601定义](https://en.wikipedia.org/wiki/ISO_8601#Week_dates) 标准计算得出，通常仅对％V有用 | 2018 | +| %g | 两位数的年份格式，与ISO 8601一致，四位数表示法的缩写 | 18 | +| %H | 24小时格式（00-23) | 22 | +| %I | 小时12h格式（01-12) | 10 | +| %j | 一年(001-366) | 002 | +| %m | 月份为十进制数（01-12) | 01 | +| %M | 分钟(00-59) | 33 | +| %n | 换行符(") | | +| %p | AM或PM指定 | PM | +| %R | 24小时HH:MM时间，相当于%H:%M | 22:33 | +| %S | 第二(00-59) | 44 | +| %t | 水平制表符(’) | | +| %T | ISO8601时间格式(HH:MM:SS)，相当于%H:%M:%S | 22:33:44 | +| %u | ISO8601平日as编号，星期一为1(1-7) | 2 | +| %V | ISO8601周编号(01-53) | 01 | +| %w | 周日为十进制数，周日为0(0-6) | 2 | +| %y | 年份，最后两位数字（00-99) | 18 | +| %Y | 年 | 2018 | +| %% | %符号 | % | + +**示例** + +查询: + +``` sql +SELECT formatDateTime(toDate('2010-01-04'), '%g') +``` + +结果: + +``` +┌─formatDateTime(toDate('2010-01-04'), '%g')─┐ +│ 10 │ +└────────────────────────────────────────────┘ +``` + +[Original article](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) + +## FROM_UNIXTIME + +当只有单个整数类型的参数时，它的作用与`toDateTime`相同，并返回[DateTime](../../sql-reference/data-types/datetime.md)类型。 + +例如: + +```sql +SELECT FROM_UNIXTIME(423543535) +``` + +```text +┌─FROM_UNIXTIME(423543535)─┐ +│ 1983-06-04 10:58:55 │ +└──────────────────────────┘ +``` + +当有两个参数时，第一个是整型或DateTime，第二个是常量格式字符串，它的作用与`formatDateTime`相同，并返回`String`类型。 + +例如: + +```sql +SELECT FROM_UNIXTIME(1234334543, '%Y-%m-%d %R:%S') AS DateTime +``` + +```text +┌─DateTime────────────┐ +│ 2009-02-11 14:42:23 │ +└─────────────────────┘ +``` From 5556c482982fc556cbdd1b0c638f15dd6619444d Mon Sep 17 00:00:00 2001 From: Vasily Nemkov Date: Tue, 8 Dec 2020 13:51:50 +0300 Subject: [PATCH 094/174] Added test for DateTime vs DateTime64 comparison --- ...eTime_and_DateTime64_comparision.reference | 3 ++ ...60_DateTime_and_DateTime64_comparision.sql | 42 +++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference create mode 100644 tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql diff --git a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference new file mode 100644 index 00000000000..088030bbc28 --- /dev/null +++ b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference @@ -0,0 +1,3 @@ +-1 DateTime64(1, \'UTC\') < 1 1 1 <= 1 1 1 = 0 0 0 >= 0 0 0 > 0 0 0 != 1 1 1 +0 DateTime64(1, \'UTC\') < 0 0 0 <= 1 1 1 = 1 1 1 >= 1 1 1 > 0 0 0 != 0 0 0 +1 DateTime64(1, \'UTC\') < 0 0 0 <= 0 0 0 = 0 0 0 >= 1 1 1 > 1 1 1 != 1 1 1 diff --git a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql new file mode 100644 index 00000000000..afee0ebadaa --- /dev/null +++ b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql @@ -0,0 +1,42 @@ +SELECT + n, + toTypeName(dt64) AS dt64_typename, + + '<', + dt64 < dt, + toDateTime(dt64) < dt, + dt64 < toDateTime64(dt, 1, 'UTC'), + + '<=', + dt64 <= dt, + toDateTime(dt64) <= dt, + dt64 <= toDateTime64(dt, 1, 'UTC'), + + '=', + dt64 = dt, + toDateTime(dt64) = dt, + dt64 = toDateTime64(dt, 1, 'UTC'), + + '>=', + dt64 >= dt, + toDateTime(dt64) >= dt, + dt64 >= toDateTime64(dt, 1, 'UTC'), + + '>', + dt64 > dt, + toDateTime(dt64) > dt, + dt64 > toDateTime64(dt, 1, 'UTC'), + + '!=', + dt64 != dt, + toDateTime(dt64) != dt, + dt64 != toDateTime64(dt, 1, 'UTC') +FROM +( + SELECT + number - 1 as n, + toDateTime64(toStartOfInterval(now(), toIntervalSecond(1), 'UTC'), 1, 'UTC') + n AS dt64, + toStartOfInterval(now(), toIntervalSecond(1), 'UTC') AS dt + FROM system.numbers + LIMIT 3 +) From f4231d552e17163f9940be57503361d56d069a38 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 8 Dec 2020 16:28:39 +0300 Subject: [PATCH 095/174] Banch of fixes --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 83 +++-- src/Common/ZooKeeper/ZooKeeperCommon.cpp | 3 +- src/Common/ZooKeeper/ZooKeeperCommon.h | 14 +- src/Common/ZooKeeper/ZooKeeperConstants.cpp | 3 + src/Common/ZooKeeper/ZooKeeperConstants.h | 2 + src/Common/ZooKeeper/ZooKeeperIO.cpp | 5 + src/Common/ZooKeeper/ZooKeeperIO.h | 1 + src/Server/TestKeeperTCPHandler.cpp | 6 +- .../test_testkeeper_back_to_back/__init__.py | 1 + .../configs/enable_test_keeper.xml | 7 + .../test_testkeeper_back_to_back/test.py | 337 ++++++++++++++++++ 11 files changed, 428 insertions(+), 34 deletions(-) create mode 100644 tests/integration/test_testkeeper_back_to_back/__init__.py create mode 100644 tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml create mode 100644 tests/integration/test_testkeeper_back_to_back/test.py diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 146eb374e28..aa44f3142f2 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -39,34 +39,37 @@ static String baseName(const String & path) return path.substr(rslash_pos + 1); } -static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) +static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches, Coordination::Event event_type) { - Coordination::ZooKeeperWatchResponse watch_response; - watch_response.path = path; - watch_response.xid = -1; - watch_response.zxid = -1; - - auto it = watches.find(watch_response.path); + auto it = watches.find(path); if (it != watches.end()) { + std::shared_ptr watch_response = std::make_shared(); + watch_response->path = path; + watch_response->xid = -1; + watch_response->zxid = -1; + watch_response->type = event_type; + watch_response->state = Coordination::State::CONNECTED; for (auto & watcher : it->second) if (watcher.watch_callback) - watcher.watch_callback(std::make_shared(watch_response)); + watcher.watch_callback(watch_response); watches.erase(it); } - Coordination::ZooKeeperWatchResponse watch_list_response; - watch_list_response.path = parentPath(path); - watch_list_response.xid = -1; - watch_list_response.zxid = -1; - - it = list_watches.find(watch_list_response.path); + auto parent_path = parentPath(path); + it = list_watches.find(parent_path); if (it != list_watches.end()) { + std::shared_ptr watch_list_response = std::make_shared(); + watch_list_response->path = parent_path; + watch_list_response->xid = -1; + watch_list_response->zxid = -1; + watch_list_response->type = Coordination::Event::CHILD; + watch_list_response->state = Coordination::State::CONNECTED; for (auto & watcher : it->second) if (watcher.watch_callback) - watcher.watch_callback(std::make_shared(watch_list_response)); + watcher.watch_callback(watch_list_response); list_watches.erase(it); } @@ -110,7 +113,7 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override { - processWatchesImpl(zk_request->getPath(), watches, list_watches); + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CREATED); } std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const override @@ -154,7 +157,6 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest if (request.is_sequential) { auto seq_num = it->second.seq_num; - ++it->second.seq_num; std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM seq_num_str.exceptions(std::ios::failbit); @@ -163,13 +165,16 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest path_created += seq_num_str.str(); } + /// Increment sequential number even if node is not sequential + ++it->second.seq_num; + response.path_created = path_created; container.emplace(path_created, std::move(created_node)); if (request.is_ephemeral) ephemerals[session_id].emplace(path_created); - undo = [&container, &ephemerals, session_id, path_created, is_sequential = request.is_sequential, is_ephemeral = request.is_ephemeral, parent_path = it->first] + undo = [&container, &ephemerals, session_id, path_created, is_ephemeral = request.is_ephemeral, parent_path = it->first] { container.erase(path_created); if (is_ephemeral) @@ -177,9 +182,7 @@ struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest auto & undo_parent = container.at(parent_path); --undo_parent.stat.cversion; --undo_parent.stat.numChildren; - - if (is_sequential) - --undo_parent.seq_num; + --undo_parent.seq_num; }; ++it->second.stat.cversion; @@ -270,7 +273,7 @@ struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override { - processWatchesImpl(zk_request->getPath(), watches, list_watches); + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::DELETED); } }; @@ -321,6 +324,7 @@ struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest ++it->second.stat.version; it->second.stat.mzxid = zxid; it->second.stat.mtime = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1); + it->second.stat.dataLength = request.data.length(); it->second.data = request.data; ++container.at(parentPath(request.path)).stat.cversion; response.stat = it->second.stat; @@ -342,7 +346,7 @@ struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override { - processWatchesImpl(zk_request->getPath(), watches, list_watches); + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CHANGED); } }; @@ -457,9 +461,23 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest for (const auto & concrete_request : concrete_requests) { auto [ cur_response, undo_action ] = concrete_request->process(container, ephemerals, zxid, session_id); + response.responses[i] = cur_response; if (cur_response->error != Coordination::Error::ZOK) { + for (size_t j = 0; j <= i; ++j) + { + auto response_error = response.responses[j]->error; + response.responses[j] = std::make_shared(); + response.responses[j]->error = response_error; + } + + for (size_t j = i + 1; j < response.responses.size(); ++j) + { + response.responses[j] = std::make_shared(); + response.responses[j]->error = Coordination::Error::ZRUNTIMEINCONSISTENCY; + } + for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) if (*it) (*it)(); @@ -468,6 +486,7 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest } else undo_actions.emplace_back(std::move(undo_action)); + ++i; } @@ -525,7 +544,7 @@ void TestKeeperStorage::processingThread() for (const auto & ephemeral_path : it->second) { container.erase(ephemeral_path); - processWatchesImpl(ephemeral_path, watches, list_watches); + processWatchesImpl(ephemeral_path, watches, list_watches, Coordination::Event::DELETED); } ephemerals.erase(it); } @@ -539,7 +558,7 @@ void TestKeeperStorage::processingThread() { if (response->error == Coordination::Error::ZOK) { - auto & watches_type = zk_request->getOpNum() == Coordination::OpNum::List + auto & watches_type = zk_request->getOpNum() == Coordination::OpNum::List || zk_request->getOpNum() == Coordination::OpNum::SimpleList ? list_watches : watches; @@ -553,11 +572,12 @@ void TestKeeperStorage::processingThread() } else { - Coordination::ZooKeeperWatchResponse watch_response; - watch_response.path = zk_request->getPath(); - watch_response.xid = -1; - watch_response.error = response->error; - info.watch_callback(std::make_shared(watch_response)); + std::shared_ptr watch_response = std::make_shared(); + watch_response->path = zk_request->getPath(); + watch_response->xid = -1; + watch_response->error = response->error; + watch_response->type = Coordination::Event::NOTWATCHING; + info.watch_callback(watch_response); } } @@ -700,6 +720,7 @@ TestKeeperWrapperFactory::TestKeeperWrapperFactory() registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); registerTestKeeperRequestWrapper(*this); } @@ -740,7 +761,9 @@ void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & req request_info.session_id = session_id; request_info.response_callback = callback; if (request->has_watch) + { request_info.watch_callback = watch_callback; + } std::lock_guard lock(push_request_mutex); if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp index 7f7e5f4abaf..9c699ee298a 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.cpp +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -393,7 +393,7 @@ void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const Coordination::write(op_num, out); Coordination::write(done, out); Coordination::write(op_error, out); - if (op_error == Error::ZOK) + if (op_error == Error::ZOK || op_num == OpNum::Error) zk_response.writeImpl(out); } @@ -472,6 +472,7 @@ ZooKeeperRequestFactory::ZooKeeperRequestFactory() registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); registerZooKeeperRequest(*this); diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h index fbf3e13fbfe..9adb0c06e4c 100644 --- a/src/Common/ZooKeeper/ZooKeeperCommon.h +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -226,7 +226,7 @@ struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse OpNum getOpNum() const override { return OpNum::Set; } }; -struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest +struct ZooKeeperListRequest : ListRequest, ZooKeeperRequest { OpNum getOpNum() const override { return OpNum::List; } void writeImpl(WriteBuffer & out) const override; @@ -234,13 +234,23 @@ struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest ZooKeeperResponsePtr makeResponse() const override; }; -struct ZooKeeperListResponse final : ListResponse, ZooKeeperResponse +struct ZooKeeperSimpleListRequest final : ZooKeeperListRequest +{ + OpNum getOpNum() const override { return OpNum::SimpleList; } +}; + +struct ZooKeeperListResponse : ListResponse, ZooKeeperResponse { void readImpl(ReadBuffer & in) override; void writeImpl(WriteBuffer & out) const override; OpNum getOpNum() const override { return OpNum::List; } }; +struct ZooKeeperSimpleListResponse final : ZooKeeperListResponse +{ + OpNum getOpNum() const override { return OpNum::SimpleList; } +}; + struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest { ZooKeeperCheckRequest() = default; diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.cpp b/src/Common/ZooKeeper/ZooKeeperConstants.cpp index 9a1a43d43a7..b4cb9feb518 100644 --- a/src/Common/ZooKeeper/ZooKeeperConstants.cpp +++ b/src/Common/ZooKeeper/ZooKeeperConstants.cpp @@ -14,6 +14,7 @@ static const std::unordered_set VALID_OPERATIONS = static_cast(OpNum::Exists), static_cast(OpNum::Get), static_cast(OpNum::Set), + static_cast(OpNum::SimpleList), static_cast(OpNum::Heartbeat), static_cast(OpNum::List), static_cast(OpNum::Check), @@ -39,6 +40,8 @@ std::string toString(OpNum op_num) return "Get"; case OpNum::Set: return "Set"; + case OpNum::SimpleList: + return "SimpleList"; case OpNum::List: return "List"; case OpNum::Check: diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.h b/src/Common/ZooKeeper/ZooKeeperConstants.h index 292bccd86b4..8a20330a2d7 100644 --- a/src/Common/ZooKeeper/ZooKeeperConstants.h +++ b/src/Common/ZooKeeper/ZooKeeperConstants.h @@ -23,6 +23,7 @@ enum class OpNum : int32_t Exists = 3, Get = 4, Set = 5, + SimpleList = 8, Heartbeat = 11, List = 12, Check = 13, @@ -35,6 +36,7 @@ OpNum getOpNum(int32_t raw_op_num); static constexpr int32_t ZOOKEEPER_PROTOCOL_VERSION = 0; static constexpr int32_t CLIENT_HANDSHAKE_LENGTH = 44; +static constexpr int32_t CLIENT_HANDSHAKE_LENGTH_WITH_READONLY = 45; static constexpr int32_t SERVER_HANDSHAKE_LENGTH = 36; static constexpr int32_t PASSWORD_LENGTH = 16; diff --git a/src/Common/ZooKeeper/ZooKeeperIO.cpp b/src/Common/ZooKeeper/ZooKeeperIO.cpp index 07227669366..a0e4161f111 100644 --- a/src/Common/ZooKeeper/ZooKeeperIO.cpp +++ b/src/Common/ZooKeeper/ZooKeeperIO.cpp @@ -81,6 +81,11 @@ void read(bool & x, ReadBuffer & in) readBinary(x, in); } +void read(int8_t & x, ReadBuffer & in) +{ + readBinary(x, in); +} + void read(std::string & s, ReadBuffer & in) { int32_t size = 0; diff --git a/src/Common/ZooKeeper/ZooKeeperIO.h b/src/Common/ZooKeeper/ZooKeeperIO.h index bd6d6a6f849..edeb995f27b 100644 --- a/src/Common/ZooKeeper/ZooKeeperIO.h +++ b/src/Common/ZooKeeper/ZooKeeperIO.h @@ -41,6 +41,7 @@ void read(int64_t & x, ReadBuffer & in); void read(int32_t & x, ReadBuffer & in); void read(OpNum & x, ReadBuffer & in); void read(bool & x, ReadBuffer & in); +void read(int8_t & x, ReadBuffer & in); void read(std::string & s, ReadBuffer & in); void read(ACL & acl, ReadBuffer & in); void read(Stat & stat, ReadBuffer & in); diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index fff6c344b4e..998e734af0a 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -231,7 +231,7 @@ void TestKeeperTCPHandler::receiveHandshake() std::array passwd {}; Coordination::read(handshake_length, *in); - if (handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH) + if (handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH && handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH_WITH_READONLY) throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(protocol_version, *in); @@ -251,6 +251,10 @@ void TestKeeperTCPHandler::receiveHandshake() throw Exception("Non zero previous session id is not supported", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); Coordination::read(passwd, *in); + + int8_t readonly; + if (handshake_length == Coordination::CLIENT_HANDSHAKE_LENGTH_WITH_READONLY) + Coordination::read(readonly, *in); } diff --git a/tests/integration/test_testkeeper_back_to_back/__init__.py b/tests/integration/test_testkeeper_back_to_back/__init__.py new file mode 100644 index 00000000000..e5a0d9b4834 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/__init__.py @@ -0,0 +1 @@ +#!/usr/bin/env python3 diff --git a/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml b/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml new file mode 100644 index 00000000000..79e993b41f7 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml @@ -0,0 +1,7 @@ + + + 9181 + 10000 + 30000 + + diff --git a/tests/integration/test_testkeeper_back_to_back/test.py b/tests/integration/test_testkeeper_back_to_back/test.py new file mode 100644 index 00000000000..5ceb97572a2 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/test.py @@ -0,0 +1,337 @@ +import pytest +from helpers.cluster import ClickHouseCluster +import random +import string +import os +import time + +cluster = ClickHouseCluster(__file__) +node = cluster.add_instance('node', main_configs=['configs/enable_test_keeper.xml'], with_zookeeper=True) +from kazoo.client import KazooClient + +_genuine_zk_instance = None +_fake_zk_instance = None + +def get_genuine_zk(): + global _genuine_zk_instance + if not _genuine_zk_instance: + print("Zoo1", cluster.get_instance_ip("zoo1")) + _genuine_zk_instance = cluster.get_kazoo_client('zoo1') + return _genuine_zk_instance + + +def get_fake_zk(): + global _fake_zk_instance + if not _fake_zk_instance: + print("node", cluster.get_instance_ip("node")) + _fake_zk_instance = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + _fake_zk_instance.start() + return _fake_zk_instance + +def random_string(length): + return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length)) + +def create_random_path(prefix="", depth=1): + if depth == 0: + return prefix + return create_random_path(os.path.join(prefix, random_string(3)), depth - 1) + +@pytest.fixture(scope="module") +def started_cluster(): + try: + cluster.start() + + yield cluster + + finally: + cluster.shutdown() + _genuine_zk_instance.stop() + _genuine_zk_instance.close() + _fake_zk_instance.stop() + _fake_zk_instance.close() + + +def test_simple_commands(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + + for zk in [genuine_zk, fake_zk]: + zk.create("/test_simple_commands", b"") + zk.create("/test_simple_commands/somenode1", b"hello") + zk.set("/test_simple_commands/somenode1", b"world") + + for zk in [genuine_zk, fake_zk]: + assert zk.exists("/test_simple_commands") + assert zk.exists("/test_simple_commands/somenode1") + print(zk.get("/test_simple_commands/somenode1")) + assert zk.get("/test_simple_commands/somenode1")[0] == b"world" + + +def test_sequential_nodes(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_sequential_nodes") + fake_zk.create("/test_sequential_nodes") + for i in range(1, 11): + genuine_zk.create("/test_sequential_nodes/" + ("a" * i) + "-", sequence=True) + genuine_zk.create("/test_sequential_nodes/" + ("b" * i)) + fake_zk.create("/test_sequential_nodes/" + ("a" * i) + "-", sequence=True) + fake_zk.create("/test_sequential_nodes/" + ("b" * i)) + + genuine_childs = list(sorted(genuine_zk.get_children("/test_sequential_nodes"))) + fake_childs = list(sorted(fake_zk.get_children("/test_sequential_nodes"))) + assert genuine_childs == fake_childs + + +def assert_eq_stats(stat1, stat2): + assert stat1.version == stat2.version + assert stat1.cversion == stat2.cversion + assert stat1.aversion == stat2.aversion + assert stat1.aversion == stat2.aversion + assert stat1.dataLength == stat2.dataLength + assert stat1.numChildren == stat2.numChildren + +def test_stats(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_stats_nodes") + fake_zk.create("/test_stats_nodes") + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + assert_eq_stats(genuine_stats, fake_stats) + for i in range(1, 11): + genuine_zk.create("/test_stats_nodes/" + ("a" * i) + "-", sequence=True) + genuine_zk.create("/test_stats_nodes/" + ("b" * i)) + fake_zk.create("/test_stats_nodes/" + ("a" * i) + "-", sequence=True) + fake_zk.create("/test_stats_nodes/" + ("b" * i)) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + assert_eq_stats(genuine_stats, fake_stats) + for i in range(1, 11): + print("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + genuine_zk.delete("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + genuine_zk.delete("/test_stats_nodes/" + ("b" * i)) + fake_zk.delete("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + fake_zk.delete("/test_stats_nodes/" + ("b" * i)) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + print(genuine_stats) + print(fake_stats) + assert_eq_stats(genuine_stats, fake_stats) + for i in range(100): + genuine_zk.set("/test_stats_nodes", ("q" * i).encode()) + fake_zk.set("/test_stats_nodes", ("q" * i).encode()) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + print(genuine_stats) + print(fake_stats) + assert_eq_stats(genuine_stats, fake_stats) + +def test_watchers(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_data_watches") + fake_zk.create("/test_data_watches") + genuine_data_watch_data = None + + def genuine_callback(event): + print("Genuine data watch called") + nonlocal genuine_data_watch_data + genuine_data_watch_data = event + + fake_data_watch_data = None + def fake_callback(event): + print("Fake data watch called") + nonlocal fake_data_watch_data + fake_data_watch_data = event + + genuine_zk.get("/test_data_watches", watch=genuine_callback) + fake_zk.get("/test_data_watches", watch=fake_callback) + + print("Calling set genuine") + genuine_zk.set("/test_data_watches", b"a") + print("Calling set fake") + fake_zk.set("/test_data_watches", b"a") + time.sleep(5) + + print("Genuine data", genuine_data_watch_data) + print("Fake data", fake_data_watch_data) + assert genuine_data_watch_data == fake_data_watch_data + + genuine_children = None + def genuine_child_callback(event): + print("Genuine child watch called") + nonlocal genuine_children + genuine_children = event + + fake_children = None + def fake_child_callback(event): + print("Fake child watch called") + nonlocal fake_children + fake_children = event + + genuine_zk.get_children("/test_data_watches", watch=genuine_child_callback) + fake_zk.get_children("/test_data_watches", watch=fake_child_callback) + + print("Calling genuine child") + genuine_zk.create("/test_data_watches/child", b"b") + print("Calling fake child") + fake_zk.create("/test_data_watches/child", b"b") + + time.sleep(5) + + print("Genuine children", genuine_children) + print("Fake children", fake_children) + assert genuine_children == fake_children + +def test_multitransactions(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + for zk in [genuine_zk, fake_zk]: + zk.create('/test_multitransactions') + t = zk.transaction() + t.create('/test_multitransactions/freddy') + t.create('/test_multitransactions/fred', ephemeral=True) + t.create('/test_multitransactions/smith', sequence=True) + results = t.commit() + assert len(results) == 3 + assert results[0] == '/test_multitransactions/freddy' + assert results[2].startswith('/test_multitransactions/smith0') is True + + from kazoo.exceptions import RolledBackError, NoNodeError + for i, zk in enumerate([genuine_zk, fake_zk]): + print("Processing ZK", i) + t = zk.transaction() + t.create('/test_multitransactions/q') + t.delete('/test_multitransactions/a') + t.create('/test_multitransactions/x') + results = t.commit() + print("Results", results) + assert results[0].__class__ == RolledBackError + assert results[1].__class__ == NoNodeError + assert zk.exists('/test_multitransactions/q') is None + assert zk.exists('/test_multitransactions/a') is None + assert zk.exists('/test_multitransactions/x') is None + + +def exists(zk, path): + result = zk.exists(path) + return result is not None + +def get(zk, path): + result = zk.get(path) + return result[0] + +def get_children(zk, path): + return [elem for elem in list(sorted(zk.get_children(path))) if elem not in ('clickhouse', 'zookeeper')] + +READ_REQUESTS = [ + ("exists", exists), + ("get", get), + ("get_children", get_children), +] + + +def create(zk, path, data): + zk.create(path, data.encode()) + + +def set_data(zk, path, data): + zk.set(path, data.encode()) + + +WRITE_REQUESTS = [ + ("create", create), + ("set_data", set_data), +] + + +def delete(zk, path): + zk.delete(path) + +DELETE_REQUESTS = [ + ("delete", delete) +] + + +class Request(object): + def __init__(self, name, arguments, callback, is_return): + self.name = name + self.arguments = arguments + self.callback = callback + self.is_return = is_return + + def __str__(self): + arg_str = ', '.join([str(k) + "=" + str(v) for k, v in self.arguments.items()]) + return "ZKRequest name {} with arguments {}".format(self.name, arg_str) + +def generate_requests(iters=1): + requests = [] + existing_paths = [] + for i in range(iters): + for _ in range(100): + rand_length = random.randint(0, 10) + path = "/" + for j in range(1, rand_length): + path = create_random_path(path, 1) + existing_paths.append(path) + value = random_string(1000) + request = Request("create", {"path" : path, "value": value[0:10]}, lambda zk, path=path, value=value: create(zk, path, value), False) + requests.append(request) + + for _ in range(100): + path = random.choice(existing_paths) + value = random_string(100) + request = Request("set", {"path": path, "value": value[0:10]}, lambda zk, path=path, value=value: set_data(zk, path, value), False) + requests.append(request) + + for _ in range(100): + path = random.choice(existing_paths) + callback = random.choice(READ_REQUESTS) + def read_func1(zk, path=path, callback=callback): + return callback[1](zk, path) + + request = Request(callback[0], {"path": path}, read_func1, True) + requests.append(request) + + for _ in range(30): + path = random.choice(existing_paths) + request = Request("delete", {"path": path}, lambda zk, path=path: delete(zk, path), False) + + for _ in range(100): + path = random.choice(existing_paths) + callback = random.choice(READ_REQUESTS) + def read_func2(zk, path=path, callback=callback): + return callback[1](zk, path) + request = Request(callback[0], {"path": path}, read_func2, True) + requests.append(request) + return requests + + +def test_random_requests(started_cluster): + requests = generate_requests(10) + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + for i, request in enumerate(requests): + genuine_throw = False + fake_throw = False + fake_result = None + genuine_result = None + try: + genuine_result = request.callback(genuine_zk) + except Exception as ex: + genuine_throw = True + + try: + fake_result = request.callback(fake_zk) + except Exception as ex: + fake_throw = True + + assert fake_throw == genuine_throw, "Fake throw genuine not or vise versa" + assert fake_result == genuine_result, "Zookeeper results differ" + root_children_genuine = [elem for elem in list(sorted(genuine_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] + root_children_fake = [elem for elem in list(sorted(fake_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] + assert root_children_fake == root_children_genuine From 0c2d73adcb5b1b38e75fdece37baa8ef48d88ecc Mon Sep 17 00:00:00 2001 From: Vasily Nemkov Date: Tue, 8 Dec 2020 17:17:02 +0300 Subject: [PATCH 096/174] Fixed comparing Date and DateTime64 --- src/Functions/FunctionsComparison.h | 5 ++- ..._Date_and_DateTime64_comparision.reference | 3 ++ .../01561_Date_and_DateTime64_comparision.sql | 43 +++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference create mode 100644 tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql diff --git a/src/Functions/FunctionsComparison.h b/src/Functions/FunctionsComparison.h index e674f8690ff..957c7e0ab3e 100644 --- a/src/Functions/FunctionsComparison.h +++ b/src/Functions/FunctionsComparison.h @@ -1216,7 +1216,10 @@ public: { return res; } - else if (isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type)) + else if ((isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type)) + // Comparing Date and DateTime64 requires implicit conversion, + // otherwise Date is treated as number. + && !(date_and_datetime && (isDate(left_type) || isDate(right_type)))) { // compare if (!allowDecimalComparison(left_type, right_type) && !date_and_datetime) diff --git a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference new file mode 100644 index 00000000000..e5183ec6a8a --- /dev/null +++ b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference @@ -0,0 +1,3 @@ +-1 DateTime64(1, \'UTC\') < 1 1 1 <= 1 1 1 = 0 0 0 >= 0 0 0 > 0 0 0 != 1 1 1 +0 DateTime64(1, \'UTC\') < 0 0 0 <= 0 1 0 = 0 1 0 >= 1 1 1 > 1 0 1 != 1 0 1 +1 DateTime64(1, \'UTC\') < 0 0 0 <= 0 0 0 = 0 0 0 >= 1 1 1 > 1 1 1 != 1 1 1 diff --git a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql new file mode 100644 index 00000000000..b780793a777 --- /dev/null +++ b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql @@ -0,0 +1,43 @@ +SELECT + n, + toTypeName(dt64) AS dt64_typename, + + '<', + dt64 < d, + toDate(dt64) < d, + dt64 < toDateTime64(d, 1, 'UTC'), + + '<=', + dt64 <= d, + toDate(dt64) <= d, + dt64 <= toDateTime64(d, 1, 'UTC'), + + '=', + dt64 = d, + toDate(dt64) = d, + dt64 = toDateTime64(d, 1, 'UTC'), + + '>=', + dt64 >= d, + toDate(dt64) >= d, + dt64 >= toDateTime64(d, 1, 'UTC'), + + '>', + dt64 > d, + toDate(dt64) > d, + dt64 > toDateTime64(d, 1, 'UTC'), + + '!=', + dt64 != d, + toDate(dt64) != d, + dt64 != toDateTime64(d, 1, 'UTC') +FROM +( + SELECT + number - 1 as n, + toDateTime64(toStartOfInterval(now(), toIntervalSecond(1), 'UTC'), 1, 'UTC') AS dt64, + toDate(now(), 'UTC') - n as d + FROM system.numbers + LIMIT 3 +) +FORMAT TabSeparated From 91e3162e09b5310e98b5b4e7d52c8c36084efc84 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 8 Dec 2020 17:28:51 +0300 Subject: [PATCH 097/174] More correct session close --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 20 ++--- src/Common/ZooKeeper/TestKeeperStorage.h | 2 - src/Server/TestKeeperTCPHandler.cpp | 78 ++++++++++++------- src/Server/TestKeeperTCPHandler.h | 2 +- .../test_testkeeper_back_to_back/test.py | 73 ++++++++++++++++- 5 files changed, 127 insertions(+), 48 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index aa44f3142f2..ea5b74a93ba 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -549,6 +549,12 @@ void TestKeeperStorage::processingThread() ephemerals.erase(it); } clearDeadWatches(info.session_id); + + /// Finish connection + auto response = std::make_shared(); + response->xid = zk_request->xid; + response->zxid = getZXID(); + info.response_callback(response); } else { @@ -725,20 +731,6 @@ TestKeeperWrapperFactory::TestKeeperWrapperFactory() registerTestKeeperRequestWrapper(*this); } - -void TestKeeperStorage::putCloseRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id) -{ - TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); - RequestInfo request_info; - request_info.time = clock::now(); - request_info.request = storage_request; - request_info.session_id = session_id; - std::lock_guard lock(push_request_mutex); - if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) - throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); -} - - void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback) { TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h index 80b871554d2..afb0a7add82 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.h +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -91,8 +91,6 @@ public: void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback); void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback); - void putCloseRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id); - int64_t getSessionID() { return session_id_counter.fetch_add(1); diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index 998e734af0a..aeb80c111a7 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -290,6 +290,7 @@ void TestKeeperTCPHandler::runImpl() sendHandshake(); session_stopwatch.start(); + bool close_received = false; while (true) { @@ -304,6 +305,17 @@ void TestKeeperTCPHandler::runImpl() if (received_op == Coordination::OpNum::Close) { LOG_DEBUG(log, "Received close request for session #{}", session_id); + if (responses.back().wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + { + LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + } + else + { + LOG_DEBUG(log, "Sent close for session #{}", session_id); + responses.back().get()->write(*out); + } + close_received = true; + break; } else if (received_op == Coordination::OpNum::Heartbeat) @@ -315,6 +327,9 @@ void TestKeeperTCPHandler::runImpl() while (in->available()); } + if (close_received) + break; + if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) { while (!responses.empty()) @@ -354,16 +369,32 @@ void TestKeeperTCPHandler::runImpl() if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) { LOG_DEBUG(log, "Session #{} expired", session_id); - putCloseRequest(); + auto response = putCloseRequest(); + if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + { + LOG_DEBUG(log, "Cannot sent close for expired session #{}", session_id); + } + else + { + response.get()->write(*out); + } + break; } } } -void TestKeeperTCPHandler::putCloseRequest() +zkutil::TestKeeperStorage::AsyncResponse TestKeeperTCPHandler::putCloseRequest() { Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(Coordination::OpNum::Close); - test_keeper_storage->putCloseRequest(request, session_id); + request->xid = Coordination::CLOSE_XID; + auto promise = std::make_shared>(); + zkutil::ResponseCallback callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) + { + promise->set_value(response); + }; + test_keeper_storage->putRequest(request, session_id, callback); + return promise->get_future(); } Coordination::OpNum TestKeeperTCPHandler::receiveRequest() @@ -380,36 +411,29 @@ Coordination::OpNum TestKeeperTCPHandler::receiveRequest() request->xid = xid; request->readImpl(*in); int response_fd = poll_wrapper->getResponseFD(); - if (opnum != Coordination::OpNum::Close) + auto promise = std::make_shared>(); + zkutil::ResponseCallback callback = [response_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) { - auto promise = std::make_shared>(); - zkutil::ResponseCallback callback = [response_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) - { - promise->set_value(response); - [[maybe_unused]] int result = write(response_fd, &RESPONSE_BYTE, sizeof(RESPONSE_BYTE)); - }; + promise->set_value(response); + [[maybe_unused]] int result = write(response_fd, &RESPONSE_BYTE, sizeof(RESPONSE_BYTE)); + }; - if (request->has_watch) + if (request->has_watch) + { + auto watch_promise = std::make_shared>(); + zkutil::ResponseCallback watch_callback = [response_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) { - auto watch_promise = std::make_shared>(); - zkutil::ResponseCallback watch_callback = [response_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) - { - watch_promise->set_value(response); - [[maybe_unused]] int result = write(response_fd, &WATCH_RESPONSE_BYTE, sizeof(WATCH_RESPONSE_BYTE)); - }; - test_keeper_storage->putRequest(request, session_id, callback, watch_callback); - responses.push(promise->get_future()); - watch_responses.emplace_back(watch_promise->get_future()); - } - else - { - test_keeper_storage->putRequest(request, session_id, callback); - responses.push(promise->get_future()); - } + watch_promise->set_value(response); + [[maybe_unused]] int result = write(response_fd, &WATCH_RESPONSE_BYTE, sizeof(WATCH_RESPONSE_BYTE)); + }; + test_keeper_storage->putRequest(request, session_id, callback, watch_callback); + responses.push(promise->get_future()); + watch_responses.emplace_back(watch_promise->get_future()); } else { - test_keeper_storage->putCloseRequest(request, session_id); + test_keeper_storage->putRequest(request, session_id, callback); + responses.push(promise->get_future()); } return opnum; diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h index f2674ff0f75..2f36dfafddf 100644 --- a/src/Server/TestKeeperTCPHandler.h +++ b/src/Server/TestKeeperTCPHandler.h @@ -46,7 +46,7 @@ private: void receiveHandshake(); Coordination::OpNum receiveRequest(); - void putCloseRequest(); + zkutil::TestKeeperStorage::AsyncResponse putCloseRequest(); }; } diff --git a/tests/integration/test_testkeeper_back_to_back/test.py b/tests/integration/test_testkeeper_back_to_back/test.py index 5ceb97572a2..a5c59396268 100644 --- a/tests/integration/test_testkeeper_back_to_back/test.py +++ b/tests/integration/test_testkeeper_back_to_back/test.py @@ -45,10 +45,12 @@ def started_cluster(): finally: cluster.shutdown() - _genuine_zk_instance.stop() - _genuine_zk_instance.close() - _fake_zk_instance.stop() - _fake_zk_instance.close() + if _genuine_zk_instance: + _genuine_zk_instance.stop() + _genuine_zk_instance.close() + if _fake_zk_instance: + _fake_zk_instance.stop() + _fake_zk_instance.close() def test_simple_commands(started_cluster): @@ -335,3 +337,66 @@ def test_random_requests(started_cluster): root_children_genuine = [elem for elem in list(sorted(genuine_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] root_children_fake = [elem for elem in list(sorted(fake_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] assert root_children_fake == root_children_genuine + +def test_end_of_session(started_cluster): + fake_zk1 = None + fake_zk2 = None + genuine_zk1 = None + genuine_zk2 = None + + try: + fake_zk1 = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + fake_zk1.start() + fake_zk2 = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + fake_zk2.start() + genuine_zk1 = cluster.get_kazoo_client('zoo1') + genuine_zk1.start() + genuine_zk2 = cluster.get_kazoo_client('zoo1') + genuine_zk2.start() + + fake_zk1.create("/test_end_of_session") + genuine_zk1.create("/test_end_of_session") + fake_ephemeral_event = None + def fake_ephemeral_callback(event): + print("Fake watch triggered") + nonlocal fake_ephemeral_event + fake_ephemeral_event = event + + genuine_ephemeral_event = None + def genuine_ephemeral_callback(event): + print("Genuine watch triggered") + nonlocal genuine_ephemeral_event + genuine_ephemeral_event = event + + assert fake_zk2.exists("/test_end_of_session") is not None + assert genuine_zk2.exists("/test_end_of_session") is not None + + fake_zk1.create("/test_end_of_session/ephemeral_node", ephemeral=True) + genuine_zk1.create("/test_end_of_session/ephemeral_node", ephemeral=True) + + assert fake_zk2.exists("/test_end_of_session/ephemeral_node", watch=fake_ephemeral_callback) is not None + assert genuine_zk2.exists("/test_end_of_session/ephemeral_node", watch=genuine_ephemeral_callback) is not None + + print("Stopping genuine zk") + genuine_zk1.stop() + print("Closing genuine zk") + genuine_zk1.close() + + print("Stopping fake zk") + fake_zk1.stop() + print("Closing fake zk") + fake_zk1.close() + + assert fake_zk2.exists("/test_end_of_session/ephemeral_node") is None + assert genuine_zk2.exists("/test_end_of_session/ephemeral_node") is None + + assert fake_ephemeral_event == genuine_ephemeral_event + + finally: + try: + for zk in [fake_zk1, fake_zk2, genuine_zk1, genuine_zk2]: + if zk: + zk.stop() + zk.close() + except: + pass From 8b0fe1404aedbd8025d117d1701437bbf93947c4 Mon Sep 17 00:00:00 2001 From: alesapin Date: Tue, 8 Dec 2020 17:50:10 +0300 Subject: [PATCH 098/174] Tiny sleeps --- tests/integration/test_testkeeper_back_to_back/test.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/integration/test_testkeeper_back_to_back/test.py b/tests/integration/test_testkeeper_back_to_back/test.py index 6295d5e6b02..04a2150e7f8 100644 --- a/tests/integration/test_testkeeper_back_to_back/test.py +++ b/tests/integration/test_testkeeper_back_to_back/test.py @@ -157,6 +157,7 @@ def test_watchers(started_cluster): genuine_zk.set("/test_data_watches", b"a") print("Calling set fake") fake_zk.set("/test_data_watches", b"a") + time.sleep(3) print("Genuine data", genuine_data_watch_data) print("Fake data", fake_data_watch_data) @@ -182,6 +183,8 @@ def test_watchers(started_cluster): print("Calling fake child") fake_zk.create("/test_data_watches/child", b"b") + time.sleep(3) + print("Genuine children", genuine_children) print("Fake children", fake_children) assert genuine_children == fake_children From 767d1fc98b7b08df3ea9f7cd71dd0f02dbb8b827 Mon Sep 17 00:00:00 2001 From: Nikita Mikhaylov Date: Tue, 8 Dec 2020 18:32:55 +0300 Subject: [PATCH 099/174] Update date-time-functions.md --- .../functions/date-time-functions.md | 52 +++++++++---------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index f7504d9efcb..233585194e9 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -693,14 +693,14 @@ SELECT FROM_UNIXTIME(1234334543, '%Y-%m-%d %R:%S') AS DateTime └─────────────────────┘ ``` -## toMJD {#tomjd} +## toModifiedJulianDay {#tomodifiedjulianday} Converts a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD` to a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number in Int32. This function supports date from `0000-01-01` to `9999-12-31`. It raises an exception if the argument cannot be parsed as a date, or the date is invalid. **Syntax** ``` sql -toMJD(date) +toModifiedJulianDay(date) ``` **Parameters** @@ -718,25 +718,25 @@ Type: [Int32](../../sql-reference/data-types/int-uint.md). Query: ``` sql -SELECT toMJD('2020-01-01'); +SELECT toModifiedJulianDay('2020-01-01'); ``` Result: ``` text -┌─toMJD('2020-01-01')─┐ -│ 58849 │ -└─────────────────────┘ +┌─toModifiedJulianDay('2020-01-01')─┐ +│ 58849 │ +└───────────────────────────────────┘ ``` -## toMJDOrNull {#tomjdornull} +## toModifiedJulianDayOrNull {#tomodifiedjuliandayornull} -Similar to [toMJD()](#tomjd), but instead of raising exceptions it returns `NULL`. +Similar to [toModifiedJulianDay()](#tomodifiedjulianday), but instead of raising exceptions it returns `NULL`. **Syntax** ``` sql -toMJDOrNull(date) +toModifiedJulianDayOrNull(date) ``` **Parameters** @@ -754,25 +754,25 @@ Type: [Nullable(Int32)](../../sql-reference/data-types/int-uint.md). Query: ``` sql -SELECT toMJDOrNull('2020-01-01'); +SELECT toModifiedJulianDayOrNull('2020-01-01'); ``` Result: ``` text -┌─toMJDOrNull('2020-01-01')─┐ -│ 58849 │ -└───────────────────────────┘ +┌─toModifiedJulianDayOrNull('2020-01-01')─┐ +│ 58849 │ +└─────────────────────────────────────────┘ ``` -## fromMJD {#frommjd} +## fromModifiedJulianDay {#frommodifiedjulianday} Converts a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number to a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD`. This function supports day number from `-678941` to `2973119` (which represent 0000-01-01 and 9999-12-31 respectively). It raises an exception if the day number is outside of the supported range. **Syntax** ``` sql -fromMJD(day) +fromModifiedJulianDay(day) ``` **Parameters** @@ -790,25 +790,25 @@ Type: [String](../../sql-reference/data-types/string.md) Query: ``` sql -SELECT fromMJD(58849); +SELECT fromModifiedJulianDay(58849); ``` Result: ``` text -┌─fromMJD(58849)─┐ -│ 2020-01-01 │ -└────────────────┘ +┌─fromModifiedJulianDay(58849)─┐ +│ 2020-01-01 │ +└──────────────────────────────┘ ``` -## fromMJDOrNull {#frommjdornull} +## fromModifiedJulianDayOrNull {#frommodifiedjuliandayornull} -Similar to [fromMJD()](#frommjd), but instead of raising exceptions it returns `NULL`. +Similar to [fromModifiedJulianDayOrNull()](#frommodifiedjuliandayornull), but instead of raising exceptions it returns `NULL`. **Syntax** ``` sql -fromMJDOrNull(day) +fromModifiedJulianDayOrNull(day) ``` **Parameters** @@ -826,13 +826,13 @@ Type: [Nullable(String)](../../sql-reference/data-types/string.md) Query: ``` sql -SELECT fromMJDOrNull(58849); +SELECT fromModifiedJulianDayOrNull(58849); ``` Result: ``` text -┌─fromMJDOrNull(58849)─┐ -│ 2020-01-01 │ -└──────────────────────┘ +┌─fromModifiedJulianDayOrNull(58849)─┐ +│ 2020-01-01 │ +└────────────────────────────────────┘ ``` From 71e9a8d9d0eb33c8b0c5a2ecb63f1b180ebea9ac Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Tue, 8 Dec 2020 18:39:49 +0300 Subject: [PATCH 100/174] build --- src/Functions/{fromMJD.cpp => fromModifiedJulianDay.cpp} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename src/Functions/{fromMJD.cpp => fromModifiedJulianDay.cpp} (97%) diff --git a/src/Functions/fromMJD.cpp b/src/Functions/fromModifiedJulianDay.cpp similarity index 97% rename from src/Functions/fromMJD.cpp rename to src/Functions/fromModifiedJulianDay.cpp index 3f8251e2caf..18d0b299c2e 100644 --- a/src/Functions/fromMJD.cpp +++ b/src/Functions/fromModifiedJulianDay.cpp @@ -208,14 +208,14 @@ namespace DB "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } - DataTypePtr baseType = std::make_shared(); + DataTypePtr base_type = std::make_shared(); if constexpr (nullOnErrors) { - return std::make_shared(baseType); + return std::make_shared(base_type); } else { - return baseType; + return base_type; } } From 52995b4d33444ea68da7b43920ebe1323896c06b Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov Date: Tue, 8 Dec 2020 19:42:03 +0300 Subject: [PATCH 101/174] Fix integer overflow in max log size calculation. If the `` was more than 4G, the logs were not rotated properly. Also log errors that occur during rotation of logs. CLICKHOUSE-4975 --- contrib/poco | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/poco b/contrib/poco index b5523bb9b4b..288320bf640 160000 --- a/contrib/poco +++ b/contrib/poco @@ -1 +1 @@ -Subproject commit b5523bb9b4bc4239640cbfec4d734be8b8585639 +Subproject commit 288320bf640425dbcde526422e371087f6f2bb81 From cb1a71fd3f935b83a0b48f70115bf0ca461dc933 Mon Sep 17 00:00:00 2001 From: Alexander Kuzmenkov Date: Tue, 8 Dec 2020 20:29:17 +0300 Subject: [PATCH 102/174] fixup --- contrib/poco | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/poco b/contrib/poco index 288320bf640..08974cc024b 160000 --- a/contrib/poco +++ b/contrib/poco @@ -1 +1 @@ -Subproject commit 288320bf640425dbcde526422e371087f6f2bb81 +Subproject commit 08974cc024b2e748f5b1d45415396706b3521d0f From e3a9cae4c538c1a9eed48550d7f5dddac9b994f0 Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Tue, 8 Dec 2020 20:43:43 +0300 Subject: [PATCH 103/174] better --- src/Functions/toModifiedJulianDay.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/Functions/toModifiedJulianDay.cpp b/src/Functions/toModifiedJulianDay.cpp index bd1ff7183ff..f11cfe5181c 100644 --- a/src/Functions/toModifiedJulianDay.cpp +++ b/src/Functions/toModifiedJulianDay.cpp @@ -201,14 +201,14 @@ namespace DB "The argument of function " + getName() + " must be String or FixedString", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } - DataTypePtr baseType = std::make_shared(); + DataTypePtr base_type = std::make_shared(); if constexpr (nullOnErrors) { - return std::make_shared(baseType); + return std::make_shared(base_type); } else { - return baseType; + return base_type; } } From 95ce4bc54c0a206e06ae0904638e70c26d486c1a Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Tue, 8 Dec 2020 21:28:18 +0300 Subject: [PATCH 104/174] style --- src/Functions/GregorianDate.h | 68 +++++++++---------------- src/Functions/fromModifiedJulianDay.cpp | 17 ++++--- src/Functions/toModifiedJulianDay.cpp | 2 + src/Functions/ya.make | 2 + 4 files changed, 39 insertions(+), 50 deletions(-) diff --git a/src/Functions/GregorianDate.h b/src/Functions/GregorianDate.h index e5caff3ae46..c5bb478f9d5 100644 --- a/src/Functions/GregorianDate.h +++ b/src/Functions/GregorianDate.h @@ -149,17 +149,14 @@ namespace DB /* Implementation */ -namespace gd { +namespace gd +{ using namespace DB; template static inline constexpr bool isLeapYear(YearT year) { - return ( (year % 4 == 0) - && - ( (year % 400 == 0) - || - (! (year % 100 == 0)) ) ); + return (year % 4 == 0) && ((year % 400 == 0) || (year % 100 != 0)); } static inline constexpr uint8_t monthLength(bool isLeapYear, uint8_t month) @@ -189,15 +186,12 @@ namespace gd { static inline constexpr I div(I x, J y) { const auto y_ = static_cast(y); - if (x > 0 && y_ < 0) { + if (x > 0 && y_ < 0) return ((x - 1) / y_) - 1; - } - else if (x < 0 && y_ > 0) { + else if (x < 0 && y_ > 0) return ((x + 1) / y_) - 1; - } - else { + else return x / y_; - } } /** Integer modulus, satisfying div(x, y)*y + mod(x, y) == x. @@ -207,12 +201,10 @@ namespace gd { { const auto y_ = static_cast(y); const auto r = x % y_; - if ((x > 0 && y_ < 0) || (x < 0 && y_ > 0)) { + if ((x > 0 && y_ < 0) || (x < 0 && y_ > 0)) return r == 0 ? static_cast(0) : r + y_; - } - else { + else return r; - } } /** Like std::min(), but the type of operands may differ. @@ -227,19 +219,16 @@ namespace gd { static inline char readDigit(ReadBuffer & in) { char c; - if (!in.read(c)) { + if (!in.read(c)) throw Exception( "Cannot parse input: expected a digit at the end of stream", ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); - } - else if (c < '0' || c > '9') { + else if (c < '0' || c > '9') throw Exception( "Cannot read input: expected a digit but got something else", ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); - } - else { + else return c - '0'; - } } } @@ -303,7 +292,8 @@ namespace DB "Impossible to stringify: year too big or small: " + DB::toString(year_), ErrorCodes::CANNOT_FORMAT_DATETIME); } - else { + else + { auto y = year_; writeChar('0' + y / 1000, buf); y %= 1000; writeChar('0' + y / 100, buf); y %= 100; @@ -367,22 +357,21 @@ namespace DB { const auto y = year_ - 1; return dayOfYear_ - + 365 * y - + gd::div(y, 4) - - gd::div(y, 100) - + gd::div(y, 400) - - 678576; + + 365 * y + + gd::div(y, 4) + - gd::div(y, 100) + + gd::div(y, 400) + - 678576; } inline MonthDay::MonthDay(uint8_t month, uint8_t dayOfMonth) : month_(month) , dayOfMonth_(dayOfMonth) { - if (month < 1 || month > 12) { + if (month < 1 || month > 12) throw Exception( "Invalid month: " + DB::toString(month), ErrorCodes::LOGICAL_ERROR); - } /* We can't validate dayOfMonth here, because we don't know if * it's a leap year. */ } @@ -390,25 +379,20 @@ namespace DB inline MonthDay::MonthDay(bool isLeapYear, uint16_t dayOfYear) { if (dayOfYear < 1 || dayOfYear > (isLeapYear ? 366 : 365)) - { throw Exception( std::string("Invalid day of year: ") + (isLeapYear ? "leap, " : "non-leap, ") + DB::toString(dayOfYear), ErrorCodes::LOGICAL_ERROR); - } month_ = 1; uint16_t d = dayOfYear; - while (true) { + while (true) + { const auto len = gd::monthLength(isLeapYear, month_); - if (d > len) - { - month_++; - d -= len; - } - else { + if (d <= len) break; - } + month_++; + d -= len; } dayOfMonth_ = d; } @@ -423,9 +407,7 @@ namespace DB "-" + DB::toString(dayOfMonth_), ErrorCodes::LOGICAL_ERROR); } - const auto k = month_ <= 2 ? 0 - : isLeapYear ? -1 - : -2; + const auto k = month_ <= 2 ? 0 : isLeapYear ? -1 :-2; return (367 * month_ - 362) / 12 + k + dayOfMonth_; } } diff --git a/src/Functions/fromModifiedJulianDay.cpp b/src/Functions/fromModifiedJulianDay.cpp index 18d0b299c2e..e62b6d4a38d 100644 --- a/src/Functions/fromModifiedJulianDay.cpp +++ b/src/Functions/fromModifiedJulianDay.cpp @@ -15,6 +15,13 @@ namespace DB { + + namespace ErrorCodes + { + extern const int CANNOT_FORMAT_DATETIME; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + } + template class ExecutableFunctionFromModifiedJulianDay : public IExecutableFunctionImpl { @@ -58,18 +65,15 @@ namespace DB catch (const Exception & e) { if (e.code() == ErrorCodes::CANNOT_FORMAT_DATETIME) - { (*vec_null_map_to)[i] = true; - } else - { throw; - } } writeChar(0, write_buffer); offsets_to[i] = write_buffer.count(); } - else { + else + { const GregorianDate<> gd(vec_from[i]); gd.write(write_buffer); writeChar(0, write_buffer); @@ -192,11 +196,10 @@ namespace DB { return std::make_unique>(argument_types, return_type); } - else { + else // Should not happen. throw Exception( "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - } } } diff --git a/src/Functions/toModifiedJulianDay.cpp b/src/Functions/toModifiedJulianDay.cpp index f11cfe5181c..ca024b007f7 100644 --- a/src/Functions/toModifiedJulianDay.cpp +++ b/src/Functions/toModifiedJulianDay.cpp @@ -17,6 +17,8 @@ namespace DB { extern const int ILLEGAL_COLUMN; extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int CANNOT_PARSE_INPUT_ASSERTION_FAILED; + extern const int CANNOT_PARSE_DATE; } template diff --git a/src/Functions/ya.make b/src/Functions/ya.make index 436a6a89996..b94c7112da3 100644 --- a/src/Functions/ya.make +++ b/src/Functions/ya.make @@ -247,6 +247,7 @@ SRCS( formatReadableTimeDelta.cpp formatRow.cpp formatString.cpp + fromModifiedJulianDay.cpp fromUnixTimestamp64Micro.cpp fromUnixTimestamp64Milli.cpp fromUnixTimestamp64Nano.cpp @@ -455,6 +456,7 @@ SRCS( toISOYear.cpp toLowCardinality.cpp toMinute.cpp + toModifiedJulianDay.cpp toMonday.cpp toMonth.cpp toNullable.cpp From dfbb31174e15597da7e83bcbc27887259820e29d Mon Sep 17 00:00:00 2001 From: Pavel Kovalenko Date: Tue, 8 Dec 2020 21:31:57 +0300 Subject: [PATCH 105/174] Ability to set metadata when put S3 object --- src/Disks/S3/DiskS3.cpp | 30 +++++++++++++++---- src/Disks/S3/DiskS3.h | 7 ++++- src/Disks/S3/registerDiskS3.cpp | 3 +- src/IO/WriteBufferFromS3.cpp | 6 ++++ src/IO/WriteBufferFromS3.h | 2 ++ .../test_log_family_s3/configs/config.xml | 12 -------- .../test_log_family_s3/configs/minio.xml | 1 + tests/integration/test_log_family_s3/test.py | 21 ++++++++----- 8 files changed, 56 insertions(+), 26 deletions(-) diff --git a/src/Disks/S3/DiskS3.cpp b/src/Disks/S3/DiskS3.cpp index 507af58f9fa..6090f00b4e2 100644 --- a/src/Disks/S3/DiskS3.cpp +++ b/src/Disks/S3/DiskS3.cpp @@ -3,6 +3,7 @@ #include "Disks/DiskFactory.h" #include +#include #include #include #include @@ -326,11 +327,19 @@ namespace const String & bucket_, Metadata metadata_, const String & s3_path_, + std::optional object_metadata_, bool is_multipart, size_t min_upload_part_size, size_t buf_size_) : WriteBufferFromFileBase(buf_size_, nullptr, 0) - , impl(WriteBufferFromS3(client_ptr_, bucket_, metadata_.s3_root_path + s3_path_, min_upload_part_size, is_multipart, buf_size_)) + , impl(WriteBufferFromS3( + client_ptr_, + bucket_, + metadata_.s3_root_path + s3_path_, + min_upload_part_size, + is_multipart, + std::move(object_metadata_), + buf_size_)) , metadata(std::move(metadata_)) , s3_path(s3_path_) { @@ -522,7 +531,8 @@ DiskS3::DiskS3( String metadata_path_, size_t min_upload_part_size_, size_t min_multi_part_upload_size_, - size_t min_bytes_for_seek_) + size_t min_bytes_for_seek_, + bool send_metadata_) : IDisk(std::make_unique()) , name(std::move(name_)) , client(std::move(client_)) @@ -533,6 +543,7 @@ DiskS3::DiskS3( , min_upload_part_size(min_upload_part_size_) , min_multi_part_upload_size(min_multi_part_upload_size_) , min_bytes_for_seek(min_bytes_for_seek_) + , send_metadata(send_metadata_) { } @@ -653,6 +664,7 @@ std::unique_ptr DiskS3::writeFile(const String & path, } /// Path to store new S3 object. auto s3_path = getRandomName(); + auto object_metadata = createObjectMetadata(path); bool is_multipart = estimated_size >= min_multi_part_upload_size; if (!exist || mode == WriteMode::Rewrite) { @@ -664,9 +676,9 @@ std::unique_ptr DiskS3::writeFile(const String & path, /// Save empty metadata to disk to have ability to get file size while buffer is not finalized. metadata.save(); - LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Write to file by path: {} New S3 path: {}", backQuote(metadata_path + path), s3_root_path + s3_path); + LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Write to file by path: {}. New S3 path: {}", backQuote(metadata_path + path), s3_root_path + s3_path); - return std::make_unique(client, bucket, metadata, s3_path, is_multipart, min_upload_part_size, buf_size); + return std::make_unique(client, bucket, metadata, s3_path, object_metadata, is_multipart, min_upload_part_size, buf_size); } else { @@ -675,7 +687,7 @@ std::unique_ptr DiskS3::writeFile(const String & path, LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Append to file by path: {}. New S3 path: {}. Existing S3 objects: {}.", backQuote(metadata_path + path), s3_root_path + s3_path, metadata.s3_objects.size()); - return std::make_unique(client, bucket, metadata, s3_path, is_multipart, min_upload_part_size, buf_size); + return std::make_unique(client, bucket, metadata, s3_path, object_metadata, is_multipart, min_upload_part_size, buf_size); } } @@ -847,4 +859,12 @@ void DiskS3::shutdown() client->DisableRequestProcessing(); } +std::optional DiskS3::createObjectMetadata(const String & path) const +{ + if (send_metadata) + return (DiskS3::ObjectMetadata){{"path", path}}; + + return {}; +} + } diff --git a/src/Disks/S3/DiskS3.h b/src/Disks/S3/DiskS3.h index fe8c47931b5..f3d20da5ca0 100644 --- a/src/Disks/S3/DiskS3.h +++ b/src/Disks/S3/DiskS3.h @@ -19,6 +19,8 @@ namespace DB class DiskS3 : public IDisk { public: + using ObjectMetadata = std::map; + friend class DiskS3Reservation; class AwsS3KeyKeeper; @@ -32,7 +34,8 @@ public: String metadata_path_, size_t min_upload_part_size_, size_t min_multi_part_upload_size_, - size_t min_bytes_for_seek_); + size_t min_bytes_for_seek_, + bool send_metadata_); const String & getName() const override { return name; } @@ -116,6 +119,7 @@ private: void removeMeta(const String & path, AwsS3KeyKeeper & keys); void removeMetaRecursive(const String & path, AwsS3KeyKeeper & keys); void removeAws(const AwsS3KeyKeeper & keys); + std::optional createObjectMetadata(const String & path) const; private: const String name; @@ -127,6 +131,7 @@ private: size_t min_upload_part_size; size_t min_multi_part_upload_size; size_t min_bytes_for_seek; + bool send_metadata; UInt64 reserved_bytes = 0; UInt64 reservation_count = 0; diff --git a/src/Disks/S3/registerDiskS3.cpp b/src/Disks/S3/registerDiskS3.cpp index 5078a7e06ae..fbf5b40dac4 100644 --- a/src/Disks/S3/registerDiskS3.cpp +++ b/src/Disks/S3/registerDiskS3.cpp @@ -148,7 +148,8 @@ void registerDiskS3(DiskFactory & factory) metadata_path, context.getSettingsRef().s3_min_upload_part_size, config.getUInt64(config_prefix + ".min_multi_part_upload_size", 10 * 1024 * 1024), - config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024)); + config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024), + config.getBool(config_prefix + ".send_object_metadata", false)); /// This code is used only to check access to the corresponding disk. if (!config.getBool(config_prefix + ".skip_access_check", false)) diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index a32aa4acdc9..e4fb0634bd0 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -43,11 +43,13 @@ WriteBufferFromS3::WriteBufferFromS3( const String & key_, size_t minimum_upload_part_size_, bool is_multipart_, + std::optional> object_metadata_, size_t buffer_size_) : BufferWithOwnMemory(buffer_size_, nullptr, 0) , is_multipart(is_multipart_) , bucket(bucket_) , key(key_) + , object_metadata(std::move(object_metadata_)) , client_ptr(std::move(client_ptr_)) , minimum_upload_part_size{minimum_upload_part_size_} , temporary_buffer{std::make_unique()} @@ -116,6 +118,8 @@ void WriteBufferFromS3::initiate() Aws::S3::Model::CreateMultipartUploadRequest req; req.SetBucket(bucket); req.SetKey(key); + if (object_metadata.has_value()) + req.SetMetadata(object_metadata.value()); auto outcome = client_ptr->CreateMultipartUpload(req); @@ -217,6 +221,8 @@ void WriteBufferFromS3::complete() Aws::S3::Model::PutObjectRequest req; req.SetBucket(bucket); req.SetKey(key); + if (object_metadata.has_value()) + req.SetMetadata(object_metadata.value()); /// This could be improved using an adapter to WriteBuffer. const std::shared_ptr input_data = Aws::MakeShared("temporary buffer", temporary_buffer->str()); diff --git a/src/IO/WriteBufferFromS3.h b/src/IO/WriteBufferFromS3.h index 1a1e859d913..9bd7b896d7f 100644 --- a/src/IO/WriteBufferFromS3.h +++ b/src/IO/WriteBufferFromS3.h @@ -28,6 +28,7 @@ private: String bucket; String key; + std::optional> object_metadata; std::shared_ptr client_ptr; size_t minimum_upload_part_size; std::unique_ptr temporary_buffer; @@ -47,6 +48,7 @@ public: const String & key_, size_t minimum_upload_part_size_, bool is_multipart, + std::optional> object_metadata_ = std::nullopt, size_t buffer_size_ = DBMS_DEFAULT_BUFFER_SIZE); void nextImpl() override; diff --git a/tests/integration/test_log_family_s3/configs/config.xml b/tests/integration/test_log_family_s3/configs/config.xml index 63b4d951eb7..5b9b5b5843a 100644 --- a/tests/integration/test_log_family_s3/configs/config.xml +++ b/tests/integration/test_log_family_s3/configs/config.xml @@ -8,18 +8,6 @@ 10 - - - - s3 - http://minio1:9001/root/data/ - minio - minio123 - - - - - 9000 127.0.0.1 diff --git a/tests/integration/test_log_family_s3/configs/minio.xml b/tests/integration/test_log_family_s3/configs/minio.xml index 6c9329a2bbc..edcf1225142 100644 --- a/tests/integration/test_log_family_s3/configs/minio.xml +++ b/tests/integration/test_log_family_s3/configs/minio.xml @@ -7,6 +7,7 @@ http://minio1:9001/root/data/ minio minio123 + true diff --git a/tests/integration/test_log_family_s3/test.py b/tests/integration/test_log_family_s3/test.py index 40e263c9c69..946e96f6ab2 100644 --- a/tests/integration/test_log_family_s3/test.py +++ b/tests/integration/test_log_family_s3/test.py @@ -23,30 +23,37 @@ def cluster(): cluster.shutdown() +def assert_objects_count(cluster, objects_count, path='data/'): + minio = cluster.minio_client + s3_objects = list(minio.list_objects(cluster.minio_bucket, path)) + if objects_count != len(s3_objects): + for s3_object in s3_objects: + object_meta = minio.stat_object(cluster.minio_bucket, s3_object.object_name) + logging.info("Existing S3 object: %s", str(object_meta)) + assert objects_count == len(s3_objects) + + @pytest.mark.parametrize( "log_engine,files_overhead,files_overhead_per_insert", [("TinyLog", 1, 1), ("Log", 2, 1), ("StripeLog", 1, 2)]) def test_log_family_s3(cluster, log_engine, files_overhead, files_overhead_per_insert): node = cluster.instances["node"] - minio = cluster.minio_client node.query("CREATE TABLE s3_test (id UInt64) Engine={}".format(log_engine)) node.query("INSERT INTO s3_test SELECT number FROM numbers(5)") assert node.query("SELECT * FROM s3_test") == "0\n1\n2\n3\n4\n" - assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert + files_overhead + assert_objects_count(cluster, files_overhead_per_insert + files_overhead) node.query("INSERT INTO s3_test SELECT number + 5 FROM numbers(3)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n" - assert len( - list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 2 + files_overhead + assert_objects_count(cluster, files_overhead_per_insert * 2 + files_overhead) node.query("INSERT INTO s3_test SELECT number + 8 FROM numbers(1)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n8\n" - assert len( - list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 3 + files_overhead + assert_objects_count(cluster, files_overhead_per_insert * 3 + files_overhead) node.query("TRUNCATE TABLE s3_test") - assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == 0 + assert_objects_count(cluster, 0) node.query("DROP TABLE s3_test") From e95e44324654692afba3383d32a216ad06f444bb Mon Sep 17 00:00:00 2001 From: feng lv Date: Wed, 9 Dec 2020 04:30:38 +0000 Subject: [PATCH 106/174] fix fix --- src/Interpreters/TreeRewriter.cpp | 68 ++++++++++++------- .../01562_agg_null_for_empty_ahead.reference | 6 +- .../01562_agg_null_for_empty_ahead.sql | 1 + 3 files changed, 49 insertions(+), 26 deletions(-) diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index c86b1892b16..5df5d6f7a42 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -121,43 +121,61 @@ struct CustomizeAggregateFunctionsSuffixData void visit(ASTFunction & func, ASTPtr &) const { const auto & instance = AggregateFunctionFactory::instance(); - if (instance.isAggregateFunctionName(func.name) && !endsWith(func.name, customized_func_suffix)) + if (instance.isAggregateFunctionName(func.name)) { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) { - auto name_size = func.name.size(); - if (endsWith(func.name, "MergeState")) - { - func.name = func.name.substr(0, name_size - 10) + customized_func_suffix + "MergeState"; - return; - } + if (!endsWith(func.name, customized_func_suffix)) + func.name += customized_func_suffix; + } + } + } +}; - if (endsWith(func.name, "Merge")) - { - func.name = func.name.substr(0, name_size - 5) + customized_func_suffix + "Merge"; - return; - } +struct CustomizeAggregateFunctionsMoveSuffixData +{ + using TypeToVisit = ASTFunction; - if (endsWith(func.name, "State")) - { - func.name = func.name.substr(0, name_size - 5) + customized_func_suffix + "State"; - return; - } + const String & customized_func_suffix; - if (endsWith(func.name, "If")) - { - func.name = func.name.substr(0, name_size - 2) + customized_func_suffix + "If"; - return; - } + String moveSuffixAhead(const String & name) const + { + auto prefix = name.substr(0, name.size() - customized_func_suffix.size()); - func.name = func.name + customized_func_suffix; + auto prefix_size = prefix.size(); + + if (endsWith(prefix, "MergeState")) + return prefix.substr(0, prefix_size - 10) + customized_func_suffix + "MergeState"; + + if (endsWith(prefix, "Merge")) + return prefix.substr(0, prefix_size - 5) + customized_func_suffix + "Merge"; + + if (endsWith(prefix, "State")) + return prefix.substr(0, prefix_size - 5) + customized_func_suffix + "State"; + + if (endsWith(prefix, "If")) + return prefix.substr(0, prefix_size - 2) + customized_func_suffix + "If"; + + return name; + } + + void visit(ASTFunction & func, ASTPtr &) const + { + const auto & instance = AggregateFunctionFactory::instance(); + if (instance.isAggregateFunctionName(func.name)) + { + auto properties = instance.tryGetProperties(func.name); + if (properties && !properties->returns_default_when_only_null) + { + func.name = moveSuffixAhead(func.name); } } } }; using CustomizeAggregateFunctionsOrNullVisitor = InDepthNodeVisitor, true>; +using CustomizeAggregateFunctionsMoveOrNullVisitor = InDepthNodeVisitor, true>; /// Translate qualified names such as db.table.column, table.column, table_alias.column to names' normal form. /// Expand asterisks and qualified asterisks with column names. @@ -780,6 +798,10 @@ void TreeRewriter::normalize(ASTPtr & query, Aliases & aliases, const Settings & CustomizeAggregateFunctionsOrNullVisitor(data_or_null).visit(query); } + /// Move -OrNull suffix ahead, this should execute after add -OrNull suffix + CustomizeAggregateFunctionsMoveOrNullVisitor::Data data_or_null{"OrNull"}; + CustomizeAggregateFunctionsMoveOrNullVisitor(data_or_null).visit(query); + /// Creates a dictionary `aliases`: alias -> ASTPtr QueryAliasesVisitor(aliases).visit(query); diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference index 31133cfb3a1..bda1162f4e9 100644 --- a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference @@ -2,9 +2,9 @@ 0 0 0 -0 +\N 1 -0 +\N \N 1 \N @@ -14,7 +14,7 @@ 0 \N 1 -0 +\N \N 1 \N diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql index 50ce4a84ee0..834204fedb9 100644 --- a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql @@ -8,6 +8,7 @@ SELECT sumIf(1, 0); SELECT sumIf(1, 1); +-- should return Null even if we donn't set aggregate_functions_null_for_empty SELECT sumIfOrNull(1, 0); SELECT sumOrNullIf(1, 0); From 7546c611fbb546920a4876b9f0a8727ad2b7bbfd Mon Sep 17 00:00:00 2001 From: feng lv Date: Wed, 9 Dec 2020 08:11:14 +0000 Subject: [PATCH 107/174] add test --- .../01600_min_max_compress_block_size.reference | 1 + .../0_stateless/01600_min_max_compress_block_size.sql | 9 +++++++++ 2 files changed, 10 insertions(+) create mode 100644 tests/queries/0_stateless/01600_min_max_compress_block_size.reference create mode 100644 tests/queries/0_stateless/01600_min_max_compress_block_size.sql diff --git a/tests/queries/0_stateless/01600_min_max_compress_block_size.reference b/tests/queries/0_stateless/01600_min_max_compress_block_size.reference new file mode 100644 index 00000000000..83b33d238da --- /dev/null +++ b/tests/queries/0_stateless/01600_min_max_compress_block_size.reference @@ -0,0 +1 @@ +1000 diff --git a/tests/queries/0_stateless/01600_min_max_compress_block_size.sql b/tests/queries/0_stateless/01600_min_max_compress_block_size.sql new file mode 100644 index 00000000000..747f0b736ce --- /dev/null +++ b/tests/queries/0_stateless/01600_min_max_compress_block_size.sql @@ -0,0 +1,9 @@ +DROP TABLE IF EXISTS ms; + +CREATE TABLE ms (n Int32) ENGINE = MergeTree() ORDER BY n SETTINGS min_compress_block_size = 1024, max_compress_block_size = 10240; + +INSERT INTO ms SELECT * FROM numbers(1000); + +SELECT COUNT(*) FROM ms; + +DROP TABLE ms; From 2f735aa61d064a6271270621501b32f6f3c86f69 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Wed, 9 Dec 2020 16:19:09 +0800 Subject: [PATCH 108/174] ISSUES-16835 try fix review comment --- base/mysqlxx/ResultBase.cpp | 8 +++ base/mysqlxx/ResultBase.h | 2 + base/mysqlxx/Row.cpp | 8 --- base/mysqlxx/Row.h | 2 - src/Formats/MySQLBlockInputStream.cpp | 99 +++++++++++++-------------- src/Formats/MySQLBlockInputStream.h | 1 + 6 files changed, 60 insertions(+), 60 deletions(-) diff --git a/base/mysqlxx/ResultBase.cpp b/base/mysqlxx/ResultBase.cpp index eac1e22ca3d..2461ad7c9ce 100644 --- a/base/mysqlxx/ResultBase.cpp +++ b/base/mysqlxx/ResultBase.cpp @@ -22,4 +22,12 @@ ResultBase::~ResultBase() mysql_free_result(res); } +std::string ResultBase::getFieldName(size_t n) const +{ + if (num_fields <= n) + throw Exception(std::string("Unknown column position ") + std::to_string(n)); + + return fields[n].name; +} + } diff --git a/base/mysqlxx/ResultBase.h b/base/mysqlxx/ResultBase.h index b72b5682122..4f2ab2eb0a2 100644 --- a/base/mysqlxx/ResultBase.h +++ b/base/mysqlxx/ResultBase.h @@ -31,6 +31,8 @@ public: MYSQL_RES * getRes() { return res; } const Query * getQuery() const { return query; } + std::string getFieldName(size_t n) const; + virtual ~ResultBase(); protected: diff --git a/base/mysqlxx/Row.cpp b/base/mysqlxx/Row.cpp index bfc18689403..aecec46e519 100644 --- a/base/mysqlxx/Row.cpp +++ b/base/mysqlxx/Row.cpp @@ -21,12 +21,4 @@ Value Row::operator[] (const char * name) const throw Exception(std::string("Unknown column ") + name); } -std::string Row::getFieldName(size_t n) const -{ - if (res->getNumFields() <= n) - throw Exception(std::string("Unknown column position ") + std::to_string(n)); - - return res->getFields()[n].name; -} - } diff --git a/base/mysqlxx/Row.h b/base/mysqlxx/Row.h index 547ee35e52e..a0b88638546 100644 --- a/base/mysqlxx/Row.h +++ b/base/mysqlxx/Row.h @@ -79,8 +79,6 @@ public: */ operator private_bool_type() const { return row == nullptr ? nullptr : &Row::row; } - std::string getFieldName(size_t n) const; - private: MYSQL_ROW row{}; ResultBase * res{}; diff --git a/src/Formats/MySQLBlockInputStream.cpp b/src/Formats/MySQLBlockInputStream.cpp index 6cbec0650af..2ff8e8e5fb2 100644 --- a/src/Formats/MySQLBlockInputStream.cpp +++ b/src/Formats/MySQLBlockInputStream.cpp @@ -45,12 +45,8 @@ MySQLBlockInputStream::MySQLBlockInputStream( , auto_close{auto_close_} , fetch_by_name(fetch_by_name_) { - if (!fetch_by_name && sample_block.columns() != connection->result.getNumFields()) - throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " - + toString(sample_block.columns()) + " expected", - ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; - description.init(sample_block); + initPositionMappingFromQueryResultStructure(); } @@ -135,47 +131,6 @@ Block MySQLBlockInputStream::readImpl() for (const auto i : ext::range(0, columns.size())) columns[i] = description.sample_block.getByPosition(i).column->cloneEmpty(); - if (unlikely(position_mapping.size() != description.sample_block.columns())) - { - position_mapping.resize(description.sample_block.columns()); - - if (!fetch_by_name) - { - for (const auto idx : ext::range(0, row.size())) - position_mapping[idx] = idx; - } - else - { - const auto & sample_names = description.sample_block.getNames(); - std::unordered_set missing_names(sample_names.begin(), sample_names.end()); - - for (const auto idx : ext::range(0, row.size())) - { - const auto & field_name = row.getFieldName(idx); - if (description.sample_block.has(field_name)) - { - const auto & position = description.sample_block.getPositionByName(field_name); - position_mapping[position] = idx; - missing_names.erase(field_name); - } - } - - if (!missing_names.empty()) - { - WriteBufferFromOwnString exception_message; - for (auto iter = missing_names.begin(); iter != missing_names.end(); ++iter) - { - if (iter != missing_names.begin()) - exception_message << ", "; - exception_message << *iter; - } - - throw Exception("mysqlxx::UseQueryResult must be contain the" + exception_message.str() + " columns.", - ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); - } - } - } - size_t num_rows = 0; while (row) { @@ -221,6 +176,53 @@ MySQLBlockInputStream::MySQLBlockInputStream( description.init(sample_block_); } +void MySQLBlockInputStream::initPositionMappingFromQueryResultStructure() +{ + position_mapping.resize(description.sample_block.columns()); + + if (!fetch_by_name) + { + if (description.sample_block.columns() != connection->result.getNumFields()) + throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " + + toString(description.sample_block.columns()) + " expected", ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; + + for (const auto idx : ext::range(0, connection->result.getNumFields())) + position_mapping[idx] = idx; + } + else + { + const auto & sample_names = description.sample_block.getNames(); + std::unordered_set missing_names(sample_names.begin(), sample_names.end()); + + size_t fields_size = connection->result.getNumFields(); + + for (const size_t & idx : ext::range(0, fields_size)) + { + const auto & field_name = connection->result.getFieldName(idx); + if (description.sample_block.has(field_name)) + { + const auto & position = description.sample_block.getPositionByName(field_name); + position_mapping[position] = idx; + missing_names.erase(field_name); + } + } + + if (!missing_names.empty()) + { + WriteBufferFromOwnString exception_message; + for (auto iter = missing_names.begin(); iter != missing_names.end(); ++iter) + { + if (iter != missing_names.begin()) + exception_message << ", "; + exception_message << *iter; + } + + throw Exception("mysqlxx::UseQueryResult must be contain the" + exception_message.str() + " columns.", + ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); + } + } +} + MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( mysqlxx::Pool & pool_, const std::string & query_str_, @@ -237,10 +239,7 @@ MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( void MySQLLazyBlockInputStream::readPrefix() { connection = std::make_unique(pool.get(), query_str); - if (!fetch_by_name && description.sample_block.columns() != connection->result.getNumFields()) - throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " - + toString(description.sample_block.columns()) + " expected", - ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; + initPositionMappingFromQueryResultStructure(); } } diff --git a/src/Formats/MySQLBlockInputStream.h b/src/Formats/MySQLBlockInputStream.h index a516e81ac6c..269b630fcc7 100644 --- a/src/Formats/MySQLBlockInputStream.h +++ b/src/Formats/MySQLBlockInputStream.h @@ -30,6 +30,7 @@ public: protected: MySQLBlockInputStream(const Block & sample_block_, UInt64 max_block_size_, bool auto_close_, bool fetch_by_name_); Block readImpl() override; + void initPositionMappingFromQueryResultStructure(); struct Connection { From 7ff50fb35278932360ebfc2f009aaac20198aef8 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 11:19:52 +0300 Subject: [PATCH 109/174] Close requests without timeout --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 11 +- src/Server/TestKeeperTCPHandler.cpp | 149 +++++++++++---------- 2 files changed, 86 insertions(+), 74 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index ea5b74a93ba..6513c9c1050 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -739,8 +739,11 @@ void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & req request_info.request = storage_request; request_info.session_id = session_id; request_info.response_callback = callback; + + /// Put close requests without timeouts + auto timeout = request->getOpNum() == Coordination::OpNum::Close ? 0 : operation_timeout.totalMilliseconds(); std::lock_guard lock(push_request_mutex); - if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) + if (!requests_queue.tryPush(std::move(request_info), timeout)) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); } @@ -753,12 +756,12 @@ void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & req request_info.session_id = session_id; request_info.response_callback = callback; if (request->has_watch) - { request_info.watch_callback = watch_callback; - } + /// Put close requests without timeouts + auto timeout = request->getOpNum() == Coordination::OpNum::Close ? 0 : operation_timeout.totalMilliseconds(); std::lock_guard lock(push_request_mutex); - if (!requests_queue.tryPush(std::move(request_info), operation_timeout.totalMilliseconds())) + if (!requests_queue.tryPush(std::move(request_info), timeout)) throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); } diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp index aeb80c111a7..ffb73985824 100644 --- a/src/Server/TestKeeperTCPHandler.cpp +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -291,97 +291,106 @@ void TestKeeperTCPHandler::runImpl() sendHandshake(); session_stopwatch.start(); bool close_received = false; - - while (true) + try { - using namespace std::chrono_literals; - - auto state = poll_wrapper->poll(session_timeout); - if (state & SocketInterruptablePollWrapper::HAS_REQUEST) + while (true) { - do + using namespace std::chrono_literals; + + auto state = poll_wrapper->poll(session_timeout); + if (state & SocketInterruptablePollWrapper::HAS_REQUEST) { - auto received_op = receiveRequest(); - if (received_op == Coordination::OpNum::Close) + do { - LOG_DEBUG(log, "Received close request for session #{}", session_id); - if (responses.back().wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + Coordination::OpNum received_op = receiveRequest(); + + if (received_op == Coordination::OpNum::Close) { - LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + LOG_DEBUG(log, "Received close request for session #{}", session_id); + if (responses.back().wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + { + LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + } + else + { + LOG_DEBUG(log, "Sent close for session #{}", session_id); + responses.back().get()->write(*out); + } + close_received = true; + + break; + } + else if (received_op == Coordination::OpNum::Heartbeat) + { + LOG_TRACE(log, "Received heartbeat for session #{}", session_id); + session_stopwatch.restart(); + } + } + while (in->available()); + } + + if (close_received) + break; + + if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) + { + while (!responses.empty()) + { + if (responses.front().wait_for(0s) != std::future_status::ready) + break; + + auto response = responses.front().get(); + response->write(*out); + responses.pop(); + } + } + + if (state & SocketInterruptablePollWrapper::HAS_WATCH_RESPONSE) + { + for (auto it = watch_responses.begin(); it != watch_responses.end();) + { + if (it->wait_for(0s) == std::future_status::ready) + { + auto response = it->get(); + if (response->error == Coordination::Error::ZOK) + response->write(*out); + it = watch_responses.erase(it); } else { - LOG_DEBUG(log, "Sent close for session #{}", session_id); - responses.back().get()->write(*out); + ++it; } - close_received = true; - - break; - } - else if (received_op == Coordination::OpNum::Heartbeat) - { - LOG_TRACE(log, "Received heartbeat for session #{}", session_id); - session_stopwatch.restart(); } } - while (in->available()); - } - if (close_received) - break; - - if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) - { - while (!responses.empty()) + if (state == SocketInterruptablePollWrapper::ERROR) { - if (responses.front().wait_for(0s) != std::future_status::ready) - break; - - auto response = responses.front().get(); - response->write(*out); - responses.pop(); + throw Exception("Exception happened while reading from socket", ErrorCodes::SYSTEM_ERROR); } - } - if (state & SocketInterruptablePollWrapper::HAS_WATCH_RESPONSE) - { - for (auto it = watch_responses.begin(); it != watch_responses.end();) + if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) { - if (it->wait_for(0s) == std::future_status::ready) - { - auto response = it->get(); - if (response->error == Coordination::Error::ZOK) - response->write(*out); - it = watch_responses.erase(it); - } + LOG_DEBUG(log, "Session #{} expired", session_id); + auto response = putCloseRequest(); + if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + LOG_DEBUG(log, "Cannot sent close for expired session #{}", session_id); else - { - ++it; - } - } - } + response.get()->write(*out); - if (state == SocketInterruptablePollWrapper::ERROR) - { - throw Exception("Exception happened while reading from socket", ErrorCodes::SYSTEM_ERROR); - } - - if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) - { - LOG_DEBUG(log, "Session #{} expired", session_id); - auto response = putCloseRequest(); - if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) - { - LOG_DEBUG(log, "Cannot sent close for expired session #{}", session_id); + break; } - else - { - response.get()->write(*out); - } - - break; } } + catch (const Exception & ex) + { + LOG_INFO(log, "Got exception processing session #{}: {}", session_id, getExceptionMessage(ex, true)); + auto response = putCloseRequest(); + if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + else + response.get()->write(*out); + } + } zkutil::TestKeeperStorage::AsyncResponse TestKeeperTCPHandler::putCloseRequest() From a44081011e78f3864ac9bb1ab32e2d7720494748 Mon Sep 17 00:00:00 2001 From: "tiger.yan" Date: Wed, 9 Dec 2020 16:22:46 +0800 Subject: [PATCH 110/174] update typo of readme typo --- docs/zh/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/index.md b/docs/zh/index.md index 2bef22f3de4..fb473020f22 100644 --- a/docs/zh/index.md +++ b/docs/zh/index.md @@ -46,7 +46,7 @@ ClickHouse是一个用于联机分析(OLAP)的列式数据库管理系统(DBMS) - 处理单个查询时需要高吞吐量(每台服务器每秒可达数十亿行) - 事务不是必须的 - 对数据一致性要求低 -- 每个查询有一个大表。除了他意以外，其他的都很小。 +- 每个查询有一个大表。除了他以外，其他的都很小。 - 查询结果明显小于源数据。换句话说，数据经过过滤或聚合，因此结果适合于单个服务器的RAM中 很容易可以看出，OLAP场景与其他通常业务场景(例如,OLTP或K/V)有很大的不同， 因此想要使用OLTP或Key-Value数据库去高效的处理分析查询场景，并不是非常完美的适用方案。例如，使用OLAP数据库去处理分析请求通常要优于使用MongoDB或Redis去处理分析请求。 From 758dcd197266d2b16d44c0052fa31fe3c5e885d2 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 11:45:36 +0300 Subject: [PATCH 111/174] Separate factory for test keeper handlers --- programs/server/Server.cpp | 3 +- src/Server/TCPHandlerFactory.h | 10 ++---- src/Server/TestKeeperTCPHandlerFactory.h | 44 ++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 9 deletions(-) create mode 100644 src/Server/TestKeeperTCPHandlerFactory.h diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index f31960b05dd..2e09700255e 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -57,6 +57,7 @@ #include #include #include +#include #include "MetricsTransmitter.h" #include #include @@ -762,7 +763,7 @@ int Server::main(const std::vector & /*args*/) socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( - new TCPHandlerFactory(*this, false, false, true), + new TestKeeperTCPHandlerFactory(*this), server_pool, socket, new Poco::Net::TCPServerParams)); diff --git a/src/Server/TCPHandlerFactory.h b/src/Server/TCPHandlerFactory.h index 1c4d4d4a7af..73318fea9da 100644 --- a/src/Server/TCPHandlerFactory.h +++ b/src/Server/TCPHandlerFactory.h @@ -5,7 +5,6 @@ #include #include #include -#include namespace Poco { class Logger; } @@ -18,7 +17,6 @@ private: IServer & server; bool parse_proxy_protocol = false; Poco::Logger * log; - bool test_keeper; class DummyTCPHandler : public Poco::Net::TCPServerConnection { @@ -32,10 +30,9 @@ public: * and set the information about forwarded address accordingly. * See https://github.com/wolfeidau/proxyv2/blob/master/docs/proxy-protocol.txt */ - TCPHandlerFactory(IServer & server_, bool secure_, bool parse_proxy_protocol_, bool test_keeper_ = false) + TCPHandlerFactory(IServer & server_, bool secure_, bool parse_proxy_protocol_) : server(server_), parse_proxy_protocol(parse_proxy_protocol_) , log(&Poco::Logger::get(std::string("TCP") + (secure_ ? "S" : "") + "HandlerFactory")) - , test_keeper(test_keeper_) { } @@ -45,10 +42,7 @@ public: { LOG_TRACE(log, "TCP Request. Address: {}", socket.peerAddress().toString()); - if (test_keeper) - return new TestKeeperTCPHandler(server, socket); - else - return new TCPHandler(server, socket, parse_proxy_protocol); + return new TCPHandler(server, socket, parse_proxy_protocol); } catch (const Poco::Net::NetException &) { diff --git a/src/Server/TestKeeperTCPHandlerFactory.h b/src/Server/TestKeeperTCPHandlerFactory.h new file mode 100644 index 00000000000..ebf91aa31d4 --- /dev/null +++ b/src/Server/TestKeeperTCPHandlerFactory.h @@ -0,0 +1,44 @@ +#pragma once +#include +#include +#include +#include +#include + +namespace DB +{ + +class TestKeeperTCPHandlerFactory : public Poco::Net::TCPServerConnectionFactory +{ +private: + IServer & server; + Poco::Logger * log; + class DummyTCPHandler : public Poco::Net::TCPServerConnection + { + public: + using Poco::Net::TCPServerConnection::TCPServerConnection; + void run() override {} + }; +public: + TestKeeperTCPHandlerFactory(IServer & server_) + : server(server_) + , log(&Poco::Logger::get("TestKeeperTCPHandlerFactory")) + { + } + + Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override + { + try + { + LOG_TRACE(log, "Test keeper request. Address: {}", socket.peerAddress().toString()); + return new TestKeeperTCPHandler(server, socket); + } + catch (const Poco::Net::NetException &) + { + LOG_TRACE(log, "TCP Request. Client is not connected (most likely RST packet was sent)."); + return new DummyTCPHandler(socket); + } + } +}; + +} From ef15b1f1fb586946758192558bd9afd9c2168558 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Wed, 9 Dec 2020 16:46:54 +0800 Subject: [PATCH 112/174] Better exception message for MaterializeMySQL --- .../MySQL/MaterializeMySQLSyncThread.cpp | 61 ++++++++++++------- 1 file changed, 40 insertions(+), 21 deletions(-) diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp index 02400a28214..4b769c1e991 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp @@ -246,15 +246,24 @@ void MaterializeMySQLSyncThread::startSynchronization() static inline void cleanOutdatedTables(const String & database_name, const Context & context) { - auto ddl_guard = DatabaseCatalog::instance().getDDLGuard(database_name, ""); - const DatabasePtr & clean_database = DatabaseCatalog::instance().getDatabase(database_name); - - for (auto iterator = clean_database->getTablesIterator(context); iterator->isValid(); iterator->next()) + String cleaning_table_name; + try { - Context query_context = createQueryContext(context); - String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; - String table_name = backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(iterator->name()); - tryToExecuteQuery(" DROP TABLE " + table_name, query_context, database_name, comment); + auto ddl_guard = DatabaseCatalog::instance().getDDLGuard(database_name, ""); + const DatabasePtr & clean_database = DatabaseCatalog::instance().getDatabase(database_name); + + for (auto iterator = clean_database->getTablesIterator(context); iterator->isValid(); iterator->next()) + { + Context query_context = createQueryContext(context); + String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; + cleaning_table_name = backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(iterator->name()); + tryToExecuteQuery(" DROP TABLE " + cleaning_table_name, query_context, database_name, comment); + } + } + catch (Exception exception) + { + exception.addMessage("While executing " + (cleaning_table_name.empty() ? "cleanOutdatedTables" : cleaning_table_name)); + throw; } } @@ -295,24 +304,32 @@ static inline void dumpDataForTables( auto iterator = master_info.need_dumping_tables.begin(); for (; iterator != master_info.need_dumping_tables.end() && !is_cancelled(); ++iterator) { - const auto & table_name = iterator->first; - Context query_context = createQueryContext(context); - String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; - tryToExecuteQuery(query_prefix + " " + iterator->second, query_context, database_name, comment); /// create table. + try + { + const auto & table_name = iterator->first; + Context query_context = createQueryContext(context); + String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; + tryToExecuteQuery(query_prefix + " " + iterator->second, query_context, database_name, comment); /// create table. - auto out = std::make_shared(getTableOutput(database_name, table_name, query_context)); - MySQLBlockInputStream input( - connection, "SELECT * FROM " + backQuoteIfNeed(mysql_database_name) + "." + backQuoteIfNeed(table_name), - out->getHeader(), DEFAULT_BLOCK_SIZE); + auto out = std::make_shared(getTableOutput(database_name, table_name, query_context)); + MySQLBlockInputStream input( + connection, "SELECT * FROM " + backQuoteIfNeed(mysql_database_name) + "." + backQuoteIfNeed(table_name), + out->getHeader(), DEFAULT_BLOCK_SIZE); - Stopwatch watch; - copyData(input, *out, is_cancelled); - const Progress & progress = out->getProgress(); - LOG_INFO(&Poco::Logger::get("MaterializeMySQLSyncThread(" + database_name + ")"), - "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." + Stopwatch watch; + copyData(input, *out, is_cancelled); + const Progress & progress = out->getProgress(); + LOG_INFO(&Poco::Logger::get("MaterializeMySQLSyncThread(" + database_name + ")"), + "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." , table_name, formatReadableQuantity(progress.written_rows), formatReadableSizeWithBinarySuffix(progress.written_bytes) , watch.elapsedSeconds(), formatReadableQuantity(static_cast(progress.written_rows / watch.elapsedSeconds())) , formatReadableSizeWithBinarySuffix(static_cast(progress.written_bytes / watch.elapsedSeconds()))); + } + catch (Exception exception) + { + exception.addMessage("While executing dump MySQL {}.{} table data.", mysql_database_name, iterator->first); + throw; + } } } @@ -685,6 +702,8 @@ void MaterializeMySQLSyncThread::executeDDLAtomic(const QueryEvent & query_event } catch (Exception & exception) { + exception.addMessage("While executing MYSQL_QUERY_EVENT. The query: " + query_event.query); + tryLogCurrentException(log); /// If some DDL query was not successfully parsed and executed From bf30aa5ff1b8b2b0b3e5904e89bbe5272d2cadf7 Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Wed, 9 Dec 2020 16:50:40 +0800 Subject: [PATCH 113/174] Code style --- src/Databases/MySQL/MaterializeMySQLSyncThread.cpp | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp index 4b769c1e991..059f4a9a630 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp @@ -260,7 +260,7 @@ static inline void cleanOutdatedTables(const String & database_name, const Conte tryToExecuteQuery(" DROP TABLE " + cleaning_table_name, query_context, database_name, comment); } } - catch (Exception exception) + catch (Exception & exception) { exception.addMessage("While executing " + (cleaning_table_name.empty() ? "cleanOutdatedTables" : cleaning_table_name)); throw; @@ -320,12 +320,12 @@ static inline void dumpDataForTables( copyData(input, *out, is_cancelled); const Progress & progress = out->getProgress(); LOG_INFO(&Poco::Logger::get("MaterializeMySQLSyncThread(" + database_name + ")"), - "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." - , table_name, formatReadableQuantity(progress.written_rows), formatReadableSizeWithBinarySuffix(progress.written_bytes) - , watch.elapsedSeconds(), formatReadableQuantity(static_cast(progress.written_rows / watch.elapsedSeconds())) - , formatReadableSizeWithBinarySuffix(static_cast(progress.written_bytes / watch.elapsedSeconds()))); + "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." + , table_name, formatReadableQuantity(progress.written_rows), formatReadableSizeWithBinarySuffix(progress.written_bytes) + , watch.elapsedSeconds(), formatReadableQuantity(static_cast(progress.written_rows / watch.elapsedSeconds())) + , formatReadableSizeWithBinarySuffix(static_cast(progress.written_bytes / watch.elapsedSeconds()))); } - catch (Exception exception) + catch (Exception & exception) { exception.addMessage("While executing dump MySQL {}.{} table data.", mysql_database_name, iterator->first); throw; From 0dd9c720f46e55e31c829706e44e5c12a716bbbf Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 11:58:41 +0300 Subject: [PATCH 114/174] Better server --- programs/server/Server.cpp | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 2e09700255e..6a8944f6881 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -741,7 +741,7 @@ int Server::main(const std::vector & /*args*/) http_params->setTimeout(settings.http_receive_timeout); http_params->setKeepAliveTimeout(keep_alive_timeout); - std::vector servers; + std::vector servers_to_start_before_tables; std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); @@ -762,7 +762,7 @@ int Server::main(const std::vector & /*args*/) auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); - servers.emplace_back(std::make_unique( + servers_to_start_before_tables.emplace_back(std::make_unique( new TestKeeperTCPHandlerFactory(*this), server_pool, socket, @@ -772,11 +772,9 @@ int Server::main(const std::vector & /*args*/) }); } - for (auto & server : servers) + for (auto & server : servers_to_start_before_tables) server.start(); - size_t already_started_servers = servers.size(); - SCOPE_EXIT({ /** Ask to cancel background jobs all table engines, * and also query_log. @@ -789,8 +787,8 @@ int Server::main(const std::vector & /*args*/) LOG_DEBUG(log, "Shut down storages."); - for (size_t i = 0; i < already_started_servers; ++i) - servers[i].stop(); + for (auto & server : servers_to_start_before_tables) + server.stop(); /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. * At this moment, no one could own shared part of Context. @@ -930,6 +928,7 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "TaskStats is not implemented for this OS. IO accounting will be disabled."); #endif + std::vector servers; { /// This object will periodically calculate some metrics. AsynchronousMetrics async_metrics(*global_context, @@ -1114,8 +1113,8 @@ int Server::main(const std::vector & /*args*/) global_context->enableNamedSessions(); - for (size_t i = already_started_servers; i < servers.size(); ++i) - servers[i].start(); + for (auto & server : servers) + server.start(); { String level_str = config().getString("text_log.level", ""); @@ -1146,10 +1145,10 @@ int Server::main(const std::vector & /*args*/) is_cancelled = true; int current_connections = 0; - for (size_t i = already_started_servers; i < servers.size(); ++i) + for (auto & server : servers) { - servers[i].stop(); - current_connections += servers[i].currentConnections(); + server.stop(); + current_connections += server.currentConnections(); } if (current_connections) @@ -1168,10 +1167,10 @@ int Server::main(const std::vector & /*args*/) while (sleep_current_ms < sleep_max_ms) { current_connections = 0; - for (size_t i = already_started_servers; i < servers.size(); ++i) + for (auto & server : servers) { - servers[i].stop(); - current_connections += servers[i].currentConnections(); + server.stop(); + current_connections += server.currentConnections(); } if (!current_connections) break; From 91c3de966416f3e5a1338ca7a82293984d92a672 Mon Sep 17 00:00:00 2001 From: feng lv Date: Wed, 9 Dec 2020 07:24:36 +0000 Subject: [PATCH 115/174] fix fix fix --- src/Interpreters/TreeRewriter.cpp | 11 +++++++---- .../01562_agg_null_for_empty_ahead.reference | 2 +- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index 5df5d6f7a42..0a656383dc3 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -121,7 +121,7 @@ struct CustomizeAggregateFunctionsSuffixData void visit(ASTFunction & func, ASTPtr &) const { const auto & instance = AggregateFunctionFactory::instance(); - if (instance.isAggregateFunctionName(func.name)) + if (instance.isAggregateFunctionName(func.name) && !endsWith(func.name, "OrNull")) { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) @@ -165,10 +165,13 @@ struct CustomizeAggregateFunctionsMoveSuffixData const auto & instance = AggregateFunctionFactory::instance(); if (instance.isAggregateFunctionName(func.name)) { - auto properties = instance.tryGetProperties(func.name); - if (properties && !properties->returns_default_when_only_null) + if (endsWith(func.name, "OrNull")) { - func.name = moveSuffixAhead(func.name); + auto properties = instance.tryGetProperties(func.name); + if (properties && !properties->returns_default_when_only_null) + { + func.name = moveSuffixAhead(func.name); + } } } } diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference index bda1162f4e9..a197ceee71f 100644 --- a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference @@ -2,7 +2,7 @@ 0 0 0 -\N +0 1 \N \N From 858390b7cf4b5422254989304b7c9cfa00890129 Mon Sep 17 00:00:00 2001 From: Evgeniia Sudarikova Date: Wed, 9 Dec 2020 13:33:42 +0300 Subject: [PATCH 116/174] add example to EN version --- docs/en/operations/settings/settings.md | 46 +++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index f1c281ef2aa..074ac07907c 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2364,6 +2364,52 @@ Defines the representation of `NULL` for [TSV](../../interfaces/formats.md#tabse Default value: `\N`. +**Examples** + +Query + +```sql +SELECT * +FROM tsv_custom_null +FORMAT TSV +``` + +Result + +```text +\N +788 +\N +``` + +Query + +```sql +SET output_format_tsv_null_representation = 'My NULL' +``` + +Result + +```text +Ok. +``` + +Query + +```sql +SELECT * +FROM tsv_custom_null +FORMAT TSV +``` + +Result + +```text +788 +My NULL +My NULL +``` + ## allow_nullable_key {#allow-nullable-key} Allows using of the [Nullable](../../sql-reference/data-types/nullable.md#data_type-nullable)-typed values in a sorting and a primary key for [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md#table_engines-mergetree) tables. From 7783ddb3afcbde3dfd45a28969d55f1567e3de39 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 14:05:48 +0300 Subject: [PATCH 117/174] Revert "Fix index granularity calculation on block borders" --- .../MergeTree/IMergeTreeDataPartWriter.cpp | 23 --------- .../MergeTree/IMergeTreeDataPartWriter.h | 23 ++------- .../MergeTreeDataPartWriterOnDisk.cpp | 47 ++++++------------- .../MergeTree/MergeTreeDataPartWriterOnDisk.h | 13 ++--- .../MergeTree/MergeTreeDataPartWriterWide.cpp | 3 -- .../MergeTree/MergeTreeIndexGranularity.cpp | 11 ----- .../MergeTree/MergeTreeIndexGranularity.h | 4 -- .../0_stateless/00955_test_final_mark.sql | 8 ++-- ...aptive_granularity_block_borders.reference | 2 - ...577_adaptive_granularity_block_borders.sql | 28 ----------- 10 files changed, 27 insertions(+), 135 deletions(-) delete mode 100644 tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference delete mode 100644 tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql diff --git a/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp b/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp index d291a447d1c..143a9920b93 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp @@ -44,29 +44,6 @@ void IMergeTreeDataPartWriter::next() index_offset = next_index_offset; } -bool IMergeTreeDataPartWriter::adjustLastUnfinishedMark(size_t new_block_index_granularity) -{ - /// If amount of rest rows in the last granule more then granularity of the new block - /// than finish it. - if (!index_granularity.empty() && index_offset > new_block_index_granularity) - { - size_t already_written_rows_in_last_granule = index_granularity.getLastMarkRows() - index_offset; - /// We can still write some rows to the last granule - if (already_written_rows_in_last_granule < new_block_index_granularity) - { - index_granularity.setLastMarkRows(new_block_index_granularity); - index_offset = new_block_index_granularity - already_written_rows_in_last_granule; - } - else /// Our last granule is already full, let's start the new one - { - index_granularity.setLastMarkRows(already_written_rows_in_last_granule); - index_offset = 0; - } - return true; - } - return false; -} - IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() = default; } diff --git a/src/Storages/MergeTree/IMergeTreeDataPartWriter.h b/src/Storages/MergeTree/IMergeTreeDataPartWriter.h index 9a6ffca1adb..4a42a58a65b 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPartWriter.h +++ b/src/Storages/MergeTree/IMergeTreeDataPartWriter.h @@ -62,41 +62,28 @@ public: protected: size_t getCurrentMark() const { return current_mark; } size_t getIndexOffset() const { return index_offset; } - /// Finishes our current unfinished mark if we have already written more rows for it - /// than granularity in the new block. Return true if last mark actually was adjusted. - /// Example: - /// __|________|___. <- previous block with granularity 8 and last unfinished mark with 3 rows - /// new_block_index_granularity = 2, so - /// __|________|___|__|__|__| - /// ^ finish last unfinished mark, new marks will have granularity 2 - bool adjustLastUnfinishedMark(size_t new_block_index_granularity); using SerializationState = IDataType::SerializeBinaryBulkStatePtr; using SerializationStates = std::unordered_map; MergeTreeData::DataPartPtr data_part; const MergeTreeData & storage; - const StorageMetadataPtr metadata_snapshot; - const NamesAndTypesList columns_list; - const MergeTreeIndices skip_indices; + StorageMetadataPtr metadata_snapshot; + NamesAndTypesList columns_list; + MergeTreeIndices skip_indices; MergeTreeIndexGranularity index_granularity; - const MergeTreeWriterSettings settings; - const bool with_final_mark; + MergeTreeWriterSettings settings; + bool with_final_mark; size_t next_mark = 0; size_t next_index_offset = 0; - /// When we were writing fresh block granularity of the last mark was adjusted - /// See adjustLastUnfinishedMark - bool last_granule_was_adjusted = false; - MutableColumns index_columns; private: /// Data is already written up to this mark. size_t current_mark = 0; /// The offset to the first row of the block for which you want to write the index. - /// Or how many rows we have to write for this last unfinished mark. size_t index_offset = 0; }; diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp index 31d78dd48a7..0ec10f9db76 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp @@ -184,8 +184,7 @@ void MergeTreeDataPartWriterOnDisk::initSkipIndices() default_codec, settings.max_compress_block_size, 0, settings.aio_threshold)); skip_indices_aggregators.push_back(index_helper->createIndexAggregator()); - marks_in_skip_index_aggregator.push_back(0); - rows_in_skip_index_aggregator_last_mark.push_back(0); + skip_index_filling.push_back(0); } skip_indices_initialized = true; @@ -257,11 +256,9 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block skip_index_current_data_mark = skip_index_data_mark; while (prev_pos < rows) { - bool new_block_started = prev_pos == 0; UInt64 limit = 0; size_t current_index_offset = getIndexOffset(); - /// We start new block, but have an offset from previous one - if (new_block_started && current_index_offset != 0) + if (prev_pos == 0 && current_index_offset != 0) { limit = current_index_offset; } @@ -273,15 +270,10 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block else { limit = index_granularity.getMarkRows(skip_index_current_data_mark); - /// We just started new block serialization but last unfinished mark was shrinked to it's current_size - /// it may happen that we have already aggregated current_size of rows of more for skip_index, but not flushed it to disk - /// because previous granule size was bigger. So do it here. - if (new_block_started && last_granule_was_adjusted && rows_in_skip_index_aggregator_last_mark[i] >= limit) - accountMarkForSkipIdxAndFlushIfNeeded(i); - if (skip_indices_aggregators[i]->empty()) { skip_indices_aggregators[i] = index_helper->createIndexAggregator(); + skip_index_filling[i] = 0; if (stream.compressed.offset() >= settings.min_compress_block_size) stream.compressed.next(); @@ -293,19 +285,24 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block if (settings.can_use_adaptive_granularity) writeIntBinary(1UL, stream.marks); } - /// this mark is aggregated, go to the next one skip_index_current_data_mark++; } size_t pos = prev_pos; skip_indices_aggregators[i]->update(skip_indexes_block, &pos, limit); - rows_in_skip_index_aggregator_last_mark[i] = (pos - prev_pos); - /// We just aggregated all rows in current mark, add new mark to skip_index marks counter - /// and flush on disk if we already aggregated required amount of marks. - if (rows_in_skip_index_aggregator_last_mark[i] == limit) - accountMarkForSkipIdxAndFlushIfNeeded(i); + if (pos == prev_pos + limit) + { + ++skip_index_filling[i]; + + /// write index if it is filled + if (skip_index_filling[i] == index_helper->index.granularity) + { + skip_indices_aggregators[i]->getGranuleAndReset()->serializeBinary(stream.compressed); + skip_index_filling[i] = 0; + } + } prev_pos = pos; } } @@ -363,21 +360,7 @@ void MergeTreeDataPartWriterOnDisk::finishSkipIndicesSerialization( skip_indices_streams.clear(); skip_indices_aggregators.clear(); - marks_in_skip_index_aggregator.clear(); - rows_in_skip_index_aggregator_last_mark.clear(); -} - -void MergeTreeDataPartWriterOnDisk::accountMarkForSkipIdxAndFlushIfNeeded(size_t skip_index_pos) -{ - ++marks_in_skip_index_aggregator[skip_index_pos]; - - /// write index if it is filled - if (marks_in_skip_index_aggregator[skip_index_pos] == skip_indices[skip_index_pos]->index.granularity) - { - skip_indices_aggregators[skip_index_pos]->getGranuleAndReset()->serializeBinary(skip_indices_streams[skip_index_pos]->compressed); - marks_in_skip_index_aggregator[skip_index_pos] = 0; - rows_in_skip_index_aggregator_last_mark[skip_index_pos] = 0; - } + skip_index_filling.clear(); } } diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h index 743b130188d..55da413c9fb 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h @@ -97,7 +97,8 @@ protected: const String marks_file_extension; CompressionCodecPtr default_codec; - const bool compute_granularity; + bool compute_granularity; + bool need_finish_last_granule; /// Number of marsk in data from which skip indices have to start /// aggregation. I.e. it's data mark number, not skip indices mark. @@ -105,10 +106,7 @@ protected: std::vector skip_indices_streams; MergeTreeIndexAggregators skip_indices_aggregators; - /// Amount of marks currently serialized in skip index aggregator - std::vector marks_in_skip_index_aggregator; - /// Amount of rows currently serialized in skip index aggregator for last mark - std::vector rows_in_skip_index_aggregator_last_mark; + std::vector skip_index_filling; std::unique_ptr index_file_stream; std::unique_ptr index_stream; @@ -127,11 +125,6 @@ protected: private: /// Index is already serialized up to this mark. size_t index_mark = 0; - - /// Increment corresponding marks_in_skip_index_aggregator[skip_index_pos] - /// value and flush skip_indices_streams[skip_index_pos] to disk if we have - /// aggregated enough marks - void accountMarkForSkipIdxAndFlushIfNeeded(size_t skip_index_pos); }; } diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp b/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp index 21ae8335af1..c15c39e7b7f 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp @@ -95,9 +95,6 @@ void MergeTreeDataPartWriterWide::write(const Block & block, if (compute_granularity) { size_t index_granularity_for_block = computeIndexGranularity(block); - /// Finish last unfinished mark rows it it's required - last_granule_was_adjusted = adjustLastUnfinishedMark(index_granularity_for_block); - /// Fill index granularity with granules of new size fillIndexGranularity(index_granularity_for_block, block.rows()); } diff --git a/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp b/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp index 2db087a394f..bca0d0cb883 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp @@ -55,17 +55,6 @@ void MergeTreeIndexGranularity::addRowsToLastMark(size_t rows_count) marks_rows_partial_sums.back() += rows_count; } -void MergeTreeIndexGranularity::setLastMarkRows(size_t rows_count) -{ - if (marks_rows_partial_sums.empty()) - marks_rows_partial_sums.push_back(rows_count); - else - { - marks_rows_partial_sums.back() -= getLastMarkRows(); - marks_rows_partial_sums.back() += rows_count; - } -} - void MergeTreeIndexGranularity::popMark() { if (!marks_rows_partial_sums.empty()) diff --git a/src/Storages/MergeTree/MergeTreeIndexGranularity.h b/src/Storages/MergeTree/MergeTreeIndexGranularity.h index bfb48511285..5aefd0f102b 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranularity.h +++ b/src/Storages/MergeTree/MergeTreeIndexGranularity.h @@ -98,10 +98,6 @@ public: /// Extends last mark by rows_count. void addRowsToLastMark(size_t rows_count); - /// Set amount of rows to last mark - /// (add new mark if new have nothing) - void setLastMarkRows(size_t rows_count); - /// Drops last mark if any exists. void popMark(); diff --git a/tests/queries/0_stateless/00955_test_final_mark.sql b/tests/queries/0_stateless/00955_test_final_mark.sql index 929ac931a94..50ca3d008f9 100644 --- a/tests/queries/0_stateless/00955_test_final_mark.sql +++ b/tests/queries/0_stateless/00955_test_final_mark.sql @@ -119,7 +119,7 @@ INSERT INTO mt_without_pk (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-1 SELECT COUNT(*) FROM mt_without_pk WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1; INSERT INTO mt_without_pk (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-10-01'), toDateTime('2018-10-01 07:57:57'), [4, 4, 4], 14, [111, 222], ['Lui', 'Dave']), (toDate('2018-10-01'), toDateTime('2018-10-01 08:57:57'), [5, 5, 5], 15, [333, 444], ['John', 'Mike']), (toDate('2018-10-01'), toDateTime('2018-10-01 09:57:57'), [6, 6, 6], 16, [555, 666, 777], ['Alex', 'Jim', 'Tom']); @@ -127,7 +127,7 @@ OPTIMIZE TABLE mt_without_pk FINAL; SELECT COUNT(*) FROM mt_without_pk WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1; DROP TABLE IF EXISTS mt_without_pk; @@ -149,7 +149,7 @@ INSERT INTO mt_with_small_granularity (d, x, y, z, `n.Age`, `n.Name`) VALUES (to SELECT COUNT(*) FROM mt_with_small_granularity WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1; INSERT INTO mt_with_small_granularity (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-10-01'), toDateTime('2018-10-01 07:57:57'), [4, 4, 4], 14, [111, 222], ['Lui', 'Dave']), (toDate('2018-10-01'), toDateTime('2018-10-01 08:57:57'), [5, 5, 5], 15, [333, 444], ['John', 'Mike']), (toDate('2018-10-01'), toDateTime('2018-10-01 09:57:57'), [6, 6, 6], 16, [555, 666, 777], ['Alex', 'Jim', 'Tom']); @@ -157,6 +157,6 @@ OPTIMIZE TABLE mt_with_small_granularity FINAL; SELECT COUNT(*) FROM mt_with_small_granularity WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1; DROP TABLE IF EXISTS mt_with_small_granularity; diff --git a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference b/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference deleted file mode 100644 index 81c7e6e4df0..00000000000 --- a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference +++ /dev/null @@ -1,2 +0,0 @@ -849 -102400 diff --git a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql b/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql deleted file mode 100644 index a73045f5a6f..00000000000 --- a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql +++ /dev/null @@ -1,28 +0,0 @@ -DROP TABLE IF EXISTS adaptive_table; - ---- If granularity of consequent blocks differs a lot, then adaptive ---- granularity will adjust amout of marks correctly. Data for test empirically ---- derived, it's quite hard to get good parameters. - -CREATE TABLE adaptive_table( - key UInt64, - value String -) ENGINE MergeTree() -ORDER BY key -SETTINGS index_granularity_bytes=1048576, min_bytes_for_wide_part = 0, enable_vertical_merge_algorithm = 0; - -SET max_block_size=900; - --- There are about 900 marks for our settings. -INSERT INTO adaptive_table SELECT number, if(number > 700, randomPrintableASCII(102400), randomPrintableASCII(1)) FROM numbers(10000); - -OPTIMIZE TABLE adaptive_table FINAL; - -SELECT marks FROM system.parts WHERE table = 'adaptive_table' and database=currentDatabase() and active; - --- If we have computed granularity incorrectly than we will exceed this limit. -SET max_memory_usage='30M'; - -SELECT max(length(value)) FROM adaptive_table; - -DROP TABLE IF EXISTS adaptive_table; From 0f4056fd95688c276e7846f50942822dd3172aed Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 14:23:37 +0300 Subject: [PATCH 118/174] Add additional size check in debug mode --- src/Storages/MergeTree/IMergeTreeDataPart.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.cpp b/src/Storages/MergeTree/IMergeTreeDataPart.cpp index aa37e62d099..4d313cc55ea 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPart.cpp @@ -659,6 +659,14 @@ void IMergeTreeDataPart::loadRowsCount() "Column {} has rows count {} according to size in memory " "and size of single value, but data part {} has {} rows", backQuote(column.name), rows_in_column, name, rows_count); } + + if (rows_in_column != index_granularity.getTotalRows()) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Column {} has rows count {} according to size in memory " + "and size of single value, but index granularity in data part {} has {} rows", backQuote(column.name), rows_in_column, name, index_granularity.getTotalRows()); + } } } #endif From b307e545a95087aed89c9de74606742b247b76a6 Mon Sep 17 00:00:00 2001 From: alesapin Date: Wed, 9 Dec 2020 14:46:04 +0300 Subject: [PATCH 119/174] Fix check --- src/Storages/MergeTree/IMergeTreeDataPart.cpp | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.cpp b/src/Storages/MergeTree/IMergeTreeDataPart.cpp index 4d313cc55ea..6a98529966c 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPart.cpp @@ -660,12 +660,27 @@ void IMergeTreeDataPart::loadRowsCount() "and size of single value, but data part {} has {} rows", backQuote(column.name), rows_in_column, name, rows_count); } - if (rows_in_column != index_granularity.getTotalRows()) + size_t last_possibly_incomplete_mark_rows = index_granularity.getLastNonFinalMarkRows(); + /// All this rows have to be written in column + size_t index_granularity_without_last_mark = index_granularity.getTotalRows() - last_possibly_incomplete_mark_rows; + /// We have more rows in column than in index granularity without last possibly incomplete mark + if (rows_in_column < index_granularity_without_last_mark) { throw Exception( ErrorCodes::LOGICAL_ERROR, "Column {} has rows count {} according to size in memory " - "and size of single value, but index granularity in data part {} has {} rows", backQuote(column.name), rows_in_column, name, index_granularity.getTotalRows()); + "and size of single value, but index granularity in part {} without last mark has {} rows, which is more than in column", + backQuote(column.name), rows_in_column, name, index_granularity.getTotalRows()); + } + + /// In last mark we actually written less or equal rows than stored in last mark of index granularity + if (rows_in_column - index_granularity_without_last_mark > last_possibly_incomplete_mark_rows) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Column {} has rows count {} in last mark according to size in memory " + "and size of single value, but index granularity in part {} in last mark has {} rows which is less than in column", + backQuote(column.name), rows_in_column - index_granularity_without_last_mark, name, last_possibly_incomplete_mark_rows); } } } From 56ecc3a9d7e214975c2cbb635e3e71146abcc471 Mon Sep 17 00:00:00 2001 From: annvsh Date: Wed, 9 Dec 2020 21:17:03 +0700 Subject: [PATCH 120/174] Fixed From b13873356c27b458dc426510d42f727dafb2a27c Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Wed, 9 Dec 2020 18:30:44 +0300 Subject: [PATCH 121/174] also style --- src/Common/RadixSort.h | 1 - src/Functions/GregorianDate.h | 108 ++++++++++++------------ src/Functions/fromModifiedJulianDay.cpp | 38 +++------ src/Functions/toModifiedJulianDay.cpp | 15 +--- 4 files changed, 67 insertions(+), 95 deletions(-) diff --git a/src/Common/RadixSort.h b/src/Common/RadixSort.h index f3a19068f5a..f490df3e4bb 100644 --- a/src/Common/RadixSort.h +++ b/src/Common/RadixSort.h @@ -10,7 +10,6 @@ #include #include #include -#include #include #include diff --git a/src/Functions/GregorianDate.h b/src/Functions/GregorianDate.h index c5bb478f9d5..844fb180e5e 100644 --- a/src/Functions/GregorianDate.h +++ b/src/Functions/GregorianDate.h @@ -46,7 +46,7 @@ namespace DB * signed. */ template ()> * = nullptr> - T toMJD() const; + T toModifiedJulianDay() const; /** Write the date in text form 'YYYY-MM-DD' to a buffer. */ @@ -66,15 +66,15 @@ namespace DB return month_; } - uint8_t dayOfMonth() const noexcept + uint8_t day_of_month() const noexcept { - return dayOfMonth_; + return day_of_month_; } private: YearT year_; uint8_t month_; - uint8_t dayOfMonth_; + uint8_t day_of_month_; }; /** ISO 8601 Ordinal Date. YearT is an integral type which should @@ -84,7 +84,7 @@ namespace DB class OrdinalDate { public: - OrdinalDate(YearT year, uint16_t dayOfYear); + OrdinalDate(YearT year, uint16_t day_of_year); /** Construct from Modified Julian Day. The type T is an * integral type which should be at least 32 bits wide, and @@ -98,7 +98,7 @@ namespace DB * preferably be signed. */ template ()> * = nullptr> - T toMJD() const noexcept; + T toModifiedJulianDay() const noexcept; YearT year() const noexcept { @@ -107,43 +107,43 @@ namespace DB uint16_t dayOfYear() const noexcept { - return dayOfYear_; + return day_of_year_; } private: YearT year_; - uint16_t dayOfYear_; + uint16_t day_of_year_; }; class MonthDay { public: /** Construct from month and day. */ - MonthDay(uint8_t month, uint8_t dayOfMonth); + MonthDay(uint8_t month, uint8_t day_of_month); /** Construct from day of year in Gregorian or Julian * calendars to month and day. */ - MonthDay(bool isLeapYear, uint16_t dayOfYear); + MonthDay(bool is_leap_year, uint16_t day_of_year); /** Convert month and day in Gregorian or Julian calendars to * day of year. */ - uint16_t dayOfYear(bool isLeapYear) const; + uint16_t dayOfYear(bool is_leap_year) const; uint8_t month() const noexcept { return month_; } - uint8_t dayOfMonth() const noexcept + uint8_t day_of_month() const noexcept { - return dayOfMonth_; + return day_of_month_; } private: uint8_t month_; - uint8_t dayOfMonth_; + uint8_t day_of_month_; }; } @@ -154,17 +154,17 @@ namespace gd using namespace DB; template - static inline constexpr bool isLeapYear(YearT year) + static inline constexpr bool is_leap_year(YearT year) { return (year % 4 == 0) && ((year % 400 == 0) || (year % 100 != 0)); } - static inline constexpr uint8_t monthLength(bool isLeapYear, uint8_t month) + static inline constexpr uint8_t monthLength(bool is_leap_year, uint8_t month) { switch (month) { case 1: return 31; - case 2: return isLeapYear ? 29 : 28; + case 2: return is_leap_year ? 29 : 28; case 3: return 31; case 4: return 30; case 5: return 31; @@ -238,8 +238,8 @@ namespace DB GregorianDate::GregorianDate(ReadBuffer & in) { year_ = gd::readDigit(in) * 1000 - + gd::readDigit(in) * 100 - + gd::readDigit(in) * 10 + + gd::readDigit(in) * 100 + + gd::readDigit(in) * 10 + gd::readDigit(in); assertChar('-', in); @@ -249,17 +249,13 @@ namespace DB assertChar('-', in); - dayOfMonth_ = gd::readDigit(in) * 10 + day_of_month_ = gd::readDigit(in) * 10 + gd::readDigit(in); assertEOF(in); - if (month_ < 1 || month_ > 12 || - dayOfMonth_ < 1 || dayOfMonth_ > gd::monthLength(gd::isLeapYear(year_), month_)) - { - throw Exception( - "Invalid date: " + toString(), ErrorCodes::CANNOT_PARSE_DATE); - } + if (month_ < 1 || month_ > 12 || day_of_month_ < 1 || day_of_month_ > gd::monthLength(gd::is_leap_year(year_), month_)) + throw Exception("Invalid date: " + toString(), ErrorCodes::CANNOT_PARSE_DATE); } template @@ -267,20 +263,20 @@ namespace DB GregorianDate::GregorianDate(T mjd) { const OrdinalDate ord(mjd); - const MonthDay md(gd::isLeapYear(ord.year()), ord.dayOfYear()); + const MonthDay md(gd::is_leap_year(ord.year()), ord.dayOfYear()); year_ = ord.year(); month_ = md.month(); - dayOfMonth_ = md.dayOfMonth(); + day_of_month_ = md.day_of_month(); } template template ()> *> - T GregorianDate::toMJD() const + T GregorianDate::toModifiedJulianDay() const { - const MonthDay md(month_, dayOfMonth_); - const auto dayOfYear = md.dayOfYear(gd::isLeapYear(year_)); - const OrdinalDate ord(year_, dayOfYear); - return ord.template toMJD(); + const MonthDay md(month_, day_of_month_); + const auto day_of_year = md.dayOfYear(gd::is_leap_year(year_)); + const OrdinalDate ord(year_, day_of_year); + return ord.template toModifiedJulianDay(); } template @@ -308,7 +304,7 @@ namespace DB writeChar('-', buf); - auto d = dayOfMonth_; + auto d = day_of_month_; writeChar('0' + d / 10, buf); d %= 10; writeChar('0' + d , buf); } @@ -323,14 +319,14 @@ namespace DB } template - OrdinalDate::OrdinalDate(YearT year, uint16_t dayOfYear) + OrdinalDate::OrdinalDate(YearT year, uint16_t day_of_year) : year_(year) - , dayOfYear_(dayOfYear) + , day_of_year_(day_of_year) { - if (dayOfYear < 1 || dayOfYear > (gd::isLeapYear(year) ? 366 : 365)) + if (day_of_year < 1 || day_of_year > (gd::is_leap_year(year) ? 366 : 365)) { throw Exception( - "Invalid ordinal date: " + toString(year) + "-" + toString(dayOfYear), + "Invalid ordinal date: " + toString(year) + "-" + toString(day_of_year), ErrorCodes::LOGICAL_ERROR); } } @@ -347,16 +343,16 @@ namespace DB const auto quad = gd::div(c, 1461); const auto d = gd::mod(c, 1461); const auto y = gd::min(gd::div(d, 365), 3); - dayOfYear_ = d - y * 365 + 1; + day_of_year_ = d - y * 365 + 1; year_ = quad_cent * 400 + cent * 100 + quad * 4 + y + 1; } template template ()> *> - T OrdinalDate::toMJD() const noexcept + T OrdinalDate::toModifiedJulianDay() const noexcept { const auto y = year_ - 1; - return dayOfYear_ + return day_of_year_ + 365 * y + gd::div(y, 4) - gd::div(y, 100) @@ -364,50 +360,50 @@ namespace DB - 678576; } - inline MonthDay::MonthDay(uint8_t month, uint8_t dayOfMonth) + inline MonthDay::MonthDay(uint8_t month, uint8_t day_of_month) : month_(month) - , dayOfMonth_(dayOfMonth) + , day_of_month_(day_of_month) { if (month < 1 || month > 12) throw Exception( "Invalid month: " + DB::toString(month), ErrorCodes::LOGICAL_ERROR); - /* We can't validate dayOfMonth here, because we don't know if + /* We can't validate day_of_month here, because we don't know if * it's a leap year. */ } - inline MonthDay::MonthDay(bool isLeapYear, uint16_t dayOfYear) + inline MonthDay::MonthDay(bool is_leap_year, uint16_t day_of_year) { - if (dayOfYear < 1 || dayOfYear > (isLeapYear ? 366 : 365)) + if (day_of_year < 1 || day_of_year > (is_leap_year ? 366 : 365)) throw Exception( std::string("Invalid day of year: ") + - (isLeapYear ? "leap, " : "non-leap, ") + DB::toString(dayOfYear), + (is_leap_year ? "leap, " : "non-leap, ") + DB::toString(day_of_year), ErrorCodes::LOGICAL_ERROR); month_ = 1; - uint16_t d = dayOfYear; + uint16_t d = day_of_year; while (true) { - const auto len = gd::monthLength(isLeapYear, month_); + const auto len = gd::monthLength(is_leap_year, month_); if (d <= len) break; month_++; d -= len; } - dayOfMonth_ = d; + day_of_month_ = d; } - inline uint16_t MonthDay::dayOfYear(bool isLeapYear) const + inline uint16_t MonthDay::dayOfYear(bool is_leap_year) const { - if (dayOfMonth_ < 1 || dayOfMonth_ > gd::monthLength(isLeapYear, month_)) + if (day_of_month_ < 1 || day_of_month_ > gd::monthLength(is_leap_year, month_)) { throw Exception( std::string("Invalid day of month: ") + - (isLeapYear ? "leap, " : "non-leap, ") + DB::toString(month_) + - "-" + DB::toString(dayOfMonth_), + (is_leap_year ? "leap, " : "non-leap, ") + DB::toString(month_) + + "-" + DB::toString(day_of_month_), ErrorCodes::LOGICAL_ERROR); } - const auto k = month_ <= 2 ? 0 : isLeapYear ? -1 :-2; - return (367 * month_ - 362) / 12 + k + dayOfMonth_; + const auto k = month_ <= 2 ? 0 : is_leap_year ? -1 :-2; + return (367 * month_ - 362) / 12 + k + day_of_month_; } } diff --git a/src/Functions/fromModifiedJulianDay.cpp b/src/Functions/fromModifiedJulianDay.cpp index e62b6d4a38d..636512db0de 100644 --- a/src/Functions/fromModifiedJulianDay.cpp +++ b/src/Functions/fromModifiedJulianDay.cpp @@ -83,13 +83,9 @@ namespace DB write_buffer.finalize(); if constexpr (nullOnErrors) - { return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); - } else - { return col_to; - } } bool useDefaultImplementationForConstants() const override @@ -181,26 +177,20 @@ namespace DB }; bool built = callOnBasicType(from_type->getTypeId(), call); if (built) - { return base; - } + + /* When the argument is a NULL constant, the resulting + * function base will not be actually called but it + * will still be inspected. Returning a NULL pointer + * here causes a SEGV. So we must somehow create a + * dummy implementation and return it. + */ + if (WhichDataType(from_type).isNullable()) // Nullable(Nothing) + return std::make_unique>(argument_types, return_type); else - { - /* When the argument is a NULL constant, the resulting - * function base will not be actually called but it - * will still be inspected. Returning a NULL pointer - * here causes a SEGV. So we must somehow create a - * dummy implementation and return it. - */ - if (WhichDataType(from_type).isNullable()) // Nullable(Nothing) - { - return std::make_unique>(argument_types, return_type); - } - else - // Should not happen. - throw Exception( - "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - } + // Should not happen. + throw Exception( + "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } DataTypePtr getReturnType(const DataTypes & arguments) const override @@ -213,13 +203,9 @@ namespace DB DataTypePtr base_type = std::make_shared(); if constexpr (nullOnErrors) - { return std::make_shared(base_type); - } else - { return base_type; - } } size_t getNumberOfArguments() const override diff --git a/src/Functions/toModifiedJulianDay.cpp b/src/Functions/toModifiedJulianDay.cpp index ca024b007f7..df3066a5005 100644 --- a/src/Functions/toModifiedJulianDay.cpp +++ b/src/Functions/toModifiedJulianDay.cpp @@ -82,37 +82,28 @@ namespace DB try { const GregorianDate<> date(read_buffer); - vec_to[i] = date.toMJD(); + vec_to[i] = date.toModifiedJulianDay(); (*vec_null_map_to)[i] = false; } catch (const Exception & e) { - if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || - e.code() == ErrorCodes::CANNOT_PARSE_DATE) - { + if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || e.code() == ErrorCodes::CANNOT_PARSE_DATE) (*vec_null_map_to)[i] = true; - } else - { throw; - } } } else { const GregorianDate<> date(read_buffer); - vec_to[i] = date.toMJD(); + vec_to[i] = date.toModifiedJulianDay(); } } if constexpr (nullOnErrors) - { return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); - } else - { return col_to; - } } bool useDefaultImplementationForConstants() const override From 89a3a86b71837486d2c57d89ed0f823306148eb8 Mon Sep 17 00:00:00 2001 From: robot-clickhouse Date: Wed, 9 Dec 2020 19:14:28 +0300 Subject: [PATCH 122/174] Update version_date.tsv after release 20.12.3.3 --- utils/list-versions/version_date.tsv | 1 + 1 file changed, 1 insertion(+) diff --git a/utils/list-versions/version_date.tsv b/utils/list-versions/version_date.tsv index 27eebc26b7e..8df84943084 100644 --- a/utils/list-versions/version_date.tsv +++ b/utils/list-versions/version_date.tsv @@ -1,3 +1,4 @@ +v20.12.3.3-stable 2020-12-09 v20.12.2.1-stable 2020-12-09 v20.11.5.18-stable 2020-12-06 v20.11.4.13-stable 2020-11-20 From 916cbd6610615f17c82905fec7bbf9da638ae6f6 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 123/174] Add ability to use custom TLD list v2: Add a note that top_level_domains_lists aren not applied w/o restart v3: Remove ExtractFirstSignificantSubdomain{Default,Custom}Lookup.h headers v4: TLDListsHolder: remove FIXME for dense_hash_map (this is not significant) --- .../sql-reference/functions/url-functions.md | 32 +++++ programs/server/Server.cpp | 4 + programs/server/config.xml | 11 ++ src/Common/ErrorCodes.cpp | 1 + src/Common/TLDListsHolder.cpp | 94 +++++++++++++++ src/Common/TLDListsHolder.h | 60 ++++++++++ src/Common/ya.make | 1 + ...n.h => ExtractFirstSignificantSubdomain.h} | 17 ++- .../URL/FirstSignificantSubdomainCustomImpl.h | 112 ++++++++++++++++++ .../URL/cutToFirstSignificantSubdomain.cpp | 2 +- .../cutToFirstSignificantSubdomainCustom.cpp | 43 +++++++ .../URL/firstSignificantSubdomain.cpp | 3 +- .../URL/firstSignificantSubdomainCustom.cpp | 18 +++ src/Functions/URL/registerFunctionsURL.cpp | 4 + src/Functions/URL/tldLookup.h | 2 + src/Functions/ya.make | 2 + 16 files changed, 403 insertions(+), 3 deletions(-) create mode 100644 src/Common/TLDListsHolder.cpp create mode 100644 src/Common/TLDListsHolder.h rename src/Functions/URL/{firstSignificantSubdomain.h => ExtractFirstSignificantSubdomain.h} (77%) create mode 100644 src/Functions/URL/FirstSignificantSubdomainCustomImpl.h create mode 100644 src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp create mode 100644 src/Functions/URL/firstSignificantSubdomainCustom.cpp diff --git a/docs/en/sql-reference/functions/url-functions.md b/docs/en/sql-reference/functions/url-functions.md index 0da74ce1b0e..2250c0ead20 100644 --- a/docs/en/sql-reference/functions/url-functions.md +++ b/docs/en/sql-reference/functions/url-functions.md @@ -131,6 +131,38 @@ For example: - `cutToFirstSignificantSubdomain('www.tr') = 'www.tr'`. - `cutToFirstSignificantSubdomain('tr') = ''`. +### cutToFirstSignificantSubdomainCustom {#cuttofirstsignificantsubdomaincustom} + +Same as `cutToFirstSignificantSubdomain` but accept custom TLD list name, useful if: + +- you need fresh TLD list, +- or you have custom. + +Configuration example: + +```xml + + + /path/to/public_suffix_list.dat + +``` + +Example: + +- `cutToFirstSignificantSubdomain('https://news.yandex.com.tr/', 'public_suffix_list') = 'yandex.com.tr'`. + +### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} + +Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. + +### firstSignificantSubdomainCustom {#firstsignificantsubdomaincustom} + +Same as `firstSignificantSubdomain` but accept custom TLD list name. + +### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} + +Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. + ### port(URL\[, default_port = 0\]) {#port} Returns the port or `default_port` if there is no port in the URL (or in case of validation error). diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index ad849f3c6e5..4480f34bb60 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -483,6 +484,9 @@ int Server::main(const std::vector & /*args*/) Poco::File(dictionaries_lib_path).createDirectories(); } + /// top_level_domains_lists + TLDListsHolder::getInstance().parseConfig(config()); + { Poco::File(path + "data/").createDirectories(); Poco::File(path + "metadata/").createDirectories(); diff --git a/programs/server/config.xml b/programs/server/config.xml index 851a7654d53..a46c6ae6eec 100644 --- a/programs/server/config.xml +++ b/programs/server/config.xml @@ -724,6 +724,17 @@ + + + + + diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index 384c29ed675..1e381808d16 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -528,6 +528,7 @@ M(559, INVALID_GRPC_QUERY_INFO) \ M(560, ZSTD_ENCODER_FAILED) \ M(561, ZSTD_DECODER_FAILED) \ + M(562, TLD_LIST_NOT_FOUND) \ \ M(999, KEEPER_EXCEPTION) \ M(1000, POCO_EXCEPTION) \ diff --git a/src/Common/TLDListsHolder.cpp b/src/Common/TLDListsHolder.cpp new file mode 100644 index 00000000000..cbad8beaa7d --- /dev/null +++ b/src/Common/TLDListsHolder.cpp @@ -0,0 +1,94 @@ +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int TLD_LIST_NOT_FOUND; +} + +/// +/// TLDList +/// +bool TLDList::insert(const StringRef & host) +{ + StringRefHash hash; + return tld_container.insert(hash(host)).second; +} +bool TLDList::has(const StringRef & host) const +{ + StringRefHash hash; + return tld_container.has(hash(host)); +} + +/// +/// TLDListsHolder +/// +TLDListsHolder & TLDListsHolder::getInstance() +{ + static TLDListsHolder instance; + return instance; +} +TLDListsHolder::TLDListsHolder() = default; + +void TLDListsHolder::parseConfig(const Poco::Util::AbstractConfiguration & config) +{ + Poco::Util::AbstractConfiguration::Keys config_keys; + config.keys("top_level_domains_lists", config_keys); + + Poco::Logger * log = &Poco::Logger::get("TLDListsHolder"); + + for (const auto & key : config_keys) + { + auto path = config.getString("top_level_domains_lists." + key); + LOG_TRACE(log, "{} loading from {}", key, path); + size_t hosts = parseAndAddTldList(key, path); + LOG_INFO(log, "{} was added ({} hosts)", key, hosts); + } +} + +size_t TLDListsHolder::parseAndAddTldList(const std::string & name, const std::string & path) +{ + TLDList tld_list; + + ReadBufferFromFile in(path); + while (!in.eof()) + { + char * newline = find_first_symbols<'\n'>(in.position(), in.buffer().end()); + if (newline >= in.buffer().end()) + break; + + std::string_view line(in.position(), newline - in.position()); + in.position() = newline + 1; + + /// Skip comments + if (line.size() > 2 && line[0] == '/' && line[1] == '/') + continue; + trim(line); + /// Skip empty line + if (line.empty()) + continue; + tld_list.insert(StringRef{line.data(), line.size()}); + } + + size_t tld_list_size = tld_list.size(); + std::lock_guard lock(tld_lists_map_mutex); + tld_lists_map.emplace(name, std::move(tld_list)); + return tld_list_size; +} + +const TLDList & TLDListsHolder::getTldList(const std::string & name) +{ + std::lock_guard lock(tld_lists_map_mutex); + auto it = tld_lists_map.find(name); + if (it == tld_lists_map.end()) + throw Exception(ErrorCodes::TLD_LIST_NOT_FOUND, "TLD list {} does not exist", name); + return it->second; +} + +} diff --git a/src/Common/TLDListsHolder.h b/src/Common/TLDListsHolder.h new file mode 100644 index 00000000000..9ce394267ec --- /dev/null +++ b/src/Common/TLDListsHolder.h @@ -0,0 +1,60 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +/// Custom TLD List +/// Unlike tldLookup (which uses gperf) this one uses plain HashSet. +class TLDList +{ +public: + /// Uses StringRefHash + using Container = HashSet; + + /// Return true if the tld_container does not contains such element. + bool insert(const StringRef & host); + bool has(const StringRef & host) const; + size_t size() const { return tld_container.size(); } + +private: + Container tld_container; +}; + +class TLDListsHolder +{ +public: + using Map = std::unordered_map; + + static TLDListsHolder & getInstance(); + + /// Parse "top_level_domains_lists" section, + /// And add each found dictionary. + void parseConfig(const Poco::Util::AbstractConfiguration & config); + + /// Parse file and add it as a Set to the list of TLDs + /// - "//" -- comment, + /// - empty lines will be ignored. + /// + /// Example: https://publicsuffix.org/list/public_suffix_list.dat + /// + /// Return size of the list. + size_t parseAndAddTldList(const std::string & name, const std::string & path); + /// Throws TLD_LIST_NOT_FOUND if list does not exist + const TLDList & getTldList(const std::string & name); + +protected: + TLDListsHolder(); + + std::mutex tld_lists_map_mutex; + Map tld_lists_map; +}; + +} diff --git a/src/Common/ya.make b/src/Common/ya.make index 71c0edaea95..558ae25228a 100644 --- a/src/Common/ya.make +++ b/src/Common/ya.make @@ -68,6 +68,7 @@ SRCS( StringUtils/StringUtils.cpp StudentTTest.cpp SymbolIndex.cpp + TLDListsHolder.cpp TaskStatsInfoGetter.cpp TerminalSize.cpp ThreadFuzzer.cpp diff --git a/src/Functions/URL/firstSignificantSubdomain.h b/src/Functions/URL/ExtractFirstSignificantSubdomain.h similarity index 77% rename from src/Functions/URL/firstSignificantSubdomain.h rename to src/Functions/URL/ExtractFirstSignificantSubdomain.h index 522e7905f69..c13b5f50156 100644 --- a/src/Functions/URL/firstSignificantSubdomain.h +++ b/src/Functions/URL/ExtractFirstSignificantSubdomain.h @@ -7,12 +7,27 @@ namespace DB { +struct FirstSignificantSubdomainDefaultLookup +{ + bool operator()(const char *src, size_t len) const + { + return tldLookup::isValid(src, len); + } +}; + template struct ExtractFirstSignificantSubdomain { static size_t getReserveLengthForElement() { return 10; } static void execute(const Pos data, const size_t size, Pos & res_data, size_t & res_size, Pos * out_domain_end = nullptr) + { + FirstSignificantSubdomainDefaultLookup loookup; + return execute(loookup, data, size, res_data, res_size, out_domain_end); + } + + template + static void execute(const Lookup & lookup, const Pos data, const size_t size, Pos & res_data, size_t & res_size, Pos * out_domain_end = nullptr) { res_data = data; res_size = 0; @@ -65,7 +80,7 @@ struct ExtractFirstSignificantSubdomain end_of_level_domain = end; } - if (tldLookup::isValid(last_3_periods[1] + 1, end_of_level_domain - last_3_periods[1] - 1) != nullptr) + if (lookup(last_3_periods[1] + 1, end_of_level_domain - last_3_periods[1] - 1)) { res_data += last_3_periods[2] + 1 - begin; res_size = last_3_periods[1] - last_3_periods[2] - 1; diff --git a/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h b/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h new file mode 100644 index 00000000000..244b32459c1 --- /dev/null +++ b/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h @@ -0,0 +1,112 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int ILLEGAL_COLUMN; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; +} + +struct FirstSignificantSubdomainCustomtLookup +{ + const TLDList & tld_list; + FirstSignificantSubdomainCustomtLookup(const std::string & tld_list_name) + : tld_list(TLDListsHolder::getInstance().getTldList(tld_list_name)) + { + } + + bool operator()(const char *pos, size_t len) const + { + return tld_list.has(StringRef{pos, len}); + } +}; + +template +class FunctionCutToFirstSignificantSubdomainCustomImpl : public IFunction +{ +public: + static constexpr auto name = Name::name; + static FunctionPtr create(const Context &) { return std::make_shared(); } + + String getName() const override { return name; } + size_t getNumberOfArguments() const override { return 2; } + + DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override + { + if (!isString(arguments[0].type)) + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + "Illegal type {} of first argument of function {}. Must be String.", + arguments[0].type->getName(), getName()); + if (!isString(arguments[1].type)) + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + "Illegal type {} of second argument (TLD_list_name) of function {}. Must be String/FixedString.", + arguments[1].type->getName(), getName()); + const auto * column = arguments[1].column.get(); + if (!column || !checkAndGetColumnConstStringOrFixedString(column)) + throw Exception(ErrorCodes::ILLEGAL_COLUMN, + "The second argument of function {} should be a constant string with the name of the custom TLD", + getName()); + + return arguments[0].type; + } + + ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & /*result_type*/, size_t /*input_rows_count*/) const override + { + const ColumnConst * column_tld_list_name = checkAndGetColumnConstStringOrFixedString(arguments[1].column.get()); + FirstSignificantSubdomainCustomtLookup tld_lookup(column_tld_list_name->getValue()); + + /// FIXME: convertToFullColumnIfConst() is suboptimal + auto column = arguments[0].column->convertToFullColumnIfConst(); + if (const ColumnString * col = checkAndGetColumn(*column)) + { + auto col_res = ColumnString::create(); + vector(tld_lookup, col->getChars(), col->getOffsets(), col_res->getChars(), col_res->getOffsets()); + return col_res; + } + else + throw Exception( + "Illegal column " + arguments[0].column->getName() + " of argument of function " + getName(), + ErrorCodes::ILLEGAL_COLUMN); + } + + static void vector(FirstSignificantSubdomainCustomtLookup & tld_lookup, + const ColumnString::Chars & data, const ColumnString::Offsets & offsets, + ColumnString::Chars & res_data, ColumnString::Offsets & res_offsets) + { + size_t size = offsets.size(); + res_offsets.resize(size); + res_data.reserve(size * Extractor::getReserveLengthForElement()); + + size_t prev_offset = 0; + size_t res_offset = 0; + + /// Matched part. + Pos start; + size_t length; + + for (size_t i = 0; i < size; ++i) + { + Extractor::execute(tld_lookup, reinterpret_cast(&data[prev_offset]), offsets[i] - prev_offset - 1, start, length); + + res_data.resize(res_data.size() + length + 1); + memcpySmallAllowReadWriteOverflow15(&res_data[res_offset], start, length); + res_offset += length + 1; + res_data[res_offset - 1] = 0; + + res_offsets[i] = res_offset; + prev_offset = offsets[i]; + } + } +}; + +} diff --git a/src/Functions/URL/cutToFirstSignificantSubdomain.cpp b/src/Functions/URL/cutToFirstSignificantSubdomain.cpp index 43d614a7036..82eb366dae6 100644 --- a/src/Functions/URL/cutToFirstSignificantSubdomain.cpp +++ b/src/Functions/URL/cutToFirstSignificantSubdomain.cpp @@ -1,6 +1,6 @@ #include #include -#include "firstSignificantSubdomain.h" +#include "ExtractFirstSignificantSubdomain.h" namespace DB diff --git a/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp b/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp new file mode 100644 index 00000000000..11fd27e317b --- /dev/null +++ b/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp @@ -0,0 +1,43 @@ +#include +#include "ExtractFirstSignificantSubdomain.h" +#include "FirstSignificantSubdomainCustomImpl.h" + +namespace DB +{ + +template +struct CutToFirstSignificantSubdomainCustom +{ + static size_t getReserveLengthForElement() { return 15; } + + static void execute(FirstSignificantSubdomainCustomtLookup & tld_lookup, const Pos data, const size_t size, Pos & res_data, size_t & res_size) + { + res_data = data; + res_size = 0; + + Pos tmp_data; + size_t tmp_length; + Pos domain_end; + ExtractFirstSignificantSubdomain::execute(tld_lookup, data, size, tmp_data, tmp_length, &domain_end); + + if (tmp_length == 0) + return; + + res_data = tmp_data; + res_size = domain_end - tmp_data; + } +}; + +struct NameCutToFirstSignificantSubdomainCustom { static constexpr auto name = "cutToFirstSignificantSubdomainCustom"; }; +using FunctionCutToFirstSignificantSubdomainCustom = FunctionCutToFirstSignificantSubdomainCustomImpl, NameCutToFirstSignificantSubdomainCustom>; + +struct NameCutToFirstSignificantSubdomainCustomWithWWW { static constexpr auto name = "cutToFirstSignificantSubdomainCustomWithWWW"; }; +using FunctionCutToFirstSignificantSubdomainCustomWithWWW = FunctionCutToFirstSignificantSubdomainCustomImpl, NameCutToFirstSignificantSubdomainCustomWithWWW>; + +void registerFunctionCutToFirstSignificantSubdomainCustom(FunctionFactory & factory) +{ + factory.registerFunction(); + factory.registerFunction(); +} + +} diff --git a/src/Functions/URL/firstSignificantSubdomain.cpp b/src/Functions/URL/firstSignificantSubdomain.cpp index 7db18824375..87659940938 100644 --- a/src/Functions/URL/firstSignificantSubdomain.cpp +++ b/src/Functions/URL/firstSignificantSubdomain.cpp @@ -1,12 +1,13 @@ #include #include -#include "firstSignificantSubdomain.h" +#include "ExtractFirstSignificantSubdomain.h" namespace DB { struct NameFirstSignificantSubdomain { static constexpr auto name = "firstSignificantSubdomain"; }; + using FunctionFirstSignificantSubdomain = FunctionStringToString>, NameFirstSignificantSubdomain>; void registerFunctionFirstSignificantSubdomain(FunctionFactory & factory) diff --git a/src/Functions/URL/firstSignificantSubdomainCustom.cpp b/src/Functions/URL/firstSignificantSubdomainCustom.cpp new file mode 100644 index 00000000000..675b4a346de --- /dev/null +++ b/src/Functions/URL/firstSignificantSubdomainCustom.cpp @@ -0,0 +1,18 @@ +#include +#include "ExtractFirstSignificantSubdomain.h" +#include "FirstSignificantSubdomainCustomImpl.h" + + +namespace DB +{ + +struct NameFirstSignificantSubdomainCustom { static constexpr auto name = "firstSignificantSubdomainCustom"; }; + +using FunctionFirstSignificantSubdomainCustom = FunctionCutToFirstSignificantSubdomainCustomImpl, NameFirstSignificantSubdomainCustom>; + +void registerFunctionFirstSignificantSubdomainCustom(FunctionFactory & factory) +{ + factory.registerFunction(); +} + +} diff --git a/src/Functions/URL/registerFunctionsURL.cpp b/src/Functions/URL/registerFunctionsURL.cpp index f3906c2723e..91118074b7a 100644 --- a/src/Functions/URL/registerFunctionsURL.cpp +++ b/src/Functions/URL/registerFunctionsURL.cpp @@ -7,6 +7,7 @@ void registerFunctionProtocol(FunctionFactory & factory); void registerFunctionDomain(FunctionFactory & factory); void registerFunctionDomainWithoutWWW(FunctionFactory & factory); void registerFunctionFirstSignificantSubdomain(FunctionFactory & factory); +void registerFunctionFirstSignificantSubdomainCustom(FunctionFactory & factory); void registerFunctionTopLevelDomain(FunctionFactory & factory); void registerFunctionPort(FunctionFactory & factory); void registerFunctionPath(FunctionFactory & factory); @@ -20,6 +21,7 @@ void registerFunctionExtractURLParameterNames(FunctionFactory & factory); void registerFunctionURLHierarchy(FunctionFactory & factory); void registerFunctionURLPathHierarchy(FunctionFactory & factory); void registerFunctionCutToFirstSignificantSubdomain(FunctionFactory & factory); +void registerFunctionCutToFirstSignificantSubdomainCustom(FunctionFactory & factory); void registerFunctionCutWWW(FunctionFactory & factory); void registerFunctionCutQueryString(FunctionFactory & factory); void registerFunctionCutFragment(FunctionFactory & factory); @@ -34,6 +36,7 @@ void registerFunctionsURL(FunctionFactory & factory) registerFunctionDomain(factory); registerFunctionDomainWithoutWWW(factory); registerFunctionFirstSignificantSubdomain(factory); + registerFunctionFirstSignificantSubdomainCustom(factory); registerFunctionTopLevelDomain(factory); registerFunctionPort(factory); registerFunctionPath(factory); @@ -47,6 +50,7 @@ void registerFunctionsURL(FunctionFactory & factory) registerFunctionURLHierarchy(factory); registerFunctionURLPathHierarchy(factory); registerFunctionCutToFirstSignificantSubdomain(factory); + registerFunctionCutToFirstSignificantSubdomainCustom(factory); registerFunctionCutWWW(factory); registerFunctionCutQueryString(factory); registerFunctionCutFragment(factory); diff --git a/src/Functions/URL/tldLookup.h b/src/Functions/URL/tldLookup.h index 25857be3dd2..38c118b6bb1 100644 --- a/src/Functions/URL/tldLookup.h +++ b/src/Functions/URL/tldLookup.h @@ -1,5 +1,7 @@ #pragma once +#include + // Definition of the class generated by gperf, present on gperf/tldLookup.gperf class TopLevelDomainLookupHash { diff --git a/src/Functions/ya.make b/src/Functions/ya.make index 436a6a89996..f768ef0c374 100644 --- a/src/Functions/ya.make +++ b/src/Functions/ya.make @@ -80,6 +80,7 @@ SRCS( URL/cutQueryString.cpp URL/cutQueryStringAndFragment.cpp URL/cutToFirstSignificantSubdomain.cpp + URL/cutToFirstSignificantSubdomainCustom.cpp URL/cutURLParameter.cpp URL/cutWWW.cpp URL/decodeURLComponent.cpp @@ -89,6 +90,7 @@ SRCS( URL/extractURLParameterNames.cpp URL/extractURLParameters.cpp URL/firstSignificantSubdomain.cpp + URL/firstSignificantSubdomainCustom.cpp URL/fragment.cpp URL/netloc.cpp URL/path.cpp From 8875767b87f6fa6cb04a706711bf567beccde7d7 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 124/174] Add a test for custom TLD --- .../config.d/top_level_domains_lists.xml | 9 + tests/config/install.sh | 7 + tests/config/public_suffix_list.dat | 13491 ++++++++++++++++ .../0_stateless/01601_custom_tld.reference | 11 + .../queries/0_stateless/01601_custom_tld.sql | 16 + 5 files changed, 13534 insertions(+) create mode 100644 tests/config/config.d/top_level_domains_lists.xml create mode 100644 tests/config/public_suffix_list.dat create mode 100644 tests/queries/0_stateless/01601_custom_tld.reference create mode 100644 tests/queries/0_stateless/01601_custom_tld.sql diff --git a/tests/config/config.d/top_level_domains_lists.xml b/tests/config/config.d/top_level_domains_lists.xml new file mode 100644 index 00000000000..1b60e555ef9 --- /dev/null +++ b/tests/config/config.d/top_level_domains_lists.xml @@ -0,0 +1,9 @@ + + + + public_suffix_list.dat + + diff --git a/tests/config/install.sh b/tests/config/install.sh index d7127a8f690..126bc2eb086 100755 --- a/tests/config/install.sh +++ b/tests/config/install.sh @@ -30,6 +30,7 @@ ln -sf $SRC_PATH/config.d/database_atomic.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/test_cluster_with_incorrect_pw.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/logging_no_rotate.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/tcp_with_proxy.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/top_level_domains_lists.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/users.d/log_queries.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/readonly.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/access_management.xml $DEST_SERVER_PATH/users.d/ @@ -41,6 +42,12 @@ ln -sf $SRC_PATH/strings_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/decimals_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/executable_dictionary.xml $DEST_SERVER_PATH/ +# For CI when it is runned via init service +mkdir -p /var/lib/clickhouse/cores +ln -sf $SRC_PATH/public_suffix_list.dat /var/lib/clickhouse/cores/ +# For fasttest, that runs server just in background +ln -sf $SRC_PATH/public_suffix_list.dat ./ + ln -sf $SRC_PATH/server.key $DEST_SERVER_PATH/ ln -sf $SRC_PATH/server.crt $DEST_SERVER_PATH/ ln -sf $SRC_PATH/dhparam.pem $DEST_SERVER_PATH/ diff --git a/tests/config/public_suffix_list.dat b/tests/config/public_suffix_list.dat new file mode 100644 index 00000000000..1ede2b929a0 --- /dev/null +++ b/tests/config/public_suffix_list.dat @@ -0,0 +1,13491 @@ +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this +// file, You can obtain one at https://mozilla.org/MPL/2.0/. + +// Please pull this list from, and only from https://publicsuffix.org/list/public_suffix_list.dat, +// rather than any other VCS sites. Pulling from any other URL is not guaranteed to be supported. + +// Instructions on pulling and using this list can be found at https://publicsuffix.org/list/. + +// ===BEGIN ICANN DOMAINS=== + +// ac : https://en.wikipedia.org/wiki/.ac +ac +com.ac +edu.ac +gov.ac +net.ac +mil.ac +org.ac + +// ad : https://en.wikipedia.org/wiki/.ad +ad +nom.ad + +// ae : https://en.wikipedia.org/wiki/.ae +// see also: "Domain Name Eligibility Policy" at http://www.aeda.ae/eng/aepolicy.php +ae +co.ae +net.ae +org.ae +sch.ae +ac.ae +gov.ae +mil.ae + +// aero : see https://www.information.aero/index.php?id=66 +aero +accident-investigation.aero +accident-prevention.aero +aerobatic.aero +aeroclub.aero +aerodrome.aero +agents.aero +aircraft.aero +airline.aero +airport.aero +air-surveillance.aero +airtraffic.aero +air-traffic-control.aero +ambulance.aero +amusement.aero +association.aero +author.aero +ballooning.aero +broker.aero +caa.aero +cargo.aero +catering.aero +certification.aero +championship.aero +charter.aero +civilaviation.aero +club.aero +conference.aero +consultant.aero +consulting.aero +control.aero +council.aero +crew.aero +design.aero +dgca.aero +educator.aero +emergency.aero +engine.aero +engineer.aero +entertainment.aero +equipment.aero +exchange.aero +express.aero +federation.aero +flight.aero +fuel.aero +gliding.aero +government.aero +groundhandling.aero +group.aero +hanggliding.aero +homebuilt.aero +insurance.aero +journal.aero +journalist.aero +leasing.aero +logistics.aero +magazine.aero +maintenance.aero +media.aero +microlight.aero +modelling.aero +navigation.aero +parachuting.aero +paragliding.aero +passenger-association.aero +pilot.aero +press.aero +production.aero +recreation.aero +repbody.aero +res.aero +research.aero +rotorcraft.aero +safety.aero +scientist.aero +services.aero +show.aero +skydiving.aero +software.aero +student.aero +trader.aero +trading.aero +trainer.aero +union.aero +workinggroup.aero +works.aero + +// af : http://www.nic.af/help.jsp +af +gov.af +com.af +org.af +net.af +edu.af + +// ag : http://www.nic.ag/prices.htm +ag +com.ag +org.ag +net.ag +co.ag +nom.ag + +// ai : http://nic.com.ai/ +ai +off.ai +com.ai +net.ai +org.ai + +// al : http://www.ert.gov.al/ert_alb/faq_det.html?Id=31 +al +com.al +edu.al +gov.al +mil.al +net.al +org.al + +// am : https://www.amnic.net/policy/en/Policy_EN.pdf +am +co.am +com.am +commune.am +net.am +org.am + +// ao : https://en.wikipedia.org/wiki/.ao +// http://www.dns.ao/REGISTR.DOC +ao +ed.ao +gv.ao +og.ao +co.ao +pb.ao +it.ao + +// aq : https://en.wikipedia.org/wiki/.aq +aq + +// ar : https://nic.ar/nic-argentina/normativa-vigente +ar +com.ar +edu.ar +gob.ar +gov.ar +int.ar +mil.ar +musica.ar +net.ar +org.ar +tur.ar + +// arpa : https://en.wikipedia.org/wiki/.arpa +// Confirmed by registry 2008-06-18 +arpa +e164.arpa +in-addr.arpa +ip6.arpa +iris.arpa +uri.arpa +urn.arpa + +// as : https://en.wikipedia.org/wiki/.as +as +gov.as + +// asia : https://en.wikipedia.org/wiki/.asia +asia + +// at : https://en.wikipedia.org/wiki/.at +// Confirmed by registry 2008-06-17 +at +ac.at +co.at +gv.at +or.at +sth.ac.at + +// au : https://en.wikipedia.org/wiki/.au +// http://www.auda.org.au/ +au +// 2LDs +com.au +net.au +org.au +edu.au +gov.au +asn.au +id.au +// Historic 2LDs (closed to new registration, but sites still exist) +info.au +conf.au +oz.au +// CGDNs - http://www.cgdn.org.au/ +act.au +nsw.au +nt.au +qld.au +sa.au +tas.au +vic.au +wa.au +// 3LDs +act.edu.au +catholic.edu.au +// eq.edu.au - Removed at the request of the Queensland Department of Education +nsw.edu.au +nt.edu.au +qld.edu.au +sa.edu.au +tas.edu.au +vic.edu.au +wa.edu.au +// act.gov.au Bug 984824 - Removed at request of Greg Tankard +// nsw.gov.au Bug 547985 - Removed at request of +// nt.gov.au Bug 940478 - Removed at request of Greg Connors +qld.gov.au +sa.gov.au +tas.gov.au +vic.gov.au +wa.gov.au +// 4LDs +// education.tas.edu.au - Removed at the request of the Department of Education Tasmania +schools.nsw.edu.au + +// aw : https://en.wikipedia.org/wiki/.aw +aw +com.aw + +// ax : https://en.wikipedia.org/wiki/.ax +ax + +// az : https://en.wikipedia.org/wiki/.az +az +com.az +net.az +int.az +gov.az +org.az +edu.az +info.az +pp.az +mil.az +name.az +pro.az +biz.az + +// ba : http://nic.ba/users_data/files/pravilnik_o_registraciji.pdf +ba +com.ba +edu.ba +gov.ba +mil.ba +net.ba +org.ba + +// bb : https://en.wikipedia.org/wiki/.bb +bb +biz.bb +co.bb +com.bb +edu.bb +gov.bb +info.bb +net.bb +org.bb +store.bb +tv.bb + +// bd : https://en.wikipedia.org/wiki/.bd +*.bd + +// be : https://en.wikipedia.org/wiki/.be +// Confirmed by registry 2008-06-08 +be +ac.be + +// bf : https://en.wikipedia.org/wiki/.bf +bf +gov.bf + +// bg : https://en.wikipedia.org/wiki/.bg +// https://www.register.bg/user/static/rules/en/index.html +bg +a.bg +b.bg +c.bg +d.bg +e.bg +f.bg +g.bg +h.bg +i.bg +j.bg +k.bg +l.bg +m.bg +n.bg +o.bg +p.bg +q.bg +r.bg +s.bg +t.bg +u.bg +v.bg +w.bg +x.bg +y.bg +z.bg +0.bg +1.bg +2.bg +3.bg +4.bg +5.bg +6.bg +7.bg +8.bg +9.bg + +// bh : https://en.wikipedia.org/wiki/.bh +bh +com.bh +edu.bh +net.bh +org.bh +gov.bh + +// bi : https://en.wikipedia.org/wiki/.bi +// http://whois.nic.bi/ +bi +co.bi +com.bi +edu.bi +or.bi +org.bi + +// biz : https://en.wikipedia.org/wiki/.biz +biz + +// bj : https://en.wikipedia.org/wiki/.bj +bj +asso.bj +barreau.bj +gouv.bj + +// bm : http://www.bermudanic.bm/dnr-text.txt +bm +com.bm +edu.bm +gov.bm +net.bm +org.bm + +// bn : http://www.bnnic.bn/faqs +bn +com.bn +edu.bn +gov.bn +net.bn +org.bn + +// bo : https://nic.bo/delegacion2015.php#h-1.10 +bo +com.bo +edu.bo +gob.bo +int.bo +org.bo +net.bo +mil.bo +tv.bo +web.bo +// Social Domains +academia.bo +agro.bo +arte.bo +blog.bo +bolivia.bo +ciencia.bo +cooperativa.bo +democracia.bo +deporte.bo +ecologia.bo +economia.bo +empresa.bo +indigena.bo +industria.bo +info.bo +medicina.bo +movimiento.bo +musica.bo +natural.bo +nombre.bo +noticias.bo +patria.bo +politica.bo +profesional.bo +plurinacional.bo +pueblo.bo +revista.bo +salud.bo +tecnologia.bo +tksat.bo +transporte.bo +wiki.bo + +// br : http://registro.br/dominio/categoria.html +// Submitted by registry +br +9guacu.br +abc.br +adm.br +adv.br +agr.br +aju.br +am.br +anani.br +aparecida.br +app.br +arq.br +art.br +ato.br +b.br +barueri.br +belem.br +bhz.br +bib.br +bio.br +blog.br +bmd.br +boavista.br +bsb.br +campinagrande.br +campinas.br +caxias.br +cim.br +cng.br +cnt.br +com.br +contagem.br +coop.br +coz.br +cri.br +cuiaba.br +curitiba.br +def.br +des.br +det.br +dev.br +ecn.br +eco.br +edu.br +emp.br +enf.br +eng.br +esp.br +etc.br +eti.br +far.br +feira.br +flog.br +floripa.br +fm.br +fnd.br +fortal.br +fot.br +foz.br +fst.br +g12.br +geo.br +ggf.br +goiania.br +gov.br +// gov.br 26 states + df https://en.wikipedia.org/wiki/States_of_Brazil +ac.gov.br +al.gov.br +am.gov.br +ap.gov.br +ba.gov.br +ce.gov.br +df.gov.br +es.gov.br +go.gov.br +ma.gov.br +mg.gov.br +ms.gov.br +mt.gov.br +pa.gov.br +pb.gov.br +pe.gov.br +pi.gov.br +pr.gov.br +rj.gov.br +rn.gov.br +ro.gov.br +rr.gov.br +rs.gov.br +sc.gov.br +se.gov.br +sp.gov.br +to.gov.br +gru.br +imb.br +ind.br +inf.br +jab.br +jampa.br +jdf.br +joinville.br +jor.br +jus.br +leg.br +lel.br +log.br +londrina.br +macapa.br +maceio.br +manaus.br +maringa.br +mat.br +med.br +mil.br +morena.br +mp.br +mus.br +natal.br +net.br +niteroi.br +*.nom.br +not.br +ntr.br +odo.br +ong.br +org.br +osasco.br +palmas.br +poa.br +ppg.br +pro.br +psc.br +psi.br +pvh.br +qsl.br +radio.br +rec.br +recife.br +rep.br +ribeirao.br +rio.br +riobranco.br +riopreto.br +salvador.br +sampa.br +santamaria.br +santoandre.br +saobernardo.br +saogonca.br +seg.br +sjc.br +slg.br +slz.br +sorocaba.br +srv.br +taxi.br +tc.br +tec.br +teo.br +the.br +tmp.br +trd.br +tur.br +tv.br +udi.br +vet.br +vix.br +vlog.br +wiki.br +zlg.br + +// bs : http://www.nic.bs/rules.html +bs +com.bs +net.bs +org.bs +edu.bs +gov.bs + +// bt : https://en.wikipedia.org/wiki/.bt +bt +com.bt +edu.bt +gov.bt +net.bt +org.bt + +// bv : No registrations at this time. +// Submitted by registry +bv + +// bw : https://en.wikipedia.org/wiki/.bw +// http://www.gobin.info/domainname/bw.doc +// list of other 2nd level tlds ? +bw +co.bw +org.bw + +// by : https://en.wikipedia.org/wiki/.by +// http://tld.by/rules_2006_en.html +// list of other 2nd level tlds ? +by +gov.by +mil.by +// Official information does not indicate that com.by is a reserved +// second-level domain, but it's being used as one (see www.google.com.by and +// www.yahoo.com.by, for example), so we list it here for safety's sake. +com.by + +// http://hoster.by/ +of.by + +// bz : https://en.wikipedia.org/wiki/.bz +// http://www.belizenic.bz/ +bz +com.bz +net.bz +org.bz +edu.bz +gov.bz + +// ca : https://en.wikipedia.org/wiki/.ca +ca +// ca geographical names +ab.ca +bc.ca +mb.ca +nb.ca +nf.ca +nl.ca +ns.ca +nt.ca +nu.ca +on.ca +pe.ca +qc.ca +sk.ca +yk.ca +// gc.ca: https://en.wikipedia.org/wiki/.gc.ca +// see also: http://registry.gc.ca/en/SubdomainFAQ +gc.ca + +// cat : https://en.wikipedia.org/wiki/.cat +cat + +// cc : https://en.wikipedia.org/wiki/.cc +cc + +// cd : https://en.wikipedia.org/wiki/.cd +// see also: https://www.nic.cd/domain/insertDomain_2.jsp?act=1 +cd +gov.cd + +// cf : https://en.wikipedia.org/wiki/.cf +cf + +// cg : https://en.wikipedia.org/wiki/.cg +cg + +// ch : https://en.wikipedia.org/wiki/.ch +ch + +// ci : https://en.wikipedia.org/wiki/.ci +// http://www.nic.ci/index.php?page=charte +ci +org.ci +or.ci +com.ci +co.ci +edu.ci +ed.ci +ac.ci +net.ci +go.ci +asso.ci +aéroport.ci +int.ci +presse.ci +md.ci +gouv.ci + +// ck : https://en.wikipedia.org/wiki/.ck +*.ck +!www.ck + +// cl : https://www.nic.cl +// Confirmed by .CL registry +cl +aprendemas.cl +co.cl +gob.cl +gov.cl +mil.cl + +// cm : https://en.wikipedia.org/wiki/.cm plus bug 981927 +cm +co.cm +com.cm +gov.cm +net.cm + +// cn : https://en.wikipedia.org/wiki/.cn +// Submitted by registry +cn +ac.cn +com.cn +edu.cn +gov.cn +net.cn +org.cn +mil.cn +公司.cn +网络.cn +網絡.cn +// cn geographic names +ah.cn +bj.cn +cq.cn +fj.cn +gd.cn +gs.cn +gz.cn +gx.cn +ha.cn +hb.cn +he.cn +hi.cn +hl.cn +hn.cn +jl.cn +js.cn +jx.cn +ln.cn +nm.cn +nx.cn +qh.cn +sc.cn +sd.cn +sh.cn +sn.cn +sx.cn +tj.cn +xj.cn +xz.cn +yn.cn +zj.cn +hk.cn +mo.cn +tw.cn + +// co : https://en.wikipedia.org/wiki/.co +// Submitted by registry +co +arts.co +com.co +edu.co +firm.co +gov.co +info.co +int.co +mil.co +net.co +nom.co +org.co +rec.co +web.co + +// com : https://en.wikipedia.org/wiki/.com +com + +// coop : https://en.wikipedia.org/wiki/.coop +coop + +// cr : http://www.nic.cr/niccr_publico/showRegistroDominiosScreen.do +cr +ac.cr +co.cr +ed.cr +fi.cr +go.cr +or.cr +sa.cr + +// cu : https://en.wikipedia.org/wiki/.cu +cu +com.cu +edu.cu +org.cu +net.cu +gov.cu +inf.cu + +// cv : https://en.wikipedia.org/wiki/.cv +cv + +// cw : http://www.una.cw/cw_registry/ +// Confirmed by registry 2013-03-26 +cw +com.cw +edu.cw +net.cw +org.cw + +// cx : https://en.wikipedia.org/wiki/.cx +// list of other 2nd level tlds ? +cx +gov.cx + +// cy : http://www.nic.cy/ +// Submitted by registry Panayiotou Fotia +cy +ac.cy +biz.cy +com.cy +ekloges.cy +gov.cy +ltd.cy +name.cy +net.cy +org.cy +parliament.cy +press.cy +pro.cy +tm.cy + +// cz : https://en.wikipedia.org/wiki/.cz +cz + +// de : https://en.wikipedia.org/wiki/.de +// Confirmed by registry (with technical +// reservations) 2008-07-01 +de + +// dj : https://en.wikipedia.org/wiki/.dj +dj + +// dk : https://en.wikipedia.org/wiki/.dk +// Confirmed by registry 2008-06-17 +dk + +// dm : https://en.wikipedia.org/wiki/.dm +dm +com.dm +net.dm +org.dm +edu.dm +gov.dm + +// do : https://en.wikipedia.org/wiki/.do +do +art.do +com.do +edu.do +gob.do +gov.do +mil.do +net.do +org.do +sld.do +web.do + +// dz : http://www.nic.dz/images/pdf_nic/charte.pdf +dz +art.dz +asso.dz +com.dz +edu.dz +gov.dz +org.dz +net.dz +pol.dz +soc.dz +tm.dz + +// ec : http://www.nic.ec/reg/paso1.asp +// Submitted by registry +ec +com.ec +info.ec +net.ec +fin.ec +k12.ec +med.ec +pro.ec +org.ec +edu.ec +gov.ec +gob.ec +mil.ec + +// edu : https://en.wikipedia.org/wiki/.edu +edu + +// ee : http://www.eenet.ee/EENet/dom_reeglid.html#lisa_B +ee +edu.ee +gov.ee +riik.ee +lib.ee +med.ee +com.ee +pri.ee +aip.ee +org.ee +fie.ee + +// eg : https://en.wikipedia.org/wiki/.eg +eg +com.eg +edu.eg +eun.eg +gov.eg +mil.eg +name.eg +net.eg +org.eg +sci.eg + +// er : https://en.wikipedia.org/wiki/.er +*.er + +// es : https://www.nic.es/site_ingles/ingles/dominios/index.html +es +com.es +nom.es +org.es +gob.es +edu.es + +// et : https://en.wikipedia.org/wiki/.et +et +com.et +gov.et +org.et +edu.et +biz.et +name.et +info.et +net.et + +// eu : https://en.wikipedia.org/wiki/.eu +eu + +// fi : https://en.wikipedia.org/wiki/.fi +fi +// aland.fi : https://en.wikipedia.org/wiki/.ax +// This domain is being phased out in favor of .ax. As there are still many +// domains under aland.fi, we still keep it on the list until aland.fi is +// completely removed. +// TODO: Check for updates (expected to be phased out around Q1/2009) +aland.fi + +// fj : http://domains.fj/ +// Submitted by registry 2020-02-11 +fj +ac.fj +biz.fj +com.fj +gov.fj +info.fj +mil.fj +name.fj +net.fj +org.fj +pro.fj + +// fk : https://en.wikipedia.org/wiki/.fk +*.fk + +// fm : https://en.wikipedia.org/wiki/.fm +com.fm +edu.fm +net.fm +org.fm +fm + +// fo : https://en.wikipedia.org/wiki/.fo +fo + +// fr : http://www.afnic.fr/ +// domaines descriptifs : https://www.afnic.fr/medias/documents/Cadre_legal/Afnic_Naming_Policy_12122016_VEN.pdf +fr +asso.fr +com.fr +gouv.fr +nom.fr +prd.fr +tm.fr +// domaines sectoriels : https://www.afnic.fr/en/products-and-services/the-fr-tld/sector-based-fr-domains-4.html +aeroport.fr +avocat.fr +avoues.fr +cci.fr +chambagri.fr +chirurgiens-dentistes.fr +experts-comptables.fr +geometre-expert.fr +greta.fr +huissier-justice.fr +medecin.fr +notaires.fr +pharmacien.fr +port.fr +veterinaire.fr + +// ga : https://en.wikipedia.org/wiki/.ga +ga + +// gb : This registry is effectively dormant +// Submitted by registry +gb + +// gd : https://en.wikipedia.org/wiki/.gd +edu.gd +gov.gd +gd + +// ge : http://www.nic.net.ge/policy_en.pdf +ge +com.ge +edu.ge +gov.ge +org.ge +mil.ge +net.ge +pvt.ge + +// gf : https://en.wikipedia.org/wiki/.gf +gf + +// gg : http://www.channelisles.net/register-domains/ +// Confirmed by registry 2013-11-28 +gg +co.gg +net.gg +org.gg + +// gh : https://en.wikipedia.org/wiki/.gh +// see also: http://www.nic.gh/reg_now.php +// Although domains directly at second level are not possible at the moment, +// they have been possible for some time and may come back. +gh +com.gh +edu.gh +gov.gh +org.gh +mil.gh + +// gi : http://www.nic.gi/rules.html +gi +com.gi +ltd.gi +gov.gi +mod.gi +edu.gi +org.gi + +// gl : https://en.wikipedia.org/wiki/.gl +// http://nic.gl +gl +co.gl +com.gl +edu.gl +net.gl +org.gl + +// gm : http://www.nic.gm/htmlpages%5Cgm-policy.htm +gm + +// gn : http://psg.com/dns/gn/gn.txt +// Submitted by registry +gn +ac.gn +com.gn +edu.gn +gov.gn +org.gn +net.gn + +// gov : https://en.wikipedia.org/wiki/.gov +gov + +// gp : http://www.nic.gp/index.php?lang=en +gp +com.gp +net.gp +mobi.gp +edu.gp +org.gp +asso.gp + +// gq : https://en.wikipedia.org/wiki/.gq +gq + +// gr : https://grweb.ics.forth.gr/english/1617-B-2005.html +// Submitted by registry +gr +com.gr +edu.gr +net.gr +org.gr +gov.gr + +// gs : https://en.wikipedia.org/wiki/.gs +gs + +// gt : https://www.gt/sitio/registration_policy.php?lang=en +gt +com.gt +edu.gt +gob.gt +ind.gt +mil.gt +net.gt +org.gt + +// gu : http://gadao.gov.gu/register.html +// University of Guam : https://www.uog.edu +// Submitted by uognoc@triton.uog.edu +gu +com.gu +edu.gu +gov.gu +guam.gu +info.gu +net.gu +org.gu +web.gu + +// gw : https://en.wikipedia.org/wiki/.gw +gw + +// gy : https://en.wikipedia.org/wiki/.gy +// http://registry.gy/ +gy +co.gy +com.gy +edu.gy +gov.gy +net.gy +org.gy + +// hk : https://www.hkirc.hk +// Submitted by registry +hk +com.hk +edu.hk +gov.hk +idv.hk +net.hk +org.hk +公司.hk +教育.hk +敎育.hk +政府.hk +個人.hk +个人.hk +箇人.hk +網络.hk +网络.hk +组織.hk +網絡.hk +网絡.hk +组织.hk +組織.hk +組织.hk + +// hm : https://en.wikipedia.org/wiki/.hm +hm + +// hn : http://www.nic.hn/politicas/ps02,,05.html +hn +com.hn +edu.hn +org.hn +net.hn +mil.hn +gob.hn + +// hr : http://www.dns.hr/documents/pdf/HRTLD-regulations.pdf +hr +iz.hr +from.hr +name.hr +com.hr + +// ht : http://www.nic.ht/info/charte.cfm +ht +com.ht +shop.ht +firm.ht +info.ht +adult.ht +net.ht +pro.ht +org.ht +med.ht +art.ht +coop.ht +pol.ht +asso.ht +edu.ht +rel.ht +gouv.ht +perso.ht + +// hu : http://www.domain.hu/domain/English/sld.html +// Confirmed by registry 2008-06-12 +hu +co.hu +info.hu +org.hu +priv.hu +sport.hu +tm.hu +2000.hu +agrar.hu +bolt.hu +casino.hu +city.hu +erotica.hu +erotika.hu +film.hu +forum.hu +games.hu +hotel.hu +ingatlan.hu +jogasz.hu +konyvelo.hu +lakas.hu +media.hu +news.hu +reklam.hu +sex.hu +shop.hu +suli.hu +szex.hu +tozsde.hu +utazas.hu +video.hu + +// id : https://pandi.id/en/domain/registration-requirements/ +id +ac.id +biz.id +co.id +desa.id +go.id +mil.id +my.id +net.id +or.id +ponpes.id +sch.id +web.id + +// ie : https://en.wikipedia.org/wiki/.ie +ie +gov.ie + +// il : http://www.isoc.org.il/domains/ +il +ac.il +co.il +gov.il +idf.il +k12.il +muni.il +net.il +org.il + +// im : https://www.nic.im/ +// Submitted by registry +im +ac.im +co.im +com.im +ltd.co.im +net.im +org.im +plc.co.im +tt.im +tv.im + +// in : https://en.wikipedia.org/wiki/.in +// see also: https://registry.in/Policies +// Please note, that nic.in is not an official eTLD, but used by most +// government institutions. +in +co.in +firm.in +net.in +org.in +gen.in +ind.in +nic.in +ac.in +edu.in +res.in +gov.in +mil.in + +// info : https://en.wikipedia.org/wiki/.info +info + +// int : https://en.wikipedia.org/wiki/.int +// Confirmed by registry 2008-06-18 +int +eu.int + +// io : http://www.nic.io/rules.html +// list of other 2nd level tlds ? +io +com.io + +// iq : http://www.cmc.iq/english/iq/iqregister1.htm +iq +gov.iq +edu.iq +mil.iq +com.iq +org.iq +net.iq + +// ir : http://www.nic.ir/Terms_and_Conditions_ir,_Appendix_1_Domain_Rules +// Also see http://www.nic.ir/Internationalized_Domain_Names +// Two .ir entries added at request of , 2010-04-16 +ir +ac.ir +co.ir +gov.ir +id.ir +net.ir +org.ir +sch.ir +// xn--mgba3a4f16a.ir (.ir, Persian YEH) +ایران.ir +// xn--mgba3a4fra.ir (.ir, Arabic YEH) +ايران.ir + +// is : http://www.isnic.is/domain/rules.php +// Confirmed by registry 2008-12-06 +is +net.is +com.is +edu.is +gov.is +org.is +int.is + +// it : https://en.wikipedia.org/wiki/.it +it +gov.it +edu.it +// Reserved geo-names (regions and provinces): +// https://www.nic.it/sites/default/files/archivio/docs/Regulation_assignation_v7.1.pdf +// Regions +abr.it +abruzzo.it +aosta-valley.it +aostavalley.it +bas.it +basilicata.it +cal.it +calabria.it +cam.it +campania.it +emilia-romagna.it +emiliaromagna.it +emr.it +friuli-v-giulia.it +friuli-ve-giulia.it +friuli-vegiulia.it +friuli-venezia-giulia.it +friuli-veneziagiulia.it +friuli-vgiulia.it +friuliv-giulia.it +friulive-giulia.it +friulivegiulia.it +friulivenezia-giulia.it +friuliveneziagiulia.it +friulivgiulia.it +fvg.it +laz.it +lazio.it +lig.it +liguria.it +lom.it +lombardia.it +lombardy.it +lucania.it +mar.it +marche.it +mol.it +molise.it +piedmont.it +piemonte.it +pmn.it +pug.it +puglia.it +sar.it +sardegna.it +sardinia.it +sic.it +sicilia.it +sicily.it +taa.it +tos.it +toscana.it +trentin-sud-tirol.it +trentin-süd-tirol.it +trentin-sudtirol.it +trentin-südtirol.it +trentin-sued-tirol.it +trentin-suedtirol.it +trentino-a-adige.it +trentino-aadige.it +trentino-alto-adige.it +trentino-altoadige.it +trentino-s-tirol.it +trentino-stirol.it +trentino-sud-tirol.it +trentino-süd-tirol.it +trentino-sudtirol.it +trentino-südtirol.it +trentino-sued-tirol.it +trentino-suedtirol.it +trentino.it +trentinoa-adige.it +trentinoaadige.it +trentinoalto-adige.it +trentinoaltoadige.it +trentinos-tirol.it +trentinostirol.it +trentinosud-tirol.it +trentinosüd-tirol.it +trentinosudtirol.it +trentinosüdtirol.it +trentinosued-tirol.it +trentinosuedtirol.it +trentinsud-tirol.it +trentinsüd-tirol.it +trentinsudtirol.it +trentinsüdtirol.it +trentinsued-tirol.it +trentinsuedtirol.it +tuscany.it +umb.it +umbria.it +val-d-aosta.it +val-daosta.it +vald-aosta.it +valdaosta.it +valle-aosta.it +valle-d-aosta.it +valle-daosta.it +valleaosta.it +valled-aosta.it +valledaosta.it +vallee-aoste.it +vallée-aoste.it +vallee-d-aoste.it +vallée-d-aoste.it +valleeaoste.it +valléeaoste.it +valleedaoste.it +valléedaoste.it +vao.it +vda.it +ven.it +veneto.it +// Provinces +ag.it +agrigento.it +al.it +alessandria.it +alto-adige.it +altoadige.it +an.it +ancona.it +andria-barletta-trani.it +andria-trani-barletta.it +andriabarlettatrani.it +andriatranibarletta.it +ao.it +aosta.it +aoste.it +ap.it +aq.it +aquila.it +ar.it +arezzo.it +ascoli-piceno.it +ascolipiceno.it +asti.it +at.it +av.it +avellino.it +ba.it +balsan-sudtirol.it +balsan-südtirol.it +balsan-suedtirol.it +balsan.it +bari.it +barletta-trani-andria.it +barlettatraniandria.it +belluno.it +benevento.it +bergamo.it +bg.it +bi.it +biella.it +bl.it +bn.it +bo.it +bologna.it +bolzano-altoadige.it +bolzano.it +bozen-sudtirol.it +bozen-südtirol.it +bozen-suedtirol.it +bozen.it +br.it +brescia.it +brindisi.it +bs.it +bt.it +bulsan-sudtirol.it +bulsan-südtirol.it +bulsan-suedtirol.it +bulsan.it +bz.it +ca.it +cagliari.it +caltanissetta.it +campidano-medio.it +campidanomedio.it +campobasso.it +carbonia-iglesias.it +carboniaiglesias.it +carrara-massa.it +carraramassa.it +caserta.it +catania.it +catanzaro.it +cb.it +ce.it +cesena-forli.it +cesena-forlì.it +cesenaforli.it +cesenaforlì.it +ch.it +chieti.it +ci.it +cl.it +cn.it +co.it +como.it +cosenza.it +cr.it +cremona.it +crotone.it +cs.it +ct.it +cuneo.it +cz.it +dell-ogliastra.it +dellogliastra.it +en.it +enna.it +fc.it +fe.it +fermo.it +ferrara.it +fg.it +fi.it +firenze.it +florence.it +fm.it +foggia.it +forli-cesena.it +forlì-cesena.it +forlicesena.it +forlìcesena.it +fr.it +frosinone.it +ge.it +genoa.it +genova.it +go.it +gorizia.it +gr.it +grosseto.it +iglesias-carbonia.it +iglesiascarbonia.it +im.it +imperia.it +is.it +isernia.it +kr.it +la-spezia.it +laquila.it +laspezia.it +latina.it +lc.it +le.it +lecce.it +lecco.it +li.it +livorno.it +lo.it +lodi.it +lt.it +lu.it +lucca.it +macerata.it +mantova.it +massa-carrara.it +massacarrara.it +matera.it +mb.it +mc.it +me.it +medio-campidano.it +mediocampidano.it +messina.it +mi.it +milan.it +milano.it +mn.it +mo.it +modena.it +monza-brianza.it +monza-e-della-brianza.it +monza.it +monzabrianza.it +monzaebrianza.it +monzaedellabrianza.it +ms.it +mt.it +na.it +naples.it +napoli.it +no.it +novara.it +nu.it +nuoro.it +og.it +ogliastra.it +olbia-tempio.it +olbiatempio.it +or.it +oristano.it +ot.it +pa.it +padova.it +padua.it +palermo.it +parma.it +pavia.it +pc.it +pd.it +pe.it +perugia.it +pesaro-urbino.it +pesarourbino.it +pescara.it +pg.it +pi.it +piacenza.it +pisa.it +pistoia.it +pn.it +po.it +pordenone.it +potenza.it +pr.it +prato.it +pt.it +pu.it +pv.it +pz.it +ra.it +ragusa.it +ravenna.it +rc.it +re.it +reggio-calabria.it +reggio-emilia.it +reggiocalabria.it +reggioemilia.it +rg.it +ri.it +rieti.it +rimini.it +rm.it +rn.it +ro.it +roma.it +rome.it +rovigo.it +sa.it +salerno.it +sassari.it +savona.it +si.it +siena.it +siracusa.it +so.it +sondrio.it +sp.it +sr.it +ss.it +suedtirol.it +südtirol.it +sv.it +ta.it +taranto.it +te.it +tempio-olbia.it +tempioolbia.it +teramo.it +terni.it +tn.it +to.it +torino.it +tp.it +tr.it +trani-andria-barletta.it +trani-barletta-andria.it +traniandriabarletta.it +tranibarlettaandria.it +trapani.it +trento.it +treviso.it +trieste.it +ts.it +turin.it +tv.it +ud.it +udine.it +urbino-pesaro.it +urbinopesaro.it +va.it +varese.it +vb.it +vc.it +ve.it +venezia.it +venice.it +verbania.it +vercelli.it +verona.it +vi.it +vibo-valentia.it +vibovalentia.it +vicenza.it +viterbo.it +vr.it +vs.it +vt.it +vv.it + +// je : http://www.channelisles.net/register-domains/ +// Confirmed by registry 2013-11-28 +je +co.je +net.je +org.je + +// jm : http://www.com.jm/register.html +*.jm + +// jo : http://www.dns.jo/Registration_policy.aspx +jo +com.jo +org.jo +net.jo +edu.jo +sch.jo +gov.jo +mil.jo +name.jo + +// jobs : https://en.wikipedia.org/wiki/.jobs +jobs + +// jp : https://en.wikipedia.org/wiki/.jp +// http://jprs.co.jp/en/jpdomain.html +// Submitted by registry +jp +// jp organizational type names +ac.jp +ad.jp +co.jp +ed.jp +go.jp +gr.jp +lg.jp +ne.jp +or.jp +// jp prefecture type names +aichi.jp +akita.jp +aomori.jp +chiba.jp +ehime.jp +fukui.jp +fukuoka.jp +fukushima.jp +gifu.jp +gunma.jp +hiroshima.jp +hokkaido.jp +hyogo.jp +ibaraki.jp +ishikawa.jp +iwate.jp +kagawa.jp +kagoshima.jp +kanagawa.jp +kochi.jp +kumamoto.jp +kyoto.jp +mie.jp +miyagi.jp +miyazaki.jp +nagano.jp +nagasaki.jp +nara.jp +niigata.jp +oita.jp +okayama.jp +okinawa.jp +osaka.jp +saga.jp +saitama.jp +shiga.jp +shimane.jp +shizuoka.jp +tochigi.jp +tokushima.jp +tokyo.jp +tottori.jp +toyama.jp +wakayama.jp +yamagata.jp +yamaguchi.jp +yamanashi.jp +栃木.jp +愛知.jp +愛媛.jp +兵庫.jp +熊本.jp +茨城.jp +北海道.jp +千葉.jp +和歌山.jp +長崎.jp +長野.jp +新潟.jp +青森.jp +静岡.jp +東京.jp +石川.jp +埼玉.jp +三重.jp +京都.jp +佐賀.jp +大分.jp +大阪.jp +奈良.jp +宮城.jp +宮崎.jp +富山.jp +山口.jp +山形.jp +山梨.jp +岩手.jp +岐阜.jp +岡山.jp +島根.jp +広島.jp +徳島.jp +沖縄.jp +滋賀.jp +神奈川.jp +福井.jp +福岡.jp +福島.jp +秋田.jp +群馬.jp +香川.jp +高知.jp +鳥取.jp +鹿児島.jp +// jp geographic type names +// http://jprs.jp/doc/rule/saisoku-1.html +*.kawasaki.jp +*.kitakyushu.jp +*.kobe.jp +*.nagoya.jp +*.sapporo.jp +*.sendai.jp +*.yokohama.jp +!city.kawasaki.jp +!city.kitakyushu.jp +!city.kobe.jp +!city.nagoya.jp +!city.sapporo.jp +!city.sendai.jp +!city.yokohama.jp +// 4th level registration +aisai.aichi.jp +ama.aichi.jp +anjo.aichi.jp +asuke.aichi.jp +chiryu.aichi.jp +chita.aichi.jp +fuso.aichi.jp +gamagori.aichi.jp +handa.aichi.jp +hazu.aichi.jp +hekinan.aichi.jp +higashiura.aichi.jp +ichinomiya.aichi.jp +inazawa.aichi.jp +inuyama.aichi.jp +isshiki.aichi.jp +iwakura.aichi.jp +kanie.aichi.jp +kariya.aichi.jp +kasugai.aichi.jp +kira.aichi.jp +kiyosu.aichi.jp +komaki.aichi.jp +konan.aichi.jp +kota.aichi.jp +mihama.aichi.jp +miyoshi.aichi.jp +nishio.aichi.jp +nisshin.aichi.jp +obu.aichi.jp +oguchi.aichi.jp +oharu.aichi.jp +okazaki.aichi.jp +owariasahi.aichi.jp +seto.aichi.jp +shikatsu.aichi.jp +shinshiro.aichi.jp +shitara.aichi.jp +tahara.aichi.jp +takahama.aichi.jp +tobishima.aichi.jp +toei.aichi.jp +togo.aichi.jp +tokai.aichi.jp +tokoname.aichi.jp +toyoake.aichi.jp +toyohashi.aichi.jp +toyokawa.aichi.jp +toyone.aichi.jp +toyota.aichi.jp +tsushima.aichi.jp +yatomi.aichi.jp +akita.akita.jp +daisen.akita.jp +fujisato.akita.jp +gojome.akita.jp +hachirogata.akita.jp +happou.akita.jp +higashinaruse.akita.jp +honjo.akita.jp +honjyo.akita.jp +ikawa.akita.jp +kamikoani.akita.jp +kamioka.akita.jp +katagami.akita.jp +kazuno.akita.jp +kitaakita.akita.jp +kosaka.akita.jp +kyowa.akita.jp +misato.akita.jp +mitane.akita.jp +moriyoshi.akita.jp +nikaho.akita.jp +noshiro.akita.jp +odate.akita.jp +oga.akita.jp +ogata.akita.jp +semboku.akita.jp +yokote.akita.jp +yurihonjo.akita.jp +aomori.aomori.jp +gonohe.aomori.jp +hachinohe.aomori.jp +hashikami.aomori.jp +hiranai.aomori.jp +hirosaki.aomori.jp +itayanagi.aomori.jp +kuroishi.aomori.jp +misawa.aomori.jp +mutsu.aomori.jp +nakadomari.aomori.jp +noheji.aomori.jp +oirase.aomori.jp +owani.aomori.jp +rokunohe.aomori.jp +sannohe.aomori.jp +shichinohe.aomori.jp +shingo.aomori.jp +takko.aomori.jp +towada.aomori.jp +tsugaru.aomori.jp +tsuruta.aomori.jp +abiko.chiba.jp +asahi.chiba.jp +chonan.chiba.jp +chosei.chiba.jp +choshi.chiba.jp +chuo.chiba.jp +funabashi.chiba.jp +futtsu.chiba.jp +hanamigawa.chiba.jp +ichihara.chiba.jp +ichikawa.chiba.jp +ichinomiya.chiba.jp +inzai.chiba.jp +isumi.chiba.jp +kamagaya.chiba.jp +kamogawa.chiba.jp +kashiwa.chiba.jp +katori.chiba.jp +katsuura.chiba.jp +kimitsu.chiba.jp +kisarazu.chiba.jp +kozaki.chiba.jp +kujukuri.chiba.jp +kyonan.chiba.jp +matsudo.chiba.jp +midori.chiba.jp +mihama.chiba.jp +minamiboso.chiba.jp +mobara.chiba.jp +mutsuzawa.chiba.jp +nagara.chiba.jp +nagareyama.chiba.jp +narashino.chiba.jp +narita.chiba.jp +noda.chiba.jp +oamishirasato.chiba.jp +omigawa.chiba.jp +onjuku.chiba.jp +otaki.chiba.jp +sakae.chiba.jp +sakura.chiba.jp +shimofusa.chiba.jp +shirako.chiba.jp +shiroi.chiba.jp +shisui.chiba.jp +sodegaura.chiba.jp +sosa.chiba.jp +tako.chiba.jp +tateyama.chiba.jp +togane.chiba.jp +tohnosho.chiba.jp +tomisato.chiba.jp +urayasu.chiba.jp +yachimata.chiba.jp +yachiyo.chiba.jp +yokaichiba.chiba.jp +yokoshibahikari.chiba.jp +yotsukaido.chiba.jp +ainan.ehime.jp +honai.ehime.jp +ikata.ehime.jp +imabari.ehime.jp +iyo.ehime.jp +kamijima.ehime.jp +kihoku.ehime.jp +kumakogen.ehime.jp +masaki.ehime.jp +matsuno.ehime.jp +matsuyama.ehime.jp +namikata.ehime.jp +niihama.ehime.jp +ozu.ehime.jp +saijo.ehime.jp +seiyo.ehime.jp +shikokuchuo.ehime.jp +tobe.ehime.jp +toon.ehime.jp +uchiko.ehime.jp +uwajima.ehime.jp +yawatahama.ehime.jp +echizen.fukui.jp +eiheiji.fukui.jp +fukui.fukui.jp +ikeda.fukui.jp +katsuyama.fukui.jp +mihama.fukui.jp +minamiechizen.fukui.jp +obama.fukui.jp +ohi.fukui.jp +ono.fukui.jp +sabae.fukui.jp +sakai.fukui.jp +takahama.fukui.jp +tsuruga.fukui.jp +wakasa.fukui.jp +ashiya.fukuoka.jp +buzen.fukuoka.jp +chikugo.fukuoka.jp +chikuho.fukuoka.jp +chikujo.fukuoka.jp +chikushino.fukuoka.jp +chikuzen.fukuoka.jp +chuo.fukuoka.jp +dazaifu.fukuoka.jp +fukuchi.fukuoka.jp +hakata.fukuoka.jp +higashi.fukuoka.jp +hirokawa.fukuoka.jp +hisayama.fukuoka.jp +iizuka.fukuoka.jp +inatsuki.fukuoka.jp +kaho.fukuoka.jp +kasuga.fukuoka.jp +kasuya.fukuoka.jp +kawara.fukuoka.jp +keisen.fukuoka.jp +koga.fukuoka.jp +kurate.fukuoka.jp +kurogi.fukuoka.jp +kurume.fukuoka.jp +minami.fukuoka.jp +miyako.fukuoka.jp +miyama.fukuoka.jp +miyawaka.fukuoka.jp +mizumaki.fukuoka.jp +munakata.fukuoka.jp +nakagawa.fukuoka.jp +nakama.fukuoka.jp +nishi.fukuoka.jp +nogata.fukuoka.jp +ogori.fukuoka.jp +okagaki.fukuoka.jp +okawa.fukuoka.jp +oki.fukuoka.jp +omuta.fukuoka.jp +onga.fukuoka.jp +onojo.fukuoka.jp +oto.fukuoka.jp +saigawa.fukuoka.jp +sasaguri.fukuoka.jp +shingu.fukuoka.jp +shinyoshitomi.fukuoka.jp +shonai.fukuoka.jp +soeda.fukuoka.jp +sue.fukuoka.jp +tachiarai.fukuoka.jp +tagawa.fukuoka.jp +takata.fukuoka.jp +toho.fukuoka.jp +toyotsu.fukuoka.jp +tsuiki.fukuoka.jp +ukiha.fukuoka.jp +umi.fukuoka.jp +usui.fukuoka.jp +yamada.fukuoka.jp +yame.fukuoka.jp +yanagawa.fukuoka.jp +yukuhashi.fukuoka.jp +aizubange.fukushima.jp +aizumisato.fukushima.jp +aizuwakamatsu.fukushima.jp +asakawa.fukushima.jp +bandai.fukushima.jp +date.fukushima.jp +fukushima.fukushima.jp +furudono.fukushima.jp +futaba.fukushima.jp +hanawa.fukushima.jp +higashi.fukushima.jp +hirata.fukushima.jp +hirono.fukushima.jp +iitate.fukushima.jp +inawashiro.fukushima.jp +ishikawa.fukushima.jp +iwaki.fukushima.jp +izumizaki.fukushima.jp +kagamiishi.fukushima.jp +kaneyama.fukushima.jp +kawamata.fukushima.jp +kitakata.fukushima.jp +kitashiobara.fukushima.jp +koori.fukushima.jp +koriyama.fukushima.jp +kunimi.fukushima.jp +miharu.fukushima.jp +mishima.fukushima.jp +namie.fukushima.jp +nango.fukushima.jp +nishiaizu.fukushima.jp +nishigo.fukushima.jp +okuma.fukushima.jp +omotego.fukushima.jp +ono.fukushima.jp +otama.fukushima.jp +samegawa.fukushima.jp +shimogo.fukushima.jp +shirakawa.fukushima.jp +showa.fukushima.jp +soma.fukushima.jp +sukagawa.fukushima.jp +taishin.fukushima.jp +tamakawa.fukushima.jp +tanagura.fukushima.jp +tenei.fukushima.jp +yabuki.fukushima.jp +yamato.fukushima.jp +yamatsuri.fukushima.jp +yanaizu.fukushima.jp +yugawa.fukushima.jp +anpachi.gifu.jp +ena.gifu.jp +gifu.gifu.jp +ginan.gifu.jp +godo.gifu.jp +gujo.gifu.jp +hashima.gifu.jp +hichiso.gifu.jp +hida.gifu.jp +higashishirakawa.gifu.jp +ibigawa.gifu.jp +ikeda.gifu.jp +kakamigahara.gifu.jp +kani.gifu.jp +kasahara.gifu.jp +kasamatsu.gifu.jp +kawaue.gifu.jp +kitagata.gifu.jp +mino.gifu.jp +minokamo.gifu.jp +mitake.gifu.jp +mizunami.gifu.jp +motosu.gifu.jp +nakatsugawa.gifu.jp +ogaki.gifu.jp +sakahogi.gifu.jp +seki.gifu.jp +sekigahara.gifu.jp +shirakawa.gifu.jp +tajimi.gifu.jp +takayama.gifu.jp +tarui.gifu.jp +toki.gifu.jp +tomika.gifu.jp +wanouchi.gifu.jp +yamagata.gifu.jp +yaotsu.gifu.jp +yoro.gifu.jp +annaka.gunma.jp +chiyoda.gunma.jp +fujioka.gunma.jp +higashiagatsuma.gunma.jp +isesaki.gunma.jp +itakura.gunma.jp +kanna.gunma.jp +kanra.gunma.jp +katashina.gunma.jp +kawaba.gunma.jp +kiryu.gunma.jp +kusatsu.gunma.jp +maebashi.gunma.jp +meiwa.gunma.jp +midori.gunma.jp +minakami.gunma.jp +naganohara.gunma.jp +nakanojo.gunma.jp +nanmoku.gunma.jp +numata.gunma.jp +oizumi.gunma.jp +ora.gunma.jp +ota.gunma.jp +shibukawa.gunma.jp +shimonita.gunma.jp +shinto.gunma.jp +showa.gunma.jp +takasaki.gunma.jp +takayama.gunma.jp +tamamura.gunma.jp +tatebayashi.gunma.jp +tomioka.gunma.jp +tsukiyono.gunma.jp +tsumagoi.gunma.jp +ueno.gunma.jp +yoshioka.gunma.jp +asaminami.hiroshima.jp +daiwa.hiroshima.jp +etajima.hiroshima.jp +fuchu.hiroshima.jp +fukuyama.hiroshima.jp +hatsukaichi.hiroshima.jp +higashihiroshima.hiroshima.jp +hongo.hiroshima.jp +jinsekikogen.hiroshima.jp +kaita.hiroshima.jp +kui.hiroshima.jp +kumano.hiroshima.jp +kure.hiroshima.jp +mihara.hiroshima.jp +miyoshi.hiroshima.jp +naka.hiroshima.jp +onomichi.hiroshima.jp +osakikamijima.hiroshima.jp +otake.hiroshima.jp +saka.hiroshima.jp +sera.hiroshima.jp +seranishi.hiroshima.jp +shinichi.hiroshima.jp +shobara.hiroshima.jp +takehara.hiroshima.jp +abashiri.hokkaido.jp +abira.hokkaido.jp +aibetsu.hokkaido.jp +akabira.hokkaido.jp +akkeshi.hokkaido.jp +asahikawa.hokkaido.jp +ashibetsu.hokkaido.jp +ashoro.hokkaido.jp +assabu.hokkaido.jp +atsuma.hokkaido.jp +bibai.hokkaido.jp +biei.hokkaido.jp +bifuka.hokkaido.jp +bihoro.hokkaido.jp +biratori.hokkaido.jp +chippubetsu.hokkaido.jp +chitose.hokkaido.jp +date.hokkaido.jp +ebetsu.hokkaido.jp +embetsu.hokkaido.jp +eniwa.hokkaido.jp +erimo.hokkaido.jp +esan.hokkaido.jp +esashi.hokkaido.jp +fukagawa.hokkaido.jp +fukushima.hokkaido.jp +furano.hokkaido.jp +furubira.hokkaido.jp +haboro.hokkaido.jp +hakodate.hokkaido.jp +hamatonbetsu.hokkaido.jp +hidaka.hokkaido.jp +higashikagura.hokkaido.jp +higashikawa.hokkaido.jp +hiroo.hokkaido.jp +hokuryu.hokkaido.jp +hokuto.hokkaido.jp +honbetsu.hokkaido.jp +horokanai.hokkaido.jp +horonobe.hokkaido.jp +ikeda.hokkaido.jp +imakane.hokkaido.jp +ishikari.hokkaido.jp +iwamizawa.hokkaido.jp +iwanai.hokkaido.jp +kamifurano.hokkaido.jp +kamikawa.hokkaido.jp +kamishihoro.hokkaido.jp +kamisunagawa.hokkaido.jp +kamoenai.hokkaido.jp +kayabe.hokkaido.jp +kembuchi.hokkaido.jp +kikonai.hokkaido.jp +kimobetsu.hokkaido.jp +kitahiroshima.hokkaido.jp +kitami.hokkaido.jp +kiyosato.hokkaido.jp +koshimizu.hokkaido.jp +kunneppu.hokkaido.jp +kuriyama.hokkaido.jp +kuromatsunai.hokkaido.jp +kushiro.hokkaido.jp +kutchan.hokkaido.jp +kyowa.hokkaido.jp +mashike.hokkaido.jp +matsumae.hokkaido.jp +mikasa.hokkaido.jp +minamifurano.hokkaido.jp +mombetsu.hokkaido.jp +moseushi.hokkaido.jp +mukawa.hokkaido.jp +muroran.hokkaido.jp +naie.hokkaido.jp +nakagawa.hokkaido.jp +nakasatsunai.hokkaido.jp +nakatombetsu.hokkaido.jp +nanae.hokkaido.jp +nanporo.hokkaido.jp +nayoro.hokkaido.jp +nemuro.hokkaido.jp +niikappu.hokkaido.jp +niki.hokkaido.jp +nishiokoppe.hokkaido.jp +noboribetsu.hokkaido.jp +numata.hokkaido.jp +obihiro.hokkaido.jp +obira.hokkaido.jp +oketo.hokkaido.jp +okoppe.hokkaido.jp +otaru.hokkaido.jp +otobe.hokkaido.jp +otofuke.hokkaido.jp +otoineppu.hokkaido.jp +oumu.hokkaido.jp +ozora.hokkaido.jp +pippu.hokkaido.jp +rankoshi.hokkaido.jp +rebun.hokkaido.jp +rikubetsu.hokkaido.jp +rishiri.hokkaido.jp +rishirifuji.hokkaido.jp +saroma.hokkaido.jp +sarufutsu.hokkaido.jp +shakotan.hokkaido.jp +shari.hokkaido.jp +shibecha.hokkaido.jp +shibetsu.hokkaido.jp +shikabe.hokkaido.jp +shikaoi.hokkaido.jp +shimamaki.hokkaido.jp +shimizu.hokkaido.jp +shimokawa.hokkaido.jp +shinshinotsu.hokkaido.jp +shintoku.hokkaido.jp +shiranuka.hokkaido.jp +shiraoi.hokkaido.jp +shiriuchi.hokkaido.jp +sobetsu.hokkaido.jp +sunagawa.hokkaido.jp +taiki.hokkaido.jp +takasu.hokkaido.jp +takikawa.hokkaido.jp +takinoue.hokkaido.jp +teshikaga.hokkaido.jp +tobetsu.hokkaido.jp +tohma.hokkaido.jp +tomakomai.hokkaido.jp +tomari.hokkaido.jp +toya.hokkaido.jp +toyako.hokkaido.jp +toyotomi.hokkaido.jp +toyoura.hokkaido.jp +tsubetsu.hokkaido.jp +tsukigata.hokkaido.jp +urakawa.hokkaido.jp +urausu.hokkaido.jp +uryu.hokkaido.jp +utashinai.hokkaido.jp +wakkanai.hokkaido.jp +wassamu.hokkaido.jp +yakumo.hokkaido.jp +yoichi.hokkaido.jp +aioi.hyogo.jp +akashi.hyogo.jp +ako.hyogo.jp +amagasaki.hyogo.jp +aogaki.hyogo.jp +asago.hyogo.jp +ashiya.hyogo.jp +awaji.hyogo.jp +fukusaki.hyogo.jp +goshiki.hyogo.jp +harima.hyogo.jp +himeji.hyogo.jp +ichikawa.hyogo.jp +inagawa.hyogo.jp +itami.hyogo.jp +kakogawa.hyogo.jp +kamigori.hyogo.jp +kamikawa.hyogo.jp +kasai.hyogo.jp +kasuga.hyogo.jp +kawanishi.hyogo.jp +miki.hyogo.jp +minamiawaji.hyogo.jp +nishinomiya.hyogo.jp +nishiwaki.hyogo.jp +ono.hyogo.jp +sanda.hyogo.jp +sannan.hyogo.jp +sasayama.hyogo.jp +sayo.hyogo.jp +shingu.hyogo.jp +shinonsen.hyogo.jp +shiso.hyogo.jp +sumoto.hyogo.jp +taishi.hyogo.jp +taka.hyogo.jp +takarazuka.hyogo.jp +takasago.hyogo.jp +takino.hyogo.jp +tamba.hyogo.jp +tatsuno.hyogo.jp +toyooka.hyogo.jp +yabu.hyogo.jp +yashiro.hyogo.jp +yoka.hyogo.jp +yokawa.hyogo.jp +ami.ibaraki.jp +asahi.ibaraki.jp +bando.ibaraki.jp +chikusei.ibaraki.jp +daigo.ibaraki.jp +fujishiro.ibaraki.jp +hitachi.ibaraki.jp +hitachinaka.ibaraki.jp +hitachiomiya.ibaraki.jp +hitachiota.ibaraki.jp +ibaraki.ibaraki.jp +ina.ibaraki.jp +inashiki.ibaraki.jp +itako.ibaraki.jp +iwama.ibaraki.jp +joso.ibaraki.jp +kamisu.ibaraki.jp +kasama.ibaraki.jp +kashima.ibaraki.jp +kasumigaura.ibaraki.jp +koga.ibaraki.jp +miho.ibaraki.jp +mito.ibaraki.jp +moriya.ibaraki.jp +naka.ibaraki.jp +namegata.ibaraki.jp +oarai.ibaraki.jp +ogawa.ibaraki.jp +omitama.ibaraki.jp +ryugasaki.ibaraki.jp +sakai.ibaraki.jp +sakuragawa.ibaraki.jp +shimodate.ibaraki.jp +shimotsuma.ibaraki.jp +shirosato.ibaraki.jp +sowa.ibaraki.jp +suifu.ibaraki.jp +takahagi.ibaraki.jp +tamatsukuri.ibaraki.jp +tokai.ibaraki.jp +tomobe.ibaraki.jp +tone.ibaraki.jp +toride.ibaraki.jp +tsuchiura.ibaraki.jp +tsukuba.ibaraki.jp +uchihara.ibaraki.jp +ushiku.ibaraki.jp +yachiyo.ibaraki.jp +yamagata.ibaraki.jp +yawara.ibaraki.jp +yuki.ibaraki.jp +anamizu.ishikawa.jp +hakui.ishikawa.jp +hakusan.ishikawa.jp +kaga.ishikawa.jp +kahoku.ishikawa.jp +kanazawa.ishikawa.jp +kawakita.ishikawa.jp +komatsu.ishikawa.jp +nakanoto.ishikawa.jp +nanao.ishikawa.jp +nomi.ishikawa.jp +nonoichi.ishikawa.jp +noto.ishikawa.jp +shika.ishikawa.jp +suzu.ishikawa.jp +tsubata.ishikawa.jp +tsurugi.ishikawa.jp +uchinada.ishikawa.jp +wajima.ishikawa.jp +fudai.iwate.jp +fujisawa.iwate.jp +hanamaki.iwate.jp +hiraizumi.iwate.jp +hirono.iwate.jp +ichinohe.iwate.jp +ichinoseki.iwate.jp +iwaizumi.iwate.jp +iwate.iwate.jp +joboji.iwate.jp +kamaishi.iwate.jp +kanegasaki.iwate.jp +karumai.iwate.jp +kawai.iwate.jp +kitakami.iwate.jp +kuji.iwate.jp +kunohe.iwate.jp +kuzumaki.iwate.jp +miyako.iwate.jp +mizusawa.iwate.jp +morioka.iwate.jp +ninohe.iwate.jp +noda.iwate.jp +ofunato.iwate.jp +oshu.iwate.jp +otsuchi.iwate.jp +rikuzentakata.iwate.jp +shiwa.iwate.jp +shizukuishi.iwate.jp +sumita.iwate.jp +tanohata.iwate.jp +tono.iwate.jp +yahaba.iwate.jp +yamada.iwate.jp +ayagawa.kagawa.jp +higashikagawa.kagawa.jp +kanonji.kagawa.jp +kotohira.kagawa.jp +manno.kagawa.jp +marugame.kagawa.jp +mitoyo.kagawa.jp +naoshima.kagawa.jp +sanuki.kagawa.jp +tadotsu.kagawa.jp +takamatsu.kagawa.jp +tonosho.kagawa.jp +uchinomi.kagawa.jp +utazu.kagawa.jp +zentsuji.kagawa.jp +akune.kagoshima.jp +amami.kagoshima.jp +hioki.kagoshima.jp +isa.kagoshima.jp +isen.kagoshima.jp +izumi.kagoshima.jp +kagoshima.kagoshima.jp +kanoya.kagoshima.jp +kawanabe.kagoshima.jp +kinko.kagoshima.jp +kouyama.kagoshima.jp +makurazaki.kagoshima.jp +matsumoto.kagoshima.jp +minamitane.kagoshima.jp +nakatane.kagoshima.jp +nishinoomote.kagoshima.jp +satsumasendai.kagoshima.jp +soo.kagoshima.jp +tarumizu.kagoshima.jp +yusui.kagoshima.jp +aikawa.kanagawa.jp +atsugi.kanagawa.jp +ayase.kanagawa.jp +chigasaki.kanagawa.jp +ebina.kanagawa.jp +fujisawa.kanagawa.jp +hadano.kanagawa.jp +hakone.kanagawa.jp +hiratsuka.kanagawa.jp +isehara.kanagawa.jp +kaisei.kanagawa.jp +kamakura.kanagawa.jp +kiyokawa.kanagawa.jp +matsuda.kanagawa.jp +minamiashigara.kanagawa.jp +miura.kanagawa.jp +nakai.kanagawa.jp +ninomiya.kanagawa.jp +odawara.kanagawa.jp +oi.kanagawa.jp +oiso.kanagawa.jp +sagamihara.kanagawa.jp +samukawa.kanagawa.jp +tsukui.kanagawa.jp +yamakita.kanagawa.jp +yamato.kanagawa.jp +yokosuka.kanagawa.jp +yugawara.kanagawa.jp +zama.kanagawa.jp +zushi.kanagawa.jp +aki.kochi.jp +geisei.kochi.jp +hidaka.kochi.jp +higashitsuno.kochi.jp +ino.kochi.jp +kagami.kochi.jp +kami.kochi.jp +kitagawa.kochi.jp +kochi.kochi.jp +mihara.kochi.jp +motoyama.kochi.jp +muroto.kochi.jp +nahari.kochi.jp +nakamura.kochi.jp +nankoku.kochi.jp +nishitosa.kochi.jp +niyodogawa.kochi.jp +ochi.kochi.jp +okawa.kochi.jp +otoyo.kochi.jp +otsuki.kochi.jp +sakawa.kochi.jp +sukumo.kochi.jp +susaki.kochi.jp +tosa.kochi.jp +tosashimizu.kochi.jp +toyo.kochi.jp +tsuno.kochi.jp +umaji.kochi.jp +yasuda.kochi.jp +yusuhara.kochi.jp +amakusa.kumamoto.jp +arao.kumamoto.jp +aso.kumamoto.jp +choyo.kumamoto.jp +gyokuto.kumamoto.jp +kamiamakusa.kumamoto.jp +kikuchi.kumamoto.jp +kumamoto.kumamoto.jp +mashiki.kumamoto.jp +mifune.kumamoto.jp +minamata.kumamoto.jp +minamioguni.kumamoto.jp +nagasu.kumamoto.jp +nishihara.kumamoto.jp +oguni.kumamoto.jp +ozu.kumamoto.jp +sumoto.kumamoto.jp +takamori.kumamoto.jp +uki.kumamoto.jp +uto.kumamoto.jp +yamaga.kumamoto.jp +yamato.kumamoto.jp +yatsushiro.kumamoto.jp +ayabe.kyoto.jp +fukuchiyama.kyoto.jp +higashiyama.kyoto.jp +ide.kyoto.jp +ine.kyoto.jp +joyo.kyoto.jp +kameoka.kyoto.jp +kamo.kyoto.jp +kita.kyoto.jp +kizu.kyoto.jp +kumiyama.kyoto.jp +kyotamba.kyoto.jp +kyotanabe.kyoto.jp +kyotango.kyoto.jp +maizuru.kyoto.jp +minami.kyoto.jp +minamiyamashiro.kyoto.jp +miyazu.kyoto.jp +muko.kyoto.jp +nagaokakyo.kyoto.jp +nakagyo.kyoto.jp +nantan.kyoto.jp +oyamazaki.kyoto.jp +sakyo.kyoto.jp +seika.kyoto.jp +tanabe.kyoto.jp +uji.kyoto.jp +ujitawara.kyoto.jp +wazuka.kyoto.jp +yamashina.kyoto.jp +yawata.kyoto.jp +asahi.mie.jp +inabe.mie.jp +ise.mie.jp +kameyama.mie.jp +kawagoe.mie.jp +kiho.mie.jp +kisosaki.mie.jp +kiwa.mie.jp +komono.mie.jp +kumano.mie.jp +kuwana.mie.jp +matsusaka.mie.jp +meiwa.mie.jp +mihama.mie.jp +minamiise.mie.jp +misugi.mie.jp +miyama.mie.jp +nabari.mie.jp +shima.mie.jp +suzuka.mie.jp +tado.mie.jp +taiki.mie.jp +taki.mie.jp +tamaki.mie.jp +toba.mie.jp +tsu.mie.jp +udono.mie.jp +ureshino.mie.jp +watarai.mie.jp +yokkaichi.mie.jp +furukawa.miyagi.jp +higashimatsushima.miyagi.jp +ishinomaki.miyagi.jp +iwanuma.miyagi.jp +kakuda.miyagi.jp +kami.miyagi.jp +kawasaki.miyagi.jp +marumori.miyagi.jp +matsushima.miyagi.jp +minamisanriku.miyagi.jp +misato.miyagi.jp +murata.miyagi.jp +natori.miyagi.jp +ogawara.miyagi.jp +ohira.miyagi.jp +onagawa.miyagi.jp +osaki.miyagi.jp +rifu.miyagi.jp +semine.miyagi.jp +shibata.miyagi.jp +shichikashuku.miyagi.jp +shikama.miyagi.jp +shiogama.miyagi.jp +shiroishi.miyagi.jp +tagajo.miyagi.jp +taiwa.miyagi.jp +tome.miyagi.jp +tomiya.miyagi.jp +wakuya.miyagi.jp +watari.miyagi.jp +yamamoto.miyagi.jp +zao.miyagi.jp +aya.miyazaki.jp +ebino.miyazaki.jp +gokase.miyazaki.jp +hyuga.miyazaki.jp +kadogawa.miyazaki.jp +kawaminami.miyazaki.jp +kijo.miyazaki.jp +kitagawa.miyazaki.jp +kitakata.miyazaki.jp +kitaura.miyazaki.jp +kobayashi.miyazaki.jp +kunitomi.miyazaki.jp +kushima.miyazaki.jp +mimata.miyazaki.jp +miyakonojo.miyazaki.jp +miyazaki.miyazaki.jp +morotsuka.miyazaki.jp +nichinan.miyazaki.jp +nishimera.miyazaki.jp +nobeoka.miyazaki.jp +saito.miyazaki.jp +shiiba.miyazaki.jp +shintomi.miyazaki.jp +takaharu.miyazaki.jp +takanabe.miyazaki.jp +takazaki.miyazaki.jp +tsuno.miyazaki.jp +achi.nagano.jp +agematsu.nagano.jp +anan.nagano.jp +aoki.nagano.jp +asahi.nagano.jp +azumino.nagano.jp +chikuhoku.nagano.jp +chikuma.nagano.jp +chino.nagano.jp +fujimi.nagano.jp +hakuba.nagano.jp +hara.nagano.jp +hiraya.nagano.jp +iida.nagano.jp +iijima.nagano.jp +iiyama.nagano.jp +iizuna.nagano.jp +ikeda.nagano.jp +ikusaka.nagano.jp +ina.nagano.jp +karuizawa.nagano.jp +kawakami.nagano.jp +kiso.nagano.jp +kisofukushima.nagano.jp +kitaaiki.nagano.jp +komagane.nagano.jp +komoro.nagano.jp +matsukawa.nagano.jp +matsumoto.nagano.jp +miasa.nagano.jp +minamiaiki.nagano.jp +minamimaki.nagano.jp +minamiminowa.nagano.jp +minowa.nagano.jp +miyada.nagano.jp +miyota.nagano.jp +mochizuki.nagano.jp +nagano.nagano.jp +nagawa.nagano.jp +nagiso.nagano.jp +nakagawa.nagano.jp +nakano.nagano.jp +nozawaonsen.nagano.jp +obuse.nagano.jp +ogawa.nagano.jp +okaya.nagano.jp +omachi.nagano.jp +omi.nagano.jp +ookuwa.nagano.jp +ooshika.nagano.jp +otaki.nagano.jp +otari.nagano.jp +sakae.nagano.jp +sakaki.nagano.jp +saku.nagano.jp +sakuho.nagano.jp +shimosuwa.nagano.jp +shinanomachi.nagano.jp +shiojiri.nagano.jp +suwa.nagano.jp +suzaka.nagano.jp +takagi.nagano.jp +takamori.nagano.jp +takayama.nagano.jp +tateshina.nagano.jp +tatsuno.nagano.jp +togakushi.nagano.jp +togura.nagano.jp +tomi.nagano.jp +ueda.nagano.jp +wada.nagano.jp +yamagata.nagano.jp +yamanouchi.nagano.jp +yasaka.nagano.jp +yasuoka.nagano.jp +chijiwa.nagasaki.jp +futsu.nagasaki.jp +goto.nagasaki.jp +hasami.nagasaki.jp +hirado.nagasaki.jp +iki.nagasaki.jp +isahaya.nagasaki.jp +kawatana.nagasaki.jp +kuchinotsu.nagasaki.jp +matsuura.nagasaki.jp +nagasaki.nagasaki.jp +obama.nagasaki.jp +omura.nagasaki.jp +oseto.nagasaki.jp +saikai.nagasaki.jp +sasebo.nagasaki.jp +seihi.nagasaki.jp +shimabara.nagasaki.jp +shinkamigoto.nagasaki.jp +togitsu.nagasaki.jp +tsushima.nagasaki.jp +unzen.nagasaki.jp +ando.nara.jp +gose.nara.jp +heguri.nara.jp +higashiyoshino.nara.jp +ikaruga.nara.jp +ikoma.nara.jp +kamikitayama.nara.jp +kanmaki.nara.jp +kashiba.nara.jp +kashihara.nara.jp +katsuragi.nara.jp +kawai.nara.jp +kawakami.nara.jp +kawanishi.nara.jp +koryo.nara.jp +kurotaki.nara.jp +mitsue.nara.jp +miyake.nara.jp +nara.nara.jp +nosegawa.nara.jp +oji.nara.jp +ouda.nara.jp +oyodo.nara.jp +sakurai.nara.jp +sango.nara.jp +shimoichi.nara.jp +shimokitayama.nara.jp +shinjo.nara.jp +soni.nara.jp +takatori.nara.jp +tawaramoto.nara.jp +tenkawa.nara.jp +tenri.nara.jp +uda.nara.jp +yamatokoriyama.nara.jp +yamatotakada.nara.jp +yamazoe.nara.jp +yoshino.nara.jp +aga.niigata.jp +agano.niigata.jp +gosen.niigata.jp +itoigawa.niigata.jp +izumozaki.niigata.jp +joetsu.niigata.jp +kamo.niigata.jp +kariwa.niigata.jp +kashiwazaki.niigata.jp +minamiuonuma.niigata.jp +mitsuke.niigata.jp +muika.niigata.jp +murakami.niigata.jp +myoko.niigata.jp +nagaoka.niigata.jp +niigata.niigata.jp +ojiya.niigata.jp +omi.niigata.jp +sado.niigata.jp +sanjo.niigata.jp +seiro.niigata.jp +seirou.niigata.jp +sekikawa.niigata.jp +shibata.niigata.jp +tagami.niigata.jp +tainai.niigata.jp +tochio.niigata.jp +tokamachi.niigata.jp +tsubame.niigata.jp +tsunan.niigata.jp +uonuma.niigata.jp +yahiko.niigata.jp +yoita.niigata.jp +yuzawa.niigata.jp +beppu.oita.jp +bungoono.oita.jp +bungotakada.oita.jp +hasama.oita.jp +hiji.oita.jp +himeshima.oita.jp +hita.oita.jp +kamitsue.oita.jp +kokonoe.oita.jp +kuju.oita.jp +kunisaki.oita.jp +kusu.oita.jp +oita.oita.jp +saiki.oita.jp +taketa.oita.jp +tsukumi.oita.jp +usa.oita.jp +usuki.oita.jp +yufu.oita.jp +akaiwa.okayama.jp +asakuchi.okayama.jp +bizen.okayama.jp +hayashima.okayama.jp +ibara.okayama.jp +kagamino.okayama.jp +kasaoka.okayama.jp +kibichuo.okayama.jp +kumenan.okayama.jp +kurashiki.okayama.jp +maniwa.okayama.jp +misaki.okayama.jp +nagi.okayama.jp +niimi.okayama.jp +nishiawakura.okayama.jp +okayama.okayama.jp +satosho.okayama.jp +setouchi.okayama.jp +shinjo.okayama.jp +shoo.okayama.jp +soja.okayama.jp +takahashi.okayama.jp +tamano.okayama.jp +tsuyama.okayama.jp +wake.okayama.jp +yakage.okayama.jp +aguni.okinawa.jp +ginowan.okinawa.jp +ginoza.okinawa.jp +gushikami.okinawa.jp +haebaru.okinawa.jp +higashi.okinawa.jp +hirara.okinawa.jp +iheya.okinawa.jp +ishigaki.okinawa.jp +ishikawa.okinawa.jp +itoman.okinawa.jp +izena.okinawa.jp +kadena.okinawa.jp +kin.okinawa.jp +kitadaito.okinawa.jp +kitanakagusuku.okinawa.jp +kumejima.okinawa.jp +kunigami.okinawa.jp +minamidaito.okinawa.jp +motobu.okinawa.jp +nago.okinawa.jp +naha.okinawa.jp +nakagusuku.okinawa.jp +nakijin.okinawa.jp +nanjo.okinawa.jp +nishihara.okinawa.jp +ogimi.okinawa.jp +okinawa.okinawa.jp +onna.okinawa.jp +shimoji.okinawa.jp +taketomi.okinawa.jp +tarama.okinawa.jp +tokashiki.okinawa.jp +tomigusuku.okinawa.jp +tonaki.okinawa.jp +urasoe.okinawa.jp +uruma.okinawa.jp +yaese.okinawa.jp +yomitan.okinawa.jp +yonabaru.okinawa.jp +yonaguni.okinawa.jp +zamami.okinawa.jp +abeno.osaka.jp +chihayaakasaka.osaka.jp +chuo.osaka.jp +daito.osaka.jp +fujiidera.osaka.jp +habikino.osaka.jp +hannan.osaka.jp +higashiosaka.osaka.jp +higashisumiyoshi.osaka.jp +higashiyodogawa.osaka.jp +hirakata.osaka.jp +ibaraki.osaka.jp +ikeda.osaka.jp +izumi.osaka.jp +izumiotsu.osaka.jp +izumisano.osaka.jp +kadoma.osaka.jp +kaizuka.osaka.jp +kanan.osaka.jp +kashiwara.osaka.jp +katano.osaka.jp +kawachinagano.osaka.jp +kishiwada.osaka.jp +kita.osaka.jp +kumatori.osaka.jp +matsubara.osaka.jp +minato.osaka.jp +minoh.osaka.jp +misaki.osaka.jp +moriguchi.osaka.jp +neyagawa.osaka.jp +nishi.osaka.jp +nose.osaka.jp +osakasayama.osaka.jp +sakai.osaka.jp +sayama.osaka.jp +sennan.osaka.jp +settsu.osaka.jp +shijonawate.osaka.jp +shimamoto.osaka.jp +suita.osaka.jp +tadaoka.osaka.jp +taishi.osaka.jp +tajiri.osaka.jp +takaishi.osaka.jp +takatsuki.osaka.jp +tondabayashi.osaka.jp +toyonaka.osaka.jp +toyono.osaka.jp +yao.osaka.jp +ariake.saga.jp +arita.saga.jp +fukudomi.saga.jp +genkai.saga.jp +hamatama.saga.jp +hizen.saga.jp +imari.saga.jp +kamimine.saga.jp +kanzaki.saga.jp +karatsu.saga.jp +kashima.saga.jp +kitagata.saga.jp +kitahata.saga.jp +kiyama.saga.jp +kouhoku.saga.jp +kyuragi.saga.jp +nishiarita.saga.jp +ogi.saga.jp +omachi.saga.jp +ouchi.saga.jp +saga.saga.jp +shiroishi.saga.jp +taku.saga.jp +tara.saga.jp +tosu.saga.jp +yoshinogari.saga.jp +arakawa.saitama.jp +asaka.saitama.jp +chichibu.saitama.jp +fujimi.saitama.jp +fujimino.saitama.jp +fukaya.saitama.jp +hanno.saitama.jp +hanyu.saitama.jp +hasuda.saitama.jp +hatogaya.saitama.jp +hatoyama.saitama.jp +hidaka.saitama.jp +higashichichibu.saitama.jp +higashimatsuyama.saitama.jp +honjo.saitama.jp +ina.saitama.jp +iruma.saitama.jp +iwatsuki.saitama.jp +kamiizumi.saitama.jp +kamikawa.saitama.jp +kamisato.saitama.jp +kasukabe.saitama.jp +kawagoe.saitama.jp +kawaguchi.saitama.jp +kawajima.saitama.jp +kazo.saitama.jp +kitamoto.saitama.jp +koshigaya.saitama.jp +kounosu.saitama.jp +kuki.saitama.jp +kumagaya.saitama.jp +matsubushi.saitama.jp +minano.saitama.jp +misato.saitama.jp +miyashiro.saitama.jp +miyoshi.saitama.jp +moroyama.saitama.jp +nagatoro.saitama.jp +namegawa.saitama.jp +niiza.saitama.jp +ogano.saitama.jp +ogawa.saitama.jp +ogose.saitama.jp +okegawa.saitama.jp +omiya.saitama.jp +otaki.saitama.jp +ranzan.saitama.jp +ryokami.saitama.jp +saitama.saitama.jp +sakado.saitama.jp +satte.saitama.jp +sayama.saitama.jp +shiki.saitama.jp +shiraoka.saitama.jp +soka.saitama.jp +sugito.saitama.jp +toda.saitama.jp +tokigawa.saitama.jp +tokorozawa.saitama.jp +tsurugashima.saitama.jp +urawa.saitama.jp +warabi.saitama.jp +yashio.saitama.jp +yokoze.saitama.jp +yono.saitama.jp +yorii.saitama.jp +yoshida.saitama.jp +yoshikawa.saitama.jp +yoshimi.saitama.jp +aisho.shiga.jp +gamo.shiga.jp +higashiomi.shiga.jp +hikone.shiga.jp +koka.shiga.jp +konan.shiga.jp +kosei.shiga.jp +koto.shiga.jp +kusatsu.shiga.jp +maibara.shiga.jp +moriyama.shiga.jp +nagahama.shiga.jp +nishiazai.shiga.jp +notogawa.shiga.jp +omihachiman.shiga.jp +otsu.shiga.jp +ritto.shiga.jp +ryuoh.shiga.jp +takashima.shiga.jp +takatsuki.shiga.jp +torahime.shiga.jp +toyosato.shiga.jp +yasu.shiga.jp +akagi.shimane.jp +ama.shimane.jp +gotsu.shimane.jp +hamada.shimane.jp +higashiizumo.shimane.jp +hikawa.shimane.jp +hikimi.shimane.jp +izumo.shimane.jp +kakinoki.shimane.jp +masuda.shimane.jp +matsue.shimane.jp +misato.shimane.jp +nishinoshima.shimane.jp +ohda.shimane.jp +okinoshima.shimane.jp +okuizumo.shimane.jp +shimane.shimane.jp +tamayu.shimane.jp +tsuwano.shimane.jp +unnan.shimane.jp +yakumo.shimane.jp +yasugi.shimane.jp +yatsuka.shimane.jp +arai.shizuoka.jp +atami.shizuoka.jp +fuji.shizuoka.jp +fujieda.shizuoka.jp +fujikawa.shizuoka.jp +fujinomiya.shizuoka.jp +fukuroi.shizuoka.jp +gotemba.shizuoka.jp +haibara.shizuoka.jp +hamamatsu.shizuoka.jp +higashiizu.shizuoka.jp +ito.shizuoka.jp +iwata.shizuoka.jp +izu.shizuoka.jp +izunokuni.shizuoka.jp +kakegawa.shizuoka.jp +kannami.shizuoka.jp +kawanehon.shizuoka.jp +kawazu.shizuoka.jp +kikugawa.shizuoka.jp +kosai.shizuoka.jp +makinohara.shizuoka.jp +matsuzaki.shizuoka.jp +minamiizu.shizuoka.jp +mishima.shizuoka.jp +morimachi.shizuoka.jp +nishiizu.shizuoka.jp +numazu.shizuoka.jp +omaezaki.shizuoka.jp +shimada.shizuoka.jp +shimizu.shizuoka.jp +shimoda.shizuoka.jp +shizuoka.shizuoka.jp +susono.shizuoka.jp +yaizu.shizuoka.jp +yoshida.shizuoka.jp +ashikaga.tochigi.jp +bato.tochigi.jp +haga.tochigi.jp +ichikai.tochigi.jp +iwafune.tochigi.jp +kaminokawa.tochigi.jp +kanuma.tochigi.jp +karasuyama.tochigi.jp +kuroiso.tochigi.jp +mashiko.tochigi.jp +mibu.tochigi.jp +moka.tochigi.jp +motegi.tochigi.jp +nasu.tochigi.jp +nasushiobara.tochigi.jp +nikko.tochigi.jp +nishikata.tochigi.jp +nogi.tochigi.jp +ohira.tochigi.jp +ohtawara.tochigi.jp +oyama.tochigi.jp +sakura.tochigi.jp +sano.tochigi.jp +shimotsuke.tochigi.jp +shioya.tochigi.jp +takanezawa.tochigi.jp +tochigi.tochigi.jp +tsuga.tochigi.jp +ujiie.tochigi.jp +utsunomiya.tochigi.jp +yaita.tochigi.jp +aizumi.tokushima.jp +anan.tokushima.jp +ichiba.tokushima.jp +itano.tokushima.jp +kainan.tokushima.jp +komatsushima.tokushima.jp +matsushige.tokushima.jp +mima.tokushima.jp +minami.tokushima.jp +miyoshi.tokushima.jp +mugi.tokushima.jp +nakagawa.tokushima.jp +naruto.tokushima.jp +sanagochi.tokushima.jp +shishikui.tokushima.jp +tokushima.tokushima.jp +wajiki.tokushima.jp +adachi.tokyo.jp +akiruno.tokyo.jp +akishima.tokyo.jp +aogashima.tokyo.jp +arakawa.tokyo.jp +bunkyo.tokyo.jp +chiyoda.tokyo.jp +chofu.tokyo.jp +chuo.tokyo.jp +edogawa.tokyo.jp +fuchu.tokyo.jp +fussa.tokyo.jp +hachijo.tokyo.jp +hachioji.tokyo.jp +hamura.tokyo.jp +higashikurume.tokyo.jp +higashimurayama.tokyo.jp +higashiyamato.tokyo.jp +hino.tokyo.jp +hinode.tokyo.jp +hinohara.tokyo.jp +inagi.tokyo.jp +itabashi.tokyo.jp +katsushika.tokyo.jp +kita.tokyo.jp +kiyose.tokyo.jp +kodaira.tokyo.jp +koganei.tokyo.jp +kokubunji.tokyo.jp +komae.tokyo.jp +koto.tokyo.jp +kouzushima.tokyo.jp +kunitachi.tokyo.jp +machida.tokyo.jp +meguro.tokyo.jp +minato.tokyo.jp +mitaka.tokyo.jp +mizuho.tokyo.jp +musashimurayama.tokyo.jp +musashino.tokyo.jp +nakano.tokyo.jp +nerima.tokyo.jp +ogasawara.tokyo.jp +okutama.tokyo.jp +ome.tokyo.jp +oshima.tokyo.jp +ota.tokyo.jp +setagaya.tokyo.jp +shibuya.tokyo.jp +shinagawa.tokyo.jp +shinjuku.tokyo.jp +suginami.tokyo.jp +sumida.tokyo.jp +tachikawa.tokyo.jp +taito.tokyo.jp +tama.tokyo.jp +toshima.tokyo.jp +chizu.tottori.jp +hino.tottori.jp +kawahara.tottori.jp +koge.tottori.jp +kotoura.tottori.jp +misasa.tottori.jp +nanbu.tottori.jp +nichinan.tottori.jp +sakaiminato.tottori.jp +tottori.tottori.jp +wakasa.tottori.jp +yazu.tottori.jp +yonago.tottori.jp +asahi.toyama.jp +fuchu.toyama.jp +fukumitsu.toyama.jp +funahashi.toyama.jp +himi.toyama.jp +imizu.toyama.jp +inami.toyama.jp +johana.toyama.jp +kamiichi.toyama.jp +kurobe.toyama.jp +nakaniikawa.toyama.jp +namerikawa.toyama.jp +nanto.toyama.jp +nyuzen.toyama.jp +oyabe.toyama.jp +taira.toyama.jp +takaoka.toyama.jp +tateyama.toyama.jp +toga.toyama.jp +tonami.toyama.jp +toyama.toyama.jp +unazuki.toyama.jp +uozu.toyama.jp +yamada.toyama.jp +arida.wakayama.jp +aridagawa.wakayama.jp +gobo.wakayama.jp +hashimoto.wakayama.jp +hidaka.wakayama.jp +hirogawa.wakayama.jp +inami.wakayama.jp +iwade.wakayama.jp +kainan.wakayama.jp +kamitonda.wakayama.jp +katsuragi.wakayama.jp +kimino.wakayama.jp +kinokawa.wakayama.jp +kitayama.wakayama.jp +koya.wakayama.jp +koza.wakayama.jp +kozagawa.wakayama.jp +kudoyama.wakayama.jp +kushimoto.wakayama.jp +mihama.wakayama.jp +misato.wakayama.jp +nachikatsuura.wakayama.jp +shingu.wakayama.jp +shirahama.wakayama.jp +taiji.wakayama.jp +tanabe.wakayama.jp +wakayama.wakayama.jp +yuasa.wakayama.jp +yura.wakayama.jp +asahi.yamagata.jp +funagata.yamagata.jp +higashine.yamagata.jp +iide.yamagata.jp +kahoku.yamagata.jp +kaminoyama.yamagata.jp +kaneyama.yamagata.jp +kawanishi.yamagata.jp +mamurogawa.yamagata.jp +mikawa.yamagata.jp +murayama.yamagata.jp +nagai.yamagata.jp +nakayama.yamagata.jp +nanyo.yamagata.jp +nishikawa.yamagata.jp +obanazawa.yamagata.jp +oe.yamagata.jp +oguni.yamagata.jp +ohkura.yamagata.jp +oishida.yamagata.jp +sagae.yamagata.jp +sakata.yamagata.jp +sakegawa.yamagata.jp +shinjo.yamagata.jp +shirataka.yamagata.jp +shonai.yamagata.jp +takahata.yamagata.jp +tendo.yamagata.jp +tozawa.yamagata.jp +tsuruoka.yamagata.jp +yamagata.yamagata.jp +yamanobe.yamagata.jp +yonezawa.yamagata.jp +yuza.yamagata.jp +abu.yamaguchi.jp +hagi.yamaguchi.jp +hikari.yamaguchi.jp +hofu.yamaguchi.jp +iwakuni.yamaguchi.jp +kudamatsu.yamaguchi.jp +mitou.yamaguchi.jp +nagato.yamaguchi.jp +oshima.yamaguchi.jp +shimonoseki.yamaguchi.jp +shunan.yamaguchi.jp +tabuse.yamaguchi.jp +tokuyama.yamaguchi.jp +toyota.yamaguchi.jp +ube.yamaguchi.jp +yuu.yamaguchi.jp +chuo.yamanashi.jp +doshi.yamanashi.jp +fuefuki.yamanashi.jp +fujikawa.yamanashi.jp +fujikawaguchiko.yamanashi.jp +fujiyoshida.yamanashi.jp +hayakawa.yamanashi.jp +hokuto.yamanashi.jp +ichikawamisato.yamanashi.jp +kai.yamanashi.jp +kofu.yamanashi.jp +koshu.yamanashi.jp +kosuge.yamanashi.jp +minami-alps.yamanashi.jp +minobu.yamanashi.jp +nakamichi.yamanashi.jp +nanbu.yamanashi.jp +narusawa.yamanashi.jp +nirasaki.yamanashi.jp +nishikatsura.yamanashi.jp +oshino.yamanashi.jp +otsuki.yamanashi.jp +showa.yamanashi.jp +tabayama.yamanashi.jp +tsuru.yamanashi.jp +uenohara.yamanashi.jp +yamanakako.yamanashi.jp +yamanashi.yamanashi.jp + +// ke : http://www.kenic.or.ke/index.php/en/ke-domains/ke-domains +ke +ac.ke +co.ke +go.ke +info.ke +me.ke +mobi.ke +ne.ke +or.ke +sc.ke + +// kg : http://www.domain.kg/dmn_n.html +kg +org.kg +net.kg +com.kg +edu.kg +gov.kg +mil.kg + +// kh : http://www.mptc.gov.kh/dns_registration.htm +*.kh + +// ki : http://www.ki/dns/index.html +ki +edu.ki +biz.ki +net.ki +org.ki +gov.ki +info.ki +com.ki + +// km : https://en.wikipedia.org/wiki/.km +// http://www.domaine.km/documents/charte.doc +km +org.km +nom.km +gov.km +prd.km +tm.km +edu.km +mil.km +ass.km +com.km +// These are only mentioned as proposed suggestions at domaine.km, but +// https://en.wikipedia.org/wiki/.km says they're available for registration: +coop.km +asso.km +presse.km +medecin.km +notaires.km +pharmaciens.km +veterinaire.km +gouv.km + +// kn : https://en.wikipedia.org/wiki/.kn +// http://www.dot.kn/domainRules.html +kn +net.kn +org.kn +edu.kn +gov.kn + +// kp : http://www.kcce.kp/en_index.php +kp +com.kp +edu.kp +gov.kp +org.kp +rep.kp +tra.kp + +// kr : https://en.wikipedia.org/wiki/.kr +// see also: http://domain.nida.or.kr/eng/registration.jsp +kr +ac.kr +co.kr +es.kr +go.kr +hs.kr +kg.kr +mil.kr +ms.kr +ne.kr +or.kr +pe.kr +re.kr +sc.kr +// kr geographical names +busan.kr +chungbuk.kr +chungnam.kr +daegu.kr +daejeon.kr +gangwon.kr +gwangju.kr +gyeongbuk.kr +gyeonggi.kr +gyeongnam.kr +incheon.kr +jeju.kr +jeonbuk.kr +jeonnam.kr +seoul.kr +ulsan.kr + +// kw : https://www.nic.kw/policies/ +// Confirmed by registry +kw +com.kw +edu.kw +emb.kw +gov.kw +ind.kw +net.kw +org.kw + +// ky : http://www.icta.ky/da_ky_reg_dom.php +// Confirmed by registry 2008-06-17 +ky +edu.ky +gov.ky +com.ky +org.ky +net.ky + +// kz : https://en.wikipedia.org/wiki/.kz +// see also: http://www.nic.kz/rules/index.jsp +kz +org.kz +edu.kz +net.kz +gov.kz +mil.kz +com.kz + +// la : https://en.wikipedia.org/wiki/.la +// Submitted by registry +la +int.la +net.la +info.la +edu.la +gov.la +per.la +com.la +org.la + +// lb : https://en.wikipedia.org/wiki/.lb +// Submitted by registry +lb +com.lb +edu.lb +gov.lb +net.lb +org.lb + +// lc : https://en.wikipedia.org/wiki/.lc +// see also: http://www.nic.lc/rules.htm +lc +com.lc +net.lc +co.lc +org.lc +edu.lc +gov.lc + +// li : https://en.wikipedia.org/wiki/.li +li + +// lk : https://www.nic.lk/index.php/domain-registration/lk-domain-naming-structure +lk +gov.lk +sch.lk +net.lk +int.lk +com.lk +org.lk +edu.lk +ngo.lk +soc.lk +web.lk +ltd.lk +assn.lk +grp.lk +hotel.lk +ac.lk + +// lr : http://psg.com/dns/lr/lr.txt +// Submitted by registry +lr +com.lr +edu.lr +gov.lr +org.lr +net.lr + +// ls : http://www.nic.ls/ +// Confirmed by registry +ls +ac.ls +biz.ls +co.ls +edu.ls +gov.ls +info.ls +net.ls +org.ls +sc.ls + +// lt : https://en.wikipedia.org/wiki/.lt +lt +// gov.lt : http://www.gov.lt/index_en.php +gov.lt + +// lu : http://www.dns.lu/en/ +lu + +// lv : http://www.nic.lv/DNS/En/generic.php +lv +com.lv +edu.lv +gov.lv +org.lv +mil.lv +id.lv +net.lv +asn.lv +conf.lv + +// ly : http://www.nic.ly/regulations.php +ly +com.ly +net.ly +gov.ly +plc.ly +edu.ly +sch.ly +med.ly +org.ly +id.ly + +// ma : https://en.wikipedia.org/wiki/.ma +// http://www.anrt.ma/fr/admin/download/upload/file_fr782.pdf +ma +co.ma +net.ma +gov.ma +org.ma +ac.ma +press.ma + +// mc : http://www.nic.mc/ +mc +tm.mc +asso.mc + +// md : https://en.wikipedia.org/wiki/.md +md + +// me : https://en.wikipedia.org/wiki/.me +me +co.me +net.me +org.me +edu.me +ac.me +gov.me +its.me +priv.me + +// mg : http://nic.mg/nicmg/?page_id=39 +mg +org.mg +nom.mg +gov.mg +prd.mg +tm.mg +edu.mg +mil.mg +com.mg +co.mg + +// mh : https://en.wikipedia.org/wiki/.mh +mh + +// mil : https://en.wikipedia.org/wiki/.mil +mil + +// mk : https://en.wikipedia.org/wiki/.mk +// see also: http://dns.marnet.net.mk/postapka.php +mk +com.mk +org.mk +net.mk +edu.mk +gov.mk +inf.mk +name.mk + +// ml : http://www.gobin.info/domainname/ml-template.doc +// see also: https://en.wikipedia.org/wiki/.ml +ml +com.ml +edu.ml +gouv.ml +gov.ml +net.ml +org.ml +presse.ml + +// mm : https://en.wikipedia.org/wiki/.mm +*.mm + +// mn : https://en.wikipedia.org/wiki/.mn +mn +gov.mn +edu.mn +org.mn + +// mo : http://www.monic.net.mo/ +mo +com.mo +net.mo +org.mo +edu.mo +gov.mo + +// mobi : https://en.wikipedia.org/wiki/.mobi +mobi + +// mp : http://www.dot.mp/ +// Confirmed by registry 2008-06-17 +mp + +// mq : https://en.wikipedia.org/wiki/.mq +mq + +// mr : https://en.wikipedia.org/wiki/.mr +mr +gov.mr + +// ms : http://www.nic.ms/pdf/MS_Domain_Name_Rules.pdf +ms +com.ms +edu.ms +gov.ms +net.ms +org.ms + +// mt : https://www.nic.org.mt/go/policy +// Submitted by registry +mt +com.mt +edu.mt +net.mt +org.mt + +// mu : https://en.wikipedia.org/wiki/.mu +mu +com.mu +net.mu +org.mu +gov.mu +ac.mu +co.mu +or.mu + +// museum : http://about.museum/naming/ +// http://index.museum/ +museum +academy.museum +agriculture.museum +air.museum +airguard.museum +alabama.museum +alaska.museum +amber.museum +ambulance.museum +american.museum +americana.museum +americanantiques.museum +americanart.museum +amsterdam.museum +and.museum +annefrank.museum +anthro.museum +anthropology.museum +antiques.museum +aquarium.museum +arboretum.museum +archaeological.museum +archaeology.museum +architecture.museum +art.museum +artanddesign.museum +artcenter.museum +artdeco.museum +arteducation.museum +artgallery.museum +arts.museum +artsandcrafts.museum +asmatart.museum +assassination.museum +assisi.museum +association.museum +astronomy.museum +atlanta.museum +austin.museum +australia.museum +automotive.museum +aviation.museum +axis.museum +badajoz.museum +baghdad.museum +bahn.museum +bale.museum +baltimore.museum +barcelona.museum +baseball.museum +basel.museum +baths.museum +bauern.museum +beauxarts.museum +beeldengeluid.museum +bellevue.museum +bergbau.museum +berkeley.museum +berlin.museum +bern.museum +bible.museum +bilbao.museum +bill.museum +birdart.museum +birthplace.museum +bonn.museum +boston.museum +botanical.museum +botanicalgarden.museum +botanicgarden.museum +botany.museum +brandywinevalley.museum +brasil.museum +bristol.museum +british.museum +britishcolumbia.museum +broadcast.museum +brunel.museum +brussel.museum +brussels.museum +bruxelles.museum +building.museum +burghof.museum +bus.museum +bushey.museum +cadaques.museum +california.museum +cambridge.museum +can.museum +canada.museum +capebreton.museum +carrier.museum +cartoonart.museum +casadelamoneda.museum +castle.museum +castres.museum +celtic.museum +center.museum +chattanooga.museum +cheltenham.museum +chesapeakebay.museum +chicago.museum +children.museum +childrens.museum +childrensgarden.museum +chiropractic.museum +chocolate.museum +christiansburg.museum +cincinnati.museum +cinema.museum +circus.museum +civilisation.museum +civilization.museum +civilwar.museum +clinton.museum +clock.museum +coal.museum +coastaldefence.museum +cody.museum +coldwar.museum +collection.museum +colonialwilliamsburg.museum +coloradoplateau.museum +columbia.museum +columbus.museum +communication.museum +communications.museum +community.museum +computer.museum +computerhistory.museum +comunicações.museum +contemporary.museum +contemporaryart.museum +convent.museum +copenhagen.museum +corporation.museum +correios-e-telecomunicações.museum +corvette.museum +costume.museum +countryestate.museum +county.museum +crafts.museum +cranbrook.museum +creation.museum +cultural.museum +culturalcenter.museum +culture.museum +cyber.museum +cymru.museum +dali.museum +dallas.museum +database.museum +ddr.museum +decorativearts.museum +delaware.museum +delmenhorst.museum +denmark.museum +depot.museum +design.museum +detroit.museum +dinosaur.museum +discovery.museum +dolls.museum +donostia.museum +durham.museum +eastafrica.museum +eastcoast.museum +education.museum +educational.museum +egyptian.museum +eisenbahn.museum +elburg.museum +elvendrell.museum +embroidery.museum +encyclopedic.museum +england.museum +entomology.museum +environment.museum +environmentalconservation.museum +epilepsy.museum +essex.museum +estate.museum +ethnology.museum +exeter.museum +exhibition.museum +family.museum +farm.museum +farmequipment.museum +farmers.museum +farmstead.museum +field.museum +figueres.museum +filatelia.museum +film.museum +fineart.museum +finearts.museum +finland.museum +flanders.museum +florida.museum +force.museum +fortmissoula.museum +fortworth.museum +foundation.museum +francaise.museum +frankfurt.museum +franziskaner.museum +freemasonry.museum +freiburg.museum +fribourg.museum +frog.museum +fundacio.museum +furniture.museum +gallery.museum +garden.museum +gateway.museum +geelvinck.museum +gemological.museum +geology.museum +georgia.museum +giessen.museum +glas.museum +glass.museum +gorge.museum +grandrapids.museum +graz.museum +guernsey.museum +halloffame.museum +hamburg.museum +handson.museum +harvestcelebration.museum +hawaii.museum +health.museum +heimatunduhren.museum +hellas.museum +helsinki.museum +hembygdsforbund.museum +heritage.museum +histoire.museum +historical.museum +historicalsociety.museum +historichouses.museum +historisch.museum +historisches.museum +history.museum +historyofscience.museum +horology.museum +house.museum +humanities.museum +illustration.museum +imageandsound.museum +indian.museum +indiana.museum +indianapolis.museum +indianmarket.museum +intelligence.museum +interactive.museum +iraq.museum +iron.museum +isleofman.museum +jamison.museum +jefferson.museum +jerusalem.museum +jewelry.museum +jewish.museum +jewishart.museum +jfk.museum +journalism.museum +judaica.museum +judygarland.museum +juedisches.museum +juif.museum +karate.museum +karikatur.museum +kids.museum +koebenhavn.museum +koeln.museum +kunst.museum +kunstsammlung.museum +kunstunddesign.museum +labor.museum +labour.museum +lajolla.museum +lancashire.museum +landes.museum +lans.museum +läns.museum +larsson.museum +lewismiller.museum +lincoln.museum +linz.museum +living.museum +livinghistory.museum +localhistory.museum +london.museum +losangeles.museum +louvre.museum +loyalist.museum +lucerne.museum +luxembourg.museum +luzern.museum +mad.museum +madrid.museum +mallorca.museum +manchester.museum +mansion.museum +mansions.museum +manx.museum +marburg.museum +maritime.museum +maritimo.museum +maryland.museum +marylhurst.museum +media.museum +medical.museum +medizinhistorisches.museum +meeres.museum +memorial.museum +mesaverde.museum +michigan.museum +midatlantic.museum +military.museum +mill.museum +miners.museum +mining.museum +minnesota.museum +missile.museum +missoula.museum +modern.museum +moma.museum +money.museum +monmouth.museum +monticello.museum +montreal.museum +moscow.museum +motorcycle.museum +muenchen.museum +muenster.museum +mulhouse.museum +muncie.museum +museet.museum +museumcenter.museum +museumvereniging.museum +music.museum +national.museum +nationalfirearms.museum +nationalheritage.museum +nativeamerican.museum +naturalhistory.museum +naturalhistorymuseum.museum +naturalsciences.museum +nature.museum +naturhistorisches.museum +natuurwetenschappen.museum +naumburg.museum +naval.museum +nebraska.museum +neues.museum +newhampshire.museum +newjersey.museum +newmexico.museum +newport.museum +newspaper.museum +newyork.museum +niepce.museum +norfolk.museum +north.museum +nrw.museum +nyc.museum +nyny.museum +oceanographic.museum +oceanographique.museum +omaha.museum +online.museum +ontario.museum +openair.museum +oregon.museum +oregontrail.museum +otago.museum +oxford.museum +pacific.museum +paderborn.museum +palace.museum +paleo.museum +palmsprings.museum +panama.museum +paris.museum +pasadena.museum +pharmacy.museum +philadelphia.museum +philadelphiaarea.museum +philately.museum +phoenix.museum +photography.museum +pilots.museum +pittsburgh.museum +planetarium.museum +plantation.museum +plants.museum +plaza.museum +portal.museum +portland.museum +portlligat.museum +posts-and-telecommunications.museum +preservation.museum +presidio.museum +press.museum +project.museum +public.museum +pubol.museum +quebec.museum +railroad.museum +railway.museum +research.museum +resistance.museum +riodejaneiro.museum +rochester.museum +rockart.museum +roma.museum +russia.museum +saintlouis.museum +salem.museum +salvadordali.museum +salzburg.museum +sandiego.museum +sanfrancisco.museum +santabarbara.museum +santacruz.museum +santafe.museum +saskatchewan.museum +satx.museum +savannahga.museum +schlesisches.museum +schoenbrunn.museum +schokoladen.museum +school.museum +schweiz.museum +science.museum +scienceandhistory.museum +scienceandindustry.museum +sciencecenter.museum +sciencecenters.museum +science-fiction.museum +sciencehistory.museum +sciences.museum +sciencesnaturelles.museum +scotland.museum +seaport.museum +settlement.museum +settlers.museum +shell.museum +sherbrooke.museum +sibenik.museum +silk.museum +ski.museum +skole.museum +society.museum +sologne.museum +soundandvision.museum +southcarolina.museum +southwest.museum +space.museum +spy.museum +square.museum +stadt.museum +stalbans.museum +starnberg.museum +state.museum +stateofdelaware.museum +station.museum +steam.museum +steiermark.museum +stjohn.museum +stockholm.museum +stpetersburg.museum +stuttgart.museum +suisse.museum +surgeonshall.museum +surrey.museum +svizzera.museum +sweden.museum +sydney.museum +tank.museum +tcm.museum +technology.museum +telekommunikation.museum +television.museum +texas.museum +textile.museum +theater.museum +time.museum +timekeeping.museum +topology.museum +torino.museum +touch.museum +town.museum +transport.museum +tree.museum +trolley.museum +trust.museum +trustee.museum +uhren.museum +ulm.museum +undersea.museum +university.museum +usa.museum +usantiques.museum +usarts.museum +uscountryestate.museum +usculture.museum +usdecorativearts.museum +usgarden.museum +ushistory.museum +ushuaia.museum +uslivinghistory.museum +utah.museum +uvic.museum +valley.museum +vantaa.museum +versailles.museum +viking.museum +village.museum +virginia.museum +virtual.museum +virtuel.museum +vlaanderen.museum +volkenkunde.museum +wales.museum +wallonie.museum +war.museum +washingtondc.museum +watchandclock.museum +watch-and-clock.museum +western.museum +westfalen.museum +whaling.museum +wildlife.museum +williamsburg.museum +windmill.museum +workshop.museum +york.museum +yorkshire.museum +yosemite.museum +youth.museum +zoological.museum +zoology.museum +ירושלים.museum +иком.museum + +// mv : https://en.wikipedia.org/wiki/.mv +// "mv" included because, contra Wikipedia, google.mv exists. +mv +aero.mv +biz.mv +com.mv +coop.mv +edu.mv +gov.mv +info.mv +int.mv +mil.mv +museum.mv +name.mv +net.mv +org.mv +pro.mv + +// mw : http://www.registrar.mw/ +mw +ac.mw +biz.mw +co.mw +com.mw +coop.mw +edu.mw +gov.mw +int.mw +museum.mw +net.mw +org.mw + +// mx : http://www.nic.mx/ +// Submitted by registry +mx +com.mx +org.mx +gob.mx +edu.mx +net.mx + +// my : http://www.mynic.net.my/ +my +com.my +net.my +org.my +gov.my +edu.my +mil.my +name.my + +// mz : http://www.uem.mz/ +// Submitted by registry +mz +ac.mz +adv.mz +co.mz +edu.mz +gov.mz +mil.mz +net.mz +org.mz + +// na : http://www.na-nic.com.na/ +// http://www.info.na/domain/ +na +info.na +pro.na +name.na +school.na +or.na +dr.na +us.na +mx.na +ca.na +in.na +cc.na +tv.na +ws.na +mobi.na +co.na +com.na +org.na + +// name : has 2nd-level tlds, but there's no list of them +name + +// nc : http://www.cctld.nc/ +nc +asso.nc +nom.nc + +// ne : https://en.wikipedia.org/wiki/.ne +ne + +// net : https://en.wikipedia.org/wiki/.net +net + +// nf : https://en.wikipedia.org/wiki/.nf +nf +com.nf +net.nf +per.nf +rec.nf +web.nf +arts.nf +firm.nf +info.nf +other.nf +store.nf + +// ng : http://www.nira.org.ng/index.php/join-us/register-ng-domain/189-nira-slds +ng +com.ng +edu.ng +gov.ng +i.ng +mil.ng +mobi.ng +name.ng +net.ng +org.ng +sch.ng + +// ni : http://www.nic.ni/ +ni +ac.ni +biz.ni +co.ni +com.ni +edu.ni +gob.ni +in.ni +info.ni +int.ni +mil.ni +net.ni +nom.ni +org.ni +web.ni + +// nl : https://en.wikipedia.org/wiki/.nl +// https://www.sidn.nl/ +// ccTLD for the Netherlands +nl + +// no : https://www.norid.no/en/om-domenenavn/regelverk-for-no/ +// Norid geographical second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-b/ +// Norid category second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-c/ +// Norid category second-level domains managed by parties other than Norid : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-d/ +// RSS feed: https://teknisk.norid.no/en/feed/ +no +// Norid category second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-c/ +fhs.no +vgs.no +fylkesbibl.no +folkebibl.no +museum.no +idrett.no +priv.no +// Norid category second-level domains managed by parties other than Norid : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-d/ +mil.no +stat.no +dep.no +kommune.no +herad.no +// Norid geographical second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-b/ +// counties +aa.no +ah.no +bu.no +fm.no +hl.no +hm.no +jan-mayen.no +mr.no +nl.no +nt.no +of.no +ol.no +oslo.no +rl.no +sf.no +st.no +svalbard.no +tm.no +tr.no +va.no +vf.no +// primary and lower secondary schools per county +gs.aa.no +gs.ah.no +gs.bu.no +gs.fm.no +gs.hl.no +gs.hm.no +gs.jan-mayen.no +gs.mr.no +gs.nl.no +gs.nt.no +gs.of.no +gs.ol.no +gs.oslo.no +gs.rl.no +gs.sf.no +gs.st.no +gs.svalbard.no +gs.tm.no +gs.tr.no +gs.va.no +gs.vf.no +// cities +akrehamn.no +åkrehamn.no +algard.no +ålgård.no +arna.no +brumunddal.no +bryne.no +bronnoysund.no +brønnøysund.no +drobak.no +drøbak.no +egersund.no +fetsund.no +floro.no +florø.no +fredrikstad.no +hokksund.no +honefoss.no +hønefoss.no +jessheim.no +jorpeland.no +jørpeland.no +kirkenes.no +kopervik.no +krokstadelva.no +langevag.no +langevåg.no +leirvik.no +mjondalen.no +mjøndalen.no +mo-i-rana.no +mosjoen.no +mosjøen.no +nesoddtangen.no +orkanger.no +osoyro.no +osøyro.no +raholt.no +råholt.no +sandnessjoen.no +sandnessjøen.no +skedsmokorset.no +slattum.no +spjelkavik.no +stathelle.no +stavern.no +stjordalshalsen.no +stjørdalshalsen.no +tananger.no +tranby.no +vossevangen.no +// communities +afjord.no +åfjord.no +agdenes.no +al.no +ål.no +alesund.no +ålesund.no +alstahaug.no +alta.no +áltá.no +alaheadju.no +álaheadju.no +alvdal.no +amli.no +åmli.no +amot.no +åmot.no +andebu.no +andoy.no +andøy.no +andasuolo.no +ardal.no +årdal.no +aremark.no +arendal.no +ås.no +aseral.no +åseral.no +asker.no +askim.no +askvoll.no +askoy.no +askøy.no +asnes.no +åsnes.no +audnedaln.no +aukra.no +aure.no +aurland.no +aurskog-holand.no +aurskog-høland.no +austevoll.no +austrheim.no +averoy.no +averøy.no +balestrand.no +ballangen.no +balat.no +bálát.no +balsfjord.no +bahccavuotna.no +báhccavuotna.no +bamble.no +bardu.no +beardu.no +beiarn.no +bajddar.no +bájddar.no +baidar.no +báidár.no +berg.no +bergen.no +berlevag.no +berlevåg.no +bearalvahki.no +bearalváhki.no +bindal.no +birkenes.no +bjarkoy.no +bjarkøy.no +bjerkreim.no +bjugn.no +bodo.no +bodø.no +badaddja.no +bådåddjå.no +budejju.no +bokn.no +bremanger.no +bronnoy.no +brønnøy.no +bygland.no +bykle.no +barum.no +bærum.no +bo.telemark.no +bø.telemark.no +bo.nordland.no +bø.nordland.no +bievat.no +bievát.no +bomlo.no +bømlo.no +batsfjord.no +båtsfjord.no +bahcavuotna.no +báhcavuotna.no +dovre.no +drammen.no +drangedal.no +dyroy.no +dyrøy.no +donna.no +dønna.no +eid.no +eidfjord.no +eidsberg.no +eidskog.no +eidsvoll.no +eigersund.no +elverum.no +enebakk.no +engerdal.no +etne.no +etnedal.no +evenes.no +evenassi.no +evenášši.no +evje-og-hornnes.no +farsund.no +fauske.no +fuossko.no +fuoisku.no +fedje.no +fet.no +finnoy.no +finnøy.no +fitjar.no +fjaler.no +fjell.no +flakstad.no +flatanger.no +flekkefjord.no +flesberg.no +flora.no +fla.no +flå.no +folldal.no +forsand.no +fosnes.no +frei.no +frogn.no +froland.no +frosta.no +frana.no +fræna.no +froya.no +frøya.no +fusa.no +fyresdal.no +forde.no +førde.no +gamvik.no +gangaviika.no +gáŋgaviika.no +gaular.no +gausdal.no +gildeskal.no +gildeskål.no +giske.no +gjemnes.no +gjerdrum.no +gjerstad.no +gjesdal.no +gjovik.no +gjøvik.no +gloppen.no +gol.no +gran.no +grane.no +granvin.no +gratangen.no +grimstad.no +grong.no +kraanghke.no +kråanghke.no +grue.no +gulen.no +hadsel.no +halden.no +halsa.no +hamar.no +hamaroy.no +habmer.no +hábmer.no +hapmir.no +hápmir.no +hammerfest.no +hammarfeasta.no +hámmárfeasta.no +haram.no +hareid.no +harstad.no +hasvik.no +aknoluokta.no +ákŋoluokta.no +hattfjelldal.no +aarborte.no +haugesund.no +hemne.no +hemnes.no +hemsedal.no +heroy.more-og-romsdal.no +herøy.møre-og-romsdal.no +heroy.nordland.no +herøy.nordland.no +hitra.no +hjartdal.no +hjelmeland.no +hobol.no +hobøl.no +hof.no +hol.no +hole.no +holmestrand.no +holtalen.no +holtålen.no +hornindal.no +horten.no +hurdal.no +hurum.no +hvaler.no +hyllestad.no +hagebostad.no +hægebostad.no +hoyanger.no +høyanger.no +hoylandet.no +høylandet.no +ha.no +hå.no +ibestad.no +inderoy.no +inderøy.no +iveland.no +jevnaker.no +jondal.no +jolster.no +jølster.no +karasjok.no +karasjohka.no +kárášjohka.no +karlsoy.no +galsa.no +gálsá.no +karmoy.no +karmøy.no +kautokeino.no +guovdageaidnu.no +klepp.no +klabu.no +klæbu.no +kongsberg.no +kongsvinger.no +kragero.no +kragerø.no +kristiansand.no +kristiansund.no +krodsherad.no +krødsherad.no +kvalsund.no +rahkkeravju.no +ráhkkerávju.no +kvam.no +kvinesdal.no +kvinnherad.no +kviteseid.no +kvitsoy.no +kvitsøy.no +kvafjord.no +kvæfjord.no +giehtavuoatna.no +kvanangen.no +kvænangen.no +navuotna.no +návuotna.no +kafjord.no +kåfjord.no +gaivuotna.no +gáivuotna.no +larvik.no +lavangen.no +lavagis.no +loabat.no +loabát.no +lebesby.no +davvesiida.no +leikanger.no +leirfjord.no +leka.no +leksvik.no +lenvik.no +leangaviika.no +leaŋgaviika.no +lesja.no +levanger.no +lier.no +lierne.no +lillehammer.no +lillesand.no +lindesnes.no +lindas.no +lindås.no +lom.no +loppa.no +lahppi.no +láhppi.no +lund.no +lunner.no +luroy.no +lurøy.no +luster.no +lyngdal.no +lyngen.no +ivgu.no +lardal.no +lerdal.no +lærdal.no +lodingen.no +lødingen.no +lorenskog.no +lørenskog.no +loten.no +løten.no +malvik.no +masoy.no +måsøy.no +muosat.no +muosát.no +mandal.no +marker.no +marnardal.no +masfjorden.no +meland.no +meldal.no +melhus.no +meloy.no +meløy.no +meraker.no +meråker.no +moareke.no +moåreke.no +midsund.no +midtre-gauldal.no +modalen.no +modum.no +molde.no +moskenes.no +moss.no +mosvik.no +malselv.no +målselv.no +malatvuopmi.no +málatvuopmi.no +namdalseid.no +aejrie.no +namsos.no +namsskogan.no +naamesjevuemie.no +nååmesjevuemie.no +laakesvuemie.no +nannestad.no +narvik.no +narviika.no +naustdal.no +nedre-eiker.no +nes.akershus.no +nes.buskerud.no +nesna.no +nesodden.no +nesseby.no +unjarga.no +unjárga.no +nesset.no +nissedal.no +nittedal.no +nord-aurdal.no +nord-fron.no +nord-odal.no +norddal.no +nordkapp.no +davvenjarga.no +davvenjárga.no +nordre-land.no +nordreisa.no +raisa.no +ráisa.no +nore-og-uvdal.no +notodden.no +naroy.no +nærøy.no +notteroy.no +nøtterøy.no +odda.no +oksnes.no +øksnes.no +oppdal.no +oppegard.no +oppegård.no +orkdal.no +orland.no +ørland.no +orskog.no +ørskog.no +orsta.no +ørsta.no +os.hedmark.no +os.hordaland.no +osen.no +osteroy.no +osterøy.no +ostre-toten.no +østre-toten.no +overhalla.no +ovre-eiker.no +øvre-eiker.no +oyer.no +øyer.no +oygarden.no +øygarden.no +oystre-slidre.no +øystre-slidre.no +porsanger.no +porsangu.no +porsáŋgu.no +porsgrunn.no +radoy.no +radøy.no +rakkestad.no +rana.no +ruovat.no +randaberg.no +rauma.no +rendalen.no +rennebu.no +rennesoy.no +rennesøy.no +rindal.no +ringebu.no +ringerike.no +ringsaker.no +rissa.no +risor.no +risør.no +roan.no +rollag.no +rygge.no +ralingen.no +rælingen.no +rodoy.no +rødøy.no +romskog.no +rømskog.no +roros.no +røros.no +rost.no +røst.no +royken.no +røyken.no +royrvik.no +røyrvik.no +rade.no +råde.no +salangen.no +siellak.no +saltdal.no +salat.no +sálát.no +sálat.no +samnanger.no +sande.more-og-romsdal.no +sande.møre-og-romsdal.no +sande.vestfold.no +sandefjord.no +sandnes.no +sandoy.no +sandøy.no +sarpsborg.no +sauda.no +sauherad.no +sel.no +selbu.no +selje.no +seljord.no +sigdal.no +siljan.no +sirdal.no +skaun.no +skedsmo.no +ski.no +skien.no +skiptvet.no +skjervoy.no +skjervøy.no +skierva.no +skiervá.no +skjak.no +skjåk.no +skodje.no +skanland.no +skånland.no +skanit.no +skánit.no +smola.no +smøla.no +snillfjord.no +snasa.no +snåsa.no +snoasa.no +snaase.no +snåase.no +sogndal.no +sokndal.no +sola.no +solund.no +songdalen.no +sortland.no +spydeberg.no +stange.no +stavanger.no +steigen.no +steinkjer.no +stjordal.no +stjørdal.no +stokke.no +stor-elvdal.no +stord.no +stordal.no +storfjord.no +omasvuotna.no +strand.no +stranda.no +stryn.no +sula.no +suldal.no +sund.no +sunndal.no +surnadal.no +sveio.no +svelvik.no +sykkylven.no +sogne.no +søgne.no +somna.no +sømna.no +sondre-land.no +søndre-land.no +sor-aurdal.no +sør-aurdal.no +sor-fron.no +sør-fron.no +sor-odal.no +sør-odal.no +sor-varanger.no +sør-varanger.no +matta-varjjat.no +mátta-várjjat.no +sorfold.no +sørfold.no +sorreisa.no +sørreisa.no +sorum.no +sørum.no +tana.no +deatnu.no +time.no +tingvoll.no +tinn.no +tjeldsund.no +dielddanuorri.no +tjome.no +tjøme.no +tokke.no +tolga.no +torsken.no +tranoy.no +tranøy.no +tromso.no +tromsø.no +tromsa.no +romsa.no +trondheim.no +troandin.no +trysil.no +trana.no +træna.no +trogstad.no +trøgstad.no +tvedestrand.no +tydal.no +tynset.no +tysfjord.no +divtasvuodna.no +divttasvuotna.no +tysnes.no +tysvar.no +tysvær.no +tonsberg.no +tønsberg.no +ullensaker.no +ullensvang.no +ulvik.no +utsira.no +vadso.no +vadsø.no +cahcesuolo.no +čáhcesuolo.no +vaksdal.no +valle.no +vang.no +vanylven.no +vardo.no +vardø.no +varggat.no +várggát.no +vefsn.no +vaapste.no +vega.no +vegarshei.no +vegårshei.no +vennesla.no +verdal.no +verran.no +vestby.no +vestnes.no +vestre-slidre.no +vestre-toten.no +vestvagoy.no +vestvågøy.no +vevelstad.no +vik.no +vikna.no +vindafjord.no +volda.no +voss.no +varoy.no +værøy.no +vagan.no +vågan.no +voagat.no +vagsoy.no +vågsøy.no +vaga.no +vågå.no +valer.ostfold.no +våler.østfold.no +valer.hedmark.no +våler.hedmark.no + +// np : http://www.mos.com.np/register.html +*.np + +// nr : http://cenpac.net.nr/dns/index.html +// Submitted by registry +nr +biz.nr +info.nr +gov.nr +edu.nr +org.nr +net.nr +com.nr + +// nu : https://en.wikipedia.org/wiki/.nu +nu + +// nz : https://en.wikipedia.org/wiki/.nz +// Submitted by registry +nz +ac.nz +co.nz +cri.nz +geek.nz +gen.nz +govt.nz +health.nz +iwi.nz +kiwi.nz +maori.nz +mil.nz +māori.nz +net.nz +org.nz +parliament.nz +school.nz + +// om : https://en.wikipedia.org/wiki/.om +om +co.om +com.om +edu.om +gov.om +med.om +museum.om +net.om +org.om +pro.om + +// onion : https://tools.ietf.org/html/rfc7686 +onion + +// org : https://en.wikipedia.org/wiki/.org +org + +// pa : http://www.nic.pa/ +// Some additional second level "domains" resolve directly as hostnames, such as +// pannet.pa, so we add a rule for "pa". +pa +ac.pa +gob.pa +com.pa +org.pa +sld.pa +edu.pa +net.pa +ing.pa +abo.pa +med.pa +nom.pa + +// pe : https://www.nic.pe/InformeFinalComision.pdf +pe +edu.pe +gob.pe +nom.pe +mil.pe +org.pe +com.pe +net.pe + +// pf : http://www.gobin.info/domainname/formulaire-pf.pdf +pf +com.pf +org.pf +edu.pf + +// pg : https://en.wikipedia.org/wiki/.pg +*.pg + +// ph : http://www.domains.ph/FAQ2.asp +// Submitted by registry +ph +com.ph +net.ph +org.ph +gov.ph +edu.ph +ngo.ph +mil.ph +i.ph + +// pk : http://pk5.pknic.net.pk/pk5/msgNamepk.PK +pk +com.pk +net.pk +edu.pk +org.pk +fam.pk +biz.pk +web.pk +gov.pk +gob.pk +gok.pk +gon.pk +gop.pk +gos.pk +info.pk + +// pl http://www.dns.pl/english/index.html +// Submitted by registry +pl +com.pl +net.pl +org.pl +// pl functional domains (http://www.dns.pl/english/index.html) +aid.pl +agro.pl +atm.pl +auto.pl +biz.pl +edu.pl +gmina.pl +gsm.pl +info.pl +mail.pl +miasta.pl +media.pl +mil.pl +nieruchomosci.pl +nom.pl +pc.pl +powiat.pl +priv.pl +realestate.pl +rel.pl +sex.pl +shop.pl +sklep.pl +sos.pl +szkola.pl +targi.pl +tm.pl +tourism.pl +travel.pl +turystyka.pl +// Government domains +gov.pl +ap.gov.pl +ic.gov.pl +is.gov.pl +us.gov.pl +kmpsp.gov.pl +kppsp.gov.pl +kwpsp.gov.pl +psp.gov.pl +wskr.gov.pl +kwp.gov.pl +mw.gov.pl +ug.gov.pl +um.gov.pl +umig.gov.pl +ugim.gov.pl +upow.gov.pl +uw.gov.pl +starostwo.gov.pl +pa.gov.pl +po.gov.pl +psse.gov.pl +pup.gov.pl +rzgw.gov.pl +sa.gov.pl +so.gov.pl +sr.gov.pl +wsa.gov.pl +sko.gov.pl +uzs.gov.pl +wiih.gov.pl +winb.gov.pl +pinb.gov.pl +wios.gov.pl +witd.gov.pl +wzmiuw.gov.pl +piw.gov.pl +wiw.gov.pl +griw.gov.pl +wif.gov.pl +oum.gov.pl +sdn.gov.pl +zp.gov.pl +uppo.gov.pl +mup.gov.pl +wuoz.gov.pl +konsulat.gov.pl +oirm.gov.pl +// pl regional domains (http://www.dns.pl/english/index.html) +augustow.pl +babia-gora.pl +bedzin.pl +beskidy.pl +bialowieza.pl +bialystok.pl +bielawa.pl +bieszczady.pl +boleslawiec.pl +bydgoszcz.pl +bytom.pl +cieszyn.pl +czeladz.pl +czest.pl +dlugoleka.pl +elblag.pl +elk.pl +glogow.pl +gniezno.pl +gorlice.pl +grajewo.pl +ilawa.pl +jaworzno.pl +jelenia-gora.pl +jgora.pl +kalisz.pl +kazimierz-dolny.pl +karpacz.pl +kartuzy.pl +kaszuby.pl +katowice.pl +kepno.pl +ketrzyn.pl +klodzko.pl +kobierzyce.pl +kolobrzeg.pl +konin.pl +konskowola.pl +kutno.pl +lapy.pl +lebork.pl +legnica.pl +lezajsk.pl +limanowa.pl +lomza.pl +lowicz.pl +lubin.pl +lukow.pl +malbork.pl +malopolska.pl +mazowsze.pl +mazury.pl +mielec.pl +mielno.pl +mragowo.pl +naklo.pl +nowaruda.pl +nysa.pl +olawa.pl +olecko.pl +olkusz.pl +olsztyn.pl +opoczno.pl +opole.pl +ostroda.pl +ostroleka.pl +ostrowiec.pl +ostrowwlkp.pl +pila.pl +pisz.pl +podhale.pl +podlasie.pl +polkowice.pl +pomorze.pl +pomorskie.pl +prochowice.pl +pruszkow.pl +przeworsk.pl +pulawy.pl +radom.pl +rawa-maz.pl +rybnik.pl +rzeszow.pl +sanok.pl +sejny.pl +slask.pl +slupsk.pl +sosnowiec.pl +stalowa-wola.pl +skoczow.pl +starachowice.pl +stargard.pl +suwalki.pl +swidnica.pl +swiebodzin.pl +swinoujscie.pl +szczecin.pl +szczytno.pl +tarnobrzeg.pl +tgory.pl +turek.pl +tychy.pl +ustka.pl +walbrzych.pl +warmia.pl +warszawa.pl +waw.pl +wegrow.pl +wielun.pl +wlocl.pl +wloclawek.pl +wodzislaw.pl +wolomin.pl +wroclaw.pl +zachpomor.pl +zagan.pl +zarow.pl +zgora.pl +zgorzelec.pl + +// pm : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +pm + +// pn : http://www.government.pn/PnRegistry/policies.htm +pn +gov.pn +co.pn +org.pn +edu.pn +net.pn + +// post : https://en.wikipedia.org/wiki/.post +post + +// pr : http://www.nic.pr/index.asp?f=1 +pr +com.pr +net.pr +org.pr +gov.pr +edu.pr +isla.pr +pro.pr +biz.pr +info.pr +name.pr +// these aren't mentioned on nic.pr, but on https://en.wikipedia.org/wiki/.pr +est.pr +prof.pr +ac.pr + +// pro : http://registry.pro/get-pro +pro +aaa.pro +aca.pro +acct.pro +avocat.pro +bar.pro +cpa.pro +eng.pro +jur.pro +law.pro +med.pro +recht.pro + +// ps : https://en.wikipedia.org/wiki/.ps +// http://www.nic.ps/registration/policy.html#reg +ps +edu.ps +gov.ps +sec.ps +plo.ps +com.ps +org.ps +net.ps + +// pt : http://online.dns.pt/dns/start_dns +pt +net.pt +gov.pt +org.pt +edu.pt +int.pt +publ.pt +com.pt +nome.pt + +// pw : https://en.wikipedia.org/wiki/.pw +pw +co.pw +ne.pw +or.pw +ed.pw +go.pw +belau.pw + +// py : http://www.nic.py/pautas.html#seccion_9 +// Submitted by registry +py +com.py +coop.py +edu.py +gov.py +mil.py +net.py +org.py + +// qa : http://domains.qa/en/ +qa +com.qa +edu.qa +gov.qa +mil.qa +name.qa +net.qa +org.qa +sch.qa + +// re : http://www.afnic.re/obtenir/chartes/nommage-re/annexe-descriptifs +re +asso.re +com.re +nom.re + +// ro : http://www.rotld.ro/ +ro +arts.ro +com.ro +firm.ro +info.ro +nom.ro +nt.ro +org.ro +rec.ro +store.ro +tm.ro +www.ro + +// rs : https://www.rnids.rs/en/domains/national-domains +rs +ac.rs +co.rs +edu.rs +gov.rs +in.rs +org.rs + +// ru : https://cctld.ru/files/pdf/docs/en/rules_ru-rf.pdf +// Submitted by George Georgievsky +ru + +// rw : https://www.ricta.org.rw/sites/default/files/resources/registry_registrar_contract_0.pdf +rw +ac.rw +co.rw +coop.rw +gov.rw +mil.rw +net.rw +org.rw + +// sa : http://www.nic.net.sa/ +sa +com.sa +net.sa +org.sa +gov.sa +med.sa +pub.sa +edu.sa +sch.sa + +// sb : http://www.sbnic.net.sb/ +// Submitted by registry +sb +com.sb +edu.sb +gov.sb +net.sb +org.sb + +// sc : http://www.nic.sc/ +sc +com.sc +gov.sc +net.sc +org.sc +edu.sc + +// sd : http://www.isoc.sd/sudanic.isoc.sd/billing_pricing.htm +// Submitted by registry +sd +com.sd +net.sd +org.sd +edu.sd +med.sd +tv.sd +gov.sd +info.sd + +// se : https://en.wikipedia.org/wiki/.se +// Submitted by registry +se +a.se +ac.se +b.se +bd.se +brand.se +c.se +d.se +e.se +f.se +fh.se +fhsk.se +fhv.se +g.se +h.se +i.se +k.se +komforb.se +kommunalforbund.se +komvux.se +l.se +lanbib.se +m.se +n.se +naturbruksgymn.se +o.se +org.se +p.se +parti.se +pp.se +press.se +r.se +s.se +t.se +tm.se +u.se +w.se +x.se +y.se +z.se + +// sg : http://www.nic.net.sg/page/registration-policies-procedures-and-guidelines +sg +com.sg +net.sg +org.sg +gov.sg +edu.sg +per.sg + +// sh : http://www.nic.sh/registrar.html +sh +com.sh +net.sh +gov.sh +org.sh +mil.sh + +// si : https://en.wikipedia.org/wiki/.si +si + +// sj : No registrations at this time. +// Submitted by registry +sj + +// sk : https://en.wikipedia.org/wiki/.sk +// list of 2nd level domains ? +sk + +// sl : http://www.nic.sl +// Submitted by registry +sl +com.sl +net.sl +edu.sl +gov.sl +org.sl + +// sm : https://en.wikipedia.org/wiki/.sm +sm + +// sn : https://en.wikipedia.org/wiki/.sn +sn +art.sn +com.sn +edu.sn +gouv.sn +org.sn +perso.sn +univ.sn + +// so : http://sonic.so/policies/ +so +com.so +edu.so +gov.so +me.so +net.so +org.so + +// sr : https://en.wikipedia.org/wiki/.sr +sr + +// ss : https://registry.nic.ss/ +// Submitted by registry +ss +biz.ss +com.ss +edu.ss +gov.ss +net.ss +org.ss + +// st : http://www.nic.st/html/policyrules/ +st +co.st +com.st +consulado.st +edu.st +embaixada.st +gov.st +mil.st +net.st +org.st +principe.st +saotome.st +store.st + +// su : https://en.wikipedia.org/wiki/.su +su + +// sv : http://www.svnet.org.sv/niveldos.pdf +sv +com.sv +edu.sv +gob.sv +org.sv +red.sv + +// sx : https://en.wikipedia.org/wiki/.sx +// Submitted by registry +sx +gov.sx + +// sy : https://en.wikipedia.org/wiki/.sy +// see also: http://www.gobin.info/domainname/sy.doc +sy +edu.sy +gov.sy +net.sy +mil.sy +com.sy +org.sy + +// sz : https://en.wikipedia.org/wiki/.sz +// http://www.sispa.org.sz/ +sz +co.sz +ac.sz +org.sz + +// tc : https://en.wikipedia.org/wiki/.tc +tc + +// td : https://en.wikipedia.org/wiki/.td +td + +// tel: https://en.wikipedia.org/wiki/.tel +// http://www.telnic.org/ +tel + +// tf : https://en.wikipedia.org/wiki/.tf +tf + +// tg : https://en.wikipedia.org/wiki/.tg +// http://www.nic.tg/ +tg + +// th : https://en.wikipedia.org/wiki/.th +// Submitted by registry +th +ac.th +co.th +go.th +in.th +mi.th +net.th +or.th + +// tj : http://www.nic.tj/policy.html +tj +ac.tj +biz.tj +co.tj +com.tj +edu.tj +go.tj +gov.tj +int.tj +mil.tj +name.tj +net.tj +nic.tj +org.tj +test.tj +web.tj + +// tk : https://en.wikipedia.org/wiki/.tk +tk + +// tl : https://en.wikipedia.org/wiki/.tl +tl +gov.tl + +// tm : http://www.nic.tm/local.html +tm +com.tm +co.tm +org.tm +net.tm +nom.tm +gov.tm +mil.tm +edu.tm + +// tn : https://en.wikipedia.org/wiki/.tn +// http://whois.ati.tn/ +tn +com.tn +ens.tn +fin.tn +gov.tn +ind.tn +intl.tn +nat.tn +net.tn +org.tn +info.tn +perso.tn +tourism.tn +edunet.tn +rnrt.tn +rns.tn +rnu.tn +mincom.tn +agrinet.tn +defense.tn +turen.tn + +// to : https://en.wikipedia.org/wiki/.to +// Submitted by registry +to +com.to +gov.to +net.to +org.to +edu.to +mil.to + +// tr : https://nic.tr/ +// https://nic.tr/forms/eng/policies.pdf +// https://nic.tr/index.php?USRACTN=PRICELST +tr +av.tr +bbs.tr +bel.tr +biz.tr +com.tr +dr.tr +edu.tr +gen.tr +gov.tr +info.tr +mil.tr +k12.tr +kep.tr +name.tr +net.tr +org.tr +pol.tr +tel.tr +tsk.tr +tv.tr +web.tr +// Used by Northern Cyprus +nc.tr +// Used by government agencies of Northern Cyprus +gov.nc.tr + +// tt : http://www.nic.tt/ +tt +co.tt +com.tt +org.tt +net.tt +biz.tt +info.tt +pro.tt +int.tt +coop.tt +jobs.tt +mobi.tt +travel.tt +museum.tt +aero.tt +name.tt +gov.tt +edu.tt + +// tv : https://en.wikipedia.org/wiki/.tv +// Not listing any 2LDs as reserved since none seem to exist in practice, +// Wikipedia notwithstanding. +tv + +// tw : https://en.wikipedia.org/wiki/.tw +tw +edu.tw +gov.tw +mil.tw +com.tw +net.tw +org.tw +idv.tw +game.tw +ebiz.tw +club.tw +網路.tw +組織.tw +商業.tw + +// tz : http://www.tznic.or.tz/index.php/domains +// Submitted by registry +tz +ac.tz +co.tz +go.tz +hotel.tz +info.tz +me.tz +mil.tz +mobi.tz +ne.tz +or.tz +sc.tz +tv.tz + +// ua : https://hostmaster.ua/policy/?ua +// Submitted by registry +ua +// ua 2LD +com.ua +edu.ua +gov.ua +in.ua +net.ua +org.ua +// ua geographic names +// https://hostmaster.ua/2ld/ +cherkassy.ua +cherkasy.ua +chernigov.ua +chernihiv.ua +chernivtsi.ua +chernovtsy.ua +ck.ua +cn.ua +cr.ua +crimea.ua +cv.ua +dn.ua +dnepropetrovsk.ua +dnipropetrovsk.ua +donetsk.ua +dp.ua +if.ua +ivano-frankivsk.ua +kh.ua +kharkiv.ua +kharkov.ua +kherson.ua +khmelnitskiy.ua +khmelnytskyi.ua +kiev.ua +kirovograd.ua +km.ua +kr.ua +krym.ua +ks.ua +kv.ua +kyiv.ua +lg.ua +lt.ua +lugansk.ua +lutsk.ua +lv.ua +lviv.ua +mk.ua +mykolaiv.ua +nikolaev.ua +od.ua +odesa.ua +odessa.ua +pl.ua +poltava.ua +rivne.ua +rovno.ua +rv.ua +sb.ua +sebastopol.ua +sevastopol.ua +sm.ua +sumy.ua +te.ua +ternopil.ua +uz.ua +uzhgorod.ua +vinnica.ua +vinnytsia.ua +vn.ua +volyn.ua +yalta.ua +zaporizhzhe.ua +zaporizhzhia.ua +zhitomir.ua +zhytomyr.ua +zp.ua +zt.ua + +// ug : https://www.registry.co.ug/ +ug +co.ug +or.ug +ac.ug +sc.ug +go.ug +ne.ug +com.ug +org.ug + +// uk : https://en.wikipedia.org/wiki/.uk +// Submitted by registry +uk +ac.uk +co.uk +gov.uk +ltd.uk +me.uk +net.uk +nhs.uk +org.uk +plc.uk +police.uk +*.sch.uk + +// us : https://en.wikipedia.org/wiki/.us +us +dni.us +fed.us +isa.us +kids.us +nsn.us +// us geographic names +ak.us +al.us +ar.us +as.us +az.us +ca.us +co.us +ct.us +dc.us +de.us +fl.us +ga.us +gu.us +hi.us +ia.us +id.us +il.us +in.us +ks.us +ky.us +la.us +ma.us +md.us +me.us +mi.us +mn.us +mo.us +ms.us +mt.us +nc.us +nd.us +ne.us +nh.us +nj.us +nm.us +nv.us +ny.us +oh.us +ok.us +or.us +pa.us +pr.us +ri.us +sc.us +sd.us +tn.us +tx.us +ut.us +vi.us +vt.us +va.us +wa.us +wi.us +wv.us +wy.us +// The registrar notes several more specific domains available in each state, +// such as state.*.us, dst.*.us, etc., but resolution of these is somewhat +// haphazard; in some states these domains resolve as addresses, while in others +// only subdomains are available, or even nothing at all. We include the +// most common ones where it's clear that different sites are different +// entities. +k12.ak.us +k12.al.us +k12.ar.us +k12.as.us +k12.az.us +k12.ca.us +k12.co.us +k12.ct.us +k12.dc.us +k12.de.us +k12.fl.us +k12.ga.us +k12.gu.us +// k12.hi.us Bug 614565 - Hawaii has a state-wide DOE login +k12.ia.us +k12.id.us +k12.il.us +k12.in.us +k12.ks.us +k12.ky.us +k12.la.us +k12.ma.us +k12.md.us +k12.me.us +k12.mi.us +k12.mn.us +k12.mo.us +k12.ms.us +k12.mt.us +k12.nc.us +// k12.nd.us Bug 1028347 - Removed at request of Travis Rosso +k12.ne.us +k12.nh.us +k12.nj.us +k12.nm.us +k12.nv.us +k12.ny.us +k12.oh.us +k12.ok.us +k12.or.us +k12.pa.us +k12.pr.us +// k12.ri.us Removed at request of Kim Cournoyer +k12.sc.us +// k12.sd.us Bug 934131 - Removed at request of James Booze +k12.tn.us +k12.tx.us +k12.ut.us +k12.vi.us +k12.vt.us +k12.va.us +k12.wa.us +k12.wi.us +// k12.wv.us Bug 947705 - Removed at request of Verne Britton +k12.wy.us +cc.ak.us +cc.al.us +cc.ar.us +cc.as.us +cc.az.us +cc.ca.us +cc.co.us +cc.ct.us +cc.dc.us +cc.de.us +cc.fl.us +cc.ga.us +cc.gu.us +cc.hi.us +cc.ia.us +cc.id.us +cc.il.us +cc.in.us +cc.ks.us +cc.ky.us +cc.la.us +cc.ma.us +cc.md.us +cc.me.us +cc.mi.us +cc.mn.us +cc.mo.us +cc.ms.us +cc.mt.us +cc.nc.us +cc.nd.us +cc.ne.us +cc.nh.us +cc.nj.us +cc.nm.us +cc.nv.us +cc.ny.us +cc.oh.us +cc.ok.us +cc.or.us +cc.pa.us +cc.pr.us +cc.ri.us +cc.sc.us +cc.sd.us +cc.tn.us +cc.tx.us +cc.ut.us +cc.vi.us +cc.vt.us +cc.va.us +cc.wa.us +cc.wi.us +cc.wv.us +cc.wy.us +lib.ak.us +lib.al.us +lib.ar.us +lib.as.us +lib.az.us +lib.ca.us +lib.co.us +lib.ct.us +lib.dc.us +// lib.de.us Issue #243 - Moved to Private section at request of Ed Moore +lib.fl.us +lib.ga.us +lib.gu.us +lib.hi.us +lib.ia.us +lib.id.us +lib.il.us +lib.in.us +lib.ks.us +lib.ky.us +lib.la.us +lib.ma.us +lib.md.us +lib.me.us +lib.mi.us +lib.mn.us +lib.mo.us +lib.ms.us +lib.mt.us +lib.nc.us +lib.nd.us +lib.ne.us +lib.nh.us +lib.nj.us +lib.nm.us +lib.nv.us +lib.ny.us +lib.oh.us +lib.ok.us +lib.or.us +lib.pa.us +lib.pr.us +lib.ri.us +lib.sc.us +lib.sd.us +lib.tn.us +lib.tx.us +lib.ut.us +lib.vi.us +lib.vt.us +lib.va.us +lib.wa.us +lib.wi.us +// lib.wv.us Bug 941670 - Removed at request of Larry W Arnold +lib.wy.us +// k12.ma.us contains school districts in Massachusetts. The 4LDs are +// managed independently except for private (PVT), charter (CHTR) and +// parochial (PAROCH) schools. Those are delegated directly to the +// 5LD operators. +pvt.k12.ma.us +chtr.k12.ma.us +paroch.k12.ma.us +// Merit Network, Inc. maintains the registry for =~ /(k12|cc|lib).mi.us/ and the following +// see also: http://domreg.merit.edu +// see also: whois -h whois.domreg.merit.edu help +ann-arbor.mi.us +cog.mi.us +dst.mi.us +eaton.mi.us +gen.mi.us +mus.mi.us +tec.mi.us +washtenaw.mi.us + +// uy : http://www.nic.org.uy/ +uy +com.uy +edu.uy +gub.uy +mil.uy +net.uy +org.uy + +// uz : http://www.reg.uz/ +uz +co.uz +com.uz +net.uz +org.uz + +// va : https://en.wikipedia.org/wiki/.va +va + +// vc : https://en.wikipedia.org/wiki/.vc +// Submitted by registry +vc +com.vc +net.vc +org.vc +gov.vc +mil.vc +edu.vc + +// ve : https://registro.nic.ve/ +// Submitted by registry +ve +arts.ve +co.ve +com.ve +e12.ve +edu.ve +firm.ve +gob.ve +gov.ve +info.ve +int.ve +mil.ve +net.ve +org.ve +rec.ve +store.ve +tec.ve +web.ve + +// vg : https://en.wikipedia.org/wiki/.vg +vg + +// vi : http://www.nic.vi/newdomainform.htm +// http://www.nic.vi/Domain_Rules/body_domain_rules.html indicates some other +// TLDs are "reserved", such as edu.vi and gov.vi, but doesn't actually say they +// are available for registration (which they do not seem to be). +vi +co.vi +com.vi +k12.vi +net.vi +org.vi + +// vn : https://www.dot.vn/vnnic/vnnic/domainregistration.jsp +vn +com.vn +net.vn +org.vn +edu.vn +gov.vn +int.vn +ac.vn +biz.vn +info.vn +name.vn +pro.vn +health.vn + +// vu : https://en.wikipedia.org/wiki/.vu +// http://www.vunic.vu/ +vu +com.vu +edu.vu +net.vu +org.vu + +// wf : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +wf + +// ws : https://en.wikipedia.org/wiki/.ws +// http://samoanic.ws/index.dhtml +ws +com.ws +net.ws +org.ws +gov.ws +edu.ws + +// yt : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +yt + +// IDN ccTLDs +// When submitting patches, please maintain a sort by ISO 3166 ccTLD, then +// U-label, and follow this format: +// // A-Label ("", [, variant info]) : +// // [sponsoring org] +// U-Label + +// xn--mgbaam7a8h ("Emerat", Arabic) : AE +// http://nic.ae/english/arabicdomain/rules.jsp +امارات + +// xn--y9a3aq ("hye", Armenian) : AM +// ISOC AM (operated by .am Registry) +հայ + +// xn--54b7fta0cc ("Bangla", Bangla) : BD +বাংলা + +// xn--90ae ("bg", Bulgarian) : BG +бг + +// xn--90ais ("bel", Belarusian/Russian Cyrillic) : BY +// Operated by .by registry +бел + +// xn--fiqs8s ("Zhongguo/China", Chinese, Simplified) : CN +// CNNIC +// http://cnnic.cn/html/Dir/2005/10/11/3218.htm +中国 + +// xn--fiqz9s ("Zhongguo/China", Chinese, Traditional) : CN +// CNNIC +// http://cnnic.cn/html/Dir/2005/10/11/3218.htm +中國 + +// xn--lgbbat1ad8j ("Algeria/Al Jazair", Arabic) : DZ +الجزائر + +// xn--wgbh1c ("Egypt/Masr", Arabic) : EG +// http://www.dotmasr.eg/ +مصر + +// xn--e1a4c ("eu", Cyrillic) : EU +// https://eurid.eu +ею + +// xn--qxa6a ("eu", Greek) : EU +// https://eurid.eu +ευ + +// xn--mgbah1a3hjkrd ("Mauritania", Arabic) : MR +موريتانيا + +// xn--node ("ge", Georgian Mkhedruli) : GE +გე + +// xn--qxam ("el", Greek) : GR +// Hellenic Ministry of Infrastructure, Transport, and Networks +ελ + +// xn--j6w193g ("Hong Kong", Chinese) : HK +// https://www.hkirc.hk +// Submitted by registry +// https://www.hkirc.hk/content.jsp?id=30#!/34 +香港 +公司.香港 +教育.香港 +政府.香港 +個人.香港 +網絡.香港 +組織.香港 + +// xn--2scrj9c ("Bharat", Kannada) : IN +// India +ಭಾರತ + +// xn--3hcrj9c ("Bharat", Oriya) : IN +// India +ଭାରତ + +// xn--45br5cyl ("Bharatam", Assamese) : IN +// India +ভাৰত + +// xn--h2breg3eve ("Bharatam", Sanskrit) : IN +// India +भारतम् + +// xn--h2brj9c8c ("Bharot", Santali) : IN +// India +भारोत + +// xn--mgbgu82a ("Bharat", Sindhi) : IN +// India +ڀارت + +// xn--rvc1e0am3e ("Bharatam", Malayalam) : IN +// India +ഭാരതം + +// xn--h2brj9c ("Bharat", Devanagari) : IN +// India +भारत + +// xn--mgbbh1a ("Bharat", Kashmiri) : IN +// India +بارت + +// xn--mgbbh1a71e ("Bharat", Arabic) : IN +// India +بھارت + +// xn--fpcrj9c3d ("Bharat", Telugu) : IN +// India +భారత్ + +// xn--gecrj9c ("Bharat", Gujarati) : IN +// India +ભારત + +// xn--s9brj9c ("Bharat", Gurmukhi) : IN +// India +ਭਾਰਤ + +// xn--45brj9c ("Bharat", Bengali) : IN +// India +ভারত + +// xn--xkc2dl3a5ee0h ("India", Tamil) : IN +// India +இந்தியா + +// xn--mgba3a4f16a ("Iran", Persian) : IR +ایران + +// xn--mgba3a4fra ("Iran", Arabic) : IR +ايران + +// xn--mgbtx2b ("Iraq", Arabic) : IQ +// Communications and Media Commission +عراق + +// xn--mgbayh7gpa ("al-Ordon", Arabic) : JO +// National Information Technology Center (NITC) +// Royal Scientific Society, Al-Jubeiha +الاردن + +// xn--3e0b707e ("Republic of Korea", Hangul) : KR +한국 + +// xn--80ao21a ("Kaz", Kazakh) : KZ +қаз + +// xn--fzc2c9e2c ("Lanka", Sinhalese-Sinhala) : LK +// https://nic.lk +ලංකා + +// xn--xkc2al3hye2a ("Ilangai", Tamil) : LK +// https://nic.lk +இலங்கை + +// xn--mgbc0a9azcg ("Morocco/al-Maghrib", Arabic) : MA +المغرب + +// xn--d1alf ("mkd", Macedonian) : MK +// MARnet +мкд + +// xn--l1acc ("mon", Mongolian) : MN +мон + +// xn--mix891f ("Macao", Chinese, Traditional) : MO +// MONIC / HNET Asia (Registry Operator for .mo) +澳門 + +// xn--mix082f ("Macao", Chinese, Simplified) : MO +澳门 + +// xn--mgbx4cd0ab ("Malaysia", Malay) : MY +مليسيا + +// xn--mgb9awbf ("Oman", Arabic) : OM +عمان + +// xn--mgbai9azgqp6j ("Pakistan", Urdu/Arabic) : PK +پاکستان + +// xn--mgbai9a5eva00b ("Pakistan", Urdu/Arabic, variant) : PK +پاكستان + +// xn--ygbi2ammx ("Falasteen", Arabic) : PS +// The Palestinian National Internet Naming Authority (PNINA) +// http://www.pnina.ps +فلسطين + +// xn--90a3ac ("srb", Cyrillic) : RS +// https://www.rnids.rs/en/domains/national-domains +срб +пр.срб +орг.срб +обр.срб +од.срб +упр.срб +ак.срб + +// xn--p1ai ("rf", Russian-Cyrillic) : RU +// https://cctld.ru/files/pdf/docs/en/rules_ru-rf.pdf +// Submitted by George Georgievsky +рф + +// xn--wgbl6a ("Qatar", Arabic) : QA +// http://www.ict.gov.qa/ +قطر + +// xn--mgberp4a5d4ar ("AlSaudiah", Arabic) : SA +// http://www.nic.net.sa/ +السعودية + +// xn--mgberp4a5d4a87g ("AlSaudiah", Arabic, variant) : SA +السعودیة + +// xn--mgbqly7c0a67fbc ("AlSaudiah", Arabic, variant) : SA +السعودیۃ + +// xn--mgbqly7cvafr ("AlSaudiah", Arabic, variant) : SA +السعوديه + +// xn--mgbpl2fh ("sudan", Arabic) : SD +// Operated by .sd registry +سودان + +// xn--yfro4i67o Singapore ("Singapore", Chinese) : SG +新加坡 + +// xn--clchc0ea0b2g2a9gcd ("Singapore", Tamil) : SG +சிங்கப்பூர் + +// xn--ogbpf8fl ("Syria", Arabic) : SY +سورية + +// xn--mgbtf8fl ("Syria", Arabic, variant) : SY +سوريا + +// xn--o3cw4h ("Thai", Thai) : TH +// http://www.thnic.co.th +ไทย +ศึกษา.ไทย +ธุรกิจ.ไทย +รัฐบาล.ไทย +ทหาร.ไทย +เน็ต.ไทย +องค์กร.ไทย + +// xn--pgbs0dh ("Tunisia", Arabic) : TN +// http://nic.tn +تونس + +// xn--kpry57d ("Taiwan", Chinese, Traditional) : TW +// http://www.twnic.net/english/dn/dn_07a.htm +台灣 + +// xn--kprw13d ("Taiwan", Chinese, Simplified) : TW +// http://www.twnic.net/english/dn/dn_07a.htm +台湾 + +// xn--nnx388a ("Taiwan", Chinese, variant) : TW +臺灣 + +// xn--j1amh ("ukr", Cyrillic) : UA +укр + +// xn--mgb2ddes ("AlYemen", Arabic) : YE +اليمن + +// xxx : http://icmregistry.com +xxx + +// ye : http://www.y.net.ye/services/domain_name.htm +*.ye + +// za : https://www.zadna.org.za/content/page/domain-information/ +ac.za +agric.za +alt.za +co.za +edu.za +gov.za +grondar.za +law.za +mil.za +net.za +ngo.za +nic.za +nis.za +nom.za +org.za +school.za +tm.za +web.za + +// zm : https://zicta.zm/ +// Submitted by registry +zm +ac.zm +biz.zm +co.zm +com.zm +edu.zm +gov.zm +info.zm +mil.zm +net.zm +org.zm +sch.zm + +// zw : https://www.potraz.gov.zw/ +// Confirmed by registry 2017-01-25 +zw +ac.zw +co.zw +gov.zw +mil.zw +org.zw + + +// newGTLDs + +// List of new gTLDs imported from https://www.icann.org/resources/registries/gtlds/v2/gtlds.json on 2020-11-30T20:26:10Z +// This list is auto-generated, don't edit it manually. +// aaa : 2015-02-26 American Automobile Association, Inc. +aaa + +// aarp : 2015-05-21 AARP +aarp + +// abarth : 2015-07-30 Fiat Chrysler Automobiles N.V. +abarth + +// abb : 2014-10-24 ABB Ltd +abb + +// abbott : 2014-07-24 Abbott Laboratories, Inc. +abbott + +// abbvie : 2015-07-30 AbbVie Inc. +abbvie + +// abc : 2015-07-30 Disney Enterprises, Inc. +abc + +// able : 2015-06-25 Able Inc. +able + +// abogado : 2014-04-24 Minds + Machines Group Limited +abogado + +// abudhabi : 2015-07-30 Abu Dhabi Systems and Information Centre +abudhabi + +// academy : 2013-11-07 Binky Moon, LLC +academy + +// accenture : 2014-08-15 Accenture plc +accenture + +// accountant : 2014-11-20 dot Accountant Limited +accountant + +// accountants : 2014-03-20 Binky Moon, LLC +accountants + +// aco : 2015-01-08 ACO Severin Ahlmann GmbH & Co. KG +aco + +// actor : 2013-12-12 Dog Beach, LLC +actor + +// adac : 2015-07-16 Allgemeiner Deutscher Automobil-Club e.V. (ADAC) +adac + +// ads : 2014-12-04 Charleston Road Registry Inc. +ads + +// adult : 2014-10-16 ICM Registry AD LLC +adult + +// aeg : 2015-03-19 Aktiebolaget Electrolux +aeg + +// aetna : 2015-05-21 Aetna Life Insurance Company +aetna + +// afamilycompany : 2015-07-23 Johnson Shareholdings, Inc. +afamilycompany + +// afl : 2014-10-02 Australian Football League +afl + +// africa : 2014-03-24 ZA Central Registry NPC trading as Registry.Africa +africa + +// agakhan : 2015-04-23 Fondation Aga Khan (Aga Khan Foundation) +agakhan + +// agency : 2013-11-14 Binky Moon, LLC +agency + +// aig : 2014-12-18 American International Group, Inc. +aig + +// airbus : 2015-07-30 Airbus S.A.S. +airbus + +// airforce : 2014-03-06 Dog Beach, LLC +airforce + +// airtel : 2014-10-24 Bharti Airtel Limited +airtel + +// akdn : 2015-04-23 Fondation Aga Khan (Aga Khan Foundation) +akdn + +// alfaromeo : 2015-07-31 Fiat Chrysler Automobiles N.V. +alfaromeo + +// alibaba : 2015-01-15 Alibaba Group Holding Limited +alibaba + +// alipay : 2015-01-15 Alibaba Group Holding Limited +alipay + +// allfinanz : 2014-07-03 Allfinanz Deutsche Vermögensberatung Aktiengesellschaft +allfinanz + +// allstate : 2015-07-31 Allstate Fire and Casualty Insurance Company +allstate + +// ally : 2015-06-18 Ally Financial Inc. +ally + +// alsace : 2014-07-02 Region Grand Est +alsace + +// alstom : 2015-07-30 ALSTOM +alstom + +// amazon : 2019-12-19 Amazon Registry Services, Inc. +amazon + +// americanexpress : 2015-07-31 American Express Travel Related Services Company, Inc. +americanexpress + +// americanfamily : 2015-07-23 AmFam, Inc. +americanfamily + +// amex : 2015-07-31 American Express Travel Related Services Company, Inc. +amex + +// amfam : 2015-07-23 AmFam, Inc. +amfam + +// amica : 2015-05-28 Amica Mutual Insurance Company +amica + +// amsterdam : 2014-07-24 Gemeente Amsterdam +amsterdam + +// analytics : 2014-12-18 Campus IP LLC +analytics + +// android : 2014-08-07 Charleston Road Registry Inc. +android + +// anquan : 2015-01-08 Beijing Qihu Keji Co., Ltd. +anquan + +// anz : 2015-07-31 Australia and New Zealand Banking Group Limited +anz + +// aol : 2015-09-17 Oath Inc. +aol + +// apartments : 2014-12-11 Binky Moon, LLC +apartments + +// app : 2015-05-14 Charleston Road Registry Inc. +app + +// apple : 2015-05-14 Apple Inc. +apple + +// aquarelle : 2014-07-24 Aquarelle.com +aquarelle + +// arab : 2015-11-12 League of Arab States +arab + +// aramco : 2014-11-20 Aramco Services Company +aramco + +// archi : 2014-02-06 Afilias Limited +archi + +// army : 2014-03-06 Dog Beach, LLC +army + +// art : 2016-03-24 UK Creative Ideas Limited +art + +// arte : 2014-12-11 Association Relative à la Télévision Européenne G.E.I.E. +arte + +// asda : 2015-07-31 Wal-Mart Stores, Inc. +asda + +// associates : 2014-03-06 Binky Moon, LLC +associates + +// athleta : 2015-07-30 The Gap, Inc. +athleta + +// attorney : 2014-03-20 Dog Beach, LLC +attorney + +// auction : 2014-03-20 Dog Beach, LLC +auction + +// audi : 2015-05-21 AUDI Aktiengesellschaft +audi + +// audible : 2015-06-25 Amazon Registry Services, Inc. +audible + +// audio : 2014-03-20 UNR Corp. +audio + +// auspost : 2015-08-13 Australian Postal Corporation +auspost + +// author : 2014-12-18 Amazon Registry Services, Inc. +author + +// auto : 2014-11-13 XYZ.COM LLC +auto + +// autos : 2014-01-09 XYZ.COM LLC +autos + +// avianca : 2015-01-08 Avianca Holdings S.A. +avianca + +// aws : 2015-06-25 Amazon Registry Services, Inc. +aws + +// axa : 2013-12-19 AXA Group Operations SAS +axa + +// azure : 2014-12-18 Microsoft Corporation +azure + +// baby : 2015-04-09 XYZ.COM LLC +baby + +// baidu : 2015-01-08 Baidu, Inc. +baidu + +// banamex : 2015-07-30 Citigroup Inc. +banamex + +// bananarepublic : 2015-07-31 The Gap, Inc. +bananarepublic + +// band : 2014-06-12 Dog Beach, LLC +band + +// bank : 2014-09-25 fTLD Registry Services LLC +bank + +// bar : 2013-12-12 Punto 2012 Sociedad Anonima Promotora de Inversion de Capital Variable +bar + +// barcelona : 2014-07-24 Municipi de Barcelona +barcelona + +// barclaycard : 2014-11-20 Barclays Bank PLC +barclaycard + +// barclays : 2014-11-20 Barclays Bank PLC +barclays + +// barefoot : 2015-06-11 Gallo Vineyards, Inc. +barefoot + +// bargains : 2013-11-14 Binky Moon, LLC +bargains + +// baseball : 2015-10-29 MLB Advanced Media DH, LLC +baseball + +// basketball : 2015-08-20 Fédération Internationale de Basketball (FIBA) +basketball + +// bauhaus : 2014-04-17 Werkhaus GmbH +bauhaus + +// bayern : 2014-01-23 Bayern Connect GmbH +bayern + +// bbc : 2014-12-18 British Broadcasting Corporation +bbc + +// bbt : 2015-07-23 BB&T Corporation +bbt + +// bbva : 2014-10-02 BANCO BILBAO VIZCAYA ARGENTARIA, S.A. +bbva + +// bcg : 2015-04-02 The Boston Consulting Group, Inc. +bcg + +// bcn : 2014-07-24 Municipi de Barcelona +bcn + +// beats : 2015-05-14 Beats Electronics, LLC +beats + +// beauty : 2015-12-03 XYZ.COM LLC +beauty + +// beer : 2014-01-09 Minds + Machines Group Limited +beer + +// bentley : 2014-12-18 Bentley Motors Limited +bentley + +// berlin : 2013-10-31 dotBERLIN GmbH & Co. KG +berlin + +// best : 2013-12-19 BestTLD Pty Ltd +best + +// bestbuy : 2015-07-31 BBY Solutions, Inc. +bestbuy + +// bet : 2015-05-07 Afilias Limited +bet + +// bharti : 2014-01-09 Bharti Enterprises (Holding) Private Limited +bharti + +// bible : 2014-06-19 American Bible Society +bible + +// bid : 2013-12-19 dot Bid Limited +bid + +// bike : 2013-08-27 Binky Moon, LLC +bike + +// bing : 2014-12-18 Microsoft Corporation +bing + +// bingo : 2014-12-04 Binky Moon, LLC +bingo + +// bio : 2014-03-06 Afilias Limited +bio + +// black : 2014-01-16 Afilias Limited +black + +// blackfriday : 2014-01-16 UNR Corp. +blackfriday + +// blockbuster : 2015-07-30 Dish DBS Corporation +blockbuster + +// blog : 2015-05-14 Knock Knock WHOIS There, LLC +blog + +// bloomberg : 2014-07-17 Bloomberg IP Holdings LLC +bloomberg + +// blue : 2013-11-07 Afilias Limited +blue + +// bms : 2014-10-30 Bristol-Myers Squibb Company +bms + +// bmw : 2014-01-09 Bayerische Motoren Werke Aktiengesellschaft +bmw + +// bnpparibas : 2014-05-29 BNP Paribas +bnpparibas + +// boats : 2014-12-04 XYZ.COM LLC +boats + +// boehringer : 2015-07-09 Boehringer Ingelheim International GmbH +boehringer + +// bofa : 2015-07-31 Bank of America Corporation +bofa + +// bom : 2014-10-16 Núcleo de Informação e Coordenação do Ponto BR - NIC.br +bom + +// bond : 2014-06-05 ShortDot SA +bond + +// boo : 2014-01-30 Charleston Road Registry Inc. +boo + +// book : 2015-08-27 Amazon Registry Services, Inc. +book + +// booking : 2015-07-16 Booking.com B.V. +booking + +// bosch : 2015-06-18 Robert Bosch GMBH +bosch + +// bostik : 2015-05-28 Bostik SA +bostik + +// boston : 2015-12-10 Boston TLD Management, LLC +boston + +// bot : 2014-12-18 Amazon Registry Services, Inc. +bot + +// boutique : 2013-11-14 Binky Moon, LLC +boutique + +// box : 2015-11-12 Intercap Registry Inc. +box + +// bradesco : 2014-12-18 Banco Bradesco S.A. +bradesco + +// bridgestone : 2014-12-18 Bridgestone Corporation +bridgestone + +// broadway : 2014-12-22 Celebrate Broadway, Inc. +broadway + +// broker : 2014-12-11 Dotbroker Registry Limited +broker + +// brother : 2015-01-29 Brother Industries, Ltd. +brother + +// brussels : 2014-02-06 DNS.be vzw +brussels + +// budapest : 2013-11-21 Minds + Machines Group Limited +budapest + +// bugatti : 2015-07-23 Bugatti International SA +bugatti + +// build : 2013-11-07 Plan Bee LLC +build + +// builders : 2013-11-07 Binky Moon, LLC +builders + +// business : 2013-11-07 Binky Moon, LLC +business + +// buy : 2014-12-18 Amazon Registry Services, Inc. +buy + +// buzz : 2013-10-02 DOTSTRATEGY CO. +buzz + +// bzh : 2014-02-27 Association www.bzh +bzh + +// cab : 2013-10-24 Binky Moon, LLC +cab + +// cafe : 2015-02-11 Binky Moon, LLC +cafe + +// cal : 2014-07-24 Charleston Road Registry Inc. +cal + +// call : 2014-12-18 Amazon Registry Services, Inc. +call + +// calvinklein : 2015-07-30 PVH gTLD Holdings LLC +calvinklein + +// cam : 2016-04-21 AC Webconnecting Holding B.V. +cam + +// camera : 2013-08-27 Binky Moon, LLC +camera + +// camp : 2013-11-07 Binky Moon, LLC +camp + +// cancerresearch : 2014-05-15 Australian Cancer Research Foundation +cancerresearch + +// canon : 2014-09-12 Canon Inc. +canon + +// capetown : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +capetown + +// capital : 2014-03-06 Binky Moon, LLC +capital + +// capitalone : 2015-08-06 Capital One Financial Corporation +capitalone + +// car : 2015-01-22 XYZ.COM LLC +car + +// caravan : 2013-12-12 Caravan International, Inc. +caravan + +// cards : 2013-12-05 Binky Moon, LLC +cards + +// care : 2014-03-06 Binky Moon, LLC +care + +// career : 2013-10-09 dotCareer LLC +career + +// careers : 2013-10-02 Binky Moon, LLC +careers + +// cars : 2014-11-13 XYZ.COM LLC +cars + +// casa : 2013-11-21 Minds + Machines Group Limited +casa + +// case : 2015-09-03 CNH Industrial N.V. +case + +// caseih : 2015-09-03 CNH Industrial N.V. +caseih + +// cash : 2014-03-06 Binky Moon, LLC +cash + +// casino : 2014-12-18 Binky Moon, LLC +casino + +// catering : 2013-12-05 Binky Moon, LLC +catering + +// catholic : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +catholic + +// cba : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +cba + +// cbn : 2014-08-22 The Christian Broadcasting Network, Inc. +cbn + +// cbre : 2015-07-02 CBRE, Inc. +cbre + +// cbs : 2015-08-06 CBS Domains Inc. +cbs + +// ceb : 2015-04-09 The Corporate Executive Board Company +ceb + +// center : 2013-11-07 Binky Moon, LLC +center + +// ceo : 2013-11-07 CEOTLD Pty Ltd +ceo + +// cern : 2014-06-05 European Organization for Nuclear Research ("CERN") +cern + +// cfa : 2014-08-28 CFA Institute +cfa + +// cfd : 2014-12-11 DotCFD Registry Limited +cfd + +// chanel : 2015-04-09 Chanel International B.V. +chanel + +// channel : 2014-05-08 Charleston Road Registry Inc. +channel + +// charity : 2018-04-11 Binky Moon, LLC +charity + +// chase : 2015-04-30 JPMorgan Chase Bank, National Association +chase + +// chat : 2014-12-04 Binky Moon, LLC +chat + +// cheap : 2013-11-14 Binky Moon, LLC +cheap + +// chintai : 2015-06-11 CHINTAI Corporation +chintai + +// christmas : 2013-11-21 UNR Corp. +christmas + +// chrome : 2014-07-24 Charleston Road Registry Inc. +chrome + +// church : 2014-02-06 Binky Moon, LLC +church + +// cipriani : 2015-02-19 Hotel Cipriani Srl +cipriani + +// circle : 2014-12-18 Amazon Registry Services, Inc. +circle + +// cisco : 2014-12-22 Cisco Technology, Inc. +cisco + +// citadel : 2015-07-23 Citadel Domain LLC +citadel + +// citi : 2015-07-30 Citigroup Inc. +citi + +// citic : 2014-01-09 CITIC Group Corporation +citic + +// city : 2014-05-29 Binky Moon, LLC +city + +// cityeats : 2014-12-11 Lifestyle Domain Holdings, Inc. +cityeats + +// claims : 2014-03-20 Binky Moon, LLC +claims + +// cleaning : 2013-12-05 Binky Moon, LLC +cleaning + +// click : 2014-06-05 UNR Corp. +click + +// clinic : 2014-03-20 Binky Moon, LLC +clinic + +// clinique : 2015-10-01 The Estée Lauder Companies Inc. +clinique + +// clothing : 2013-08-27 Binky Moon, LLC +clothing + +// cloud : 2015-04-16 Aruba PEC S.p.A. +cloud + +// club : 2013-11-08 .CLUB DOMAINS, LLC +club + +// clubmed : 2015-06-25 Club Méditerranée S.A. +clubmed + +// coach : 2014-10-09 Binky Moon, LLC +coach + +// codes : 2013-10-31 Binky Moon, LLC +codes + +// coffee : 2013-10-17 Binky Moon, LLC +coffee + +// college : 2014-01-16 XYZ.COM LLC +college + +// cologne : 2014-02-05 dotKoeln GmbH +cologne + +// comcast : 2015-07-23 Comcast IP Holdings I, LLC +comcast + +// commbank : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +commbank + +// community : 2013-12-05 Binky Moon, LLC +community + +// company : 2013-11-07 Binky Moon, LLC +company + +// compare : 2015-10-08 Registry Services, LLC +compare + +// computer : 2013-10-24 Binky Moon, LLC +computer + +// comsec : 2015-01-08 VeriSign, Inc. +comsec + +// condos : 2013-12-05 Binky Moon, LLC +condos + +// construction : 2013-09-16 Binky Moon, LLC +construction + +// consulting : 2013-12-05 Dog Beach, LLC +consulting + +// contact : 2015-01-08 Dog Beach, LLC +contact + +// contractors : 2013-09-10 Binky Moon, LLC +contractors + +// cooking : 2013-11-21 Minds + Machines Group Limited +cooking + +// cookingchannel : 2015-07-02 Lifestyle Domain Holdings, Inc. +cookingchannel + +// cool : 2013-11-14 Binky Moon, LLC +cool + +// corsica : 2014-09-25 Collectivité de Corse +corsica + +// country : 2013-12-19 DotCountry LLC +country + +// coupon : 2015-02-26 Amazon Registry Services, Inc. +coupon + +// coupons : 2015-03-26 Binky Moon, LLC +coupons + +// courses : 2014-12-04 OPEN UNIVERSITIES AUSTRALIA PTY LTD +courses + +// cpa : 2019-06-10 American Institute of Certified Public Accountants +cpa + +// credit : 2014-03-20 Binky Moon, LLC +credit + +// creditcard : 2014-03-20 Binky Moon, LLC +creditcard + +// creditunion : 2015-01-22 CUNA Performance Resources, LLC +creditunion + +// cricket : 2014-10-09 dot Cricket Limited +cricket + +// crown : 2014-10-24 Crown Equipment Corporation +crown + +// crs : 2014-04-03 Federated Co-operatives Limited +crs + +// cruise : 2015-12-10 Viking River Cruises (Bermuda) Ltd. +cruise + +// cruises : 2013-12-05 Binky Moon, LLC +cruises + +// csc : 2014-09-25 Alliance-One Services, Inc. +csc + +// cuisinella : 2014-04-03 SCHMIDT GROUPE S.A.S. +cuisinella + +// cymru : 2014-05-08 Nominet UK +cymru + +// cyou : 2015-01-22 ShortDot SA +cyou + +// dabur : 2014-02-06 Dabur India Limited +dabur + +// dad : 2014-01-23 Charleston Road Registry Inc. +dad + +// dance : 2013-10-24 Dog Beach, LLC +dance + +// data : 2016-06-02 Dish DBS Corporation +data + +// date : 2014-11-20 dot Date Limited +date + +// dating : 2013-12-05 Binky Moon, LLC +dating + +// datsun : 2014-03-27 NISSAN MOTOR CO., LTD. +datsun + +// day : 2014-01-30 Charleston Road Registry Inc. +day + +// dclk : 2014-11-20 Charleston Road Registry Inc. +dclk + +// dds : 2015-05-07 Minds + Machines Group Limited +dds + +// deal : 2015-06-25 Amazon Registry Services, Inc. +deal + +// dealer : 2014-12-22 Intercap Registry Inc. +dealer + +// deals : 2014-05-22 Binky Moon, LLC +deals + +// degree : 2014-03-06 Dog Beach, LLC +degree + +// delivery : 2014-09-11 Binky Moon, LLC +delivery + +// dell : 2014-10-24 Dell Inc. +dell + +// deloitte : 2015-07-31 Deloitte Touche Tohmatsu +deloitte + +// delta : 2015-02-19 Delta Air Lines, Inc. +delta + +// democrat : 2013-10-24 Dog Beach, LLC +democrat + +// dental : 2014-03-20 Binky Moon, LLC +dental + +// dentist : 2014-03-20 Dog Beach, LLC +dentist + +// desi : 2013-11-14 Desi Networks LLC +desi + +// design : 2014-11-07 Top Level Design, LLC +design + +// dev : 2014-10-16 Charleston Road Registry Inc. +dev + +// dhl : 2015-07-23 Deutsche Post AG +dhl + +// diamonds : 2013-09-22 Binky Moon, LLC +diamonds + +// diet : 2014-06-26 UNR Corp. +diet + +// digital : 2014-03-06 Binky Moon, LLC +digital + +// direct : 2014-04-10 Binky Moon, LLC +direct + +// directory : 2013-09-20 Binky Moon, LLC +directory + +// discount : 2014-03-06 Binky Moon, LLC +discount + +// discover : 2015-07-23 Discover Financial Services +discover + +// dish : 2015-07-30 Dish DBS Corporation +dish + +// diy : 2015-11-05 Lifestyle Domain Holdings, Inc. +diy + +// dnp : 2013-12-13 Dai Nippon Printing Co., Ltd. +dnp + +// docs : 2014-10-16 Charleston Road Registry Inc. +docs + +// doctor : 2016-06-02 Binky Moon, LLC +doctor + +// dog : 2014-12-04 Binky Moon, LLC +dog + +// domains : 2013-10-17 Binky Moon, LLC +domains + +// dot : 2015-05-21 Dish DBS Corporation +dot + +// download : 2014-11-20 dot Support Limited +download + +// drive : 2015-03-05 Charleston Road Registry Inc. +drive + +// dtv : 2015-06-04 Dish DBS Corporation +dtv + +// dubai : 2015-01-01 Dubai Smart Government Department +dubai + +// duck : 2015-07-23 Johnson Shareholdings, Inc. +duck + +// dunlop : 2015-07-02 The Goodyear Tire & Rubber Company +dunlop + +// dupont : 2015-06-25 E. I. du Pont de Nemours and Company +dupont + +// durban : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +durban + +// dvag : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +dvag + +// dvr : 2016-05-26 DISH Technologies L.L.C. +dvr + +// earth : 2014-12-04 Interlink Co., Ltd. +earth + +// eat : 2014-01-23 Charleston Road Registry Inc. +eat + +// eco : 2016-07-08 Big Room Inc. +eco + +// edeka : 2014-12-18 EDEKA Verband kaufmännischer Genossenschaften e.V. +edeka + +// education : 2013-11-07 Binky Moon, LLC +education + +// email : 2013-10-31 Binky Moon, LLC +email + +// emerck : 2014-04-03 Merck KGaA +emerck + +// energy : 2014-09-11 Binky Moon, LLC +energy + +// engineer : 2014-03-06 Dog Beach, LLC +engineer + +// engineering : 2014-03-06 Binky Moon, LLC +engineering + +// enterprises : 2013-09-20 Binky Moon, LLC +enterprises + +// epson : 2014-12-04 Seiko Epson Corporation +epson + +// equipment : 2013-08-27 Binky Moon, LLC +equipment + +// ericsson : 2015-07-09 Telefonaktiebolaget L M Ericsson +ericsson + +// erni : 2014-04-03 ERNI Group Holding AG +erni + +// esq : 2014-05-08 Charleston Road Registry Inc. +esq + +// estate : 2013-08-27 Binky Moon, LLC +estate + +// etisalat : 2015-09-03 Emirates Telecommunications Corporation (trading as Etisalat) +etisalat + +// eurovision : 2014-04-24 European Broadcasting Union (EBU) +eurovision + +// eus : 2013-12-12 Puntueus Fundazioa +eus + +// events : 2013-12-05 Binky Moon, LLC +events + +// exchange : 2014-03-06 Binky Moon, LLC +exchange + +// expert : 2013-11-21 Binky Moon, LLC +expert + +// exposed : 2013-12-05 Binky Moon, LLC +exposed + +// express : 2015-02-11 Binky Moon, LLC +express + +// extraspace : 2015-05-14 Extra Space Storage LLC +extraspace + +// fage : 2014-12-18 Fage International S.A. +fage + +// fail : 2014-03-06 Binky Moon, LLC +fail + +// fairwinds : 2014-11-13 FairWinds Partners, LLC +fairwinds + +// faith : 2014-11-20 dot Faith Limited +faith + +// family : 2015-04-02 Dog Beach, LLC +family + +// fan : 2014-03-06 Dog Beach, LLC +fan + +// fans : 2014-11-07 ZDNS International Limited +fans + +// farm : 2013-11-07 Binky Moon, LLC +farm + +// farmers : 2015-07-09 Farmers Insurance Exchange +farmers + +// fashion : 2014-07-03 Minds + Machines Group Limited +fashion + +// fast : 2014-12-18 Amazon Registry Services, Inc. +fast + +// fedex : 2015-08-06 Federal Express Corporation +fedex + +// feedback : 2013-12-19 Top Level Spectrum, Inc. +feedback + +// ferrari : 2015-07-31 Fiat Chrysler Automobiles N.V. +ferrari + +// ferrero : 2014-12-18 Ferrero Trading Lux S.A. +ferrero + +// fiat : 2015-07-31 Fiat Chrysler Automobiles N.V. +fiat + +// fidelity : 2015-07-30 Fidelity Brokerage Services LLC +fidelity + +// fido : 2015-08-06 Rogers Communications Canada Inc. +fido + +// film : 2015-01-08 Motion Picture Domain Registry Pty Ltd +film + +// final : 2014-10-16 Núcleo de Informação e Coordenação do Ponto BR - NIC.br +final + +// finance : 2014-03-20 Binky Moon, LLC +finance + +// financial : 2014-03-06 Binky Moon, LLC +financial + +// fire : 2015-06-25 Amazon Registry Services, Inc. +fire + +// firestone : 2014-12-18 Bridgestone Licensing Services, Inc +firestone + +// firmdale : 2014-03-27 Firmdale Holdings Limited +firmdale + +// fish : 2013-12-12 Binky Moon, LLC +fish + +// fishing : 2013-11-21 Minds + Machines Group Limited +fishing + +// fit : 2014-11-07 Minds + Machines Group Limited +fit + +// fitness : 2014-03-06 Binky Moon, LLC +fitness + +// flickr : 2015-04-02 Flickr, Inc. +flickr + +// flights : 2013-12-05 Binky Moon, LLC +flights + +// flir : 2015-07-23 FLIR Systems, Inc. +flir + +// florist : 2013-11-07 Binky Moon, LLC +florist + +// flowers : 2014-10-09 UNR Corp. +flowers + +// fly : 2014-05-08 Charleston Road Registry Inc. +fly + +// foo : 2014-01-23 Charleston Road Registry Inc. +foo + +// food : 2016-04-21 Lifestyle Domain Holdings, Inc. +food + +// foodnetwork : 2015-07-02 Lifestyle Domain Holdings, Inc. +foodnetwork + +// football : 2014-12-18 Binky Moon, LLC +football + +// ford : 2014-11-13 Ford Motor Company +ford + +// forex : 2014-12-11 Dotforex Registry Limited +forex + +// forsale : 2014-05-22 Dog Beach, LLC +forsale + +// forum : 2015-04-02 Fegistry, LLC +forum + +// foundation : 2013-12-05 Binky Moon, LLC +foundation + +// fox : 2015-09-11 FOX Registry, LLC +fox + +// free : 2015-12-10 Amazon Registry Services, Inc. +free + +// fresenius : 2015-07-30 Fresenius Immobilien-Verwaltungs-GmbH +fresenius + +// frl : 2014-05-15 FRLregistry B.V. +frl + +// frogans : 2013-12-19 OP3FT +frogans + +// frontdoor : 2015-07-02 Lifestyle Domain Holdings, Inc. +frontdoor + +// frontier : 2015-02-05 Frontier Communications Corporation +frontier + +// ftr : 2015-07-16 Frontier Communications Corporation +ftr + +// fujitsu : 2015-07-30 Fujitsu Limited +fujitsu + +// fujixerox : 2015-07-23 Xerox DNHC LLC +fujixerox + +// fun : 2016-01-14 DotSpace Inc. +fun + +// fund : 2014-03-20 Binky Moon, LLC +fund + +// furniture : 2014-03-20 Binky Moon, LLC +furniture + +// futbol : 2013-09-20 Dog Beach, LLC +futbol + +// fyi : 2015-04-02 Binky Moon, LLC +fyi + +// gal : 2013-11-07 Asociación puntoGAL +gal + +// gallery : 2013-09-13 Binky Moon, LLC +gallery + +// gallo : 2015-06-11 Gallo Vineyards, Inc. +gallo + +// gallup : 2015-02-19 Gallup, Inc. +gallup + +// game : 2015-05-28 UNR Corp. +game + +// games : 2015-05-28 Dog Beach, LLC +games + +// gap : 2015-07-31 The Gap, Inc. +gap + +// garden : 2014-06-26 Minds + Machines Group Limited +garden + +// gay : 2019-05-23 Top Level Design, LLC +gay + +// gbiz : 2014-07-17 Charleston Road Registry Inc. +gbiz + +// gdn : 2014-07-31 Joint Stock Company "Navigation-information systems" +gdn + +// gea : 2014-12-04 GEA Group Aktiengesellschaft +gea + +// gent : 2014-01-23 COMBELL NV +gent + +// genting : 2015-03-12 Resorts World Inc Pte. Ltd. +genting + +// george : 2015-07-31 Wal-Mart Stores, Inc. +george + +// ggee : 2014-01-09 GMO Internet, Inc. +ggee + +// gift : 2013-10-17 DotGift, LLC +gift + +// gifts : 2014-07-03 Binky Moon, LLC +gifts + +// gives : 2014-03-06 Dog Beach, LLC +gives + +// giving : 2014-11-13 Giving Limited +giving + +// glade : 2015-07-23 Johnson Shareholdings, Inc. +glade + +// glass : 2013-11-07 Binky Moon, LLC +glass + +// gle : 2014-07-24 Charleston Road Registry Inc. +gle + +// global : 2014-04-17 Dot Global Domain Registry Limited +global + +// globo : 2013-12-19 Globo Comunicação e Participações S.A +globo + +// gmail : 2014-05-01 Charleston Road Registry Inc. +gmail + +// gmbh : 2016-01-29 Binky Moon, LLC +gmbh + +// gmo : 2014-01-09 GMO Internet, Inc. +gmo + +// gmx : 2014-04-24 1&1 Mail & Media GmbH +gmx + +// godaddy : 2015-07-23 Go Daddy East, LLC +godaddy + +// gold : 2015-01-22 Binky Moon, LLC +gold + +// goldpoint : 2014-11-20 YODOBASHI CAMERA CO.,LTD. +goldpoint + +// golf : 2014-12-18 Binky Moon, LLC +golf + +// goo : 2014-12-18 NTT Resonant Inc. +goo + +// goodyear : 2015-07-02 The Goodyear Tire & Rubber Company +goodyear + +// goog : 2014-11-20 Charleston Road Registry Inc. +goog + +// google : 2014-07-24 Charleston Road Registry Inc. +google + +// gop : 2014-01-16 Republican State Leadership Committee, Inc. +gop + +// got : 2014-12-18 Amazon Registry Services, Inc. +got + +// grainger : 2015-05-07 Grainger Registry Services, LLC +grainger + +// graphics : 2013-09-13 Binky Moon, LLC +graphics + +// gratis : 2014-03-20 Binky Moon, LLC +gratis + +// green : 2014-05-08 Afilias Limited +green + +// gripe : 2014-03-06 Binky Moon, LLC +gripe + +// grocery : 2016-06-16 Wal-Mart Stores, Inc. +grocery + +// group : 2014-08-15 Binky Moon, LLC +group + +// guardian : 2015-07-30 The Guardian Life Insurance Company of America +guardian + +// gucci : 2014-11-13 Guccio Gucci S.p.a. +gucci + +// guge : 2014-08-28 Charleston Road Registry Inc. +guge + +// guide : 2013-09-13 Binky Moon, LLC +guide + +// guitars : 2013-11-14 UNR Corp. +guitars + +// guru : 2013-08-27 Binky Moon, LLC +guru + +// hair : 2015-12-03 XYZ.COM LLC +hair + +// hamburg : 2014-02-20 Hamburg Top-Level-Domain GmbH +hamburg + +// hangout : 2014-11-13 Charleston Road Registry Inc. +hangout + +// haus : 2013-12-05 Dog Beach, LLC +haus + +// hbo : 2015-07-30 HBO Registry Services, Inc. +hbo + +// hdfc : 2015-07-30 HOUSING DEVELOPMENT FINANCE CORPORATION LIMITED +hdfc + +// hdfcbank : 2015-02-12 HDFC Bank Limited +hdfcbank + +// health : 2015-02-11 DotHealth, LLC +health + +// healthcare : 2014-06-12 Binky Moon, LLC +healthcare + +// help : 2014-06-26 UNR Corp. +help + +// helsinki : 2015-02-05 City of Helsinki +helsinki + +// here : 2014-02-06 Charleston Road Registry Inc. +here + +// hermes : 2014-07-10 HERMES INTERNATIONAL +hermes + +// hgtv : 2015-07-02 Lifestyle Domain Holdings, Inc. +hgtv + +// hiphop : 2014-03-06 UNR Corp. +hiphop + +// hisamitsu : 2015-07-16 Hisamitsu Pharmaceutical Co.,Inc. +hisamitsu + +// hitachi : 2014-10-31 Hitachi, Ltd. +hitachi + +// hiv : 2014-03-13 UNR Corp. +hiv + +// hkt : 2015-05-14 PCCW-HKT DataCom Services Limited +hkt + +// hockey : 2015-03-19 Binky Moon, LLC +hockey + +// holdings : 2013-08-27 Binky Moon, LLC +holdings + +// holiday : 2013-11-07 Binky Moon, LLC +holiday + +// homedepot : 2015-04-02 Home Depot Product Authority, LLC +homedepot + +// homegoods : 2015-07-16 The TJX Companies, Inc. +homegoods + +// homes : 2014-01-09 XYZ.COM LLC +homes + +// homesense : 2015-07-16 The TJX Companies, Inc. +homesense + +// honda : 2014-12-18 Honda Motor Co., Ltd. +honda + +// horse : 2013-11-21 Minds + Machines Group Limited +horse + +// hospital : 2016-10-20 Binky Moon, LLC +hospital + +// host : 2014-04-17 DotHost Inc. +host + +// hosting : 2014-05-29 UNR Corp. +hosting + +// hot : 2015-08-27 Amazon Registry Services, Inc. +hot + +// hoteles : 2015-03-05 Travel Reservations SRL +hoteles + +// hotels : 2016-04-07 Booking.com B.V. +hotels + +// hotmail : 2014-12-18 Microsoft Corporation +hotmail + +// house : 2013-11-07 Binky Moon, LLC +house + +// how : 2014-01-23 Charleston Road Registry Inc. +how + +// hsbc : 2014-10-24 HSBC Global Services (UK) Limited +hsbc + +// hughes : 2015-07-30 Hughes Satellite Systems Corporation +hughes + +// hyatt : 2015-07-30 Hyatt GTLD, L.L.C. +hyatt + +// hyundai : 2015-07-09 Hyundai Motor Company +hyundai + +// ibm : 2014-07-31 International Business Machines Corporation +ibm + +// icbc : 2015-02-19 Industrial and Commercial Bank of China Limited +icbc + +// ice : 2014-10-30 IntercontinentalExchange, Inc. +ice + +// icu : 2015-01-08 ShortDot SA +icu + +// ieee : 2015-07-23 IEEE Global LLC +ieee + +// ifm : 2014-01-30 ifm electronic gmbh +ifm + +// ikano : 2015-07-09 Ikano S.A. +ikano + +// imamat : 2015-08-06 Fondation Aga Khan (Aga Khan Foundation) +imamat + +// imdb : 2015-06-25 Amazon Registry Services, Inc. +imdb + +// immo : 2014-07-10 Binky Moon, LLC +immo + +// immobilien : 2013-11-07 Dog Beach, LLC +immobilien + +// inc : 2018-03-10 Intercap Registry Inc. +inc + +// industries : 2013-12-05 Binky Moon, LLC +industries + +// infiniti : 2014-03-27 NISSAN MOTOR CO., LTD. +infiniti + +// ing : 2014-01-23 Charleston Road Registry Inc. +ing + +// ink : 2013-12-05 Top Level Design, LLC +ink + +// institute : 2013-11-07 Binky Moon, LLC +institute + +// insurance : 2015-02-19 fTLD Registry Services LLC +insurance + +// insure : 2014-03-20 Binky Moon, LLC +insure + +// international : 2013-11-07 Binky Moon, LLC +international + +// intuit : 2015-07-30 Intuit Administrative Services, Inc. +intuit + +// investments : 2014-03-20 Binky Moon, LLC +investments + +// ipiranga : 2014-08-28 Ipiranga Produtos de Petroleo S.A. +ipiranga + +// irish : 2014-08-07 Binky Moon, LLC +irish + +// ismaili : 2015-08-06 Fondation Aga Khan (Aga Khan Foundation) +ismaili + +// ist : 2014-08-28 Istanbul Metropolitan Municipality +ist + +// istanbul : 2014-08-28 Istanbul Metropolitan Municipality +istanbul + +// itau : 2014-10-02 Itau Unibanco Holding S.A. +itau + +// itv : 2015-07-09 ITV Services Limited +itv + +// iveco : 2015-09-03 CNH Industrial N.V. +iveco + +// jaguar : 2014-11-13 Jaguar Land Rover Ltd +jaguar + +// java : 2014-06-19 Oracle Corporation +java + +// jcb : 2014-11-20 JCB Co., Ltd. +jcb + +// jeep : 2015-07-30 FCA US LLC. +jeep + +// jetzt : 2014-01-09 Binky Moon, LLC +jetzt + +// jewelry : 2015-03-05 Binky Moon, LLC +jewelry + +// jio : 2015-04-02 Reliance Industries Limited +jio + +// jll : 2015-04-02 Jones Lang LaSalle Incorporated +jll + +// jmp : 2015-03-26 Matrix IP LLC +jmp + +// jnj : 2015-06-18 Johnson & Johnson Services, Inc. +jnj + +// joburg : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +joburg + +// jot : 2014-12-18 Amazon Registry Services, Inc. +jot + +// joy : 2014-12-18 Amazon Registry Services, Inc. +joy + +// jpmorgan : 2015-04-30 JPMorgan Chase Bank, National Association +jpmorgan + +// jprs : 2014-09-18 Japan Registry Services Co., Ltd. +jprs + +// juegos : 2014-03-20 UNR Corp. +juegos + +// juniper : 2015-07-30 JUNIPER NETWORKS, INC. +juniper + +// kaufen : 2013-11-07 Dog Beach, LLC +kaufen + +// kddi : 2014-09-12 KDDI CORPORATION +kddi + +// kerryhotels : 2015-04-30 Kerry Trading Co. Limited +kerryhotels + +// kerrylogistics : 2015-04-09 Kerry Trading Co. Limited +kerrylogistics + +// kerryproperties : 2015-04-09 Kerry Trading Co. Limited +kerryproperties + +// kfh : 2014-12-04 Kuwait Finance House +kfh + +// kia : 2015-07-09 KIA MOTORS CORPORATION +kia + +// kim : 2013-09-23 Afilias Limited +kim + +// kinder : 2014-11-07 Ferrero Trading Lux S.A. +kinder + +// kindle : 2015-06-25 Amazon Registry Services, Inc. +kindle + +// kitchen : 2013-09-20 Binky Moon, LLC +kitchen + +// kiwi : 2013-09-20 DOT KIWI LIMITED +kiwi + +// koeln : 2014-01-09 dotKoeln GmbH +koeln + +// komatsu : 2015-01-08 Komatsu Ltd. +komatsu + +// kosher : 2015-08-20 Kosher Marketing Assets LLC +kosher + +// kpmg : 2015-04-23 KPMG International Cooperative (KPMG International Genossenschaft) +kpmg + +// kpn : 2015-01-08 Koninklijke KPN N.V. +kpn + +// krd : 2013-12-05 KRG Department of Information Technology +krd + +// kred : 2013-12-19 KredTLD Pty Ltd +kred + +// kuokgroup : 2015-04-09 Kerry Trading Co. Limited +kuokgroup + +// kyoto : 2014-11-07 Academic Institution: Kyoto Jyoho Gakuen +kyoto + +// lacaixa : 2014-01-09 Fundación Bancaria Caixa d’Estalvis i Pensions de Barcelona, “la Caixa” +lacaixa + +// lamborghini : 2015-06-04 Automobili Lamborghini S.p.A. +lamborghini + +// lamer : 2015-10-01 The Estée Lauder Companies Inc. +lamer + +// lancaster : 2015-02-12 LANCASTER +lancaster + +// lancia : 2015-07-31 Fiat Chrysler Automobiles N.V. +lancia + +// land : 2013-09-10 Binky Moon, LLC +land + +// landrover : 2014-11-13 Jaguar Land Rover Ltd +landrover + +// lanxess : 2015-07-30 LANXESS Corporation +lanxess + +// lasalle : 2015-04-02 Jones Lang LaSalle Incorporated +lasalle + +// lat : 2014-10-16 ECOM-LAC Federaciòn de Latinoamèrica y el Caribe para Internet y el Comercio Electrònico +lat + +// latino : 2015-07-30 Dish DBS Corporation +latino + +// latrobe : 2014-06-16 La Trobe University +latrobe + +// law : 2015-01-22 LW TLD Limited +law + +// lawyer : 2014-03-20 Dog Beach, LLC +lawyer + +// lds : 2014-03-20 IRI Domain Management, LLC +lds + +// lease : 2014-03-06 Binky Moon, LLC +lease + +// leclerc : 2014-08-07 A.C.D. LEC Association des Centres Distributeurs Edouard Leclerc +leclerc + +// lefrak : 2015-07-16 LeFrak Organization, Inc. +lefrak + +// legal : 2014-10-16 Binky Moon, LLC +legal + +// lego : 2015-07-16 LEGO Juris A/S +lego + +// lexus : 2015-04-23 TOYOTA MOTOR CORPORATION +lexus + +// lgbt : 2014-05-08 Afilias Limited +lgbt + +// lidl : 2014-09-18 Schwarz Domains und Services GmbH & Co. KG +lidl + +// life : 2014-02-06 Binky Moon, LLC +life + +// lifeinsurance : 2015-01-15 American Council of Life Insurers +lifeinsurance + +// lifestyle : 2014-12-11 Lifestyle Domain Holdings, Inc. +lifestyle + +// lighting : 2013-08-27 Binky Moon, LLC +lighting + +// like : 2014-12-18 Amazon Registry Services, Inc. +like + +// lilly : 2015-07-31 Eli Lilly and Company +lilly + +// limited : 2014-03-06 Binky Moon, LLC +limited + +// limo : 2013-10-17 Binky Moon, LLC +limo + +// lincoln : 2014-11-13 Ford Motor Company +lincoln + +// linde : 2014-12-04 Linde Aktiengesellschaft +linde + +// link : 2013-11-14 UNR Corp. +link + +// lipsy : 2015-06-25 Lipsy Ltd +lipsy + +// live : 2014-12-04 Dog Beach, LLC +live + +// living : 2015-07-30 Lifestyle Domain Holdings, Inc. +living + +// lixil : 2015-03-19 LIXIL Group Corporation +lixil + +// llc : 2017-12-14 Afilias Limited +llc + +// llp : 2019-08-26 UNR Corp. +llp + +// loan : 2014-11-20 dot Loan Limited +loan + +// loans : 2014-03-20 Binky Moon, LLC +loans + +// locker : 2015-06-04 Dish DBS Corporation +locker + +// locus : 2015-06-25 Locus Analytics LLC +locus + +// loft : 2015-07-30 Annco, Inc. +loft + +// lol : 2015-01-30 UNR Corp. +lol + +// london : 2013-11-14 Dot London Domains Limited +london + +// lotte : 2014-11-07 Lotte Holdings Co., Ltd. +lotte + +// lotto : 2014-04-10 Afilias Limited +lotto + +// love : 2014-12-22 Merchant Law Group LLP +love + +// lpl : 2015-07-30 LPL Holdings, Inc. +lpl + +// lplfinancial : 2015-07-30 LPL Holdings, Inc. +lplfinancial + +// ltd : 2014-09-25 Binky Moon, LLC +ltd + +// ltda : 2014-04-17 InterNetX, Corp +ltda + +// lundbeck : 2015-08-06 H. Lundbeck A/S +lundbeck + +// lupin : 2014-11-07 LUPIN LIMITED +lupin + +// luxe : 2014-01-09 Minds + Machines Group Limited +luxe + +// luxury : 2013-10-17 Luxury Partners, LLC +luxury + +// macys : 2015-07-31 Macys, Inc. +macys + +// madrid : 2014-05-01 Comunidad de Madrid +madrid + +// maif : 2014-10-02 Mutuelle Assurance Instituteur France (MAIF) +maif + +// maison : 2013-12-05 Binky Moon, LLC +maison + +// makeup : 2015-01-15 XYZ.COM LLC +makeup + +// man : 2014-12-04 MAN SE +man + +// management : 2013-11-07 Binky Moon, LLC +management + +// mango : 2013-10-24 PUNTO FA S.L. +mango + +// map : 2016-06-09 Charleston Road Registry Inc. +map + +// market : 2014-03-06 Dog Beach, LLC +market + +// marketing : 2013-11-07 Binky Moon, LLC +marketing + +// markets : 2014-12-11 Dotmarkets Registry Limited +markets + +// marriott : 2014-10-09 Marriott Worldwide Corporation +marriott + +// marshalls : 2015-07-16 The TJX Companies, Inc. +marshalls + +// maserati : 2015-07-31 Fiat Chrysler Automobiles N.V. +maserati + +// mattel : 2015-08-06 Mattel Sites, Inc. +mattel + +// mba : 2015-04-02 Binky Moon, LLC +mba + +// mckinsey : 2015-07-31 McKinsey Holdings, Inc. +mckinsey + +// med : 2015-08-06 Medistry LLC +med + +// media : 2014-03-06 Binky Moon, LLC +media + +// meet : 2014-01-16 Charleston Road Registry Inc. +meet + +// melbourne : 2014-05-29 The Crown in right of the State of Victoria, represented by its Department of State Development, Business and Innovation +melbourne + +// meme : 2014-01-30 Charleston Road Registry Inc. +meme + +// memorial : 2014-10-16 Dog Beach, LLC +memorial + +// men : 2015-02-26 Exclusive Registry Limited +men + +// menu : 2013-09-11 Dot Menu Registry, LLC +menu + +// merckmsd : 2016-07-14 MSD Registry Holdings, Inc. +merckmsd + +// miami : 2013-12-19 Minds + Machines Group Limited +miami + +// microsoft : 2014-12-18 Microsoft Corporation +microsoft + +// mini : 2014-01-09 Bayerische Motoren Werke Aktiengesellschaft +mini + +// mint : 2015-07-30 Intuit Administrative Services, Inc. +mint + +// mit : 2015-07-02 Massachusetts Institute of Technology +mit + +// mitsubishi : 2015-07-23 Mitsubishi Corporation +mitsubishi + +// mlb : 2015-05-21 MLB Advanced Media DH, LLC +mlb + +// mls : 2015-04-23 The Canadian Real Estate Association +mls + +// mma : 2014-11-07 MMA IARD +mma + +// mobile : 2016-06-02 Dish DBS Corporation +mobile + +// moda : 2013-11-07 Dog Beach, LLC +moda + +// moe : 2013-11-13 Interlink Co., Ltd. +moe + +// moi : 2014-12-18 Amazon Registry Services, Inc. +moi + +// mom : 2015-04-16 UNR Corp. +mom + +// monash : 2013-09-30 Monash University +monash + +// money : 2014-10-16 Binky Moon, LLC +money + +// monster : 2015-09-11 XYZ.COM LLC +monster + +// mormon : 2013-12-05 IRI Domain Management, LLC +mormon + +// mortgage : 2014-03-20 Dog Beach, LLC +mortgage + +// moscow : 2013-12-19 Foundation for Assistance for Internet Technologies and Infrastructure Development (FAITID) +moscow + +// moto : 2015-06-04 Motorola Trademark Holdings, LLC +moto + +// motorcycles : 2014-01-09 XYZ.COM LLC +motorcycles + +// mov : 2014-01-30 Charleston Road Registry Inc. +mov + +// movie : 2015-02-05 Binky Moon, LLC +movie + +// msd : 2015-07-23 MSD Registry Holdings, Inc. +msd + +// mtn : 2014-12-04 MTN Dubai Limited +mtn + +// mtr : 2015-03-12 MTR Corporation Limited +mtr + +// mutual : 2015-04-02 Northwestern Mutual MU TLD Registry, LLC +mutual + +// nab : 2015-08-20 National Australia Bank Limited +nab + +// nagoya : 2013-10-24 GMO Registry, Inc. +nagoya + +// nationwide : 2015-07-23 Nationwide Mutual Insurance Company +nationwide + +// natura : 2015-03-12 NATURA COSMÉTICOS S.A. +natura + +// navy : 2014-03-06 Dog Beach, LLC +navy + +// nba : 2015-07-31 NBA REGISTRY, LLC +nba + +// nec : 2015-01-08 NEC Corporation +nec + +// netbank : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +netbank + +// netflix : 2015-06-18 Netflix, Inc. +netflix + +// network : 2013-11-14 Binky Moon, LLC +network + +// neustar : 2013-12-05 NeuStar, Inc. +neustar + +// new : 2014-01-30 Charleston Road Registry Inc. +new + +// newholland : 2015-09-03 CNH Industrial N.V. +newholland + +// news : 2014-12-18 Dog Beach, LLC +news + +// next : 2015-06-18 Next plc +next + +// nextdirect : 2015-06-18 Next plc +nextdirect + +// nexus : 2014-07-24 Charleston Road Registry Inc. +nexus + +// nfl : 2015-07-23 NFL Reg Ops LLC +nfl + +// ngo : 2014-03-06 Public Interest Registry +ngo + +// nhk : 2014-02-13 Japan Broadcasting Corporation (NHK) +nhk + +// nico : 2014-12-04 DWANGO Co., Ltd. +nico + +// nike : 2015-07-23 NIKE, Inc. +nike + +// nikon : 2015-05-21 NIKON CORPORATION +nikon + +// ninja : 2013-11-07 Dog Beach, LLC +ninja + +// nissan : 2014-03-27 NISSAN MOTOR CO., LTD. +nissan + +// nissay : 2015-10-29 Nippon Life Insurance Company +nissay + +// nokia : 2015-01-08 Nokia Corporation +nokia + +// northwesternmutual : 2015-06-18 Northwestern Mutual Registry, LLC +northwesternmutual + +// norton : 2014-12-04 NortonLifeLock Inc. +norton + +// now : 2015-06-25 Amazon Registry Services, Inc. +now + +// nowruz : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +nowruz + +// nowtv : 2015-05-14 Starbucks (HK) Limited +nowtv + +// nra : 2014-05-22 NRA Holdings Company, INC. +nra + +// nrw : 2013-11-21 Minds + Machines GmbH +nrw + +// ntt : 2014-10-31 NIPPON TELEGRAPH AND TELEPHONE CORPORATION +ntt + +// nyc : 2014-01-23 The City of New York by and through the New York City Department of Information Technology & Telecommunications +nyc + +// obi : 2014-09-25 OBI Group Holding SE & Co. KGaA +obi + +// observer : 2015-04-30 Top Level Spectrum, Inc. +observer + +// off : 2015-07-23 Johnson Shareholdings, Inc. +off + +// office : 2015-03-12 Microsoft Corporation +office + +// okinawa : 2013-12-05 BRregistry, Inc. +okinawa + +// olayan : 2015-05-14 Crescent Holding GmbH +olayan + +// olayangroup : 2015-05-14 Crescent Holding GmbH +olayangroup + +// oldnavy : 2015-07-31 The Gap, Inc. +oldnavy + +// ollo : 2015-06-04 Dish DBS Corporation +ollo + +// omega : 2015-01-08 The Swatch Group Ltd +omega + +// one : 2014-11-07 One.com A/S +one + +// ong : 2014-03-06 Public Interest Registry +ong + +// onl : 2013-09-16 iRegistry GmbH +onl + +// online : 2015-01-15 DotOnline Inc. +online + +// onyourside : 2015-07-23 Nationwide Mutual Insurance Company +onyourside + +// ooo : 2014-01-09 INFIBEAM AVENUES LIMITED +ooo + +// open : 2015-07-31 American Express Travel Related Services Company, Inc. +open + +// oracle : 2014-06-19 Oracle Corporation +oracle + +// orange : 2015-03-12 Orange Brand Services Limited +orange + +// organic : 2014-03-27 Afilias Limited +organic + +// origins : 2015-10-01 The Estée Lauder Companies Inc. +origins + +// osaka : 2014-09-04 Osaka Registry Co., Ltd. +osaka + +// otsuka : 2013-10-11 Otsuka Holdings Co., Ltd. +otsuka + +// ott : 2015-06-04 Dish DBS Corporation +ott + +// ovh : 2014-01-16 MédiaBC +ovh + +// page : 2014-12-04 Charleston Road Registry Inc. +page + +// panasonic : 2015-07-30 Panasonic Corporation +panasonic + +// paris : 2014-01-30 City of Paris +paris + +// pars : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +pars + +// partners : 2013-12-05 Binky Moon, LLC +partners + +// parts : 2013-12-05 Binky Moon, LLC +parts + +// party : 2014-09-11 Blue Sky Registry Limited +party + +// passagens : 2015-03-05 Travel Reservations SRL +passagens + +// pay : 2015-08-27 Amazon Registry Services, Inc. +pay + +// pccw : 2015-05-14 PCCW Enterprises Limited +pccw + +// pet : 2015-05-07 Afilias Limited +pet + +// pfizer : 2015-09-11 Pfizer Inc. +pfizer + +// pharmacy : 2014-06-19 National Association of Boards of Pharmacy +pharmacy + +// phd : 2016-07-28 Charleston Road Registry Inc. +phd + +// philips : 2014-11-07 Koninklijke Philips N.V. +philips + +// phone : 2016-06-02 Dish DBS Corporation +phone + +// photo : 2013-11-14 UNR Corp. +photo + +// photography : 2013-09-20 Binky Moon, LLC +photography + +// photos : 2013-10-17 Binky Moon, LLC +photos + +// physio : 2014-05-01 PhysBiz Pty Ltd +physio + +// pics : 2013-11-14 UNR Corp. +pics + +// pictet : 2014-06-26 Pictet Europe S.A. +pictet + +// pictures : 2014-03-06 Binky Moon, LLC +pictures + +// pid : 2015-01-08 Top Level Spectrum, Inc. +pid + +// pin : 2014-12-18 Amazon Registry Services, Inc. +pin + +// ping : 2015-06-11 Ping Registry Provider, Inc. +ping + +// pink : 2013-10-01 Afilias Limited +pink + +// pioneer : 2015-07-16 Pioneer Corporation +pioneer + +// pizza : 2014-06-26 Binky Moon, LLC +pizza + +// place : 2014-04-24 Binky Moon, LLC +place + +// play : 2015-03-05 Charleston Road Registry Inc. +play + +// playstation : 2015-07-02 Sony Interactive Entertainment Inc. +playstation + +// plumbing : 2013-09-10 Binky Moon, LLC +plumbing + +// plus : 2015-02-05 Binky Moon, LLC +plus + +// pnc : 2015-07-02 PNC Domain Co., LLC +pnc + +// pohl : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +pohl + +// poker : 2014-07-03 Afilias Limited +poker + +// politie : 2015-08-20 Politie Nederland +politie + +// porn : 2014-10-16 ICM Registry PN LLC +porn + +// pramerica : 2015-07-30 Prudential Financial, Inc. +pramerica + +// praxi : 2013-12-05 Praxi S.p.A. +praxi + +// press : 2014-04-03 DotPress Inc. +press + +// prime : 2015-06-25 Amazon Registry Services, Inc. +prime + +// prod : 2014-01-23 Charleston Road Registry Inc. +prod + +// productions : 2013-12-05 Binky Moon, LLC +productions + +// prof : 2014-07-24 Charleston Road Registry Inc. +prof + +// progressive : 2015-07-23 Progressive Casualty Insurance Company +progressive + +// promo : 2014-12-18 Afilias Limited +promo + +// properties : 2013-12-05 Binky Moon, LLC +properties + +// property : 2014-05-22 UNR Corp. +property + +// protection : 2015-04-23 XYZ.COM LLC +protection + +// pru : 2015-07-30 Prudential Financial, Inc. +pru + +// prudential : 2015-07-30 Prudential Financial, Inc. +prudential + +// pub : 2013-12-12 Dog Beach, LLC +pub + +// pwc : 2015-10-29 PricewaterhouseCoopers LLP +pwc + +// qpon : 2013-11-14 dotCOOL, Inc. +qpon + +// quebec : 2013-12-19 PointQuébec Inc +quebec + +// quest : 2015-03-26 XYZ.COM LLC +quest + +// qvc : 2015-07-30 QVC, Inc. +qvc + +// racing : 2014-12-04 Premier Registry Limited +racing + +// radio : 2016-07-21 European Broadcasting Union (EBU) +radio + +// raid : 2015-07-23 Johnson Shareholdings, Inc. +raid + +// read : 2014-12-18 Amazon Registry Services, Inc. +read + +// realestate : 2015-09-11 dotRealEstate LLC +realestate + +// realtor : 2014-05-29 Real Estate Domains LLC +realtor + +// realty : 2015-03-19 Fegistry, LLC +realty + +// recipes : 2013-10-17 Binky Moon, LLC +recipes + +// red : 2013-11-07 Afilias Limited +red + +// redstone : 2014-10-31 Redstone Haute Couture Co., Ltd. +redstone + +// redumbrella : 2015-03-26 Travelers TLD, LLC +redumbrella + +// rehab : 2014-03-06 Dog Beach, LLC +rehab + +// reise : 2014-03-13 Binky Moon, LLC +reise + +// reisen : 2014-03-06 Binky Moon, LLC +reisen + +// reit : 2014-09-04 National Association of Real Estate Investment Trusts, Inc. +reit + +// reliance : 2015-04-02 Reliance Industries Limited +reliance + +// ren : 2013-12-12 ZDNS International Limited +ren + +// rent : 2014-12-04 XYZ.COM LLC +rent + +// rentals : 2013-12-05 Binky Moon, LLC +rentals + +// repair : 2013-11-07 Binky Moon, LLC +repair + +// report : 2013-12-05 Binky Moon, LLC +report + +// republican : 2014-03-20 Dog Beach, LLC +republican + +// rest : 2013-12-19 Punto 2012 Sociedad Anonima Promotora de Inversion de Capital Variable +rest + +// restaurant : 2014-07-03 Binky Moon, LLC +restaurant + +// review : 2014-11-20 dot Review Limited +review + +// reviews : 2013-09-13 Dog Beach, LLC +reviews + +// rexroth : 2015-06-18 Robert Bosch GMBH +rexroth + +// rich : 2013-11-21 iRegistry GmbH +rich + +// richardli : 2015-05-14 Pacific Century Asset Management (HK) Limited +richardli + +// ricoh : 2014-11-20 Ricoh Company, Ltd. +ricoh + +// ril : 2015-04-02 Reliance Industries Limited +ril + +// rio : 2014-02-27 Empresa Municipal de Informática SA - IPLANRIO +rio + +// rip : 2014-07-10 Dog Beach, LLC +rip + +// rmit : 2015-11-19 Royal Melbourne Institute of Technology +rmit + +// rocher : 2014-12-18 Ferrero Trading Lux S.A. +rocher + +// rocks : 2013-11-14 Dog Beach, LLC +rocks + +// rodeo : 2013-12-19 Minds + Machines Group Limited +rodeo + +// rogers : 2015-08-06 Rogers Communications Canada Inc. +rogers + +// room : 2014-12-18 Amazon Registry Services, Inc. +room + +// rsvp : 2014-05-08 Charleston Road Registry Inc. +rsvp + +// rugby : 2016-12-15 World Rugby Strategic Developments Limited +rugby + +// ruhr : 2013-10-02 regiodot GmbH & Co. KG +ruhr + +// run : 2015-03-19 Binky Moon, LLC +run + +// rwe : 2015-04-02 RWE AG +rwe + +// ryukyu : 2014-01-09 BRregistry, Inc. +ryukyu + +// saarland : 2013-12-12 dotSaarland GmbH +saarland + +// safe : 2014-12-18 Amazon Registry Services, Inc. +safe + +// safety : 2015-01-08 Safety Registry Services, LLC. +safety + +// sakura : 2014-12-18 SAKURA Internet Inc. +sakura + +// sale : 2014-10-16 Dog Beach, LLC +sale + +// salon : 2014-12-11 Binky Moon, LLC +salon + +// samsclub : 2015-07-31 Wal-Mart Stores, Inc. +samsclub + +// samsung : 2014-04-03 SAMSUNG SDS CO., LTD +samsung + +// sandvik : 2014-11-13 Sandvik AB +sandvik + +// sandvikcoromant : 2014-11-07 Sandvik AB +sandvikcoromant + +// sanofi : 2014-10-09 Sanofi +sanofi + +// sap : 2014-03-27 SAP AG +sap + +// sarl : 2014-07-03 Binky Moon, LLC +sarl + +// sas : 2015-04-02 Research IP LLC +sas + +// save : 2015-06-25 Amazon Registry Services, Inc. +save + +// saxo : 2014-10-31 Saxo Bank A/S +saxo + +// sbi : 2015-03-12 STATE BANK OF INDIA +sbi + +// sbs : 2014-11-07 SPECIAL BROADCASTING SERVICE CORPORATION +sbs + +// sca : 2014-03-13 SVENSKA CELLULOSA AKTIEBOLAGET SCA (publ) +sca + +// scb : 2014-02-20 The Siam Commercial Bank Public Company Limited ("SCB") +scb + +// schaeffler : 2015-08-06 Schaeffler Technologies AG & Co. KG +schaeffler + +// schmidt : 2014-04-03 SCHMIDT GROUPE S.A.S. +schmidt + +// scholarships : 2014-04-24 Scholarships.com, LLC +scholarships + +// school : 2014-12-18 Binky Moon, LLC +school + +// schule : 2014-03-06 Binky Moon, LLC +schule + +// schwarz : 2014-09-18 Schwarz Domains und Services GmbH & Co. KG +schwarz + +// science : 2014-09-11 dot Science Limited +science + +// scjohnson : 2015-07-23 Johnson Shareholdings, Inc. +scjohnson + +// scot : 2014-01-23 Dot Scot Registry Limited +scot + +// search : 2016-06-09 Charleston Road Registry Inc. +search + +// seat : 2014-05-22 SEAT, S.A. (Sociedad Unipersonal) +seat + +// secure : 2015-08-27 Amazon Registry Services, Inc. +secure + +// security : 2015-05-14 XYZ.COM LLC +security + +// seek : 2014-12-04 Seek Limited +seek + +// select : 2015-10-08 Registry Services, LLC +select + +// sener : 2014-10-24 Sener Ingeniería y Sistemas, S.A. +sener + +// services : 2014-02-27 Binky Moon, LLC +services + +// ses : 2015-07-23 SES +ses + +// seven : 2015-08-06 Seven West Media Ltd +seven + +// sew : 2014-07-17 SEW-EURODRIVE GmbH & Co KG +sew + +// sex : 2014-11-13 ICM Registry SX LLC +sex + +// sexy : 2013-09-11 UNR Corp. +sexy + +// sfr : 2015-08-13 Societe Francaise du Radiotelephone - SFR +sfr + +// shangrila : 2015-09-03 Shangri‐La International Hotel Management Limited +shangrila + +// sharp : 2014-05-01 Sharp Corporation +sharp + +// shaw : 2015-04-23 Shaw Cablesystems G.P. +shaw + +// shell : 2015-07-30 Shell Information Technology International Inc +shell + +// shia : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +shia + +// shiksha : 2013-11-14 Afilias Limited +shiksha + +// shoes : 2013-10-02 Binky Moon, LLC +shoes + +// shop : 2016-04-08 GMO Registry, Inc. +shop + +// shopping : 2016-03-31 Binky Moon, LLC +shopping + +// shouji : 2015-01-08 Beijing Qihu Keji Co., Ltd. +shouji + +// show : 2015-03-05 Binky Moon, LLC +show + +// showtime : 2015-08-06 CBS Domains Inc. +showtime + +// silk : 2015-06-25 Amazon Registry Services, Inc. +silk + +// sina : 2015-03-12 Sina Corporation +sina + +// singles : 2013-08-27 Binky Moon, LLC +singles + +// site : 2015-01-15 DotSite Inc. +site + +// ski : 2015-04-09 Afilias Limited +ski + +// skin : 2015-01-15 XYZ.COM LLC +skin + +// sky : 2014-06-19 Sky International AG +sky + +// skype : 2014-12-18 Microsoft Corporation +skype + +// sling : 2015-07-30 DISH Technologies L.L.C. +sling + +// smart : 2015-07-09 Smart Communications, Inc. (SMART) +smart + +// smile : 2014-12-18 Amazon Registry Services, Inc. +smile + +// sncf : 2015-02-19 Société Nationale des Chemins de fer Francais S N C F +sncf + +// soccer : 2015-03-26 Binky Moon, LLC +soccer + +// social : 2013-11-07 Dog Beach, LLC +social + +// softbank : 2015-07-02 SoftBank Group Corp. +softbank + +// software : 2014-03-20 Dog Beach, LLC +software + +// sohu : 2013-12-19 Sohu.com Limited +sohu + +// solar : 2013-11-07 Binky Moon, LLC +solar + +// solutions : 2013-11-07 Binky Moon, LLC +solutions + +// song : 2015-02-26 Amazon Registry Services, Inc. +song + +// sony : 2015-01-08 Sony Corporation +sony + +// soy : 2014-01-23 Charleston Road Registry Inc. +soy + +// spa : 2019-09-19 Asia Spa and Wellness Promotion Council Limited +spa + +// space : 2014-04-03 DotSpace Inc. +space + +// sport : 2017-11-16 Global Association of International Sports Federations (GAISF) +sport + +// spot : 2015-02-26 Amazon Registry Services, Inc. +spot + +// spreadbetting : 2014-12-11 Dotspreadbetting Registry Limited +spreadbetting + +// srl : 2015-05-07 InterNetX, Corp +srl + +// stada : 2014-11-13 STADA Arzneimittel AG +stada + +// staples : 2015-07-30 Staples, Inc. +staples + +// star : 2015-01-08 Star India Private Limited +star + +// statebank : 2015-03-12 STATE BANK OF INDIA +statebank + +// statefarm : 2015-07-30 State Farm Mutual Automobile Insurance Company +statefarm + +// stc : 2014-10-09 Saudi Telecom Company +stc + +// stcgroup : 2014-10-09 Saudi Telecom Company +stcgroup + +// stockholm : 2014-12-18 Stockholms kommun +stockholm + +// storage : 2014-12-22 XYZ.COM LLC +storage + +// store : 2015-04-09 DotStore Inc. +store + +// stream : 2016-01-08 dot Stream Limited +stream + +// studio : 2015-02-11 Dog Beach, LLC +studio + +// study : 2014-12-11 OPEN UNIVERSITIES AUSTRALIA PTY LTD +study + +// style : 2014-12-04 Binky Moon, LLC +style + +// sucks : 2014-12-22 Vox Populi Registry Ltd. +sucks + +// supplies : 2013-12-19 Binky Moon, LLC +supplies + +// supply : 2013-12-19 Binky Moon, LLC +supply + +// support : 2013-10-24 Binky Moon, LLC +support + +// surf : 2014-01-09 Minds + Machines Group Limited +surf + +// surgery : 2014-03-20 Binky Moon, LLC +surgery + +// suzuki : 2014-02-20 SUZUKI MOTOR CORPORATION +suzuki + +// swatch : 2015-01-08 The Swatch Group Ltd +swatch + +// swiftcover : 2015-07-23 Swiftcover Insurance Services Limited +swiftcover + +// swiss : 2014-10-16 Swiss Confederation +swiss + +// sydney : 2014-09-18 State of New South Wales, Department of Premier and Cabinet +sydney + +// systems : 2013-11-07 Binky Moon, LLC +systems + +// tab : 2014-12-04 Tabcorp Holdings Limited +tab + +// taipei : 2014-07-10 Taipei City Government +taipei + +// talk : 2015-04-09 Amazon Registry Services, Inc. +talk + +// taobao : 2015-01-15 Alibaba Group Holding Limited +taobao + +// target : 2015-07-31 Target Domain Holdings, LLC +target + +// tatamotors : 2015-03-12 Tata Motors Ltd +tatamotors + +// tatar : 2014-04-24 Limited Liability Company "Coordination Center of Regional Domain of Tatarstan Republic" +tatar + +// tattoo : 2013-08-30 UNR Corp. +tattoo + +// tax : 2014-03-20 Binky Moon, LLC +tax + +// taxi : 2015-03-19 Binky Moon, LLC +taxi + +// tci : 2014-09-12 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +tci + +// tdk : 2015-06-11 TDK Corporation +tdk + +// team : 2015-03-05 Binky Moon, LLC +team + +// tech : 2015-01-30 Personals TLD Inc. +tech + +// technology : 2013-09-13 Binky Moon, LLC +technology + +// temasek : 2014-08-07 Temasek Holdings (Private) Limited +temasek + +// tennis : 2014-12-04 Binky Moon, LLC +tennis + +// teva : 2015-07-02 Teva Pharmaceutical Industries Limited +teva + +// thd : 2015-04-02 Home Depot Product Authority, LLC +thd + +// theater : 2015-03-19 Binky Moon, LLC +theater + +// theatre : 2015-05-07 XYZ.COM LLC +theatre + +// tiaa : 2015-07-23 Teachers Insurance and Annuity Association of America +tiaa + +// tickets : 2015-02-05 Accent Media Limited +tickets + +// tienda : 2013-11-14 Binky Moon, LLC +tienda + +// tiffany : 2015-01-30 Tiffany and Company +tiffany + +// tips : 2013-09-20 Binky Moon, LLC +tips + +// tires : 2014-11-07 Binky Moon, LLC +tires + +// tirol : 2014-04-24 punkt Tirol GmbH +tirol + +// tjmaxx : 2015-07-16 The TJX Companies, Inc. +tjmaxx + +// tjx : 2015-07-16 The TJX Companies, Inc. +tjx + +// tkmaxx : 2015-07-16 The TJX Companies, Inc. +tkmaxx + +// tmall : 2015-01-15 Alibaba Group Holding Limited +tmall + +// today : 2013-09-20 Binky Moon, LLC +today + +// tokyo : 2013-11-13 GMO Registry, Inc. +tokyo + +// tools : 2013-11-21 Binky Moon, LLC +tools + +// top : 2014-03-20 .TOP Registry +top + +// toray : 2014-12-18 Toray Industries, Inc. +toray + +// toshiba : 2014-04-10 TOSHIBA Corporation +toshiba + +// total : 2015-08-06 Total SA +total + +// tours : 2015-01-22 Binky Moon, LLC +tours + +// town : 2014-03-06 Binky Moon, LLC +town + +// toyota : 2015-04-23 TOYOTA MOTOR CORPORATION +toyota + +// toys : 2014-03-06 Binky Moon, LLC +toys + +// trade : 2014-01-23 Elite Registry Limited +trade + +// trading : 2014-12-11 Dottrading Registry Limited +trading + +// training : 2013-11-07 Binky Moon, LLC +training + +// travel : 2015-10-09 Dog Beach, LLC +travel + +// travelchannel : 2015-07-02 Lifestyle Domain Holdings, Inc. +travelchannel + +// travelers : 2015-03-26 Travelers TLD, LLC +travelers + +// travelersinsurance : 2015-03-26 Travelers TLD, LLC +travelersinsurance + +// trust : 2014-10-16 UNR Corp. +trust + +// trv : 2015-03-26 Travelers TLD, LLC +trv + +// tube : 2015-06-11 Latin American Telecom LLC +tube + +// tui : 2014-07-03 TUI AG +tui + +// tunes : 2015-02-26 Amazon Registry Services, Inc. +tunes + +// tushu : 2014-12-18 Amazon Registry Services, Inc. +tushu + +// tvs : 2015-02-19 T V SUNDRAM IYENGAR & SONS LIMITED +tvs + +// ubank : 2015-08-20 National Australia Bank Limited +ubank + +// ubs : 2014-12-11 UBS AG +ubs + +// unicom : 2015-10-15 China United Network Communications Corporation Limited +unicom + +// university : 2014-03-06 Binky Moon, LLC +university + +// uno : 2013-09-11 DotSite Inc. +uno + +// uol : 2014-05-01 UBN INTERNET LTDA. +uol + +// ups : 2015-06-25 UPS Market Driver, Inc. +ups + +// vacations : 2013-12-05 Binky Moon, LLC +vacations + +// vana : 2014-12-11 Lifestyle Domain Holdings, Inc. +vana + +// vanguard : 2015-09-03 The Vanguard Group, Inc. +vanguard + +// vegas : 2014-01-16 Dot Vegas, Inc. +vegas + +// ventures : 2013-08-27 Binky Moon, LLC +ventures + +// verisign : 2015-08-13 VeriSign, Inc. +verisign + +// versicherung : 2014-03-20 tldbox GmbH +versicherung + +// vet : 2014-03-06 Dog Beach, LLC +vet + +// viajes : 2013-10-17 Binky Moon, LLC +viajes + +// video : 2014-10-16 Dog Beach, LLC +video + +// vig : 2015-05-14 VIENNA INSURANCE GROUP AG Wiener Versicherung Gruppe +vig + +// viking : 2015-04-02 Viking River Cruises (Bermuda) Ltd. +viking + +// villas : 2013-12-05 Binky Moon, LLC +villas + +// vin : 2015-06-18 Binky Moon, LLC +vin + +// vip : 2015-01-22 Minds + Machines Group Limited +vip + +// virgin : 2014-09-25 Virgin Enterprises Limited +virgin + +// visa : 2015-07-30 Visa Worldwide Pte. Limited +visa + +// vision : 2013-12-05 Binky Moon, LLC +vision + +// viva : 2014-11-07 Saudi Telecom Company +viva + +// vivo : 2015-07-31 Telefonica Brasil S.A. +vivo + +// vlaanderen : 2014-02-06 DNS.be vzw +vlaanderen + +// vodka : 2013-12-19 Minds + Machines Group Limited +vodka + +// volkswagen : 2015-05-14 Volkswagen Group of America Inc. +volkswagen + +// volvo : 2015-11-12 Volvo Holding Sverige Aktiebolag +volvo + +// vote : 2013-11-21 Monolith Registry LLC +vote + +// voting : 2013-11-13 Valuetainment Corp. +voting + +// voto : 2013-11-21 Monolith Registry LLC +voto + +// voyage : 2013-08-27 Binky Moon, LLC +voyage + +// vuelos : 2015-03-05 Travel Reservations SRL +vuelos + +// wales : 2014-05-08 Nominet UK +wales + +// walmart : 2015-07-31 Wal-Mart Stores, Inc. +walmart + +// walter : 2014-11-13 Sandvik AB +walter + +// wang : 2013-10-24 Zodiac Wang Limited +wang + +// wanggou : 2014-12-18 Amazon Registry Services, Inc. +wanggou + +// watch : 2013-11-14 Binky Moon, LLC +watch + +// watches : 2014-12-22 Richemont DNS Inc. +watches + +// weather : 2015-01-08 International Business Machines Corporation +weather + +// weatherchannel : 2015-03-12 International Business Machines Corporation +weatherchannel + +// webcam : 2014-01-23 dot Webcam Limited +webcam + +// weber : 2015-06-04 Saint-Gobain Weber SA +weber + +// website : 2014-04-03 DotWebsite Inc. +website + +// wedding : 2014-04-24 Minds + Machines Group Limited +wedding + +// weibo : 2015-03-05 Sina Corporation +weibo + +// weir : 2015-01-29 Weir Group IP Limited +weir + +// whoswho : 2014-02-20 Who's Who Registry +whoswho + +// wien : 2013-10-28 punkt.wien GmbH +wien + +// wiki : 2013-11-07 Top Level Design, LLC +wiki + +// williamhill : 2014-03-13 William Hill Organization Limited +williamhill + +// win : 2014-11-20 First Registry Limited +win + +// windows : 2014-12-18 Microsoft Corporation +windows + +// wine : 2015-06-18 Binky Moon, LLC +wine + +// winners : 2015-07-16 The TJX Companies, Inc. +winners + +// wme : 2014-02-13 William Morris Endeavor Entertainment, LLC +wme + +// wolterskluwer : 2015-08-06 Wolters Kluwer N.V. +wolterskluwer + +// woodside : 2015-07-09 Woodside Petroleum Limited +woodside + +// work : 2013-12-19 Minds + Machines Group Limited +work + +// works : 2013-11-14 Binky Moon, LLC +works + +// world : 2014-06-12 Binky Moon, LLC +world + +// wow : 2015-10-08 Amazon Registry Services, Inc. +wow + +// wtc : 2013-12-19 World Trade Centers Association, Inc. +wtc + +// wtf : 2014-03-06 Binky Moon, LLC +wtf + +// xbox : 2014-12-18 Microsoft Corporation +xbox + +// xerox : 2014-10-24 Xerox DNHC LLC +xerox + +// xfinity : 2015-07-09 Comcast IP Holdings I, LLC +xfinity + +// xihuan : 2015-01-08 Beijing Qihu Keji Co., Ltd. +xihuan + +// xin : 2014-12-11 Elegant Leader Limited +xin + +// xn--11b4c3d : 2015-01-15 VeriSign Sarl +कॉम + +// xn--1ck2e1b : 2015-02-26 Amazon Registry Services, Inc. +セール + +// xn--1qqw23a : 2014-01-09 Guangzhou YU Wei Information Technology Co., Ltd. +佛山 + +// xn--30rr7y : 2014-06-12 Excellent First Limited +慈善 + +// xn--3bst00m : 2013-09-13 Eagle Horizon Limited +集团 + +// xn--3ds443g : 2013-09-08 TLD REGISTRY LIMITED OY +在线 + +// xn--3oq18vl8pn36a : 2015-07-02 Volkswagen (China) Investment Co., Ltd. +大众汽车 + +// xn--3pxu8k : 2015-01-15 VeriSign Sarl +点看 + +// xn--42c2d9a : 2015-01-15 VeriSign Sarl +คอม + +// xn--45q11c : 2013-11-21 Zodiac Gemini Ltd +八卦 + +// xn--4gbrim : 2013-10-04 Fans TLD Limited +موقع + +// xn--55qw42g : 2013-11-08 China Organizational Name Administration Center +公益 + +// xn--55qx5d : 2013-11-14 China Internet Network Information Center (CNNIC) +公司 + +// xn--5su34j936bgsg : 2015-09-03 Shangri‐La International Hotel Management Limited +香格里拉 + +// xn--5tzm5g : 2014-12-22 Global Website TLD Asia Limited +网站 + +// xn--6frz82g : 2013-09-23 Afilias Limited +移动 + +// xn--6qq986b3xl : 2013-09-13 Tycoon Treasure Limited +我爱你 + +// xn--80adxhks : 2013-12-19 Foundation for Assistance for Internet Technologies and Infrastructure Development (FAITID) +москва + +// xn--80aqecdr1a : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +католик + +// xn--80asehdb : 2013-07-14 CORE Association +онлайн + +// xn--80aswg : 2013-07-14 CORE Association +сайт + +// xn--8y0a063a : 2015-03-26 China United Network Communications Corporation Limited +联通 + +// xn--9dbq2a : 2015-01-15 VeriSign Sarl +קום + +// xn--9et52u : 2014-06-12 RISE VICTORY LIMITED +时尚 + +// xn--9krt00a : 2015-03-12 Sina Corporation +微博 + +// xn--b4w605ferd : 2014-08-07 Temasek Holdings (Private) Limited +淡马锡 + +// xn--bck1b9a5dre4c : 2015-02-26 Amazon Registry Services, Inc. +ファッション + +// xn--c1avg : 2013-11-14 Public Interest Registry +орг + +// xn--c2br7g : 2015-01-15 VeriSign Sarl +नेट + +// xn--cck2b3b : 2015-02-26 Amazon Registry Services, Inc. +ストア + +// xn--cckwcxetd : 2019-12-19 Amazon Registry Services, Inc. +アマゾン + +// xn--cg4bki : 2013-09-27 SAMSUNG SDS CO., LTD +삼성 + +// xn--czr694b : 2014-01-16 Internet DotTrademark Organisation Limited +商标 + +// xn--czrs0t : 2013-12-19 Binky Moon, LLC +商店 + +// xn--czru2d : 2013-11-21 Zodiac Aquarius Limited +商城 + +// xn--d1acj3b : 2013-11-20 The Foundation for Network Initiatives “The Smart Internet” +дети + +// xn--eckvdtc9d : 2014-12-18 Amazon Registry Services, Inc. +ポイント + +// xn--efvy88h : 2014-08-22 Guangzhou YU Wei Information Technology Co., Ltd. +新闻 + +// xn--fct429k : 2015-04-09 Amazon Registry Services, Inc. +家電 + +// xn--fhbei : 2015-01-15 VeriSign Sarl +كوم + +// xn--fiq228c5hs : 2013-09-08 TLD REGISTRY LIMITED OY +中文网 + +// xn--fiq64b : 2013-10-14 CITIC Group Corporation +中信 + +// xn--fjq720a : 2014-05-22 Binky Moon, LLC +娱乐 + +// xn--flw351e : 2014-07-31 Charleston Road Registry Inc. +谷歌 + +// xn--fzys8d69uvgm : 2015-05-14 PCCW Enterprises Limited +電訊盈科 + +// xn--g2xx48c : 2015-01-30 Nawang Heli(Xiamen) Network Service Co., LTD. +购物 + +// xn--gckr3f0f : 2015-02-26 Amazon Registry Services, Inc. +クラウド + +// xn--gk3at1e : 2015-10-08 Amazon Registry Services, Inc. +通販 + +// xn--hxt814e : 2014-05-15 Zodiac Taurus Limited +网店 + +// xn--i1b6b1a6a2e : 2013-11-14 Public Interest Registry +संगठन + +// xn--imr513n : 2014-12-11 Internet DotTrademark Organisation Limited +餐厅 + +// xn--io0a7i : 2013-11-14 China Internet Network Information Center (CNNIC) +网络 + +// xn--j1aef : 2015-01-15 VeriSign Sarl +ком + +// xn--jlq480n2rg : 2019-12-19 Amazon Registry Services, Inc. +亚马逊 + +// xn--jlq61u9w7b : 2015-01-08 Nokia Corporation +诺基亚 + +// xn--jvr189m : 2015-02-26 Amazon Registry Services, Inc. +食品 + +// xn--kcrx77d1x4a : 2014-11-07 Koninklijke Philips N.V. +飞利浦 + +// xn--kput3i : 2014-02-13 Beijing RITT-Net Technology Development Co., Ltd +手机 + +// xn--mgba3a3ejt : 2014-11-20 Aramco Services Company +ارامكو + +// xn--mgba7c0bbn0a : 2015-05-14 Crescent Holding GmbH +العليان + +// xn--mgbaakc7dvf : 2015-09-03 Emirates Telecommunications Corporation (trading as Etisalat) +اتصالات + +// xn--mgbab2bd : 2013-10-31 CORE Association +بازار + +// xn--mgbca7dzdo : 2015-07-30 Abu Dhabi Systems and Information Centre +ابوظبي + +// xn--mgbi4ecexp : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +كاثوليك + +// xn--mgbt3dhd : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +همراه + +// xn--mk1bu44c : 2015-01-15 VeriSign Sarl +닷컴 + +// xn--mxtq1m : 2014-03-06 Net-Chinese Co., Ltd. +政府 + +// xn--ngbc5azd : 2013-07-13 International Domain Registry Pty. Ltd. +شبكة + +// xn--ngbe9e0a : 2014-12-04 Kuwait Finance House +بيتك + +// xn--ngbrx : 2015-11-12 League of Arab States +عرب + +// xn--nqv7f : 2013-11-14 Public Interest Registry +机构 + +// xn--nqv7fs00ema : 2013-11-14 Public Interest Registry +组织机构 + +// xn--nyqy26a : 2014-11-07 Stable Tone Limited +健康 + +// xn--otu796d : 2017-08-06 Jiang Yu Liang Cai Technology Company Limited +招聘 + +// xn--p1acf : 2013-12-12 Rusnames Limited +рус + +// xn--pssy2u : 2015-01-15 VeriSign Sarl +大拿 + +// xn--q9jyb4c : 2013-09-17 Charleston Road Registry Inc. +みんな + +// xn--qcka1pmc : 2014-07-31 Charleston Road Registry Inc. +グーグル + +// xn--rhqv96g : 2013-09-11 Stable Tone Limited +世界 + +// xn--rovu88b : 2015-02-26 Amazon Registry Services, Inc. +書籍 + +// xn--ses554g : 2014-01-16 KNET Co., Ltd. +网址 + +// xn--t60b56a : 2015-01-15 VeriSign Sarl +닷넷 + +// xn--tckwe : 2015-01-15 VeriSign Sarl +コム + +// xn--tiq49xqyj : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +天主教 + +// xn--unup4y : 2013-07-14 Binky Moon, LLC +游戏 + +// xn--vermgensberater-ctb : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +vermögensberater + +// xn--vermgensberatung-pwb : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +vermögensberatung + +// xn--vhquv : 2013-08-27 Binky Moon, LLC +企业 + +// xn--vuq861b : 2014-10-16 Beijing Tele-info Network Technology Co., Ltd. +信息 + +// xn--w4r85el8fhu5dnra : 2015-04-30 Kerry Trading Co. Limited +嘉里大酒店 + +// xn--w4rs40l : 2015-07-30 Kerry Trading Co. Limited +嘉里 + +// xn--xhq521b : 2013-11-14 Guangzhou YU Wei Information Technology Co., Ltd. +广东 + +// xn--zfr164b : 2013-11-08 China Organizational Name Administration Center +政务 + +// xyz : 2013-12-05 XYZ.COM LLC +xyz + +// yachts : 2014-01-09 XYZ.COM LLC +yachts + +// yahoo : 2015-04-02 Yahoo! Domain Services Inc. +yahoo + +// yamaxun : 2014-12-18 Amazon Registry Services, Inc. +yamaxun + +// yandex : 2014-04-10 Yandex Europe B.V. +yandex + +// yodobashi : 2014-11-20 YODOBASHI CAMERA CO.,LTD. +yodobashi + +// yoga : 2014-05-29 Minds + Machines Group Limited +yoga + +// yokohama : 2013-12-12 GMO Registry, Inc. +yokohama + +// you : 2015-04-09 Amazon Registry Services, Inc. +you + +// youtube : 2014-05-01 Charleston Road Registry Inc. +youtube + +// yun : 2015-01-08 Beijing Qihu Keji Co., Ltd. +yun + +// zappos : 2015-06-25 Amazon Registry Services, Inc. +zappos + +// zara : 2014-11-07 Industria de Diseño Textil, S.A. (INDITEX, S.A.) +zara + +// zero : 2014-12-18 Amazon Registry Services, Inc. +zero + +// zip : 2014-05-08 Charleston Road Registry Inc. +zip + +// zone : 2013-11-14 Binky Moon, LLC +zone + +// zuerich : 2014-11-07 Kanton Zürich (Canton of Zurich) +zuerich + + +// ===END ICANN DOMAINS=== +// ===BEGIN PRIVATE DOMAINS=== +// (Note: these are in alphabetical order by company name) + +// 1GB LLC : https://www.1gb.ua/ +// Submitted by 1GB LLC +cc.ua +inf.ua +ltd.ua + +// 611coin : https://611project.org/ +611.to + +// Adobe : https://www.adobe.com/ +// Submitted by Ian Boston +adobeaemcloud.com +adobeaemcloud.net +*.dev.adobeaemcloud.com + +// Agnat sp. z o.o. : https://domena.pl +// Submitted by Przemyslaw Plewa +beep.pl + +// alboto.ca : http://alboto.ca +// Submitted by Anton Avramov +barsy.ca + +// Alces Software Ltd : http://alces-software.com +// Submitted by Mark J. Titorenko +*.compute.estate +*.alces.network + +// all-inkl.com : https://all-inkl.com +// Submitted by Werner Kaltofen +kasserver.com + +// Altervista: https://www.altervista.org +// Submitted by Carlo Cannas +altervista.org + +// alwaysdata : https://www.alwaysdata.com +// Submitted by Cyril +alwaysdata.net + +// Amazon CloudFront : https://aws.amazon.com/cloudfront/ +// Submitted by Donavan Miller +cloudfront.net + +// Amazon Elastic Compute Cloud : https://aws.amazon.com/ec2/ +// Submitted by Luke Wells +*.compute.amazonaws.com +*.compute-1.amazonaws.com +*.compute.amazonaws.com.cn +us-east-1.amazonaws.com + +// Amazon Elastic Beanstalk : https://aws.amazon.com/elasticbeanstalk/ +// Submitted by Luke Wells +cn-north-1.eb.amazonaws.com.cn +cn-northwest-1.eb.amazonaws.com.cn +elasticbeanstalk.com +ap-northeast-1.elasticbeanstalk.com +ap-northeast-2.elasticbeanstalk.com +ap-northeast-3.elasticbeanstalk.com +ap-south-1.elasticbeanstalk.com +ap-southeast-1.elasticbeanstalk.com +ap-southeast-2.elasticbeanstalk.com +ca-central-1.elasticbeanstalk.com +eu-central-1.elasticbeanstalk.com +eu-west-1.elasticbeanstalk.com +eu-west-2.elasticbeanstalk.com +eu-west-3.elasticbeanstalk.com +sa-east-1.elasticbeanstalk.com +us-east-1.elasticbeanstalk.com +us-east-2.elasticbeanstalk.com +us-gov-west-1.elasticbeanstalk.com +us-west-1.elasticbeanstalk.com +us-west-2.elasticbeanstalk.com + +// Amazon Elastic Load Balancing : https://aws.amazon.com/elasticloadbalancing/ +// Submitted by Luke Wells +*.elb.amazonaws.com +*.elb.amazonaws.com.cn + +// Amazon S3 : https://aws.amazon.com/s3/ +// Submitted by Luke Wells +s3.amazonaws.com +s3-ap-northeast-1.amazonaws.com +s3-ap-northeast-2.amazonaws.com +s3-ap-south-1.amazonaws.com +s3-ap-southeast-1.amazonaws.com +s3-ap-southeast-2.amazonaws.com +s3-ca-central-1.amazonaws.com +s3-eu-central-1.amazonaws.com +s3-eu-west-1.amazonaws.com +s3-eu-west-2.amazonaws.com +s3-eu-west-3.amazonaws.com +s3-external-1.amazonaws.com +s3-fips-us-gov-west-1.amazonaws.com +s3-sa-east-1.amazonaws.com +s3-us-gov-west-1.amazonaws.com +s3-us-east-2.amazonaws.com +s3-us-west-1.amazonaws.com +s3-us-west-2.amazonaws.com +s3.ap-northeast-2.amazonaws.com +s3.ap-south-1.amazonaws.com +s3.cn-north-1.amazonaws.com.cn +s3.ca-central-1.amazonaws.com +s3.eu-central-1.amazonaws.com +s3.eu-west-2.amazonaws.com +s3.eu-west-3.amazonaws.com +s3.us-east-2.amazonaws.com +s3.dualstack.ap-northeast-1.amazonaws.com +s3.dualstack.ap-northeast-2.amazonaws.com +s3.dualstack.ap-south-1.amazonaws.com +s3.dualstack.ap-southeast-1.amazonaws.com +s3.dualstack.ap-southeast-2.amazonaws.com +s3.dualstack.ca-central-1.amazonaws.com +s3.dualstack.eu-central-1.amazonaws.com +s3.dualstack.eu-west-1.amazonaws.com +s3.dualstack.eu-west-2.amazonaws.com +s3.dualstack.eu-west-3.amazonaws.com +s3.dualstack.sa-east-1.amazonaws.com +s3.dualstack.us-east-1.amazonaws.com +s3.dualstack.us-east-2.amazonaws.com +s3-website-us-east-1.amazonaws.com +s3-website-us-west-1.amazonaws.com +s3-website-us-west-2.amazonaws.com +s3-website-ap-northeast-1.amazonaws.com +s3-website-ap-southeast-1.amazonaws.com +s3-website-ap-southeast-2.amazonaws.com +s3-website-eu-west-1.amazonaws.com +s3-website-sa-east-1.amazonaws.com +s3-website.ap-northeast-2.amazonaws.com +s3-website.ap-south-1.amazonaws.com +s3-website.ca-central-1.amazonaws.com +s3-website.eu-central-1.amazonaws.com +s3-website.eu-west-2.amazonaws.com +s3-website.eu-west-3.amazonaws.com +s3-website.us-east-2.amazonaws.com + +// Amsterdam Wireless: https://www.amsterdamwireless.nl/ +// Submitted by Imre Jonk +amsw.nl + +// Amune : https://amune.org/ +// Submitted by Team Amune +t3l3p0rt.net +tele.amune.org + +// Apigee : https://apigee.com/ +// Submitted by Apigee Security Team +apigee.io + +// Aptible : https://www.aptible.com/ +// Submitted by Thomas Orozco +on-aptible.com + +// ASEINet : https://www.aseinet.com/ +// Submitted by Asei SEKIGUCHI +user.aseinet.ne.jp +gv.vc +d.gv.vc + +// Asociación Amigos de la Informática "Euskalamiga" : http://encounter.eus/ +// Submitted by Hector Martin +user.party.eus + +// Association potager.org : https://potager.org/ +// Submitted by Lunar +pimienta.org +poivron.org +potager.org +sweetpepper.org + +// ASUSTOR Inc. : http://www.asustor.com +// Submitted by Vincent Tseng +myasustor.com + +// AVM : https://avm.de +// Submitted by Andreas Weise +myfritz.net + +// AW AdvisorWebsites.com Software Inc : https://advisorwebsites.com +// Submitted by James Kennedy +*.awdev.ca +*.advisor.ws + +// b-data GmbH : https://www.b-data.io +// Submitted by Olivier Benz +b-data.io + +// backplane : https://www.backplane.io +// Submitted by Anthony Voutas +backplaneapp.io + +// Balena : https://www.balena.io +// Submitted by Petros Angelatos +balena-devices.com + +// Banzai Cloud +// Submitted by Janos Matyas +*.banzai.cloud +app.banzaicloud.io +*.backyards.banzaicloud.io + + +// BetaInABox +// Submitted by Adrian +betainabox.com + +// BinaryLane : http://www.binarylane.com +// Submitted by Nathan O'Sullivan +bnr.la + +// Blackbaud, Inc. : https://www.blackbaud.com +// Submitted by Paul Crowder +blackbaudcdn.net + +// Blatech : http://www.blatech.net +// Submitted by Luke Bratch +of.je + +// Boomla : https://boomla.com +// Submitted by Tibor Halter +boomla.net + +// Boxfuse : https://boxfuse.com +// Submitted by Axel Fontaine +boxfuse.io + +// bplaced : https://www.bplaced.net/ +// Submitted by Miroslav Bozic +square7.ch +bplaced.com +bplaced.de +square7.de +bplaced.net +square7.net + +// BrowserSafetyMark +// Submitted by Dave Tharp +browsersafetymark.io + +// Bytemark Hosting : https://www.bytemark.co.uk +// Submitted by Paul Cammish +uk0.bigv.io +dh.bytemark.co.uk +vm.bytemark.co.uk + +// callidomus : https://www.callidomus.com/ +// Submitted by Marcus Popp +mycd.eu + +// Carrd : https://carrd.co +// Submitted by AJ +carrd.co +crd.co +uwu.ai + +// CentralNic : http://www.centralnic.com/names/domains +// Submitted by registry +ae.org +br.com +cn.com +com.de +com.se +de.com +eu.com +gb.net +hu.net +jp.net +jpn.com +mex.com +ru.com +sa.com +se.net +uk.com +uk.net +us.com +za.bz +za.com + +// No longer operated by CentralNic, these entries should be adopted and/or removed by current operators +// Submitted by Gavin Brown +ar.com +gb.com +hu.com +kr.com +no.com +qc.com +uy.com + +// Africa.com Web Solutions Ltd : https://registry.africa.com +// Submitted by Gavin Brown +africa.com + +// iDOT Services Limited : http://www.domain.gr.com +// Submitted by Gavin Brown +gr.com + +// Radix FZC : http://domains.in.net +// Submitted by Gavin Brown +in.net +web.in + +// US REGISTRY LLC : http://us.org +// Submitted by Gavin Brown +us.org + +// co.com Registry, LLC : https://registry.co.com +// Submitted by Gavin Brown +co.com + +// Roar Domains LLC : https://roar.basketball/ +// Submitted by Gavin Brown +aus.basketball +nz.basketball + +// BRS Media : https://brsmedia.com/ +// Submitted by Gavin Brown +radio.am +radio.fm + +// Globe Hosting SRL : https://www.globehosting.com/ +// Submitted by Gavin Brown +co.ro +shop.ro + +// c.la : http://www.c.la/ +c.la + +// certmgr.org : https://certmgr.org +// Submitted by B. Blechschmidt +certmgr.org + +// Civilized Discourse Construction Kit, Inc. : https://www.discourse.org/ +// Submitted by Rishabh Nambiar & Michael Brown +discourse.group +discourse.team + +// ClearVox : http://www.clearvox.nl/ +// Submitted by Leon Rowland +virtueeldomein.nl + +// Clever Cloud : https://www.clever-cloud.com/ +// Submitted by Quentin Adam +cleverapps.io + +// Clerk : https://www.clerk.dev +// Submitted by Colin Sidoti +*.lcl.dev +*.stg.dev + +// Clic2000 : https://clic2000.fr +// Submitted by Mathilde Blanchemanche +clic2000.net + +// Cloud66 : https://www.cloud66.com/ +// Submitted by Khash Sajadi +c66.me +cloud66.ws +cloud66.zone + +// CloudAccess.net : https://www.cloudaccess.net/ +// Submitted by Pawel Panek +jdevcloud.com +wpdevcloud.com +cloudaccess.host +freesite.host +cloudaccess.net + +// cloudControl : https://www.cloudcontrol.com/ +// Submitted by Tobias Wilken +cloudcontrolled.com +cloudcontrolapp.com + +// Cloudera, Inc. : https://www.cloudera.com/ +// Submitted by Philip Langdale +cloudera.site + +// Cloudflare, Inc. : https://www.cloudflare.com/ +// Submitted by Cloudflare Team +pages.dev +trycloudflare.com +workers.dev + +// Clovyr : https://clovyr.io +// Submitted by Patrick Nielsen +wnext.app + +// co.ca : http://registry.co.ca/ +co.ca + +// Co & Co : https://co-co.nl/ +// Submitted by Govert Versluis +*.otap.co + +// i-registry s.r.o. : http://www.i-registry.cz/ +// Submitted by Martin Semrad +co.cz + +// CDN77.com : http://www.cdn77.com +// Submitted by Jan Krpes +c.cdn77.org +cdn77-ssl.net +r.cdn77.net +rsc.cdn77.org +ssl.origin.cdn77-secure.org + +// Cloud DNS Ltd : http://www.cloudns.net +// Submitted by Aleksander Hristov +cloudns.asia +cloudns.biz +cloudns.club +cloudns.cc +cloudns.eu +cloudns.in +cloudns.info +cloudns.org +cloudns.pro +cloudns.pw +cloudns.us + +// CNPY : https://cnpy.gdn +// Submitted by Angelo Gladding +cnpy.gdn + +// CoDNS B.V. +co.nl +co.no + +// Combell.com : https://www.combell.com +// Submitted by Thomas Wouters +webhosting.be +hosting-cluster.nl + +// Coordination Center for TLD RU and XN--P1AI : https://cctld.ru/en/domains/domens_ru/reserved/ +// Submitted by George Georgievsky +ac.ru +edu.ru +gov.ru +int.ru +mil.ru +test.ru + +// COSIMO GmbH : http://www.cosimo.de +// Submitted by Rene Marticke +dyn.cosidns.de +dynamisches-dns.de +dnsupdater.de +internet-dns.de +l-o-g-i-n.de +dynamic-dns.info +feste-ip.net +knx-server.net +static-access.net + +// Craynic, s.r.o. : http://www.craynic.com/ +// Submitted by Ales Krajnik +realm.cz + +// Cryptonomic : https://cryptonomic.net/ +// Submitted by Andrew Cady +*.cryptonomic.net + +// Cupcake : https://cupcake.io/ +// Submitted by Jonathan Rudenberg +cupcake.is + +// Curv UG : https://curv-labs.de/ +// Submitted by Marvin Wiesner +curv.dev + +// Customer OCI - Oracle Dyn https://cloud.oracle.com/home https://dyn.com/dns/ +// Submitted by Gregory Drake +// Note: This is intended to also include customer-oci.com due to wildcards implicitly including the current label +*.customer-oci.com +*.oci.customer-oci.com +*.ocp.customer-oci.com +*.ocs.customer-oci.com + +// cyon GmbH : https://www.cyon.ch/ +// Submitted by Dominic Luechinger +cyon.link +cyon.site + +// Danger Science Group: https://dangerscience.com/ +// Submitted by Skylar MacDonald +fnwk.site +folionetwork.site +platform0.app + +// Daplie, Inc : https://daplie.com +// Submitted by AJ ONeal +daplie.me +localhost.daplie.me + +// Datto, Inc. : https://www.datto.com/ +// Submitted by Philipp Heckel +dattolocal.com +dattorelay.com +dattoweb.com +mydatto.com +dattolocal.net +mydatto.net + +// Dansk.net : http://www.dansk.net/ +// Submitted by Anani Voule +biz.dk +co.dk +firm.dk +reg.dk +store.dk + +// dappnode.io : https://dappnode.io/ +// Submitted by Abel Boldu / DAppNode Team +dyndns.dappnode.io + +// dapps.earth : https://dapps.earth/ +// Submitted by Daniil Burdakov +*.dapps.earth +*.bzz.dapps.earth + +// Dark, Inc. : https://darklang.com +// Submitted by Paul Biggar +builtwithdark.com + +// Datawire, Inc : https://www.datawire.io +// Submitted by Richard Li +edgestack.me + +// Debian : https://www.debian.org/ +// Submitted by Peter Palfrader / Debian Sysadmin Team +debian.net + +// deSEC : https://desec.io/ +// Submitted by Peter Thomassen +dedyn.io + +// DNS Africa Ltd https://dns.business +// Submitted by Calvin Browne +jozi.biz + +// DNShome : https://www.dnshome.de/ +// Submitted by Norbert Auler +dnshome.de + +// DotArai : https://www.dotarai.com/ +// Submitted by Atsadawat Netcharadsang +online.th +shop.th + +// DrayTek Corp. : https://www.draytek.com/ +// Submitted by Paul Fang +drayddns.com + +// DreamHost : http://www.dreamhost.com/ +// Submitted by Andrew Farmer +dreamhosters.com + +// Drobo : http://www.drobo.com/ +// Submitted by Ricardo Padilha +mydrobo.com + +// Drud Holdings, LLC. : https://www.drud.com/ +// Submitted by Kevin Bridges +drud.io +drud.us + +// DuckDNS : http://www.duckdns.org/ +// Submitted by Richard Harper +duckdns.org + +// Bip : https://bip.sh +// Submitted by Joel Kennedy +bip.sh + +// bitbridge.net : Submitted by Craig Welch, abeliidev@gmail.com +bitbridge.net + +// dy.fi : http://dy.fi/ +// Submitted by Heikki Hannikainen +dy.fi +tunk.org + +// DynDNS.com : http://www.dyndns.com/services/dns/dyndns/ +dyndns-at-home.com +dyndns-at-work.com +dyndns-blog.com +dyndns-free.com +dyndns-home.com +dyndns-ip.com +dyndns-mail.com +dyndns-office.com +dyndns-pics.com +dyndns-remote.com +dyndns-server.com +dyndns-web.com +dyndns-wiki.com +dyndns-work.com +dyndns.biz +dyndns.info +dyndns.org +dyndns.tv +at-band-camp.net +ath.cx +barrel-of-knowledge.info +barrell-of-knowledge.info +better-than.tv +blogdns.com +blogdns.net +blogdns.org +blogsite.org +boldlygoingnowhere.org +broke-it.net +buyshouses.net +cechire.com +dnsalias.com +dnsalias.net +dnsalias.org +dnsdojo.com +dnsdojo.net +dnsdojo.org +does-it.net +doesntexist.com +doesntexist.org +dontexist.com +dontexist.net +dontexist.org +doomdns.com +doomdns.org +dvrdns.org +dyn-o-saur.com +dynalias.com +dynalias.net +dynalias.org +dynathome.net +dyndns.ws +endofinternet.net +endofinternet.org +endoftheinternet.org +est-a-la-maison.com +est-a-la-masion.com +est-le-patron.com +est-mon-blogueur.com +for-better.biz +for-more.biz +for-our.info +for-some.biz +for-the.biz +forgot.her.name +forgot.his.name +from-ak.com +from-al.com +from-ar.com +from-az.net +from-ca.com +from-co.net +from-ct.com +from-dc.com +from-de.com +from-fl.com +from-ga.com +from-hi.com +from-ia.com +from-id.com +from-il.com +from-in.com +from-ks.com +from-ky.com +from-la.net +from-ma.com +from-md.com +from-me.org +from-mi.com +from-mn.com +from-mo.com +from-ms.com +from-mt.com +from-nc.com +from-nd.com +from-ne.com +from-nh.com +from-nj.com +from-nm.com +from-nv.com +from-ny.net +from-oh.com +from-ok.com +from-or.com +from-pa.com +from-pr.com +from-ri.com +from-sc.com +from-sd.com +from-tn.com +from-tx.com +from-ut.com +from-va.com +from-vt.com +from-wa.com +from-wi.com +from-wv.com +from-wy.com +ftpaccess.cc +fuettertdasnetz.de +game-host.org +game-server.cc +getmyip.com +gets-it.net +go.dyndns.org +gotdns.com +gotdns.org +groks-the.info +groks-this.info +ham-radio-op.net +here-for-more.info +hobby-site.com +hobby-site.org +home.dyndns.org +homedns.org +homeftp.net +homeftp.org +homeip.net +homelinux.com +homelinux.net +homelinux.org +homeunix.com +homeunix.net +homeunix.org +iamallama.com +in-the-band.net +is-a-anarchist.com +is-a-blogger.com +is-a-bookkeeper.com +is-a-bruinsfan.org +is-a-bulls-fan.com +is-a-candidate.org +is-a-caterer.com +is-a-celticsfan.org +is-a-chef.com +is-a-chef.net +is-a-chef.org +is-a-conservative.com +is-a-cpa.com +is-a-cubicle-slave.com +is-a-democrat.com +is-a-designer.com +is-a-doctor.com +is-a-financialadvisor.com +is-a-geek.com +is-a-geek.net +is-a-geek.org +is-a-green.com +is-a-guru.com +is-a-hard-worker.com +is-a-hunter.com +is-a-knight.org +is-a-landscaper.com +is-a-lawyer.com +is-a-liberal.com +is-a-libertarian.com +is-a-linux-user.org +is-a-llama.com +is-a-musician.com +is-a-nascarfan.com +is-a-nurse.com +is-a-painter.com +is-a-patsfan.org +is-a-personaltrainer.com +is-a-photographer.com +is-a-player.com +is-a-republican.com +is-a-rockstar.com +is-a-socialist.com +is-a-soxfan.org +is-a-student.com +is-a-teacher.com +is-a-techie.com +is-a-therapist.com +is-an-accountant.com +is-an-actor.com +is-an-actress.com +is-an-anarchist.com +is-an-artist.com +is-an-engineer.com +is-an-entertainer.com +is-by.us +is-certified.com +is-found.org +is-gone.com +is-into-anime.com +is-into-cars.com +is-into-cartoons.com +is-into-games.com +is-leet.com +is-lost.org +is-not-certified.com +is-saved.org +is-slick.com +is-uberleet.com +is-very-bad.org +is-very-evil.org +is-very-good.org +is-very-nice.org +is-very-sweet.org +is-with-theband.com +isa-geek.com +isa-geek.net +isa-geek.org +isa-hockeynut.com +issmarterthanyou.com +isteingeek.de +istmein.de +kicks-ass.net +kicks-ass.org +knowsitall.info +land-4-sale.us +lebtimnetz.de +leitungsen.de +likes-pie.com +likescandy.com +merseine.nu +mine.nu +misconfused.org +mypets.ws +myphotos.cc +neat-url.com +office-on-the.net +on-the-web.tv +podzone.net +podzone.org +readmyblog.org +saves-the-whales.com +scrapper-site.net +scrapping.cc +selfip.biz +selfip.com +selfip.info +selfip.net +selfip.org +sells-for-less.com +sells-for-u.com +sells-it.net +sellsyourhome.org +servebbs.com +servebbs.net +servebbs.org +serveftp.net +serveftp.org +servegame.org +shacknet.nu +simple-url.com +space-to-rent.com +stuff-4-sale.org +stuff-4-sale.us +teaches-yoga.com +thruhere.net +traeumtgerade.de +webhop.biz +webhop.info +webhop.net +webhop.org +worse-than.tv +writesthisblog.com + +// ddnss.de : https://www.ddnss.de/ +// Submitted by Robert Niedziela +ddnss.de +dyn.ddnss.de +dyndns.ddnss.de +dyndns1.de +dyn-ip24.de +home-webserver.de +dyn.home-webserver.de +myhome-server.de +ddnss.org + +// Definima : http://www.definima.com/ +// Submitted by Maxence Bitterli +definima.net +definima.io + +// DigitalOcean : https://digitalocean.com/ +// Submitted by Braxton Huggins +ondigitalocean.app + +// dnstrace.pro : https://dnstrace.pro/ +// Submitted by Chris Partridge +bci.dnstrace.pro + +// Dynu.com : https://www.dynu.com/ +// Submitted by Sue Ye +ddnsfree.com +ddnsgeek.com +giize.com +gleeze.com +kozow.com +loseyourip.com +ooguy.com +theworkpc.com +casacam.net +dynu.net +accesscam.org +camdvr.org +freeddns.org +mywire.org +webredirect.org +myddns.rocks +blogsite.xyz + +// dynv6 : https://dynv6.com +// Submitted by Dominik Menke +dynv6.net + +// E4YOU spol. s.r.o. : https://e4you.cz/ +// Submitted by Vladimir Dudr +e4.cz + +// En root‽ : https://en-root.org +// Submitted by Emmanuel Raviart +en-root.fr + +// Enalean SAS: https://www.enalean.com +// Submitted by Thomas Cottier +mytuleap.com + +// ECG Robotics, Inc: https://ecgrobotics.org +// Submitted by +onred.one +staging.onred.one + +// One.com: https://www.one.com/ +// Submitted by Jacob Bunk Nielsen +service.one + +// Enonic : http://enonic.com/ +// Submitted by Erik Kaareng-Sunde +enonic.io +customer.enonic.io + +// EU.org https://eu.org/ +// Submitted by Pierre Beyssac +eu.org +al.eu.org +asso.eu.org +at.eu.org +au.eu.org +be.eu.org +bg.eu.org +ca.eu.org +cd.eu.org +ch.eu.org +cn.eu.org +cy.eu.org +cz.eu.org +de.eu.org +dk.eu.org +edu.eu.org +ee.eu.org +es.eu.org +fi.eu.org +fr.eu.org +gr.eu.org +hr.eu.org +hu.eu.org +ie.eu.org +il.eu.org +in.eu.org +int.eu.org +is.eu.org +it.eu.org +jp.eu.org +kr.eu.org +lt.eu.org +lu.eu.org +lv.eu.org +mc.eu.org +me.eu.org +mk.eu.org +mt.eu.org +my.eu.org +net.eu.org +ng.eu.org +nl.eu.org +no.eu.org +nz.eu.org +paris.eu.org +pl.eu.org +pt.eu.org +q-a.eu.org +ro.eu.org +ru.eu.org +se.eu.org +si.eu.org +sk.eu.org +tr.eu.org +uk.eu.org +us.eu.org + +// Evennode : http://www.evennode.com/ +// Submitted by Michal Kralik +eu-1.evennode.com +eu-2.evennode.com +eu-3.evennode.com +eu-4.evennode.com +us-1.evennode.com +us-2.evennode.com +us-3.evennode.com +us-4.evennode.com + +// eDirect Corp. : https://hosting.url.com.tw/ +// Submitted by C.S. chang +twmail.cc +twmail.net +twmail.org +mymailer.com.tw +url.tw + +// Fabrica Technologies, Inc. : https://www.fabrica.dev/ +// Submitted by Eric Jiang +onfabrica.com + +// Facebook, Inc. +// Submitted by Peter Ruibal +apps.fbsbx.com + +// FAITID : https://faitid.org/ +// Submitted by Maxim Alzoba +// https://www.flexireg.net/stat_info +ru.net +adygeya.ru +bashkiria.ru +bir.ru +cbg.ru +com.ru +dagestan.ru +grozny.ru +kalmykia.ru +kustanai.ru +marine.ru +mordovia.ru +msk.ru +mytis.ru +nalchik.ru +nov.ru +pyatigorsk.ru +spb.ru +vladikavkaz.ru +vladimir.ru +abkhazia.su +adygeya.su +aktyubinsk.su +arkhangelsk.su +armenia.su +ashgabad.su +azerbaijan.su +balashov.su +bashkiria.su +bryansk.su +bukhara.su +chimkent.su +dagestan.su +east-kazakhstan.su +exnet.su +georgia.su +grozny.su +ivanovo.su +jambyl.su +kalmykia.su +kaluga.su +karacol.su +karaganda.su +karelia.su +khakassia.su +krasnodar.su +kurgan.su +kustanai.su +lenug.su +mangyshlak.su +mordovia.su +msk.su +murmansk.su +nalchik.su +navoi.su +north-kazakhstan.su +nov.su +obninsk.su +penza.su +pokrovsk.su +sochi.su +spb.su +tashkent.su +termez.su +togliatti.su +troitsk.su +tselinograd.su +tula.su +tuva.su +vladikavkaz.su +vladimir.su +vologda.su + +// Fancy Bits, LLC : http://getchannels.com +// Submitted by Aman Gupta +channelsdvr.net +u.channelsdvr.net + +// Fastly Inc. : http://www.fastly.com/ +// Submitted by Fastly Security +fastly-terrarium.com +fastlylb.net +map.fastlylb.net +freetls.fastly.net +map.fastly.net +a.prod.fastly.net +global.prod.fastly.net +a.ssl.fastly.net +b.ssl.fastly.net +global.ssl.fastly.net + +// FASTVPS EESTI OU : https://fastvps.ru/ +// Submitted by Likhachev Vasiliy +fastvps-server.com +fastvps.host +myfast.host +fastvps.site +myfast.space + +// Featherhead : https://featherhead.xyz/ +// Submitted by Simon Menke +fhapp.xyz + +// Fedora : https://fedoraproject.org/ +// submitted by Patrick Uiterwijk +fedorainfracloud.org +fedorapeople.org +cloud.fedoraproject.org +app.os.fedoraproject.org +app.os.stg.fedoraproject.org + +// FearWorks Media Ltd. : https://fearworksmedia.co.uk +// submitted by Keith Fairley +conn.uk +copro.uk +couk.me +ukco.me + +// Fermax : https://fermax.com/ +// submitted by Koen Van Isterdael +mydobiss.com + +// FH Muenster : https://www.fh-muenster.de +// Submitted by Robin Naundorf +fh-muenster.io + +// Filegear Inc. : https://www.filegear.com +// Submitted by Jason Zhu +filegear.me +filegear-au.me +filegear-de.me +filegear-gb.me +filegear-ie.me +filegear-jp.me +filegear-sg.me + +// Firebase, Inc. +// Submitted by Chris Raynor +firebaseapp.com + +// fly.io: https://fly.io +// Submitted by Kurt Mackey +fly.dev +edgeapp.net +shw.io + +// Flynn : https://flynn.io +// Submitted by Jonathan Rudenberg +flynnhosting.net + +// Frederik Braun https://frederik-braun.com +// Submitted by Frederik Braun +0e.vc + +// Freebox : http://www.freebox.fr +// Submitted by Romain Fliedel +freebox-os.com +freeboxos.com +fbx-os.fr +fbxos.fr +freebox-os.fr +freeboxos.fr + +// freedesktop.org : https://www.freedesktop.org +// Submitted by Daniel Stone +freedesktop.org + +// FunkFeuer - Verein zur Förderung freier Netze : https://www.funkfeuer.at +// Submitted by Daniel A. Maierhofer +wien.funkfeuer.at + +// Futureweb OG : http://www.futureweb.at +// Submitted by Andreas Schnederle-Wagner +*.futurecms.at +*.ex.futurecms.at +*.in.futurecms.at +futurehosting.at +futuremailing.at +*.ex.ortsinfo.at +*.kunden.ortsinfo.at +*.statics.cloud + +// GDS : https://www.gov.uk/service-manual/operations/operating-servicegovuk-subdomains +// Submitted by David Illsley +service.gov.uk + +// Gehirn Inc. : https://www.gehirn.co.jp/ +// Submitted by Kohei YOSHIDA +gehirn.ne.jp +usercontent.jp + +// Gentlent, Inc. : https://www.gentlent.com +// Submitted by Tom Klein +gentapps.com +gentlentapis.com +lab.ms +cdn-edges.net + +// GitHub, Inc. +// Submitted by Patrick Toomey +github.io +githubusercontent.com + +// GitLab, Inc. +// Submitted by Alex Hanselka +gitlab.io + +// Gitplac.si - https://gitplac.si +// Submitted by Aljaž Starc +gitapp.si +gitpage.si + +// Glitch, Inc : https://glitch.com +// Submitted by Mads Hartmann +glitch.me + +// GMO Pepabo, Inc. : https://pepabo.com/ +// Submitted by dojineko +lolipop.io + +// GOV.UK Platform as a Service : https://www.cloud.service.gov.uk/ +// Submitted by Tom Whitwell +cloudapps.digital +london.cloudapps.digital + +// GOV.UK Pay : https://www.payments.service.gov.uk/ +// Submitted by Richard Baker +pymnt.uk + +// UKHomeOffice : https://www.gov.uk/government/organisations/home-office +// Submitted by Jon Shanks +homeoffice.gov.uk + +// GlobeHosting, Inc. +// Submitted by Zoltan Egresi +ro.im + +// GoIP DNS Services : http://www.goip.de +// Submitted by Christian Poulter +goip.de + +// Google, Inc. +// Submitted by Eduardo Vela +run.app +a.run.app +web.app +*.0emm.com +appspot.com +*.r.appspot.com +codespot.com +googleapis.com +googlecode.com +pagespeedmobilizer.com +publishproxy.com +withgoogle.com +withyoutube.com +*.gateway.dev +cloud.goog +translate.goog +cloudfunctions.net + +blogspot.ae +blogspot.al +blogspot.am +blogspot.ba +blogspot.be +blogspot.bg +blogspot.bj +blogspot.ca +blogspot.cf +blogspot.ch +blogspot.cl +blogspot.co.at +blogspot.co.id +blogspot.co.il +blogspot.co.ke +blogspot.co.nz +blogspot.co.uk +blogspot.co.za +blogspot.com +blogspot.com.ar +blogspot.com.au +blogspot.com.br +blogspot.com.by +blogspot.com.co +blogspot.com.cy +blogspot.com.ee +blogspot.com.eg +blogspot.com.es +blogspot.com.mt +blogspot.com.ng +blogspot.com.tr +blogspot.com.uy +blogspot.cv +blogspot.cz +blogspot.de +blogspot.dk +blogspot.fi +blogspot.fr +blogspot.gr +blogspot.hk +blogspot.hr +blogspot.hu +blogspot.ie +blogspot.in +blogspot.is +blogspot.it +blogspot.jp +blogspot.kr +blogspot.li +blogspot.lt +blogspot.lu +blogspot.md +blogspot.mk +blogspot.mr +blogspot.mx +blogspot.my +blogspot.nl +blogspot.no +blogspot.pe +blogspot.pt +blogspot.qa +blogspot.re +blogspot.ro +blogspot.rs +blogspot.ru +blogspot.se +blogspot.sg +blogspot.si +blogspot.sk +blogspot.sn +blogspot.td +blogspot.tw +blogspot.ug +blogspot.vn + +// Aaron Marais' Gitlab pages: https://lab.aaronleem.co.za +// Submitted by Aaron Marais +graphox.us + +// Group 53, LLC : https://www.group53.com +// Submitted by Tyler Todd +awsmppl.com + +// Hakaran group: http://hakaran.cz +// Submited by Arseniy Sokolov +fin.ci +free.hr +caa.li +ua.rs +conf.se + +// Handshake : https://handshake.org +// Submitted by Mike Damm +hs.zone +hs.run + +// Hashbang : https://hashbang.sh +hashbang.sh + +// Hasura : https://hasura.io +// Submitted by Shahidh K Muhammed +hasura.app +hasura-app.io + +// Hepforge : https://www.hepforge.org +// Submitted by David Grellscheid +hepforge.org + +// Heroku : https://www.heroku.com/ +// Submitted by Tom Maher +herokuapp.com +herokussl.com + +// Hibernating Rhinos +// Submitted by Oren Eini +myravendb.com +ravendb.community +ravendb.me +development.run +ravendb.run + +// Hong Kong Productivity Council: https://www.hkpc.org/ +// Submitted by SECaaS Team +secaas.hk + +// HOSTBIP REGISTRY : https://www.hostbip.com/ +// Submitted by Atanunu Igbunuroghene +bpl.biz +orx.biz +ng.city +biz.gl +ng.ink +col.ng +firm.ng +gen.ng +ltd.ng +ngo.ng +ng.school +sch.so + +// HostyHosting (hostyhosting.com) +hostyhosting.io + +// Häkkinen.fi +// Submitted by Eero Häkkinen +häkkinen.fi + +// Ici la Lune : http://www.icilalune.com/ +// Submitted by Simon Morvan +*.moonscale.io +moonscale.net + +// iki.fi +// Submitted by Hannu Aronsson +iki.fi + +// Individual Network Berlin e.V. : https://www.in-berlin.de/ +// Submitted by Christian Seitz +dyn-berlin.de +in-berlin.de +in-brb.de +in-butter.de +in-dsl.de +in-dsl.net +in-dsl.org +in-vpn.de +in-vpn.net +in-vpn.org + +// info.at : http://www.info.at/ +biz.at +info.at + +// info.cx : http://info.cx +// Submitted by Jacob Slater +info.cx + +// Interlegis : http://www.interlegis.leg.br +// Submitted by Gabriel Ferreira +ac.leg.br +al.leg.br +am.leg.br +ap.leg.br +ba.leg.br +ce.leg.br +df.leg.br +es.leg.br +go.leg.br +ma.leg.br +mg.leg.br +ms.leg.br +mt.leg.br +pa.leg.br +pb.leg.br +pe.leg.br +pi.leg.br +pr.leg.br +rj.leg.br +rn.leg.br +ro.leg.br +rr.leg.br +rs.leg.br +sc.leg.br +se.leg.br +sp.leg.br +to.leg.br + +// intermetrics GmbH : https://pixolino.com/ +// Submitted by Wolfgang Schwarz +pixolino.com + +// Internet-Pro, LLP: https://netangels.ru/ +// Submited by Vasiliy Sheredeko +na4u.ru + +// iopsys software solutions AB : https://iopsys.eu/ +// Submitted by Roman Azarenko +iopsys.se + +// IPiFony Systems, Inc. : https://www.ipifony.com/ +// Submitted by Matthew Hardeman +ipifony.net + +// IServ GmbH : https://iserv.eu +// Submitted by Kim-Alexander Brodowski +mein-iserv.de +schulserver.de +test-iserv.de +iserv.dev + +// I-O DATA DEVICE, INC. : http://www.iodata.com/ +// Submitted by Yuji Minagawa +iobb.net + +// Jelastic, Inc. : https://jelastic.com/ +// Submited by Ihor Kolodyuk +mel.cloudlets.com.au +cloud.interhostsolutions.be +users.scale.virtualcloud.com.br +mycloud.by +alp1.ae.flow.ch +appengine.flow.ch +es-1.axarnet.cloud +diadem.cloud +vip.jelastic.cloud +jele.cloud +it1.eur.aruba.jenv-aruba.cloud +it1.jenv-aruba.cloud +it1-eur.jenv-arubabiz.cloud +oxa.cloud +tn.oxa.cloud +uk.oxa.cloud +primetel.cloud +uk.primetel.cloud +ca.reclaim.cloud +uk.reclaim.cloud +us.reclaim.cloud +ch.trendhosting.cloud +de.trendhosting.cloud +jele.club +clicketcloud.com +ams.cloudswitches.com +au.cloudswitches.com +sg.cloudswitches.com +dopaas.com +elastyco.com +nv.elastyco.com +hidora.com +paas.hosted-by-previder.com +rag-cloud.hosteur.com +rag-cloud-ch.hosteur.com +jcloud.ik-server.com +jcloud-ver-jpc.ik-server.com +demo.jelastic.com +kilatiron.com +paas.massivegrid.com +jed.wafaicloud.com +lon.wafaicloud.com +ryd.wafaicloud.com +j.scaleforce.com.cy +jelastic.dogado.eu +paas.leviracloud.eu +fi.cloudplatform.fi +demo.datacenter.fi +paas.datacenter.fi +jele.host +mircloud.host +jele.io +ocs.opusinteractive.io +cloud.unispace.io +cloud-de.unispace.io +cloud-fr1.unispace.io +jc.neen.it +cloud.jelastic.open.tim.it +jcloud.kz +upaas.kazteleport.kz +jl.serv.net.mx +cloudjiffy.net +fra1-de.cloudjiffy.net +west1-us.cloudjiffy.net +ams1.jls.docktera.net +jls-sto1.elastx.net +jls-sto2.elastx.net +jls-sto3.elastx.net +fr-1.paas.massivegrid.net +lon-1.paas.massivegrid.net +lon-2.paas.massivegrid.net +ny-1.paas.massivegrid.net +ny-2.paas.massivegrid.net +sg-1.paas.massivegrid.net +jelastic.saveincloud.net +nordeste-idc.saveincloud.net +j.scaleforce.net +jelastic.tsukaeru.net +atl.jelastic.vps-host.net +njs.jelastic.vps-host.net +unicloud.pl +mircloud.ru +jelastic.regruhosting.ru +enscaled.sg +jele.site +jelastic.team +orangecloud.tn +j.layershift.co.uk +phx.enscaled.us +mircloud.us + +// Jino : https://www.jino.ru +// Submitted by Sergey Ulyashin +myjino.ru +*.hosting.myjino.ru +*.landing.myjino.ru +*.spectrum.myjino.ru +*.vps.myjino.ru + +// Joyent : https://www.joyent.com/ +// Submitted by Brian Bennett +*.triton.zone +*.cns.joyent.com + +// JS.ORG : http://dns.js.org +// Submitted by Stefan Keim +js.org + +// KaasHosting : http://www.kaashosting.nl/ +// Submitted by Wouter Bakker +kaas.gg +khplay.nl + +// Keyweb AG : https://www.keyweb.de +// Submitted by Martin Dannehl +keymachine.de + +// KingHost : https://king.host +// Submitted by Felipe Keller Braz +kinghost.net +uni5.net + +// KnightPoint Systems, LLC : http://www.knightpoint.com/ +// Submitted by Roy Keene +knightpoint.systems + +// KUROKU LTD : https://kuroku.ltd/ +// Submitted by DisposaBoy +oya.to + +// .KRD : http://nic.krd/data/krd/Registration%20Policy.pdf +co.krd +edu.krd + +// LCube - Professional hosting e.K. : https://www.lcube-webhosting.de +// Submitted by Lars Laehn +git-repos.de +lcube-server.de +svn-repos.de + +// Leadpages : https://www.leadpages.net +// Submitted by Greg Dallavalle +leadpages.co +lpages.co +lpusercontent.com + +// Lelux.fi : https://lelux.fi/ +// Submitted by Lelux Admin +lelux.site + +// Lifetime Hosting : https://Lifetime.Hosting/ +// Submitted by Mike Fillator +co.business +co.education +co.events +co.financial +co.network +co.place +co.technology + +// Lightmaker Property Manager, Inc. : https://app.lmpm.com/ +// Submitted by Greg Holland +app.lmpm.com + +// linkyard ldt: https://www.linkyard.ch/ +// Submitted by Mario Siegenthaler +linkyard.cloud +linkyard-cloud.ch + +// Linode : https://linode.com +// Submitted by +members.linode.com +*.nodebalancer.linode.com +*.linodeobjects.com + +// LiquidNet Ltd : http://www.liquidnetlimited.com/ +// Submitted by Victor Velchev +we.bs + +// localzone.xyz +// Submitted by Kenny Niehage +localzone.xyz + +// Log'in Line : https://www.loginline.com/ +// Submitted by Rémi Mach +loginline.app +loginline.dev +loginline.io +loginline.services +loginline.site + +// LubMAN UMCS Sp. z o.o : https://lubman.pl/ +// Submitted by Ireneusz Maliszewski +krasnik.pl +leczna.pl +lubartow.pl +lublin.pl +poniatowa.pl +swidnik.pl + +// Lug.org.uk : https://lug.org.uk +// Submitted by Jon Spriggs +glug.org.uk +lug.org.uk +lugs.org.uk + +// Lukanet Ltd : https://lukanet.com +// Submitted by Anton Avramov +barsy.bg +barsy.co.uk +barsyonline.co.uk +barsycenter.com +barsyonline.com +barsy.club +barsy.de +barsy.eu +barsy.in +barsy.info +barsy.io +barsy.me +barsy.menu +barsy.mobi +barsy.net +barsy.online +barsy.org +barsy.pro +barsy.pub +barsy.shop +barsy.site +barsy.support +barsy.uk + +// Magento Commerce +// Submitted by Damien Tournoud +*.magentosite.cloud + +// May First - People Link : https://mayfirst.org/ +// Submitted by Jamie McClelland +mayfirst.info +mayfirst.org + +// Mail.Ru Group : https://hb.cldmail.ru +// Submitted by Ilya Zaretskiy +hb.cldmail.ru + +// mcpe.me : https://mcpe.me +// Submitted by Noa Heyl +mcpe.me + +// McHost : https://mchost.ru +// Submitted by Evgeniy Subbotin +mcdir.ru +vps.mcdir.ru + +// Memset hosting : https://www.memset.com +// Submitted by Tom Whitwell +miniserver.com +memset.net + +// MetaCentrum, CESNET z.s.p.o. : https://www.metacentrum.cz/en/ +// Submitted by Zdeněk Šustr +*.cloud.metacentrum.cz +custom.metacentrum.cz + +// MetaCentrum, CESNET z.s.p.o. : https://www.metacentrum.cz/en/ +// Submitted by Radim Janča +flt.cloud.muni.cz +usr.cloud.muni.cz + +// Meteor Development Group : https://www.meteor.com/hosting +// Submitted by Pierre Carrier +meteorapp.com +eu.meteorapp.com + +// Michau Enterprises Limited : http://www.co.pl/ +co.pl + +// Microsoft Corporation : http://microsoft.com +// Submitted by Mitch Webster +*.azurecontainer.io +azurewebsites.net +azure-mobile.net +cloudapp.net +azurestaticapps.net +centralus.azurestaticapps.net +eastasia.azurestaticapps.net +eastus2.azurestaticapps.net +westeurope.azurestaticapps.net +westus2.azurestaticapps.net + +// minion.systems : http://minion.systems +// Submitted by Robert Böttinger +csx.cc + +// MobileEducation, LLC : https://joinforte.com +// Submitted by Grayson Martin +forte.id + +// Mozilla Corporation : https://mozilla.com +// Submitted by Ben Francis +mozilla-iot.org + +// Mozilla Foundation : https://mozilla.org/ +// Submitted by glob +bmoattachments.org + +// MSK-IX : https://www.msk-ix.ru/ +// Submitted by Khannanov Roman +net.ru +org.ru +pp.ru + +// Mythic Beasts : https://www.mythic-beasts.com +// Submitted by Paul Cammish +hostedpi.com +customer.mythic-beasts.com +lynx.mythic-beasts.com +ocelot.mythic-beasts.com +onza.mythic-beasts.com +sphinx.mythic-beasts.com +vs.mythic-beasts.com +x.mythic-beasts.com +yali.mythic-beasts.com +cust.retrosnub.co.uk + +// Nabu Casa : https://www.nabucasa.com +// Submitted by Paulus Schoutsen +ui.nabu.casa + +// Names.of.London : https://names.of.london/ +// Submitted by James Stevens or +pony.club +of.fashion +in.london +of.london +from.marketing +with.marketing +for.men +repair.men +and.mom +for.mom +for.one +under.one +for.sale +that.win +from.work +to.work + +// NCTU.ME : https://nctu.me/ +// Submitted by Tocknicsu +nctu.me + +// Netlify : https://www.netlify.com +// Submitted by Jessica Parsons +netlify.app + +// Neustar Inc. +// Submitted by Trung Tran +4u.com + +// ngrok : https://ngrok.com/ +// Submitted by Alan Shreve +ngrok.io + +// Nimbus Hosting Ltd. : https://www.nimbushosting.co.uk/ +// Submitted by Nicholas Ford +nh-serv.co.uk + +// NFSN, Inc. : https://www.NearlyFreeSpeech.NET/ +// Submitted by Jeff Wheelhouse +nfshost.com + +// Now-DNS : https://now-dns.com +// Submitted by Steve Russell +dnsking.ch +mypi.co +n4t.co +001www.com +ddnslive.com +myiphost.com +forumz.info +16-b.it +32-b.it +64-b.it +soundcast.me +tcp4.me +dnsup.net +hicam.net +now-dns.net +ownip.net +vpndns.net +dynserv.org +now-dns.org +x443.pw +now-dns.top +ntdll.top +freeddns.us +crafting.xyz +zapto.xyz + +// nsupdate.info : https://www.nsupdate.info/ +// Submitted by Thomas Waldmann +nsupdate.info +nerdpol.ovh + +// No-IP.com : https://noip.com/ +// Submitted by Deven Reza +blogsyte.com +brasilia.me +cable-modem.org +ciscofreak.com +collegefan.org +couchpotatofries.org +damnserver.com +ddns.me +ditchyourip.com +dnsfor.me +dnsiskinky.com +dvrcam.info +dynns.com +eating-organic.net +fantasyleague.cc +geekgalaxy.com +golffan.us +health-carereform.com +homesecuritymac.com +homesecuritypc.com +hopto.me +ilovecollege.info +loginto.me +mlbfan.org +mmafan.biz +myactivedirectory.com +mydissent.net +myeffect.net +mymediapc.net +mypsx.net +mysecuritycamera.com +mysecuritycamera.net +mysecuritycamera.org +net-freaks.com +nflfan.org +nhlfan.net +no-ip.ca +no-ip.co.uk +no-ip.net +noip.us +onthewifi.com +pgafan.net +point2this.com +pointto.us +privatizehealthinsurance.net +quicksytes.com +read-books.org +securitytactics.com +serveexchange.com +servehumour.com +servep2p.com +servesarcasm.com +stufftoread.com +ufcfan.org +unusualperson.com +workisboring.com +3utilities.com +bounceme.net +ddns.net +ddnsking.com +gotdns.ch +hopto.org +myftp.biz +myftp.org +myvnc.com +no-ip.biz +no-ip.info +no-ip.org +noip.me +redirectme.net +servebeer.com +serveblog.net +servecounterstrike.com +serveftp.com +servegame.com +servehalflife.com +servehttp.com +serveirc.com +serveminecraft.net +servemp3.com +servepics.com +servequake.com +sytes.net +webhop.me +zapto.org + +// NodeArt : https://nodeart.io +// Submitted by Konstantin Nosov +stage.nodeart.io + +// Nodum B.V. : https://nodum.io/ +// Submitted by Wietse Wind +nodum.co +nodum.io + +// Nucleos Inc. : https://nucleos.com +// Submitted by Piotr Zduniak +pcloud.host + +// NYC.mn : http://www.information.nyc.mn +// Submitted by Matthew Brown +nyc.mn + +// NymNom : https://nymnom.com/ +// Submitted by NymNom +nom.ae +nom.af +nom.ai +nom.al +nym.by +nom.bz +nym.bz +nom.cl +nym.ec +nom.gd +nom.ge +nom.gl +nym.gr +nom.gt +nym.gy +nym.hk +nom.hn +nym.ie +nom.im +nom.ke +nym.kz +nym.la +nym.lc +nom.li +nym.li +nym.lt +nym.lu +nom.lv +nym.me +nom.mk +nym.mn +nym.mx +nom.nu +nym.nz +nym.pe +nym.pt +nom.pw +nom.qa +nym.ro +nom.rs +nom.si +nym.sk +nom.st +nym.su +nym.sx +nom.tj +nym.tw +nom.ug +nom.uy +nom.vc +nom.vg + +// Observable, Inc. : https://observablehq.com +// Submitted by Mike Bostock +static.observableusercontent.com + +// Octopodal Solutions, LLC. : https://ulterius.io/ +// Submitted by Andrew Sampson +cya.gg + +// OMG.LOL : +// Submitted by Adam Newbold +omg.lol + +// Omnibond Systems, LLC. : https://www.omnibond.com +// Submitted by Cole Estep +cloudycluster.net + +// OmniWe Limited: https://omniwe.com +// Submitted by Vicary Archangel +omniwe.site + +// One Fold Media : http://www.onefoldmedia.com/ +// Submitted by Eddie Jones +nid.io + +// Open Social : https://www.getopensocial.com/ +// Submitted by Alexander Varwijk +opensocial.site + +// OpenCraft GmbH : http://opencraft.com/ +// Submitted by Sven Marnach +opencraft.hosting + +// Opera Software, A.S.A. +// Submitted by Yngve Pettersen +operaunite.com + +// Oursky Limited : https://skygear.io/ +// Submited by Skygear Developer +skygearapp.com + +// OutSystems +// Submitted by Duarte Santos +outsystemscloud.com + +// OwnProvider GmbH: http://www.ownprovider.com +// Submitted by Jan Moennich +ownprovider.com +own.pm + +// OwO : https://whats-th.is/ +// Submitted by Dean Sheather +*.owo.codes + +// OX : http://www.ox.rs +// Submitted by Adam Grand +ox.rs + +// oy.lc +// Submitted by Charly Coste +oy.lc + +// Pagefog : https://pagefog.com/ +// Submitted by Derek Myers +pgfog.com + +// Pagefront : https://www.pagefronthq.com/ +// Submitted by Jason Kriss +pagefrontapp.com + +// PageXL : https://pagexl.com +// Submitted by Yann Guichard +pagexl.com + +// pcarrier.ca Software Inc: https://pcarrier.ca/ +// Submitted by Pierre Carrier +bar0.net +bar1.net +bar2.net +rdv.to + +// .pl domains (grandfathered) +art.pl +gliwice.pl +krakow.pl +poznan.pl +wroc.pl +zakopane.pl + +// Pantheon Systems, Inc. : https://pantheon.io/ +// Submitted by Gary Dylina +pantheonsite.io +gotpantheon.com + +// Peplink | Pepwave : http://peplink.com/ +// Submitted by Steve Leung +mypep.link + +// Perspecta : https://perspecta.com/ +// Submitted by Kenneth Van Alstyne +perspecta.cloud + +// PE Ulyanov Kirill Sergeevich : https://airy.host +// Submitted by Kirill Ulyanov +lk3.ru +ra-ru.ru +zsew.ru + +// Planet-Work : https://www.planet-work.com/ +// Submitted by Frédéric VANNIÈRE +on-web.fr + +// Platform.sh : https://platform.sh +// Submitted by Nikola Kotur +bc.platform.sh +ent.platform.sh +eu.platform.sh +us.platform.sh +*.platformsh.site + +// Platter: https://platter.dev +// Submitted by Patrick Flor +platter-app.com +platter-app.dev +platterp.us + +// Plesk : https://www.plesk.com/ +// Submitted by Anton Akhtyamov +pdns.page +plesk.page +pleskns.com + +// Port53 : https://port53.io/ +// Submitted by Maximilian Schieder +dyn53.io + +// Positive Codes Technology Company : http://co.bn/faq.html +// Submitted by Zulfais +co.bn + +// prgmr.com : https://prgmr.com/ +// Submitted by Sarah Newman +xen.prgmr.com + +// priv.at : http://www.nic.priv.at/ +// Submitted by registry +priv.at + +// privacytools.io : https://www.privacytools.io/ +// Submitted by Jonah Aragon +prvcy.page + +// Protocol Labs : https://protocol.ai/ +// Submitted by Michael Burns +*.dweb.link + +// Protonet GmbH : http://protonet.io +// Submitted by Martin Meier +protonet.io + +// Publication Presse Communication SARL : https://ppcom.fr +// Submitted by Yaacov Akiba Slama +chirurgiens-dentistes-en-france.fr +byen.site + +// pubtls.org: https://www.pubtls.org +// Submitted by Kor Nielsen +pubtls.org + +// QOTO, Org. +// Submitted by Jeffrey Phillips Freeman +qoto.io + +// Qualifio : https://qualifio.com/ +// Submitted by Xavier De Cock +qualifioapp.com + +// QuickBackend: https://www.quickbackend.com +// Submitted by Dani Biro +qbuser.com + +// Redstar Consultants : https://www.redstarconsultants.com/ +// Submitted by Jons Slemmer +instantcloud.cn + +// Russian Academy of Sciences +// Submitted by Tech Support +ras.ru + +// QA2 +// Submitted by Daniel Dent (https://www.danieldent.com/) +qa2.com + +// QCX +// Submitted by Cassandra Beelen +qcx.io +*.sys.qcx.io + +// QNAP System Inc : https://www.qnap.com +// Submitted by Nick Chang +dev-myqnapcloud.com +alpha-myqnapcloud.com +myqnapcloud.com + +// Quip : https://quip.com +// Submitted by Patrick Linehan +*.quipelements.com + +// Qutheory LLC : http://qutheory.io +// Submitted by Jonas Schwartz +vapor.cloud +vaporcloud.io + +// Rackmaze LLC : https://www.rackmaze.com +// Submitted by Kirill Pertsev +rackmaze.com +rackmaze.net + +// Rakuten Games, Inc : https://dev.viberplay.io +// Submitted by Joshua Zhang +g.vbrplsbx.io + +// Rancher Labs, Inc : https://rancher.com +// Submitted by Vincent Fiduccia +*.on-k3s.io +*.on-rancher.cloud +*.on-rio.io + +// Read The Docs, Inc : https://www.readthedocs.org +// Submitted by David Fischer +readthedocs.io + +// Red Hat, Inc. OpenShift : https://openshift.redhat.com/ +// Submitted by Tim Kramer +rhcloud.com + +// Render : https://render.com +// Submitted by Anurag Goel +app.render.com +onrender.com + +// Repl.it : https://repl.it +// Submitted by Mason Clayton +repl.co +repl.run + +// Resin.io : https://resin.io +// Submitted by Tim Perry +resindevice.io +devices.resinstaging.io + +// RethinkDB : https://www.rethinkdb.com/ +// Submitted by Chris Kastorff +hzc.io + +// Revitalised Limited : http://www.revitalised.co.uk +// Submitted by Jack Price +wellbeingzone.eu +wellbeingzone.co.uk + +// Rochester Institute of Technology : http://www.rit.edu/ +// Submitted by Jennifer Herting +git-pages.rit.edu + +// Sandstorm Development Group, Inc. : https://sandcats.io/ +// Submitted by Asheesh Laroia +sandcats.io + +// SBE network solutions GmbH : https://www.sbe.de/ +// Submitted by Norman Meilick +logoip.de +logoip.com + +// schokokeks.org GbR : https://schokokeks.org/ +// Submitted by Hanno Böck +schokokeks.net + +// Scottish Government: https://www.gov.scot +// Submitted by Martin Ellis +gov.scot + +// Scry Security : http://www.scrysec.com +// Submitted by Shante Adam +scrysec.com + +// Securepoint GmbH : https://www.securepoint.de +// Submitted by Erik Anders +firewall-gateway.com +firewall-gateway.de +my-gateway.de +my-router.de +spdns.de +spdns.eu +firewall-gateway.net +my-firewall.org +myfirewall.org +spdns.org + +// Seidat : https://www.seidat.com +// Submitted by Artem Kondratev +seidat.net + +// Senseering GmbH : https://www.senseering.de +// Submitted by Felix Mönckemeyer +senseering.net + +// Service Online LLC : http://drs.ua/ +// Submitted by Serhii Bulakh +biz.ua +co.ua +pp.ua + +// ShiftEdit : https://shiftedit.net/ +// Submitted by Adam Jimenez +shiftedit.io + +// Shopblocks : http://www.shopblocks.com/ +// Submitted by Alex Bowers +myshopblocks.com + +// Shopit : https://www.shopitcommerce.com/ +// Submitted by Craig McMahon +shopitsite.com + +// shopware AG : https://shopware.com +// Submitted by Jens Küper +shopware.store + +// Siemens Mobility GmbH +// Submitted by Oliver Graebner +mo-siemens.io + +// SinaAppEngine : http://sae.sina.com.cn/ +// Submitted by SinaAppEngine +1kapp.com +appchizi.com +applinzi.com +sinaapp.com +vipsinaapp.com + +// Siteleaf : https://www.siteleaf.com/ +// Submitted by Skylar Challand +siteleaf.net + +// Skyhat : http://www.skyhat.io +// Submitted by Shante Adam +bounty-full.com +alpha.bounty-full.com +beta.bounty-full.com + +// Small Technology Foundation : https://small-tech.org +// Submitted by Aral Balkan +small-web.org + +// Stackhero : https://www.stackhero.io +// Submitted by Adrien Gillon +stackhero-network.com + +// staticland : https://static.land +// Submitted by Seth Vincent +static.land +dev.static.land +sites.static.land + +// Sony Interactive Entertainment LLC : https://sie.com/ +// Submitted by David Coles +playstation-cloud.com + +// SourceLair PC : https://www.sourcelair.com +// Submitted by Antonis Kalipetis +apps.lair.io +*.stolos.io + +// SpaceKit : https://www.spacekit.io/ +// Submitted by Reza Akhavan +spacekit.io + +// SpeedPartner GmbH: https://www.speedpartner.de/ +// Submitted by Stefan Neufeind +customer.speedpartner.de + +// Standard Library : https://stdlib.com +// Submitted by Jacob Lee +api.stdlib.com + +// Storj Labs Inc. : https://storj.io/ +// Submitted by Philip Hutchins +storj.farm + +// Studenten Net Twente : http://www.snt.utwente.nl/ +// Submitted by Silke Hofstra +utwente.io + +// Student-Run Computing Facility : https://www.srcf.net/ +// Submitted by Edwin Balani +soc.srcf.net +user.srcf.net + +// Sub 6 Limited: http://www.sub6.com +// Submitted by Dan Miller +temp-dns.com + +// Swisscom Application Cloud: https://developer.swisscom.com +// Submitted by Matthias.Winzeler +applicationcloud.io +scapp.io + +// Symfony, SAS : https://symfony.com/ +// Submitted by Fabien Potencier +*.s5y.io +*.sensiosite.cloud + +// Syncloud : https://syncloud.org +// Submitted by Boris Rybalkin +syncloud.it + +// Synology, Inc. : https://www.synology.com/ +// Submitted by Rony Weng +diskstation.me +dscloud.biz +dscloud.me +dscloud.mobi +dsmynas.com +dsmynas.net +dsmynas.org +familyds.com +familyds.net +familyds.org +i234.me +myds.me +synology.me +vpnplus.to +direct.quickconnect.to + +// TAIFUN Software AG : http://taifun-software.de +// Submitted by Bjoern Henke +taifun-dns.de + +// TASK geographical domains (www.task.gda.pl/uslugi/dns) +gda.pl +gdansk.pl +gdynia.pl +med.pl +sopot.pl + +// Teckids e.V. : https://www.teckids.org +// Submitted by Dominik George +edugit.org + +// Telebit : https://telebit.cloud +// Submitted by AJ ONeal +telebit.app +telebit.io +*.telebit.xyz + +// The Gwiddle Foundation : https://gwiddlefoundation.org.uk +// Submitted by Joshua Bayfield +gwiddle.co.uk + +// Thingdust AG : https://thingdust.com/ +// Submitted by Adrian Imboden +thingdustdata.com +cust.dev.thingdust.io +cust.disrec.thingdust.io +cust.prod.thingdust.io +cust.testing.thingdust.io +*.firenet.ch +*.svc.firenet.ch + +// Tlon.io : https://tlon.io +// Submitted by Mark Staarink +arvo.network +azimuth.network +tlon.network + +// TownNews.com : http://www.townnews.com +// Submitted by Dustin Ward +bloxcms.com +townnews-staging.com + +// TrafficPlex GmbH : https://www.trafficplex.de/ +// Submitted by Phillipp Röll +12hp.at +2ix.at +4lima.at +lima-city.at +12hp.ch +2ix.ch +4lima.ch +lima-city.ch +trafficplex.cloud +de.cool +12hp.de +2ix.de +4lima.de +lima-city.de +1337.pictures +clan.rip +lima-city.rocks +webspace.rocks +lima.zone + +// TransIP : https://www.transip.nl +// Submitted by Rory Breuk +*.transurl.be +*.transurl.eu +*.transurl.nl + +// TuxFamily : http://tuxfamily.org +// Submitted by TuxFamily administrators +tuxfamily.org + +// TwoDNS : https://www.twodns.de/ +// Submitted by TwoDNS-Support +dd-dns.de +diskstation.eu +diskstation.org +dray-dns.de +draydns.de +dyn-vpn.de +dynvpn.de +mein-vigor.de +my-vigor.de +my-wan.de +syno-ds.de +synology-diskstation.de +synology-ds.de + +// Uberspace : https://uberspace.de +// Submitted by Moritz Werner +uber.space +*.uberspace.de + +// UDR Limited : http://www.udr.hk.com +// Submitted by registry +hk.com +hk.org +ltd.hk +inc.hk + +// United Gameserver GmbH : https://united-gameserver.de +// Submitted by Stefan Schwarz +virtualuser.de +virtual-user.de + +// urown.net : https://urown.net +// Submitted by Hostmaster +urown.cloud +dnsupdate.info + +// .US +// Submitted by Ed Moore +lib.de.us + +// VeryPositive SIA : http://very.lv +// Submitted by Danko Aleksejevs +2038.io + +// Vercel, Inc : https://vercel.com/ +// Submitted by Connor Davis +vercel.app +vercel.dev +now.sh + +// Viprinet Europe GmbH : http://www.viprinet.com +// Submitted by Simon Kissel +router.management + +// Virtual-Info : https://www.virtual-info.info/ +// Submitted by Adnan RIHAN +v-info.info + +// Voorloper.com: https://voorloper.com +// Submitted by Nathan van Bakel +voorloper.cloud + +// Voxel.sh DNS : https://voxel.sh/dns/ +// Submitted by Mia Rehlinger +neko.am +nyaa.am +be.ax +cat.ax +es.ax +eu.ax +gg.ax +mc.ax +us.ax +xy.ax +nl.ci +xx.gl +app.gp +blog.gt +de.gt +to.gt +be.gy +cc.hn +blog.kg +io.kg +jp.kg +tv.kg +uk.kg +us.kg +de.ls +at.md +de.md +jp.md +to.md +uwu.nu +indie.porn +vxl.sh +ch.tc +me.tc +we.tc +nyan.to +at.vg +blog.vu +dev.vu +me.vu + +// V.UA Domain Administrator : https://domain.v.ua/ +// Submitted by Serhii Rostilo +v.ua + +// Waffle Computer Inc., Ltd. : https://docs.waffleinfo.com +// Submitted by Masayuki Note +wafflecell.com + +// WapBlog.ID : https://www.wapblog.id +// Submitted by Fajar Sodik +idnblogger.com +indowapblog.com +bloger.id +wblog.id +wbq.me +fastblog.net + +// WebHare bv: https://www.webhare.com/ +// Submitted by Arnold Hendriks +*.webhare.dev + +// WeDeploy by Liferay, Inc. : https://www.wedeploy.com +// Submitted by Henrique Vicente +wedeploy.io +wedeploy.me +wedeploy.sh + +// Western Digital Technologies, Inc : https://www.wdc.com +// Submitted by Jung Jin +remotewd.com + +// WIARD Enterprises : https://wiardweb.com +// Submitted by Kidd Hustle +pages.wiardweb.com + +// Wikimedia Labs : https://wikitech.wikimedia.org +// Submitted by Arturo Borrero Gonzalez +wmflabs.org +toolforge.org +wmcloud.org + +// WISP : https://wisp.gg +// Submitted by Stepan Fedotov +panel.gg +daemon.panel.gg + +// WoltLab GmbH : https://www.woltlab.com +// Submitted by Tim Düsterhus +myforum.community +community-pro.de +diskussionsbereich.de +community-pro.net +meinforum.net + +// www.com.vc : http://www.com.vc +// Submitted by Li Hui +cn.vu + +// XenonCloud GbR: https://xenoncloud.net +// Submitted by Julian Uphoff +half.host + +// XnBay Technology : http://www.xnbay.com/ +// Submitted by XnBay Developer +xnbay.com +u2.xnbay.com +u2-local.xnbay.com + +// XS4ALL Internet bv : https://www.xs4all.nl/ +// Submitted by Daniel Mostertman +cistron.nl +demon.nl +xs4all.space + +// Yandex.Cloud LLC: https://cloud.yandex.com +// Submitted by Alexander Lodin +yandexcloud.net +storage.yandexcloud.net +website.yandexcloud.net + +// YesCourse Pty Ltd : https://yescourse.com +// Submitted by Atul Bhouraskar +official.academy + +// Yola : https://www.yola.com/ +// Submitted by Stefano Rivera +yolasite.com + +// Yombo : https://yombo.net +// Submitted by Mitch Schwenk +ybo.faith +yombo.me +homelink.one +ybo.party +ybo.review +ybo.science +ybo.trade + +// Yunohost : https://yunohost.org +// Submitted by Valentin Grimaud +nohost.me +noho.st + +// ZaNiC : http://www.za.net/ +// Submitted by registry +za.net +za.org + +// Zine EOOD : https://zine.bg/ +// Submitted by Martin Angelov +bss.design + +// Zitcom A/S : https://www.zitcom.dk +// Submitted by Emil Stahl +basicserver.io +virtualserver.io +enterprisecloud.nu + +// Mintere : https://mintere.com/ +// Submitted by Ben Aubin +mintere.site + +// Cityhost LLC : https://cityhost.ua +// Submitted by Maksym Rivtin +cx.ua + +// WP Engine : https://wpengine.com/ +// Submitted by Michael Smith +// Submitted by Brandon DuRette +wpenginepowered.com +js.wpenginepowered.com + +// Impertrix Solutions : +// Submitted by Zhixiang Zhao +impertrixcdn.com +impertrix.com + +// GignoSystemJapan: http://gsj.bz +// Submitted by GignoSystemJapan +gsj.bz +// ===END PRIVATE DOMAINS=== diff --git a/tests/queries/0_stateless/01601_custom_tld.reference b/tests/queries/0_stateless/01601_custom_tld.reference new file mode 100644 index 00000000000..98b99778396 --- /dev/null +++ b/tests/queries/0_stateless/01601_custom_tld.reference @@ -0,0 +1,11 @@ +no-tld + +foo.there-is-no-such-domain +foo.there-is-no-such-domain +foo +generic +kernel +kernel.biz.ss +difference +biz.ss +kernel.biz.ss diff --git a/tests/queries/0_stateless/01601_custom_tld.sql b/tests/queries/0_stateless/01601_custom_tld.sql new file mode 100644 index 00000000000..6d68299c07d --- /dev/null +++ b/tests/queries/0_stateless/01601_custom_tld.sql @@ -0,0 +1,16 @@ +select 'no-tld'; +select cutToFirstSignificantSubdomainCustom('there-is-no-such-domain', 'public_suffix_list'); +-- even if there is no TLD, 2-nd level by default anyway +-- FIXME: make this behavior optional (so that TLD for host never changed, either empty or something real) +select cutToFirstSignificantSubdomainCustom('foo.there-is-no-such-domain', 'public_suffix_list'); +select cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +select firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); + +select 'generic'; +select firstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss +select cutToFirstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss + +select 'difference'; +-- biz.ss is not in the default TLD list, hence: +select cutToFirstSignificantSubdomain('foo.kernel.biz.ss'); -- biz.ss +select cutToFirstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss From 8b6256dc4baa694fc550a789af1d49a7fd502ae2 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 125/174] Add performance test for custom TLD And seems works with the same speed as default (that uses gperf): - cutToFirstSignificantSubdomain SELECT cutToFirstSignificantSubdomain(URL) FROM datasets.hits SETTINGS max_threads = 1 FORMAT Null SETTINGS max_threads = 1 0 rows in set. Elapsed: 0.904 sec. Processed 8.87 million rows, 762.68 MB (9.82 million rows/s., 843.61 MB/s.) - cutToFirstSignificantSubdomainCustom SELECT cutToFirstSignificantSubdomainCustom(URL, 'public_suffix_list') FROM datasets.hits SETTINGS max_threads = 1 FORMAT Null SETTINGS max_threads = 1 0 rows in set. Elapsed: 0.909 sec. Processed 8.87 million rows, 762.68 MB (9.76 million rows/s., 838.83 MB/s.) --- tests/performance/custom_tld.xml | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 tests/performance/custom_tld.xml diff --git a/tests/performance/custom_tld.xml b/tests/performance/custom_tld.xml new file mode 100644 index 00000000000..e4a3d2e5f9f --- /dev/null +++ b/tests/performance/custom_tld.xml @@ -0,0 +1,11 @@ + + + hits_100m_single + + + SELECT count() FROM hits_100m_single WHERE NOT ignore(firstSignificantSubdomain(URL)) + SELECT count() FROM hits_100m_single WHERE NOT ignore(cutToFirstSignificantSubdomain(URL)) + + SELECT count() FROM hits_100m_single WHERE NOT ignore(firstSignificantSubdomainCustom(URL, 'public_suffix_list')) + SELECT count() FROM hits_100m_single WHERE NOT ignore(cutToFirstSignificantSubdomainCustom(URL, 'public_suffix_list')) + From 8a0487b1af38aeb8826912c7f8eb052df63fdf94 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 126/174] Add StringHashSet --- src/Common/HashTable/StringHashSet.h | 101 +++++++++++++++++++++ src/Interpreters/tests/CMakeLists.txt | 3 + src/Interpreters/tests/string_hash_set.cpp | 83 +++++++++++++++++ 3 files changed, 187 insertions(+) create mode 100644 src/Common/HashTable/StringHashSet.h create mode 100644 src/Interpreters/tests/string_hash_set.cpp diff --git a/src/Common/HashTable/StringHashSet.h b/src/Common/HashTable/StringHashSet.h new file mode 100644 index 00000000000..8714a0e1fe4 --- /dev/null +++ b/src/Common/HashTable/StringHashSet.h @@ -0,0 +1,101 @@ +#pragma once + +#include +#include +#include + +template +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; +}; + +template <> +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; + + bool isZero(const HashTableNoState & state) const { return isZero(this->key, state); } + // Zero means unoccupied cells in hash table. Use key with last word = 0 as + // zero keys, because such keys are unrepresentable (no way to encode length). + static bool isZero(const StringKey16 & key_, const HashTableNoState &) + { return key_.high == 0; } + void setZero() { this->key.high = 0; } +}; + +template <> +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; + + bool isZero(const HashTableNoState & state) const { return isZero(this->key, state); } + // Zero means unoccupied cells in hash table. Use key with last word = 0 as + // zero keys, because such keys are unrepresentable (no way to encode length). + static bool isZero(const StringKey24 & key_, const HashTableNoState &) + { return key_.c == 0; } + void setZero() { this->key.c = 0; } +}; + +template <> +struct StringHashSetCell : public HashSetCellWithSavedHash +{ + using Base = HashSetCellWithSavedHash; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; +}; + +template +struct StringHashSetSubMaps +{ + using T0 = StringHashTableEmpty>; + using T1 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using T2 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using T3 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using Ts = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; +}; + +template +class StringHashSet : public StringHashTable> +{ +public: + using Key = StringRef; + using Base = StringHashTable>; + using Self = StringHashSet; + using LookupResult = typename Base::LookupResult; + + using Base::Base; + + template + void ALWAYS_INLINE emplace(KeyHolder && key_holder, bool & inserted) + { + LookupResult it; + Base::emplace(key_holder, it, inserted); + } + +}; diff --git a/src/Interpreters/tests/CMakeLists.txt b/src/Interpreters/tests/CMakeLists.txt index 20aa73166fb..1bc9d7fbacb 100644 --- a/src/Interpreters/tests/CMakeLists.txt +++ b/src/Interpreters/tests/CMakeLists.txt @@ -29,6 +29,9 @@ target_link_libraries (string_hash_map PRIVATE dbms) add_executable (string_hash_map_aggregation string_hash_map.cpp) target_link_libraries (string_hash_map_aggregation PRIVATE dbms) +add_executable (string_hash_set string_hash_set.cpp) +target_link_libraries (string_hash_set PRIVATE dbms) + add_executable (two_level_hash_map two_level_hash_map.cpp) target_include_directories (two_level_hash_map SYSTEM BEFORE PRIVATE ${SPARSEHASH_INCLUDE_DIR}) target_link_libraries (two_level_hash_map PRIVATE dbms) diff --git a/src/Interpreters/tests/string_hash_set.cpp b/src/Interpreters/tests/string_hash_set.cpp new file mode 100644 index 00000000000..d9d6453da34 --- /dev/null +++ b/src/Interpreters/tests/string_hash_set.cpp @@ -0,0 +1,83 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/// NOTE: see string_hash_map.cpp for usage example + +template +void NO_INLINE bench(const std::vector & data, DB::Arena & pool, const char * name) +{ + std::cerr << "method " << name << std::endl; + for (auto t = 0ul; t < 7; ++t) + { + Stopwatch watch; + Set set; + typename Set::LookupResult it; + bool inserted; + + for (const auto & value : data) + { + if constexpr (std::is_same_v, Set>) + set.emplace(DB::ArenaKeyHolder{value, pool}, inserted); + else + set.emplace(DB::ArenaKeyHolder{value, pool}, it, inserted); + } + watch.stop(); + + std::cerr << "arena-memory " << pool.size() + set.getBufferSizeInBytes() << std::endl; + std::cerr << "single-run " << std::setprecision(3) + << watch.elapsedSeconds() << std::endl; + } +} + +int main(int argc, char ** argv) +{ + if (argc < 3) + { + std::cerr << "Usage: program n m\n"; + return 1; + } + + size_t n = std::stol(argv[1]); + size_t m = std::stol(argv[2]); + + DB::Arena pool(128 * 1024 * 1024); + std::vector data(n); + + std::cerr << "sizeof(Key) = " << sizeof(StringRef) << std::endl; + + { + Stopwatch watch; + DB::ReadBufferFromFileDescriptor in1(STDIN_FILENO); + DB::CompressedReadBuffer in2(in1); + + std::string tmp; + for (size_t i = 0; i < n && !in2.eof(); ++i) + { + DB::readStringBinary(tmp, in2); + data[i] = StringRef(pool.insert(tmp.data(), tmp.size()), tmp.size()); + } + + watch.stop(); + std::cerr << std::fixed << std::setprecision(2) << "Vector. Size: " << n << ", elapsed: " << watch.elapsedSeconds() << " (" + << n / watch.elapsedSeconds() << " elem/sec.)" << std::endl; + } + + if (!m || m == 1) + bench>(data, pool, "StringHashSet"); + if (!m || m == 2) + bench>(data, pool, "HashSetWithSavedHash"); + if (!m || m == 3) + bench>(data, pool, "HashSet"); + return 0; +} From 89cb2185a8fc2f26ded0c01188c6a666448acfdc Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 127/174] Fix StringHashTable move constructor --- src/Common/HashTable/StringHashTable.h | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/Common/HashTable/StringHashTable.h b/src/Common/HashTable/StringHashTable.h index 06389825e60..9f91de5585b 100644 --- a/src/Common/HashTable/StringHashTable.h +++ b/src/Common/HashTable/StringHashTable.h @@ -212,7 +212,7 @@ public: using LookupResult = StringHashTableLookupResult; using ConstLookupResult = StringHashTableLookupResult; - StringHashTable() {} + StringHashTable() = default; StringHashTable(size_t reserve_for_num_elements) : m1{reserve_for_num_elements / 4} @@ -222,8 +222,15 @@ public: { } - StringHashTable(StringHashTable && rhs) { *this = std::move(rhs); } - ~StringHashTable() {} + StringHashTable(StringHashTable && rhs) + : m1(std::move(rhs.m1)) + , m2(std::move(rhs.m2)) + , m3(std::move(rhs.m3)) + , ms(std::move(rhs.ms)) + { + } + + ~StringHashTable() = default; public: // Dispatch is written in a way that maximizes the performance: From c987be632f205e2783e84bc156a08a99c1d22fe1 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Fri, 4 Dec 2020 00:11:38 +0300 Subject: [PATCH 128/174] Switch TLDList to StringHashSet (to avoid errors on collisions) --- src/Common/TLDListsHolder.cpp | 26 +++++++++++++++++++------- src/Common/TLDListsHolder.h | 13 +++++++++---- 2 files changed, 28 insertions(+), 11 deletions(-) diff --git a/src/Common/TLDListsHolder.cpp b/src/Common/TLDListsHolder.cpp index cbad8beaa7d..e29a89a03d6 100644 --- a/src/Common/TLDListsHolder.cpp +++ b/src/Common/TLDListsHolder.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB { @@ -15,15 +16,19 @@ namespace ErrorCodes /// /// TLDList /// +TLDList::TLDList(size_t size) + : tld_container(size) + , pool(std::make_unique(10 << 20)) +{} bool TLDList::insert(const StringRef & host) { - StringRefHash hash; - return tld_container.insert(hash(host)).second; + bool inserted; + tld_container.emplace(DB::ArenaKeyHolder{host, *pool}, inserted); + return inserted; } bool TLDList::has(const StringRef & host) const { - StringRefHash hash; - return tld_container.has(hash(host)); + return tld_container.has(host); } /// @@ -54,7 +59,7 @@ void TLDListsHolder::parseConfig(const Poco::Util::AbstractConfiguration & confi size_t TLDListsHolder::parseAndAddTldList(const std::string & name, const std::string & path) { - TLDList tld_list; + std::unordered_set tld_list_tmp; ReadBufferFromFile in(path); while (!in.eof()) @@ -73,12 +78,19 @@ size_t TLDListsHolder::parseAndAddTldList(const std::string & name, const std::s /// Skip empty line if (line.empty()) continue; - tld_list.insert(StringRef{line.data(), line.size()}); + tld_list_tmp.emplace(line); + } + + TLDList tld_list(tld_list_tmp.size()); + for (const auto & host : tld_list_tmp) + { + StringRef host_ref{host.data(), host.size()}; + tld_list.insert(host_ref); } size_t tld_list_size = tld_list.size(); std::lock_guard lock(tld_lists_map_mutex); - tld_lists_map.emplace(name, std::move(tld_list)); + tld_lists_map.insert(std::make_pair(name, std::move(tld_list))); return tld_list_size; } diff --git a/src/Common/TLDListsHolder.h b/src/Common/TLDListsHolder.h index 9ce394267ec..209ccba217b 100644 --- a/src/Common/TLDListsHolder.h +++ b/src/Common/TLDListsHolder.h @@ -2,7 +2,8 @@ #include #include -#include +#include +#include #include #include #include @@ -12,20 +13,24 @@ namespace DB { /// Custom TLD List -/// Unlike tldLookup (which uses gperf) this one uses plain HashSet. +/// +/// Unlike tldLookup (which uses gperf) this one uses plain StringHashSet. class TLDList { public: - /// Uses StringRefHash - using Container = HashSet; + using Container = StringHashSet<>; + + TLDList(size_t size); /// Return true if the tld_container does not contains such element. bool insert(const StringRef & host); + /// Check is there such TLD bool has(const StringRef & host) const; size_t size() const { return tld_container.size(); } private: Container tld_container; + std::unique_ptr pool; }; class TLDListsHolder From 9b6b2b175f53719e4545e52557eecca5fe6cde6d Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Mon, 7 Dec 2020 21:19:11 +0300 Subject: [PATCH 129/174] perf: merge custom_tld.xml/first_significant_subdomain.xml into url_hits.xml v2: smaller table for *firstSignificantSubdomain* (max_threads=1) --- tests/performance/custom_tld.xml | 11 ----------- tests/performance/first_significant_subdomain.xml | 14 -------------- tests/performance/url_hits.xml | 11 ++++++++--- 3 files changed, 8 insertions(+), 28 deletions(-) delete mode 100644 tests/performance/custom_tld.xml delete mode 100644 tests/performance/first_significant_subdomain.xml diff --git a/tests/performance/custom_tld.xml b/tests/performance/custom_tld.xml deleted file mode 100644 index e4a3d2e5f9f..00000000000 --- a/tests/performance/custom_tld.xml +++ /dev/null @@ -1,11 +0,0 @@ - - - hits_100m_single - - - SELECT count() FROM hits_100m_single WHERE NOT ignore(firstSignificantSubdomain(URL)) - SELECT count() FROM hits_100m_single WHERE NOT ignore(cutToFirstSignificantSubdomain(URL)) - - SELECT count() FROM hits_100m_single WHERE NOT ignore(firstSignificantSubdomainCustom(URL, 'public_suffix_list')) - SELECT count() FROM hits_100m_single WHERE NOT ignore(cutToFirstSignificantSubdomainCustom(URL, 'public_suffix_list')) - diff --git a/tests/performance/first_significant_subdomain.xml b/tests/performance/first_significant_subdomain.xml deleted file mode 100644 index b8418401986..00000000000 --- a/tests/performance/first_significant_subdomain.xml +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - test.hits - - - - 1 - - - SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomain(URL)) - diff --git a/tests/performance/url_hits.xml b/tests/performance/url_hits.xml index c8cf119a7d7..072fb5b94e7 100644 --- a/tests/performance/url_hits.xml +++ b/tests/performance/url_hits.xml @@ -1,11 +1,10 @@ - hits_100m_single + test.hits - func @@ -32,6 +31,12 @@ - SELECT count() FROM hits_100m_single WHERE NOT ignore({func}(URL)) + + + SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomain(URL)) SETTINGS max_threads=1 + SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomainCustom(URL, 'public_suffix_list')) SETTINGS max_threads=1 + + SELECT count() FROM test.hits WHERE NOT ignore(cutToFirstSignificantSubdomain(URL)) SETTINGS max_threads=1 + SELECT count() FROM test.hits WHERE NOT ignore(cutToFirstSignificantSubdomainCustom(URL, 'public_suffix_list')) SETTINGS max_threads=1 From 840a21d0734528d81b2b9f09ba941696e31923dd Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Tue, 8 Dec 2020 23:54:03 +0300 Subject: [PATCH 130/174] Add top_level_domains_path for easier overriding --- docs/en/sql-reference/functions/url-functions.md | 4 +++- programs/server/Server.cpp | 5 ++++- programs/server/config.xml | 2 ++ src/Common/TLDListsHolder.cpp | 4 ++-- src/Common/TLDListsHolder.h | 2 +- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/en/sql-reference/functions/url-functions.md b/docs/en/sql-reference/functions/url-functions.md index 2250c0ead20..006542f494a 100644 --- a/docs/en/sql-reference/functions/url-functions.md +++ b/docs/en/sql-reference/functions/url-functions.md @@ -141,9 +141,11 @@ Same as `cutToFirstSignificantSubdomain` but accept custom TLD list name, useful Configuration example: ```xml + - /path/to/public_suffix_list.dat + public_suffix_list.dat + ``` diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 4480f34bb60..27c503e744a 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -485,7 +485,10 @@ int Server::main(const std::vector & /*args*/) } /// top_level_domains_lists - TLDListsHolder::getInstance().parseConfig(config()); + { + const std::string & top_level_domains_path = config().getString("top_level_domains_path", path + "top_level_domains/") + "/"; + TLDListsHolder::getInstance().parseConfig(top_level_domains_path, config()); + } { Poco::File(path + "data/").createDirectories(); diff --git a/programs/server/config.xml b/programs/server/config.xml index a46c6ae6eec..f41c346bbed 100644 --- a/programs/server/config.xml +++ b/programs/server/config.xml @@ -724,10 +724,12 @@ + public_suffix_list.dat diff --git a/tests/config/config.d/top_level_domains_path.xml b/tests/config/config.d/top_level_domains_path.xml new file mode 100644 index 00000000000..0ab836e5818 --- /dev/null +++ b/tests/config/config.d/top_level_domains_path.xml @@ -0,0 +1,3 @@ + + /etc/clickhouse-server/top_level_domains/ + diff --git a/tests/config/install.sh b/tests/config/install.sh index 126bc2eb086..6af0449ebec 100755 --- a/tests/config/install.sh +++ b/tests/config/install.sh @@ -31,6 +31,7 @@ ln -sf $SRC_PATH/config.d/test_cluster_with_incorrect_pw.xml $DEST_SERVER_PATH/c ln -sf $SRC_PATH/config.d/logging_no_rotate.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/tcp_with_proxy.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/top_level_domains_lists.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/top_level_domains_path.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/users.d/log_queries.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/readonly.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/access_management.xml $DEST_SERVER_PATH/users.d/ @@ -42,11 +43,7 @@ ln -sf $SRC_PATH/strings_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/decimals_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/executable_dictionary.xml $DEST_SERVER_PATH/ -# For CI when it is runned via init service -mkdir -p /var/lib/clickhouse/cores -ln -sf $SRC_PATH/public_suffix_list.dat /var/lib/clickhouse/cores/ -# For fasttest, that runs server just in background -ln -sf $SRC_PATH/public_suffix_list.dat ./ +ln -sf $SRC_PATH/top_level_domains $DEST_SERVER_PATH/ ln -sf $SRC_PATH/server.key $DEST_SERVER_PATH/ ln -sf $SRC_PATH/server.crt $DEST_SERVER_PATH/ diff --git a/tests/config/public_suffix_list.dat b/tests/config/top_level_domains/public_suffix_list.dat similarity index 100% rename from tests/config/public_suffix_list.dat rename to tests/config/top_level_domains/public_suffix_list.dat From 8a4b29495b28c5759c25a987907bc3f4dff47a8e Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Wed, 9 Dec 2020 21:06:29 +0300 Subject: [PATCH 133/174] perf: override top_level_domains_path via command line arguments --- docker/test/performance-comparison/compare.sh | 3 +++ .../config/config.d/top_level_domains_path.xml | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-) delete mode 100644 docker/test/performance-comparison/config/config.d/top_level_domains_path.xml diff --git a/docker/test/performance-comparison/compare.sh b/docker/test/performance-comparison/compare.sh index 6068a12f8a7..59d7cc98063 100755 --- a/docker/test/performance-comparison/compare.sh +++ b/docker/test/performance-comparison/compare.sh @@ -55,6 +55,7 @@ function configure # server *config* directives overrides --path db0 --user_files_path db0/user_files + --top_level_domains_path /top_level_domains --tcp_port $LEFT_SERVER_PORT ) left/clickhouse-server "${setup_left_server_opts[@]}" &> setup-server-log.log & @@ -102,6 +103,7 @@ function restart # server *config* directives overrides --path left/db --user_files_path left/db/user_files + --top_level_domains_path /top_level_domains --tcp_port $LEFT_SERVER_PORT ) left/clickhouse-server "${left_server_opts[@]}" &>> left-server-log.log & @@ -116,6 +118,7 @@ function restart # server *config* directives overrides --path right/db --user_files_path right/db/user_files + --top_level_domains_path /top_level_domains --tcp_port $RIGHT_SERVER_PORT ) right/clickhouse-server "${right_server_opts[@]}" &>> right-server-log.log & diff --git a/docker/test/performance-comparison/config/config.d/top_level_domains_path.xml b/docker/test/performance-comparison/config/config.d/top_level_domains_path.xml deleted file mode 100644 index 7ed88b0f546..00000000000 --- a/docker/test/performance-comparison/config/config.d/top_level_domains_path.xml +++ /dev/null @@ -1,3 +0,0 @@ - - /top_level_domains/ - From 0f1d9f3a6fe56b627b71fa7e18656e8a80ef8587 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Wed, 9 Dec 2020 21:06:29 +0300 Subject: [PATCH 134/174] Put top_level_domains into the docker/test/performance-comparison/config Since in docker/test/performance-comparison there is Dockerfile that is runned from that directory, so it cannot COPY anything outside of it. --- docker/test/performance-comparison/config/top_level_domains | 1 - .../config/top_level_domains/public_suffix_list.dat | 0 tests/config/top_level_domains | 1 + 3 files changed, 1 insertion(+), 1 deletion(-) delete mode 120000 docker/test/performance-comparison/config/top_level_domains rename {tests => docker/test/performance-comparison}/config/top_level_domains/public_suffix_list.dat (100%) create mode 120000 tests/config/top_level_domains diff --git a/docker/test/performance-comparison/config/top_level_domains b/docker/test/performance-comparison/config/top_level_domains deleted file mode 120000 index 53cbfda110d..00000000000 --- a/docker/test/performance-comparison/config/top_level_domains +++ /dev/null @@ -1 +0,0 @@ -../../../../tests/config/top_level_domains \ No newline at end of file diff --git a/tests/config/top_level_domains/public_suffix_list.dat b/docker/test/performance-comparison/config/top_level_domains/public_suffix_list.dat similarity index 100% rename from tests/config/top_level_domains/public_suffix_list.dat rename to docker/test/performance-comparison/config/top_level_domains/public_suffix_list.dat diff --git a/tests/config/top_level_domains b/tests/config/top_level_domains new file mode 120000 index 00000000000..7e12ab4ba2c --- /dev/null +++ b/tests/config/top_level_domains @@ -0,0 +1 @@ +../../docker/test/performance-comparison/config/top_level_domains \ No newline at end of file From 123923b5e5d55a27f2307fe0db05594875d08894 Mon Sep 17 00:00:00 2001 From: Dmitriy Date: Wed, 9 Dec 2020 21:14:55 +0300 Subject: [PATCH 135/174] Create distribution_queue.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Задокументировал системную таблицу distribution_queue. --- .../system-tables/distribution_queue.md | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 docs/en/operations/system-tables/distribution_queue.md diff --git a/docs/en/operations/system-tables/distribution_queue.md b/docs/en/operations/system-tables/distribution_queue.md new file mode 100644 index 00000000000..cc3822bad1a --- /dev/null +++ b/docs/en/operations/system-tables/distribution_queue.md @@ -0,0 +1,42 @@ +# system.distribution_queue {#system_tables-distribution_queue} + +Contains information about local files that are in the queue to be sent to the shards. This local files contain new parts that are created by inserting new data into the table. + +Columns: + +- `database` ([String](../../sql-reference/data-types/string.md)) — Name of the database. + +- `table` ([String](../../sql-reference/data-types/string.md)) — Name of the table. + +- `data_path` ([String](../../sql-reference/data-types/string.md)) — Path to the folder with local files. + +- `is_blocked` ([UInt8](../../sql-reference/data-types/int-uint.md)) — Flag indicates whether sending local files to the server is blocked. + +- `error_count` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Number of errors. + +- `data_files` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Number of local files in a folder. + +- `data_compressed_bytes` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Size of compressed data in local files, in bytes. + +- `last_exception` ([String](../../sql-reference/data-types/string.md)) — Text message about the last error that occurred. + +**Example** + +``` sql +SELECT * FROM system.distribution_queue LIMIT 1 FORMAT Vertical; +``` + +``` text +Row 1: +────── +database: default +table: dist +data_path: ./store/268/268bc070-3aad-4b1a-9cf2-4987580161af/default@127%2E0%2E0%2E2:9000/ +is_blocked: 1 +error_count: 0 +data_files: 1 +data_compressed_bytes: 499 +last_exception: +``` + +[Original article](https://clickhouse.tech/docs/en/operations/system_tables/distribution_queue) From 88798f201b6d9964ab8e2de08b9c89df058e27ea Mon Sep 17 00:00:00 2001 From: zhang2014 Date: Thu, 10 Dec 2020 02:50:56 +0800 Subject: [PATCH 136/174] ISSUES-15187 support prefix index --- .../MySQL/InterpretersMySQLDDLQuery.cpp | 33 +++++++++++++++++-- .../MySQL/tests/gtest_create_rewritten.cpp | 11 +++++++ 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp index 688ab786b56..37714c1a83e 100644 --- a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp +++ b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp @@ -164,20 +164,47 @@ static inline std::tuplechildren.empty()) { + NameSet columns_name_set; + const Names & columns_name = columns.getNames(); + columns_name_set.insert(columns_name.begin(), columns_name.end()); + + const auto & remove_prefix_key = [&](const ASTPtr & node) -> ASTPtr + { + auto res = std::make_shared(); + for (const auto & index_expression : node->children) + { + res->children.emplace_back(index_expression); + + if (const auto & function = index_expression->as()) + { + /// column_name(int64 literal) + if (columns_name_set.contains(function->name) && function->arguments->children.size() == 1) + { + const auto & prefix_limit = function->arguments->children[0]->as(); + + if (prefix_limit && isInt64FieldType(prefix_limit->value.getType())) + res->children.back() = std::make_shared(function->name); + } + } + } + return res; + }; + for (const auto & declare_index_ast : indices_define->children) { const auto & declare_index = declare_index_ast->as(); + const auto & index_columns = remove_prefix_key(declare_index->index_columns); /// flatten if (startsWith(declare_index->index_type, "KEY_")) keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); else if (startsWith(declare_index->index_type, "UNIQUE_")) unique_keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); if (startsWith(declare_index->index_type, "PRIMARY_KEY_")) primary_keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); } } diff --git a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp index 2221b7b1588..d7209330ec6 100644 --- a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp +++ b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp @@ -184,3 +184,14 @@ TEST(MySQLCreateRewritten, RewrittenQueryWithPrimaryKey) "ReplacingMergeTree(_version) PARTITION BY intDiv(key_2, 4294967) ORDER BY (key_1, key_2)"); } +TEST(MySQLCreateRewritten, RewrittenQueryWithPrefixKey) +{ + tryRegisterFunctions(); + const auto & context_holder = getContext(); + + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` int NOT NULL PRIMARY KEY, `prefix_key` varchar(200) NOT NULL, KEY prefix_key_index(prefix_key(2))) ENGINE=InnoDB DEFAULT CHARSET=utf8", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` Int32, `prefix_key` String, `_sign` Int8() MATERIALIZED 1, `_version` UInt64() MATERIALIZED 1) ENGINE = " + "ReplacingMergeTree(_version) PARTITION BY intDiv(key, 4294967) ORDER BY (key, prefix_key)"); +} + From 2760332a2a09909380b3d44812d229d8e7bd7e25 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 02:26:40 +0300 Subject: [PATCH 137/174] Add GH dataset description --- .../getting-started/example-datasets/github-events.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 docs/en/getting-started/example-datasets/github-events.md diff --git a/docs/en/getting-started/example-datasets/github-events.md b/docs/en/getting-started/example-datasets/github-events.md new file mode 100644 index 00000000000..a6c71733832 --- /dev/null +++ b/docs/en/getting-started/example-datasets/github-events.md @@ -0,0 +1,11 @@ +--- +toc_priority: 11 +toc_title: GitHub Events +--- + +# GitHub Events Dataset + +Dataset contains all events on GitHub from 2011 to Dec 6 2020, the size is 3.1 billion records. Download size is 75 GB and it will require up to 200 GB space on disk if stored in a table with lz4 compression. + +Full dataset description, insights, download instruction and interactive queries are posted [here](https://github-sql.github.io/explorer/). + From 9a1f64cd15d39aedbefe2eb641904f6df7c0a606 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 02:27:12 +0300 Subject: [PATCH 138/174] Add GH dataset description --- docs/en/getting-started/example-datasets/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/getting-started/example-datasets/index.md b/docs/en/getting-started/example-datasets/index.md index 35ac90f9beb..b769e8fcb45 100644 --- a/docs/en/getting-started/example-datasets/index.md +++ b/docs/en/getting-started/example-datasets/index.md @@ -1,6 +1,6 @@ --- toc_folder_title: Example Datasets -toc_priority: 14 +toc_priority: 10 toc_title: Introduction --- @@ -18,4 +18,4 @@ The list of documented datasets: - [New York Taxi Data](../../getting-started/example-datasets/nyc-taxi.md) - [OnTime](../../getting-started/example-datasets/ontime.md) -[Original article](https://clickhouse.tech/docs/en/getting_started/example_datasets) \ No newline at end of file +[Original article](https://clickhouse.tech/docs/en/getting_started/example_datasets) From b42ee3d99327733f50a37aac7dbf79136e8e50cb Mon Sep 17 00:00:00 2001 From: 243f6a8885a308d313198a2e037 <243f6a8885a308d313198a2e037@gmail.com> Date: Sun, 15 Nov 2020 15:16:08 +0900 Subject: [PATCH 139/174] docs: fix broken changelog link --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c722e4a1ca0..fb4e4dd6b5a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2998,7 +2998,7 @@ No changes compared to v20.4.3.16-stable. * Added check for extra parts of `MergeTree` at different disks, in order to not allow to miss data parts at undefined disks. [#8118](https://github.com/ClickHouse/ClickHouse/pull/8118) ([Vladimir Chebotarev](https://github.com/excitoon)) * Enable SSL support for Mac client and server. [#8297](https://github.com/ClickHouse/ClickHouse/pull/8297) ([Ivan](https://github.com/abyss7)) * Now ClickHouse can work as MySQL federated server (see https://dev.mysql.com/doc/refman/5.7/en/federated-create-server.html). [#7717](https://github.com/ClickHouse/ClickHouse/pull/7717) ([Maxim Fedotov](https://github.com/MaxFedotov)) -* `clickhouse-client` now only enable `bracketed-paste` when multiquery is on and multiline is off. This fixes (#7757)[https://github.com/ClickHouse/ClickHouse/issues/7757]. [#7761](https://github.com/ClickHouse/ClickHouse/pull/7761) ([Amos Bird](https://github.com/amosbird)) +* `clickhouse-client` now only enable `bracketed-paste` when multiquery is on and multiline is off. This fixes [#7757](https://github.com/ClickHouse/ClickHouse/issues/7757). [#7761](https://github.com/ClickHouse/ClickHouse/pull/7761) ([Amos Bird](https://github.com/amosbird)) * Support `Array(Decimal)` in `if` function. [#7721](https://github.com/ClickHouse/ClickHouse/pull/7721) ([Artem Zuikov](https://github.com/4ertus2)) * Support Decimals in `arrayDifference`, `arrayCumSum` and `arrayCumSumNegative` functions. [#7724](https://github.com/ClickHouse/ClickHouse/pull/7724) ([Artem Zuikov](https://github.com/4ertus2)) * Added `lifetime` column to `system.dictionaries` table. [#6820](https://github.com/ClickHouse/ClickHouse/issues/6820) [#7727](https://github.com/ClickHouse/ClickHouse/pull/7727) ([kekekekule](https://github.com/kekekekule)) From cf7ec95d36ab5d7222897c3c2779373edfa5ed87 Mon Sep 17 00:00:00 2001 From: 243f6a8885a308d313198a2e037 <243f6a8885a308d313198a2e037@gmail.com> Date: Sun, 15 Nov 2020 15:24:35 +0900 Subject: [PATCH 140/174] docs: fix raw link to #ddddd --- CHANGELOG.md | 74 ++++++++++++++++++++++++++-------------------------- 1 file changed, 37 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fb4e4dd6b5a..1812fadf664 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ * Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)). * Remove `ANALYZE` and `AST` queries, and make the setting `enable_debug_queries` obsolete since now it is the part of full featured `EXPLAIN` query. [#16536](https://github.com/ClickHouse/ClickHouse/pull/16536) ([Ivan](https://github.com/abyss7)). * Aggregate functions `boundingRatio`, `rankCorr`, `retention`, `timeSeriesGroupSum`, `timeSeriesGroupRateSum`, `windowFunnel` were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. [#16407](https://github.com/ClickHouse/ClickHouse/pull/16407) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Make `rankCorr` function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). +* Make `rankCorr` function return nan on insufficient data [#16124](https://github.com/ClickHouse/ClickHouse/issues/16124). [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). #### New Feature @@ -243,18 +243,18 @@ * Fix hang of queries with a lot of subqueries to same table of `MySQL` engine. Previously, if there were more than 16 subqueries to same `MySQL` table in query, it hang forever. [#15299](https://github.com/ClickHouse/ClickHouse/pull/15299) ([Anton Popov](https://github.com/CurtizJ)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. [#15242](https://github.com/ClickHouse/ClickHouse/pull/15242) ([Artem Zuikov](https://github.com/4ertus2)). -* Fix instance crash when using `joinGet` with `LowCardinality` types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using `joinGet` with `LowCardinality` types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust Decimal field size in MySQL column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). * Fixes `Data compressed with different methods` in `join_algorithm='auto'`. Keep LowCardinality as type for left table join key in `join_algorithm='partial_merge'`. [#15088](https://github.com/ClickHouse/ClickHouse/pull/15088) ([Artem Zuikov](https://github.com/4ertus2)). * Update `jemalloc` to fix `percpu_arena` with affinity mask. [#15035](https://github.com/ClickHouse/ClickHouse/pull/15035) ([Azat Khuzhin](https://github.com/azat)). [#14957](https://github.com/ClickHouse/ClickHouse/pull/14957) ([Azat Khuzhin](https://github.com/azat)). -* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). * If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in Docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). * Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. [#15002](https://github.com/ClickHouse/ClickHouse/pull/15002) ([Artem Zuikov](https://github.com/4ertus2)). * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains `finalizeAggregation` function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). * `MaterializedMySQL` (experimental feature): Fixed `.metadata.tmp File exists` error. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile with file descriptor. [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). @@ -263,14 +263,14 @@ * Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes [#14531](https://github.com/ClickHouse/ClickHouse/issues/14531). [#14845](https://github.com/ClickHouse/ClickHouse/pull/14845) ([alesapin](https://github.com/alesapin)). * Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes [#14814](https://github.com/ClickHouse/ClickHouse/issues/14814). [#14843](https://github.com/ClickHouse/ClickHouse/pull/14843) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)). -* `Replace` column transformer should replace identifiers with cloned ASTs. This fixes https://github.com/ClickHouse/ClickHouse/issues/14695 . [#14734](https://github.com/ClickHouse/ClickHouse/pull/14734) ([Amos Bird](https://github.com/amosbird)). +* `Replace` column transformer should replace identifiers with cloned ASTs. This fixes [#14695](https://github.com/ClickHouse/ClickHouse/issues/14695) . [#14734](https://github.com/ClickHouse/ClickHouse/pull/14734) ([Amos Bird](https://github.com/amosbird)). * Fixed missed default database name in metadata of materialized view when executing `ALTER ... MODIFY QUERY`. [#14664](https://github.com/ClickHouse/ClickHouse/pull/14664) ([tavplubix](https://github.com/tavplubix)). * Fix bug when `ALTER UPDATE` mutation with `Nullable` column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). * Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)). * Fix function `has` with `LowCardinality` of `Nullable`. [#14591](https://github.com/ClickHouse/ClickHouse/pull/14591) ([Mike](https://github.com/myrrc)). * Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. [#14563](https://github.com/ClickHouse/ClickHouse/pull/14563) ([Bharat Nallan](https://github.com/bharatnc)). * Fix rare segfaults in functions with combinator `-Resample`, which could appear in result of overflow with very large parameters. [#14562](https://github.com/ClickHouse/ClickHouse/pull/14562) ([Anton Popov](https://github.com/CurtizJ)). -* Fix a bug when converting `Nullable(String)` to Enum. Introduced by https://github.com/ClickHouse/ClickHouse/pull/12745. This fixes https://github.com/ClickHouse/ClickHouse/issues/14435. [#14530](https://github.com/ClickHouse/ClickHouse/pull/14530) ([Amos Bird](https://github.com/amosbird)). +* Fix a bug when converting `Nullable(String)` to Enum. Introduced by [#12745](https://github.com/ClickHouse/ClickHouse/pull/12745). This fixes [#14435](https://github.com/ClickHouse/ClickHouse/issues/14435). [#14530](https://github.com/ClickHouse/ClickHouse/pull/14530) ([Amos Bird](https://github.com/amosbird)). * Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix `currentDatabase()` function cannot be used in `ON CLUSTER` ddl query. [#14211](https://github.com/ClickHouse/ClickHouse/pull/14211) ([Winter Zhang](https://github.com/zhang2014)). * `MaterializedMySQL` (experimental feature): Fixed `Packet payload is not fully read` error in `MaterializeMySQL` database engine. [#14696](https://github.com/ClickHouse/ClickHouse/pull/14696) ([BohuTANG](https://github.com/BohuTANG)). @@ -308,7 +308,7 @@ * Add an option to skip access checks for `DiskS3`. `s3` disk is an experimental feature. [#14497](https://github.com/ClickHouse/ClickHouse/pull/14497) ([Pavel Kovalenko](https://github.com/Jokser)). * Speed up server shutdown process if there are ongoing S3 requests. [#14496](https://github.com/ClickHouse/ClickHouse/pull/14496) ([Pavel Kovalenko](https://github.com/Jokser)). * `SYSTEM RELOAD CONFIG` now throws an exception if failed to reload and continues using the previous users.xml. The background periodic reloading also continues using the previous users.xml if failed to reload. [#14492](https://github.com/ClickHouse/ClickHouse/pull/14492) ([Vitaly Baranov](https://github.com/vitlibar)). -* For INSERTs with inline data in VALUES format in the script mode of `clickhouse-client`, support semicolon as the data terminator, in addition to the new line. Closes https://github.com/ClickHouse/ClickHouse/issues/12288. [#13192](https://github.com/ClickHouse/ClickHouse/pull/13192) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* For INSERTs with inline data in VALUES format in the script mode of `clickhouse-client`, support semicolon as the data terminator, in addition to the new line. Closes [#12288](https://github.com/ClickHouse/ClickHouse/issues/12288). [#13192](https://github.com/ClickHouse/ClickHouse/pull/13192) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Support custom codecs in compact parts. [#12183](https://github.com/ClickHouse/ClickHouse/pull/12183) ([Anton Popov](https://github.com/CurtizJ)). #### Performance Improvement @@ -320,7 +320,7 @@ * Improve performance of 256-bit types using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. [#14859](https://github.com/ClickHouse/ClickHouse/pull/14859) ([Artem Zuikov](https://github.com/4ertus2)). * Explicitly use a temporary disk to store vertical merge temporary data. [#15639](https://github.com/ClickHouse/ClickHouse/pull/15639) ([Grigory Pervakov](https://github.com/GrigoryPervakov)). * Use one S3 DeleteObjects request instead of multiple DeleteObject in a loop. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. [#15238](https://github.com/ClickHouse/ClickHouse/pull/15238) ([ianton-ru](https://github.com/ianton-ru)). -* Fix `DateTime DateTime` mistakenly choosing the slow generic implementation. This fixes https://github.com/ClickHouse/ClickHouse/issues/15153. [#15178](https://github.com/ClickHouse/ClickHouse/pull/15178) ([Amos Bird](https://github.com/amosbird)). +* Fix `DateTime DateTime` mistakenly choosing the slow generic implementation. This fixes [#15153](https://github.com/ClickHouse/ClickHouse/issues/15153). [#15178](https://github.com/ClickHouse/ClickHouse/pull/15178) ([Amos Bird](https://github.com/amosbird)). * Improve performance of GROUP BY key of type `FixedString`. [#15034](https://github.com/ClickHouse/ClickHouse/pull/15034) ([Amos Bird](https://github.com/amosbird)). * Only `mlock` code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. [#14929](https://github.com/ClickHouse/ClickHouse/pull/14929) ([alexey-milovidov](https://github.com/alexey-milovidov)). * ClickHouse binary become smaller due to link time optimization. @@ -387,7 +387,7 @@ * Allow to use direct layout for dictionaries with complex keys. [#16007](https://github.com/ClickHouse/ClickHouse/pull/16007) ([Anton Popov](https://github.com/CurtizJ)). * Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. [#15987](https://github.com/ClickHouse/ClickHouse/pull/15987) ([filimonov](https://github.com/filimonov)). * Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). [#15984](https://github.com/ClickHouse/ClickHouse/pull/15984) ([tavplubix](https://github.com/tavplubix)). -* Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes https://github.com/ClickHouse/ClickHouse/issues/15628. [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes [#15628](https://github.com/ClickHouse/ClickHouse/issues/15628). [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix a crash when database creation fails. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). * Fixed `DROP TABLE IF EXISTS` failure with `Table ... doesn't exist` error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like `DROP DATABASE` and `RENAME TABLE`) Fixed `DROP/DETACH DATABASE` failure with `Table ... doesn't exist` when concurrently executing `DROP/DETACH TABLE`. [#15934](https://github.com/ClickHouse/ClickHouse/pull/15934) ([tavplubix](https://github.com/tavplubix)). * Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes [#15792](https://github.com/ClickHouse/ClickHouse/issues/15792). [#15933](https://github.com/ClickHouse/ClickHouse/pull/15933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -398,7 +398,7 @@ * Fixed too low default value of `max_replicated_logs_to_keep` setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. [#15701](https://github.com/ClickHouse/ClickHouse/pull/15701) ([tavplubix](https://github.com/tavplubix)). * Fix error `Cannot add simple transform to empty Pipe` which happened while reading from `Buffer` table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes [#15529](https://github.com/ClickHouse/ClickHouse/issues/15529). [#15662](https://github.com/ClickHouse/ClickHouse/pull/15662) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. [#15646](https://github.com/ClickHouse/ClickHouse/pull/15646) ([Vladimir Chebotarev](https://github.com/excitoon)). -* Decrement the `ReadonlyReplica` metric when detaching read-only tables. This fixes https://github.com/ClickHouse/ClickHouse/issues/15598. [#15592](https://github.com/ClickHouse/ClickHouse/pull/15592) ([sundyli](https://github.com/sundy-li)). +* Decrement the `ReadonlyReplica` metric when detaching read-only tables. This fixes [#15598](https://github.com/ClickHouse/ClickHouse/issues/15598). [#15592](https://github.com/ClickHouse/ClickHouse/pull/15592) ([sundyli](https://github.com/sundy-li)). * Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. [#15516](https://github.com/ClickHouse/ClickHouse/pull/15516) ([nvartolomei](https://github.com/nvartolomei)). #### Improvement @@ -426,7 +426,7 @@ * Fix race condition during MergeTree table rename and background cleanup. [#15304](https://github.com/ClickHouse/ClickHouse/pull/15304) ([alesapin](https://github.com/alesapin)). * Fix rare race condition on server startup when system.logs are enabled. [#15300](https://github.com/ClickHouse/ClickHouse/pull/15300) ([alesapin](https://github.com/alesapin)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using joinGet with LowCardinality types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust decimals field size in mysql column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). @@ -455,10 +455,10 @@ * Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). * Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* Fixed inconsistent comparison with primary key of type `FixedString` on index analysis if they're compered with a string of less size. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* Fixed inconsistent comparison with primary key of type `FixedString` on index analysis if they're compered with a string of less size. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). * Fix bug which leads to wrong merges assignment if table has partitions with a single part. [#14444](https://github.com/ClickHouse/ClickHouse/pull/14444) ([alesapin](https://github.com/alesapin)). * If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). @@ -579,15 +579,15 @@ * Fix race condition during MergeTree table rename and background cleanup. [#15304](https://github.com/ClickHouse/ClickHouse/pull/15304) ([alesapin](https://github.com/alesapin)). * Fix rare race condition on server startup when system.logs are enabled. [#15300](https://github.com/ClickHouse/ClickHouse/pull/15300) ([alesapin](https://github.com/alesapin)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using joinGet with LowCardinality types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust decimals field size in mysql column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). -* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). * If function `bar` was called with specifically crafter arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes [#14814](https://github.com/ClickHouse/ClickHouse/issues/14814). [#14843](https://github.com/ClickHouse/ClickHouse/pull/14843) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)). @@ -653,10 +653,10 @@ * Fix crash during `ALTER` query for table which was created `AS table_function`. Fixes [#14212](https://github.com/ClickHouse/ClickHouse/issues/14212). [#14326](https://github.com/ClickHouse/ClickHouse/pull/14326) ([alesapin](https://github.com/alesapin)). * Fix exception during ALTER LIVE VIEW query with REFRESH command. Live view is an experimental feature. [#14320](https://github.com/ClickHouse/ClickHouse/pull/14320) ([Bharat Nallan](https://github.com/bharatnc)). * Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. [#14315](https://github.com/ClickHouse/ClickHouse/pull/14315) ([Azat Khuzhin](https://github.com/azat)). -* Fix segfault in `clickhouse-odbc-bridge` during schema fetch from some external sources. This PR fixes https://github.com/ClickHouse/ClickHouse/issues/13861. [#14267](https://github.com/ClickHouse/ClickHouse/pull/14267) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash in mark inclusion search introduced in https://github.com/ClickHouse/ClickHouse/pull/12277. [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). +* Fix segfault in `clickhouse-odbc-bridge` during schema fetch from some external sources. This PR fixes [#13861](https://github.com/ClickHouse/ClickHouse/issues/13861). [#14267](https://github.com/ClickHouse/ClickHouse/pull/14267) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash in mark inclusion search introduced in [#12277](https://github.com/ClickHouse/ClickHouse/pull/12277). [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). * Fix creation of tables with named tuples. This fixes [#13027](https://github.com/ClickHouse/ClickHouse/issues/13027). [#14143](https://github.com/ClickHouse/ClickHouse/pull/14143) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix formatting of minimal negative decimal numbers. This fixes https://github.com/ClickHouse/ClickHouse/issues/14111. [#14119](https://github.com/ClickHouse/ClickHouse/pull/14119) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Fix formatting of minimal negative decimal numbers. This fixes [#14111](https://github.com/ClickHouse/ClickHouse/issues/14111). [#14119](https://github.com/ClickHouse/ClickHouse/pull/14119) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix `DistributedFilesToInsert` metric (zeroed when it should not). [#14095](https://github.com/ClickHouse/ClickHouse/pull/14095) ([Azat Khuzhin](https://github.com/azat)). * Fix `pointInPolygon` with const 2d array as polygon. [#14079](https://github.com/ClickHouse/ClickHouse/pull/14079) ([Alexey Ilyukhov](https://github.com/livace)). * Fixed wrong mount point in extra info for `Poco::Exception: no space left on device`. [#14050](https://github.com/ClickHouse/ClickHouse/pull/14050) ([tavplubix](https://github.com/tavplubix)). @@ -685,10 +685,10 @@ * Fix wrong code in function `netloc`. This fixes [#13335](https://github.com/ClickHouse/ClickHouse/issues/13335). [#13446](https://github.com/ClickHouse/ClickHouse/pull/13446) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix possible race in `StorageMemory`. [#13416](https://github.com/ClickHouse/ClickHouse/pull/13416) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix missing or excessive headers in `TSV/CSVWithNames` formats in HTTP protocol. This fixes [#12504](https://github.com/ClickHouse/ClickHouse/issues/12504). [#13343](https://github.com/ClickHouse/ClickHouse/pull/13343) ([Azat Khuzhin](https://github.com/azat)). -* Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes https://github.com/ClickHouse/ClickHouse/issues/5779, https://github.com/ClickHouse/ClickHouse/issues/12527. [#13199](https://github.com/ClickHouse/ClickHouse/pull/13199) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes [#5779](https://github.com/ClickHouse/ClickHouse/issues/5779), [#12527](https://github.com/ClickHouse/ClickHouse/issues/12527). [#13199](https://github.com/ClickHouse/ClickHouse/pull/13199) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix access to `redis` dictionary after connection was dropped once. It may happen with `cache` and `direct` dictionary layouts. [#13082](https://github.com/ClickHouse/ClickHouse/pull/13082) ([Anton Popov](https://github.com/CurtizJ)). * Removed wrong auth access check when using ClickHouseDictionarySource to query remote tables. [#12756](https://github.com/ClickHouse/ClickHouse/pull/12756) ([sundyli](https://github.com/sundy-li)). -* Properly distinguish subqueries in some cases for common subexpression elimination. https://github.com/ClickHouse/ClickHouse/issues/8333. [#8367](https://github.com/ClickHouse/ClickHouse/pull/8367) ([Amos Bird](https://github.com/amosbird)). +* Properly distinguish subqueries in some cases for common subexpression elimination. [#8333](https://github.com/ClickHouse/ClickHouse/issues/8333). [#8367](https://github.com/ClickHouse/ClickHouse/pull/8367) ([Amos Bird](https://github.com/amosbird)). #### Improvement @@ -756,7 +756,7 @@ * Updating LDAP user authentication suite to check that it works with RBAC. [#13656](https://github.com/ClickHouse/ClickHouse/pull/13656) ([vzakaznikov](https://github.com/vzakaznikov)). * Removed `-DENABLE_CURL_CLIENT` for `contrib/aws`. [#13628](https://github.com/ClickHouse/ClickHouse/pull/13628) ([Vladimir Chebotarev](https://github.com/excitoon)). * Increasing health-check timeouts for ClickHouse nodes and adding support to dump docker-compose logs if unhealthy containers found. [#13612](https://github.com/ClickHouse/ClickHouse/pull/13612) ([vzakaznikov](https://github.com/vzakaznikov)). -* Make sure https://github.com/ClickHouse/ClickHouse/issues/10977 is invalid. [#13539](https://github.com/ClickHouse/ClickHouse/pull/13539) ([Amos Bird](https://github.com/amosbird)). +* Make sure [#10977](https://github.com/ClickHouse/ClickHouse/issues/10977) is invalid. [#13539](https://github.com/ClickHouse/ClickHouse/pull/13539) ([Amos Bird](https://github.com/amosbird)). * Skip PR's from robot-clickhouse. [#13489](https://github.com/ClickHouse/ClickHouse/pull/13489) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Move Dockerfiles from integration tests to `docker/test` directory. docker_compose files are available in `runner` docker container. Docker images are built in CI and not in integration tests. [#13448](https://github.com/ClickHouse/ClickHouse/pull/13448) ([Ilya Yatsishin](https://github.com/qoega)). @@ -1213,7 +1213,7 @@ * Fix wrong result of comparison of FixedString with constant String. This fixes [#11393](https://github.com/ClickHouse/ClickHouse/issues/11393). This bug appeared in version 20.4. [#11828](https://github.com/ClickHouse/ClickHouse/pull/11828) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix wrong result for `if` with NULLs in condition. [#11807](https://github.com/ClickHouse/ClickHouse/pull/11807) ([Artem Zuikov](https://github.com/4ertus2)). * Fix using too many threads for queries. [#11788](https://github.com/ClickHouse/ClickHouse/pull/11788) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). -* Fixed `Scalar doesn't exist` exception when using `WITH ...` in `SELECT ... FROM merge_tree_table ...` https://github.com/ClickHouse/ClickHouse/issues/11621. [#11767](https://github.com/ClickHouse/ClickHouse/pull/11767) ([Amos Bird](https://github.com/amosbird)). +* Fixed `Scalar doesn't exist` exception when using `WITH ...` in `SELECT ... FROM merge_tree_table ...` [#11621](https://github.com/ClickHouse/ClickHouse/issues/11621). [#11767](https://github.com/ClickHouse/ClickHouse/pull/11767) ([Amos Bird](https://github.com/amosbird)). * Fix unexpected behaviour of queries like `SELECT *, xyz.*` which were success while an error expected. [#11753](https://github.com/ClickHouse/ClickHouse/pull/11753) ([hexiaoting](https://github.com/hexiaoting)). * Now replicated fetches will be cancelled during metadata alter. [#11744](https://github.com/ClickHouse/ClickHouse/pull/11744) ([alesapin](https://github.com/alesapin)). * Parse metadata stored in zookeeper before checking for equality. [#11739](https://github.com/ClickHouse/ClickHouse/pull/11739) ([Azat Khuzhin](https://github.com/azat)). @@ -1264,8 +1264,8 @@ * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash when `SET DEFAULT ROLE` is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash while reading malformed data in `Protobuf` format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash when `SET DEFAULT ROLE` is called with wrong arguments. This fixes [#10586](https://github.com/ClickHouse/ClickHouse/issues/10586). [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in `Protobuf` format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when `cache` dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1488,7 +1488,7 @@ * Lower memory usage in tests. [#10617](https://github.com/ClickHouse/ClickHouse/pull/10617) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixing hard coded timeouts in new live view tests. [#10604](https://github.com/ClickHouse/ClickHouse/pull/10604) ([vzakaznikov](https://github.com/vzakaznikov)). * Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. [#10599](https://github.com/ClickHouse/ClickHouse/pull/10599) ([vzakaznikov](https://github.com/vzakaznikov)). -* Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. [#10585](https://github.com/ClickHouse/ClickHouse/pull/10585) ([Amos Bird](https://github.com/amosbird)). +* Enable ThinLTO for clang builds, continuation of [#10435](https://github.com/ClickHouse/ClickHouse/pull/10435). [#10585](https://github.com/ClickHouse/ClickHouse/pull/10585) ([Amos Bird](https://github.com/amosbird)). * Adding fuzzers and preparing for oss-fuzz integration. [#10546](https://github.com/ClickHouse/ClickHouse/pull/10546) ([kyprizel](https://github.com/kyprizel)). * Fix FreeBSD build. [#10150](https://github.com/ClickHouse/ClickHouse/pull/10150) ([Ivan](https://github.com/abyss7)). * Add new build for query tests using pytest framework. [#10039](https://github.com/ClickHouse/ClickHouse/pull/10039) ([Ivan](https://github.com/abyss7)). @@ -1617,7 +1617,7 @@ * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. [#11420](https://github.com/ClickHouse/ClickHouse/pull/11420) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). * Fix possible `Pipeline stuck` error for queries with external sort and limit. Fixes [#11359](https://github.com/ClickHouse/ClickHouse/issues/11359). [#11366](https://github.com/ClickHouse/ClickHouse/pull/11366) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1633,8 +1633,8 @@ * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes [#10586](https://github.com/ClickHouse/ClickHouse/issues/10586). [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1932,7 +1932,7 @@ No changes compared to v20.4.3.16-stable. * Move integration tests docker files to docker/ directory. [#10335](https://github.com/ClickHouse/ClickHouse/pull/10335) ([Ilya Yatsishin](https://github.com/qoega)) * Allow to use `clang-10` in CI. It ensures that [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238) is fixed. [#10384](https://github.com/ClickHouse/ClickHouse/pull/10384) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message `OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error` and `SSL Exception: error:2400006E:random number generator::error retrieving entropy`. The issue was present in version 20.1. [#8956](https://github.com/ClickHouse/ClickHouse/pull/8956) ([alexey-milovidov](https://github.com/alexey-milovidov)) -* Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238 [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)) +* Fix clang-10 build. [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238) [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)) * Add performance test for [Parallel INSERT for materialized view](https://github.com/ClickHouse/ClickHouse/pull/10052). [#10345](https://github.com/ClickHouse/ClickHouse/pull/10345) ([vxider](https://github.com/Vxider)) * Fix flaky test `test_settings_constraints_distributed.test_insert_clamps_settings`. [#10346](https://github.com/ClickHouse/ClickHouse/pull/10346) ([Vitaly Baranov](https://github.com/vitlibar)) * Add util to test results upload in CI ClickHouse [#10330](https://github.com/ClickHouse/ClickHouse/pull/10330) ([Ilya Yatsishin](https://github.com/qoega)) @@ -2148,7 +2148,7 @@ No changes compared to v20.4.3.16-stable. * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fix crash in JOIN over LowCarinality(T) and Nullable(T). [#11380](https://github.com/ClickHouse/ClickHouse/issues/11380). [#11414](https://github.com/ClickHouse/ClickHouse/pull/11414) ([Artem Zuikov](https://github.com/4ertus2)). * Fix error code for wrong `USING` key. [#11373](https://github.com/ClickHouse/ClickHouse/issues/11373). [#11404](https://github.com/ClickHouse/ClickHouse/pull/11404) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). @@ -2165,7 +2165,7 @@ No changes compared to v20.4.3.16-stable. * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2225,7 +2225,7 @@ No changes compared to v20.4.3.16-stable. #### Build/Testing/Packaging Improvement * Fix UBSan report in LZ4 library. [#10631](https://github.com/ClickHouse/ClickHouse/pull/10631) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238. [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)). +* Fix clang-10 build. [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238). [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)). * Added failing tests about `max_rows_to_sort` setting. [#10268](https://github.com/ClickHouse/ClickHouse/pull/10268) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added some improvements in printing diagnostic info in input formats. Fixes [#10204](https://github.com/ClickHouse/ClickHouse/issues/10204). [#10418](https://github.com/ClickHouse/ClickHouse/pull/10418) ([tavplubix](https://github.com/tavplubix)). * Added CA certificates to clickhouse-server docker image. [#10476](https://github.com/ClickHouse/ClickHouse/pull/10476) ([filimonov](https://github.com/filimonov)). @@ -2424,7 +2424,7 @@ No changes compared to v20.4.3.16-stable. * Fixed the behaviour of `match` and `extract` functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes [#9160](https://github.com/ClickHouse/ClickHouse/issues/9160) [#9163](https://github.com/ClickHouse/ClickHouse/pull/9163) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#9345](https://github.com/ClickHouse/ClickHouse/pull/9345) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Avoid throwing from destructor in Apache Avro 3rd-party library. [#9066](https://github.com/ClickHouse/ClickHouse/pull/9066) ([Andrew Onyshchuk](https://github.com/oandrew)) * Don't commit a batch polled from `Kafka` partially as it can lead to holes in data. [#8876](https://github.com/ClickHouse/ClickHouse/pull/8876) ([filimonov](https://github.com/filimonov)) -* Fix `joinGet` with nullable return types. https://github.com/ClickHouse/ClickHouse/issues/8919 [#9014](https://github.com/ClickHouse/ClickHouse/pull/9014) ([Amos Bird](https://github.com/amosbird)) +* Fix `joinGet` with nullable return types. [#8919](https://github.com/ClickHouse/ClickHouse/issues/8919) [#9014](https://github.com/ClickHouse/ClickHouse/pull/9014) ([Amos Bird](https://github.com/amosbird)) * Fix data incompatibility when compressed with `T64` codec. [#9016](https://github.com/ClickHouse/ClickHouse/pull/9016) ([Artem Zuikov](https://github.com/4ertus2)) Fix data type ids in `T64` compression codec that leads to wrong (de)compression in affected versions. [#9033](https://github.com/ClickHouse/ClickHouse/pull/9033) ([Artem Zuikov](https://github.com/4ertus2)) * Add setting `enable_early_constant_folding` and disable it in some cases that leads to errors. [#9010](https://github.com/ClickHouse/ClickHouse/pull/9010) ([Artem Zuikov](https://github.com/4ertus2)) * Fix pushdown predicate optimizer with VIEW and enable the test [#9011](https://github.com/ClickHouse/ClickHouse/pull/9011) ([Winter Zhang](https://github.com/zhang2014)) @@ -2626,7 +2626,7 @@ No changes compared to v20.4.3.16-stable. * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). * Fix possible `Pipeline stuck` error for queries with external sort and limit. Fixes [#11359](https://github.com/ClickHouse/ClickHouse/issues/11359). [#11366](https://github.com/ClickHouse/ClickHouse/pull/11366) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix crash in `quantilesExactWeightedArray`. [#11337](https://github.com/ClickHouse/ClickHouse/pull/11337) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2636,7 +2636,7 @@ No changes compared to v20.4.3.16-stable. * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix possible error `Cannot capture column` for higher-order functions with `Array(Array(LowCardinality))` captured argument. [#11185](https://github.com/ClickHouse/ClickHouse/pull/11185) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. [#11162](https://github.com/ClickHouse/ClickHouse/pull/11162) ([Azat Khuzhin](https://github.com/azat)). * Remove logging from mutation finalization task if nothing was finalized. [#11109](https://github.com/ClickHouse/ClickHouse/pull/11109) ([alesapin](https://github.com/alesapin)). @@ -2914,7 +2914,7 @@ No changes compared to v20.4.3.16-stable. * Several improvements ClickHouse grammar in `.g4` file. [#8294](https://github.com/ClickHouse/ClickHouse/pull/8294) ([taiyang-li](https://github.com/taiyang-li)) * Fix bug that leads to crashes in `JOIN`s with tables with engine `Join`. This fixes [#7556](https://github.com/ClickHouse/ClickHouse/issues/7556) [#8254](https://github.com/ClickHouse/ClickHouse/issues/8254) [#7915](https://github.com/ClickHouse/ClickHouse/issues/7915) [#8100](https://github.com/ClickHouse/ClickHouse/issues/8100). [#8298](https://github.com/ClickHouse/ClickHouse/pull/8298) ([Artem Zuikov](https://github.com/4ertus2)) * Fix redundant dictionaries reload on `CREATE DATABASE`. [#7916](https://github.com/ClickHouse/ClickHouse/pull/7916) ([Azat Khuzhin](https://github.com/azat)) -* Limit maximum number of streams for read from `StorageFile` and `StorageHDFS`. Fixes https://github.com/ClickHouse/ClickHouse/issues/7650. [#7981](https://github.com/ClickHouse/ClickHouse/pull/7981) ([alesapin](https://github.com/alesapin)) +* Limit maximum number of streams for read from `StorageFile` and `StorageHDFS`. Fixes [#7650](https://github.com/ClickHouse/ClickHouse/issues/7650). [#7981](https://github.com/ClickHouse/ClickHouse/pull/7981) ([alesapin](https://github.com/alesapin)) * Fix bug in `ALTER ... MODIFY ... CODEC` query, when user specify both default expression and codec. Fixes [8593](https://github.com/ClickHouse/ClickHouse/issues/8593). [#8614](https://github.com/ClickHouse/ClickHouse/pull/8614) ([alesapin](https://github.com/alesapin)) * Fix error in background merge of columns with `SimpleAggregateFunction(LowCardinality)` type. [#8613](https://github.com/ClickHouse/ClickHouse/pull/8613) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Fixed type check in function `toDateTime64`. [#8375](https://github.com/ClickHouse/ClickHouse/pull/8375) ([Vasily Nemkov](https://github.com/Enmk)) From e77ab918849981cf4d95a6d23cca3a0e56c7aa46 Mon Sep 17 00:00:00 2001 From: 243f6a8885a308d313198a2e037 <243f6a8885a308d313198a2e037@gmail.com> Date: Thu, 10 Dec 2020 08:53:05 +0900 Subject: [PATCH 141/174] fix: typo and code blocks, missing explanations --- CHANGELOG.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1812fadf664..8ccb42c3603 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,7 +33,7 @@ * Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. [#16594](https://github.com/ClickHouse/ClickHouse/pull/16594) ([Amos Bird](https://github.com/amosbird)). * Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. [#16642](https://github.com/ClickHouse/ClickHouse/pull/16642) ([Maksim Kita](https://github.com/kitaisreal)). * Add `farmFingerprint64` function (non-cryptographic string hashing). [#16570](https://github.com/ClickHouse/ClickHouse/pull/16570) ([Jacob Hayes](https://github.com/JacobHayes)). -* Add `log_queries_min_query_duration_ms`, only queries slower then the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)). +* Add `log_queries_min_query_duration_ms`, only queries slower than the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)). * Ability to create a docker image on the top of `Alpine`. Uses precompiled binary and glibc components from ubuntu 20.04. [#16479](https://github.com/ClickHouse/ClickHouse/pull/16479) ([filimonov](https://github.com/filimonov)). * Added `toUUIDOrNull`, `toUUIDOrZero` cast functions. [#16337](https://github.com/ClickHouse/ClickHouse/pull/16337) ([Maksim Kita](https://github.com/kitaisreal)). * Add `max_concurrent_queries_for_all_users` setting, see [#6636](https://github.com/ClickHouse/ClickHouse/issues/6636) for use cases. [#16154](https://github.com/ClickHouse/ClickHouse/pull/16154) ([nvartolomei](https://github.com/nvartolomei)). @@ -422,7 +422,7 @@ * Fix `Missing columns` errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes [#15530](https://github.com/ClickHouse/ClickHouse/issues/15530). [#15532](https://github.com/ClickHouse/ClickHouse/pull/15532) ([alesapin](https://github.com/alesapin)). * Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in `ON CLUSTER`. Introduced in [#13450](https://github.com/ClickHouse/ClickHouse/issues/13450). [#15477](https://github.com/ClickHouse/ClickHouse/pull/15477) ([alesapin](https://github.com/alesapin)). * Report proper error when the second argument of `boundingRatio` aggregate function has a wrong type. [#15407](https://github.com/ClickHouse/ClickHouse/pull/15407) ([detailyang](https://github.com/detailyang)). -* Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. [#15319](https://github.com/ClickHouse/ClickHouse/pull/15319) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix bug where queries like `SELECT toStartOfDay(today())` fail complaining about empty time_zone argument. [#15319](https://github.com/ClickHouse/ClickHouse/pull/15319) ([Bharat Nallan](https://github.com/bharatnc)). * Fix race condition during MergeTree table rename and background cleanup. [#15304](https://github.com/ClickHouse/ClickHouse/pull/15304) ([alesapin](https://github.com/alesapin)). * Fix rare race condition on server startup when system.logs are enabled. [#15300](https://github.com/ClickHouse/ClickHouse/pull/15300) ([alesapin](https://github.com/alesapin)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -501,7 +501,7 @@ #### Performance Improvement -* Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under optimize_skip_unused_shards and optimize_distributed_group_by_sharding_key). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)). +* Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under `optimize_skip_unused_shards` and `optimize_distributed_group_by_sharding_key`). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)). * Creating sets for multiple `JOIN` and `IN` in parallel. It may slightly improve performance for queries with several different `IN subquery` expressions. [#14412](https://github.com/ClickHouse/ClickHouse/pull/14412) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Improve Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). [#13939](https://github.com/ClickHouse/ClickHouse/pull/13939) ([fastio](https://github.com/fastio)). @@ -583,7 +583,7 @@ * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust decimals field size in mysql column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). * We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). -* If function `bar` was called with specifically crafter arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). @@ -647,7 +647,7 @@ * Fix visible data clobbering by progress bar in client in interactive mode. This fixes [#12562](https://github.com/ClickHouse/ClickHouse/issues/12562) and [#13369](https://github.com/ClickHouse/ClickHouse/issues/13369) and [#13584](https://github.com/ClickHouse/ClickHouse/issues/13584) and fixes [#12964](https://github.com/ClickHouse/ClickHouse/issues/12964). [#13691](https://github.com/ClickHouse/ClickHouse/pull/13691) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect sorting order if `LowCardinality` column when sorting by multiple columns. This fixes [#13958](https://github.com/ClickHouse/ClickHouse/issues/13958). [#14223](https://github.com/ClickHouse/ClickHouse/pull/14223) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* Check for array size overflow in `topK` aggregate function. Without this check the user may send a query with carefully crafter parameters that will lead to server crash. This closes [#14452](https://github.com/ClickHouse/ClickHouse/issues/14452). [#14467](https://github.com/ClickHouse/ClickHouse/pull/14467) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Check for array size overflow in `topK` aggregate function. Without this check the user may send a query with carefully crafted parameters that will lead to server crash. This closes [#14452](https://github.com/ClickHouse/ClickHouse/issues/14452). [#14467](https://github.com/ClickHouse/ClickHouse/pull/14467) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug which can lead to wrong merges assignment if table has partitions with a single part. [#14444](https://github.com/ClickHouse/ClickHouse/pull/14444) ([alesapin](https://github.com/alesapin)). * Stop query execution if exception happened in `PipelineExecutor` itself. This could prevent rare possible query hung. Continuation of [#14334](https://github.com/ClickHouse/ClickHouse/issues/14334). [#14402](https://github.com/ClickHouse/ClickHouse/pull/14402) [#14334](https://github.com/ClickHouse/ClickHouse/pull/14334) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix crash during `ALTER` query for table which was created `AS table_function`. Fixes [#14212](https://github.com/ClickHouse/ClickHouse/issues/14212). [#14326](https://github.com/ClickHouse/ClickHouse/pull/14326) ([alesapin](https://github.com/alesapin)). @@ -788,7 +788,7 @@ * Add `FROM_UNIXTIME` function for compatibility with MySQL, related to [12149](https://github.com/ClickHouse/ClickHouse/issues/12149). [#12484](https://github.com/ClickHouse/ClickHouse/pull/12484) ([flynn](https://github.com/ucasFL)). * Allow Nullable types as keys in MergeTree tables if `allow_nullable_key` table setting is enabled. Closes [#5319](https://github.com/ClickHouse/ClickHouse/issues/5319). [#12433](https://github.com/ClickHouse/ClickHouse/pull/12433) ([Amos Bird](https://github.com/amosbird)). * Integration with [COS](https://intl.cloud.tencent.com/product/cos). [#12386](https://github.com/ClickHouse/ClickHouse/pull/12386) ([fastio](https://github.com/fastio)). -* Add mapAdd and mapSubtract functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). +* Add `mapAdd` and `mapSubtract` functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). #### Bug Fix @@ -1071,7 +1071,7 @@ * Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with `optimize_aggregation_in_order` setting, disabled by default). [#11696](https://github.com/ClickHouse/ClickHouse/pull/11696) ([Anton Popov](https://github.com/CurtizJ)). * Removed injective functions inside `uniq*()` if `set optimize_injective_functions_inside_uniq=1`. [#12337](https://github.com/ClickHouse/ClickHouse/pull/12337) ([Ruslan Kamalov](https://github.com/kamalov-ruslan)). -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). * Implemented single part uploads for DiskS3 (experimental feature). [#12026](https://github.com/ClickHouse/ClickHouse/pull/12026) ([Vladimir Chebotarev](https://github.com/excitoon)). #### Experimental Feature @@ -1133,7 +1133,7 @@ #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). #### Build/Testing/Packaging Improvement @@ -1331,7 +1331,7 @@ * Fix error `the BloomFilter false positive must be a double number between 0 and 1` [#10551](https://github.com/ClickHouse/ClickHouse/issues/10551). [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). * Fix SELECT of column ALIAS which default expression type different from column type. [#10563](https://github.com/ClickHouse/ClickHouse/pull/10563) ([Azat Khuzhin](https://github.com/azat)). * Implemented comparison between DateTime64 and String values (just like for DateTime). [#10560](https://github.com/ClickHouse/ClickHouse/pull/10560) ([Vasily Nemkov](https://github.com/Enmk)). -* Fix index corruption, which may accur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). +* Fix index corruption, which may occur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). * Disable GROUP BY sharding_key optimization by default (`optimize_distributed_group_by_sharding_key` had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is `if` in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. [#10516](https://github.com/ClickHouse/ClickHouse/pull/10516) ([Azat Khuzhin](https://github.com/azat)). * Fixes: [#10263](https://github.com/ClickHouse/ClickHouse/issues/10263) (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: [#8756](https://github.com/ClickHouse/ClickHouse/issues/8756) (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): `internal_replication == false`, multiple local shards (activates the hardlinking code) and `distributed_storage_policy` (makes `link(2)` fails on `EXDEV`)). [#10486](https://github.com/ClickHouse/ClickHouse/pull/10486) ([Azat Khuzhin](https://github.com/azat)). * Fixed error with "max_rows_to_sort" limit. [#10268](https://github.com/ClickHouse/ClickHouse/pull/10268) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1563,7 +1563,7 @@ #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). #### Build/Testing/Packaging Improvement @@ -1679,7 +1679,7 @@ No changes compared to v20.4.3.16-stable. * Now constraints are updated if the column participating in `CONSTRAINT` expression was renamed. Fixes [#10844](https://github.com/ClickHouse/ClickHouse/issues/10844). [#10847](https://github.com/ClickHouse/ClickHouse/pull/10847) ([alesapin](https://github.com/alesapin)). * Fixed potential read of uninitialized memory in cache-dictionary. [#10834](https://github.com/ClickHouse/ClickHouse/pull/10834) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed columns order after `Block::sortColumns()`. [#10826](https://github.com/ClickHouse/ClickHouse/pull/10826) ([Azat Khuzhin](https://github.com/azat)). -* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984](https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UBSan` and `MSan` report in `DateLUT`. [#10798](https://github.com/ClickHouse/ClickHouse/pull/10798) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect type conversion in key conditions. Fixes [#6287](https://github.com/ClickHouse/ClickHouse/issues/6287). [#10791](https://github.com/ClickHouse/ClickHouse/pull/10791) ([Andrew Onyshchuk](https://github.com/oandrew)). * Fixed `parallel_view_processing` behavior. Now all insertions into `MATERIALIZED VIEW` without exception should be finished if exception happened. Fixes [#10241](https://github.com/ClickHouse/ClickHouse/issues/10241). [#10757](https://github.com/ClickHouse/ClickHouse/pull/10757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1707,15 +1707,15 @@ No changes compared to v20.4.3.16-stable. #### New Feature * Add support for secured connection from ClickHouse to Zookeeper [#10184](https://github.com/ClickHouse/ClickHouse/pull/10184) ([Konstantin Lebedev](https://github.com/xzkostyan)) -* Support custom HTTP handlers. See ISSUES-5436 for description. [#7572](https://github.com/ClickHouse/ClickHouse/pull/7572) ([Winter Zhang](https://github.com/zhang2014)) +* Support custom HTTP handlers. See [#5436](https://github.com/ClickHouse/ClickHouse/issues/5436) for description. [#7572](https://github.com/ClickHouse/ClickHouse/pull/7572) ([Winter Zhang](https://github.com/zhang2014)) * Add MessagePack Input/Output format. [#9889](https://github.com/ClickHouse/ClickHouse/pull/9889) ([Kruglov Pavel](https://github.com/Avogar)) * Add Regexp input format. [#9196](https://github.com/ClickHouse/ClickHouse/pull/9196) ([Kruglov Pavel](https://github.com/Avogar)) * Added output format `Markdown` for embedding tables in markdown documents. [#10317](https://github.com/ClickHouse/ClickHouse/pull/10317) ([Kruglov Pavel](https://github.com/Avogar)) * Added support for custom settings section in dictionaries. Also fixes issue [#2829](https://github.com/ClickHouse/ClickHouse/issues/2829). [#10137](https://github.com/ClickHouse/ClickHouse/pull/10137) ([Artem Streltsov](https://github.com/kekekekule)) -* Added custom settings support in DDL-queries for CREATE DICTIONARY [#10465](https://github.com/ClickHouse/ClickHouse/pull/10465) ([Artem Streltsov](https://github.com/kekekekule)) +* Added custom settings support in DDL-queries for `CREATE DICTIONARY` [#10465](https://github.com/ClickHouse/ClickHouse/pull/10465) ([Artem Streltsov](https://github.com/kekekekule)) * Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. [#10444](https://github.com/ClickHouse/ClickHouse/pull/10444) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add setting `always_fetch_merged_part` which restrict replica to merge parts by itself and always prefer dowloading from other replicas. [#10379](https://github.com/ClickHouse/ClickHouse/pull/10379) ([alesapin](https://github.com/alesapin)) -* Add function JSONExtractKeysAndValuesRaw which extracts raw data from JSON objects [#10378](https://github.com/ClickHouse/ClickHouse/pull/10378) ([hcz](https://github.com/hczhcz)) +* Add function `JSONExtractKeysAndValuesRaw` which extracts raw data from JSON objects [#10378](https://github.com/ClickHouse/ClickHouse/pull/10378) ([hcz](https://github.com/hczhcz)) * Add memory usage from OS to `system.asynchronous_metrics`. [#10361](https://github.com/ClickHouse/ClickHouse/pull/10361) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Added generic variants for functions `least` and `greatest`. Now they work with arbitrary number of arguments of arbitrary types. This fixes [#4767](https://github.com/ClickHouse/ClickHouse/issues/4767) [#10318](https://github.com/ClickHouse/ClickHouse/pull/10318) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: `strict_max_lifetime_seconds`, which is `max_lifetime` by default, and `query_wait_timeout_milliseconds`, which is one minute by default. The first settings is also useful with `allow_read_expired_keys` settings (to forbid reading very expired keys). [#10337](https://github.com/ClickHouse/ClickHouse/pull/10337) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) @@ -1728,7 +1728,7 @@ No changes compared to v20.4.3.16-stable. * Add ability to query Distributed over Distributed (w/o `distributed_group_by_no_merge`) ... [#9923](https://github.com/ClickHouse/ClickHouse/pull/9923) ([Azat Khuzhin](https://github.com/azat)) * Add function `arrayReduceInRanges` which aggregates array elements in given ranges. [#9598](https://github.com/ClickHouse/ClickHouse/pull/9598) ([hcz](https://github.com/hczhcz)) * Add Dictionary Status on prometheus exporter. [#9622](https://github.com/ClickHouse/ClickHouse/pull/9622) ([Guillaume Tassery](https://github.com/YiuRULE)) -* Add function arrayAUC [#8698](https://github.com/ClickHouse/ClickHouse/pull/8698) ([taiyang-li](https://github.com/taiyang-li)) +* Add function `arrayAUC` [#8698](https://github.com/ClickHouse/ClickHouse/pull/8698) ([taiyang-li](https://github.com/taiyang-li)) * Support `DROP VIEW` statement for better TPC-H compatibility. [#9831](https://github.com/ClickHouse/ClickHouse/pull/9831) ([Amos Bird](https://github.com/amosbird)) * Add 'strict_order' option to windowFunnel() [#9773](https://github.com/ClickHouse/ClickHouse/pull/9773) ([achimbab](https://github.com/achimbab)) * Support `DATE` and `TIMESTAMP` SQL operators, e.g. `SELECT date '2001-01-01'` [#9691](https://github.com/ClickHouse/ClickHouse/pull/9691) ([Artem Zuikov](https://github.com/4ertus2)) @@ -2106,7 +2106,7 @@ No changes compared to v20.4.3.16-stable. #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). ### ClickHouse release v20.3.12.112-lts 2020-06-25 @@ -2196,7 +2196,7 @@ No changes compared to v20.4.3.16-stable. * Fixed `SIGSEGV` in `StringHashTable` if such a key does not exist. [#10870](https://github.com/ClickHouse/ClickHouse/pull/10870) ([Azat Khuzhin](https://github.com/azat)). * Fixed bug in `ReplicatedMergeTree` which might cause some `ALTER` on `OPTIMIZE` query to hang waiting for some replica after it become inactive. [#10849](https://github.com/ClickHouse/ClickHouse/pull/10849) ([tavplubix](https://github.com/tavplubix)). * Fixed columns order after `Block::sortColumns()`. [#10826](https://github.com/ClickHouse/ClickHouse/pull/10826) ([Azat Khuzhin](https://github.com/azat)). -* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984](https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UBSan` and `MSan` report in `DateLUT`. [#10798](https://github.com/ClickHouse/ClickHouse/pull/10798) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect type conversion in key conditions. Fixes [#6287](https://github.com/ClickHouse/ClickHouse/issues/6287). [#10791](https://github.com/ClickHouse/ClickHouse/pull/10791) ([Andrew Onyshchuk](https://github.com/oandrew)) * Fixed `parallel_view_processing` behavior. Now all insertions into `MATERIALIZED VIEW` without exception should be finished if exception happened. Fixes [#10241](https://github.com/ClickHouse/ClickHouse/issues/10241). [#10757](https://github.com/ClickHouse/ClickHouse/pull/10757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2215,7 +2215,7 @@ No changes compared to v20.4.3.16-stable. * Fixed incorrect scalar results inside inner query of `MATERIALIZED VIEW` in case if this query contained dependent table. [#10603](https://github.com/ClickHouse/ClickHouse/pull/10603) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed `SELECT` of column `ALIAS` which default expression type different from column type. [#10563](https://github.com/ClickHouse/ClickHouse/pull/10563) ([Azat Khuzhin](https://github.com/azat)). * Implemented comparison between DateTime64 and String values. [#10560](https://github.com/ClickHouse/ClickHouse/pull/10560) ([Vasily Nemkov](https://github.com/Enmk)). -* Fixed index corruption, which may accur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). +* Fixed index corruption, which may occur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). * Fixed the situation, when mutation finished all parts, but hung up in `is_done=0`. [#10526](https://github.com/ClickHouse/ClickHouse/pull/10526) ([alesapin](https://github.com/alesapin)). * Fixed overflow at beginning of unix epoch for timezones with fractional offset from `UTC`. This fixes [#9335](https://github.com/ClickHouse/ClickHouse/issues/9335). [#10513](https://github.com/ClickHouse/ClickHouse/pull/10513) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed improper shutdown of `Distributed` storage. [#10491](https://github.com/ClickHouse/ClickHouse/pull/10491) ([Azat Khuzhin](https://github.com/azat)). @@ -2232,7 +2232,7 @@ No changes compared to v20.4.3.16-stable. #### Bug fix -* #10551. [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). +* Fix error `the BloomFilter false positive must be a double number between 0 and 1` [#10551](https://github.com/ClickHouse/ClickHouse/issues/10551). [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). ### ClickHouse release v20.3.8.53, 2020-04-23 From b3419c5d328ab404f84f8fd0f4a4eaf856d5f21c Mon Sep 17 00:00:00 2001 From: 243f6a8885a308d313198a2e037 <243f6a8885a308d313198a2e037@gmail.com> Date: Thu, 10 Dec 2020 09:36:55 +0900 Subject: [PATCH 142/174] fix: Materialize d MySQL -> MaterializeMySQL --- CHANGELOG.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ccb42c3603..4d525798505 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -178,7 +178,7 @@ * Add `JSONStrings` format which output data in arrays of strings. [#14333](https://github.com/ClickHouse/ClickHouse/pull/14333) ([hcz](https://github.com/hczhcz)). * Add support for "Raw" column format for `Regexp` format. It allows to simply extract subpatterns as a whole without any escaping rules. [#15363](https://github.com/ClickHouse/ClickHouse/pull/15363) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow configurable `NULL` representation for `TSV` output format. It is controlled by the setting `output_format_tsv_null_representation` which is `\N` by default. This closes [#9375](https://github.com/ClickHouse/ClickHouse/issues/9375). Note that the setting only controls output format and `\N` is the only supported `NULL` representation for `TSV` input format. [#14586](https://github.com/ClickHouse/ClickHouse/pull/14586) ([Kruglov Pavel](https://github.com/Avogar)). -* Support Decimal data type for `MaterializedMySQL`. `MaterializedMySQL` is an experimental feature. [#14535](https://github.com/ClickHouse/ClickHouse/pull/14535) ([Winter Zhang](https://github.com/zhang2014)). +* Support Decimal data type for `MaterializeMySQL`. `MaterializeMySQL` is an experimental feature. [#14535](https://github.com/ClickHouse/ClickHouse/pull/14535) ([Winter Zhang](https://github.com/zhang2014)). * Add new feature: `SHOW DATABASES LIKE 'xxx'`. [#14521](https://github.com/ClickHouse/ClickHouse/pull/14521) ([hexiaoting](https://github.com/hexiaoting)). * Added a script to import (arbitrary) git repository to ClickHouse as a sample dataset. [#14471](https://github.com/ClickHouse/ClickHouse/pull/14471) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Now insert statements can have asterisk (or variants) with column transformers in the column list. [#14453](https://github.com/ClickHouse/ClickHouse/pull/14453) ([Amos Bird](https://github.com/amosbird)). @@ -200,18 +200,18 @@ * Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. [#16264](https://github.com/ClickHouse/ClickHouse/pull/16264) ([Amos Bird](https://github.com/amosbird)). * Fix segfault in some cases of wrong aggregation in lambdas. [#16082](https://github.com/ClickHouse/ClickHouse/pull/16082) ([Anton Popov](https://github.com/CurtizJ)). * Fix `ALTER MODIFY ... ORDER BY` query hang for `ReplicatedVersionedCollapsingMergeTree`. This fixes [#15980](https://github.com/ClickHouse/ClickHouse/issues/15980). [#16011](https://github.com/ClickHouse/ClickHouse/pull/16011) ([alesapin](https://github.com/alesapin)). -* `MaterializedMySQL` (experimental feature): Fix collate name & charset name parser and support `length = 0` for string type. [#16008](https://github.com/ClickHouse/ClickHouse/pull/16008) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fix collate name & charset name parser and support `length = 0` for string type. [#16008](https://github.com/ClickHouse/ClickHouse/pull/16008) ([Winter Zhang](https://github.com/zhang2014)). * Allow to use `direct` layout for dictionaries with complex keys. [#16007](https://github.com/ClickHouse/ClickHouse/pull/16007) ([Anton Popov](https://github.com/CurtizJ)). * Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. [#15987](https://github.com/ClickHouse/ClickHouse/pull/15987) ([filimonov](https://github.com/filimonov)). * Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). [#15984](https://github.com/ClickHouse/ClickHouse/pull/15984) ([tavplubix](https://github.com/tavplubix)). * Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes [#15628](https://github.com/ClickHouse/ClickHouse/issues/15628). [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). -* `MaterializedMySQL` (experimental feature): Fix crash on create database failure. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fix crash on create database failure. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). * Fixed `DROP TABLE IF EXISTS` failure with `Table ... doesn't exist` error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like `DROP DATABASE` and `RENAME TABLE`) - Fixed `DROP/DETACH DATABASE` failure with `Table ... doesn't exist` when concurrently executing `DROP/DETACH TABLE`. [#15934](https://github.com/ClickHouse/ClickHouse/pull/15934) ([tavplubix](https://github.com/tavplubix)). * Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes [#15792](https://github.com/ClickHouse/ClickHouse/issues/15792). [#15933](https://github.com/ClickHouse/ClickHouse/pull/15933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixes [#12513](https://github.com/ClickHouse/ClickHouse/issues/12513): difference expressions with same alias when query is reanalyzed. [#15886](https://github.com/ClickHouse/ClickHouse/pull/15886) ([Winter Zhang](https://github.com/zhang2014)). * Fix possible very rare deadlocks in RBAC implementation. [#15875](https://github.com/ClickHouse/ClickHouse/pull/15875) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix exception `Block structure mismatch` in `SELECT ... ORDER BY DESC` queries which were executed after `ALTER MODIFY COLUMN` query. Fixes [#15800](https://github.com/ClickHouse/ClickHouse/issues/15800). [#15852](https://github.com/ClickHouse/ClickHouse/pull/15852) ([alesapin](https://github.com/alesapin)). -* `MaterializedMySQL` (experimental feature): Fix `select count()` inaccuracy. [#15767](https://github.com/ClickHouse/ClickHouse/pull/15767) ([tavplubix](https://github.com/tavplubix)). +* `MaterializeMySQL` (experimental feature): Fix `select count()` inaccuracy. [#15767](https://github.com/ClickHouse/ClickHouse/pull/15767) ([tavplubix](https://github.com/tavplubix)). * Fix some cases of queries, in which only virtual columns are selected. Previously `Not found column _nothing in block` exception may be thrown. Fixes [#12298](https://github.com/ClickHouse/ClickHouse/issues/12298). [#15756](https://github.com/ClickHouse/ClickHouse/pull/15756) ([Anton Popov](https://github.com/CurtizJ)). * Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). [#15743](https://github.com/ClickHouse/ClickHouse/pull/15743) ([Azat Khuzhin](https://github.com/azat)). * Possibility to move part to another disk/volume if the first attempt was failed. [#15723](https://github.com/ClickHouse/ClickHouse/pull/15723) ([Pavel Kovalenko](https://github.com/Jokser)). @@ -255,11 +255,11 @@ * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains `finalizeAggregation` function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). * Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). -* `MaterializedMySQL` (experimental feature): Fixed `.metadata.tmp File exists` error. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fixed `.metadata.tmp File exists` error. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile with file descriptor. [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). * Fixed segfault in `cache` dictionary [#14837](https://github.com/ClickHouse/ClickHouse/issues/14837). [#14879](https://github.com/ClickHouse/ClickHouse/pull/14879) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* `MaterializedMySQL` (experimental feature): Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. [#14852](https://github.com/ClickHouse/ClickHouse/pull/14852) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. [#14852](https://github.com/ClickHouse/ClickHouse/pull/14852) ([Winter Zhang](https://github.com/zhang2014)). * Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes [#14531](https://github.com/ClickHouse/ClickHouse/issues/14531). [#14845](https://github.com/ClickHouse/ClickHouse/pull/14845) ([alesapin](https://github.com/alesapin)). * Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes [#14814](https://github.com/ClickHouse/ClickHouse/issues/14814). [#14843](https://github.com/ClickHouse/ClickHouse/pull/14843) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)). @@ -273,7 +273,7 @@ * Fix a bug when converting `Nullable(String)` to Enum. Introduced by [#12745](https://github.com/ClickHouse/ClickHouse/pull/12745). This fixes [#14435](https://github.com/ClickHouse/ClickHouse/issues/14435). [#14530](https://github.com/ClickHouse/ClickHouse/pull/14530) ([Amos Bird](https://github.com/amosbird)). * Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix `currentDatabase()` function cannot be used in `ON CLUSTER` ddl query. [#14211](https://github.com/ClickHouse/ClickHouse/pull/14211) ([Winter Zhang](https://github.com/zhang2014)). -* `MaterializedMySQL` (experimental feature): Fixed `Packet payload is not fully read` error in `MaterializeMySQL` database engine. [#14696](https://github.com/ClickHouse/ClickHouse/pull/14696) ([BohuTANG](https://github.com/BohuTANG)). +* `MaterializeMySQL` (experimental feature): Fixed `Packet payload is not fully read` error in `MaterializeMySQL` database engine. [#14696](https://github.com/ClickHouse/ClickHouse/pull/14696) ([BohuTANG](https://github.com/BohuTANG)). #### Improvement From b63274e999b1c3565a88e2499f906ba4718fea59 Mon Sep 17 00:00:00 2001 From: Robert Hodges Date: Wed, 9 Dec 2020 19:12:28 -0800 Subject: [PATCH 143/174] Added blog article on ClickHouse Community. --- .../blog/en/2020/the-clickhouse-community.md | 245 ++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 website/blog/en/2020/the-clickhouse-community.md diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md new file mode 100644 index 00000000000..993f30cdf59 --- /dev/null +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -0,0 +1,245 @@ +--- +title: 'The ClickHouse Community' +image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png +date: '2020-12-10' +author: '[Robert Hodges](https://github.com/hodgesrm)' +tags: ['community', 'open source', 'telegram', 'meetup'] +--- + +One of the great “features” of ClickHouse is a friendly and welcoming +community. In this article we would like to outline how the ClickHouse +community arose, what it is today, and how you can get involved. There +is a role for everyone, from end users to contributors to corporate +friends. Our goal is to make the community welcoming to every person +who wants to join. + +But first, let’s review a bit of history, starting with how ClickHouse +first developed at [Yandex](https://yandex.com/company/). + +## Origins at Yandex + +ClickHouse began as a solution for web analytics in [Yandex +Metrica](https://metrica.yandex.com/about?). Metrica is a popular service for analyzing website traffic that +is now #2 in the market behind Google Analytics. In 2008 [Alexey Milovidov](https://github.com/alexey-milovidov), +an engineer on the Metrica team, was looking for a database that could +create reports on metrics like number of page views per day, unique +visitors, and bounce rate, without aggregating the data in advance. The +idea was to provide a wide range of metric data and let users ask any +question about them. + +This is a classic problem for data warehouses. However, Alexey could +not find one that met Yandex requirements, specifically large datasets, +linear scaling, high efficiency, and compatibility with SQL tools. In +a nutshell: like MySQL but for analytic applications. So Alexey wrote +one. It started as a prototype to do GROUP BY operations. + +The prototype evolved into a full solution with a name, ClickHouse, +short for “Clickstream Data Warehouse”. Alexey added additional +features including SQL support and the MergeTree engine. The SQL dialect +was superficially similar to MySQL, [which was also used in Metrica](https://clickhouse.tech/blog/en/2016/evolution-of-data-structures-in-yandex-metrica/) but +could not handle query workloads without complex pre-aggregation. By +2011 ClickHouse was in production for Metrica. + +Over the next 5 years Alexey and a growing team of developers extended +ClickHouse to cover new use cases. By 2016 ClickHouse was a core Metrica +backend service. It was also becoming entrenched as a data warehouse +within Yandex, extending to use cases like service monitoring, network +flow logs, and event management. ClickHouse had evolved from the original +one-person project to business critical software with a full team of a +dozen engineers led by Alexey. + +By 2016, ClickHouse had an 8 year history and was ready to become a major +open source project. Here’s a timeline that tracks major developments +as a time series. + + + +## ClickHouse goes open source + +Yandex open sourced ClickHouse under an Apache 2.0 license in 2016. There +were numerous reasons for this step. + +* Promote adoption within Yandex by making it easier for internal departments to get builds. +* Ensure that ClickHouse would continue to evolve by creating a community to nurture it. +* Motivate developers to contribute to and use ClickHouse due to the open source “cool” factor. +* Improve ClickHouse quality by making the code public. Nobody wants their name visible on bad code. ;-) +* Showcase Yandex innovation to a worldwide audience + +Alexey and the development team moved ClickHouse code to a Github repo +under the Yandex organization and began issuing community builds as well +as accepting external contributions. They simultaneously began regular +meetups to popularize ClickHouse and build a community around it. The +result was a burst of adoption across multiple regions of the globe. + +ClickHouse quickly picked up steam in Eastern Europe. The first ClickHouse +meetups started in 2016 and have grown to include 200 participants for +in-person meetings and up to 400 for online meetings. ClickHouse is +now widely used in start-ups in Russia as well as other Eastern European +countries. Developers located in Eastern Europe continue to supply more +contributions to ClickHouse than any other region. + +ClickHouse also started to gain recognition in the US and Western Europe. +[CloudFlare](https://www.cloudflare.com/) published a widely read blog +article about [their success using ClickHouse for DNS analytics](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/). Alexander +Zaitsev successfully migrated an ad tech analytics system from a +commercial DBMS to a ClickHouse cluster. This success prompted him to +found [Altinity](https://altinity.com) in 2017 with help from friends at +[Percona](https://www.percona.com). US meetups +started in the same year. With support from Altinity these have grown +to over 100 attendees for online meetings. + +ClickHouse also took off in China. The first meetup in China took place +in 2018 and attracted enormous interest. In-person meetups included over +400 participants. Online meetings have reached up to 1000 online viewers. + +In 2019 a further step occurred as ClickHouse moved out from under the +Yandex Github organization into a separate [ClickHouse organization](https://github.com/ClickHouse). The +new organization includes ClickHouse server code plus core ecosystem +projects like the cpp and ODBC drivers. + +ClickHouse community events shifted online following world-wide +disruptions due to COVID-19, but growth in usage continued. One +interesting development has been the increasing number of startups using +ClickHouse as a backend. Many of these are listed on the +[ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) +page. Also, additional prominent companies like eBay, Uber, and +Flipcart went public in 2020 with stories of successful ClickHouse usage. + +## The ClickHouse community today + +As of 2020 the ClickHouse community includes developers and users from +virtually every region of the globe. Yandex engineers continue to supply +a majority of pull requests to ClickHouse itself. Altinity follows +in second place with contributions to ClickHouse core and ecosystem +projects. There is also substantial in-house development on ClickHouse +(e.g. on private forks) within Chinese internet providers. + +The real success, however, has been the huge number of commits to ClickHouse core from people in outside organizations. The following list shows the main outside contributors: + +* Azat Khuzhin +* Amos Bird +* Winter Zhang +* Denny Crane +* Danila Kutenin +* Hczhcz +* Marek Vavruša +* Guillaume Tassery +* Sundy Li +* Mikhail Shiryaev +* Nicolae Vartolomei +* Igor Hatarist +* Andrew Onyshchuk +* BohuTANG +* Yu Zhi Chang +* Kirill Shvakov +* Alexander Krasheninnikov +* Simon Podlipsky +* Silviu Caragea +* Flynn ucasFL +* [And over 550 more](https://github.com/ClickHouse/ClickHouse/graphs/contributors)... + +ClickHouse ecosystem projects are also growing rapidly. Here is a selected list of active Github projects that help enable ClickHouse applications, sorted by number of stars. + +* [sqlpad/sqlpad](https://github.com/sqlpad/sqlpad) -- Web-based SQL editor that supports ClickHouse +* [mindsdb/mindsdb](https://github.com/mindsdb/mindsdb) -- Predictive AI layer for databases with ClickHouse support +* [x-ream/sqli](https://github.com/x-ream/sqli) -- ORM SQL interface +* [tricksterproxy/trickster](https://github.com/tricksterproxy/trickster) -- HTTP reverse proxy cache and time series dashboard accelerator +* [ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go) -- Golang driver for ClickHouse +* [gohouse/gorose](https://github.com/gohouse/gorose) -- A mini database ORM for Golang +* [ClickHouse/clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) -- JDBC driver for ClickHouse +* [brockercap/Bifrost](https://github.com/brokercap/Bifrost) -- Middleware to sync MySQL binlog to ClickHouse +* [mymarilyn/clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) -- ClickHouse Python driver with native interface support +* [Vertamedia/clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) -- Grafana datasource for ClickHouse +* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) -- PHP Clickhouse client +* [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) -- Kubernetes operator for ClickHouse +* [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) -- ClickHouse backup and restore using cloud storage +* [And almost 1200 more](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories)... + +## Resources + +With the community growth numerous resources are available to users. At +the center is the [ClickHouse org on Github](https://github.com/ClickHouse), which hosts [ClickHouse +server code](https://github.com/ClickHouse/ClickHouse). ClickHouse server documentation is available at the +[clickhouse.tech](https://clickhouse.tech/) website. It has [installation instructions](https://clickhouse.tech/docs/en/getting-started/install/) and links to +ClickHouse community builds for major Linux distributions as well as Mac, +FreeBSD, and Docker. + +In addition, ClickHouse users have a wide range of ways to engage with +the community and get help on applications. These include both chat +applications as well as meetups. Here are some links to get started. + +* Yandex Meetups -- Yandex has regular in-person and online Russian-language meetups. Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). +* [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) -- The largest US ClickHouse meetup, with meetings approximately every 2 months. +* Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. +* Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. + * [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“Clickhouse does not slow down”) - Russian language Telegram group with 4,629 members currently. + * [ClickHouse](https://t.me/clickhouse_en) -- English language group with 1,286 members. +* [ClickHouse Community Slack Channel](http://clickhousedb.slack.com) -- Public channel for Slack users. It currently has 551 members. +* [Clickhouse.com.cn](http://clickhouse.com.cn/) -- Chinese language site for ClickHouse-related announcements and questions. +* [Conference Presentations](https://github.com/ClickHouse/clickhouse-presentations) -- ClickHouse developers like to talk and do so whenever they can. Many recent presentations are stored in Github. Also, look for ClickHouse presentations at Linux Foundation conferences, Data Con LA, Percona Live, and many other venues where there are presentations about data. +* Technical webinars -- Altinity has a large library of technical presentations on ClickHouse and related applications on the [Altinity Youtube channel](https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA/featured). + +If you know of additional resources please bring them to our attention. + +## How you can contribute to ClickHouse + +We welcome users to join the ClickHouse community in every capacity. There are four main ways to participate. + +### Use ClickHouse and share your experiences + +Start with the documentation. Download ClickHouse and try +it out. Join the chat channels. If you encounter bugs, [log +issues](https://github.com/ClickHouse/ClickHouse/issues) so we can get +them fixed. Also, it’s easy to make contributions to the documentation +if you have basic Github and markdown skills. Press the pencil icon on +any page of the clickhouse.tech website to edit pages and automatically +generate pull requests to merge your changes. + +If your company has deployed ClickHouse and is comfortable talking about +it, please don't be shy. Add them to the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page so +that others can learn from your experience. + +### Become a ClickHouse developer + +Write code to make ClickHouse better. Here are your choices. + +* ClickHouse server -- Start with the [“For Beginners” documentation](https://clickhouse.tech/docs/en/development/developer-instruction/) to learn how to build ClickHouse and submit PRs. Check out the current ClickHouse issues if you are looking for work. PRs that follow the development standards will be merged faster. + +* Ecosystem projects -- Most projects in the ClickHouse ecosystem accept PRs. Check with each project for specific practices. + +ClickHouse is also a great target for research problems. Overall the +years many dozens of university CS students have worked on ClickHouse +features. Alexey Milovidov maintains an especially rich set of [project +suggestions for students](https://github.com/ClickHouse/ClickHouse/issues/15065). Join Telegram and ask for help if you are +interested. Both Yandex and Altinity also offer internships. + +### Write ClickHouse applications + +ClickHouse enables a host of new applications that depend on low +latency access to large datasets. If you write something interesting, +blog about it and present at local meetups. Altinity has a program to +highlight startups who are developing ClickHouse applications and help +with marketing as well as resources for development. Send email to +[info@altinity.com](mailto:info@altinity.com) for more information. + +### Become a corporate sponsor + +The ClickHouse community has been assisted by many corporate +users who have helped organize meetups, funded development, and +guided growth of ClickHouse. Contact community members directly at +[clickhouse-feedback@yandex-team.ru](mailto:clickhouse-feedback@yandex-team.ru), [info@altinity.com](mailto:info@altinity.com), or via Telegram +to find out more about how to chip in as a corporate sponsor. + +## Where we go from here + +ClickHouse has grown enormously from its origins as a basic prototype in +2008 to the popular SQL data warehouse users see today. Our community is +the rock that will enable ClickHouse to become the default data warehouse +worldwide. We are working together to create an inclusive environment +where everyone feels welcome and has an opportunity to contribute. We +welcome you to join! + +This article was written with kind assistance from Alexey Milovidov, +Ivan Blinkov, and Alexander Zaitsev + +_2020-12-11 [Robert Hodges](https://github.com/hodgesrm)_ From f68a231a4631f16c5ef19116182b171dc7411bd7 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Thu, 10 Dec 2020 05:19:51 +0300 Subject: [PATCH 144/174] Fix installing top_level_domains for tests --- tests/CMakeLists.txt | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3ef09e5658f..9e5a2e29dc9 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -15,6 +15,16 @@ install ( COMPONENT clickhouse PATTERN "CMakeLists.txt" EXCLUDE PATTERN ".gitignore" EXCLUDE + PATTERN "top_level_domains" EXCLUDE +) + +# Dereference symlink +get_filename_component(TOP_LEVEL_DOMAINS_ABS_DIR config/top_level_domains REALPATH) +install ( + DIRECTORY "${TOP_LEVEL_DOMAINS_ABS_DIR}" + DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/clickhouse-test/config + USE_SOURCE_PERMISSIONS + COMPONENT clickhouse ) install (FILES server-test.xml DESTINATION ${CLICKHOUSE_ETC_DIR}/clickhouse-server COMPONENT clickhouse) From 5fc67379ea1cd9444a47613246b73e9a873ff804 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Steve-=E9=87=91=E5=8B=87?= Date: Thu, 10 Dec 2020 16:36:05 +0800 Subject: [PATCH 145/174] Update introspection.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Improve translation of Chinese. There are several language parts that hard to understand. 优化一些翻译，主要是语句不通的地方。 --- docs/zh/sql-reference/functions/introspection.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/zh/sql-reference/functions/introspection.md b/docs/zh/sql-reference/functions/introspection.md index 8d3b909b199..c0bbc10131d 100644 --- a/docs/zh/sql-reference/functions/introspection.md +++ b/docs/zh/sql-reference/functions/introspection.md @@ -18,7 +18,7 @@ toc_title: "\u81EA\u7701" - 设置 [allow_introspection_functions](../../operations/settings/settings.md#settings-allow_introspection_functions) 设置为1。 - For security reasons introspection functions are disabled by default. + 出于安全考虑，内省函数默认是关闭的。 ClickHouse将探查器报告保存到 [trace_log](../../operations/system-tables/trace_log.md#system_tables-trace_log) 系统表. 确保正确配置了表和探查器。 @@ -36,17 +36,17 @@ addressToLine(address_of_binary_instruction) **参数** -- `address_of_binary_instruction` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Address of instruction in a running process. +- `address_of_binary_instruction` ([UInt64](../../sql-reference/data-types/int-uint.md)) — 正在运行进程的指令地址。 **返回值** -- 源代码文件名和此文件中用冒号分隔的行号。 +- 源代码文件名和行号（用冒号分隔的行号） - For example, `/build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:199`, where `199` is a line number. + 示例, `/build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:199`, where `199` is a line number. -- 二进制文件的名称，如果函数找不到调试信息。 +- 如果函数找不到调试信息，返回二进制文件的名称。 -- 空字符串，如果地址无效。 +- 如果地址无效，返回空字符串。 类型: [字符串](../../sql-reference/data-types/string.md). @@ -132,7 +132,7 @@ addressToSymbol(address_of_binary_instruction) **返回值** - 来自ClickHouse对象文件的符号。 -- 空字符串，如果地址无效。 +- 如果地址无效，返回空字符串。 类型: [字符串](../../sql-reference/data-types/string.md). From 39d867e56d4a36c45d2600d694857a5786bfc0ea Mon Sep 17 00:00:00 2001 From: Vladimir Chebotarev Date: Thu, 10 Dec 2020 12:19:42 +0300 Subject: [PATCH 146/174] Added proper authentication for S3 client (#16856) --- src/Disks/S3/registerDiskS3.cpp | 1 + src/IO/S3Common.cpp | 120 ++++++++++++++++++- src/IO/S3Common.h | 3 + src/Storages/StorageS3.cpp | 10 +- src/Storages/StorageS3Settings.cpp | 7 +- src/Storages/StorageS3Settings.h | 3 + tests/integration/test_log_family_s3/test.py | 5 + 7 files changed, 141 insertions(+), 8 deletions(-) diff --git a/src/Disks/S3/registerDiskS3.cpp b/src/Disks/S3/registerDiskS3.cpp index 5078a7e06ae..809d6728189 100644 --- a/src/Disks/S3/registerDiskS3.cpp +++ b/src/Disks/S3/registerDiskS3.cpp @@ -134,6 +134,7 @@ void registerDiskS3(DiskFactory & factory) uri.is_virtual_hosted_style, config.getString(config_prefix + ".access_key_id", ""), config.getString(config_prefix + ".secret_access_key", ""), + config.getBool(config_prefix + ".use_environment_credentials", config.getBool("s3.use_environment_credentials", false)), context.getRemoteHostFilter(), context.getGlobalContext().getSettingsRef().s3_max_redirects); diff --git a/src/IO/S3Common.cpp b/src/IO/S3Common.cpp index 97aa95f3c0d..bc49c2641a0 100644 --- a/src/IO/S3Common.cpp +++ b/src/IO/S3Common.cpp @@ -7,6 +7,10 @@ # include # include +# include +# include +# include +# include # include # include # include @@ -85,15 +89,107 @@ private: std::unordered_map tag_loggers; }; +class S3CredentialsProviderChain : public Aws::Auth::AWSCredentialsProviderChain +{ +public: + explicit S3CredentialsProviderChain(const DB::S3::PocoHTTPClientConfiguration & configuration, const Aws::Auth::AWSCredentials & credentials, bool use_environment_credentials) + { + if (use_environment_credentials) + { + const DB::RemoteHostFilter & remote_host_filter = configuration.remote_host_filter; + const unsigned int s3_max_redirects = configuration.s3_max_redirects; + + static const char AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI[] = "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"; + static const char AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI[] = "AWS_CONTAINER_CREDENTIALS_FULL_URI"; + static const char AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN[] = "AWS_CONTAINER_AUTHORIZATION_TOKEN"; + static const char AWS_EC2_METADATA_DISABLED[] = "AWS_EC2_METADATA_DISABLED"; + + auto * logger = &Poco::Logger::get("S3CredentialsProviderChain"); + + /// The only difference from DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() + /// is that this chain uses custom ClientConfiguration. + + AddProvider(std::make_shared()); + AddProvider(std::make_shared()); + AddProvider(std::make_shared()); + + /// ECS TaskRole Credentials only available when ENVIRONMENT VARIABLE is set. + const auto relative_uri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI, + relative_uri); + + const auto absolute_uri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI, + absolute_uri); + + const auto ec2_metadata_disabled = Aws::Environment::GetEnv(AWS_EC2_METADATA_DISABLED); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_EC2_METADATA_DISABLED, + ec2_metadata_disabled); + + if (!relative_uri.empty()) + { + AddProvider(std::make_shared(relative_uri.c_str())); + LOG_INFO(logger, "Added ECS metadata service credentials provider with relative path: [{}] to the provider chain.", + relative_uri); + } + else if (!absolute_uri.empty()) + { + const auto token = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN); + AddProvider(std::make_shared(absolute_uri.c_str(), token.c_str())); + + /// DO NOT log the value of the authorization token for security purposes. + LOG_INFO(logger, "Added ECS credentials provider with URI: [{}] to the provider chain with a{} authorization token.", + absolute_uri, token.empty() ? "n empty" : " non-empty"); + } + else if (Aws::Utils::StringUtils::ToLower(ec2_metadata_disabled.c_str()) != "true") + { + Aws::Client::ClientConfiguration aws_client_configuration; + + /// See MakeDefaultHttpResourceClientConfiguration(). + /// This is part of EC2 metadata client, but unfortunately it can't be accessed from outside + /// of contrib/aws/aws-cpp-sdk-core/source/internal/AWSHttpResourceClient.cpp + aws_client_configuration.maxConnections = 2; + aws_client_configuration.scheme = Aws::Http::Scheme::HTTP; + + /// Explicitly set the proxy settings to empty/zero to avoid relying on defaults that could potentially change + /// in the future. + aws_client_configuration.proxyHost = ""; + aws_client_configuration.proxyUserName = ""; + aws_client_configuration.proxyPassword = ""; + aws_client_configuration.proxyPort = 0; + + /// EC2MetadataService throttles by delaying the response so the service client should set a large read timeout. + /// EC2MetadataService delay is in order of seconds so it only make sense to retry after a couple of seconds. + aws_client_configuration.connectTimeoutMs = 1000; + aws_client_configuration.requestTimeoutMs = 1000; + aws_client_configuration.retryStrategy = std::make_shared(1, 1000); + + DB::S3::PocoHTTPClientConfiguration client_configuration(aws_client_configuration, remote_host_filter, s3_max_redirects); + auto ec2_metadata_client = std::make_shared(client_configuration); + auto config_loader = std::make_shared(ec2_metadata_client); + + AddProvider(std::make_shared(config_loader)); + LOG_INFO(logger, "Added EC2 metadata service credentials provider to the provider chain."); + } + } + + AddProvider(std::make_shared(credentials)); + } +}; + class S3AuthSigner : public Aws::Client::AWSAuthV4Signer { public: S3AuthSigner( const Aws::Client::ClientConfiguration & client_configuration, const Aws::Auth::AWSCredentials & credentials, - const DB::HeaderCollection & headers_) + const DB::HeaderCollection & headers_, + bool use_environment_credentials) : Aws::Client::AWSAuthV4Signer( - std::make_shared(credentials), + std::make_shared( + static_cast(client_configuration), + credentials, + use_environment_credentials), "s3", client_configuration.region, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, @@ -164,6 +260,7 @@ namespace S3 bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -172,7 +269,13 @@ namespace S3 if (!endpoint.empty()) cfg.endpointOverride = endpoint; - return create(cfg, is_virtual_hosted_style, access_key_id, secret_access_key, remote_host_filter, s3_max_redirects); + return create(cfg, + is_virtual_hosted_style, + access_key_id, + secret_access_key, + use_environment_credentials, + remote_host_filter, + s3_max_redirects); } std::shared_ptr ClientFactory::create( // NOLINT @@ -180,6 +283,7 @@ namespace S3 bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -190,7 +294,10 @@ namespace S3 client_configuration.updateSchemeAndRegion(); return std::make_shared( - credentials, // Aws credentials. + std::make_shared( + client_configuration, + credentials, + use_environment_credentials), // AWS credentials provider. std::move(client_configuration), // Client configuration. Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, // Sign policy. is_virtual_hosted_style || cfg.endpointOverride.empty() // Use virtual addressing if endpoint is not specified. @@ -203,6 +310,7 @@ namespace S3 const String & access_key_id, const String & secret_access_key, HeaderCollection headers, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -214,8 +322,10 @@ namespace S3 client_configuration.updateSchemeAndRegion(); Aws::Auth::AWSCredentials credentials(access_key_id, secret_access_key); + + auto auth_signer = std::make_shared(client_configuration, std::move(credentials), std::move(headers), use_environment_credentials); return std::make_shared( - std::make_shared(client_configuration, std::move(credentials), std::move(headers)), + std::move(auth_signer), std::move(client_configuration), // Client configuration. is_virtual_hosted_style || client_configuration.endpointOverride.empty() // Use virtual addressing only if endpoint is not specified. ); diff --git a/src/IO/S3Common.h b/src/IO/S3Common.h index 39d9068a6f4..664c07d5bf4 100644 --- a/src/IO/S3Common.h +++ b/src/IO/S3Common.h @@ -36,6 +36,7 @@ public: bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); @@ -44,6 +45,7 @@ public: bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); @@ -53,6 +55,7 @@ public: const String & access_key_id, const String & secret_access_key, HeaderCollection headers, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); diff --git a/src/Storages/StorageS3.cpp b/src/Storages/StorageS3.cpp index ea9319d4693..fff44bb1f4c 100644 --- a/src/Storages/StorageS3.cpp +++ b/src/Storages/StorageS3.cpp @@ -216,8 +216,14 @@ StorageS3::StorageS3( credentials = Aws::Auth::AWSCredentials(std::move(settings.access_key_id), std::move(settings.secret_access_key)); client = S3::ClientFactory::instance().create( - uri_.endpoint, uri_.is_virtual_hosted_style, access_key_id_, secret_access_key_, std::move(settings.headers), - context_.getRemoteHostFilter(), context_.getGlobalContext().getSettingsRef().s3_max_redirects); + uri_.endpoint, + uri_.is_virtual_hosted_style, + credentials.GetAWSAccessKeyId(), + credentials.GetAWSSecretKey(), + std::move(settings.headers), + settings.use_environment_credentials.value_or(global_context.getConfigRef().getBool("s3.use_environment_credentials", false)), + context_.getRemoteHostFilter(), + context_.getGlobalContext().getSettingsRef().s3_max_redirects); } diff --git a/src/Storages/StorageS3Settings.cpp b/src/Storages/StorageS3Settings.cpp index 5b443de6b9a..9c4dc5831dc 100644 --- a/src/Storages/StorageS3Settings.cpp +++ b/src/Storages/StorageS3Settings.cpp @@ -26,6 +26,11 @@ void StorageS3Settings::loadFromConfig(const String & config_elem, const Poco::U auto endpoint = config.getString(config_elem + "." + key + ".endpoint"); auto access_key_id = config.getString(config_elem + "." + key + ".access_key_id", ""); auto secret_access_key = config.getString(config_elem + "." + key + ".secret_access_key", ""); + std::optional use_environment_credentials; + if (config.has(config_elem + "." + key + ".use_environment_credentials")) + { + use_environment_credentials = config.getBool(config_elem + "." + key + ".use_environment_credentials"); + } HeaderCollection headers; Poco::Util::AbstractConfiguration::Keys subconfig_keys; @@ -42,7 +47,7 @@ void StorageS3Settings::loadFromConfig(const String & config_elem, const Poco::U } } - settings.emplace(endpoint, S3AuthSettings{std::move(access_key_id), std::move(secret_access_key), std::move(headers)}); + settings.emplace(endpoint, S3AuthSettings{std::move(access_key_id), std::move(secret_access_key), std::move(headers), use_environment_credentials}); } } diff --git a/src/Storages/StorageS3Settings.h b/src/Storages/StorageS3Settings.h index 19b2bf48bd8..88f964774c6 100644 --- a/src/Storages/StorageS3Settings.h +++ b/src/Storages/StorageS3Settings.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -28,6 +29,8 @@ struct S3AuthSettings const String secret_access_key; const HeaderCollection headers; + + std::optional use_environment_credentials; }; /// Settings for the StorageS3. diff --git a/tests/integration/test_log_family_s3/test.py b/tests/integration/test_log_family_s3/test.py index 40e263c9c69..8b262bf6760 100644 --- a/tests/integration/test_log_family_s3/test.py +++ b/tests/integration/test_log_family_s3/test.py @@ -1,4 +1,5 @@ import logging +import sys import pytest from helpers.cluster import ClickHouseCluster @@ -34,19 +35,23 @@ def test_log_family_s3(cluster, log_engine, files_overhead, files_overhead_per_i node.query("INSERT INTO s3_test SELECT number FROM numbers(5)") assert node.query("SELECT * FROM s3_test") == "0\n1\n2\n3\n4\n" + print(list(minio.list_objects(cluster.minio_bucket, 'data/')), file=sys.stderr) assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert + files_overhead node.query("INSERT INTO s3_test SELECT number + 5 FROM numbers(3)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n" + print(list(minio.list_objects(cluster.minio_bucket, 'data/')), file=sys.stderr) assert len( list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 2 + files_overhead node.query("INSERT INTO s3_test SELECT number + 8 FROM numbers(1)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n8\n" + print(list(minio.list_objects(cluster.minio_bucket, 'data/')), file=sys.stderr) assert len( list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 3 + files_overhead node.query("TRUNCATE TABLE s3_test") + print(list(minio.list_objects(cluster.minio_bucket, 'data/')), file=sys.stderr) assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == 0 node.query("DROP TABLE s3_test") From c34c0bf50764aadaae0f906091e281c6af2d4799 Mon Sep 17 00:00:00 2001 From: robot-clickhouse Date: Thu, 10 Dec 2020 12:45:44 +0300 Subject: [PATCH 147/174] Update version_date.tsv after release 20.8.9.6 --- utils/list-versions/version_date.tsv | 1 + 1 file changed, 1 insertion(+) diff --git a/utils/list-versions/version_date.tsv b/utils/list-versions/version_date.tsv index 8df84943084..cec9dbf1b08 100644 --- a/utils/list-versions/version_date.tsv +++ b/utils/list-versions/version_date.tsv @@ -15,6 +15,7 @@ v20.9.5.5-stable 2020-11-13 v20.9.4.76-stable 2020-10-29 v20.9.3.45-stable 2020-10-09 v20.9.2.20-stable 2020-09-22 +v20.8.9.6-lts 2020-12-10 v20.8.8.2-lts 2020-12-07 v20.8.7.15-lts 2020-11-20 v20.8.6.6-lts 2020-11-13 From c63d3593b09324e7359707f2d2a6e9c50b107576 Mon Sep 17 00:00:00 2001 From: Evgeniia Sudarikova Date: Thu, 10 Dec 2020 13:01:27 +0300 Subject: [PATCH 148/174] edited example --- docs/en/operations/settings/settings.md | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 074ac07907c..95332d1d841 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2369,37 +2369,22 @@ Default value: `\N`. Query ```sql -SELECT * -FROM tsv_custom_null -FORMAT TSV +SELECT * FROM tsv_custom_null FORMAT TSV; ``` Result ```text -\N 788 \N +\N ``` Query ```sql -SET output_format_tsv_null_representation = 'My NULL' -``` - -Result - -```text -Ok. -``` - -Query - -```sql -SELECT * -FROM tsv_custom_null -FORMAT TSV +SET output_format_tsv_null_representation = 'My NULL'; +SELECT * FROM tsv_custom_null FORMAT TSV; ``` Result From 955033d6a1874f35117b0c5e998d3790b299c3de Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Thu, 10 Dec 2020 16:18:37 +0300 Subject: [PATCH 149/174] done --- src/IO/WriteHelpers.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/src/IO/WriteHelpers.h b/src/IO/WriteHelpers.h index 1997aede564..772ab5fad9b 100644 --- a/src/IO/WriteHelpers.h +++ b/src/IO/WriteHelpers.h @@ -29,7 +29,12 @@ #include #include -#include +/// There is no dragonbox in Arcadia +#if !defined(ARCADIA_BUILD) +# include +#else +# include +#endif #include @@ -228,14 +233,22 @@ inline size_t writeFloatTextFastPath(T x, char * buffer) if (DecomposedFloat64(x).is_inside_int64()) result = itoa(Int64(x), buffer) - buffer; else +#if !defined(ARCADIA_BUILD) result = jkj::dragonbox::to_chars_n(x, buffer) - buffer; +#else + result = d2s_buffered_n(x, buffer); +#endif } else { if (DecomposedFloat32(x).is_inside_int32()) result = itoa(Int32(x), buffer) - buffer; else +#if !defined(ARCADIA_BUILD) result = jkj::dragonbox::to_chars_n(x, buffer) - buffer; +#else + result = f2s_buffered_n(x, buffer); +#endif } if (result <= 0) From 6f2c559398ad3073123f270eb2504dabcf711554 Mon Sep 17 00:00:00 2001 From: Azat Khuzhin Date: Thu, 10 Dec 2020 21:01:17 +0300 Subject: [PATCH 150/174] Convert top_level_domains_lists.xml into regular file docker/test/performance-comparison/config/config.d is not allowed to has symlinks, since it is copied with the COPY docker command. --- .../config/config.d/top_level_domains_lists.xml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) mode change 120000 => 100644 docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml diff --git a/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml b/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml deleted file mode 120000 index c1fc2c14dfc..00000000000 --- a/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml +++ /dev/null @@ -1 +0,0 @@ -../../../../../tests/config/config.d/top_level_domains_lists.xml \ No newline at end of file diff --git a/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml b/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml new file mode 100644 index 00000000000..7b5e6a5638a --- /dev/null +++ b/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml @@ -0,0 +1,5 @@ + + + public_suffix_list.dat + + From 17310c6c50ad125f86617425959c1eeb07bc734f Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Thu, 10 Dec 2020 21:08:16 +0300 Subject: [PATCH 151/174] fix-arcadia --- src/IO/WriteHelpers.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/IO/WriteHelpers.h b/src/IO/WriteHelpers.h index 772ab5fad9b..54e9ac9e09e 100644 --- a/src/IO/WriteHelpers.h +++ b/src/IO/WriteHelpers.h @@ -32,9 +32,9 @@ /// There is no dragonbox in Arcadia #if !defined(ARCADIA_BUILD) # include -#else +#else # include -#endif +#endif #include From f1ee1ef16ad769793565bbefa21b17637c2ec1b5 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Thu, 10 Dec 2020 22:50:54 +0300 Subject: [PATCH 152/174] Update MergeTreeSettings.h --- src/Storages/MergeTree/MergeTreeSettings.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Storages/MergeTree/MergeTreeSettings.h b/src/Storages/MergeTree/MergeTreeSettings.h index 5941e18506b..2f3931786a6 100644 --- a/src/Storages/MergeTree/MergeTreeSettings.h +++ b/src/Storages/MergeTree/MergeTreeSettings.h @@ -17,8 +17,8 @@ struct Settings; #define LIST_OF_MERGE_TREE_SETTINGS(M) \ - M(UInt64, min_compress_block_size, 0, "The actual size of the block to compress of each table, if this setting doesn't set, using corresponding global setting.", 0) \ - M(UInt64, max_compress_block_size, 0, "The maximum size of blocks of uncompressed data of each table before compressing for writing to a table, if this setting doesn't set, using correspondin global setting.", 0) \ + M(UInt64, min_compress_block_size, 0, "When granule is written, compress the data in buffer if the size of pending uncompressed data is larger or equal than the specified threshold. If this setting is not set, the corresponding global setting is used.", 0) \ + M(UInt64, max_compress_block_size, 0, "Compress the pending uncompressed data in buffer if its size is larger or equal than the specified threshold. Block of data will be compressed even if the current granule is not finished. If this setting is not set, the corresponding global setting is used.", 0) \ M(UInt64, index_granularity, 8192, "How many rows correspond to one primary key value.", 0) \ \ /** Data storing format settings. */ \ From 3123089ba841c615ad2e24b201c954bf1a322748 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Thu, 10 Dec 2020 23:05:51 +0300 Subject: [PATCH 153/174] Update the-clickhouse-community.md --- website/blog/en/2020/the-clickhouse-community.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 993f30cdf59..2ab5e07e5d5 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -1,6 +1,6 @@ --- title: 'The ClickHouse Community' -image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png +image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png' date: '2020-12-10' author: '[Robert Hodges](https://github.com/hodgesrm)' tags: ['community', 'open source', 'telegram', 'meetup'] From c0624e497201868bcc0d8379c53b1b48193524ca Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:33:35 +0300 Subject: [PATCH 154/174] Corrections --- .../blog/en/2020/the-clickhouse-community.md | 218 +++++------------- 1 file changed, 61 insertions(+), 157 deletions(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 2ab5e07e5d5..5c667046a5f 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -1,120 +1,58 @@ ---- +—- title: 'The ClickHouse Community' image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png' date: '2020-12-10' author: '[Robert Hodges](https://github.com/hodgesrm)' tags: ['community', 'open source', 'telegram', 'meetup'] ---- +—- -One of the great “features” of ClickHouse is a friendly and welcoming -community. In this article we would like to outline how the ClickHouse -community arose, what it is today, and how you can get involved. There -is a role for everyone, from end users to contributors to corporate -friends. Our goal is to make the community welcoming to every person -who wants to join. +One of the great “features” of ClickHouse is a friendly and welcoming community. In this article we would like to outline how the ClickHouse +community arose, what it is today, and how you can get involved. There is a role for everyone, from end users to contributors to corporate +friends. Our goal is to make the community welcoming to every person who wants to join. -But first, let’s review a bit of history, starting with how ClickHouse -first developed at [Yandex](https://yandex.com/company/). +But first, let’s review a bit of history, starting with how ClickHouse first developed at [Yandex](https://yandex.com/company/). ## Origins at Yandex -ClickHouse began as a solution for web analytics in [Yandex -Metrica](https://metrica.yandex.com/about?). Metrica is a popular service for analyzing website traffic that -is now #2 in the market behind Google Analytics. In 2008 [Alexey Milovidov](https://github.com/alexey-milovidov), -an engineer on the Metrica team, was looking for a database that could -create reports on metrics like number of page views per day, unique -visitors, and bounce rate, without aggregating the data in advance. The -idea was to provide a wide range of metric data and let users ask any -question about them. +ClickHouse began as a solution for web analytics in [Yandex Metrica](https://metrica.yandex.com/about?). Metrica is a popular service for analyzing website traffic that is now #2 in the market behind Google Analytics. In 2008 [Alexey Milovidov](https://github.com/alexey-milovidov), an engineer on the Metrica team, was looking for a database that could create reports on metrics like number of page views per day, unique visitors, and bounce rate, without aggregating the data in advance. The idea was to provide a wide range of metric data and let users ask any question about them. -This is a classic problem for data warehouses. However, Alexey could -not find one that met Yandex requirements, specifically large datasets, -linear scaling, high efficiency, and compatibility with SQL tools. In -a nutshell: like MySQL but for analytic applications. So Alexey wrote -one. It started as a prototype to do GROUP BY operations. +This is a classic problem for data warehouses. However, Alexey could not find one that met Yandex requirements, specifically large datasets, linear scaling, high efficiency, and compatibility with SQL tools. In a nutshell: like MySQL but for analytic applications. So Alexey wrote one. It started as a prototype to do GROUP BY operations. -The prototype evolved into a full solution with a name, ClickHouse, -short for “Clickstream Data Warehouse”. Alexey added additional -features including SQL support and the MergeTree engine. The SQL dialect -was superficially similar to MySQL, [which was also used in Metrica](https://clickhouse.tech/blog/en/2016/evolution-of-data-structures-in-yandex-metrica/) but -could not handle query workloads without complex pre-aggregation. By -2011 ClickHouse was in production for Metrica. +The prototype evolved into a full solution with a name, ClickHouse, short for “Clickstream Data Warehouse”. Alexey added additional features including SQL support and the MergeTree engine. The SQL dialect was superficially similar to MySQL, [which was also used in Metrica](https://clickhouse.tech/blog/en/2016/evolution-of-data-structures-in-yandex-metrica/) but could not handle query workloads without complex pre-aggregation. By 2011 ClickHouse was in production for Metrica. -Over the next 5 years Alexey and a growing team of developers extended -ClickHouse to cover new use cases. By 2016 ClickHouse was a core Metrica -backend service. It was also becoming entrenched as a data warehouse -within Yandex, extending to use cases like service monitoring, network -flow logs, and event management. ClickHouse had evolved from the original -one-person project to business critical software with a full team of a -dozen engineers led by Alexey. +Over the next 5 years Alexey and a growing team of developers extended ClickHouse to cover new use cases. By 2016 ClickHouse was a core Metrica backend service. It was also becoming entrenched as a data warehouse within Yandex, extending to use cases like service monitoring, network flow logs, and event management. ClickHouse had evolved from the original one-person project to business critical software with a full team of a dozen engineers led by Alexey. -By 2016, ClickHouse had an 8 year history and was ready to become a major -open source project. Here’s a timeline that tracks major developments -as a time series. +By 2016, ClickHouse had an 8 year history and was ready to become a major open source project. Here’s a timeline that tracks major developments as a time series. ## ClickHouse goes open source -Yandex open sourced ClickHouse under an Apache 2.0 license in 2016. There -were numerous reasons for this step. +Yandex open sourced ClickHouse under an Apache 2.0 license in 2016. There were numerous reasons for this step. -* Promote adoption within Yandex by making it easier for internal departments to get builds. +* Promote adoption within Yandex by making it easier for internal departments to get builds. * Ensure that ClickHouse would continue to evolve by creating a community to nurture it. -* Motivate developers to contribute to and use ClickHouse due to the open source “cool” factor. +* Motivate developers to contribute to and use ClickHouse due to the open source “cool” factor. * Improve ClickHouse quality by making the code public. Nobody wants their name visible on bad code. ;-) * Showcase Yandex innovation to a worldwide audience -Alexey and the development team moved ClickHouse code to a Github repo -under the Yandex organization and began issuing community builds as well -as accepting external contributions. They simultaneously began regular -meetups to popularize ClickHouse and build a community around it. The -result was a burst of adoption across multiple regions of the globe. +Alexey and the development team moved ClickHouse code to a Github repo under the Yandex organization and began issuing community builds as well as accepting external contributions. They simultaneously began regular meetups to popularize ClickHouse and build a community around it. The result was a burst of adoption across multiple regions of the globe. -ClickHouse quickly picked up steam in Eastern Europe. The first ClickHouse -meetups started in 2016 and have grown to include 200 participants for -in-person meetings and up to 400 for online meetings. ClickHouse is -now widely used in start-ups in Russia as well as other Eastern European -countries. Developers located in Eastern Europe continue to supply more -contributions to ClickHouse than any other region. +ClickHouse quickly picked up steam in Eastern Europe. The first ClickHouse meetups started in 2016 and have grown to include 200 participants for in-person meetings and up to 400 for online meetings. ClickHouse is now widely used in start-ups in Russia as well as other Eastern European countries. Developers located in Eastern Europe continue to supply more contributions to ClickHouse than any other region. -ClickHouse also started to gain recognition in the US and Western Europe. -[CloudFlare](https://www.cloudflare.com/) published a widely read blog -article about [their success using ClickHouse for DNS analytics](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/). Alexander -Zaitsev successfully migrated an ad tech analytics system from a -commercial DBMS to a ClickHouse cluster. This success prompted him to -found [Altinity](https://altinity.com) in 2017 with help from friends at -[Percona](https://www.percona.com). US meetups -started in the same year. With support from Altinity these have grown -to over 100 attendees for online meetings. +ClickHouse also started to gain recognition in the US and Western Europe. [CloudFlare](https://www.cloudflare.com/) published a widely read blog article about [their success using ClickHouse for DNS analytics](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/). Alexander Zaitsev successfully migrated an ad tech analytics system from a commercial DBMS to a ClickHouse cluster. This success prompted him to found [Altinity](https://altinity.com) in 2017 with help from friends at [Percona](https://www.percona.com). US meetups started in the same year. With support from Altinity these have grown to over 100 attendees for online meetings. -ClickHouse also took off in China. The first meetup in China took place -in 2018 and attracted enormous interest. In-person meetups included over -400 participants. Online meetings have reached up to 1000 online viewers. +ClickHouse also took off in China. The first meetup in China took place in 2018 and attracted enormous interest. In-person meetups included over 400 participants. Online meetings have reached up to 1000 online viewers. -In 2019 a further step occurred as ClickHouse moved out from under the -Yandex Github organization into a separate [ClickHouse organization](https://github.com/ClickHouse). The -new organization includes ClickHouse server code plus core ecosystem -projects like the cpp and ODBC drivers. +In 2019 a further step occurred as ClickHouse moved out from under the Yandex Github organization into a separate [ClickHouse organization](https://github.com/ClickHouse). The new organization includes ClickHouse server code plus core ecosystem projects like the cpp and ODBC drivers. -ClickHouse community events shifted online following world-wide -disruptions due to COVID-19, but growth in usage continued. One -interesting development has been the increasing number of startups using -ClickHouse as a backend. Many of these are listed on the -[ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) -page. Also, additional prominent companies like eBay, Uber, and -Flipcart went public in 2020 with stories of successful ClickHouse usage. +ClickHouse community events shifted online following world-wide disruptions due to COVID-19, but growth in usage continued. One interesting development has been the increasing number of startups using ClickHouse as a backend. Many of these are listed on the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page. Also, additional prominent companies like eBay, Uber, and Flipcart went public in 2020 with stories of successful ClickHouse usage. ## The ClickHouse community today -As of 2020 the ClickHouse community includes developers and users from -virtually every region of the globe. Yandex engineers continue to supply -a majority of pull requests to ClickHouse itself. Altinity follows -in second place with contributions to ClickHouse core and ecosystem -projects. There is also substantial in-house development on ClickHouse -(e.g. on private forks) within Chinese internet providers. +As of 2020 the ClickHouse community includes developers and users from virtually every region of the globe. Yandex engineers continue to supply a majority of pull requests to ClickHouse itself. Altinity follows in second place with contributions to ClickHouse core and ecosystem projects. There is also substantial in-house development on ClickHouse (e.g. on private forks) within Chinese internet providers. -The real success, however, has been the huge number of commits to ClickHouse core from people in outside organizations. The following list shows the main outside contributors: +The real success, however, has been the huge number of commits to ClickHouse core from people in outside organizations. The following list shows the main outside contributors: * Azat Khuzhin * Amos Bird @@ -138,108 +76,74 @@ The real success, however, has been the huge number of commits to ClickHouse cor * Flynn ucasFL * [And over 550 more](https://github.com/ClickHouse/ClickHouse/graphs/contributors)... -ClickHouse ecosystem projects are also growing rapidly. Here is a selected list of active Github projects that help enable ClickHouse applications, sorted by number of stars. +ClickHouse ecosystem projects are also growing rapidly. Here is a selected list of active Github projects that help enable ClickHouse applications, sorted by number of stars. -* [sqlpad/sqlpad](https://github.com/sqlpad/sqlpad) -- Web-based SQL editor that supports ClickHouse -* [mindsdb/mindsdb](https://github.com/mindsdb/mindsdb) -- Predictive AI layer for databases with ClickHouse support -* [x-ream/sqli](https://github.com/x-ream/sqli) -- ORM SQL interface -* [tricksterproxy/trickster](https://github.com/tricksterproxy/trickster) -- HTTP reverse proxy cache and time series dashboard accelerator -* [ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go) -- Golang driver for ClickHouse -* [gohouse/gorose](https://github.com/gohouse/gorose) -- A mini database ORM for Golang -* [ClickHouse/clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) -- JDBC driver for ClickHouse -* [brockercap/Bifrost](https://github.com/brokercap/Bifrost) -- Middleware to sync MySQL binlog to ClickHouse -* [mymarilyn/clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) -- ClickHouse Python driver with native interface support -* [Vertamedia/clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) -- Grafana datasource for ClickHouse -* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) -- PHP Clickhouse client -* [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) -- Kubernetes operator for ClickHouse -* [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) -- ClickHouse backup and restore using cloud storage +* [sqlpad/sqlpad](https://github.com/sqlpad/sqlpad) — Web-based SQL editor that supports ClickHouse +* [mindsdb/mindsdb](https://github.com/mindsdb/mindsdb) — Predictive AI layer for databases with ClickHouse support +* [x-ream/sqli](https://github.com/x-ream/sqli) — ORM SQL interface +* [tricksterproxy/trickster](https://github.com/tricksterproxy/trickster) — HTTP reverse proxy cache and time series dashboard accelerator +* [ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go) — Golang driver for ClickHouse +* [gohouse/gorose](https://github.com/gohouse/gorose) — A mini database ORM for Golang +* [ClickHouse/clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) — JDBC driver for ClickHouse +* [brockercap/Bifrost](https://github.com/brokercap/Bifrost) — Middleware to sync MySQL binlog to ClickHouse +* [mymarilyn/clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) — ClickHouse Python driver with native interface support +* [Vertamedia/clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) — Grafana datasource for ClickHouse +* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) — PHP Clickhouse client +* [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) — Kubernetes operator for ClickHouse +* [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) — ClickHouse backup and restore using cloud storage * [And almost 1200 more](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories)... -## Resources +## Resources -With the community growth numerous resources are available to users. At -the center is the [ClickHouse org on Github](https://github.com/ClickHouse), which hosts [ClickHouse -server code](https://github.com/ClickHouse/ClickHouse). ClickHouse server documentation is available at the -[clickhouse.tech](https://clickhouse.tech/) website. It has [installation instructions](https://clickhouse.tech/docs/en/getting-started/install/) and links to -ClickHouse community builds for major Linux distributions as well as Mac, -FreeBSD, and Docker. +With the community growth numerous resources are available to users. At the center is the [ClickHouse org on Github](https://github.com/ClickHouse), which hosts [ClickHouse server code](https://github.com/ClickHouse/ClickHouse). ClickHouse server documentation is available at the [clickhouse.tech](https://clickhouse.tech/) website. It has [installation instructions](https://clickhouse.tech/docs/en/getting-started/install/) and links to ClickHouse community builds for major Linux distributions as well as Mac, FreeBSD, and Docker. -In addition, ClickHouse users have a wide range of ways to engage with -the community and get help on applications. These include both chat -applications as well as meetups. Here are some links to get started. +In addition, ClickHouse users have a wide range of ways to engage with the community and get help on applications. These include both chat applications as well as meetups. Here are some links to get started. -* Yandex Meetups -- Yandex has regular in-person and online Russian-language meetups. Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). -* [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) -- The largest US ClickHouse meetup, with meetings approximately every 2 months. -* Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. -* Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. - * [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“Clickhouse does not slow down”) - Russian language Telegram group with 4,629 members currently. - * [ClickHouse](https://t.me/clickhouse_en) -- English language group with 1,286 members. -* [ClickHouse Community Slack Channel](http://clickhousedb.slack.com) -- Public channel for Slack users. It currently has 551 members. -* [Clickhouse.com.cn](http://clickhouse.com.cn/) -- Chinese language site for ClickHouse-related announcements and questions. -* [Conference Presentations](https://github.com/ClickHouse/clickhouse-presentations) -- ClickHouse developers like to talk and do so whenever they can. Many recent presentations are stored in Github. Also, look for ClickHouse presentations at Linux Foundation conferences, Data Con LA, Percona Live, and many other venues where there are presentations about data. -* Technical webinars -- Altinity has a large library of technical presentations on ClickHouse and related applications on the [Altinity Youtube channel](https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA/featured). +* Yandex Meetups — Yandex has regular in-person and online Russian-language meetups. Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). +* [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) — The largest US ClickHouse meetup, with meetings approximately every 2 months. +* Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. +* Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. +* [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“Clickhouse does not slow down”) - Russian language Telegram group with 4,629 members currently. +* [ClickHouse](https://t.me/clickhouse_en) — English language group with 1,286 members. +* [ClickHouse Community Slack Channel](http://clickhousedb.slack.com) — Public channel for Slack users. It currently has 551 members. +* [Clickhouse.com.cn](http://clickhouse.com.cn/) — Chinese language site for ClickHouse-related announcements and questions. +* [Conference Presentations](https://github.com/ClickHouse/clickhouse-presentations) — ClickHouse developers like to talk and do so whenever they can. Many recent presentations are stored in Github. Also, look for ClickHouse presentations at Linux Foundation conferences, Data Con LA, Percona Live, and many other venues where there are presentations about data. +* Technical webinars — Altinity has a large library of technical presentations on ClickHouse and related applications on the [Altinity Youtube channel](https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA/featured). -If you know of additional resources please bring them to our attention. +If you know of additional resources please bring them to our attention. ## How you can contribute to ClickHouse -We welcome users to join the ClickHouse community in every capacity. There are four main ways to participate. +We welcome users to join the ClickHouse community in every capacity. There are four main ways to participate. ### Use ClickHouse and share your experiences -Start with the documentation. Download ClickHouse and try -it out. Join the chat channels. If you encounter bugs, [log -issues](https://github.com/ClickHouse/ClickHouse/issues) so we can get -them fixed. Also, it’s easy to make contributions to the documentation -if you have basic Github and markdown skills. Press the pencil icon on -any page of the clickhouse.tech website to edit pages and automatically -generate pull requests to merge your changes. +Start with the documentation. Download ClickHouse and try it out. Join the chat channels. If you encounter bugs, [log issues](https://github.com/ClickHouse/ClickHouse/issues) so we can get them fixed. Also, it’s easy to make contributions to the documentation if you have basic Github and markdown skills. Press the pencil icon on any page of the clickhouse.tech website to edit pages and automatically generate pull requests to merge your changes. -If your company has deployed ClickHouse and is comfortable talking about -it, please don't be shy. Add them to the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page so -that others can learn from your experience. +If your company has deployed ClickHouse and is comfortable talking about it, please don't be shy. Add them to the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page so that others can learn from your experience. ### Become a ClickHouse developer -Write code to make ClickHouse better. Here are your choices. +Write code to make ClickHouse better. Here are your choices. -* ClickHouse server -- Start with the [“For Beginners” documentation](https://clickhouse.tech/docs/en/development/developer-instruction/) to learn how to build ClickHouse and submit PRs. Check out the current ClickHouse issues if you are looking for work. PRs that follow the development standards will be merged faster. +* ClickHouse server — Start with the [“For Beginners” documentation](https://clickhouse.tech/docs/en/development/developer-instruction/) to learn how to build ClickHouse and submit PRs. Check out the current ClickHouse issues if you are looking for work. PRs that follow the development standards will be merged faster. -* Ecosystem projects -- Most projects in the ClickHouse ecosystem accept PRs. Check with each project for specific practices. +* Ecosystem projects — Most projects in the ClickHouse ecosystem accept PRs. Check with each project for specific practices. -ClickHouse is also a great target for research problems. Overall the -years many dozens of university CS students have worked on ClickHouse -features. Alexey Milovidov maintains an especially rich set of [project -suggestions for students](https://github.com/ClickHouse/ClickHouse/issues/15065). Join Telegram and ask for help if you are -interested. Both Yandex and Altinity also offer internships. +ClickHouse is also a great target for research problems. Overall the years many dozens of university CS students have worked on ClickHouse features. Alexey Milovidov maintains an especially rich set of [project suggestions for students](https://github.com/ClickHouse/ClickHouse/issues/15065). Join Telegram and ask for help if you are interested. Both Yandex and Altinity also offer internships. ### Write ClickHouse applications -ClickHouse enables a host of new applications that depend on low -latency access to large datasets. If you write something interesting, -blog about it and present at local meetups. Altinity has a program to -highlight startups who are developing ClickHouse applications and help -with marketing as well as resources for development. Send email to -[info@altinity.com](mailto:info@altinity.com) for more information. +ClickHouse enables a host of new applications that depend on low latency access to large datasets. If you write something interesting, blog about it and present at local meetups. Altinity has a program to highlight startups who are developing ClickHouse applications and help with marketing as well as resources for development. Send email to [info@altinity.com](mailto:info@altinity.com) for more information. ### Become a corporate sponsor -The ClickHouse community has been assisted by many corporate -users who have helped organize meetups, funded development, and -guided growth of ClickHouse. Contact community members directly at -[clickhouse-feedback@yandex-team.ru](mailto:clickhouse-feedback@yandex-team.ru), [info@altinity.com](mailto:info@altinity.com), or via Telegram -to find out more about how to chip in as a corporate sponsor. +The ClickHouse community has been assisted by many corporate users who have helped organize meetups, funded development, and guided growth of ClickHouse. Contact community members directly at [clickhouse-feedback@yandex-team.ru](mailto:clickhouse-feedback@yandex-team.ru), [info@altinity.com](mailto:info@altinity.com), or via Telegram to find out more about how to chip in as a corporate sponsor. ## Where we go from here -ClickHouse has grown enormously from its origins as a basic prototype in -2008 to the popular SQL data warehouse users see today. Our community is -the rock that will enable ClickHouse to become the default data warehouse -worldwide. We are working together to create an inclusive environment -where everyone feels welcome and has an opportunity to contribute. We -welcome you to join! +ClickHouse has grown enormously from its origins as a basic prototype in 2008 to the popular SQL data warehouse users see today. Our community is the rock that will enable ClickHouse to become the default data warehouse worldwide. We are working together to create an inclusive environment where everyone feels welcome and has an opportunity to contribute. We welcome you to join! -This article was written with kind assistance from Alexey Milovidov, -Ivan Blinkov, and Alexander Zaitsev +This article was written with kind assistance from Alexey Milovidov, Ivan Blinkov, and Alexander Zaitsev. _2020-12-11 [Robert Hodges](https://github.com/hodgesrm)_ From 24f53727438bb711f4244c3762cfd1fe84ba59d7 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:36:11 +0300 Subject: [PATCH 155/174] Corrections --- website/blog/en/2020/the-clickhouse-community.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 5c667046a5f..4b9eb252b48 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -1,10 +1,10 @@ -—- +--- title: 'The ClickHouse Community' image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png' date: '2020-12-10' author: '[Robert Hodges](https://github.com/hodgesrm)' tags: ['community', 'open source', 'telegram', 'meetup'] -—- +--- One of the great “features” of ClickHouse is a friendly and welcoming community. In this article we would like to outline how the ClickHouse community arose, what it is today, and how you can get involved. There is a role for everyone, from end users to contributors to corporate From ca0e53bb502d87430aaad0d14e3b15d00a401283 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:38:22 +0300 Subject: [PATCH 156/174] Corrections --- website/blog/en/2020/the-clickhouse-community.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 4b9eb252b48..be409dfe568 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -6,9 +6,7 @@ author: '[Robert Hodges](https://github.com/hodgesrm)' tags: ['community', 'open source', 'telegram', 'meetup'] --- -One of the great “features” of ClickHouse is a friendly and welcoming community. In this article we would like to outline how the ClickHouse -community arose, what it is today, and how you can get involved. There is a role for everyone, from end users to contributors to corporate -friends. Our goal is to make the community welcoming to every person who wants to join. +One of the great “features” of ClickHouse is a friendly and welcoming community. In this article we would like to outline how the ClickHouse community arose, what it is today, and how you can get involved. There is a role for everyone, from end users to contributors to corporate friends. Our goal is to make the community welcoming to every person who wants to join. But first, let’s review a bit of history, starting with how ClickHouse first developed at [Yandex](https://yandex.com/company/). From 67b228e6c1bb2f5456f23da527edfb927b4df2b9 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:43:31 +0300 Subject: [PATCH 157/174] Corrections --- website/blog/en/2020/the-clickhouse-community.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index be409dfe568..89c65b70a83 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -32,7 +32,7 @@ Yandex open sourced ClickHouse under an Apache 2.0 license in 2016. There were n * Ensure that ClickHouse would continue to evolve by creating a community to nurture it. * Motivate developers to contribute to and use ClickHouse due to the open source “cool” factor. * Improve ClickHouse quality by making the code public. Nobody wants their name visible on bad code. ;-) -* Showcase Yandex innovation to a worldwide audience +* Showcase Yandex innovation to a worldwide audience. Alexey and the development team moved ClickHouse code to a Github repo under the Yandex organization and began issuing community builds as well as accepting external contributions. They simultaneously began regular meetups to popularize ClickHouse and build a community around it. The result was a burst of adoption across multiple regions of the globe. From 2a7391dea5b517e302a2159ca4c3d566c24eb210 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:45:16 +0300 Subject: [PATCH 158/174] Corrections --- website/blog/en/2020/the-clickhouse-community.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 89c65b70a83..689fbf44b48 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -86,7 +86,7 @@ ClickHouse ecosystem projects are also growing rapidly. Here is a selected list * [brockercap/Bifrost](https://github.com/brokercap/Bifrost) — Middleware to sync MySQL binlog to ClickHouse * [mymarilyn/clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) — ClickHouse Python driver with native interface support * [Vertamedia/clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) — Grafana datasource for ClickHouse -* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) — PHP Clickhouse client +* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) — PHP ClickHouse client * [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) — Kubernetes operator for ClickHouse * [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) — ClickHouse backup and restore using cloud storage * [And almost 1200 more](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories)... @@ -101,10 +101,10 @@ In addition, ClickHouse users have a wide range of ways to engage with the commu * [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) — The largest US ClickHouse meetup, with meetings approximately every 2 months. * Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. * Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. -* [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“Clickhouse does not slow down”) - Russian language Telegram group with 4,629 members currently. +* [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“ClickHouse does not slow down”) - Russian language Telegram group with 4,629 members currently. * [ClickHouse](https://t.me/clickhouse_en) — English language group with 1,286 members. * [ClickHouse Community Slack Channel](http://clickhousedb.slack.com) — Public channel for Slack users. It currently has 551 members. -* [Clickhouse.com.cn](http://clickhouse.com.cn/) — Chinese language site for ClickHouse-related announcements and questions. +* [ClickHouse.com.cn](http://clickhouse.com.cn/) — Chinese language site for ClickHouse-related announcements and questions. * [Conference Presentations](https://github.com/ClickHouse/clickhouse-presentations) — ClickHouse developers like to talk and do so whenever they can. Many recent presentations are stored in Github. Also, look for ClickHouse presentations at Linux Foundation conferences, Data Con LA, Percona Live, and many other venues where there are presentations about data. * Technical webinars — Altinity has a large library of technical presentations on ClickHouse and related applications on the [Altinity Youtube channel](https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA/featured). From c8b8d22e0e18f135f3c20e6dd0f282f290c61780 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:49:40 +0300 Subject: [PATCH 159/174] Add YouTube --- website/blog/en/2020/the-clickhouse-community.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index 689fbf44b48..ec90e3e19b7 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -97,7 +97,7 @@ With the community growth numerous resources are available to users. At the cent In addition, ClickHouse users have a wide range of ways to engage with the community and get help on applications. These include both chat applications as well as meetups. Here are some links to get started. -* Yandex Meetups — Yandex has regular in-person and online Russian-language meetups. Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). +* Yandex Meetups — Yandex has regular in-person and online international and Russian-language meetups. Video recordings and online translations are available at the official [YouTube channel](https://www.youtube.com/c/ClickHouseDB/videos). Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). * [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) — The largest US ClickHouse meetup, with meetings approximately every 2 months. * Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. * Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. From a618bbce4ce0045d8a650c6eca93c93b558d280e Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 10 Dec 2020 23:52:50 +0300 Subject: [PATCH 160/174] Slightly better --- website/blog/en/2020/the-clickhouse-community.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md index ec90e3e19b7..7080fed6479 100644 --- a/website/blog/en/2020/the-clickhouse-community.md +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -72,7 +72,7 @@ The real success, however, has been the huge number of commits to ClickHouse cor * Simon Podlipsky * Silviu Caragea * Flynn ucasFL -* [And over 550 more](https://github.com/ClickHouse/ClickHouse/graphs/contributors)... +* [And over 550 more...](https://github.com/ClickHouse/ClickHouse/graphs/contributors) ClickHouse ecosystem projects are also growing rapidly. Here is a selected list of active Github projects that help enable ClickHouse applications, sorted by number of stars. @@ -89,7 +89,7 @@ ClickHouse ecosystem projects are also growing rapidly. Here is a selected list * [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) — PHP ClickHouse client * [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) — Kubernetes operator for ClickHouse * [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) — ClickHouse backup and restore using cloud storage -* [And almost 1200 more](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories)... +* [And almost 1200 more...](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories) ## Resources From 6e417088dc00ec93973f8d77768aef9503b57a40 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 00:34:10 +0300 Subject: [PATCH 161/174] Update distribution_queue.md --- docs/en/operations/system-tables/distribution_queue.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/operations/system-tables/distribution_queue.md b/docs/en/operations/system-tables/distribution_queue.md index cc3822bad1a..da4effd0ff5 100644 --- a/docs/en/operations/system-tables/distribution_queue.md +++ b/docs/en/operations/system-tables/distribution_queue.md @@ -1,6 +1,6 @@ # system.distribution_queue {#system_tables-distribution_queue} -Contains information about local files that are in the queue to be sent to the shards. This local files contain new parts that are created by inserting new data into the table. +Contains information about local files that are in the queue to be sent to the shards. This local files contain new parts that are created by inserting new data into the Distributed table in asynchronous mode. Columns: @@ -18,7 +18,7 @@ Columns: - `data_compressed_bytes` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Size of compressed data in local files, in bytes. -- `last_exception` ([String](../../sql-reference/data-types/string.md)) — Text message about the last error that occurred. +- `last_exception` ([String](../../sql-reference/data-types/string.md)) — Text message about the last error that occurred (if any). **Example** From 2fb7ce8340496626a0b8cb8ec4348a2afbc81aa0 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:09:01 +0300 Subject: [PATCH 162/174] Revert "Date vs DateTime64 comparison" --- src/Functions/FunctionsComparison.h | 5 +-- ...eTime_and_DateTime64_comparision.reference | 3 -- ...60_DateTime_and_DateTime64_comparision.sql | 42 ------------------ ..._Date_and_DateTime64_comparision.reference | 3 -- .../01561_Date_and_DateTime64_comparision.sql | 43 ------------------- 5 files changed, 1 insertion(+), 95 deletions(-) delete mode 100644 tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference delete mode 100644 tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql delete mode 100644 tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference delete mode 100644 tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql diff --git a/src/Functions/FunctionsComparison.h b/src/Functions/FunctionsComparison.h index 957c7e0ab3e..e674f8690ff 100644 --- a/src/Functions/FunctionsComparison.h +++ b/src/Functions/FunctionsComparison.h @@ -1216,10 +1216,7 @@ public: { return res; } - else if ((isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type)) - // Comparing Date and DateTime64 requires implicit conversion, - // otherwise Date is treated as number. - && !(date_and_datetime && (isDate(left_type) || isDate(right_type)))) + else if (isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type)) { // compare if (!allowDecimalComparison(left_type, right_type) && !date_and_datetime) diff --git a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference deleted file mode 100644 index 088030bbc28..00000000000 --- a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.reference +++ /dev/null @@ -1,3 +0,0 @@ --1 DateTime64(1, \'UTC\') < 1 1 1 <= 1 1 1 = 0 0 0 >= 0 0 0 > 0 0 0 != 1 1 1 -0 DateTime64(1, \'UTC\') < 0 0 0 <= 1 1 1 = 1 1 1 >= 1 1 1 > 0 0 0 != 0 0 0 -1 DateTime64(1, \'UTC\') < 0 0 0 <= 0 0 0 = 0 0 0 >= 1 1 1 > 1 1 1 != 1 1 1 diff --git a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql b/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql deleted file mode 100644 index afee0ebadaa..00000000000 --- a/tests/queries/0_stateless/01560_DateTime_and_DateTime64_comparision.sql +++ /dev/null @@ -1,42 +0,0 @@ -SELECT - n, - toTypeName(dt64) AS dt64_typename, - - '<', - dt64 < dt, - toDateTime(dt64) < dt, - dt64 < toDateTime64(dt, 1, 'UTC'), - - '<=', - dt64 <= dt, - toDateTime(dt64) <= dt, - dt64 <= toDateTime64(dt, 1, 'UTC'), - - '=', - dt64 = dt, - toDateTime(dt64) = dt, - dt64 = toDateTime64(dt, 1, 'UTC'), - - '>=', - dt64 >= dt, - toDateTime(dt64) >= dt, - dt64 >= toDateTime64(dt, 1, 'UTC'), - - '>', - dt64 > dt, - toDateTime(dt64) > dt, - dt64 > toDateTime64(dt, 1, 'UTC'), - - '!=', - dt64 != dt, - toDateTime(dt64) != dt, - dt64 != toDateTime64(dt, 1, 'UTC') -FROM -( - SELECT - number - 1 as n, - toDateTime64(toStartOfInterval(now(), toIntervalSecond(1), 'UTC'), 1, 'UTC') + n AS dt64, - toStartOfInterval(now(), toIntervalSecond(1), 'UTC') AS dt - FROM system.numbers - LIMIT 3 -) diff --git a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference deleted file mode 100644 index e5183ec6a8a..00000000000 --- a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.reference +++ /dev/null @@ -1,3 +0,0 @@ --1 DateTime64(1, \'UTC\') < 1 1 1 <= 1 1 1 = 0 0 0 >= 0 0 0 > 0 0 0 != 1 1 1 -0 DateTime64(1, \'UTC\') < 0 0 0 <= 0 1 0 = 0 1 0 >= 1 1 1 > 1 0 1 != 1 0 1 -1 DateTime64(1, \'UTC\') < 0 0 0 <= 0 0 0 = 0 0 0 >= 1 1 1 > 1 1 1 != 1 1 1 diff --git a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql b/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql deleted file mode 100644 index b780793a777..00000000000 --- a/tests/queries/0_stateless/01561_Date_and_DateTime64_comparision.sql +++ /dev/null @@ -1,43 +0,0 @@ -SELECT - n, - toTypeName(dt64) AS dt64_typename, - - '<', - dt64 < d, - toDate(dt64) < d, - dt64 < toDateTime64(d, 1, 'UTC'), - - '<=', - dt64 <= d, - toDate(dt64) <= d, - dt64 <= toDateTime64(d, 1, 'UTC'), - - '=', - dt64 = d, - toDate(dt64) = d, - dt64 = toDateTime64(d, 1, 'UTC'), - - '>=', - dt64 >= d, - toDate(dt64) >= d, - dt64 >= toDateTime64(d, 1, 'UTC'), - - '>', - dt64 > d, - toDate(dt64) > d, - dt64 > toDateTime64(d, 1, 'UTC'), - - '!=', - dt64 != d, - toDate(dt64) != d, - dt64 != toDateTime64(d, 1, 'UTC') -FROM -( - SELECT - number - 1 as n, - toDateTime64(toStartOfInterval(now(), toIntervalSecond(1), 'UTC'), 1, 'UTC') AS dt64, - toDate(now(), 'UTC') - n as d - FROM system.numbers - LIMIT 3 -) -FORMAT TabSeparated From cbf7e947c862fe1b9a4d20dabc1b4fdff0adaaaf Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:43:07 +0300 Subject: [PATCH 163/174] Update system.md --- docs/ru/sql-reference/statements/system.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ru/sql-reference/statements/system.md b/docs/ru/sql-reference/statements/system.md index e64248096d2..a6a6c5047af 100644 --- a/docs/ru/sql-reference/statements/system.md +++ b/docs/ru/sql-reference/statements/system.md @@ -82,8 +82,8 @@ SYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/path/to/table/in/zk'; Первая команда удаляет метаданные реплики `'replica_name'` для таблицы `database.table`. Вторая команда удаляет метаданные реплики `'replica_name'` для всех таблиц базы данных `database`. -Треться команда удаляет метаданные реплики `'replica_name'` для всех таблиц существующих на локальной реплике (список таблиц генерируется из локальной реплики). -Четверая команда полезна для удаления метадданных мертвой реплики когда все другие реплики таблицы уже были удалены ранее, поэтому необходимо явно указать Zookeeper путь таблицы. Zookeeper путь это первый аргумент для `ReplicatedMergeTree` движка при создании таблицы. +Третья команда удаляет метаданные реплики `'replica_name'` для всех таблиц, существующих на локальном сервере (список таблиц генерируется из локальной реплики). +Четверая команда полезна для удаления метаданных мертвой реплики когда все другие реплики таблицы уже были удалены ранее, поэтому необходимо явно указать ZooKeeper путь таблицы. ZooKeeper путь это первый аргумент для `ReplicatedMergeTree` движка при создании таблицы. ## DROP UNCOMPRESSED CACHE {#query_language-system-drop-uncompressed-cache} From 85a8cc35b3e669499ed868e9739e1b48684e3c15 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:51:07 +0300 Subject: [PATCH 164/174] Update date-time-functions.md --- docs/ru/sql-reference/functions/date-time-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 50152001072..0ab6b83c622 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -303,7 +303,7 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d ## toISOYear {#toisoyear} -Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого 1 января. +Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого января. Пример: From 91224948a2bca9b87c59a5a818cd8d92eb2ef1ed Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:51:55 +0300 Subject: [PATCH 165/174] Update date-time-functions.md --- docs/ru/sql-reference/functions/date-time-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 0ab6b83c622..0ca2c58b793 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -139,7 +139,7 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp ## toStartOfISOYear {#tostartofisoyear} Округляет дату или дату-с-временем вниз до первого дня ISO года. Возвращается дата. -Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года это неделя с 4-мя или более днями в этом году. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года - это неделя с четырьмя или более днями в этом году. 1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это 2016 ISO-год, который начался 2016-01-04. From 4ab123f6c7b56b721cd0c70202cf87e8b15287db Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:52:50 +0300 Subject: [PATCH 166/174] Update date-time-functions.md --- docs/ru/sql-reference/functions/date-time-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 0ca2c58b793..72442557c2d 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -322,7 +322,7 @@ SELECT ## toISOWeek {#toisoweek} Переводит дату-с-временем или дату в число типа UInt8, содержащее номер ISO недели. -Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года это неделя с 4-мя или более днями в этом году. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года - это неделя с четырьмя или более днями в этом году. 1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это последняя неделя 2016 года. From 822eb0c80510acaadc5ce64a0d59c13d0523dc8a Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:53:53 +0300 Subject: [PATCH 167/174] Update date-time-functions.md --- .../sql-reference/functions/date-time-functions.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 72442557c2d..b7a077b3bd6 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -347,16 +347,16 @@ SELECT | Mode | Первый день недели | Диапазон | Неделя 1 это первая неделя … | | ----------- | -------- | -------- | ------------------ | -|0|Воскресенье|0-53|с Воскресеньем в этом году +|0|Воскресенье|0-53|с воскресеньем в этом году |1|Понедельник|0-53|с 4-мя или более днями в этом году -|2|Воскресенье|1-53|с Воскресеньем в этом году +|2|Воскресенье|1-53|с воскресеньем в этом году |3|Понедельник|1-53|с 4-мя или более днями в этом году |4|Воскресенье|0-53|с 4-мя или более днями в этом году -|5|Понедельник|0-53|с Понедельником в этом году +|5|Понедельник|0-53|с понедельником в этом году |6|Воскресенье|1-53|с 4-мя или более днями в этом году -|7|Понедельник|1-53|с Понедельником в этом году -|8|Воскресенье|1-53|содержащая 1 Января -|9|Понедельник|1-53|содержащая 1 Января +|7|Понедельник|1-53|с понедельником в этом году +|8|Воскресенье|1-53|содержащая 1 января +|9|Понедельник|1-53|содержащая 1 января Для режимов со значением «с 4 или более днями в этом году» недели нумеруются в соответствии с ISO 8601:1988: From 14e0b8dd349f1994dfa9f673c853480307959b1a Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:57:18 +0300 Subject: [PATCH 168/174] Update formats.md --- docs/en/interfaces/formats.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 9b6164c67e7..4f9210f3477 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1379,9 +1379,9 @@ Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparat - there are no delimiters between values; - no newline at the end of each value. [TabSeparatedRaw] (#tabseparatedraw): -- data is output without screening; -- the string contains values ​​separated by tabs; -- there is a line feed after the last value in the line. +- data is output without escaping; +- the rows contain values separated by tabs; +- there is a line feed after the last value in every row. The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) formats. `RawBLOB`: From 56a873c789f30b1261504a74c6f451d4016e7c56 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 01:58:27 +0300 Subject: [PATCH 169/174] Update formats.md --- docs/en/interfaces/formats.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 4f9210f3477..f3b2af75486 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1385,11 +1385,10 @@ Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparat The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) formats. `RawBLOB`: -- lines are output without their length. -[TabSeparatedRaw](#tabseparatedraw): +- strings are output without their length. - strings are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. -When an empty value is passed to the `RawBLOB` input, ClickHouse generates an exception: +When an empty data is passed to the `RawBLOB` input, ClickHouse throws an exception: ``` text Code: 108. DB::Exception: No data to insert From 3874d62db8168e31e6220f613ce4417d0f37bcb5 Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 02:00:43 +0300 Subject: [PATCH 170/174] Update formats.md --- docs/ru/interfaces/formats.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index bd9ca6bb1ad..dd949c1c34b 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -1256,7 +1256,7 @@ SELECT * FROM line_as_string; `RawBLOB`: - данные выводятся в бинарном виде, без экранирования; - нет разделителей между значениями; -- нет новой строки в конце каждого значения. +- нет перевода строки в конце каждого значения. [TabSeparatedRaw](#tabseparatedraw): - данные выводятся без экранирования; - строка содержит значения, разделённые табуляцией; @@ -1264,11 +1264,11 @@ SELECT * FROM line_as_string; Далее рассмотрено сравнение форматов `RawBLOB` и [RowBinary](#rowbinary). `RawBLOB`: -- строки выводятся без их длины. -[TabSeparatedRaw](#tabseparatedraw): +- строки выводятся без их длины в начале. +`RowBinary`: - строки представлены как длина в формате varint (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), а затем байты строки. -При передаче на вход `RawBLOB` пустого значения ClickHouse сгенерирует исключение: +При передаче на вход `RawBLOB` пустых данных, ClickHouse бросает исключение: ``` text Code: 108. DB::Exception: No data to insert From 27ba035aa179c88233d98a06cf734537972d44eb Mon Sep 17 00:00:00 2001 From: alexey-milovidov Date: Fri, 11 Dec 2020 02:01:39 +0300 Subject: [PATCH 171/174] Update formats.md --- docs/en/interfaces/formats.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index f3b2af75486..b2f3da84d59 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -1385,8 +1385,9 @@ Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparat The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) formats. `RawBLOB`: -- strings are output without their length. -- strings are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. +- String fields are output without being prefixed by length. +`RowBinary`: +- String fields are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. When an empty data is passed to the `RawBLOB` input, ClickHouse throws an exception: From b6d69eb63ba4f588d56fdebab0f64018ab4f2623 Mon Sep 17 00:00:00 2001 From: nikitamikhaylov Date: Fri, 11 Dec 2020 00:33:13 +0300 Subject: [PATCH 172/174] done --- src/Common/ZooKeeper/TestKeeperStorage.cpp | 2 +- src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp index 6513c9c1050..e3e4edc23a7 100644 --- a/src/Common/ZooKeeper/TestKeeperStorage.cpp +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -427,7 +427,7 @@ struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest for (const auto & sub_request : request.requests) { - auto sub_zk_request = dynamic_pointer_cast(sub_request); + auto sub_zk_request = std::dynamic_pointer_cast(sub_request); if (sub_zk_request->getOpNum() == Coordination::OpNum::Create) { concrete_requests.push_back(std::make_shared(sub_zk_request)); diff --git a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp index 37714c1a83e..f045a92c394 100644 --- a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp +++ b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp @@ -178,7 +178,7 @@ static inline std::tupleas()) { /// column_name(int64 literal) - if (columns_name_set.contains(function->name) && function->arguments->children.size() == 1) + if (columns_name_set.count(function->name) && function->arguments->children.size() == 1) { const auto & prefix_limit = function->arguments->children[0]->as(); From 3ee6b6382b093dd693b5de8b0c5ec36f536ff1bd Mon Sep 17 00:00:00 2001 From: feng lv Date: Thu, 10 Dec 2020 08:26:45 +0000 Subject: [PATCH 173/174] fix fix --- src/Interpreters/TreeRewriter.cpp | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index 0a656383dc3..1025e64e2f4 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -121,18 +121,18 @@ struct CustomizeAggregateFunctionsSuffixData void visit(ASTFunction & func, ASTPtr &) const { const auto & instance = AggregateFunctionFactory::instance(); - if (instance.isAggregateFunctionName(func.name) && !endsWith(func.name, "OrNull")) + if (instance.isAggregateFunctionName(func.name) && !endsWith(func.name, customized_func_suffix)) { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) { - if (!endsWith(func.name, customized_func_suffix)) - func.name += customized_func_suffix; + func.name += customized_func_suffix; } } } }; +// Used to rewrite aggregate functions with -OrNull suffix in some cases, such as sumIfOrNull, we shoule rewrite to sumOrNullIf struct CustomizeAggregateFunctionsMoveSuffixData { using TypeToVisit = ASTFunction; @@ -165,7 +165,7 @@ struct CustomizeAggregateFunctionsMoveSuffixData const auto & instance = AggregateFunctionFactory::instance(); if (instance.isAggregateFunctionName(func.name)) { - if (endsWith(func.name, "OrNull")) + if (endsWith(func.name, customized_func_suffix)) { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) From 401f5024683195623761d4279d33b676040bd63d Mon Sep 17 00:00:00 2001 From: Evgeny Markov Date: Fri, 11 Dec 2020 17:30:15 +0300 Subject: [PATCH 174/174] Fix typo in related links --- docs/ru/operations/settings/query-complexity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ru/operations/settings/query-complexity.md b/docs/ru/operations/settings/query-complexity.md index a62e7523207..b0eac5d96e7 100644 --- a/docs/ru/operations/settings/query-complexity.md +++ b/docs/ru/operations/settings/query-complexity.md @@ -297,7 +297,7 @@ FORMAT Null; **Смотрите также** - [Секция JOIN](../../sql-reference/statements/select/join.md#select-join) -- [Движоy таблиц Join](../../engines/table-engines/special/join.md) +- [Движок таблиц Join](../../engines/table-engines/special/join.md) ## max_partitions_per_insert_block {#max-partitions-per-insert-block}