diff --git a/.gitmodules b/.gitmodules index 27aff5d95c4..1d996fb84cc 100644 --- a/.gitmodules +++ b/.gitmodules @@ -208,3 +208,6 @@ [submodule "contrib/dragonbox"] path = contrib/dragonbox url = https://github.com/ClickHouse-Extras/dragonbox.git +[submodule "contrib/fast_float"] + path = contrib/fast_float + url = https://github.com/fastfloat/fast_float diff --git a/CHANGELOG.md b/CHANGELOG.md index c722e4a1ca0..4d525798505 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ * Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)). * Remove `ANALYZE` and `AST` queries, and make the setting `enable_debug_queries` obsolete since now it is the part of full featured `EXPLAIN` query. [#16536](https://github.com/ClickHouse/ClickHouse/pull/16536) ([Ivan](https://github.com/abyss7)). * Aggregate functions `boundingRatio`, `rankCorr`, `retention`, `timeSeriesGroupSum`, `timeSeriesGroupRateSum`, `windowFunnel` were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. [#16407](https://github.com/ClickHouse/ClickHouse/pull/16407) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Make `rankCorr` function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). +* Make `rankCorr` function return nan on insufficient data [#16124](https://github.com/ClickHouse/ClickHouse/issues/16124). [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). * When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to `Part ... intersects previous part` errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version). #### New Feature @@ -33,7 +33,7 @@ * Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. [#16594](https://github.com/ClickHouse/ClickHouse/pull/16594) ([Amos Bird](https://github.com/amosbird)). * Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. [#16642](https://github.com/ClickHouse/ClickHouse/pull/16642) ([Maksim Kita](https://github.com/kitaisreal)). * Add `farmFingerprint64` function (non-cryptographic string hashing). [#16570](https://github.com/ClickHouse/ClickHouse/pull/16570) ([Jacob Hayes](https://github.com/JacobHayes)). -* Add `log_queries_min_query_duration_ms`, only queries slower then the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)). +* Add `log_queries_min_query_duration_ms`, only queries slower than the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)). * Ability to create a docker image on the top of `Alpine`. Uses precompiled binary and glibc components from ubuntu 20.04. [#16479](https://github.com/ClickHouse/ClickHouse/pull/16479) ([filimonov](https://github.com/filimonov)). * Added `toUUIDOrNull`, `toUUIDOrZero` cast functions. [#16337](https://github.com/ClickHouse/ClickHouse/pull/16337) ([Maksim Kita](https://github.com/kitaisreal)). * Add `max_concurrent_queries_for_all_users` setting, see [#6636](https://github.com/ClickHouse/ClickHouse/issues/6636) for use cases. [#16154](https://github.com/ClickHouse/ClickHouse/pull/16154) ([nvartolomei](https://github.com/nvartolomei)). @@ -178,7 +178,7 @@ * Add `JSONStrings` format which output data in arrays of strings. [#14333](https://github.com/ClickHouse/ClickHouse/pull/14333) ([hcz](https://github.com/hczhcz)). * Add support for "Raw" column format for `Regexp` format. It allows to simply extract subpatterns as a whole without any escaping rules. [#15363](https://github.com/ClickHouse/ClickHouse/pull/15363) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow configurable `NULL` representation for `TSV` output format. It is controlled by the setting `output_format_tsv_null_representation` which is `\N` by default. This closes [#9375](https://github.com/ClickHouse/ClickHouse/issues/9375). Note that the setting only controls output format and `\N` is the only supported `NULL` representation for `TSV` input format. [#14586](https://github.com/ClickHouse/ClickHouse/pull/14586) ([Kruglov Pavel](https://github.com/Avogar)). -* Support Decimal data type for `MaterializedMySQL`. `MaterializedMySQL` is an experimental feature. [#14535](https://github.com/ClickHouse/ClickHouse/pull/14535) ([Winter Zhang](https://github.com/zhang2014)). +* Support Decimal data type for `MaterializeMySQL`. `MaterializeMySQL` is an experimental feature. [#14535](https://github.com/ClickHouse/ClickHouse/pull/14535) ([Winter Zhang](https://github.com/zhang2014)). * Add new feature: `SHOW DATABASES LIKE 'xxx'`. [#14521](https://github.com/ClickHouse/ClickHouse/pull/14521) ([hexiaoting](https://github.com/hexiaoting)). * Added a script to import (arbitrary) git repository to ClickHouse as a sample dataset. [#14471](https://github.com/ClickHouse/ClickHouse/pull/14471) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Now insert statements can have asterisk (or variants) with column transformers in the column list. [#14453](https://github.com/ClickHouse/ClickHouse/pull/14453) ([Amos Bird](https://github.com/amosbird)). @@ -200,18 +200,18 @@ * Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. [#16264](https://github.com/ClickHouse/ClickHouse/pull/16264) ([Amos Bird](https://github.com/amosbird)). * Fix segfault in some cases of wrong aggregation in lambdas. [#16082](https://github.com/ClickHouse/ClickHouse/pull/16082) ([Anton Popov](https://github.com/CurtizJ)). * Fix `ALTER MODIFY ... ORDER BY` query hang for `ReplicatedVersionedCollapsingMergeTree`. This fixes [#15980](https://github.com/ClickHouse/ClickHouse/issues/15980). [#16011](https://github.com/ClickHouse/ClickHouse/pull/16011) ([alesapin](https://github.com/alesapin)). -* `MaterializedMySQL` (experimental feature): Fix collate name & charset name parser and support `length = 0` for string type. [#16008](https://github.com/ClickHouse/ClickHouse/pull/16008) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fix collate name & charset name parser and support `length = 0` for string type. [#16008](https://github.com/ClickHouse/ClickHouse/pull/16008) ([Winter Zhang](https://github.com/zhang2014)). * Allow to use `direct` layout for dictionaries with complex keys. [#16007](https://github.com/ClickHouse/ClickHouse/pull/16007) ([Anton Popov](https://github.com/CurtizJ)). * Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. [#15987](https://github.com/ClickHouse/ClickHouse/pull/15987) ([filimonov](https://github.com/filimonov)). * Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). [#15984](https://github.com/ClickHouse/ClickHouse/pull/15984) ([tavplubix](https://github.com/tavplubix)). * Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes [#15628](https://github.com/ClickHouse/ClickHouse/issues/15628). [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). -* `MaterializedMySQL` (experimental feature): Fix crash on create database failure. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fix crash on create database failure. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). * Fixed `DROP TABLE IF EXISTS` failure with `Table ... doesn't exist` error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like `DROP DATABASE` and `RENAME TABLE`) - Fixed `DROP/DETACH DATABASE` failure with `Table ... doesn't exist` when concurrently executing `DROP/DETACH TABLE`. [#15934](https://github.com/ClickHouse/ClickHouse/pull/15934) ([tavplubix](https://github.com/tavplubix)). * Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes [#15792](https://github.com/ClickHouse/ClickHouse/issues/15792). [#15933](https://github.com/ClickHouse/ClickHouse/pull/15933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixes [#12513](https://github.com/ClickHouse/ClickHouse/issues/12513): difference expressions with same alias when query is reanalyzed. [#15886](https://github.com/ClickHouse/ClickHouse/pull/15886) ([Winter Zhang](https://github.com/zhang2014)). * Fix possible very rare deadlocks in RBAC implementation. [#15875](https://github.com/ClickHouse/ClickHouse/pull/15875) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix exception `Block structure mismatch` in `SELECT ... ORDER BY DESC` queries which were executed after `ALTER MODIFY COLUMN` query. Fixes [#15800](https://github.com/ClickHouse/ClickHouse/issues/15800). [#15852](https://github.com/ClickHouse/ClickHouse/pull/15852) ([alesapin](https://github.com/alesapin)). -* `MaterializedMySQL` (experimental feature): Fix `select count()` inaccuracy. [#15767](https://github.com/ClickHouse/ClickHouse/pull/15767) ([tavplubix](https://github.com/tavplubix)). +* `MaterializeMySQL` (experimental feature): Fix `select count()` inaccuracy. [#15767](https://github.com/ClickHouse/ClickHouse/pull/15767) ([tavplubix](https://github.com/tavplubix)). * Fix some cases of queries, in which only virtual columns are selected. Previously `Not found column _nothing in block` exception may be thrown. Fixes [#12298](https://github.com/ClickHouse/ClickHouse/issues/12298). [#15756](https://github.com/ClickHouse/ClickHouse/pull/15756) ([Anton Popov](https://github.com/CurtizJ)). * Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). [#15743](https://github.com/ClickHouse/ClickHouse/pull/15743) ([Azat Khuzhin](https://github.com/azat)). * Possibility to move part to another disk/volume if the first attempt was failed. [#15723](https://github.com/ClickHouse/ClickHouse/pull/15723) ([Pavel Kovalenko](https://github.com/Jokser)). @@ -243,37 +243,37 @@ * Fix hang of queries with a lot of subqueries to same table of `MySQL` engine. Previously, if there were more than 16 subqueries to same `MySQL` table in query, it hang forever. [#15299](https://github.com/ClickHouse/ClickHouse/pull/15299) ([Anton Popov](https://github.com/CurtizJ)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. [#15242](https://github.com/ClickHouse/ClickHouse/pull/15242) ([Artem Zuikov](https://github.com/4ertus2)). -* Fix instance crash when using `joinGet` with `LowCardinality` types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using `joinGet` with `LowCardinality` types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust Decimal field size in MySQL column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). * Fixes `Data compressed with different methods` in `join_algorithm='auto'`. Keep LowCardinality as type for left table join key in `join_algorithm='partial_merge'`. [#15088](https://github.com/ClickHouse/ClickHouse/pull/15088) ([Artem Zuikov](https://github.com/4ertus2)). * Update `jemalloc` to fix `percpu_arena` with affinity mask. [#15035](https://github.com/ClickHouse/ClickHouse/pull/15035) ([Azat Khuzhin](https://github.com/azat)). [#14957](https://github.com/ClickHouse/ClickHouse/pull/14957) ([Azat Khuzhin](https://github.com/azat)). -* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). * If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in Docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). * Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. [#15002](https://github.com/ClickHouse/ClickHouse/pull/15002) ([Artem Zuikov](https://github.com/4ertus2)). * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains `finalizeAggregation` function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). -* `MaterializedMySQL` (experimental feature): Fixed `.metadata.tmp File exists` error. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* `MaterializeMySQL` (experimental feature): Fixed `.metadata.tmp File exists` error. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile with file descriptor. [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). * Fixed segfault in `cache` dictionary [#14837](https://github.com/ClickHouse/ClickHouse/issues/14837). [#14879](https://github.com/ClickHouse/ClickHouse/pull/14879) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* `MaterializedMySQL` (experimental feature): Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. [#14852](https://github.com/ClickHouse/ClickHouse/pull/14852) ([Winter Zhang](https://github.com/zhang2014)). +* `MaterializeMySQL` (experimental feature): Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. [#14852](https://github.com/ClickHouse/ClickHouse/pull/14852) ([Winter Zhang](https://github.com/zhang2014)). * Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes [#14531](https://github.com/ClickHouse/ClickHouse/issues/14531). [#14845](https://github.com/ClickHouse/ClickHouse/pull/14845) ([alesapin](https://github.com/alesapin)). * Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes [#14814](https://github.com/ClickHouse/ClickHouse/issues/14814). [#14843](https://github.com/ClickHouse/ClickHouse/pull/14843) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)). -* `Replace` column transformer should replace identifiers with cloned ASTs. This fixes https://github.com/ClickHouse/ClickHouse/issues/14695 . [#14734](https://github.com/ClickHouse/ClickHouse/pull/14734) ([Amos Bird](https://github.com/amosbird)). +* `Replace` column transformer should replace identifiers with cloned ASTs. This fixes [#14695](https://github.com/ClickHouse/ClickHouse/issues/14695) . [#14734](https://github.com/ClickHouse/ClickHouse/pull/14734) ([Amos Bird](https://github.com/amosbird)). * Fixed missed default database name in metadata of materialized view when executing `ALTER ... MODIFY QUERY`. [#14664](https://github.com/ClickHouse/ClickHouse/pull/14664) ([tavplubix](https://github.com/tavplubix)). * Fix bug when `ALTER UPDATE` mutation with `Nullable` column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). * Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)). * Fix function `has` with `LowCardinality` of `Nullable`. [#14591](https://github.com/ClickHouse/ClickHouse/pull/14591) ([Mike](https://github.com/myrrc)). * Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. [#14563](https://github.com/ClickHouse/ClickHouse/pull/14563) ([Bharat Nallan](https://github.com/bharatnc)). * Fix rare segfaults in functions with combinator `-Resample`, which could appear in result of overflow with very large parameters. [#14562](https://github.com/ClickHouse/ClickHouse/pull/14562) ([Anton Popov](https://github.com/CurtizJ)). -* Fix a bug when converting `Nullable(String)` to Enum. Introduced by https://github.com/ClickHouse/ClickHouse/pull/12745. This fixes https://github.com/ClickHouse/ClickHouse/issues/14435. [#14530](https://github.com/ClickHouse/ClickHouse/pull/14530) ([Amos Bird](https://github.com/amosbird)). +* Fix a bug when converting `Nullable(String)` to Enum. Introduced by [#12745](https://github.com/ClickHouse/ClickHouse/pull/12745). This fixes [#14435](https://github.com/ClickHouse/ClickHouse/issues/14435). [#14530](https://github.com/ClickHouse/ClickHouse/pull/14530) ([Amos Bird](https://github.com/amosbird)). * Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix `currentDatabase()` function cannot be used in `ON CLUSTER` ddl query. [#14211](https://github.com/ClickHouse/ClickHouse/pull/14211) ([Winter Zhang](https://github.com/zhang2014)). -* `MaterializedMySQL` (experimental feature): Fixed `Packet payload is not fully read` error in `MaterializeMySQL` database engine. [#14696](https://github.com/ClickHouse/ClickHouse/pull/14696) ([BohuTANG](https://github.com/BohuTANG)). +* `MaterializeMySQL` (experimental feature): Fixed `Packet payload is not fully read` error in `MaterializeMySQL` database engine. [#14696](https://github.com/ClickHouse/ClickHouse/pull/14696) ([BohuTANG](https://github.com/BohuTANG)). #### Improvement @@ -308,7 +308,7 @@ * Add an option to skip access checks for `DiskS3`. `s3` disk is an experimental feature. [#14497](https://github.com/ClickHouse/ClickHouse/pull/14497) ([Pavel Kovalenko](https://github.com/Jokser)). * Speed up server shutdown process if there are ongoing S3 requests. [#14496](https://github.com/ClickHouse/ClickHouse/pull/14496) ([Pavel Kovalenko](https://github.com/Jokser)). * `SYSTEM RELOAD CONFIG` now throws an exception if failed to reload and continues using the previous users.xml. The background periodic reloading also continues using the previous users.xml if failed to reload. [#14492](https://github.com/ClickHouse/ClickHouse/pull/14492) ([Vitaly Baranov](https://github.com/vitlibar)). -* For INSERTs with inline data in VALUES format in the script mode of `clickhouse-client`, support semicolon as the data terminator, in addition to the new line. Closes https://github.com/ClickHouse/ClickHouse/issues/12288. [#13192](https://github.com/ClickHouse/ClickHouse/pull/13192) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* For INSERTs with inline data in VALUES format in the script mode of `clickhouse-client`, support semicolon as the data terminator, in addition to the new line. Closes [#12288](https://github.com/ClickHouse/ClickHouse/issues/12288). [#13192](https://github.com/ClickHouse/ClickHouse/pull/13192) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Support custom codecs in compact parts. [#12183](https://github.com/ClickHouse/ClickHouse/pull/12183) ([Anton Popov](https://github.com/CurtizJ)). #### Performance Improvement @@ -320,7 +320,7 @@ * Improve performance of 256-bit types using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. [#14859](https://github.com/ClickHouse/ClickHouse/pull/14859) ([Artem Zuikov](https://github.com/4ertus2)). * Explicitly use a temporary disk to store vertical merge temporary data. [#15639](https://github.com/ClickHouse/ClickHouse/pull/15639) ([Grigory Pervakov](https://github.com/GrigoryPervakov)). * Use one S3 DeleteObjects request instead of multiple DeleteObject in a loop. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. [#15238](https://github.com/ClickHouse/ClickHouse/pull/15238) ([ianton-ru](https://github.com/ianton-ru)). -* Fix `DateTime DateTime` mistakenly choosing the slow generic implementation. This fixes https://github.com/ClickHouse/ClickHouse/issues/15153. [#15178](https://github.com/ClickHouse/ClickHouse/pull/15178) ([Amos Bird](https://github.com/amosbird)). +* Fix `DateTime DateTime` mistakenly choosing the slow generic implementation. This fixes [#15153](https://github.com/ClickHouse/ClickHouse/issues/15153). [#15178](https://github.com/ClickHouse/ClickHouse/pull/15178) ([Amos Bird](https://github.com/amosbird)). * Improve performance of GROUP BY key of type `FixedString`. [#15034](https://github.com/ClickHouse/ClickHouse/pull/15034) ([Amos Bird](https://github.com/amosbird)). * Only `mlock` code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. [#14929](https://github.com/ClickHouse/ClickHouse/pull/14929) ([alexey-milovidov](https://github.com/alexey-milovidov)). * ClickHouse binary become smaller due to link time optimization. @@ -387,7 +387,7 @@ * Allow to use direct layout for dictionaries with complex keys. [#16007](https://github.com/ClickHouse/ClickHouse/pull/16007) ([Anton Popov](https://github.com/CurtizJ)). * Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. [#15987](https://github.com/ClickHouse/ClickHouse/pull/15987) ([filimonov](https://github.com/filimonov)). * Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). [#15984](https://github.com/ClickHouse/ClickHouse/pull/15984) ([tavplubix](https://github.com/tavplubix)). -* Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes https://github.com/ClickHouse/ClickHouse/issues/15628. [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes [#15628](https://github.com/ClickHouse/ClickHouse/issues/15628). [#15982](https://github.com/ClickHouse/ClickHouse/pull/15982) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix a crash when database creation fails. [#15954](https://github.com/ClickHouse/ClickHouse/pull/15954) ([Winter Zhang](https://github.com/zhang2014)). * Fixed `DROP TABLE IF EXISTS` failure with `Table ... doesn't exist` error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like `DROP DATABASE` and `RENAME TABLE`) Fixed `DROP/DETACH DATABASE` failure with `Table ... doesn't exist` when concurrently executing `DROP/DETACH TABLE`. [#15934](https://github.com/ClickHouse/ClickHouse/pull/15934) ([tavplubix](https://github.com/tavplubix)). * Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes [#15792](https://github.com/ClickHouse/ClickHouse/issues/15792). [#15933](https://github.com/ClickHouse/ClickHouse/pull/15933) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -398,7 +398,7 @@ * Fixed too low default value of `max_replicated_logs_to_keep` setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. [#15701](https://github.com/ClickHouse/ClickHouse/pull/15701) ([tavplubix](https://github.com/tavplubix)). * Fix error `Cannot add simple transform to empty Pipe` which happened while reading from `Buffer` table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes [#15529](https://github.com/ClickHouse/ClickHouse/issues/15529). [#15662](https://github.com/ClickHouse/ClickHouse/pull/15662) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. [#15646](https://github.com/ClickHouse/ClickHouse/pull/15646) ([Vladimir Chebotarev](https://github.com/excitoon)). -* Decrement the `ReadonlyReplica` metric when detaching read-only tables. This fixes https://github.com/ClickHouse/ClickHouse/issues/15598. [#15592](https://github.com/ClickHouse/ClickHouse/pull/15592) ([sundyli](https://github.com/sundy-li)). +* Decrement the `ReadonlyReplica` metric when detaching read-only tables. This fixes [#15598](https://github.com/ClickHouse/ClickHouse/issues/15598). [#15592](https://github.com/ClickHouse/ClickHouse/pull/15592) ([sundyli](https://github.com/sundy-li)). * Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. [#15516](https://github.com/ClickHouse/ClickHouse/pull/15516) ([nvartolomei](https://github.com/nvartolomei)). #### Improvement @@ -422,11 +422,11 @@ * Fix `Missing columns` errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes [#15530](https://github.com/ClickHouse/ClickHouse/issues/15530). [#15532](https://github.com/ClickHouse/ClickHouse/pull/15532) ([alesapin](https://github.com/alesapin)). * Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in `ON CLUSTER`. Introduced in [#13450](https://github.com/ClickHouse/ClickHouse/issues/13450). [#15477](https://github.com/ClickHouse/ClickHouse/pull/15477) ([alesapin](https://github.com/alesapin)). * Report proper error when the second argument of `boundingRatio` aggregate function has a wrong type. [#15407](https://github.com/ClickHouse/ClickHouse/pull/15407) ([detailyang](https://github.com/detailyang)). -* Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. [#15319](https://github.com/ClickHouse/ClickHouse/pull/15319) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix bug where queries like `SELECT toStartOfDay(today())` fail complaining about empty time_zone argument. [#15319](https://github.com/ClickHouse/ClickHouse/pull/15319) ([Bharat Nallan](https://github.com/bharatnc)). * Fix race condition during MergeTree table rename and background cleanup. [#15304](https://github.com/ClickHouse/ClickHouse/pull/15304) ([alesapin](https://github.com/alesapin)). * Fix rare race condition on server startup when system.logs are enabled. [#15300](https://github.com/ClickHouse/ClickHouse/pull/15300) ([alesapin](https://github.com/alesapin)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using joinGet with LowCardinality types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust decimals field size in mysql column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). @@ -455,10 +455,10 @@ * Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes [#13634](https://github.com/ClickHouse/ClickHouse/issues/13634), [#14045](https://github.com/ClickHouse/ClickHouse/issues/14045). [#14646](https://github.com/ClickHouse/ClickHouse/pull/14646) ([alesapin](https://github.com/alesapin)). * Fix wrong Decimal multiplication result caused wrong decimal scale of result column. [#14603](https://github.com/ClickHouse/ClickHouse/pull/14603) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed the incorrect sorting order of `Nullable` column. This fixes [#14344](https://github.com/ClickHouse/ClickHouse/issues/14344). [#14495](https://github.com/ClickHouse/ClickHouse/pull/14495) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* Fixed inconsistent comparison with primary key of type `FixedString` on index analysis if they're compered with a string of less size. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* Fixed inconsistent comparison with primary key of type `FixedString` on index analysis if they're compered with a string of less size. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). * Fix bug which leads to wrong merges assignment if table has partitions with a single part. [#14444](https://github.com/ClickHouse/ClickHouse/pull/14444) ([alesapin](https://github.com/alesapin)). * If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes [#13383](https://github.com/ClickHouse/ClickHouse/issues/13383). [#14889](https://github.com/ClickHouse/ClickHouse/pull/14889) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). [#14887](https://github.com/ClickHouse/ClickHouse/pull/14887) ([Azat Khuzhin](https://github.com/azat)). @@ -501,7 +501,7 @@ #### Performance Improvement -* Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under optimize_skip_unused_shards and optimize_distributed_group_by_sharding_key). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)). +* Optimize queries with LIMIT/LIMIT BY/ORDER BY for distributed with GROUP BY sharding_key (under `optimize_skip_unused_shards` and `optimize_distributed_group_by_sharding_key`). [#10373](https://github.com/ClickHouse/ClickHouse/pull/10373) ([Azat Khuzhin](https://github.com/azat)). * Creating sets for multiple `JOIN` and `IN` in parallel. It may slightly improve performance for queries with several different `IN subquery` expressions. [#14412](https://github.com/ClickHouse/ClickHouse/pull/14412) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Improve Kafka engine performance by providing independent thread for each consumer. Separate thread pool for streaming engines (like Kafka). [#13939](https://github.com/ClickHouse/ClickHouse/pull/13939) ([fastio](https://github.com/fastio)). @@ -579,15 +579,15 @@ * Fix race condition during MergeTree table rename and background cleanup. [#15304](https://github.com/ClickHouse/ClickHouse/pull/15304) ([alesapin](https://github.com/alesapin)). * Fix rare race condition on server startup when system.logs are enabled. [#15300](https://github.com/ClickHouse/ClickHouse/pull/15300) ([alesapin](https://github.com/alesapin)). * Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. [#15258](https://github.com/ClickHouse/ClickHouse/pull/15258) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). +* Fix instance crash when using joinGet with LowCardinality types. This fixes [#15214](https://github.com/ClickHouse/ClickHouse/issues/15214). [#15220](https://github.com/ClickHouse/ClickHouse/pull/15220) ([Amos Bird](https://github.com/amosbird)). * Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes [#15117](https://github.com/ClickHouse/ClickHouse/issues/15117). [#15192](https://github.com/ClickHouse/ClickHouse/pull/15192) ([alesapin](https://github.com/alesapin)). * Adjust decimals field size in mysql column definition packet. [#15152](https://github.com/ClickHouse/ClickHouse/pull/15152) ([maqroll](https://github.com/maqroll)). -* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). -* If function `bar` was called with specifically crafter arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes [#14908](https://github.com/ClickHouse/ClickHouse/issues/14908). [#15033](https://github.com/ClickHouse/ClickHouse/pull/15033) ([Amos Bird](https://github.com/amosbird)). +* If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes [#13926](https://github.com/ClickHouse/ClickHouse/issues/13926). [#15028](https://github.com/ClickHouse/ClickHouse/pull/15028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. [#15024](https://github.com/ClickHouse/ClickHouse/pull/15024) ([tavplubix](https://github.com/tavplubix)). * Now settings `number_of_free_entries_in_pool_to_execute_mutation` and `number_of_free_entries_in_pool_to_lower_max_size_of_merge` can be equal to `background_pool_size`. [#14975](https://github.com/ClickHouse/ClickHouse/pull/14975) ([alesapin](https://github.com/alesapin)). * Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes [#14847](https://github.com/ClickHouse/ClickHouse/issues/14847). [#14937](https://github.com/ClickHouse/ClickHouse/pull/14937) ([filimonov](https://github.com/filimonov)). -* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes [#14923](https://github.com/ClickHouse/ClickHouse/issues/14923). [#14924](https://github.com/ClickHouse/ClickHouse/pull/14924) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. [#14898](https://github.com/ClickHouse/ClickHouse/pull/14898) ([Winter Zhang](https://github.com/zhang2014)). * Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes [#14814](https://github.com/ClickHouse/ClickHouse/issues/14814). [#14843](https://github.com/ClickHouse/ClickHouse/pull/14843) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in [#14513](https://github.com/ClickHouse/ClickHouse/issues/14513). [#14783](https://github.com/ClickHouse/ClickHouse/pull/14783) ([Amos Bird](https://github.com/amosbird)). @@ -647,16 +647,16 @@ * Fix visible data clobbering by progress bar in client in interactive mode. This fixes [#12562](https://github.com/ClickHouse/ClickHouse/issues/12562) and [#13369](https://github.com/ClickHouse/ClickHouse/issues/13369) and [#13584](https://github.com/ClickHouse/ClickHouse/issues/13584) and fixes [#12964](https://github.com/ClickHouse/ClickHouse/issues/12964). [#13691](https://github.com/ClickHouse/ClickHouse/pull/13691) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect sorting order if `LowCardinality` column when sorting by multiple columns. This fixes [#13958](https://github.com/ClickHouse/ClickHouse/issues/13958). [#14223](https://github.com/ClickHouse/ClickHouse/pull/14223) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). -* Check for array size overflow in `topK` aggregate function. Without this check the user may send a query with carefully crafter parameters that will lead to server crash. This closes [#14452](https://github.com/ClickHouse/ClickHouse/issues/14452). [#14467](https://github.com/ClickHouse/ClickHouse/pull/14467) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Check for array size overflow in `topK` aggregate function. Without this check the user may send a query with carefully crafted parameters that will lead to server crash. This closes [#14452](https://github.com/ClickHouse/ClickHouse/issues/14452). [#14467](https://github.com/ClickHouse/ClickHouse/pull/14467) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug which can lead to wrong merges assignment if table has partitions with a single part. [#14444](https://github.com/ClickHouse/ClickHouse/pull/14444) ([alesapin](https://github.com/alesapin)). * Stop query execution if exception happened in `PipelineExecutor` itself. This could prevent rare possible query hung. Continuation of [#14334](https://github.com/ClickHouse/ClickHouse/issues/14334). [#14402](https://github.com/ClickHouse/ClickHouse/pull/14402) [#14334](https://github.com/ClickHouse/ClickHouse/pull/14334) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix crash during `ALTER` query for table which was created `AS table_function`. Fixes [#14212](https://github.com/ClickHouse/ClickHouse/issues/14212). [#14326](https://github.com/ClickHouse/ClickHouse/pull/14326) ([alesapin](https://github.com/alesapin)). * Fix exception during ALTER LIVE VIEW query with REFRESH command. Live view is an experimental feature. [#14320](https://github.com/ClickHouse/ClickHouse/pull/14320) ([Bharat Nallan](https://github.com/bharatnc)). * Fix QueryPlan lifetime (for EXPLAIN PIPELINE graph=1) for queries with nested interpreter. [#14315](https://github.com/ClickHouse/ClickHouse/pull/14315) ([Azat Khuzhin](https://github.com/azat)). -* Fix segfault in `clickhouse-odbc-bridge` during schema fetch from some external sources. This PR fixes https://github.com/ClickHouse/ClickHouse/issues/13861. [#14267](https://github.com/ClickHouse/ClickHouse/pull/14267) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash in mark inclusion search introduced in https://github.com/ClickHouse/ClickHouse/pull/12277. [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). +* Fix segfault in `clickhouse-odbc-bridge` during schema fetch from some external sources. This PR fixes [#13861](https://github.com/ClickHouse/ClickHouse/issues/13861). [#14267](https://github.com/ClickHouse/ClickHouse/pull/14267) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash in mark inclusion search introduced in [#12277](https://github.com/ClickHouse/ClickHouse/pull/12277). [#14225](https://github.com/ClickHouse/ClickHouse/pull/14225) ([Amos Bird](https://github.com/amosbird)). * Fix creation of tables with named tuples. This fixes [#13027](https://github.com/ClickHouse/ClickHouse/issues/13027). [#14143](https://github.com/ClickHouse/ClickHouse/pull/14143) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix formatting of minimal negative decimal numbers. This fixes https://github.com/ClickHouse/ClickHouse/issues/14111. [#14119](https://github.com/ClickHouse/ClickHouse/pull/14119) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Fix formatting of minimal negative decimal numbers. This fixes [#14111](https://github.com/ClickHouse/ClickHouse/issues/14111). [#14119](https://github.com/ClickHouse/ClickHouse/pull/14119) ([Alexander Kuzmenkov](https://github.com/akuzm)). * Fix `DistributedFilesToInsert` metric (zeroed when it should not). [#14095](https://github.com/ClickHouse/ClickHouse/pull/14095) ([Azat Khuzhin](https://github.com/azat)). * Fix `pointInPolygon` with const 2d array as polygon. [#14079](https://github.com/ClickHouse/ClickHouse/pull/14079) ([Alexey Ilyukhov](https://github.com/livace)). * Fixed wrong mount point in extra info for `Poco::Exception: no space left on device`. [#14050](https://github.com/ClickHouse/ClickHouse/pull/14050) ([tavplubix](https://github.com/tavplubix)). @@ -685,10 +685,10 @@ * Fix wrong code in function `netloc`. This fixes [#13335](https://github.com/ClickHouse/ClickHouse/issues/13335). [#13446](https://github.com/ClickHouse/ClickHouse/pull/13446) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix possible race in `StorageMemory`. [#13416](https://github.com/ClickHouse/ClickHouse/pull/13416) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix missing or excessive headers in `TSV/CSVWithNames` formats in HTTP protocol. This fixes [#12504](https://github.com/ClickHouse/ClickHouse/issues/12504). [#13343](https://github.com/ClickHouse/ClickHouse/pull/13343) ([Azat Khuzhin](https://github.com/azat)). -* Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes https://github.com/ClickHouse/ClickHouse/issues/5779, https://github.com/ClickHouse/ClickHouse/issues/12527. [#13199](https://github.com/ClickHouse/ClickHouse/pull/13199) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes [#5779](https://github.com/ClickHouse/ClickHouse/issues/5779), [#12527](https://github.com/ClickHouse/ClickHouse/issues/12527). [#13199](https://github.com/ClickHouse/ClickHouse/pull/13199) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix access to `redis` dictionary after connection was dropped once. It may happen with `cache` and `direct` dictionary layouts. [#13082](https://github.com/ClickHouse/ClickHouse/pull/13082) ([Anton Popov](https://github.com/CurtizJ)). * Removed wrong auth access check when using ClickHouseDictionarySource to query remote tables. [#12756](https://github.com/ClickHouse/ClickHouse/pull/12756) ([sundyli](https://github.com/sundy-li)). -* Properly distinguish subqueries in some cases for common subexpression elimination. https://github.com/ClickHouse/ClickHouse/issues/8333. [#8367](https://github.com/ClickHouse/ClickHouse/pull/8367) ([Amos Bird](https://github.com/amosbird)). +* Properly distinguish subqueries in some cases for common subexpression elimination. [#8333](https://github.com/ClickHouse/ClickHouse/issues/8333). [#8367](https://github.com/ClickHouse/ClickHouse/pull/8367) ([Amos Bird](https://github.com/amosbird)). #### Improvement @@ -756,7 +756,7 @@ * Updating LDAP user authentication suite to check that it works with RBAC. [#13656](https://github.com/ClickHouse/ClickHouse/pull/13656) ([vzakaznikov](https://github.com/vzakaznikov)). * Removed `-DENABLE_CURL_CLIENT` for `contrib/aws`. [#13628](https://github.com/ClickHouse/ClickHouse/pull/13628) ([Vladimir Chebotarev](https://github.com/excitoon)). * Increasing health-check timeouts for ClickHouse nodes and adding support to dump docker-compose logs if unhealthy containers found. [#13612](https://github.com/ClickHouse/ClickHouse/pull/13612) ([vzakaznikov](https://github.com/vzakaznikov)). -* Make sure https://github.com/ClickHouse/ClickHouse/issues/10977 is invalid. [#13539](https://github.com/ClickHouse/ClickHouse/pull/13539) ([Amos Bird](https://github.com/amosbird)). +* Make sure [#10977](https://github.com/ClickHouse/ClickHouse/issues/10977) is invalid. [#13539](https://github.com/ClickHouse/ClickHouse/pull/13539) ([Amos Bird](https://github.com/amosbird)). * Skip PR's from robot-clickhouse. [#13489](https://github.com/ClickHouse/ClickHouse/pull/13489) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Move Dockerfiles from integration tests to `docker/test` directory. docker_compose files are available in `runner` docker container. Docker images are built in CI and not in integration tests. [#13448](https://github.com/ClickHouse/ClickHouse/pull/13448) ([Ilya Yatsishin](https://github.com/qoega)). @@ -788,7 +788,7 @@ * Add `FROM_UNIXTIME` function for compatibility with MySQL, related to [12149](https://github.com/ClickHouse/ClickHouse/issues/12149). [#12484](https://github.com/ClickHouse/ClickHouse/pull/12484) ([flynn](https://github.com/ucasFL)). * Allow Nullable types as keys in MergeTree tables if `allow_nullable_key` table setting is enabled. Closes [#5319](https://github.com/ClickHouse/ClickHouse/issues/5319). [#12433](https://github.com/ClickHouse/ClickHouse/pull/12433) ([Amos Bird](https://github.com/amosbird)). * Integration with [COS](https://intl.cloud.tencent.com/product/cos). [#12386](https://github.com/ClickHouse/ClickHouse/pull/12386) ([fastio](https://github.com/fastio)). -* Add mapAdd and mapSubtract functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). +* Add `mapAdd` and `mapSubtract` functions for adding/subtracting key-mapped values. [#11735](https://github.com/ClickHouse/ClickHouse/pull/11735) ([Ildus Kurbangaliev](https://github.com/ildus)). #### Bug Fix @@ -1071,7 +1071,7 @@ * Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with `optimize_aggregation_in_order` setting, disabled by default). [#11696](https://github.com/ClickHouse/ClickHouse/pull/11696) ([Anton Popov](https://github.com/CurtizJ)). * Removed injective functions inside `uniq*()` if `set optimize_injective_functions_inside_uniq=1`. [#12337](https://github.com/ClickHouse/ClickHouse/pull/12337) ([Ruslan Kamalov](https://github.com/kamalov-ruslan)). -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). * Implemented single part uploads for DiskS3 (experimental feature). [#12026](https://github.com/ClickHouse/ClickHouse/pull/12026) ([Vladimir Chebotarev](https://github.com/excitoon)). #### Experimental Feature @@ -1133,7 +1133,7 @@ #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). #### Build/Testing/Packaging Improvement @@ -1213,7 +1213,7 @@ * Fix wrong result of comparison of FixedString with constant String. This fixes [#11393](https://github.com/ClickHouse/ClickHouse/issues/11393). This bug appeared in version 20.4. [#11828](https://github.com/ClickHouse/ClickHouse/pull/11828) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix wrong result for `if` with NULLs in condition. [#11807](https://github.com/ClickHouse/ClickHouse/pull/11807) ([Artem Zuikov](https://github.com/4ertus2)). * Fix using too many threads for queries. [#11788](https://github.com/ClickHouse/ClickHouse/pull/11788) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). -* Fixed `Scalar doesn't exist` exception when using `WITH ...` in `SELECT ... FROM merge_tree_table ...` https://github.com/ClickHouse/ClickHouse/issues/11621. [#11767](https://github.com/ClickHouse/ClickHouse/pull/11767) ([Amos Bird](https://github.com/amosbird)). +* Fixed `Scalar doesn't exist` exception when using `WITH ...` in `SELECT ... FROM merge_tree_table ...` [#11621](https://github.com/ClickHouse/ClickHouse/issues/11621). [#11767](https://github.com/ClickHouse/ClickHouse/pull/11767) ([Amos Bird](https://github.com/amosbird)). * Fix unexpected behaviour of queries like `SELECT *, xyz.*` which were success while an error expected. [#11753](https://github.com/ClickHouse/ClickHouse/pull/11753) ([hexiaoting](https://github.com/hexiaoting)). * Now replicated fetches will be cancelled during metadata alter. [#11744](https://github.com/ClickHouse/ClickHouse/pull/11744) ([alesapin](https://github.com/alesapin)). * Parse metadata stored in zookeeper before checking for equality. [#11739](https://github.com/ClickHouse/ClickHouse/pull/11739) ([Azat Khuzhin](https://github.com/azat)). @@ -1264,8 +1264,8 @@ * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash when `SET DEFAULT ROLE` is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash while reading malformed data in `Protobuf` format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash when `SET DEFAULT ROLE` is called with wrong arguments. This fixes [#10586](https://github.com/ClickHouse/ClickHouse/issues/10586). [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in `Protobuf` format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when `cache` dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1331,7 +1331,7 @@ * Fix error `the BloomFilter false positive must be a double number between 0 and 1` [#10551](https://github.com/ClickHouse/ClickHouse/issues/10551). [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). * Fix SELECT of column ALIAS which default expression type different from column type. [#10563](https://github.com/ClickHouse/ClickHouse/pull/10563) ([Azat Khuzhin](https://github.com/azat)). * Implemented comparison between DateTime64 and String values (just like for DateTime). [#10560](https://github.com/ClickHouse/ClickHouse/pull/10560) ([Vasily Nemkov](https://github.com/Enmk)). -* Fix index corruption, which may accur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). +* Fix index corruption, which may occur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). * Disable GROUP BY sharding_key optimization by default (`optimize_distributed_group_by_sharding_key` had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is `if` in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. [#10516](https://github.com/ClickHouse/ClickHouse/pull/10516) ([Azat Khuzhin](https://github.com/azat)). * Fixes: [#10263](https://github.com/ClickHouse/ClickHouse/issues/10263) (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: [#8756](https://github.com/ClickHouse/ClickHouse/issues/8756) (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): `internal_replication == false`, multiple local shards (activates the hardlinking code) and `distributed_storage_policy` (makes `link(2)` fails on `EXDEV`)). [#10486](https://github.com/ClickHouse/ClickHouse/pull/10486) ([Azat Khuzhin](https://github.com/azat)). * Fixed error with "max_rows_to_sort" limit. [#10268](https://github.com/ClickHouse/ClickHouse/pull/10268) ([alexey-milovidov](https://github.com/alexey-milovidov)). @@ -1488,7 +1488,7 @@ * Lower memory usage in tests. [#10617](https://github.com/ClickHouse/ClickHouse/pull/10617) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixing hard coded timeouts in new live view tests. [#10604](https://github.com/ClickHouse/ClickHouse/pull/10604) ([vzakaznikov](https://github.com/vzakaznikov)). * Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. [#10599](https://github.com/ClickHouse/ClickHouse/pull/10599) ([vzakaznikov](https://github.com/vzakaznikov)). -* Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. [#10585](https://github.com/ClickHouse/ClickHouse/pull/10585) ([Amos Bird](https://github.com/amosbird)). +* Enable ThinLTO for clang builds, continuation of [#10435](https://github.com/ClickHouse/ClickHouse/pull/10435). [#10585](https://github.com/ClickHouse/ClickHouse/pull/10585) ([Amos Bird](https://github.com/amosbird)). * Adding fuzzers and preparing for oss-fuzz integration. [#10546](https://github.com/ClickHouse/ClickHouse/pull/10546) ([kyprizel](https://github.com/kyprizel)). * Fix FreeBSD build. [#10150](https://github.com/ClickHouse/ClickHouse/pull/10150) ([Ivan](https://github.com/abyss7)). * Add new build for query tests using pytest framework. [#10039](https://github.com/ClickHouse/ClickHouse/pull/10039) ([Ivan](https://github.com/abyss7)). @@ -1563,7 +1563,7 @@ #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). #### Build/Testing/Packaging Improvement @@ -1617,7 +1617,7 @@ * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. [#11420](https://github.com/ClickHouse/ClickHouse/pull/11420) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). * Fix possible `Pipeline stuck` error for queries with external sort and limit. Fixes [#11359](https://github.com/ClickHouse/ClickHouse/issues/11359). [#11366](https://github.com/ClickHouse/ClickHouse/pull/11366) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1633,8 +1633,8 @@ * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes [#10586](https://github.com/ClickHouse/ClickHouse/issues/10586). [#11278](https://github.com/ClickHouse/ClickHouse/pull/11278) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1679,7 +1679,7 @@ No changes compared to v20.4.3.16-stable. * Now constraints are updated if the column participating in `CONSTRAINT` expression was renamed. Fixes [#10844](https://github.com/ClickHouse/ClickHouse/issues/10844). [#10847](https://github.com/ClickHouse/ClickHouse/pull/10847) ([alesapin](https://github.com/alesapin)). * Fixed potential read of uninitialized memory in cache-dictionary. [#10834](https://github.com/ClickHouse/ClickHouse/pull/10834) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed columns order after `Block::sortColumns()`. [#10826](https://github.com/ClickHouse/ClickHouse/pull/10826) ([Azat Khuzhin](https://github.com/azat)). -* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984](https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UBSan` and `MSan` report in `DateLUT`. [#10798](https://github.com/ClickHouse/ClickHouse/pull/10798) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect type conversion in key conditions. Fixes [#6287](https://github.com/ClickHouse/ClickHouse/issues/6287). [#10791](https://github.com/ClickHouse/ClickHouse/pull/10791) ([Andrew Onyshchuk](https://github.com/oandrew)). * Fixed `parallel_view_processing` behavior. Now all insertions into `MATERIALIZED VIEW` without exception should be finished if exception happened. Fixes [#10241](https://github.com/ClickHouse/ClickHouse/issues/10241). [#10757](https://github.com/ClickHouse/ClickHouse/pull/10757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -1707,15 +1707,15 @@ No changes compared to v20.4.3.16-stable. #### New Feature * Add support for secured connection from ClickHouse to Zookeeper [#10184](https://github.com/ClickHouse/ClickHouse/pull/10184) ([Konstantin Lebedev](https://github.com/xzkostyan)) -* Support custom HTTP handlers. See ISSUES-5436 for description. [#7572](https://github.com/ClickHouse/ClickHouse/pull/7572) ([Winter Zhang](https://github.com/zhang2014)) +* Support custom HTTP handlers. See [#5436](https://github.com/ClickHouse/ClickHouse/issues/5436) for description. [#7572](https://github.com/ClickHouse/ClickHouse/pull/7572) ([Winter Zhang](https://github.com/zhang2014)) * Add MessagePack Input/Output format. [#9889](https://github.com/ClickHouse/ClickHouse/pull/9889) ([Kruglov Pavel](https://github.com/Avogar)) * Add Regexp input format. [#9196](https://github.com/ClickHouse/ClickHouse/pull/9196) ([Kruglov Pavel](https://github.com/Avogar)) * Added output format `Markdown` for embedding tables in markdown documents. [#10317](https://github.com/ClickHouse/ClickHouse/pull/10317) ([Kruglov Pavel](https://github.com/Avogar)) * Added support for custom settings section in dictionaries. Also fixes issue [#2829](https://github.com/ClickHouse/ClickHouse/issues/2829). [#10137](https://github.com/ClickHouse/ClickHouse/pull/10137) ([Artem Streltsov](https://github.com/kekekekule)) -* Added custom settings support in DDL-queries for CREATE DICTIONARY [#10465](https://github.com/ClickHouse/ClickHouse/pull/10465) ([Artem Streltsov](https://github.com/kekekekule)) +* Added custom settings support in DDL-queries for `CREATE DICTIONARY` [#10465](https://github.com/ClickHouse/ClickHouse/pull/10465) ([Artem Streltsov](https://github.com/kekekekule)) * Add simple server-wide memory profiler that will collect allocation contexts when server memory usage becomes higher than the next allocation threshold. [#10444](https://github.com/ClickHouse/ClickHouse/pull/10444) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Add setting `always_fetch_merged_part` which restrict replica to merge parts by itself and always prefer dowloading from other replicas. [#10379](https://github.com/ClickHouse/ClickHouse/pull/10379) ([alesapin](https://github.com/alesapin)) -* Add function JSONExtractKeysAndValuesRaw which extracts raw data from JSON objects [#10378](https://github.com/ClickHouse/ClickHouse/pull/10378) ([hcz](https://github.com/hczhcz)) +* Add function `JSONExtractKeysAndValuesRaw` which extracts raw data from JSON objects [#10378](https://github.com/ClickHouse/ClickHouse/pull/10378) ([hcz](https://github.com/hczhcz)) * Add memory usage from OS to `system.asynchronous_metrics`. [#10361](https://github.com/ClickHouse/ClickHouse/pull/10361) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Added generic variants for functions `least` and `greatest`. Now they work with arbitrary number of arguments of arbitrary types. This fixes [#4767](https://github.com/ClickHouse/ClickHouse/issues/4767) [#10318](https://github.com/ClickHouse/ClickHouse/pull/10318) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Now ClickHouse controls timeouts of dictionary sources on its side. Two new settings added to cache dictionary configuration: `strict_max_lifetime_seconds`, which is `max_lifetime` by default, and `query_wait_timeout_milliseconds`, which is one minute by default. The first settings is also useful with `allow_read_expired_keys` settings (to forbid reading very expired keys). [#10337](https://github.com/ClickHouse/ClickHouse/pull/10337) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)) @@ -1728,7 +1728,7 @@ No changes compared to v20.4.3.16-stable. * Add ability to query Distributed over Distributed (w/o `distributed_group_by_no_merge`) ... [#9923](https://github.com/ClickHouse/ClickHouse/pull/9923) ([Azat Khuzhin](https://github.com/azat)) * Add function `arrayReduceInRanges` which aggregates array elements in given ranges. [#9598](https://github.com/ClickHouse/ClickHouse/pull/9598) ([hcz](https://github.com/hczhcz)) * Add Dictionary Status on prometheus exporter. [#9622](https://github.com/ClickHouse/ClickHouse/pull/9622) ([Guillaume Tassery](https://github.com/YiuRULE)) -* Add function arrayAUC [#8698](https://github.com/ClickHouse/ClickHouse/pull/8698) ([taiyang-li](https://github.com/taiyang-li)) +* Add function `arrayAUC` [#8698](https://github.com/ClickHouse/ClickHouse/pull/8698) ([taiyang-li](https://github.com/taiyang-li)) * Support `DROP VIEW` statement for better TPC-H compatibility. [#9831](https://github.com/ClickHouse/ClickHouse/pull/9831) ([Amos Bird](https://github.com/amosbird)) * Add 'strict_order' option to windowFunnel() [#9773](https://github.com/ClickHouse/ClickHouse/pull/9773) ([achimbab](https://github.com/achimbab)) * Support `DATE` and `TIMESTAMP` SQL operators, e.g. `SELECT date '2001-01-01'` [#9691](https://github.com/ClickHouse/ClickHouse/pull/9691) ([Artem Zuikov](https://github.com/4ertus2)) @@ -1932,7 +1932,7 @@ No changes compared to v20.4.3.16-stable. * Move integration tests docker files to docker/ directory. [#10335](https://github.com/ClickHouse/ClickHouse/pull/10335) ([Ilya Yatsishin](https://github.com/qoega)) * Allow to use `clang-10` in CI. It ensures that [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238) is fixed. [#10384](https://github.com/ClickHouse/ClickHouse/pull/10384) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Update OpenSSL to upstream master. Fixed the issue when TLS connections may fail with the message `OpenSSL SSL_read: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error` and `SSL Exception: error:2400006E:random number generator::error retrieving entropy`. The issue was present in version 20.1. [#8956](https://github.com/ClickHouse/ClickHouse/pull/8956) ([alexey-milovidov](https://github.com/alexey-milovidov)) -* Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238 [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)) +* Fix clang-10 build. [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238) [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)) * Add performance test for [Parallel INSERT for materialized view](https://github.com/ClickHouse/ClickHouse/pull/10052). [#10345](https://github.com/ClickHouse/ClickHouse/pull/10345) ([vxider](https://github.com/Vxider)) * Fix flaky test `test_settings_constraints_distributed.test_insert_clamps_settings`. [#10346](https://github.com/ClickHouse/ClickHouse/pull/10346) ([Vitaly Baranov](https://github.com/vitlibar)) * Add util to test results upload in CI ClickHouse [#10330](https://github.com/ClickHouse/ClickHouse/pull/10330) ([Ilya Yatsishin](https://github.com/qoega)) @@ -2106,7 +2106,7 @@ No changes compared to v20.4.3.16-stable. #### Performance Improvement -* Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "[#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). +* Index not used for IN operator with literals, performance regression introduced around v19.3. This fixes [#10574](https://github.com/ClickHouse/ClickHouse/issues/10574). [#12062](https://github.com/ClickHouse/ClickHouse/pull/12062) ([nvartolomei](https://github.com/nvartolomei)). ### ClickHouse release v20.3.12.112-lts 2020-06-25 @@ -2148,7 +2148,7 @@ No changes compared to v20.4.3.16-stable. * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fix crash in JOIN over LowCarinality(T) and Nullable(T). [#11380](https://github.com/ClickHouse/ClickHouse/issues/11380). [#11414](https://github.com/ClickHouse/ClickHouse/pull/11414) ([Artem Zuikov](https://github.com/4ertus2)). * Fix error code for wrong `USING` key. [#11373](https://github.com/ClickHouse/ClickHouse/issues/11373). [#11404](https://github.com/ClickHouse/ClickHouse/pull/11404) ([Artem Zuikov](https://github.com/4ertus2)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). @@ -2165,7 +2165,7 @@ No changes compared to v20.4.3.16-stable. * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fixed a bug when cache-dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. [#11233](https://github.com/ClickHouse/ClickHouse/pull/11233) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). * Fix error `Block structure mismatch in QueryPipeline` while reading from `VIEW` with constants in inner query. Fixes [#11181](https://github.com/ClickHouse/ClickHouse/issues/11181). [#11205](https://github.com/ClickHouse/ClickHouse/pull/11205) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix possible exception `Invalid status for associated output`. [#11200](https://github.com/ClickHouse/ClickHouse/pull/11200) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2196,7 +2196,7 @@ No changes compared to v20.4.3.16-stable. * Fixed `SIGSEGV` in `StringHashTable` if such a key does not exist. [#10870](https://github.com/ClickHouse/ClickHouse/pull/10870) ([Azat Khuzhin](https://github.com/azat)). * Fixed bug in `ReplicatedMergeTree` which might cause some `ALTER` on `OPTIMIZE` query to hang waiting for some replica after it become inactive. [#10849](https://github.com/ClickHouse/ClickHouse/pull/10849) ([tavplubix](https://github.com/tavplubix)). * Fixed columns order after `Block::sortColumns()`. [#10826](https://github.com/ClickHouse/ClickHouse/pull/10826) ([Azat Khuzhin](https://github.com/azat)). -* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984] (https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed the issue with `ODBC` bridge when no quoting of identifiers is requested. Fixes [#7984](https://github.com/ClickHouse/ClickHouse/issues/7984). [#10821](https://github.com/ClickHouse/ClickHouse/pull/10821) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UBSan` and `MSan` report in `DateLUT`. [#10798](https://github.com/ClickHouse/ClickHouse/pull/10798) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect type conversion in key conditions. Fixes [#6287](https://github.com/ClickHouse/ClickHouse/issues/6287). [#10791](https://github.com/ClickHouse/ClickHouse/pull/10791) ([Andrew Onyshchuk](https://github.com/oandrew)) * Fixed `parallel_view_processing` behavior. Now all insertions into `MATERIALIZED VIEW` without exception should be finished if exception happened. Fixes [#10241](https://github.com/ClickHouse/ClickHouse/issues/10241). [#10757](https://github.com/ClickHouse/ClickHouse/pull/10757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2215,7 +2215,7 @@ No changes compared to v20.4.3.16-stable. * Fixed incorrect scalar results inside inner query of `MATERIALIZED VIEW` in case if this query contained dependent table. [#10603](https://github.com/ClickHouse/ClickHouse/pull/10603) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fixed `SELECT` of column `ALIAS` which default expression type different from column type. [#10563](https://github.com/ClickHouse/ClickHouse/pull/10563) ([Azat Khuzhin](https://github.com/azat)). * Implemented comparison between DateTime64 and String values. [#10560](https://github.com/ClickHouse/ClickHouse/pull/10560) ([Vasily Nemkov](https://github.com/Enmk)). -* Fixed index corruption, which may accur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). +* Fixed index corruption, which may occur in some cases after merge compact parts into another compact part. [#10531](https://github.com/ClickHouse/ClickHouse/pull/10531) ([Anton Popov](https://github.com/CurtizJ)). * Fixed the situation, when mutation finished all parts, but hung up in `is_done=0`. [#10526](https://github.com/ClickHouse/ClickHouse/pull/10526) ([alesapin](https://github.com/alesapin)). * Fixed overflow at beginning of unix epoch for timezones with fractional offset from `UTC`. This fixes [#9335](https://github.com/ClickHouse/ClickHouse/issues/9335). [#10513](https://github.com/ClickHouse/ClickHouse/pull/10513) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed improper shutdown of `Distributed` storage. [#10491](https://github.com/ClickHouse/ClickHouse/pull/10491) ([Azat Khuzhin](https://github.com/azat)). @@ -2225,14 +2225,14 @@ No changes compared to v20.4.3.16-stable. #### Build/Testing/Packaging Improvement * Fix UBSan report in LZ4 library. [#10631](https://github.com/ClickHouse/ClickHouse/pull/10631) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix clang-10 build. https://github.com/ClickHouse/ClickHouse/issues/10238. [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)). +* Fix clang-10 build. [#10238](https://github.com/ClickHouse/ClickHouse/issues/10238). [#10370](https://github.com/ClickHouse/ClickHouse/pull/10370) ([Amos Bird](https://github.com/amosbird)). * Added failing tests about `max_rows_to_sort` setting. [#10268](https://github.com/ClickHouse/ClickHouse/pull/10268) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added some improvements in printing diagnostic info in input formats. Fixes [#10204](https://github.com/ClickHouse/ClickHouse/issues/10204). [#10418](https://github.com/ClickHouse/ClickHouse/pull/10418) ([tavplubix](https://github.com/tavplubix)). * Added CA certificates to clickhouse-server docker image. [#10476](https://github.com/ClickHouse/ClickHouse/pull/10476) ([filimonov](https://github.com/filimonov)). #### Bug fix -* #10551. [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). +* Fix error `the BloomFilter false positive must be a double number between 0 and 1` [#10551](https://github.com/ClickHouse/ClickHouse/issues/10551). [#10569](https://github.com/ClickHouse/ClickHouse/pull/10569) ([Winter Zhang](https://github.com/zhang2014)). ### ClickHouse release v20.3.8.53, 2020-04-23 @@ -2424,7 +2424,7 @@ No changes compared to v20.4.3.16-stable. * Fixed the behaviour of `match` and `extract` functions when haystack has zero bytes. The behaviour was wrong when haystack was constant. This fixes [#9160](https://github.com/ClickHouse/ClickHouse/issues/9160) [#9163](https://github.com/ClickHouse/ClickHouse/pull/9163) ([alexey-milovidov](https://github.com/alexey-milovidov)) [#9345](https://github.com/ClickHouse/ClickHouse/pull/9345) ([alexey-milovidov](https://github.com/alexey-milovidov)) * Avoid throwing from destructor in Apache Avro 3rd-party library. [#9066](https://github.com/ClickHouse/ClickHouse/pull/9066) ([Andrew Onyshchuk](https://github.com/oandrew)) * Don't commit a batch polled from `Kafka` partially as it can lead to holes in data. [#8876](https://github.com/ClickHouse/ClickHouse/pull/8876) ([filimonov](https://github.com/filimonov)) -* Fix `joinGet` with nullable return types. https://github.com/ClickHouse/ClickHouse/issues/8919 [#9014](https://github.com/ClickHouse/ClickHouse/pull/9014) ([Amos Bird](https://github.com/amosbird)) +* Fix `joinGet` with nullable return types. [#8919](https://github.com/ClickHouse/ClickHouse/issues/8919) [#9014](https://github.com/ClickHouse/ClickHouse/pull/9014) ([Amos Bird](https://github.com/amosbird)) * Fix data incompatibility when compressed with `T64` codec. [#9016](https://github.com/ClickHouse/ClickHouse/pull/9016) ([Artem Zuikov](https://github.com/4ertus2)) Fix data type ids in `T64` compression codec that leads to wrong (de)compression in affected versions. [#9033](https://github.com/ClickHouse/ClickHouse/pull/9033) ([Artem Zuikov](https://github.com/4ertus2)) * Add setting `enable_early_constant_folding` and disable it in some cases that leads to errors. [#9010](https://github.com/ClickHouse/ClickHouse/pull/9010) ([Artem Zuikov](https://github.com/4ertus2)) * Fix pushdown predicate optimizer with VIEW and enable the test [#9011](https://github.com/ClickHouse/ClickHouse/pull/9011) ([Winter Zhang](https://github.com/zhang2014)) @@ -2626,7 +2626,7 @@ No changes compared to v20.4.3.16-stable. * Fix the error `Data compressed with different methods` that can happen if `min_bytes_to_use_direct_io` is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes [#11539](https://github.com/ClickHouse/ClickHouse/issues/11539). [#11540](https://github.com/ClickHouse/ClickHouse/pull/11540) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix return compressed size for codecs. [#11448](https://github.com/ClickHouse/ClickHouse/pull/11448) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix server crash when a column has compression codec with non-literal arguments. Fixes [#11365](https://github.com/ClickHouse/ClickHouse/issues/11365). [#11431](https://github.com/ClickHouse/ClickHouse/pull/11431) ([alesapin](https://github.com/alesapin)). -* Fix pointInPolygon with nan as point. Fixes https://github.com/ClickHouse/ClickHouse/issues/11375. [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). +* Fix pointInPolygon with nan as point. Fixes [#11375](https://github.com/ClickHouse/ClickHouse/issues/11375). [#11421](https://github.com/ClickHouse/ClickHouse/pull/11421) ([Alexey Ilyukhov](https://github.com/livace)). * Fixed geohashesInBox with arguments outside of latitude/longitude range. [#11403](https://github.com/ClickHouse/ClickHouse/pull/11403) ([Vasily Nemkov](https://github.com/Enmk)). * Fix possible `Pipeline stuck` error for queries with external sort and limit. Fixes [#11359](https://github.com/ClickHouse/ClickHouse/issues/11359). [#11366](https://github.com/ClickHouse/ClickHouse/pull/11366) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * Fix crash in `quantilesExactWeightedArray`. [#11337](https://github.com/ClickHouse/ClickHouse/pull/11337) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). @@ -2636,7 +2636,7 @@ No changes compared to v20.4.3.16-stable. * Fix potential uninitialized memory in conversion. Example: `SELECT toIntervalSecond(now64())`. [#11311](https://github.com/ClickHouse/ClickHouse/pull/11311) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with `empty` or `notEmpty` functions. This fixes [#11286](https://github.com/ClickHouse/ClickHouse/issues/11286). [#11303](https://github.com/ClickHouse/ClickHouse/pull/11303) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix bug when query speed estimation can be incorrect and the limit of `min_execution_speed` may not work or work incorrectly if the query is throttled by `max_network_bandwidth`, `max_execution_speed` or `priority` settings. Change the default value of `timeout_before_checking_execution_speed` to non-zero, because otherwise the settings `min_execution_speed` and `max_execution_speed` have no effect. This fixes [#11297](https://github.com/ClickHouse/ClickHouse/issues/11297). This fixes [#5732](https://github.com/ClickHouse/ClickHouse/issues/5732). This fixes [#6228](https://github.com/ClickHouse/ClickHouse/issues/6228). Usability improvement: avoid concatenation of exception message with progress bar in `clickhouse-client`. [#11296](https://github.com/ClickHouse/ClickHouse/pull/11296) ([alexey-milovidov](https://github.com/alexey-milovidov)). -* Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash while reading malformed data in Protobuf format. This fixes [#5957](https://github.com/ClickHouse/ClickHouse/issues/5957), fixes [#11203](https://github.com/ClickHouse/ClickHouse/issues/11203). [#11258](https://github.com/ClickHouse/ClickHouse/pull/11258) ([Vitaly Baranov](https://github.com/vitlibar)). * Fix possible error `Cannot capture column` for higher-order functions with `Array(Array(LowCardinality))` captured argument. [#11185](https://github.com/ClickHouse/ClickHouse/pull/11185) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). * If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. [#11162](https://github.com/ClickHouse/ClickHouse/pull/11162) ([Azat Khuzhin](https://github.com/azat)). * Remove logging from mutation finalization task if nothing was finalized. [#11109](https://github.com/ClickHouse/ClickHouse/pull/11109) ([alesapin](https://github.com/alesapin)). @@ -2914,7 +2914,7 @@ No changes compared to v20.4.3.16-stable. * Several improvements ClickHouse grammar in `.g4` file. [#8294](https://github.com/ClickHouse/ClickHouse/pull/8294) ([taiyang-li](https://github.com/taiyang-li)) * Fix bug that leads to crashes in `JOIN`s with tables with engine `Join`. This fixes [#7556](https://github.com/ClickHouse/ClickHouse/issues/7556) [#8254](https://github.com/ClickHouse/ClickHouse/issues/8254) [#7915](https://github.com/ClickHouse/ClickHouse/issues/7915) [#8100](https://github.com/ClickHouse/ClickHouse/issues/8100). [#8298](https://github.com/ClickHouse/ClickHouse/pull/8298) ([Artem Zuikov](https://github.com/4ertus2)) * Fix redundant dictionaries reload on `CREATE DATABASE`. [#7916](https://github.com/ClickHouse/ClickHouse/pull/7916) ([Azat Khuzhin](https://github.com/azat)) -* Limit maximum number of streams for read from `StorageFile` and `StorageHDFS`. Fixes https://github.com/ClickHouse/ClickHouse/issues/7650. [#7981](https://github.com/ClickHouse/ClickHouse/pull/7981) ([alesapin](https://github.com/alesapin)) +* Limit maximum number of streams for read from `StorageFile` and `StorageHDFS`. Fixes [#7650](https://github.com/ClickHouse/ClickHouse/issues/7650). [#7981](https://github.com/ClickHouse/ClickHouse/pull/7981) ([alesapin](https://github.com/alesapin)) * Fix bug in `ALTER ... MODIFY ... CODEC` query, when user specify both default expression and codec. Fixes [8593](https://github.com/ClickHouse/ClickHouse/issues/8593). [#8614](https://github.com/ClickHouse/ClickHouse/pull/8614) ([alesapin](https://github.com/alesapin)) * Fix error in background merge of columns with `SimpleAggregateFunction(LowCardinality)` type. [#8613](https://github.com/ClickHouse/ClickHouse/pull/8613) ([Nikolai Kochetov](https://github.com/KochetovNicolai)) * Fixed type check in function `toDateTime64`. [#8375](https://github.com/ClickHouse/ClickHouse/pull/8375) ([Vasily Nemkov](https://github.com/Enmk)) @@ -2998,7 +2998,7 @@ No changes compared to v20.4.3.16-stable. * Added check for extra parts of `MergeTree` at different disks, in order to not allow to miss data parts at undefined disks. [#8118](https://github.com/ClickHouse/ClickHouse/pull/8118) ([Vladimir Chebotarev](https://github.com/excitoon)) * Enable SSL support for Mac client and server. [#8297](https://github.com/ClickHouse/ClickHouse/pull/8297) ([Ivan](https://github.com/abyss7)) * Now ClickHouse can work as MySQL federated server (see https://dev.mysql.com/doc/refman/5.7/en/federated-create-server.html). [#7717](https://github.com/ClickHouse/ClickHouse/pull/7717) ([Maxim Fedotov](https://github.com/MaxFedotov)) -* `clickhouse-client` now only enable `bracketed-paste` when multiquery is on and multiline is off. This fixes (#7757)[https://github.com/ClickHouse/ClickHouse/issues/7757]. [#7761](https://github.com/ClickHouse/ClickHouse/pull/7761) ([Amos Bird](https://github.com/amosbird)) +* `clickhouse-client` now only enable `bracketed-paste` when multiquery is on and multiline is off. This fixes [#7757](https://github.com/ClickHouse/ClickHouse/issues/7757). [#7761](https://github.com/ClickHouse/ClickHouse/pull/7761) ([Amos Bird](https://github.com/amosbird)) * Support `Array(Decimal)` in `if` function. [#7721](https://github.com/ClickHouse/ClickHouse/pull/7721) ([Artem Zuikov](https://github.com/4ertus2)) * Support Decimals in `arrayDifference`, `arrayCumSum` and `arrayCumSumNegative` functions. [#7724](https://github.com/ClickHouse/ClickHouse/pull/7724) ([Artem Zuikov](https://github.com/4ertus2)) * Added `lifetime` column to `system.dictionaries` table. [#6820](https://github.com/ClickHouse/ClickHouse/issues/6820) [#7727](https://github.com/ClickHouse/ClickHouse/pull/7727) ([kekekekule](https://github.com/kekekekule)) diff --git a/CMakeLists.txt b/CMakeLists.txt index e3ddfb0c7ad..367af2bbf92 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -223,8 +223,8 @@ if (ARCH_NATIVE) set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native") endif () -if (UNBUNDLED AND (COMPILER_GCC OR COMPILER_CLANG)) - # to make numeric_limits<__int128> works for unbundled build +if (COMPILER_GCC OR COMPILER_CLANG) + # to make numeric_limits<__int128> works with GCC set (_CXX_STANDARD "-std=gnu++2a") else() set (_CXX_STANDARD "-std=c++2a") @@ -457,6 +457,7 @@ include (cmake/find/s3.cmake) include (cmake/find/base64.cmake) include (cmake/find/parquet.cmake) include (cmake/find/simdjson.cmake) +include (cmake/find/fast_float.cmake) include (cmake/find/rapidjson.cmake) include (cmake/find/fastops.cmake) include (cmake/find/odbc.cmake) diff --git a/base/common/wide_integer.h b/base/common/wide_integer.h index 61d88bdcaf3..c9d1eaa32aa 100644 --- a/base/common/wide_integer.h +++ b/base/common/wide_integer.h @@ -58,8 +58,7 @@ public: using signed_base_type = int64_t; // ctors - integer() = default; - + constexpr integer() noexcept; template constexpr integer(T rhs) noexcept; template diff --git a/base/common/wide_integer_impl.h b/base/common/wide_integer_impl.h index 2a889819c11..9a80660ea85 100644 --- a/base/common/wide_integer_impl.h +++ b/base/common/wide_integer_impl.h @@ -916,6 +916,11 @@ public: // Members +template +constexpr integer::integer() noexcept + : items{} +{} + template template constexpr integer::integer(T rhs) noexcept diff --git a/base/daemon/BaseDaemon.cpp b/base/daemon/BaseDaemon.cpp index 22455d09cf2..331f9da56dd 100644 --- a/base/daemon/BaseDaemon.cpp +++ b/base/daemon/BaseDaemon.cpp @@ -761,14 +761,14 @@ void BaseDaemon::initializeTerminationAndSignalProcessing() static KillingErrorHandler killing_error_handler; Poco::ErrorHandler::set(&killing_error_handler); - signal_pipe.setNonBlocking(); + signal_pipe.setNonBlockingWrite(); signal_pipe.tryIncreaseSize(1 << 20); signal_listener = std::make_unique(*this); signal_listener_thread.start(*signal_listener); #if defined(__ELF__) && !defined(__FreeBSD__) - String build_id_hex = DB::SymbolIndex::instance().getBuildIDHex(); + String build_id_hex = DB::SymbolIndex::instance()->getBuildIDHex(); if (build_id_hex.empty()) build_id_info = "no build id"; else diff --git a/base/daemon/SentryWriter.cpp b/base/daemon/SentryWriter.cpp index b8f2e5073ab..29430b65983 100644 --- a/base/daemon/SentryWriter.cpp +++ b/base/daemon/SentryWriter.cpp @@ -179,7 +179,7 @@ void SentryWriter::onFault(int sig, const std::string & error_message, const Sta sentry_set_extra("signal_number", sentry_value_new_int32(sig)); #if defined(__ELF__) && !defined(__FreeBSD__) - const String & build_id_hex = DB::SymbolIndex::instance().getBuildIDHex(); + const String & build_id_hex = DB::SymbolIndex::instance()->getBuildIDHex(); sentry_set_tag("build_id", build_id_hex.c_str()); #endif diff --git a/base/mysqlxx/Pool.cpp b/base/mysqlxx/Pool.cpp index 2058429d3da..2cb3e62db84 100644 --- a/base/mysqlxx/Pool.cpp +++ b/base/mysqlxx/Pool.cpp @@ -248,7 +248,7 @@ bool Pool::Entry::tryForceConnected() const if (prev_connection_id != current_connection_id) { auto & logger = Poco::Util::Application::instance().logger(); - logger.information("Connection to mysql server has been reestablished. Connection id changed: %d -> %d", + logger.information("Connection to mysql server has been reestablished. Connection id changed: %lu -> %lu", prev_connection_id, current_connection_id); } return true; diff --git a/base/mysqlxx/ResultBase.cpp b/base/mysqlxx/ResultBase.cpp index eac1e22ca3d..2461ad7c9ce 100644 --- a/base/mysqlxx/ResultBase.cpp +++ b/base/mysqlxx/ResultBase.cpp @@ -22,4 +22,12 @@ ResultBase::~ResultBase() mysql_free_result(res); } +std::string ResultBase::getFieldName(size_t n) const +{ + if (num_fields <= n) + throw Exception(std::string("Unknown column position ") + std::to_string(n)); + + return fields[n].name; +} + } diff --git a/base/mysqlxx/ResultBase.h b/base/mysqlxx/ResultBase.h index b72b5682122..4f2ab2eb0a2 100644 --- a/base/mysqlxx/ResultBase.h +++ b/base/mysqlxx/ResultBase.h @@ -31,6 +31,8 @@ public: MYSQL_RES * getRes() { return res; } const Query * getQuery() const { return query; } + std::string getFieldName(size_t n) const; + virtual ~ResultBase(); protected: diff --git a/cmake/find/fast_float.cmake b/cmake/find/fast_float.cmake new file mode 100644 index 00000000000..4b215c710ad --- /dev/null +++ b/cmake/find/fast_float.cmake @@ -0,0 +1,6 @@ +if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/fast_float/include/fast_float/fast_float.h") + message (FATAL_ERROR "submodule contrib/fast_float is missing. to fix try run: \n git submodule update --init --recursive") +endif () + +set(FAST_FLOAT_LIBRARY fast_float) +set(FAST_FLOAT_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/fast_float/include/") diff --git a/contrib/CMakeLists.txt b/contrib/CMakeLists.txt index a674d542c78..48d760e7540 100644 --- a/contrib/CMakeLists.txt +++ b/contrib/CMakeLists.txt @@ -26,6 +26,7 @@ add_subdirectory (boost-cmake) add_subdirectory (cctz-cmake) add_subdirectory (consistent-hashing-sumbur) add_subdirectory (consistent-hashing) +add_subdirectory (dragonbox-cmake) add_subdirectory (FastMemcpy) add_subdirectory (hyperscan-cmake) add_subdirectory (jemalloc-cmake) @@ -322,4 +323,5 @@ if (USE_INTERNAL_ROCKSDB_LIBRARY) add_subdirectory(rocksdb-cmake) endif() -add_subdirectory(dragonbox) +add_subdirectory(fast_float) + diff --git a/contrib/boost b/contrib/boost index a04e72c0464..a7ceabe4747 160000 --- a/contrib/boost +++ b/contrib/boost @@ -1 +1 @@ -Subproject commit a04e72c0464f0c31d3384f18f0c0db36a05538e0 +Subproject commit a7ceabe4747ecc3309dd3dcd9de4b29660dfd298 diff --git a/contrib/dragonbox-cmake/CMakeLists.txt b/contrib/dragonbox-cmake/CMakeLists.txt new file mode 100644 index 00000000000..604394c6dce --- /dev/null +++ b/contrib/dragonbox-cmake/CMakeLists.txt @@ -0,0 +1,5 @@ +set(LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/dragonbox") + +add_library(dragonbox_to_chars "${LIBRARY_DIR}/source/dragonbox_to_chars.cpp") + +target_include_directories(dragonbox_to_chars SYSTEM BEFORE PUBLIC "${LIBRARY_DIR}/include/") diff --git a/contrib/fast_float b/contrib/fast_float new file mode 160000 index 00000000000..7eae925b51f --- /dev/null +++ b/contrib/fast_float @@ -0,0 +1 @@ +Subproject commit 7eae925b51fd0f570ccd5c880c12e3e27a23b86f diff --git a/contrib/poco b/contrib/poco index b5523bb9b4b..08974cc024b 160000 --- a/contrib/poco +++ b/contrib/poco @@ -1 +1 @@ -Subproject commit b5523bb9b4bc4239640cbfec4d734be8b8585639 +Subproject commit 08974cc024b2e748f5b1d45415396706b3521d0f diff --git a/docker/test/fasttest/run.sh b/docker/test/fasttest/run.sh index fb905bdaa5d..a918cc44420 100755 --- a/docker/test/fasttest/run.sh +++ b/docker/test/fasttest/run.sh @@ -64,7 +64,14 @@ function stop_server function start_server { set -m # Spawn server in its own process groups - clickhouse-server --config-file="$FASTTEST_DATA/config.xml" -- --path "$FASTTEST_DATA" --user_files_path "$FASTTEST_DATA/user_files" &>> "$FASTTEST_OUTPUT/server.log" & + local opts=( + --config-file="$FASTTEST_DATA/config.xml" + -- + --path "$FASTTEST_DATA" + --user_files_path "$FASTTEST_DATA/user_files" + --top_level_domains_path "$FASTTEST_DATA/top_level_domains" + ) + clickhouse-server "${opts[@]}" &>> "$FASTTEST_OUTPUT/server.log" & server_pid=$! set +m @@ -155,6 +162,7 @@ function clone_submodules contrib/miniselect contrib/xz contrib/dragonbox + contrib/fast_float ) git submodule sync @@ -318,6 +326,9 @@ function run_tests 01545_system_errors # Checks system.errors 01563_distributed_query_finish + + # nc - command not found + 01601_proxy_protocol ) time clickhouse-test -j 8 --order=random --no-long --testname --shard --zookeeper --skip "${TESTS_TO_SKIP[@]}" -- "$FASTTEST_FOCUS" 2>&1 | ts '%Y-%m-%d %H:%M:%S' | tee "$FASTTEST_OUTPUT/test_log.txt" diff --git a/docker/test/performance-comparison/Dockerfile b/docker/test/performance-comparison/Dockerfile index 8734e47e80f..5ec048de657 100644 --- a/docker/test/performance-comparison/Dockerfile +++ b/docker/test/performance-comparison/Dockerfile @@ -53,4 +53,3 @@ COPY * / CMD ["bash", "-c", "node=$((RANDOM % $(numactl --hardware | sed -n 's/^.*available:\\(.*\\)nodes.*$/\\1/p'))); echo Will bind to NUMA node $node; numactl --cpunodebind=$node --membind=$node /entrypoint.sh"] # docker run --network=host --volume :/workspace --volume=:/output -e PR_TO_TEST=<> -e SHA_TO_TEST=<> yandex/clickhouse-performance-comparison - diff --git a/docker/test/performance-comparison/compare.sh b/docker/test/performance-comparison/compare.sh index 92f134011ab..59d7cc98063 100755 --- a/docker/test/performance-comparison/compare.sh +++ b/docker/test/performance-comparison/compare.sh @@ -7,6 +7,11 @@ trap 'kill $(jobs -pr) ||:' EXIT stage=${stage:-} script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" +# upstream/master +LEFT_SERVER_PORT=9001 +# patched version +RIGHT_SERVER_PORT=9002 + function wait_for_server # port, pid { for _ in {1..60} @@ -37,25 +42,33 @@ function configure rm right/config/config.d/text_log.xml ||: cp -rv right/config left ||: - sed -i 's/900./9001/g' left/config/config.xml - sed -i 's/900./9002/g' right/config/config.xml - # Start a temporary server to rename the tables while killall clickhouse-server; do echo . ; sleep 1 ; done echo all killed set -m # Spawn temporary in its own process groups - left/clickhouse-server --config-file=left/config/config.xml -- --path db0 --user_files_path db0/user_files &> setup-server-log.log & + + local setup_left_server_opts=( + # server options + --config-file=left/config/config.xml + -- + # server *config* directives overrides + --path db0 + --user_files_path db0/user_files + --top_level_domains_path /top_level_domains + --tcp_port $LEFT_SERVER_PORT + ) + left/clickhouse-server "${setup_left_server_opts[@]}" &> setup-server-log.log & left_pid=$! kill -0 $left_pid disown $left_pid set +m - wait_for_server 9001 $left_pid + wait_for_server $LEFT_SERVER_PORT $left_pid echo Server for setup started - clickhouse-client --port 9001 --query "create database test" ||: - clickhouse-client --port 9001 --query "rename table datasets.hits_v1 to test.hits" ||: + clickhouse-client --port $LEFT_SERVER_PORT --query "create database test" ||: + clickhouse-client --port $LEFT_SERVER_PORT --query "rename table datasets.hits_v1 to test.hits" ||: while killall clickhouse-server; do echo . ; sleep 1 ; done echo all killed @@ -83,16 +96,32 @@ function restart set -m # Spawn servers in their own process groups - left/clickhouse-server --config-file=left/config/config.xml \ - -- --path left/db --user_files_path left/db/user_files \ - &>> left-server-log.log & + local left_server_opts=( + # server options + --config-file=left/config/config.xml + -- + # server *config* directives overrides + --path left/db + --user_files_path left/db/user_files + --top_level_domains_path /top_level_domains + --tcp_port $LEFT_SERVER_PORT + ) + left/clickhouse-server "${left_server_opts[@]}" &>> left-server-log.log & left_pid=$! kill -0 $left_pid disown $left_pid - right/clickhouse-server --config-file=right/config/config.xml \ - -- --path right/db --user_files_path right/db/user_files \ - &>> right-server-log.log & + local right_server_opts=( + # server options + --config-file=right/config/config.xml + -- + # server *config* directives overrides + --path right/db + --user_files_path right/db/user_files + --top_level_domains_path /top_level_domains + --tcp_port $RIGHT_SERVER_PORT + ) + right/clickhouse-server "${right_server_opts[@]}" &>> right-server-log.log & right_pid=$! kill -0 $right_pid disown $right_pid @@ -101,16 +130,16 @@ function restart unset MALLOC_CONF - wait_for_server 9001 $left_pid + wait_for_server $LEFT_SERVER_PORT $left_pid echo left ok - wait_for_server 9002 $right_pid + wait_for_server $RIGHT_SERVER_PORT $right_pid echo right ok - clickhouse-client --port 9001 --query "select * from system.tables where database != 'system'" - clickhouse-client --port 9001 --query "select * from system.build_options" - clickhouse-client --port 9002 --query "select * from system.tables where database != 'system'" - clickhouse-client --port 9002 --query "select * from system.build_options" + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.tables where database != 'system'" + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.build_options" + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.tables where database != 'system'" + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.build_options" # Check again that both servers we started are running -- this is important # for running locally, when there might be some other servers started and we @@ -199,9 +228,9 @@ function run_tests for test in $test_files do # Check that both servers are alive, and restart them if they die. - clickhouse-client --port 9001 --query "select 1 format Null" \ + clickhouse-client --port $LEFT_SERVER_PORT --query "select 1 format Null" \ || { echo $test_name >> left-server-died.log ; restart ; } - clickhouse-client --port 9002 --query "select 1 format Null" \ + clickhouse-client --port $RIGHT_SERVER_PORT --query "select 1 format Null" \ || { echo $test_name >> right-server-died.log ; restart ; } test_name=$(basename "$test" ".xml") @@ -215,7 +244,7 @@ function run_tests # The grep is to filter out set -x output and keep only time output. # The '2>&1 >/dev/null' redirects stderr to stdout, and discards stdout. { \ - time "$script_dir/perf.py" --host localhost localhost --port 9001 9002 \ + time "$script_dir/perf.py" --host localhost localhost --port $LEFT_SERVER_PORT $RIGHT_SERVER_PORT \ --runs "$CHPC_RUNS" --max-queries "$CHPC_MAX_QUERIES" \ --profile-seconds "$profile_seconds" \ -- "$test" > "$test_name-raw.tsv" 2> "$test_name-err.log" ; \ @@ -257,36 +286,36 @@ function get_profiles_watchdog function get_profiles { # Collect the profiles - clickhouse-client --port 9001 --query "set query_profiler_cpu_time_period_ns = 0" - clickhouse-client --port 9001 --query "set query_profiler_real_time_period_ns = 0" - clickhouse-client --port 9001 --query "system flush logs" & + clickhouse-client --port $LEFT_SERVER_PORT --query "set query_profiler_cpu_time_period_ns = 0" + clickhouse-client --port $LEFT_SERVER_PORT --query "set query_profiler_real_time_period_ns = 0" + clickhouse-client --port $LEFT_SERVER_PORT --query "system flush logs" & - clickhouse-client --port 9002 --query "set query_profiler_cpu_time_period_ns = 0" - clickhouse-client --port 9002 --query "set query_profiler_real_time_period_ns = 0" - clickhouse-client --port 9002 --query "system flush logs" & + clickhouse-client --port $RIGHT_SERVER_PORT --query "set query_profiler_cpu_time_period_ns = 0" + clickhouse-client --port $RIGHT_SERVER_PORT --query "set query_profiler_real_time_period_ns = 0" + clickhouse-client --port $RIGHT_SERVER_PORT --query "system flush logs" & wait - clickhouse-client --port 9001 --query "select * from system.query_log where type = 2 format TSVWithNamesAndTypes" > left-query-log.tsv ||: & - clickhouse-client --port 9001 --query "select * from system.query_thread_log format TSVWithNamesAndTypes" > left-query-thread-log.tsv ||: & - clickhouse-client --port 9001 --query "select * from system.trace_log format TSVWithNamesAndTypes" > left-trace-log.tsv ||: & - clickhouse-client --port 9001 --query "select arrayJoin(trace) addr, concat(splitByChar('/', addressToLine(addr))[-1], '#', demangle(addressToSymbol(addr)) ) name from system.trace_log group by addr format TSVWithNamesAndTypes" > left-addresses.tsv ||: & - clickhouse-client --port 9001 --query "select * from system.metric_log format TSVWithNamesAndTypes" > left-metric-log.tsv ||: & - clickhouse-client --port 9001 --query "select * from system.asynchronous_metric_log format TSVWithNamesAndTypes" > left-async-metric-log.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.query_log where type = 2 format TSVWithNamesAndTypes" > left-query-log.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.query_thread_log format TSVWithNamesAndTypes" > left-query-thread-log.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.trace_log format TSVWithNamesAndTypes" > left-trace-log.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select arrayJoin(trace) addr, concat(splitByChar('/', addressToLine(addr))[-1], '#', demangle(addressToSymbol(addr)) ) name from system.trace_log group by addr format TSVWithNamesAndTypes" > left-addresses.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.metric_log format TSVWithNamesAndTypes" > left-metric-log.tsv ||: & + clickhouse-client --port $LEFT_SERVER_PORT --query "select * from system.asynchronous_metric_log format TSVWithNamesAndTypes" > left-async-metric-log.tsv ||: & - clickhouse-client --port 9002 --query "select * from system.query_log where type = 2 format TSVWithNamesAndTypes" > right-query-log.tsv ||: & - clickhouse-client --port 9002 --query "select * from system.query_thread_log format TSVWithNamesAndTypes" > right-query-thread-log.tsv ||: & - clickhouse-client --port 9002 --query "select * from system.trace_log format TSVWithNamesAndTypes" > right-trace-log.tsv ||: & - clickhouse-client --port 9002 --query "select arrayJoin(trace) addr, concat(splitByChar('/', addressToLine(addr))[-1], '#', demangle(addressToSymbol(addr)) ) name from system.trace_log group by addr format TSVWithNamesAndTypes" > right-addresses.tsv ||: & - clickhouse-client --port 9002 --query "select * from system.metric_log format TSVWithNamesAndTypes" > right-metric-log.tsv ||: & - clickhouse-client --port 9002 --query "select * from system.asynchronous_metric_log format TSVWithNamesAndTypes" > right-async-metric-log.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.query_log where type = 2 format TSVWithNamesAndTypes" > right-query-log.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.query_thread_log format TSVWithNamesAndTypes" > right-query-thread-log.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.trace_log format TSVWithNamesAndTypes" > right-trace-log.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select arrayJoin(trace) addr, concat(splitByChar('/', addressToLine(addr))[-1], '#', demangle(addressToSymbol(addr)) ) name from system.trace_log group by addr format TSVWithNamesAndTypes" > right-addresses.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.metric_log format TSVWithNamesAndTypes" > right-metric-log.tsv ||: & + clickhouse-client --port $RIGHT_SERVER_PORT --query "select * from system.asynchronous_metric_log format TSVWithNamesAndTypes" > right-async-metric-log.tsv ||: & wait # Just check that the servers are alive so that we return a proper exit code. # We don't consistently check the return codes of the above background jobs. - clickhouse-client --port 9001 --query "select 1" - clickhouse-client --port 9002 --query "select 1" + clickhouse-client --port $LEFT_SERVER_PORT --query "select 1" + clickhouse-client --port $RIGHT_SERVER_PORT --query "select 1" } function build_log_column_definitions diff --git a/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml b/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml new file mode 100644 index 00000000000..7b5e6a5638a --- /dev/null +++ b/docker/test/performance-comparison/config/config.d/top_level_domains_lists.xml @@ -0,0 +1,5 @@ + + + public_suffix_list.dat + + diff --git a/docker/test/performance-comparison/config/config.d/perf-comparison-tweaks-config.xml b/docker/test/performance-comparison/config/config.d/zzz-perf-comparison-tweaks-config.xml similarity index 94% rename from docker/test/performance-comparison/config/config.d/perf-comparison-tweaks-config.xml rename to docker/test/performance-comparison/config/config.d/zzz-perf-comparison-tweaks-config.xml index bc7ddf1fbbb..81dab1a48b0 100644 --- a/docker/test/performance-comparison/config/config.d/perf-comparison-tweaks-config.xml +++ b/docker/test/performance-comparison/config/config.d/zzz-perf-comparison-tweaks-config.xml @@ -2,6 +2,7 @@ + :: diff --git a/docker/test/performance-comparison/config/top_level_domains/public_suffix_list.dat b/docker/test/performance-comparison/config/top_level_domains/public_suffix_list.dat new file mode 100644 index 00000000000..1ede2b929a0 --- /dev/null +++ b/docker/test/performance-comparison/config/top_level_domains/public_suffix_list.dat @@ -0,0 +1,13491 @@ +// This Source Code Form is subject to the terms of the Mozilla Public +// License, v. 2.0. If a copy of the MPL was not distributed with this +// file, You can obtain one at https://mozilla.org/MPL/2.0/. + +// Please pull this list from, and only from https://publicsuffix.org/list/public_suffix_list.dat, +// rather than any other VCS sites. Pulling from any other URL is not guaranteed to be supported. + +// Instructions on pulling and using this list can be found at https://publicsuffix.org/list/. + +// ===BEGIN ICANN DOMAINS=== + +// ac : https://en.wikipedia.org/wiki/.ac +ac +com.ac +edu.ac +gov.ac +net.ac +mil.ac +org.ac + +// ad : https://en.wikipedia.org/wiki/.ad +ad +nom.ad + +// ae : https://en.wikipedia.org/wiki/.ae +// see also: "Domain Name Eligibility Policy" at http://www.aeda.ae/eng/aepolicy.php +ae +co.ae +net.ae +org.ae +sch.ae +ac.ae +gov.ae +mil.ae + +// aero : see https://www.information.aero/index.php?id=66 +aero +accident-investigation.aero +accident-prevention.aero +aerobatic.aero +aeroclub.aero +aerodrome.aero +agents.aero +aircraft.aero +airline.aero +airport.aero +air-surveillance.aero +airtraffic.aero +air-traffic-control.aero +ambulance.aero +amusement.aero +association.aero +author.aero +ballooning.aero +broker.aero +caa.aero +cargo.aero +catering.aero +certification.aero +championship.aero +charter.aero +civilaviation.aero +club.aero +conference.aero +consultant.aero +consulting.aero +control.aero +council.aero +crew.aero +design.aero +dgca.aero +educator.aero +emergency.aero +engine.aero +engineer.aero +entertainment.aero +equipment.aero +exchange.aero +express.aero +federation.aero +flight.aero +fuel.aero +gliding.aero +government.aero +groundhandling.aero +group.aero +hanggliding.aero +homebuilt.aero +insurance.aero +journal.aero +journalist.aero +leasing.aero +logistics.aero +magazine.aero +maintenance.aero +media.aero +microlight.aero +modelling.aero +navigation.aero +parachuting.aero +paragliding.aero +passenger-association.aero +pilot.aero +press.aero +production.aero +recreation.aero +repbody.aero +res.aero +research.aero +rotorcraft.aero +safety.aero +scientist.aero +services.aero +show.aero +skydiving.aero +software.aero +student.aero +trader.aero +trading.aero +trainer.aero +union.aero +workinggroup.aero +works.aero + +// af : http://www.nic.af/help.jsp +af +gov.af +com.af +org.af +net.af +edu.af + +// ag : http://www.nic.ag/prices.htm +ag +com.ag +org.ag +net.ag +co.ag +nom.ag + +// ai : http://nic.com.ai/ +ai +off.ai +com.ai +net.ai +org.ai + +// al : http://www.ert.gov.al/ert_alb/faq_det.html?Id=31 +al +com.al +edu.al +gov.al +mil.al +net.al +org.al + +// am : https://www.amnic.net/policy/en/Policy_EN.pdf +am +co.am +com.am +commune.am +net.am +org.am + +// ao : https://en.wikipedia.org/wiki/.ao +// http://www.dns.ao/REGISTR.DOC +ao +ed.ao +gv.ao +og.ao +co.ao +pb.ao +it.ao + +// aq : https://en.wikipedia.org/wiki/.aq +aq + +// ar : https://nic.ar/nic-argentina/normativa-vigente +ar +com.ar +edu.ar +gob.ar +gov.ar +int.ar +mil.ar +musica.ar +net.ar +org.ar +tur.ar + +// arpa : https://en.wikipedia.org/wiki/.arpa +// Confirmed by registry 2008-06-18 +arpa +e164.arpa +in-addr.arpa +ip6.arpa +iris.arpa +uri.arpa +urn.arpa + +// as : https://en.wikipedia.org/wiki/.as +as +gov.as + +// asia : https://en.wikipedia.org/wiki/.asia +asia + +// at : https://en.wikipedia.org/wiki/.at +// Confirmed by registry 2008-06-17 +at +ac.at +co.at +gv.at +or.at +sth.ac.at + +// au : https://en.wikipedia.org/wiki/.au +// http://www.auda.org.au/ +au +// 2LDs +com.au +net.au +org.au +edu.au +gov.au +asn.au +id.au +// Historic 2LDs (closed to new registration, but sites still exist) +info.au +conf.au +oz.au +// CGDNs - http://www.cgdn.org.au/ +act.au +nsw.au +nt.au +qld.au +sa.au +tas.au +vic.au +wa.au +// 3LDs +act.edu.au +catholic.edu.au +// eq.edu.au - Removed at the request of the Queensland Department of Education +nsw.edu.au +nt.edu.au +qld.edu.au +sa.edu.au +tas.edu.au +vic.edu.au +wa.edu.au +// act.gov.au Bug 984824 - Removed at request of Greg Tankard +// nsw.gov.au Bug 547985 - Removed at request of +// nt.gov.au Bug 940478 - Removed at request of Greg Connors +qld.gov.au +sa.gov.au +tas.gov.au +vic.gov.au +wa.gov.au +// 4LDs +// education.tas.edu.au - Removed at the request of the Department of Education Tasmania +schools.nsw.edu.au + +// aw : https://en.wikipedia.org/wiki/.aw +aw +com.aw + +// ax : https://en.wikipedia.org/wiki/.ax +ax + +// az : https://en.wikipedia.org/wiki/.az +az +com.az +net.az +int.az +gov.az +org.az +edu.az +info.az +pp.az +mil.az +name.az +pro.az +biz.az + +// ba : http://nic.ba/users_data/files/pravilnik_o_registraciji.pdf +ba +com.ba +edu.ba +gov.ba +mil.ba +net.ba +org.ba + +// bb : https://en.wikipedia.org/wiki/.bb +bb +biz.bb +co.bb +com.bb +edu.bb +gov.bb +info.bb +net.bb +org.bb +store.bb +tv.bb + +// bd : https://en.wikipedia.org/wiki/.bd +*.bd + +// be : https://en.wikipedia.org/wiki/.be +// Confirmed by registry 2008-06-08 +be +ac.be + +// bf : https://en.wikipedia.org/wiki/.bf +bf +gov.bf + +// bg : https://en.wikipedia.org/wiki/.bg +// https://www.register.bg/user/static/rules/en/index.html +bg +a.bg +b.bg +c.bg +d.bg +e.bg +f.bg +g.bg +h.bg +i.bg +j.bg +k.bg +l.bg +m.bg +n.bg +o.bg +p.bg +q.bg +r.bg +s.bg +t.bg +u.bg +v.bg +w.bg +x.bg +y.bg +z.bg +0.bg +1.bg +2.bg +3.bg +4.bg +5.bg +6.bg +7.bg +8.bg +9.bg + +// bh : https://en.wikipedia.org/wiki/.bh +bh +com.bh +edu.bh +net.bh +org.bh +gov.bh + +// bi : https://en.wikipedia.org/wiki/.bi +// http://whois.nic.bi/ +bi +co.bi +com.bi +edu.bi +or.bi +org.bi + +// biz : https://en.wikipedia.org/wiki/.biz +biz + +// bj : https://en.wikipedia.org/wiki/.bj +bj +asso.bj +barreau.bj +gouv.bj + +// bm : http://www.bermudanic.bm/dnr-text.txt +bm +com.bm +edu.bm +gov.bm +net.bm +org.bm + +// bn : http://www.bnnic.bn/faqs +bn +com.bn +edu.bn +gov.bn +net.bn +org.bn + +// bo : https://nic.bo/delegacion2015.php#h-1.10 +bo +com.bo +edu.bo +gob.bo +int.bo +org.bo +net.bo +mil.bo +tv.bo +web.bo +// Social Domains +academia.bo +agro.bo +arte.bo +blog.bo +bolivia.bo +ciencia.bo +cooperativa.bo +democracia.bo +deporte.bo +ecologia.bo +economia.bo +empresa.bo +indigena.bo +industria.bo +info.bo +medicina.bo +movimiento.bo +musica.bo +natural.bo +nombre.bo +noticias.bo +patria.bo +politica.bo +profesional.bo +plurinacional.bo +pueblo.bo +revista.bo +salud.bo +tecnologia.bo +tksat.bo +transporte.bo +wiki.bo + +// br : http://registro.br/dominio/categoria.html +// Submitted by registry +br +9guacu.br +abc.br +adm.br +adv.br +agr.br +aju.br +am.br +anani.br +aparecida.br +app.br +arq.br +art.br +ato.br +b.br +barueri.br +belem.br +bhz.br +bib.br +bio.br +blog.br +bmd.br +boavista.br +bsb.br +campinagrande.br +campinas.br +caxias.br +cim.br +cng.br +cnt.br +com.br +contagem.br +coop.br +coz.br +cri.br +cuiaba.br +curitiba.br +def.br +des.br +det.br +dev.br +ecn.br +eco.br +edu.br +emp.br +enf.br +eng.br +esp.br +etc.br +eti.br +far.br +feira.br +flog.br +floripa.br +fm.br +fnd.br +fortal.br +fot.br +foz.br +fst.br +g12.br +geo.br +ggf.br +goiania.br +gov.br +// gov.br 26 states + df https://en.wikipedia.org/wiki/States_of_Brazil +ac.gov.br +al.gov.br +am.gov.br +ap.gov.br +ba.gov.br +ce.gov.br +df.gov.br +es.gov.br +go.gov.br +ma.gov.br +mg.gov.br +ms.gov.br +mt.gov.br +pa.gov.br +pb.gov.br +pe.gov.br +pi.gov.br +pr.gov.br +rj.gov.br +rn.gov.br +ro.gov.br +rr.gov.br +rs.gov.br +sc.gov.br +se.gov.br +sp.gov.br +to.gov.br +gru.br +imb.br +ind.br +inf.br +jab.br +jampa.br +jdf.br +joinville.br +jor.br +jus.br +leg.br +lel.br +log.br +londrina.br +macapa.br +maceio.br +manaus.br +maringa.br +mat.br +med.br +mil.br +morena.br +mp.br +mus.br +natal.br +net.br +niteroi.br +*.nom.br +not.br +ntr.br +odo.br +ong.br +org.br +osasco.br +palmas.br +poa.br +ppg.br +pro.br +psc.br +psi.br +pvh.br +qsl.br +radio.br +rec.br +recife.br +rep.br +ribeirao.br +rio.br +riobranco.br +riopreto.br +salvador.br +sampa.br +santamaria.br +santoandre.br +saobernardo.br +saogonca.br +seg.br +sjc.br +slg.br +slz.br +sorocaba.br +srv.br +taxi.br +tc.br +tec.br +teo.br +the.br +tmp.br +trd.br +tur.br +tv.br +udi.br +vet.br +vix.br +vlog.br +wiki.br +zlg.br + +// bs : http://www.nic.bs/rules.html +bs +com.bs +net.bs +org.bs +edu.bs +gov.bs + +// bt : https://en.wikipedia.org/wiki/.bt +bt +com.bt +edu.bt +gov.bt +net.bt +org.bt + +// bv : No registrations at this time. +// Submitted by registry +bv + +// bw : https://en.wikipedia.org/wiki/.bw +// http://www.gobin.info/domainname/bw.doc +// list of other 2nd level tlds ? +bw +co.bw +org.bw + +// by : https://en.wikipedia.org/wiki/.by +// http://tld.by/rules_2006_en.html +// list of other 2nd level tlds ? +by +gov.by +mil.by +// Official information does not indicate that com.by is a reserved +// second-level domain, but it's being used as one (see www.google.com.by and +// www.yahoo.com.by, for example), so we list it here for safety's sake. +com.by + +// http://hoster.by/ +of.by + +// bz : https://en.wikipedia.org/wiki/.bz +// http://www.belizenic.bz/ +bz +com.bz +net.bz +org.bz +edu.bz +gov.bz + +// ca : https://en.wikipedia.org/wiki/.ca +ca +// ca geographical names +ab.ca +bc.ca +mb.ca +nb.ca +nf.ca +nl.ca +ns.ca +nt.ca +nu.ca +on.ca +pe.ca +qc.ca +sk.ca +yk.ca +// gc.ca: https://en.wikipedia.org/wiki/.gc.ca +// see also: http://registry.gc.ca/en/SubdomainFAQ +gc.ca + +// cat : https://en.wikipedia.org/wiki/.cat +cat + +// cc : https://en.wikipedia.org/wiki/.cc +cc + +// cd : https://en.wikipedia.org/wiki/.cd +// see also: https://www.nic.cd/domain/insertDomain_2.jsp?act=1 +cd +gov.cd + +// cf : https://en.wikipedia.org/wiki/.cf +cf + +// cg : https://en.wikipedia.org/wiki/.cg +cg + +// ch : https://en.wikipedia.org/wiki/.ch +ch + +// ci : https://en.wikipedia.org/wiki/.ci +// http://www.nic.ci/index.php?page=charte +ci +org.ci +or.ci +com.ci +co.ci +edu.ci +ed.ci +ac.ci +net.ci +go.ci +asso.ci +aéroport.ci +int.ci +presse.ci +md.ci +gouv.ci + +// ck : https://en.wikipedia.org/wiki/.ck +*.ck +!www.ck + +// cl : https://www.nic.cl +// Confirmed by .CL registry +cl +aprendemas.cl +co.cl +gob.cl +gov.cl +mil.cl + +// cm : https://en.wikipedia.org/wiki/.cm plus bug 981927 +cm +co.cm +com.cm +gov.cm +net.cm + +// cn : https://en.wikipedia.org/wiki/.cn +// Submitted by registry +cn +ac.cn +com.cn +edu.cn +gov.cn +net.cn +org.cn +mil.cn +公司.cn +网络.cn +網絡.cn +// cn geographic names +ah.cn +bj.cn +cq.cn +fj.cn +gd.cn +gs.cn +gz.cn +gx.cn +ha.cn +hb.cn +he.cn +hi.cn +hl.cn +hn.cn +jl.cn +js.cn +jx.cn +ln.cn +nm.cn +nx.cn +qh.cn +sc.cn +sd.cn +sh.cn +sn.cn +sx.cn +tj.cn +xj.cn +xz.cn +yn.cn +zj.cn +hk.cn +mo.cn +tw.cn + +// co : https://en.wikipedia.org/wiki/.co +// Submitted by registry +co +arts.co +com.co +edu.co +firm.co +gov.co +info.co +int.co +mil.co +net.co +nom.co +org.co +rec.co +web.co + +// com : https://en.wikipedia.org/wiki/.com +com + +// coop : https://en.wikipedia.org/wiki/.coop +coop + +// cr : http://www.nic.cr/niccr_publico/showRegistroDominiosScreen.do +cr +ac.cr +co.cr +ed.cr +fi.cr +go.cr +or.cr +sa.cr + +// cu : https://en.wikipedia.org/wiki/.cu +cu +com.cu +edu.cu +org.cu +net.cu +gov.cu +inf.cu + +// cv : https://en.wikipedia.org/wiki/.cv +cv + +// cw : http://www.una.cw/cw_registry/ +// Confirmed by registry 2013-03-26 +cw +com.cw +edu.cw +net.cw +org.cw + +// cx : https://en.wikipedia.org/wiki/.cx +// list of other 2nd level tlds ? +cx +gov.cx + +// cy : http://www.nic.cy/ +// Submitted by registry Panayiotou Fotia +cy +ac.cy +biz.cy +com.cy +ekloges.cy +gov.cy +ltd.cy +name.cy +net.cy +org.cy +parliament.cy +press.cy +pro.cy +tm.cy + +// cz : https://en.wikipedia.org/wiki/.cz +cz + +// de : https://en.wikipedia.org/wiki/.de +// Confirmed by registry (with technical +// reservations) 2008-07-01 +de + +// dj : https://en.wikipedia.org/wiki/.dj +dj + +// dk : https://en.wikipedia.org/wiki/.dk +// Confirmed by registry 2008-06-17 +dk + +// dm : https://en.wikipedia.org/wiki/.dm +dm +com.dm +net.dm +org.dm +edu.dm +gov.dm + +// do : https://en.wikipedia.org/wiki/.do +do +art.do +com.do +edu.do +gob.do +gov.do +mil.do +net.do +org.do +sld.do +web.do + +// dz : http://www.nic.dz/images/pdf_nic/charte.pdf +dz +art.dz +asso.dz +com.dz +edu.dz +gov.dz +org.dz +net.dz +pol.dz +soc.dz +tm.dz + +// ec : http://www.nic.ec/reg/paso1.asp +// Submitted by registry +ec +com.ec +info.ec +net.ec +fin.ec +k12.ec +med.ec +pro.ec +org.ec +edu.ec +gov.ec +gob.ec +mil.ec + +// edu : https://en.wikipedia.org/wiki/.edu +edu + +// ee : http://www.eenet.ee/EENet/dom_reeglid.html#lisa_B +ee +edu.ee +gov.ee +riik.ee +lib.ee +med.ee +com.ee +pri.ee +aip.ee +org.ee +fie.ee + +// eg : https://en.wikipedia.org/wiki/.eg +eg +com.eg +edu.eg +eun.eg +gov.eg +mil.eg +name.eg +net.eg +org.eg +sci.eg + +// er : https://en.wikipedia.org/wiki/.er +*.er + +// es : https://www.nic.es/site_ingles/ingles/dominios/index.html +es +com.es +nom.es +org.es +gob.es +edu.es + +// et : https://en.wikipedia.org/wiki/.et +et +com.et +gov.et +org.et +edu.et +biz.et +name.et +info.et +net.et + +// eu : https://en.wikipedia.org/wiki/.eu +eu + +// fi : https://en.wikipedia.org/wiki/.fi +fi +// aland.fi : https://en.wikipedia.org/wiki/.ax +// This domain is being phased out in favor of .ax. As there are still many +// domains under aland.fi, we still keep it on the list until aland.fi is +// completely removed. +// TODO: Check for updates (expected to be phased out around Q1/2009) +aland.fi + +// fj : http://domains.fj/ +// Submitted by registry 2020-02-11 +fj +ac.fj +biz.fj +com.fj +gov.fj +info.fj +mil.fj +name.fj +net.fj +org.fj +pro.fj + +// fk : https://en.wikipedia.org/wiki/.fk +*.fk + +// fm : https://en.wikipedia.org/wiki/.fm +com.fm +edu.fm +net.fm +org.fm +fm + +// fo : https://en.wikipedia.org/wiki/.fo +fo + +// fr : http://www.afnic.fr/ +// domaines descriptifs : https://www.afnic.fr/medias/documents/Cadre_legal/Afnic_Naming_Policy_12122016_VEN.pdf +fr +asso.fr +com.fr +gouv.fr +nom.fr +prd.fr +tm.fr +// domaines sectoriels : https://www.afnic.fr/en/products-and-services/the-fr-tld/sector-based-fr-domains-4.html +aeroport.fr +avocat.fr +avoues.fr +cci.fr +chambagri.fr +chirurgiens-dentistes.fr +experts-comptables.fr +geometre-expert.fr +greta.fr +huissier-justice.fr +medecin.fr +notaires.fr +pharmacien.fr +port.fr +veterinaire.fr + +// ga : https://en.wikipedia.org/wiki/.ga +ga + +// gb : This registry is effectively dormant +// Submitted by registry +gb + +// gd : https://en.wikipedia.org/wiki/.gd +edu.gd +gov.gd +gd + +// ge : http://www.nic.net.ge/policy_en.pdf +ge +com.ge +edu.ge +gov.ge +org.ge +mil.ge +net.ge +pvt.ge + +// gf : https://en.wikipedia.org/wiki/.gf +gf + +// gg : http://www.channelisles.net/register-domains/ +// Confirmed by registry 2013-11-28 +gg +co.gg +net.gg +org.gg + +// gh : https://en.wikipedia.org/wiki/.gh +// see also: http://www.nic.gh/reg_now.php +// Although domains directly at second level are not possible at the moment, +// they have been possible for some time and may come back. +gh +com.gh +edu.gh +gov.gh +org.gh +mil.gh + +// gi : http://www.nic.gi/rules.html +gi +com.gi +ltd.gi +gov.gi +mod.gi +edu.gi +org.gi + +// gl : https://en.wikipedia.org/wiki/.gl +// http://nic.gl +gl +co.gl +com.gl +edu.gl +net.gl +org.gl + +// gm : http://www.nic.gm/htmlpages%5Cgm-policy.htm +gm + +// gn : http://psg.com/dns/gn/gn.txt +// Submitted by registry +gn +ac.gn +com.gn +edu.gn +gov.gn +org.gn +net.gn + +// gov : https://en.wikipedia.org/wiki/.gov +gov + +// gp : http://www.nic.gp/index.php?lang=en +gp +com.gp +net.gp +mobi.gp +edu.gp +org.gp +asso.gp + +// gq : https://en.wikipedia.org/wiki/.gq +gq + +// gr : https://grweb.ics.forth.gr/english/1617-B-2005.html +// Submitted by registry +gr +com.gr +edu.gr +net.gr +org.gr +gov.gr + +// gs : https://en.wikipedia.org/wiki/.gs +gs + +// gt : https://www.gt/sitio/registration_policy.php?lang=en +gt +com.gt +edu.gt +gob.gt +ind.gt +mil.gt +net.gt +org.gt + +// gu : http://gadao.gov.gu/register.html +// University of Guam : https://www.uog.edu +// Submitted by uognoc@triton.uog.edu +gu +com.gu +edu.gu +gov.gu +guam.gu +info.gu +net.gu +org.gu +web.gu + +// gw : https://en.wikipedia.org/wiki/.gw +gw + +// gy : https://en.wikipedia.org/wiki/.gy +// http://registry.gy/ +gy +co.gy +com.gy +edu.gy +gov.gy +net.gy +org.gy + +// hk : https://www.hkirc.hk +// Submitted by registry +hk +com.hk +edu.hk +gov.hk +idv.hk +net.hk +org.hk +公司.hk +教育.hk +敎育.hk +政府.hk +個人.hk +个人.hk +箇人.hk +網络.hk +网络.hk +组織.hk +網絡.hk +网絡.hk +组织.hk +組織.hk +組织.hk + +// hm : https://en.wikipedia.org/wiki/.hm +hm + +// hn : http://www.nic.hn/politicas/ps02,,05.html +hn +com.hn +edu.hn +org.hn +net.hn +mil.hn +gob.hn + +// hr : http://www.dns.hr/documents/pdf/HRTLD-regulations.pdf +hr +iz.hr +from.hr +name.hr +com.hr + +// ht : http://www.nic.ht/info/charte.cfm +ht +com.ht +shop.ht +firm.ht +info.ht +adult.ht +net.ht +pro.ht +org.ht +med.ht +art.ht +coop.ht +pol.ht +asso.ht +edu.ht +rel.ht +gouv.ht +perso.ht + +// hu : http://www.domain.hu/domain/English/sld.html +// Confirmed by registry 2008-06-12 +hu +co.hu +info.hu +org.hu +priv.hu +sport.hu +tm.hu +2000.hu +agrar.hu +bolt.hu +casino.hu +city.hu +erotica.hu +erotika.hu +film.hu +forum.hu +games.hu +hotel.hu +ingatlan.hu +jogasz.hu +konyvelo.hu +lakas.hu +media.hu +news.hu +reklam.hu +sex.hu +shop.hu +suli.hu +szex.hu +tozsde.hu +utazas.hu +video.hu + +// id : https://pandi.id/en/domain/registration-requirements/ +id +ac.id +biz.id +co.id +desa.id +go.id +mil.id +my.id +net.id +or.id +ponpes.id +sch.id +web.id + +// ie : https://en.wikipedia.org/wiki/.ie +ie +gov.ie + +// il : http://www.isoc.org.il/domains/ +il +ac.il +co.il +gov.il +idf.il +k12.il +muni.il +net.il +org.il + +// im : https://www.nic.im/ +// Submitted by registry +im +ac.im +co.im +com.im +ltd.co.im +net.im +org.im +plc.co.im +tt.im +tv.im + +// in : https://en.wikipedia.org/wiki/.in +// see also: https://registry.in/Policies +// Please note, that nic.in is not an official eTLD, but used by most +// government institutions. +in +co.in +firm.in +net.in +org.in +gen.in +ind.in +nic.in +ac.in +edu.in +res.in +gov.in +mil.in + +// info : https://en.wikipedia.org/wiki/.info +info + +// int : https://en.wikipedia.org/wiki/.int +// Confirmed by registry 2008-06-18 +int +eu.int + +// io : http://www.nic.io/rules.html +// list of other 2nd level tlds ? +io +com.io + +// iq : http://www.cmc.iq/english/iq/iqregister1.htm +iq +gov.iq +edu.iq +mil.iq +com.iq +org.iq +net.iq + +// ir : http://www.nic.ir/Terms_and_Conditions_ir,_Appendix_1_Domain_Rules +// Also see http://www.nic.ir/Internationalized_Domain_Names +// Two .ir entries added at request of , 2010-04-16 +ir +ac.ir +co.ir +gov.ir +id.ir +net.ir +org.ir +sch.ir +// xn--mgba3a4f16a.ir (.ir, Persian YEH) +ایران.ir +// xn--mgba3a4fra.ir (.ir, Arabic YEH) +ايران.ir + +// is : http://www.isnic.is/domain/rules.php +// Confirmed by registry 2008-12-06 +is +net.is +com.is +edu.is +gov.is +org.is +int.is + +// it : https://en.wikipedia.org/wiki/.it +it +gov.it +edu.it +// Reserved geo-names (regions and provinces): +// https://www.nic.it/sites/default/files/archivio/docs/Regulation_assignation_v7.1.pdf +// Regions +abr.it +abruzzo.it +aosta-valley.it +aostavalley.it +bas.it +basilicata.it +cal.it +calabria.it +cam.it +campania.it +emilia-romagna.it +emiliaromagna.it +emr.it +friuli-v-giulia.it +friuli-ve-giulia.it +friuli-vegiulia.it +friuli-venezia-giulia.it +friuli-veneziagiulia.it +friuli-vgiulia.it +friuliv-giulia.it +friulive-giulia.it +friulivegiulia.it +friulivenezia-giulia.it +friuliveneziagiulia.it +friulivgiulia.it +fvg.it +laz.it +lazio.it +lig.it +liguria.it +lom.it +lombardia.it +lombardy.it +lucania.it +mar.it +marche.it +mol.it +molise.it +piedmont.it +piemonte.it +pmn.it +pug.it +puglia.it +sar.it +sardegna.it +sardinia.it +sic.it +sicilia.it +sicily.it +taa.it +tos.it +toscana.it +trentin-sud-tirol.it +trentin-süd-tirol.it +trentin-sudtirol.it +trentin-südtirol.it +trentin-sued-tirol.it +trentin-suedtirol.it +trentino-a-adige.it +trentino-aadige.it +trentino-alto-adige.it +trentino-altoadige.it +trentino-s-tirol.it +trentino-stirol.it +trentino-sud-tirol.it +trentino-süd-tirol.it +trentino-sudtirol.it +trentino-südtirol.it +trentino-sued-tirol.it +trentino-suedtirol.it +trentino.it +trentinoa-adige.it +trentinoaadige.it +trentinoalto-adige.it +trentinoaltoadige.it +trentinos-tirol.it +trentinostirol.it +trentinosud-tirol.it +trentinosüd-tirol.it +trentinosudtirol.it +trentinosüdtirol.it +trentinosued-tirol.it +trentinosuedtirol.it +trentinsud-tirol.it +trentinsüd-tirol.it +trentinsudtirol.it +trentinsüdtirol.it +trentinsued-tirol.it +trentinsuedtirol.it +tuscany.it +umb.it +umbria.it +val-d-aosta.it +val-daosta.it +vald-aosta.it +valdaosta.it +valle-aosta.it +valle-d-aosta.it +valle-daosta.it +valleaosta.it +valled-aosta.it +valledaosta.it +vallee-aoste.it +vallée-aoste.it +vallee-d-aoste.it +vallée-d-aoste.it +valleeaoste.it +valléeaoste.it +valleedaoste.it +valléedaoste.it +vao.it +vda.it +ven.it +veneto.it +// Provinces +ag.it +agrigento.it +al.it +alessandria.it +alto-adige.it +altoadige.it +an.it +ancona.it +andria-barletta-trani.it +andria-trani-barletta.it +andriabarlettatrani.it +andriatranibarletta.it +ao.it +aosta.it +aoste.it +ap.it +aq.it +aquila.it +ar.it +arezzo.it +ascoli-piceno.it +ascolipiceno.it +asti.it +at.it +av.it +avellino.it +ba.it +balsan-sudtirol.it +balsan-südtirol.it +balsan-suedtirol.it +balsan.it +bari.it +barletta-trani-andria.it +barlettatraniandria.it +belluno.it +benevento.it +bergamo.it +bg.it +bi.it +biella.it +bl.it +bn.it +bo.it +bologna.it +bolzano-altoadige.it +bolzano.it +bozen-sudtirol.it +bozen-südtirol.it +bozen-suedtirol.it +bozen.it +br.it +brescia.it +brindisi.it +bs.it +bt.it +bulsan-sudtirol.it +bulsan-südtirol.it +bulsan-suedtirol.it +bulsan.it +bz.it +ca.it +cagliari.it +caltanissetta.it +campidano-medio.it +campidanomedio.it +campobasso.it +carbonia-iglesias.it +carboniaiglesias.it +carrara-massa.it +carraramassa.it +caserta.it +catania.it +catanzaro.it +cb.it +ce.it +cesena-forli.it +cesena-forlì.it +cesenaforli.it +cesenaforlì.it +ch.it +chieti.it +ci.it +cl.it +cn.it +co.it +como.it +cosenza.it +cr.it +cremona.it +crotone.it +cs.it +ct.it +cuneo.it +cz.it +dell-ogliastra.it +dellogliastra.it +en.it +enna.it +fc.it +fe.it +fermo.it +ferrara.it +fg.it +fi.it +firenze.it +florence.it +fm.it +foggia.it +forli-cesena.it +forlì-cesena.it +forlicesena.it +forlìcesena.it +fr.it +frosinone.it +ge.it +genoa.it +genova.it +go.it +gorizia.it +gr.it +grosseto.it +iglesias-carbonia.it +iglesiascarbonia.it +im.it +imperia.it +is.it +isernia.it +kr.it +la-spezia.it +laquila.it +laspezia.it +latina.it +lc.it +le.it +lecce.it +lecco.it +li.it +livorno.it +lo.it +lodi.it +lt.it +lu.it +lucca.it +macerata.it +mantova.it +massa-carrara.it +massacarrara.it +matera.it +mb.it +mc.it +me.it +medio-campidano.it +mediocampidano.it +messina.it +mi.it +milan.it +milano.it +mn.it +mo.it +modena.it +monza-brianza.it +monza-e-della-brianza.it +monza.it +monzabrianza.it +monzaebrianza.it +monzaedellabrianza.it +ms.it +mt.it +na.it +naples.it +napoli.it +no.it +novara.it +nu.it +nuoro.it +og.it +ogliastra.it +olbia-tempio.it +olbiatempio.it +or.it +oristano.it +ot.it +pa.it +padova.it +padua.it +palermo.it +parma.it +pavia.it +pc.it +pd.it +pe.it +perugia.it +pesaro-urbino.it +pesarourbino.it +pescara.it +pg.it +pi.it +piacenza.it +pisa.it +pistoia.it +pn.it +po.it +pordenone.it +potenza.it +pr.it +prato.it +pt.it +pu.it +pv.it +pz.it +ra.it +ragusa.it +ravenna.it +rc.it +re.it +reggio-calabria.it +reggio-emilia.it +reggiocalabria.it +reggioemilia.it +rg.it +ri.it +rieti.it +rimini.it +rm.it +rn.it +ro.it +roma.it +rome.it +rovigo.it +sa.it +salerno.it +sassari.it +savona.it +si.it +siena.it +siracusa.it +so.it +sondrio.it +sp.it +sr.it +ss.it +suedtirol.it +südtirol.it +sv.it +ta.it +taranto.it +te.it +tempio-olbia.it +tempioolbia.it +teramo.it +terni.it +tn.it +to.it +torino.it +tp.it +tr.it +trani-andria-barletta.it +trani-barletta-andria.it +traniandriabarletta.it +tranibarlettaandria.it +trapani.it +trento.it +treviso.it +trieste.it +ts.it +turin.it +tv.it +ud.it +udine.it +urbino-pesaro.it +urbinopesaro.it +va.it +varese.it +vb.it +vc.it +ve.it +venezia.it +venice.it +verbania.it +vercelli.it +verona.it +vi.it +vibo-valentia.it +vibovalentia.it +vicenza.it +viterbo.it +vr.it +vs.it +vt.it +vv.it + +// je : http://www.channelisles.net/register-domains/ +// Confirmed by registry 2013-11-28 +je +co.je +net.je +org.je + +// jm : http://www.com.jm/register.html +*.jm + +// jo : http://www.dns.jo/Registration_policy.aspx +jo +com.jo +org.jo +net.jo +edu.jo +sch.jo +gov.jo +mil.jo +name.jo + +// jobs : https://en.wikipedia.org/wiki/.jobs +jobs + +// jp : https://en.wikipedia.org/wiki/.jp +// http://jprs.co.jp/en/jpdomain.html +// Submitted by registry +jp +// jp organizational type names +ac.jp +ad.jp +co.jp +ed.jp +go.jp +gr.jp +lg.jp +ne.jp +or.jp +// jp prefecture type names +aichi.jp +akita.jp +aomori.jp +chiba.jp +ehime.jp +fukui.jp +fukuoka.jp +fukushima.jp +gifu.jp +gunma.jp +hiroshima.jp +hokkaido.jp +hyogo.jp +ibaraki.jp +ishikawa.jp +iwate.jp +kagawa.jp +kagoshima.jp +kanagawa.jp +kochi.jp +kumamoto.jp +kyoto.jp +mie.jp +miyagi.jp +miyazaki.jp +nagano.jp +nagasaki.jp +nara.jp +niigata.jp +oita.jp +okayama.jp +okinawa.jp +osaka.jp +saga.jp +saitama.jp +shiga.jp +shimane.jp +shizuoka.jp +tochigi.jp +tokushima.jp +tokyo.jp +tottori.jp +toyama.jp +wakayama.jp +yamagata.jp +yamaguchi.jp +yamanashi.jp +栃木.jp +愛知.jp +愛媛.jp +兵庫.jp +熊本.jp +茨城.jp +北海道.jp +千葉.jp +和歌山.jp +長崎.jp +長野.jp +新潟.jp +青森.jp +静岡.jp +東京.jp +石川.jp +埼玉.jp +三重.jp +京都.jp +佐賀.jp +大分.jp +大阪.jp +奈良.jp +宮城.jp +宮崎.jp +富山.jp +山口.jp +山形.jp +山梨.jp +岩手.jp +岐阜.jp +岡山.jp +島根.jp +広島.jp +徳島.jp +沖縄.jp +滋賀.jp +神奈川.jp +福井.jp +福岡.jp +福島.jp +秋田.jp +群馬.jp +香川.jp +高知.jp +鳥取.jp +鹿児島.jp +// jp geographic type names +// http://jprs.jp/doc/rule/saisoku-1.html +*.kawasaki.jp +*.kitakyushu.jp +*.kobe.jp +*.nagoya.jp +*.sapporo.jp +*.sendai.jp +*.yokohama.jp +!city.kawasaki.jp +!city.kitakyushu.jp +!city.kobe.jp +!city.nagoya.jp +!city.sapporo.jp +!city.sendai.jp +!city.yokohama.jp +// 4th level registration +aisai.aichi.jp +ama.aichi.jp +anjo.aichi.jp +asuke.aichi.jp +chiryu.aichi.jp +chita.aichi.jp +fuso.aichi.jp +gamagori.aichi.jp +handa.aichi.jp +hazu.aichi.jp +hekinan.aichi.jp +higashiura.aichi.jp +ichinomiya.aichi.jp +inazawa.aichi.jp +inuyama.aichi.jp +isshiki.aichi.jp +iwakura.aichi.jp +kanie.aichi.jp +kariya.aichi.jp +kasugai.aichi.jp +kira.aichi.jp +kiyosu.aichi.jp +komaki.aichi.jp +konan.aichi.jp +kota.aichi.jp +mihama.aichi.jp +miyoshi.aichi.jp +nishio.aichi.jp +nisshin.aichi.jp +obu.aichi.jp +oguchi.aichi.jp +oharu.aichi.jp +okazaki.aichi.jp +owariasahi.aichi.jp +seto.aichi.jp +shikatsu.aichi.jp +shinshiro.aichi.jp +shitara.aichi.jp +tahara.aichi.jp +takahama.aichi.jp +tobishima.aichi.jp +toei.aichi.jp +togo.aichi.jp +tokai.aichi.jp +tokoname.aichi.jp +toyoake.aichi.jp +toyohashi.aichi.jp +toyokawa.aichi.jp +toyone.aichi.jp +toyota.aichi.jp +tsushima.aichi.jp +yatomi.aichi.jp +akita.akita.jp +daisen.akita.jp +fujisato.akita.jp +gojome.akita.jp +hachirogata.akita.jp +happou.akita.jp +higashinaruse.akita.jp +honjo.akita.jp +honjyo.akita.jp +ikawa.akita.jp +kamikoani.akita.jp +kamioka.akita.jp +katagami.akita.jp +kazuno.akita.jp +kitaakita.akita.jp +kosaka.akita.jp +kyowa.akita.jp +misato.akita.jp +mitane.akita.jp +moriyoshi.akita.jp +nikaho.akita.jp +noshiro.akita.jp +odate.akita.jp +oga.akita.jp +ogata.akita.jp +semboku.akita.jp +yokote.akita.jp +yurihonjo.akita.jp +aomori.aomori.jp +gonohe.aomori.jp +hachinohe.aomori.jp +hashikami.aomori.jp +hiranai.aomori.jp +hirosaki.aomori.jp +itayanagi.aomori.jp +kuroishi.aomori.jp +misawa.aomori.jp +mutsu.aomori.jp +nakadomari.aomori.jp +noheji.aomori.jp +oirase.aomori.jp +owani.aomori.jp +rokunohe.aomori.jp +sannohe.aomori.jp +shichinohe.aomori.jp +shingo.aomori.jp +takko.aomori.jp +towada.aomori.jp +tsugaru.aomori.jp +tsuruta.aomori.jp +abiko.chiba.jp +asahi.chiba.jp +chonan.chiba.jp +chosei.chiba.jp +choshi.chiba.jp +chuo.chiba.jp +funabashi.chiba.jp +futtsu.chiba.jp +hanamigawa.chiba.jp +ichihara.chiba.jp +ichikawa.chiba.jp +ichinomiya.chiba.jp +inzai.chiba.jp +isumi.chiba.jp +kamagaya.chiba.jp +kamogawa.chiba.jp +kashiwa.chiba.jp +katori.chiba.jp +katsuura.chiba.jp +kimitsu.chiba.jp +kisarazu.chiba.jp +kozaki.chiba.jp +kujukuri.chiba.jp +kyonan.chiba.jp +matsudo.chiba.jp +midori.chiba.jp +mihama.chiba.jp +minamiboso.chiba.jp +mobara.chiba.jp +mutsuzawa.chiba.jp +nagara.chiba.jp +nagareyama.chiba.jp +narashino.chiba.jp +narita.chiba.jp +noda.chiba.jp +oamishirasato.chiba.jp +omigawa.chiba.jp +onjuku.chiba.jp +otaki.chiba.jp +sakae.chiba.jp +sakura.chiba.jp +shimofusa.chiba.jp +shirako.chiba.jp +shiroi.chiba.jp +shisui.chiba.jp +sodegaura.chiba.jp +sosa.chiba.jp +tako.chiba.jp +tateyama.chiba.jp +togane.chiba.jp +tohnosho.chiba.jp +tomisato.chiba.jp +urayasu.chiba.jp +yachimata.chiba.jp +yachiyo.chiba.jp +yokaichiba.chiba.jp +yokoshibahikari.chiba.jp +yotsukaido.chiba.jp +ainan.ehime.jp +honai.ehime.jp +ikata.ehime.jp +imabari.ehime.jp +iyo.ehime.jp +kamijima.ehime.jp +kihoku.ehime.jp +kumakogen.ehime.jp +masaki.ehime.jp +matsuno.ehime.jp +matsuyama.ehime.jp +namikata.ehime.jp +niihama.ehime.jp +ozu.ehime.jp +saijo.ehime.jp +seiyo.ehime.jp +shikokuchuo.ehime.jp +tobe.ehime.jp +toon.ehime.jp +uchiko.ehime.jp +uwajima.ehime.jp +yawatahama.ehime.jp +echizen.fukui.jp +eiheiji.fukui.jp +fukui.fukui.jp +ikeda.fukui.jp +katsuyama.fukui.jp +mihama.fukui.jp +minamiechizen.fukui.jp +obama.fukui.jp +ohi.fukui.jp +ono.fukui.jp +sabae.fukui.jp +sakai.fukui.jp +takahama.fukui.jp +tsuruga.fukui.jp +wakasa.fukui.jp +ashiya.fukuoka.jp +buzen.fukuoka.jp +chikugo.fukuoka.jp +chikuho.fukuoka.jp +chikujo.fukuoka.jp +chikushino.fukuoka.jp +chikuzen.fukuoka.jp +chuo.fukuoka.jp +dazaifu.fukuoka.jp +fukuchi.fukuoka.jp +hakata.fukuoka.jp +higashi.fukuoka.jp +hirokawa.fukuoka.jp +hisayama.fukuoka.jp +iizuka.fukuoka.jp +inatsuki.fukuoka.jp +kaho.fukuoka.jp +kasuga.fukuoka.jp +kasuya.fukuoka.jp +kawara.fukuoka.jp +keisen.fukuoka.jp +koga.fukuoka.jp +kurate.fukuoka.jp +kurogi.fukuoka.jp +kurume.fukuoka.jp +minami.fukuoka.jp +miyako.fukuoka.jp +miyama.fukuoka.jp +miyawaka.fukuoka.jp +mizumaki.fukuoka.jp +munakata.fukuoka.jp +nakagawa.fukuoka.jp +nakama.fukuoka.jp +nishi.fukuoka.jp +nogata.fukuoka.jp +ogori.fukuoka.jp +okagaki.fukuoka.jp +okawa.fukuoka.jp +oki.fukuoka.jp +omuta.fukuoka.jp +onga.fukuoka.jp +onojo.fukuoka.jp +oto.fukuoka.jp +saigawa.fukuoka.jp +sasaguri.fukuoka.jp +shingu.fukuoka.jp +shinyoshitomi.fukuoka.jp +shonai.fukuoka.jp +soeda.fukuoka.jp +sue.fukuoka.jp +tachiarai.fukuoka.jp +tagawa.fukuoka.jp +takata.fukuoka.jp +toho.fukuoka.jp +toyotsu.fukuoka.jp +tsuiki.fukuoka.jp +ukiha.fukuoka.jp +umi.fukuoka.jp +usui.fukuoka.jp +yamada.fukuoka.jp +yame.fukuoka.jp +yanagawa.fukuoka.jp +yukuhashi.fukuoka.jp +aizubange.fukushima.jp +aizumisato.fukushima.jp +aizuwakamatsu.fukushima.jp +asakawa.fukushima.jp +bandai.fukushima.jp +date.fukushima.jp +fukushima.fukushima.jp +furudono.fukushima.jp +futaba.fukushima.jp +hanawa.fukushima.jp +higashi.fukushima.jp +hirata.fukushima.jp +hirono.fukushima.jp +iitate.fukushima.jp +inawashiro.fukushima.jp +ishikawa.fukushima.jp +iwaki.fukushima.jp +izumizaki.fukushima.jp +kagamiishi.fukushima.jp +kaneyama.fukushima.jp +kawamata.fukushima.jp +kitakata.fukushima.jp +kitashiobara.fukushima.jp +koori.fukushima.jp +koriyama.fukushima.jp +kunimi.fukushima.jp +miharu.fukushima.jp +mishima.fukushima.jp +namie.fukushima.jp +nango.fukushima.jp +nishiaizu.fukushima.jp +nishigo.fukushima.jp +okuma.fukushima.jp +omotego.fukushima.jp +ono.fukushima.jp +otama.fukushima.jp +samegawa.fukushima.jp +shimogo.fukushima.jp +shirakawa.fukushima.jp +showa.fukushima.jp +soma.fukushima.jp +sukagawa.fukushima.jp +taishin.fukushima.jp +tamakawa.fukushima.jp +tanagura.fukushima.jp +tenei.fukushima.jp +yabuki.fukushima.jp +yamato.fukushima.jp +yamatsuri.fukushima.jp +yanaizu.fukushima.jp +yugawa.fukushima.jp +anpachi.gifu.jp +ena.gifu.jp +gifu.gifu.jp +ginan.gifu.jp +godo.gifu.jp +gujo.gifu.jp +hashima.gifu.jp +hichiso.gifu.jp +hida.gifu.jp +higashishirakawa.gifu.jp +ibigawa.gifu.jp +ikeda.gifu.jp +kakamigahara.gifu.jp +kani.gifu.jp +kasahara.gifu.jp +kasamatsu.gifu.jp +kawaue.gifu.jp +kitagata.gifu.jp +mino.gifu.jp +minokamo.gifu.jp +mitake.gifu.jp +mizunami.gifu.jp +motosu.gifu.jp +nakatsugawa.gifu.jp +ogaki.gifu.jp +sakahogi.gifu.jp +seki.gifu.jp +sekigahara.gifu.jp +shirakawa.gifu.jp +tajimi.gifu.jp +takayama.gifu.jp +tarui.gifu.jp +toki.gifu.jp +tomika.gifu.jp +wanouchi.gifu.jp +yamagata.gifu.jp +yaotsu.gifu.jp +yoro.gifu.jp +annaka.gunma.jp +chiyoda.gunma.jp +fujioka.gunma.jp +higashiagatsuma.gunma.jp +isesaki.gunma.jp +itakura.gunma.jp +kanna.gunma.jp +kanra.gunma.jp +katashina.gunma.jp +kawaba.gunma.jp +kiryu.gunma.jp +kusatsu.gunma.jp +maebashi.gunma.jp +meiwa.gunma.jp +midori.gunma.jp +minakami.gunma.jp +naganohara.gunma.jp +nakanojo.gunma.jp +nanmoku.gunma.jp +numata.gunma.jp +oizumi.gunma.jp +ora.gunma.jp +ota.gunma.jp +shibukawa.gunma.jp +shimonita.gunma.jp +shinto.gunma.jp +showa.gunma.jp +takasaki.gunma.jp +takayama.gunma.jp +tamamura.gunma.jp +tatebayashi.gunma.jp +tomioka.gunma.jp +tsukiyono.gunma.jp +tsumagoi.gunma.jp +ueno.gunma.jp +yoshioka.gunma.jp +asaminami.hiroshima.jp +daiwa.hiroshima.jp +etajima.hiroshima.jp +fuchu.hiroshima.jp +fukuyama.hiroshima.jp +hatsukaichi.hiroshima.jp +higashihiroshima.hiroshima.jp +hongo.hiroshima.jp +jinsekikogen.hiroshima.jp +kaita.hiroshima.jp +kui.hiroshima.jp +kumano.hiroshima.jp +kure.hiroshima.jp +mihara.hiroshima.jp +miyoshi.hiroshima.jp +naka.hiroshima.jp +onomichi.hiroshima.jp +osakikamijima.hiroshima.jp +otake.hiroshima.jp +saka.hiroshima.jp +sera.hiroshima.jp +seranishi.hiroshima.jp +shinichi.hiroshima.jp +shobara.hiroshima.jp +takehara.hiroshima.jp +abashiri.hokkaido.jp +abira.hokkaido.jp +aibetsu.hokkaido.jp +akabira.hokkaido.jp +akkeshi.hokkaido.jp +asahikawa.hokkaido.jp +ashibetsu.hokkaido.jp +ashoro.hokkaido.jp +assabu.hokkaido.jp +atsuma.hokkaido.jp +bibai.hokkaido.jp +biei.hokkaido.jp +bifuka.hokkaido.jp +bihoro.hokkaido.jp +biratori.hokkaido.jp +chippubetsu.hokkaido.jp +chitose.hokkaido.jp +date.hokkaido.jp +ebetsu.hokkaido.jp +embetsu.hokkaido.jp +eniwa.hokkaido.jp +erimo.hokkaido.jp +esan.hokkaido.jp +esashi.hokkaido.jp +fukagawa.hokkaido.jp +fukushima.hokkaido.jp +furano.hokkaido.jp +furubira.hokkaido.jp +haboro.hokkaido.jp +hakodate.hokkaido.jp +hamatonbetsu.hokkaido.jp +hidaka.hokkaido.jp +higashikagura.hokkaido.jp +higashikawa.hokkaido.jp +hiroo.hokkaido.jp +hokuryu.hokkaido.jp +hokuto.hokkaido.jp +honbetsu.hokkaido.jp +horokanai.hokkaido.jp +horonobe.hokkaido.jp +ikeda.hokkaido.jp +imakane.hokkaido.jp +ishikari.hokkaido.jp +iwamizawa.hokkaido.jp +iwanai.hokkaido.jp +kamifurano.hokkaido.jp +kamikawa.hokkaido.jp +kamishihoro.hokkaido.jp +kamisunagawa.hokkaido.jp +kamoenai.hokkaido.jp +kayabe.hokkaido.jp +kembuchi.hokkaido.jp +kikonai.hokkaido.jp +kimobetsu.hokkaido.jp +kitahiroshima.hokkaido.jp +kitami.hokkaido.jp +kiyosato.hokkaido.jp +koshimizu.hokkaido.jp +kunneppu.hokkaido.jp +kuriyama.hokkaido.jp +kuromatsunai.hokkaido.jp +kushiro.hokkaido.jp +kutchan.hokkaido.jp +kyowa.hokkaido.jp +mashike.hokkaido.jp +matsumae.hokkaido.jp +mikasa.hokkaido.jp +minamifurano.hokkaido.jp +mombetsu.hokkaido.jp +moseushi.hokkaido.jp +mukawa.hokkaido.jp +muroran.hokkaido.jp +naie.hokkaido.jp +nakagawa.hokkaido.jp +nakasatsunai.hokkaido.jp +nakatombetsu.hokkaido.jp +nanae.hokkaido.jp +nanporo.hokkaido.jp +nayoro.hokkaido.jp +nemuro.hokkaido.jp +niikappu.hokkaido.jp +niki.hokkaido.jp +nishiokoppe.hokkaido.jp +noboribetsu.hokkaido.jp +numata.hokkaido.jp +obihiro.hokkaido.jp +obira.hokkaido.jp +oketo.hokkaido.jp +okoppe.hokkaido.jp +otaru.hokkaido.jp +otobe.hokkaido.jp +otofuke.hokkaido.jp +otoineppu.hokkaido.jp +oumu.hokkaido.jp +ozora.hokkaido.jp +pippu.hokkaido.jp +rankoshi.hokkaido.jp +rebun.hokkaido.jp +rikubetsu.hokkaido.jp +rishiri.hokkaido.jp +rishirifuji.hokkaido.jp +saroma.hokkaido.jp +sarufutsu.hokkaido.jp +shakotan.hokkaido.jp +shari.hokkaido.jp +shibecha.hokkaido.jp +shibetsu.hokkaido.jp +shikabe.hokkaido.jp +shikaoi.hokkaido.jp +shimamaki.hokkaido.jp +shimizu.hokkaido.jp +shimokawa.hokkaido.jp +shinshinotsu.hokkaido.jp +shintoku.hokkaido.jp +shiranuka.hokkaido.jp +shiraoi.hokkaido.jp +shiriuchi.hokkaido.jp +sobetsu.hokkaido.jp +sunagawa.hokkaido.jp +taiki.hokkaido.jp +takasu.hokkaido.jp +takikawa.hokkaido.jp +takinoue.hokkaido.jp +teshikaga.hokkaido.jp +tobetsu.hokkaido.jp +tohma.hokkaido.jp +tomakomai.hokkaido.jp +tomari.hokkaido.jp +toya.hokkaido.jp +toyako.hokkaido.jp +toyotomi.hokkaido.jp +toyoura.hokkaido.jp +tsubetsu.hokkaido.jp +tsukigata.hokkaido.jp +urakawa.hokkaido.jp +urausu.hokkaido.jp +uryu.hokkaido.jp +utashinai.hokkaido.jp +wakkanai.hokkaido.jp +wassamu.hokkaido.jp +yakumo.hokkaido.jp +yoichi.hokkaido.jp +aioi.hyogo.jp +akashi.hyogo.jp +ako.hyogo.jp +amagasaki.hyogo.jp +aogaki.hyogo.jp +asago.hyogo.jp +ashiya.hyogo.jp +awaji.hyogo.jp +fukusaki.hyogo.jp +goshiki.hyogo.jp +harima.hyogo.jp +himeji.hyogo.jp +ichikawa.hyogo.jp +inagawa.hyogo.jp +itami.hyogo.jp +kakogawa.hyogo.jp +kamigori.hyogo.jp +kamikawa.hyogo.jp +kasai.hyogo.jp +kasuga.hyogo.jp +kawanishi.hyogo.jp +miki.hyogo.jp +minamiawaji.hyogo.jp +nishinomiya.hyogo.jp +nishiwaki.hyogo.jp +ono.hyogo.jp +sanda.hyogo.jp +sannan.hyogo.jp +sasayama.hyogo.jp +sayo.hyogo.jp +shingu.hyogo.jp +shinonsen.hyogo.jp +shiso.hyogo.jp +sumoto.hyogo.jp +taishi.hyogo.jp +taka.hyogo.jp +takarazuka.hyogo.jp +takasago.hyogo.jp +takino.hyogo.jp +tamba.hyogo.jp +tatsuno.hyogo.jp +toyooka.hyogo.jp +yabu.hyogo.jp +yashiro.hyogo.jp +yoka.hyogo.jp +yokawa.hyogo.jp +ami.ibaraki.jp +asahi.ibaraki.jp +bando.ibaraki.jp +chikusei.ibaraki.jp +daigo.ibaraki.jp +fujishiro.ibaraki.jp +hitachi.ibaraki.jp +hitachinaka.ibaraki.jp +hitachiomiya.ibaraki.jp +hitachiota.ibaraki.jp +ibaraki.ibaraki.jp +ina.ibaraki.jp +inashiki.ibaraki.jp +itako.ibaraki.jp +iwama.ibaraki.jp +joso.ibaraki.jp +kamisu.ibaraki.jp +kasama.ibaraki.jp +kashima.ibaraki.jp +kasumigaura.ibaraki.jp +koga.ibaraki.jp +miho.ibaraki.jp +mito.ibaraki.jp +moriya.ibaraki.jp +naka.ibaraki.jp +namegata.ibaraki.jp +oarai.ibaraki.jp +ogawa.ibaraki.jp +omitama.ibaraki.jp +ryugasaki.ibaraki.jp +sakai.ibaraki.jp +sakuragawa.ibaraki.jp +shimodate.ibaraki.jp +shimotsuma.ibaraki.jp +shirosato.ibaraki.jp +sowa.ibaraki.jp +suifu.ibaraki.jp +takahagi.ibaraki.jp +tamatsukuri.ibaraki.jp +tokai.ibaraki.jp +tomobe.ibaraki.jp +tone.ibaraki.jp +toride.ibaraki.jp +tsuchiura.ibaraki.jp +tsukuba.ibaraki.jp +uchihara.ibaraki.jp +ushiku.ibaraki.jp +yachiyo.ibaraki.jp +yamagata.ibaraki.jp +yawara.ibaraki.jp +yuki.ibaraki.jp +anamizu.ishikawa.jp +hakui.ishikawa.jp +hakusan.ishikawa.jp +kaga.ishikawa.jp +kahoku.ishikawa.jp +kanazawa.ishikawa.jp +kawakita.ishikawa.jp +komatsu.ishikawa.jp +nakanoto.ishikawa.jp +nanao.ishikawa.jp +nomi.ishikawa.jp +nonoichi.ishikawa.jp +noto.ishikawa.jp +shika.ishikawa.jp +suzu.ishikawa.jp +tsubata.ishikawa.jp +tsurugi.ishikawa.jp +uchinada.ishikawa.jp +wajima.ishikawa.jp +fudai.iwate.jp +fujisawa.iwate.jp +hanamaki.iwate.jp +hiraizumi.iwate.jp +hirono.iwate.jp +ichinohe.iwate.jp +ichinoseki.iwate.jp +iwaizumi.iwate.jp +iwate.iwate.jp +joboji.iwate.jp +kamaishi.iwate.jp +kanegasaki.iwate.jp +karumai.iwate.jp +kawai.iwate.jp +kitakami.iwate.jp +kuji.iwate.jp +kunohe.iwate.jp +kuzumaki.iwate.jp +miyako.iwate.jp +mizusawa.iwate.jp +morioka.iwate.jp +ninohe.iwate.jp +noda.iwate.jp +ofunato.iwate.jp +oshu.iwate.jp +otsuchi.iwate.jp +rikuzentakata.iwate.jp +shiwa.iwate.jp +shizukuishi.iwate.jp +sumita.iwate.jp +tanohata.iwate.jp +tono.iwate.jp +yahaba.iwate.jp +yamada.iwate.jp +ayagawa.kagawa.jp +higashikagawa.kagawa.jp +kanonji.kagawa.jp +kotohira.kagawa.jp +manno.kagawa.jp +marugame.kagawa.jp +mitoyo.kagawa.jp +naoshima.kagawa.jp +sanuki.kagawa.jp +tadotsu.kagawa.jp +takamatsu.kagawa.jp +tonosho.kagawa.jp +uchinomi.kagawa.jp +utazu.kagawa.jp +zentsuji.kagawa.jp +akune.kagoshima.jp +amami.kagoshima.jp +hioki.kagoshima.jp +isa.kagoshima.jp +isen.kagoshima.jp +izumi.kagoshima.jp +kagoshima.kagoshima.jp +kanoya.kagoshima.jp +kawanabe.kagoshima.jp +kinko.kagoshima.jp +kouyama.kagoshima.jp +makurazaki.kagoshima.jp +matsumoto.kagoshima.jp +minamitane.kagoshima.jp +nakatane.kagoshima.jp +nishinoomote.kagoshima.jp +satsumasendai.kagoshima.jp +soo.kagoshima.jp +tarumizu.kagoshima.jp +yusui.kagoshima.jp +aikawa.kanagawa.jp +atsugi.kanagawa.jp +ayase.kanagawa.jp +chigasaki.kanagawa.jp +ebina.kanagawa.jp +fujisawa.kanagawa.jp +hadano.kanagawa.jp +hakone.kanagawa.jp +hiratsuka.kanagawa.jp +isehara.kanagawa.jp +kaisei.kanagawa.jp +kamakura.kanagawa.jp +kiyokawa.kanagawa.jp +matsuda.kanagawa.jp +minamiashigara.kanagawa.jp +miura.kanagawa.jp +nakai.kanagawa.jp +ninomiya.kanagawa.jp +odawara.kanagawa.jp +oi.kanagawa.jp +oiso.kanagawa.jp +sagamihara.kanagawa.jp +samukawa.kanagawa.jp +tsukui.kanagawa.jp +yamakita.kanagawa.jp +yamato.kanagawa.jp +yokosuka.kanagawa.jp +yugawara.kanagawa.jp +zama.kanagawa.jp +zushi.kanagawa.jp +aki.kochi.jp +geisei.kochi.jp +hidaka.kochi.jp +higashitsuno.kochi.jp +ino.kochi.jp +kagami.kochi.jp +kami.kochi.jp +kitagawa.kochi.jp +kochi.kochi.jp +mihara.kochi.jp +motoyama.kochi.jp +muroto.kochi.jp +nahari.kochi.jp +nakamura.kochi.jp +nankoku.kochi.jp +nishitosa.kochi.jp +niyodogawa.kochi.jp +ochi.kochi.jp +okawa.kochi.jp +otoyo.kochi.jp +otsuki.kochi.jp +sakawa.kochi.jp +sukumo.kochi.jp +susaki.kochi.jp +tosa.kochi.jp +tosashimizu.kochi.jp +toyo.kochi.jp +tsuno.kochi.jp +umaji.kochi.jp +yasuda.kochi.jp +yusuhara.kochi.jp +amakusa.kumamoto.jp +arao.kumamoto.jp +aso.kumamoto.jp +choyo.kumamoto.jp +gyokuto.kumamoto.jp +kamiamakusa.kumamoto.jp +kikuchi.kumamoto.jp +kumamoto.kumamoto.jp +mashiki.kumamoto.jp +mifune.kumamoto.jp +minamata.kumamoto.jp +minamioguni.kumamoto.jp +nagasu.kumamoto.jp +nishihara.kumamoto.jp +oguni.kumamoto.jp +ozu.kumamoto.jp +sumoto.kumamoto.jp +takamori.kumamoto.jp +uki.kumamoto.jp +uto.kumamoto.jp +yamaga.kumamoto.jp +yamato.kumamoto.jp +yatsushiro.kumamoto.jp +ayabe.kyoto.jp +fukuchiyama.kyoto.jp +higashiyama.kyoto.jp +ide.kyoto.jp +ine.kyoto.jp +joyo.kyoto.jp +kameoka.kyoto.jp +kamo.kyoto.jp +kita.kyoto.jp +kizu.kyoto.jp +kumiyama.kyoto.jp +kyotamba.kyoto.jp +kyotanabe.kyoto.jp +kyotango.kyoto.jp +maizuru.kyoto.jp +minami.kyoto.jp +minamiyamashiro.kyoto.jp +miyazu.kyoto.jp +muko.kyoto.jp +nagaokakyo.kyoto.jp +nakagyo.kyoto.jp +nantan.kyoto.jp +oyamazaki.kyoto.jp +sakyo.kyoto.jp +seika.kyoto.jp +tanabe.kyoto.jp +uji.kyoto.jp +ujitawara.kyoto.jp +wazuka.kyoto.jp +yamashina.kyoto.jp +yawata.kyoto.jp +asahi.mie.jp +inabe.mie.jp +ise.mie.jp +kameyama.mie.jp +kawagoe.mie.jp +kiho.mie.jp +kisosaki.mie.jp +kiwa.mie.jp +komono.mie.jp +kumano.mie.jp +kuwana.mie.jp +matsusaka.mie.jp +meiwa.mie.jp +mihama.mie.jp +minamiise.mie.jp +misugi.mie.jp +miyama.mie.jp +nabari.mie.jp +shima.mie.jp +suzuka.mie.jp +tado.mie.jp +taiki.mie.jp +taki.mie.jp +tamaki.mie.jp +toba.mie.jp +tsu.mie.jp +udono.mie.jp +ureshino.mie.jp +watarai.mie.jp +yokkaichi.mie.jp +furukawa.miyagi.jp +higashimatsushima.miyagi.jp +ishinomaki.miyagi.jp +iwanuma.miyagi.jp +kakuda.miyagi.jp +kami.miyagi.jp +kawasaki.miyagi.jp +marumori.miyagi.jp +matsushima.miyagi.jp +minamisanriku.miyagi.jp +misato.miyagi.jp +murata.miyagi.jp +natori.miyagi.jp +ogawara.miyagi.jp +ohira.miyagi.jp +onagawa.miyagi.jp +osaki.miyagi.jp +rifu.miyagi.jp +semine.miyagi.jp +shibata.miyagi.jp +shichikashuku.miyagi.jp +shikama.miyagi.jp +shiogama.miyagi.jp +shiroishi.miyagi.jp +tagajo.miyagi.jp +taiwa.miyagi.jp +tome.miyagi.jp +tomiya.miyagi.jp +wakuya.miyagi.jp +watari.miyagi.jp +yamamoto.miyagi.jp +zao.miyagi.jp +aya.miyazaki.jp +ebino.miyazaki.jp +gokase.miyazaki.jp +hyuga.miyazaki.jp +kadogawa.miyazaki.jp +kawaminami.miyazaki.jp +kijo.miyazaki.jp +kitagawa.miyazaki.jp +kitakata.miyazaki.jp +kitaura.miyazaki.jp +kobayashi.miyazaki.jp +kunitomi.miyazaki.jp +kushima.miyazaki.jp +mimata.miyazaki.jp +miyakonojo.miyazaki.jp +miyazaki.miyazaki.jp +morotsuka.miyazaki.jp +nichinan.miyazaki.jp +nishimera.miyazaki.jp +nobeoka.miyazaki.jp +saito.miyazaki.jp +shiiba.miyazaki.jp +shintomi.miyazaki.jp +takaharu.miyazaki.jp +takanabe.miyazaki.jp +takazaki.miyazaki.jp +tsuno.miyazaki.jp +achi.nagano.jp +agematsu.nagano.jp +anan.nagano.jp +aoki.nagano.jp +asahi.nagano.jp +azumino.nagano.jp +chikuhoku.nagano.jp +chikuma.nagano.jp +chino.nagano.jp +fujimi.nagano.jp +hakuba.nagano.jp +hara.nagano.jp +hiraya.nagano.jp +iida.nagano.jp +iijima.nagano.jp +iiyama.nagano.jp +iizuna.nagano.jp +ikeda.nagano.jp +ikusaka.nagano.jp +ina.nagano.jp +karuizawa.nagano.jp +kawakami.nagano.jp +kiso.nagano.jp +kisofukushima.nagano.jp +kitaaiki.nagano.jp +komagane.nagano.jp +komoro.nagano.jp +matsukawa.nagano.jp +matsumoto.nagano.jp +miasa.nagano.jp +minamiaiki.nagano.jp +minamimaki.nagano.jp +minamiminowa.nagano.jp +minowa.nagano.jp +miyada.nagano.jp +miyota.nagano.jp +mochizuki.nagano.jp +nagano.nagano.jp +nagawa.nagano.jp +nagiso.nagano.jp +nakagawa.nagano.jp +nakano.nagano.jp +nozawaonsen.nagano.jp +obuse.nagano.jp +ogawa.nagano.jp +okaya.nagano.jp +omachi.nagano.jp +omi.nagano.jp +ookuwa.nagano.jp +ooshika.nagano.jp +otaki.nagano.jp +otari.nagano.jp +sakae.nagano.jp +sakaki.nagano.jp +saku.nagano.jp +sakuho.nagano.jp +shimosuwa.nagano.jp +shinanomachi.nagano.jp +shiojiri.nagano.jp +suwa.nagano.jp +suzaka.nagano.jp +takagi.nagano.jp +takamori.nagano.jp +takayama.nagano.jp +tateshina.nagano.jp +tatsuno.nagano.jp +togakushi.nagano.jp +togura.nagano.jp +tomi.nagano.jp +ueda.nagano.jp +wada.nagano.jp +yamagata.nagano.jp +yamanouchi.nagano.jp +yasaka.nagano.jp +yasuoka.nagano.jp +chijiwa.nagasaki.jp +futsu.nagasaki.jp +goto.nagasaki.jp +hasami.nagasaki.jp +hirado.nagasaki.jp +iki.nagasaki.jp +isahaya.nagasaki.jp +kawatana.nagasaki.jp +kuchinotsu.nagasaki.jp +matsuura.nagasaki.jp +nagasaki.nagasaki.jp +obama.nagasaki.jp +omura.nagasaki.jp +oseto.nagasaki.jp +saikai.nagasaki.jp +sasebo.nagasaki.jp +seihi.nagasaki.jp +shimabara.nagasaki.jp +shinkamigoto.nagasaki.jp +togitsu.nagasaki.jp +tsushima.nagasaki.jp +unzen.nagasaki.jp +ando.nara.jp +gose.nara.jp +heguri.nara.jp +higashiyoshino.nara.jp +ikaruga.nara.jp +ikoma.nara.jp +kamikitayama.nara.jp +kanmaki.nara.jp +kashiba.nara.jp +kashihara.nara.jp +katsuragi.nara.jp +kawai.nara.jp +kawakami.nara.jp +kawanishi.nara.jp +koryo.nara.jp +kurotaki.nara.jp +mitsue.nara.jp +miyake.nara.jp +nara.nara.jp +nosegawa.nara.jp +oji.nara.jp +ouda.nara.jp +oyodo.nara.jp +sakurai.nara.jp +sango.nara.jp +shimoichi.nara.jp +shimokitayama.nara.jp +shinjo.nara.jp +soni.nara.jp +takatori.nara.jp +tawaramoto.nara.jp +tenkawa.nara.jp +tenri.nara.jp +uda.nara.jp +yamatokoriyama.nara.jp +yamatotakada.nara.jp +yamazoe.nara.jp +yoshino.nara.jp +aga.niigata.jp +agano.niigata.jp +gosen.niigata.jp +itoigawa.niigata.jp +izumozaki.niigata.jp +joetsu.niigata.jp +kamo.niigata.jp +kariwa.niigata.jp +kashiwazaki.niigata.jp +minamiuonuma.niigata.jp +mitsuke.niigata.jp +muika.niigata.jp +murakami.niigata.jp +myoko.niigata.jp +nagaoka.niigata.jp +niigata.niigata.jp +ojiya.niigata.jp +omi.niigata.jp +sado.niigata.jp +sanjo.niigata.jp +seiro.niigata.jp +seirou.niigata.jp +sekikawa.niigata.jp +shibata.niigata.jp +tagami.niigata.jp +tainai.niigata.jp +tochio.niigata.jp +tokamachi.niigata.jp +tsubame.niigata.jp +tsunan.niigata.jp +uonuma.niigata.jp +yahiko.niigata.jp +yoita.niigata.jp +yuzawa.niigata.jp +beppu.oita.jp +bungoono.oita.jp +bungotakada.oita.jp +hasama.oita.jp +hiji.oita.jp +himeshima.oita.jp +hita.oita.jp +kamitsue.oita.jp +kokonoe.oita.jp +kuju.oita.jp +kunisaki.oita.jp +kusu.oita.jp +oita.oita.jp +saiki.oita.jp +taketa.oita.jp +tsukumi.oita.jp +usa.oita.jp +usuki.oita.jp +yufu.oita.jp +akaiwa.okayama.jp +asakuchi.okayama.jp +bizen.okayama.jp +hayashima.okayama.jp +ibara.okayama.jp +kagamino.okayama.jp +kasaoka.okayama.jp +kibichuo.okayama.jp +kumenan.okayama.jp +kurashiki.okayama.jp +maniwa.okayama.jp +misaki.okayama.jp +nagi.okayama.jp +niimi.okayama.jp +nishiawakura.okayama.jp +okayama.okayama.jp +satosho.okayama.jp +setouchi.okayama.jp +shinjo.okayama.jp +shoo.okayama.jp +soja.okayama.jp +takahashi.okayama.jp +tamano.okayama.jp +tsuyama.okayama.jp +wake.okayama.jp +yakage.okayama.jp +aguni.okinawa.jp +ginowan.okinawa.jp +ginoza.okinawa.jp +gushikami.okinawa.jp +haebaru.okinawa.jp +higashi.okinawa.jp +hirara.okinawa.jp +iheya.okinawa.jp +ishigaki.okinawa.jp +ishikawa.okinawa.jp +itoman.okinawa.jp +izena.okinawa.jp +kadena.okinawa.jp +kin.okinawa.jp +kitadaito.okinawa.jp +kitanakagusuku.okinawa.jp +kumejima.okinawa.jp +kunigami.okinawa.jp +minamidaito.okinawa.jp +motobu.okinawa.jp +nago.okinawa.jp +naha.okinawa.jp +nakagusuku.okinawa.jp +nakijin.okinawa.jp +nanjo.okinawa.jp +nishihara.okinawa.jp +ogimi.okinawa.jp +okinawa.okinawa.jp +onna.okinawa.jp +shimoji.okinawa.jp +taketomi.okinawa.jp +tarama.okinawa.jp +tokashiki.okinawa.jp +tomigusuku.okinawa.jp +tonaki.okinawa.jp +urasoe.okinawa.jp +uruma.okinawa.jp +yaese.okinawa.jp +yomitan.okinawa.jp +yonabaru.okinawa.jp +yonaguni.okinawa.jp +zamami.okinawa.jp +abeno.osaka.jp +chihayaakasaka.osaka.jp +chuo.osaka.jp +daito.osaka.jp +fujiidera.osaka.jp +habikino.osaka.jp +hannan.osaka.jp +higashiosaka.osaka.jp +higashisumiyoshi.osaka.jp +higashiyodogawa.osaka.jp +hirakata.osaka.jp +ibaraki.osaka.jp +ikeda.osaka.jp +izumi.osaka.jp +izumiotsu.osaka.jp +izumisano.osaka.jp +kadoma.osaka.jp +kaizuka.osaka.jp +kanan.osaka.jp +kashiwara.osaka.jp +katano.osaka.jp +kawachinagano.osaka.jp +kishiwada.osaka.jp +kita.osaka.jp +kumatori.osaka.jp +matsubara.osaka.jp +minato.osaka.jp +minoh.osaka.jp +misaki.osaka.jp +moriguchi.osaka.jp +neyagawa.osaka.jp +nishi.osaka.jp +nose.osaka.jp +osakasayama.osaka.jp +sakai.osaka.jp +sayama.osaka.jp +sennan.osaka.jp +settsu.osaka.jp +shijonawate.osaka.jp +shimamoto.osaka.jp +suita.osaka.jp +tadaoka.osaka.jp +taishi.osaka.jp +tajiri.osaka.jp +takaishi.osaka.jp +takatsuki.osaka.jp +tondabayashi.osaka.jp +toyonaka.osaka.jp +toyono.osaka.jp +yao.osaka.jp +ariake.saga.jp +arita.saga.jp +fukudomi.saga.jp +genkai.saga.jp +hamatama.saga.jp +hizen.saga.jp +imari.saga.jp +kamimine.saga.jp +kanzaki.saga.jp +karatsu.saga.jp +kashima.saga.jp +kitagata.saga.jp +kitahata.saga.jp +kiyama.saga.jp +kouhoku.saga.jp +kyuragi.saga.jp +nishiarita.saga.jp +ogi.saga.jp +omachi.saga.jp +ouchi.saga.jp +saga.saga.jp +shiroishi.saga.jp +taku.saga.jp +tara.saga.jp +tosu.saga.jp +yoshinogari.saga.jp +arakawa.saitama.jp +asaka.saitama.jp +chichibu.saitama.jp +fujimi.saitama.jp +fujimino.saitama.jp +fukaya.saitama.jp +hanno.saitama.jp +hanyu.saitama.jp +hasuda.saitama.jp +hatogaya.saitama.jp +hatoyama.saitama.jp +hidaka.saitama.jp +higashichichibu.saitama.jp +higashimatsuyama.saitama.jp +honjo.saitama.jp +ina.saitama.jp +iruma.saitama.jp +iwatsuki.saitama.jp +kamiizumi.saitama.jp +kamikawa.saitama.jp +kamisato.saitama.jp +kasukabe.saitama.jp +kawagoe.saitama.jp +kawaguchi.saitama.jp +kawajima.saitama.jp +kazo.saitama.jp +kitamoto.saitama.jp +koshigaya.saitama.jp +kounosu.saitama.jp +kuki.saitama.jp +kumagaya.saitama.jp +matsubushi.saitama.jp +minano.saitama.jp +misato.saitama.jp +miyashiro.saitama.jp +miyoshi.saitama.jp +moroyama.saitama.jp +nagatoro.saitama.jp +namegawa.saitama.jp +niiza.saitama.jp +ogano.saitama.jp +ogawa.saitama.jp +ogose.saitama.jp +okegawa.saitama.jp +omiya.saitama.jp +otaki.saitama.jp +ranzan.saitama.jp +ryokami.saitama.jp +saitama.saitama.jp +sakado.saitama.jp +satte.saitama.jp +sayama.saitama.jp +shiki.saitama.jp +shiraoka.saitama.jp +soka.saitama.jp +sugito.saitama.jp +toda.saitama.jp +tokigawa.saitama.jp +tokorozawa.saitama.jp +tsurugashima.saitama.jp +urawa.saitama.jp +warabi.saitama.jp +yashio.saitama.jp +yokoze.saitama.jp +yono.saitama.jp +yorii.saitama.jp +yoshida.saitama.jp +yoshikawa.saitama.jp +yoshimi.saitama.jp +aisho.shiga.jp +gamo.shiga.jp +higashiomi.shiga.jp +hikone.shiga.jp +koka.shiga.jp +konan.shiga.jp +kosei.shiga.jp +koto.shiga.jp +kusatsu.shiga.jp +maibara.shiga.jp +moriyama.shiga.jp +nagahama.shiga.jp +nishiazai.shiga.jp +notogawa.shiga.jp +omihachiman.shiga.jp +otsu.shiga.jp +ritto.shiga.jp +ryuoh.shiga.jp +takashima.shiga.jp +takatsuki.shiga.jp +torahime.shiga.jp +toyosato.shiga.jp +yasu.shiga.jp +akagi.shimane.jp +ama.shimane.jp +gotsu.shimane.jp +hamada.shimane.jp +higashiizumo.shimane.jp +hikawa.shimane.jp +hikimi.shimane.jp +izumo.shimane.jp +kakinoki.shimane.jp +masuda.shimane.jp +matsue.shimane.jp +misato.shimane.jp +nishinoshima.shimane.jp +ohda.shimane.jp +okinoshima.shimane.jp +okuizumo.shimane.jp +shimane.shimane.jp +tamayu.shimane.jp +tsuwano.shimane.jp +unnan.shimane.jp +yakumo.shimane.jp +yasugi.shimane.jp +yatsuka.shimane.jp +arai.shizuoka.jp +atami.shizuoka.jp +fuji.shizuoka.jp +fujieda.shizuoka.jp +fujikawa.shizuoka.jp +fujinomiya.shizuoka.jp +fukuroi.shizuoka.jp +gotemba.shizuoka.jp +haibara.shizuoka.jp +hamamatsu.shizuoka.jp +higashiizu.shizuoka.jp +ito.shizuoka.jp +iwata.shizuoka.jp +izu.shizuoka.jp +izunokuni.shizuoka.jp +kakegawa.shizuoka.jp +kannami.shizuoka.jp +kawanehon.shizuoka.jp +kawazu.shizuoka.jp +kikugawa.shizuoka.jp +kosai.shizuoka.jp +makinohara.shizuoka.jp +matsuzaki.shizuoka.jp +minamiizu.shizuoka.jp +mishima.shizuoka.jp +morimachi.shizuoka.jp +nishiizu.shizuoka.jp +numazu.shizuoka.jp +omaezaki.shizuoka.jp +shimada.shizuoka.jp +shimizu.shizuoka.jp +shimoda.shizuoka.jp +shizuoka.shizuoka.jp +susono.shizuoka.jp +yaizu.shizuoka.jp +yoshida.shizuoka.jp +ashikaga.tochigi.jp +bato.tochigi.jp +haga.tochigi.jp +ichikai.tochigi.jp +iwafune.tochigi.jp +kaminokawa.tochigi.jp +kanuma.tochigi.jp +karasuyama.tochigi.jp +kuroiso.tochigi.jp +mashiko.tochigi.jp +mibu.tochigi.jp +moka.tochigi.jp +motegi.tochigi.jp +nasu.tochigi.jp +nasushiobara.tochigi.jp +nikko.tochigi.jp +nishikata.tochigi.jp +nogi.tochigi.jp +ohira.tochigi.jp +ohtawara.tochigi.jp +oyama.tochigi.jp +sakura.tochigi.jp +sano.tochigi.jp +shimotsuke.tochigi.jp +shioya.tochigi.jp +takanezawa.tochigi.jp +tochigi.tochigi.jp +tsuga.tochigi.jp +ujiie.tochigi.jp +utsunomiya.tochigi.jp +yaita.tochigi.jp +aizumi.tokushima.jp +anan.tokushima.jp +ichiba.tokushima.jp +itano.tokushima.jp +kainan.tokushima.jp +komatsushima.tokushima.jp +matsushige.tokushima.jp +mima.tokushima.jp +minami.tokushima.jp +miyoshi.tokushima.jp +mugi.tokushima.jp +nakagawa.tokushima.jp +naruto.tokushima.jp +sanagochi.tokushima.jp +shishikui.tokushima.jp +tokushima.tokushima.jp +wajiki.tokushima.jp +adachi.tokyo.jp +akiruno.tokyo.jp +akishima.tokyo.jp +aogashima.tokyo.jp +arakawa.tokyo.jp +bunkyo.tokyo.jp +chiyoda.tokyo.jp +chofu.tokyo.jp +chuo.tokyo.jp +edogawa.tokyo.jp +fuchu.tokyo.jp +fussa.tokyo.jp +hachijo.tokyo.jp +hachioji.tokyo.jp +hamura.tokyo.jp +higashikurume.tokyo.jp +higashimurayama.tokyo.jp +higashiyamato.tokyo.jp +hino.tokyo.jp +hinode.tokyo.jp +hinohara.tokyo.jp +inagi.tokyo.jp +itabashi.tokyo.jp +katsushika.tokyo.jp +kita.tokyo.jp +kiyose.tokyo.jp +kodaira.tokyo.jp +koganei.tokyo.jp +kokubunji.tokyo.jp +komae.tokyo.jp +koto.tokyo.jp +kouzushima.tokyo.jp +kunitachi.tokyo.jp +machida.tokyo.jp +meguro.tokyo.jp +minato.tokyo.jp +mitaka.tokyo.jp +mizuho.tokyo.jp +musashimurayama.tokyo.jp +musashino.tokyo.jp +nakano.tokyo.jp +nerima.tokyo.jp +ogasawara.tokyo.jp +okutama.tokyo.jp +ome.tokyo.jp +oshima.tokyo.jp +ota.tokyo.jp +setagaya.tokyo.jp +shibuya.tokyo.jp +shinagawa.tokyo.jp +shinjuku.tokyo.jp +suginami.tokyo.jp +sumida.tokyo.jp +tachikawa.tokyo.jp +taito.tokyo.jp +tama.tokyo.jp +toshima.tokyo.jp +chizu.tottori.jp +hino.tottori.jp +kawahara.tottori.jp +koge.tottori.jp +kotoura.tottori.jp +misasa.tottori.jp +nanbu.tottori.jp +nichinan.tottori.jp +sakaiminato.tottori.jp +tottori.tottori.jp +wakasa.tottori.jp +yazu.tottori.jp +yonago.tottori.jp +asahi.toyama.jp +fuchu.toyama.jp +fukumitsu.toyama.jp +funahashi.toyama.jp +himi.toyama.jp +imizu.toyama.jp +inami.toyama.jp +johana.toyama.jp +kamiichi.toyama.jp +kurobe.toyama.jp +nakaniikawa.toyama.jp +namerikawa.toyama.jp +nanto.toyama.jp +nyuzen.toyama.jp +oyabe.toyama.jp +taira.toyama.jp +takaoka.toyama.jp +tateyama.toyama.jp +toga.toyama.jp +tonami.toyama.jp +toyama.toyama.jp +unazuki.toyama.jp +uozu.toyama.jp +yamada.toyama.jp +arida.wakayama.jp +aridagawa.wakayama.jp +gobo.wakayama.jp +hashimoto.wakayama.jp +hidaka.wakayama.jp +hirogawa.wakayama.jp +inami.wakayama.jp +iwade.wakayama.jp +kainan.wakayama.jp +kamitonda.wakayama.jp +katsuragi.wakayama.jp +kimino.wakayama.jp +kinokawa.wakayama.jp +kitayama.wakayama.jp +koya.wakayama.jp +koza.wakayama.jp +kozagawa.wakayama.jp +kudoyama.wakayama.jp +kushimoto.wakayama.jp +mihama.wakayama.jp +misato.wakayama.jp +nachikatsuura.wakayama.jp +shingu.wakayama.jp +shirahama.wakayama.jp +taiji.wakayama.jp +tanabe.wakayama.jp +wakayama.wakayama.jp +yuasa.wakayama.jp +yura.wakayama.jp +asahi.yamagata.jp +funagata.yamagata.jp +higashine.yamagata.jp +iide.yamagata.jp +kahoku.yamagata.jp +kaminoyama.yamagata.jp +kaneyama.yamagata.jp +kawanishi.yamagata.jp +mamurogawa.yamagata.jp +mikawa.yamagata.jp +murayama.yamagata.jp +nagai.yamagata.jp +nakayama.yamagata.jp +nanyo.yamagata.jp +nishikawa.yamagata.jp +obanazawa.yamagata.jp +oe.yamagata.jp +oguni.yamagata.jp +ohkura.yamagata.jp +oishida.yamagata.jp +sagae.yamagata.jp +sakata.yamagata.jp +sakegawa.yamagata.jp +shinjo.yamagata.jp +shirataka.yamagata.jp +shonai.yamagata.jp +takahata.yamagata.jp +tendo.yamagata.jp +tozawa.yamagata.jp +tsuruoka.yamagata.jp +yamagata.yamagata.jp +yamanobe.yamagata.jp +yonezawa.yamagata.jp +yuza.yamagata.jp +abu.yamaguchi.jp +hagi.yamaguchi.jp +hikari.yamaguchi.jp +hofu.yamaguchi.jp +iwakuni.yamaguchi.jp +kudamatsu.yamaguchi.jp +mitou.yamaguchi.jp +nagato.yamaguchi.jp +oshima.yamaguchi.jp +shimonoseki.yamaguchi.jp +shunan.yamaguchi.jp +tabuse.yamaguchi.jp +tokuyama.yamaguchi.jp +toyota.yamaguchi.jp +ube.yamaguchi.jp +yuu.yamaguchi.jp +chuo.yamanashi.jp +doshi.yamanashi.jp +fuefuki.yamanashi.jp +fujikawa.yamanashi.jp +fujikawaguchiko.yamanashi.jp +fujiyoshida.yamanashi.jp +hayakawa.yamanashi.jp +hokuto.yamanashi.jp +ichikawamisato.yamanashi.jp +kai.yamanashi.jp +kofu.yamanashi.jp +koshu.yamanashi.jp +kosuge.yamanashi.jp +minami-alps.yamanashi.jp +minobu.yamanashi.jp +nakamichi.yamanashi.jp +nanbu.yamanashi.jp +narusawa.yamanashi.jp +nirasaki.yamanashi.jp +nishikatsura.yamanashi.jp +oshino.yamanashi.jp +otsuki.yamanashi.jp +showa.yamanashi.jp +tabayama.yamanashi.jp +tsuru.yamanashi.jp +uenohara.yamanashi.jp +yamanakako.yamanashi.jp +yamanashi.yamanashi.jp + +// ke : http://www.kenic.or.ke/index.php/en/ke-domains/ke-domains +ke +ac.ke +co.ke +go.ke +info.ke +me.ke +mobi.ke +ne.ke +or.ke +sc.ke + +// kg : http://www.domain.kg/dmn_n.html +kg +org.kg +net.kg +com.kg +edu.kg +gov.kg +mil.kg + +// kh : http://www.mptc.gov.kh/dns_registration.htm +*.kh + +// ki : http://www.ki/dns/index.html +ki +edu.ki +biz.ki +net.ki +org.ki +gov.ki +info.ki +com.ki + +// km : https://en.wikipedia.org/wiki/.km +// http://www.domaine.km/documents/charte.doc +km +org.km +nom.km +gov.km +prd.km +tm.km +edu.km +mil.km +ass.km +com.km +// These are only mentioned as proposed suggestions at domaine.km, but +// https://en.wikipedia.org/wiki/.km says they're available for registration: +coop.km +asso.km +presse.km +medecin.km +notaires.km +pharmaciens.km +veterinaire.km +gouv.km + +// kn : https://en.wikipedia.org/wiki/.kn +// http://www.dot.kn/domainRules.html +kn +net.kn +org.kn +edu.kn +gov.kn + +// kp : http://www.kcce.kp/en_index.php +kp +com.kp +edu.kp +gov.kp +org.kp +rep.kp +tra.kp + +// kr : https://en.wikipedia.org/wiki/.kr +// see also: http://domain.nida.or.kr/eng/registration.jsp +kr +ac.kr +co.kr +es.kr +go.kr +hs.kr +kg.kr +mil.kr +ms.kr +ne.kr +or.kr +pe.kr +re.kr +sc.kr +// kr geographical names +busan.kr +chungbuk.kr +chungnam.kr +daegu.kr +daejeon.kr +gangwon.kr +gwangju.kr +gyeongbuk.kr +gyeonggi.kr +gyeongnam.kr +incheon.kr +jeju.kr +jeonbuk.kr +jeonnam.kr +seoul.kr +ulsan.kr + +// kw : https://www.nic.kw/policies/ +// Confirmed by registry +kw +com.kw +edu.kw +emb.kw +gov.kw +ind.kw +net.kw +org.kw + +// ky : http://www.icta.ky/da_ky_reg_dom.php +// Confirmed by registry 2008-06-17 +ky +edu.ky +gov.ky +com.ky +org.ky +net.ky + +// kz : https://en.wikipedia.org/wiki/.kz +// see also: http://www.nic.kz/rules/index.jsp +kz +org.kz +edu.kz +net.kz +gov.kz +mil.kz +com.kz + +// la : https://en.wikipedia.org/wiki/.la +// Submitted by registry +la +int.la +net.la +info.la +edu.la +gov.la +per.la +com.la +org.la + +// lb : https://en.wikipedia.org/wiki/.lb +// Submitted by registry +lb +com.lb +edu.lb +gov.lb +net.lb +org.lb + +// lc : https://en.wikipedia.org/wiki/.lc +// see also: http://www.nic.lc/rules.htm +lc +com.lc +net.lc +co.lc +org.lc +edu.lc +gov.lc + +// li : https://en.wikipedia.org/wiki/.li +li + +// lk : https://www.nic.lk/index.php/domain-registration/lk-domain-naming-structure +lk +gov.lk +sch.lk +net.lk +int.lk +com.lk +org.lk +edu.lk +ngo.lk +soc.lk +web.lk +ltd.lk +assn.lk +grp.lk +hotel.lk +ac.lk + +// lr : http://psg.com/dns/lr/lr.txt +// Submitted by registry +lr +com.lr +edu.lr +gov.lr +org.lr +net.lr + +// ls : http://www.nic.ls/ +// Confirmed by registry +ls +ac.ls +biz.ls +co.ls +edu.ls +gov.ls +info.ls +net.ls +org.ls +sc.ls + +// lt : https://en.wikipedia.org/wiki/.lt +lt +// gov.lt : http://www.gov.lt/index_en.php +gov.lt + +// lu : http://www.dns.lu/en/ +lu + +// lv : http://www.nic.lv/DNS/En/generic.php +lv +com.lv +edu.lv +gov.lv +org.lv +mil.lv +id.lv +net.lv +asn.lv +conf.lv + +// ly : http://www.nic.ly/regulations.php +ly +com.ly +net.ly +gov.ly +plc.ly +edu.ly +sch.ly +med.ly +org.ly +id.ly + +// ma : https://en.wikipedia.org/wiki/.ma +// http://www.anrt.ma/fr/admin/download/upload/file_fr782.pdf +ma +co.ma +net.ma +gov.ma +org.ma +ac.ma +press.ma + +// mc : http://www.nic.mc/ +mc +tm.mc +asso.mc + +// md : https://en.wikipedia.org/wiki/.md +md + +// me : https://en.wikipedia.org/wiki/.me +me +co.me +net.me +org.me +edu.me +ac.me +gov.me +its.me +priv.me + +// mg : http://nic.mg/nicmg/?page_id=39 +mg +org.mg +nom.mg +gov.mg +prd.mg +tm.mg +edu.mg +mil.mg +com.mg +co.mg + +// mh : https://en.wikipedia.org/wiki/.mh +mh + +// mil : https://en.wikipedia.org/wiki/.mil +mil + +// mk : https://en.wikipedia.org/wiki/.mk +// see also: http://dns.marnet.net.mk/postapka.php +mk +com.mk +org.mk +net.mk +edu.mk +gov.mk +inf.mk +name.mk + +// ml : http://www.gobin.info/domainname/ml-template.doc +// see also: https://en.wikipedia.org/wiki/.ml +ml +com.ml +edu.ml +gouv.ml +gov.ml +net.ml +org.ml +presse.ml + +// mm : https://en.wikipedia.org/wiki/.mm +*.mm + +// mn : https://en.wikipedia.org/wiki/.mn +mn +gov.mn +edu.mn +org.mn + +// mo : http://www.monic.net.mo/ +mo +com.mo +net.mo +org.mo +edu.mo +gov.mo + +// mobi : https://en.wikipedia.org/wiki/.mobi +mobi + +// mp : http://www.dot.mp/ +// Confirmed by registry 2008-06-17 +mp + +// mq : https://en.wikipedia.org/wiki/.mq +mq + +// mr : https://en.wikipedia.org/wiki/.mr +mr +gov.mr + +// ms : http://www.nic.ms/pdf/MS_Domain_Name_Rules.pdf +ms +com.ms +edu.ms +gov.ms +net.ms +org.ms + +// mt : https://www.nic.org.mt/go/policy +// Submitted by registry +mt +com.mt +edu.mt +net.mt +org.mt + +// mu : https://en.wikipedia.org/wiki/.mu +mu +com.mu +net.mu +org.mu +gov.mu +ac.mu +co.mu +or.mu + +// museum : http://about.museum/naming/ +// http://index.museum/ +museum +academy.museum +agriculture.museum +air.museum +airguard.museum +alabama.museum +alaska.museum +amber.museum +ambulance.museum +american.museum +americana.museum +americanantiques.museum +americanart.museum +amsterdam.museum +and.museum +annefrank.museum +anthro.museum +anthropology.museum +antiques.museum +aquarium.museum +arboretum.museum +archaeological.museum +archaeology.museum +architecture.museum +art.museum +artanddesign.museum +artcenter.museum +artdeco.museum +arteducation.museum +artgallery.museum +arts.museum +artsandcrafts.museum +asmatart.museum +assassination.museum +assisi.museum +association.museum +astronomy.museum +atlanta.museum +austin.museum +australia.museum +automotive.museum +aviation.museum +axis.museum +badajoz.museum +baghdad.museum +bahn.museum +bale.museum +baltimore.museum +barcelona.museum +baseball.museum +basel.museum +baths.museum +bauern.museum +beauxarts.museum +beeldengeluid.museum +bellevue.museum +bergbau.museum +berkeley.museum +berlin.museum +bern.museum +bible.museum +bilbao.museum +bill.museum +birdart.museum +birthplace.museum +bonn.museum +boston.museum +botanical.museum +botanicalgarden.museum +botanicgarden.museum +botany.museum +brandywinevalley.museum +brasil.museum +bristol.museum +british.museum +britishcolumbia.museum +broadcast.museum +brunel.museum +brussel.museum +brussels.museum +bruxelles.museum +building.museum +burghof.museum +bus.museum +bushey.museum +cadaques.museum +california.museum +cambridge.museum +can.museum +canada.museum +capebreton.museum +carrier.museum +cartoonart.museum +casadelamoneda.museum +castle.museum +castres.museum +celtic.museum +center.museum +chattanooga.museum +cheltenham.museum +chesapeakebay.museum +chicago.museum +children.museum +childrens.museum +childrensgarden.museum +chiropractic.museum +chocolate.museum +christiansburg.museum +cincinnati.museum +cinema.museum +circus.museum +civilisation.museum +civilization.museum +civilwar.museum +clinton.museum +clock.museum +coal.museum +coastaldefence.museum +cody.museum +coldwar.museum +collection.museum +colonialwilliamsburg.museum +coloradoplateau.museum +columbia.museum +columbus.museum +communication.museum +communications.museum +community.museum +computer.museum +computerhistory.museum +comunicações.museum +contemporary.museum +contemporaryart.museum +convent.museum +copenhagen.museum +corporation.museum +correios-e-telecomunicações.museum +corvette.museum +costume.museum +countryestate.museum +county.museum +crafts.museum +cranbrook.museum +creation.museum +cultural.museum +culturalcenter.museum +culture.museum +cyber.museum +cymru.museum +dali.museum +dallas.museum +database.museum +ddr.museum +decorativearts.museum +delaware.museum +delmenhorst.museum +denmark.museum +depot.museum +design.museum +detroit.museum +dinosaur.museum +discovery.museum +dolls.museum +donostia.museum +durham.museum +eastafrica.museum +eastcoast.museum +education.museum +educational.museum +egyptian.museum +eisenbahn.museum +elburg.museum +elvendrell.museum +embroidery.museum +encyclopedic.museum +england.museum +entomology.museum +environment.museum +environmentalconservation.museum +epilepsy.museum +essex.museum +estate.museum +ethnology.museum +exeter.museum +exhibition.museum +family.museum +farm.museum +farmequipment.museum +farmers.museum +farmstead.museum +field.museum +figueres.museum +filatelia.museum +film.museum +fineart.museum +finearts.museum +finland.museum +flanders.museum +florida.museum +force.museum +fortmissoula.museum +fortworth.museum +foundation.museum +francaise.museum +frankfurt.museum +franziskaner.museum +freemasonry.museum +freiburg.museum +fribourg.museum +frog.museum +fundacio.museum +furniture.museum +gallery.museum +garden.museum +gateway.museum +geelvinck.museum +gemological.museum +geology.museum +georgia.museum +giessen.museum +glas.museum +glass.museum +gorge.museum +grandrapids.museum +graz.museum +guernsey.museum +halloffame.museum +hamburg.museum +handson.museum +harvestcelebration.museum +hawaii.museum +health.museum +heimatunduhren.museum +hellas.museum +helsinki.museum +hembygdsforbund.museum +heritage.museum +histoire.museum +historical.museum +historicalsociety.museum +historichouses.museum +historisch.museum +historisches.museum +history.museum +historyofscience.museum +horology.museum +house.museum +humanities.museum +illustration.museum +imageandsound.museum +indian.museum +indiana.museum +indianapolis.museum +indianmarket.museum +intelligence.museum +interactive.museum +iraq.museum +iron.museum +isleofman.museum +jamison.museum +jefferson.museum +jerusalem.museum +jewelry.museum +jewish.museum +jewishart.museum +jfk.museum +journalism.museum +judaica.museum +judygarland.museum +juedisches.museum +juif.museum +karate.museum +karikatur.museum +kids.museum +koebenhavn.museum +koeln.museum +kunst.museum +kunstsammlung.museum +kunstunddesign.museum +labor.museum +labour.museum +lajolla.museum +lancashire.museum +landes.museum +lans.museum +läns.museum +larsson.museum +lewismiller.museum +lincoln.museum +linz.museum +living.museum +livinghistory.museum +localhistory.museum +london.museum +losangeles.museum +louvre.museum +loyalist.museum +lucerne.museum +luxembourg.museum +luzern.museum +mad.museum +madrid.museum +mallorca.museum +manchester.museum +mansion.museum +mansions.museum +manx.museum +marburg.museum +maritime.museum +maritimo.museum +maryland.museum +marylhurst.museum +media.museum +medical.museum +medizinhistorisches.museum +meeres.museum +memorial.museum +mesaverde.museum +michigan.museum +midatlantic.museum +military.museum +mill.museum +miners.museum +mining.museum +minnesota.museum +missile.museum +missoula.museum +modern.museum +moma.museum +money.museum +monmouth.museum +monticello.museum +montreal.museum +moscow.museum +motorcycle.museum +muenchen.museum +muenster.museum +mulhouse.museum +muncie.museum +museet.museum +museumcenter.museum +museumvereniging.museum +music.museum +national.museum +nationalfirearms.museum +nationalheritage.museum +nativeamerican.museum +naturalhistory.museum +naturalhistorymuseum.museum +naturalsciences.museum +nature.museum +naturhistorisches.museum +natuurwetenschappen.museum +naumburg.museum +naval.museum +nebraska.museum +neues.museum +newhampshire.museum +newjersey.museum +newmexico.museum +newport.museum +newspaper.museum +newyork.museum +niepce.museum +norfolk.museum +north.museum +nrw.museum +nyc.museum +nyny.museum +oceanographic.museum +oceanographique.museum +omaha.museum +online.museum +ontario.museum +openair.museum +oregon.museum +oregontrail.museum +otago.museum +oxford.museum +pacific.museum +paderborn.museum +palace.museum +paleo.museum +palmsprings.museum +panama.museum +paris.museum +pasadena.museum +pharmacy.museum +philadelphia.museum +philadelphiaarea.museum +philately.museum +phoenix.museum +photography.museum +pilots.museum +pittsburgh.museum +planetarium.museum +plantation.museum +plants.museum +plaza.museum +portal.museum +portland.museum +portlligat.museum +posts-and-telecommunications.museum +preservation.museum +presidio.museum +press.museum +project.museum +public.museum +pubol.museum +quebec.museum +railroad.museum +railway.museum +research.museum +resistance.museum +riodejaneiro.museum +rochester.museum +rockart.museum +roma.museum +russia.museum +saintlouis.museum +salem.museum +salvadordali.museum +salzburg.museum +sandiego.museum +sanfrancisco.museum +santabarbara.museum +santacruz.museum +santafe.museum +saskatchewan.museum +satx.museum +savannahga.museum +schlesisches.museum +schoenbrunn.museum +schokoladen.museum +school.museum +schweiz.museum +science.museum +scienceandhistory.museum +scienceandindustry.museum +sciencecenter.museum +sciencecenters.museum +science-fiction.museum +sciencehistory.museum +sciences.museum +sciencesnaturelles.museum +scotland.museum +seaport.museum +settlement.museum +settlers.museum +shell.museum +sherbrooke.museum +sibenik.museum +silk.museum +ski.museum +skole.museum +society.museum +sologne.museum +soundandvision.museum +southcarolina.museum +southwest.museum +space.museum +spy.museum +square.museum +stadt.museum +stalbans.museum +starnberg.museum +state.museum +stateofdelaware.museum +station.museum +steam.museum +steiermark.museum +stjohn.museum +stockholm.museum +stpetersburg.museum +stuttgart.museum +suisse.museum +surgeonshall.museum +surrey.museum +svizzera.museum +sweden.museum +sydney.museum +tank.museum +tcm.museum +technology.museum +telekommunikation.museum +television.museum +texas.museum +textile.museum +theater.museum +time.museum +timekeeping.museum +topology.museum +torino.museum +touch.museum +town.museum +transport.museum +tree.museum +trolley.museum +trust.museum +trustee.museum +uhren.museum +ulm.museum +undersea.museum +university.museum +usa.museum +usantiques.museum +usarts.museum +uscountryestate.museum +usculture.museum +usdecorativearts.museum +usgarden.museum +ushistory.museum +ushuaia.museum +uslivinghistory.museum +utah.museum +uvic.museum +valley.museum +vantaa.museum +versailles.museum +viking.museum +village.museum +virginia.museum +virtual.museum +virtuel.museum +vlaanderen.museum +volkenkunde.museum +wales.museum +wallonie.museum +war.museum +washingtondc.museum +watchandclock.museum +watch-and-clock.museum +western.museum +westfalen.museum +whaling.museum +wildlife.museum +williamsburg.museum +windmill.museum +workshop.museum +york.museum +yorkshire.museum +yosemite.museum +youth.museum +zoological.museum +zoology.museum +ירושלים.museum +иком.museum + +// mv : https://en.wikipedia.org/wiki/.mv +// "mv" included because, contra Wikipedia, google.mv exists. +mv +aero.mv +biz.mv +com.mv +coop.mv +edu.mv +gov.mv +info.mv +int.mv +mil.mv +museum.mv +name.mv +net.mv +org.mv +pro.mv + +// mw : http://www.registrar.mw/ +mw +ac.mw +biz.mw +co.mw +com.mw +coop.mw +edu.mw +gov.mw +int.mw +museum.mw +net.mw +org.mw + +// mx : http://www.nic.mx/ +// Submitted by registry +mx +com.mx +org.mx +gob.mx +edu.mx +net.mx + +// my : http://www.mynic.net.my/ +my +com.my +net.my +org.my +gov.my +edu.my +mil.my +name.my + +// mz : http://www.uem.mz/ +// Submitted by registry +mz +ac.mz +adv.mz +co.mz +edu.mz +gov.mz +mil.mz +net.mz +org.mz + +// na : http://www.na-nic.com.na/ +// http://www.info.na/domain/ +na +info.na +pro.na +name.na +school.na +or.na +dr.na +us.na +mx.na +ca.na +in.na +cc.na +tv.na +ws.na +mobi.na +co.na +com.na +org.na + +// name : has 2nd-level tlds, but there's no list of them +name + +// nc : http://www.cctld.nc/ +nc +asso.nc +nom.nc + +// ne : https://en.wikipedia.org/wiki/.ne +ne + +// net : https://en.wikipedia.org/wiki/.net +net + +// nf : https://en.wikipedia.org/wiki/.nf +nf +com.nf +net.nf +per.nf +rec.nf +web.nf +arts.nf +firm.nf +info.nf +other.nf +store.nf + +// ng : http://www.nira.org.ng/index.php/join-us/register-ng-domain/189-nira-slds +ng +com.ng +edu.ng +gov.ng +i.ng +mil.ng +mobi.ng +name.ng +net.ng +org.ng +sch.ng + +// ni : http://www.nic.ni/ +ni +ac.ni +biz.ni +co.ni +com.ni +edu.ni +gob.ni +in.ni +info.ni +int.ni +mil.ni +net.ni +nom.ni +org.ni +web.ni + +// nl : https://en.wikipedia.org/wiki/.nl +// https://www.sidn.nl/ +// ccTLD for the Netherlands +nl + +// no : https://www.norid.no/en/om-domenenavn/regelverk-for-no/ +// Norid geographical second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-b/ +// Norid category second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-c/ +// Norid category second-level domains managed by parties other than Norid : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-d/ +// RSS feed: https://teknisk.norid.no/en/feed/ +no +// Norid category second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-c/ +fhs.no +vgs.no +fylkesbibl.no +folkebibl.no +museum.no +idrett.no +priv.no +// Norid category second-level domains managed by parties other than Norid : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-d/ +mil.no +stat.no +dep.no +kommune.no +herad.no +// Norid geographical second level domains : https://www.norid.no/en/om-domenenavn/regelverk-for-no/vedlegg-b/ +// counties +aa.no +ah.no +bu.no +fm.no +hl.no +hm.no +jan-mayen.no +mr.no +nl.no +nt.no +of.no +ol.no +oslo.no +rl.no +sf.no +st.no +svalbard.no +tm.no +tr.no +va.no +vf.no +// primary and lower secondary schools per county +gs.aa.no +gs.ah.no +gs.bu.no +gs.fm.no +gs.hl.no +gs.hm.no +gs.jan-mayen.no +gs.mr.no +gs.nl.no +gs.nt.no +gs.of.no +gs.ol.no +gs.oslo.no +gs.rl.no +gs.sf.no +gs.st.no +gs.svalbard.no +gs.tm.no +gs.tr.no +gs.va.no +gs.vf.no +// cities +akrehamn.no +åkrehamn.no +algard.no +ålgård.no +arna.no +brumunddal.no +bryne.no +bronnoysund.no +brønnøysund.no +drobak.no +drøbak.no +egersund.no +fetsund.no +floro.no +florø.no +fredrikstad.no +hokksund.no +honefoss.no +hønefoss.no +jessheim.no +jorpeland.no +jørpeland.no +kirkenes.no +kopervik.no +krokstadelva.no +langevag.no +langevåg.no +leirvik.no +mjondalen.no +mjøndalen.no +mo-i-rana.no +mosjoen.no +mosjøen.no +nesoddtangen.no +orkanger.no +osoyro.no +osøyro.no +raholt.no +råholt.no +sandnessjoen.no +sandnessjøen.no +skedsmokorset.no +slattum.no +spjelkavik.no +stathelle.no +stavern.no +stjordalshalsen.no +stjørdalshalsen.no +tananger.no +tranby.no +vossevangen.no +// communities +afjord.no +åfjord.no +agdenes.no +al.no +ål.no +alesund.no +ålesund.no +alstahaug.no +alta.no +áltá.no +alaheadju.no +álaheadju.no +alvdal.no +amli.no +åmli.no +amot.no +åmot.no +andebu.no +andoy.no +andøy.no +andasuolo.no +ardal.no +årdal.no +aremark.no +arendal.no +ås.no +aseral.no +åseral.no +asker.no +askim.no +askvoll.no +askoy.no +askøy.no +asnes.no +åsnes.no +audnedaln.no +aukra.no +aure.no +aurland.no +aurskog-holand.no +aurskog-høland.no +austevoll.no +austrheim.no +averoy.no +averøy.no +balestrand.no +ballangen.no +balat.no +bálát.no +balsfjord.no +bahccavuotna.no +báhccavuotna.no +bamble.no +bardu.no +beardu.no +beiarn.no +bajddar.no +bájddar.no +baidar.no +báidár.no +berg.no +bergen.no +berlevag.no +berlevåg.no +bearalvahki.no +bearalváhki.no +bindal.no +birkenes.no +bjarkoy.no +bjarkøy.no +bjerkreim.no +bjugn.no +bodo.no +bodø.no +badaddja.no +bådåddjå.no +budejju.no +bokn.no +bremanger.no +bronnoy.no +brønnøy.no +bygland.no +bykle.no +barum.no +bærum.no +bo.telemark.no +bø.telemark.no +bo.nordland.no +bø.nordland.no +bievat.no +bievát.no +bomlo.no +bømlo.no +batsfjord.no +båtsfjord.no +bahcavuotna.no +báhcavuotna.no +dovre.no +drammen.no +drangedal.no +dyroy.no +dyrøy.no +donna.no +dønna.no +eid.no +eidfjord.no +eidsberg.no +eidskog.no +eidsvoll.no +eigersund.no +elverum.no +enebakk.no +engerdal.no +etne.no +etnedal.no +evenes.no +evenassi.no +evenášši.no +evje-og-hornnes.no +farsund.no +fauske.no +fuossko.no +fuoisku.no +fedje.no +fet.no +finnoy.no +finnøy.no +fitjar.no +fjaler.no +fjell.no +flakstad.no +flatanger.no +flekkefjord.no +flesberg.no +flora.no +fla.no +flå.no +folldal.no +forsand.no +fosnes.no +frei.no +frogn.no +froland.no +frosta.no +frana.no +fræna.no +froya.no +frøya.no +fusa.no +fyresdal.no +forde.no +førde.no +gamvik.no +gangaviika.no +gáŋgaviika.no +gaular.no +gausdal.no +gildeskal.no +gildeskål.no +giske.no +gjemnes.no +gjerdrum.no +gjerstad.no +gjesdal.no +gjovik.no +gjøvik.no +gloppen.no +gol.no +gran.no +grane.no +granvin.no +gratangen.no +grimstad.no +grong.no +kraanghke.no +kråanghke.no +grue.no +gulen.no +hadsel.no +halden.no +halsa.no +hamar.no +hamaroy.no +habmer.no +hábmer.no +hapmir.no +hápmir.no +hammerfest.no +hammarfeasta.no +hámmárfeasta.no +haram.no +hareid.no +harstad.no +hasvik.no +aknoluokta.no +ákŋoluokta.no +hattfjelldal.no +aarborte.no +haugesund.no +hemne.no +hemnes.no +hemsedal.no +heroy.more-og-romsdal.no +herøy.møre-og-romsdal.no +heroy.nordland.no +herøy.nordland.no +hitra.no +hjartdal.no +hjelmeland.no +hobol.no +hobøl.no +hof.no +hol.no +hole.no +holmestrand.no +holtalen.no +holtålen.no +hornindal.no +horten.no +hurdal.no +hurum.no +hvaler.no +hyllestad.no +hagebostad.no +hægebostad.no +hoyanger.no +høyanger.no +hoylandet.no +høylandet.no +ha.no +hå.no +ibestad.no +inderoy.no +inderøy.no +iveland.no +jevnaker.no +jondal.no +jolster.no +jølster.no +karasjok.no +karasjohka.no +kárášjohka.no +karlsoy.no +galsa.no +gálsá.no +karmoy.no +karmøy.no +kautokeino.no +guovdageaidnu.no +klepp.no +klabu.no +klæbu.no +kongsberg.no +kongsvinger.no +kragero.no +kragerø.no +kristiansand.no +kristiansund.no +krodsherad.no +krødsherad.no +kvalsund.no +rahkkeravju.no +ráhkkerávju.no +kvam.no +kvinesdal.no +kvinnherad.no +kviteseid.no +kvitsoy.no +kvitsøy.no +kvafjord.no +kvæfjord.no +giehtavuoatna.no +kvanangen.no +kvænangen.no +navuotna.no +návuotna.no +kafjord.no +kåfjord.no +gaivuotna.no +gáivuotna.no +larvik.no +lavangen.no +lavagis.no +loabat.no +loabát.no +lebesby.no +davvesiida.no +leikanger.no +leirfjord.no +leka.no +leksvik.no +lenvik.no +leangaviika.no +leaŋgaviika.no +lesja.no +levanger.no +lier.no +lierne.no +lillehammer.no +lillesand.no +lindesnes.no +lindas.no +lindås.no +lom.no +loppa.no +lahppi.no +láhppi.no +lund.no +lunner.no +luroy.no +lurøy.no +luster.no +lyngdal.no +lyngen.no +ivgu.no +lardal.no +lerdal.no +lærdal.no +lodingen.no +lødingen.no +lorenskog.no +lørenskog.no +loten.no +løten.no +malvik.no +masoy.no +måsøy.no +muosat.no +muosát.no +mandal.no +marker.no +marnardal.no +masfjorden.no +meland.no +meldal.no +melhus.no +meloy.no +meløy.no +meraker.no +meråker.no +moareke.no +moåreke.no +midsund.no +midtre-gauldal.no +modalen.no +modum.no +molde.no +moskenes.no +moss.no +mosvik.no +malselv.no +målselv.no +malatvuopmi.no +málatvuopmi.no +namdalseid.no +aejrie.no +namsos.no +namsskogan.no +naamesjevuemie.no +nååmesjevuemie.no +laakesvuemie.no +nannestad.no +narvik.no +narviika.no +naustdal.no +nedre-eiker.no +nes.akershus.no +nes.buskerud.no +nesna.no +nesodden.no +nesseby.no +unjarga.no +unjárga.no +nesset.no +nissedal.no +nittedal.no +nord-aurdal.no +nord-fron.no +nord-odal.no +norddal.no +nordkapp.no +davvenjarga.no +davvenjárga.no +nordre-land.no +nordreisa.no +raisa.no +ráisa.no +nore-og-uvdal.no +notodden.no +naroy.no +nærøy.no +notteroy.no +nøtterøy.no +odda.no +oksnes.no +øksnes.no +oppdal.no +oppegard.no +oppegård.no +orkdal.no +orland.no +ørland.no +orskog.no +ørskog.no +orsta.no +ørsta.no +os.hedmark.no +os.hordaland.no +osen.no +osteroy.no +osterøy.no +ostre-toten.no +østre-toten.no +overhalla.no +ovre-eiker.no +øvre-eiker.no +oyer.no +øyer.no +oygarden.no +øygarden.no +oystre-slidre.no +øystre-slidre.no +porsanger.no +porsangu.no +porsáŋgu.no +porsgrunn.no +radoy.no +radøy.no +rakkestad.no +rana.no +ruovat.no +randaberg.no +rauma.no +rendalen.no +rennebu.no +rennesoy.no +rennesøy.no +rindal.no +ringebu.no +ringerike.no +ringsaker.no +rissa.no +risor.no +risør.no +roan.no +rollag.no +rygge.no +ralingen.no +rælingen.no +rodoy.no +rødøy.no +romskog.no +rømskog.no +roros.no +røros.no +rost.no +røst.no +royken.no +røyken.no +royrvik.no +røyrvik.no +rade.no +råde.no +salangen.no +siellak.no +saltdal.no +salat.no +sálát.no +sálat.no +samnanger.no +sande.more-og-romsdal.no +sande.møre-og-romsdal.no +sande.vestfold.no +sandefjord.no +sandnes.no +sandoy.no +sandøy.no +sarpsborg.no +sauda.no +sauherad.no +sel.no +selbu.no +selje.no +seljord.no +sigdal.no +siljan.no +sirdal.no +skaun.no +skedsmo.no +ski.no +skien.no +skiptvet.no +skjervoy.no +skjervøy.no +skierva.no +skiervá.no +skjak.no +skjåk.no +skodje.no +skanland.no +skånland.no +skanit.no +skánit.no +smola.no +smøla.no +snillfjord.no +snasa.no +snåsa.no +snoasa.no +snaase.no +snåase.no +sogndal.no +sokndal.no +sola.no +solund.no +songdalen.no +sortland.no +spydeberg.no +stange.no +stavanger.no +steigen.no +steinkjer.no +stjordal.no +stjørdal.no +stokke.no +stor-elvdal.no +stord.no +stordal.no +storfjord.no +omasvuotna.no +strand.no +stranda.no +stryn.no +sula.no +suldal.no +sund.no +sunndal.no +surnadal.no +sveio.no +svelvik.no +sykkylven.no +sogne.no +søgne.no +somna.no +sømna.no +sondre-land.no +søndre-land.no +sor-aurdal.no +sør-aurdal.no +sor-fron.no +sør-fron.no +sor-odal.no +sør-odal.no +sor-varanger.no +sør-varanger.no +matta-varjjat.no +mátta-várjjat.no +sorfold.no +sørfold.no +sorreisa.no +sørreisa.no +sorum.no +sørum.no +tana.no +deatnu.no +time.no +tingvoll.no +tinn.no +tjeldsund.no +dielddanuorri.no +tjome.no +tjøme.no +tokke.no +tolga.no +torsken.no +tranoy.no +tranøy.no +tromso.no +tromsø.no +tromsa.no +romsa.no +trondheim.no +troandin.no +trysil.no +trana.no +træna.no +trogstad.no +trøgstad.no +tvedestrand.no +tydal.no +tynset.no +tysfjord.no +divtasvuodna.no +divttasvuotna.no +tysnes.no +tysvar.no +tysvær.no +tonsberg.no +tønsberg.no +ullensaker.no +ullensvang.no +ulvik.no +utsira.no +vadso.no +vadsø.no +cahcesuolo.no +čáhcesuolo.no +vaksdal.no +valle.no +vang.no +vanylven.no +vardo.no +vardø.no +varggat.no +várggát.no +vefsn.no +vaapste.no +vega.no +vegarshei.no +vegårshei.no +vennesla.no +verdal.no +verran.no +vestby.no +vestnes.no +vestre-slidre.no +vestre-toten.no +vestvagoy.no +vestvågøy.no +vevelstad.no +vik.no +vikna.no +vindafjord.no +volda.no +voss.no +varoy.no +værøy.no +vagan.no +vågan.no +voagat.no +vagsoy.no +vågsøy.no +vaga.no +vågå.no +valer.ostfold.no +våler.østfold.no +valer.hedmark.no +våler.hedmark.no + +// np : http://www.mos.com.np/register.html +*.np + +// nr : http://cenpac.net.nr/dns/index.html +// Submitted by registry +nr +biz.nr +info.nr +gov.nr +edu.nr +org.nr +net.nr +com.nr + +// nu : https://en.wikipedia.org/wiki/.nu +nu + +// nz : https://en.wikipedia.org/wiki/.nz +// Submitted by registry +nz +ac.nz +co.nz +cri.nz +geek.nz +gen.nz +govt.nz +health.nz +iwi.nz +kiwi.nz +maori.nz +mil.nz +māori.nz +net.nz +org.nz +parliament.nz +school.nz + +// om : https://en.wikipedia.org/wiki/.om +om +co.om +com.om +edu.om +gov.om +med.om +museum.om +net.om +org.om +pro.om + +// onion : https://tools.ietf.org/html/rfc7686 +onion + +// org : https://en.wikipedia.org/wiki/.org +org + +// pa : http://www.nic.pa/ +// Some additional second level "domains" resolve directly as hostnames, such as +// pannet.pa, so we add a rule for "pa". +pa +ac.pa +gob.pa +com.pa +org.pa +sld.pa +edu.pa +net.pa +ing.pa +abo.pa +med.pa +nom.pa + +// pe : https://www.nic.pe/InformeFinalComision.pdf +pe +edu.pe +gob.pe +nom.pe +mil.pe +org.pe +com.pe +net.pe + +// pf : http://www.gobin.info/domainname/formulaire-pf.pdf +pf +com.pf +org.pf +edu.pf + +// pg : https://en.wikipedia.org/wiki/.pg +*.pg + +// ph : http://www.domains.ph/FAQ2.asp +// Submitted by registry +ph +com.ph +net.ph +org.ph +gov.ph +edu.ph +ngo.ph +mil.ph +i.ph + +// pk : http://pk5.pknic.net.pk/pk5/msgNamepk.PK +pk +com.pk +net.pk +edu.pk +org.pk +fam.pk +biz.pk +web.pk +gov.pk +gob.pk +gok.pk +gon.pk +gop.pk +gos.pk +info.pk + +// pl http://www.dns.pl/english/index.html +// Submitted by registry +pl +com.pl +net.pl +org.pl +// pl functional domains (http://www.dns.pl/english/index.html) +aid.pl +agro.pl +atm.pl +auto.pl +biz.pl +edu.pl +gmina.pl +gsm.pl +info.pl +mail.pl +miasta.pl +media.pl +mil.pl +nieruchomosci.pl +nom.pl +pc.pl +powiat.pl +priv.pl +realestate.pl +rel.pl +sex.pl +shop.pl +sklep.pl +sos.pl +szkola.pl +targi.pl +tm.pl +tourism.pl +travel.pl +turystyka.pl +// Government domains +gov.pl +ap.gov.pl +ic.gov.pl +is.gov.pl +us.gov.pl +kmpsp.gov.pl +kppsp.gov.pl +kwpsp.gov.pl +psp.gov.pl +wskr.gov.pl +kwp.gov.pl +mw.gov.pl +ug.gov.pl +um.gov.pl +umig.gov.pl +ugim.gov.pl +upow.gov.pl +uw.gov.pl +starostwo.gov.pl +pa.gov.pl +po.gov.pl +psse.gov.pl +pup.gov.pl +rzgw.gov.pl +sa.gov.pl +so.gov.pl +sr.gov.pl +wsa.gov.pl +sko.gov.pl +uzs.gov.pl +wiih.gov.pl +winb.gov.pl +pinb.gov.pl +wios.gov.pl +witd.gov.pl +wzmiuw.gov.pl +piw.gov.pl +wiw.gov.pl +griw.gov.pl +wif.gov.pl +oum.gov.pl +sdn.gov.pl +zp.gov.pl +uppo.gov.pl +mup.gov.pl +wuoz.gov.pl +konsulat.gov.pl +oirm.gov.pl +// pl regional domains (http://www.dns.pl/english/index.html) +augustow.pl +babia-gora.pl +bedzin.pl +beskidy.pl +bialowieza.pl +bialystok.pl +bielawa.pl +bieszczady.pl +boleslawiec.pl +bydgoszcz.pl +bytom.pl +cieszyn.pl +czeladz.pl +czest.pl +dlugoleka.pl +elblag.pl +elk.pl +glogow.pl +gniezno.pl +gorlice.pl +grajewo.pl +ilawa.pl +jaworzno.pl +jelenia-gora.pl +jgora.pl +kalisz.pl +kazimierz-dolny.pl +karpacz.pl +kartuzy.pl +kaszuby.pl +katowice.pl +kepno.pl +ketrzyn.pl +klodzko.pl +kobierzyce.pl +kolobrzeg.pl +konin.pl +konskowola.pl +kutno.pl +lapy.pl +lebork.pl +legnica.pl +lezajsk.pl +limanowa.pl +lomza.pl +lowicz.pl +lubin.pl +lukow.pl +malbork.pl +malopolska.pl +mazowsze.pl +mazury.pl +mielec.pl +mielno.pl +mragowo.pl +naklo.pl +nowaruda.pl +nysa.pl +olawa.pl +olecko.pl +olkusz.pl +olsztyn.pl +opoczno.pl +opole.pl +ostroda.pl +ostroleka.pl +ostrowiec.pl +ostrowwlkp.pl +pila.pl +pisz.pl +podhale.pl +podlasie.pl +polkowice.pl +pomorze.pl +pomorskie.pl +prochowice.pl +pruszkow.pl +przeworsk.pl +pulawy.pl +radom.pl +rawa-maz.pl +rybnik.pl +rzeszow.pl +sanok.pl +sejny.pl +slask.pl +slupsk.pl +sosnowiec.pl +stalowa-wola.pl +skoczow.pl +starachowice.pl +stargard.pl +suwalki.pl +swidnica.pl +swiebodzin.pl +swinoujscie.pl +szczecin.pl +szczytno.pl +tarnobrzeg.pl +tgory.pl +turek.pl +tychy.pl +ustka.pl +walbrzych.pl +warmia.pl +warszawa.pl +waw.pl +wegrow.pl +wielun.pl +wlocl.pl +wloclawek.pl +wodzislaw.pl +wolomin.pl +wroclaw.pl +zachpomor.pl +zagan.pl +zarow.pl +zgora.pl +zgorzelec.pl + +// pm : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +pm + +// pn : http://www.government.pn/PnRegistry/policies.htm +pn +gov.pn +co.pn +org.pn +edu.pn +net.pn + +// post : https://en.wikipedia.org/wiki/.post +post + +// pr : http://www.nic.pr/index.asp?f=1 +pr +com.pr +net.pr +org.pr +gov.pr +edu.pr +isla.pr +pro.pr +biz.pr +info.pr +name.pr +// these aren't mentioned on nic.pr, but on https://en.wikipedia.org/wiki/.pr +est.pr +prof.pr +ac.pr + +// pro : http://registry.pro/get-pro +pro +aaa.pro +aca.pro +acct.pro +avocat.pro +bar.pro +cpa.pro +eng.pro +jur.pro +law.pro +med.pro +recht.pro + +// ps : https://en.wikipedia.org/wiki/.ps +// http://www.nic.ps/registration/policy.html#reg +ps +edu.ps +gov.ps +sec.ps +plo.ps +com.ps +org.ps +net.ps + +// pt : http://online.dns.pt/dns/start_dns +pt +net.pt +gov.pt +org.pt +edu.pt +int.pt +publ.pt +com.pt +nome.pt + +// pw : https://en.wikipedia.org/wiki/.pw +pw +co.pw +ne.pw +or.pw +ed.pw +go.pw +belau.pw + +// py : http://www.nic.py/pautas.html#seccion_9 +// Submitted by registry +py +com.py +coop.py +edu.py +gov.py +mil.py +net.py +org.py + +// qa : http://domains.qa/en/ +qa +com.qa +edu.qa +gov.qa +mil.qa +name.qa +net.qa +org.qa +sch.qa + +// re : http://www.afnic.re/obtenir/chartes/nommage-re/annexe-descriptifs +re +asso.re +com.re +nom.re + +// ro : http://www.rotld.ro/ +ro +arts.ro +com.ro +firm.ro +info.ro +nom.ro +nt.ro +org.ro +rec.ro +store.ro +tm.ro +www.ro + +// rs : https://www.rnids.rs/en/domains/national-domains +rs +ac.rs +co.rs +edu.rs +gov.rs +in.rs +org.rs + +// ru : https://cctld.ru/files/pdf/docs/en/rules_ru-rf.pdf +// Submitted by George Georgievsky +ru + +// rw : https://www.ricta.org.rw/sites/default/files/resources/registry_registrar_contract_0.pdf +rw +ac.rw +co.rw +coop.rw +gov.rw +mil.rw +net.rw +org.rw + +// sa : http://www.nic.net.sa/ +sa +com.sa +net.sa +org.sa +gov.sa +med.sa +pub.sa +edu.sa +sch.sa + +// sb : http://www.sbnic.net.sb/ +// Submitted by registry +sb +com.sb +edu.sb +gov.sb +net.sb +org.sb + +// sc : http://www.nic.sc/ +sc +com.sc +gov.sc +net.sc +org.sc +edu.sc + +// sd : http://www.isoc.sd/sudanic.isoc.sd/billing_pricing.htm +// Submitted by registry +sd +com.sd +net.sd +org.sd +edu.sd +med.sd +tv.sd +gov.sd +info.sd + +// se : https://en.wikipedia.org/wiki/.se +// Submitted by registry +se +a.se +ac.se +b.se +bd.se +brand.se +c.se +d.se +e.se +f.se +fh.se +fhsk.se +fhv.se +g.se +h.se +i.se +k.se +komforb.se +kommunalforbund.se +komvux.se +l.se +lanbib.se +m.se +n.se +naturbruksgymn.se +o.se +org.se +p.se +parti.se +pp.se +press.se +r.se +s.se +t.se +tm.se +u.se +w.se +x.se +y.se +z.se + +// sg : http://www.nic.net.sg/page/registration-policies-procedures-and-guidelines +sg +com.sg +net.sg +org.sg +gov.sg +edu.sg +per.sg + +// sh : http://www.nic.sh/registrar.html +sh +com.sh +net.sh +gov.sh +org.sh +mil.sh + +// si : https://en.wikipedia.org/wiki/.si +si + +// sj : No registrations at this time. +// Submitted by registry +sj + +// sk : https://en.wikipedia.org/wiki/.sk +// list of 2nd level domains ? +sk + +// sl : http://www.nic.sl +// Submitted by registry +sl +com.sl +net.sl +edu.sl +gov.sl +org.sl + +// sm : https://en.wikipedia.org/wiki/.sm +sm + +// sn : https://en.wikipedia.org/wiki/.sn +sn +art.sn +com.sn +edu.sn +gouv.sn +org.sn +perso.sn +univ.sn + +// so : http://sonic.so/policies/ +so +com.so +edu.so +gov.so +me.so +net.so +org.so + +// sr : https://en.wikipedia.org/wiki/.sr +sr + +// ss : https://registry.nic.ss/ +// Submitted by registry +ss +biz.ss +com.ss +edu.ss +gov.ss +net.ss +org.ss + +// st : http://www.nic.st/html/policyrules/ +st +co.st +com.st +consulado.st +edu.st +embaixada.st +gov.st +mil.st +net.st +org.st +principe.st +saotome.st +store.st + +// su : https://en.wikipedia.org/wiki/.su +su + +// sv : http://www.svnet.org.sv/niveldos.pdf +sv +com.sv +edu.sv +gob.sv +org.sv +red.sv + +// sx : https://en.wikipedia.org/wiki/.sx +// Submitted by registry +sx +gov.sx + +// sy : https://en.wikipedia.org/wiki/.sy +// see also: http://www.gobin.info/domainname/sy.doc +sy +edu.sy +gov.sy +net.sy +mil.sy +com.sy +org.sy + +// sz : https://en.wikipedia.org/wiki/.sz +// http://www.sispa.org.sz/ +sz +co.sz +ac.sz +org.sz + +// tc : https://en.wikipedia.org/wiki/.tc +tc + +// td : https://en.wikipedia.org/wiki/.td +td + +// tel: https://en.wikipedia.org/wiki/.tel +// http://www.telnic.org/ +tel + +// tf : https://en.wikipedia.org/wiki/.tf +tf + +// tg : https://en.wikipedia.org/wiki/.tg +// http://www.nic.tg/ +tg + +// th : https://en.wikipedia.org/wiki/.th +// Submitted by registry +th +ac.th +co.th +go.th +in.th +mi.th +net.th +or.th + +// tj : http://www.nic.tj/policy.html +tj +ac.tj +biz.tj +co.tj +com.tj +edu.tj +go.tj +gov.tj +int.tj +mil.tj +name.tj +net.tj +nic.tj +org.tj +test.tj +web.tj + +// tk : https://en.wikipedia.org/wiki/.tk +tk + +// tl : https://en.wikipedia.org/wiki/.tl +tl +gov.tl + +// tm : http://www.nic.tm/local.html +tm +com.tm +co.tm +org.tm +net.tm +nom.tm +gov.tm +mil.tm +edu.tm + +// tn : https://en.wikipedia.org/wiki/.tn +// http://whois.ati.tn/ +tn +com.tn +ens.tn +fin.tn +gov.tn +ind.tn +intl.tn +nat.tn +net.tn +org.tn +info.tn +perso.tn +tourism.tn +edunet.tn +rnrt.tn +rns.tn +rnu.tn +mincom.tn +agrinet.tn +defense.tn +turen.tn + +// to : https://en.wikipedia.org/wiki/.to +// Submitted by registry +to +com.to +gov.to +net.to +org.to +edu.to +mil.to + +// tr : https://nic.tr/ +// https://nic.tr/forms/eng/policies.pdf +// https://nic.tr/index.php?USRACTN=PRICELST +tr +av.tr +bbs.tr +bel.tr +biz.tr +com.tr +dr.tr +edu.tr +gen.tr +gov.tr +info.tr +mil.tr +k12.tr +kep.tr +name.tr +net.tr +org.tr +pol.tr +tel.tr +tsk.tr +tv.tr +web.tr +// Used by Northern Cyprus +nc.tr +// Used by government agencies of Northern Cyprus +gov.nc.tr + +// tt : http://www.nic.tt/ +tt +co.tt +com.tt +org.tt +net.tt +biz.tt +info.tt +pro.tt +int.tt +coop.tt +jobs.tt +mobi.tt +travel.tt +museum.tt +aero.tt +name.tt +gov.tt +edu.tt + +// tv : https://en.wikipedia.org/wiki/.tv +// Not listing any 2LDs as reserved since none seem to exist in practice, +// Wikipedia notwithstanding. +tv + +// tw : https://en.wikipedia.org/wiki/.tw +tw +edu.tw +gov.tw +mil.tw +com.tw +net.tw +org.tw +idv.tw +game.tw +ebiz.tw +club.tw +網路.tw +組織.tw +商業.tw + +// tz : http://www.tznic.or.tz/index.php/domains +// Submitted by registry +tz +ac.tz +co.tz +go.tz +hotel.tz +info.tz +me.tz +mil.tz +mobi.tz +ne.tz +or.tz +sc.tz +tv.tz + +// ua : https://hostmaster.ua/policy/?ua +// Submitted by registry +ua +// ua 2LD +com.ua +edu.ua +gov.ua +in.ua +net.ua +org.ua +// ua geographic names +// https://hostmaster.ua/2ld/ +cherkassy.ua +cherkasy.ua +chernigov.ua +chernihiv.ua +chernivtsi.ua +chernovtsy.ua +ck.ua +cn.ua +cr.ua +crimea.ua +cv.ua +dn.ua +dnepropetrovsk.ua +dnipropetrovsk.ua +donetsk.ua +dp.ua +if.ua +ivano-frankivsk.ua +kh.ua +kharkiv.ua +kharkov.ua +kherson.ua +khmelnitskiy.ua +khmelnytskyi.ua +kiev.ua +kirovograd.ua +km.ua +kr.ua +krym.ua +ks.ua +kv.ua +kyiv.ua +lg.ua +lt.ua +lugansk.ua +lutsk.ua +lv.ua +lviv.ua +mk.ua +mykolaiv.ua +nikolaev.ua +od.ua +odesa.ua +odessa.ua +pl.ua +poltava.ua +rivne.ua +rovno.ua +rv.ua +sb.ua +sebastopol.ua +sevastopol.ua +sm.ua +sumy.ua +te.ua +ternopil.ua +uz.ua +uzhgorod.ua +vinnica.ua +vinnytsia.ua +vn.ua +volyn.ua +yalta.ua +zaporizhzhe.ua +zaporizhzhia.ua +zhitomir.ua +zhytomyr.ua +zp.ua +zt.ua + +// ug : https://www.registry.co.ug/ +ug +co.ug +or.ug +ac.ug +sc.ug +go.ug +ne.ug +com.ug +org.ug + +// uk : https://en.wikipedia.org/wiki/.uk +// Submitted by registry +uk +ac.uk +co.uk +gov.uk +ltd.uk +me.uk +net.uk +nhs.uk +org.uk +plc.uk +police.uk +*.sch.uk + +// us : https://en.wikipedia.org/wiki/.us +us +dni.us +fed.us +isa.us +kids.us +nsn.us +// us geographic names +ak.us +al.us +ar.us +as.us +az.us +ca.us +co.us +ct.us +dc.us +de.us +fl.us +ga.us +gu.us +hi.us +ia.us +id.us +il.us +in.us +ks.us +ky.us +la.us +ma.us +md.us +me.us +mi.us +mn.us +mo.us +ms.us +mt.us +nc.us +nd.us +ne.us +nh.us +nj.us +nm.us +nv.us +ny.us +oh.us +ok.us +or.us +pa.us +pr.us +ri.us +sc.us +sd.us +tn.us +tx.us +ut.us +vi.us +vt.us +va.us +wa.us +wi.us +wv.us +wy.us +// The registrar notes several more specific domains available in each state, +// such as state.*.us, dst.*.us, etc., but resolution of these is somewhat +// haphazard; in some states these domains resolve as addresses, while in others +// only subdomains are available, or even nothing at all. We include the +// most common ones where it's clear that different sites are different +// entities. +k12.ak.us +k12.al.us +k12.ar.us +k12.as.us +k12.az.us +k12.ca.us +k12.co.us +k12.ct.us +k12.dc.us +k12.de.us +k12.fl.us +k12.ga.us +k12.gu.us +// k12.hi.us Bug 614565 - Hawaii has a state-wide DOE login +k12.ia.us +k12.id.us +k12.il.us +k12.in.us +k12.ks.us +k12.ky.us +k12.la.us +k12.ma.us +k12.md.us +k12.me.us +k12.mi.us +k12.mn.us +k12.mo.us +k12.ms.us +k12.mt.us +k12.nc.us +// k12.nd.us Bug 1028347 - Removed at request of Travis Rosso +k12.ne.us +k12.nh.us +k12.nj.us +k12.nm.us +k12.nv.us +k12.ny.us +k12.oh.us +k12.ok.us +k12.or.us +k12.pa.us +k12.pr.us +// k12.ri.us Removed at request of Kim Cournoyer +k12.sc.us +// k12.sd.us Bug 934131 - Removed at request of James Booze +k12.tn.us +k12.tx.us +k12.ut.us +k12.vi.us +k12.vt.us +k12.va.us +k12.wa.us +k12.wi.us +// k12.wv.us Bug 947705 - Removed at request of Verne Britton +k12.wy.us +cc.ak.us +cc.al.us +cc.ar.us +cc.as.us +cc.az.us +cc.ca.us +cc.co.us +cc.ct.us +cc.dc.us +cc.de.us +cc.fl.us +cc.ga.us +cc.gu.us +cc.hi.us +cc.ia.us +cc.id.us +cc.il.us +cc.in.us +cc.ks.us +cc.ky.us +cc.la.us +cc.ma.us +cc.md.us +cc.me.us +cc.mi.us +cc.mn.us +cc.mo.us +cc.ms.us +cc.mt.us +cc.nc.us +cc.nd.us +cc.ne.us +cc.nh.us +cc.nj.us +cc.nm.us +cc.nv.us +cc.ny.us +cc.oh.us +cc.ok.us +cc.or.us +cc.pa.us +cc.pr.us +cc.ri.us +cc.sc.us +cc.sd.us +cc.tn.us +cc.tx.us +cc.ut.us +cc.vi.us +cc.vt.us +cc.va.us +cc.wa.us +cc.wi.us +cc.wv.us +cc.wy.us +lib.ak.us +lib.al.us +lib.ar.us +lib.as.us +lib.az.us +lib.ca.us +lib.co.us +lib.ct.us +lib.dc.us +// lib.de.us Issue #243 - Moved to Private section at request of Ed Moore +lib.fl.us +lib.ga.us +lib.gu.us +lib.hi.us +lib.ia.us +lib.id.us +lib.il.us +lib.in.us +lib.ks.us +lib.ky.us +lib.la.us +lib.ma.us +lib.md.us +lib.me.us +lib.mi.us +lib.mn.us +lib.mo.us +lib.ms.us +lib.mt.us +lib.nc.us +lib.nd.us +lib.ne.us +lib.nh.us +lib.nj.us +lib.nm.us +lib.nv.us +lib.ny.us +lib.oh.us +lib.ok.us +lib.or.us +lib.pa.us +lib.pr.us +lib.ri.us +lib.sc.us +lib.sd.us +lib.tn.us +lib.tx.us +lib.ut.us +lib.vi.us +lib.vt.us +lib.va.us +lib.wa.us +lib.wi.us +// lib.wv.us Bug 941670 - Removed at request of Larry W Arnold +lib.wy.us +// k12.ma.us contains school districts in Massachusetts. The 4LDs are +// managed independently except for private (PVT), charter (CHTR) and +// parochial (PAROCH) schools. Those are delegated directly to the +// 5LD operators. +pvt.k12.ma.us +chtr.k12.ma.us +paroch.k12.ma.us +// Merit Network, Inc. maintains the registry for =~ /(k12|cc|lib).mi.us/ and the following +// see also: http://domreg.merit.edu +// see also: whois -h whois.domreg.merit.edu help +ann-arbor.mi.us +cog.mi.us +dst.mi.us +eaton.mi.us +gen.mi.us +mus.mi.us +tec.mi.us +washtenaw.mi.us + +// uy : http://www.nic.org.uy/ +uy +com.uy +edu.uy +gub.uy +mil.uy +net.uy +org.uy + +// uz : http://www.reg.uz/ +uz +co.uz +com.uz +net.uz +org.uz + +// va : https://en.wikipedia.org/wiki/.va +va + +// vc : https://en.wikipedia.org/wiki/.vc +// Submitted by registry +vc +com.vc +net.vc +org.vc +gov.vc +mil.vc +edu.vc + +// ve : https://registro.nic.ve/ +// Submitted by registry +ve +arts.ve +co.ve +com.ve +e12.ve +edu.ve +firm.ve +gob.ve +gov.ve +info.ve +int.ve +mil.ve +net.ve +org.ve +rec.ve +store.ve +tec.ve +web.ve + +// vg : https://en.wikipedia.org/wiki/.vg +vg + +// vi : http://www.nic.vi/newdomainform.htm +// http://www.nic.vi/Domain_Rules/body_domain_rules.html indicates some other +// TLDs are "reserved", such as edu.vi and gov.vi, but doesn't actually say they +// are available for registration (which they do not seem to be). +vi +co.vi +com.vi +k12.vi +net.vi +org.vi + +// vn : https://www.dot.vn/vnnic/vnnic/domainregistration.jsp +vn +com.vn +net.vn +org.vn +edu.vn +gov.vn +int.vn +ac.vn +biz.vn +info.vn +name.vn +pro.vn +health.vn + +// vu : https://en.wikipedia.org/wiki/.vu +// http://www.vunic.vu/ +vu +com.vu +edu.vu +net.vu +org.vu + +// wf : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +wf + +// ws : https://en.wikipedia.org/wiki/.ws +// http://samoanic.ws/index.dhtml +ws +com.ws +net.ws +org.ws +gov.ws +edu.ws + +// yt : http://www.afnic.fr/medias/documents/AFNIC-naming-policy2012.pdf +yt + +// IDN ccTLDs +// When submitting patches, please maintain a sort by ISO 3166 ccTLD, then +// U-label, and follow this format: +// // A-Label ("", [, variant info]) : +// // [sponsoring org] +// U-Label + +// xn--mgbaam7a8h ("Emerat", Arabic) : AE +// http://nic.ae/english/arabicdomain/rules.jsp +امارات + +// xn--y9a3aq ("hye", Armenian) : AM +// ISOC AM (operated by .am Registry) +հայ + +// xn--54b7fta0cc ("Bangla", Bangla) : BD +বাংলা + +// xn--90ae ("bg", Bulgarian) : BG +бг + +// xn--90ais ("bel", Belarusian/Russian Cyrillic) : BY +// Operated by .by registry +бел + +// xn--fiqs8s ("Zhongguo/China", Chinese, Simplified) : CN +// CNNIC +// http://cnnic.cn/html/Dir/2005/10/11/3218.htm +中国 + +// xn--fiqz9s ("Zhongguo/China", Chinese, Traditional) : CN +// CNNIC +// http://cnnic.cn/html/Dir/2005/10/11/3218.htm +中國 + +// xn--lgbbat1ad8j ("Algeria/Al Jazair", Arabic) : DZ +الجزائر + +// xn--wgbh1c ("Egypt/Masr", Arabic) : EG +// http://www.dotmasr.eg/ +مصر + +// xn--e1a4c ("eu", Cyrillic) : EU +// https://eurid.eu +ею + +// xn--qxa6a ("eu", Greek) : EU +// https://eurid.eu +ευ + +// xn--mgbah1a3hjkrd ("Mauritania", Arabic) : MR +موريتانيا + +// xn--node ("ge", Georgian Mkhedruli) : GE +გე + +// xn--qxam ("el", Greek) : GR +// Hellenic Ministry of Infrastructure, Transport, and Networks +ελ + +// xn--j6w193g ("Hong Kong", Chinese) : HK +// https://www.hkirc.hk +// Submitted by registry +// https://www.hkirc.hk/content.jsp?id=30#!/34 +香港 +公司.香港 +教育.香港 +政府.香港 +個人.香港 +網絡.香港 +組織.香港 + +// xn--2scrj9c ("Bharat", Kannada) : IN +// India +ಭಾರತ + +// xn--3hcrj9c ("Bharat", Oriya) : IN +// India +ଭାରତ + +// xn--45br5cyl ("Bharatam", Assamese) : IN +// India +ভাৰত + +// xn--h2breg3eve ("Bharatam", Sanskrit) : IN +// India +भारतम् + +// xn--h2brj9c8c ("Bharot", Santali) : IN +// India +भारोत + +// xn--mgbgu82a ("Bharat", Sindhi) : IN +// India +ڀارت + +// xn--rvc1e0am3e ("Bharatam", Malayalam) : IN +// India +ഭാരതം + +// xn--h2brj9c ("Bharat", Devanagari) : IN +// India +भारत + +// xn--mgbbh1a ("Bharat", Kashmiri) : IN +// India +بارت + +// xn--mgbbh1a71e ("Bharat", Arabic) : IN +// India +بھارت + +// xn--fpcrj9c3d ("Bharat", Telugu) : IN +// India +భారత్ + +// xn--gecrj9c ("Bharat", Gujarati) : IN +// India +ભારત + +// xn--s9brj9c ("Bharat", Gurmukhi) : IN +// India +ਭਾਰਤ + +// xn--45brj9c ("Bharat", Bengali) : IN +// India +ভারত + +// xn--xkc2dl3a5ee0h ("India", Tamil) : IN +// India +இந்தியா + +// xn--mgba3a4f16a ("Iran", Persian) : IR +ایران + +// xn--mgba3a4fra ("Iran", Arabic) : IR +ايران + +// xn--mgbtx2b ("Iraq", Arabic) : IQ +// Communications and Media Commission +عراق + +// xn--mgbayh7gpa ("al-Ordon", Arabic) : JO +// National Information Technology Center (NITC) +// Royal Scientific Society, Al-Jubeiha +الاردن + +// xn--3e0b707e ("Republic of Korea", Hangul) : KR +한국 + +// xn--80ao21a ("Kaz", Kazakh) : KZ +қаз + +// xn--fzc2c9e2c ("Lanka", Sinhalese-Sinhala) : LK +// https://nic.lk +ලංකා + +// xn--xkc2al3hye2a ("Ilangai", Tamil) : LK +// https://nic.lk +இலங்கை + +// xn--mgbc0a9azcg ("Morocco/al-Maghrib", Arabic) : MA +المغرب + +// xn--d1alf ("mkd", Macedonian) : MK +// MARnet +мкд + +// xn--l1acc ("mon", Mongolian) : MN +мон + +// xn--mix891f ("Macao", Chinese, Traditional) : MO +// MONIC / HNET Asia (Registry Operator for .mo) +澳門 + +// xn--mix082f ("Macao", Chinese, Simplified) : MO +澳门 + +// xn--mgbx4cd0ab ("Malaysia", Malay) : MY +مليسيا + +// xn--mgb9awbf ("Oman", Arabic) : OM +عمان + +// xn--mgbai9azgqp6j ("Pakistan", Urdu/Arabic) : PK +پاکستان + +// xn--mgbai9a5eva00b ("Pakistan", Urdu/Arabic, variant) : PK +پاكستان + +// xn--ygbi2ammx ("Falasteen", Arabic) : PS +// The Palestinian National Internet Naming Authority (PNINA) +// http://www.pnina.ps +فلسطين + +// xn--90a3ac ("srb", Cyrillic) : RS +// https://www.rnids.rs/en/domains/national-domains +срб +пр.срб +орг.срб +обр.срб +од.срб +упр.срб +ак.срб + +// xn--p1ai ("rf", Russian-Cyrillic) : RU +// https://cctld.ru/files/pdf/docs/en/rules_ru-rf.pdf +// Submitted by George Georgievsky +рф + +// xn--wgbl6a ("Qatar", Arabic) : QA +// http://www.ict.gov.qa/ +قطر + +// xn--mgberp4a5d4ar ("AlSaudiah", Arabic) : SA +// http://www.nic.net.sa/ +السعودية + +// xn--mgberp4a5d4a87g ("AlSaudiah", Arabic, variant) : SA +السعودیة + +// xn--mgbqly7c0a67fbc ("AlSaudiah", Arabic, variant) : SA +السعودیۃ + +// xn--mgbqly7cvafr ("AlSaudiah", Arabic, variant) : SA +السعوديه + +// xn--mgbpl2fh ("sudan", Arabic) : SD +// Operated by .sd registry +سودان + +// xn--yfro4i67o Singapore ("Singapore", Chinese) : SG +新加坡 + +// xn--clchc0ea0b2g2a9gcd ("Singapore", Tamil) : SG +சிங்கப்பூர் + +// xn--ogbpf8fl ("Syria", Arabic) : SY +سورية + +// xn--mgbtf8fl ("Syria", Arabic, variant) : SY +سوريا + +// xn--o3cw4h ("Thai", Thai) : TH +// http://www.thnic.co.th +ไทย +ศึกษา.ไทย +ธุรกิจ.ไทย +รัฐบาล.ไทย +ทหาร.ไทย +เน็ต.ไทย +องค์กร.ไทย + +// xn--pgbs0dh ("Tunisia", Arabic) : TN +// http://nic.tn +تونس + +// xn--kpry57d ("Taiwan", Chinese, Traditional) : TW +// http://www.twnic.net/english/dn/dn_07a.htm +台灣 + +// xn--kprw13d ("Taiwan", Chinese, Simplified) : TW +// http://www.twnic.net/english/dn/dn_07a.htm +台湾 + +// xn--nnx388a ("Taiwan", Chinese, variant) : TW +臺灣 + +// xn--j1amh ("ukr", Cyrillic) : UA +укр + +// xn--mgb2ddes ("AlYemen", Arabic) : YE +اليمن + +// xxx : http://icmregistry.com +xxx + +// ye : http://www.y.net.ye/services/domain_name.htm +*.ye + +// za : https://www.zadna.org.za/content/page/domain-information/ +ac.za +agric.za +alt.za +co.za +edu.za +gov.za +grondar.za +law.za +mil.za +net.za +ngo.za +nic.za +nis.za +nom.za +org.za +school.za +tm.za +web.za + +// zm : https://zicta.zm/ +// Submitted by registry +zm +ac.zm +biz.zm +co.zm +com.zm +edu.zm +gov.zm +info.zm +mil.zm +net.zm +org.zm +sch.zm + +// zw : https://www.potraz.gov.zw/ +// Confirmed by registry 2017-01-25 +zw +ac.zw +co.zw +gov.zw +mil.zw +org.zw + + +// newGTLDs + +// List of new gTLDs imported from https://www.icann.org/resources/registries/gtlds/v2/gtlds.json on 2020-11-30T20:26:10Z +// This list is auto-generated, don't edit it manually. +// aaa : 2015-02-26 American Automobile Association, Inc. +aaa + +// aarp : 2015-05-21 AARP +aarp + +// abarth : 2015-07-30 Fiat Chrysler Automobiles N.V. +abarth + +// abb : 2014-10-24 ABB Ltd +abb + +// abbott : 2014-07-24 Abbott Laboratories, Inc. +abbott + +// abbvie : 2015-07-30 AbbVie Inc. +abbvie + +// abc : 2015-07-30 Disney Enterprises, Inc. +abc + +// able : 2015-06-25 Able Inc. +able + +// abogado : 2014-04-24 Minds + Machines Group Limited +abogado + +// abudhabi : 2015-07-30 Abu Dhabi Systems and Information Centre +abudhabi + +// academy : 2013-11-07 Binky Moon, LLC +academy + +// accenture : 2014-08-15 Accenture plc +accenture + +// accountant : 2014-11-20 dot Accountant Limited +accountant + +// accountants : 2014-03-20 Binky Moon, LLC +accountants + +// aco : 2015-01-08 ACO Severin Ahlmann GmbH & Co. KG +aco + +// actor : 2013-12-12 Dog Beach, LLC +actor + +// adac : 2015-07-16 Allgemeiner Deutscher Automobil-Club e.V. (ADAC) +adac + +// ads : 2014-12-04 Charleston Road Registry Inc. +ads + +// adult : 2014-10-16 ICM Registry AD LLC +adult + +// aeg : 2015-03-19 Aktiebolaget Electrolux +aeg + +// aetna : 2015-05-21 Aetna Life Insurance Company +aetna + +// afamilycompany : 2015-07-23 Johnson Shareholdings, Inc. +afamilycompany + +// afl : 2014-10-02 Australian Football League +afl + +// africa : 2014-03-24 ZA Central Registry NPC trading as Registry.Africa +africa + +// agakhan : 2015-04-23 Fondation Aga Khan (Aga Khan Foundation) +agakhan + +// agency : 2013-11-14 Binky Moon, LLC +agency + +// aig : 2014-12-18 American International Group, Inc. +aig + +// airbus : 2015-07-30 Airbus S.A.S. +airbus + +// airforce : 2014-03-06 Dog Beach, LLC +airforce + +// airtel : 2014-10-24 Bharti Airtel Limited +airtel + +// akdn : 2015-04-23 Fondation Aga Khan (Aga Khan Foundation) +akdn + +// alfaromeo : 2015-07-31 Fiat Chrysler Automobiles N.V. +alfaromeo + +// alibaba : 2015-01-15 Alibaba Group Holding Limited +alibaba + +// alipay : 2015-01-15 Alibaba Group Holding Limited +alipay + +// allfinanz : 2014-07-03 Allfinanz Deutsche Vermögensberatung Aktiengesellschaft +allfinanz + +// allstate : 2015-07-31 Allstate Fire and Casualty Insurance Company +allstate + +// ally : 2015-06-18 Ally Financial Inc. +ally + +// alsace : 2014-07-02 Region Grand Est +alsace + +// alstom : 2015-07-30 ALSTOM +alstom + +// amazon : 2019-12-19 Amazon Registry Services, Inc. +amazon + +// americanexpress : 2015-07-31 American Express Travel Related Services Company, Inc. +americanexpress + +// americanfamily : 2015-07-23 AmFam, Inc. +americanfamily + +// amex : 2015-07-31 American Express Travel Related Services Company, Inc. +amex + +// amfam : 2015-07-23 AmFam, Inc. +amfam + +// amica : 2015-05-28 Amica Mutual Insurance Company +amica + +// amsterdam : 2014-07-24 Gemeente Amsterdam +amsterdam + +// analytics : 2014-12-18 Campus IP LLC +analytics + +// android : 2014-08-07 Charleston Road Registry Inc. +android + +// anquan : 2015-01-08 Beijing Qihu Keji Co., Ltd. +anquan + +// anz : 2015-07-31 Australia and New Zealand Banking Group Limited +anz + +// aol : 2015-09-17 Oath Inc. +aol + +// apartments : 2014-12-11 Binky Moon, LLC +apartments + +// app : 2015-05-14 Charleston Road Registry Inc. +app + +// apple : 2015-05-14 Apple Inc. +apple + +// aquarelle : 2014-07-24 Aquarelle.com +aquarelle + +// arab : 2015-11-12 League of Arab States +arab + +// aramco : 2014-11-20 Aramco Services Company +aramco + +// archi : 2014-02-06 Afilias Limited +archi + +// army : 2014-03-06 Dog Beach, LLC +army + +// art : 2016-03-24 UK Creative Ideas Limited +art + +// arte : 2014-12-11 Association Relative à la Télévision Européenne G.E.I.E. +arte + +// asda : 2015-07-31 Wal-Mart Stores, Inc. +asda + +// associates : 2014-03-06 Binky Moon, LLC +associates + +// athleta : 2015-07-30 The Gap, Inc. +athleta + +// attorney : 2014-03-20 Dog Beach, LLC +attorney + +// auction : 2014-03-20 Dog Beach, LLC +auction + +// audi : 2015-05-21 AUDI Aktiengesellschaft +audi + +// audible : 2015-06-25 Amazon Registry Services, Inc. +audible + +// audio : 2014-03-20 UNR Corp. +audio + +// auspost : 2015-08-13 Australian Postal Corporation +auspost + +// author : 2014-12-18 Amazon Registry Services, Inc. +author + +// auto : 2014-11-13 XYZ.COM LLC +auto + +// autos : 2014-01-09 XYZ.COM LLC +autos + +// avianca : 2015-01-08 Avianca Holdings S.A. +avianca + +// aws : 2015-06-25 Amazon Registry Services, Inc. +aws + +// axa : 2013-12-19 AXA Group Operations SAS +axa + +// azure : 2014-12-18 Microsoft Corporation +azure + +// baby : 2015-04-09 XYZ.COM LLC +baby + +// baidu : 2015-01-08 Baidu, Inc. +baidu + +// banamex : 2015-07-30 Citigroup Inc. +banamex + +// bananarepublic : 2015-07-31 The Gap, Inc. +bananarepublic + +// band : 2014-06-12 Dog Beach, LLC +band + +// bank : 2014-09-25 fTLD Registry Services LLC +bank + +// bar : 2013-12-12 Punto 2012 Sociedad Anonima Promotora de Inversion de Capital Variable +bar + +// barcelona : 2014-07-24 Municipi de Barcelona +barcelona + +// barclaycard : 2014-11-20 Barclays Bank PLC +barclaycard + +// barclays : 2014-11-20 Barclays Bank PLC +barclays + +// barefoot : 2015-06-11 Gallo Vineyards, Inc. +barefoot + +// bargains : 2013-11-14 Binky Moon, LLC +bargains + +// baseball : 2015-10-29 MLB Advanced Media DH, LLC +baseball + +// basketball : 2015-08-20 Fédération Internationale de Basketball (FIBA) +basketball + +// bauhaus : 2014-04-17 Werkhaus GmbH +bauhaus + +// bayern : 2014-01-23 Bayern Connect GmbH +bayern + +// bbc : 2014-12-18 British Broadcasting Corporation +bbc + +// bbt : 2015-07-23 BB&T Corporation +bbt + +// bbva : 2014-10-02 BANCO BILBAO VIZCAYA ARGENTARIA, S.A. +bbva + +// bcg : 2015-04-02 The Boston Consulting Group, Inc. +bcg + +// bcn : 2014-07-24 Municipi de Barcelona +bcn + +// beats : 2015-05-14 Beats Electronics, LLC +beats + +// beauty : 2015-12-03 XYZ.COM LLC +beauty + +// beer : 2014-01-09 Minds + Machines Group Limited +beer + +// bentley : 2014-12-18 Bentley Motors Limited +bentley + +// berlin : 2013-10-31 dotBERLIN GmbH & Co. KG +berlin + +// best : 2013-12-19 BestTLD Pty Ltd +best + +// bestbuy : 2015-07-31 BBY Solutions, Inc. +bestbuy + +// bet : 2015-05-07 Afilias Limited +bet + +// bharti : 2014-01-09 Bharti Enterprises (Holding) Private Limited +bharti + +// bible : 2014-06-19 American Bible Society +bible + +// bid : 2013-12-19 dot Bid Limited +bid + +// bike : 2013-08-27 Binky Moon, LLC +bike + +// bing : 2014-12-18 Microsoft Corporation +bing + +// bingo : 2014-12-04 Binky Moon, LLC +bingo + +// bio : 2014-03-06 Afilias Limited +bio + +// black : 2014-01-16 Afilias Limited +black + +// blackfriday : 2014-01-16 UNR Corp. +blackfriday + +// blockbuster : 2015-07-30 Dish DBS Corporation +blockbuster + +// blog : 2015-05-14 Knock Knock WHOIS There, LLC +blog + +// bloomberg : 2014-07-17 Bloomberg IP Holdings LLC +bloomberg + +// blue : 2013-11-07 Afilias Limited +blue + +// bms : 2014-10-30 Bristol-Myers Squibb Company +bms + +// bmw : 2014-01-09 Bayerische Motoren Werke Aktiengesellschaft +bmw + +// bnpparibas : 2014-05-29 BNP Paribas +bnpparibas + +// boats : 2014-12-04 XYZ.COM LLC +boats + +// boehringer : 2015-07-09 Boehringer Ingelheim International GmbH +boehringer + +// bofa : 2015-07-31 Bank of America Corporation +bofa + +// bom : 2014-10-16 Núcleo de Informação e Coordenação do Ponto BR - NIC.br +bom + +// bond : 2014-06-05 ShortDot SA +bond + +// boo : 2014-01-30 Charleston Road Registry Inc. +boo + +// book : 2015-08-27 Amazon Registry Services, Inc. +book + +// booking : 2015-07-16 Booking.com B.V. +booking + +// bosch : 2015-06-18 Robert Bosch GMBH +bosch + +// bostik : 2015-05-28 Bostik SA +bostik + +// boston : 2015-12-10 Boston TLD Management, LLC +boston + +// bot : 2014-12-18 Amazon Registry Services, Inc. +bot + +// boutique : 2013-11-14 Binky Moon, LLC +boutique + +// box : 2015-11-12 Intercap Registry Inc. +box + +// bradesco : 2014-12-18 Banco Bradesco S.A. +bradesco + +// bridgestone : 2014-12-18 Bridgestone Corporation +bridgestone + +// broadway : 2014-12-22 Celebrate Broadway, Inc. +broadway + +// broker : 2014-12-11 Dotbroker Registry Limited +broker + +// brother : 2015-01-29 Brother Industries, Ltd. +brother + +// brussels : 2014-02-06 DNS.be vzw +brussels + +// budapest : 2013-11-21 Minds + Machines Group Limited +budapest + +// bugatti : 2015-07-23 Bugatti International SA +bugatti + +// build : 2013-11-07 Plan Bee LLC +build + +// builders : 2013-11-07 Binky Moon, LLC +builders + +// business : 2013-11-07 Binky Moon, LLC +business + +// buy : 2014-12-18 Amazon Registry Services, Inc. +buy + +// buzz : 2013-10-02 DOTSTRATEGY CO. +buzz + +// bzh : 2014-02-27 Association www.bzh +bzh + +// cab : 2013-10-24 Binky Moon, LLC +cab + +// cafe : 2015-02-11 Binky Moon, LLC +cafe + +// cal : 2014-07-24 Charleston Road Registry Inc. +cal + +// call : 2014-12-18 Amazon Registry Services, Inc. +call + +// calvinklein : 2015-07-30 PVH gTLD Holdings LLC +calvinklein + +// cam : 2016-04-21 AC Webconnecting Holding B.V. +cam + +// camera : 2013-08-27 Binky Moon, LLC +camera + +// camp : 2013-11-07 Binky Moon, LLC +camp + +// cancerresearch : 2014-05-15 Australian Cancer Research Foundation +cancerresearch + +// canon : 2014-09-12 Canon Inc. +canon + +// capetown : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +capetown + +// capital : 2014-03-06 Binky Moon, LLC +capital + +// capitalone : 2015-08-06 Capital One Financial Corporation +capitalone + +// car : 2015-01-22 XYZ.COM LLC +car + +// caravan : 2013-12-12 Caravan International, Inc. +caravan + +// cards : 2013-12-05 Binky Moon, LLC +cards + +// care : 2014-03-06 Binky Moon, LLC +care + +// career : 2013-10-09 dotCareer LLC +career + +// careers : 2013-10-02 Binky Moon, LLC +careers + +// cars : 2014-11-13 XYZ.COM LLC +cars + +// casa : 2013-11-21 Minds + Machines Group Limited +casa + +// case : 2015-09-03 CNH Industrial N.V. +case + +// caseih : 2015-09-03 CNH Industrial N.V. +caseih + +// cash : 2014-03-06 Binky Moon, LLC +cash + +// casino : 2014-12-18 Binky Moon, LLC +casino + +// catering : 2013-12-05 Binky Moon, LLC +catering + +// catholic : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +catholic + +// cba : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +cba + +// cbn : 2014-08-22 The Christian Broadcasting Network, Inc. +cbn + +// cbre : 2015-07-02 CBRE, Inc. +cbre + +// cbs : 2015-08-06 CBS Domains Inc. +cbs + +// ceb : 2015-04-09 The Corporate Executive Board Company +ceb + +// center : 2013-11-07 Binky Moon, LLC +center + +// ceo : 2013-11-07 CEOTLD Pty Ltd +ceo + +// cern : 2014-06-05 European Organization for Nuclear Research ("CERN") +cern + +// cfa : 2014-08-28 CFA Institute +cfa + +// cfd : 2014-12-11 DotCFD Registry Limited +cfd + +// chanel : 2015-04-09 Chanel International B.V. +chanel + +// channel : 2014-05-08 Charleston Road Registry Inc. +channel + +// charity : 2018-04-11 Binky Moon, LLC +charity + +// chase : 2015-04-30 JPMorgan Chase Bank, National Association +chase + +// chat : 2014-12-04 Binky Moon, LLC +chat + +// cheap : 2013-11-14 Binky Moon, LLC +cheap + +// chintai : 2015-06-11 CHINTAI Corporation +chintai + +// christmas : 2013-11-21 UNR Corp. +christmas + +// chrome : 2014-07-24 Charleston Road Registry Inc. +chrome + +// church : 2014-02-06 Binky Moon, LLC +church + +// cipriani : 2015-02-19 Hotel Cipriani Srl +cipriani + +// circle : 2014-12-18 Amazon Registry Services, Inc. +circle + +// cisco : 2014-12-22 Cisco Technology, Inc. +cisco + +// citadel : 2015-07-23 Citadel Domain LLC +citadel + +// citi : 2015-07-30 Citigroup Inc. +citi + +// citic : 2014-01-09 CITIC Group Corporation +citic + +// city : 2014-05-29 Binky Moon, LLC +city + +// cityeats : 2014-12-11 Lifestyle Domain Holdings, Inc. +cityeats + +// claims : 2014-03-20 Binky Moon, LLC +claims + +// cleaning : 2013-12-05 Binky Moon, LLC +cleaning + +// click : 2014-06-05 UNR Corp. +click + +// clinic : 2014-03-20 Binky Moon, LLC +clinic + +// clinique : 2015-10-01 The Estée Lauder Companies Inc. +clinique + +// clothing : 2013-08-27 Binky Moon, LLC +clothing + +// cloud : 2015-04-16 Aruba PEC S.p.A. +cloud + +// club : 2013-11-08 .CLUB DOMAINS, LLC +club + +// clubmed : 2015-06-25 Club Méditerranée S.A. +clubmed + +// coach : 2014-10-09 Binky Moon, LLC +coach + +// codes : 2013-10-31 Binky Moon, LLC +codes + +// coffee : 2013-10-17 Binky Moon, LLC +coffee + +// college : 2014-01-16 XYZ.COM LLC +college + +// cologne : 2014-02-05 dotKoeln GmbH +cologne + +// comcast : 2015-07-23 Comcast IP Holdings I, LLC +comcast + +// commbank : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +commbank + +// community : 2013-12-05 Binky Moon, LLC +community + +// company : 2013-11-07 Binky Moon, LLC +company + +// compare : 2015-10-08 Registry Services, LLC +compare + +// computer : 2013-10-24 Binky Moon, LLC +computer + +// comsec : 2015-01-08 VeriSign, Inc. +comsec + +// condos : 2013-12-05 Binky Moon, LLC +condos + +// construction : 2013-09-16 Binky Moon, LLC +construction + +// consulting : 2013-12-05 Dog Beach, LLC +consulting + +// contact : 2015-01-08 Dog Beach, LLC +contact + +// contractors : 2013-09-10 Binky Moon, LLC +contractors + +// cooking : 2013-11-21 Minds + Machines Group Limited +cooking + +// cookingchannel : 2015-07-02 Lifestyle Domain Holdings, Inc. +cookingchannel + +// cool : 2013-11-14 Binky Moon, LLC +cool + +// corsica : 2014-09-25 Collectivité de Corse +corsica + +// country : 2013-12-19 DotCountry LLC +country + +// coupon : 2015-02-26 Amazon Registry Services, Inc. +coupon + +// coupons : 2015-03-26 Binky Moon, LLC +coupons + +// courses : 2014-12-04 OPEN UNIVERSITIES AUSTRALIA PTY LTD +courses + +// cpa : 2019-06-10 American Institute of Certified Public Accountants +cpa + +// credit : 2014-03-20 Binky Moon, LLC +credit + +// creditcard : 2014-03-20 Binky Moon, LLC +creditcard + +// creditunion : 2015-01-22 CUNA Performance Resources, LLC +creditunion + +// cricket : 2014-10-09 dot Cricket Limited +cricket + +// crown : 2014-10-24 Crown Equipment Corporation +crown + +// crs : 2014-04-03 Federated Co-operatives Limited +crs + +// cruise : 2015-12-10 Viking River Cruises (Bermuda) Ltd. +cruise + +// cruises : 2013-12-05 Binky Moon, LLC +cruises + +// csc : 2014-09-25 Alliance-One Services, Inc. +csc + +// cuisinella : 2014-04-03 SCHMIDT GROUPE S.A.S. +cuisinella + +// cymru : 2014-05-08 Nominet UK +cymru + +// cyou : 2015-01-22 ShortDot SA +cyou + +// dabur : 2014-02-06 Dabur India Limited +dabur + +// dad : 2014-01-23 Charleston Road Registry Inc. +dad + +// dance : 2013-10-24 Dog Beach, LLC +dance + +// data : 2016-06-02 Dish DBS Corporation +data + +// date : 2014-11-20 dot Date Limited +date + +// dating : 2013-12-05 Binky Moon, LLC +dating + +// datsun : 2014-03-27 NISSAN MOTOR CO., LTD. +datsun + +// day : 2014-01-30 Charleston Road Registry Inc. +day + +// dclk : 2014-11-20 Charleston Road Registry Inc. +dclk + +// dds : 2015-05-07 Minds + Machines Group Limited +dds + +// deal : 2015-06-25 Amazon Registry Services, Inc. +deal + +// dealer : 2014-12-22 Intercap Registry Inc. +dealer + +// deals : 2014-05-22 Binky Moon, LLC +deals + +// degree : 2014-03-06 Dog Beach, LLC +degree + +// delivery : 2014-09-11 Binky Moon, LLC +delivery + +// dell : 2014-10-24 Dell Inc. +dell + +// deloitte : 2015-07-31 Deloitte Touche Tohmatsu +deloitte + +// delta : 2015-02-19 Delta Air Lines, Inc. +delta + +// democrat : 2013-10-24 Dog Beach, LLC +democrat + +// dental : 2014-03-20 Binky Moon, LLC +dental + +// dentist : 2014-03-20 Dog Beach, LLC +dentist + +// desi : 2013-11-14 Desi Networks LLC +desi + +// design : 2014-11-07 Top Level Design, LLC +design + +// dev : 2014-10-16 Charleston Road Registry Inc. +dev + +// dhl : 2015-07-23 Deutsche Post AG +dhl + +// diamonds : 2013-09-22 Binky Moon, LLC +diamonds + +// diet : 2014-06-26 UNR Corp. +diet + +// digital : 2014-03-06 Binky Moon, LLC +digital + +// direct : 2014-04-10 Binky Moon, LLC +direct + +// directory : 2013-09-20 Binky Moon, LLC +directory + +// discount : 2014-03-06 Binky Moon, LLC +discount + +// discover : 2015-07-23 Discover Financial Services +discover + +// dish : 2015-07-30 Dish DBS Corporation +dish + +// diy : 2015-11-05 Lifestyle Domain Holdings, Inc. +diy + +// dnp : 2013-12-13 Dai Nippon Printing Co., Ltd. +dnp + +// docs : 2014-10-16 Charleston Road Registry Inc. +docs + +// doctor : 2016-06-02 Binky Moon, LLC +doctor + +// dog : 2014-12-04 Binky Moon, LLC +dog + +// domains : 2013-10-17 Binky Moon, LLC +domains + +// dot : 2015-05-21 Dish DBS Corporation +dot + +// download : 2014-11-20 dot Support Limited +download + +// drive : 2015-03-05 Charleston Road Registry Inc. +drive + +// dtv : 2015-06-04 Dish DBS Corporation +dtv + +// dubai : 2015-01-01 Dubai Smart Government Department +dubai + +// duck : 2015-07-23 Johnson Shareholdings, Inc. +duck + +// dunlop : 2015-07-02 The Goodyear Tire & Rubber Company +dunlop + +// dupont : 2015-06-25 E. I. du Pont de Nemours and Company +dupont + +// durban : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +durban + +// dvag : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +dvag + +// dvr : 2016-05-26 DISH Technologies L.L.C. +dvr + +// earth : 2014-12-04 Interlink Co., Ltd. +earth + +// eat : 2014-01-23 Charleston Road Registry Inc. +eat + +// eco : 2016-07-08 Big Room Inc. +eco + +// edeka : 2014-12-18 EDEKA Verband kaufmännischer Genossenschaften e.V. +edeka + +// education : 2013-11-07 Binky Moon, LLC +education + +// email : 2013-10-31 Binky Moon, LLC +email + +// emerck : 2014-04-03 Merck KGaA +emerck + +// energy : 2014-09-11 Binky Moon, LLC +energy + +// engineer : 2014-03-06 Dog Beach, LLC +engineer + +// engineering : 2014-03-06 Binky Moon, LLC +engineering + +// enterprises : 2013-09-20 Binky Moon, LLC +enterprises + +// epson : 2014-12-04 Seiko Epson Corporation +epson + +// equipment : 2013-08-27 Binky Moon, LLC +equipment + +// ericsson : 2015-07-09 Telefonaktiebolaget L M Ericsson +ericsson + +// erni : 2014-04-03 ERNI Group Holding AG +erni + +// esq : 2014-05-08 Charleston Road Registry Inc. +esq + +// estate : 2013-08-27 Binky Moon, LLC +estate + +// etisalat : 2015-09-03 Emirates Telecommunications Corporation (trading as Etisalat) +etisalat + +// eurovision : 2014-04-24 European Broadcasting Union (EBU) +eurovision + +// eus : 2013-12-12 Puntueus Fundazioa +eus + +// events : 2013-12-05 Binky Moon, LLC +events + +// exchange : 2014-03-06 Binky Moon, LLC +exchange + +// expert : 2013-11-21 Binky Moon, LLC +expert + +// exposed : 2013-12-05 Binky Moon, LLC +exposed + +// express : 2015-02-11 Binky Moon, LLC +express + +// extraspace : 2015-05-14 Extra Space Storage LLC +extraspace + +// fage : 2014-12-18 Fage International S.A. +fage + +// fail : 2014-03-06 Binky Moon, LLC +fail + +// fairwinds : 2014-11-13 FairWinds Partners, LLC +fairwinds + +// faith : 2014-11-20 dot Faith Limited +faith + +// family : 2015-04-02 Dog Beach, LLC +family + +// fan : 2014-03-06 Dog Beach, LLC +fan + +// fans : 2014-11-07 ZDNS International Limited +fans + +// farm : 2013-11-07 Binky Moon, LLC +farm + +// farmers : 2015-07-09 Farmers Insurance Exchange +farmers + +// fashion : 2014-07-03 Minds + Machines Group Limited +fashion + +// fast : 2014-12-18 Amazon Registry Services, Inc. +fast + +// fedex : 2015-08-06 Federal Express Corporation +fedex + +// feedback : 2013-12-19 Top Level Spectrum, Inc. +feedback + +// ferrari : 2015-07-31 Fiat Chrysler Automobiles N.V. +ferrari + +// ferrero : 2014-12-18 Ferrero Trading Lux S.A. +ferrero + +// fiat : 2015-07-31 Fiat Chrysler Automobiles N.V. +fiat + +// fidelity : 2015-07-30 Fidelity Brokerage Services LLC +fidelity + +// fido : 2015-08-06 Rogers Communications Canada Inc. +fido + +// film : 2015-01-08 Motion Picture Domain Registry Pty Ltd +film + +// final : 2014-10-16 Núcleo de Informação e Coordenação do Ponto BR - NIC.br +final + +// finance : 2014-03-20 Binky Moon, LLC +finance + +// financial : 2014-03-06 Binky Moon, LLC +financial + +// fire : 2015-06-25 Amazon Registry Services, Inc. +fire + +// firestone : 2014-12-18 Bridgestone Licensing Services, Inc +firestone + +// firmdale : 2014-03-27 Firmdale Holdings Limited +firmdale + +// fish : 2013-12-12 Binky Moon, LLC +fish + +// fishing : 2013-11-21 Minds + Machines Group Limited +fishing + +// fit : 2014-11-07 Minds + Machines Group Limited +fit + +// fitness : 2014-03-06 Binky Moon, LLC +fitness + +// flickr : 2015-04-02 Flickr, Inc. +flickr + +// flights : 2013-12-05 Binky Moon, LLC +flights + +// flir : 2015-07-23 FLIR Systems, Inc. +flir + +// florist : 2013-11-07 Binky Moon, LLC +florist + +// flowers : 2014-10-09 UNR Corp. +flowers + +// fly : 2014-05-08 Charleston Road Registry Inc. +fly + +// foo : 2014-01-23 Charleston Road Registry Inc. +foo + +// food : 2016-04-21 Lifestyle Domain Holdings, Inc. +food + +// foodnetwork : 2015-07-02 Lifestyle Domain Holdings, Inc. +foodnetwork + +// football : 2014-12-18 Binky Moon, LLC +football + +// ford : 2014-11-13 Ford Motor Company +ford + +// forex : 2014-12-11 Dotforex Registry Limited +forex + +// forsale : 2014-05-22 Dog Beach, LLC +forsale + +// forum : 2015-04-02 Fegistry, LLC +forum + +// foundation : 2013-12-05 Binky Moon, LLC +foundation + +// fox : 2015-09-11 FOX Registry, LLC +fox + +// free : 2015-12-10 Amazon Registry Services, Inc. +free + +// fresenius : 2015-07-30 Fresenius Immobilien-Verwaltungs-GmbH +fresenius + +// frl : 2014-05-15 FRLregistry B.V. +frl + +// frogans : 2013-12-19 OP3FT +frogans + +// frontdoor : 2015-07-02 Lifestyle Domain Holdings, Inc. +frontdoor + +// frontier : 2015-02-05 Frontier Communications Corporation +frontier + +// ftr : 2015-07-16 Frontier Communications Corporation +ftr + +// fujitsu : 2015-07-30 Fujitsu Limited +fujitsu + +// fujixerox : 2015-07-23 Xerox DNHC LLC +fujixerox + +// fun : 2016-01-14 DotSpace Inc. +fun + +// fund : 2014-03-20 Binky Moon, LLC +fund + +// furniture : 2014-03-20 Binky Moon, LLC +furniture + +// futbol : 2013-09-20 Dog Beach, LLC +futbol + +// fyi : 2015-04-02 Binky Moon, LLC +fyi + +// gal : 2013-11-07 Asociación puntoGAL +gal + +// gallery : 2013-09-13 Binky Moon, LLC +gallery + +// gallo : 2015-06-11 Gallo Vineyards, Inc. +gallo + +// gallup : 2015-02-19 Gallup, Inc. +gallup + +// game : 2015-05-28 UNR Corp. +game + +// games : 2015-05-28 Dog Beach, LLC +games + +// gap : 2015-07-31 The Gap, Inc. +gap + +// garden : 2014-06-26 Minds + Machines Group Limited +garden + +// gay : 2019-05-23 Top Level Design, LLC +gay + +// gbiz : 2014-07-17 Charleston Road Registry Inc. +gbiz + +// gdn : 2014-07-31 Joint Stock Company "Navigation-information systems" +gdn + +// gea : 2014-12-04 GEA Group Aktiengesellschaft +gea + +// gent : 2014-01-23 COMBELL NV +gent + +// genting : 2015-03-12 Resorts World Inc Pte. Ltd. +genting + +// george : 2015-07-31 Wal-Mart Stores, Inc. +george + +// ggee : 2014-01-09 GMO Internet, Inc. +ggee + +// gift : 2013-10-17 DotGift, LLC +gift + +// gifts : 2014-07-03 Binky Moon, LLC +gifts + +// gives : 2014-03-06 Dog Beach, LLC +gives + +// giving : 2014-11-13 Giving Limited +giving + +// glade : 2015-07-23 Johnson Shareholdings, Inc. +glade + +// glass : 2013-11-07 Binky Moon, LLC +glass + +// gle : 2014-07-24 Charleston Road Registry Inc. +gle + +// global : 2014-04-17 Dot Global Domain Registry Limited +global + +// globo : 2013-12-19 Globo Comunicação e Participações S.A +globo + +// gmail : 2014-05-01 Charleston Road Registry Inc. +gmail + +// gmbh : 2016-01-29 Binky Moon, LLC +gmbh + +// gmo : 2014-01-09 GMO Internet, Inc. +gmo + +// gmx : 2014-04-24 1&1 Mail & Media GmbH +gmx + +// godaddy : 2015-07-23 Go Daddy East, LLC +godaddy + +// gold : 2015-01-22 Binky Moon, LLC +gold + +// goldpoint : 2014-11-20 YODOBASHI CAMERA CO.,LTD. +goldpoint + +// golf : 2014-12-18 Binky Moon, LLC +golf + +// goo : 2014-12-18 NTT Resonant Inc. +goo + +// goodyear : 2015-07-02 The Goodyear Tire & Rubber Company +goodyear + +// goog : 2014-11-20 Charleston Road Registry Inc. +goog + +// google : 2014-07-24 Charleston Road Registry Inc. +google + +// gop : 2014-01-16 Republican State Leadership Committee, Inc. +gop + +// got : 2014-12-18 Amazon Registry Services, Inc. +got + +// grainger : 2015-05-07 Grainger Registry Services, LLC +grainger + +// graphics : 2013-09-13 Binky Moon, LLC +graphics + +// gratis : 2014-03-20 Binky Moon, LLC +gratis + +// green : 2014-05-08 Afilias Limited +green + +// gripe : 2014-03-06 Binky Moon, LLC +gripe + +// grocery : 2016-06-16 Wal-Mart Stores, Inc. +grocery + +// group : 2014-08-15 Binky Moon, LLC +group + +// guardian : 2015-07-30 The Guardian Life Insurance Company of America +guardian + +// gucci : 2014-11-13 Guccio Gucci S.p.a. +gucci + +// guge : 2014-08-28 Charleston Road Registry Inc. +guge + +// guide : 2013-09-13 Binky Moon, LLC +guide + +// guitars : 2013-11-14 UNR Corp. +guitars + +// guru : 2013-08-27 Binky Moon, LLC +guru + +// hair : 2015-12-03 XYZ.COM LLC +hair + +// hamburg : 2014-02-20 Hamburg Top-Level-Domain GmbH +hamburg + +// hangout : 2014-11-13 Charleston Road Registry Inc. +hangout + +// haus : 2013-12-05 Dog Beach, LLC +haus + +// hbo : 2015-07-30 HBO Registry Services, Inc. +hbo + +// hdfc : 2015-07-30 HOUSING DEVELOPMENT FINANCE CORPORATION LIMITED +hdfc + +// hdfcbank : 2015-02-12 HDFC Bank Limited +hdfcbank + +// health : 2015-02-11 DotHealth, LLC +health + +// healthcare : 2014-06-12 Binky Moon, LLC +healthcare + +// help : 2014-06-26 UNR Corp. +help + +// helsinki : 2015-02-05 City of Helsinki +helsinki + +// here : 2014-02-06 Charleston Road Registry Inc. +here + +// hermes : 2014-07-10 HERMES INTERNATIONAL +hermes + +// hgtv : 2015-07-02 Lifestyle Domain Holdings, Inc. +hgtv + +// hiphop : 2014-03-06 UNR Corp. +hiphop + +// hisamitsu : 2015-07-16 Hisamitsu Pharmaceutical Co.,Inc. +hisamitsu + +// hitachi : 2014-10-31 Hitachi, Ltd. +hitachi + +// hiv : 2014-03-13 UNR Corp. +hiv + +// hkt : 2015-05-14 PCCW-HKT DataCom Services Limited +hkt + +// hockey : 2015-03-19 Binky Moon, LLC +hockey + +// holdings : 2013-08-27 Binky Moon, LLC +holdings + +// holiday : 2013-11-07 Binky Moon, LLC +holiday + +// homedepot : 2015-04-02 Home Depot Product Authority, LLC +homedepot + +// homegoods : 2015-07-16 The TJX Companies, Inc. +homegoods + +// homes : 2014-01-09 XYZ.COM LLC +homes + +// homesense : 2015-07-16 The TJX Companies, Inc. +homesense + +// honda : 2014-12-18 Honda Motor Co., Ltd. +honda + +// horse : 2013-11-21 Minds + Machines Group Limited +horse + +// hospital : 2016-10-20 Binky Moon, LLC +hospital + +// host : 2014-04-17 DotHost Inc. +host + +// hosting : 2014-05-29 UNR Corp. +hosting + +// hot : 2015-08-27 Amazon Registry Services, Inc. +hot + +// hoteles : 2015-03-05 Travel Reservations SRL +hoteles + +// hotels : 2016-04-07 Booking.com B.V. +hotels + +// hotmail : 2014-12-18 Microsoft Corporation +hotmail + +// house : 2013-11-07 Binky Moon, LLC +house + +// how : 2014-01-23 Charleston Road Registry Inc. +how + +// hsbc : 2014-10-24 HSBC Global Services (UK) Limited +hsbc + +// hughes : 2015-07-30 Hughes Satellite Systems Corporation +hughes + +// hyatt : 2015-07-30 Hyatt GTLD, L.L.C. +hyatt + +// hyundai : 2015-07-09 Hyundai Motor Company +hyundai + +// ibm : 2014-07-31 International Business Machines Corporation +ibm + +// icbc : 2015-02-19 Industrial and Commercial Bank of China Limited +icbc + +// ice : 2014-10-30 IntercontinentalExchange, Inc. +ice + +// icu : 2015-01-08 ShortDot SA +icu + +// ieee : 2015-07-23 IEEE Global LLC +ieee + +// ifm : 2014-01-30 ifm electronic gmbh +ifm + +// ikano : 2015-07-09 Ikano S.A. +ikano + +// imamat : 2015-08-06 Fondation Aga Khan (Aga Khan Foundation) +imamat + +// imdb : 2015-06-25 Amazon Registry Services, Inc. +imdb + +// immo : 2014-07-10 Binky Moon, LLC +immo + +// immobilien : 2013-11-07 Dog Beach, LLC +immobilien + +// inc : 2018-03-10 Intercap Registry Inc. +inc + +// industries : 2013-12-05 Binky Moon, LLC +industries + +// infiniti : 2014-03-27 NISSAN MOTOR CO., LTD. +infiniti + +// ing : 2014-01-23 Charleston Road Registry Inc. +ing + +// ink : 2013-12-05 Top Level Design, LLC +ink + +// institute : 2013-11-07 Binky Moon, LLC +institute + +// insurance : 2015-02-19 fTLD Registry Services LLC +insurance + +// insure : 2014-03-20 Binky Moon, LLC +insure + +// international : 2013-11-07 Binky Moon, LLC +international + +// intuit : 2015-07-30 Intuit Administrative Services, Inc. +intuit + +// investments : 2014-03-20 Binky Moon, LLC +investments + +// ipiranga : 2014-08-28 Ipiranga Produtos de Petroleo S.A. +ipiranga + +// irish : 2014-08-07 Binky Moon, LLC +irish + +// ismaili : 2015-08-06 Fondation Aga Khan (Aga Khan Foundation) +ismaili + +// ist : 2014-08-28 Istanbul Metropolitan Municipality +ist + +// istanbul : 2014-08-28 Istanbul Metropolitan Municipality +istanbul + +// itau : 2014-10-02 Itau Unibanco Holding S.A. +itau + +// itv : 2015-07-09 ITV Services Limited +itv + +// iveco : 2015-09-03 CNH Industrial N.V. +iveco + +// jaguar : 2014-11-13 Jaguar Land Rover Ltd +jaguar + +// java : 2014-06-19 Oracle Corporation +java + +// jcb : 2014-11-20 JCB Co., Ltd. +jcb + +// jeep : 2015-07-30 FCA US LLC. +jeep + +// jetzt : 2014-01-09 Binky Moon, LLC +jetzt + +// jewelry : 2015-03-05 Binky Moon, LLC +jewelry + +// jio : 2015-04-02 Reliance Industries Limited +jio + +// jll : 2015-04-02 Jones Lang LaSalle Incorporated +jll + +// jmp : 2015-03-26 Matrix IP LLC +jmp + +// jnj : 2015-06-18 Johnson & Johnson Services, Inc. +jnj + +// joburg : 2014-03-24 ZA Central Registry NPC trading as ZA Central Registry +joburg + +// jot : 2014-12-18 Amazon Registry Services, Inc. +jot + +// joy : 2014-12-18 Amazon Registry Services, Inc. +joy + +// jpmorgan : 2015-04-30 JPMorgan Chase Bank, National Association +jpmorgan + +// jprs : 2014-09-18 Japan Registry Services Co., Ltd. +jprs + +// juegos : 2014-03-20 UNR Corp. +juegos + +// juniper : 2015-07-30 JUNIPER NETWORKS, INC. +juniper + +// kaufen : 2013-11-07 Dog Beach, LLC +kaufen + +// kddi : 2014-09-12 KDDI CORPORATION +kddi + +// kerryhotels : 2015-04-30 Kerry Trading Co. Limited +kerryhotels + +// kerrylogistics : 2015-04-09 Kerry Trading Co. Limited +kerrylogistics + +// kerryproperties : 2015-04-09 Kerry Trading Co. Limited +kerryproperties + +// kfh : 2014-12-04 Kuwait Finance House +kfh + +// kia : 2015-07-09 KIA MOTORS CORPORATION +kia + +// kim : 2013-09-23 Afilias Limited +kim + +// kinder : 2014-11-07 Ferrero Trading Lux S.A. +kinder + +// kindle : 2015-06-25 Amazon Registry Services, Inc. +kindle + +// kitchen : 2013-09-20 Binky Moon, LLC +kitchen + +// kiwi : 2013-09-20 DOT KIWI LIMITED +kiwi + +// koeln : 2014-01-09 dotKoeln GmbH +koeln + +// komatsu : 2015-01-08 Komatsu Ltd. +komatsu + +// kosher : 2015-08-20 Kosher Marketing Assets LLC +kosher + +// kpmg : 2015-04-23 KPMG International Cooperative (KPMG International Genossenschaft) +kpmg + +// kpn : 2015-01-08 Koninklijke KPN N.V. +kpn + +// krd : 2013-12-05 KRG Department of Information Technology +krd + +// kred : 2013-12-19 KredTLD Pty Ltd +kred + +// kuokgroup : 2015-04-09 Kerry Trading Co. Limited +kuokgroup + +// kyoto : 2014-11-07 Academic Institution: Kyoto Jyoho Gakuen +kyoto + +// lacaixa : 2014-01-09 Fundación Bancaria Caixa d’Estalvis i Pensions de Barcelona, “la Caixa” +lacaixa + +// lamborghini : 2015-06-04 Automobili Lamborghini S.p.A. +lamborghini + +// lamer : 2015-10-01 The Estée Lauder Companies Inc. +lamer + +// lancaster : 2015-02-12 LANCASTER +lancaster + +// lancia : 2015-07-31 Fiat Chrysler Automobiles N.V. +lancia + +// land : 2013-09-10 Binky Moon, LLC +land + +// landrover : 2014-11-13 Jaguar Land Rover Ltd +landrover + +// lanxess : 2015-07-30 LANXESS Corporation +lanxess + +// lasalle : 2015-04-02 Jones Lang LaSalle Incorporated +lasalle + +// lat : 2014-10-16 ECOM-LAC Federaciòn de Latinoamèrica y el Caribe para Internet y el Comercio Electrònico +lat + +// latino : 2015-07-30 Dish DBS Corporation +latino + +// latrobe : 2014-06-16 La Trobe University +latrobe + +// law : 2015-01-22 LW TLD Limited +law + +// lawyer : 2014-03-20 Dog Beach, LLC +lawyer + +// lds : 2014-03-20 IRI Domain Management, LLC +lds + +// lease : 2014-03-06 Binky Moon, LLC +lease + +// leclerc : 2014-08-07 A.C.D. LEC Association des Centres Distributeurs Edouard Leclerc +leclerc + +// lefrak : 2015-07-16 LeFrak Organization, Inc. +lefrak + +// legal : 2014-10-16 Binky Moon, LLC +legal + +// lego : 2015-07-16 LEGO Juris A/S +lego + +// lexus : 2015-04-23 TOYOTA MOTOR CORPORATION +lexus + +// lgbt : 2014-05-08 Afilias Limited +lgbt + +// lidl : 2014-09-18 Schwarz Domains und Services GmbH & Co. KG +lidl + +// life : 2014-02-06 Binky Moon, LLC +life + +// lifeinsurance : 2015-01-15 American Council of Life Insurers +lifeinsurance + +// lifestyle : 2014-12-11 Lifestyle Domain Holdings, Inc. +lifestyle + +// lighting : 2013-08-27 Binky Moon, LLC +lighting + +// like : 2014-12-18 Amazon Registry Services, Inc. +like + +// lilly : 2015-07-31 Eli Lilly and Company +lilly + +// limited : 2014-03-06 Binky Moon, LLC +limited + +// limo : 2013-10-17 Binky Moon, LLC +limo + +// lincoln : 2014-11-13 Ford Motor Company +lincoln + +// linde : 2014-12-04 Linde Aktiengesellschaft +linde + +// link : 2013-11-14 UNR Corp. +link + +// lipsy : 2015-06-25 Lipsy Ltd +lipsy + +// live : 2014-12-04 Dog Beach, LLC +live + +// living : 2015-07-30 Lifestyle Domain Holdings, Inc. +living + +// lixil : 2015-03-19 LIXIL Group Corporation +lixil + +// llc : 2017-12-14 Afilias Limited +llc + +// llp : 2019-08-26 UNR Corp. +llp + +// loan : 2014-11-20 dot Loan Limited +loan + +// loans : 2014-03-20 Binky Moon, LLC +loans + +// locker : 2015-06-04 Dish DBS Corporation +locker + +// locus : 2015-06-25 Locus Analytics LLC +locus + +// loft : 2015-07-30 Annco, Inc. +loft + +// lol : 2015-01-30 UNR Corp. +lol + +// london : 2013-11-14 Dot London Domains Limited +london + +// lotte : 2014-11-07 Lotte Holdings Co., Ltd. +lotte + +// lotto : 2014-04-10 Afilias Limited +lotto + +// love : 2014-12-22 Merchant Law Group LLP +love + +// lpl : 2015-07-30 LPL Holdings, Inc. +lpl + +// lplfinancial : 2015-07-30 LPL Holdings, Inc. +lplfinancial + +// ltd : 2014-09-25 Binky Moon, LLC +ltd + +// ltda : 2014-04-17 InterNetX, Corp +ltda + +// lundbeck : 2015-08-06 H. Lundbeck A/S +lundbeck + +// lupin : 2014-11-07 LUPIN LIMITED +lupin + +// luxe : 2014-01-09 Minds + Machines Group Limited +luxe + +// luxury : 2013-10-17 Luxury Partners, LLC +luxury + +// macys : 2015-07-31 Macys, Inc. +macys + +// madrid : 2014-05-01 Comunidad de Madrid +madrid + +// maif : 2014-10-02 Mutuelle Assurance Instituteur France (MAIF) +maif + +// maison : 2013-12-05 Binky Moon, LLC +maison + +// makeup : 2015-01-15 XYZ.COM LLC +makeup + +// man : 2014-12-04 MAN SE +man + +// management : 2013-11-07 Binky Moon, LLC +management + +// mango : 2013-10-24 PUNTO FA S.L. +mango + +// map : 2016-06-09 Charleston Road Registry Inc. +map + +// market : 2014-03-06 Dog Beach, LLC +market + +// marketing : 2013-11-07 Binky Moon, LLC +marketing + +// markets : 2014-12-11 Dotmarkets Registry Limited +markets + +// marriott : 2014-10-09 Marriott Worldwide Corporation +marriott + +// marshalls : 2015-07-16 The TJX Companies, Inc. +marshalls + +// maserati : 2015-07-31 Fiat Chrysler Automobiles N.V. +maserati + +// mattel : 2015-08-06 Mattel Sites, Inc. +mattel + +// mba : 2015-04-02 Binky Moon, LLC +mba + +// mckinsey : 2015-07-31 McKinsey Holdings, Inc. +mckinsey + +// med : 2015-08-06 Medistry LLC +med + +// media : 2014-03-06 Binky Moon, LLC +media + +// meet : 2014-01-16 Charleston Road Registry Inc. +meet + +// melbourne : 2014-05-29 The Crown in right of the State of Victoria, represented by its Department of State Development, Business and Innovation +melbourne + +// meme : 2014-01-30 Charleston Road Registry Inc. +meme + +// memorial : 2014-10-16 Dog Beach, LLC +memorial + +// men : 2015-02-26 Exclusive Registry Limited +men + +// menu : 2013-09-11 Dot Menu Registry, LLC +menu + +// merckmsd : 2016-07-14 MSD Registry Holdings, Inc. +merckmsd + +// miami : 2013-12-19 Minds + Machines Group Limited +miami + +// microsoft : 2014-12-18 Microsoft Corporation +microsoft + +// mini : 2014-01-09 Bayerische Motoren Werke Aktiengesellschaft +mini + +// mint : 2015-07-30 Intuit Administrative Services, Inc. +mint + +// mit : 2015-07-02 Massachusetts Institute of Technology +mit + +// mitsubishi : 2015-07-23 Mitsubishi Corporation +mitsubishi + +// mlb : 2015-05-21 MLB Advanced Media DH, LLC +mlb + +// mls : 2015-04-23 The Canadian Real Estate Association +mls + +// mma : 2014-11-07 MMA IARD +mma + +// mobile : 2016-06-02 Dish DBS Corporation +mobile + +// moda : 2013-11-07 Dog Beach, LLC +moda + +// moe : 2013-11-13 Interlink Co., Ltd. +moe + +// moi : 2014-12-18 Amazon Registry Services, Inc. +moi + +// mom : 2015-04-16 UNR Corp. +mom + +// monash : 2013-09-30 Monash University +monash + +// money : 2014-10-16 Binky Moon, LLC +money + +// monster : 2015-09-11 XYZ.COM LLC +monster + +// mormon : 2013-12-05 IRI Domain Management, LLC +mormon + +// mortgage : 2014-03-20 Dog Beach, LLC +mortgage + +// moscow : 2013-12-19 Foundation for Assistance for Internet Technologies and Infrastructure Development (FAITID) +moscow + +// moto : 2015-06-04 Motorola Trademark Holdings, LLC +moto + +// motorcycles : 2014-01-09 XYZ.COM LLC +motorcycles + +// mov : 2014-01-30 Charleston Road Registry Inc. +mov + +// movie : 2015-02-05 Binky Moon, LLC +movie + +// msd : 2015-07-23 MSD Registry Holdings, Inc. +msd + +// mtn : 2014-12-04 MTN Dubai Limited +mtn + +// mtr : 2015-03-12 MTR Corporation Limited +mtr + +// mutual : 2015-04-02 Northwestern Mutual MU TLD Registry, LLC +mutual + +// nab : 2015-08-20 National Australia Bank Limited +nab + +// nagoya : 2013-10-24 GMO Registry, Inc. +nagoya + +// nationwide : 2015-07-23 Nationwide Mutual Insurance Company +nationwide + +// natura : 2015-03-12 NATURA COSMÉTICOS S.A. +natura + +// navy : 2014-03-06 Dog Beach, LLC +navy + +// nba : 2015-07-31 NBA REGISTRY, LLC +nba + +// nec : 2015-01-08 NEC Corporation +nec + +// netbank : 2014-06-26 COMMONWEALTH BANK OF AUSTRALIA +netbank + +// netflix : 2015-06-18 Netflix, Inc. +netflix + +// network : 2013-11-14 Binky Moon, LLC +network + +// neustar : 2013-12-05 NeuStar, Inc. +neustar + +// new : 2014-01-30 Charleston Road Registry Inc. +new + +// newholland : 2015-09-03 CNH Industrial N.V. +newholland + +// news : 2014-12-18 Dog Beach, LLC +news + +// next : 2015-06-18 Next plc +next + +// nextdirect : 2015-06-18 Next plc +nextdirect + +// nexus : 2014-07-24 Charleston Road Registry Inc. +nexus + +// nfl : 2015-07-23 NFL Reg Ops LLC +nfl + +// ngo : 2014-03-06 Public Interest Registry +ngo + +// nhk : 2014-02-13 Japan Broadcasting Corporation (NHK) +nhk + +// nico : 2014-12-04 DWANGO Co., Ltd. +nico + +// nike : 2015-07-23 NIKE, Inc. +nike + +// nikon : 2015-05-21 NIKON CORPORATION +nikon + +// ninja : 2013-11-07 Dog Beach, LLC +ninja + +// nissan : 2014-03-27 NISSAN MOTOR CO., LTD. +nissan + +// nissay : 2015-10-29 Nippon Life Insurance Company +nissay + +// nokia : 2015-01-08 Nokia Corporation +nokia + +// northwesternmutual : 2015-06-18 Northwestern Mutual Registry, LLC +northwesternmutual + +// norton : 2014-12-04 NortonLifeLock Inc. +norton + +// now : 2015-06-25 Amazon Registry Services, Inc. +now + +// nowruz : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +nowruz + +// nowtv : 2015-05-14 Starbucks (HK) Limited +nowtv + +// nra : 2014-05-22 NRA Holdings Company, INC. +nra + +// nrw : 2013-11-21 Minds + Machines GmbH +nrw + +// ntt : 2014-10-31 NIPPON TELEGRAPH AND TELEPHONE CORPORATION +ntt + +// nyc : 2014-01-23 The City of New York by and through the New York City Department of Information Technology & Telecommunications +nyc + +// obi : 2014-09-25 OBI Group Holding SE & Co. KGaA +obi + +// observer : 2015-04-30 Top Level Spectrum, Inc. +observer + +// off : 2015-07-23 Johnson Shareholdings, Inc. +off + +// office : 2015-03-12 Microsoft Corporation +office + +// okinawa : 2013-12-05 BRregistry, Inc. +okinawa + +// olayan : 2015-05-14 Crescent Holding GmbH +olayan + +// olayangroup : 2015-05-14 Crescent Holding GmbH +olayangroup + +// oldnavy : 2015-07-31 The Gap, Inc. +oldnavy + +// ollo : 2015-06-04 Dish DBS Corporation +ollo + +// omega : 2015-01-08 The Swatch Group Ltd +omega + +// one : 2014-11-07 One.com A/S +one + +// ong : 2014-03-06 Public Interest Registry +ong + +// onl : 2013-09-16 iRegistry GmbH +onl + +// online : 2015-01-15 DotOnline Inc. +online + +// onyourside : 2015-07-23 Nationwide Mutual Insurance Company +onyourside + +// ooo : 2014-01-09 INFIBEAM AVENUES LIMITED +ooo + +// open : 2015-07-31 American Express Travel Related Services Company, Inc. +open + +// oracle : 2014-06-19 Oracle Corporation +oracle + +// orange : 2015-03-12 Orange Brand Services Limited +orange + +// organic : 2014-03-27 Afilias Limited +organic + +// origins : 2015-10-01 The Estée Lauder Companies Inc. +origins + +// osaka : 2014-09-04 Osaka Registry Co., Ltd. +osaka + +// otsuka : 2013-10-11 Otsuka Holdings Co., Ltd. +otsuka + +// ott : 2015-06-04 Dish DBS Corporation +ott + +// ovh : 2014-01-16 MédiaBC +ovh + +// page : 2014-12-04 Charleston Road Registry Inc. +page + +// panasonic : 2015-07-30 Panasonic Corporation +panasonic + +// paris : 2014-01-30 City of Paris +paris + +// pars : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +pars + +// partners : 2013-12-05 Binky Moon, LLC +partners + +// parts : 2013-12-05 Binky Moon, LLC +parts + +// party : 2014-09-11 Blue Sky Registry Limited +party + +// passagens : 2015-03-05 Travel Reservations SRL +passagens + +// pay : 2015-08-27 Amazon Registry Services, Inc. +pay + +// pccw : 2015-05-14 PCCW Enterprises Limited +pccw + +// pet : 2015-05-07 Afilias Limited +pet + +// pfizer : 2015-09-11 Pfizer Inc. +pfizer + +// pharmacy : 2014-06-19 National Association of Boards of Pharmacy +pharmacy + +// phd : 2016-07-28 Charleston Road Registry Inc. +phd + +// philips : 2014-11-07 Koninklijke Philips N.V. +philips + +// phone : 2016-06-02 Dish DBS Corporation +phone + +// photo : 2013-11-14 UNR Corp. +photo + +// photography : 2013-09-20 Binky Moon, LLC +photography + +// photos : 2013-10-17 Binky Moon, LLC +photos + +// physio : 2014-05-01 PhysBiz Pty Ltd +physio + +// pics : 2013-11-14 UNR Corp. +pics + +// pictet : 2014-06-26 Pictet Europe S.A. +pictet + +// pictures : 2014-03-06 Binky Moon, LLC +pictures + +// pid : 2015-01-08 Top Level Spectrum, Inc. +pid + +// pin : 2014-12-18 Amazon Registry Services, Inc. +pin + +// ping : 2015-06-11 Ping Registry Provider, Inc. +ping + +// pink : 2013-10-01 Afilias Limited +pink + +// pioneer : 2015-07-16 Pioneer Corporation +pioneer + +// pizza : 2014-06-26 Binky Moon, LLC +pizza + +// place : 2014-04-24 Binky Moon, LLC +place + +// play : 2015-03-05 Charleston Road Registry Inc. +play + +// playstation : 2015-07-02 Sony Interactive Entertainment Inc. +playstation + +// plumbing : 2013-09-10 Binky Moon, LLC +plumbing + +// plus : 2015-02-05 Binky Moon, LLC +plus + +// pnc : 2015-07-02 PNC Domain Co., LLC +pnc + +// pohl : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +pohl + +// poker : 2014-07-03 Afilias Limited +poker + +// politie : 2015-08-20 Politie Nederland +politie + +// porn : 2014-10-16 ICM Registry PN LLC +porn + +// pramerica : 2015-07-30 Prudential Financial, Inc. +pramerica + +// praxi : 2013-12-05 Praxi S.p.A. +praxi + +// press : 2014-04-03 DotPress Inc. +press + +// prime : 2015-06-25 Amazon Registry Services, Inc. +prime + +// prod : 2014-01-23 Charleston Road Registry Inc. +prod + +// productions : 2013-12-05 Binky Moon, LLC +productions + +// prof : 2014-07-24 Charleston Road Registry Inc. +prof + +// progressive : 2015-07-23 Progressive Casualty Insurance Company +progressive + +// promo : 2014-12-18 Afilias Limited +promo + +// properties : 2013-12-05 Binky Moon, LLC +properties + +// property : 2014-05-22 UNR Corp. +property + +// protection : 2015-04-23 XYZ.COM LLC +protection + +// pru : 2015-07-30 Prudential Financial, Inc. +pru + +// prudential : 2015-07-30 Prudential Financial, Inc. +prudential + +// pub : 2013-12-12 Dog Beach, LLC +pub + +// pwc : 2015-10-29 PricewaterhouseCoopers LLP +pwc + +// qpon : 2013-11-14 dotCOOL, Inc. +qpon + +// quebec : 2013-12-19 PointQuébec Inc +quebec + +// quest : 2015-03-26 XYZ.COM LLC +quest + +// qvc : 2015-07-30 QVC, Inc. +qvc + +// racing : 2014-12-04 Premier Registry Limited +racing + +// radio : 2016-07-21 European Broadcasting Union (EBU) +radio + +// raid : 2015-07-23 Johnson Shareholdings, Inc. +raid + +// read : 2014-12-18 Amazon Registry Services, Inc. +read + +// realestate : 2015-09-11 dotRealEstate LLC +realestate + +// realtor : 2014-05-29 Real Estate Domains LLC +realtor + +// realty : 2015-03-19 Fegistry, LLC +realty + +// recipes : 2013-10-17 Binky Moon, LLC +recipes + +// red : 2013-11-07 Afilias Limited +red + +// redstone : 2014-10-31 Redstone Haute Couture Co., Ltd. +redstone + +// redumbrella : 2015-03-26 Travelers TLD, LLC +redumbrella + +// rehab : 2014-03-06 Dog Beach, LLC +rehab + +// reise : 2014-03-13 Binky Moon, LLC +reise + +// reisen : 2014-03-06 Binky Moon, LLC +reisen + +// reit : 2014-09-04 National Association of Real Estate Investment Trusts, Inc. +reit + +// reliance : 2015-04-02 Reliance Industries Limited +reliance + +// ren : 2013-12-12 ZDNS International Limited +ren + +// rent : 2014-12-04 XYZ.COM LLC +rent + +// rentals : 2013-12-05 Binky Moon, LLC +rentals + +// repair : 2013-11-07 Binky Moon, LLC +repair + +// report : 2013-12-05 Binky Moon, LLC +report + +// republican : 2014-03-20 Dog Beach, LLC +republican + +// rest : 2013-12-19 Punto 2012 Sociedad Anonima Promotora de Inversion de Capital Variable +rest + +// restaurant : 2014-07-03 Binky Moon, LLC +restaurant + +// review : 2014-11-20 dot Review Limited +review + +// reviews : 2013-09-13 Dog Beach, LLC +reviews + +// rexroth : 2015-06-18 Robert Bosch GMBH +rexroth + +// rich : 2013-11-21 iRegistry GmbH +rich + +// richardli : 2015-05-14 Pacific Century Asset Management (HK) Limited +richardli + +// ricoh : 2014-11-20 Ricoh Company, Ltd. +ricoh + +// ril : 2015-04-02 Reliance Industries Limited +ril + +// rio : 2014-02-27 Empresa Municipal de Informática SA - IPLANRIO +rio + +// rip : 2014-07-10 Dog Beach, LLC +rip + +// rmit : 2015-11-19 Royal Melbourne Institute of Technology +rmit + +// rocher : 2014-12-18 Ferrero Trading Lux S.A. +rocher + +// rocks : 2013-11-14 Dog Beach, LLC +rocks + +// rodeo : 2013-12-19 Minds + Machines Group Limited +rodeo + +// rogers : 2015-08-06 Rogers Communications Canada Inc. +rogers + +// room : 2014-12-18 Amazon Registry Services, Inc. +room + +// rsvp : 2014-05-08 Charleston Road Registry Inc. +rsvp + +// rugby : 2016-12-15 World Rugby Strategic Developments Limited +rugby + +// ruhr : 2013-10-02 regiodot GmbH & Co. KG +ruhr + +// run : 2015-03-19 Binky Moon, LLC +run + +// rwe : 2015-04-02 RWE AG +rwe + +// ryukyu : 2014-01-09 BRregistry, Inc. +ryukyu + +// saarland : 2013-12-12 dotSaarland GmbH +saarland + +// safe : 2014-12-18 Amazon Registry Services, Inc. +safe + +// safety : 2015-01-08 Safety Registry Services, LLC. +safety + +// sakura : 2014-12-18 SAKURA Internet Inc. +sakura + +// sale : 2014-10-16 Dog Beach, LLC +sale + +// salon : 2014-12-11 Binky Moon, LLC +salon + +// samsclub : 2015-07-31 Wal-Mart Stores, Inc. +samsclub + +// samsung : 2014-04-03 SAMSUNG SDS CO., LTD +samsung + +// sandvik : 2014-11-13 Sandvik AB +sandvik + +// sandvikcoromant : 2014-11-07 Sandvik AB +sandvikcoromant + +// sanofi : 2014-10-09 Sanofi +sanofi + +// sap : 2014-03-27 SAP AG +sap + +// sarl : 2014-07-03 Binky Moon, LLC +sarl + +// sas : 2015-04-02 Research IP LLC +sas + +// save : 2015-06-25 Amazon Registry Services, Inc. +save + +// saxo : 2014-10-31 Saxo Bank A/S +saxo + +// sbi : 2015-03-12 STATE BANK OF INDIA +sbi + +// sbs : 2014-11-07 SPECIAL BROADCASTING SERVICE CORPORATION +sbs + +// sca : 2014-03-13 SVENSKA CELLULOSA AKTIEBOLAGET SCA (publ) +sca + +// scb : 2014-02-20 The Siam Commercial Bank Public Company Limited ("SCB") +scb + +// schaeffler : 2015-08-06 Schaeffler Technologies AG & Co. KG +schaeffler + +// schmidt : 2014-04-03 SCHMIDT GROUPE S.A.S. +schmidt + +// scholarships : 2014-04-24 Scholarships.com, LLC +scholarships + +// school : 2014-12-18 Binky Moon, LLC +school + +// schule : 2014-03-06 Binky Moon, LLC +schule + +// schwarz : 2014-09-18 Schwarz Domains und Services GmbH & Co. KG +schwarz + +// science : 2014-09-11 dot Science Limited +science + +// scjohnson : 2015-07-23 Johnson Shareholdings, Inc. +scjohnson + +// scot : 2014-01-23 Dot Scot Registry Limited +scot + +// search : 2016-06-09 Charleston Road Registry Inc. +search + +// seat : 2014-05-22 SEAT, S.A. (Sociedad Unipersonal) +seat + +// secure : 2015-08-27 Amazon Registry Services, Inc. +secure + +// security : 2015-05-14 XYZ.COM LLC +security + +// seek : 2014-12-04 Seek Limited +seek + +// select : 2015-10-08 Registry Services, LLC +select + +// sener : 2014-10-24 Sener Ingeniería y Sistemas, S.A. +sener + +// services : 2014-02-27 Binky Moon, LLC +services + +// ses : 2015-07-23 SES +ses + +// seven : 2015-08-06 Seven West Media Ltd +seven + +// sew : 2014-07-17 SEW-EURODRIVE GmbH & Co KG +sew + +// sex : 2014-11-13 ICM Registry SX LLC +sex + +// sexy : 2013-09-11 UNR Corp. +sexy + +// sfr : 2015-08-13 Societe Francaise du Radiotelephone - SFR +sfr + +// shangrila : 2015-09-03 Shangri‐La International Hotel Management Limited +shangrila + +// sharp : 2014-05-01 Sharp Corporation +sharp + +// shaw : 2015-04-23 Shaw Cablesystems G.P. +shaw + +// shell : 2015-07-30 Shell Information Technology International Inc +shell + +// shia : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +shia + +// shiksha : 2013-11-14 Afilias Limited +shiksha + +// shoes : 2013-10-02 Binky Moon, LLC +shoes + +// shop : 2016-04-08 GMO Registry, Inc. +shop + +// shopping : 2016-03-31 Binky Moon, LLC +shopping + +// shouji : 2015-01-08 Beijing Qihu Keji Co., Ltd. +shouji + +// show : 2015-03-05 Binky Moon, LLC +show + +// showtime : 2015-08-06 CBS Domains Inc. +showtime + +// silk : 2015-06-25 Amazon Registry Services, Inc. +silk + +// sina : 2015-03-12 Sina Corporation +sina + +// singles : 2013-08-27 Binky Moon, LLC +singles + +// site : 2015-01-15 DotSite Inc. +site + +// ski : 2015-04-09 Afilias Limited +ski + +// skin : 2015-01-15 XYZ.COM LLC +skin + +// sky : 2014-06-19 Sky International AG +sky + +// skype : 2014-12-18 Microsoft Corporation +skype + +// sling : 2015-07-30 DISH Technologies L.L.C. +sling + +// smart : 2015-07-09 Smart Communications, Inc. (SMART) +smart + +// smile : 2014-12-18 Amazon Registry Services, Inc. +smile + +// sncf : 2015-02-19 Société Nationale des Chemins de fer Francais S N C F +sncf + +// soccer : 2015-03-26 Binky Moon, LLC +soccer + +// social : 2013-11-07 Dog Beach, LLC +social + +// softbank : 2015-07-02 SoftBank Group Corp. +softbank + +// software : 2014-03-20 Dog Beach, LLC +software + +// sohu : 2013-12-19 Sohu.com Limited +sohu + +// solar : 2013-11-07 Binky Moon, LLC +solar + +// solutions : 2013-11-07 Binky Moon, LLC +solutions + +// song : 2015-02-26 Amazon Registry Services, Inc. +song + +// sony : 2015-01-08 Sony Corporation +sony + +// soy : 2014-01-23 Charleston Road Registry Inc. +soy + +// spa : 2019-09-19 Asia Spa and Wellness Promotion Council Limited +spa + +// space : 2014-04-03 DotSpace Inc. +space + +// sport : 2017-11-16 Global Association of International Sports Federations (GAISF) +sport + +// spot : 2015-02-26 Amazon Registry Services, Inc. +spot + +// spreadbetting : 2014-12-11 Dotspreadbetting Registry Limited +spreadbetting + +// srl : 2015-05-07 InterNetX, Corp +srl + +// stada : 2014-11-13 STADA Arzneimittel AG +stada + +// staples : 2015-07-30 Staples, Inc. +staples + +// star : 2015-01-08 Star India Private Limited +star + +// statebank : 2015-03-12 STATE BANK OF INDIA +statebank + +// statefarm : 2015-07-30 State Farm Mutual Automobile Insurance Company +statefarm + +// stc : 2014-10-09 Saudi Telecom Company +stc + +// stcgroup : 2014-10-09 Saudi Telecom Company +stcgroup + +// stockholm : 2014-12-18 Stockholms kommun +stockholm + +// storage : 2014-12-22 XYZ.COM LLC +storage + +// store : 2015-04-09 DotStore Inc. +store + +// stream : 2016-01-08 dot Stream Limited +stream + +// studio : 2015-02-11 Dog Beach, LLC +studio + +// study : 2014-12-11 OPEN UNIVERSITIES AUSTRALIA PTY LTD +study + +// style : 2014-12-04 Binky Moon, LLC +style + +// sucks : 2014-12-22 Vox Populi Registry Ltd. +sucks + +// supplies : 2013-12-19 Binky Moon, LLC +supplies + +// supply : 2013-12-19 Binky Moon, LLC +supply + +// support : 2013-10-24 Binky Moon, LLC +support + +// surf : 2014-01-09 Minds + Machines Group Limited +surf + +// surgery : 2014-03-20 Binky Moon, LLC +surgery + +// suzuki : 2014-02-20 SUZUKI MOTOR CORPORATION +suzuki + +// swatch : 2015-01-08 The Swatch Group Ltd +swatch + +// swiftcover : 2015-07-23 Swiftcover Insurance Services Limited +swiftcover + +// swiss : 2014-10-16 Swiss Confederation +swiss + +// sydney : 2014-09-18 State of New South Wales, Department of Premier and Cabinet +sydney + +// systems : 2013-11-07 Binky Moon, LLC +systems + +// tab : 2014-12-04 Tabcorp Holdings Limited +tab + +// taipei : 2014-07-10 Taipei City Government +taipei + +// talk : 2015-04-09 Amazon Registry Services, Inc. +talk + +// taobao : 2015-01-15 Alibaba Group Holding Limited +taobao + +// target : 2015-07-31 Target Domain Holdings, LLC +target + +// tatamotors : 2015-03-12 Tata Motors Ltd +tatamotors + +// tatar : 2014-04-24 Limited Liability Company "Coordination Center of Regional Domain of Tatarstan Republic" +tatar + +// tattoo : 2013-08-30 UNR Corp. +tattoo + +// tax : 2014-03-20 Binky Moon, LLC +tax + +// taxi : 2015-03-19 Binky Moon, LLC +taxi + +// tci : 2014-09-12 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +tci + +// tdk : 2015-06-11 TDK Corporation +tdk + +// team : 2015-03-05 Binky Moon, LLC +team + +// tech : 2015-01-30 Personals TLD Inc. +tech + +// technology : 2013-09-13 Binky Moon, LLC +technology + +// temasek : 2014-08-07 Temasek Holdings (Private) Limited +temasek + +// tennis : 2014-12-04 Binky Moon, LLC +tennis + +// teva : 2015-07-02 Teva Pharmaceutical Industries Limited +teva + +// thd : 2015-04-02 Home Depot Product Authority, LLC +thd + +// theater : 2015-03-19 Binky Moon, LLC +theater + +// theatre : 2015-05-07 XYZ.COM LLC +theatre + +// tiaa : 2015-07-23 Teachers Insurance and Annuity Association of America +tiaa + +// tickets : 2015-02-05 Accent Media Limited +tickets + +// tienda : 2013-11-14 Binky Moon, LLC +tienda + +// tiffany : 2015-01-30 Tiffany and Company +tiffany + +// tips : 2013-09-20 Binky Moon, LLC +tips + +// tires : 2014-11-07 Binky Moon, LLC +tires + +// tirol : 2014-04-24 punkt Tirol GmbH +tirol + +// tjmaxx : 2015-07-16 The TJX Companies, Inc. +tjmaxx + +// tjx : 2015-07-16 The TJX Companies, Inc. +tjx + +// tkmaxx : 2015-07-16 The TJX Companies, Inc. +tkmaxx + +// tmall : 2015-01-15 Alibaba Group Holding Limited +tmall + +// today : 2013-09-20 Binky Moon, LLC +today + +// tokyo : 2013-11-13 GMO Registry, Inc. +tokyo + +// tools : 2013-11-21 Binky Moon, LLC +tools + +// top : 2014-03-20 .TOP Registry +top + +// toray : 2014-12-18 Toray Industries, Inc. +toray + +// toshiba : 2014-04-10 TOSHIBA Corporation +toshiba + +// total : 2015-08-06 Total SA +total + +// tours : 2015-01-22 Binky Moon, LLC +tours + +// town : 2014-03-06 Binky Moon, LLC +town + +// toyota : 2015-04-23 TOYOTA MOTOR CORPORATION +toyota + +// toys : 2014-03-06 Binky Moon, LLC +toys + +// trade : 2014-01-23 Elite Registry Limited +trade + +// trading : 2014-12-11 Dottrading Registry Limited +trading + +// training : 2013-11-07 Binky Moon, LLC +training + +// travel : 2015-10-09 Dog Beach, LLC +travel + +// travelchannel : 2015-07-02 Lifestyle Domain Holdings, Inc. +travelchannel + +// travelers : 2015-03-26 Travelers TLD, LLC +travelers + +// travelersinsurance : 2015-03-26 Travelers TLD, LLC +travelersinsurance + +// trust : 2014-10-16 UNR Corp. +trust + +// trv : 2015-03-26 Travelers TLD, LLC +trv + +// tube : 2015-06-11 Latin American Telecom LLC +tube + +// tui : 2014-07-03 TUI AG +tui + +// tunes : 2015-02-26 Amazon Registry Services, Inc. +tunes + +// tushu : 2014-12-18 Amazon Registry Services, Inc. +tushu + +// tvs : 2015-02-19 T V SUNDRAM IYENGAR & SONS LIMITED +tvs + +// ubank : 2015-08-20 National Australia Bank Limited +ubank + +// ubs : 2014-12-11 UBS AG +ubs + +// unicom : 2015-10-15 China United Network Communications Corporation Limited +unicom + +// university : 2014-03-06 Binky Moon, LLC +university + +// uno : 2013-09-11 DotSite Inc. +uno + +// uol : 2014-05-01 UBN INTERNET LTDA. +uol + +// ups : 2015-06-25 UPS Market Driver, Inc. +ups + +// vacations : 2013-12-05 Binky Moon, LLC +vacations + +// vana : 2014-12-11 Lifestyle Domain Holdings, Inc. +vana + +// vanguard : 2015-09-03 The Vanguard Group, Inc. +vanguard + +// vegas : 2014-01-16 Dot Vegas, Inc. +vegas + +// ventures : 2013-08-27 Binky Moon, LLC +ventures + +// verisign : 2015-08-13 VeriSign, Inc. +verisign + +// versicherung : 2014-03-20 tldbox GmbH +versicherung + +// vet : 2014-03-06 Dog Beach, LLC +vet + +// viajes : 2013-10-17 Binky Moon, LLC +viajes + +// video : 2014-10-16 Dog Beach, LLC +video + +// vig : 2015-05-14 VIENNA INSURANCE GROUP AG Wiener Versicherung Gruppe +vig + +// viking : 2015-04-02 Viking River Cruises (Bermuda) Ltd. +viking + +// villas : 2013-12-05 Binky Moon, LLC +villas + +// vin : 2015-06-18 Binky Moon, LLC +vin + +// vip : 2015-01-22 Minds + Machines Group Limited +vip + +// virgin : 2014-09-25 Virgin Enterprises Limited +virgin + +// visa : 2015-07-30 Visa Worldwide Pte. Limited +visa + +// vision : 2013-12-05 Binky Moon, LLC +vision + +// viva : 2014-11-07 Saudi Telecom Company +viva + +// vivo : 2015-07-31 Telefonica Brasil S.A. +vivo + +// vlaanderen : 2014-02-06 DNS.be vzw +vlaanderen + +// vodka : 2013-12-19 Minds + Machines Group Limited +vodka + +// volkswagen : 2015-05-14 Volkswagen Group of America Inc. +volkswagen + +// volvo : 2015-11-12 Volvo Holding Sverige Aktiebolag +volvo + +// vote : 2013-11-21 Monolith Registry LLC +vote + +// voting : 2013-11-13 Valuetainment Corp. +voting + +// voto : 2013-11-21 Monolith Registry LLC +voto + +// voyage : 2013-08-27 Binky Moon, LLC +voyage + +// vuelos : 2015-03-05 Travel Reservations SRL +vuelos + +// wales : 2014-05-08 Nominet UK +wales + +// walmart : 2015-07-31 Wal-Mart Stores, Inc. +walmart + +// walter : 2014-11-13 Sandvik AB +walter + +// wang : 2013-10-24 Zodiac Wang Limited +wang + +// wanggou : 2014-12-18 Amazon Registry Services, Inc. +wanggou + +// watch : 2013-11-14 Binky Moon, LLC +watch + +// watches : 2014-12-22 Richemont DNS Inc. +watches + +// weather : 2015-01-08 International Business Machines Corporation +weather + +// weatherchannel : 2015-03-12 International Business Machines Corporation +weatherchannel + +// webcam : 2014-01-23 dot Webcam Limited +webcam + +// weber : 2015-06-04 Saint-Gobain Weber SA +weber + +// website : 2014-04-03 DotWebsite Inc. +website + +// wedding : 2014-04-24 Minds + Machines Group Limited +wedding + +// weibo : 2015-03-05 Sina Corporation +weibo + +// weir : 2015-01-29 Weir Group IP Limited +weir + +// whoswho : 2014-02-20 Who's Who Registry +whoswho + +// wien : 2013-10-28 punkt.wien GmbH +wien + +// wiki : 2013-11-07 Top Level Design, LLC +wiki + +// williamhill : 2014-03-13 William Hill Organization Limited +williamhill + +// win : 2014-11-20 First Registry Limited +win + +// windows : 2014-12-18 Microsoft Corporation +windows + +// wine : 2015-06-18 Binky Moon, LLC +wine + +// winners : 2015-07-16 The TJX Companies, Inc. +winners + +// wme : 2014-02-13 William Morris Endeavor Entertainment, LLC +wme + +// wolterskluwer : 2015-08-06 Wolters Kluwer N.V. +wolterskluwer + +// woodside : 2015-07-09 Woodside Petroleum Limited +woodside + +// work : 2013-12-19 Minds + Machines Group Limited +work + +// works : 2013-11-14 Binky Moon, LLC +works + +// world : 2014-06-12 Binky Moon, LLC +world + +// wow : 2015-10-08 Amazon Registry Services, Inc. +wow + +// wtc : 2013-12-19 World Trade Centers Association, Inc. +wtc + +// wtf : 2014-03-06 Binky Moon, LLC +wtf + +// xbox : 2014-12-18 Microsoft Corporation +xbox + +// xerox : 2014-10-24 Xerox DNHC LLC +xerox + +// xfinity : 2015-07-09 Comcast IP Holdings I, LLC +xfinity + +// xihuan : 2015-01-08 Beijing Qihu Keji Co., Ltd. +xihuan + +// xin : 2014-12-11 Elegant Leader Limited +xin + +// xn--11b4c3d : 2015-01-15 VeriSign Sarl +कॉम + +// xn--1ck2e1b : 2015-02-26 Amazon Registry Services, Inc. +セール + +// xn--1qqw23a : 2014-01-09 Guangzhou YU Wei Information Technology Co., Ltd. +佛山 + +// xn--30rr7y : 2014-06-12 Excellent First Limited +慈善 + +// xn--3bst00m : 2013-09-13 Eagle Horizon Limited +集团 + +// xn--3ds443g : 2013-09-08 TLD REGISTRY LIMITED OY +在线 + +// xn--3oq18vl8pn36a : 2015-07-02 Volkswagen (China) Investment Co., Ltd. +大众汽车 + +// xn--3pxu8k : 2015-01-15 VeriSign Sarl +点看 + +// xn--42c2d9a : 2015-01-15 VeriSign Sarl +คอม + +// xn--45q11c : 2013-11-21 Zodiac Gemini Ltd +八卦 + +// xn--4gbrim : 2013-10-04 Fans TLD Limited +موقع + +// xn--55qw42g : 2013-11-08 China Organizational Name Administration Center +公益 + +// xn--55qx5d : 2013-11-14 China Internet Network Information Center (CNNIC) +公司 + +// xn--5su34j936bgsg : 2015-09-03 Shangri‐La International Hotel Management Limited +香格里拉 + +// xn--5tzm5g : 2014-12-22 Global Website TLD Asia Limited +网站 + +// xn--6frz82g : 2013-09-23 Afilias Limited +移动 + +// xn--6qq986b3xl : 2013-09-13 Tycoon Treasure Limited +我爱你 + +// xn--80adxhks : 2013-12-19 Foundation for Assistance for Internet Technologies and Infrastructure Development (FAITID) +москва + +// xn--80aqecdr1a : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +католик + +// xn--80asehdb : 2013-07-14 CORE Association +онлайн + +// xn--80aswg : 2013-07-14 CORE Association +сайт + +// xn--8y0a063a : 2015-03-26 China United Network Communications Corporation Limited +联通 + +// xn--9dbq2a : 2015-01-15 VeriSign Sarl +קום + +// xn--9et52u : 2014-06-12 RISE VICTORY LIMITED +时尚 + +// xn--9krt00a : 2015-03-12 Sina Corporation +微博 + +// xn--b4w605ferd : 2014-08-07 Temasek Holdings (Private) Limited +淡马锡 + +// xn--bck1b9a5dre4c : 2015-02-26 Amazon Registry Services, Inc. +ファッション + +// xn--c1avg : 2013-11-14 Public Interest Registry +орг + +// xn--c2br7g : 2015-01-15 VeriSign Sarl +नेट + +// xn--cck2b3b : 2015-02-26 Amazon Registry Services, Inc. +ストア + +// xn--cckwcxetd : 2019-12-19 Amazon Registry Services, Inc. +アマゾン + +// xn--cg4bki : 2013-09-27 SAMSUNG SDS CO., LTD +삼성 + +// xn--czr694b : 2014-01-16 Internet DotTrademark Organisation Limited +商标 + +// xn--czrs0t : 2013-12-19 Binky Moon, LLC +商店 + +// xn--czru2d : 2013-11-21 Zodiac Aquarius Limited +商城 + +// xn--d1acj3b : 2013-11-20 The Foundation for Network Initiatives “The Smart Internet” +дети + +// xn--eckvdtc9d : 2014-12-18 Amazon Registry Services, Inc. +ポイント + +// xn--efvy88h : 2014-08-22 Guangzhou YU Wei Information Technology Co., Ltd. +新闻 + +// xn--fct429k : 2015-04-09 Amazon Registry Services, Inc. +家電 + +// xn--fhbei : 2015-01-15 VeriSign Sarl +كوم + +// xn--fiq228c5hs : 2013-09-08 TLD REGISTRY LIMITED OY +中文网 + +// xn--fiq64b : 2013-10-14 CITIC Group Corporation +中信 + +// xn--fjq720a : 2014-05-22 Binky Moon, LLC +娱乐 + +// xn--flw351e : 2014-07-31 Charleston Road Registry Inc. +谷歌 + +// xn--fzys8d69uvgm : 2015-05-14 PCCW Enterprises Limited +電訊盈科 + +// xn--g2xx48c : 2015-01-30 Nawang Heli(Xiamen) Network Service Co., LTD. +购物 + +// xn--gckr3f0f : 2015-02-26 Amazon Registry Services, Inc. +クラウド + +// xn--gk3at1e : 2015-10-08 Amazon Registry Services, Inc. +通販 + +// xn--hxt814e : 2014-05-15 Zodiac Taurus Limited +网店 + +// xn--i1b6b1a6a2e : 2013-11-14 Public Interest Registry +संगठन + +// xn--imr513n : 2014-12-11 Internet DotTrademark Organisation Limited +餐厅 + +// xn--io0a7i : 2013-11-14 China Internet Network Information Center (CNNIC) +网络 + +// xn--j1aef : 2015-01-15 VeriSign Sarl +ком + +// xn--jlq480n2rg : 2019-12-19 Amazon Registry Services, Inc. +亚马逊 + +// xn--jlq61u9w7b : 2015-01-08 Nokia Corporation +诺基亚 + +// xn--jvr189m : 2015-02-26 Amazon Registry Services, Inc. +食品 + +// xn--kcrx77d1x4a : 2014-11-07 Koninklijke Philips N.V. +飞利浦 + +// xn--kput3i : 2014-02-13 Beijing RITT-Net Technology Development Co., Ltd +手机 + +// xn--mgba3a3ejt : 2014-11-20 Aramco Services Company +ارامكو + +// xn--mgba7c0bbn0a : 2015-05-14 Crescent Holding GmbH +العليان + +// xn--mgbaakc7dvf : 2015-09-03 Emirates Telecommunications Corporation (trading as Etisalat) +اتصالات + +// xn--mgbab2bd : 2013-10-31 CORE Association +بازار + +// xn--mgbca7dzdo : 2015-07-30 Abu Dhabi Systems and Information Centre +ابوظبي + +// xn--mgbi4ecexp : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +كاثوليك + +// xn--mgbt3dhd : 2014-09-04 Asia Green IT System Bilgisayar San. ve Tic. Ltd. Sti. +همراه + +// xn--mk1bu44c : 2015-01-15 VeriSign Sarl +닷컴 + +// xn--mxtq1m : 2014-03-06 Net-Chinese Co., Ltd. +政府 + +// xn--ngbc5azd : 2013-07-13 International Domain Registry Pty. Ltd. +شبكة + +// xn--ngbe9e0a : 2014-12-04 Kuwait Finance House +بيتك + +// xn--ngbrx : 2015-11-12 League of Arab States +عرب + +// xn--nqv7f : 2013-11-14 Public Interest Registry +机构 + +// xn--nqv7fs00ema : 2013-11-14 Public Interest Registry +组织机构 + +// xn--nyqy26a : 2014-11-07 Stable Tone Limited +健康 + +// xn--otu796d : 2017-08-06 Jiang Yu Liang Cai Technology Company Limited +招聘 + +// xn--p1acf : 2013-12-12 Rusnames Limited +рус + +// xn--pssy2u : 2015-01-15 VeriSign Sarl +大拿 + +// xn--q9jyb4c : 2013-09-17 Charleston Road Registry Inc. +みんな + +// xn--qcka1pmc : 2014-07-31 Charleston Road Registry Inc. +グーグル + +// xn--rhqv96g : 2013-09-11 Stable Tone Limited +世界 + +// xn--rovu88b : 2015-02-26 Amazon Registry Services, Inc. +書籍 + +// xn--ses554g : 2014-01-16 KNET Co., Ltd. +网址 + +// xn--t60b56a : 2015-01-15 VeriSign Sarl +닷넷 + +// xn--tckwe : 2015-01-15 VeriSign Sarl +コム + +// xn--tiq49xqyj : 2015-10-21 Pontificium Consilium de Comunicationibus Socialibus (PCCS) (Pontifical Council for Social Communication) +天主教 + +// xn--unup4y : 2013-07-14 Binky Moon, LLC +游戏 + +// xn--vermgensberater-ctb : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +vermögensberater + +// xn--vermgensberatung-pwb : 2014-06-23 Deutsche Vermögensberatung Aktiengesellschaft DVAG +vermögensberatung + +// xn--vhquv : 2013-08-27 Binky Moon, LLC +企业 + +// xn--vuq861b : 2014-10-16 Beijing Tele-info Network Technology Co., Ltd. +信息 + +// xn--w4r85el8fhu5dnra : 2015-04-30 Kerry Trading Co. Limited +嘉里大酒店 + +// xn--w4rs40l : 2015-07-30 Kerry Trading Co. Limited +嘉里 + +// xn--xhq521b : 2013-11-14 Guangzhou YU Wei Information Technology Co., Ltd. +广东 + +// xn--zfr164b : 2013-11-08 China Organizational Name Administration Center +政务 + +// xyz : 2013-12-05 XYZ.COM LLC +xyz + +// yachts : 2014-01-09 XYZ.COM LLC +yachts + +// yahoo : 2015-04-02 Yahoo! Domain Services Inc. +yahoo + +// yamaxun : 2014-12-18 Amazon Registry Services, Inc. +yamaxun + +// yandex : 2014-04-10 Yandex Europe B.V. +yandex + +// yodobashi : 2014-11-20 YODOBASHI CAMERA CO.,LTD. +yodobashi + +// yoga : 2014-05-29 Minds + Machines Group Limited +yoga + +// yokohama : 2013-12-12 GMO Registry, Inc. +yokohama + +// you : 2015-04-09 Amazon Registry Services, Inc. +you + +// youtube : 2014-05-01 Charleston Road Registry Inc. +youtube + +// yun : 2015-01-08 Beijing Qihu Keji Co., Ltd. +yun + +// zappos : 2015-06-25 Amazon Registry Services, Inc. +zappos + +// zara : 2014-11-07 Industria de Diseño Textil, S.A. (INDITEX, S.A.) +zara + +// zero : 2014-12-18 Amazon Registry Services, Inc. +zero + +// zip : 2014-05-08 Charleston Road Registry Inc. +zip + +// zone : 2013-11-14 Binky Moon, LLC +zone + +// zuerich : 2014-11-07 Kanton Zürich (Canton of Zurich) +zuerich + + +// ===END ICANN DOMAINS=== +// ===BEGIN PRIVATE DOMAINS=== +// (Note: these are in alphabetical order by company name) + +// 1GB LLC : https://www.1gb.ua/ +// Submitted by 1GB LLC +cc.ua +inf.ua +ltd.ua + +// 611coin : https://611project.org/ +611.to + +// Adobe : https://www.adobe.com/ +// Submitted by Ian Boston +adobeaemcloud.com +adobeaemcloud.net +*.dev.adobeaemcloud.com + +// Agnat sp. z o.o. : https://domena.pl +// Submitted by Przemyslaw Plewa +beep.pl + +// alboto.ca : http://alboto.ca +// Submitted by Anton Avramov +barsy.ca + +// Alces Software Ltd : http://alces-software.com +// Submitted by Mark J. Titorenko +*.compute.estate +*.alces.network + +// all-inkl.com : https://all-inkl.com +// Submitted by Werner Kaltofen +kasserver.com + +// Altervista: https://www.altervista.org +// Submitted by Carlo Cannas +altervista.org + +// alwaysdata : https://www.alwaysdata.com +// Submitted by Cyril +alwaysdata.net + +// Amazon CloudFront : https://aws.amazon.com/cloudfront/ +// Submitted by Donavan Miller +cloudfront.net + +// Amazon Elastic Compute Cloud : https://aws.amazon.com/ec2/ +// Submitted by Luke Wells +*.compute.amazonaws.com +*.compute-1.amazonaws.com +*.compute.amazonaws.com.cn +us-east-1.amazonaws.com + +// Amazon Elastic Beanstalk : https://aws.amazon.com/elasticbeanstalk/ +// Submitted by Luke Wells +cn-north-1.eb.amazonaws.com.cn +cn-northwest-1.eb.amazonaws.com.cn +elasticbeanstalk.com +ap-northeast-1.elasticbeanstalk.com +ap-northeast-2.elasticbeanstalk.com +ap-northeast-3.elasticbeanstalk.com +ap-south-1.elasticbeanstalk.com +ap-southeast-1.elasticbeanstalk.com +ap-southeast-2.elasticbeanstalk.com +ca-central-1.elasticbeanstalk.com +eu-central-1.elasticbeanstalk.com +eu-west-1.elasticbeanstalk.com +eu-west-2.elasticbeanstalk.com +eu-west-3.elasticbeanstalk.com +sa-east-1.elasticbeanstalk.com +us-east-1.elasticbeanstalk.com +us-east-2.elasticbeanstalk.com +us-gov-west-1.elasticbeanstalk.com +us-west-1.elasticbeanstalk.com +us-west-2.elasticbeanstalk.com + +// Amazon Elastic Load Balancing : https://aws.amazon.com/elasticloadbalancing/ +// Submitted by Luke Wells +*.elb.amazonaws.com +*.elb.amazonaws.com.cn + +// Amazon S3 : https://aws.amazon.com/s3/ +// Submitted by Luke Wells +s3.amazonaws.com +s3-ap-northeast-1.amazonaws.com +s3-ap-northeast-2.amazonaws.com +s3-ap-south-1.amazonaws.com +s3-ap-southeast-1.amazonaws.com +s3-ap-southeast-2.amazonaws.com +s3-ca-central-1.amazonaws.com +s3-eu-central-1.amazonaws.com +s3-eu-west-1.amazonaws.com +s3-eu-west-2.amazonaws.com +s3-eu-west-3.amazonaws.com +s3-external-1.amazonaws.com +s3-fips-us-gov-west-1.amazonaws.com +s3-sa-east-1.amazonaws.com +s3-us-gov-west-1.amazonaws.com +s3-us-east-2.amazonaws.com +s3-us-west-1.amazonaws.com +s3-us-west-2.amazonaws.com +s3.ap-northeast-2.amazonaws.com +s3.ap-south-1.amazonaws.com +s3.cn-north-1.amazonaws.com.cn +s3.ca-central-1.amazonaws.com +s3.eu-central-1.amazonaws.com +s3.eu-west-2.amazonaws.com +s3.eu-west-3.amazonaws.com +s3.us-east-2.amazonaws.com +s3.dualstack.ap-northeast-1.amazonaws.com +s3.dualstack.ap-northeast-2.amazonaws.com +s3.dualstack.ap-south-1.amazonaws.com +s3.dualstack.ap-southeast-1.amazonaws.com +s3.dualstack.ap-southeast-2.amazonaws.com +s3.dualstack.ca-central-1.amazonaws.com +s3.dualstack.eu-central-1.amazonaws.com +s3.dualstack.eu-west-1.amazonaws.com +s3.dualstack.eu-west-2.amazonaws.com +s3.dualstack.eu-west-3.amazonaws.com +s3.dualstack.sa-east-1.amazonaws.com +s3.dualstack.us-east-1.amazonaws.com +s3.dualstack.us-east-2.amazonaws.com +s3-website-us-east-1.amazonaws.com +s3-website-us-west-1.amazonaws.com +s3-website-us-west-2.amazonaws.com +s3-website-ap-northeast-1.amazonaws.com +s3-website-ap-southeast-1.amazonaws.com +s3-website-ap-southeast-2.amazonaws.com +s3-website-eu-west-1.amazonaws.com +s3-website-sa-east-1.amazonaws.com +s3-website.ap-northeast-2.amazonaws.com +s3-website.ap-south-1.amazonaws.com +s3-website.ca-central-1.amazonaws.com +s3-website.eu-central-1.amazonaws.com +s3-website.eu-west-2.amazonaws.com +s3-website.eu-west-3.amazonaws.com +s3-website.us-east-2.amazonaws.com + +// Amsterdam Wireless: https://www.amsterdamwireless.nl/ +// Submitted by Imre Jonk +amsw.nl + +// Amune : https://amune.org/ +// Submitted by Team Amune +t3l3p0rt.net +tele.amune.org + +// Apigee : https://apigee.com/ +// Submitted by Apigee Security Team +apigee.io + +// Aptible : https://www.aptible.com/ +// Submitted by Thomas Orozco +on-aptible.com + +// ASEINet : https://www.aseinet.com/ +// Submitted by Asei SEKIGUCHI +user.aseinet.ne.jp +gv.vc +d.gv.vc + +// Asociación Amigos de la Informática "Euskalamiga" : http://encounter.eus/ +// Submitted by Hector Martin +user.party.eus + +// Association potager.org : https://potager.org/ +// Submitted by Lunar +pimienta.org +poivron.org +potager.org +sweetpepper.org + +// ASUSTOR Inc. : http://www.asustor.com +// Submitted by Vincent Tseng +myasustor.com + +// AVM : https://avm.de +// Submitted by Andreas Weise +myfritz.net + +// AW AdvisorWebsites.com Software Inc : https://advisorwebsites.com +// Submitted by James Kennedy +*.awdev.ca +*.advisor.ws + +// b-data GmbH : https://www.b-data.io +// Submitted by Olivier Benz +b-data.io + +// backplane : https://www.backplane.io +// Submitted by Anthony Voutas +backplaneapp.io + +// Balena : https://www.balena.io +// Submitted by Petros Angelatos +balena-devices.com + +// Banzai Cloud +// Submitted by Janos Matyas +*.banzai.cloud +app.banzaicloud.io +*.backyards.banzaicloud.io + + +// BetaInABox +// Submitted by Adrian +betainabox.com + +// BinaryLane : http://www.binarylane.com +// Submitted by Nathan O'Sullivan +bnr.la + +// Blackbaud, Inc. : https://www.blackbaud.com +// Submitted by Paul Crowder +blackbaudcdn.net + +// Blatech : http://www.blatech.net +// Submitted by Luke Bratch +of.je + +// Boomla : https://boomla.com +// Submitted by Tibor Halter +boomla.net + +// Boxfuse : https://boxfuse.com +// Submitted by Axel Fontaine +boxfuse.io + +// bplaced : https://www.bplaced.net/ +// Submitted by Miroslav Bozic +square7.ch +bplaced.com +bplaced.de +square7.de +bplaced.net +square7.net + +// BrowserSafetyMark +// Submitted by Dave Tharp +browsersafetymark.io + +// Bytemark Hosting : https://www.bytemark.co.uk +// Submitted by Paul Cammish +uk0.bigv.io +dh.bytemark.co.uk +vm.bytemark.co.uk + +// callidomus : https://www.callidomus.com/ +// Submitted by Marcus Popp +mycd.eu + +// Carrd : https://carrd.co +// Submitted by AJ +carrd.co +crd.co +uwu.ai + +// CentralNic : http://www.centralnic.com/names/domains +// Submitted by registry +ae.org +br.com +cn.com +com.de +com.se +de.com +eu.com +gb.net +hu.net +jp.net +jpn.com +mex.com +ru.com +sa.com +se.net +uk.com +uk.net +us.com +za.bz +za.com + +// No longer operated by CentralNic, these entries should be adopted and/or removed by current operators +// Submitted by Gavin Brown +ar.com +gb.com +hu.com +kr.com +no.com +qc.com +uy.com + +// Africa.com Web Solutions Ltd : https://registry.africa.com +// Submitted by Gavin Brown +africa.com + +// iDOT Services Limited : http://www.domain.gr.com +// Submitted by Gavin Brown +gr.com + +// Radix FZC : http://domains.in.net +// Submitted by Gavin Brown +in.net +web.in + +// US REGISTRY LLC : http://us.org +// Submitted by Gavin Brown +us.org + +// co.com Registry, LLC : https://registry.co.com +// Submitted by Gavin Brown +co.com + +// Roar Domains LLC : https://roar.basketball/ +// Submitted by Gavin Brown +aus.basketball +nz.basketball + +// BRS Media : https://brsmedia.com/ +// Submitted by Gavin Brown +radio.am +radio.fm + +// Globe Hosting SRL : https://www.globehosting.com/ +// Submitted by Gavin Brown +co.ro +shop.ro + +// c.la : http://www.c.la/ +c.la + +// certmgr.org : https://certmgr.org +// Submitted by B. Blechschmidt +certmgr.org + +// Civilized Discourse Construction Kit, Inc. : https://www.discourse.org/ +// Submitted by Rishabh Nambiar & Michael Brown +discourse.group +discourse.team + +// ClearVox : http://www.clearvox.nl/ +// Submitted by Leon Rowland +virtueeldomein.nl + +// Clever Cloud : https://www.clever-cloud.com/ +// Submitted by Quentin Adam +cleverapps.io + +// Clerk : https://www.clerk.dev +// Submitted by Colin Sidoti +*.lcl.dev +*.stg.dev + +// Clic2000 : https://clic2000.fr +// Submitted by Mathilde Blanchemanche +clic2000.net + +// Cloud66 : https://www.cloud66.com/ +// Submitted by Khash Sajadi +c66.me +cloud66.ws +cloud66.zone + +// CloudAccess.net : https://www.cloudaccess.net/ +// Submitted by Pawel Panek +jdevcloud.com +wpdevcloud.com +cloudaccess.host +freesite.host +cloudaccess.net + +// cloudControl : https://www.cloudcontrol.com/ +// Submitted by Tobias Wilken +cloudcontrolled.com +cloudcontrolapp.com + +// Cloudera, Inc. : https://www.cloudera.com/ +// Submitted by Philip Langdale +cloudera.site + +// Cloudflare, Inc. : https://www.cloudflare.com/ +// Submitted by Cloudflare Team +pages.dev +trycloudflare.com +workers.dev + +// Clovyr : https://clovyr.io +// Submitted by Patrick Nielsen +wnext.app + +// co.ca : http://registry.co.ca/ +co.ca + +// Co & Co : https://co-co.nl/ +// Submitted by Govert Versluis +*.otap.co + +// i-registry s.r.o. : http://www.i-registry.cz/ +// Submitted by Martin Semrad +co.cz + +// CDN77.com : http://www.cdn77.com +// Submitted by Jan Krpes +c.cdn77.org +cdn77-ssl.net +r.cdn77.net +rsc.cdn77.org +ssl.origin.cdn77-secure.org + +// Cloud DNS Ltd : http://www.cloudns.net +// Submitted by Aleksander Hristov +cloudns.asia +cloudns.biz +cloudns.club +cloudns.cc +cloudns.eu +cloudns.in +cloudns.info +cloudns.org +cloudns.pro +cloudns.pw +cloudns.us + +// CNPY : https://cnpy.gdn +// Submitted by Angelo Gladding +cnpy.gdn + +// CoDNS B.V. +co.nl +co.no + +// Combell.com : https://www.combell.com +// Submitted by Thomas Wouters +webhosting.be +hosting-cluster.nl + +// Coordination Center for TLD RU and XN--P1AI : https://cctld.ru/en/domains/domens_ru/reserved/ +// Submitted by George Georgievsky +ac.ru +edu.ru +gov.ru +int.ru +mil.ru +test.ru + +// COSIMO GmbH : http://www.cosimo.de +// Submitted by Rene Marticke +dyn.cosidns.de +dynamisches-dns.de +dnsupdater.de +internet-dns.de +l-o-g-i-n.de +dynamic-dns.info +feste-ip.net +knx-server.net +static-access.net + +// Craynic, s.r.o. : http://www.craynic.com/ +// Submitted by Ales Krajnik +realm.cz + +// Cryptonomic : https://cryptonomic.net/ +// Submitted by Andrew Cady +*.cryptonomic.net + +// Cupcake : https://cupcake.io/ +// Submitted by Jonathan Rudenberg +cupcake.is + +// Curv UG : https://curv-labs.de/ +// Submitted by Marvin Wiesner +curv.dev + +// Customer OCI - Oracle Dyn https://cloud.oracle.com/home https://dyn.com/dns/ +// Submitted by Gregory Drake +// Note: This is intended to also include customer-oci.com due to wildcards implicitly including the current label +*.customer-oci.com +*.oci.customer-oci.com +*.ocp.customer-oci.com +*.ocs.customer-oci.com + +// cyon GmbH : https://www.cyon.ch/ +// Submitted by Dominic Luechinger +cyon.link +cyon.site + +// Danger Science Group: https://dangerscience.com/ +// Submitted by Skylar MacDonald +fnwk.site +folionetwork.site +platform0.app + +// Daplie, Inc : https://daplie.com +// Submitted by AJ ONeal +daplie.me +localhost.daplie.me + +// Datto, Inc. : https://www.datto.com/ +// Submitted by Philipp Heckel +dattolocal.com +dattorelay.com +dattoweb.com +mydatto.com +dattolocal.net +mydatto.net + +// Dansk.net : http://www.dansk.net/ +// Submitted by Anani Voule +biz.dk +co.dk +firm.dk +reg.dk +store.dk + +// dappnode.io : https://dappnode.io/ +// Submitted by Abel Boldu / DAppNode Team +dyndns.dappnode.io + +// dapps.earth : https://dapps.earth/ +// Submitted by Daniil Burdakov +*.dapps.earth +*.bzz.dapps.earth + +// Dark, Inc. : https://darklang.com +// Submitted by Paul Biggar +builtwithdark.com + +// Datawire, Inc : https://www.datawire.io +// Submitted by Richard Li +edgestack.me + +// Debian : https://www.debian.org/ +// Submitted by Peter Palfrader / Debian Sysadmin Team +debian.net + +// deSEC : https://desec.io/ +// Submitted by Peter Thomassen +dedyn.io + +// DNS Africa Ltd https://dns.business +// Submitted by Calvin Browne +jozi.biz + +// DNShome : https://www.dnshome.de/ +// Submitted by Norbert Auler +dnshome.de + +// DotArai : https://www.dotarai.com/ +// Submitted by Atsadawat Netcharadsang +online.th +shop.th + +// DrayTek Corp. : https://www.draytek.com/ +// Submitted by Paul Fang +drayddns.com + +// DreamHost : http://www.dreamhost.com/ +// Submitted by Andrew Farmer +dreamhosters.com + +// Drobo : http://www.drobo.com/ +// Submitted by Ricardo Padilha +mydrobo.com + +// Drud Holdings, LLC. : https://www.drud.com/ +// Submitted by Kevin Bridges +drud.io +drud.us + +// DuckDNS : http://www.duckdns.org/ +// Submitted by Richard Harper +duckdns.org + +// Bip : https://bip.sh +// Submitted by Joel Kennedy +bip.sh + +// bitbridge.net : Submitted by Craig Welch, abeliidev@gmail.com +bitbridge.net + +// dy.fi : http://dy.fi/ +// Submitted by Heikki Hannikainen +dy.fi +tunk.org + +// DynDNS.com : http://www.dyndns.com/services/dns/dyndns/ +dyndns-at-home.com +dyndns-at-work.com +dyndns-blog.com +dyndns-free.com +dyndns-home.com +dyndns-ip.com +dyndns-mail.com +dyndns-office.com +dyndns-pics.com +dyndns-remote.com +dyndns-server.com +dyndns-web.com +dyndns-wiki.com +dyndns-work.com +dyndns.biz +dyndns.info +dyndns.org +dyndns.tv +at-band-camp.net +ath.cx +barrel-of-knowledge.info +barrell-of-knowledge.info +better-than.tv +blogdns.com +blogdns.net +blogdns.org +blogsite.org +boldlygoingnowhere.org +broke-it.net +buyshouses.net +cechire.com +dnsalias.com +dnsalias.net +dnsalias.org +dnsdojo.com +dnsdojo.net +dnsdojo.org +does-it.net +doesntexist.com +doesntexist.org +dontexist.com +dontexist.net +dontexist.org +doomdns.com +doomdns.org +dvrdns.org +dyn-o-saur.com +dynalias.com +dynalias.net +dynalias.org +dynathome.net +dyndns.ws +endofinternet.net +endofinternet.org +endoftheinternet.org +est-a-la-maison.com +est-a-la-masion.com +est-le-patron.com +est-mon-blogueur.com +for-better.biz +for-more.biz +for-our.info +for-some.biz +for-the.biz +forgot.her.name +forgot.his.name +from-ak.com +from-al.com +from-ar.com +from-az.net +from-ca.com +from-co.net +from-ct.com +from-dc.com +from-de.com +from-fl.com +from-ga.com +from-hi.com +from-ia.com +from-id.com +from-il.com +from-in.com +from-ks.com +from-ky.com +from-la.net +from-ma.com +from-md.com +from-me.org +from-mi.com +from-mn.com +from-mo.com +from-ms.com +from-mt.com +from-nc.com +from-nd.com +from-ne.com +from-nh.com +from-nj.com +from-nm.com +from-nv.com +from-ny.net +from-oh.com +from-ok.com +from-or.com +from-pa.com +from-pr.com +from-ri.com +from-sc.com +from-sd.com +from-tn.com +from-tx.com +from-ut.com +from-va.com +from-vt.com +from-wa.com +from-wi.com +from-wv.com +from-wy.com +ftpaccess.cc +fuettertdasnetz.de +game-host.org +game-server.cc +getmyip.com +gets-it.net +go.dyndns.org +gotdns.com +gotdns.org +groks-the.info +groks-this.info +ham-radio-op.net +here-for-more.info +hobby-site.com +hobby-site.org +home.dyndns.org +homedns.org +homeftp.net +homeftp.org +homeip.net +homelinux.com +homelinux.net +homelinux.org +homeunix.com +homeunix.net +homeunix.org +iamallama.com +in-the-band.net +is-a-anarchist.com +is-a-blogger.com +is-a-bookkeeper.com +is-a-bruinsfan.org +is-a-bulls-fan.com +is-a-candidate.org +is-a-caterer.com +is-a-celticsfan.org +is-a-chef.com +is-a-chef.net +is-a-chef.org +is-a-conservative.com +is-a-cpa.com +is-a-cubicle-slave.com +is-a-democrat.com +is-a-designer.com +is-a-doctor.com +is-a-financialadvisor.com +is-a-geek.com +is-a-geek.net +is-a-geek.org +is-a-green.com +is-a-guru.com +is-a-hard-worker.com +is-a-hunter.com +is-a-knight.org +is-a-landscaper.com +is-a-lawyer.com +is-a-liberal.com +is-a-libertarian.com +is-a-linux-user.org +is-a-llama.com +is-a-musician.com +is-a-nascarfan.com +is-a-nurse.com +is-a-painter.com +is-a-patsfan.org +is-a-personaltrainer.com +is-a-photographer.com +is-a-player.com +is-a-republican.com +is-a-rockstar.com +is-a-socialist.com +is-a-soxfan.org +is-a-student.com +is-a-teacher.com +is-a-techie.com +is-a-therapist.com +is-an-accountant.com +is-an-actor.com +is-an-actress.com +is-an-anarchist.com +is-an-artist.com +is-an-engineer.com +is-an-entertainer.com +is-by.us +is-certified.com +is-found.org +is-gone.com +is-into-anime.com +is-into-cars.com +is-into-cartoons.com +is-into-games.com +is-leet.com +is-lost.org +is-not-certified.com +is-saved.org +is-slick.com +is-uberleet.com +is-very-bad.org +is-very-evil.org +is-very-good.org +is-very-nice.org +is-very-sweet.org +is-with-theband.com +isa-geek.com +isa-geek.net +isa-geek.org +isa-hockeynut.com +issmarterthanyou.com +isteingeek.de +istmein.de +kicks-ass.net +kicks-ass.org +knowsitall.info +land-4-sale.us +lebtimnetz.de +leitungsen.de +likes-pie.com +likescandy.com +merseine.nu +mine.nu +misconfused.org +mypets.ws +myphotos.cc +neat-url.com +office-on-the.net +on-the-web.tv +podzone.net +podzone.org +readmyblog.org +saves-the-whales.com +scrapper-site.net +scrapping.cc +selfip.biz +selfip.com +selfip.info +selfip.net +selfip.org +sells-for-less.com +sells-for-u.com +sells-it.net +sellsyourhome.org +servebbs.com +servebbs.net +servebbs.org +serveftp.net +serveftp.org +servegame.org +shacknet.nu +simple-url.com +space-to-rent.com +stuff-4-sale.org +stuff-4-sale.us +teaches-yoga.com +thruhere.net +traeumtgerade.de +webhop.biz +webhop.info +webhop.net +webhop.org +worse-than.tv +writesthisblog.com + +// ddnss.de : https://www.ddnss.de/ +// Submitted by Robert Niedziela +ddnss.de +dyn.ddnss.de +dyndns.ddnss.de +dyndns1.de +dyn-ip24.de +home-webserver.de +dyn.home-webserver.de +myhome-server.de +ddnss.org + +// Definima : http://www.definima.com/ +// Submitted by Maxence Bitterli +definima.net +definima.io + +// DigitalOcean : https://digitalocean.com/ +// Submitted by Braxton Huggins +ondigitalocean.app + +// dnstrace.pro : https://dnstrace.pro/ +// Submitted by Chris Partridge +bci.dnstrace.pro + +// Dynu.com : https://www.dynu.com/ +// Submitted by Sue Ye +ddnsfree.com +ddnsgeek.com +giize.com +gleeze.com +kozow.com +loseyourip.com +ooguy.com +theworkpc.com +casacam.net +dynu.net +accesscam.org +camdvr.org +freeddns.org +mywire.org +webredirect.org +myddns.rocks +blogsite.xyz + +// dynv6 : https://dynv6.com +// Submitted by Dominik Menke +dynv6.net + +// E4YOU spol. s.r.o. : https://e4you.cz/ +// Submitted by Vladimir Dudr +e4.cz + +// En root‽ : https://en-root.org +// Submitted by Emmanuel Raviart +en-root.fr + +// Enalean SAS: https://www.enalean.com +// Submitted by Thomas Cottier +mytuleap.com + +// ECG Robotics, Inc: https://ecgrobotics.org +// Submitted by +onred.one +staging.onred.one + +// One.com: https://www.one.com/ +// Submitted by Jacob Bunk Nielsen +service.one + +// Enonic : http://enonic.com/ +// Submitted by Erik Kaareng-Sunde +enonic.io +customer.enonic.io + +// EU.org https://eu.org/ +// Submitted by Pierre Beyssac +eu.org +al.eu.org +asso.eu.org +at.eu.org +au.eu.org +be.eu.org +bg.eu.org +ca.eu.org +cd.eu.org +ch.eu.org +cn.eu.org +cy.eu.org +cz.eu.org +de.eu.org +dk.eu.org +edu.eu.org +ee.eu.org +es.eu.org +fi.eu.org +fr.eu.org +gr.eu.org +hr.eu.org +hu.eu.org +ie.eu.org +il.eu.org +in.eu.org +int.eu.org +is.eu.org +it.eu.org +jp.eu.org +kr.eu.org +lt.eu.org +lu.eu.org +lv.eu.org +mc.eu.org +me.eu.org +mk.eu.org +mt.eu.org +my.eu.org +net.eu.org +ng.eu.org +nl.eu.org +no.eu.org +nz.eu.org +paris.eu.org +pl.eu.org +pt.eu.org +q-a.eu.org +ro.eu.org +ru.eu.org +se.eu.org +si.eu.org +sk.eu.org +tr.eu.org +uk.eu.org +us.eu.org + +// Evennode : http://www.evennode.com/ +// Submitted by Michal Kralik +eu-1.evennode.com +eu-2.evennode.com +eu-3.evennode.com +eu-4.evennode.com +us-1.evennode.com +us-2.evennode.com +us-3.evennode.com +us-4.evennode.com + +// eDirect Corp. : https://hosting.url.com.tw/ +// Submitted by C.S. chang +twmail.cc +twmail.net +twmail.org +mymailer.com.tw +url.tw + +// Fabrica Technologies, Inc. : https://www.fabrica.dev/ +// Submitted by Eric Jiang +onfabrica.com + +// Facebook, Inc. +// Submitted by Peter Ruibal +apps.fbsbx.com + +// FAITID : https://faitid.org/ +// Submitted by Maxim Alzoba +// https://www.flexireg.net/stat_info +ru.net +adygeya.ru +bashkiria.ru +bir.ru +cbg.ru +com.ru +dagestan.ru +grozny.ru +kalmykia.ru +kustanai.ru +marine.ru +mordovia.ru +msk.ru +mytis.ru +nalchik.ru +nov.ru +pyatigorsk.ru +spb.ru +vladikavkaz.ru +vladimir.ru +abkhazia.su +adygeya.su +aktyubinsk.su +arkhangelsk.su +armenia.su +ashgabad.su +azerbaijan.su +balashov.su +bashkiria.su +bryansk.su +bukhara.su +chimkent.su +dagestan.su +east-kazakhstan.su +exnet.su +georgia.su +grozny.su +ivanovo.su +jambyl.su +kalmykia.su +kaluga.su +karacol.su +karaganda.su +karelia.su +khakassia.su +krasnodar.su +kurgan.su +kustanai.su +lenug.su +mangyshlak.su +mordovia.su +msk.su +murmansk.su +nalchik.su +navoi.su +north-kazakhstan.su +nov.su +obninsk.su +penza.su +pokrovsk.su +sochi.su +spb.su +tashkent.su +termez.su +togliatti.su +troitsk.su +tselinograd.su +tula.su +tuva.su +vladikavkaz.su +vladimir.su +vologda.su + +// Fancy Bits, LLC : http://getchannels.com +// Submitted by Aman Gupta +channelsdvr.net +u.channelsdvr.net + +// Fastly Inc. : http://www.fastly.com/ +// Submitted by Fastly Security +fastly-terrarium.com +fastlylb.net +map.fastlylb.net +freetls.fastly.net +map.fastly.net +a.prod.fastly.net +global.prod.fastly.net +a.ssl.fastly.net +b.ssl.fastly.net +global.ssl.fastly.net + +// FASTVPS EESTI OU : https://fastvps.ru/ +// Submitted by Likhachev Vasiliy +fastvps-server.com +fastvps.host +myfast.host +fastvps.site +myfast.space + +// Featherhead : https://featherhead.xyz/ +// Submitted by Simon Menke +fhapp.xyz + +// Fedora : https://fedoraproject.org/ +// submitted by Patrick Uiterwijk +fedorainfracloud.org +fedorapeople.org +cloud.fedoraproject.org +app.os.fedoraproject.org +app.os.stg.fedoraproject.org + +// FearWorks Media Ltd. : https://fearworksmedia.co.uk +// submitted by Keith Fairley +conn.uk +copro.uk +couk.me +ukco.me + +// Fermax : https://fermax.com/ +// submitted by Koen Van Isterdael +mydobiss.com + +// FH Muenster : https://www.fh-muenster.de +// Submitted by Robin Naundorf +fh-muenster.io + +// Filegear Inc. : https://www.filegear.com +// Submitted by Jason Zhu +filegear.me +filegear-au.me +filegear-de.me +filegear-gb.me +filegear-ie.me +filegear-jp.me +filegear-sg.me + +// Firebase, Inc. +// Submitted by Chris Raynor +firebaseapp.com + +// fly.io: https://fly.io +// Submitted by Kurt Mackey +fly.dev +edgeapp.net +shw.io + +// Flynn : https://flynn.io +// Submitted by Jonathan Rudenberg +flynnhosting.net + +// Frederik Braun https://frederik-braun.com +// Submitted by Frederik Braun +0e.vc + +// Freebox : http://www.freebox.fr +// Submitted by Romain Fliedel +freebox-os.com +freeboxos.com +fbx-os.fr +fbxos.fr +freebox-os.fr +freeboxos.fr + +// freedesktop.org : https://www.freedesktop.org +// Submitted by Daniel Stone +freedesktop.org + +// FunkFeuer - Verein zur Förderung freier Netze : https://www.funkfeuer.at +// Submitted by Daniel A. Maierhofer +wien.funkfeuer.at + +// Futureweb OG : http://www.futureweb.at +// Submitted by Andreas Schnederle-Wagner +*.futurecms.at +*.ex.futurecms.at +*.in.futurecms.at +futurehosting.at +futuremailing.at +*.ex.ortsinfo.at +*.kunden.ortsinfo.at +*.statics.cloud + +// GDS : https://www.gov.uk/service-manual/operations/operating-servicegovuk-subdomains +// Submitted by David Illsley +service.gov.uk + +// Gehirn Inc. : https://www.gehirn.co.jp/ +// Submitted by Kohei YOSHIDA +gehirn.ne.jp +usercontent.jp + +// Gentlent, Inc. : https://www.gentlent.com +// Submitted by Tom Klein +gentapps.com +gentlentapis.com +lab.ms +cdn-edges.net + +// GitHub, Inc. +// Submitted by Patrick Toomey +github.io +githubusercontent.com + +// GitLab, Inc. +// Submitted by Alex Hanselka +gitlab.io + +// Gitplac.si - https://gitplac.si +// Submitted by Aljaž Starc +gitapp.si +gitpage.si + +// Glitch, Inc : https://glitch.com +// Submitted by Mads Hartmann +glitch.me + +// GMO Pepabo, Inc. : https://pepabo.com/ +// Submitted by dojineko +lolipop.io + +// GOV.UK Platform as a Service : https://www.cloud.service.gov.uk/ +// Submitted by Tom Whitwell +cloudapps.digital +london.cloudapps.digital + +// GOV.UK Pay : https://www.payments.service.gov.uk/ +// Submitted by Richard Baker +pymnt.uk + +// UKHomeOffice : https://www.gov.uk/government/organisations/home-office +// Submitted by Jon Shanks +homeoffice.gov.uk + +// GlobeHosting, Inc. +// Submitted by Zoltan Egresi +ro.im + +// GoIP DNS Services : http://www.goip.de +// Submitted by Christian Poulter +goip.de + +// Google, Inc. +// Submitted by Eduardo Vela +run.app +a.run.app +web.app +*.0emm.com +appspot.com +*.r.appspot.com +codespot.com +googleapis.com +googlecode.com +pagespeedmobilizer.com +publishproxy.com +withgoogle.com +withyoutube.com +*.gateway.dev +cloud.goog +translate.goog +cloudfunctions.net + +blogspot.ae +blogspot.al +blogspot.am +blogspot.ba +blogspot.be +blogspot.bg +blogspot.bj +blogspot.ca +blogspot.cf +blogspot.ch +blogspot.cl +blogspot.co.at +blogspot.co.id +blogspot.co.il +blogspot.co.ke +blogspot.co.nz +blogspot.co.uk +blogspot.co.za +blogspot.com +blogspot.com.ar +blogspot.com.au +blogspot.com.br +blogspot.com.by +blogspot.com.co +blogspot.com.cy +blogspot.com.ee +blogspot.com.eg +blogspot.com.es +blogspot.com.mt +blogspot.com.ng +blogspot.com.tr +blogspot.com.uy +blogspot.cv +blogspot.cz +blogspot.de +blogspot.dk +blogspot.fi +blogspot.fr +blogspot.gr +blogspot.hk +blogspot.hr +blogspot.hu +blogspot.ie +blogspot.in +blogspot.is +blogspot.it +blogspot.jp +blogspot.kr +blogspot.li +blogspot.lt +blogspot.lu +blogspot.md +blogspot.mk +blogspot.mr +blogspot.mx +blogspot.my +blogspot.nl +blogspot.no +blogspot.pe +blogspot.pt +blogspot.qa +blogspot.re +blogspot.ro +blogspot.rs +blogspot.ru +blogspot.se +blogspot.sg +blogspot.si +blogspot.sk +blogspot.sn +blogspot.td +blogspot.tw +blogspot.ug +blogspot.vn + +// Aaron Marais' Gitlab pages: https://lab.aaronleem.co.za +// Submitted by Aaron Marais +graphox.us + +// Group 53, LLC : https://www.group53.com +// Submitted by Tyler Todd +awsmppl.com + +// Hakaran group: http://hakaran.cz +// Submited by Arseniy Sokolov +fin.ci +free.hr +caa.li +ua.rs +conf.se + +// Handshake : https://handshake.org +// Submitted by Mike Damm +hs.zone +hs.run + +// Hashbang : https://hashbang.sh +hashbang.sh + +// Hasura : https://hasura.io +// Submitted by Shahidh K Muhammed +hasura.app +hasura-app.io + +// Hepforge : https://www.hepforge.org +// Submitted by David Grellscheid +hepforge.org + +// Heroku : https://www.heroku.com/ +// Submitted by Tom Maher +herokuapp.com +herokussl.com + +// Hibernating Rhinos +// Submitted by Oren Eini +myravendb.com +ravendb.community +ravendb.me +development.run +ravendb.run + +// Hong Kong Productivity Council: https://www.hkpc.org/ +// Submitted by SECaaS Team +secaas.hk + +// HOSTBIP REGISTRY : https://www.hostbip.com/ +// Submitted by Atanunu Igbunuroghene +bpl.biz +orx.biz +ng.city +biz.gl +ng.ink +col.ng +firm.ng +gen.ng +ltd.ng +ngo.ng +ng.school +sch.so + +// HostyHosting (hostyhosting.com) +hostyhosting.io + +// Häkkinen.fi +// Submitted by Eero Häkkinen +häkkinen.fi + +// Ici la Lune : http://www.icilalune.com/ +// Submitted by Simon Morvan +*.moonscale.io +moonscale.net + +// iki.fi +// Submitted by Hannu Aronsson +iki.fi + +// Individual Network Berlin e.V. : https://www.in-berlin.de/ +// Submitted by Christian Seitz +dyn-berlin.de +in-berlin.de +in-brb.de +in-butter.de +in-dsl.de +in-dsl.net +in-dsl.org +in-vpn.de +in-vpn.net +in-vpn.org + +// info.at : http://www.info.at/ +biz.at +info.at + +// info.cx : http://info.cx +// Submitted by Jacob Slater +info.cx + +// Interlegis : http://www.interlegis.leg.br +// Submitted by Gabriel Ferreira +ac.leg.br +al.leg.br +am.leg.br +ap.leg.br +ba.leg.br +ce.leg.br +df.leg.br +es.leg.br +go.leg.br +ma.leg.br +mg.leg.br +ms.leg.br +mt.leg.br +pa.leg.br +pb.leg.br +pe.leg.br +pi.leg.br +pr.leg.br +rj.leg.br +rn.leg.br +ro.leg.br +rr.leg.br +rs.leg.br +sc.leg.br +se.leg.br +sp.leg.br +to.leg.br + +// intermetrics GmbH : https://pixolino.com/ +// Submitted by Wolfgang Schwarz +pixolino.com + +// Internet-Pro, LLP: https://netangels.ru/ +// Submited by Vasiliy Sheredeko +na4u.ru + +// iopsys software solutions AB : https://iopsys.eu/ +// Submitted by Roman Azarenko +iopsys.se + +// IPiFony Systems, Inc. : https://www.ipifony.com/ +// Submitted by Matthew Hardeman +ipifony.net + +// IServ GmbH : https://iserv.eu +// Submitted by Kim-Alexander Brodowski +mein-iserv.de +schulserver.de +test-iserv.de +iserv.dev + +// I-O DATA DEVICE, INC. : http://www.iodata.com/ +// Submitted by Yuji Minagawa +iobb.net + +// Jelastic, Inc. : https://jelastic.com/ +// Submited by Ihor Kolodyuk +mel.cloudlets.com.au +cloud.interhostsolutions.be +users.scale.virtualcloud.com.br +mycloud.by +alp1.ae.flow.ch +appengine.flow.ch +es-1.axarnet.cloud +diadem.cloud +vip.jelastic.cloud +jele.cloud +it1.eur.aruba.jenv-aruba.cloud +it1.jenv-aruba.cloud +it1-eur.jenv-arubabiz.cloud +oxa.cloud +tn.oxa.cloud +uk.oxa.cloud +primetel.cloud +uk.primetel.cloud +ca.reclaim.cloud +uk.reclaim.cloud +us.reclaim.cloud +ch.trendhosting.cloud +de.trendhosting.cloud +jele.club +clicketcloud.com +ams.cloudswitches.com +au.cloudswitches.com +sg.cloudswitches.com +dopaas.com +elastyco.com +nv.elastyco.com +hidora.com +paas.hosted-by-previder.com +rag-cloud.hosteur.com +rag-cloud-ch.hosteur.com +jcloud.ik-server.com +jcloud-ver-jpc.ik-server.com +demo.jelastic.com +kilatiron.com +paas.massivegrid.com +jed.wafaicloud.com +lon.wafaicloud.com +ryd.wafaicloud.com +j.scaleforce.com.cy +jelastic.dogado.eu +paas.leviracloud.eu +fi.cloudplatform.fi +demo.datacenter.fi +paas.datacenter.fi +jele.host +mircloud.host +jele.io +ocs.opusinteractive.io +cloud.unispace.io +cloud-de.unispace.io +cloud-fr1.unispace.io +jc.neen.it +cloud.jelastic.open.tim.it +jcloud.kz +upaas.kazteleport.kz +jl.serv.net.mx +cloudjiffy.net +fra1-de.cloudjiffy.net +west1-us.cloudjiffy.net +ams1.jls.docktera.net +jls-sto1.elastx.net +jls-sto2.elastx.net +jls-sto3.elastx.net +fr-1.paas.massivegrid.net +lon-1.paas.massivegrid.net +lon-2.paas.massivegrid.net +ny-1.paas.massivegrid.net +ny-2.paas.massivegrid.net +sg-1.paas.massivegrid.net +jelastic.saveincloud.net +nordeste-idc.saveincloud.net +j.scaleforce.net +jelastic.tsukaeru.net +atl.jelastic.vps-host.net +njs.jelastic.vps-host.net +unicloud.pl +mircloud.ru +jelastic.regruhosting.ru +enscaled.sg +jele.site +jelastic.team +orangecloud.tn +j.layershift.co.uk +phx.enscaled.us +mircloud.us + +// Jino : https://www.jino.ru +// Submitted by Sergey Ulyashin +myjino.ru +*.hosting.myjino.ru +*.landing.myjino.ru +*.spectrum.myjino.ru +*.vps.myjino.ru + +// Joyent : https://www.joyent.com/ +// Submitted by Brian Bennett +*.triton.zone +*.cns.joyent.com + +// JS.ORG : http://dns.js.org +// Submitted by Stefan Keim +js.org + +// KaasHosting : http://www.kaashosting.nl/ +// Submitted by Wouter Bakker +kaas.gg +khplay.nl + +// Keyweb AG : https://www.keyweb.de +// Submitted by Martin Dannehl +keymachine.de + +// KingHost : https://king.host +// Submitted by Felipe Keller Braz +kinghost.net +uni5.net + +// KnightPoint Systems, LLC : http://www.knightpoint.com/ +// Submitted by Roy Keene +knightpoint.systems + +// KUROKU LTD : https://kuroku.ltd/ +// Submitted by DisposaBoy +oya.to + +// .KRD : http://nic.krd/data/krd/Registration%20Policy.pdf +co.krd +edu.krd + +// LCube - Professional hosting e.K. : https://www.lcube-webhosting.de +// Submitted by Lars Laehn +git-repos.de +lcube-server.de +svn-repos.de + +// Leadpages : https://www.leadpages.net +// Submitted by Greg Dallavalle +leadpages.co +lpages.co +lpusercontent.com + +// Lelux.fi : https://lelux.fi/ +// Submitted by Lelux Admin +lelux.site + +// Lifetime Hosting : https://Lifetime.Hosting/ +// Submitted by Mike Fillator +co.business +co.education +co.events +co.financial +co.network +co.place +co.technology + +// Lightmaker Property Manager, Inc. : https://app.lmpm.com/ +// Submitted by Greg Holland +app.lmpm.com + +// linkyard ldt: https://www.linkyard.ch/ +// Submitted by Mario Siegenthaler +linkyard.cloud +linkyard-cloud.ch + +// Linode : https://linode.com +// Submitted by +members.linode.com +*.nodebalancer.linode.com +*.linodeobjects.com + +// LiquidNet Ltd : http://www.liquidnetlimited.com/ +// Submitted by Victor Velchev +we.bs + +// localzone.xyz +// Submitted by Kenny Niehage +localzone.xyz + +// Log'in Line : https://www.loginline.com/ +// Submitted by Rémi Mach +loginline.app +loginline.dev +loginline.io +loginline.services +loginline.site + +// LubMAN UMCS Sp. z o.o : https://lubman.pl/ +// Submitted by Ireneusz Maliszewski +krasnik.pl +leczna.pl +lubartow.pl +lublin.pl +poniatowa.pl +swidnik.pl + +// Lug.org.uk : https://lug.org.uk +// Submitted by Jon Spriggs +glug.org.uk +lug.org.uk +lugs.org.uk + +// Lukanet Ltd : https://lukanet.com +// Submitted by Anton Avramov +barsy.bg +barsy.co.uk +barsyonline.co.uk +barsycenter.com +barsyonline.com +barsy.club +barsy.de +barsy.eu +barsy.in +barsy.info +barsy.io +barsy.me +barsy.menu +barsy.mobi +barsy.net +barsy.online +barsy.org +barsy.pro +barsy.pub +barsy.shop +barsy.site +barsy.support +barsy.uk + +// Magento Commerce +// Submitted by Damien Tournoud +*.magentosite.cloud + +// May First - People Link : https://mayfirst.org/ +// Submitted by Jamie McClelland +mayfirst.info +mayfirst.org + +// Mail.Ru Group : https://hb.cldmail.ru +// Submitted by Ilya Zaretskiy +hb.cldmail.ru + +// mcpe.me : https://mcpe.me +// Submitted by Noa Heyl +mcpe.me + +// McHost : https://mchost.ru +// Submitted by Evgeniy Subbotin +mcdir.ru +vps.mcdir.ru + +// Memset hosting : https://www.memset.com +// Submitted by Tom Whitwell +miniserver.com +memset.net + +// MetaCentrum, CESNET z.s.p.o. : https://www.metacentrum.cz/en/ +// Submitted by Zdeněk Šustr +*.cloud.metacentrum.cz +custom.metacentrum.cz + +// MetaCentrum, CESNET z.s.p.o. : https://www.metacentrum.cz/en/ +// Submitted by Radim Janča +flt.cloud.muni.cz +usr.cloud.muni.cz + +// Meteor Development Group : https://www.meteor.com/hosting +// Submitted by Pierre Carrier +meteorapp.com +eu.meteorapp.com + +// Michau Enterprises Limited : http://www.co.pl/ +co.pl + +// Microsoft Corporation : http://microsoft.com +// Submitted by Mitch Webster +*.azurecontainer.io +azurewebsites.net +azure-mobile.net +cloudapp.net +azurestaticapps.net +centralus.azurestaticapps.net +eastasia.azurestaticapps.net +eastus2.azurestaticapps.net +westeurope.azurestaticapps.net +westus2.azurestaticapps.net + +// minion.systems : http://minion.systems +// Submitted by Robert Böttinger +csx.cc + +// MobileEducation, LLC : https://joinforte.com +// Submitted by Grayson Martin +forte.id + +// Mozilla Corporation : https://mozilla.com +// Submitted by Ben Francis +mozilla-iot.org + +// Mozilla Foundation : https://mozilla.org/ +// Submitted by glob +bmoattachments.org + +// MSK-IX : https://www.msk-ix.ru/ +// Submitted by Khannanov Roman +net.ru +org.ru +pp.ru + +// Mythic Beasts : https://www.mythic-beasts.com +// Submitted by Paul Cammish +hostedpi.com +customer.mythic-beasts.com +lynx.mythic-beasts.com +ocelot.mythic-beasts.com +onza.mythic-beasts.com +sphinx.mythic-beasts.com +vs.mythic-beasts.com +x.mythic-beasts.com +yali.mythic-beasts.com +cust.retrosnub.co.uk + +// Nabu Casa : https://www.nabucasa.com +// Submitted by Paulus Schoutsen +ui.nabu.casa + +// Names.of.London : https://names.of.london/ +// Submitted by James Stevens or +pony.club +of.fashion +in.london +of.london +from.marketing +with.marketing +for.men +repair.men +and.mom +for.mom +for.one +under.one +for.sale +that.win +from.work +to.work + +// NCTU.ME : https://nctu.me/ +// Submitted by Tocknicsu +nctu.me + +// Netlify : https://www.netlify.com +// Submitted by Jessica Parsons +netlify.app + +// Neustar Inc. +// Submitted by Trung Tran +4u.com + +// ngrok : https://ngrok.com/ +// Submitted by Alan Shreve +ngrok.io + +// Nimbus Hosting Ltd. : https://www.nimbushosting.co.uk/ +// Submitted by Nicholas Ford +nh-serv.co.uk + +// NFSN, Inc. : https://www.NearlyFreeSpeech.NET/ +// Submitted by Jeff Wheelhouse +nfshost.com + +// Now-DNS : https://now-dns.com +// Submitted by Steve Russell +dnsking.ch +mypi.co +n4t.co +001www.com +ddnslive.com +myiphost.com +forumz.info +16-b.it +32-b.it +64-b.it +soundcast.me +tcp4.me +dnsup.net +hicam.net +now-dns.net +ownip.net +vpndns.net +dynserv.org +now-dns.org +x443.pw +now-dns.top +ntdll.top +freeddns.us +crafting.xyz +zapto.xyz + +// nsupdate.info : https://www.nsupdate.info/ +// Submitted by Thomas Waldmann +nsupdate.info +nerdpol.ovh + +// No-IP.com : https://noip.com/ +// Submitted by Deven Reza +blogsyte.com +brasilia.me +cable-modem.org +ciscofreak.com +collegefan.org +couchpotatofries.org +damnserver.com +ddns.me +ditchyourip.com +dnsfor.me +dnsiskinky.com +dvrcam.info +dynns.com +eating-organic.net +fantasyleague.cc +geekgalaxy.com +golffan.us +health-carereform.com +homesecuritymac.com +homesecuritypc.com +hopto.me +ilovecollege.info +loginto.me +mlbfan.org +mmafan.biz +myactivedirectory.com +mydissent.net +myeffect.net +mymediapc.net +mypsx.net +mysecuritycamera.com +mysecuritycamera.net +mysecuritycamera.org +net-freaks.com +nflfan.org +nhlfan.net +no-ip.ca +no-ip.co.uk +no-ip.net +noip.us +onthewifi.com +pgafan.net +point2this.com +pointto.us +privatizehealthinsurance.net +quicksytes.com +read-books.org +securitytactics.com +serveexchange.com +servehumour.com +servep2p.com +servesarcasm.com +stufftoread.com +ufcfan.org +unusualperson.com +workisboring.com +3utilities.com +bounceme.net +ddns.net +ddnsking.com +gotdns.ch +hopto.org +myftp.biz +myftp.org +myvnc.com +no-ip.biz +no-ip.info +no-ip.org +noip.me +redirectme.net +servebeer.com +serveblog.net +servecounterstrike.com +serveftp.com +servegame.com +servehalflife.com +servehttp.com +serveirc.com +serveminecraft.net +servemp3.com +servepics.com +servequake.com +sytes.net +webhop.me +zapto.org + +// NodeArt : https://nodeart.io +// Submitted by Konstantin Nosov +stage.nodeart.io + +// Nodum B.V. : https://nodum.io/ +// Submitted by Wietse Wind +nodum.co +nodum.io + +// Nucleos Inc. : https://nucleos.com +// Submitted by Piotr Zduniak +pcloud.host + +// NYC.mn : http://www.information.nyc.mn +// Submitted by Matthew Brown +nyc.mn + +// NymNom : https://nymnom.com/ +// Submitted by NymNom +nom.ae +nom.af +nom.ai +nom.al +nym.by +nom.bz +nym.bz +nom.cl +nym.ec +nom.gd +nom.ge +nom.gl +nym.gr +nom.gt +nym.gy +nym.hk +nom.hn +nym.ie +nom.im +nom.ke +nym.kz +nym.la +nym.lc +nom.li +nym.li +nym.lt +nym.lu +nom.lv +nym.me +nom.mk +nym.mn +nym.mx +nom.nu +nym.nz +nym.pe +nym.pt +nom.pw +nom.qa +nym.ro +nom.rs +nom.si +nym.sk +nom.st +nym.su +nym.sx +nom.tj +nym.tw +nom.ug +nom.uy +nom.vc +nom.vg + +// Observable, Inc. : https://observablehq.com +// Submitted by Mike Bostock +static.observableusercontent.com + +// Octopodal Solutions, LLC. : https://ulterius.io/ +// Submitted by Andrew Sampson +cya.gg + +// OMG.LOL : +// Submitted by Adam Newbold +omg.lol + +// Omnibond Systems, LLC. : https://www.omnibond.com +// Submitted by Cole Estep +cloudycluster.net + +// OmniWe Limited: https://omniwe.com +// Submitted by Vicary Archangel +omniwe.site + +// One Fold Media : http://www.onefoldmedia.com/ +// Submitted by Eddie Jones +nid.io + +// Open Social : https://www.getopensocial.com/ +// Submitted by Alexander Varwijk +opensocial.site + +// OpenCraft GmbH : http://opencraft.com/ +// Submitted by Sven Marnach +opencraft.hosting + +// Opera Software, A.S.A. +// Submitted by Yngve Pettersen +operaunite.com + +// Oursky Limited : https://skygear.io/ +// Submited by Skygear Developer +skygearapp.com + +// OutSystems +// Submitted by Duarte Santos +outsystemscloud.com + +// OwnProvider GmbH: http://www.ownprovider.com +// Submitted by Jan Moennich +ownprovider.com +own.pm + +// OwO : https://whats-th.is/ +// Submitted by Dean Sheather +*.owo.codes + +// OX : http://www.ox.rs +// Submitted by Adam Grand +ox.rs + +// oy.lc +// Submitted by Charly Coste +oy.lc + +// Pagefog : https://pagefog.com/ +// Submitted by Derek Myers +pgfog.com + +// Pagefront : https://www.pagefronthq.com/ +// Submitted by Jason Kriss +pagefrontapp.com + +// PageXL : https://pagexl.com +// Submitted by Yann Guichard +pagexl.com + +// pcarrier.ca Software Inc: https://pcarrier.ca/ +// Submitted by Pierre Carrier +bar0.net +bar1.net +bar2.net +rdv.to + +// .pl domains (grandfathered) +art.pl +gliwice.pl +krakow.pl +poznan.pl +wroc.pl +zakopane.pl + +// Pantheon Systems, Inc. : https://pantheon.io/ +// Submitted by Gary Dylina +pantheonsite.io +gotpantheon.com + +// Peplink | Pepwave : http://peplink.com/ +// Submitted by Steve Leung +mypep.link + +// Perspecta : https://perspecta.com/ +// Submitted by Kenneth Van Alstyne +perspecta.cloud + +// PE Ulyanov Kirill Sergeevich : https://airy.host +// Submitted by Kirill Ulyanov +lk3.ru +ra-ru.ru +zsew.ru + +// Planet-Work : https://www.planet-work.com/ +// Submitted by Frédéric VANNIÈRE +on-web.fr + +// Platform.sh : https://platform.sh +// Submitted by Nikola Kotur +bc.platform.sh +ent.platform.sh +eu.platform.sh +us.platform.sh +*.platformsh.site + +// Platter: https://platter.dev +// Submitted by Patrick Flor +platter-app.com +platter-app.dev +platterp.us + +// Plesk : https://www.plesk.com/ +// Submitted by Anton Akhtyamov +pdns.page +plesk.page +pleskns.com + +// Port53 : https://port53.io/ +// Submitted by Maximilian Schieder +dyn53.io + +// Positive Codes Technology Company : http://co.bn/faq.html +// Submitted by Zulfais +co.bn + +// prgmr.com : https://prgmr.com/ +// Submitted by Sarah Newman +xen.prgmr.com + +// priv.at : http://www.nic.priv.at/ +// Submitted by registry +priv.at + +// privacytools.io : https://www.privacytools.io/ +// Submitted by Jonah Aragon +prvcy.page + +// Protocol Labs : https://protocol.ai/ +// Submitted by Michael Burns +*.dweb.link + +// Protonet GmbH : http://protonet.io +// Submitted by Martin Meier +protonet.io + +// Publication Presse Communication SARL : https://ppcom.fr +// Submitted by Yaacov Akiba Slama +chirurgiens-dentistes-en-france.fr +byen.site + +// pubtls.org: https://www.pubtls.org +// Submitted by Kor Nielsen +pubtls.org + +// QOTO, Org. +// Submitted by Jeffrey Phillips Freeman +qoto.io + +// Qualifio : https://qualifio.com/ +// Submitted by Xavier De Cock +qualifioapp.com + +// QuickBackend: https://www.quickbackend.com +// Submitted by Dani Biro +qbuser.com + +// Redstar Consultants : https://www.redstarconsultants.com/ +// Submitted by Jons Slemmer +instantcloud.cn + +// Russian Academy of Sciences +// Submitted by Tech Support +ras.ru + +// QA2 +// Submitted by Daniel Dent (https://www.danieldent.com/) +qa2.com + +// QCX +// Submitted by Cassandra Beelen +qcx.io +*.sys.qcx.io + +// QNAP System Inc : https://www.qnap.com +// Submitted by Nick Chang +dev-myqnapcloud.com +alpha-myqnapcloud.com +myqnapcloud.com + +// Quip : https://quip.com +// Submitted by Patrick Linehan +*.quipelements.com + +// Qutheory LLC : http://qutheory.io +// Submitted by Jonas Schwartz +vapor.cloud +vaporcloud.io + +// Rackmaze LLC : https://www.rackmaze.com +// Submitted by Kirill Pertsev +rackmaze.com +rackmaze.net + +// Rakuten Games, Inc : https://dev.viberplay.io +// Submitted by Joshua Zhang +g.vbrplsbx.io + +// Rancher Labs, Inc : https://rancher.com +// Submitted by Vincent Fiduccia +*.on-k3s.io +*.on-rancher.cloud +*.on-rio.io + +// Read The Docs, Inc : https://www.readthedocs.org +// Submitted by David Fischer +readthedocs.io + +// Red Hat, Inc. OpenShift : https://openshift.redhat.com/ +// Submitted by Tim Kramer +rhcloud.com + +// Render : https://render.com +// Submitted by Anurag Goel +app.render.com +onrender.com + +// Repl.it : https://repl.it +// Submitted by Mason Clayton +repl.co +repl.run + +// Resin.io : https://resin.io +// Submitted by Tim Perry +resindevice.io +devices.resinstaging.io + +// RethinkDB : https://www.rethinkdb.com/ +// Submitted by Chris Kastorff +hzc.io + +// Revitalised Limited : http://www.revitalised.co.uk +// Submitted by Jack Price +wellbeingzone.eu +wellbeingzone.co.uk + +// Rochester Institute of Technology : http://www.rit.edu/ +// Submitted by Jennifer Herting +git-pages.rit.edu + +// Sandstorm Development Group, Inc. : https://sandcats.io/ +// Submitted by Asheesh Laroia +sandcats.io + +// SBE network solutions GmbH : https://www.sbe.de/ +// Submitted by Norman Meilick +logoip.de +logoip.com + +// schokokeks.org GbR : https://schokokeks.org/ +// Submitted by Hanno Böck +schokokeks.net + +// Scottish Government: https://www.gov.scot +// Submitted by Martin Ellis +gov.scot + +// Scry Security : http://www.scrysec.com +// Submitted by Shante Adam +scrysec.com + +// Securepoint GmbH : https://www.securepoint.de +// Submitted by Erik Anders +firewall-gateway.com +firewall-gateway.de +my-gateway.de +my-router.de +spdns.de +spdns.eu +firewall-gateway.net +my-firewall.org +myfirewall.org +spdns.org + +// Seidat : https://www.seidat.com +// Submitted by Artem Kondratev +seidat.net + +// Senseering GmbH : https://www.senseering.de +// Submitted by Felix Mönckemeyer +senseering.net + +// Service Online LLC : http://drs.ua/ +// Submitted by Serhii Bulakh +biz.ua +co.ua +pp.ua + +// ShiftEdit : https://shiftedit.net/ +// Submitted by Adam Jimenez +shiftedit.io + +// Shopblocks : http://www.shopblocks.com/ +// Submitted by Alex Bowers +myshopblocks.com + +// Shopit : https://www.shopitcommerce.com/ +// Submitted by Craig McMahon +shopitsite.com + +// shopware AG : https://shopware.com +// Submitted by Jens Küper +shopware.store + +// Siemens Mobility GmbH +// Submitted by Oliver Graebner +mo-siemens.io + +// SinaAppEngine : http://sae.sina.com.cn/ +// Submitted by SinaAppEngine +1kapp.com +appchizi.com +applinzi.com +sinaapp.com +vipsinaapp.com + +// Siteleaf : https://www.siteleaf.com/ +// Submitted by Skylar Challand +siteleaf.net + +// Skyhat : http://www.skyhat.io +// Submitted by Shante Adam +bounty-full.com +alpha.bounty-full.com +beta.bounty-full.com + +// Small Technology Foundation : https://small-tech.org +// Submitted by Aral Balkan +small-web.org + +// Stackhero : https://www.stackhero.io +// Submitted by Adrien Gillon +stackhero-network.com + +// staticland : https://static.land +// Submitted by Seth Vincent +static.land +dev.static.land +sites.static.land + +// Sony Interactive Entertainment LLC : https://sie.com/ +// Submitted by David Coles +playstation-cloud.com + +// SourceLair PC : https://www.sourcelair.com +// Submitted by Antonis Kalipetis +apps.lair.io +*.stolos.io + +// SpaceKit : https://www.spacekit.io/ +// Submitted by Reza Akhavan +spacekit.io + +// SpeedPartner GmbH: https://www.speedpartner.de/ +// Submitted by Stefan Neufeind +customer.speedpartner.de + +// Standard Library : https://stdlib.com +// Submitted by Jacob Lee +api.stdlib.com + +// Storj Labs Inc. : https://storj.io/ +// Submitted by Philip Hutchins +storj.farm + +// Studenten Net Twente : http://www.snt.utwente.nl/ +// Submitted by Silke Hofstra +utwente.io + +// Student-Run Computing Facility : https://www.srcf.net/ +// Submitted by Edwin Balani +soc.srcf.net +user.srcf.net + +// Sub 6 Limited: http://www.sub6.com +// Submitted by Dan Miller +temp-dns.com + +// Swisscom Application Cloud: https://developer.swisscom.com +// Submitted by Matthias.Winzeler +applicationcloud.io +scapp.io + +// Symfony, SAS : https://symfony.com/ +// Submitted by Fabien Potencier +*.s5y.io +*.sensiosite.cloud + +// Syncloud : https://syncloud.org +// Submitted by Boris Rybalkin +syncloud.it + +// Synology, Inc. : https://www.synology.com/ +// Submitted by Rony Weng +diskstation.me +dscloud.biz +dscloud.me +dscloud.mobi +dsmynas.com +dsmynas.net +dsmynas.org +familyds.com +familyds.net +familyds.org +i234.me +myds.me +synology.me +vpnplus.to +direct.quickconnect.to + +// TAIFUN Software AG : http://taifun-software.de +// Submitted by Bjoern Henke +taifun-dns.de + +// TASK geographical domains (www.task.gda.pl/uslugi/dns) +gda.pl +gdansk.pl +gdynia.pl +med.pl +sopot.pl + +// Teckids e.V. : https://www.teckids.org +// Submitted by Dominik George +edugit.org + +// Telebit : https://telebit.cloud +// Submitted by AJ ONeal +telebit.app +telebit.io +*.telebit.xyz + +// The Gwiddle Foundation : https://gwiddlefoundation.org.uk +// Submitted by Joshua Bayfield +gwiddle.co.uk + +// Thingdust AG : https://thingdust.com/ +// Submitted by Adrian Imboden +thingdustdata.com +cust.dev.thingdust.io +cust.disrec.thingdust.io +cust.prod.thingdust.io +cust.testing.thingdust.io +*.firenet.ch +*.svc.firenet.ch + +// Tlon.io : https://tlon.io +// Submitted by Mark Staarink +arvo.network +azimuth.network +tlon.network + +// TownNews.com : http://www.townnews.com +// Submitted by Dustin Ward +bloxcms.com +townnews-staging.com + +// TrafficPlex GmbH : https://www.trafficplex.de/ +// Submitted by Phillipp Röll +12hp.at +2ix.at +4lima.at +lima-city.at +12hp.ch +2ix.ch +4lima.ch +lima-city.ch +trafficplex.cloud +de.cool +12hp.de +2ix.de +4lima.de +lima-city.de +1337.pictures +clan.rip +lima-city.rocks +webspace.rocks +lima.zone + +// TransIP : https://www.transip.nl +// Submitted by Rory Breuk +*.transurl.be +*.transurl.eu +*.transurl.nl + +// TuxFamily : http://tuxfamily.org +// Submitted by TuxFamily administrators +tuxfamily.org + +// TwoDNS : https://www.twodns.de/ +// Submitted by TwoDNS-Support +dd-dns.de +diskstation.eu +diskstation.org +dray-dns.de +draydns.de +dyn-vpn.de +dynvpn.de +mein-vigor.de +my-vigor.de +my-wan.de +syno-ds.de +synology-diskstation.de +synology-ds.de + +// Uberspace : https://uberspace.de +// Submitted by Moritz Werner +uber.space +*.uberspace.de + +// UDR Limited : http://www.udr.hk.com +// Submitted by registry +hk.com +hk.org +ltd.hk +inc.hk + +// United Gameserver GmbH : https://united-gameserver.de +// Submitted by Stefan Schwarz +virtualuser.de +virtual-user.de + +// urown.net : https://urown.net +// Submitted by Hostmaster +urown.cloud +dnsupdate.info + +// .US +// Submitted by Ed Moore +lib.de.us + +// VeryPositive SIA : http://very.lv +// Submitted by Danko Aleksejevs +2038.io + +// Vercel, Inc : https://vercel.com/ +// Submitted by Connor Davis +vercel.app +vercel.dev +now.sh + +// Viprinet Europe GmbH : http://www.viprinet.com +// Submitted by Simon Kissel +router.management + +// Virtual-Info : https://www.virtual-info.info/ +// Submitted by Adnan RIHAN +v-info.info + +// Voorloper.com: https://voorloper.com +// Submitted by Nathan van Bakel +voorloper.cloud + +// Voxel.sh DNS : https://voxel.sh/dns/ +// Submitted by Mia Rehlinger +neko.am +nyaa.am +be.ax +cat.ax +es.ax +eu.ax +gg.ax +mc.ax +us.ax +xy.ax +nl.ci +xx.gl +app.gp +blog.gt +de.gt +to.gt +be.gy +cc.hn +blog.kg +io.kg +jp.kg +tv.kg +uk.kg +us.kg +de.ls +at.md +de.md +jp.md +to.md +uwu.nu +indie.porn +vxl.sh +ch.tc +me.tc +we.tc +nyan.to +at.vg +blog.vu +dev.vu +me.vu + +// V.UA Domain Administrator : https://domain.v.ua/ +// Submitted by Serhii Rostilo +v.ua + +// Waffle Computer Inc., Ltd. : https://docs.waffleinfo.com +// Submitted by Masayuki Note +wafflecell.com + +// WapBlog.ID : https://www.wapblog.id +// Submitted by Fajar Sodik +idnblogger.com +indowapblog.com +bloger.id +wblog.id +wbq.me +fastblog.net + +// WebHare bv: https://www.webhare.com/ +// Submitted by Arnold Hendriks +*.webhare.dev + +// WeDeploy by Liferay, Inc. : https://www.wedeploy.com +// Submitted by Henrique Vicente +wedeploy.io +wedeploy.me +wedeploy.sh + +// Western Digital Technologies, Inc : https://www.wdc.com +// Submitted by Jung Jin +remotewd.com + +// WIARD Enterprises : https://wiardweb.com +// Submitted by Kidd Hustle +pages.wiardweb.com + +// Wikimedia Labs : https://wikitech.wikimedia.org +// Submitted by Arturo Borrero Gonzalez +wmflabs.org +toolforge.org +wmcloud.org + +// WISP : https://wisp.gg +// Submitted by Stepan Fedotov +panel.gg +daemon.panel.gg + +// WoltLab GmbH : https://www.woltlab.com +// Submitted by Tim Düsterhus +myforum.community +community-pro.de +diskussionsbereich.de +community-pro.net +meinforum.net + +// www.com.vc : http://www.com.vc +// Submitted by Li Hui +cn.vu + +// XenonCloud GbR: https://xenoncloud.net +// Submitted by Julian Uphoff +half.host + +// XnBay Technology : http://www.xnbay.com/ +// Submitted by XnBay Developer +xnbay.com +u2.xnbay.com +u2-local.xnbay.com + +// XS4ALL Internet bv : https://www.xs4all.nl/ +// Submitted by Daniel Mostertman +cistron.nl +demon.nl +xs4all.space + +// Yandex.Cloud LLC: https://cloud.yandex.com +// Submitted by Alexander Lodin +yandexcloud.net +storage.yandexcloud.net +website.yandexcloud.net + +// YesCourse Pty Ltd : https://yescourse.com +// Submitted by Atul Bhouraskar +official.academy + +// Yola : https://www.yola.com/ +// Submitted by Stefano Rivera +yolasite.com + +// Yombo : https://yombo.net +// Submitted by Mitch Schwenk +ybo.faith +yombo.me +homelink.one +ybo.party +ybo.review +ybo.science +ybo.trade + +// Yunohost : https://yunohost.org +// Submitted by Valentin Grimaud +nohost.me +noho.st + +// ZaNiC : http://www.za.net/ +// Submitted by registry +za.net +za.org + +// Zine EOOD : https://zine.bg/ +// Submitted by Martin Angelov +bss.design + +// Zitcom A/S : https://www.zitcom.dk +// Submitted by Emil Stahl +basicserver.io +virtualserver.io +enterprisecloud.nu + +// Mintere : https://mintere.com/ +// Submitted by Ben Aubin +mintere.site + +// Cityhost LLC : https://cityhost.ua +// Submitted by Maksym Rivtin +cx.ua + +// WP Engine : https://wpengine.com/ +// Submitted by Michael Smith +// Submitted by Brandon DuRette +wpenginepowered.com +js.wpenginepowered.com + +// Impertrix Solutions : +// Submitted by Zhixiang Zhao +impertrixcdn.com +impertrix.com + +// GignoSystemJapan: http://gsj.bz +// Submitted by GignoSystemJapan +gsj.bz +// ===END PRIVATE DOMAINS=== diff --git a/docs/en/development/architecture.md b/docs/en/development/architecture.md index 19caa5241b0..4ef01f4e4fb 100644 --- a/docs/en/development/architecture.md +++ b/docs/en/development/architecture.md @@ -177,8 +177,6 @@ When you `INSERT` a bunch of data into `MergeTree`, that bunch is sorted by prim `MergeTree` is not an LSM tree because it doesn’t contain “memtable” and “log”: inserted data is written directly to the filesystem. This makes it suitable only to INSERT data in batches, not by individual row and not very frequently – about once per second is ok, but a thousand times a second is not. We did it this way for simplicity’s sake, and because we are already inserting data in batches in our applications. -> MergeTree tables can only have one (primary) index: there aren’t any secondary indices. It would be nice to allow multiple physical representations under one logical table, for example, to store data in more than one physical order or even to allow representations with pre-aggregated data along with original data. - There are MergeTree engines that are doing additional work during background merges. Examples are `CollapsingMergeTree` and `AggregatingMergeTree`. This could be treated as special support for updates. Keep in mind that these are not real updates because users usually have no control over the time when background merges are executed, and data in a `MergeTree` table is almost always stored in more than one part, not in completely merged form. ## Replication {#replication} diff --git a/docs/en/getting-started/example-datasets/github-events.md b/docs/en/getting-started/example-datasets/github-events.md new file mode 100644 index 00000000000..a6c71733832 --- /dev/null +++ b/docs/en/getting-started/example-datasets/github-events.md @@ -0,0 +1,11 @@ +--- +toc_priority: 11 +toc_title: GitHub Events +--- + +# GitHub Events Dataset + +Dataset contains all events on GitHub from 2011 to Dec 6 2020, the size is 3.1 billion records. Download size is 75 GB and it will require up to 200 GB space on disk if stored in a table with lz4 compression. + +Full dataset description, insights, download instruction and interactive queries are posted [here](https://github-sql.github.io/explorer/). + diff --git a/docs/en/getting-started/example-datasets/index.md b/docs/en/getting-started/example-datasets/index.md index 35ac90f9beb..b769e8fcb45 100644 --- a/docs/en/getting-started/example-datasets/index.md +++ b/docs/en/getting-started/example-datasets/index.md @@ -1,6 +1,6 @@ --- toc_folder_title: Example Datasets -toc_priority: 14 +toc_priority: 10 toc_title: Introduction --- @@ -18,4 +18,4 @@ The list of documented datasets: - [New York Taxi Data](../../getting-started/example-datasets/nyc-taxi.md) - [OnTime](../../getting-started/example-datasets/ontime.md) -[Original article](https://clickhouse.tech/docs/en/getting_started/example_datasets) \ No newline at end of file +[Original article](https://clickhouse.tech/docs/en/getting_started/example_datasets) diff --git a/docs/en/interfaces/cli.md b/docs/en/interfaces/cli.md index 42416383860..a2ea6edf24a 100644 --- a/docs/en/interfaces/cli.md +++ b/docs/en/interfaces/cli.md @@ -113,7 +113,8 @@ You can pass parameters to `clickhouse-client` (all parameters have a default va - `--port` – The port to connect to. Default value: 9000. Note that the HTTP interface and the native interface use different ports. - `--user, -u` – The username. Default value: default. - `--password` – The password. Default value: empty string. -- `--query, -q` – The query to process when using non-interactive mode. +- `--query, -q` – The query to process when using non-interactive mode. You must specify either `query` or `queries-file` option. +- `--queries-file, -qf` - file path with queries to execute. You must specify either `query` or `queries-file` option. - `--database, -d` – Select the current default database. Default value: the current database from the server settings (‘default’ by default). - `--multiline, -m` – If specified, allow multiline queries (do not send the query on Enter). - `--multiquery, -n` – If specified, allow processing multiple queries separated by semicolons. diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md index 618ae374e8a..95b339a3269 100644 --- a/docs/en/interfaces/formats.md +++ b/docs/en/interfaces/formats.md @@ -58,6 +58,7 @@ The supported formats are: | [XML](#xml) | ✗ | ✔ | | [CapnProto](#capnproto) | ✔ | ✗ | | [LineAsString](#lineasstring) | ✔ | ✗ | +| [RawBLOB](#rawblob) | ✔ | ✔ | You can control some format processing parameters with the ClickHouse settings. For more information read the [Settings](../operations/settings/settings.md) section. @@ -457,7 +458,10 @@ This format is only appropriate for outputting a query result, but not for parsi ClickHouse supports [NULL](../sql-reference/syntax.md), which is displayed as `null` in the JSON output. To enable `+nan`, `-nan`, `+inf`, `-inf` values in output, set the [output_format_json_quote_denormals](../operations/settings/settings.md#settings-output_format_json_quote_denormals) to 1. -See also the [JSONEachRow](#jsoneachrow) format. +**See Also** + +- [JSONEachRow](#jsoneachrow) format +- [output_format_json_array_of_rows](../operations/settings/settings.md#output-format-json-array-of-rows) setting ## JSONString {#jsonstring} @@ -1367,4 +1371,45 @@ Result: └───────────────────────────────────────────────────┘ ``` +## RawBLOB {#rawblob} + +In this format, all input data is read to a single value. It is possible to parse only a table with a single field of type [String](../sql-reference/data-types/string.md) or similar. +The result is output in binary format without delimiters and escaping. If more than one value is output, the format is ambiguous, and it will be impossible to read the data back. + +Below is a comparison of the formats `RawBLOB` and [TabSeparatedRaw](#tabseparatedraw). +`RawBLOB`: +- data is output in binary format, no escaping; +- there are no delimiters between values; +- no newline at the end of each value. +[TabSeparatedRaw] (#tabseparatedraw): +- data is output without escaping; +- the rows contain values separated by tabs; +- there is a line feed after the last value in every row. + +The following is a comparison of the `RawBLOB` and [RowBinary](#rowbinary) formats. +`RawBLOB`: +- String fields are output without being prefixed by length. +`RowBinary`: +- String fields are represented as length in varint format (unsigned [LEB128] (https://en.wikipedia.org/wiki/LEB128)), followed by the bytes of the string. + +When an empty data is passed to the `RawBLOB` input, ClickHouse throws an exception: + +``` text +Code: 108. DB::Exception: No data to insert +``` + +**Example** + +``` bash +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" +$ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum +``` + +Result: + +``` text +f9725a22f9191e064120d718e26862a9 - +``` + [Original article](https://clickhouse.tech/docs/en/interfaces/formats/) diff --git a/docs/en/operations/settings/settings.md b/docs/en/operations/settings/settings.md index 29c74b76c29..fc921f2ef7e 100644 --- a/docs/en/operations/settings/settings.md +++ b/docs/en/operations/settings/settings.md @@ -2360,10 +2360,89 @@ Default value: `1`. ## output_format_tsv_null_representation {#output_format_tsv_null_representation} -Allows configurable `NULL` representation for [TSV](../../interfaces/formats.md#tabseparated) output format. The setting only controls output format and `\N` is the only supported `NULL` representation for TSV input format. +Defines the representation of `NULL` for [TSV](../../interfaces/formats.md#tabseparated) output format. User can set any string as a value, for example, `My NULL`. Default value: `\N`. +**Examples** + +Query + +```sql +SELECT * FROM tsv_custom_null FORMAT TSV; +``` + +Result + +```text +788 +\N +\N +``` + +Query + +```sql +SET output_format_tsv_null_representation = 'My NULL'; +SELECT * FROM tsv_custom_null FORMAT TSV; +``` + +Result + +```text +788 +My NULL +My NULL +``` + +## output_format_json_array_of_rows {#output-format-json-array-of-rows} + +Enables the ability to output all rows as a JSON array in the [JSONEachRow](../../interfaces/formats.md#jsoneachrow) format. + +Possible values: + +- 1 — ClickHouse outputs all rows as an array, each row in the `JSONEachRow` format. +- 0 — ClickHouse outputs each row separately in the `JSONEachRow` format. + +Default value: `0`. + +**Example of a query with the enabled setting** + +Query: + +```sql +SET output_format_json_array_of_rows = 1; +SELECT number FROM numbers(3) FORMAT JSONEachRow; +``` + +Result: + +```text +[ +{"number":"0"}, +{"number":"1"}, +{"number":"2"} +] +``` + +**Example of a query with the disabled setting** + +Query: + +```sql +SET output_format_json_array_of_rows = 0; +SELECT number FROM numbers(3) FORMAT JSONEachRow; +``` + +Result: + +```text +{"number":"0"} +{"number":"1"} +{"number":"2"} +``` + +======= ## allow_nullable_key {#allow-nullable-key} Allows using of the [Nullable](../../sql-reference/data-types/nullable.md#data_type-nullable)-typed values in a sorting and a primary key for [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md#table_engines-mergetree) tables. diff --git a/docs/en/operations/system-tables/distribution_queue.md b/docs/en/operations/system-tables/distribution_queue.md new file mode 100644 index 00000000000..da4effd0ff5 --- /dev/null +++ b/docs/en/operations/system-tables/distribution_queue.md @@ -0,0 +1,42 @@ +# system.distribution_queue {#system_tables-distribution_queue} + +Contains information about local files that are in the queue to be sent to the shards. This local files contain new parts that are created by inserting new data into the Distributed table in asynchronous mode. + +Columns: + +- `database` ([String](../../sql-reference/data-types/string.md)) — Name of the database. + +- `table` ([String](../../sql-reference/data-types/string.md)) — Name of the table. + +- `data_path` ([String](../../sql-reference/data-types/string.md)) — Path to the folder with local files. + +- `is_blocked` ([UInt8](../../sql-reference/data-types/int-uint.md)) — Flag indicates whether sending local files to the server is blocked. + +- `error_count` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Number of errors. + +- `data_files` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Number of local files in a folder. + +- `data_compressed_bytes` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Size of compressed data in local files, in bytes. + +- `last_exception` ([String](../../sql-reference/data-types/string.md)) — Text message about the last error that occurred (if any). + +**Example** + +``` sql +SELECT * FROM system.distribution_queue LIMIT 1 FORMAT Vertical; +``` + +``` text +Row 1: +────── +database: default +table: dist +data_path: ./store/268/268bc070-3aad-4b1a-9cf2-4987580161af/default@127%2E0%2E0%2E2:9000/ +is_blocked: 1 +error_count: 0 +data_files: 1 +data_compressed_bytes: 499 +last_exception: +``` + +[Original article](https://clickhouse.tech/docs/en/operations/system_tables/distribution_queue) diff --git a/docs/en/operations/utilities/clickhouse-local.md b/docs/en/operations/utilities/clickhouse-local.md index af3d06898fd..f93ba139cae 100644 --- a/docs/en/operations/utilities/clickhouse-local.md +++ b/docs/en/operations/utilities/clickhouse-local.md @@ -32,7 +32,8 @@ Arguments: - `-S`, `--structure` — table structure for input data. - `-if`, `--input-format` — input format, `TSV` by default. - `-f`, `--file` — path to data, `stdin` by default. -- `-q` `--query` — queries to execute with `;` as delimeter. +- `-q` `--query` — queries to execute with `;` as delimeter. You must specify either `query` or `queries-file` option. +- `-qf` `--queries-file` - file path with queries to execute. You must specify either `query` or `queries-file` option. - `-N`, `--table` — table name where to put output data, `table` by default. - `-of`, `--format`, `--output-format` — output format, `TSV` by default. - `--stacktrace` — whether to dump debug output in case of exception. diff --git a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md index 957f2b6ae53..e86ac7fe105 100644 --- a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md +++ b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md @@ -93,6 +93,8 @@ Setting fields: - `path` – The absolute path to the file. - `format` – The file format. All the formats described in “[Formats](../../../interfaces/formats.md#formats)” are supported. +When dictionary with FILE source is created via DDL command (`CREATE DICTIONARY ...`), source of the dictionary have to be located in `user_files` directory, to prevent DB users accessing arbitrary file on clickhouse node. + ## Executable File {#dicts-external_dicts_dict_sources-executable} Working with executable files depends on [how the dictionary is stored in memory](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file’s STDIN. Otherwise, ClickHouse starts executable file and treats its output as dictionary data. @@ -108,17 +110,13 @@ Example of settings: ``` -or - -``` sql -SOURCE(EXECUTABLE(command 'cat /opt/dictionaries/os.tsv' format 'TabSeparated')) -``` - Setting fields: - `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`). - `format` – The file format. All the formats described in “[Formats](../../../interfaces/formats.md#formats)” are supported. +That dictionary source can be configured only via XML configuration. Creating dictionaries with executable source via DDL is disabled, otherwise, the DB user would be able to execute arbitrary binary on clickhouse node. + ## Http(s) {#dicts-external_dicts_dict_sources-http} Working with an HTTP(s) server depends on [how the dictionary is stored in memory](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method. @@ -169,6 +167,8 @@ Setting fields: - `name` – Identifiant name used for the header send on the request. - `value` – Value set for a specific identifiant name. +When creating a dictionary using the DDL command (`CREATE DICTIONARY ...`) remote hosts for HTTP dictionaries checked with the `remote_url_allow_hosts` section from config to prevent database users to access arbitrary HTTP server. + ## ODBC {#dicts-external_dicts_dict_sources-odbc} You can use this method to connect any database that has an ODBC driver. diff --git a/docs/en/sql-reference/functions/date-time-functions.md b/docs/en/sql-reference/functions/date-time-functions.md index 75db3fafe36..628c321adee 100644 --- a/docs/en/sql-reference/functions/date-time-functions.md +++ b/docs/en/sql-reference/functions/date-time-functions.md @@ -366,7 +366,7 @@ SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(d └────────────┴───────────┴───────────┴───────────┘ ``` -## date_trunc {#date_trunc} +## date\_trunc {#date_trunc} Truncates date and time data to the specified part of date. @@ -435,7 +435,7 @@ Result: - [toStartOfInterval](#tostartofintervaltime-or-data-interval-x-unit-time-zone) -# now {#now} +## now {#now} Returns the current date and time. @@ -662,7 +662,7 @@ Result: [Original article](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) -## FROM_UNIXTIME +## FROM\_UNIXTIME {#fromunixfime} When there is only single argument of integer type, it act in the same way as `toDateTime` and return [DateTime](../../sql-reference/data-types/datetime.md). type. @@ -692,3 +692,147 @@ SELECT FROM_UNIXTIME(1234334543, '%Y-%m-%d %R:%S') AS DateTime │ 2009-02-11 14:42:23 │ └─────────────────────┘ ``` + +## toModifiedJulianDay {#tomodifiedjulianday} + +Converts a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD` to a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number in Int32. This function supports date from `0000-01-01` to `9999-12-31`. It raises an exception if the argument cannot be parsed as a date, or the date is invalid. + +**Syntax** + +``` sql +toModifiedJulianDay(date) +``` + +**Parameters** + +- `date` — Date in text form. [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md). + +**Returned value** + +- Modified Julian Day number. + +Type: [Int32](../../sql-reference/data-types/int-uint.md). + +**Example** + +Query: + +``` sql +SELECT toModifiedJulianDay('2020-01-01'); +``` + +Result: + +``` text +┌─toModifiedJulianDay('2020-01-01')─┐ +│ 58849 │ +└───────────────────────────────────┘ +``` + +## toModifiedJulianDayOrNull {#tomodifiedjuliandayornull} + +Similar to [toModifiedJulianDay()](#tomodifiedjulianday), but instead of raising exceptions it returns `NULL`. + +**Syntax** + +``` sql +toModifiedJulianDayOrNull(date) +``` + +**Parameters** + +- `date` — Date in text form. [String](../../sql-reference/data-types/string.md) or [FixedString](../../sql-reference/data-types/fixedstring.md). + +**Returned value** + +- Modified Julian Day number. + +Type: [Nullable(Int32)](../../sql-reference/data-types/int-uint.md). + +**Example** + +Query: + +``` sql +SELECT toModifiedJulianDayOrNull('2020-01-01'); +``` + +Result: + +``` text +┌─toModifiedJulianDayOrNull('2020-01-01')─┐ +│ 58849 │ +└─────────────────────────────────────────┘ +``` + +## fromModifiedJulianDay {#frommodifiedjulianday} + +Converts a [Modified Julian Day](https://en.wikipedia.org/wiki/Julian_day#Variants) number to a [Proleptic Gregorian calendar](https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar) date in text form `YYYY-MM-DD`. This function supports day number from `-678941` to `2973119` (which represent 0000-01-01 and 9999-12-31 respectively). It raises an exception if the day number is outside of the supported range. + +**Syntax** + +``` sql +fromModifiedJulianDay(day) +``` + +**Parameters** + +- `day` — Modified Julian Day number. [Any integral types](../../sql-reference/data-types/int-uint.md). + +**Returned value** + +- Date in text form. + +Type: [String](../../sql-reference/data-types/string.md) + +**Example** + +Query: + +``` sql +SELECT fromModifiedJulianDay(58849); +``` + +Result: + +``` text +┌─fromModifiedJulianDay(58849)─┐ +│ 2020-01-01 │ +└──────────────────────────────┘ +``` + +## fromModifiedJulianDayOrNull {#frommodifiedjuliandayornull} + +Similar to [fromModifiedJulianDayOrNull()](#frommodifiedjuliandayornull), but instead of raising exceptions it returns `NULL`. + +**Syntax** + +``` sql +fromModifiedJulianDayOrNull(day) +``` + +**Parameters** + +- `day` — Modified Julian Day number. [Any integral types](../../sql-reference/data-types/int-uint.md). + +**Returned value** + +- Date in text form. + +Type: [Nullable(String)](../../sql-reference/data-types/string.md) + +**Example** + +Query: + +``` sql +SELECT fromModifiedJulianDayOrNull(58849); +``` + +Result: + +``` text +┌─fromModifiedJulianDayOrNull(58849)─┐ +│ 2020-01-01 │ +└────────────────────────────────────┘ +``` diff --git a/docs/en/sql-reference/functions/math-functions.md b/docs/en/sql-reference/functions/math-functions.md index acb88afd219..1f45b69f7a1 100644 --- a/docs/en/sql-reference/functions/math-functions.md +++ b/docs/en/sql-reference/functions/math-functions.md @@ -111,4 +111,306 @@ Accepts a numeric argument and returns a UInt64 number close to 2 to the power o Accepts a numeric argument and returns a UInt64 number close to 10 to the power of x. +## cosh(x) {#coshx} + +[Hyperbolic cosine](https://in.mathworks.com/help/matlab/ref/cosh.html). + +**Syntax** + +``` sql +cosh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values from the interval: `1 <= cosh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT cosh(0); +``` + +Result: + +``` text +┌─cosh(0)──┐ +│ 1 │ +└──────────┘ +``` + +## acosh(x) {#acoshx} + +[Inverse hyperbolic cosine](https://www.mathworks.com/help/matlab/ref/acosh.html). + +**Syntax** + +``` sql +acosh(x) +``` + +**Parameters** + +- `x` — Hyperbolic cosine of angle. Values from the interval: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The angle, in radians. Values from the interval: `0 <= acosh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT acosh(1); +``` + +Result: + +``` text +┌─acosh(1)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [cosh(x)](../../sql-reference/functions/math-functions.md#coshx) + +## sinh(x) {#sinhx} + +[Hyperbolic sine](https://www.mathworks.com/help/matlab/ref/sinh.html). + +**Syntax** + +``` sql +sinh(x) +``` + +**Parameters** + +- `x` — The angle, in radians. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values from the interval: `-∞ < sinh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT sinh(0); +``` + +Result: + +``` text +┌─sinh(0)──┐ +│ 0 │ +└──────────┘ +``` + +## asinh(x) {#asinhx} + +[Inverse hyperbolic sine](https://www.mathworks.com/help/matlab/ref/asinh.html). + +**Syntax** + +``` sql +asinh(x) +``` + +**Parameters** + +- `x` — Hyperbolic sine of angle. Values from the interval: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The angle, in radians. Values from the interval: `-∞ < asinh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT asinh(0); +``` + +Result: + +``` text +┌─asinh(0)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [sinh(x)](../../sql-reference/functions/math-functions.md#sinhx) + +## atanh(x) {#atanhx} + +[Inverse hyperbolic tangent](https://www.mathworks.com/help/matlab/ref/atanh.html). + +**Syntax** + +``` sql +atanh(x) +``` + +**Parameters** + +- `x` — Hyperbolic tangent of angle. Values from the interval: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The angle, in radians. Values from the interval: `-∞ < atanh(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT atanh(0); +``` + +Result: + +``` text +┌─atanh(0)─┐ +│ 0 │ +└──────────┘ +``` + +## atan2(y, x) {#atan2yx} + +The [function](https://en.wikipedia.org/wiki/Atan2) calculates the angle in the Euclidean plane, given in radians, between the positive x axis and the ray to the point `(x, y) ≠ (0, 0)`. + +**Syntax** + +``` sql +atan2(y, x) +``` + +**Parameters** + +- `y` — y-coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — x-coordinate of the point through which the ray passes. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The angle `θ` such that `−π < θ ≤ π`, in radians. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT atan2(1, 1); +``` + +Result: + +``` text +┌────────atan2(1, 1)─┐ +│ 0.7853981633974483 │ +└────────────────────┘ +``` + +## hypot(x, y) {#hypotxy} + +Calculates the length of the hypotenuse of a right-angle triangle. The [function](https://en.wikipedia.org/wiki/Hypot) avoids problems that occur when squaring very large or very small numbers. + +**Syntax** + +``` sql +hypot(x, y) +``` + +**Parameters** + +- `x` — The first cathetus of a right-angle triangle. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `y` — The second cathetus of a right-angle triangle. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- The length of the hypotenuse of a right-angle triangle. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT hypot(1, 1); +``` + +Result: + +``` text +┌────────hypot(1, 1)─┐ +│ 1.4142135623730951 │ +└────────────────────┘ +``` + +## log1p(x) {#log1px} + +Calculates `log(1+x)`. The [function](https://en.wikipedia.org/wiki/Natural_logarithm#lnp1) `log1p(x)` is more accurate than `log(1+x)` for small values of x. + +**Syntax** + +``` sql +log1p(x) +``` + +**Parameters** + +- `x` — Values from the interval: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Returned value** + +- Values from the interval: `-∞ < log1p(x) < +∞`. + +Type: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Example** + +Query: + +``` sql +SELECT log1p(0); +``` + +Result: + +``` text +┌─log1p(0)─┐ +│ 0 │ +└──────────┘ +``` + +**See Also** + +- [log(x)](../../sql-reference/functions/math-functions.md#logx-lnx) + [Original article](https://clickhouse.tech/docs/en/query_language/functions/math_functions/) diff --git a/docs/en/sql-reference/functions/url-functions.md b/docs/en/sql-reference/functions/url-functions.md index 0da74ce1b0e..006542f494a 100644 --- a/docs/en/sql-reference/functions/url-functions.md +++ b/docs/en/sql-reference/functions/url-functions.md @@ -131,6 +131,40 @@ For example: - `cutToFirstSignificantSubdomain('www.tr') = 'www.tr'`. - `cutToFirstSignificantSubdomain('tr') = ''`. +### cutToFirstSignificantSubdomainCustom {#cuttofirstsignificantsubdomaincustom} + +Same as `cutToFirstSignificantSubdomain` but accept custom TLD list name, useful if: + +- you need fresh TLD list, +- or you have custom. + +Configuration example: + +```xml + + + + public_suffix_list.dat + + +``` + +Example: + +- `cutToFirstSignificantSubdomain('https://news.yandex.com.tr/', 'public_suffix_list') = 'yandex.com.tr'`. + +### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} + +Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. + +### firstSignificantSubdomainCustom {#firstsignificantsubdomaincustom} + +Same as `firstSignificantSubdomain` but accept custom TLD list name. + +### cutToFirstSignificantSubdomainCustomWithWWW {#cuttofirstsignificantsubdomaincustomwithwww} + +Same as `cutToFirstSignificantSubdomainWithWWW` but accept custom TLD list name. + ### port(URL\[, default_port = 0\]) {#port} Returns the port or `default_port` if there is no port in the URL (or in case of validation error). diff --git a/docs/en/sql-reference/statements/create/quota.md b/docs/en/sql-reference/statements/create/quota.md index 29752050b69..20537b98a46 100644 --- a/docs/en/sql-reference/statements/create/quota.md +++ b/docs/en/sql-reference/statements/create/quota.md @@ -11,7 +11,7 @@ Syntax: ``` sql CREATE QUOTA [IF NOT EXISTS | OR REPLACE] name [ON CLUSTER cluster_name] - [KEYED BY {'none' | 'user name' | 'ip address' | 'client key' | 'client key or user name' | 'client key or ip address'}] + [KEYED BY {'none' | 'user name' | 'ip address' | 'forwarded ip address' | 'client key' | 'client key or user name' | 'client key or ip address'}] [FOR [RANDOMIZED] INTERVAL number {SECOND | MINUTE | HOUR | DAY | WEEK | MONTH | QUARTER | YEAR} {MAX { {QUERIES | ERRORS | RESULT ROWS | RESULT BYTES | READ ROWS | READ BYTES | EXECUTION TIME} = number } [,...] | NO LIMITS | TRACKING ONLY} [,...]] diff --git a/docs/en/sql-reference/statements/kill.md b/docs/en/sql-reference/statements/kill.md index d3f2d9bb5c6..6aa09cca4ef 100644 --- a/docs/en/sql-reference/statements/kill.md +++ b/docs/en/sql-reference/statements/kill.md @@ -53,7 +53,7 @@ KILL MUTATION [ON CLUSTER cluster] Tries to cancel and remove [mutations](../../sql-reference/statements/alter/index.md#alter-mutations) that are currently executing. Mutations to cancel are selected from the [`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) table using the filter specified by the `WHERE` clause of the `KILL` query. -A test query (`TEST`) only checks the user’s rights and displays a list of queries to stop. +A test query (`TEST`) only checks the user’s rights and displays a list of mutations to stop. Examples: diff --git a/docs/ru/interfaces/formats.md b/docs/ru/interfaces/formats.md index 042c62e310c..482e4999cea 100644 --- a/docs/ru/interfaces/formats.md +++ b/docs/ru/interfaces/formats.md @@ -9,6 +9,7 @@ ClickHouse может принимать (`INSERT`) и отдавать (`SELECT Поддерживаемые форматы и возможность использовать их в запросах `INSERT` и `SELECT` перечислены в таблице ниже. +======= | Формат | INSERT | SELECT | |-----------------------------------------------------------------------------------------|--------|--------| | [TabSeparated](#tabseparated) | ✔ | ✔ | @@ -56,6 +57,7 @@ ClickHouse может принимать (`INSERT`) и отдавать (`SELECT | [XML](#xml) | ✗ | ✔ | | [CapnProto](#capnproto) | ✔ | ✗ | | [LineAsString](#lineasstring) | ✔ | ✗ | +| [RawBLOB](#rawblob) | ✔ | ✔ | Вы можете регулировать некоторые параметры работы с форматами с помощью настроек ClickHouse. За дополнительной информацией обращайтесь к разделу [Настройки](../operations/settings/settings.md). @@ -434,7 +436,10 @@ JSON совместим с JavaScript. Для этого, дополнитель ClickHouse поддерживает [NULL](../sql-reference/syntax.md), который при выводе JSON будет отображен как `null`. Чтобы включить отображение в результате значений `+nan`, `-nan`, `+inf`, `-inf`, установите параметр [output_format_json_quote_denormals](../operations/settings/settings.md#settings-output_format_json_quote_denormals) равным 1. -Смотрите также формат [JSONEachRow](#jsoneachrow). +**Смотрите также** + +- Формат [JSONEachRow](#jsoneachrow) +- Настройка [output_format_json_array_of_rows](../operations/settings/settings.md#output-format-json-array-of-rows) ## JSONString {#jsonstring} @@ -1245,4 +1250,45 @@ SELECT * FROM line_as_string; └───────────────────────────────────────────────────┘ ``` +## RawBLOB {#rawblob} + +В этом формате все входные данные считываются в одно значение. Парсить можно только таблицу с одним полем типа [String](../sql-reference/data-types/string.md) или подобным ему. +Результат выводится в бинарном виде без разделителей и экранирования. При выводе более одного значения формат неоднозначен и будет невозможно прочитать данные снова. + +Ниже приведено сравнение форматов `RawBLOB` и [TabSeparatedRaw](#tabseparatedraw). +`RawBLOB`: +- данные выводятся в бинарном виде, без экранирования; +- нет разделителей между значениями; +- нет перевода строки в конце каждого значения. +[TabSeparatedRaw](#tabseparatedraw): +- данные выводятся без экранирования; +- строка содержит значения, разделённые табуляцией; +- после последнего значения в строке есть перевод строки. + +Далее рассмотрено сравнение форматов `RawBLOB` и [RowBinary](#rowbinary). +`RawBLOB`: +- строки выводятся без их длины в начале. +`RowBinary`: +- строки представлены как длина в формате varint (unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128)), а затем байты строки. + +При передаче на вход `RawBLOB` пустых данных, ClickHouse бросает исключение: + +``` text +Code: 108. DB::Exception: No data to insert +``` + +**Пример** + +``` bash +$ clickhouse-client --query "CREATE TABLE {some_table} (a String) ENGINE = Memory;" +$ cat {filename} | clickhouse-client --query="INSERT INTO {some_table} FORMAT RawBLOB" +$ clickhouse-client --query "SELECT * FROM {some_table} FORMAT RawBLOB" | md5sum +``` + +Результат: + +``` text +f9725a22f9191e064120d718e26862a9 - +``` + [Оригинальная статья](https://clickhouse.tech/docs/ru/interfaces/formats/) diff --git a/docs/ru/operations/settings/query-complexity.md b/docs/ru/operations/settings/query-complexity.md index a62e7523207..b0eac5d96e7 100644 --- a/docs/ru/operations/settings/query-complexity.md +++ b/docs/ru/operations/settings/query-complexity.md @@ -297,7 +297,7 @@ FORMAT Null; **Смотрите также** - [Секция JOIN](../../sql-reference/statements/select/join.md#select-join) -- [Движоy таблиц Join](../../engines/table-engines/special/join.md) +- [Движок таблиц Join](../../engines/table-engines/special/join.md) ## max_partitions_per_insert_block {#max-partitions-per-insert-block} diff --git a/docs/ru/operations/settings/settings.md b/docs/ru/operations/settings/settings.md index 9750f777990..0a8094231c2 100644 --- a/docs/ru/operations/settings/settings.md +++ b/docs/ru/operations/settings/settings.md @@ -2231,10 +2231,88 @@ SELECT CAST(toNullable(toInt32(0)) AS Int32) as x, toTypeName(x); ## output_format_tsv_null_representation {#output_format_tsv_null_representation} -Позволяет настраивать представление `NULL` для формата выходных данных [TSV](../../interfaces/formats.md#tabseparated). Настройка управляет форматом выходных данных, `\N` является единственным поддерживаемым представлением для формата входных данных TSV. +Определяет представление `NULL` для формата выходных данных [TSV](../../interfaces/formats.md#tabseparated). Пользователь может установить в качестве значения любую строку. Значение по умолчанию: `\N`. +**Примеры** + +Запрос + +```sql +SELECT * FROM tsv_custom_null FORMAT TSV; +``` + +Результат + +```text +788 +\N +\N +``` + +Запрос + +```sql +SET output_format_tsv_null_representation = 'My NULL'; +SELECT * FROM tsv_custom_null FORMAT TSV; +``` + +Результат + +```text +788 +My NULL +My NULL +``` + +## output_format_json_array_of_rows {#output-format-json-array-of-rows} + +Позволяет выводить все строки в виде массива JSON в формате [JSONEachRow](../../interfaces/formats.md#jsoneachrow). + +Возможные значения: + +- 1 — ClickHouse выводит все строки в виде массива и при этом каждую строку в формате `JSONEachRow`. +- 0 — ClickHouse выводит каждую строку отдельно в формате `JSONEachRow`. + +Значение по умолчанию: `0`. + +**Пример запроса с включенной настройкой** + +Запрос: + +```sql +SET output_format_json_array_of_rows = 1; +SELECT number FROM numbers(3) FORMAT JSONEachRow; +``` + +Результат: + +```text +[ +{"number":"0"}, +{"number":"1"}, +{"number":"2"} +] +``` + +**Пример запроса с отключенной настройкой** + +Запрос: + +```sql +SET output_format_json_array_of_rows = 0; +SELECT number FROM numbers(3) FORMAT JSONEachRow; +``` + +Результат: + +```text +{"number":"0"} +{"number":"1"} +{"number":"2"} +``` + ## allow_nullable_key {#allow-nullable-key} Включает или отключает поддержку типа [Nullable](../../sql-reference/data-types/nullable.md#data_type-nullable) для ключей таблиц [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md#table_engines-mergetree). diff --git a/docs/ru/sql-reference/functions/date-time-functions.md b/docs/ru/sql-reference/functions/date-time-functions.md index 3c9bd99de57..b7a077b3bd6 100644 --- a/docs/ru/sql-reference/functions/date-time-functions.md +++ b/docs/ru/sql-reference/functions/date-time-functions.md @@ -63,10 +63,18 @@ int32samoa: 1546300800 Переводит дату или дату-с-временем в число типа UInt16, содержащее номер года (AD). +## toQuarter {#toquarter} + +Переводит дату или дату-с-временем в число типа UInt8, содержащее номер квартала. + ## toMonth {#tomonth} Переводит дату или дату-с-временем в число типа UInt8, содержащее номер месяца (1-12). +## toDayOfYear {#todayofyear} + +Переводит дату или дату-с-временем в число типа UInt16, содержащее номер дня года (1-366). + ## toDayOfMonth {#todayofmonth} Переводит дату или дату-с-временем в число типа UInt8, содержащее номер дня в месяце (1-31). @@ -128,6 +136,22 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp Округляет дату или дату-с-временем вниз до первого дня года. Возвращается дата. +## toStartOfISOYear {#tostartofisoyear} + +Округляет дату или дату-с-временем вниз до первого дня ISO года. Возвращается дата. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года - это неделя с четырьмя или более днями в этом году. + +1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это 2016 ISO-год, который начался 2016-01-04. + +```sql +SELECT toStartOfISOYear(toDate('2017-01-01')) AS ISOYear20170101; +``` +```text +┌─ISOYear20170101─┐ +│ 2016-01-04 │ +└─────────────────┘ +``` + ## toStartOfQuarter {#tostartofquarter} Округляет дату или дату-с-временем вниз до первого дня квартала. @@ -147,6 +171,12 @@ SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp Округляет дату или дату-с-временем вниз до ближайшего понедельника. Возвращается дата. +## toStartOfWeek(t[,mode]) {#tostartofweek} + +Округляет дату или дату со временем до ближайшего воскресенья или понедельника в соответствии с mode. +Возвращается дата. +Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если аргумент mode опущен, то используется режим 0. + ## toStartOfDay {#tostartofday} Округляет дату-с-временем вниз до начала дня. Возвращается дата-с-временем. @@ -243,6 +273,10 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d Переводит дату-с-временем или дату в номер года, начиная с некоторого фиксированного момента в прошлом. +## toRelativeQuarterNum {#torelativequarternum} + +Переводит дату-с-временем или дату в номер квартала, начиная с некоторого фиксированного момента в прошлом. + ## toRelativeMonthNum {#torelativemonthnum} Переводит дату-с-временем или дату в номер месяца, начиная с некоторого фиксированного момента в прошлом. @@ -267,6 +301,102 @@ WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 SELECT toStartOfSecond(d Переводит дату-с-временем в номер секунды, начиная с некоторого фиксированного момента в прошлом. +## toISOYear {#toisoyear} + +Переводит дату-с-временем или дату в число типа UInt16, содержащее номер ISO года. ISO год отличается от обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) ISO год начинается необязательно первого января. + +Пример: + +```sql +SELECT + toDate('2017-01-01') AS date, + toYear(date), + toISOYear(date) +``` +```text +┌───────date─┬─toYear(toDate('2017-01-01'))─┬─toISOYear(toDate('2017-01-01'))─┐ +│ 2017-01-01 │ 2017 │ 2016 │ +└────────────┴──────────────────────────────┴─────────────────────────────────┘ +``` + +## toISOWeek {#toisoweek} + +Переводит дату-с-временем или дату в число типа UInt8, содержащее номер ISO недели. +Начало ISO года отличается от начала обычного года, потому что в соответствии с [ISO 8601:1988](https://en.wikipedia.org/wiki/ISO_8601) первая неделя года - это неделя с четырьмя или более днями в этом году. + +1 Января 2017 г. - воскресение, т.е. первая ISO неделя 2017 года началась в понедельник 2 января, поэтому 1 января 2017 это последняя неделя 2016 года. + +```sql +SELECT + toISOWeek(toDate('2017-01-01')) AS ISOWeek20170101, + toISOWeek(toDate('2017-01-02')) AS ISOWeek20170102 +``` + +```text +┌─ISOWeek20170101─┬─ISOWeek20170102─┐ +│ 52 │ 1 │ +└─────────────────┴─────────────────┘ +``` + +## toWeek(date\[, mode\]\[, timezone\]) {#toweek} +Переводит дату-с-временем или дату в число UInt8, содержащее номер недели. Второй аргументам mode задает режим, начинается ли неделя с воскресенья или с понедельника и должно ли возвращаемое значение находиться в диапазоне от 0 до 53 или от 1 до 53. Если аргумент mode опущен, то используется режим 0. + +`toISOWeek() ` эквивалентно `toWeek(date,3)`. + +Описание режимов (mode): + +| Mode | Первый день недели | Диапазон | Неделя 1 это первая неделя … | +| ----------- | -------- | -------- | ------------------ | +|0|Воскресенье|0-53|с воскресеньем в этом году +|1|Понедельник|0-53|с 4-мя или более днями в этом году +|2|Воскресенье|1-53|с воскресеньем в этом году +|3|Понедельник|1-53|с 4-мя или более днями в этом году +|4|Воскресенье|0-53|с 4-мя или более днями в этом году +|5|Понедельник|0-53|с понедельником в этом году +|6|Воскресенье|1-53|с 4-мя или более днями в этом году +|7|Понедельник|1-53|с понедельником в этом году +|8|Воскресенье|1-53|содержащая 1 января +|9|Понедельник|1-53|содержащая 1 января + +Для режимов со значением «с 4 или более днями в этом году» недели нумеруются в соответствии с ISO 8601:1988: + +- Если неделя, содержащая 1 января, имеет 4 или более дней в новом году, это неделя 1. + +- В противном случае это последняя неделя предыдущего года, а следующая неделя - неделя 1. + +Для режимов со значением «содержит 1 января», неделя 1 – это неделя содержащая 1 января. Не имеет значения, сколько дней в новом году содержала неделя, даже если она содержала только один день. + +**Пример** + +```sql +SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS week1, toWeek(date,9) AS week9; +``` + +```text +┌───────date─┬─week0─┬─week1─┬─week9─┐ +│ 2016-12-27 │ 52 │ 52 │ 1 │ +└────────────┴───────┴───────┴───────┘ +``` + +## toYearWeek(date[,mode]) {#toyearweek} +Возвращает год и неделю для даты. Год в результате может отличаться от года в аргументе даты для первой и последней недели года. + +Аргумент mode работает точно так же, как аргумент mode [toWeek()](#toweek). Если mode не задан, используется режим 0. + +`toISOYear() ` эквивалентно `intDiv(toYearWeek(date,3),100)`. + +**Пример** + +```sql +SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(date,1) AS yearWeek1, toYearWeek(date,9) AS yearWeek9; +``` + +```text +┌───────date─┬─yearWeek0─┬─yearWeek1─┬─yearWeek9─┐ +│ 2016-12-27 │ 201652 │ 201652 │ 201701 │ +└────────────┴───────────┴───────────┴───────────┘ +``` + ## date_trunc {#date_trunc} Отсекает от даты и времени части, меньшие чем указанная часть. diff --git a/docs/ru/sql-reference/functions/math-functions.md b/docs/ru/sql-reference/functions/math-functions.md index 6df366d129f..2e57aca6a0a 100644 --- a/docs/ru/sql-reference/functions/math-functions.md +++ b/docs/ru/sql-reference/functions/math-functions.md @@ -103,4 +103,306 @@ SELECT erf(3 / sqrt(2)) Принимает два числовых аргумента x и y. Возвращает число типа Float64, близкое к x в степени y. +## cosh(x) {#coshx} + +[Гиперболический косинус](https://help.scilab.org/docs/5.4.0/ru_RU/cosh.html). + +**Синтаксис** + +``` sql +cosh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `1 <= cosh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT cosh(0); +``` + +Результат: + +``` text +┌─cosh(0)──┐ +│ 1 │ +└──────────┘ +``` + +## acosh(x) {#acoshx} + +[Обратный гиперболический косинус](https://help.scilab.org/docs/5.4.0/ru_RU/acosh.html). + +**Синтаксис** + +``` sql +acosh(x) +``` + +**Параметры** + +- `x` — гиперболический косинус угла. Значения из интервала: `1 <= x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Угол в радианах. Значения из интервала: `0 <= acosh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT acosh(1); +``` + +Результат: + +``` text +┌─acosh(1)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [cosh(x)](../../sql-reference/functions/math-functions.md#coshx) + +## sinh(x) {#sinhx} + +[Гиперболический синус](https://help.scilab.org/docs/5.4.0/ru_RU/sinh.html). + +**Синтаксис** + +``` sql +sinh(x) +``` + +**Параметры** + +- `x` — угол в радианах. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < sinh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT sinh(0); +``` + +Результат: + +``` text +┌─sinh(0)──┐ +│ 0 │ +└──────────┘ +``` + +## asinh(x) {#asinhx} + +[Обратный гиперболический синус](https://help.scilab.org/docs/5.4.0/ru_RU/asinh.html). + +**Синтаксис** + +``` sql +asinh(x) +``` + +**Параметры** + +- `x` — гиперболический синус угла. Значения из интервала: `-∞ < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Угол в радианах. Значения из интервала: `-∞ < asinh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT asinh(0); +``` + +Результат: + +``` text +┌─asinh(0)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [sinh(x)](../../sql-reference/functions/math-functions.md#sinhx) + +## atanh(x) {#atanhx} + +[Обратный гиперболический тангенс](https://help.scilab.org/docs/5.4.0/ru_RU/atanh.html). + +**Синтаксис** + +``` sql +atanh(x) +``` + +**Параметры** + +- `x` — гиперболический тангенс угла. Значения из интервала: `–1 < x < 1`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Угол в радианах. Значения из интервала: `-∞ < atanh(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT atanh(0); +``` + +Результат: + +``` text +┌─atanh(0)─┐ +│ 0 │ +└──────────┘ +``` + +## atan2(y, x) {#atan2yx} + +[Функция](https://msoffice-prowork.com/ref/excel/excelfunc/math/atan2/) вычисляет угол в радианах между положительной осью x и линией, проведенной из начала координат в точку `(x, y) ≠ (0, 0)`. + +**Синтаксис** + +``` sql +atan2(y, x) +``` + +**Параметры** + +- `y` — координата y точки, в которую проведена линия. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `x` — координата х точки, в которую проведена линия. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Угол `θ` в радианах из интервала: `−π < θ ≤ π`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT atan2(1, 1); +``` + +Результат: + +``` text +┌────────atan2(1, 1)─┐ +│ 0.7853981633974483 │ +└────────────────────┘ +``` + +## hypot(x, y) {#hypotxy} + +Вычисляет длину гипотенузы прямоугольного треугольника. При использовании этой [функции](https://php.ru/manual/function.hypot.html) не возникает проблем при возведении в квадрат очень больших или очень малых чисел. + +**Синтаксис** + +``` sql +hypot(x, y) +``` + +**Параметры** + +- `x` — первый катет прямоугольного треугольника. [Float64](../../sql-reference/data-types/float.md#float32-float64). +- `y` — второй катет прямоугольного треугольника. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Длина гипотенузы прямоугольного треугольника. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT hypot(1, 1); +``` + +Результат: + +``` text +┌────────hypot(1, 1)─┐ +│ 1.4142135623730951 │ +└────────────────────┘ +``` + +## log1p(x) {#log1px} + +Вычисляет `log(1+x)`. [Функция](https://help.scilab.org/docs/6.0.1/ru_RU/log1p.html) `log1p(x)` является более точной, чем функция `log(1+x)` для малых значений x. + +**Синтаксис** + +``` sql +log1p(x) +``` + +**Параметры** + +- `x` — значения из интервала: `-1 < x < +∞`. [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Возвращаемое значение** + +- Значения из интервала: `-∞ < log1p(x) < +∞`. + +Тип: [Float64](../../sql-reference/data-types/float.md#float32-float64). + +**Пример** + +Запрос: + +``` sql +SELECT log1p(0); +``` + +Результат: + +``` text +┌─log1p(0)─┐ +│ 0 │ +└──────────┘ +``` + +**Смотрите также** + +- [log(x)](../../sql-reference/functions/math-functions.md#logx) + [Оригинальная статья](https://clickhouse.tech/docs/ru/query_language/functions/math_functions/) diff --git a/docs/ru/sql-reference/statements/system.md b/docs/ru/sql-reference/statements/system.md index 4f7ac98807d..a6a6c5047af 100644 --- a/docs/ru/sql-reference/statements/system.md +++ b/docs/ru/sql-reference/statements/system.md @@ -12,6 +12,7 @@ toc_title: SYSTEM - [DROP MARK CACHE](#query_language-system-drop-mark-cache) - [DROP UNCOMPRESSED CACHE](#query_language-system-drop-uncompressed-cache) - [DROP COMPILED EXPRESSION CACHE](#query_language-system-drop-compiled-expression-cache) +- [DROP REPLICA](#query_language-system-drop-replica) - [FLUSH LOGS](#query_language-system-flush_logs) - [RELOAD CONFIG](#query_language-system-reload-config) - [SHUTDOWN](#query_language-system-shutdown) @@ -66,6 +67,24 @@ SELECT name, status FROM system.dictionaries; Сбрасывает кеш «засечек» (`mark cache`). Используется при разработке ClickHouse и тестах производительности. +## DROP REPLICA {#query_language-system-drop-replica} + +Мертвые реплики можно удалить, используя следующий синтаксис: + +``` sql +SYSTEM DROP REPLICA 'replica_name' FROM TABLE database.table; +SYSTEM DROP REPLICA 'replica_name' FROM DATABASE database; +SYSTEM DROP REPLICA 'replica_name'; +SYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/path/to/table/in/zk'; +``` + +Удаляет путь реплики из ZooKeeper-а. Это полезно, когда реплика мертва и ее метаданные не могут быть удалены из ZooKeeper с помощью `DROP TABLE`, потому что такой таблицы больше нет. `DROP REPLICA` может удалить только неактивную / устаревшую реплику и не может удалить локальную реплику, используйте для этого `DROP TABLE`. `DROP REPLICA` не удаляет таблицы и не удаляет данные или метаданные с диска. + +Первая команда удаляет метаданные реплики `'replica_name'` для таблицы `database.table`. +Вторая команда удаляет метаданные реплики `'replica_name'` для всех таблиц базы данных `database`. +Третья команда удаляет метаданные реплики `'replica_name'` для всех таблиц, существующих на локальном сервере (список таблиц генерируется из локальной реплики). +Четверая команда полезна для удаления метаданных мертвой реплики когда все другие реплики таблицы уже были удалены ранее, поэтому необходимо явно указать ZooKeeper путь таблицы. ZooKeeper путь это первый аргумент для `ReplicatedMergeTree` движка при создании таблицы. + ## DROP UNCOMPRESSED CACHE {#query_language-system-drop-uncompressed-cache} Сбрасывает кеш не сжатых данных. Используется при разработке ClickHouse и тестах производительности. diff --git a/docs/zh/index.md b/docs/zh/index.md index 2bef22f3de4..fb473020f22 100644 --- a/docs/zh/index.md +++ b/docs/zh/index.md @@ -46,7 +46,7 @@ ClickHouse是一个用于联机分析(OLAP)的列式数据库管理系统(DBMS) - 处理单个查询时需要高吞吐量(每台服务器每秒可达数十亿行) - 事务不是必须的 - 对数据一致性要求低 -- 每个查询有一个大表。除了他意以外,其他的都很小。 +- 每个查询有一个大表。除了他以外,其他的都很小。 - 查询结果明显小于源数据。换句话说,数据经过过滤或聚合,因此结果适合于单个服务器的RAM中 很容易可以看出,OLAP场景与其他通常业务场景(例如,OLTP或K/V)有很大的不同, 因此想要使用OLTP或Key-Value数据库去高效的处理分析查询场景,并不是非常完美的适用方案。例如,使用OLAP数据库去处理分析请求通常要优于使用MongoDB或Redis去处理分析请求。 diff --git a/docs/zh/interfaces/mysql.md b/docs/zh/interfaces/mysql.md index dc442b4d056..fa9fc0053a6 100644 --- a/docs/zh/interfaces/mysql.md +++ b/docs/zh/interfaces/mysql.md @@ -5,13 +5,13 @@ toc_title: MySQL接口 # MySQL接口 {#mysql-interface} -ClickHouse支持MySQL协议。它可以通过[mysql_port](../operations/server-configuration-parameters/settings.md#server_configuration_parameters-mysql_port)在配置文件中设置: +ClickHouse支持MySQL wire通讯协议。可以通过在配置文件中设置 [mysql_port](../operations/server-configuration-parameters/settings.md#server_configuration_parameters-mysql_port) 来启用它: ``` xml 9004 ``` -使用命令行工具连接`mysql`示例: +使用命令行工具 `mysql` 进行连接的示例: ``` bash $ mysql --protocol tcp -u default -P 9004 @@ -35,12 +35,12 @@ Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> ``` -为了与所有MySQL客户端兼容,建议在配置文件中使用[双SHA1](../operations/settings/settings-users.md#password_double_sha1_hex)指定用户密码。 -如果使用[SHA256](../operations/settings/settings-users.md#password_sha256_hex)指定用户密码,一些客户端将无法进行身份验证(mysqljs和旧版本的命令行工具mysql)。 +为了与所有MySQL客户端兼容,建议在配置文件中使用 [double SHA1](../operations/settings/settings-users.md#password_double_sha1_hex) 来指定用户密码。 +如果使用 [SHA256](../operations/settings/settings-users.md#password_sha256_hex) 指定用户密码,一些客户端将无法进行身份验证(比如mysqljs和旧版本的命令行工具mysql)。 限制: -- 不支持准备好的查询 +- 不支持prepared queries - 某些数据类型以字符串形式发送 diff --git a/docs/zh/sql-reference/functions/date-time-functions.md b/docs/zh/sql-reference/functions/date-time-functions.md index 65d331a7846..00dab5ee680 100644 --- a/docs/zh/sql-reference/functions/date-time-functions.md +++ b/docs/zh/sql-reference/functions/date-time-functions.md @@ -20,7 +20,37 @@ SELECT ## toTimeZone {#totimezone} -将Date或DateTime转换为指定的时区。 +将Date或DateTime转换为指定的时区。 时区是Date/DateTime类型的属性。 表字段或结果集的列的内部值(秒数)不会更改,列的类型会更改,并且其字符串表示形式也会相应更改。 + +```sql +SELECT + toDateTime('2019-01-01 00:00:00', 'UTC') AS time_utc, + toTypeName(time_utc) AS type_utc, + toInt32(time_utc) AS int32utc, + toTimeZone(time_utc, 'Asia/Yekaterinburg') AS time_yekat, + toTypeName(time_yekat) AS type_yekat, + toInt32(time_yekat) AS int32yekat, + toTimeZone(time_utc, 'US/Samoa') AS time_samoa, + toTypeName(time_samoa) AS type_samoa, + toInt32(time_samoa) AS int32samoa +FORMAT Vertical; +``` + +```text +Row 1: +────── +time_utc: 2019-01-01 00:00:00 +type_utc: DateTime('UTC') +int32utc: 1546300800 +time_yekat: 2019-01-01 05:00:00 +type_yekat: DateTime('Asia/Yekaterinburg') +int32yekat: 1546300800 +time_samoa: 2018-12-31 13:00:00 +type_samoa: DateTime('US/Samoa') +int32samoa: 1546300800 +``` + +`toTimeZone(time_utc, 'Asia/Yekaterinburg')` 把 `DateTime('UTC')` 类型转换为 `DateTime('Asia/Yekaterinburg')`. 内部值 (Unixtimestamp) 1546300800 保持不变, 但是字符串表示(toString() 函数的结果值) 由 `time_utc: 2019-01-01 00:00:00` 转换为o `time_yekat: 2019-01-01 05:00:00`. ## toYear {#toyear} @@ -34,15 +64,15 @@ SELECT 将Date或DateTime转换为包含月份编号(1-12)的UInt8类型的数字。 -## 今天一年 {#todayofyear} +## toDayOfYear {#todayofyear} 将Date或DateTime转换为包含一年中的某一天的编号的UInt16(1-366)类型的数字。 -## 今天月 {#todayofmonth} +## toDayOfMonth {#todayofmonth} 将Date或DateTime转换为包含一月中的某一天的编号的UInt8(1-31)类型的数字。 -## 今天一周 {#todayofweek} +## toDayOfWeek {#todayofweek} 将Date或DateTime转换为包含一周中的某一天的编号的UInt8(周一是1, 周日是7)类型的数字。 @@ -55,31 +85,61 @@ SELECT 将DateTime转换为包含一小时中分钟数(0-59)的UInt8数字。 -## 秒 {#tosecond} +## toSecond {#tosecond} 将DateTime转换为包含一分钟中秒数(0-59)的UInt8数字。 闰秒不计算在内。 -## toUnixTimestamp {#tounixtimestamp} +## toUnixTimestamp {#to-unix-timestamp} -将DateTime转换为unix时间戳。 +对于DateTime参数:将值转换为UInt32类型的数字-Unix时间戳(https://en.wikipedia.org/wiki/Unix_time)。 +对于String参数:根据时区将输入字符串转换为日期时间(可选的第二个参数,默认使用服务器时区),并返回相应的unix时间戳。 -## 开始一年 {#tostartofyear} +**语法** + +``` sql +toUnixTimestamp(datetime) +toUnixTimestamp(str, [timezone]) +``` + +**返回值** + +- 返回 unix timestamp. + +类型: `UInt32`. + +**示例** + +查询: + +``` sql +SELECT toUnixTimestamp('2017-11-05 08:07:47', 'Asia/Tokyo') AS unix_timestamp +``` + +结果: + +``` text +┌─unix_timestamp─┐ +│ 1509836867 │ +└────────────────┘ +``` + +## toStartOfYear {#tostartofyear} 将Date或DateTime向前取整到本年的第一天。 返回Date类型。 -## 今年开始 {#tostartofisoyear} +## toStartOfISOYear {#tostartofisoyear} 将Date或DateTime向前取整到ISO本年的第一天。 返回Date类型。 -## 四分之一开始 {#tostartofquarter} +## toStartOfQuarter {#tostartofquarter} 将Date或DateTime向前取整到本季度的第一天。 返回Date类型。 -## 到月份开始 {#tostartofmonth} +## toStartOfMonth {#tostartofmonth} 将Date或DateTime向前取整到本月的第一天。 返回Date类型。 @@ -92,27 +152,90 @@ SELECT 将Date或DateTime向前取整到本周的星期一。 返回Date类型。 -## 今天开始 {#tostartofday} +## toStartOfWeek(t\[,mode\]) {#tostartofweek} -将DateTime向前取整到当日的开始。 +按mode将Date或DateTime向前取整到最近的星期日或星期一。 +返回Date类型。 +mode参数的工作方式与toWeek()的mode参数完全相同。 对于单参数语法,mode使用默认值0。 -## 开始一小时 {#tostartofhour} +## toStartOfDay {#tostartofday} + +将DateTime向前取整到今天的开始。 + +## toStartOfHour {#tostartofhour} 将DateTime向前取整到当前小时的开始。 -## to startofminute {#tostartofminute} +## toStartOfMinute {#tostartofminute} 将DateTime向前取整到当前分钟的开始。 -## to startoffiveminute {#tostartoffiveminute} +## toStartOfSecond {#tostartofsecond} + +将DateTime向前取整到当前秒数的开始。 + +**语法** + +``` sql +toStartOfSecond(value[, timezone]) +``` + +**参数** + +- `value` — 时间和日期[DateTime64](../../sql-reference/data-types/datetime64.md). +- `timezone` — 返回值的[Timezone](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) (可选参数)。 如果未指定将使用 `value` 参数的时区。 [String](../../sql-reference/data-types/string.md)。 + +**返回值** + +- 输入值毫秒部分为零。 + +类型: [DateTime64](../../sql-reference/data-types/datetime64.md). + +**示例** + +不指定时区查询: + +``` sql +WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 +SELECT toStartOfSecond(dt64); +``` + +结果: + +``` text +┌───toStartOfSecond(dt64)─┐ +│ 2020-01-01 10:20:30.000 │ +└─────────────────────────┘ +``` + +指定时区查询: + +``` sql +WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64 +SELECT toStartOfSecond(dt64, 'Europe/Moscow'); +``` + +结果: + +``` text +┌─toStartOfSecond(dt64, 'Europe/Moscow')─┐ +│ 2020-01-01 13:20:30.000 │ +└────────────────────────────────────────┘ +``` + +**参考** + +- [Timezone](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 服务器配置选项。 + +## toStartOfFiveMinute {#tostartoffiveminute} 将DateTime以五分钟为单位向前取整到最接近的时间点。 -## 开始分钟 {#tostartoftenminutes} +## toStartOfTenMinutes {#tostartoftenminutes} 将DateTime以十分钟为单位向前取整到最接近的时间点。 -## 开始几分钟 {#tostartoffifteenminutes} +## toStartOfFifteenMinutes {#tostartoffifteenminutes} 将DateTime以十五分钟为单位向前取整到最接近的时间点。 @@ -168,31 +291,214 @@ SELECT 将Date或DateTime转换为包含ISO周数的UInt8类型的编号。 -## 现在 {#now} +## toWeek(date\[,mode\]) {#toweekdatemode} -不接受任何参数并在请求执行时的某一刻返回当前时间(DateTime)。 -此函数返回一个常量,即时请求需要很长时间能够完成。 +返回Date或DateTime的周数。两个参数形式可以指定星期是从星期日还是星期一开始,以及返回值应在0到53还是从1到53的范围内。如果省略了mode参数,则默认 模式为0。 +`toISOWeek()`是一个兼容函数,等效于`toWeek(date,3)`。 +下表描述了mode参数的工作方式。 -## 今天 {#today} +| Mode | First day of week | Range | Week 1 is the first week … | +|------|-------------------|-------|-------------------------------| +| 0 | Sunday | 0-53 | with a Sunday in this year | +| 1 | Monday | 0-53 | with 4 or more days this year | +| 2 | Sunday | 1-53 | with a Sunday in this year | +| 3 | Monday | 1-53 | with 4 or more days this year | +| 4 | Sunday | 0-53 | with 4 or more days this year | +| 5 | Monday | 0-53 | with a Monday in this year | +| 6 | Sunday | 1-53 | with 4 or more days this year | +| 7 | Monday | 1-53 | with a Monday in this year | +| 8 | Sunday | 1-53 | contains January 1 | +| 9 | Monday | 1-53 | contains January 1 | + +对于象“with 4 or more days this year,”的mode值,根据ISO 8601:1988对周进行编号: + +- 如果包含1月1日的一周在后一年度中有4天或更多天,则为第1周。 + +- 否则,它是上一年的最后一周,下周是第1周。 + +对于像“contains January 1”的mode值, 包含1月1日的那周为本年度的第1周。 + +``` sql +toWeek(date, [, mode][, Timezone]) +``` + +**参数** + +- `date` – Date 或 DateTime. +- `mode` – 可选参数, 取值范围 \[0,9\], 默认0。 +- `Timezone` – 可选参数, 可其他时间日期转换参数的行为一致。 + +**示例** + +``` sql +SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS week1, toWeek(date,9) AS week9; +``` + +``` text +┌───────date─┬─week0─┬─week1─┬─week9─┐ +│ 2016-12-27 │ 52 │ 52 │ 1 │ +└────────────┴───────┴───────┴───────┘ +``` + +## toYearWeek(date\[,mode\]) {#toyearweekdatemode} + +返回Date的年和周。 结果中的年份可能因为Date为该年份的第一周和最后一周而于Date的年份不同。 + +mode参数的工作方式与toWeek()的mode参数完全相同。 对于单参数语法,mode使用默认值0。 + +`toISOYear()`是一个兼容函数,等效于`intDiv(toYearWeek(date,3),100)`. + +**示例** + +``` sql +SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(date,1) AS yearWeek1, toYearWeek(date,9) AS yearWeek9; +``` + +``` text +┌───────date─┬─yearWeek0─┬─yearWeek1─┬─yearWeek9─┐ +│ 2016-12-27 │ 201652 │ 201652 │ 201701 │ +└────────────┴───────────┴───────────┴───────────┘ +``` + +## date_trunc {#date_trunc} + +将Date或DateTime按指定的单位向前取整到最接近的时间点。 + +**语法** + +``` sql +date_trunc(unit, value[, timezone]) +``` + +别名: `dateTrunc`. + +**参数** + +- `unit` — 单位. [String](../syntax.md#syntax-string-literal). + 可选值: + + - `second` + - `minute` + - `hour` + - `day` + - `week` + - `month` + - `quarter` + - `year` + +- `value` — [DateTime](../../sql-reference/data-types/datetime.md) 或者 [DateTime64](../../sql-reference/data-types/datetime64.md). +- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 返回值的时区(可选值)。如果未指定将使用`value`的时区。 [String](../../sql-reference/data-types/string.md). + +**返回值** + +- 按指定的单位向前取整后的DateTime。 + +类型: [Datetime](../../sql-reference/data-types/datetime.md). + +**示例** + +不指定时区查询: + +``` sql +SELECT now(), date_trunc('hour', now()); +``` + +结果: + +``` text +┌───────────────now()─┬─date_trunc('hour', now())─┐ +│ 2020-09-28 10:40:45 │ 2020-09-28 10:00:00 │ +└─────────────────────┴───────────────────────────┘ +``` + +指定时区查询: + +```sql +SELECT now(), date_trunc('hour', now(), 'Europe/Moscow'); +``` + +结果: + +```text +┌───────────────now()─┬─date_trunc('hour', now(), 'Europe/Moscow')─┐ +│ 2020-09-28 10:46:26 │ 2020-09-28 13:00:00 │ +└─────────────────────┴────────────────────────────────────────────┘ +``` + +**参考** + +- [toStartOfInterval](#tostartofintervaltime-or-data-interval-x-unit-time-zone) + +# now {#now} + +返回当前日期和时间。 + +**语法** + +``` sql +now([timezone]) +``` + +**参数** + +- `timezone` — [Timezone name](../../operations/server-configuration-parameters/settings.md#server_configuration_parameters-timezone) 返回结果的时区(可先参数). [String](../../sql-reference/data-types/string.md). + +**返回值** + +- 当前日期和时间。 + +类型: [Datetime](../../sql-reference/data-types/datetime.md). + +**示例** + +不指定时区查询: + +``` sql +SELECT now(); +``` + +结果: + +``` text +┌───────────────now()─┐ +│ 2020-10-17 07:42:09 │ +└─────────────────────┘ +``` + +指定时区查询: + +``` sql +SELECT now('Europe/Moscow'); +``` + +结果: + +``` text +┌─now('Europe/Moscow')─┐ +│ 2020-10-17 10:42:23 │ +└──────────────────────┘ +``` + +## today {#today} 不接受任何参数并在请求执行时的某一刻返回当前日期(Date)。 -其功能与’toDate(now())’相同。 +其功能与’toDate(now())’相同。 -## 昨天 {#yesterday} +## yesterday {#yesterday} 不接受任何参数并在请求执行时的某一刻返回昨天的日期(Date)。 -其功能与’today() - 1’相同。 +其功能与’today() - 1’相同。 -## 时隙 {#timeslot} +## timeSlot {#timeslot} 将时间向前取整半小时。 此功能用于Yandex.Metrica,因为如果跟踪标记显示单个用户的连续综合浏览量在时间上严格超过此数量,则半小时是将会话分成两个会话的最短时间。这意味着(tag id,user id,time slot)可用于搜索相应会话中包含的综合浏览量。 -## toyyymm {#toyyyymm} +## toYYYMM {#toyyyymm} 将Date或DateTime转换为包含年份和月份编号的UInt32类型的数字(YYYY \* 100 + MM)。 -## toyyymmdd {#toyyyymmdd} +## toYYYMMDD {#toyyyymmdd} 将Date或DateTime转换为包含年份和月份编号的UInt32类型的数字(YYYY \* 10000 + MM \* 100 + DD)。 @@ -200,7 +506,7 @@ SELECT 将Date或DateTime转换为包含年份和月份编号的UInt64类型的数字(YYYY \* 10000000000 + MM \* 100000000 + DD \* 1000000 + hh \* 10000 + mm \* 100 + ss)。 -## 隆隆隆隆路虏脢,,陇,貌,垄拢卢虏禄quar陇,貌路,隆拢脳枚脢虏,麓脢,脱,,,录,禄庐戮,utes, {#addyears-addmonths-addweeks-adddays-addhours-addminutes-addseconds-addquarters} +## addYears, addMonths, addWeeks, addDays, addHours, addMinutes, addSeconds, addQuarters {#addyears-addmonths-addweeks-adddays-addhours-addminutes-addseconds-addquarters} 函数将一段时间间隔添加到Date/DateTime,然后返回Date/DateTime。例如: @@ -234,59 +540,145 @@ SELECT │ 2018-01-01 │ 2018-01-01 00:00:00 │ └──────────────────────────┴───────────────────────────────┘ -## dateDiff(‘unit’,t1,t2,\[时区\]) {#datediffunit-t1-t2-timezone} +## dateDiff {#datediff} -返回以’unit’为单位表示的两个时间之间的差异,例如`'hours'`。 ‘t1’和’t2’可以是Date或DateTime,如果指定’timezone’,它将应用于两个参数。如果不是,则使用来自数据类型’t1’和’t2’的时区。如果时区不相同,则结果将是未定义的。 +返回两个Date或DateTime类型之间的时差。 -支持的单位值: +**语法** -| 单位 | -|------| -| 第二 | -| 分钟 | -| 小时 | -| 日 | -| 周 | -| 月 | -| 季 | -| 年 | +``` sql +dateDiff('unit', startdate, enddate, [timezone]) +``` -## 时隙(开始时间,持续时间,\[,大小\]) {#timeslotsstarttime-duration-size} +**参数** + +- `unit` — 返回结果的时间单位。 [String](../../sql-reference/syntax.md#syntax-string-literal). + + 支持的时间单位: second, minute, hour, day, week, month, quarter, year. + +- `startdate` — 第一个待比较值。 [Date](../../sql-reference/data-types/date.md) 或 [DateTime](../../sql-reference/data-types/datetime.md). + +- `enddate` — 第二个待比较值。 [Date](../../sql-reference/data-types/date.md) 或 [DateTime](../../sql-reference/data-types/datetime.md). + +- `timezone` — 可选参数。 如果指定了,则同时适用于`startdate`和`enddate`。如果不指定,则使用`startdate`和`enddate`的时区。如果两个时区不一致,则结果不可预料。 + +**返回值** + +以`unit`为单位的`startdate`和`enddate`之间的时差。 + +类型: `int`. + +**示例** + +查询: + +``` sql +SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00')); +``` + +结果: + +``` text +┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐ +│ 25 │ +└────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## timeSlots(StartTime, Duration,\[, Size\]) {#timeslotsstarttime-duration-size} 它返回一个时间数组,其中包括从从«StartTime»开始到«StartTime + Duration 秒»内的所有符合«size»(以秒为单位)步长的时间点。其中«size»是一个可选参数,默认为1800。 例如,`timeSlots(toDateTime('2012-01-01 12:20:00'),600) = [toDateTime('2012-01-01 12:00:00'),toDateTime('2012-01-01 12:30:00' )]`。 这对于搜索在相应会话中综合浏览量是非常有用的。 -## formatDateTime(时间,格式\[,时区\]) {#formatdatetimetime-format-timezone} +## formatDateTime {#formatdatetime} 函数根据给定的格式字符串来格式化时间。请注意:格式字符串必须是常量表达式,例如:单个结果列不能有多种格式字符串。 -支持的格式修饰符: -(«Example» 列是对`2018-01-02 22:33:44`的格式化结果) +**语法** -| 修饰符 | 产品描述 | 示例 | -|--------|-------------------------------------------|------------| -| %C | 年除以100并截断为整数(00-99) | 20 | -| %d | 月中的一天,零填充(01-31) | 02 | -| %D | 短MM/DD/YY日期,相当于%m/%d/%y | 01/02/2018 | -| %e | 月中的一天,空格填充(1-31) | 2 | -| %F | 短YYYY-MM-DD日期,相当于%Y-%m-%d | 2018-01-02 | -| %H | 24小时格式(00-23) | 22 | -| %I | 小时12h格式(01-12) | 10 | -| %j | 一年(001-366) | 002 | -| %m | 月份为十进制数(01-12) | 01 | -| %M | 分钟(00-59) | 33 | -| %n | 换行符(") | | -| %p | AM或PM指定 | PM | -| %R | 24小时HH:MM时间,相当于%H:%M | 22:33 | -| %S | 第二(00-59) | 44 | -| %t | 水平制表符(’) | | -| %T | ISO8601时间格式(HH:MM:SS),相当于%H:%M:%S | 22:33:44 | -| %u | ISO8601平日as编号,星期一为1(1-7) | 2 | -| %V | ISO8601周编号(01-53) | 01 | -| %w | 周日为十进制数,周日为0(0-6) | 2 | -| %y | 年份,最后两位数字(00-99) | 18 | -| %Y | 年 | 2018 | -| %% | %符号 | % | +``` sql +formatDateTime(Time, Format\[, Timezone\]) +``` -[来源文章](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) +**返回值** + +根据指定格式返回的日期和时间。 + +**支持的格式修饰符** + +使用格式修饰符来指定结果字符串的样式。«Example» 列是对`2018-01-02 22:33:44`的格式化结果。 + +| 修饰符 | 描述 | 示例 | +|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| +| %C | 年除以100并截断为整数(00-99) | 20 | +| %d | 月中的一天,零填充(01-31) | 02 | +| %D | 短MM/DD/YY日期,相当于%m/%d/%y | 01/02/2018 | +| %e | 月中的一天,空格填充(1-31) | 2 | +| %F | 短YYYY-MM-DD日期,相当于%Y-%m-%d | 2018-01-02 | +| %G | ISO周号的四位数年份格式, 从基于周的年份[由ISO 8601定义](https://en.wikipedia.org/wiki/ISO_8601#Week_dates) 标准计算得出,通常仅对%V有用 | 2018 | +| %g | 两位数的年份格式,与ISO 8601一致,四位数表示法的缩写 | 18 | +| %H | 24小时格式(00-23) | 22 | +| %I | 小时12h格式(01-12) | 10 | +| %j | 一年(001-366) | 002 | +| %m | 月份为十进制数(01-12) | 01 | +| %M | 分钟(00-59) | 33 | +| %n | 换行符(") | | +| %p | AM或PM指定 | PM | +| %R | 24小时HH:MM时间,相当于%H:%M | 22:33 | +| %S | 第二(00-59) | 44 | +| %t | 水平制表符(’) | | +| %T | ISO8601时间格式(HH:MM:SS),相当于%H:%M:%S | 22:33:44 | +| %u | ISO8601平日as编号,星期一为1(1-7) | 2 | +| %V | ISO8601周编号(01-53) | 01 | +| %w | 周日为十进制数,周日为0(0-6) | 2 | +| %y | 年份,最后两位数字(00-99) | 18 | +| %Y | 年 | 2018 | +| %% | %符号 | % | + +**示例** + +查询: + +``` sql +SELECT formatDateTime(toDate('2010-01-04'), '%g') +``` + +结果: + +``` +┌─formatDateTime(toDate('2010-01-04'), '%g')─┐ +│ 10 │ +└────────────────────────────────────────────┘ +``` + +[Original article](https://clickhouse.tech/docs/en/query_language/functions/date_time_functions/) + +## FROM_UNIXTIME + +当只有单个整数类型的参数时,它的作用与`toDateTime`相同,并返回[DateTime](../../sql-reference/data-types/datetime.md)类型。 + +例如: + +```sql +SELECT FROM_UNIXTIME(423543535) +``` + +```text +┌─FROM_UNIXTIME(423543535)─┐ +│ 1983-06-04 10:58:55 │ +└──────────────────────────┘ +``` + +当有两个参数时,第一个是整型或DateTime,第二个是常量格式字符串,它的作用与`formatDateTime`相同,并返回`String`类型。 + +例如: + +```sql +SELECT FROM_UNIXTIME(1234334543, '%Y-%m-%d %R:%S') AS DateTime +``` + +```text +┌─DateTime────────────┐ +│ 2009-02-11 14:42:23 │ +└─────────────────────┘ +``` diff --git a/docs/zh/sql-reference/functions/introspection.md b/docs/zh/sql-reference/functions/introspection.md index 8d3b909b199..c0bbc10131d 100644 --- a/docs/zh/sql-reference/functions/introspection.md +++ b/docs/zh/sql-reference/functions/introspection.md @@ -18,7 +18,7 @@ toc_title: "\u81EA\u7701" - 设置 [allow_introspection_functions](../../operations/settings/settings.md#settings-allow_introspection_functions) 设置为1。 - For security reasons introspection functions are disabled by default. + 出于安全考虑,内省函数默认是关闭的。 ClickHouse将探查器报告保存到 [trace_log](../../operations/system-tables/trace_log.md#system_tables-trace_log) 系统表. 确保正确配置了表和探查器。 @@ -36,17 +36,17 @@ addressToLine(address_of_binary_instruction) **参数** -- `address_of_binary_instruction` ([UInt64](../../sql-reference/data-types/int-uint.md)) — Address of instruction in a running process. +- `address_of_binary_instruction` ([UInt64](../../sql-reference/data-types/int-uint.md)) — 正在运行进程的指令地址。 **返回值** -- 源代码文件名和此文件中用冒号分隔的行号。 +- 源代码文件名和行号(用冒号分隔的行号) - For example, `/build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:199`, where `199` is a line number. + 示例, `/build/obj-x86_64-linux-gnu/../src/Common/ThreadPool.cpp:199`, where `199` is a line number. -- 二进制文件的名称,如果函数找不到调试信息。 +- 如果函数找不到调试信息,返回二进制文件的名称。 -- 空字符串,如果地址无效。 +- 如果地址无效,返回空字符串。 类型: [字符串](../../sql-reference/data-types/string.md). @@ -132,7 +132,7 @@ addressToSymbol(address_of_binary_instruction) **返回值** - 来自ClickHouse对象文件的符号。 -- 空字符串,如果地址无效。 +- 如果地址无效,返回空字符串。 类型: [字符串](../../sql-reference/data-types/string.md). diff --git a/docs/zh/sql-reference/statements/misc.md b/docs/zh/sql-reference/statements/misc.md index a736ed2af5b..b4297d1ed4f 100644 --- a/docs/zh/sql-reference/statements/misc.md +++ b/docs/zh/sql-reference/statements/misc.md @@ -41,25 +41,25 @@ CHECK TABLE [db.]name 该 `CHECK TABLE` 查询支持下表引擎: -- [日志](../../engines/table-engines/log-family/log.md) +- [Log](../../engines/table-engines/log-family/log.md) - [TinyLog](../../engines/table-engines/log-family/tinylog.md) - [StripeLog](../../engines/table-engines/log-family/stripelog.md) -- [梅树家族](../../engines/table-engines/mergetree-family/mergetree.md) +- [MergeTree 家族](../../engines/table-engines/mergetree-family/mergetree.md) -使用另一个表引擎对表执行会导致异常。 +对其他不支持的表引擎的表执行会导致异常。 -从发动机 `*Log` 家庭不提供故障自动数据恢复。 使用 `CHECK TABLE` 查询以及时跟踪数据丢失。 +来自 `*Log` 家族的引擎不提供故障自动数据恢复。 使用 `CHECK TABLE` 查询及时跟踪数据丢失。 -为 `MergeTree` 家庭发动机, `CHECK TABLE` 查询显示本地服务器上表的每个单独数据部分的检查状态。 +对于 `MergeTree` 家族引擎, `CHECK TABLE` 查询显示本地服务器上表的每个单独数据部分的检查状态。 **如果数据已损坏** 如果表已损坏,则可以将未损坏的数据复制到另一个表。 要做到这一点: -1. 创建具有与损坏的表相同结构的新表。 要执行此操作,请执行查询 `CREATE TABLE AS `. -2. 设置 [max_threads](../../operations/settings/settings.md#settings-max_threads) 值为1以在单个线程中处理下一个查询。 要执行此操作,请运行查询 `SET max_threads = 1`. +1. 创建一个与损坏的表结构相同的新表。 要做到这一点,请执行查询 `CREATE TABLE AS `. +2. 将 [max_threads](../../operations/settings/settings.md#settings-max_threads) 值设置为1,以在单个线程中处理下一个查询。 要这样做,请运行查询 `SET max_threads = 1`. 3. 执行查询 `INSERT INTO SELECT * FROM `. 此请求将未损坏的数据从损坏的表复制到另一个表。 只有损坏部分之前的数据才会被复制。 -4. 重新启动 `clickhouse-client` 要重置 `max_threads` 价值。 +4. 重新启动 `clickhouse-client` 以重置 `max_threads` 值。 ## DESCRIBE TABLE {#misc-describe-table} @@ -67,57 +67,65 @@ CHECK TABLE [db.]name DESC|DESCRIBE TABLE [db.]table [INTO OUTFILE filename] [FORMAT format] ``` -返回以下内容 `String` 类型列: +返回以下 `String` 类型列: -- `name` — Column name. -- `type`— Column type. -- `default_type` — Clause that is used in [默认表达式](create.md#create-default-values) (`DEFAULT`, `MATERIALIZED` 或 `ALIAS`). 如果未指定默认表达式,则Column包含一个空字符串。 -- `default_expression` — Value specified in the `DEFAULT` 条款 -- `comment_expression` — Comment text. +- `name` — 列名。 +- `type`— 列的类型。 +- `default_type` — [默认表达式](create.md#create-default-values) (`DEFAULT`, `MATERIALIZED` 或 `ALIAS`)中使用的子句。 如果没有指定默认表达式,则列包含一个空字符串。 +- `default_expression` — `DEFAULT` 子句中指定的值。 +- `comment_expression` — 注释。 -嵌套的数据结构输出 “expanded” 格式。 每列分别显示,名称后面有一个点。 +嵌套数据结构以 “expanded” 格式输出。 每列分别显示,列名后加点号。 ## DETACH {#detach} -删除有关 ‘name’ 表从服务器。 服务器停止了解表的存在。 +从服务器中删除有关 ‘name’ 表的信息。 服务器停止了解该表的存在。 ``` sql DETACH TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -这不会删除表的数据或元数据。 在下一次服务器启动时,服务器将读取元数据并再次查找有关表的信息。 -同样,一个 “detached” 表可以使用重新连接 `ATTACH` 查询(系统表除外,它们没有为它们存储元数据)。 - -没有 `DETACH DATABASE` 查询。 +这不会删除表的数据或元数据。 在下一次服务器启动时,服务器将读取元数据并再次查找该表。 +同样,可以使用 `ATTACH` 查询重新连接一个 “detached” 的表(系统表除外,没有为它们存储元数据)。 ## DROP {#drop} -此查询有两种类型: `DROP DATABASE` 和 `DROP TABLE`. +删除已经存在的实体。如果指定 `IF EXISTS`, 则如果实体不存在,则不返回错误。 + +## DROP DATABASE {#drop-database} + +删除 `db` 数据库中的所有表,然后删除 `db` 数据库本身。 + +语法: ``` sql DROP DATABASE [IF EXISTS] db [ON CLUSTER cluster] ``` +## DROP TABLE {#drop-table} -删除内部的所有表 ‘db’ 数据库,然后删除 ‘db’ 数据库本身。 -如果 `IF EXISTS` 如果数据库不存在,则不会返回错误。 +删除表。 + +语法: ``` sql DROP [TEMPORARY] TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -删除表。 -如果 `IF EXISTS` 如果表不存在或数据库不存在,则不会返回错误。 - - DROP DICTIONARY [IF EXISTS] [db.]name +## DROP DICTIONARY {#drop-dictionary} 删除字典。 -如果 `IF EXISTS` 如果表不存在或数据库不存在,则不会返回错误。 + +语法: + +``` sql +DROP DICTIONARY [IF EXISTS] [db.]name +``` ## DROP USER {#drop-user-statement} 删除用户。 -### 语法 {#drop-user-syntax} +语法: ``` sql DROP USER [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -129,7 +137,7 @@ DROP USER [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除的角色将从授予该角色的所有实体撤销。 -### 语法 {#drop-role-syntax} +语法: ``` sql DROP ROLE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -141,7 +149,7 @@ DROP ROLE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除行策略将从分配该策略的所有实体撤销。 -### 语法 {#drop-row-policy-syntax} +语法: ``` sql DROP [ROW] POLICY [IF EXISTS] name [,...] ON [database.]table [,...] [ON CLUSTER cluster_name] @@ -153,7 +161,7 @@ DROP [ROW] POLICY [IF EXISTS] name [,...] ON [database.]table [,...] [ON CLUSTER 已删除的配额将从分配该配额的所有实体撤销。 -### 语法 {#drop-quota-syntax} +语法: ``` sql DROP QUOTA [IF EXISTS] name [,...] [ON CLUSTER cluster_name] @@ -165,12 +173,22 @@ DROP QUOTA [IF EXISTS] name [,...] [ON CLUSTER cluster_name] 已删除的settings配置将从分配该settings配置的所有实体撤销。 -### 语法 {#drop-settings-profile-syntax} +语法: ``` sql DROP [SETTINGS] PROFILE [IF EXISTS] name [,...] [ON CLUSTER cluster_name] ``` +## DROP VIEW {#drop-view} + +删除视图。视图也可以通过 `DROP TABLE` 删除,但是 `DROP VIEW` 检查 `[db.]name` 是视图。 + +语法: + +``` sql +DROP VIEW [IF EXISTS] [db.]name [ON CLUSTER cluster] +``` + ## EXISTS {#exists-statement} ``` sql @@ -189,7 +207,7 @@ KILL QUERY [ON CLUSTER cluster] ``` 尝试强制终止当前正在运行的查询。 -要终止的查询是从系统中选择的。使用在定义的标准进程表 `WHERE` 《公约》条款 `KILL` 查询。 +要终止的查询是使用 `KILL` 查询的 `WHERE` 子句定义的标准从system.processes表中选择的。 例: @@ -206,13 +224,13 @@ KILL QUERY WHERE user='username' SYNC 默认情况下,使用异步版本的查询 (`ASYNC`),不等待确认查询已停止。 同步版本 (`SYNC`)等待所有查询停止,并在停止时显示有关每个进程的信息。 -响应包含 `kill_status` 列,它可以采用以下值: +响应包含 `kill_status` 列,该列可以采用以下值: -1. ‘finished’ – The query was terminated successfully. -2. ‘waiting’ – Waiting for the query to end after sending it a signal to terminate. -3. The other values ​​explain why the query can't be stopped. +1. ‘finished’ – 查询已成功终止。 +2. ‘waiting’ – 发送查询信号终止后,等待查询结束。 +3. 其他值解释为什么查询不能停止。 -测试查询 (`TEST`)仅检查用户的权限并显示要停止的查询列表。 +测试查询 (`TEST`)仅检查用户的权限,并显示要停止的查询列表。 ## KILL MUTATION {#kill-mutation} @@ -223,9 +241,9 @@ KILL MUTATION [ON CLUSTER cluster] [FORMAT format] ``` -尝试取消和删除 [突变](alter.md#alter-mutations) 当前正在执行。 要取消的突变选自 [`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) 表使用由指定的过滤器 `WHERE` 《公约》条款 `KILL` 查询。 +尝试取消和删除当前正在执行的 [mutations](alter.md#alter-mutations) 。 要取消的mutation是使用 `KILL` 查询的WHERE子句指定的过滤器从[`system.mutations`](../../operations/system-tables/mutations.md#system_tables-mutations) 表中选择的。 -测试查询 (`TEST`)仅检查用户的权限并显示要停止的查询列表。 +测试查询 (`TEST`)仅检查用户的权限并显示要停止的mutations列表。 例: @@ -237,9 +255,9 @@ KILL MUTATION WHERE database = 'default' AND table = 'table' KILL MUTATION WHERE database = 'default' AND table = 'table' AND mutation_id = 'mutation_3.txt' ``` -The query is useful when a mutation is stuck and cannot finish (e.g. if some function in the mutation query throws an exception when applied to the data contained in the table). +当mutation卡住且无法完成时,该查询是有用的(例如,当mutation查询中的某些函数在应用于表中包含的数据时抛出异常)。 -已经由突变所做的更改不会回滚。 +Mutation已经做出的更改不会回滚。 ## OPTIMIZE {#misc_operations-optimize} @@ -247,19 +265,19 @@ The query is useful when a mutation is stuck and cannot finish (e.g. if some fu OPTIMIZE TABLE [db.]name [ON CLUSTER cluster] [PARTITION partition | PARTITION ID 'partition_id'] [FINAL] [DEDUPLICATE] ``` -此查询尝试使用来自表引擎的表初始化表的数据部分的非计划合并 [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md) 家人 +此查询尝试初始化 [MergeTree](../../engines/table-engines/mergetree-family/mergetree.md)家族的表引擎的表中未计划合并数据部分。 -该 `OPTMIZE` 查询也支持 [MaterializedView](../../engines/table-engines/special/materializedview.md) 和 [缓冲区](../../engines/table-engines/special/buffer.md) 引擎 不支持其他表引擎。 +该 `OPTMIZE` 查询也支持 [MaterializedView](../../engines/table-engines/special/materializedview.md) 和 [Buffer](../../engines/table-engines/special/buffer.md) 引擎。 不支持其他表引擎。 -当 `OPTIMIZE` 与使用 [ReplicatedMergeTree](../../engines/table-engines/mergetree-family/replication.md) 表引擎的家族,ClickHouse创建合并任务,并等待在所有节点上执行(如果 `replication_alter_partitions_sync` 设置已启用)。 +当 `OPTIMIZE` 与 [ReplicatedMergeTree](../../engines/table-engines/mergetree-family/replication.md) 家族的表引擎一起使用时,ClickHouse将创建一个合并任务,并等待所有节点上的执行(如果 `replication_alter_partitions_sync` 设置已启用)。 - 如果 `OPTIMIZE` 出于任何原因不执行合并,它不通知客户端。 要启用通知,请使用 [optimize_throw_if_noop](../../operations/settings/settings.md#setting-optimize_throw_if_noop) 设置。 - 如果您指定 `PARTITION`,仅优化指定的分区。 [如何设置分区表达式](alter.md#alter-how-to-specify-part-expr). - 如果您指定 `FINAL`,即使所有数据已经在一个部分中,也会执行优化。 -- 如果您指定 `DEDUPLICATE`,然后完全相同的行将被重复数据删除(所有列进行比较),这仅适用于MergeTree引擎。 +- 如果您指定 `DEDUPLICATE`,则将对完全相同的行进行重复数据删除(所有列进行比较),这仅适用于MergeTree引擎。 !!! warning "警告" - `OPTIMIZE` 无法修复 “Too many parts” 错误 + `OPTIMIZE` 无法修复 “Too many parts” 错误。 ## RENAME {#misc_operations-rename} @@ -270,6 +288,7 @@ RENAME TABLE [db11.]name11 TO [db12.]name12, [db21.]name21 TO [db22.]name22, ... ``` 所有表都在全局锁定下重命名。 重命名表是一个轻型操作。 如果您在TO之后指定了另一个数据库,则表将被移动到此数据库。 但是,包含数据库的目录必须位于同一文件系统中(否则,将返回错误)。 +如果您在一个查询中重命名多个表,这是一个非原子操作,它可能被部分执行,其他会话中的查询可能会接收错误 Table ... doesn't exist ...。 ## SET {#query-set} @@ -277,9 +296,9 @@ RENAME TABLE [db11.]name11 TO [db12.]name12, [db21.]name21 TO [db22.]name22, ... SET param = value ``` -分配 `value` 到 `param` [设置](../../operations/settings/index.md) 对于当前会话。 你不能改变 [服务器设置](../../operations/server-configuration-parameters/index.md) 这边 +为当前会话的 [设置](../../operations/settings/index.md) `param` 分配值 `value`。 您不能以这种方式更改 [服务器设置](../../operations/server-configuration-parameters/index.md)。 -您还可以在单个查询中设置指定设置配置文件中的所有值。 +您还可以在单个查询中从指定的设置配置文件中设置所有值。 ``` sql SET profile = 'profile-name-from-the-settings-file' @@ -291,8 +310,6 @@ SET profile = 'profile-name-from-the-settings-file' 激活当前用户的角色。 -### 语法 {#set-role-syntax} - ``` sql SET ROLE {DEFAULT | NONE | role [,...] | ALL | ALL EXCEPT role [,...]} ``` @@ -301,15 +318,13 @@ SET ROLE {DEFAULT | NONE | role [,...] | ALL | ALL EXCEPT role [,...]} 将默认角色设置为用户。 -默认角色在用户登录时自动激活。 您只能将以前授予的角色设置为默认值。 如果未向用户授予角色,ClickHouse将引发异常。 - -### 语法 {#set-default-role-syntax} +默认角色在用户登录时自动激活。 您只能将以前授予的角色设置为默认值。 如果角色没有授予用户,ClickHouse会抛出异常。 ``` sql SET DEFAULT ROLE {NONE | role [,...] | ALL | ALL EXCEPT role [,...]} TO {user|CURRENT_USER} [,...] ``` -### 例 {#set-default-role-examples} +### 示例 {#set-default-role-examples} 为用户设置多个默认角色: @@ -317,19 +332,19 @@ SET DEFAULT ROLE {NONE | role [,...] | ALL | ALL EXCEPT role [,...]} TO {user|CU SET DEFAULT ROLE role1, role2, ... TO user ``` -将所有授予的角色设置为用户的默认值: +将所有授予的角色设置为用户的默认角色: ``` sql SET DEFAULT ROLE ALL TO user ``` -从用户清除默认角色: +清除用户的默认角色: ``` sql SET DEFAULT ROLE NONE TO user ``` -将所有授予的角色设置为默认角色,其中一些角色除外: +将所有授予的角色设置为默认角色,但其中一些角色除外: ``` sql SET DEFAULT ROLE ALL EXCEPT role1, role2 TO user @@ -341,9 +356,9 @@ SET DEFAULT ROLE ALL EXCEPT role1, role2 TO user TRUNCATE TABLE [IF EXISTS] [db.]name [ON CLUSTER cluster] ``` -从表中删除所有数据。 当条款 `IF EXISTS` 如果该表不存在,则查询返回错误。 +从表中删除所有数据。 当省略 `IF EXISTS`子句时,如果该表不存在,则查询返回错误。 -该 `TRUNCATE` 查询不支持 [查看](../../engines/table-engines/special/view.md), [文件](../../engines/table-engines/special/file.md), [URL](../../engines/table-engines/special/url.md) 和 [Null](../../engines/table-engines/special/null.md) 表引擎. +该 `TRUNCATE` 查询不支持 [View](../../engines/table-engines/special/view.md), [File](../../engines/table-engines/special/file.md), [URL](../../engines/table-engines/special/url.md) 和 [Null](../../engines/table-engines/special/null.md) 表引擎. ## USE {#use} diff --git a/programs/client/Client.cpp b/programs/client/Client.cpp index e4858eeda8b..92ae37123fa 100644 --- a/programs/client/Client.cpp +++ b/programs/client/Client.cpp @@ -45,6 +45,7 @@ #include #include #include +#include #include #include #include @@ -475,9 +476,16 @@ private: /// The value of the option is used as the text of query (or of multiple queries). /// If stdin is not a terminal, INSERT data for the first query is read from it. /// - stdin is not a terminal. In this case queries are read from it. - if (!stdin_is_a_tty || config().has("query")) + /// - -qf (--queries-file) command line option is present. + /// The value of the option is used as file with query (or of multiple queries) to execute. + if (!stdin_is_a_tty || config().has("query") || config().has("queries-file")) is_interactive = false; + if (config().has("query") && config().has("queries-file")) + { + throw Exception("Specify either `query` or `queries-file` option", ErrorCodes::BAD_ARGUMENTS); + } + std::cout << std::fixed << std::setprecision(3); std::cerr << std::fixed << std::setprecision(3); @@ -786,8 +794,15 @@ private: { String text; - if (config().has("query")) - text = config().getRawString("query"); /// Poco configuration should not process substitutions in form of ${...} inside query. + if (config().has("queries-file")) + { + ReadBufferFromFile in(config().getString("queries-file")); + readStringUntilEOF(text, in); + processMultiQuery(text); + return; + } + else if (config().has("query")) + text = config().getRawString("query"); /// Poco configuration should not process substitutions in form of ${...} inside query. else { /// If 'query' parameter is not set, read a query from stdin. @@ -2320,6 +2335,7 @@ public: "Suggestion limit for how many databases, tables and columns to fetch.") ("multiline,m", "multiline") ("multiquery,n", "multiquery") + ("queries-file,qf", po::value(), "file path with queries to execute") ("format,f", po::value(), "default output format") ("testmode,T", "enable test hints in comments") ("ignore-error", "do not stop processing in multiquery mode") @@ -2448,6 +2464,8 @@ public: config().setString("query_id", options["query_id"].as()); if (options.count("query")) config().setString("query", options["query"].as()); + if (options.count("queries-file")) + config().setString("queries-file", options["queries-file"].as()); if (options.count("database")) config().setString("database", options["database"].as()); if (options.count("pager")) diff --git a/programs/local/LocalServer.cpp b/programs/local/LocalServer.cpp index 15e71198eb1..dbf153eeb81 100644 --- a/programs/local/LocalServer.cpp +++ b/programs/local/LocalServer.cpp @@ -20,9 +20,11 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -195,7 +197,7 @@ try ThreadStatus thread_status; UseSSL use_ssl; - if (!config().has("query") && !config().has("table-structure")) /// Nothing to process + if (!config().has("query") && !config().has("table-structure") && !config().has("queries-file")) /// Nothing to process { if (config().hasOption("verbose")) std::cerr << "There are no queries to process." << '\n'; @@ -203,6 +205,11 @@ try return Application::EXIT_OK; } + if (config().has("query") && config().has("queries-file")) + { + throw Exception("Specify either `query` or `queries-file` option", ErrorCodes::BAD_ARGUMENTS); + } + shared_context = Context::createShared(); global_context = std::make_unique(Context::createGlobal(shared_context.get())); global_context->makeGlobalContext(); @@ -340,7 +347,17 @@ std::string LocalServer::getInitialCreateTableQuery() void LocalServer::processQueries() { String initial_create_query = getInitialCreateTableQuery(); - String queries_str = initial_create_query + config().getRawString("query"); + String queries_str = initial_create_query; + + if (config().has("query")) + queries_str += config().getRawString("query"); + else + { + String queries_from_file; + ReadBufferFromFile in(config().getString("queries-file")); + readStringUntilEOF(queries_from_file, in); + queries_str += queries_from_file; + } const auto & settings = global_context->getSettingsRef(); @@ -505,6 +522,7 @@ void LocalServer::init(int argc, char ** argv) ("help", "produce help message") ("config-file,c", po::value(), "config-file path") ("query,q", po::value(), "query") + ("queries-file, qf", po::value(), "file path with queries to execute") ("database,d", po::value(), "database") ("table,N", po::value(), "name of the initial table") @@ -552,6 +570,8 @@ void LocalServer::init(int argc, char ** argv) config().setString("config-file", options["config-file"].as()); if (options.count("query")) config().setString("query", options["query"].as()); + if (options.count("queries-file")) + config().setString("queries-file", options["queries-file"].as()); if (options.count("database")) config().setString("default_database", options["database"].as()); diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 26339c5ad3f..28d8301b920 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -57,6 +58,7 @@ #include #include #include +#include #include "MetricsTransmitter.h" #include #include @@ -186,6 +188,85 @@ static std::string getUserName(uid_t user_id) return toString(user_id); } +Poco::Net::SocketAddress makeSocketAddress(const std::string & host, UInt16 port, Poco::Logger * log) +{ + Poco::Net::SocketAddress socket_address; + try + { + socket_address = Poco::Net::SocketAddress(host, port); + } + catch (const Poco::Net::DNSException & e) + { + const auto code = e.code(); + if (code == EAI_FAMILY +#if defined(EAI_ADDRFAMILY) + || code == EAI_ADDRFAMILY +#endif + ) + { + LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " + "If it is an IPv6 address and your host has disabled IPv6, then consider to " + "specify IPv4 address to listen in element of configuration " + "file. Example: 0.0.0.0", + host, e.code(), e.message()); + } + + throw; + } + return socket_address; +} + +Poco::Net::SocketAddress Server::socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure) const +{ + auto address = makeSocketAddress(host, port, &logger()); +#if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100 + if (secure) + /// Bug in old (<1.9.1) poco, listen() after bind() with reusePort param will fail because have no implementation in SecureServerSocketImpl + /// https://github.com/pocoproject/poco/pull/2257 + socket.bind(address, /* reuseAddress = */ true); + else +#endif +#if POCO_VERSION < 0x01080000 + socket.bind(address, /* reuseAddress = */ true); +#else + socket.bind(address, /* reuseAddress = */ true, /* reusePort = */ config().getBool("listen_reuse_port", false)); +#endif + + socket.listen(/* backlog = */ config().getUInt("listen_backlog", 64)); + + return address; +} + +void Server::createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const +{ + /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file. + if (!config().has(port_name)) + return; + + auto port = config().getInt(port_name); + try + { + func(port); + } + catch (const Poco::Exception &) + { + std::string message = "Listen [" + listen_host + "]:" + std::to_string(port) + " failed: " + getCurrentExceptionMessage(false); + + if (listen_try) + { + LOG_WARNING(&logger(), "{}. If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to " + "specify not disabled IPv4 or IPv6 address to listen in element of configuration " + "file. Example for disabled IPv6: 0.0.0.0 ." + " Example for disabled IPv4: ::", + message); + } + else + { + throw Exception{message, ErrorCodes::NETWORK_ERROR}; + } + } +} + void Server::uninitialize() { logger().information("shutting down"); @@ -399,27 +480,6 @@ int Server::main(const std::vector & /*args*/) StatusFile status{path + "status", StatusFile::write_full_info}; - SCOPE_EXIT({ - /** Ask to cancel background jobs all table engines, - * and also query_log. - * It is important to do early, not in destructor of Context, because - * table engines could use Context on destroy. - */ - LOG_INFO(log, "Shutting down storages."); - - global_context->shutdown(); - - LOG_DEBUG(log, "Shut down storages."); - - /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. - * At this moment, no one could own shared part of Context. - */ - global_context_ptr = nullptr; - global_context.reset(); - shared_context.reset(); - LOG_DEBUG(log, "Destroyed global context."); - }); - /// Try to increase limit on number of open files. { rlimit rlim; @@ -483,6 +543,12 @@ int Server::main(const std::vector & /*args*/) Poco::File(dictionaries_lib_path).createDirectories(); } + /// top_level_domains_lists + { + const std::string & top_level_domains_path = config().getString("top_level_domains_path", path + "top_level_domains/") + "/"; + TLDListsHolder::getInstance().parseConfig(top_level_domains_path, config()); + } + { Poco::File(path + "data/").createDirectories(); Poco::File(path + "metadata/").createDirectories(); @@ -675,6 +741,71 @@ int Server::main(const std::vector & /*args*/) total_memory_tracker.setDescription("(total)"); total_memory_tracker.setMetric(CurrentMetrics::MemoryTracking); + Poco::Timespan keep_alive_timeout(config().getUInt("keep_alive_timeout", 10), 0); + + Poco::ThreadPool server_pool(3, config().getUInt("max_connections", 1024)); + Poco::Net::HTTPServerParams::Ptr http_params = new Poco::Net::HTTPServerParams; + http_params->setTimeout(settings.http_receive_timeout); + http_params->setKeepAliveTimeout(keep_alive_timeout); + + std::vector servers_to_start_before_tables; + + std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); + + bool listen_try = config().getBool("listen_try", false); + if (listen_hosts.empty()) + { + listen_hosts.emplace_back("::1"); + listen_hosts.emplace_back("127.0.0.1"); + listen_try = true; + } + + for (const auto & listen_host : listen_hosts) + { + /// TCP TestKeeper + createServer(listen_host, "test_keeper_server.tcp_port", listen_try, [&](UInt16 port) + { + Poco::Net::ServerSocket socket; + auto address = socketBindListen(socket, listen_host, port); + socket.setReceiveTimeout(settings.receive_timeout); + socket.setSendTimeout(settings.send_timeout); + servers_to_start_before_tables.emplace_back(std::make_unique( + new TestKeeperTCPHandlerFactory(*this), + server_pool, + socket, + new Poco::Net::TCPServerParams)); + + LOG_INFO(log, "Listening for connections to fake zookeeper (tcp): {}", address.toString()); + }); + } + + for (auto & server : servers_to_start_before_tables) + server.start(); + + SCOPE_EXIT({ + /** Ask to cancel background jobs all table engines, + * and also query_log. + * It is important to do early, not in destructor of Context, because + * table engines could use Context on destroy. + */ + LOG_INFO(log, "Shutting down storages."); + + global_context->shutdown(); + + LOG_DEBUG(log, "Shut down storages."); + + for (auto & server : servers_to_start_before_tables) + server.stop(); + + /** Explicitly destroy Context. It is more convenient than in destructor of Server, because logger is still available. + * At this moment, no one could own shared part of Context. + */ + global_context_ptr = nullptr; + global_context.reset(); + shared_context.reset(); + LOG_DEBUG(log, "Destroyed global context."); + }); + /// Set current database name before loading tables and databases because /// system logs may copy global context. global_context->setCurrentDatabaseNameInGlobalContext(default_database); @@ -804,75 +935,8 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "TaskStats is not implemented for this OS. IO accounting will be disabled."); #endif + std::vector servers; { - Poco::Timespan keep_alive_timeout(config().getUInt("keep_alive_timeout", 10), 0); - - Poco::ThreadPool server_pool(3, config().getUInt("max_connections", 1024)); - Poco::Net::HTTPServerParams::Ptr http_params = new Poco::Net::HTTPServerParams; - http_params->setTimeout(settings.http_receive_timeout); - http_params->setKeepAliveTimeout(keep_alive_timeout); - - std::vector servers; - - std::vector listen_hosts = DB::getMultipleValuesFromConfig(config(), "", "listen_host"); - - bool listen_try = config().getBool("listen_try", false); - if (listen_hosts.empty()) - { - listen_hosts.emplace_back("::1"); - listen_hosts.emplace_back("127.0.0.1"); - listen_try = true; - } - - auto make_socket_address = [&](const std::string & host, UInt16 port) - { - Poco::Net::SocketAddress socket_address; - try - { - socket_address = Poco::Net::SocketAddress(host, port); - } - catch (const Poco::Net::DNSException & e) - { - const auto code = e.code(); - if (code == EAI_FAMILY -#if defined(EAI_ADDRFAMILY) - || code == EAI_ADDRFAMILY -#endif - ) - { - LOG_ERROR(log, "Cannot resolve listen_host ({}), error {}: {}. " - "If it is an IPv6 address and your host has disabled IPv6, then consider to " - "specify IPv4 address to listen in element of configuration " - "file. Example: 0.0.0.0", - host, e.code(), e.message()); - } - - throw; - } - return socket_address; - }; - - auto socket_bind_listen = [&](auto & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false) - { - auto address = make_socket_address(host, port); -#if !defined(POCO_CLICKHOUSE_PATCH) || POCO_VERSION < 0x01090100 - if (secure) - /// Bug in old (<1.9.1) poco, listen() after bind() with reusePort param will fail because have no implementation in SecureServerSocketImpl - /// https://github.com/pocoproject/poco/pull/2257 - socket.bind(address, /* reuseAddress = */ true); - else -#endif -#if POCO_VERSION < 0x01080000 - socket.bind(address, /* reuseAddress = */ true); -#else - socket.bind(address, /* reuseAddress = */ true, /* reusePort = */ config().getBool("listen_reuse_port", false)); -#endif - - socket.listen(/* backlog = */ config().getUInt("listen_backlog", 64)); - - return address; - }; - /// This object will periodically calculate some metrics. AsynchronousMetrics async_metrics(*global_context, config().getUInt("asynchronous_metrics_update_period_s", 60)); @@ -880,41 +944,11 @@ int Server::main(const std::vector & /*args*/) for (const auto & listen_host : listen_hosts) { - auto create_server = [&](const char * port_name, auto && func) - { - /// For testing purposes, user may omit tcp_port or http_port or https_port in configuration file. - if (!config().has(port_name)) - return; - - auto port = config().getInt(port_name); - try - { - func(port); - } - catch (const Poco::Exception &) - { - std::string message = "Listen [" + listen_host + "]:" + std::to_string(port) + " failed: " + getCurrentExceptionMessage(false); - - if (listen_try) - { - LOG_WARNING(log, "{}. If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to " - "specify not disabled IPv4 or IPv6 address to listen in element of configuration " - "file. Example for disabled IPv6: 0.0.0.0 ." - " Example for disabled IPv4: ::", - message); - } - else - { - throw Exception{message, ErrorCodes::NETWORK_ERROR}; - } - } - }; - /// HTTP - create_server("http_port", [&](UInt16 port) + createServer(listen_host, "http_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); @@ -925,11 +959,11 @@ int Server::main(const std::vector & /*args*/) }); /// HTTPS - create_server("https_port", [&](UInt16 port) + createServer(listen_host, "https_port", listen_try, [&](UInt16 port) { #if USE_SSL Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -944,14 +978,14 @@ int Server::main(const std::vector & /*args*/) }); /// TCP - create_server("tcp_port", [&](UInt16 port) + createServer(listen_host, "tcp_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( - new TCPHandlerFactory(*this), + new TCPHandlerFactory(*this, /* secure */ false, /* proxy protocol */ false), server_pool, socket, new Poco::Net::TCPServerParams)); @@ -959,16 +993,32 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for connections with native protocol (tcp): {}", address.toString()); }); - /// TCP with SSL - create_server("tcp_port_secure", [&](UInt16 port) + /// TCP with PROXY protocol, see https://github.com/wolfeidau/proxyv2/blob/master/docs/proxy-protocol.txt + createServer(listen_host, "tcp_with_proxy_port", listen_try, [&](UInt16 port) { -#if USE_SSL - Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + Poco::Net::ServerSocket socket; + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.receive_timeout); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( - new TCPHandlerFactory(*this, /* secure= */ true), + new TCPHandlerFactory(*this, /* secure */ false, /* proxy protocol */ true), + server_pool, + socket, + new Poco::Net::TCPServerParams)); + + LOG_INFO(log, "Listening for connections with native protocol (tcp) with PROXY: {}", address.toString()); + }); + + /// TCP with SSL + createServer(listen_host, "tcp_port_secure", listen_try, [&](UInt16 port) + { +#if USE_SSL + Poco::Net::SecureServerSocket socket; + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); + socket.setReceiveTimeout(settings.receive_timeout); + socket.setSendTimeout(settings.send_timeout); + servers.emplace_back(std::make_unique( + new TCPHandlerFactory(*this, /* secure */ true, /* proxy protocol */ false), server_pool, socket, new Poco::Net::TCPServerParams)); @@ -981,10 +1031,10 @@ int Server::main(const std::vector & /*args*/) }); /// Interserver IO HTTP - create_server("interserver_http_port", [&](UInt16 port) + createServer(listen_host, "interserver_http_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -993,11 +1043,11 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for replica communication (interserver): http://{}", address.toString()); }); - create_server("interserver_https_port", [&](UInt16 port) + createServer(listen_host, "interserver_https_port", listen_try, [&](UInt16 port) { #if USE_SSL Poco::Net::SecureServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -1011,10 +1061,10 @@ int Server::main(const std::vector & /*args*/) #endif }); - create_server("mysql_port", [&](UInt16 port) + createServer(listen_host, "mysql_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(Poco::Timespan()); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -1026,10 +1076,10 @@ int Server::main(const std::vector & /*args*/) LOG_INFO(log, "Listening for MySQL compatibility protocol: {}", address.toString()); }); - create_server("postgresql_port", [&](UInt16 port) + createServer(listen_host, "postgresql_port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port, /* secure = */ true); + auto address = socketBindListen(socket, listen_host, port, /* secure = */ true); socket.setReceiveTimeout(Poco::Timespan()); socket.setSendTimeout(settings.send_timeout); servers.emplace_back(std::make_unique( @@ -1042,19 +1092,19 @@ int Server::main(const std::vector & /*args*/) }); #if USE_GRPC - create_server("grpc_port", [&](UInt16 port) + createServer(listen_host, "grpc_port", listen_try, [&](UInt16 port) { Poco::Net::SocketAddress server_address(listen_host, port); - servers.emplace_back(std::make_unique(*this, make_socket_address(listen_host, port))); + servers.emplace_back(std::make_unique(*this, makeSocketAddress(listen_host, port, log))); LOG_INFO(log, "Listening for gRPC protocol: " + server_address.toString()); }); #endif /// Prometheus (if defined and not setup yet with http_port) - create_server("prometheus.port", [&](UInt16 port) + createServer(listen_host, "prometheus.port", listen_try, [&](UInt16 port) { Poco::Net::ServerSocket socket; - auto address = socket_bind_listen(socket, listen_host, port); + auto address = socketBindListen(socket, listen_host, port); socket.setReceiveTimeout(settings.http_receive_timeout); socket.setSendTimeout(settings.http_send_timeout); servers.emplace_back(std::make_unique( @@ -1078,6 +1128,7 @@ int Server::main(const std::vector & /*args*/) int level = level_str.empty() ? INT_MAX : Poco::Logger::parseLevel(level_str); setTextLog(global_context->getTextLog(), level); } + buildLoggers(config(), logger()); main_config_reloader->start(); @@ -1124,7 +1175,10 @@ int Server::main(const std::vector & /*args*/) { current_connections = 0; for (auto & server : servers) + { + server.stop(); current_connections += server.currentConnections(); + } if (!current_connections) break; sleep_current_ms += sleep_one_ms; diff --git a/programs/server/Server.h b/programs/server/Server.h index ad9e51c881c..c582e475308 100644 --- a/programs/server/Server.h +++ b/programs/server/Server.h @@ -14,6 +14,13 @@ * 3. Interserver HTTP - for replication. */ +namespace Poco +{ + namespace Net + { + class ServerSocket; + } +} namespace DB { @@ -57,6 +64,13 @@ protected: private: Context * global_context_ptr = nullptr; + +private: + + Poco::Net::SocketAddress socketBindListen(Poco::Net::ServerSocket & socket, const std::string & host, UInt16 port, [[maybe_unused]] bool secure = false) const; + + using CreateServerFunc = std::function; + void createServer(const std::string & listen_host, const char * port_name, bool listen_try, CreateServerFunc && func) const; }; } diff --git a/programs/server/config.d/path.xml b/programs/server/config.d/path.xml index 8db1d18e8c7..466ed0d1663 100644 --- a/programs/server/config.d/path.xml +++ b/programs/server/config.d/path.xml @@ -4,4 +4,5 @@ ./user_files/ ./format_schemas/ ./access/ + ./top_level_domains/ diff --git a/programs/server/config.d/tcp_with_proxy.xml b/programs/server/config.d/tcp_with_proxy.xml new file mode 120000 index 00000000000..6dc2b3fd1b9 --- /dev/null +++ b/programs/server/config.d/tcp_with_proxy.xml @@ -0,0 +1 @@ +../../../tests/config/config.d/tcp_with_proxy.xml \ No newline at end of file diff --git a/programs/server/config.xml b/programs/server/config.xml index dde3702a44b..f41c346bbed 100644 --- a/programs/server/config.xml +++ b/programs/server/config.xml @@ -64,11 +64,18 @@ 8123 9000 9004 + + + + + @@ -137,29 +144,32 @@ - true + false - /path/to/ssl_cert_file - /path/to/ssl_key_file --> + /path/to/ssl_key_file - + + false - + + /path/to/ssl_ca_cert_file + Supported algorithms: none, deflate, gzip, stream_gzip --> + deflate + Supported levels: none, low, medium, high --> + medium - -1 - 4194304 --> + -1 + + + false @@ -714,6 +724,19 @@ + + + + + + diff --git a/src/Access/AccessControlManager.cpp b/src/Access/AccessControlManager.cpp index a95d65ebb59..e874bda5b69 100644 --- a/src/Access/AccessControlManager.cpp +++ b/src/Access/AccessControlManager.cpp @@ -15,6 +15,8 @@ #include #include #include +#include +#include #include #include @@ -419,6 +421,18 @@ std::shared_ptr AccessControlManager::getContextAccess( params.http_method = client_info.http_method; params.address = client_info.current_address.host(); params.quota_key = client_info.quota_key; + + /// Extract the last entry from comma separated list of X-Forwarded-For addresses. + /// Only the last proxy can be trusted (if any). + Strings forwarded_addresses; + boost::split(forwarded_addresses, client_info.forwarded_for, boost::is_any_of(",")); + if (!forwarded_addresses.empty()) + { + String & last_forwarded_address = forwarded_addresses.back(); + boost::trim(last_forwarded_address); + params.forwarded_address = last_forwarded_address; + } + return getContextAccess(params); } @@ -444,9 +458,14 @@ std::shared_ptr AccessControlManager::getEnabledRowPol std::shared_ptr AccessControlManager::getEnabledQuota( - const UUID & user_id, const String & user_name, const boost::container::flat_set & enabled_roles, const Poco::Net::IPAddress & address, const String & custom_quota_key) const + const UUID & user_id, + const String & user_name, + const boost::container::flat_set & enabled_roles, + const Poco::Net::IPAddress & address, + const String & forwarded_address, + const String & custom_quota_key) const { - return quota_cache->getEnabledQuota(user_id, user_name, enabled_roles, address, custom_quota_key); + return quota_cache->getEnabledQuota(user_id, user_name, enabled_roles, address, forwarded_address, custom_quota_key); } diff --git a/src/Access/AccessControlManager.h b/src/Access/AccessControlManager.h index c960b330ee6..07edfd93475 100644 --- a/src/Access/AccessControlManager.h +++ b/src/Access/AccessControlManager.h @@ -135,6 +135,7 @@ public: const String & user_name, const boost::container::flat_set & enabled_roles, const Poco::Net::IPAddress & address, + const String & forwarded_address, const String & custom_quota_key) const; std::vector getAllQuotasUsage() const; diff --git a/src/Access/AccessType.h b/src/Access/AccessType.h index 11896f628d9..1a070420fd1 100644 --- a/src/Access/AccessType.h +++ b/src/Access/AccessType.h @@ -127,6 +127,7 @@ enum class AccessType M(SYSTEM_DROP_COMPILED_EXPRESSION_CACHE, "SYSTEM DROP COMPILED EXPRESSION, DROP COMPILED EXPRESSION CACHE, DROP COMPILED EXPRESSIONS", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_CACHE, "DROP CACHE", GROUP, SYSTEM) \ M(SYSTEM_RELOAD_CONFIG, "RELOAD CONFIG", GLOBAL, SYSTEM_RELOAD) \ + M(SYSTEM_RELOAD_SYMBOLS, "RELOAD SYMBOLS", GLOBAL, SYSTEM_RELOAD) \ M(SYSTEM_RELOAD_DICTIONARY, "SYSTEM RELOAD DICTIONARIES, RELOAD DICTIONARY, RELOAD DICTIONARIES", GLOBAL, SYSTEM_RELOAD) \ M(SYSTEM_RELOAD_EMBEDDED_DICTIONARIES, "RELOAD EMBEDDED DICTIONARIES", GLOBAL, SYSTEM_RELOAD) /* implicitly enabled by the grant SYSTEM_RELOAD_DICTIONARY ON *.* */\ M(SYSTEM_RELOAD, "", GROUP, SYSTEM) \ diff --git a/src/Access/ContextAccess.cpp b/src/Access/ContextAccess.cpp index 0459022cb1a..0e4f3fe7871 100644 --- a/src/Access/ContextAccess.cpp +++ b/src/Access/ContextAccess.cpp @@ -258,9 +258,12 @@ void ContextAccess::setRolesInfo(const std::shared_ptr & { assert(roles_info_); roles_info = roles_info_; - enabled_row_policies = manager->getEnabledRowPolicies(*params.user_id, roles_info->enabled_roles); - enabled_quota = manager->getEnabledQuota(*params.user_id, user_name, roles_info->enabled_roles, params.address, params.quota_key); - enabled_settings = manager->getEnabledSettings(*params.user_id, user->settings, roles_info->enabled_roles, roles_info->settings_from_enabled_roles); + enabled_row_policies = manager->getEnabledRowPolicies( + *params.user_id, roles_info->enabled_roles); + enabled_quota = manager->getEnabledQuota( + *params.user_id, user_name, roles_info->enabled_roles, params.address, params.forwarded_address, params.quota_key); + enabled_settings = manager->getEnabledSettings( + *params.user_id, user->settings, roles_info->enabled_roles, roles_info->settings_from_enabled_roles); calculateAccessRights(); } diff --git a/src/Access/ContextAccess.h b/src/Access/ContextAccess.h index fd7d7c01922..319c8edb076 100644 --- a/src/Access/ContextAccess.h +++ b/src/Access/ContextAccess.h @@ -41,9 +41,16 @@ struct ContextAccessParams ClientInfo::Interface interface = ClientInfo::Interface::TCP; ClientInfo::HTTPMethod http_method = ClientInfo::HTTPMethod::UNKNOWN; Poco::Net::IPAddress address; + String forwarded_address; String quota_key; - auto toTuple() const { return std::tie(user_id, current_roles, use_default_roles, readonly, allow_ddl, allow_introspection, current_database, interface, http_method, address, quota_key); } + auto toTuple() const + { + return std::tie( + user_id, current_roles, use_default_roles, readonly, allow_ddl, allow_introspection, + current_database, interface, http_method, address, forwarded_address, quota_key); + } + friend bool operator ==(const ContextAccessParams & lhs, const ContextAccessParams & rhs) { return lhs.toTuple() == rhs.toTuple(); } friend bool operator !=(const ContextAccessParams & lhs, const ContextAccessParams & rhs) { return !(lhs == rhs); } friend bool operator <(const ContextAccessParams & lhs, const ContextAccessParams & rhs) { return lhs.toTuple() < rhs.toTuple(); } diff --git a/src/Access/EnabledQuota.h b/src/Access/EnabledQuota.h index 25e804dd050..7ae107e45e3 100644 --- a/src/Access/EnabledQuota.h +++ b/src/Access/EnabledQuota.h @@ -25,9 +25,10 @@ public: String user_name; boost::container::flat_set enabled_roles; Poco::Net::IPAddress client_address; + String forwarded_address; String client_key; - auto toTuple() const { return std::tie(user_id, enabled_roles, user_name, client_address, client_key); } + auto toTuple() const { return std::tie(user_id, enabled_roles, user_name, client_address, forwarded_address, client_key); } friend bool operator ==(const Params & lhs, const Params & rhs) { return lhs.toTuple() == rhs.toTuple(); } friend bool operator !=(const Params & lhs, const Params & rhs) { return !(lhs == rhs); } friend bool operator <(const Params & lhs, const Params & rhs) { return lhs.toTuple() < rhs.toTuple(); } diff --git a/src/Access/Quota.h b/src/Access/Quota.h index 5bbea36cfda..b636e83ec40 100644 --- a/src/Access/Quota.h +++ b/src/Access/Quota.h @@ -76,6 +76,7 @@ struct Quota : public IAccessEntity NONE, /// All users share the same quota. USER_NAME, /// Connections with the same user name share the same quota. IP_ADDRESS, /// Connections from the same IP share the same quota. + FORWARDED_IP_ADDRESS, /// Use X-Forwarded-For HTTP header instead of IP address. CLIENT_KEY, /// Client should explicitly supply a key to use. CLIENT_KEY_OR_USER_NAME, /// Same as CLIENT_KEY, but use USER_NAME if the client doesn't supply a key. CLIENT_KEY_OR_IP_ADDRESS, /// Same as CLIENT_KEY, but use IP_ADDRESS if the client doesn't supply a key. @@ -205,12 +206,16 @@ inline const Quota::KeyTypeInfo & Quota::KeyTypeInfo::get(KeyType type) if (tokens.size() > 1) { for (const auto & token : tokens) + { for (auto kt : ext::range(KeyType::MAX)) + { if (KeyTypeInfo::get(kt).name == token) { init_base_types.push_back(kt); break; } + } + } } return KeyTypeInfo{raw_name_, std::move(init_name), std::move(init_base_types)}; }; @@ -232,6 +237,11 @@ inline const Quota::KeyTypeInfo & Quota::KeyTypeInfo::get(KeyType type) static const auto info = make_info("IP_ADDRESS"); return info; } + case KeyType::FORWARDED_IP_ADDRESS: + { + static const auto info = make_info("FORWARDED_IP_ADDRESS"); + return info; + } case KeyType::CLIENT_KEY: { static const auto info = make_info("CLIENT_KEY"); diff --git a/src/Access/QuotaCache.cpp b/src/Access/QuotaCache.cpp index d3ebca9529c..0f814bced4e 100644 --- a/src/Access/QuotaCache.cpp +++ b/src/Access/QuotaCache.cpp @@ -48,11 +48,21 @@ String QuotaCache::QuotaInfo::calculateKey(const EnabledQuota & enabled) const switch (quota->key_type) { case KeyType::NONE: + { return ""; + } case KeyType::USER_NAME: + { return params.user_name; + } case KeyType::IP_ADDRESS: + { return params.client_address.toString(); + } + case KeyType::FORWARDED_IP_ADDRESS: + { + return params.forwarded_address; + } case KeyType::CLIENT_KEY: { if (!params.client_key.empty()) @@ -170,7 +180,7 @@ QuotaCache::QuotaCache(const AccessControlManager & access_control_manager_) QuotaCache::~QuotaCache() = default; -std::shared_ptr QuotaCache::getEnabledQuota(const UUID & user_id, const String & user_name, const boost::container::flat_set & enabled_roles, const Poco::Net::IPAddress & client_address, const String & client_key) +std::shared_ptr QuotaCache::getEnabledQuota(const UUID & user_id, const String & user_name, const boost::container::flat_set & enabled_roles, const Poco::Net::IPAddress & client_address, const String & forwarded_address, const String & client_key) { std::lock_guard lock{mutex}; ensureAllQuotasRead(); @@ -180,6 +190,7 @@ std::shared_ptr QuotaCache::getEnabledQuota(const UUID & use params.user_name = user_name; params.enabled_roles = enabled_roles; params.client_address = client_address; + params.forwarded_address = forwarded_address; params.client_key = client_key; auto it = enabled_quotas.find(params); if (it != enabled_quotas.end()) diff --git a/src/Access/QuotaCache.h b/src/Access/QuotaCache.h index 0bb5c11a82b..31d274c4ff6 100644 --- a/src/Access/QuotaCache.h +++ b/src/Access/QuotaCache.h @@ -20,7 +20,14 @@ public: QuotaCache(const AccessControlManager & access_control_manager_); ~QuotaCache(); - std::shared_ptr getEnabledQuota(const UUID & user_id, const String & user_name, const boost::container::flat_set & enabled_roles, const Poco::Net::IPAddress & address, const String & client_key); + std::shared_ptr getEnabledQuota( + const UUID & user_id, + const String & user_name, + const boost::container::flat_set & enabled_roles, + const Poco::Net::IPAddress & address, + const String & forwarded_address, + const String & client_key); + std::vector getAllQuotasUsage() const; private: diff --git a/src/Access/UsersConfigAccessStorage.cpp b/src/Access/UsersConfigAccessStorage.cpp index eb993d696c6..b3f151c3030 100644 --- a/src/Access/UsersConfigAccessStorage.cpp +++ b/src/Access/UsersConfigAccessStorage.cpp @@ -215,6 +215,8 @@ namespace String quota_config = "quotas." + quota_name; if (config.has(quota_config + ".keyed_by_ip")) quota->key_type = KeyType::IP_ADDRESS; + else if (config.has(quota_config + ".keyed_by_forwarded_ip")) + quota->key_type = KeyType::FORWARDED_IP_ADDRESS; else if (config.has(quota_config + ".keyed")) quota->key_type = KeyType::CLIENT_KEY_OR_USER_NAME; else diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 3f5801e310a..ac735cb3bc3 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -403,6 +403,9 @@ if (USE_MSGPACK) target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${MSGPACK_INCLUDE_DIR}) endif() +target_link_libraries (clickhouse_common_io PUBLIC ${FAST_FLOAT_LIBRARY}) +target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${FAST_FLOAT_INCLUDE_DIR}) + if (USE_ORC) dbms_target_link_libraries(PUBLIC ${ORC_LIBRARIES}) dbms_target_include_directories(SYSTEM BEFORE PUBLIC ${ORC_INCLUDE_DIR} ${CMAKE_BINARY_DIR}/contrib/orc/c++/include) diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index 384c29ed675..1e381808d16 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -528,6 +528,7 @@ M(559, INVALID_GRPC_QUERY_INFO) \ M(560, ZSTD_ENCODER_FAILED) \ M(561, ZSTD_DECODER_FAILED) \ + M(562, TLD_LIST_NOT_FOUND) \ \ M(999, KEEPER_EXCEPTION) \ M(1000, POCO_EXCEPTION) \ diff --git a/src/Common/HashTable/StringHashSet.h b/src/Common/HashTable/StringHashSet.h new file mode 100644 index 00000000000..8714a0e1fe4 --- /dev/null +++ b/src/Common/HashTable/StringHashSet.h @@ -0,0 +1,101 @@ +#pragma once + +#include +#include +#include + +template +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; +}; + +template <> +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; + + bool isZero(const HashTableNoState & state) const { return isZero(this->key, state); } + // Zero means unoccupied cells in hash table. Use key with last word = 0 as + // zero keys, because such keys are unrepresentable (no way to encode length). + static bool isZero(const StringKey16 & key_, const HashTableNoState &) + { return key_.high == 0; } + void setZero() { this->key.high = 0; } +}; + +template <> +struct StringHashSetCell : public HashTableCell +{ + using Base = HashTableCell; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; + + bool isZero(const HashTableNoState & state) const { return isZero(this->key, state); } + // Zero means unoccupied cells in hash table. Use key with last word = 0 as + // zero keys, because such keys are unrepresentable (no way to encode length). + static bool isZero(const StringKey24 & key_, const HashTableNoState &) + { return key_.c == 0; } + void setZero() { this->key.c = 0; } +}; + +template <> +struct StringHashSetCell : public HashSetCellWithSavedHash +{ + using Base = HashSetCellWithSavedHash; + using Base::Base; + + VoidMapped void_map; + VoidMapped & getMapped() { return void_map; } + const VoidMapped & getMapped() const { return void_map; } + + static constexpr bool need_zero_value_storage = false; +}; + +template +struct StringHashSetSubMaps +{ + using T0 = StringHashTableEmpty>; + using T1 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using T2 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using T3 = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; + using Ts = HashSetTable, StringHashTableHash, StringHashTableGrower<>, Allocator>; +}; + +template +class StringHashSet : public StringHashTable> +{ +public: + using Key = StringRef; + using Base = StringHashTable>; + using Self = StringHashSet; + using LookupResult = typename Base::LookupResult; + + using Base::Base; + + template + void ALWAYS_INLINE emplace(KeyHolder && key_holder, bool & inserted) + { + LookupResult it; + Base::emplace(key_holder, it, inserted); + } + +}; diff --git a/src/Common/HashTable/StringHashTable.h b/src/Common/HashTable/StringHashTable.h index 06389825e60..9f91de5585b 100644 --- a/src/Common/HashTable/StringHashTable.h +++ b/src/Common/HashTable/StringHashTable.h @@ -212,7 +212,7 @@ public: using LookupResult = StringHashTableLookupResult; using ConstLookupResult = StringHashTableLookupResult; - StringHashTable() {} + StringHashTable() = default; StringHashTable(size_t reserve_for_num_elements) : m1{reserve_for_num_elements / 4} @@ -222,8 +222,15 @@ public: { } - StringHashTable(StringHashTable && rhs) { *this = std::move(rhs); } - ~StringHashTable() {} + StringHashTable(StringHashTable && rhs) + : m1(std::move(rhs.m1)) + , m2(std::move(rhs.m2)) + , m3(std::move(rhs.m3)) + , ms(std::move(rhs.ms)) + { + } + + ~StringHashTable() = default; public: // Dispatch is written in a way that maximizes the performance: diff --git a/src/Common/PipeFDs.cpp b/src/Common/PipeFDs.cpp index d91917c23a4..a5c21e3d872 100644 --- a/src/Common/PipeFDs.cpp +++ b/src/Common/PipeFDs.cpp @@ -70,7 +70,7 @@ LazyPipeFDs::~LazyPipeFDs() } -void LazyPipeFDs::setNonBlocking() +void LazyPipeFDs::setNonBlockingWrite() { int flags = fcntl(fds_rw[1], F_GETFL, 0); if (-1 == flags) @@ -79,6 +79,21 @@ void LazyPipeFDs::setNonBlocking() throwFromErrno("Cannot set non-blocking mode of pipe", ErrorCodes::CANNOT_FCNTL); } +void LazyPipeFDs::setNonBlockingRead() +{ + int flags = fcntl(fds_rw[0], F_GETFL, 0); + if (-1 == flags) + throwFromErrno("Cannot get file status flags of pipe", ErrorCodes::CANNOT_FCNTL); + if (-1 == fcntl(fds_rw[0], F_SETFL, flags | O_NONBLOCK)) + throwFromErrno("Cannot set non-blocking mode of pipe", ErrorCodes::CANNOT_FCNTL); +} + +void LazyPipeFDs::setNonBlockingReadWrite() +{ + setNonBlockingRead(); + setNonBlockingWrite(); +} + void LazyPipeFDs::tryIncreaseSize(int desired_size) { #if defined(OS_LINUX) diff --git a/src/Common/PipeFDs.h b/src/Common/PipeFDs.h index fe76740da70..20bd847c077 100644 --- a/src/Common/PipeFDs.h +++ b/src/Common/PipeFDs.h @@ -17,7 +17,12 @@ struct LazyPipeFDs void open(); void close(); - void setNonBlocking(); + /// Set O_NONBLOCK to different ends of pipe preserving existing flags. + /// Throws an exception if fcntl was not successful. + void setNonBlockingWrite(); + void setNonBlockingRead(); + void setNonBlockingReadWrite(); + void tryIncreaseSize(int desired_size); ~LazyPipeFDs(); diff --git a/src/Common/StackTrace.cpp b/src/Common/StackTrace.cpp index 9cfdba2687f..b285a45bdc5 100644 --- a/src/Common/StackTrace.cpp +++ b/src/Common/StackTrace.cpp @@ -195,7 +195,8 @@ void StackTrace::symbolize(const StackTrace::FramePointers & frame_pointers, siz { #if defined(__ELF__) && !defined(__FreeBSD__) && !defined(ARCADIA_BUILD) - const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance(); + auto symbol_index_ptr = DB::SymbolIndex::instance(); + const DB::SymbolIndex & symbol_index = *symbol_index_ptr; std::unordered_map dwarfs; for (size_t i = 0; i < offset; ++i) @@ -316,7 +317,8 @@ static void toStringEveryLineImpl( return callback(""); #if defined(__ELF__) && !defined(__FreeBSD__) - const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance(); + auto symbol_index_ptr = DB::SymbolIndex::instance(); + const DB::SymbolIndex & symbol_index = *symbol_index_ptr; std::unordered_map dwarfs; std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM diff --git a/src/Common/SymbolIndex.cpp b/src/Common/SymbolIndex.cpp index a738512bb30..a9b657e671b 100644 --- a/src/Common/SymbolIndex.cpp +++ b/src/Common/SymbolIndex.cpp @@ -300,13 +300,13 @@ void collectSymbolsFromELF(dl_phdr_info * info, String our_build_id = getBuildIDFromProgramHeaders(info); - /// If the name is empty - it's main executable. - /// Find a elf file for the main executable. - + /// If the name is empty and there is a non-empty build-id - it's main executable. + /// Find a elf file for the main executable and set the build-id. if (object_name.empty()) { object_name = "/proc/self/exe"; - build_id = our_build_id; + if (build_id.empty()) + build_id = our_build_id; } std::error_code ec; @@ -316,9 +316,16 @@ void collectSymbolsFromELF(dl_phdr_info * info, return; /// Debug info and symbol table sections may be split to separate binary. + std::filesystem::path local_debug_info_path = canonical_path.parent_path() / canonical_path.stem(); + local_debug_info_path += ".debug"; std::filesystem::path debug_info_path = std::filesystem::path("/usr/lib/debug") / canonical_path.relative_path(); - object_name = std::filesystem::exists(debug_info_path) ? debug_info_path : canonical_path; + if (std::filesystem::exists(local_debug_info_path)) + object_name = local_debug_info_path; + else if (std::filesystem::exists(debug_info_path)) + object_name = debug_info_path; + else + object_name = canonical_path; /// But we have to compare Build ID to check that debug info corresponds to the same executable. @@ -434,10 +441,12 @@ String SymbolIndex::getBuildIDHex() const return build_id_hex; } -SymbolIndex & SymbolIndex::instance() +MultiVersion::Version SymbolIndex::instance(bool reload) { - static SymbolIndex instance; - return instance; + static MultiVersion instance(std::unique_ptr(new SymbolIndex)); + if (reload) + instance.set(std::unique_ptr(new SymbolIndex)); + return instance.get(); } } diff --git a/src/Common/SymbolIndex.h b/src/Common/SymbolIndex.h index 8ef0a949515..b310f90988e 100644 --- a/src/Common/SymbolIndex.h +++ b/src/Common/SymbolIndex.h @@ -7,6 +7,7 @@ #include #include +#include namespace DB { @@ -21,7 +22,7 @@ protected: SymbolIndex() { update(); } public: - static SymbolIndex & instance(); + static MultiVersion::Version instance(bool reload = false); struct Symbol { diff --git a/src/Common/TLDListsHolder.cpp b/src/Common/TLDListsHolder.cpp new file mode 100644 index 00000000000..f0702f37e93 --- /dev/null +++ b/src/Common/TLDListsHolder.cpp @@ -0,0 +1,106 @@ +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int TLD_LIST_NOT_FOUND; +} + +/// +/// TLDList +/// +TLDList::TLDList(size_t size) + : tld_container(size) + , pool(std::make_unique(10 << 20)) +{} +bool TLDList::insert(const StringRef & host) +{ + bool inserted; + tld_container.emplace(DB::ArenaKeyHolder{host, *pool}, inserted); + return inserted; +} +bool TLDList::has(const StringRef & host) const +{ + return tld_container.has(host); +} + +/// +/// TLDListsHolder +/// +TLDListsHolder & TLDListsHolder::getInstance() +{ + static TLDListsHolder instance; + return instance; +} +TLDListsHolder::TLDListsHolder() = default; + +void TLDListsHolder::parseConfig(const std::string & top_level_domains_path, const Poco::Util::AbstractConfiguration & config) +{ + Poco::Util::AbstractConfiguration::Keys config_keys; + config.keys("top_level_domains_lists", config_keys); + + Poco::Logger * log = &Poco::Logger::get("TLDListsHolder"); + + for (const auto & key : config_keys) + { + const std::string & path = top_level_domains_path + config.getString("top_level_domains_lists." + key); + LOG_TRACE(log, "{} loading from {}", key, path); + size_t hosts = parseAndAddTldList(key, path); + LOG_INFO(log, "{} was added ({} hosts)", key, hosts); + } +} + +size_t TLDListsHolder::parseAndAddTldList(const std::string & name, const std::string & path) +{ + std::unordered_set tld_list_tmp; + + ReadBufferFromFile in(path); + while (!in.eof()) + { + char * newline = find_first_symbols<'\n'>(in.position(), in.buffer().end()); + if (newline >= in.buffer().end()) + break; + + std::string_view line(in.position(), newline - in.position()); + in.position() = newline + 1; + + /// Skip comments + if (line.size() > 2 && line[0] == '/' && line[1] == '/') + continue; + trim(line); + /// Skip empty line + if (line.empty()) + continue; + tld_list_tmp.emplace(line); + } + + TLDList tld_list(tld_list_tmp.size()); + for (const auto & host : tld_list_tmp) + { + StringRef host_ref{host.data(), host.size()}; + tld_list.insert(host_ref); + } + + size_t tld_list_size = tld_list.size(); + std::lock_guard lock(tld_lists_map_mutex); + tld_lists_map.insert(std::make_pair(name, std::move(tld_list))); + return tld_list_size; +} + +const TLDList & TLDListsHolder::getTldList(const std::string & name) +{ + std::lock_guard lock(tld_lists_map_mutex); + auto it = tld_lists_map.find(name); + if (it == tld_lists_map.end()) + throw Exception(ErrorCodes::TLD_LIST_NOT_FOUND, "TLD list {} does not exist", name); + return it->second; +} + +} diff --git a/src/Common/TLDListsHolder.h b/src/Common/TLDListsHolder.h new file mode 100644 index 00000000000..3900f9c30d2 --- /dev/null +++ b/src/Common/TLDListsHolder.h @@ -0,0 +1,65 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +/// Custom TLD List +/// +/// Unlike tldLookup (which uses gperf) this one uses plain StringHashSet. +class TLDList +{ +public: + using Container = StringHashSet<>; + + TLDList(size_t size); + + /// Return true if the tld_container does not contains such element. + bool insert(const StringRef & host); + /// Check is there such TLD + bool has(const StringRef & host) const; + size_t size() const { return tld_container.size(); } + +private: + Container tld_container; + std::unique_ptr pool; +}; + +class TLDListsHolder +{ +public: + using Map = std::unordered_map; + + static TLDListsHolder & getInstance(); + + /// Parse "top_level_domains_lists" section, + /// And add each found dictionary. + void parseConfig(const std::string & top_level_domains_path, const Poco::Util::AbstractConfiguration & config); + + /// Parse file and add it as a Set to the list of TLDs + /// - "//" -- comment, + /// - empty lines will be ignored. + /// + /// Example: https://publicsuffix.org/list/public_suffix_list.dat + /// + /// Return size of the list. + size_t parseAndAddTldList(const std::string & name, const std::string & path); + /// Throws TLD_LIST_NOT_FOUND if list does not exist + const TLDList & getTldList(const std::string & name); + +protected: + TLDListsHolder(); + + std::mutex tld_lists_map_mutex; + Map tld_lists_map; +}; + +} diff --git a/src/Common/TraceCollector.cpp b/src/Common/TraceCollector.cpp index 1548af50d98..cbac9cd1a19 100644 --- a/src/Common/TraceCollector.cpp +++ b/src/Common/TraceCollector.cpp @@ -36,7 +36,7 @@ TraceCollector::TraceCollector(std::shared_ptr trace_log_) /** Turn write end of pipe to non-blocking mode to avoid deadlocks * when QueryProfiler is invoked under locks and TraceCollector cannot pull data from pipe. */ - pipe.setNonBlocking(); + pipe.setNonBlockingWrite(); pipe.tryIncreaseSize(1 << 20); thread = ThreadFromGlobalPool(&TraceCollector::run, this); diff --git a/src/Common/ZooKeeper/TestKeeper.cpp b/src/Common/ZooKeeper/TestKeeper.cpp index 61da1137526..5951164f58f 100644 --- a/src/Common/ZooKeeper/TestKeeper.cpp +++ b/src/Common/ZooKeeper/TestKeeper.cpp @@ -31,7 +31,6 @@ using Undo = std::function; struct TestKeeperRequest : virtual Request { - virtual bool isMutable() const { return false; } virtual ResponsePtr createResponse() const = 0; virtual std::pair process(TestKeeper::Container & container, int64_t zxid) const = 0; virtual void processWatches(TestKeeper::Watches & /*watches*/, TestKeeper::Watches & /*list_watches*/) const {} @@ -85,7 +84,6 @@ struct TestKeeperRemoveRequest final : RemoveRequest, TestKeeperRequest { TestKeeperRemoveRequest() = default; explicit TestKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} - bool isMutable() const override { return true; } ResponsePtr createResponse() const override; std::pair process(TestKeeper::Container & container, int64_t zxid) const override; @@ -112,7 +110,6 @@ struct TestKeeperSetRequest final : SetRequest, TestKeeperRequest { TestKeeperSetRequest() = default; explicit TestKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} - bool isMutable() const override { return true; } ResponsePtr createResponse() const override; std::pair process(TestKeeper::Container & container, int64_t zxid) const override; diff --git a/src/Common/ZooKeeper/TestKeeper.h b/src/Common/ZooKeeper/TestKeeper.h index 01c92c98778..ca9f584304f 100644 --- a/src/Common/ZooKeeper/TestKeeper.h +++ b/src/Common/ZooKeeper/TestKeeper.h @@ -125,8 +125,6 @@ private: Watches watches; Watches list_watches; /// Watches for 'list' request (watches on children). - void createWatchCallBack(const String & path); - using RequestsQueue = ConcurrentBoundedQueue; RequestsQueue requests_queue{1}; diff --git a/src/Common/ZooKeeper/TestKeeperStorage.cpp b/src/Common/ZooKeeper/TestKeeperStorage.cpp new file mode 100644 index 00000000000..e3e4edc23a7 --- /dev/null +++ b/src/Common/ZooKeeper/TestKeeperStorage.cpp @@ -0,0 +1,806 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int LOGICAL_ERROR; + extern const int TIMEOUT_EXCEEDED; + extern const int BAD_ARGUMENTS; +} + +} + +namespace zkutil +{ + +using namespace DB; + +static String parentPath(const String & path) +{ + auto rslash_pos = path.rfind('/'); + if (rslash_pos > 0) + return path.substr(0, rslash_pos); + return "/"; +} + +static String baseName(const String & path) +{ + auto rslash_pos = path.rfind('/'); + return path.substr(rslash_pos + 1); +} + +static void processWatchesImpl(const String & path, TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches, Coordination::Event event_type) +{ + auto it = watches.find(path); + if (it != watches.end()) + { + std::shared_ptr watch_response = std::make_shared(); + watch_response->path = path; + watch_response->xid = -1; + watch_response->zxid = -1; + watch_response->type = event_type; + watch_response->state = Coordination::State::CONNECTED; + for (auto & watcher : it->second) + if (watcher.watch_callback) + watcher.watch_callback(watch_response); + + watches.erase(it); + } + + auto parent_path = parentPath(path); + it = list_watches.find(parent_path); + if (it != list_watches.end()) + { + std::shared_ptr watch_list_response = std::make_shared(); + watch_list_response->path = parent_path; + watch_list_response->xid = -1; + watch_list_response->zxid = -1; + watch_list_response->type = Coordination::Event::CHILD; + watch_list_response->state = Coordination::State::CONNECTED; + for (auto & watcher : it->second) + if (watcher.watch_callback) + watcher.watch_callback(watch_list_response); + + list_watches.erase(it); + } +} + +TestKeeperStorage::TestKeeperStorage() +{ + container.emplace("/", Node()); + + processing_thread = ThreadFromGlobalPool([this] { processingThread(); }); +} + +using Undo = std::function; + +struct TestKeeperStorageRequest +{ + Coordination::ZooKeeperRequestPtr zk_request; + + explicit TestKeeperStorageRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + : zk_request(zk_request_) + {} + virtual std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const = 0; + virtual void processWatches(TestKeeperStorage::Watches & /*watches*/, TestKeeperStorage::Watches & /*list_watches*/) const {} + + virtual ~TestKeeperStorageRequest() = default; +}; + +struct TestKeeperStorageHeartbeatRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & /* container */, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /* zxid */, int64_t /* session_id */) const override + { + return {zk_request->makeResponse(), {}}; + } +}; + + +struct TestKeeperStorageCreateRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CREATED); + } + + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Undo undo; + Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperCreateRequest & request = dynamic_cast(*zk_request); + + if (container.count(request.path)) + { + response.error = Coordination::Error::ZNODEEXISTS; + } + else + { + auto it = container.find(parentPath(request.path)); + + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (it->second.is_ephemeral) + { + response.error = Coordination::Error::ZNOCHILDRENFOREPHEMERALS; + } + else + { + TestKeeperStorage::Node created_node; + created_node.seq_num = 0; + created_node.stat.czxid = zxid; + created_node.stat.mzxid = zxid; + created_node.stat.ctime = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1); + created_node.stat.mtime = created_node.stat.ctime; + created_node.stat.numChildren = 0; + created_node.stat.dataLength = request.data.length(); + created_node.data = request.data; + created_node.is_ephemeral = request.is_ephemeral; + created_node.is_sequental = request.is_sequential; + std::string path_created = request.path; + + if (request.is_sequential) + { + auto seq_num = it->second.seq_num; + + std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM + seq_num_str.exceptions(std::ios::failbit); + seq_num_str << std::setw(10) << std::setfill('0') << seq_num; + + path_created += seq_num_str.str(); + } + + /// Increment sequential number even if node is not sequential + ++it->second.seq_num; + + response.path_created = path_created; + container.emplace(path_created, std::move(created_node)); + + if (request.is_ephemeral) + ephemerals[session_id].emplace(path_created); + + undo = [&container, &ephemerals, session_id, path_created, is_ephemeral = request.is_ephemeral, parent_path = it->first] + { + container.erase(path_created); + if (is_ephemeral) + ephemerals[session_id].erase(path_created); + auto & undo_parent = container.at(parent_path); + --undo_parent.stat.cversion; + --undo_parent.stat.numChildren; + --undo_parent.seq_num; + }; + + ++it->second.stat.cversion; + ++it->second.stat.numChildren; + + response.error = Coordination::Error::ZOK; + } + } + + return { response_ptr, undo }; + } +}; + +struct TestKeeperStorageGetRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /* zxid */, int64_t /* session_id */) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request); + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else + { + response.stat = it->second.stat; + response.data = it->second.data; + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageRemoveRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t /*zxid*/, int64_t session_id) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperRemoveResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperRemoveRequest & request = dynamic_cast(*zk_request); + Undo undo; + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version != -1 && request.version != it->second.stat.version) + { + response.error = Coordination::Error::ZBADVERSION; + } + else if (it->second.stat.numChildren) + { + response.error = Coordination::Error::ZNOTEMPTY; + } + else + { + auto prev_node = it->second; + if (prev_node.is_ephemeral) + ephemerals[session_id].erase(request.path); + + container.erase(it); + auto & parent = container.at(parentPath(request.path)); + --parent.stat.numChildren; + ++parent.stat.cversion; + response.error = Coordination::Error::ZOK; + + undo = [prev_node, &container, &ephemerals, session_id, path = request.path] + { + if (prev_node.is_ephemeral) + ephemerals[session_id].emplace(path); + + container.emplace(path, prev_node); + auto & undo_parent = container.at(parentPath(path)); + ++undo_parent.stat.numChildren; + --undo_parent.stat.cversion; + }; + } + + return { response_ptr, undo }; + } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::DELETED); + } +}; + +struct TestKeeperStorageExistsRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /* session_id */) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperExistsResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperExistsRequest & request = dynamic_cast(*zk_request); + + auto it = container.find(request.path); + if (it != container.end()) + { + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + } + else + { + response.error = Coordination::Error::ZNONODE; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageSetRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t zxid, int64_t /* session_id */) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperSetResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperSetRequest & request = dynamic_cast(*zk_request); + Undo undo; + + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version == -1 || request.version == it->second.stat.version) + { + auto prev_node = it->second; + + it->second.data = request.data; + ++it->second.stat.version; + it->second.stat.mzxid = zxid; + it->second.stat.mtime = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1); + it->second.stat.dataLength = request.data.length(); + it->second.data = request.data; + ++container.at(parentPath(request.path)).stat.cversion; + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + + undo = [prev_node, &container, path = request.path] + { + container.at(path) = prev_node; + --container.at(parentPath(path)).stat.cversion; + }; + } + else + { + response.error = Coordination::Error::ZBADVERSION; + } + + return { response_ptr, undo }; + } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CHANGED); + } + +}; + +struct TestKeeperStorageListRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /*session_id*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperListResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperListRequest & request = dynamic_cast(*zk_request); + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else + { + auto path_prefix = request.path; + if (path_prefix.empty()) + throw DB::Exception("Logical error: path cannot be empty", ErrorCodes::LOGICAL_ERROR); + + if (path_prefix.back() != '/') + path_prefix += '/'; + + /// Fairly inefficient. + for (auto child_it = container.upper_bound(path_prefix); + child_it != container.end() && startsWith(child_it->first, path_prefix); + ++child_it) + { + if (parentPath(child_it->first) == request.path) + response.names.emplace_back(baseName(child_it->first)); + } + + response.stat = it->second.stat; + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageCheckRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & /* ephemerals */, int64_t /*zxid*/, int64_t /*session_id*/) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperCheckResponse & response = dynamic_cast(*response_ptr); + Coordination::ZooKeeperCheckRequest & request = dynamic_cast(*zk_request); + auto it = container.find(request.path); + if (it == container.end()) + { + response.error = Coordination::Error::ZNONODE; + } + else if (request.version != -1 && request.version != it->second.stat.version) + { + response.error = Coordination::Error::ZBADVERSION; + } + else + { + response.error = Coordination::Error::ZOK; + } + + return { response_ptr, {} }; + } +}; + +struct TestKeeperStorageMultiRequest final : public TestKeeperStorageRequest +{ + std::vector concrete_requests; + explicit TestKeeperStorageMultiRequest(const Coordination::ZooKeeperRequestPtr & zk_request_) + : TestKeeperStorageRequest(zk_request_) + { + Coordination::ZooKeeperMultiRequest & request = dynamic_cast(*zk_request); + concrete_requests.reserve(request.requests.size()); + + for (const auto & sub_request : request.requests) + { + auto sub_zk_request = std::dynamic_pointer_cast(sub_request); + if (sub_zk_request->getOpNum() == Coordination::OpNum::Create) + { + concrete_requests.push_back(std::make_shared(sub_zk_request)); + } + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Remove) + { + concrete_requests.push_back(std::make_shared(sub_zk_request)); + } + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Set) + { + concrete_requests.push_back(std::make_shared(sub_zk_request)); + } + else if (sub_zk_request->getOpNum() == Coordination::OpNum::Check) + { + concrete_requests.push_back(std::make_shared(sub_zk_request)); + } + else + throw DB::Exception(ErrorCodes::BAD_ARGUMENTS, "Illegal command as part of multi ZooKeeper request {}", sub_zk_request->getOpNum()); + } + } + + std::pair process(TestKeeperStorage::Container & container, TestKeeperStorage::Ephemerals & ephemerals, int64_t zxid, int64_t session_id) const override + { + Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse(); + Coordination::ZooKeeperMultiResponse & response = dynamic_cast(*response_ptr); + std::vector undo_actions; + + try + { + size_t i = 0; + for (const auto & concrete_request : concrete_requests) + { + auto [ cur_response, undo_action ] = concrete_request->process(container, ephemerals, zxid, session_id); + + response.responses[i] = cur_response; + if (cur_response->error != Coordination::Error::ZOK) + { + for (size_t j = 0; j <= i; ++j) + { + auto response_error = response.responses[j]->error; + response.responses[j] = std::make_shared(); + response.responses[j]->error = response_error; + } + + for (size_t j = i + 1; j < response.responses.size(); ++j) + { + response.responses[j] = std::make_shared(); + response.responses[j]->error = Coordination::Error::ZRUNTIMEINCONSISTENCY; + } + + for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) + if (*it) + (*it)(); + + return { response_ptr, {} }; + } + else + undo_actions.emplace_back(std::move(undo_action)); + + ++i; + } + + response.error = Coordination::Error::ZOK; + return { response_ptr, {} }; + } + catch (...) + { + for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it) + if (*it) + (*it)(); + throw; + } + } + + void processWatches(TestKeeperStorage::Watches & watches, TestKeeperStorage::Watches & list_watches) const override + { + for (const auto & generic_request : concrete_requests) + generic_request->processWatches(watches, list_watches); + } +}; + +struct TestKeeperStorageCloseRequest final : public TestKeeperStorageRequest +{ + using TestKeeperStorageRequest::TestKeeperStorageRequest; + std::pair process(TestKeeperStorage::Container &, TestKeeperStorage::Ephemerals &, int64_t, int64_t) const override + { + throw DB::Exception("Called process on close request", ErrorCodes::LOGICAL_ERROR); + } +}; + +void TestKeeperStorage::processingThread() +{ + setThreadName("TestKeeperSProc"); + + try + { + while (!shutdown) + { + RequestInfo info; + + UInt64 max_wait = UInt64(operation_timeout.totalMilliseconds()); + + if (requests_queue.tryPop(info, max_wait)) + { + if (shutdown) + break; + + auto zk_request = info.request->zk_request; + if (zk_request->getOpNum() == Coordination::OpNum::Close) + { + auto it = ephemerals.find(info.session_id); + if (it != ephemerals.end()) + { + for (const auto & ephemeral_path : it->second) + { + container.erase(ephemeral_path); + processWatchesImpl(ephemeral_path, watches, list_watches, Coordination::Event::DELETED); + } + ephemerals.erase(it); + } + clearDeadWatches(info.session_id); + + /// Finish connection + auto response = std::make_shared(); + response->xid = zk_request->xid; + response->zxid = getZXID(); + info.response_callback(response); + } + else + { + auto [response, _] = info.request->process(container, ephemerals, zxid, info.session_id); + + if (info.watch_callback) + { + if (response->error == Coordination::Error::ZOK) + { + auto & watches_type = zk_request->getOpNum() == Coordination::OpNum::List || zk_request->getOpNum() == Coordination::OpNum::SimpleList + ? list_watches + : watches; + + watches_type[zk_request->getPath()].emplace_back(Watcher{info.session_id, info.watch_callback}); + sessions_and_watchers[info.session_id].emplace(zk_request->getPath()); + } + else if (response->error == Coordination::Error::ZNONODE && zk_request->getOpNum() == Coordination::OpNum::Exists) + { + watches[zk_request->getPath()].emplace_back(Watcher{info.session_id, info.watch_callback}); + sessions_and_watchers[info.session_id].emplace(zk_request->getPath()); + } + else + { + std::shared_ptr watch_response = std::make_shared(); + watch_response->path = zk_request->getPath(); + watch_response->xid = -1; + watch_response->error = response->error; + watch_response->type = Coordination::Event::NOTWATCHING; + info.watch_callback(watch_response); + } + } + + if (response->error == Coordination::Error::ZOK) + info.request->processWatches(watches, list_watches); + + response->xid = zk_request->xid; + response->zxid = getZXID(); + + info.response_callback(response); + } + } + } + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + finalize(); + } +} + + +void TestKeeperStorage::finalize() +{ + { + std::lock_guard lock(push_request_mutex); + + if (shutdown) + return; + + shutdown = true; + + if (processing_thread.joinable()) + processing_thread.join(); + } + + try + { + { + auto finish_watch = [] (const auto & watch_pair) + { + Coordination::ZooKeeperWatchResponse response; + response.type = Coordination::SESSION; + response.state = Coordination::EXPIRED_SESSION; + response.error = Coordination::Error::ZSESSIONEXPIRED; + + for (auto & watcher : watch_pair.second) + { + if (watcher.watch_callback) + { + try + { + watcher.watch_callback(std::make_shared(response)); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } + } + } + }; + for (auto & path_watch : watches) + finish_watch(path_watch); + watches.clear(); + for (auto & path_watch : list_watches) + finish_watch(path_watch); + list_watches.clear(); + sessions_and_watchers.clear(); + } + RequestInfo info; + while (requests_queue.tryPop(info)) + { + auto response = info.request->zk_request->makeResponse(); + response->error = Coordination::Error::ZSESSIONEXPIRED; + try + { + info.response_callback(response); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } + } + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } +} + + +class TestKeeperWrapperFactory final : private boost::noncopyable +{ + +public: + using Creator = std::function; + using OpNumToRequest = std::unordered_map; + + static TestKeeperWrapperFactory & instance() + { + static TestKeeperWrapperFactory factory; + return factory; + } + + TestKeeperStorageRequestPtr get(const Coordination::ZooKeeperRequestPtr & zk_request) const + { + auto it = op_num_to_request.find(zk_request->getOpNum()); + if (it == op_num_to_request.end()) + throw DB::Exception("Unknown operation type " + toString(zk_request->getOpNum()), ErrorCodes::LOGICAL_ERROR); + + return it->second(zk_request); + } + + void registerRequest(Coordination::OpNum op_num, Creator creator) + { + if (!op_num_to_request.try_emplace(op_num, creator).second) + throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Request with op num {} already registered", op_num); + } + +private: + OpNumToRequest op_num_to_request; + TestKeeperWrapperFactory(); +}; + +template +void registerTestKeeperRequestWrapper(TestKeeperWrapperFactory & factory) +{ + factory.registerRequest(num, [] (const Coordination::ZooKeeperRequestPtr & zk_request) { return std::make_shared(zk_request); }); +} + + +TestKeeperWrapperFactory::TestKeeperWrapperFactory() +{ + registerTestKeeperRequestWrapper(*this); + //registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); + registerTestKeeperRequestWrapper(*this); +} + +void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback) +{ + TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); + RequestInfo request_info; + request_info.time = clock::now(); + request_info.request = storage_request; + request_info.session_id = session_id; + request_info.response_callback = callback; + + /// Put close requests without timeouts + auto timeout = request->getOpNum() == Coordination::OpNum::Close ? 0 : operation_timeout.totalMilliseconds(); + std::lock_guard lock(push_request_mutex); + if (!requests_queue.tryPush(std::move(request_info), timeout)) + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); +} + +void TestKeeperStorage::putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback) +{ + TestKeeperStorageRequestPtr storage_request = TestKeeperWrapperFactory::instance().get(request); + RequestInfo request_info; + request_info.time = clock::now(); + request_info.request = storage_request; + request_info.session_id = session_id; + request_info.response_callback = callback; + if (request->has_watch) + request_info.watch_callback = watch_callback; + + /// Put close requests without timeouts + auto timeout = request->getOpNum() == Coordination::OpNum::Close ? 0 : operation_timeout.totalMilliseconds(); + std::lock_guard lock(push_request_mutex); + if (!requests_queue.tryPush(std::move(request_info), timeout)) + throw Exception("Cannot push request to queue within operation timeout", ErrorCodes::TIMEOUT_EXCEEDED); +} + +TestKeeperStorage::~TestKeeperStorage() +{ + try + { + finalize(); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } +} + +void TestKeeperStorage::clearDeadWatches(int64_t session_id) +{ + auto watches_it = sessions_and_watchers.find(session_id); + if (watches_it != sessions_and_watchers.end()) + { + for (const auto & watch_path : watches_it->second) + { + auto watch = watches.find(watch_path); + if (watch != watches.end()) + { + auto & watches_for_path = watch->second; + for (auto w_it = watches_for_path.begin(); w_it != watches_for_path.end();) + { + if (w_it->session_id == session_id) + w_it = watches_for_path.erase(w_it); + else + ++w_it; + } + if (watches_for_path.empty()) + watches.erase(watch); + } + } + sessions_and_watchers.erase(watches_it); + } +} + +} diff --git a/src/Common/ZooKeeper/TestKeeperStorage.h b/src/Common/ZooKeeper/TestKeeperStorage.h new file mode 100644 index 00000000000..afb0a7add82 --- /dev/null +++ b/src/Common/ZooKeeper/TestKeeperStorage.h @@ -0,0 +1,104 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace zkutil +{ + +using namespace DB; +struct TestKeeperStorageRequest; +using TestKeeperStorageRequestPtr = std::shared_ptr; +using ResponseCallback = std::function; + +class TestKeeperStorage +{ + +public: + + Poco::Timespan operation_timeout{0, Coordination::DEFAULT_OPERATION_TIMEOUT_MS * 1000}; + std::atomic session_id_counter{0}; + + struct Node + { + String data; + Coordination::ACLs acls; + bool is_ephemeral = false; + bool is_sequental = false; + Coordination::Stat stat{}; + int32_t seq_num = 0; + }; + + struct Watcher + { + int64_t session_id; + ResponseCallback watch_callback; + }; + + using Container = std::map; + using Ephemerals = std::unordered_map>; + using SessionAndWatcher = std::unordered_map>; + + using WatchCallbacks = std::vector; + using Watches = std::map; + + Container container; + Ephemerals ephemerals; + SessionAndWatcher sessions_and_watchers; + + std::atomic zxid{0}; + std::atomic shutdown{false}; + + Watches watches; + Watches list_watches; /// Watches for 'list' request (watches on children). + + using clock = std::chrono::steady_clock; + + struct RequestInfo + { + TestKeeperStorageRequestPtr request; + ResponseCallback response_callback; + ResponseCallback watch_callback; + clock::time_point time; + int64_t session_id; + }; + + std::mutex push_request_mutex; + using RequestsQueue = ConcurrentBoundedQueue; + RequestsQueue requests_queue{1}; + + void finalize(); + + ThreadFromGlobalPool processing_thread; + + void processingThread(); + void clearDeadWatches(int64_t session_id); + +public: + using AsyncResponse = std::future; + TestKeeperStorage(); + ~TestKeeperStorage(); + struct ResponsePair + { + AsyncResponse response; + std::optional watch_response; + }; + void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback); + void putRequest(const Coordination::ZooKeeperRequestPtr & request, int64_t session_id, ResponseCallback callback, ResponseCallback watch_callback); + + int64_t getSessionID() + { + return session_id_counter.fetch_add(1); + } + int64_t getZXID() + { + return zxid.fetch_add(1); + } +}; + +} diff --git a/src/Common/ZooKeeper/ZooKeeper.cpp b/src/Common/ZooKeeper/ZooKeeper.cpp index bee875d1c74..b3a0a082a9f 100644 --- a/src/Common/ZooKeeper/ZooKeeper.cpp +++ b/src/Common/ZooKeeper/ZooKeeper.cpp @@ -129,8 +129,8 @@ struct ZooKeeperArgs std::vector hosts_strings; - session_timeout_ms = DEFAULT_SESSION_TIMEOUT; - operation_timeout_ms = DEFAULT_OPERATION_TIMEOUT; + session_timeout_ms = Coordination::DEFAULT_SESSION_TIMEOUT_MS; + operation_timeout_ms = Coordination::DEFAULT_OPERATION_TIMEOUT_MS; implementation = "zookeeper"; for (const auto & key : keys) { diff --git a/src/Common/ZooKeeper/ZooKeeper.h b/src/Common/ZooKeeper/ZooKeeper.h index b1a69646db5..0d9dc104c48 100644 --- a/src/Common/ZooKeeper/ZooKeeper.h +++ b/src/Common/ZooKeeper/ZooKeeper.h @@ -11,6 +11,7 @@ #include #include #include +#include #include @@ -28,9 +29,6 @@ namespace CurrentMetrics namespace zkutil { -const UInt32 DEFAULT_SESSION_TIMEOUT = 30000; -const UInt32 DEFAULT_OPERATION_TIMEOUT = 10000; - /// Preferred size of multi() command (in number of ops) constexpr size_t MULTI_BATCH_SIZE = 100; @@ -53,8 +51,8 @@ public: using Ptr = std::shared_ptr; ZooKeeper(const std::string & hosts_, const std::string & identity_ = "", - int32_t session_timeout_ms_ = DEFAULT_SESSION_TIMEOUT, - int32_t operation_timeout_ms_ = DEFAULT_OPERATION_TIMEOUT, + int32_t session_timeout_ms_ = Coordination::DEFAULT_SESSION_TIMEOUT_MS, + int32_t operation_timeout_ms_ = Coordination::DEFAULT_OPERATION_TIMEOUT_MS, const std::string & chroot_ = "", const std::string & implementation_ = "zookeeper"); diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.cpp b/src/Common/ZooKeeper/ZooKeeperCommon.cpp new file mode 100644 index 00000000000..9c699ee298a --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperCommon.cpp @@ -0,0 +1,481 @@ +#include +#include +#include +#include +#include +#include +#include +#include + + +namespace Coordination +{ + +using namespace DB; + +void ZooKeeperResponse::write(WriteBuffer & out) const +{ + /// Excessive copy to calculate length. + WriteBufferFromOwnString buf; + Coordination::write(xid, buf); + Coordination::write(zxid, buf); + Coordination::write(error, buf); + if (error == Error::ZOK) + writeImpl(buf); + Coordination::write(buf.str(), out); + out.next(); +} + +void ZooKeeperRequest::write(WriteBuffer & out) const +{ + /// Excessive copy to calculate length. + WriteBufferFromOwnString buf; + Coordination::write(xid, buf); + Coordination::write(getOpNum(), buf); + writeImpl(buf); + Coordination::write(buf.str(), out); + out.next(); +} + +void ZooKeeperWatchResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(type, in); + Coordination::read(state, in); + Coordination::read(path, in); +} + +void ZooKeeperWatchResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(type, out); + Coordination::write(state, out); + Coordination::write(path, out); +} + +void ZooKeeperAuthRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(type, out); + Coordination::write(scheme, out); + Coordination::write(data, out); +} + +void ZooKeeperAuthRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(type, in); + Coordination::read(scheme, in); + Coordination::read(data, in); +} + +void ZooKeeperCreateRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(data, out); + Coordination::write(acls, out); + + int32_t flags = 0; + + if (is_ephemeral) + flags |= 1; + if (is_sequential) + flags |= 2; + + Coordination::write(flags, out); +} + +void ZooKeeperCreateRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(data, in); + Coordination::read(acls, in); + + int32_t flags = 0; + Coordination::read(flags, in); + + if (flags & 1) + is_ephemeral = true; + if (flags & 2) + is_sequential = true; +} + +void ZooKeeperCreateResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(path_created, in); +} + +void ZooKeeperCreateResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path_created, out); +} + +void ZooKeeperRemoveRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(version, out); +} + +void ZooKeeperRemoveRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(version, in); +} + +void ZooKeeperExistsRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperExistsRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperExistsResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(stat, in); +} + +void ZooKeeperExistsResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(stat, out); +} + +void ZooKeeperGetRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperGetRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperGetResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(data, in); + Coordination::read(stat, in); +} + +void ZooKeeperGetResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(data, out); + Coordination::write(stat, out); +} + +void ZooKeeperSetRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(data, out); + Coordination::write(version, out); +} + +void ZooKeeperSetRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(data, in); + Coordination::read(version, in); +} + +void ZooKeeperSetResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(stat, in); +} + +void ZooKeeperSetResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(stat, out); +} + +void ZooKeeperListRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(has_watch, out); +} + +void ZooKeeperListRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(has_watch, in); +} + +void ZooKeeperListResponse::readImpl(ReadBuffer & in) +{ + Coordination::read(names, in); + Coordination::read(stat, in); +} + +void ZooKeeperListResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(names, out); + Coordination::write(stat, out); +} + +void ZooKeeperCheckRequest::writeImpl(WriteBuffer & out) const +{ + Coordination::write(path, out); + Coordination::write(version, out); +} + +void ZooKeeperCheckRequest::readImpl(ReadBuffer & in) +{ + Coordination::read(path, in); + Coordination::read(version, in); +} + +void ZooKeeperErrorResponse::readImpl(ReadBuffer & in) +{ + Coordination::Error read_error; + Coordination::read(read_error, in); + + if (read_error != error) + throw Exception(fmt::format("Error code in ErrorResponse ({}) doesn't match error code in header ({})", read_error, error), + Error::ZMARSHALLINGERROR); +} + +void ZooKeeperErrorResponse::writeImpl(WriteBuffer & out) const +{ + Coordination::write(error, out); +} + +ZooKeeperMultiRequest::ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls) +{ + /// Convert nested Requests to ZooKeeperRequests. + /// Note that deep copy is required to avoid modifying path in presence of chroot prefix. + requests.reserve(generic_requests.size()); + + for (const auto & generic_request : generic_requests) + { + if (const auto * concrete_request_create = dynamic_cast(generic_request.get())) + { + auto create = std::make_shared(*concrete_request_create); + if (create->acls.empty()) + create->acls = default_acls; + requests.push_back(create); + } + else if (const auto * concrete_request_remove = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_remove)); + } + else if (const auto * concrete_request_set = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_set)); + } + else if (const auto * concrete_request_check = dynamic_cast(generic_request.get())) + { + requests.push_back(std::make_shared(*concrete_request_check)); + } + else + throw Exception("Illegal command as part of multi ZooKeeper request", Error::ZBADARGUMENTS); + } +} + +void ZooKeeperMultiRequest::writeImpl(WriteBuffer & out) const +{ + for (const auto & request : requests) + { + const auto & zk_request = dynamic_cast(*request); + + bool done = false; + int32_t error = -1; + + Coordination::write(zk_request.getOpNum(), out); + Coordination::write(done, out); + Coordination::write(error, out); + + zk_request.writeImpl(out); + } + + OpNum op_num = OpNum::Error; + bool done = true; + int32_t error = -1; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(error, out); +} + +void ZooKeeperMultiRequest::readImpl(ReadBuffer & in) +{ + + while (true) + { + OpNum op_num; + bool done; + int32_t error; + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(error, in); + + if (done) + { + if (op_num != OpNum::Error) + throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + if (error != -1) + throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + break; + } + + ZooKeeperRequestPtr request = ZooKeeperRequestFactory::instance().get(op_num); + request->readImpl(in); + requests.push_back(request); + + if (in.eof()) + throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); + } +} + +void ZooKeeperMultiResponse::readImpl(ReadBuffer & in) +{ + for (auto & response : responses) + { + OpNum op_num; + bool done; + Error op_error; + + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(op_error, in); + + if (done) + throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); + + /// op_num == -1 is special for multi transaction. + /// For unknown reason, error code is duplicated in header and in response body. + + if (op_num == OpNum::Error) + response = std::make_shared(); + + if (op_error != Error::ZOK) + { + response->error = op_error; + + /// Set error for whole transaction. + /// If some operations fail, ZK send global error as zero and then send details about each operation. + /// It will set error code for first failed operation and it will set special "runtime inconsistency" code for other operations. + if (error == Error::ZOK && op_error != Error::ZRUNTIMEINCONSISTENCY) + error = op_error; + } + + if (op_error == Error::ZOK || op_num == OpNum::Error) + dynamic_cast(*response).readImpl(in); + } + + /// Footer. + { + OpNum op_num; + bool done; + int32_t error_read; + + Coordination::read(op_num, in); + Coordination::read(done, in); + Coordination::read(error_read, in); + + if (!done) + throw Exception("Too many results received for multi transaction", Error::ZMARSHALLINGERROR); + if (op_num != OpNum::Error) + throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + if (error_read != -1) + throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); + } +} + +void ZooKeeperMultiResponse::writeImpl(WriteBuffer & out) const +{ + for (const auto & response : responses) + { + const ZooKeeperResponse & zk_response = dynamic_cast(*response); + OpNum op_num = zk_response.getOpNum(); + bool done = false; + Error op_error = zk_response.error; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(op_error, out); + if (op_error == Error::ZOK || op_num == OpNum::Error) + zk_response.writeImpl(out); + } + + /// Footer. + { + OpNum op_num = OpNum::Error; + bool done = true; + int32_t error_read = - 1; + + Coordination::write(op_num, out); + Coordination::write(done, out); + Coordination::write(error_read, out); + } +} + +ZooKeeperResponsePtr ZooKeeperHeartbeatRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperAuthRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperCreateRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperRemoveRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperExistsRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperGetRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperSetRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperListRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperCheckRequest::makeResponse() const { return std::make_shared(); } +ZooKeeperResponsePtr ZooKeeperMultiRequest::makeResponse() const { return std::make_shared(requests); } +ZooKeeperResponsePtr ZooKeeperCloseRequest::makeResponse() const { return std::make_shared(); } + +void ZooKeeperRequestFactory::registerRequest(OpNum op_num, Creator creator) +{ + if (!op_num_to_request.try_emplace(op_num, creator).second) + throw Coordination::Exception("Request type " + toString(op_num) + " already registered", Coordination::Error::ZRUNTIMEINCONSISTENCY); +} + +std::shared_ptr ZooKeeperRequest::read(ReadBuffer & in) +{ + XID xid; + OpNum op_num; + + Coordination::read(xid, in); + Coordination::read(op_num, in); + + auto request = ZooKeeperRequestFactory::instance().get(op_num); + request->xid = xid; + request->readImpl(in); + return request; +} + +ZooKeeperRequestPtr ZooKeeperRequestFactory::get(OpNum op_num) const +{ + auto it = op_num_to_request.find(op_num); + if (it == op_num_to_request.end()) + throw Exception("Unknown operation type " + toString(op_num), Error::ZBADARGUMENTS); + + return it->second(); +} + +ZooKeeperRequestFactory & ZooKeeperRequestFactory::instance() +{ + static ZooKeeperRequestFactory factory; + return factory; +} + +template +void registerZooKeeperRequest(ZooKeeperRequestFactory & factory) +{ + factory.registerRequest(num, [] { return std::make_shared(); }); +} + +ZooKeeperRequestFactory::ZooKeeperRequestFactory() +{ + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); + registerZooKeeperRequest(*this); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperCommon.h b/src/Common/ZooKeeper/ZooKeeperCommon.h new file mode 100644 index 00000000000..9adb0c06e4c --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperCommon.h @@ -0,0 +1,338 @@ +#pragma once + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + + +namespace Coordination +{ + +struct ZooKeeperResponse : virtual Response +{ + XID xid = 0; + int64_t zxid; + + virtual ~ZooKeeperResponse() override = default; + virtual void readImpl(ReadBuffer &) = 0; + virtual void writeImpl(WriteBuffer &) const = 0; + void write(WriteBuffer & out) const; + virtual OpNum getOpNum() const = 0; +}; + +using ZooKeeperResponsePtr = std::shared_ptr; + +/// Exposed in header file for Yandex.Metrica code. +struct ZooKeeperRequest : virtual Request +{ + XID xid = 0; + bool has_watch = false; + /// If the request was not send and the error happens, we definitely sure, that it has not been processed by the server. + /// If the request was sent and we didn't get the response and the error happens, then we cannot be sure was it processed or not. + bool probably_sent = false; + + ZooKeeperRequest() = default; + ZooKeeperRequest(const ZooKeeperRequest &) = default; + virtual ~ZooKeeperRequest() override = default; + + virtual OpNum getOpNum() const = 0; + + /// Writes length, xid, op_num, then the rest. + void write(WriteBuffer & out) const; + + virtual void writeImpl(WriteBuffer &) const = 0; + virtual void readImpl(ReadBuffer &) = 0; + + static std::shared_ptr read(ReadBuffer & in); + + virtual ZooKeeperResponsePtr makeResponse() const = 0; +}; + +using ZooKeeperRequestPtr = std::shared_ptr; + +struct ZooKeeperHeartbeatRequest final : ZooKeeperRequest +{ + String getPath() const override { return {}; } + OpNum getOpNum() const override { return OpNum::Heartbeat; } + void writeImpl(WriteBuffer &) const override {} + void readImpl(ReadBuffer &) override {} + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperHeartbeatResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return OpNum::Heartbeat; } +}; + +struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + + OpNum getOpNum() const override + { + throw Exception("OpNum for watch response doesn't exist", Error::ZRUNTIMEINCONSISTENCY); + } +}; + +struct ZooKeeperAuthRequest final : ZooKeeperRequest +{ + int32_t type = 0; /// ignored by the server + String scheme; + String data; + + String getPath() const override { return {}; } + OpNum getOpNum() const override { return OpNum::Auth; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperAuthResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + + OpNum getOpNum() const override { return OpNum::Auth; } +}; + +struct ZooKeeperCloseRequest final : ZooKeeperRequest +{ + String getPath() const override { return {}; } + OpNum getOpNum() const override { return OpNum::Close; } + void writeImpl(WriteBuffer &) const override {} + void readImpl(ReadBuffer &) override {} + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCloseResponse final : ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override + { + throw Exception("Received response for close request", Error::ZRUNTIMEINCONSISTENCY); + } + + void writeImpl(WriteBuffer &) const override {} + + OpNum getOpNum() const override { return OpNum::Close; } +}; + +struct ZooKeeperCreateRequest final : public CreateRequest, ZooKeeperRequest +{ + ZooKeeperCreateRequest() = default; + explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} + + OpNum getOpNum() const override { return OpNum::Create; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCreateResponse final : CreateResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + + OpNum getOpNum() const override { return OpNum::Create; } +}; + +struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest +{ + ZooKeeperRemoveRequest() = default; + explicit ZooKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} + + OpNum getOpNum() const override { return OpNum::Remove; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperRemoveResponse final : RemoveResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return OpNum::Remove; } +}; + +struct ZooKeeperExistsRequest final : ExistsRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return OpNum::Exists; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperExistsResponse final : ExistsResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return OpNum::Exists; } +}; + +struct ZooKeeperGetRequest final : GetRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return OpNum::Get; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperGetResponse final : GetResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return OpNum::Get; } +}; + +struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest +{ + ZooKeeperSetRequest() = default; + explicit ZooKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} + + OpNum getOpNum() const override { return OpNum::Set; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return OpNum::Set; } +}; + +struct ZooKeeperListRequest : ListRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return OpNum::List; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperSimpleListRequest final : ZooKeeperListRequest +{ + OpNum getOpNum() const override { return OpNum::SimpleList; } +}; + +struct ZooKeeperListResponse : ListResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + OpNum getOpNum() const override { return OpNum::List; } +}; + +struct ZooKeeperSimpleListResponse final : ZooKeeperListResponse +{ + OpNum getOpNum() const override { return OpNum::SimpleList; } +}; + +struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest +{ + ZooKeeperCheckRequest() = default; + explicit ZooKeeperCheckRequest(const CheckRequest & base) : CheckRequest(base) {} + + OpNum getOpNum() const override { return OpNum::Check; } + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperCheckResponse final : CheckResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer &) override {} + void writeImpl(WriteBuffer &) const override {} + OpNum getOpNum() const override { return OpNum::Check; } +}; + +/// This response may be received only as an element of responses in MultiResponse. +struct ZooKeeperErrorResponse final : ErrorResponse, ZooKeeperResponse +{ + void readImpl(ReadBuffer & in) override; + void writeImpl(WriteBuffer & out) const override; + + OpNum getOpNum() const override { return OpNum::Error; } +}; + +struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest +{ + OpNum getOpNum() const override { return OpNum::Multi; } + ZooKeeperMultiRequest() = default; + + ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls); + + void writeImpl(WriteBuffer & out) const override; + void readImpl(ReadBuffer & in) override; + + ZooKeeperResponsePtr makeResponse() const override; +}; + +struct ZooKeeperMultiResponse final : MultiResponse, ZooKeeperResponse +{ + OpNum getOpNum() const override { return OpNum::Multi; } + + explicit ZooKeeperMultiResponse(const Requests & requests) + { + responses.reserve(requests.size()); + + for (const auto & request : requests) + responses.emplace_back(dynamic_cast(*request).makeResponse()); + } + + explicit ZooKeeperMultiResponse(const Responses & responses_) + { + responses = responses_; + } + + void readImpl(ReadBuffer & in) override; + + void writeImpl(WriteBuffer & out) const override; + +}; + +class ZooKeeperRequestFactory final : private boost::noncopyable +{ + +public: + using Creator = std::function; + using OpNumToRequest = std::unordered_map; + + static ZooKeeperRequestFactory & instance(); + + ZooKeeperRequestPtr get(OpNum op_num) const; + + void registerRequest(OpNum op_num, Creator creator); + +private: + OpNumToRequest op_num_to_request; + +private: + ZooKeeperRequestFactory(); +}; + +} diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.cpp b/src/Common/ZooKeeper/ZooKeeperConstants.cpp new file mode 100644 index 00000000000..b4cb9feb518 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperConstants.cpp @@ -0,0 +1,67 @@ +#include +#include +#include + +namespace Coordination +{ + +static const std::unordered_set VALID_OPERATIONS = +{ + static_cast(OpNum::Close), + static_cast(OpNum::Error), + static_cast(OpNum::Create), + static_cast(OpNum::Remove), + static_cast(OpNum::Exists), + static_cast(OpNum::Get), + static_cast(OpNum::Set), + static_cast(OpNum::SimpleList), + static_cast(OpNum::Heartbeat), + static_cast(OpNum::List), + static_cast(OpNum::Check), + static_cast(OpNum::Multi), + static_cast(OpNum::Auth), +}; + +std::string toString(OpNum op_num) +{ + switch (op_num) + { + case OpNum::Close: + return "Close"; + case OpNum::Error: + return "Error"; + case OpNum::Create: + return "Create"; + case OpNum::Remove: + return "Remove"; + case OpNum::Exists: + return "Exists"; + case OpNum::Get: + return "Get"; + case OpNum::Set: + return "Set"; + case OpNum::SimpleList: + return "SimpleList"; + case OpNum::List: + return "List"; + case OpNum::Check: + return "Check"; + case OpNum::Multi: + return "Multi"; + case OpNum::Heartbeat: + return "Heartbeat"; + case OpNum::Auth: + return "Auth"; + } + int32_t raw_op = static_cast(op_num); + throw Exception("Operation " + std::to_string(raw_op) + " is unknown", Error::ZUNIMPLEMENTED); +} + +OpNum getOpNum(int32_t raw_op_num) +{ + if (!VALID_OPERATIONS.count(raw_op_num)) + throw Exception("Operation " + std::to_string(raw_op_num) + " is unknown", Error::ZUNIMPLEMENTED); + return static_cast(raw_op_num); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperConstants.h b/src/Common/ZooKeeper/ZooKeeperConstants.h new file mode 100644 index 00000000000..8a20330a2d7 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperConstants.h @@ -0,0 +1,49 @@ +#pragma once + +#include +#include + + +namespace Coordination +{ + +using XID = int32_t; + +static constexpr XID WATCH_XID = -1; +static constexpr XID PING_XID = -2; +static constexpr XID AUTH_XID = -4; +static constexpr XID CLOSE_XID = 0x7FFFFFFF; + +enum class OpNum : int32_t +{ + Close = -11, + Error = -1, + Create = 1, + Remove = 2, + Exists = 3, + Get = 4, + Set = 5, + SimpleList = 8, + Heartbeat = 11, + List = 12, + Check = 13, + Multi = 14, + Auth = 100, +}; + +std::string toString(OpNum op_num); +OpNum getOpNum(int32_t raw_op_num); + +static constexpr int32_t ZOOKEEPER_PROTOCOL_VERSION = 0; +static constexpr int32_t CLIENT_HANDSHAKE_LENGTH = 44; +static constexpr int32_t CLIENT_HANDSHAKE_LENGTH_WITH_READONLY = 45; +static constexpr int32_t SERVER_HANDSHAKE_LENGTH = 36; +static constexpr int32_t PASSWORD_LENGTH = 16; + +/// ZooKeeper has 1 MB node size and serialization limit by default, +/// but it can be raised up, so we have a slightly larger limit on our side. +static constexpr int32_t MAX_STRING_OR_ARRAY_SIZE = 1 << 28; /// 256 MiB +static constexpr int32_t DEFAULT_SESSION_TIMEOUT_MS = 30000; +static constexpr int32_t DEFAULT_OPERATION_TIMEOUT_MS = 10000; + +} diff --git a/src/Common/ZooKeeper/ZooKeeperIO.cpp b/src/Common/ZooKeeper/ZooKeeperIO.cpp new file mode 100644 index 00000000000..a0e4161f111 --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperIO.cpp @@ -0,0 +1,140 @@ +#include + +namespace Coordination +{ + +void write(int64_t x, WriteBuffer & out) +{ + x = __builtin_bswap64(x); + writeBinary(x, out); +} +void write(int32_t x, WriteBuffer & out) +{ + x = __builtin_bswap32(x); + writeBinary(x, out); +} + +void write(OpNum x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +void write(bool x, WriteBuffer & out) +{ + writeBinary(x, out); +} + +void write(const std::string & s, WriteBuffer & out) +{ + write(int32_t(s.size()), out); + out.write(s.data(), s.size()); +} + +void write(const ACL & acl, WriteBuffer & out) +{ + write(acl.permissions, out); + write(acl.scheme, out); + write(acl.id, out); +} + +void write(const Stat & stat, WriteBuffer & out) +{ + write(stat.czxid, out); + write(stat.mzxid, out); + write(stat.ctime, out); + write(stat.mtime, out); + write(stat.version, out); + write(stat.cversion, out); + write(stat.aversion, out); + write(stat.ephemeralOwner, out); + write(stat.dataLength, out); + write(stat.numChildren, out); + write(stat.pzxid, out); +} + +void write(const Error & x, WriteBuffer & out) +{ + write(static_cast(x), out); +} + +void read(int64_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap64(x); +} + +void read(int32_t & x, ReadBuffer & in) +{ + readBinary(x, in); + x = __builtin_bswap32(x); +} + +void read(OpNum & x, ReadBuffer & in) +{ + int32_t raw_op_num; + read(raw_op_num, in); + x = getOpNum(raw_op_num); +} + +void read(bool & x, ReadBuffer & in) +{ + readBinary(x, in); +} + +void read(int8_t & x, ReadBuffer & in) +{ + readBinary(x, in); +} + +void read(std::string & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + + if (size == -1) + { + /// It means that zookeeper node has NULL value. We will treat it like empty string. + s.clear(); + return; + } + + if (size < 0) + throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); + + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + + s.resize(size); + in.read(s.data(), size); +} + +void read(ACL & acl, ReadBuffer & in) +{ + read(acl.permissions, in); + read(acl.scheme, in); + read(acl.id, in); +} + +void read(Stat & stat, ReadBuffer & in) +{ + read(stat.czxid, in); + read(stat.mzxid, in); + read(stat.ctime, in); + read(stat.mtime, in); + read(stat.version, in); + read(stat.cversion, in); + read(stat.aversion, in); + read(stat.ephemeralOwner, in); + read(stat.dataLength, in); + read(stat.numChildren, in); + read(stat.pzxid, in); +} + +void read(Error & x, ReadBuffer & in) +{ + int32_t code; + read(code, in); + x = Coordination::Error(code); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperIO.h b/src/Common/ZooKeeper/ZooKeeperIO.h new file mode 100644 index 00000000000..edeb995f27b --- /dev/null +++ b/src/Common/ZooKeeper/ZooKeeperIO.h @@ -0,0 +1,74 @@ +#pragma once +#include +#include +#include +#include +#include +#include +#include +#include + +namespace Coordination +{ + +using namespace DB; + +void write(int64_t x, WriteBuffer & out); +void write(int32_t x, WriteBuffer & out); +void write(OpNum x, WriteBuffer & out); +void write(bool x, WriteBuffer & out); +void write(const std::string & s, WriteBuffer & out); +void write(const ACL & acl, WriteBuffer & out); +void write(const Stat & stat, WriteBuffer & out); +void write(const Error & x, WriteBuffer & out); + +template +void write(const std::array s, WriteBuffer & out) +{ + write(int32_t(N), out); + out.write(s.data(), N); +} + +template +void write(const std::vector & arr, WriteBuffer & out) +{ + write(int32_t(arr.size()), out); + for (const auto & elem : arr) + write(elem, out); +} + +void read(int64_t & x, ReadBuffer & in); +void read(int32_t & x, ReadBuffer & in); +void read(OpNum & x, ReadBuffer & in); +void read(bool & x, ReadBuffer & in); +void read(int8_t & x, ReadBuffer & in); +void read(std::string & s, ReadBuffer & in); +void read(ACL & acl, ReadBuffer & in); +void read(Stat & stat, ReadBuffer & in); +void read(Error & x, ReadBuffer & in); + +template +void read(std::array & s, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size != N) + throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + in.read(s.data(), N); +} + +template +void read(std::vector & arr, ReadBuffer & in) +{ + int32_t size = 0; + read(size, in); + if (size < 0) + throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); + if (size > MAX_STRING_OR_ARRAY_SIZE) + throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); + arr.resize(size); + for (auto & elem : arr) + read(elem, in); +} + +} diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.cpp b/src/Common/ZooKeeper/ZooKeeperImpl.cpp index 285dc32dcae..2314139af69 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.cpp +++ b/src/Common/ZooKeeper/ZooKeeperImpl.cpp @@ -2,11 +2,12 @@ #include #include #include - +#include #include #include #include #include +#include #if !defined(ARCADIA_BUILD) # include @@ -19,11 +20,6 @@ #include -/// ZooKeeper has 1 MB node size and serialization limit by default, -/// but it can be raised up, so we have a slightly larger limit on our side. -#define MAX_STRING_OR_ARRAY_SIZE (1 << 28) /// 256 MiB - - namespace ProfileEvents { extern const Event ZooKeeperInit; @@ -266,137 +262,6 @@ namespace Coordination using namespace DB; - -/// Assuming we are at little endian. - -static void write(int64_t x, WriteBuffer & out) -{ - x = __builtin_bswap64(x); - writeBinary(x, out); -} - -static void write(int32_t x, WriteBuffer & out) -{ - x = __builtin_bswap32(x); - writeBinary(x, out); -} - -static void write(bool x, WriteBuffer & out) -{ - writeBinary(x, out); -} - -static void write(const String & s, WriteBuffer & out) -{ - write(int32_t(s.size()), out); - out.write(s.data(), s.size()); -} - -template void write(std::array s, WriteBuffer & out) -{ - write(int32_t(N), out); - out.write(s.data(), N); -} - -template void write(const std::vector & arr, WriteBuffer & out) -{ - write(int32_t(arr.size()), out); - for (const auto & elem : arr) - write(elem, out); -} - -static void write(const ACL & acl, WriteBuffer & out) -{ - write(acl.permissions, out); - write(acl.scheme, out); - write(acl.id, out); -} - - -static void read(int64_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap64(x); -} - -static void read(int32_t & x, ReadBuffer & in) -{ - readBinary(x, in); - x = __builtin_bswap32(x); -} - -static void read(Error & x, ReadBuffer & in) -{ - int32_t code; - read(code, in); - x = Error(code); -} - -static void read(bool & x, ReadBuffer & in) -{ - readBinary(x, in); -} - -static void read(String & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - - if (size == -1) - { - /// It means that zookeeper node has NULL value. We will treat it like empty string. - s.clear(); - return; - } - - if (size < 0) - throw Exception("Negative size while reading string from ZooKeeper", Error::ZMARSHALLINGERROR); - - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large string size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - - s.resize(size); - in.read(s.data(), size); -} - -template void read(std::array & s, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size != N) - throw Exception("Unexpected array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - in.read(s.data(), N); -} - -static void read(Stat & stat, ReadBuffer & in) -{ - read(stat.czxid, in); - read(stat.mzxid, in); - read(stat.ctime, in); - read(stat.mtime, in); - read(stat.version, in); - read(stat.cversion, in); - read(stat.aversion, in); - read(stat.ephemeralOwner, in); - read(stat.dataLength, in); - read(stat.numChildren, in); - read(stat.pzxid, in); -} - -template void read(std::vector & arr, ReadBuffer & in) -{ - int32_t size = 0; - read(size, in); - if (size < 0) - throw Exception("Negative size while reading array from ZooKeeper", Error::ZMARSHALLINGERROR); - if (size > MAX_STRING_OR_ARRAY_SIZE) - throw Exception("Too large array size while reading from ZooKeeper", Error::ZMARSHALLINGERROR); - arr.resize(size); - for (auto & elem : arr) - read(elem, in); -} - - template void ZooKeeper::write(const T & x) { @@ -409,19 +274,6 @@ void ZooKeeper::read(T & x) Coordination::read(x, *in); } - -void ZooKeeperRequest::write(WriteBuffer & out) const -{ - /// Excessive copy to calculate length. - WriteBufferFromOwnString buf; - Coordination::write(xid, buf); - Coordination::write(getOpNum(), buf); - writeImpl(buf); - Coordination::write(buf.str(), out); - out.next(); -} - - static void removeRootPath(String & path, const String & root_path) { if (root_path.empty()) @@ -433,394 +285,6 @@ static void removeRootPath(String & path, const String & root_path) path = path.substr(root_path.size()); } - -struct ZooKeeperResponse : virtual Response -{ - virtual ~ZooKeeperResponse() override = default; - virtual void readImpl(ReadBuffer &) = 0; -}; - - -struct ZooKeeperHeartbeatRequest final : ZooKeeperRequest -{ - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return 11; } - void writeImpl(WriteBuffer &) const override {} - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperHeartbeatResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperWatchResponse final : WatchResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(type, in); - Coordination::read(state, in); - Coordination::read(path, in); - } -}; - -struct ZooKeeperAuthRequest final : ZooKeeperRequest -{ - int32_t type = 0; /// ignored by the server - String scheme; - String data; - - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return 100; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(type, out); - Coordination::write(scheme, out); - Coordination::write(data, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperAuthResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperCloseRequest final : ZooKeeperRequest -{ - String getPath() const override { return {}; } - ZooKeeper::OpNum getOpNum() const override { return -11; } - void writeImpl(WriteBuffer &) const override {} - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCloseResponse final : ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override - { - throw Exception("Received response for close request", Error::ZRUNTIMEINCONSISTENCY); - } -}; - -struct ZooKeeperCreateRequest final : CreateRequest, ZooKeeperRequest -{ - ZooKeeperCreateRequest() = default; - explicit ZooKeeperCreateRequest(const CreateRequest & base) : CreateRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 1; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(data, out); - Coordination::write(acls, out); - - int32_t flags = 0; - - if (is_ephemeral) - flags |= 1; - if (is_sequential) - flags |= 2; - - Coordination::write(flags, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCreateResponse final : CreateResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(path_created, in); - } -}; - -struct ZooKeeperRemoveRequest final : RemoveRequest, ZooKeeperRequest -{ - ZooKeeperRemoveRequest() = default; - explicit ZooKeeperRemoveRequest(const RemoveRequest & base) : RemoveRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 2; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperRemoveResponse final : RemoveResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -struct ZooKeeperExistsRequest final : ExistsRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 3; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperExistsResponse final : ExistsResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(stat, in); - } -}; - -struct ZooKeeperGetRequest final : GetRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 4; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperGetResponse final : GetResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(data, in); - Coordination::read(stat, in); - } -}; - -struct ZooKeeperSetRequest final : SetRequest, ZooKeeperRequest -{ - ZooKeeperSetRequest() = default; - explicit ZooKeeperSetRequest(const SetRequest & base) : SetRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 5; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(data, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperSetResponse final : SetResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(stat, in); - } -}; - -struct ZooKeeperListRequest final : ListRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 12; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(has_watch, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperListResponse final : ListResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::read(names, in); - Coordination::read(stat, in); - } -}; - -struct ZooKeeperCheckRequest final : CheckRequest, ZooKeeperRequest -{ - ZooKeeperCheckRequest() = default; - explicit ZooKeeperCheckRequest(const CheckRequest & base) : CheckRequest(base) {} - - ZooKeeper::OpNum getOpNum() const override { return 13; } - void writeImpl(WriteBuffer & out) const override - { - Coordination::write(path, out); - Coordination::write(version, out); - } - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperCheckResponse final : CheckResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer &) override {} -}; - -/// This response may be received only as an element of responses in MultiResponse. -struct ZooKeeperErrorResponse final : ErrorResponse, ZooKeeperResponse -{ - void readImpl(ReadBuffer & in) override - { - Coordination::Error read_error; - Coordination::read(read_error, in); - - if (read_error != error) - throw Exception(fmt::format("Error code in ErrorResponse ({}) doesn't match error code in header ({})", read_error, error), - Error::ZMARSHALLINGERROR); - } -}; - -struct ZooKeeperMultiRequest final : MultiRequest, ZooKeeperRequest -{ - ZooKeeper::OpNum getOpNum() const override { return 14; } - - ZooKeeperMultiRequest(const Requests & generic_requests, const ACLs & default_acls) - { - /// Convert nested Requests to ZooKeeperRequests. - /// Note that deep copy is required to avoid modifying path in presence of chroot prefix. - requests.reserve(generic_requests.size()); - - for (const auto & generic_request : generic_requests) - { - if (const auto * concrete_request_create = dynamic_cast(generic_request.get())) - { - auto create = std::make_shared(*concrete_request_create); - if (create->acls.empty()) - create->acls = default_acls; - requests.push_back(create); - } - else if (const auto * concrete_request_remove = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_remove)); - } - else if (const auto * concrete_request_set = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_set)); - } - else if (const auto * concrete_request_check = dynamic_cast(generic_request.get())) - { - requests.push_back(std::make_shared(*concrete_request_check)); - } - else - throw Exception("Illegal command as part of multi ZooKeeper request", Error::ZBADARGUMENTS); - } - } - - void writeImpl(WriteBuffer & out) const override - { - for (const auto & request : requests) - { - const auto & zk_request = dynamic_cast(*request); - - bool done = false; - int32_t error = -1; - - Coordination::write(zk_request.getOpNum(), out); - Coordination::write(done, out); - Coordination::write(error, out); - - zk_request.writeImpl(out); - } - - ZooKeeper::OpNum op_num = -1; - bool done = true; - int32_t error = -1; - - Coordination::write(op_num, out); - Coordination::write(done, out); - Coordination::write(error, out); - } - - ZooKeeperResponsePtr makeResponse() const override; -}; - -struct ZooKeeperMultiResponse final : MultiResponse, ZooKeeperResponse -{ - explicit ZooKeeperMultiResponse(const Requests & requests) - { - responses.reserve(requests.size()); - - for (const auto & request : requests) - responses.emplace_back(dynamic_cast(*request).makeResponse()); - } - - void readImpl(ReadBuffer & in) override - { - for (auto & response : responses) - { - ZooKeeper::OpNum op_num; - bool done; - Error op_error; - - Coordination::read(op_num, in); - Coordination::read(done, in); - Coordination::read(op_error, in); - - if (done) - throw Exception("Not enough results received for multi transaction", Error::ZMARSHALLINGERROR); - - /// op_num == -1 is special for multi transaction. - /// For unknown reason, error code is duplicated in header and in response body. - - if (op_num == -1) - response = std::make_shared(); - - if (op_error != Error::ZOK) - { - response->error = op_error; - - /// Set error for whole transaction. - /// If some operations fail, ZK send global error as zero and then send details about each operation. - /// It will set error code for first failed operation and it will set special "runtime inconsistency" code for other operations. - if (error == Error::ZOK && op_error != Error::ZRUNTIMEINCONSISTENCY) - error = op_error; - } - - if (op_error == Error::ZOK || op_num == -1) - dynamic_cast(*response).readImpl(in); - } - - /// Footer. - { - ZooKeeper::OpNum op_num; - bool done; - int32_t error_read; - - Coordination::read(op_num, in); - Coordination::read(done, in); - Coordination::read(error_read, in); - - if (!done) - throw Exception("Too many results received for multi transaction", Error::ZMARSHALLINGERROR); - if (op_num != -1) - throw Exception("Unexpected op_num received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); - if (error_read != -1) - throw Exception("Unexpected error value received at the end of results for multi transaction", Error::ZMARSHALLINGERROR); - } - } -}; - - -ZooKeeperResponsePtr ZooKeeperHeartbeatRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperAuthRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperCreateRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperRemoveRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperExistsRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperGetRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperSetRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperListRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperCheckRequest::makeResponse() const { return std::make_shared(); } -ZooKeeperResponsePtr ZooKeeperMultiRequest::makeResponse() const { return std::make_shared(requests); } -ZooKeeperResponsePtr ZooKeeperCloseRequest::makeResponse() const { return std::make_shared(); } - - -static constexpr int32_t protocol_version = 0; - -static constexpr ZooKeeper::XID watch_xid = -1; -static constexpr ZooKeeper::XID ping_xid = -2; -static constexpr ZooKeeper::XID auth_xid = -4; - -static constexpr ZooKeeper::XID close_xid = 0x7FFFFFFF; - - ZooKeeper::~ZooKeeper() { try @@ -995,7 +459,7 @@ void ZooKeeper::sendHandshake() std::array passwd {}; write(handshake_length); - write(protocol_version); + write(ZOOKEEPER_PROTOCOL_VERSION); write(last_zxid_seen); write(timeout); write(previous_session_id); @@ -1010,16 +474,15 @@ void ZooKeeper::receiveHandshake() int32_t handshake_length; int32_t protocol_version_read; int32_t timeout; - constexpr int32_t passwd_len = 16; - std::array passwd; + std::array passwd; read(handshake_length); - if (handshake_length != 36) - throw Exception("Unexpected handshake length received: " + toString(handshake_length), Error::ZMARSHALLINGERROR); + if (handshake_length != SERVER_HANDSHAKE_LENGTH) + throw Exception("Unexpected handshake length received: " + DB::toString(handshake_length), Error::ZMARSHALLINGERROR); read(protocol_version_read); - if (protocol_version_read != protocol_version) - throw Exception("Unexpected protocol version: " + toString(protocol_version_read), Error::ZMARSHALLINGERROR); + if (protocol_version_read != ZOOKEEPER_PROTOCOL_VERSION) + throw Exception("Unexpected protocol version: " + DB::toString(protocol_version_read), Error::ZMARSHALLINGERROR); read(timeout); if (timeout != session_timeout.totalMilliseconds()) @@ -1036,7 +499,7 @@ void ZooKeeper::sendAuth(const String & scheme, const String & data) ZooKeeperAuthRequest request; request.scheme = scheme; request.data = data; - request.xid = auth_xid; + request.xid = AUTH_XID; request.write(*out); int32_t length; @@ -1050,17 +513,17 @@ void ZooKeeper::sendAuth(const String & scheme, const String & data) read(zxid); read(err); - if (read_xid != auth_xid) - throw Exception("Unexpected event received in reply to auth request: " + toString(read_xid), + if (read_xid != AUTH_XID) + throw Exception("Unexpected event received in reply to auth request: " + DB::toString(read_xid), Error::ZMARSHALLINGERROR); int32_t actual_length = in->count() - count_before_event; if (length != actual_length) - throw Exception("Response length doesn't match. Expected: " + toString(length) + ", actual: " + toString(actual_length), + throw Exception("Response length doesn't match. Expected: " + DB::toString(length) + ", actual: " + DB::toString(actual_length), Error::ZMARSHALLINGERROR); if (err != Error::ZOK) - throw Exception("Error received in reply to auth request. Code: " + toString(int32_t(err)) + ". Message: " + String(errorMessage(err)), + throw Exception("Error received in reply to auth request. Code: " + DB::toString(int32_t(err)) + ". Message: " + String(errorMessage(err)), Error::ZMARSHALLINGERROR); } @@ -1093,7 +556,7 @@ void ZooKeeper::sendThread() /// After we popped element from the queue, we must register callbacks (even in the case when expired == true right now), /// because they must not be lost (callbacks must be called because the user will wait for them). - if (info.request->xid != close_xid) + if (info.request->xid != CLOSE_XID) { CurrentMetrics::add(CurrentMetrics::ZooKeeperRequest); std::lock_guard lock(operations_mutex); @@ -1107,7 +570,9 @@ void ZooKeeper::sendThread() } if (expired) + { break; + } info.request->addRootPath(root_path); @@ -1115,7 +580,7 @@ void ZooKeeper::sendThread() info.request->write(*out); /// We sent close request, exit - if (info.request->xid == close_xid) + if (info.request->xid == CLOSE_XID) break; } } @@ -1125,7 +590,7 @@ void ZooKeeper::sendThread() prev_heartbeat_time = clock::now(); ZooKeeperHeartbeatRequest request; - request.xid = ping_xid; + request.xid = PING_XID; request.write(*out); } @@ -1179,7 +644,9 @@ void ZooKeeper::receiveThread() else { if (earliest_operation) - throw Exception("Operation timeout (no response) for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); + { + throw Exception("Operation timeout (no response) for request " + toString(earliest_operation->request->getOpNum()) + " for path: " + earliest_operation->request->getPath(), Error::ZOPERATIONTIMEOUT); + } waited += max_wait; if (waited >= session_timeout.totalMicroseconds()) throw Exception("Nothing is received in session timeout", Error::ZOPERATIONTIMEOUT); @@ -1213,14 +680,14 @@ void ZooKeeper::receiveEvent() RequestInfo request_info; ZooKeeperResponsePtr response; - if (xid == ping_xid) + if (xid == PING_XID) { if (err != Error::ZOK) throw Exception("Received error in heartbeat response: " + String(errorMessage(err)), Error::ZRUNTIMEINCONSISTENCY); response = std::make_shared(); } - else if (xid == watch_xid) + else if (xid == WATCH_XID) { ProfileEvents::increment(ProfileEvents::ZooKeeperWatchResponse); response = std::make_shared(); @@ -1261,7 +728,7 @@ void ZooKeeper::receiveEvent() auto it = operations.find(xid); if (it == operations.end()) - throw Exception("Received response for unknown xid", Error::ZRUNTIMEINCONSISTENCY); + throw Exception("Received response for unknown xid " + DB::toString(xid), Error::ZRUNTIMEINCONSISTENCY); /// After this point, we must invoke callback, that we've grabbed from 'operations'. /// Invariant: all callbacks are invoked either in case of success or in case of error. @@ -1282,13 +749,14 @@ void ZooKeeper::receiveEvent() response = request_info.request->makeResponse(); if (err != Error::ZOK) + { response->error = err; + } else { response->readImpl(*in); response->removeRootPath(root_path); } - /// Instead of setting the watch in sendEvent, set it in receiveEvent because need to check the response. /// The watch shouldn't be set if the node does not exist and it will never exist like sequential ephemeral nodes. /// By using getData() instead of exists(), a watch won't be set if the node doesn't exist. @@ -1298,7 +766,7 @@ void ZooKeeper::receiveEvent() /// 3 indicates the ZooKeeperExistsRequest. // For exists, we set the watch on both node exist and nonexist case. // For other case like getData, we only set the watch when node exists. - if (request_info.request->getOpNum() == 3) + if (request_info.request->getOpNum() == OpNum::Exists) add_watch = (response->error == Error::ZOK || response->error == Error::ZNONODE); else add_watch = response->error == Error::ZOK; @@ -1315,7 +783,7 @@ void ZooKeeper::receiveEvent() int32_t actual_length = in->count() - count_before_event; if (length != actual_length) - throw Exception("Response length doesn't match. Expected: " + toString(length) + ", actual: " + toString(actual_length), Error::ZMARSHALLINGERROR); + throw Exception("Response length doesn't match. Expected: " + DB::toString(length) + ", actual: " + DB::toString(actual_length), Error::ZMARSHALLINGERROR); } catch (...) { @@ -1508,7 +976,7 @@ void ZooKeeper::pushRequest(RequestInfo && info) if (!info.request->xid) { info.request->xid = next_xid.fetch_add(1); - if (info.request->xid == close_xid) + if (info.request->xid == CLOSE_XID) throw Exception("xid equal to close_xid", Error::ZSESSIONEXPIRED); if (info.request->xid < 0) throw Exception("XID overflow", Error::ZSESSIONEXPIRED); @@ -1688,7 +1156,7 @@ void ZooKeeper::multi( void ZooKeeper::close() { ZooKeeperCloseRequest request; - request.xid = close_xid; + request.xid = CLOSE_XID; RequestInfo request_info; request_info.request = std::make_shared(std::move(request)); @@ -1699,5 +1167,4 @@ void ZooKeeper::close() ProfileEvents::increment(ProfileEvents::ZooKeeperClose); } - } diff --git a/src/Common/ZooKeeper/ZooKeeperImpl.h b/src/Common/ZooKeeper/ZooKeeperImpl.h index b0bb1d6caf4..71b7cd56149 100644 --- a/src/Common/ZooKeeper/ZooKeeperImpl.h +++ b/src/Common/ZooKeeper/ZooKeeperImpl.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -85,9 +86,6 @@ namespace Coordination using namespace DB; -struct ZooKeeperRequest; - - /** Usage scenario: look at the documentation for IKeeper class. */ class ZooKeeper : public IKeeper @@ -101,9 +99,6 @@ public: using Nodes = std::vector; - using XID = int32_t; - using OpNum = int32_t; - /** Connection to nodes is performed in order. If you want, shuffle them manually. * Operation timeout couldn't be greater than session timeout. * Operation timeout applies independently for network read, network write, waiting for events and synchronization. @@ -196,7 +191,7 @@ private: struct RequestInfo { - std::shared_ptr request; + ZooKeeperRequestPtr request; ResponseCallback callback; WatchCallback watch; clock::time_point time; @@ -249,31 +244,4 @@ private: CurrentMetrics::Increment active_session_metric_increment{CurrentMetrics::ZooKeeperSession}; }; -struct ZooKeeperResponse; -using ZooKeeperResponsePtr = std::shared_ptr; - -/// Exposed in header file for Yandex.Metrica code. -struct ZooKeeperRequest : virtual Request -{ - ZooKeeper::XID xid = 0; - bool has_watch = false; - /// If the request was not send and the error happens, we definitely sure, that is has not been processed by the server. - /// If the request was sent and we didn't get the response and the error happens, then we cannot be sure was it processed or not. - bool probably_sent = false; - - ZooKeeperRequest() = default; - ZooKeeperRequest(const ZooKeeperRequest &) = default; - virtual ~ZooKeeperRequest() override = default; - - virtual ZooKeeper::OpNum getOpNum() const = 0; - - /// Writes length, xid, op_num, then the rest. - void write(WriteBuffer & out) const; - - virtual void writeImpl(WriteBuffer &) const = 0; - - virtual ZooKeeperResponsePtr makeResponse() const = 0; -}; - - } diff --git a/src/Common/tests/symbol_index.cpp b/src/Common/tests/symbol_index.cpp index 67defa77242..3811bbbdd71 100644 --- a/src/Common/tests/symbol_index.cpp +++ b/src/Common/tests/symbol_index.cpp @@ -25,7 +25,8 @@ int main(int argc, char ** argv) return 1; } - const SymbolIndex & symbol_index = SymbolIndex::instance(); + auto symbol_index_ptr = SymbolIndex::instance(); + const SymbolIndex & symbol_index = *symbol_index_ptr; for (const auto & elem : symbol_index.symbols()) std::cout << elem.name << ": " << elem.address_begin << " ... " << elem.address_end << "\n"; diff --git a/src/Common/ya.make b/src/Common/ya.make index 71c0edaea95..e515741a272 100644 --- a/src/Common/ya.make +++ b/src/Common/ya.make @@ -68,6 +68,7 @@ SRCS( StringUtils/StringUtils.cpp StudentTTest.cpp SymbolIndex.cpp + TLDListsHolder.cpp TaskStatsInfoGetter.cpp TerminalSize.cpp ThreadFuzzer.cpp @@ -80,7 +81,11 @@ SRCS( WeakHash.cpp ZooKeeper/IKeeper.cpp ZooKeeper/TestKeeper.cpp + ZooKeeper/TestKeeperStorage.cpp ZooKeeper/ZooKeeper.cpp + ZooKeeper/ZooKeeperCommon.cpp + ZooKeeper/ZooKeeperConstants.cpp + ZooKeeper/ZooKeeperIO.cpp ZooKeeper/ZooKeeperImpl.cpp ZooKeeper/ZooKeeperNodeCache.cpp checkStackSize.cpp diff --git a/src/Core/Defines.h b/src/Core/Defines.h index 9b6578092c9..f1cd1a8a708 100644 --- a/src/Core/Defines.h +++ b/src/Core/Defines.h @@ -73,8 +73,10 @@ /// Minimum revision supporting interserver secret. #define DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET 54441 +#define DBMS_MIN_REVISION_WITH_X_FORWARDED_FOR_IN_CLIENT_INFO 54443 + /// Version of ClickHouse TCP protocol. Increment it manually when you change the protocol. -#define DBMS_TCP_PROTOCOL_VERSION 54442 +#define DBMS_TCP_PROTOCOL_VERSION 54443 /// The boundary on which the blocks for asynchronous file operations should be aligned. #define DEFAULT_AIO_FILE_BLOCK_SIZE 4096 diff --git a/src/Databases/MySQL/MaterializeMetadata.cpp b/src/Databases/MySQL/MaterializeMetadata.cpp index 1acc0e78b6c..84f44771bf1 100644 --- a/src/Databases/MySQL/MaterializeMetadata.cpp +++ b/src/Databases/MySQL/MaterializeMetadata.cpp @@ -36,7 +36,7 @@ static std::unordered_map fetchTablesCreateQuery( MySQLBlockInputStream show_create_table( connection, "SHOW CREATE TABLE " + backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(fetch_table_name), - show_create_table_header, DEFAULT_BLOCK_SIZE); + show_create_table_header, DEFAULT_BLOCK_SIZE, false, true); Block create_query_block = show_create_table.read(); if (!create_query_block || create_query_block.rows() != 1) @@ -77,7 +77,7 @@ void MaterializeMetadata::fetchMasterStatus(mysqlxx::PoolWithFailover::Entry & c {std::make_shared(), "Executed_Gtid_Set"}, }; - MySQLBlockInputStream input(connection, "SHOW MASTER STATUS;", header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream input(connection, "SHOW MASTER STATUS;", header, DEFAULT_BLOCK_SIZE, false, true); Block master_status = input.read(); if (!master_status || master_status.rows() != 1) @@ -99,7 +99,7 @@ void MaterializeMetadata::fetchMasterVariablesValue(const mysqlxx::PoolWithFailo }; const String & fetch_query = "SHOW VARIABLES WHERE Variable_name = 'binlog_checksum'"; - MySQLBlockInputStream variables_input(connection, fetch_query, variables_header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream variables_input(connection, fetch_query, variables_header, DEFAULT_BLOCK_SIZE, false, true); while (Block variables_block = variables_input.read()) { @@ -114,23 +114,6 @@ void MaterializeMetadata::fetchMasterVariablesValue(const mysqlxx::PoolWithFailo } } -static Block getShowMasterLogHeader(const String & mysql_version) -{ - if (startsWith(mysql_version, "5.")) - { - return Block { - {std::make_shared(), "Log_name"}, - {std::make_shared(), "File_size"} - }; - } - - return Block { - {std::make_shared(), "Log_name"}, - {std::make_shared(), "File_size"}, - {std::make_shared(), "Encrypted"} - }; -} - static bool checkSyncUserPrivImpl(const mysqlxx::PoolWithFailover::Entry & connection, WriteBuffer & out) { Block sync_user_privs_header @@ -174,9 +157,14 @@ static void checkSyncUserPriv(const mysqlxx::PoolWithFailover::Entry & connectio "But the SYNC USER grant query is: " + out.str(), ErrorCodes::SYNC_MYSQL_USER_ACCESS_ERROR); } -bool MaterializeMetadata::checkBinlogFileExists(const mysqlxx::PoolWithFailover::Entry & connection, const String & mysql_version) const +bool MaterializeMetadata::checkBinlogFileExists(const mysqlxx::PoolWithFailover::Entry & connection) const { - MySQLBlockInputStream input(connection, "SHOW MASTER LOGS", getShowMasterLogHeader(mysql_version), DEFAULT_BLOCK_SIZE); + Block logs_header { + {std::make_shared(), "Log_name"}, + {std::make_shared(), "File_size"} + }; + + MySQLBlockInputStream input(connection, "SHOW MASTER LOGS", logs_header, DEFAULT_BLOCK_SIZE, false, true); while (Block block = input.read()) { @@ -233,7 +221,7 @@ void MaterializeMetadata::transaction(const MySQLReplication::Position & positio MaterializeMetadata::MaterializeMetadata( mysqlxx::PoolWithFailover::Entry & connection, const String & path_, - const String & database, bool & opened_transaction, const String & mysql_version) + const String & database, bool & opened_transaction) : persistent_path(path_) { checkSyncUserPriv(connection); @@ -251,7 +239,7 @@ MaterializeMetadata::MaterializeMetadata( assertString("\nData Version:\t", in); readIntText(data_version, in); - if (checkBinlogFileExists(connection, mysql_version)) + if (checkBinlogFileExists(connection)) return; } diff --git a/src/Databases/MySQL/MaterializeMetadata.h b/src/Databases/MySQL/MaterializeMetadata.h index 545d162ca18..f4cd2970fce 100644 --- a/src/Databases/MySQL/MaterializeMetadata.h +++ b/src/Databases/MySQL/MaterializeMetadata.h @@ -41,13 +41,13 @@ struct MaterializeMetadata void fetchMasterVariablesValue(const mysqlxx::PoolWithFailover::Entry & connection); - bool checkBinlogFileExists(const mysqlxx::PoolWithFailover::Entry & connection, const String & mysql_version) const; + bool checkBinlogFileExists(const mysqlxx::PoolWithFailover::Entry & connection) const; void transaction(const MySQLReplication::Position & position, const std::function & fun); MaterializeMetadata( mysqlxx::PoolWithFailover::Entry & connection, const String & path - , const String & database, bool & opened_transaction, const String & mysql_version); + , const String & database, bool & opened_transaction); }; } diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp index 02400a28214..bc7c9e4c554 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp @@ -93,7 +93,7 @@ MaterializeMySQLSyncThread::~MaterializeMySQLSyncThread() } } -static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection) +static void checkMySQLVariables(const mysqlxx::Pool::Entry & connection) { Block variables_header{ {std::make_shared(), "Variable_name"}, @@ -106,7 +106,7 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection "OR (Variable_name = 'binlog_row_image' AND upper(Value) = 'FULL') " "OR (Variable_name = 'default_authentication_plugin' AND upper(Value) = 'MYSQL_NATIVE_PASSWORD');"; - MySQLBlockInputStream variables_input(connection, check_query, variables_header, DEFAULT_BLOCK_SIZE); + MySQLBlockInputStream variables_input(connection, check_query, variables_header, DEFAULT_BLOCK_SIZE, false, true); Block variables_block = variables_input.read(); if (!variables_block || variables_block.rows() != 4) @@ -140,15 +140,6 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection throw Exception(error_message.str(), ErrorCodes::ILLEGAL_MYSQL_VARIABLE); } - - Block version_header{{std::make_shared(), "version"}}; - MySQLBlockInputStream version_input(connection, "SELECT version() AS version;", version_header, DEFAULT_BLOCK_SIZE); - - Block version_block = version_input.read(); - if (!version_block || version_block.rows() != 1) - throw Exception("LOGICAL ERROR: cannot get mysql version.", ErrorCodes::LOGICAL_ERROR); - - return version_block.getByPosition(0).column->getDataAt(0).toString(); } MaterializeMySQLSyncThread::MaterializeMySQLSyncThread( @@ -160,13 +151,13 @@ MaterializeMySQLSyncThread::MaterializeMySQLSyncThread( query_prefix = "EXTERNAL DDL FROM MySQL(" + backQuoteIfNeed(database_name) + ", " + backQuoteIfNeed(mysql_database_name) + ") "; } -void MaterializeMySQLSyncThread::synchronization(const String & mysql_version) +void MaterializeMySQLSyncThread::synchronization() { setThreadName(MYSQL_BACKGROUND_THREAD_NAME); try { - if (std::optional metadata = prepareSynchronized(mysql_version)) + if (std::optional metadata = prepareSynchronized()) { Stopwatch watch; Buffers buffers(database_name); @@ -217,10 +208,8 @@ void MaterializeMySQLSyncThread::startSynchronization() { try { - const auto & mysql_server_version = checkVariableAndGetVersion(pool.get()); - - background_thread_pool = std::make_unique( - [this, mysql_server_version = mysql_server_version]() { synchronization(mysql_server_version); }); + checkMySQLVariables(pool.get()); + background_thread_pool = std::make_unique([this]() { synchronization(); }); } catch (...) { @@ -246,15 +235,24 @@ void MaterializeMySQLSyncThread::startSynchronization() static inline void cleanOutdatedTables(const String & database_name, const Context & context) { - auto ddl_guard = DatabaseCatalog::instance().getDDLGuard(database_name, ""); - const DatabasePtr & clean_database = DatabaseCatalog::instance().getDatabase(database_name); - - for (auto iterator = clean_database->getTablesIterator(context); iterator->isValid(); iterator->next()) + String cleaning_table_name; + try { - Context query_context = createQueryContext(context); - String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; - String table_name = backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(iterator->name()); - tryToExecuteQuery(" DROP TABLE " + table_name, query_context, database_name, comment); + auto ddl_guard = DatabaseCatalog::instance().getDDLGuard(database_name, ""); + const DatabasePtr & clean_database = DatabaseCatalog::instance().getDatabase(database_name); + + for (auto iterator = clean_database->getTablesIterator(context); iterator->isValid(); iterator->next()) + { + Context query_context = createQueryContext(context); + String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; + cleaning_table_name = backQuoteIfNeed(database_name) + "." + backQuoteIfNeed(iterator->name()); + tryToExecuteQuery(" DROP TABLE " + cleaning_table_name, query_context, database_name, comment); + } + } + catch (Exception & exception) + { + exception.addMessage("While executing " + (cleaning_table_name.empty() ? "cleanOutdatedTables" : cleaning_table_name)); + throw; } } @@ -295,24 +293,32 @@ static inline void dumpDataForTables( auto iterator = master_info.need_dumping_tables.begin(); for (; iterator != master_info.need_dumping_tables.end() && !is_cancelled(); ++iterator) { - const auto & table_name = iterator->first; - Context query_context = createQueryContext(context); - String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; - tryToExecuteQuery(query_prefix + " " + iterator->second, query_context, database_name, comment); /// create table. + try + { + const auto & table_name = iterator->first; + Context query_context = createQueryContext(context); + String comment = "Materialize MySQL step 1: execute MySQL DDL for dump data"; + tryToExecuteQuery(query_prefix + " " + iterator->second, query_context, database_name, comment); /// create table. - auto out = std::make_shared(getTableOutput(database_name, table_name, query_context)); - MySQLBlockInputStream input( - connection, "SELECT * FROM " + backQuoteIfNeed(mysql_database_name) + "." + backQuoteIfNeed(table_name), - out->getHeader(), DEFAULT_BLOCK_SIZE); + auto out = std::make_shared(getTableOutput(database_name, table_name, query_context)); + MySQLBlockInputStream input( + connection, "SELECT * FROM " + backQuoteIfNeed(mysql_database_name) + "." + backQuoteIfNeed(table_name), + out->getHeader(), DEFAULT_BLOCK_SIZE); - Stopwatch watch; - copyData(input, *out, is_cancelled); - const Progress & progress = out->getProgress(); - LOG_INFO(&Poco::Logger::get("MaterializeMySQLSyncThread(" + database_name + ")"), - "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." - , table_name, formatReadableQuantity(progress.written_rows), formatReadableSizeWithBinarySuffix(progress.written_bytes) - , watch.elapsedSeconds(), formatReadableQuantity(static_cast(progress.written_rows / watch.elapsedSeconds())) - , formatReadableSizeWithBinarySuffix(static_cast(progress.written_bytes / watch.elapsedSeconds()))); + Stopwatch watch; + copyData(input, *out, is_cancelled); + const Progress & progress = out->getProgress(); + LOG_INFO(&Poco::Logger::get("MaterializeMySQLSyncThread(" + database_name + ")"), + "Materialize MySQL step 1: dump {}, {} rows, {} in {} sec., {} rows/sec., {}/sec." + , table_name, formatReadableQuantity(progress.written_rows), formatReadableSizeWithBinarySuffix(progress.written_bytes) + , watch.elapsedSeconds(), formatReadableQuantity(static_cast(progress.written_rows / watch.elapsedSeconds())) + , formatReadableSizeWithBinarySuffix(static_cast(progress.written_bytes / watch.elapsedSeconds()))); + } + catch (Exception & exception) + { + exception.addMessage("While executing dump MySQL {}.{} table data.", mysql_database_name, iterator->first); + throw; + } } } @@ -324,7 +330,7 @@ static inline UInt32 randomNumber() return dist6(rng); } -std::optional MaterializeMySQLSyncThread::prepareSynchronized(const String & mysql_version) +std::optional MaterializeMySQLSyncThread::prepareSynchronized() { bool opened_transaction = false; mysqlxx::PoolWithFailover::Entry connection; @@ -337,8 +343,7 @@ std::optional MaterializeMySQLSyncThread::prepareSynchroniz opened_transaction = false; MaterializeMetadata metadata( - connection, getDatabase(database_name).getMetadataPath() + "/.metadata", - mysql_database_name, opened_transaction, mysql_version); + connection, getDatabase(database_name).getMetadataPath() + "/.metadata", mysql_database_name, opened_transaction); if (!metadata.need_dumping_tables.empty()) { @@ -685,6 +690,8 @@ void MaterializeMySQLSyncThread::executeDDLAtomic(const QueryEvent & query_event } catch (Exception & exception) { + exception.addMessage("While executing MYSQL_QUERY_EVENT. The query: " + query_event.query); + tryLogCurrentException(log); /// If some DDL query was not successfully parsed and executed diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.h b/src/Databases/MySQL/MaterializeMySQLSyncThread.h index 323ae5beb80..54f148026ad 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.h +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.h @@ -95,11 +95,11 @@ private: BufferAndSortingColumnsPtr getTableDataBuffer(const String & table, const Context & context); }; - void synchronization(const String & mysql_version); + void synchronization(); bool isCancelled() { return sync_quit.load(std::memory_order_relaxed); } - std::optional prepareSynchronized(const String & mysql_version); + std::optional prepareSynchronized(); void flushBuffersData(Buffers & buffers, MaterializeMetadata & metadata); diff --git a/src/Disks/S3/DiskS3.cpp b/src/Disks/S3/DiskS3.cpp index f5469fd762c..2e51f93cb6e 100644 --- a/src/Disks/S3/DiskS3.cpp +++ b/src/Disks/S3/DiskS3.cpp @@ -3,6 +3,7 @@ #include "Disks/DiskFactory.h" #include +#include #include #include #include @@ -344,11 +345,11 @@ public: const String & bucket_, DiskS3::Metadata metadata_, const String & s3_path_, - bool is_multipart, + std::optional object_metadata_,bool is_multipart, size_t min_upload_part_size, size_t buf_size_) : WriteBufferFromFileBase(buf_size_, nullptr, 0) - , impl(WriteBufferFromS3(client_ptr_, bucket_, metadata_.s3_root_path + s3_path_, min_upload_part_size, is_multipart, buf_size_)) + , impl(WriteBufferFromS3(client_ptr_, bucket_, metadata_.s3_root_path + s3_path_, min_upload_part_size, is_multipart,std::move(object_metadata_), buf_size_)) , metadata(std::move(metadata_)) , s3_path(s3_path_) { @@ -539,7 +540,8 @@ DiskS3::DiskS3( String metadata_path_, size_t min_upload_part_size_, size_t min_multi_part_upload_size_, - size_t min_bytes_for_seek_) + size_t min_bytes_for_seek_, + bool send_metadata_) : IDisk(std::make_unique()) , name(std::move(name_)) , client(std::move(client_)) @@ -550,6 +552,7 @@ DiskS3::DiskS3( , min_upload_part_size(min_upload_part_size_) , min_multi_part_upload_size(min_multi_part_upload_size_) , min_bytes_for_seek(min_bytes_for_seek_) + , send_metadata(send_metadata_) { } @@ -667,6 +670,7 @@ std::unique_ptr DiskS3::writeFile(const String & path, /// Path to store new S3 object. auto s3_path = getRandomName(); + auto object_metadata = createObjectMetadata(path); bool is_multipart = estimated_size >= min_multi_part_upload_size; if (!exist || mode == WriteMode::Rewrite) { @@ -678,9 +682,9 @@ std::unique_ptr DiskS3::writeFile(const String & path, /// Save empty metadata to disk to have ability to get file size while buffer is not finalized. metadata.save(); - LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Write to file by path: {} New S3 path: {}", backQuote(metadata_path + path), s3_root_path + s3_path); + LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Write to file by path: {}. New S3 path: {}", backQuote(metadata_path + path), s3_root_path + s3_path); - return std::make_unique(client, bucket, metadata, s3_path, is_multipart, min_upload_part_size, buf_size); + return std::make_unique(client, bucket, metadata, s3_path, object_metadata, is_multipart, min_upload_part_size, buf_size); } else { @@ -689,7 +693,7 @@ std::unique_ptr DiskS3::writeFile(const String & path, LOG_DEBUG(&Poco::Logger::get("DiskS3"), "Append to file by path: {}. New S3 path: {}. Existing S3 objects: {}.", backQuote(metadata_path + path), s3_root_path + s3_path, metadata.s3_objects.size()); - return std::make_unique(client, bucket, metadata, s3_path, is_multipart, min_upload_part_size, buf_size); + return std::make_unique(client, bucket, metadata, s3_path, object_metadata, is_multipart, min_upload_part_size, buf_size); } } @@ -881,4 +885,12 @@ void DiskS3::shutdown() client->DisableRequestProcessing(); } +std::optional DiskS3::createObjectMetadata(const String & path) const +{ + if (send_metadata) + return (DiskS3::ObjectMetadata){{"path", path}}; + + return {}; +} + } diff --git a/src/Disks/S3/DiskS3.h b/src/Disks/S3/DiskS3.h index 4ed95ee9b02..5bc8d51abb2 100644 --- a/src/Disks/S3/DiskS3.h +++ b/src/Disks/S3/DiskS3.h @@ -19,6 +19,8 @@ namespace DB class DiskS3 : public IDisk { public: + using ObjectMetadata = std::map; + friend class DiskS3Reservation; class AwsS3KeyKeeper; @@ -33,7 +35,8 @@ public: String metadata_path_, size_t min_upload_part_size_, size_t min_multi_part_upload_size_, - size_t min_bytes_for_seek_); + size_t min_bytes_for_seek_, + bool send_metadata_); const String & getName() const override { return name; } @@ -117,6 +120,7 @@ private: void removeMeta(const String & path, AwsS3KeyKeeper & keys); void removeMetaRecursive(const String & path, AwsS3KeyKeeper & keys); void removeAws(const AwsS3KeyKeeper & keys); + std::optional createObjectMetadata(const String & path) const; Metadata readMeta(const String & path) const; Metadata createMeta(const String & path) const; @@ -131,6 +135,7 @@ private: size_t min_upload_part_size; size_t min_multi_part_upload_size; size_t min_bytes_for_seek; + bool send_metadata; UInt64 reserved_bytes = 0; UInt64 reservation_count = 0; diff --git a/src/Disks/S3/registerDiskS3.cpp b/src/Disks/S3/registerDiskS3.cpp index 5078a7e06ae..ff92c295fe4 100644 --- a/src/Disks/S3/registerDiskS3.cpp +++ b/src/Disks/S3/registerDiskS3.cpp @@ -134,6 +134,7 @@ void registerDiskS3(DiskFactory & factory) uri.is_virtual_hosted_style, config.getString(config_prefix + ".access_key_id", ""), config.getString(config_prefix + ".secret_access_key", ""), + config.getBool(config_prefix + ".use_environment_credentials", config.getBool("s3.use_environment_credentials", false)), context.getRemoteHostFilter(), context.getGlobalContext().getSettingsRef().s3_max_redirects); @@ -148,7 +149,8 @@ void registerDiskS3(DiskFactory & factory) metadata_path, context.getSettingsRef().s3_min_upload_part_size, config.getUInt64(config_prefix + ".min_multi_part_upload_size", 10 * 1024 * 1024), - config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024)); + config.getUInt64(config_prefix + ".min_bytes_for_seek", 1024 * 1024), + config.getBool(config_prefix + ".send_object_metadata", false)); /// This code is used only to check access to the corresponding disk. if (!config.getBool(config_prefix + ".skip_access_check", false)) diff --git a/src/Formats/MySQLBlockInputStream.cpp b/src/Formats/MySQLBlockInputStream.cpp index 73def337240..2ff8e8e5fb2 100644 --- a/src/Formats/MySQLBlockInputStream.cpp +++ b/src/Formats/MySQLBlockInputStream.cpp @@ -12,6 +12,7 @@ # include # include # include +# include # include # include # include "MySQLBlockInputStream.h" @@ -37,17 +38,15 @@ MySQLBlockInputStream::MySQLBlockInputStream( const std::string & query_str, const Block & sample_block, const UInt64 max_block_size_, - const bool auto_close_) + const bool auto_close_, + const bool fetch_by_name_) : connection{std::make_unique(entry, query_str)} , max_block_size{max_block_size_} , auto_close{auto_close_} + , fetch_by_name(fetch_by_name_) { - if (sample_block.columns() != connection->result.getNumFields()) - throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " - + toString(sample_block.columns()) + " expected", - ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; - description.init(sample_block); + initPositionMappingFromQueryResultStructure(); } @@ -135,24 +134,25 @@ Block MySQLBlockInputStream::readImpl() size_t num_rows = 0; while (row) { - for (const auto idx : ext::range(0, row.size())) + for (size_t index = 0; index < position_mapping.size(); ++index) { - const auto value = row[idx]; - const auto & sample = description.sample_block.getByPosition(idx); + const auto value = row[position_mapping[index]]; + const auto & sample = description.sample_block.getByPosition(index); + if (!value.isNull()) { - if (description.types[idx].second) + if (description.types[index].second) { - ColumnNullable & column_nullable = assert_cast(*columns[idx]); + ColumnNullable & column_nullable = assert_cast(*columns[index]); const auto & data_type = assert_cast(*sample.type); - insertValue(*data_type.getNestedType(), column_nullable.getNestedColumn(), description.types[idx].first, value); + insertValue(*data_type.getNestedType(), column_nullable.getNestedColumn(), description.types[index].first, value); column_nullable.getNullMapData().emplace_back(0); } else - insertValue(*sample.type, *columns[idx], description.types[idx].first, value); + insertValue(*sample.type, *columns[index], description.types[index].first, value); } else - insertDefaultValue(*columns[idx], *sample.column); + insertDefaultValue(*columns[index], *sample.column); } ++num_rows; @@ -167,20 +167,70 @@ Block MySQLBlockInputStream::readImpl() MySQLBlockInputStream::MySQLBlockInputStream( const Block & sample_block_, UInt64 max_block_size_, - bool auto_close_) + bool auto_close_, + bool fetch_by_name_) : max_block_size(max_block_size_) , auto_close(auto_close_) + , fetch_by_name(fetch_by_name_) { description.init(sample_block_); } +void MySQLBlockInputStream::initPositionMappingFromQueryResultStructure() +{ + position_mapping.resize(description.sample_block.columns()); + + if (!fetch_by_name) + { + if (description.sample_block.columns() != connection->result.getNumFields()) + throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " + + toString(description.sample_block.columns()) + " expected", ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; + + for (const auto idx : ext::range(0, connection->result.getNumFields())) + position_mapping[idx] = idx; + } + else + { + const auto & sample_names = description.sample_block.getNames(); + std::unordered_set missing_names(sample_names.begin(), sample_names.end()); + + size_t fields_size = connection->result.getNumFields(); + + for (const size_t & idx : ext::range(0, fields_size)) + { + const auto & field_name = connection->result.getFieldName(idx); + if (description.sample_block.has(field_name)) + { + const auto & position = description.sample_block.getPositionByName(field_name); + position_mapping[position] = idx; + missing_names.erase(field_name); + } + } + + if (!missing_names.empty()) + { + WriteBufferFromOwnString exception_message; + for (auto iter = missing_names.begin(); iter != missing_names.end(); ++iter) + { + if (iter != missing_names.begin()) + exception_message << ", "; + exception_message << *iter; + } + + throw Exception("mysqlxx::UseQueryResult must be contain the" + exception_message.str() + " columns.", + ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); + } + } +} + MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( mysqlxx::Pool & pool_, const std::string & query_str_, const Block & sample_block_, const UInt64 max_block_size_, - const bool auto_close_) - : MySQLBlockInputStream(sample_block_, max_block_size_, auto_close_) + const bool auto_close_, + const bool fetch_by_name_) + : MySQLBlockInputStream(sample_block_, max_block_size_, auto_close_, fetch_by_name_) , pool(pool_) , query_str(query_str_) { @@ -189,10 +239,7 @@ MySQLLazyBlockInputStream::MySQLLazyBlockInputStream( void MySQLLazyBlockInputStream::readPrefix() { connection = std::make_unique(pool.get(), query_str); - if (description.sample_block.columns() != connection->result.getNumFields()) - throw Exception{"mysqlxx::UseQueryResult contains " + toString(connection->result.getNumFields()) + " columns while " - + toString(description.sample_block.columns()) + " expected", - ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH}; + initPositionMappingFromQueryResultStructure(); } } diff --git a/src/Formats/MySQLBlockInputStream.h b/src/Formats/MySQLBlockInputStream.h index 2eaeb5b8d59..269b630fcc7 100644 --- a/src/Formats/MySQLBlockInputStream.h +++ b/src/Formats/MySQLBlockInputStream.h @@ -20,15 +20,17 @@ public: const std::string & query_str, const Block & sample_block, const UInt64 max_block_size_, - const bool auto_close_ = false); + const bool auto_close_ = false, + const bool fetch_by_name_ = false); String getName() const override { return "MySQL"; } Block getHeader() const override { return description.sample_block.cloneEmpty(); } protected: - MySQLBlockInputStream(const Block & sample_block_, UInt64 max_block_size_, bool auto_close_); + MySQLBlockInputStream(const Block & sample_block_, UInt64 max_block_size_, bool auto_close_, bool fetch_by_name_); Block readImpl() override; + void initPositionMappingFromQueryResultStructure(); struct Connection { @@ -43,6 +45,8 @@ protected: const UInt64 max_block_size; const bool auto_close; + const bool fetch_by_name; + std::vector position_mapping; ExternalResultDescription description; }; @@ -56,7 +60,8 @@ public: const std::string & query_str_, const Block & sample_block_, const UInt64 max_block_size_, - const bool auto_close_ = false); + const bool auto_close_ = false, + const bool fetch_by_name_ = false); private: void readPrefix() override; diff --git a/src/Functions/GregorianDate.h b/src/Functions/GregorianDate.h new file mode 100644 index 00000000000..844fb180e5e --- /dev/null +++ b/src/Functions/GregorianDate.h @@ -0,0 +1,409 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#include + +namespace DB +{ + namespace ErrorCodes + { + extern const int CANNOT_PARSE_INPUT_ASSERTION_FAILED; + extern const int CANNOT_PARSE_DATE; + extern const int CANNOT_FORMAT_DATETIME; + extern const int LOGICAL_ERROR; + } + + /** Proleptic Gregorian calendar date. YearT is an integral type + * which should be at least 32 bits wide, and should preferably + * be signed. + */ + template + class GregorianDate + { + public: + GregorianDate() = delete; + + /** Construct from date in text form 'YYYY-MM-DD' by reading from + * ReadBuffer. + */ + GregorianDate(ReadBuffer & in); + + /** Construct from Modified Julian Day. The type T is an + * integral type which should be at least 32 bits wide, and + * should preferably signed. + */ + template ()> * = nullptr> + GregorianDate(T mjd); + + /** Convert to Modified Julian Day. The type T is an integral type + * which should be at least 32 bits wide, and should preferably + * signed. + */ + template ()> * = nullptr> + T toModifiedJulianDay() const; + + /** Write the date in text form 'YYYY-MM-DD' to a buffer. + */ + void write(WriteBuffer & buf) const; + + /** Convert to a string in text form 'YYYY-MM-DD'. + */ + std::string toString() const; + + YearT year() const noexcept + { + return year_; + } + + uint8_t month() const noexcept + { + return month_; + } + + uint8_t day_of_month() const noexcept + { + return day_of_month_; + } + + private: + YearT year_; + uint8_t month_; + uint8_t day_of_month_; + }; + + /** ISO 8601 Ordinal Date. YearT is an integral type which should + * be at least 32 bits wide, and should preferably signed. + */ + template + class OrdinalDate + { + public: + OrdinalDate(YearT year, uint16_t day_of_year); + + /** Construct from Modified Julian Day. The type T is an + * integral type which should be at least 32 bits wide, and + * should preferably signed. + */ + template ()> * = nullptr> + OrdinalDate(T mjd); + + /** Convert to Modified Julian Day. The type T is an integral + * type which should be at least 32 bits wide, and should + * preferably be signed. + */ + template ()> * = nullptr> + T toModifiedJulianDay() const noexcept; + + YearT year() const noexcept + { + return year_; + } + + uint16_t dayOfYear() const noexcept + { + return day_of_year_; + } + + private: + YearT year_; + uint16_t day_of_year_; + }; + + class MonthDay + { + public: + /** Construct from month and day. */ + MonthDay(uint8_t month, uint8_t day_of_month); + + /** Construct from day of year in Gregorian or Julian + * calendars to month and day. + */ + MonthDay(bool is_leap_year, uint16_t day_of_year); + + /** Convert month and day in Gregorian or Julian calendars to + * day of year. + */ + uint16_t dayOfYear(bool is_leap_year) const; + + uint8_t month() const noexcept + { + return month_; + } + + uint8_t day_of_month() const noexcept + { + return day_of_month_; + } + + private: + uint8_t month_; + uint8_t day_of_month_; + }; +} + +/* Implementation */ + +namespace gd +{ + using namespace DB; + + template + static inline constexpr bool is_leap_year(YearT year) + { + return (year % 4 == 0) && ((year % 400 == 0) || (year % 100 != 0)); + } + + static inline constexpr uint8_t monthLength(bool is_leap_year, uint8_t month) + { + switch (month) + { + case 1: return 31; + case 2: return is_leap_year ? 29 : 28; + case 3: return 31; + case 4: return 30; + case 5: return 31; + case 6: return 30; + case 7: return 31; + case 8: return 31; + case 9: return 30; + case 10: return 31; + case 11: return 30; + case 12: return 31; + default: + std::terminate(); + } + } + + /** Integer division truncated toward negative infinity. + */ + template + static inline constexpr I div(I x, J y) + { + const auto y_ = static_cast(y); + if (x > 0 && y_ < 0) + return ((x - 1) / y_) - 1; + else if (x < 0 && y_ > 0) + return ((x + 1) / y_) - 1; + else + return x / y_; + } + + /** Integer modulus, satisfying div(x, y)*y + mod(x, y) == x. + */ + template + static inline constexpr I mod(I x, J y) + { + const auto y_ = static_cast(y); + const auto r = x % y_; + if ((x > 0 && y_ < 0) || (x < 0 && y_ > 0)) + return r == 0 ? static_cast(0) : r + y_; + else + return r; + } + + /** Like std::min(), but the type of operands may differ. + */ + template + static inline constexpr I min(I x, J y) + { + const auto y_ = static_cast(y); + return x < y_ ? x : y_; + } + + static inline char readDigit(ReadBuffer & in) + { + char c; + if (!in.read(c)) + throw Exception( + "Cannot parse input: expected a digit at the end of stream", + ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); + else if (c < '0' || c > '9') + throw Exception( + "Cannot read input: expected a digit but got something else", + ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED); + else + return c - '0'; + } +} + +namespace DB +{ + template + GregorianDate::GregorianDate(ReadBuffer & in) + { + year_ = gd::readDigit(in) * 1000 + + gd::readDigit(in) * 100 + + gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertChar('-', in); + + month_ = gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertChar('-', in); + + day_of_month_ = gd::readDigit(in) * 10 + + gd::readDigit(in); + + assertEOF(in); + + if (month_ < 1 || month_ > 12 || day_of_month_ < 1 || day_of_month_ > gd::monthLength(gd::is_leap_year(year_), month_)) + throw Exception("Invalid date: " + toString(), ErrorCodes::CANNOT_PARSE_DATE); + } + + template + template ()> *> + GregorianDate::GregorianDate(T mjd) + { + const OrdinalDate ord(mjd); + const MonthDay md(gd::is_leap_year(ord.year()), ord.dayOfYear()); + year_ = ord.year(); + month_ = md.month(); + day_of_month_ = md.day_of_month(); + } + + template + template ()> *> + T GregorianDate::toModifiedJulianDay() const + { + const MonthDay md(month_, day_of_month_); + const auto day_of_year = md.dayOfYear(gd::is_leap_year(year_)); + const OrdinalDate ord(year_, day_of_year); + return ord.template toModifiedJulianDay(); + } + + template + void GregorianDate::write(WriteBuffer & buf) const + { + if (year_ < 0 || year_ > 9999) + { + throw Exception( + "Impossible to stringify: year too big or small: " + DB::toString(year_), + ErrorCodes::CANNOT_FORMAT_DATETIME); + } + else + { + auto y = year_; + writeChar('0' + y / 1000, buf); y %= 1000; + writeChar('0' + y / 100, buf); y %= 100; + writeChar('0' + y / 10, buf); y %= 10; + writeChar('0' + y , buf); + + writeChar('-', buf); + + auto m = month_; + writeChar('0' + m / 10, buf); m %= 10; + writeChar('0' + m , buf); + + writeChar('-', buf); + + auto d = day_of_month_; + writeChar('0' + d / 10, buf); d %= 10; + writeChar('0' + d , buf); + } + } + + template + std::string GregorianDate::toString() const + { + WriteBufferFromOwnString buf; + write(buf); + return buf.str(); + } + + template + OrdinalDate::OrdinalDate(YearT year, uint16_t day_of_year) + : year_(year) + , day_of_year_(day_of_year) + { + if (day_of_year < 1 || day_of_year > (gd::is_leap_year(year) ? 366 : 365)) + { + throw Exception( + "Invalid ordinal date: " + toString(year) + "-" + toString(day_of_year), + ErrorCodes::LOGICAL_ERROR); + } + } + + template + template ()> *> + OrdinalDate::OrdinalDate(T mjd) + { + const auto a = mjd + 678575; + const auto quad_cent = gd::div(a, 146097); + const auto b = gd::mod(a, 146097); + const auto cent = gd::min(gd::div(b, 36524), 3); + const auto c = b - cent * 36524; + const auto quad = gd::div(c, 1461); + const auto d = gd::mod(c, 1461); + const auto y = gd::min(gd::div(d, 365), 3); + day_of_year_ = d - y * 365 + 1; + year_ = quad_cent * 400 + cent * 100 + quad * 4 + y + 1; + } + + template + template ()> *> + T OrdinalDate::toModifiedJulianDay() const noexcept + { + const auto y = year_ - 1; + return day_of_year_ + + 365 * y + + gd::div(y, 4) + - gd::div(y, 100) + + gd::div(y, 400) + - 678576; + } + + inline MonthDay::MonthDay(uint8_t month, uint8_t day_of_month) + : month_(month) + , day_of_month_(day_of_month) + { + if (month < 1 || month > 12) + throw Exception( + "Invalid month: " + DB::toString(month), + ErrorCodes::LOGICAL_ERROR); + /* We can't validate day_of_month here, because we don't know if + * it's a leap year. */ + } + + inline MonthDay::MonthDay(bool is_leap_year, uint16_t day_of_year) + { + if (day_of_year < 1 || day_of_year > (is_leap_year ? 366 : 365)) + throw Exception( + std::string("Invalid day of year: ") + + (is_leap_year ? "leap, " : "non-leap, ") + DB::toString(day_of_year), + ErrorCodes::LOGICAL_ERROR); + + month_ = 1; + uint16_t d = day_of_year; + while (true) + { + const auto len = gd::monthLength(is_leap_year, month_); + if (d <= len) + break; + month_++; + d -= len; + } + day_of_month_ = d; + } + + inline uint16_t MonthDay::dayOfYear(bool is_leap_year) const + { + if (day_of_month_ < 1 || day_of_month_ > gd::monthLength(is_leap_year, month_)) + { + throw Exception( + std::string("Invalid day of month: ") + + (is_leap_year ? "leap, " : "non-leap, ") + DB::toString(month_) + + "-" + DB::toString(day_of_month_), + ErrorCodes::LOGICAL_ERROR); + } + const auto k = month_ <= 2 ? 0 : is_leap_year ? -1 :-2; + return (367 * month_ - 362) / 12 + k + day_of_month_; + } +} diff --git a/src/Functions/URL/firstSignificantSubdomain.h b/src/Functions/URL/ExtractFirstSignificantSubdomain.h similarity index 77% rename from src/Functions/URL/firstSignificantSubdomain.h rename to src/Functions/URL/ExtractFirstSignificantSubdomain.h index 522e7905f69..c13b5f50156 100644 --- a/src/Functions/URL/firstSignificantSubdomain.h +++ b/src/Functions/URL/ExtractFirstSignificantSubdomain.h @@ -7,12 +7,27 @@ namespace DB { +struct FirstSignificantSubdomainDefaultLookup +{ + bool operator()(const char *src, size_t len) const + { + return tldLookup::isValid(src, len); + } +}; + template struct ExtractFirstSignificantSubdomain { static size_t getReserveLengthForElement() { return 10; } static void execute(const Pos data, const size_t size, Pos & res_data, size_t & res_size, Pos * out_domain_end = nullptr) + { + FirstSignificantSubdomainDefaultLookup loookup; + return execute(loookup, data, size, res_data, res_size, out_domain_end); + } + + template + static void execute(const Lookup & lookup, const Pos data, const size_t size, Pos & res_data, size_t & res_size, Pos * out_domain_end = nullptr) { res_data = data; res_size = 0; @@ -65,7 +80,7 @@ struct ExtractFirstSignificantSubdomain end_of_level_domain = end; } - if (tldLookup::isValid(last_3_periods[1] + 1, end_of_level_domain - last_3_periods[1] - 1) != nullptr) + if (lookup(last_3_periods[1] + 1, end_of_level_domain - last_3_periods[1] - 1)) { res_data += last_3_periods[2] + 1 - begin; res_size = last_3_periods[1] - last_3_periods[2] - 1; diff --git a/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h b/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h new file mode 100644 index 00000000000..244b32459c1 --- /dev/null +++ b/src/Functions/URL/FirstSignificantSubdomainCustomImpl.h @@ -0,0 +1,112 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int ILLEGAL_COLUMN; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; +} + +struct FirstSignificantSubdomainCustomtLookup +{ + const TLDList & tld_list; + FirstSignificantSubdomainCustomtLookup(const std::string & tld_list_name) + : tld_list(TLDListsHolder::getInstance().getTldList(tld_list_name)) + { + } + + bool operator()(const char *pos, size_t len) const + { + return tld_list.has(StringRef{pos, len}); + } +}; + +template +class FunctionCutToFirstSignificantSubdomainCustomImpl : public IFunction +{ +public: + static constexpr auto name = Name::name; + static FunctionPtr create(const Context &) { return std::make_shared(); } + + String getName() const override { return name; } + size_t getNumberOfArguments() const override { return 2; } + + DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override + { + if (!isString(arguments[0].type)) + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + "Illegal type {} of first argument of function {}. Must be String.", + arguments[0].type->getName(), getName()); + if (!isString(arguments[1].type)) + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + "Illegal type {} of second argument (TLD_list_name) of function {}. Must be String/FixedString.", + arguments[1].type->getName(), getName()); + const auto * column = arguments[1].column.get(); + if (!column || !checkAndGetColumnConstStringOrFixedString(column)) + throw Exception(ErrorCodes::ILLEGAL_COLUMN, + "The second argument of function {} should be a constant string with the name of the custom TLD", + getName()); + + return arguments[0].type; + } + + ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr & /*result_type*/, size_t /*input_rows_count*/) const override + { + const ColumnConst * column_tld_list_name = checkAndGetColumnConstStringOrFixedString(arguments[1].column.get()); + FirstSignificantSubdomainCustomtLookup tld_lookup(column_tld_list_name->getValue()); + + /// FIXME: convertToFullColumnIfConst() is suboptimal + auto column = arguments[0].column->convertToFullColumnIfConst(); + if (const ColumnString * col = checkAndGetColumn(*column)) + { + auto col_res = ColumnString::create(); + vector(tld_lookup, col->getChars(), col->getOffsets(), col_res->getChars(), col_res->getOffsets()); + return col_res; + } + else + throw Exception( + "Illegal column " + arguments[0].column->getName() + " of argument of function " + getName(), + ErrorCodes::ILLEGAL_COLUMN); + } + + static void vector(FirstSignificantSubdomainCustomtLookup & tld_lookup, + const ColumnString::Chars & data, const ColumnString::Offsets & offsets, + ColumnString::Chars & res_data, ColumnString::Offsets & res_offsets) + { + size_t size = offsets.size(); + res_offsets.resize(size); + res_data.reserve(size * Extractor::getReserveLengthForElement()); + + size_t prev_offset = 0; + size_t res_offset = 0; + + /// Matched part. + Pos start; + size_t length; + + for (size_t i = 0; i < size; ++i) + { + Extractor::execute(tld_lookup, reinterpret_cast(&data[prev_offset]), offsets[i] - prev_offset - 1, start, length); + + res_data.resize(res_data.size() + length + 1); + memcpySmallAllowReadWriteOverflow15(&res_data[res_offset], start, length); + res_offset += length + 1; + res_data[res_offset - 1] = 0; + + res_offsets[i] = res_offset; + prev_offset = offsets[i]; + } + } +}; + +} diff --git a/src/Functions/URL/cutToFirstSignificantSubdomain.cpp b/src/Functions/URL/cutToFirstSignificantSubdomain.cpp index 43d614a7036..82eb366dae6 100644 --- a/src/Functions/URL/cutToFirstSignificantSubdomain.cpp +++ b/src/Functions/URL/cutToFirstSignificantSubdomain.cpp @@ -1,6 +1,6 @@ #include #include -#include "firstSignificantSubdomain.h" +#include "ExtractFirstSignificantSubdomain.h" namespace DB diff --git a/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp b/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp new file mode 100644 index 00000000000..11fd27e317b --- /dev/null +++ b/src/Functions/URL/cutToFirstSignificantSubdomainCustom.cpp @@ -0,0 +1,43 @@ +#include +#include "ExtractFirstSignificantSubdomain.h" +#include "FirstSignificantSubdomainCustomImpl.h" + +namespace DB +{ + +template +struct CutToFirstSignificantSubdomainCustom +{ + static size_t getReserveLengthForElement() { return 15; } + + static void execute(FirstSignificantSubdomainCustomtLookup & tld_lookup, const Pos data, const size_t size, Pos & res_data, size_t & res_size) + { + res_data = data; + res_size = 0; + + Pos tmp_data; + size_t tmp_length; + Pos domain_end; + ExtractFirstSignificantSubdomain::execute(tld_lookup, data, size, tmp_data, tmp_length, &domain_end); + + if (tmp_length == 0) + return; + + res_data = tmp_data; + res_size = domain_end - tmp_data; + } +}; + +struct NameCutToFirstSignificantSubdomainCustom { static constexpr auto name = "cutToFirstSignificantSubdomainCustom"; }; +using FunctionCutToFirstSignificantSubdomainCustom = FunctionCutToFirstSignificantSubdomainCustomImpl, NameCutToFirstSignificantSubdomainCustom>; + +struct NameCutToFirstSignificantSubdomainCustomWithWWW { static constexpr auto name = "cutToFirstSignificantSubdomainCustomWithWWW"; }; +using FunctionCutToFirstSignificantSubdomainCustomWithWWW = FunctionCutToFirstSignificantSubdomainCustomImpl, NameCutToFirstSignificantSubdomainCustomWithWWW>; + +void registerFunctionCutToFirstSignificantSubdomainCustom(FunctionFactory & factory) +{ + factory.registerFunction(); + factory.registerFunction(); +} + +} diff --git a/src/Functions/URL/firstSignificantSubdomain.cpp b/src/Functions/URL/firstSignificantSubdomain.cpp index 7db18824375..87659940938 100644 --- a/src/Functions/URL/firstSignificantSubdomain.cpp +++ b/src/Functions/URL/firstSignificantSubdomain.cpp @@ -1,12 +1,13 @@ #include #include -#include "firstSignificantSubdomain.h" +#include "ExtractFirstSignificantSubdomain.h" namespace DB { struct NameFirstSignificantSubdomain { static constexpr auto name = "firstSignificantSubdomain"; }; + using FunctionFirstSignificantSubdomain = FunctionStringToString>, NameFirstSignificantSubdomain>; void registerFunctionFirstSignificantSubdomain(FunctionFactory & factory) diff --git a/src/Functions/URL/firstSignificantSubdomainCustom.cpp b/src/Functions/URL/firstSignificantSubdomainCustom.cpp new file mode 100644 index 00000000000..675b4a346de --- /dev/null +++ b/src/Functions/URL/firstSignificantSubdomainCustom.cpp @@ -0,0 +1,18 @@ +#include +#include "ExtractFirstSignificantSubdomain.h" +#include "FirstSignificantSubdomainCustomImpl.h" + + +namespace DB +{ + +struct NameFirstSignificantSubdomainCustom { static constexpr auto name = "firstSignificantSubdomainCustom"; }; + +using FunctionFirstSignificantSubdomainCustom = FunctionCutToFirstSignificantSubdomainCustomImpl, NameFirstSignificantSubdomainCustom>; + +void registerFunctionFirstSignificantSubdomainCustom(FunctionFactory & factory) +{ + factory.registerFunction(); +} + +} diff --git a/src/Functions/URL/registerFunctionsURL.cpp b/src/Functions/URL/registerFunctionsURL.cpp index f3906c2723e..91118074b7a 100644 --- a/src/Functions/URL/registerFunctionsURL.cpp +++ b/src/Functions/URL/registerFunctionsURL.cpp @@ -7,6 +7,7 @@ void registerFunctionProtocol(FunctionFactory & factory); void registerFunctionDomain(FunctionFactory & factory); void registerFunctionDomainWithoutWWW(FunctionFactory & factory); void registerFunctionFirstSignificantSubdomain(FunctionFactory & factory); +void registerFunctionFirstSignificantSubdomainCustom(FunctionFactory & factory); void registerFunctionTopLevelDomain(FunctionFactory & factory); void registerFunctionPort(FunctionFactory & factory); void registerFunctionPath(FunctionFactory & factory); @@ -20,6 +21,7 @@ void registerFunctionExtractURLParameterNames(FunctionFactory & factory); void registerFunctionURLHierarchy(FunctionFactory & factory); void registerFunctionURLPathHierarchy(FunctionFactory & factory); void registerFunctionCutToFirstSignificantSubdomain(FunctionFactory & factory); +void registerFunctionCutToFirstSignificantSubdomainCustom(FunctionFactory & factory); void registerFunctionCutWWW(FunctionFactory & factory); void registerFunctionCutQueryString(FunctionFactory & factory); void registerFunctionCutFragment(FunctionFactory & factory); @@ -34,6 +36,7 @@ void registerFunctionsURL(FunctionFactory & factory) registerFunctionDomain(factory); registerFunctionDomainWithoutWWW(factory); registerFunctionFirstSignificantSubdomain(factory); + registerFunctionFirstSignificantSubdomainCustom(factory); registerFunctionTopLevelDomain(factory); registerFunctionPort(factory); registerFunctionPath(factory); @@ -47,6 +50,7 @@ void registerFunctionsURL(FunctionFactory & factory) registerFunctionURLHierarchy(factory); registerFunctionURLPathHierarchy(factory); registerFunctionCutToFirstSignificantSubdomain(factory); + registerFunctionCutToFirstSignificantSubdomainCustom(factory); registerFunctionCutWWW(factory); registerFunctionCutQueryString(factory); registerFunctionCutFragment(factory); diff --git a/src/Functions/URL/tldLookup.h b/src/Functions/URL/tldLookup.h index 25857be3dd2..38c118b6bb1 100644 --- a/src/Functions/URL/tldLookup.h +++ b/src/Functions/URL/tldLookup.h @@ -1,5 +1,7 @@ #pragma once +#include + // Definition of the class generated by gperf, present on gperf/tldLookup.gperf class TopLevelDomainLookupHash { diff --git a/src/Functions/addressToLine.cpp b/src/Functions/addressToLine.cpp index 0cee8365531..59e347dd348 100644 --- a/src/Functions/addressToLine.cpp +++ b/src/Functions/addressToLine.cpp @@ -106,7 +106,8 @@ private: StringRef impl(uintptr_t addr) const { - const SymbolIndex & symbol_index = SymbolIndex::instance(); + auto symbol_index_ptr = SymbolIndex::instance(); + const SymbolIndex & symbol_index = *symbol_index_ptr; if (const auto * object = symbol_index.findObject(reinterpret_cast(addr))) { diff --git a/src/Functions/addressToSymbol.cpp b/src/Functions/addressToSymbol.cpp index e0c87b7bbb9..5accfd1dd94 100644 --- a/src/Functions/addressToSymbol.cpp +++ b/src/Functions/addressToSymbol.cpp @@ -66,7 +66,8 @@ public: ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override { - const SymbolIndex & symbol_index = SymbolIndex::instance(); + auto symbol_index_ptr = SymbolIndex::instance(); + const SymbolIndex & symbol_index = *symbol_index_ptr; const ColumnPtr & column = arguments[0].column; const ColumnUInt64 * column_concrete = checkAndGetColumn(column.get()); diff --git a/src/Functions/buildId.cpp b/src/Functions/buildId.cpp index ffe3da27153..7361f0e4546 100644 --- a/src/Functions/buildId.cpp +++ b/src/Functions/buildId.cpp @@ -40,7 +40,7 @@ public: ColumnPtr executeImpl(const ColumnsWithTypeAndName &, const DataTypePtr &, size_t input_rows_count) const override { - return DataTypeString().createColumnConst(input_rows_count, SymbolIndex::instance().getBuildIDHex()); + return DataTypeString().createColumnConst(input_rows_count, SymbolIndex::instance()->getBuildIDHex()); } }; diff --git a/src/Functions/fromModifiedJulianDay.cpp b/src/Functions/fromModifiedJulianDay.cpp new file mode 100644 index 00000000000..636512db0de --- /dev/null +++ b/src/Functions/fromModifiedJulianDay.cpp @@ -0,0 +1,237 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + + namespace ErrorCodes + { + extern const int CANNOT_FORMAT_DATETIME; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + } + + template + class ExecutableFunctionFromModifiedJulianDay : public IExecutableFunctionImpl + { + public: + String getName() const override + { + return Name::name; + } + + ColumnPtr execute(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override + { + using ColVecType = typename FromDataType::ColumnType; + const ColVecType * col_from = checkAndGetColumn(arguments[0].column.get()); + const typename ColVecType::Container & vec_from = col_from->getData(); + + auto col_to = ColumnString::create(); + ColumnString::Chars & data_to = col_to->getChars(); + ColumnString::Offsets & offsets_to = col_to->getOffsets(); + data_to.resize(input_rows_count * strlen("YYYY-MM-DD") + 1); + offsets_to.resize(input_rows_count); + + ColumnUInt8::MutablePtr col_null_map_to; + ColumnUInt8::Container * vec_null_map_to [[maybe_unused]] = nullptr; + if constexpr (nullOnErrors) + { + col_null_map_to = ColumnUInt8::create(input_rows_count); + vec_null_map_to = &col_null_map_to->getData(); + } + + WriteBufferFromVector write_buffer(data_to); + for (size_t i = 0; i < input_rows_count; ++i) + { + if constexpr (nullOnErrors) + { + try + { + const GregorianDate<> gd(vec_from[i]); + gd.write(write_buffer); + (*vec_null_map_to)[i] = false; + } + catch (const Exception & e) + { + if (e.code() == ErrorCodes::CANNOT_FORMAT_DATETIME) + (*vec_null_map_to)[i] = true; + else + throw; + } + writeChar(0, write_buffer); + offsets_to[i] = write_buffer.count(); + } + else + { + const GregorianDate<> gd(vec_from[i]); + gd.write(write_buffer); + writeChar(0, write_buffer); + offsets_to[i] = write_buffer.count(); + } + } + write_buffer.finalize(); + + if constexpr (nullOnErrors) + return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); + else + return col_to; + } + + bool useDefaultImplementationForConstants() const override + { + return true; + } + }; + + template + class FunctionBaseFromModifiedJulianDay : public IFunctionBaseImpl + { + public: + explicit FunctionBaseFromModifiedJulianDay(DataTypes argument_types_, DataTypePtr return_type_) + : argument_types(std::move(argument_types_)) + , return_type(std::move(return_type_)) {} + + String getName() const override + { + return Name::name; + } + + const DataTypes & getArgumentTypes() const override + { + return argument_types; + } + + const DataTypePtr & getResultType() const override + { + return return_type; + } + + ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override + { + return std::make_unique>(); + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + + bool hasInformationAboutMonotonicity() const override + { + return true; + } + + Monotonicity getMonotonicityForRange(const IDataType &, const Field &, const Field &) const override + { + return Monotonicity( + true, // is_monotonic + true, // is_positive + true); // is_always_monotonic + } + + private: + DataTypes argument_types; + DataTypePtr return_type; + }; + + template + class FromModifiedJulianDayOverloadResolver : public IFunctionOverloadResolverImpl + { + public: + static constexpr auto name = Name::name; + + static FunctionOverloadResolverImplPtr create(const Context &) + { + return std::make_unique>(); + } + + String getName() const override + { + return Name::name; + } + + FunctionBaseImplPtr build(const ColumnsWithTypeAndName & arguments, const DataTypePtr & return_type) const override + { + const DataTypePtr & from_type = arguments[0].type; + DataTypes argument_types = { from_type }; + FunctionBaseImplPtr base; + auto call = [&](const auto & types) -> bool + { + using Types = std::decay_t; + using FromIntType = typename Types::RightType; + using FromDataType = DataTypeNumber; + + base = std::make_unique>(argument_types, return_type); + return true; + }; + bool built = callOnBasicType(from_type->getTypeId(), call); + if (built) + return base; + + /* When the argument is a NULL constant, the resulting + * function base will not be actually called but it + * will still be inspected. Returning a NULL pointer + * here causes a SEGV. So we must somehow create a + * dummy implementation and return it. + */ + if (WhichDataType(from_type).isNullable()) // Nullable(Nothing) + return std::make_unique>(argument_types, return_type); + else + // Should not happen. + throw Exception( + "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + + DataTypePtr getReturnType(const DataTypes & arguments) const override + { + if (!isInteger(arguments[0])) + { + throw Exception( + "The argument of function " + getName() + " must be integral", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + + DataTypePtr base_type = std::make_shared(); + if constexpr (nullOnErrors) + return std::make_shared(base_type); + else + return base_type; + } + + size_t getNumberOfArguments() const override + { + return 1; + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + }; + + struct NameFromModifiedJulianDay + { + static constexpr auto name = "fromModifiedJulianDay"; + }; + + struct NameFromModifiedJulianDayOrNull + { + static constexpr auto name = "fromModifiedJulianDayOrNull"; + }; + + void registerFunctionFromModifiedJulianDay(FunctionFactory & factory) + { + factory.registerFunction>(); + factory.registerFunction>(); + } +} diff --git a/src/Functions/registerFunctionsDateTime.cpp b/src/Functions/registerFunctionsDateTime.cpp index e95ab230087..2cb737b8c75 100644 --- a/src/Functions/registerFunctionsDateTime.cpp +++ b/src/Functions/registerFunctionsDateTime.cpp @@ -18,6 +18,7 @@ void registerFunctionToMonday(FunctionFactory &); void registerFunctionToISOWeek(FunctionFactory &); void registerFunctionToISOYear(FunctionFactory &); void registerFunctionToCustomWeek(FunctionFactory &); +void registerFunctionToModifiedJulianDay(FunctionFactory &); void registerFunctionToStartOfMonth(FunctionFactory &); void registerFunctionToStartOfQuarter(FunctionFactory &); void registerFunctionToStartOfYear(FunctionFactory &); @@ -65,6 +66,7 @@ void registerFunctionSubtractYears(FunctionFactory &); void registerFunctionDateDiff(FunctionFactory &); void registerFunctionToTimeZone(FunctionFactory &); void registerFunctionFormatDateTime(FunctionFactory &); +void registerFunctionFromModifiedJulianDay(FunctionFactory &); void registerFunctionDateTrunc(FunctionFactory &); void registerFunctionsDateTime(FunctionFactory & factory) @@ -83,6 +85,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionToISOWeek(factory); registerFunctionToISOYear(factory); registerFunctionToCustomWeek(factory); + registerFunctionToModifiedJulianDay(factory); registerFunctionToStartOfMonth(factory); registerFunctionToStartOfQuarter(factory); registerFunctionToStartOfYear(factory); @@ -131,6 +134,7 @@ void registerFunctionsDateTime(FunctionFactory & factory) registerFunctionDateDiff(factory); registerFunctionToTimeZone(factory); registerFunctionFormatDateTime(factory); + registerFunctionFromModifiedJulianDay(factory); registerFunctionDateTrunc(factory); } diff --git a/src/Functions/toModifiedJulianDay.cpp b/src/Functions/toModifiedJulianDay.cpp new file mode 100644 index 00000000000..df3066a5005 --- /dev/null +++ b/src/Functions/toModifiedJulianDay.cpp @@ -0,0 +1,234 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + namespace ErrorCodes + { + extern const int ILLEGAL_COLUMN; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int CANNOT_PARSE_INPUT_ASSERTION_FAILED; + extern const int CANNOT_PARSE_DATE; + } + + template + class ExecutableFunctionToModifiedJulianDay : public IExecutableFunctionImpl + { + public: + String getName() const override + { + return Name::name; + } + + ColumnPtr execute(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override + { + const IColumn * col_from = arguments[0].column.get(); + const ColumnString * col_from_string = checkAndGetColumn(col_from); + const ColumnFixedString * col_from_fixed_string = checkAndGetColumn(col_from); + + const ColumnString::Chars * chars = nullptr; + const IColumn::Offsets * offsets = nullptr; + size_t fixed_string_size = 0; + + if (col_from_string) + { + chars = &col_from_string->getChars(); + offsets = &col_from_string->getOffsets(); + } + else if (col_from_fixed_string) + { + chars = &col_from_fixed_string->getChars(); + fixed_string_size = col_from_fixed_string->getN(); + } + else + { + throw Exception("Illegal column " + col_from->getName() + + " of first argument of function " + Name::name, + ErrorCodes::ILLEGAL_COLUMN); + } + + using ColVecTo = typename ToDataType::ColumnType; + typename ColVecTo::MutablePtr col_to = ColVecTo::create(input_rows_count); + typename ColVecTo::Container & vec_to = col_to->getData(); + + ColumnUInt8::MutablePtr col_null_map_to; + ColumnUInt8::Container * vec_null_map_to [[maybe_unused]] = nullptr; + if constexpr (nullOnErrors) + { + col_null_map_to = ColumnUInt8::create(input_rows_count); + vec_null_map_to = &col_null_map_to->getData(); + } + + size_t current_offset = 0; + for (size_t i = 0; i < input_rows_count; ++i) + { + const size_t next_offset = offsets ? (*offsets)[i] : current_offset + fixed_string_size; + const size_t string_size = offsets ? next_offset - current_offset - 1 : fixed_string_size; + ReadBufferFromMemory read_buffer(&(*chars)[current_offset], string_size); + current_offset = next_offset; + + if constexpr (nullOnErrors) + { + try + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toModifiedJulianDay(); + (*vec_null_map_to)[i] = false; + } + catch (const Exception & e) + { + if (e.code() == ErrorCodes::CANNOT_PARSE_INPUT_ASSERTION_FAILED || e.code() == ErrorCodes::CANNOT_PARSE_DATE) + (*vec_null_map_to)[i] = true; + else + throw; + } + } + else + { + const GregorianDate<> date(read_buffer); + vec_to[i] = date.toModifiedJulianDay(); + } + } + + if constexpr (nullOnErrors) + return ColumnNullable::create(std::move(col_to), std::move(col_null_map_to)); + else + return col_to; + } + + bool useDefaultImplementationForConstants() const override + { + return true; + } + }; + + template + class FunctionBaseToModifiedJulianDay : public IFunctionBaseImpl + { + public: + explicit FunctionBaseToModifiedJulianDay(DataTypes argument_types_, DataTypePtr return_type_) + : argument_types(std::move(argument_types_)) + , return_type(std::move(return_type_)) {} + + String getName() const override + { + return Name::name; + } + + const DataTypes & getArgumentTypes() const override + { + return argument_types; + } + + const DataTypePtr & getResultType() const override + { + return return_type; + } + + ExecutableFunctionImplPtr prepare(const ColumnsWithTypeAndName &) const override + { + return std::make_unique>(); + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + + bool hasInformationAboutMonotonicity() const override + { + return true; + } + + Monotonicity getMonotonicityForRange(const IDataType &, const Field &, const Field &) const override + { + return Monotonicity( + true, // is_monotonic + true, // is_positive + true); // is_always_monotonic + } + + private: + DataTypes argument_types; + DataTypePtr return_type; + }; + + template + class ToModifiedJulianDayOverloadResolver : public IFunctionOverloadResolverImpl + { + public: + static constexpr auto name = Name::name; + + static FunctionOverloadResolverImplPtr create(const Context &) + { + return std::make_unique>(); + } + + String getName() const override + { + return Name::name; + } + + FunctionBaseImplPtr build(const ColumnsWithTypeAndName & arguments, const DataTypePtr & return_type) const override + { + DataTypes argument_types = { arguments[0].type }; + + return std::make_unique>(argument_types, return_type); + } + + DataTypePtr getReturnType(const DataTypes & arguments) const override + { + if (!isStringOrFixedString(arguments[0])) + { + throw Exception( + "The argument of function " + getName() + " must be String or FixedString", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } + + DataTypePtr base_type = std::make_shared(); + if constexpr (nullOnErrors) + { + return std::make_shared(base_type); + } + else + { + return base_type; + } + } + + size_t getNumberOfArguments() const override + { + return 1; + } + + bool isInjective(const ColumnsWithTypeAndName &) const override + { + return true; + } + }; + + struct NameToModifiedJulianDay + { + static constexpr auto name = "toModifiedJulianDay"; + }; + + struct NameToModifiedJulianDayOrNull + { + static constexpr auto name = "toModifiedJulianDayOrNull"; + }; + + void registerFunctionToModifiedJulianDay(FunctionFactory & factory) + { + factory.registerFunction>(); + factory.registerFunction>(); + } +} diff --git a/src/Functions/ya.make b/src/Functions/ya.make index 436a6a89996..6d1b69ed45c 100644 --- a/src/Functions/ya.make +++ b/src/Functions/ya.make @@ -80,6 +80,7 @@ SRCS( URL/cutQueryString.cpp URL/cutQueryStringAndFragment.cpp URL/cutToFirstSignificantSubdomain.cpp + URL/cutToFirstSignificantSubdomainCustom.cpp URL/cutURLParameter.cpp URL/cutWWW.cpp URL/decodeURLComponent.cpp @@ -89,6 +90,7 @@ SRCS( URL/extractURLParameterNames.cpp URL/extractURLParameters.cpp URL/firstSignificantSubdomain.cpp + URL/firstSignificantSubdomainCustom.cpp URL/fragment.cpp URL/netloc.cpp URL/path.cpp @@ -247,6 +249,7 @@ SRCS( formatReadableTimeDelta.cpp formatRow.cpp formatString.cpp + fromModifiedJulianDay.cpp fromUnixTimestamp64Micro.cpp fromUnixTimestamp64Milli.cpp fromUnixTimestamp64Nano.cpp @@ -455,6 +458,7 @@ SRCS( toISOYear.cpp toLowCardinality.cpp toMinute.cpp + toModifiedJulianDay.cpp toMonday.cpp toMonth.cpp toNullable.cpp diff --git a/src/IO/ReadHelpers.cpp b/src/IO/ReadHelpers.cpp index 8232ce0b22d..e290da39535 100644 --- a/src/IO/ReadHelpers.cpp +++ b/src/IO/ReadHelpers.cpp @@ -194,12 +194,12 @@ inline void appendToStringOrVector(PODArray & s, ReadBuffer & rb, const ch s.insert(rb.position(), end); } -template -void readStringInto(Vector & s, ReadBuffer & buf) +template +void readStringUntilCharsInto(Vector & s, ReadBuffer & buf) { while (!buf.eof()) { - char * next_pos = find_first_symbols<'\t', '\n'>(buf.position(), buf.buffer().end()); + char * next_pos = find_first_symbols(buf.position(), buf.buffer().end()); appendToStringOrVector(s, buf, next_pos); buf.position() = next_pos; @@ -209,22 +209,31 @@ void readStringInto(Vector & s, ReadBuffer & buf) } } +template +void readStringInto(Vector & s, ReadBuffer & buf) +{ + readStringUntilCharsInto<'\t', '\n'>(s, buf); +} + +template +void readStringUntilWhitespaceInto(Vector & s, ReadBuffer & buf) +{ + readStringUntilCharsInto<' '>(s, buf); +} + template void readNullTerminated(Vector & s, ReadBuffer & buf) { - while (!buf.eof()) - { - char * next_pos = find_first_symbols<'\0'>(buf.position(), buf.buffer().end()); - - appendToStringOrVector(s, buf, next_pos); - buf.position() = next_pos; - - if (buf.hasPendingData()) - break; - } + readStringUntilCharsInto<'\0'>(s, buf); buf.ignore(); } +void readStringUntilWhitespace(String & s, ReadBuffer & buf) +{ + s.clear(); + readStringUntilWhitespaceInto(s, buf); +} + template void readNullTerminated>(PODArray & s, ReadBuffer & buf); template void readNullTerminated(String & s, ReadBuffer & buf); diff --git a/src/IO/ReadHelpers.h b/src/IO/ReadHelpers.h index c40278bfc0b..dbbaae0816f 100644 --- a/src/IO/ReadHelpers.h +++ b/src/IO/ReadHelpers.h @@ -478,6 +478,9 @@ void readStringUntilEOF(String & s, ReadBuffer & buf); // Buffer pointer is left at EOL, don't forget to advance it. void readEscapedStringUntilEOL(String & s, ReadBuffer & buf); +/// Only 0x20 as whitespace character +void readStringUntilWhitespace(String & s, ReadBuffer & buf); + /** Read string in CSV format. * Parsing rules: @@ -529,6 +532,9 @@ bool tryReadJSONStringInto(Vector & s, ReadBuffer & buf) return readJSONStringInto(s, buf); } +template +void readStringUntilWhitespaceInto(Vector & s, ReadBuffer & buf); + /// This could be used as template parameter for functions above, if you want to just skip data. struct NullOutput { diff --git a/src/IO/S3Common.cpp b/src/IO/S3Common.cpp index 97aa95f3c0d..bc49c2641a0 100644 --- a/src/IO/S3Common.cpp +++ b/src/IO/S3Common.cpp @@ -7,6 +7,10 @@ # include # include +# include +# include +# include +# include # include # include # include @@ -85,15 +89,107 @@ private: std::unordered_map tag_loggers; }; +class S3CredentialsProviderChain : public Aws::Auth::AWSCredentialsProviderChain +{ +public: + explicit S3CredentialsProviderChain(const DB::S3::PocoHTTPClientConfiguration & configuration, const Aws::Auth::AWSCredentials & credentials, bool use_environment_credentials) + { + if (use_environment_credentials) + { + const DB::RemoteHostFilter & remote_host_filter = configuration.remote_host_filter; + const unsigned int s3_max_redirects = configuration.s3_max_redirects; + + static const char AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI[] = "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"; + static const char AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI[] = "AWS_CONTAINER_CREDENTIALS_FULL_URI"; + static const char AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN[] = "AWS_CONTAINER_AUTHORIZATION_TOKEN"; + static const char AWS_EC2_METADATA_DISABLED[] = "AWS_EC2_METADATA_DISABLED"; + + auto * logger = &Poco::Logger::get("S3CredentialsProviderChain"); + + /// The only difference from DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() + /// is that this chain uses custom ClientConfiguration. + + AddProvider(std::make_shared()); + AddProvider(std::make_shared()); + AddProvider(std::make_shared()); + + /// ECS TaskRole Credentials only available when ENVIRONMENT VARIABLE is set. + const auto relative_uri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI, + relative_uri); + + const auto absolute_uri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI, + absolute_uri); + + const auto ec2_metadata_disabled = Aws::Environment::GetEnv(AWS_EC2_METADATA_DISABLED); + LOG_DEBUG(logger, "The environment variable value {} is {}", AWS_EC2_METADATA_DISABLED, + ec2_metadata_disabled); + + if (!relative_uri.empty()) + { + AddProvider(std::make_shared(relative_uri.c_str())); + LOG_INFO(logger, "Added ECS metadata service credentials provider with relative path: [{}] to the provider chain.", + relative_uri); + } + else if (!absolute_uri.empty()) + { + const auto token = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN); + AddProvider(std::make_shared(absolute_uri.c_str(), token.c_str())); + + /// DO NOT log the value of the authorization token for security purposes. + LOG_INFO(logger, "Added ECS credentials provider with URI: [{}] to the provider chain with a{} authorization token.", + absolute_uri, token.empty() ? "n empty" : " non-empty"); + } + else if (Aws::Utils::StringUtils::ToLower(ec2_metadata_disabled.c_str()) != "true") + { + Aws::Client::ClientConfiguration aws_client_configuration; + + /// See MakeDefaultHttpResourceClientConfiguration(). + /// This is part of EC2 metadata client, but unfortunately it can't be accessed from outside + /// of contrib/aws/aws-cpp-sdk-core/source/internal/AWSHttpResourceClient.cpp + aws_client_configuration.maxConnections = 2; + aws_client_configuration.scheme = Aws::Http::Scheme::HTTP; + + /// Explicitly set the proxy settings to empty/zero to avoid relying on defaults that could potentially change + /// in the future. + aws_client_configuration.proxyHost = ""; + aws_client_configuration.proxyUserName = ""; + aws_client_configuration.proxyPassword = ""; + aws_client_configuration.proxyPort = 0; + + /// EC2MetadataService throttles by delaying the response so the service client should set a large read timeout. + /// EC2MetadataService delay is in order of seconds so it only make sense to retry after a couple of seconds. + aws_client_configuration.connectTimeoutMs = 1000; + aws_client_configuration.requestTimeoutMs = 1000; + aws_client_configuration.retryStrategy = std::make_shared(1, 1000); + + DB::S3::PocoHTTPClientConfiguration client_configuration(aws_client_configuration, remote_host_filter, s3_max_redirects); + auto ec2_metadata_client = std::make_shared(client_configuration); + auto config_loader = std::make_shared(ec2_metadata_client); + + AddProvider(std::make_shared(config_loader)); + LOG_INFO(logger, "Added EC2 metadata service credentials provider to the provider chain."); + } + } + + AddProvider(std::make_shared(credentials)); + } +}; + class S3AuthSigner : public Aws::Client::AWSAuthV4Signer { public: S3AuthSigner( const Aws::Client::ClientConfiguration & client_configuration, const Aws::Auth::AWSCredentials & credentials, - const DB::HeaderCollection & headers_) + const DB::HeaderCollection & headers_, + bool use_environment_credentials) : Aws::Client::AWSAuthV4Signer( - std::make_shared(credentials), + std::make_shared( + static_cast(client_configuration), + credentials, + use_environment_credentials), "s3", client_configuration.region, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, @@ -164,6 +260,7 @@ namespace S3 bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -172,7 +269,13 @@ namespace S3 if (!endpoint.empty()) cfg.endpointOverride = endpoint; - return create(cfg, is_virtual_hosted_style, access_key_id, secret_access_key, remote_host_filter, s3_max_redirects); + return create(cfg, + is_virtual_hosted_style, + access_key_id, + secret_access_key, + use_environment_credentials, + remote_host_filter, + s3_max_redirects); } std::shared_ptr ClientFactory::create( // NOLINT @@ -180,6 +283,7 @@ namespace S3 bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -190,7 +294,10 @@ namespace S3 client_configuration.updateSchemeAndRegion(); return std::make_shared( - credentials, // Aws credentials. + std::make_shared( + client_configuration, + credentials, + use_environment_credentials), // AWS credentials provider. std::move(client_configuration), // Client configuration. Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, // Sign policy. is_virtual_hosted_style || cfg.endpointOverride.empty() // Use virtual addressing if endpoint is not specified. @@ -203,6 +310,7 @@ namespace S3 const String & access_key_id, const String & secret_access_key, HeaderCollection headers, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects) { @@ -214,8 +322,10 @@ namespace S3 client_configuration.updateSchemeAndRegion(); Aws::Auth::AWSCredentials credentials(access_key_id, secret_access_key); + + auto auth_signer = std::make_shared(client_configuration, std::move(credentials), std::move(headers), use_environment_credentials); return std::make_shared( - std::make_shared(client_configuration, std::move(credentials), std::move(headers)), + std::move(auth_signer), std::move(client_configuration), // Client configuration. is_virtual_hosted_style || client_configuration.endpointOverride.empty() // Use virtual addressing only if endpoint is not specified. ); diff --git a/src/IO/S3Common.h b/src/IO/S3Common.h index 39d9068a6f4..664c07d5bf4 100644 --- a/src/IO/S3Common.h +++ b/src/IO/S3Common.h @@ -36,6 +36,7 @@ public: bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); @@ -44,6 +45,7 @@ public: bool is_virtual_hosted_style, const String & access_key_id, const String & secret_access_key, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); @@ -53,6 +55,7 @@ public: const String & access_key_id, const String & secret_access_key, HeaderCollection headers, + bool use_environment_credentials, const RemoteHostFilter & remote_host_filter, unsigned int s3_max_redirects); diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index a32aa4acdc9..e4fb0634bd0 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -43,11 +43,13 @@ WriteBufferFromS3::WriteBufferFromS3( const String & key_, size_t minimum_upload_part_size_, bool is_multipart_, + std::optional> object_metadata_, size_t buffer_size_) : BufferWithOwnMemory(buffer_size_, nullptr, 0) , is_multipart(is_multipart_) , bucket(bucket_) , key(key_) + , object_metadata(std::move(object_metadata_)) , client_ptr(std::move(client_ptr_)) , minimum_upload_part_size{minimum_upload_part_size_} , temporary_buffer{std::make_unique()} @@ -116,6 +118,8 @@ void WriteBufferFromS3::initiate() Aws::S3::Model::CreateMultipartUploadRequest req; req.SetBucket(bucket); req.SetKey(key); + if (object_metadata.has_value()) + req.SetMetadata(object_metadata.value()); auto outcome = client_ptr->CreateMultipartUpload(req); @@ -217,6 +221,8 @@ void WriteBufferFromS3::complete() Aws::S3::Model::PutObjectRequest req; req.SetBucket(bucket); req.SetKey(key); + if (object_metadata.has_value()) + req.SetMetadata(object_metadata.value()); /// This could be improved using an adapter to WriteBuffer. const std::shared_ptr input_data = Aws::MakeShared("temporary buffer", temporary_buffer->str()); diff --git a/src/IO/WriteBufferFromS3.h b/src/IO/WriteBufferFromS3.h index 1a1e859d913..9bd7b896d7f 100644 --- a/src/IO/WriteBufferFromS3.h +++ b/src/IO/WriteBufferFromS3.h @@ -28,6 +28,7 @@ private: String bucket; String key; + std::optional> object_metadata; std::shared_ptr client_ptr; size_t minimum_upload_part_size; std::unique_ptr temporary_buffer; @@ -47,6 +48,7 @@ public: const String & key_, size_t minimum_upload_part_size_, bool is_multipart, + std::optional> object_metadata_ = std::nullopt, size_t buffer_size_ = DBMS_DEFAULT_BUFFER_SIZE); void nextImpl() override; diff --git a/src/IO/WriteHelpers.h b/src/IO/WriteHelpers.h index 1997aede564..54e9ac9e09e 100644 --- a/src/IO/WriteHelpers.h +++ b/src/IO/WriteHelpers.h @@ -29,7 +29,12 @@ #include #include -#include +/// There is no dragonbox in Arcadia +#if !defined(ARCADIA_BUILD) +# include +#else +# include +#endif #include @@ -228,14 +233,22 @@ inline size_t writeFloatTextFastPath(T x, char * buffer) if (DecomposedFloat64(x).is_inside_int64()) result = itoa(Int64(x), buffer) - buffer; else +#if !defined(ARCADIA_BUILD) result = jkj::dragonbox::to_chars_n(x, buffer) - buffer; +#else + result = d2s_buffered_n(x, buffer); +#endif } else { if (DecomposedFloat32(x).is_inside_int32()) result = itoa(Int32(x), buffer) - buffer; else +#if !defined(ARCADIA_BUILD) result = jkj::dragonbox::to_chars_n(x, buffer) - buffer; +#else + result = f2s_buffered_n(x, buffer); +#endif } if (result <= 0) diff --git a/src/IO/readFloatText.h b/src/IO/readFloatText.h index d126dd52ef8..a721642a185 100644 --- a/src/IO/readFloatText.h +++ b/src/IO/readFloatText.h @@ -5,6 +5,7 @@ #include #include #include +#include /** Methods for reading floating point numbers from text with decimal representation. @@ -133,89 +134,23 @@ bool assertOrParseNaN(ReadBuffer & buf) } -/// Some garbage may be successfully parsed, examples: '--1' parsed as '1'. template ReturnType readFloatTextPreciseImpl(T & x, ReadBuffer & buf) { - static_assert(std::is_same_v || std::is_same_v, "Argument for readFloatTextImpl must be float or double"); + static_assert(std::is_same_v || std::is_same_v, "Argument for readFloatTextFastFloatImpl must be float or double"); + static_assert('a' > '.' && 'A' > '.' && '\n' < '.' && '\t' < '.' && '\'' < '.' && '"' < '.', "Layout of char is not like ASCII"); //-V590 + static constexpr bool throw_exception = std::is_same_v; - if (buf.eof()) - { - if constexpr (throw_exception) - throw Exception("Cannot read floating point value", ErrorCodes::CANNOT_PARSE_NUMBER); - else - return ReturnType(false); - } - - /// We use special code to read denormals (inf, nan), because we support slightly more variants that double-conversion library does: - /// Example: inf and Infinity. - - bool negative = false; - - while (true) - { - switch (*buf.position()) - { - case '+': - continue; - - case '-': - { - negative = true; - ++buf.position(); - continue; - } - - case 'i': [[fallthrough]]; - case 'I': - { - if (assertOrParseInfinity(buf)) - { - x = std::numeric_limits::infinity(); - if (negative) - x = -x; - return ReturnType(true); - } - return ReturnType(false); - } - - case 'n': [[fallthrough]]; - case 'N': - { - if (assertOrParseNaN(buf)) - { - x = std::numeric_limits::quiet_NaN(); - if (negative) - x = -x; - return ReturnType(true); - } - return ReturnType(false); - } - - default: - break; - } - break; - } - - static const double_conversion::StringToDoubleConverter converter( - double_conversion::StringToDoubleConverter::ALLOW_TRAILING_JUNK, - 0, 0, nullptr, nullptr); - /// Fast path (avoid copying) if the buffer have at least MAX_LENGTH bytes. static constexpr int MAX_LENGTH = 316; - if (buf.position() + MAX_LENGTH <= buf.buffer().end()) + if (likely(buf.position() + MAX_LENGTH <= buf.buffer().end())) { - int num_processed_characters = 0; + auto initial_position = buf.position(); + auto res = fast_float::from_chars(initial_position, buf.buffer().end(), x); - if constexpr (std::is_same_v) - x = converter.StringToDouble(buf.position(), buf.buffer().end() - buf.position(), &num_processed_characters); - else - x = converter.StringToFloat(buf.position(), buf.buffer().end() - buf.position(), &num_processed_characters); - - if (num_processed_characters < 0) + if (unlikely(res.ec != std::errc())) { if constexpr (throw_exception) throw Exception("Cannot read floating point value", ErrorCodes::CANNOT_PARSE_NUMBER); @@ -223,15 +158,64 @@ ReturnType readFloatTextPreciseImpl(T & x, ReadBuffer & buf) return ReturnType(false); } - buf.position() += num_processed_characters; + buf.position() += res.ptr - initial_position; - if (negative) - x = -x; return ReturnType(true); } else { /// Slow path. Copy characters that may be present in floating point number to temporary buffer. + bool negative = false; + + /// We check eof here because we can parse +inf +nan + while (!buf.eof()) + { + switch (*buf.position()) + { + case '+': + ++buf.position(); + continue; + + case '-': + { + negative = true; + ++buf.position(); + continue; + } + + case 'i': [[fallthrough]]; + case 'I': + { + if (assertOrParseInfinity(buf)) + { + x = std::numeric_limits::infinity(); + if (negative) + x = -x; + return ReturnType(true); + } + return ReturnType(false); + } + + case 'n': [[fallthrough]]; + case 'N': + { + if (assertOrParseNaN(buf)) + { + x = std::numeric_limits::quiet_NaN(); + if (negative) + x = -x; + return ReturnType(true); + } + return ReturnType(false); + } + + default: + break; + } + + break; + } + char tmp_buf[MAX_LENGTH]; int num_copied_chars = 0; @@ -247,14 +231,9 @@ ReturnType readFloatTextPreciseImpl(T & x, ReadBuffer & buf) ++num_copied_chars; } - int num_processed_characters = 0; + auto res = fast_float::from_chars(tmp_buf, tmp_buf + num_copied_chars, x); - if constexpr (std::is_same_v) - x = converter.StringToDouble(tmp_buf, num_copied_chars, &num_processed_characters); - else - x = converter.StringToFloat(tmp_buf, num_copied_chars, &num_processed_characters); - - if (num_processed_characters < num_copied_chars) + if (unlikely(res.ec != std::errc())) { if constexpr (throw_exception) throw Exception("Cannot read floating point value", ErrorCodes::CANNOT_PARSE_NUMBER); @@ -264,6 +243,7 @@ ReturnType readFloatTextPreciseImpl(T & x, ReadBuffer & buf) if (negative) x = -x; + return ReturnType(true); } } @@ -474,7 +454,6 @@ ReturnType readFloatTextFastImpl(T & x, ReadBuffer & in) return ReturnType(true); } - template ReturnType readFloatTextSimpleImpl(T & x, ReadBuffer & buf) { @@ -575,7 +554,6 @@ ReturnType readFloatTextSimpleImpl(T & x, ReadBuffer & buf) return ReturnType(true); } - template void readFloatTextPrecise(T & x, ReadBuffer & in) { readFloatTextPreciseImpl(x, in); } template bool tryReadFloatTextPrecise(T & x, ReadBuffer & in) { return readFloatTextPreciseImpl(x, in); } @@ -591,5 +569,4 @@ template bool tryReadFloatTextSimple(T & x, ReadBuffer & in) { retu template void readFloatText(T & x, ReadBuffer & in) { readFloatTextFast(x, in); } template bool tryReadFloatText(T & x, ReadBuffer & in) { return tryReadFloatTextFast(x, in); } - } diff --git a/src/IO/tests/read_float_perf.cpp b/src/IO/tests/read_float_perf.cpp index acfd294e0e7..40742d17b1c 100644 --- a/src/IO/tests/read_float_perf.cpp +++ b/src/IO/tests/read_float_perf.cpp @@ -13,7 +13,6 @@ #include #include - /** How to test: # Prepare data @@ -39,7 +38,6 @@ $ for i in {1..10}; do echo $i; time ./read_float_perf 2 < numbers$i.tsv; done using namespace DB; - template void NO_INLINE loop(ReadBuffer & in, WriteBuffer & out) { diff --git a/src/Interpreters/ClientInfo.cpp b/src/Interpreters/ClientInfo.cpp index 4dbec8af3a4..30aa8a0c692 100644 --- a/src/Interpreters/ClientInfo.cpp +++ b/src/Interpreters/ClientInfo.cpp @@ -49,6 +49,9 @@ void ClientInfo::write(WriteBuffer & out, const UInt64 server_protocol_revision) { writeBinary(UInt8(http_method), out); writeBinary(http_user_agent, out); + + if (server_protocol_revision >= DBMS_MIN_REVISION_WITH_X_FORWARDED_FOR_IN_CLIENT_INFO) + writeBinary(forwarded_for, out); } if (server_protocol_revision >= DBMS_MIN_REVISION_WITH_QUOTA_KEY_IN_CLIENT_INFO) @@ -120,6 +123,9 @@ void ClientInfo::read(ReadBuffer & in, const UInt64 client_protocol_revision) http_method = HTTPMethod(read_http_method); readBinary(http_user_agent, in); + + if (client_protocol_revision >= DBMS_MIN_REVISION_WITH_X_FORWARDED_FOR_IN_CLIENT_INFO) + readBinary(forwarded_for, in); } if (client_protocol_revision >= DBMS_MIN_REVISION_WITH_QUOTA_KEY_IN_CLIENT_INFO) diff --git a/src/Interpreters/ClientInfo.h b/src/Interpreters/ClientInfo.h index 36cde339995..e188420d607 100644 --- a/src/Interpreters/ClientInfo.h +++ b/src/Interpreters/ClientInfo.h @@ -83,6 +83,12 @@ public: HTTPMethod http_method = HTTPMethod::UNKNOWN; String http_user_agent; + /// Comma separated list of forwarded IP addresses (from X-Forwarded-For for HTTP interface). + /// It's expected that proxy appends the forwarded address to the end of the list. + /// The element can be trusted only if you trust the corresponding proxy. + /// NOTE This field can also be reused in future for TCP interface with PROXY v1/v2 protocols. + String forwarded_for; + /// Common String quota_key; diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index 0c9881bf895..9955c1ca11f 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -304,6 +305,8 @@ struct ContextShared mutable zkutil::ZooKeeperPtr zookeeper; /// Client for ZooKeeper. ConfigurationPtr zookeeper_config; /// Stores zookeeper configs + mutable std::mutex test_keeper_storage_mutex; + mutable std::shared_ptr test_keeper_storage; mutable std::mutex auxiliary_zookeepers_mutex; mutable std::map auxiliary_zookeepers; /// Map for auxiliary ZooKeeper clients. ConfigurationPtr auxiliary_zookeepers_config; /// Stores auxiliary zookeepers configs @@ -442,6 +445,8 @@ struct ContextShared /// Stop trace collector if any trace_collector.reset(); + /// Stop test_keeper storage + test_keeper_storage.reset(); } bool hasTraceCollector() const @@ -1505,6 +1510,15 @@ zkutil::ZooKeeperPtr Context::getZooKeeper() const return shared->zookeeper; } +std::shared_ptr & Context::getTestKeeperStorage() const +{ + std::lock_guard lock(shared->test_keeper_storage_mutex); + if (!shared->test_keeper_storage) + shared->test_keeper_storage = std::make_shared(); + + return shared->test_keeper_storage; +} + zkutil::ZooKeeperPtr Context::getAuxiliaryZooKeeper(const String & name) const { std::lock_guard lock(shared->auxiliary_zookeepers_mutex); diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index 231145eec99..ccdaf98980d 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -40,6 +40,7 @@ namespace Poco namespace zkutil { class ZooKeeper; + class TestKeeperStorage; } @@ -494,6 +495,9 @@ public: /// Same as above but return a zookeeper connection from auxiliary_zookeepers configuration entry. std::shared_ptr getAuxiliaryZooKeeper(const String & name) const; + + std::shared_ptr & getTestKeeperStorage() const; + /// Set auxiliary zookeepers configuration at server starting or configuration reloading. void reloadAuxiliaryZooKeepersConfigIfChanged(const ConfigurationPtr & config); /// Has ready or expired ZooKeeper diff --git a/src/Interpreters/CrashLog.cpp b/src/Interpreters/CrashLog.cpp index 9d84d5a18e9..bf81a2e8aba 100644 --- a/src/Interpreters/CrashLog.cpp +++ b/src/Interpreters/CrashLog.cpp @@ -53,7 +53,7 @@ void CrashLogElement::appendToBlock(MutableColumns & columns) const String build_id_hex; #if defined(__ELF__) && !defined(__FreeBSD__) - build_id_hex = SymbolIndex::instance().getBuildIDHex(); + build_id_hex = SymbolIndex::instance()->getBuildIDHex(); #endif columns[i++]->insert(build_id_hex); } diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index e59d0f925e9..fd36f3a6fd6 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -271,6 +272,14 @@ BlockIO InterpreterSystemQuery::execute() context.checkAccess(AccessType::SYSTEM_RELOAD_CONFIG); system_context.reloadConfig(); break; + case Type::RELOAD_SYMBOLS: +#if defined(__ELF__) && !defined(__FreeBSD__) + context.checkAccess(AccessType::SYSTEM_RELOAD_SYMBOLS); + (void)SymbolIndex::instance(true); + break; +#else + throw Exception("SYSTEM RELOAD SYMBOLS is not supported on current platform", ErrorCodes::NOT_IMPLEMENTED); +#endif case Type::STOP_MERGES: startStopAction(ActionLocks::PartsMerge, false); break; @@ -604,6 +613,11 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() required_access.emplace_back(AccessType::SYSTEM_RELOAD_CONFIG); break; } + case Type::RELOAD_SYMBOLS: + { + required_access.emplace_back(AccessType::SYSTEM_RELOAD_SYMBOLS); + break; + } case Type::STOP_MERGES: [[fallthrough]]; case Type::START_MERGES: { diff --git a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp index 688ab786b56..f045a92c394 100644 --- a/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp +++ b/src/Interpreters/MySQL/InterpretersMySQLDDLQuery.cpp @@ -164,20 +164,47 @@ static inline std::tuplechildren.empty()) { + NameSet columns_name_set; + const Names & columns_name = columns.getNames(); + columns_name_set.insert(columns_name.begin(), columns_name.end()); + + const auto & remove_prefix_key = [&](const ASTPtr & node) -> ASTPtr + { + auto res = std::make_shared(); + for (const auto & index_expression : node->children) + { + res->children.emplace_back(index_expression); + + if (const auto & function = index_expression->as()) + { + /// column_name(int64 literal) + if (columns_name_set.count(function->name) && function->arguments->children.size() == 1) + { + const auto & prefix_limit = function->arguments->children[0]->as(); + + if (prefix_limit && isInt64FieldType(prefix_limit->value.getType())) + res->children.back() = std::make_shared(function->name); + } + } + } + return res; + }; + for (const auto & declare_index_ast : indices_define->children) { const auto & declare_index = declare_index_ast->as(); + const auto & index_columns = remove_prefix_key(declare_index->index_columns); /// flatten if (startsWith(declare_index->index_type, "KEY_")) keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); else if (startsWith(declare_index->index_type, "UNIQUE_")) unique_keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); if (startsWith(declare_index->index_type, "PRIMARY_KEY_")) primary_keys->arguments->children.insert(keys->arguments->children.end(), - declare_index->index_columns->children.begin(), declare_index->index_columns->children.end()); + index_columns->children.begin(), index_columns->children.end()); } } diff --git a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp index 2221b7b1588..d7209330ec6 100644 --- a/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp +++ b/src/Interpreters/MySQL/tests/gtest_create_rewritten.cpp @@ -184,3 +184,14 @@ TEST(MySQLCreateRewritten, RewrittenQueryWithPrimaryKey) "ReplacingMergeTree(_version) PARTITION BY intDiv(key_2, 4294967) ORDER BY (key_1, key_2)"); } +TEST(MySQLCreateRewritten, RewrittenQueryWithPrefixKey) +{ + tryRegisterFunctions(); + const auto & context_holder = getContext(); + + EXPECT_EQ(queryToString(tryRewrittenCreateQuery( + "CREATE TABLE `test_database`.`test_table_1` (`key` int NOT NULL PRIMARY KEY, `prefix_key` varchar(200) NOT NULL, KEY prefix_key_index(prefix_key(2))) ENGINE=InnoDB DEFAULT CHARSET=utf8", context_holder.context)), + "CREATE TABLE test_database.test_table_1 (`key` Int32, `prefix_key` String, `_sign` Int8() MATERIALIZED 1, `_version` UInt64() MATERIALIZED 1) ENGINE = " + "ReplacingMergeTree(_version) PARTITION BY intDiv(key, 4294967) ORDER BY (key, prefix_key)"); +} + diff --git a/src/Interpreters/QueryLog.cpp b/src/Interpreters/QueryLog.cpp index c2273a1db2c..e5d68870f87 100644 --- a/src/Interpreters/QueryLog.cpp +++ b/src/Interpreters/QueryLog.cpp @@ -77,6 +77,7 @@ Block QueryLogElement::createBlock() {std::make_shared(), "client_version_patch"}, {std::make_shared(), "http_method"}, {std::make_shared(), "http_user_agent"}, + {std::make_shared(), "forwarded_for"}, {std::make_shared(), "quota_key"}, {std::make_shared(), "revision"}, @@ -181,6 +182,7 @@ void QueryLogElement::appendClientInfo(const ClientInfo & client_info, MutableCo columns[i++]->insert(UInt64(client_info.http_method)); columns[i++]->insert(client_info.http_user_agent); + columns[i++]->insert(client_info.forwarded_for); columns[i++]->insert(client_info.quota_key); } diff --git a/src/Interpreters/QueryThreadLog.cpp b/src/Interpreters/QueryThreadLog.cpp index 8fea360085b..986bedcfc15 100644 --- a/src/Interpreters/QueryThreadLog.cpp +++ b/src/Interpreters/QueryThreadLog.cpp @@ -60,6 +60,7 @@ Block QueryThreadLogElement::createBlock() {std::make_shared(), "client_version_patch"}, {std::make_shared(), "http_method"}, {std::make_shared(), "http_user_agent"}, + {std::make_shared(), "forwarded_for"}, {std::make_shared(), "quota_key"}, {std::make_shared(), "revision"}, diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index 069e76e430b..1025e64e2f4 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -125,12 +125,60 @@ struct CustomizeAggregateFunctionsSuffixData { auto properties = instance.tryGetProperties(func.name); if (properties && !properties->returns_default_when_only_null) - func.name = func.name + customized_func_suffix; + { + func.name += customized_func_suffix; + } + } + } +}; + +// Used to rewrite aggregate functions with -OrNull suffix in some cases, such as sumIfOrNull, we shoule rewrite to sumOrNullIf +struct CustomizeAggregateFunctionsMoveSuffixData +{ + using TypeToVisit = ASTFunction; + + const String & customized_func_suffix; + + String moveSuffixAhead(const String & name) const + { + auto prefix = name.substr(0, name.size() - customized_func_suffix.size()); + + auto prefix_size = prefix.size(); + + if (endsWith(prefix, "MergeState")) + return prefix.substr(0, prefix_size - 10) + customized_func_suffix + "MergeState"; + + if (endsWith(prefix, "Merge")) + return prefix.substr(0, prefix_size - 5) + customized_func_suffix + "Merge"; + + if (endsWith(prefix, "State")) + return prefix.substr(0, prefix_size - 5) + customized_func_suffix + "State"; + + if (endsWith(prefix, "If")) + return prefix.substr(0, prefix_size - 2) + customized_func_suffix + "If"; + + return name; + } + + void visit(ASTFunction & func, ASTPtr &) const + { + const auto & instance = AggregateFunctionFactory::instance(); + if (instance.isAggregateFunctionName(func.name)) + { + if (endsWith(func.name, customized_func_suffix)) + { + auto properties = instance.tryGetProperties(func.name); + if (properties && !properties->returns_default_when_only_null) + { + func.name = moveSuffixAhead(func.name); + } + } } } }; using CustomizeAggregateFunctionsOrNullVisitor = InDepthNodeVisitor, true>; +using CustomizeAggregateFunctionsMoveOrNullVisitor = InDepthNodeVisitor, true>; /// Translate qualified names such as db.table.column, table.column, table_alias.column to names' normal form. /// Expand asterisks and qualified asterisks with column names. @@ -753,6 +801,10 @@ void TreeRewriter::normalize(ASTPtr & query, Aliases & aliases, const Settings & CustomizeAggregateFunctionsOrNullVisitor(data_or_null).visit(query); } + /// Move -OrNull suffix ahead, this should execute after add -OrNull suffix + CustomizeAggregateFunctionsMoveOrNullVisitor::Data data_or_null{"OrNull"}; + CustomizeAggregateFunctionsMoveOrNullVisitor(data_or_null).visit(query); + /// Creates a dictionary `aliases`: alias -> ASTPtr QueryAliasesVisitor(aliases).visit(query); diff --git a/src/Interpreters/tests/CMakeLists.txt b/src/Interpreters/tests/CMakeLists.txt index 20aa73166fb..1bc9d7fbacb 100644 --- a/src/Interpreters/tests/CMakeLists.txt +++ b/src/Interpreters/tests/CMakeLists.txt @@ -29,6 +29,9 @@ target_link_libraries (string_hash_map PRIVATE dbms) add_executable (string_hash_map_aggregation string_hash_map.cpp) target_link_libraries (string_hash_map_aggregation PRIVATE dbms) +add_executable (string_hash_set string_hash_set.cpp) +target_link_libraries (string_hash_set PRIVATE dbms) + add_executable (two_level_hash_map two_level_hash_map.cpp) target_include_directories (two_level_hash_map SYSTEM BEFORE PRIVATE ${SPARSEHASH_INCLUDE_DIR}) target_link_libraries (two_level_hash_map PRIVATE dbms) diff --git a/src/Interpreters/tests/string_hash_set.cpp b/src/Interpreters/tests/string_hash_set.cpp new file mode 100644 index 00000000000..d9d6453da34 --- /dev/null +++ b/src/Interpreters/tests/string_hash_set.cpp @@ -0,0 +1,83 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/// NOTE: see string_hash_map.cpp for usage example + +template +void NO_INLINE bench(const std::vector & data, DB::Arena & pool, const char * name) +{ + std::cerr << "method " << name << std::endl; + for (auto t = 0ul; t < 7; ++t) + { + Stopwatch watch; + Set set; + typename Set::LookupResult it; + bool inserted; + + for (const auto & value : data) + { + if constexpr (std::is_same_v, Set>) + set.emplace(DB::ArenaKeyHolder{value, pool}, inserted); + else + set.emplace(DB::ArenaKeyHolder{value, pool}, it, inserted); + } + watch.stop(); + + std::cerr << "arena-memory " << pool.size() + set.getBufferSizeInBytes() << std::endl; + std::cerr << "single-run " << std::setprecision(3) + << watch.elapsedSeconds() << std::endl; + } +} + +int main(int argc, char ** argv) +{ + if (argc < 3) + { + std::cerr << "Usage: program n m\n"; + return 1; + } + + size_t n = std::stol(argv[1]); + size_t m = std::stol(argv[2]); + + DB::Arena pool(128 * 1024 * 1024); + std::vector data(n); + + std::cerr << "sizeof(Key) = " << sizeof(StringRef) << std::endl; + + { + Stopwatch watch; + DB::ReadBufferFromFileDescriptor in1(STDIN_FILENO); + DB::CompressedReadBuffer in2(in1); + + std::string tmp; + for (size_t i = 0; i < n && !in2.eof(); ++i) + { + DB::readStringBinary(tmp, in2); + data[i] = StringRef(pool.insert(tmp.data(), tmp.size()), tmp.size()); + } + + watch.stop(); + std::cerr << std::fixed << std::setprecision(2) << "Vector. Size: " << n << ", elapsed: " << watch.elapsedSeconds() << " (" + << n / watch.elapsedSeconds() << " elem/sec.)" << std::endl; + } + + if (!m || m == 1) + bench>(data, pool, "StringHashSet"); + if (!m || m == 2) + bench>(data, pool, "HashSetWithSavedHash"); + if (!m || m == 3) + bench>(data, pool, "HashSet"); + return 0; +} diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index 4ed0ecd3a91..0d6e15a3d8c 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -54,6 +54,8 @@ const char * ASTSystemQuery::typeToString(Type type) return "RELOAD EMBEDDED DICTIONARIES"; case Type::RELOAD_CONFIG: return "RELOAD CONFIG"; + case Type::RELOAD_SYMBOLS: + return "RELOAD SYMBOLS"; case Type::STOP_MERGES: return "STOP MERGES"; case Type::START_MERGES: diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index f8f803e1c0c..756b5b52600 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -36,6 +36,7 @@ public: RELOAD_DICTIONARIES, RELOAD_EMBEDDED_DICTIONARIES, RELOAD_CONFIG, + RELOAD_SYMBOLS, STOP_MERGES, START_MERGES, STOP_TTL_MERGES, diff --git a/src/Parsers/ParserCreateQuotaQuery.cpp b/src/Parsers/ParserCreateQuotaQuery.cpp index 324519b9c01..68c53d2fc1d 100644 --- a/src/Parsers/ParserCreateQuotaQuery.cpp +++ b/src/Parsers/ParserCreateQuotaQuery.cpp @@ -63,11 +63,13 @@ namespace boost::replace_all(name, " ", "_"); for (auto kt : ext::range(Quota::KeyType::MAX)) + { if (KeyTypeInfo::get(kt).name == name) { key_type = kt; return true; } + } String all_types_str; for (auto kt : ext::range(Quota::KeyType::MAX)) diff --git a/src/Parsers/ParserCreateQuotaQuery.h b/src/Parsers/ParserCreateQuotaQuery.h index 1f1df0588bd..b44838225ac 100644 --- a/src/Parsers/ParserCreateQuotaQuery.h +++ b/src/Parsers/ParserCreateQuotaQuery.h @@ -7,7 +7,7 @@ namespace DB { /** Parses queries like * CREATE QUOTA [IF NOT EXISTS | OR REPLACE] name - * [KEYED BY {none | user_name | ip_address | client_key | client_key, user_name | client_key, ip_address} | NOT KEYED] + * [KEYED BY {none | user_name | ip_address | forwarded_ip_address | client_key | client_key, user_name | client_key, ip_address} | NOT KEYED] * [FOR [RANDOMIZED] INTERVAL number {second | minute | hour | day | week | month | quarter | year} * {MAX {{queries | errors | result_rows | result_bytes | read_rows | read_bytes | execution_time} = number} [,...] | * NO LIMITS | TRACKING ONLY} [,...]] @@ -15,7 +15,7 @@ namespace DB * * ALTER QUOTA [IF EXISTS] name * [RENAME TO new_name] - * [KEYED BY {none | user_name | ip_address | client_key | client_key, user_name | client_key, ip_address} | NOT KEYED] + * [KEYED BY {none | user_name | ip_address | forwarded_ip_address | client_key | client_key, user_name | client_key, ip_address} | NOT KEYED] * [FOR [RANDOMIZED] INTERVAL number {second | minute | hour | day | week | month | quarter | year} * {MAX {{queries | errors | result_rows | result_bytes | read_rows | read_bytes | execution_time} = number} [,...] | * NO LIMITS | TRACKING ONLY} [,...]] diff --git a/src/Server/GRPCServer.cpp b/src/Server/GRPCServer.cpp index cf5c6112312..bdf6d3ba5d7 100644 --- a/src/Server/GRPCServer.cpp +++ b/src/Server/GRPCServer.cpp @@ -31,10 +31,6 @@ #include -/// For diagnosing problems use the following environment variables: -/// export GRPC_TRACE=all -/// export GRPC_VERBOSITY=DEBUG - using GRPCService = clickhouse::grpc::ClickHouse::AsyncService; using GRPCQueryInfo = clickhouse::grpc::QueryInfo; using GRPCResult = clickhouse::grpc::Result; @@ -57,6 +53,39 @@ namespace ErrorCodes namespace { + /// Make grpc to pass logging messages to ClickHouse logging system. + void initGRPCLogging(const Poco::Util::AbstractConfiguration & config) + { + static std::once_flag once_flag; + std::call_once(once_flag, [&config] + { + static Poco::Logger * logger = &Poco::Logger::get("grpc"); + gpr_set_log_function([](gpr_log_func_args* args) + { + if (args->severity == GPR_LOG_SEVERITY_DEBUG) + LOG_DEBUG(logger, "{} ({}:{})", args->message, args->file, args->line); + else if (args->severity == GPR_LOG_SEVERITY_INFO) + LOG_INFO(logger, "{} ({}:{})", args->message, args->file, args->line); + else if (args->severity == GPR_LOG_SEVERITY_ERROR) + LOG_ERROR(logger, "{} ({}:{})", args->message, args->file, args->line); + }); + + if (config.getBool("grpc.verbose_logs", false)) + { + gpr_set_log_verbosity(GPR_LOG_SEVERITY_DEBUG); + grpc_tracer_set_enabled("all", true); + } + else if (logger->is(Poco::Message::PRIO_DEBUG)) + { + gpr_set_log_verbosity(GPR_LOG_SEVERITY_DEBUG); + } + else if (logger->is(Poco::Message::PRIO_INFORMATION)) + { + gpr_set_log_verbosity(GPR_LOG_SEVERITY_INFO); + } + }); + } + grpc_compression_algorithm parseCompressionAlgorithm(const String & str) { if (str == "none") @@ -1604,6 +1633,7 @@ GRPCServer::~GRPCServer() void GRPCServer::start() { + initGRPCLogging(iserver.config()); grpc::ServerBuilder builder; builder.AddListeningPort(address_to_listen.toString(), makeCredentials(iserver.config())); builder.RegisterService(&grpc_service); diff --git a/src/Server/HTTPHandler.cpp b/src/Server/HTTPHandler.cpp index 34d510e4cb2..ed154ba65f2 100644 --- a/src/Server/HTTPHandler.cpp +++ b/src/Server/HTTPHandler.cpp @@ -280,10 +280,31 @@ void HTTPHandler::processQuery( } } + /// Set client info. It will be used for quota accounting parameters in 'setUser' method. + + ClientInfo & client_info = context.getClientInfo(); + client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; + client_info.interface = ClientInfo::Interface::HTTP; + + ClientInfo::HTTPMethod http_method = ClientInfo::HTTPMethod::UNKNOWN; + if (request.getMethod() == Poco::Net::HTTPServerRequest::HTTP_GET) + http_method = ClientInfo::HTTPMethod::GET; + else if (request.getMethod() == Poco::Net::HTTPServerRequest::HTTP_POST) + http_method = ClientInfo::HTTPMethod::POST; + + client_info.http_method = http_method; + client_info.http_user_agent = request.get("User-Agent", ""); + client_info.forwarded_for = request.get("X-Forwarded-For", ""); + + /// This will also set client_info.current_user and current_address context.setUser(user, password, request.clientAddress()); if (!quota_key.empty()) context.setQuotaKey(quota_key); + /// Query sent through HTTP interface is initial. + client_info.initial_user = client_info.current_user; + client_info.initial_address = client_info.current_address; + /// The user could specify session identifier and session timeout. /// It allows to modify settings, create temporary tables and reuse them in subsequent requests. @@ -335,6 +356,8 @@ void HTTPHandler::processQuery( context.setCurrentQueryId(params.get("query_id", request.get("X-ClickHouse-Query-Id", ""))); + client_info.initial_query_id = client_info.current_query_id; + /// The client can pass a HTTP header indicating supported compression method (gzip or deflate). String http_response_compression_methods = request.get("Accept-Encoding", ""); CompressionMethod http_response_compression_method = CompressionMethod::None; @@ -561,24 +584,6 @@ void HTTPHandler::processQuery( /// Origin header. used_output.out->addHeaderCORS(settings.add_http_cors_header && !request.get("Origin", "").empty()); - ClientInfo & client_info = context.getClientInfo(); - client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; - client_info.interface = ClientInfo::Interface::HTTP; - - /// Query sent through HTTP interface is initial. - client_info.initial_user = client_info.current_user; - client_info.initial_query_id = client_info.current_query_id; - client_info.initial_address = client_info.current_address; - - ClientInfo::HTTPMethod http_method = ClientInfo::HTTPMethod::UNKNOWN; - if (request.getMethod() == Poco::Net::HTTPServerRequest::HTTP_GET) - http_method = ClientInfo::HTTPMethod::GET; - else if (request.getMethod() == Poco::Net::HTTPServerRequest::HTTP_POST) - http_method = ClientInfo::HTTPMethod::POST; - - client_info.http_method = http_method; - client_info.http_user_agent = request.get("User-Agent", ""); - auto append_callback = [&context] (ProgressCallback callback) { auto prev = context.getProgressCallback(); diff --git a/src/Server/HTTPHandlerFactory.cpp b/src/Server/HTTPHandlerFactory.cpp index 8915ea747ca..9eac60355d2 100644 --- a/src/Server/HTTPHandlerFactory.cpp +++ b/src/Server/HTTPHandlerFactory.cpp @@ -31,10 +31,10 @@ HTTPRequestHandlerFactoryMain::HTTPRequestHandlerFactoryMain(const std::string & Poco::Net::HTTPRequestHandler * HTTPRequestHandlerFactoryMain::createRequestHandler(const Poco::Net::HTTPServerRequest & request) { - LOG_TRACE(log, "HTTP Request for {}. Method: {}, Address: {}, User-Agent: {}{}, Content Type: {}, Transfer Encoding: {}", - name, request.getMethod(), request.clientAddress().toString(), request.has("User-Agent") ? request.get("User-Agent") : "none", + LOG_TRACE(log, "HTTP Request for {}. Method: {}, Address: {}, User-Agent: {}{}, Content Type: {}, Transfer Encoding: {}, X-Forwarded-For: {}", + name, request.getMethod(), request.clientAddress().toString(), request.get("User-Agent", "(none)"), (request.hasContentLength() ? (", Length: " + std::to_string(request.getContentLength())) : ("")), - request.getContentType(), request.getTransferEncoding()); + request.getContentType(), request.getTransferEncoding(), request.get("X-Forwarded-For", "(none)")); for (auto & handler_factory : child_factories) { diff --git a/src/Server/TCPHandler.cpp b/src/Server/TCPHandler.cpp index 1f55a3af635..e84c89bd165 100644 --- a/src/Server/TCPHandler.cpp +++ b/src/Server/TCPHandler.cpp @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -76,6 +77,10 @@ void TCPHandler::runImpl() in = std::make_shared(socket()); out = std::make_shared(socket()); + /// Support for PROXY protocol + if (parse_proxy_protocol && !receiveProxyHeader()) + return; + if (in->eof()) { LOG_WARNING(log, "Client has not sent any data."); @@ -728,6 +733,78 @@ void TCPHandler::sendExtremes(const Block & extremes) } +bool TCPHandler::receiveProxyHeader() +{ + if (in->eof()) + { + LOG_WARNING(log, "Client has not sent any data."); + return false; + } + + String forwarded_address; + + /// Only PROXYv1 is supported. + /// Validation of protocol is not fully performed. + + LimitReadBuffer limit_in(*in, 107, true); /// Maximum length from the specs. + + assertString("PROXY ", limit_in); + + if (limit_in.eof()) + { + LOG_WARNING(log, "Incomplete PROXY header is received."); + return false; + } + + /// TCP4 / TCP6 / UNKNOWN + if ('T' == *limit_in.position()) + { + assertString("TCP", limit_in); + + if (limit_in.eof()) + { + LOG_WARNING(log, "Incomplete PROXY header is received."); + return false; + } + + if ('4' != *limit_in.position() && '6' != *limit_in.position()) + { + LOG_WARNING(log, "Unexpected protocol in PROXY header is received."); + return false; + } + + ++limit_in.position(); + assertChar(' ', limit_in); + + /// Read the first field and ignore other. + readStringUntilWhitespace(forwarded_address, limit_in); + + /// Skip until \r\n + while (!limit_in.eof() && *limit_in.position() != '\r') + ++limit_in.position(); + assertString("\r\n", limit_in); + } + else if (checkString("UNKNOWN", limit_in)) + { + /// This is just a health check, there is no subsequent data in this connection. + + while (!limit_in.eof() && *limit_in.position() != '\r') + ++limit_in.position(); + assertString("\r\n", limit_in); + return false; + } + else + { + LOG_WARNING(log, "Unexpected protocol in PROXY header is received."); + return false; + } + + LOG_TRACE(log, "Forwarded client address from PROXY header: {}", forwarded_address); + connection_context.getClientInfo().forwarded_for = forwarded_address; + return true; +} + + void TCPHandler::receiveHello() { /// Receive `hello` packet. diff --git a/src/Server/TCPHandler.h b/src/Server/TCPHandler.h index 2f2bf35e59e..e12e9bcf4d0 100644 --- a/src/Server/TCPHandler.h +++ b/src/Server/TCPHandler.h @@ -101,9 +101,18 @@ struct LastBlockInputParameters class TCPHandler : public Poco::Net::TCPServerConnection { public: - TCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) + /** parse_proxy_protocol_ - if true, expect and parse the header of PROXY protocol in every connection + * and set the information about forwarded address accordingly. + * See https://github.com/wolfeidau/proxyv2/blob/master/docs/proxy-protocol.txt + * + * Note: immediate IP address is always used for access control (accept-list of IP networks), + * because it allows to check the IP ranges of the trusted proxy. + * Proxy-forwarded (original client) IP address is used for quota accounting if quota is keyed by forwarded IP. + */ + TCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_, bool parse_proxy_protocol_) : Poco::Net::TCPServerConnection(socket_) , server(server_) + , parse_proxy_protocol(parse_proxy_protocol_) , log(&Poco::Logger::get("TCPHandler")) , connection_context(server.context()) , query_context(server.context()) @@ -118,6 +127,7 @@ public: private: IServer & server; + bool parse_proxy_protocol = false; Poco::Logger * log; String client_name; @@ -158,6 +168,7 @@ private: void runImpl(); + bool receiveProxyHeader(); void receiveHello(); bool receivePacket(); void receiveQuery(); diff --git a/src/Server/TCPHandlerFactory.h b/src/Server/TCPHandlerFactory.h index 5ecd427bf8b..73318fea9da 100644 --- a/src/Server/TCPHandlerFactory.h +++ b/src/Server/TCPHandlerFactory.h @@ -15,6 +15,7 @@ class TCPHandlerFactory : public Poco::Net::TCPServerConnectionFactory { private: IServer & server; + bool parse_proxy_protocol = false; Poco::Logger * log; class DummyTCPHandler : public Poco::Net::TCPServerConnection @@ -25,8 +26,12 @@ private: }; public: - explicit TCPHandlerFactory(IServer & server_, bool secure_ = false) - : server(server_) + /** parse_proxy_protocol_ - if true, expect and parse the header of PROXY protocol in every connection + * and set the information about forwarded address accordingly. + * See https://github.com/wolfeidau/proxyv2/blob/master/docs/proxy-protocol.txt + */ + TCPHandlerFactory(IServer & server_, bool secure_, bool parse_proxy_protocol_) + : server(server_), parse_proxy_protocol(parse_proxy_protocol_) , log(&Poco::Logger::get(std::string("TCP") + (secure_ ? "S" : "") + "HandlerFactory")) { } @@ -36,7 +41,8 @@ public: try { LOG_TRACE(log, "TCP Request. Address: {}", socket.peerAddress().toString()); - return new TCPHandler(server, socket); + + return new TCPHandler(server, socket, parse_proxy_protocol); } catch (const Poco::Net::NetException &) { diff --git a/src/Server/TestKeeperTCPHandler.cpp b/src/Server/TestKeeperTCPHandler.cpp new file mode 100644 index 00000000000..ffb73985824 --- /dev/null +++ b/src/Server/TestKeeperTCPHandler.cpp @@ -0,0 +1,452 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef POCO_HAVE_FD_EPOLL + #include +#else + #include +#endif + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int SYSTEM_ERROR; + extern const int UNEXPECTED_PACKET_FROM_CLIENT; +} + +static constexpr UInt8 RESPONSE_BYTE = 1; +static constexpr UInt8 WATCH_RESPONSE_BYTE = 2; + +struct SocketInterruptablePollWrapper +{ + int sockfd; + PipeFDs pipe; + +#if defined(POCO_HAVE_FD_EPOLL) + int epollfd; + epoll_event socket_event{}; + epoll_event pipe_event{}; +#endif + + using PollStatus = size_t; + static constexpr PollStatus TIMEOUT = 0x0; + static constexpr PollStatus HAS_REQUEST = 0x1; + static constexpr PollStatus HAS_RESPONSE = 0x2; + static constexpr PollStatus HAS_WATCH_RESPONSE = 0x4; + static constexpr PollStatus ERROR = 0x8; + + using InterruptCallback = std::function; + + explicit SocketInterruptablePollWrapper(const Poco::Net::StreamSocket & poco_socket_) + : sockfd(poco_socket_.impl()->sockfd()) + { + pipe.setNonBlockingReadWrite(); + +#if defined(POCO_HAVE_FD_EPOLL) + epollfd = epoll_create(2); + if (epollfd < 0) + throwFromErrno("Cannot epoll_create", ErrorCodes::SYSTEM_ERROR); + + socket_event.events = EPOLLIN | EPOLLERR; + socket_event.data.fd = sockfd; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sockfd, &socket_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } + pipe_event.events = EPOLLIN | EPOLLERR; + pipe_event.data.fd = pipe.fds_rw[0]; + if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pipe.fds_rw[0], &pipe_event) < 0) + { + ::close(epollfd); + throwFromErrno("Cannot insert socket into epoll queue", ErrorCodes::SYSTEM_ERROR); + } +#endif + } + + int getResponseFD() const + { + return pipe.fds_rw[1]; + } + + PollStatus poll(Poco::Timespan remaining_time) + { + std::array outputs = {-1, -1}; +#if defined(POCO_HAVE_FD_EPOLL) + int rc; + epoll_event evout[2]; + memset(evout, 0, sizeof(evout)); + do + { + Poco::Timestamp start; + rc = epoll_wait(epollfd, evout, 2, remaining_time.totalMilliseconds()); + if (rc < 0 && errno == EINTR) + { + Poco::Timestamp end; + Poco::Timespan waited = end - start; + if (waited < remaining_time) + remaining_time -= waited; + else + remaining_time = 0; + } + } + while (rc < 0 && errno == EINTR); + + if (rc >= 1 && evout[0].events & EPOLLIN) + outputs[0] = evout[0].data.fd; + if (rc == 2 && evout[1].events & EPOLLIN) + outputs[1] = evout[1].data.fd; +#else + pollfd poll_buf[2]; + poll_buf[0].fd = sockfd; + poll_buf[0].events = POLLIN; + poll_buf[1].fd = pipe.fds_rw[0]; + poll_buf[1].events = POLLIN; + + int rc; + do + { + Poco::Timestamp start; + rc = ::poll(poll_buf, 2, remaining_time.totalMilliseconds()); + if (rc < 0 && errno == POCO_EINTR) + { + Poco::Timestamp end; + Poco::Timespan waited = end - start; + if (waited < remaining_time) + remaining_time -= waited; + else + remaining_time = 0; + } + } + while (rc < 0 && errno == POCO_EINTR); + if (rc >= 1 && poll_buf[0].revents & POLLIN) + outputs[0] = sockfd; + if (rc == 2 && poll_buf[1].revents & POLLIN) + outputs[1] = pipe.fds_rw[0]; +#endif + + PollStatus result = TIMEOUT; + if (rc < 0) + { + return ERROR; + } + else if (rc == 0) + { + return result; + } + else + { + for (auto fd : outputs) + { + if (fd != -1) + { + if (fd == sockfd) + result |= HAS_REQUEST; + else + { + int read_result; + do + { + UInt8 byte; + read_result = read(pipe.fds_rw[0], &byte, sizeof(byte)); + if (read_result > 0) + { + if (byte == WATCH_RESPONSE_BYTE) + result |= HAS_WATCH_RESPONSE; + else if (byte == RESPONSE_BYTE) + result |= HAS_RESPONSE; + else + throw Exception("Unexpected byte received from signaling pipe", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + } + } + while (read_result > 0 || (read_result < 0 && errno == EINTR)); + + if (read_result < 0 && errno != EAGAIN) + throwFromErrno("Got error reading from pipe", ErrorCodes::SYSTEM_ERROR); + } + } + } + } + return result; + } + +#if defined(POCO_HAVE_FD_EPOLL) + ~SocketInterruptablePollWrapper() + { + ::close(epollfd); + } +#endif +}; + +TestKeeperTCPHandler::TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_) + : Poco::Net::TCPServerConnection(socket_) + , server(server_) + , log(&Poco::Logger::get("TestKeeperTCPHandler")) + , global_context(server.context()) + , test_keeper_storage(global_context.getTestKeeperStorage()) + , operation_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.operation_timeout_ms", Coordination::DEFAULT_OPERATION_TIMEOUT_MS) * 1000) + , session_timeout(0, global_context.getConfigRef().getUInt("test_keeper_server.session_timeout_ms", Coordination::DEFAULT_SESSION_TIMEOUT_MS) * 1000) + , session_id(test_keeper_storage->getSessionID()) + , poll_wrapper(std::make_unique(socket_)) +{ +} + +void TestKeeperTCPHandler::sendHandshake() +{ + Coordination::write(Coordination::SERVER_HANDSHAKE_LENGTH, *out); + Coordination::write(Coordination::ZOOKEEPER_PROTOCOL_VERSION, *out); + Coordination::write(Coordination::DEFAULT_SESSION_TIMEOUT_MS, *out); + Coordination::write(session_id, *out); + std::array passwd{}; + Coordination::write(passwd, *out); + out->next(); +} + +void TestKeeperTCPHandler::run() +{ + runImpl(); +} + +void TestKeeperTCPHandler::receiveHandshake() +{ + int32_t handshake_length; + int32_t protocol_version; + int64_t last_zxid_seen; + int32_t timeout; + int64_t previous_session_id = 0; /// We don't support session restore. So previous session_id is always zero. + std::array passwd {}; + + Coordination::read(handshake_length, *in); + if (handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH && handshake_length != Coordination::CLIENT_HANDSHAKE_LENGTH_WITH_READONLY) + throw Exception("Unexpected handshake length received: " + toString(handshake_length), ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + + Coordination::read(protocol_version, *in); + + if (protocol_version != Coordination::ZOOKEEPER_PROTOCOL_VERSION) + throw Exception("Unexpected protocol version: " + toString(protocol_version), ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + + Coordination::read(last_zxid_seen, *in); + + if (last_zxid_seen != 0) + throw Exception("Non zero last_zxid_seen is not supported", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + + Coordination::read(timeout, *in); + Coordination::read(previous_session_id, *in); + + if (previous_session_id != 0) + throw Exception("Non zero previous session id is not supported", ErrorCodes::UNEXPECTED_PACKET_FROM_CLIENT); + + Coordination::read(passwd, *in); + + int8_t readonly; + if (handshake_length == Coordination::CLIENT_HANDSHAKE_LENGTH_WITH_READONLY) + Coordination::read(readonly, *in); +} + + +void TestKeeperTCPHandler::runImpl() +{ + setThreadName("TstKprHandler"); + ThreadStatus thread_status; + auto global_receive_timeout = global_context.getSettingsRef().receive_timeout; + auto global_send_timeout = global_context.getSettingsRef().send_timeout; + + socket().setReceiveTimeout(global_receive_timeout); + socket().setSendTimeout(global_send_timeout); + socket().setNoDelay(true); + + in = std::make_shared(socket()); + out = std::make_shared(socket()); + + if (in->eof()) + { + LOG_WARNING(log, "Client has not sent any data."); + return; + } + + try + { + receiveHandshake(); + } + catch (const Exception & e) /// Typical for an incorrect username, password, or address. + { + LOG_WARNING(log, "Cannot receive handshake {}", e.displayText()); + return; + } + + sendHandshake(); + session_stopwatch.start(); + bool close_received = false; + try + { + while (true) + { + using namespace std::chrono_literals; + + auto state = poll_wrapper->poll(session_timeout); + if (state & SocketInterruptablePollWrapper::HAS_REQUEST) + { + do + { + Coordination::OpNum received_op = receiveRequest(); + + if (received_op == Coordination::OpNum::Close) + { + LOG_DEBUG(log, "Received close request for session #{}", session_id); + if (responses.back().wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + { + LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + } + else + { + LOG_DEBUG(log, "Sent close for session #{}", session_id); + responses.back().get()->write(*out); + } + close_received = true; + + break; + } + else if (received_op == Coordination::OpNum::Heartbeat) + { + LOG_TRACE(log, "Received heartbeat for session #{}", session_id); + session_stopwatch.restart(); + } + } + while (in->available()); + } + + if (close_received) + break; + + if (state & SocketInterruptablePollWrapper::HAS_RESPONSE) + { + while (!responses.empty()) + { + if (responses.front().wait_for(0s) != std::future_status::ready) + break; + + auto response = responses.front().get(); + response->write(*out); + responses.pop(); + } + } + + if (state & SocketInterruptablePollWrapper::HAS_WATCH_RESPONSE) + { + for (auto it = watch_responses.begin(); it != watch_responses.end();) + { + if (it->wait_for(0s) == std::future_status::ready) + { + auto response = it->get(); + if (response->error == Coordination::Error::ZOK) + response->write(*out); + it = watch_responses.erase(it); + } + else + { + ++it; + } + } + } + + if (state == SocketInterruptablePollWrapper::ERROR) + { + throw Exception("Exception happened while reading from socket", ErrorCodes::SYSTEM_ERROR); + } + + if (session_stopwatch.elapsedMicroseconds() > static_cast(session_timeout.totalMicroseconds())) + { + LOG_DEBUG(log, "Session #{} expired", session_id); + auto response = putCloseRequest(); + if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + LOG_DEBUG(log, "Cannot sent close for expired session #{}", session_id); + else + response.get()->write(*out); + + break; + } + } + } + catch (const Exception & ex) + { + LOG_INFO(log, "Got exception processing session #{}: {}", session_id, getExceptionMessage(ex, true)); + auto response = putCloseRequest(); + if (response.wait_for(std::chrono::microseconds(operation_timeout.totalMicroseconds())) != std::future_status::ready) + LOG_DEBUG(log, "Cannot sent close for session #{}", session_id); + else + response.get()->write(*out); + } + +} + +zkutil::TestKeeperStorage::AsyncResponse TestKeeperTCPHandler::putCloseRequest() +{ + Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(Coordination::OpNum::Close); + request->xid = Coordination::CLOSE_XID; + auto promise = std::make_shared>(); + zkutil::ResponseCallback callback = [promise] (const Coordination::ZooKeeperResponsePtr & response) + { + promise->set_value(response); + }; + test_keeper_storage->putRequest(request, session_id, callback); + return promise->get_future(); +} + +Coordination::OpNum TestKeeperTCPHandler::receiveRequest() +{ + int32_t length; + Coordination::read(length, *in); + int32_t xid; + Coordination::read(xid, *in); + + Coordination::OpNum opnum; + Coordination::read(opnum, *in); + + Coordination::ZooKeeperRequestPtr request = Coordination::ZooKeeperRequestFactory::instance().get(opnum); + request->xid = xid; + request->readImpl(*in); + int response_fd = poll_wrapper->getResponseFD(); + auto promise = std::make_shared>(); + zkutil::ResponseCallback callback = [response_fd, promise] (const Coordination::ZooKeeperResponsePtr & response) + { + promise->set_value(response); + [[maybe_unused]] int result = write(response_fd, &RESPONSE_BYTE, sizeof(RESPONSE_BYTE)); + }; + + if (request->has_watch) + { + auto watch_promise = std::make_shared>(); + zkutil::ResponseCallback watch_callback = [response_fd, watch_promise] (const Coordination::ZooKeeperResponsePtr & response) + { + watch_promise->set_value(response); + [[maybe_unused]] int result = write(response_fd, &WATCH_RESPONSE_BYTE, sizeof(WATCH_RESPONSE_BYTE)); + }; + test_keeper_storage->putRequest(request, session_id, callback, watch_callback); + responses.push(promise->get_future()); + watch_responses.emplace_back(watch_promise->get_future()); + } + else + { + test_keeper_storage->putRequest(request, session_id, callback); + responses.push(promise->get_future()); + } + + return opnum; +} + + +} diff --git a/src/Server/TestKeeperTCPHandler.h b/src/Server/TestKeeperTCPHandler.h new file mode 100644 index 00000000000..2f36dfafddf --- /dev/null +++ b/src/Server/TestKeeperTCPHandler.h @@ -0,0 +1,52 @@ +#pragma once + +#include +#include "IServer.h" +#include +#include +#include +#include +#include +#include +#include +#include + +namespace DB +{ + +struct SocketInterruptablePollWrapper; +using SocketInterruptablePollWrapperPtr = std::unique_ptr; + +class TestKeeperTCPHandler : public Poco::Net::TCPServerConnection +{ +public: + TestKeeperTCPHandler(IServer & server_, const Poco::Net::StreamSocket & socket_); + void run() override; +private: + IServer & server; + Poco::Logger * log; + Context global_context; + std::shared_ptr test_keeper_storage; + Poco::Timespan operation_timeout; + Poco::Timespan session_timeout; + int64_t session_id; + Stopwatch session_stopwatch; + SocketInterruptablePollWrapperPtr poll_wrapper; + + std::queue responses; + std::vector watch_responses; + + /// Streams for reading/writing from/to client connection socket. + std::shared_ptr in; + std::shared_ptr out; + + void runImpl(); + + void sendHandshake(); + void receiveHandshake(); + + Coordination::OpNum receiveRequest(); + zkutil::TestKeeperStorage::AsyncResponse putCloseRequest(); +}; + +} diff --git a/src/Server/TestKeeperTCPHandlerFactory.h b/src/Server/TestKeeperTCPHandlerFactory.h new file mode 100644 index 00000000000..ebf91aa31d4 --- /dev/null +++ b/src/Server/TestKeeperTCPHandlerFactory.h @@ -0,0 +1,44 @@ +#pragma once +#include +#include +#include +#include +#include + +namespace DB +{ + +class TestKeeperTCPHandlerFactory : public Poco::Net::TCPServerConnectionFactory +{ +private: + IServer & server; + Poco::Logger * log; + class DummyTCPHandler : public Poco::Net::TCPServerConnection + { + public: + using Poco::Net::TCPServerConnection::TCPServerConnection; + void run() override {} + }; +public: + TestKeeperTCPHandlerFactory(IServer & server_) + : server(server_) + , log(&Poco::Logger::get("TestKeeperTCPHandlerFactory")) + { + } + + Poco::Net::TCPServerConnection * createConnection(const Poco::Net::StreamSocket & socket) override + { + try + { + LOG_TRACE(log, "Test keeper request. Address: {}", socket.peerAddress().toString()); + return new TestKeeperTCPHandler(server, socket); + } + catch (const Poco::Net::NetException &) + { + LOG_TRACE(log, "TCP Request. Client is not connected (most likely RST packet was sent)."); + return new DummyTCPHandler(socket); + } + } +}; + +} diff --git a/src/Server/ya.make b/src/Server/ya.make index 9a728b39641..1e44577aea9 100644 --- a/src/Server/ya.make +++ b/src/Server/ya.make @@ -25,6 +25,7 @@ SRCS( ReplicasStatusHandler.cpp StaticRequestHandler.cpp TCPHandler.cpp + TestKeeperTCPHandler.cpp WebUIRequestHandler.cpp ) diff --git a/src/Storages/MergeTree/IMergeTreeDataPart.cpp b/src/Storages/MergeTree/IMergeTreeDataPart.cpp index aa37e62d099..6a98529966c 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPart.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPart.cpp @@ -659,6 +659,29 @@ void IMergeTreeDataPart::loadRowsCount() "Column {} has rows count {} according to size in memory " "and size of single value, but data part {} has {} rows", backQuote(column.name), rows_in_column, name, rows_count); } + + size_t last_possibly_incomplete_mark_rows = index_granularity.getLastNonFinalMarkRows(); + /// All this rows have to be written in column + size_t index_granularity_without_last_mark = index_granularity.getTotalRows() - last_possibly_incomplete_mark_rows; + /// We have more rows in column than in index granularity without last possibly incomplete mark + if (rows_in_column < index_granularity_without_last_mark) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Column {} has rows count {} according to size in memory " + "and size of single value, but index granularity in part {} without last mark has {} rows, which is more than in column", + backQuote(column.name), rows_in_column, name, index_granularity.getTotalRows()); + } + + /// In last mark we actually written less or equal rows than stored in last mark of index granularity + if (rows_in_column - index_granularity_without_last_mark > last_possibly_incomplete_mark_rows) + { + throw Exception( + ErrorCodes::LOGICAL_ERROR, + "Column {} has rows count {} in last mark according to size in memory " + "and size of single value, but index granularity in part {} in last mark has {} rows which is less than in column", + backQuote(column.name), rows_in_column - index_granularity_without_last_mark, name, last_possibly_incomplete_mark_rows); + } } } #endif diff --git a/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp b/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp index d291a447d1c..143a9920b93 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp +++ b/src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp @@ -44,29 +44,6 @@ void IMergeTreeDataPartWriter::next() index_offset = next_index_offset; } -bool IMergeTreeDataPartWriter::adjustLastUnfinishedMark(size_t new_block_index_granularity) -{ - /// If amount of rest rows in the last granule more then granularity of the new block - /// than finish it. - if (!index_granularity.empty() && index_offset > new_block_index_granularity) - { - size_t already_written_rows_in_last_granule = index_granularity.getLastMarkRows() - index_offset; - /// We can still write some rows to the last granule - if (already_written_rows_in_last_granule < new_block_index_granularity) - { - index_granularity.setLastMarkRows(new_block_index_granularity); - index_offset = new_block_index_granularity - already_written_rows_in_last_granule; - } - else /// Our last granule is already full, let's start the new one - { - index_granularity.setLastMarkRows(already_written_rows_in_last_granule); - index_offset = 0; - } - return true; - } - return false; -} - IMergeTreeDataPartWriter::~IMergeTreeDataPartWriter() = default; } diff --git a/src/Storages/MergeTree/IMergeTreeDataPartWriter.h b/src/Storages/MergeTree/IMergeTreeDataPartWriter.h index 9a6ffca1adb..4a42a58a65b 100644 --- a/src/Storages/MergeTree/IMergeTreeDataPartWriter.h +++ b/src/Storages/MergeTree/IMergeTreeDataPartWriter.h @@ -62,41 +62,28 @@ public: protected: size_t getCurrentMark() const { return current_mark; } size_t getIndexOffset() const { return index_offset; } - /// Finishes our current unfinished mark if we have already written more rows for it - /// than granularity in the new block. Return true if last mark actually was adjusted. - /// Example: - /// __|________|___. <- previous block with granularity 8 and last unfinished mark with 3 rows - /// new_block_index_granularity = 2, so - /// __|________|___|__|__|__| - /// ^ finish last unfinished mark, new marks will have granularity 2 - bool adjustLastUnfinishedMark(size_t new_block_index_granularity); using SerializationState = IDataType::SerializeBinaryBulkStatePtr; using SerializationStates = std::unordered_map; MergeTreeData::DataPartPtr data_part; const MergeTreeData & storage; - const StorageMetadataPtr metadata_snapshot; - const NamesAndTypesList columns_list; - const MergeTreeIndices skip_indices; + StorageMetadataPtr metadata_snapshot; + NamesAndTypesList columns_list; + MergeTreeIndices skip_indices; MergeTreeIndexGranularity index_granularity; - const MergeTreeWriterSettings settings; - const bool with_final_mark; + MergeTreeWriterSettings settings; + bool with_final_mark; size_t next_mark = 0; size_t next_index_offset = 0; - /// When we were writing fresh block granularity of the last mark was adjusted - /// See adjustLastUnfinishedMark - bool last_granule_was_adjusted = false; - MutableColumns index_columns; private: /// Data is already written up to this mark. size_t current_mark = 0; /// The offset to the first row of the block for which you want to write the index. - /// Or how many rows we have to write for this last unfinished mark. size_t index_offset = 0; }; diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp index 31d78dd48a7..0ec10f9db76 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp @@ -184,8 +184,7 @@ void MergeTreeDataPartWriterOnDisk::initSkipIndices() default_codec, settings.max_compress_block_size, 0, settings.aio_threshold)); skip_indices_aggregators.push_back(index_helper->createIndexAggregator()); - marks_in_skip_index_aggregator.push_back(0); - rows_in_skip_index_aggregator_last_mark.push_back(0); + skip_index_filling.push_back(0); } skip_indices_initialized = true; @@ -257,11 +256,9 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block skip_index_current_data_mark = skip_index_data_mark; while (prev_pos < rows) { - bool new_block_started = prev_pos == 0; UInt64 limit = 0; size_t current_index_offset = getIndexOffset(); - /// We start new block, but have an offset from previous one - if (new_block_started && current_index_offset != 0) + if (prev_pos == 0 && current_index_offset != 0) { limit = current_index_offset; } @@ -273,15 +270,10 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block else { limit = index_granularity.getMarkRows(skip_index_current_data_mark); - /// We just started new block serialization but last unfinished mark was shrinked to it's current_size - /// it may happen that we have already aggregated current_size of rows of more for skip_index, but not flushed it to disk - /// because previous granule size was bigger. So do it here. - if (new_block_started && last_granule_was_adjusted && rows_in_skip_index_aggregator_last_mark[i] >= limit) - accountMarkForSkipIdxAndFlushIfNeeded(i); - if (skip_indices_aggregators[i]->empty()) { skip_indices_aggregators[i] = index_helper->createIndexAggregator(); + skip_index_filling[i] = 0; if (stream.compressed.offset() >= settings.min_compress_block_size) stream.compressed.next(); @@ -293,19 +285,24 @@ void MergeTreeDataPartWriterOnDisk::calculateAndSerializeSkipIndices(const Block if (settings.can_use_adaptive_granularity) writeIntBinary(1UL, stream.marks); } - /// this mark is aggregated, go to the next one skip_index_current_data_mark++; } size_t pos = prev_pos; skip_indices_aggregators[i]->update(skip_indexes_block, &pos, limit); - rows_in_skip_index_aggregator_last_mark[i] = (pos - prev_pos); - /// We just aggregated all rows in current mark, add new mark to skip_index marks counter - /// and flush on disk if we already aggregated required amount of marks. - if (rows_in_skip_index_aggregator_last_mark[i] == limit) - accountMarkForSkipIdxAndFlushIfNeeded(i); + if (pos == prev_pos + limit) + { + ++skip_index_filling[i]; + + /// write index if it is filled + if (skip_index_filling[i] == index_helper->index.granularity) + { + skip_indices_aggregators[i]->getGranuleAndReset()->serializeBinary(stream.compressed); + skip_index_filling[i] = 0; + } + } prev_pos = pos; } } @@ -363,21 +360,7 @@ void MergeTreeDataPartWriterOnDisk::finishSkipIndicesSerialization( skip_indices_streams.clear(); skip_indices_aggregators.clear(); - marks_in_skip_index_aggregator.clear(); - rows_in_skip_index_aggregator_last_mark.clear(); -} - -void MergeTreeDataPartWriterOnDisk::accountMarkForSkipIdxAndFlushIfNeeded(size_t skip_index_pos) -{ - ++marks_in_skip_index_aggregator[skip_index_pos]; - - /// write index if it is filled - if (marks_in_skip_index_aggregator[skip_index_pos] == skip_indices[skip_index_pos]->index.granularity) - { - skip_indices_aggregators[skip_index_pos]->getGranuleAndReset()->serializeBinary(skip_indices_streams[skip_index_pos]->compressed); - marks_in_skip_index_aggregator[skip_index_pos] = 0; - rows_in_skip_index_aggregator_last_mark[skip_index_pos] = 0; - } + skip_index_filling.clear(); } } diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h index 743b130188d..55da413c9fb 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h @@ -97,7 +97,8 @@ protected: const String marks_file_extension; CompressionCodecPtr default_codec; - const bool compute_granularity; + bool compute_granularity; + bool need_finish_last_granule; /// Number of marsk in data from which skip indices have to start /// aggregation. I.e. it's data mark number, not skip indices mark. @@ -105,10 +106,7 @@ protected: std::vector skip_indices_streams; MergeTreeIndexAggregators skip_indices_aggregators; - /// Amount of marks currently serialized in skip index aggregator - std::vector marks_in_skip_index_aggregator; - /// Amount of rows currently serialized in skip index aggregator for last mark - std::vector rows_in_skip_index_aggregator_last_mark; + std::vector skip_index_filling; std::unique_ptr index_file_stream; std::unique_ptr index_stream; @@ -127,11 +125,6 @@ protected: private: /// Index is already serialized up to this mark. size_t index_mark = 0; - - /// Increment corresponding marks_in_skip_index_aggregator[skip_index_pos] - /// value and flush skip_indices_streams[skip_index_pos] to disk if we have - /// aggregated enough marks - void accountMarkForSkipIdxAndFlushIfNeeded(size_t skip_index_pos); }; } diff --git a/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp b/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp index 21ae8335af1..c15c39e7b7f 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp @@ -95,9 +95,6 @@ void MergeTreeDataPartWriterWide::write(const Block & block, if (compute_granularity) { size_t index_granularity_for_block = computeIndexGranularity(block); - /// Finish last unfinished mark rows it it's required - last_granule_was_adjusted = adjustLastUnfinishedMark(index_granularity_for_block); - /// Fill index granularity with granules of new size fillIndexGranularity(index_granularity_for_block, block.rows()); } diff --git a/src/Storages/MergeTree/MergeTreeIOSettings.h b/src/Storages/MergeTree/MergeTreeIOSettings.h index db01f1ae9b7..41d9be07c70 100644 --- a/src/Storages/MergeTree/MergeTreeIOSettings.h +++ b/src/Storages/MergeTree/MergeTreeIOSettings.h @@ -1,6 +1,7 @@ #pragma once #include #include +#include namespace DB { @@ -19,13 +20,22 @@ struct MergeTreeWriterSettings { MergeTreeWriterSettings() = default; - MergeTreeWriterSettings(const Settings & global_settings, bool can_use_adaptive_granularity_, - size_t aio_threshold_, bool blocks_are_granules_size_ = false) - : min_compress_block_size(global_settings.min_compress_block_size) - , max_compress_block_size(global_settings.max_compress_block_size) + MergeTreeWriterSettings( + const Settings & global_settings, + const MergeTreeSettingsPtr & storage_settings, + bool can_use_adaptive_granularity_, + size_t aio_threshold_, + bool blocks_are_granules_size_ = false) + : min_compress_block_size( + storage_settings->min_compress_block_size ? storage_settings->min_compress_block_size : global_settings.min_compress_block_size) + , max_compress_block_size( + storage_settings->max_compress_block_size ? storage_settings->max_compress_block_size + : global_settings.max_compress_block_size) , aio_threshold(aio_threshold_) , can_use_adaptive_granularity(can_use_adaptive_granularity_) - , blocks_are_granules_size(blocks_are_granules_size_) {} + , blocks_are_granules_size(blocks_are_granules_size_) + { + } size_t min_compress_block_size; size_t max_compress_block_size; diff --git a/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp b/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp index 2db087a394f..bca0d0cb883 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp +++ b/src/Storages/MergeTree/MergeTreeIndexGranularity.cpp @@ -55,17 +55,6 @@ void MergeTreeIndexGranularity::addRowsToLastMark(size_t rows_count) marks_rows_partial_sums.back() += rows_count; } -void MergeTreeIndexGranularity::setLastMarkRows(size_t rows_count) -{ - if (marks_rows_partial_sums.empty()) - marks_rows_partial_sums.push_back(rows_count); - else - { - marks_rows_partial_sums.back() -= getLastMarkRows(); - marks_rows_partial_sums.back() += rows_count; - } -} - void MergeTreeIndexGranularity::popMark() { if (!marks_rows_partial_sums.empty()) diff --git a/src/Storages/MergeTree/MergeTreeIndexGranularity.h b/src/Storages/MergeTree/MergeTreeIndexGranularity.h index bfb48511285..5aefd0f102b 100644 --- a/src/Storages/MergeTree/MergeTreeIndexGranularity.h +++ b/src/Storages/MergeTree/MergeTreeIndexGranularity.h @@ -98,10 +98,6 @@ public: /// Extends last mark by rows_count. void addRowsToLastMark(size_t rows_count); - /// Set amount of rows to last mark - /// (add new mark if new have nothing) - void setLastMarkRows(size_t rows_count); - /// Drops last mark if any exists. void popMark(); diff --git a/src/Storages/MergeTree/MergeTreeSettings.h b/src/Storages/MergeTree/MergeTreeSettings.h index b2889af4f11..2f3931786a6 100644 --- a/src/Storages/MergeTree/MergeTreeSettings.h +++ b/src/Storages/MergeTree/MergeTreeSettings.h @@ -17,6 +17,8 @@ struct Settings; #define LIST_OF_MERGE_TREE_SETTINGS(M) \ + M(UInt64, min_compress_block_size, 0, "When granule is written, compress the data in buffer if the size of pending uncompressed data is larger or equal than the specified threshold. If this setting is not set, the corresponding global setting is used.", 0) \ + M(UInt64, max_compress_block_size, 0, "Compress the pending uncompressed data in buffer if its size is larger or equal than the specified threshold. Block of data will be compressed even if the current granule is not finished. If this setting is not set, the corresponding global setting is used.", 0) \ M(UInt64, index_granularity, 8192, "How many rows correspond to one primary key value.", 0) \ \ /** Data storing format settings. */ \ diff --git a/src/Storages/MergeTree/MergedBlockOutputStream.cpp b/src/Storages/MergeTree/MergedBlockOutputStream.cpp index 42d310da485..00a4c37c60d 100644 --- a/src/Storages/MergeTree/MergedBlockOutputStream.cpp +++ b/src/Storages/MergeTree/MergedBlockOutputStream.cpp @@ -48,6 +48,7 @@ MergedBlockOutputStream::MergedBlockOutputStream( { MergeTreeWriterSettings writer_settings( storage.global_context.getSettings(), + storage.getSettings(), data_part->index_granularity_info.is_adaptive, aio_threshold, blocks_are_granules_size); diff --git a/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp b/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp index 8ce4ea126ed..12b14d13656 100644 --- a/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp +++ b/src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp @@ -22,8 +22,11 @@ MergedColumnOnlyOutputStream::MergedColumnOnlyOutputStream( , header(header_) { const auto & global_settings = data_part->storage.global_context.getSettings(); + const auto & storage_settings = data_part->storage.getSettings(); + MergeTreeWriterSettings writer_settings( global_settings, + storage_settings, index_granularity_info ? index_granularity_info->is_adaptive : data_part->storage.canUseAdaptiveGranularity(), global_settings.min_bytes_to_use_direct_io); diff --git a/src/Storages/StorageMerge.cpp b/src/Storages/StorageMerge.cpp index f15ef8a5784..97f4ccd0bba 100644 --- a/src/Storages/StorageMerge.cpp +++ b/src/Storages/StorageMerge.cpp @@ -430,7 +430,15 @@ StorageMerge::StorageListWithLocks StorageMerge::getSelectedTables( DatabaseTablesIteratorPtr StorageMerge::getDatabaseIterator(const Context & context) const { - checkStackSize(); + try + { + checkStackSize(); + } + catch (Exception & e) + { + e.addMessage("while getting table iterator of Merge table. Maybe caused by two Merge tables that will endlessly try to read each other's data"); + throw; + } auto database = DatabaseCatalog::instance().getDatabase(source_database); auto table_name_match = [this](const String & table_name_) { return table_name_regexp.match(table_name_); }; return database->getTablesIterator(context, table_name_match); diff --git a/src/Storages/StorageS3.cpp b/src/Storages/StorageS3.cpp index ea9319d4693..fff44bb1f4c 100644 --- a/src/Storages/StorageS3.cpp +++ b/src/Storages/StorageS3.cpp @@ -216,8 +216,14 @@ StorageS3::StorageS3( credentials = Aws::Auth::AWSCredentials(std::move(settings.access_key_id), std::move(settings.secret_access_key)); client = S3::ClientFactory::instance().create( - uri_.endpoint, uri_.is_virtual_hosted_style, access_key_id_, secret_access_key_, std::move(settings.headers), - context_.getRemoteHostFilter(), context_.getGlobalContext().getSettingsRef().s3_max_redirects); + uri_.endpoint, + uri_.is_virtual_hosted_style, + credentials.GetAWSAccessKeyId(), + credentials.GetAWSSecretKey(), + std::move(settings.headers), + settings.use_environment_credentials.value_or(global_context.getConfigRef().getBool("s3.use_environment_credentials", false)), + context_.getRemoteHostFilter(), + context_.getGlobalContext().getSettingsRef().s3_max_redirects); } diff --git a/src/Storages/StorageS3Settings.cpp b/src/Storages/StorageS3Settings.cpp index 5b443de6b9a..9c4dc5831dc 100644 --- a/src/Storages/StorageS3Settings.cpp +++ b/src/Storages/StorageS3Settings.cpp @@ -26,6 +26,11 @@ void StorageS3Settings::loadFromConfig(const String & config_elem, const Poco::U auto endpoint = config.getString(config_elem + "." + key + ".endpoint"); auto access_key_id = config.getString(config_elem + "." + key + ".access_key_id", ""); auto secret_access_key = config.getString(config_elem + "." + key + ".secret_access_key", ""); + std::optional use_environment_credentials; + if (config.has(config_elem + "." + key + ".use_environment_credentials")) + { + use_environment_credentials = config.getBool(config_elem + "." + key + ".use_environment_credentials"); + } HeaderCollection headers; Poco::Util::AbstractConfiguration::Keys subconfig_keys; @@ -42,7 +47,7 @@ void StorageS3Settings::loadFromConfig(const String & config_elem, const Poco::U } } - settings.emplace(endpoint, S3AuthSettings{std::move(access_key_id), std::move(secret_access_key), std::move(headers)}); + settings.emplace(endpoint, S3AuthSettings{std::move(access_key_id), std::move(secret_access_key), std::move(headers), use_environment_credentials}); } } diff --git a/src/Storages/StorageS3Settings.h b/src/Storages/StorageS3Settings.h index 19b2bf48bd8..88f964774c6 100644 --- a/src/Storages/StorageS3Settings.h +++ b/src/Storages/StorageS3Settings.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -28,6 +29,8 @@ struct S3AuthSettings const String secret_access_key; const HeaderCollection headers; + + std::optional use_environment_credentials; }; /// Settings for the StorageS3. diff --git a/src/Storages/System/StorageSystemProcesses.cpp b/src/Storages/System/StorageSystemProcesses.cpp index d899a1708bf..178eedd9ad7 100644 --- a/src/Storages/System/StorageSystemProcesses.cpp +++ b/src/Storages/System/StorageSystemProcesses.cpp @@ -43,6 +43,7 @@ NamesAndTypesList StorageSystemProcesses::getNamesAndTypes() {"http_method", std::make_shared()}, {"http_user_agent", std::make_shared()}, + {"forwarded_for", std::make_shared()}, {"quota_key", std::make_shared()}, @@ -98,6 +99,7 @@ void StorageSystemProcesses::fillData(MutableColumns & res_columns, const Contex res_columns[i++]->insert(UInt64(process.client_info.http_method)); res_columns[i++]->insert(process.client_info.http_user_agent); + res_columns[i++]->insert(process.client_info.forwarded_for); res_columns[i++]->insert(process.client_info.quota_key); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 3ef09e5658f..9e5a2e29dc9 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -15,6 +15,16 @@ install ( COMPONENT clickhouse PATTERN "CMakeLists.txt" EXCLUDE PATTERN ".gitignore" EXCLUDE + PATTERN "top_level_domains" EXCLUDE +) + +# Dereference symlink +get_filename_component(TOP_LEVEL_DOMAINS_ABS_DIR config/top_level_domains REALPATH) +install ( + DIRECTORY "${TOP_LEVEL_DOMAINS_ABS_DIR}" + DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/clickhouse-test/config + USE_SOURCE_PERMISSIONS + COMPONENT clickhouse ) install (FILES server-test.xml DESTINATION ${CLICKHOUSE_ETC_DIR}/clickhouse-server COMPONENT clickhouse) diff --git a/tests/config/config.d/tcp_with_proxy.xml b/tests/config/config.d/tcp_with_proxy.xml new file mode 100644 index 00000000000..19046054c16 --- /dev/null +++ b/tests/config/config.d/tcp_with_proxy.xml @@ -0,0 +1,3 @@ + + 9010 + diff --git a/tests/config/config.d/test_keeper_port.xml b/tests/config/config.d/test_keeper_port.xml new file mode 100644 index 00000000000..79e993b41f7 --- /dev/null +++ b/tests/config/config.d/test_keeper_port.xml @@ -0,0 +1,7 @@ + + + 9181 + 10000 + 30000 + + diff --git a/tests/config/config.d/top_level_domains_lists.xml b/tests/config/config.d/top_level_domains_lists.xml new file mode 100644 index 00000000000..7b5e6a5638a --- /dev/null +++ b/tests/config/config.d/top_level_domains_lists.xml @@ -0,0 +1,5 @@ + + + public_suffix_list.dat + + diff --git a/tests/config/config.d/top_level_domains_path.xml b/tests/config/config.d/top_level_domains_path.xml new file mode 100644 index 00000000000..0ab836e5818 --- /dev/null +++ b/tests/config/config.d/top_level_domains_path.xml @@ -0,0 +1,3 @@ + + /etc/clickhouse-server/top_level_domains/ + diff --git a/tests/config/config.d/zookeeper.xml b/tests/config/config.d/zookeeper.xml index 68c85788c98..06ed7fcd39f 100644 --- a/tests/config/config.d/zookeeper.xml +++ b/tests/config/config.d/zookeeper.xml @@ -1,5 +1,8 @@ - testkeeper + + localhost + 9181 + diff --git a/tests/config/install.sh b/tests/config/install.sh index c20fb4c8f4e..416cc21893b 100755 --- a/tests/config/install.sh +++ b/tests/config/install.sh @@ -28,7 +28,11 @@ ln -sf $SRC_PATH/config.d/clusters.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/graphite.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/database_atomic.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/test_cluster_with_incorrect_pw.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/test_keeper_port.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/config.d/logging_no_rotate.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/tcp_with_proxy.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/top_level_domains_lists.xml $DEST_SERVER_PATH/config.d/ +ln -sf $SRC_PATH/config.d/top_level_domains_path.xml $DEST_SERVER_PATH/config.d/ ln -sf $SRC_PATH/users.d/log_queries.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/readonly.xml $DEST_SERVER_PATH/users.d/ ln -sf $SRC_PATH/users.d/access_management.xml $DEST_SERVER_PATH/users.d/ @@ -40,6 +44,8 @@ ln -sf $SRC_PATH/strings_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/decimals_dictionary.xml $DEST_SERVER_PATH/ ln -sf $SRC_PATH/executable_dictionary.xml $DEST_SERVER_PATH/ +ln -sf $SRC_PATH/top_level_domains $DEST_SERVER_PATH/ + ln -sf $SRC_PATH/server.key $DEST_SERVER_PATH/ ln -sf $SRC_PATH/server.crt $DEST_SERVER_PATH/ ln -sf $SRC_PATH/dhparam.pem $DEST_SERVER_PATH/ diff --git a/tests/config/top_level_domains b/tests/config/top_level_domains new file mode 120000 index 00000000000..7e12ab4ba2c --- /dev/null +++ b/tests/config/top_level_domains @@ -0,0 +1 @@ +../../docker/test/performance-comparison/config/top_level_domains \ No newline at end of file diff --git a/tests/integration/test_grpc_protocol/configs/grpc_config.xml b/tests/integration/test_grpc_protocol/configs/grpc_config.xml new file mode 100644 index 00000000000..ad941306892 --- /dev/null +++ b/tests/integration/test_grpc_protocol/configs/grpc_config.xml @@ -0,0 +1,7 @@ + + 9100 + + + false + + diff --git a/tests/integration/test_grpc_protocol/configs/grpc_port.xml b/tests/integration/test_grpc_protocol/configs/grpc_port.xml deleted file mode 100644 index e4a8ba829f9..00000000000 --- a/tests/integration/test_grpc_protocol/configs/grpc_port.xml +++ /dev/null @@ -1,3 +0,0 @@ - - 9100 - diff --git a/tests/integration/test_grpc_protocol/test.py b/tests/integration/test_grpc_protocol/test.py index 5242bd56d01..3a3dc464155 100644 --- a/tests/integration/test_grpc_protocol/test.py +++ b/tests/integration/test_grpc_protocol/test.py @@ -28,7 +28,7 @@ import clickhouse_grpc_pb2_grpc config_dir = os.path.join(SCRIPT_DIR, './configs') cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', main_configs=['configs/grpc_port.xml']) +node = cluster.add_instance('node', main_configs=['configs/grpc_config.xml']) grpc_port = 9100 main_channel = None @@ -138,15 +138,12 @@ def reset_after_test(): # Actual tests -@pytest.mark.skip(reason="Flaky") def test_select_one(): assert query("SELECT 1") == "1\n" -@pytest.mark.skip(reason="Flaky") def test_ordinary_query(): assert query("SELECT count() FROM numbers(100)") == "100\n" -@pytest.mark.skip(reason="Flaky") def test_insert_query(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t VALUES (1),(2),(3)") @@ -155,13 +152,11 @@ def test_insert_query(): query("INSERT INTO t FORMAT TabSeparated", input_data="9\n10\n") assert query("SELECT a FROM t ORDER BY a") == "1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n" -@pytest.mark.skip(reason="Flaky") def test_insert_query_streaming(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t VALUES", input_data=["(1),(2),(3),", "(5),(4),(6),", "(7),(8),(9)"]) assert query("SELECT a FROM t ORDER BY a") == "1\n2\n3\n4\n5\n6\n7\n8\n9\n" -@pytest.mark.skip(reason="Flaky") def test_insert_query_delimiter(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t FORMAT CSV 1\n2", input_data=["3", "4\n5"], input_data_delimiter='\n') @@ -171,7 +166,6 @@ def test_insert_query_delimiter(): query("INSERT INTO t FORMAT CSV 1\n2", input_data=["3", "4\n5"]) assert query("SELECT a FROM t ORDER BY a") == "1\n5\n234\n" -@pytest.mark.skip(reason="Flaky") def test_insert_default_column(): query("CREATE TABLE t (a UInt8, b Int32 DEFAULT 100, c String DEFAULT 'c') ENGINE = Memory") query("INSERT INTO t (c, a) VALUES ('x',1),('y',2)") @@ -181,20 +175,17 @@ def test_insert_default_column(): "3\t100\tc\n" \ "4\t100\tc\n" -@pytest.mark.skip(reason="Flaky") def test_insert_splitted_row(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t VALUES", input_data=["(1),(2),(", "3),(5),(4),(6)"]) assert query("SELECT a FROM t ORDER BY a") == "1\n2\n3\n4\n5\n6\n" -@pytest.mark.skip(reason="Flaky") def test_output_format(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t VALUES (1),(2),(3)") assert query("SELECT a FROM t ORDER BY a FORMAT JSONEachRow") == '{"a":1}\n{"a":2}\n{"a":3}\n' assert query("SELECT a FROM t ORDER BY a", output_format="JSONEachRow") == '{"a":1}\n{"a":2}\n{"a":3}\n' -@pytest.mark.skip(reason="Flaky") def test_totals_and_extremes(): query("CREATE TABLE t (x UInt8, y UInt8) ENGINE = Memory") query("INSERT INTO t VALUES (1, 2), (2, 4), (3, 2), (3, 3), (3, 4)") @@ -203,7 +194,6 @@ def test_totals_and_extremes(): assert query("SELECT x, y FROM t") == "1\t2\n2\t4\n3\t2\n3\t3\n3\t4\n" assert query_and_get_extremes("SELECT x, y FROM t", settings={"extremes": "1"}) == "1\t2\n3\t4\n" -@pytest.mark.skip(reason="Flaky") def test_errors_handling(): e = query_and_get_error("") #print(e) @@ -212,19 +202,16 @@ def test_errors_handling(): e = query_and_get_error("CREATE TABLE t (a UInt8) ENGINE = Memory") assert "Table default.t already exists" in e.display_text -@pytest.mark.skip(reason="Flaky") def test_authentication(): query("CREATE USER john IDENTIFIED BY 'qwe123'") assert query("SELECT currentUser()", user_name="john", password="qwe123") == "john\n" -@pytest.mark.skip(reason="Flaky") def test_logs(): logs = query_and_get_logs("SELECT 1", settings={'send_logs_level':'debug'}) assert "SELECT 1" in logs assert "Read 1 rows" in logs assert "Peak memory usage" in logs -@pytest.mark.skip(reason="Flaky") def test_progress(): results = query_no_errors("SELECT number, sleep(0.31) FROM numbers(8) SETTINGS max_block_size=2, interactive_delay=100000", stream_output=True) #print(results) @@ -259,7 +246,6 @@ def test_progress(): } ]""" -@pytest.mark.skip(reason="Flaky") def test_session(): session_a = "session A" session_b = "session B" @@ -270,12 +256,10 @@ def test_session(): assert query("SELECT getSetting('custom_x'), getSetting('custom_y')", session_id=session_a) == "1\t2\n" assert query("SELECT getSetting('custom_x'), getSetting('custom_y')", session_id=session_b) == "3\t4\n" -@pytest.mark.skip(reason="Flaky") def test_no_session(): e = query_and_get_error("SET custom_x=1") assert "There is no session" in e.display_text -@pytest.mark.skip(reason="Flaky") def test_input_function(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") query("INSERT INTO t SELECT col1 * col2 FROM input('col1 UInt8, col2 UInt8') FORMAT CSV", input_data=["5,4\n", "8,11\n", "10,12\n"]) @@ -285,7 +269,6 @@ def test_input_function(): query("INSERT INTO t SELECT col1 * col2 FROM input('col1 UInt8, col2 UInt8') FORMAT CSV 20,10\n", input_data="15,15\n") assert query("SELECT a FROM t ORDER BY a") == "20\n88\n120\n143\n200\n225\n" -@pytest.mark.skip(reason="Flaky") def test_external_table(): columns = [clickhouse_grpc_pb2.NameAndType(name='UserID', type='UInt64'), clickhouse_grpc_pb2.NameAndType(name='UserName', type='String')] ext1 = clickhouse_grpc_pb2.ExternalTable(name='ext1', columns=columns, data='1\tAlex\n2\tBen\n3\tCarl\n', format='TabSeparated') @@ -303,7 +286,6 @@ def test_external_table(): assert query("SELECT * FROM _data ORDER BY _2", external_tables=[unnamed_table]) == "7\tFred\n"\ "6\tGeorge\n" -@pytest.mark.skip(reason="Flaky") def test_external_table_streaming(): columns = [clickhouse_grpc_pb2.NameAndType(name='UserID', type='UInt64'), clickhouse_grpc_pb2.NameAndType(name='UserName', type='String')] def send_query_info(): @@ -319,7 +301,6 @@ def test_external_table_streaming(): "4\tDaniel\n"\ "5\tEthan\n" -@pytest.mark.skip(reason="Flaky") def test_simultaneous_queries_same_channel(): threads=[] try: @@ -331,7 +312,6 @@ def test_simultaneous_queries_same_channel(): for thread in threads: thread.join() -@pytest.mark.skip(reason="Flaky") def test_simultaneous_queries_multiple_channels(): threads=[] try: @@ -343,7 +323,6 @@ def test_simultaneous_queries_multiple_channels(): for thread in threads: thread.join() -@pytest.mark.skip(reason="Flaky") def test_cancel_while_processing_input(): query("CREATE TABLE t (a UInt8) ENGINE = Memory") def send_query_info(): @@ -356,7 +335,6 @@ def test_cancel_while_processing_input(): assert result.progress.written_rows == 6 assert query("SELECT a FROM t ORDER BY a") == "1\n2\n3\n4\n5\n6\n" -@pytest.mark.skip(reason="Flaky") def test_cancel_while_generating_output(): def send_query_info(): yield clickhouse_grpc_pb2.QueryInfo(query="SELECT number, sleep(0.2) FROM numbers(10) SETTINGS max_block_size=2") diff --git a/tests/integration/test_grpc_protocol_ssl/configs/grpc_config.xml b/tests/integration/test_grpc_protocol_ssl/configs/grpc_config.xml index 396c8f9b779..79cb4fde6fc 100644 --- a/tests/integration/test_grpc_protocol_ssl/configs/grpc_config.xml +++ b/tests/integration/test_grpc_protocol_ssl/configs/grpc_config.xml @@ -1,5 +1,5 @@ - 9100 + 9100 true @@ -20,5 +20,8 @@ high + + + false diff --git a/tests/integration/test_grpc_protocol_ssl/test.py b/tests/integration/test_grpc_protocol_ssl/test.py index f831b934e7c..a311dee8781 100644 --- a/tests/integration/test_grpc_protocol_ssl/test.py +++ b/tests/integration/test_grpc_protocol_ssl/test.py @@ -73,18 +73,15 @@ def start_cluster(): # Actual tests -@pytest.mark.skip(reason="Flaky") def test_secure_channel(): with create_secure_channel() as channel: assert query("SELECT 'ok'", channel) == "ok\n" -@pytest.mark.skip(reason="Flaky") def test_insecure_channel(): with pytest.raises(grpc.FutureTimeoutError): with create_insecure_channel() as channel: query("SELECT 'ok'", channel) -@pytest.mark.skip(reason="Flaky") def test_wrong_client_certificate(): with pytest.raises(grpc.FutureTimeoutError): with create_insecure_channel() as channel: diff --git a/tests/integration/test_log_family_s3/configs/config.xml b/tests/integration/test_log_family_s3/configs/config.xml index 63b4d951eb7..5b9b5b5843a 100644 --- a/tests/integration/test_log_family_s3/configs/config.xml +++ b/tests/integration/test_log_family_s3/configs/config.xml @@ -8,18 +8,6 @@ 10 - - - - s3 - http://minio1:9001/root/data/ - minio - minio123 - - - - - 9000 127.0.0.1 diff --git a/tests/integration/test_log_family_s3/configs/minio.xml b/tests/integration/test_log_family_s3/configs/minio.xml index 6c9329a2bbc..edcf1225142 100644 --- a/tests/integration/test_log_family_s3/configs/minio.xml +++ b/tests/integration/test_log_family_s3/configs/minio.xml @@ -7,6 +7,7 @@ http://minio1:9001/root/data/ minio minio123 + true diff --git a/tests/integration/test_log_family_s3/test.py b/tests/integration/test_log_family_s3/test.py index 40e263c9c69..4ee370a7703 100644 --- a/tests/integration/test_log_family_s3/test.py +++ b/tests/integration/test_log_family_s3/test.py @@ -1,4 +1,5 @@ import logging +import sys import pytest from helpers.cluster import ClickHouseCluster @@ -23,30 +24,38 @@ def cluster(): cluster.shutdown() +def assert_objects_count(cluster, objects_count, path='data/'): + minio = cluster.minio_client + s3_objects = list(minio.list_objects(cluster.minio_bucket, path)) + print(s3_objects, file=sys.stderr) + if objects_count != len(s3_objects): + for s3_object in s3_objects: + object_meta = minio.stat_object(cluster.minio_bucket, s3_object.object_name) + logging.info("Existing S3 object: %s", str(object_meta)) + assert objects_count == len(s3_objects) + + @pytest.mark.parametrize( "log_engine,files_overhead,files_overhead_per_insert", [("TinyLog", 1, 1), ("Log", 2, 1), ("StripeLog", 1, 2)]) def test_log_family_s3(cluster, log_engine, files_overhead, files_overhead_per_insert): node = cluster.instances["node"] - minio = cluster.minio_client node.query("CREATE TABLE s3_test (id UInt64) Engine={}".format(log_engine)) node.query("INSERT INTO s3_test SELECT number FROM numbers(5)") assert node.query("SELECT * FROM s3_test") == "0\n1\n2\n3\n4\n" - assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert + files_overhead + assert_objects_count(cluster, files_overhead_per_insert + files_overhead) node.query("INSERT INTO s3_test SELECT number + 5 FROM numbers(3)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n" - assert len( - list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 2 + files_overhead + assert_objects_count(cluster, files_overhead_per_insert * 2 + files_overhead) node.query("INSERT INTO s3_test SELECT number + 8 FROM numbers(1)") assert node.query("SELECT * FROM s3_test order by id") == "0\n1\n2\n3\n4\n5\n6\n7\n8\n" - assert len( - list(minio.list_objects(cluster.minio_bucket, 'data/'))) == files_overhead_per_insert * 3 + files_overhead + assert_objects_count(cluster, files_overhead_per_insert * 3 + files_overhead) node.query("TRUNCATE TABLE s3_test") - assert len(list(minio.list_objects(cluster.minio_bucket, 'data/'))) == 0 + assert_objects_count(cluster, 0) node.query("DROP TABLE s3_test") diff --git a/tests/integration/test_testkeeper_back_to_back/__init__.py b/tests/integration/test_testkeeper_back_to_back/__init__.py new file mode 100644 index 00000000000..e5a0d9b4834 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/__init__.py @@ -0,0 +1 @@ +#!/usr/bin/env python3 diff --git a/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml b/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml new file mode 100644 index 00000000000..79e993b41f7 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/configs/enable_test_keeper.xml @@ -0,0 +1,7 @@ + + + 9181 + 10000 + 30000 + + diff --git a/tests/integration/test_testkeeper_back_to_back/test.py b/tests/integration/test_testkeeper_back_to_back/test.py new file mode 100644 index 00000000000..04a2150e7f8 --- /dev/null +++ b/tests/integration/test_testkeeper_back_to_back/test.py @@ -0,0 +1,402 @@ +import pytest +from helpers.cluster import ClickHouseCluster +import random +import string +import os +import time + +cluster = ClickHouseCluster(__file__) +node = cluster.add_instance('node', main_configs=['configs/enable_test_keeper.xml'], with_zookeeper=True) +from kazoo.client import KazooClient + +_genuine_zk_instance = None +_fake_zk_instance = None + +def get_genuine_zk(): + global _genuine_zk_instance + if not _genuine_zk_instance: + print("Zoo1", cluster.get_instance_ip("zoo1")) + _genuine_zk_instance = cluster.get_kazoo_client('zoo1') + return _genuine_zk_instance + + +def get_fake_zk(): + global _fake_zk_instance + if not _fake_zk_instance: + print("node", cluster.get_instance_ip("node")) + _fake_zk_instance = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + _fake_zk_instance.start() + return _fake_zk_instance + +def random_string(length): + return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length)) + +def create_random_path(prefix="", depth=1): + if depth == 0: + return prefix + return create_random_path(os.path.join(prefix, random_string(3)), depth - 1) + +@pytest.fixture(scope="module") +def started_cluster(): + try: + cluster.start() + + yield cluster + + finally: + cluster.shutdown() + if _genuine_zk_instance: + _genuine_zk_instance.stop() + _genuine_zk_instance.close() + if _fake_zk_instance: + _fake_zk_instance.stop() + _fake_zk_instance.close() + + +def test_simple_commands(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + + for zk in [genuine_zk, fake_zk]: + zk.create("/test_simple_commands", b"") + zk.create("/test_simple_commands/somenode1", b"hello") + zk.set("/test_simple_commands/somenode1", b"world") + + for zk in [genuine_zk, fake_zk]: + assert zk.exists("/test_simple_commands") + assert zk.exists("/test_simple_commands/somenode1") + print(zk.get("/test_simple_commands/somenode1")) + assert zk.get("/test_simple_commands/somenode1")[0] == b"world" + + +def test_sequential_nodes(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_sequential_nodes") + fake_zk.create("/test_sequential_nodes") + for i in range(1, 11): + genuine_zk.create("/test_sequential_nodes/" + ("a" * i) + "-", sequence=True) + genuine_zk.create("/test_sequential_nodes/" + ("b" * i)) + fake_zk.create("/test_sequential_nodes/" + ("a" * i) + "-", sequence=True) + fake_zk.create("/test_sequential_nodes/" + ("b" * i)) + + genuine_childs = list(sorted(genuine_zk.get_children("/test_sequential_nodes"))) + fake_childs = list(sorted(fake_zk.get_children("/test_sequential_nodes"))) + assert genuine_childs == fake_childs + + +def assert_eq_stats(stat1, stat2): + assert stat1.version == stat2.version + assert stat1.cversion == stat2.cversion + assert stat1.aversion == stat2.aversion + assert stat1.aversion == stat2.aversion + assert stat1.dataLength == stat2.dataLength + assert stat1.numChildren == stat2.numChildren + +def test_stats(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_stats_nodes") + fake_zk.create("/test_stats_nodes") + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + assert_eq_stats(genuine_stats, fake_stats) + for i in range(1, 11): + genuine_zk.create("/test_stats_nodes/" + ("a" * i) + "-", sequence=True) + genuine_zk.create("/test_stats_nodes/" + ("b" * i)) + fake_zk.create("/test_stats_nodes/" + ("a" * i) + "-", sequence=True) + fake_zk.create("/test_stats_nodes/" + ("b" * i)) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + assert_eq_stats(genuine_stats, fake_stats) + for i in range(1, 11): + print("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + genuine_zk.delete("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + genuine_zk.delete("/test_stats_nodes/" + ("b" * i)) + fake_zk.delete("/test_stats_nodes/" + ("a" * i) + "-" + "{:010d}".format((i - 1) * 2)) + fake_zk.delete("/test_stats_nodes/" + ("b" * i)) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + print(genuine_stats) + print(fake_stats) + assert_eq_stats(genuine_stats, fake_stats) + for i in range(100): + genuine_zk.set("/test_stats_nodes", ("q" * i).encode()) + fake_zk.set("/test_stats_nodes", ("q" * i).encode()) + + genuine_stats = genuine_zk.exists("/test_stats_nodes") + fake_stats = fake_zk.exists("/test_stats_nodes") + print(genuine_stats) + print(fake_stats) + assert_eq_stats(genuine_stats, fake_stats) + +def test_watchers(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + genuine_zk.create("/test_data_watches") + fake_zk.create("/test_data_watches") + genuine_data_watch_data = None + + def genuine_callback(event): + print("Genuine data watch called") + nonlocal genuine_data_watch_data + genuine_data_watch_data = event + + fake_data_watch_data = None + def fake_callback(event): + print("Fake data watch called") + nonlocal fake_data_watch_data + fake_data_watch_data = event + + genuine_zk.get("/test_data_watches", watch=genuine_callback) + fake_zk.get("/test_data_watches", watch=fake_callback) + + print("Calling set genuine") + genuine_zk.set("/test_data_watches", b"a") + print("Calling set fake") + fake_zk.set("/test_data_watches", b"a") + time.sleep(3) + + print("Genuine data", genuine_data_watch_data) + print("Fake data", fake_data_watch_data) + assert genuine_data_watch_data == fake_data_watch_data + + genuine_children = None + def genuine_child_callback(event): + print("Genuine child watch called") + nonlocal genuine_children + genuine_children = event + + fake_children = None + def fake_child_callback(event): + print("Fake child watch called") + nonlocal fake_children + fake_children = event + + genuine_zk.get_children("/test_data_watches", watch=genuine_child_callback) + fake_zk.get_children("/test_data_watches", watch=fake_child_callback) + + print("Calling genuine child") + genuine_zk.create("/test_data_watches/child", b"b") + print("Calling fake child") + fake_zk.create("/test_data_watches/child", b"b") + + time.sleep(3) + + print("Genuine children", genuine_children) + print("Fake children", fake_children) + assert genuine_children == fake_children + +def test_multitransactions(started_cluster): + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + for zk in [genuine_zk, fake_zk]: + zk.create('/test_multitransactions') + t = zk.transaction() + t.create('/test_multitransactions/freddy') + t.create('/test_multitransactions/fred', ephemeral=True) + t.create('/test_multitransactions/smith', sequence=True) + results = t.commit() + assert len(results) == 3 + assert results[0] == '/test_multitransactions/freddy' + assert results[2].startswith('/test_multitransactions/smith0') is True + + from kazoo.exceptions import RolledBackError, NoNodeError + for i, zk in enumerate([genuine_zk, fake_zk]): + print("Processing ZK", i) + t = zk.transaction() + t.create('/test_multitransactions/q') + t.delete('/test_multitransactions/a') + t.create('/test_multitransactions/x') + results = t.commit() + print("Results", results) + assert results[0].__class__ == RolledBackError + assert results[1].__class__ == NoNodeError + assert zk.exists('/test_multitransactions/q') is None + assert zk.exists('/test_multitransactions/a') is None + assert zk.exists('/test_multitransactions/x') is None + + +def exists(zk, path): + result = zk.exists(path) + return result is not None + +def get(zk, path): + result = zk.get(path) + return result[0] + +def get_children(zk, path): + return [elem for elem in list(sorted(zk.get_children(path))) if elem not in ('clickhouse', 'zookeeper')] + +READ_REQUESTS = [ + ("exists", exists), + ("get", get), + ("get_children", get_children), +] + + +def create(zk, path, data): + zk.create(path, data.encode()) + + +def set_data(zk, path, data): + zk.set(path, data.encode()) + + +WRITE_REQUESTS = [ + ("create", create), + ("set_data", set_data), +] + + +def delete(zk, path): + zk.delete(path) + +DELETE_REQUESTS = [ + ("delete", delete) +] + + +class Request(object): + def __init__(self, name, arguments, callback, is_return): + self.name = name + self.arguments = arguments + self.callback = callback + self.is_return = is_return + + def __str__(self): + arg_str = ', '.join([str(k) + "=" + str(v) for k, v in self.arguments.items()]) + return "ZKRequest name {} with arguments {}".format(self.name, arg_str) + +def generate_requests(iters=1): + requests = [] + existing_paths = [] + for i in range(iters): + for _ in range(100): + rand_length = random.randint(0, 10) + path = "/" + for j in range(1, rand_length): + path = create_random_path(path, 1) + existing_paths.append(path) + value = random_string(1000) + request = Request("create", {"path" : path, "value": value[0:10]}, lambda zk, path=path, value=value: create(zk, path, value), False) + requests.append(request) + + for _ in range(100): + path = random.choice(existing_paths) + value = random_string(100) + request = Request("set", {"path": path, "value": value[0:10]}, lambda zk, path=path, value=value: set_data(zk, path, value), False) + requests.append(request) + + for _ in range(100): + path = random.choice(existing_paths) + callback = random.choice(READ_REQUESTS) + def read_func1(zk, path=path, callback=callback): + return callback[1](zk, path) + + request = Request(callback[0], {"path": path}, read_func1, True) + requests.append(request) + + for _ in range(30): + path = random.choice(existing_paths) + request = Request("delete", {"path": path}, lambda zk, path=path: delete(zk, path), False) + + for _ in range(100): + path = random.choice(existing_paths) + callback = random.choice(READ_REQUESTS) + def read_func2(zk, path=path, callback=callback): + return callback[1](zk, path) + request = Request(callback[0], {"path": path}, read_func2, True) + requests.append(request) + return requests + + +def test_random_requests(started_cluster): + requests = generate_requests(10) + genuine_zk = get_genuine_zk() + fake_zk = get_fake_zk() + for i, request in enumerate(requests): + genuine_throw = False + fake_throw = False + fake_result = None + genuine_result = None + try: + genuine_result = request.callback(genuine_zk) + except Exception as ex: + genuine_throw = True + + try: + fake_result = request.callback(fake_zk) + except Exception as ex: + fake_throw = True + + assert fake_throw == genuine_throw, "Fake throw genuine not or vise versa" + assert fake_result == genuine_result, "Zookeeper results differ" + root_children_genuine = [elem for elem in list(sorted(genuine_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] + root_children_fake = [elem for elem in list(sorted(fake_zk.get_children("/"))) if elem not in ('clickhouse', 'zookeeper')] + assert root_children_fake == root_children_genuine + +def test_end_of_session(started_cluster): + fake_zk1 = None + fake_zk2 = None + genuine_zk1 = None + genuine_zk2 = None + + try: + fake_zk1 = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + fake_zk1.start() + fake_zk2 = KazooClient(hosts=cluster.get_instance_ip("node") + ":9181") + fake_zk2.start() + genuine_zk1 = cluster.get_kazoo_client('zoo1') + genuine_zk1.start() + genuine_zk2 = cluster.get_kazoo_client('zoo1') + genuine_zk2.start() + + fake_zk1.create("/test_end_of_session") + genuine_zk1.create("/test_end_of_session") + fake_ephemeral_event = None + def fake_ephemeral_callback(event): + print("Fake watch triggered") + nonlocal fake_ephemeral_event + fake_ephemeral_event = event + + genuine_ephemeral_event = None + def genuine_ephemeral_callback(event): + print("Genuine watch triggered") + nonlocal genuine_ephemeral_event + genuine_ephemeral_event = event + + assert fake_zk2.exists("/test_end_of_session") is not None + assert genuine_zk2.exists("/test_end_of_session") is not None + + fake_zk1.create("/test_end_of_session/ephemeral_node", ephemeral=True) + genuine_zk1.create("/test_end_of_session/ephemeral_node", ephemeral=True) + + assert fake_zk2.exists("/test_end_of_session/ephemeral_node", watch=fake_ephemeral_callback) is not None + assert genuine_zk2.exists("/test_end_of_session/ephemeral_node", watch=genuine_ephemeral_callback) is not None + + print("Stopping genuine zk") + genuine_zk1.stop() + print("Closing genuine zk") + genuine_zk1.close() + + print("Stopping fake zk") + fake_zk1.stop() + print("Closing fake zk") + fake_zk1.close() + + assert fake_zk2.exists("/test_end_of_session/ephemeral_node") is None + assert genuine_zk2.exists("/test_end_of_session/ephemeral_node") is None + + assert fake_ephemeral_event == genuine_ephemeral_event + + finally: + try: + for zk in [fake_zk1, fake_zk2, genuine_zk1, genuine_zk2]: + if zk: + zk.stop() + zk.close() + except: + pass diff --git a/tests/performance/first_significant_subdomain.xml b/tests/performance/first_significant_subdomain.xml deleted file mode 100644 index b8418401986..00000000000 --- a/tests/performance/first_significant_subdomain.xml +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - test.hits - - - - 1 - - - SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomain(URL)) - diff --git a/tests/performance/url_hits.xml b/tests/performance/url_hits.xml index c8cf119a7d7..072fb5b94e7 100644 --- a/tests/performance/url_hits.xml +++ b/tests/performance/url_hits.xml @@ -1,11 +1,10 @@ - hits_100m_single + test.hits - func @@ -32,6 +31,12 @@ - SELECT count() FROM hits_100m_single WHERE NOT ignore({func}(URL)) + + + SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomain(URL)) SETTINGS max_threads=1 + SELECT count() FROM test.hits WHERE NOT ignore(firstSignificantSubdomainCustom(URL, 'public_suffix_list')) SETTINGS max_threads=1 + + SELECT count() FROM test.hits WHERE NOT ignore(cutToFirstSignificantSubdomain(URL)) SETTINGS max_threads=1 + SELECT count() FROM test.hits WHERE NOT ignore(cutToFirstSignificantSubdomainCustom(URL, 'public_suffix_list')) SETTINGS max_threads=1 diff --git a/tests/queries/0_stateless/00955_test_final_mark.sql b/tests/queries/0_stateless/00955_test_final_mark.sql index 929ac931a94..50ca3d008f9 100644 --- a/tests/queries/0_stateless/00955_test_final_mark.sql +++ b/tests/queries/0_stateless/00955_test_final_mark.sql @@ -119,7 +119,7 @@ INSERT INTO mt_without_pk (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-1 SELECT COUNT(*) FROM mt_without_pk WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1; INSERT INTO mt_without_pk (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-10-01'), toDateTime('2018-10-01 07:57:57'), [4, 4, 4], 14, [111, 222], ['Lui', 'Dave']), (toDate('2018-10-01'), toDateTime('2018-10-01 08:57:57'), [5, 5, 5], 15, [333, 444], ['John', 'Mike']), (toDate('2018-10-01'), toDateTime('2018-10-01 09:57:57'), [6, 6, 6], 16, [555, 666, 777], ['Alex', 'Jim', 'Tom']); @@ -127,7 +127,7 @@ OPTIMIZE TABLE mt_without_pk FINAL; SELECT COUNT(*) FROM mt_without_pk WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_without_pk' AND active=1; DROP TABLE IF EXISTS mt_without_pk; @@ -149,7 +149,7 @@ INSERT INTO mt_with_small_granularity (d, x, y, z, `n.Age`, `n.Name`) VALUES (to SELECT COUNT(*) FROM mt_with_small_granularity WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1; INSERT INTO mt_with_small_granularity (d, x, y, z, `n.Age`, `n.Name`) VALUES (toDate('2018-10-01'), toDateTime('2018-10-01 07:57:57'), [4, 4, 4], 14, [111, 222], ['Lui', 'Dave']), (toDate('2018-10-01'), toDateTime('2018-10-01 08:57:57'), [5, 5, 5], 15, [333, 444], ['John', 'Mike']), (toDate('2018-10-01'), toDateTime('2018-10-01 09:57:57'), [6, 6, 6], 16, [555, 666, 777], ['Alex', 'Jim', 'Tom']); @@ -157,6 +157,6 @@ OPTIMIZE TABLE mt_with_small_granularity FINAL; SELECT COUNT(*) FROM mt_with_small_granularity WHERE x > toDateTime('2018-10-01 23:57:57'); -SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1 AND database=currentDatabase(); +SELECT sum(marks) FROM system.parts WHERE table = 'mt_with_small_granularity' AND active=1; DROP TABLE IF EXISTS mt_with_small_granularity; diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index dc972f958a8..5f5a39b906b 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -79,6 +79,7 @@ SYSTEM DROP UNCOMPRESSED CACHE ['SYSTEM DROP UNCOMPRESSED','DROP UNCOMPRESSED CA SYSTEM DROP COMPILED EXPRESSION CACHE ['SYSTEM DROP COMPILED EXPRESSION','DROP COMPILED EXPRESSION CACHE','DROP COMPILED EXPRESSIONS'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP CACHE ['DROP CACHE'] \N SYSTEM SYSTEM RELOAD CONFIG ['RELOAD CONFIG'] GLOBAL SYSTEM RELOAD +SYSTEM RELOAD SYMBOLS ['RELOAD SYMBOLS'] GLOBAL SYSTEM RELOAD SYSTEM RELOAD DICTIONARY ['SYSTEM RELOAD DICTIONARIES','RELOAD DICTIONARY','RELOAD DICTIONARIES'] GLOBAL SYSTEM RELOAD SYSTEM RELOAD EMBEDDED DICTIONARIES ['RELOAD EMBEDDED DICTIONARIES'] GLOBAL SYSTEM RELOAD SYSTEM RELOAD [] \N SYSTEM diff --git a/tests/queries/0_stateless/01523_client_local_queries_file_parameter.reference b/tests/queries/0_stateless/01523_client_local_queries_file_parameter.reference new file mode 100644 index 00000000000..2a3eb745751 --- /dev/null +++ b/tests/queries/0_stateless/01523_client_local_queries_file_parameter.reference @@ -0,0 +1,6 @@ +1 +1 +2 +3 +1 2 +3 4 diff --git a/tests/queries/0_stateless/01523_client_local_queries_file_parameter.sh b/tests/queries/0_stateless/01523_client_local_queries_file_parameter.sh new file mode 100755 index 00000000000..bd8cbc03095 --- /dev/null +++ b/tests/queries/0_stateless/01523_client_local_queries_file_parameter.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CURDIR"/../shell_config.sh + +echo "SELECT 1;" > 01523_client_local_queries_file_parameter_tmp.sql +$CLICKHOUSE_CLIENT --queries-file=01523_client_local_queries_file_parameter_tmp.sql 2>&1 + +echo "CREATE TABLE 01523_test(value Int32) ENGINE=Log; +INSERT INTO 01523_test + VALUES (1), (2), (3); +SELECT * FROM 01523_test; +DROP TABLE 01523_test;" > 01523_client_local_queries_file_parameter_tmp.sql +$CLICKHOUSE_CLIENT --queries-file=01523_client_local_queries_file_parameter_tmp.sql 2>&1 + +echo "CREATE TABLE 01523_test (a Int64, b Int64) ENGINE = File(CSV, stdin); +SELECT a, b FROM 01523_test; +DROP TABLE 01523_test;" > 01523_client_local_queries_file_parameter_tmp.sql + +echo -e "1,2\n3,4" | $CLICKHOUSE_LOCAL --queries-file=01523_client_local_queries_file_parameter_tmp.sql 2>&1 + +rm 01523_client_local_queries_file_parameter_tmp.sql diff --git a/tests/queries/0_stateless/01543_toModifiedJulianDay.reference b/tests/queries/0_stateless/01543_toModifiedJulianDay.reference new file mode 100644 index 00000000000..fbb668d09e5 --- /dev/null +++ b/tests/queries/0_stateless/01543_toModifiedJulianDay.reference @@ -0,0 +1,18 @@ +Invocation with constant +-1 +0 +59154 +\N +or null +59154 +\N +\N +\N +Invocation with String column +-1 +0 +59154 +Invocation with FixedString column +-1 +0 +59154 diff --git a/tests/queries/0_stateless/01543_toModifiedJulianDay.sql b/tests/queries/0_stateless/01543_toModifiedJulianDay.sql new file mode 100644 index 00000000000..4cc0813c8cc --- /dev/null +++ b/tests/queries/0_stateless/01543_toModifiedJulianDay.sql @@ -0,0 +1,38 @@ +-- +SELECT 'Invocation with constant'; + +SELECT toModifiedJulianDay('1858-11-16'); +SELECT toModifiedJulianDay('1858-11-17'); +SELECT toModifiedJulianDay('2020-11-01'); +SELECT toModifiedJulianDay(NULL); +SELECT toModifiedJulianDay('unparsable'); -- { serverError 27 } +SELECT toModifiedJulianDay('1999-02-29'); -- { serverError 38 } +SELECT toModifiedJulianDay('1999-13-32'); -- { serverError 38 } + +SELECT 'or null'; +SELECT toModifiedJulianDayOrNull('2020-11-01'); +SELECT toModifiedJulianDayOrNull('unparsable'); +SELECT toModifiedJulianDayOrNull('1999-02-29'); +SELECT toModifiedJulianDayOrNull('1999-13-32'); + +-- +SELECT 'Invocation with String column'; + +DROP TABLE IF EXISTS toModifiedJulianDay_test; +CREATE TABLE toModifiedJulianDay_test (d String) ENGINE = Memory; + +INSERT INTO toModifiedJulianDay_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toModifiedJulianDay(d) FROM toModifiedJulianDay_test; + +DROP TABLE toModifiedJulianDay_test; + +-- +SELECT 'Invocation with FixedString column'; + +DROP TABLE IF EXISTS toModifiedJulianDay_test; +CREATE TABLE toModifiedJulianDay_test (d FixedString(10)) ENGINE = Memory; + +INSERT INTO toModifiedJulianDay_test VALUES ('1858-11-16'), ('1858-11-17'), ('2020-11-01'); +SELECT toModifiedJulianDay(d) FROM toModifiedJulianDay_test; + +DROP TABLE toModifiedJulianDay_test; diff --git a/tests/queries/0_stateless/01544_fromModifiedJulianDay.reference b/tests/queries/0_stateless/01544_fromModifiedJulianDay.reference new file mode 100644 index 00000000000..6f71b6263c0 --- /dev/null +++ b/tests/queries/0_stateless/01544_fromModifiedJulianDay.reference @@ -0,0 +1,14 @@ +Invocation with constant +1858-11-16 +1858-11-17 +2020-11-01 +\N +or null +2020-11-01 +\N +\N +\N +Invocation with Int32 column +1858-11-16 +1858-11-17 +2020-11-01 diff --git a/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql b/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql new file mode 100644 index 00000000000..4e50351d191 --- /dev/null +++ b/tests/queries/0_stateless/01544_fromModifiedJulianDay.sql @@ -0,0 +1,26 @@ +-- +SELECT 'Invocation with constant'; + +SELECT fromModifiedJulianDay(-1); +SELECT fromModifiedJulianDay(0); +SELECT fromModifiedJulianDay(59154); +SELECT fromModifiedJulianDay(NULL); +SELECT fromModifiedJulianDay(-678942); -- { serverError 490 } +SELECT fromModifiedJulianDay(2973484); -- { serverError 490 } + +SELECT 'or null'; +SELECT fromModifiedJulianDayOrNull(59154); +SELECT fromModifiedJulianDayOrNull(NULL); +SELECT fromModifiedJulianDayOrNull(-678942); +SELECT fromModifiedJulianDayOrNull(2973484); + +-- +SELECT 'Invocation with Int32 column'; + +DROP TABLE IF EXISTS fromModifiedJulianDay_test; +CREATE TABLE fromModifiedJulianDay_test (d Int32) ENGINE = Memory; + +INSERT INTO fromModifiedJulianDay_test VALUES (-1), (0), (59154); +SELECT fromModifiedJulianDay(d) FROM fromModifiedJulianDay_test; + +DROP TABLE fromModifiedJulianDay_test; diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference new file mode 100644 index 00000000000..a197ceee71f --- /dev/null +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.reference @@ -0,0 +1,20 @@ +0 +0 +0 +0 +0 +1 +\N +\N +1 +\N +\N +0 +\N +0 +\N +1 +\N +\N +1 +\N diff --git a/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql new file mode 100644 index 00000000000..834204fedb9 --- /dev/null +++ b/tests/queries/0_stateless/01562_agg_null_for_empty_ahead.sql @@ -0,0 +1,36 @@ +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(0)); +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(1)); + +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(0))); +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(1))); + +SELECT sumIf(1, 0); + +SELECT sumIf(1, 1); + +-- should return Null even if we donn't set aggregate_functions_null_for_empty +SELECT sumIfOrNull(1, 0); +SELECT sumOrNullIf(1, 0); + +SELECT nullIf(1, 0); + +SELECT nullIf(1, 1); + +SET aggregate_functions_null_for_empty=1; + +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(0)); +SELECT sumMerge(s) FROM (SELECT sumState(number) s FROM numbers(1)); + +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(0))); +SELECT sumMerge(s) FROM (SELECT sumMergeState(n) s FROM (SELECT sumState(number) n FROM numbers(1))); + +SELECT sumIf(1, 0); + +SELECT sumIf(1, 1); + +SELECT sumIfOrNull(1, 0); +SELECT sumOrNullIf(1, 0); + +SELECT nullIf(1, 0); + +SELECT nullIf(1, 1); diff --git a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference b/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference deleted file mode 100644 index 81c7e6e4df0..00000000000 --- a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.reference +++ /dev/null @@ -1,2 +0,0 @@ -849 -102400 diff --git a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql b/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql deleted file mode 100644 index a73045f5a6f..00000000000 --- a/tests/queries/0_stateless/01577_adaptive_granularity_block_borders.sql +++ /dev/null @@ -1,28 +0,0 @@ -DROP TABLE IF EXISTS adaptive_table; - ---- If granularity of consequent blocks differs a lot, then adaptive ---- granularity will adjust amout of marks correctly. Data for test empirically ---- derived, it's quite hard to get good parameters. - -CREATE TABLE adaptive_table( - key UInt64, - value String -) ENGINE MergeTree() -ORDER BY key -SETTINGS index_granularity_bytes=1048576, min_bytes_for_wide_part = 0, enable_vertical_merge_algorithm = 0; - -SET max_block_size=900; - --- There are about 900 marks for our settings. -INSERT INTO adaptive_table SELECT number, if(number > 700, randomPrintableASCII(102400), randomPrintableASCII(1)) FROM numbers(10000); - -OPTIMIZE TABLE adaptive_table FINAL; - -SELECT marks FROM system.parts WHERE table = 'adaptive_table' and database=currentDatabase() and active; - --- If we have computed granularity incorrectly than we will exceed this limit. -SET max_memory_usage='30M'; - -SELECT max(length(value)) FROM adaptive_table; - -DROP TABLE IF EXISTS adaptive_table; diff --git a/tests/queries/0_stateless/01600_min_max_compress_block_size.reference b/tests/queries/0_stateless/01600_min_max_compress_block_size.reference new file mode 100644 index 00000000000..83b33d238da --- /dev/null +++ b/tests/queries/0_stateless/01600_min_max_compress_block_size.reference @@ -0,0 +1 @@ +1000 diff --git a/tests/queries/0_stateless/01600_min_max_compress_block_size.sql b/tests/queries/0_stateless/01600_min_max_compress_block_size.sql new file mode 100644 index 00000000000..747f0b736ce --- /dev/null +++ b/tests/queries/0_stateless/01600_min_max_compress_block_size.sql @@ -0,0 +1,9 @@ +DROP TABLE IF EXISTS ms; + +CREATE TABLE ms (n Int32) ENGINE = MergeTree() ORDER BY n SETTINGS min_compress_block_size = 1024, max_compress_block_size = 10240; + +INSERT INTO ms SELECT * FROM numbers(1000); + +SELECT COUNT(*) FROM ms; + +DROP TABLE ms; diff --git a/tests/queries/0_stateless/01600_quota_by_forwarded_ip.reference b/tests/queries/0_stateless/01600_quota_by_forwarded_ip.reference new file mode 100644 index 00000000000..6f4fa48a62f --- /dev/null +++ b/tests/queries/0_stateless/01600_quota_by_forwarded_ip.reference @@ -0,0 +1,11 @@ +--- Test with quota by immediate IP --- +10 +exceeded +exceeded +--- Test with quota by forwarded IP --- +10 +exceeded +10 +exceeded +exceeded +10 diff --git a/tests/queries/0_stateless/01600_quota_by_forwarded_ip.sh b/tests/queries/0_stateless/01600_quota_by_forwarded_ip.sh new file mode 100755 index 00000000000..33f5434dc6c --- /dev/null +++ b/tests/queries/0_stateless/01600_quota_by_forwarded_ip.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CURDIR"/../shell_config.sh + + +$CLICKHOUSE_CLIENT -n --query " +DROP USER IF EXISTS quoted_by_ip; +DROP USER IF EXISTS quoted_by_forwarded_ip; + +DROP QUOTA IF EXISTS quota_by_ip; +DROP QUOTA IF EXISTS quota_by_forwarded_ip; + +CREATE USER quoted_by_ip; +CREATE USER quoted_by_forwarded_ip; + +GRANT SELECT, CREATE ON *.* TO quoted_by_ip; +GRANT SELECT, CREATE ON *.* TO quoted_by_forwarded_ip; + +CREATE QUOTA quota_by_ip KEYED BY ip_address FOR RANDOMIZED INTERVAL 1 YEAR MAX QUERIES = 1 TO quoted_by_ip; +CREATE QUOTA quota_by_forwarded_ip KEYED BY forwarded_ip_address FOR RANDOMIZED INTERVAL 1 YEAR MAX QUERIES = 1 TO quoted_by_forwarded_ip; +" + +# Note: the test can be flaky if the randomized interval will end while the loop is run. But with year long interval it's unlikely. +# One query is allowed per quota. Actually two queries will execute successfully due to some implementation specific behaviour. + +echo '--- Test with quota by immediate IP ---' + +while true; do + $CLICKHOUSE_CLIENT --user quoted_by_ip --query "SELECT count() FROM numbers(10)" 2>/dev/null || break +done | uniq + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&user=quoted_by_ip" -d "SELECT count() FROM numbers(10)" | grep -oF 'exceeded' + +# X-Forwarded-For is ignored for quota by immediate IP address +${CLICKHOUSE_CURL} -H 'X-Forwarded-For: 1.2.3.4' -sS "${CLICKHOUSE_URL}&user=quoted_by_ip" -d "SELECT count() FROM numbers(10)" | grep -oF 'exceeded' + + +echo '--- Test with quota by forwarded IP ---' + +while true; do + $CLICKHOUSE_CLIENT --user quoted_by_forwarded_ip --query "SELECT count() FROM numbers(10)" 2>/dev/null || break +done | uniq + +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&user=quoted_by_forwarded_ip" -d "SELECT count() FROM numbers(10)" | grep -oF 'exceeded' + +# X-Forwarded-For is respected for quota by forwarded IP address +while true; do + ${CLICKHOUSE_CURL} -H 'X-Forwarded-For: 1.2.3.4' -sS "${CLICKHOUSE_URL}&user=quoted_by_forwarded_ip" -d "SELECT count() FROM numbers(10)" | grep -oP '^10$' || break +done | uniq + +${CLICKHOUSE_CURL} -H 'X-Forwarded-For: 1.2.3.4' -sS "${CLICKHOUSE_URL}&user=quoted_by_forwarded_ip" -d "SELECT count() FROM numbers(10)" | grep -oF 'exceeded' + +# Only the last IP address is trusted +${CLICKHOUSE_CURL} -H 'X-Forwarded-For: 5.6.7.8, 1.2.3.4' -sS "${CLICKHOUSE_URL}&user=quoted_by_forwarded_ip" -d "SELECT count() FROM numbers(10)" | grep -oF 'exceeded' + +${CLICKHOUSE_CURL} -H 'X-Forwarded-For: 1.2.3.4, 5.6.7.8' -sS "${CLICKHOUSE_URL}&user=quoted_by_forwarded_ip" -d "SELECT count() FROM numbers(10)" + +$CLICKHOUSE_CLIENT -n --query " +DROP QUOTA IF EXISTS quota_by_ip; +DROP QUOTA IF EXISTS quota_by_forwarded_ip; + +DROP USER IF EXISTS quoted_by_ip; +DROP USER IF EXISTS quoted_by_forwarded_ip; +" diff --git a/tests/queries/0_stateless/01601_custom_tld.reference b/tests/queries/0_stateless/01601_custom_tld.reference new file mode 100644 index 00000000000..98b99778396 --- /dev/null +++ b/tests/queries/0_stateless/01601_custom_tld.reference @@ -0,0 +1,11 @@ +no-tld + +foo.there-is-no-such-domain +foo.there-is-no-such-domain +foo +generic +kernel +kernel.biz.ss +difference +biz.ss +kernel.biz.ss diff --git a/tests/queries/0_stateless/01601_custom_tld.sql b/tests/queries/0_stateless/01601_custom_tld.sql new file mode 100644 index 00000000000..6d68299c07d --- /dev/null +++ b/tests/queries/0_stateless/01601_custom_tld.sql @@ -0,0 +1,16 @@ +select 'no-tld'; +select cutToFirstSignificantSubdomainCustom('there-is-no-such-domain', 'public_suffix_list'); +-- even if there is no TLD, 2-nd level by default anyway +-- FIXME: make this behavior optional (so that TLD for host never changed, either empty or something real) +select cutToFirstSignificantSubdomainCustom('foo.there-is-no-such-domain', 'public_suffix_list'); +select cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); +select firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list'); + +select 'generic'; +select firstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss +select cutToFirstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss + +select 'difference'; +-- biz.ss is not in the default TLD list, hence: +select cutToFirstSignificantSubdomain('foo.kernel.biz.ss'); -- biz.ss +select cutToFirstSignificantSubdomainCustom('foo.kernel.biz.ss', 'public_suffix_list'); -- kernel.biz.ss diff --git a/tests/queries/0_stateless/01601_proxy_protocol.reference b/tests/queries/0_stateless/01601_proxy_protocol.reference new file mode 100644 index 00000000000..a5c19667710 --- /dev/null +++ b/tests/queries/0_stateless/01601_proxy_protocol.reference @@ -0,0 +1 @@ +Hello, world diff --git a/tests/queries/0_stateless/01601_proxy_protocol.sh b/tests/queries/0_stateless/01601_proxy_protocol.sh new file mode 100755 index 00000000000..8a431ba6dae --- /dev/null +++ b/tests/queries/0_stateless/01601_proxy_protocol.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$CURDIR"/../shell_config.sh + +printf "PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n\0\21ClickHouse client\24\r\253\251\3\0\7default\0\4\1\0\1\0\0\t0.0.0.0:0\1\tmilovidov\21milovidov-desktop\vClickHouse \24\r\253\251\3\0\1\0\0\0\2\1\25SELECT 'Hello, world'\2\0\247\203\254l\325\\z|\265\254F\275\333\206\342\24\202\24\0\0\0\n\0\0\0\240\1\0\2\377\377\377\377\0\0\0" | nc "${CLICKHOUSE_HOST}" "${CLICKHOUSE_PORT_TCP_WITH_PROXY}" | head -c150 | grep --text -o -F 'Hello, world' diff --git a/tests/queries/shell_config.sh b/tests/queries/shell_config.sh index 88ff59c5084..9e287740c11 100644 --- a/tests/queries/shell_config.sh +++ b/tests/queries/shell_config.sh @@ -41,6 +41,8 @@ export CLICKHOUSE_PORT_TCP=${CLICKHOUSE_PORT_TCP:=$(${CLICKHOUSE_EXTRACT_CONFIG} export CLICKHOUSE_PORT_TCP=${CLICKHOUSE_PORT_TCP:="9000"} export CLICKHOUSE_PORT_TCP_SECURE=${CLICKHOUSE_PORT_TCP_SECURE:=$(${CLICKHOUSE_EXTRACT_CONFIG} --try --key=tcp_port_secure 2>/dev/null)} 2>/dev/null export CLICKHOUSE_PORT_TCP_SECURE=${CLICKHOUSE_PORT_TCP_SECURE:="9440"} +export CLICKHOUSE_PORT_TCP_WITH_PROXY=${CLICKHOUSE_PORT_TCP_WITH_PROXY:=$(${CLICKHOUSE_EXTRACT_CONFIG} --try --key=tcp_with_proxy_port 2>/dev/null)} 2>/dev/null +export CLICKHOUSE_PORT_TCP_WITH_PROXY=${CLICKHOUSE_PORT_TCP_WITH_PROXY:="9010"} export CLICKHOUSE_PORT_HTTP=${CLICKHOUSE_PORT_HTTP:=$(${CLICKHOUSE_EXTRACT_CONFIG} --key=http_port 2>/dev/null)} export CLICKHOUSE_PORT_HTTP=${CLICKHOUSE_PORT_HTTP:="8123"} export CLICKHOUSE_PORT_HTTPS=${CLICKHOUSE_PORT_HTTPS:=$(${CLICKHOUSE_EXTRACT_CONFIG} --try --key=https_port 2>/dev/null)} 2>/dev/null diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index f76490c11ce..322ad2630d1 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -13,6 +13,7 @@ if (NOT DEFINED ENABLE_UTILS OR ENABLE_UTILS) add_subdirectory (iotest) add_subdirectory (corrector_utf8) add_subdirectory (zookeeper-cli) + add_subdirectory (zookeeper-test) add_subdirectory (zookeeper-dump-tree) add_subdirectory (zookeeper-remove-by-list) add_subdirectory (zookeeper-create-entry-to-download-part) diff --git a/utils/list-versions/version_date.tsv b/utils/list-versions/version_date.tsv index eb28f4b5369..cec9dbf1b08 100644 --- a/utils/list-versions/version_date.tsv +++ b/utils/list-versions/version_date.tsv @@ -1,3 +1,5 @@ +v20.12.3.3-stable 2020-12-09 +v20.12.2.1-stable 2020-12-09 v20.11.5.18-stable 2020-12-06 v20.11.4.13-stable 2020-11-20 v20.11.3.3-stable 2020-11-13 @@ -13,6 +15,7 @@ v20.9.5.5-stable 2020-11-13 v20.9.4.76-stable 2020-10-29 v20.9.3.45-stable 2020-10-09 v20.9.2.20-stable 2020-09-22 +v20.8.9.6-lts 2020-12-10 v20.8.8.2-lts 2020-12-07 v20.8.7.15-lts 2020-11-20 v20.8.6.6-lts 2020-11-13 @@ -23,6 +26,7 @@ v20.8.2.3-stable 2020-09-08 v20.7.4.11-stable 2020-10-09 v20.7.3.7-stable 2020-09-18 v20.7.2.30-stable 2020-08-31 +v20.6.11.1-stable 2020-12-09 v20.6.10.2-stable 2020-11-19 v20.6.9.1-stable 2020-11-10 v20.6.8.5-stable 2020-10-12 diff --git a/utils/zookeeper-test/CMakeLists.txt b/utils/zookeeper-test/CMakeLists.txt new file mode 100644 index 00000000000..aa26c840ba3 --- /dev/null +++ b/utils/zookeeper-test/CMakeLists.txt @@ -0,0 +1,3 @@ +add_executable(zk-test main.cpp) +target_link_libraries(zk-test PRIVATE clickhouse_common_zookeeper) +INSTALL(TARGETS zk-test RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse-utils) diff --git a/utils/zookeeper-test/main.cpp b/utils/zookeeper-test/main.cpp new file mode 100644 index 00000000000..fe7bf93fd15 --- /dev/null +++ b/utils/zookeeper-test/main.cpp @@ -0,0 +1,178 @@ +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +using namespace std; + +/// TODO: Remove ME + +void checkEq(zkutil::ZooKeeper & zk, const std::string & path, const std::string & expected) +{ + auto result = zk.get(path); + if (result != expected) + throw std::runtime_error(fmt::format("Data on path '{}' = '{}' doesn't match expected '{}'", + path, result, expected)); +} + +void checkExists(zkutil::ZooKeeper & zk, const std::string & path) +{ + if (!zk.exists(path)) + throw std::runtime_error(fmt::format("Path '{}' doesn't exists", path)); +} + +void testCreateGetExistsNode(zkutil::ZooKeeper & zk) +{ + zk.create("/data", "test_string", zkutil::CreateMode::Persistent); + zk.create("/data/seq-", "another_string", zkutil::CreateMode::PersistentSequential); + checkEq(zk, "/data", "test_string"); + checkExists(zk, "/data/seq-0000000000"); + checkEq(zk, "/data/seq-0000000000", "another_string"); +} + +void testCreateSetNode(zkutil::ZooKeeper & zk) +{ + zk.create("/data/set", "sssss", zkutil::CreateMode::Persistent); + checkEq(zk, "/data/set", "sssss"); + zk.set("/data/set", "qqqqq"); + checkEq(zk, "/data/set", "qqqqq"); +} + +void testCreateList(zkutil::ZooKeeper & zk) +{ + zk.create("/data/lst", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d1", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d2", "", zkutil::CreateMode::Persistent); + zk.create("/data/lst/d3", "", zkutil::CreateMode::Persistent); + auto children = zk.getChildren("/data/lst"); + if (children.size() != 3) + throw std::runtime_error("Children of /data/lst doesn't equal to three"); + for (size_t i = 0; i < children.size(); ++i) + { + if (children[i] != "d" + std::to_string(i + 1)) + throw std::runtime_error(fmt::format("Incorrect children #{} got {}, expected {}", i, children[i], "d" + std::to_string(i + 1))); + } +} + +void testCreateSetVersionRequest(zkutil::ZooKeeper & zk) +{ + zk.create("/data/check_data", "d", zkutil::CreateMode::Persistent); + Coordination::Stat stat; + std::string result = zk.get("/data/check_data", &stat); + try + { + zk.set("/data/check_data", "e", stat.version + 2); + std::terminate(); + } + catch (...) + { + std::cerr << "Got exception on incorrect version (it's ok)\n"; + } + + checkEq(zk, "/data/check_data", "d"); + zk.set("/data/check_data", "e", stat.version); + + checkEq(zk, "/data/check_data", "e"); +} + +void testCreateSetWatchEvent(zkutil::ZooKeeper & zk) +{ + + std::shared_ptr event = std::make_shared(); + zk.create("/data/nodeforwatch", "", zkutil::CreateMode::Persistent); + Coordination::Stat stat; + zk.get("/data/nodeforwatch", &stat, event); + + if (event->tryWait(300)) + throw std::runtime_error(fmt::format("Event for path {} was set without any actions", "/data/nodeforwatch")); + + zk.set("/data/nodeforwatch", "x"); + if (!event->tryWait(300)) + throw std::runtime_error(fmt::format("Event for path {} was not set after set", "/data/nodeforwatch")); + else + std::cerr << "Event was set well\n"; +} + +void testCreateListWatchEvent(zkutil::ZooKeeper & zk) +{ + std::shared_ptr event = std::make_shared(); + std::string path = "/data/pathforwatch"; + zk.create(path, "", zkutil::CreateMode::Persistent); + zk.create(path + "/n1", "", zkutil::CreateMode::Persistent); + zk.create(path + "/n2", "", zkutil::CreateMode::Persistent); + zk.getChildren(path, nullptr, event); + + if (event->tryWait(300)) + throw std::runtime_error(fmt::format("ListEvent for path {} was set without any actions", path)); + + zk.create(path + "/n3", "", zkutil::CreateMode::Persistent); + if (!event->tryWait(300)) + throw std::runtime_error(fmt::format("ListEvent for path {} was not set after create", path)); + else + std::cerr << "ListEvent was set well\n"; +} + +void testMultiRequest(zkutil::ZooKeeper & zk) +{ + Coordination::Requests requests; + requests.push_back(zkutil::makeCreateRequest("/data/multirequest", "aaa", zkutil::CreateMode::Persistent)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "bbb", -1)); + zk.multi(requests); + + try + { + requests.clear(); + requests.push_back(zkutil::makeCreateRequest("/data/multirequest", "qweqwe", zkutil::CreateMode::Persistent)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "bbb", -1)); + requests.push_back(zkutil::makeSetRequest("/data/multirequest", "ccc", -1)); + zk.multi(requests); + std::terminate(); + } + catch (...) + { + std::cerr << "Got exception on multy request (it's ok)\n"; + } + + checkEq(zk, "/data/multirequest", "bbb"); +} + +int main(int argc, char *argv[]) +{ + + if (argc != 2) + { + std::cerr << "usage: " << argv[0] << " hosts" << std::endl; + return 2; + } + Poco::AutoPtr channel = new Poco::ConsoleChannel(std::cerr); + Poco::Logger::root().setChannel(channel); + Poco::Logger::root().setLevel("trace"); + + zkutil::ZooKeeper zk(argv[1]); + + try + { + zk.tryRemoveRecursive("/data"); + testCreateGetExistsNode(zk); + testCreateSetNode(zk); + testCreateList(zk); + testCreateSetVersionRequest(zk); + testMultiRequest(zk); + testCreateSetWatchEvent(zk); + testCreateListWatchEvent(zk); + } + catch (...) + { + zk.tryRemoveRecursive("/data"); + throw; + } + return 0; +} diff --git a/website/blog/en/2020/the-clickhouse-community.md b/website/blog/en/2020/the-clickhouse-community.md new file mode 100644 index 00000000000..7080fed6479 --- /dev/null +++ b/website/blog/en/2020/the-clickhouse-community.md @@ -0,0 +1,147 @@ +--- +title: 'The ClickHouse Community' +image: 'https://blog-images.clickhouse.tech/en/2020/the-clickhouse-community/clickhouse-community-history.png' +date: '2020-12-10' +author: '[Robert Hodges](https://github.com/hodgesrm)' +tags: ['community', 'open source', 'telegram', 'meetup'] +--- + +One of the great “features” of ClickHouse is a friendly and welcoming community. In this article we would like to outline how the ClickHouse community arose, what it is today, and how you can get involved. There is a role for everyone, from end users to contributors to corporate friends. Our goal is to make the community welcoming to every person who wants to join. + +But first, let’s review a bit of history, starting with how ClickHouse first developed at [Yandex](https://yandex.com/company/). + +## Origins at Yandex + +ClickHouse began as a solution for web analytics in [Yandex Metrica](https://metrica.yandex.com/about?). Metrica is a popular service for analyzing website traffic that is now #2 in the market behind Google Analytics. In 2008 [Alexey Milovidov](https://github.com/alexey-milovidov), an engineer on the Metrica team, was looking for a database that could create reports on metrics like number of page views per day, unique visitors, and bounce rate, without aggregating the data in advance. The idea was to provide a wide range of metric data and let users ask any question about them. + +This is a classic problem for data warehouses. However, Alexey could not find one that met Yandex requirements, specifically large datasets, linear scaling, high efficiency, and compatibility with SQL tools. In a nutshell: like MySQL but for analytic applications. So Alexey wrote one. It started as a prototype to do GROUP BY operations. + +The prototype evolved into a full solution with a name, ClickHouse, short for “Clickstream Data Warehouse”. Alexey added additional features including SQL support and the MergeTree engine. The SQL dialect was superficially similar to MySQL, [which was also used in Metrica](https://clickhouse.tech/blog/en/2016/evolution-of-data-structures-in-yandex-metrica/) but could not handle query workloads without complex pre-aggregation. By 2011 ClickHouse was in production for Metrica. + +Over the next 5 years Alexey and a growing team of developers extended ClickHouse to cover new use cases. By 2016 ClickHouse was a core Metrica backend service. It was also becoming entrenched as a data warehouse within Yandex, extending to use cases like service monitoring, network flow logs, and event management. ClickHouse had evolved from the original one-person project to business critical software with a full team of a dozen engineers led by Alexey. + +By 2016, ClickHouse had an 8 year history and was ready to become a major open source project. Here’s a timeline that tracks major developments as a time series. + + + +## ClickHouse goes open source + +Yandex open sourced ClickHouse under an Apache 2.0 license in 2016. There were numerous reasons for this step. + +* Promote adoption within Yandex by making it easier for internal departments to get builds. +* Ensure that ClickHouse would continue to evolve by creating a community to nurture it. +* Motivate developers to contribute to and use ClickHouse due to the open source “cool” factor. +* Improve ClickHouse quality by making the code public. Nobody wants their name visible on bad code. ;-) +* Showcase Yandex innovation to a worldwide audience. + +Alexey and the development team moved ClickHouse code to a Github repo under the Yandex organization and began issuing community builds as well as accepting external contributions. They simultaneously began regular meetups to popularize ClickHouse and build a community around it. The result was a burst of adoption across multiple regions of the globe. + +ClickHouse quickly picked up steam in Eastern Europe. The first ClickHouse meetups started in 2016 and have grown to include 200 participants for in-person meetings and up to 400 for online meetings. ClickHouse is now widely used in start-ups in Russia as well as other Eastern European countries. Developers located in Eastern Europe continue to supply more contributions to ClickHouse than any other region. + +ClickHouse also started to gain recognition in the US and Western Europe. [CloudFlare](https://www.cloudflare.com/) published a widely read blog article about [their success using ClickHouse for DNS analytics](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/). Alexander Zaitsev successfully migrated an ad tech analytics system from a commercial DBMS to a ClickHouse cluster. This success prompted him to found [Altinity](https://altinity.com) in 2017 with help from friends at [Percona](https://www.percona.com). US meetups started in the same year. With support from Altinity these have grown to over 100 attendees for online meetings. + +ClickHouse also took off in China. The first meetup in China took place in 2018 and attracted enormous interest. In-person meetups included over 400 participants. Online meetings have reached up to 1000 online viewers. + +In 2019 a further step occurred as ClickHouse moved out from under the Yandex Github organization into a separate [ClickHouse organization](https://github.com/ClickHouse). The new organization includes ClickHouse server code plus core ecosystem projects like the cpp and ODBC drivers. + +ClickHouse community events shifted online following world-wide disruptions due to COVID-19, but growth in usage continued. One interesting development has been the increasing number of startups using ClickHouse as a backend. Many of these are listed on the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page. Also, additional prominent companies like eBay, Uber, and Flipcart went public in 2020 with stories of successful ClickHouse usage. + +## The ClickHouse community today + +As of 2020 the ClickHouse community includes developers and users from virtually every region of the globe. Yandex engineers continue to supply a majority of pull requests to ClickHouse itself. Altinity follows in second place with contributions to ClickHouse core and ecosystem projects. There is also substantial in-house development on ClickHouse (e.g. on private forks) within Chinese internet providers. + +The real success, however, has been the huge number of commits to ClickHouse core from people in outside organizations. The following list shows the main outside contributors: + +* Azat Khuzhin +* Amos Bird +* Winter Zhang +* Denny Crane +* Danila Kutenin +* Hczhcz +* Marek Vavruša +* Guillaume Tassery +* Sundy Li +* Mikhail Shiryaev +* Nicolae Vartolomei +* Igor Hatarist +* Andrew Onyshchuk +* BohuTANG +* Yu Zhi Chang +* Kirill Shvakov +* Alexander Krasheninnikov +* Simon Podlipsky +* Silviu Caragea +* Flynn ucasFL +* [And over 550 more...](https://github.com/ClickHouse/ClickHouse/graphs/contributors) + +ClickHouse ecosystem projects are also growing rapidly. Here is a selected list of active Github projects that help enable ClickHouse applications, sorted by number of stars. + +* [sqlpad/sqlpad](https://github.com/sqlpad/sqlpad) — Web-based SQL editor that supports ClickHouse +* [mindsdb/mindsdb](https://github.com/mindsdb/mindsdb) — Predictive AI layer for databases with ClickHouse support +* [x-ream/sqli](https://github.com/x-ream/sqli) — ORM SQL interface +* [tricksterproxy/trickster](https://github.com/tricksterproxy/trickster) — HTTP reverse proxy cache and time series dashboard accelerator +* [ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go) — Golang driver for ClickHouse +* [gohouse/gorose](https://github.com/gohouse/gorose) — A mini database ORM for Golang +* [ClickHouse/clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) — JDBC driver for ClickHouse +* [brockercap/Bifrost](https://github.com/brokercap/Bifrost) — Middleware to sync MySQL binlog to ClickHouse +* [mymarilyn/clickhouse-driver](https://github.com/mymarilyn/clickhouse-driver) — ClickHouse Python driver with native interface support +* [Vertamedia/clickhouse-grafana](https://github.com/Vertamedia/clickhouse-grafana) — Grafana datasource for ClickHouse +* [smi2/phpClickHouse](https://github.com/smi2/phpClickHouse) — PHP ClickHouse client +* [Altinity/clickhouse-operator](https://github.com/Altinity/clickhouse-operator) — Kubernetes operator for ClickHouse +* [AlexAkulov/clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) — ClickHouse backup and restore using cloud storage +* [And almost 1200 more...](https://github.com/search?o=desc&p=1&q=clickhouse&s=stars&type=Repositories) + +## Resources + +With the community growth numerous resources are available to users. At the center is the [ClickHouse org on Github](https://github.com/ClickHouse), which hosts [ClickHouse server code](https://github.com/ClickHouse/ClickHouse). ClickHouse server documentation is available at the [clickhouse.tech](https://clickhouse.tech/) website. It has [installation instructions](https://clickhouse.tech/docs/en/getting-started/install/) and links to ClickHouse community builds for major Linux distributions as well as Mac, FreeBSD, and Docker. + +In addition, ClickHouse users have a wide range of ways to engage with the community and get help on applications. These include both chat applications as well as meetups. Here are some links to get started. + +* Yandex Meetups — Yandex has regular in-person and online international and Russian-language meetups. Video recordings and online translations are available at the official [YouTube channel](https://www.youtube.com/c/ClickHouseDB/videos). Watch for announcements on the [clickhouse.tech](https://clickhouse.tech/) site and [Telegram](https://t.me/clickhouse_ru). +* [SF Bay Area ClickHouse Meetup](https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/) — The largest US ClickHouse meetup, with meetings approximately every 2 months. +* Chinese meetups occur at regular intervals with different sponsors. Watch for announcements on clickhouse.tech. +* Telegram - By far the largest forum for ClickHouse. It is the best place to talk to ClickHouse devs. There are two groups. +* [ClickHouse не тормозит](https://t.me/clickhouse_ru) (“ClickHouse does not slow down”) - Russian language Telegram group with 4,629 members currently. +* [ClickHouse](https://t.me/clickhouse_en) — English language group with 1,286 members. +* [ClickHouse Community Slack Channel](http://clickhousedb.slack.com) — Public channel for Slack users. It currently has 551 members. +* [ClickHouse.com.cn](http://clickhouse.com.cn/) — Chinese language site for ClickHouse-related announcements and questions. +* [Conference Presentations](https://github.com/ClickHouse/clickhouse-presentations) — ClickHouse developers like to talk and do so whenever they can. Many recent presentations are stored in Github. Also, look for ClickHouse presentations at Linux Foundation conferences, Data Con LA, Percona Live, and many other venues where there are presentations about data. +* Technical webinars — Altinity has a large library of technical presentations on ClickHouse and related applications on the [Altinity Youtube channel](https://www.youtube.com/channel/UCE3Y2lDKl_ZfjaCrh62onYA/featured). + +If you know of additional resources please bring them to our attention. + +## How you can contribute to ClickHouse + +We welcome users to join the ClickHouse community in every capacity. There are four main ways to participate. + +### Use ClickHouse and share your experiences + +Start with the documentation. Download ClickHouse and try it out. Join the chat channels. If you encounter bugs, [log issues](https://github.com/ClickHouse/ClickHouse/issues) so we can get them fixed. Also, it’s easy to make contributions to the documentation if you have basic Github and markdown skills. Press the pencil icon on any page of the clickhouse.tech website to edit pages and automatically generate pull requests to merge your changes. + +If your company has deployed ClickHouse and is comfortable talking about it, please don't be shy. Add them to the [ClickHouse Adopters](https://clickhouse.tech/docs/en/introduction/adopters/) page so that others can learn from your experience. + +### Become a ClickHouse developer + +Write code to make ClickHouse better. Here are your choices. + +* ClickHouse server — Start with the [“For Beginners” documentation](https://clickhouse.tech/docs/en/development/developer-instruction/) to learn how to build ClickHouse and submit PRs. Check out the current ClickHouse issues if you are looking for work. PRs that follow the development standards will be merged faster. + +* Ecosystem projects — Most projects in the ClickHouse ecosystem accept PRs. Check with each project for specific practices. + +ClickHouse is also a great target for research problems. Overall the years many dozens of university CS students have worked on ClickHouse features. Alexey Milovidov maintains an especially rich set of [project suggestions for students](https://github.com/ClickHouse/ClickHouse/issues/15065). Join Telegram and ask for help if you are interested. Both Yandex and Altinity also offer internships. + +### Write ClickHouse applications + +ClickHouse enables a host of new applications that depend on low latency access to large datasets. If you write something interesting, blog about it and present at local meetups. Altinity has a program to highlight startups who are developing ClickHouse applications and help with marketing as well as resources for development. Send email to [info@altinity.com](mailto:info@altinity.com) for more information. + +### Become a corporate sponsor + +The ClickHouse community has been assisted by many corporate users who have helped organize meetups, funded development, and guided growth of ClickHouse. Contact community members directly at [clickhouse-feedback@yandex-team.ru](mailto:clickhouse-feedback@yandex-team.ru), [info@altinity.com](mailto:info@altinity.com), or via Telegram to find out more about how to chip in as a corporate sponsor. + +## Where we go from here + +ClickHouse has grown enormously from its origins as a basic prototype in 2008 to the popular SQL data warehouse users see today. Our community is the rock that will enable ClickHouse to become the default data warehouse worldwide. We are working together to create an inclusive environment where everyone feels welcome and has an opportunity to contribute. We welcome you to join! + +This article was written with kind assistance from Alexey Milovidov, Ivan Blinkov, and Alexander Zaitsev. + +_2020-12-11 [Robert Hodges](https://github.com/hodgesrm)_